Getting Leave to Appeal to the Supreme Court: Empirical Insights from Tax Cases

As I have posted before, I’m a fan of using empirical data to inform legal practice. Much as the evidence-based medicine movement has taken hold in the field of medicine, the practice of law should, wherever possible, rely on objective data to inform our decision-making. Unfortunately, empirical legal work remains at an early stage. While there are some academics embracing empirical analysis, much of legal academic scholarship does not involve the quantitative analysis of empirical data. In my view, it is important for lawyers to test our intuitions whenever we can by reference to whatever data is available.

For this reason, I was delighted to hear that Professor Kim Brooks, a tax law professor at Dalhousie University and Dean of the Management Faculty there, had previously assembled a dataset relating to tax cases in which leave to appeal was sought to the Supreme Court of Canada. From my correspondence with Professor Brooks, my understanding became that she had collected this dataset, but had never published any papers building on it. I asked her if she would be willing to provide me with that dataset so we could do our own analysis and see whether any useful insights could be gleaned. She was very happy to oblige, for which I am grateful.

Before getting into what the data shows, it’s important to describe what the data is and the limits of what inferences can be drawn from it. The data collected by Professor Brooks consists of tax law cases in which leave to appeal was sought to the Supreme Court of Canada between 2000 and 2010. After some data processing and removing some duplicates, the data consisted of 138 cases in which leave to appeal was sought to the Supreme Court of Canada, and in 19 of those cases (approximately 14%), leave to appeal was granted.

Clearly, there are important limitations as to what inferences can be drawn from this data. I do not practice tax law, so the subject matter is different from any area of law in which I would be interested. And the data is between 10 and 20 years old. Finally, the dataset is relatively small and unbalanced, which can pose technical challenges for statistical inference.

Notwithstanding those limitations, my view is that there is significant value in analyzing Professor Brooks’ data. While the data is older and comes from one particular type of case, the data can still be probative of current practice, assuming (reasonably) some degree of similarity in approaches over time and between subject matters. More generally, the reality of empirical research is that no single study will ever be the definitive study on a particular issue. Findings in one study that might have more limited relevance can still be a useful and interesting data point that can both inform further research efforts, and, in the absence of any other information, should inform one’s thinking about the state of the world.

With those caveats in mind, our analysis of Professor Brooks’ data did in fact show that there were a number of cases featured that correlated with the likelihood of being granted leave. Although we looked at the data using a number of different models, the model I have chosen to present here is a regression tree model. A regression tree model effectively tries to partition the data into discrete buckets of data, each of which have similar characteristics. While decision trees have a number of limitations, a significant benefit is that they are easily interpretable. In this case, by looking at the decision tree that fit best to Professor Brooks’ data, anyone can easily understand the implications of that model for their own purposes.

Here’s the decision tree that our analysis generated:

The decision tree model we calculated showed that the first branch, which is the most important predictor of a case being granted leave or not, was the length of the Court of Appeal’s decision. Among those decisions that were 59 paragraphs or more, there was a 42% chance of the Supreme Court granting leave to those cases. By contrast, of those decisions that were 58 paragraphs or less, there was only a 9% chance of getting leave to appeal.

This is a reasonable result. It’s reasonable to assume that the Supreme Court of Canada is more likely to grant leave where a case is legally contentious, raises difficult issues, or raises broad issues of public policy. It’s also reasonable to assume that those same factors would require the Court of Appeal to write relatively longer decisions. Consequently, it’s not surprising to see a relationship between the length of the Court of Appeal’s decision and the likelihood of leaving being granted.

Second, of those cases where the Court of Appeal’s decision was 58 paragraphs or less, the decision tree model showed that a loss by the government at the Court of Appeal made it more likely that the Supreme Court of Canada would grant leave. Of that subset of cases, the government obtained leave to appeal in 29% of cases where it lost at the Court of Appeal, while the taxpayer obtained leave to appeal in only 7% of those cases. This may suggest either that the Supreme Court is more willing to hear cases that the government seeks leave on, or it may simply reflect that the government has better case selection than private parties in deciding whether a particular case is likely to be granted leave.

Finally, the third variable that played a significant role in the likelihood of leave being granted was the percentage of the applicant’s factum for leave to appeal that was devoted to the question of national importance. (As background, Professor Brooks’ team actually reviewed factums from all the leave applications and made judgments as to the percentage of each factum that dealt with the issue of whether the appeal raised an issue of national importance, among other things. This was particularly impressive because the Supreme Court was not posting leave factums on its website at that time.) This analysis showed that at least in a subset of cases, spending more time on issues of national importance had an impact: of those cases at the last branch of the decision tree, those cases where at least 28% of the applicant’s factum addressed the issue of national importance had a 30% chance of getting leave, while those that devoted less than 28% of the factum to the issue of national importance had only a 4% chance of getting leave.

Our analysis provides some helpful rules of thumb that can guide one’s assessment as to the likelihood of getting leave (longer court of appeal decisions are more likely to get leave), as well as strategies that can be taken to maximize one’s chances of getting leave (spend a substantial portion of your leave application arguing that the application raises issues of national importance).

By contrast, it is worth noting a few factors that did not show a statistically significant relationship with the likelihood of getting leave.

(A word of caution is in order here regarding the absence of statistical significance. The fact that one variable was not found to be statistically significant does not mean that it is does not in fact correlate with the variable you are trying to measure. It may well be that on a larger dataset, a statistically significant correlation could be found. Consequently, the absence of evidence for a particular correlation should not necessarily be taken as proof of the absence of such a relationship.)

First, our analysis did not show that whether the Court of Appeal had allowed or dismissed the appeal had an impact on the likelihood of leave. Taken by itself, this factor had a weak statistical relation with the likelihood of leave being granted. However, after controlling for decision length and other factors, the impact of this factor disappeared entirely.

Second, we found no relationship between the length of time it took for a leave decision to be made. While the data showed that cases where leaves were granted took two weeks longer to be decided than cases where leaves were dismissed, none of the models we tried showed that this difference was statistically significant.

Third, there was no statistically significant relationship between whether a Charter issue was raised in the case and whether leave was granted. I suspect that this is because Charter cases fall into at least two very different categories. On the one hand, a well-founded Charter case is more likely to raise issues of national importance because they raise fundamental rights issues. On the other hand, the Charter is sometimes invoked by self-represented litigants who do not understand if the Charter actually applies, and the Charter can sometimes also be the last refuge of someone with a weak argument. In the former case, the Charter issue might increase the likelihood of leave being granted, while in the latter case, it might reduce it.

Fourth, whether the litigant was self-represented in seeking leave to appeal did not have a statistically significant impact, after controlling for other variables. While this may be surprising at first glance, we suspect that this feature is captured in other variables. For example, when a self-represented litigant seeks leave to the Supreme Court of Canada, it is likely they were also a self-represented litigant at the Court of Appeal. One can expect that Court of Appeal decisions involving those self-represented litigants are, on average, likely to be shorter, and the government was more likely to be successful. Consequently, the fact that a party is a self-represented litigant has little statistically significant value over and on top of other factors that were controlled for.

Finally, we note one factor that would be expected to have an impact on the likelihood of leave being granted which could not be measured on this data, and that is the presence of a dissent at the Court of Appeal. One would expect that a dissent at the Court of Appeal would increase the likelihood of leave being granted, as a dissent would likely indicate the presence of a contentious legal issue. However, in the entire dataset, there was only one case where there had been a dissent at the intermediate Court of Appeal. There was simply too little variation to come to any conclusion about the impact on the likelihood of leave being granted in the presence of a dissent.

Taken together, Professor Brooks’ dataset is a very helpful contribution to empirically understanding the factors that influence the likelihood of leave to the Supreme Court of Canada being granted. I am grateful to her for sharing it and look forward to further empirical work on these issues.