Navigating the Unknown: Setting the Standard for Legal AI Research Tools
“There is nothing more dangerous than someone who knows a little law.” My late father, a longtime criminal prosecutor turned judge, used to say this to me when I first started law school and immediately assumed I could solve any legal problem that came my way. What I lacked in knowledge, I made up for in confidence.
It was only when I started practicing law that I began to understand the importance of knowing what you don’t know. Lawyers live in the unknown. True, we master some foundational legal principles, and our experience helps guide our instincts, but the reality of legal practice, especially as a research lawyer, is that we are constantly considering novel issues. The key is acknowledging and embracing the “unknown.”
Recently, I’ve been contemplating the concept of the unknown in the context of legal AI. At Lenczner Slaght, we are piloting various legal AI platforms. As a research lawyer, I’ve approached these platforms with a mix of intrigue, awe, and admittedly, a healthy dose of skepticism.
A breakthrough moment came when one of these platforms, Alexi, a legal research and memo drafting tool, responded that it could not answer a question because it did not turn up sufficiently relevant material. In contrast, the same question, when submitted to a different legal AI platform, provided a very confident, clear, and incorrect answer. On another occasion, Alexi indicated that its provided response was by way of analogy, since it did not find case law directly on point.
To me, this is the real gold standard of any AI legal research tool. I’m not especially impressed by all the times these platforms provide helpful and correct legal analysis (although this is obviously a crucial aspect of the tool). Rather, what gave me pause was its ability to recognize its limitations and make a judgment call in the face of the unknown.
There has been so much focus on the fear of hallucinations – AI generating content that does not exist – that we don’t seem to have spent enough time thinking about the opposite: can legal AI tools recognize when they simply don’t know the answer? And if they can, how do they grapple with the unknown?
If legal AI systems can recognize the reality of the unknown and make a judgment call as to how to proceed, they will become an invaluable asset for legal research. That judgment call may require the user to reframe the question or provide reasoning by analogy to similar principles. Both reflect the reality of legal analysis as applied by humans.
The only “danger” of knowing a little law is when it is applied without first considering the limits of that knowledge. As is the case for all lawyers, for legal AI research tools, sometimes the best answer is no answer, coupled with a plan for how to move forward.
Stay tuned for Jessica’s top tips on leveraging AI for legal research.