Conveyancers are being offered AI tools that answer legal questions in fluent, confident prose. Ed Molyneux argues that fluency is precisely the wrong thing to trust, and sets out the questions every firm should ask a supplier before believing a single answer.
A conveyancer evaluating an AI tool is in an awkward position. The tool answers in complete, confident sentences, cites the right-sounding law, and arrives in seconds at a conclusion that would take a fee earner an hour. Everything about the presentation invites trust. And the presentation is exactly what you must not trust, because fluency and correctness are different things, and modern language models are far better at the first than at the second.
It is worth being concrete about how wrong a confident model can be. In one openly published test, several leading models were asked to carry out a stamp duty calculation. Most of them got it wrong, and none of them hesitated while doing so. The models are improving quickly and are genuinely capable. But “it sounds right” carries no information about whether it is right, and a busy conveyancer skimming a plausible answer under time pressure is exactly the reader that confidence is most likely to mislead.
A measure of truth
So the question that matters when a supplier demonstrates a polished AI is not “can it read a lease” but “how would I know when it is wrong?” The only honest answer to that is measurement. A tool worth trusting has been tested against the judgement of a qualified expert, on real cases the expert has labelled, and its rate of agreement is known. If a supplier cannot tell you that number, or cannot tell you what it was measured against, they are asking you to accept fluency as evidence.
But not any number will do, and this is where firms should press. Ask three things.
First, who decided the right answer? If the tool’s accuracy was graded by another AI, or by the tool effectively marking its own homework, the figure is close to meaningless; the ground truth has to be human and expert. Second, is the benchmark held still? A number quoted against a moving or hand-picked set of easy cases tells you nothing; it should be a frozen set the vendor cannot game.
Third, and most important for our work, is the accuracy reported by severity? A tool that is right ninety-five per cent of the time overall but wrong on the small number of calls that carry real risk is not ninety-five per cent safe. The whole liability lives in the exceptions.
Qualifiying questions
Two further questions separate a serious tool from a demonstration. One is coverage: does it answer every relevant question, including the ones where the data is simply missing, or only what it happens to find? A risk nobody asked about is the one that surfaces on completion. The other is the direction of its errors: over-flagging wastes time, but under-flagging misses the defect that becomes a claim, and a supplier who quotes one blended figure without saying which way it errs is hiding the number that matters most.
There is one qualification, and it cuts in the profession’s favour. Not every output from these tools is an answer you rely on. A tool that tells you there is no issue with a title is giving an assurance you would act on and put your name to; a tool that says you may wish to enquire about a covenant is only putting a question in front of you, and leaving the judgement where it belongs.
The first needs measured accuracy behind it. The second promises nothing, and a prompt to look is useful long before anyone can quote a number. So the sharper question is not only how accurate a tool is, but what a given output is actually claiming.
Direction of travel
None of this is unique to conveyancing, but conveyancing is where it bites hardest, because the answers carry liability and, increasingly, insurance. The Legal Services Board has already named redress as one of its consumer non-negotiables for AI, and the first insured AI conveyancing products, such as the title review underwritten by First Title for Orbital Witness, exist precisely because a measured, bounded error rate is something an underwriter can price.
The direction of travel is clear. The AI that earns a place in regulated work will be the AI whose accuracy can be measured, defended and stood behind, not the AI that simply sounds the most assured.
So the practical advice is simple, and it applies to every supplier who calls this year, ourselves included. Do not buy the demonstration. Ask what each answer is actually claiming, and wherever a tool offers an assurance, ask what it was measured against, who set the ground truth, whether the benchmark is frozen, and how it performs on the cases that actually carry risk.
A firm that asks those questions will adopt this technology safely, and sooner than it fears. A firm dazzled by a confident paragraph will find out the hard way that confidence was never the point.
About the author
Ed Molyneux is co-founder and CTO of Moverly, the property intelligence platform working with LMS and Connells Group to bring structured, verified data to property transactions. He is the architect of the Property Data Trust Framework (PDTF), the open standard for machine-readable property data now being adopted across the industry. Ed writes about AI, property data infrastructure, and the future of conveyancing.
