We start this week’s Wrap with some surprising data about Grok’s performance on contract review. But first, a quick announcement: Artificial Lawyer is now on editorial holiday until Thursday Oct 29 and won’t be publishing news until then.
This site and its founder will be in Boston, USA, to help moderate ClioCon’s Enterprise track and is taking the opportunity to get some R&R before Legal Innovators UK on Nov 4 + 5, and then returning to America for Legal Innovators New York on Nov 17 + 18. (See below for your express tickets).
Right, now onto Grok and other news.
Ivo has developed a new benchmark with data lab, micro1. They tested seven leading AI models across 350 real-world contract review tasks to ‘see how well they could make the same judgment calls as experienced attorneys’. The benchmark evaluated whether they could prioritize important issues, make appropriate edits, and recognize when a decision required human input, they explained.
According to the results, all seven models scored below 60% on weighted attorney-defined criteria, with even the top-performing model reaching just 58.3%.
And that’s not massively surprising, because to get above moderate success rates on complex legal work you really need more ‘context’, i.e. specific reference data, rulebooks and precedent documents that are very closely fitted to the matter, and subtle oversight that can correct initial outputs.
But, what did surprise this site is that Grok – the LLM creation of Elon Musk and team, and often overlooked by many legal AI experts – came top in terms of overall performance for contract review.
Ivo stated: ‘Grok 4.7 leads a close top tier, but completion, cost, and speed separate the leaders. Grok leads on weighted rubric pass rate at 58.3%, followed by Opus 5.5 at 58.1% and Kimi K3 at 57.1%. Completion rates range from 80.8% for Opus to 99.5% for Grok and 100% for Kimi. Kimi costs 38% less per completed rollout and finishes 13% faster than Grok.
See table below – and here’s more about the new Ivo Contractbench.
And now the rest of the news:
Ertan Bera, who worked with Lovable on core AI product development, has joined Sweden’s Aloi as Applied AI Lead. Joining him is Erik Lindberg, as CRO, who spent five years as VP Sales at Sana through its billion-dollar exit. Aloi is focused on developing the judgment layer for law firms by tapping past matters – and the work during those matters – to capture insights.
Checkbox, an AI legal front door used by SAP, PepsiCo, and BILL, has launched Matter History, which is a record of every change on a legal matter and who or what made it, from a lawyer to an AI agent. Other new features include Tasks & Events for tracking deadlines on the matter, bulk editing across matters, and multi-document DocuSign envelopes.
Lawfecta, a legal talent marketplace, said it is developing an ‘AI-powered operating system designed to help attorneys organize workflows, prepare tasks for delegation, and connect with professionals who can execute the work’.
Mary Technology has announced an integration with Harvey (and Harvey has now rolled out multiple links with other legal tech companies via its Connector Library). The move brings Mary’s ‘specialist factual investigation capabilities into Harvey’s legal AI platform’. Mary reads and maps a matter’s uploaded documents before the first question, and retains the team’s findings and review as the case develops. The connector brings that accumulated understanding into Harvey, so lawyers can investigate an issue using both the evidence and the work their team has already done, they said.
Littler, the world’s largest employment law firm, has announced strategic partnerships with Legora and Epiq, ‘to help employers navigate the legal, workforce, and operational implications of AI-enabled workforce transformation’.
BLEN, a launch partner for Google Cloud’s Gemini for government services, has announced that its Federal Register, Electronic Code of Federal Regulations (eCFR) connector is now available. The connector gives agents read-only access to official US federal regulatory data from all three sources.
—
And now some Webinar news. First, one all about agents: what are they, how to build them, how to use them, and why they are so helpful – all with Harvey and guests.
Join us on November 2 at 11 am ET, 4pm GMT, for a great new webinar with Harvey in association with Artificial Lawyer, where we will explore: ‘Agents in Legal: A Practical Guide to Choosing, Building, and Rolling Them Out’.
Get a practical, no-hype look at what legal AI agents really are and how to roll them out. In 45 minutes, Harvey’s Ryan Samii, Artificial Lawyer’s Richard Tromans, and two law firm leaders, Tania Djerrahianand Andrei Salajan, who’ve rolled out agents themselves will cover agent use cases, where human review fits in, and winning over skeptics.
–
And then a mix of webinar and live product demo with Spellbook, all about the future direction of contract management and its new ACM – autonomous contract management – capability
On Thursday, October 29 at 12:00 PM ET, 5PM UK – Spellbook’s CEO, Scott Stevenson, and Artificial Lawyer, will be holding a special live webinar: ‘The Future of Contract Management’, which will include the first ever public demonstration of their new ACM – Autonomous Contract Management – capability. It will be a lively discussion about where we are going with contract AI, with questions from the audience, plus the demo of Spellbook’s major new product for inhouse legal teams.
–
And of course, as mentioned: Legal Innovators UK is in 2 weeks on Nov 4 and 5 at the Minster Building near the Tower of London, and then Legal Innovators New York is on Nov 17 and 18 in Midtown at a great new venue – Quorum by Convene. Looking forward to chairing both. Going to be lots to explore together, that’s for sure!
Here are the express ticket registrations so you can get your complimentary passes super-fast.
To pick up your UK tickets, please follow the link below. Complimentary passes are for those working inside law firms and within corporate legal teams.
And for your New York tickets, please see here:
Legal Innovators New York Conference – Nov 17 + 18
—
That’s all for now, have a nice weekend folks!
I’m off to Boston to spill some tea! Back ‘in office’ on Thursday, October 29.
Richard Tromans, Founder, Artificial Lawyer and Chair of the Legal Innovators conferences.
Discover more from Artificial Lawyer
Subscribe to get the latest posts sent to your email.