When ChatGPT first came out, along with the whole story around artificial intelligence, its creators promised us the moon and the stars.
They told us AI would be the great “key” that opens doors for everyone, allowing a 24-year-old who just entered the job market to deliver work like a seasoned veteran.
But things are not exactly like that.
A new study from MIT brings us back down to earth, showing that AI may help you finish a job quickly, but in the end it teaches you nothing if you don’t already know the basics.
NEWSLETTER TABLE TALK
Never miss a story.
Subscribe now.
The most important news & topics every week in your inbox.
This story is very reminiscent of the old “China shock,” which hit the market when China forced its way into global trade and many factories closed.
Only now, the “blow” is not landing on industries but on offices. If young workers expect the computer to do everything for them, they risk falling into a major illusion: thinking they have become experts, while in reality they are just pushing buttons.
So let’s look at how technology, instead of sparing us the effort, may end up leaving us without real skills, and how we can use it wisely without losing our own judgment.
The Experiment With Lawyers
In a three-month experiment with patent lawyers, artificial intelligence improved the quality of drafts across the board. But when the researchers took the tool away and tested their independent judgment, only the experienced lawyers showed a moderate improvement. The younger ones did not.
“Our results challenge the idea that AI is an automatic skill equalizer,” economist David Autor said in written answers to Fortune.
“Our data suggest that it is a performance equalizer, but a skill non-equalizer, in the sense that only professionals who already had foundational mental models were able to improve their underlying skill sets.”
The study classified lawyers by experience, not age. “Senior” meant seven or more years of practice, and the researchers did not study Gen Z as a group. However, early-career workers are the people the findings concern most directly.
From The China Shock To Artificial Intelligence
Autor heads MIT’s economics department and is a visiting fellow at Google Technology and Society. He is best known for the “China shock,” which he developed with economists David Dorn and Gordon Hanson.
Their research showed that import competition from China caused lasting damage to American manufacturing communities, which did not adjust as smoothly as optimistic reports on globalization had predicted. In heavily exposed labor markets, wages and labor force participation stayed low and unemployment stayed high for at least a decade.
More recently, Autor and Hanson warned of a “China shock 2.0”: Chinese competition in advanced industries that threatens American technological leadership and the high-paying jobs that come with it.
Autor said AI “will not be, in any sense, a repeat of the China trade shock,” because that was experienced by American companies as a “pure negative competitive shock,” while AI will have “a very different texture” in boosting productivity.
The debate about AI rests on a version of the assumption the China shock overturned: that once technology takes over routine work, workers will naturally move on to higher-level skills. In his answers to Fortune, Autor said his concern about AI is not mass job loss.
“The risk we may face with AI is not that it eliminates a large number of jobs on net, but that it eliminates certain areas of expertise while creating new ones,” he said.
“That sounds like a reasonable deal, but the workers who lose those old careers are generally not the ones who will be able to benefit from the new roles that emerge.”
The patent law experiment looks at a narrower question: whether workers using a powerful new tool actually build the expertise they need to advance.
Does AI Assistance Improve Or Erode Expertise?
Whether AI assistance strengthens or erodes professional expertise is unclear.
In a preregistered three-month randomized controlled trial, 133 patent lawyers at eleven US intellectual property law firms were given access to a custom AI drafting assistant, and both their performance while using AI and their professional judgment after using it were measured.
The study examined how AI affects the work of 133 lawyers who draft patents for Google.
The research was funded by Google and carried out in collaboration with economist David Autor.
The study’s main findings are the following:
- Better drafts: After 90 days of using the AI tool, the lawyers improved their patent drafting scores by about 11 percentage points.
- Where the improvement came from: AI did not turn good work into “excellent” work, but it helped reduce errors and poor performance (it made mediocre work better).
- Unequal learning (experienced vs. junior): At the end, the lawyers were tested on finding errors in a patent without AI help. The experienced lawyers who had trained with AI in the previous days did much better than the rest. The junior lawyers, by contrast, showed no gain.
- Weaknesses of the study: The study had some problems, as several lawyers did not complete the final test, their skills were not measured before the study began, and there were suspicions that some secretly used AI on the final test even though it was not allowed.
The “Illusion Of Competence”
The juniors’ scores split, according to the study: significantly more poor scores, fewer average ones, more good ones and no gains at the top. The authors write that AI “acted as a springboard for some juniors and as a cushion for others.”
The blog post describes how the juniors worked. They tended to go from top to bottom, polishing the introductory text before reaching the main claims.
Some spotted serious flaws but left comments describing them instead of fixing them.
The juniors in the control group did the same, a pattern the authors call a “junior baseline deficit,” which three months of access to AI did not fix.
The juniors did notice one difference: they liked the tool. In his answers to Fortune, Autor said junior lawyers using AI reported an increase in job satisfaction, which was the largest effect in the study. They felt that skipping the “blank page problem” allowed them to step straight into the role of reviewer, he said.
“Young professionals need to beware of the illusion of competence,” he said. “The only way to really know whether you are building skills is to perform tasks without AI assistance and evaluate your performance.”
Why The Senior Lawyers Improved
According to the blog post, the experienced lawyers described AI in follow-up interviews as a “logic checker” rather than a finished product.
The authors write that it loosened their attachment to existing prose and forced them to explain the why and how of their structural changes.
“You can only use AI as a ‘logic checker’ if you already know the law well enough to spot when fluent-sounding text has legal flaws,” Autor told Fortune. He said he believes that being freed from basic drafting allowed the senior lawyers to focus on strategic scope, “on the forest, not the trees.”
On the unassisted test, the blog post says, the senior lawyers in the treatment group skipped low-stakes boilerplate text, rebuilt claims from scratch, cut language that could limit legal protection and tied many of their changes to legal theory.
This does not mean juniors should abandon the tool.
Autor told Fortune that beginners should work the way the seniors do: first lay out the argument without help, and then use AI as a critic to stress-test the logic. He acknowledged that this is a big ask.
“Most of us don’t have the self-discipline to do things by hand when a readily available, low-effort tool is sitting there offering to do it for us,” he said.
Don’t Cut Off The Apprenticeship
Automating tasks through AI carries the risk of leading employers to short-sighted decisions, such as hiring fewer junior employees, which threatens to disrupt the process of developing future senior professionals.
According to Autor, companies need to separate which tasks can be done permanently with AI and which require independent human judgment, while also introducing skill checks and mentoring.
Despite the limitations of this particular study, the point stands that AI cannot replace human expertise that is built through hard work.
In the long run, human and machine intelligence must work together and complement each other.
Source: MIT