🔍 Read the full analysis: Data Science With AI: What Consultants’ ChatGPT Results Suggest on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
In a randomized trial involving BCG consultants, participants given GPT-4 and brief training scored higher than a control group on coding, statistics and prediction tasks. After ChatGPT was removed, researchers found no evidence that the participants had gained an advantage on technical questions they answered alone.
A randomized trial of Boston Consulting Group consultants found that access to GPT-4 improved performance on unfamiliar coding, statistics and prediction tasks, but researchers found no evidence of an advantage when participants later answered technical questions without ChatGPT. Published October 26 in Nature Human Behaviour, the study suggests that AI can raise workers’ performance while they use it without necessarily building skills they can apply independently.
The researchers assigned consultants to either a ChatGPT group or a comparison group trained to use Google and resources such as Stack Overflow and Khan Academy. The ChatGPT group received 15 to 20 minutes of prompting instruction and used GPT-4. Participants completed two of three 90-minute tasks: cleaning and combining data with Python, evaluating statistical models, or predicting the predictability of international football matches.
In the study’s main analysis, based on 487 consultants who submitted both assigned tasks, the ChatGPT group scored 49 percentage points higher than the comparison group on coding, 20 points higher on statistics and 17 points higher on prediction. The researchers set the average performance of 44 BCG data scientists, who completed the tasks without AI, as a benchmark. The ChatGPT group remained below that benchmark on average across all three tasks.
After the work tasks, participants answered five technical questions on topics including probability, machine learning and coding, with no ChatGPT access. The authors reported no evidence that the ChatGPT group performed better on this unaided test. They also found that the group using ChatGPT was more confident that the system could solve problems it could not, an analysis the authors describe as exploratory.
The trial in motion · BCG consultants · 2024
With ChatGPT: close to the experts. Without it: no measurable gain.
A randomized trial with a control group that used Google and Stack Overflow instead of ChatGPT.
- 1 · Training15–20 minutes on prompting ChatGPT
- 2 · TasksTwo 90-minute data science tasks, with ChatGPT
- 3 · ScoresGap to 44 data scientists closes, not fully
- 4 · Tool removedNo ChatGPT from here on
- 5 · TestFive technical questions
Coding task score
0 = data scientists’ average
ChatGPT on+49 pp on codingStill below the data scientists on average
Knowledge test without ChatGPT
ChatGPT group vs. control, five questions
ChatGPT off−7.7 to +4.8 ppNone significant · joint test p = 0.531
Source: Wiles, Krayer et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (2026); authors’ manuscript, Sections 2.1 and 2.3, Tables B7 and B9. Bars scaled to the five published differences.
Graphic: Powered by Thorsten Meyer AI
AI Raises Task Performance, Not Skill
The results matter to employers weighing whether AI tools count as workforce training. In this experiment, access to ChatGPT helped some consultants complete technical work they were not trained to do, but the follow-up test did not show that the experience had made them better at answering related questions alone.
That distinction affects how companies evaluate productivity, training and quality control. A worker can produce a stronger result with an AI assistant without having acquired the knowledge needed to reproduce or check that result independently. The study does not establish how often this happens across workplaces, but it cautions against treating tool-assisted performance as proof of lasting upskilling.
The authors also found differences by task. Coding showed the largest gap between the ChatGPT and comparison groups, while the statistics result was smaller and the prediction result smaller still. The findings point to benefits that can depend on the task and the worker; they do not show that AI improves every kind of knowledge work by the same amount.
Products Worth Considering
Logitech MX Mechanical Wireless Illuminated Keyboard, Tactile Quiet, Graphite
Bose QuietComfort Wireless Noise-Cancelling Over-Ear Headphones, Cypress Green
Bose QuietComfort Wireless Headphones – Twilight Blue (Limited Edition)
As an affiliate, we earn on qualifying purchases.
How the BCG Trial Was Designed
The trial took place at BCG in late March and early April 2024 and was preregistered before data collection. Of 986 consultants allocated to the groups, 573 began the survey and 487 submitted work on both tasks used in the main analysis. Participants had at least a bachelor’s degree, and the authors said most reported no coding experience.
BCG data scientists and OpenAI’s Economic Impacts Research team designed the tasks so ChatGPT could provide useful guidance but could not correctly answer every question on its own. The researchers compared participants’ work with that of 44 BCG data scientists working without AI. The expert benchmark was not a separate randomized comparison of AI use among data scientists; it served as a reference point for task scores.
Among consultants who already had some coding experience, the authors reported scores on coding and statistics that were not detectably different from the data scientists’ average. That finding applied to this subgroup and those two tasks; it does not establish that experienced workers generally match experts when using AI.
“The authors find no evidence that the ChatGPT group did better on the technical questions answered without the tool.”
— The study authors, as described in their paper
Limits of the Skill Test
The unaided follow-up consisted of five technical questions, not a long-term assessment of participants’ performance or learning. The study therefore does not show whether repeated use, different training, or later practice could produce durable skills. Nor does it establish how the results would translate to other employers, occupations, AI systems or workplace tasks.
The main analysis included 487 of the 986 consultants allocated to the groups, after fewer participants began the survey and submitted both tasks. The supplied study summary does not explain in detail how outcomes for those who did not complete both tasks may affect interpretation. The reported increase in confidence about ChatGPT’s abilities was exploratory, so it should not be treated as a definitive finding about users’ beliefs.
The results also do not show that AI assistance is useless or that every AI-supported output is unreliable. They measure performance on particular tasks under the trial’s conditions. The study leaves open how employers should combine tool access with instruction, review and repeated practice.
What Employers Should Measure
The study does not announce a new company policy or a follow-up trial. For employers, its findings point to a practical distinction to test: whether AI improves completed work, whether workers can explain and verify that work, and whether they retain the relevant skills when the system is unavailable.
Further research would be needed to determine whether longer training or repeated use changes the unaided results, and whether the same pattern holds across different tasks and workplaces. Until then, the trial supports a limited conclusion: ChatGPT improved performance on these tasks during use, while this study found no evidence of a corresponding advantage on its short, tool-free technical test.
Key Questions
What did the BCG trial find about ChatGPT?
Participants using GPT-4 scored higher than the comparison group on coding, statistics and prediction tasks. The reported differences were 49, 20 and 17 percentage points, respectively.
Did the consultants learn to do the work without ChatGPT?
The researchers found no evidence of better performance by the ChatGPT group on five technical questions answered after the tool was removed. The test was limited and does not settle whether different or longer training could build lasting skills.
Did the ChatGPT group match expert data scientists?
No. On average, the ChatGPT group remained below the benchmark set by 44 BCG data scientists working without AI. Some participants with coding experience had scores on coding and statistics that were not detectably different from the benchmark.
How many consultants were included in the main analysis?
The main analysis included 487 consultants who submitted work on both assigned tasks. The study allocated 986 consultants to the groups; 573 began the survey.
When was the study conducted and published?
The experiment took place in late March and early April 2024. The paper was published in Nature Human Behaviour on October 26, 2026.
Source: ThorstenMeyerAI.com
