How I’m Writing a Paper About Human Research Assistants Without Any

How I’m Writing a Paper About Human Research Assistants Without Any

My new paper asks whether AI is replacing research assistants in economics. To answer it, I had AI read the acknowledgments of all 11,718 NBER working papers from 2018 through October 2026. No human research assistant worked on the paper. Two AI systems did that work, and I ran both.

This post explains how I did it.

Step 1. Start with a question you can test, just like with any other empirical project

The project began as one panel in a comic op-ed I was making with ChatGPT. The comic was about how I use AI instead of human research assistants and worry about the future of the profession as this rung on the career ladder goes away. Whether AI is replacing human research assistants is, in principle, an empirical question. Economists thank their research assistants in the acknowledgments. If AI is replacing research assistants, fewer papers should be thanking them. That gave me a question with a yes-or-no answer and a place to look.

I used NBER papers because they are easy to study the same way over time. The National Bureau of Economic Research is a private, nonprofit research organization whose affiliated researchers are mostly economists at leading American universities. The working papers it issues all follow the same format, with the acknowledgments on the first page. However, NBER researchers are a fairly elite group, likely with bigger research budgets and more students to hire than most economists. They may also be among the first to try new tools. As a result, the paper describes what is happening at the top of the profession, which may not be what is happening everywhere else.

Step 2. Run the study twice, and keep the two runs apart

On October 13, I asked ChatGPT to start collecting acknowledgments from NBER papers. The next day, I gave Claude the same question and told it not to look at anything ChatGPT had done.

The two systems never talked to each other. Everything that passed between them went through me, and I passed almost nothing. Claude got the list of paper ID numbers that ChatGPT had built, and that was all. Each system wrote its own rules for what counts as thanking a research assistant. Each collected and read the papers on its own, downloaded its own data about the authors, and ran its own statistical tests.

This project would barely be feasible without AI. Coding 11,718 papers by hand would be grueling and error-prone, and the best way to catch errors is to have a second team do the same work for comparison. Two AI systems, working separately with their own rules, solve both problems. Each can do the job in days, and if both find the same answer, that answer is less likely to be an accident of how one system read the papers.

Step 3. Write the rules before you look at the data

Before either system coded a single paper, I approved a plan that said what we would measure, how we would measure it, and which tests would count as the main results. Then we froze it. Any change after that is an amendment, dated and logged with the reason. The paper has 40 amendments. It says which tests were planned in advance, which ones outside readers asked for after seeing early results, and which ones I chose after seeing the data. Those are different kinds of evidence, and a reader should know which is which.

Step 4. Get permission

This should have been step 2. You should not read thousands of web pages from one site without asking. As soon as I realized that AI agents downloading and reading papers at that pace could run up against NBER’s access rules, I stopped both systems and asked NBER for permission. NBER said yes, with limits on how fast we could read and what we could keep. The study worked under those limits.

Step 5. Let the systems do the reading

The figure shows how I organized the work. Each branch is a group of sub-agents, each with one job. Some read the NBER papers and recorded what they found. Others checked the first group’s work. Others audited the results. Others wrote referee reports on the drafts.

A coder had to pass a test on sample papers before the study accepted its work. Then it read each acknowledgment and recorded whether the paper thanked a human for research help and whether it said the authors used AI. A different model then re-read samples without seeing the first codes and agreed with them 97 to 100 percent of the time. Auditors re-read 356 papers coded as “no research assistant” and found no missed credits. Every credit and every AI statement the paper counts comes with an exact quote, checked against the stored text.

Step 6. Check the work yourself

I audited 100 acknowledgments by hand against the codes. I also read every paper on which the two systems disagreed and decided how to code it.

Step 7. Have the systems referee each other

Each system wrote referee reports on its own drafts, and I had ChatGPT review Claude’s draft. That review found an inconsistent coding rule, an unfair date cutoff in one comparison, and a mistake in a formula. The systems fixed and logged all three. Colleagues read early drafts and suggested tests that had not occurred to me.

Step 8. Compare the two answers

The two systems read almost every paper the same way. Of the 11,271 papers both coded as thanking or not thanking a research assistant, they disagreed on 14. On whether the authors said they used AI, they disagreed on one paper out of 11,663. The two studies also agree on the answer. Each found that the share of papers thanking a research assistant fell by roughly three percentage points after ChatGPT arrived, and their estimates differ by less than one point.

The comparison was not fully blind. On October 23, I sent Claude’s codes to the ChatGPT study so it could compare them, which means ChatGPT’s later coding ran with those codes nearby. The 2,602 papers ChatGPT coded before that date agree with Claude’s at the same rate as the 9,106 it coded afterward. That is reassuring, but it is not proof.

Step 9. Save everything

I keep a repository with the plan, the coding rules, the amendment log, the referee reports, and the comparison between the two studies.

What it cost

Reading 11,718 acknowledgments and 9,644 full papers, coding them, checking samples, auditing the negatives, and writing up the results would have taken several research assistants months. It took days. I subscribe to Claude’s Ultra plan and ChatGPT’s Pro plan. I did not buy extra credits. My time went to designing the study, editing, and making judgment calls.

What this does and does not show

This method is itself evidence about what these tools could do in late 2026. The paper’s data cannot show that yet, because papers appear long after the work is done, and researchers take years to learn what a new tool can do. The acknowledgments of 2026 mostly record decisions made before tools like these existed.

The method has limits too. I designed the checks, and the systems being checked ran them. The replication was not fully blind. A hand audit of 100 papers is small. I report all of that in the paper, because the paper is where it has to be judged, and because the point of working this way was to find out whether the work could be judged at all.

Scott Wallsten is President and Senior Fellow at the Technology Policy Institute, a Policy Fellow at the Stanford Institute for Economic Policy Research (SIEPR), and a senior fellow at the Georgetown Center for Business and Public Policy. He is an economist with expertise in industrial organization and public policy, and his research focuses on competition, regulation, telecommunications, the economics of digitization, and technology policy. He was the economics director for the FCC's National Broadband Plan and has been a lecturer in Stanford University's public policy program, director of communications policy studies and senior fellow at the Progress & Freedom Foundation, a senior fellow at the AEI–Brookings Joint Center for Regulatory Studies, a resident scholar at the American Enterprise Institute, an economist at the World Bank, and a staff economist at the U.S. President's Council of Economic Advisers. He holds a PhD in economics from Stanford University.

Share This Article

artificial intelligence, research methods

Recommended Reads

Is AI Replacing Human Research Assistants? Evidence from NBER Working Papers

The 3 Percent Data Center Scenario Is Not a Forecast

AI Can Produce Research. Who Will Produce Researchers?

Explore More Topics

Antitrust and Competition 193
Artificial Intelligence 53
Broadband 401
Content Moderation 15
Economics and Methods 44
Evidence-Based Policy 19
Free Speech 23
Intellectual Property 56
Miscellaneous 338
Privacy and Security 154
Regulation 31
Trade 3

View More Publications by

Related Articles

Is AI Replacing Human Research Assistants? Evidence from NBER Working Papers

Research Roundup – October 2026

The 3 Percent Data Center Scenario Is Not a Forecast

AI Can Produce Research. Who Will Produce Researchers?

What I Learned Running a Hundred-Page Empirical Study With AI

Research Roundup – September 2026

TPI Aspen Discussion Recap: AI Finds the Flaws. Can We Fix Them Fast Enough?

The AI Labs Asked for a Waiver. Congress Should Ask for a Justification

SIGN UP FOR UPDATES

"(Required)" indicates required fields

This field is for validation purposes and should be left unchanged.

Secret Link