I’m using AI to study whether AI is replacing research assistants. I’ll post the findings when the study is done. In the meantime, here is how I’m making the sausage
It started as a cartoon. I was making a comic op-ed with ChatGPT when I realized the question deserved a real study. On October 13 I asked ChatGPT to start collecting acknowledgments from NBER working papers. The next day I gave Claude the same question and asked for an independent study, with instructions not to look at ChatGPT’s work.
The figure below shows what happened next. Each system ran its own study, and each branch off a line is a group of agents it sent out to code papers, check the first coders’ work, audit the results, or referee the drafts. Each system eventually read the acknowledgments in all 11,718 NBER working papers from 2018 through October 2026. That is months of research-assistant work, and it took days.
The two systems never talked to each other. Every exchange went through me, and so did every decision. I wrote the question, approved the protocol, and decided what to do when things went wrong. At one point that meant stopping both systems for a week while I asked NBER for permission to read its pages.
Coding every paper independently, the two systems disagreed on 15 of the 11,271 papers both could code. That shows the coding is reproducible. It does not show the coding is right, which I’m working on now.
Scott Wallsten is President and Senior Fellow at the Technology Policy Institute, a Policy Fellow at the Stanford Institute for Economic Policy Research (SIEPR), and a senior fellow at the Georgetown Center for Business and Public Policy. He is an economist with expertise in industrial organization and public policy, and his research focuses on competition, regulation, telecommunications, the economics of digitization, and technology policy. He was the economics director for the FCC's National Broadband Plan and has been a lecturer in Stanford University's public policy program, director of communications policy studies and senior fellow at the Progress & Freedom Foundation, a senior fellow at the AEI–Brookings Joint Center for Regulatory Studies, a resident scholar at the American Enterprise Institute, an economist at the World Bank, and a staff economist at the U.S. President's Council of Economic Advisers. He holds a PhD in economics from Stanford University.
