A randomized experiment with junior engineers. One group wrote code with AI, the other without. The task: use a Python library that no one had seen before. Then, a comprehension test.
Results:
- The AI group finished about 2 minutes faster
- But on the test, the AI group scored 17% worse (approximately a 2-grade difference)
An interesting nuance: some in the AI group still achieved high scores. The difference was in how they used AI. They asked conceptual and clarifying questions to understand the code, rather than just delegating the task.
My takeaway: AI speeds things up, but if you really want to learn, ask "why" instead of "do it for me."
📝 Paper: https://arxiv.org/abs/2601.20245
