Accelerating Large-Scale Test Migration with LLMs – A Game Changer at Airbnb

What if a project estimated to take 1.5 years of manual labor could be completed in just six weeks? That’s exactly what Airbnb pulled off by leveraging large language models (LLMs) to automate the migration of nearly 3,500 test files from Enzyme to React Testing Library (RTL). This article walks us through how Airbnb structured an LLM-driven automation pipeline to refactor tests at scale—reducing not just engineering time, but also maintaining test intent and code coverage. Using iterative prompt tuning, dynamic retry loops, and an ever-expanding pool of test context, they reached a 97% automation success rate, only needing manual intervention for the final 3% of cases. More impressively, their approach demonstrates how LLMs can be used not just for generating code, but intelligently adapting existing codebases without breaking functionality.

Key Takeaways:

• The Challenge: With Enzyme no longer aligning with modern React practices, Airbnb needed to migrate to RTL but couldn’t afford to lose test coverage or invest years in a manual migration.
• LLM to the Rescue: Inspired by a 2023 hackathon project, Airbnb built an LLM-powered migration pipeline that broke down the transformation into automated validation steps, leveraging retry loops and expanding context dynamically.
• Step-Based Approach: Using a state machine model, each test file passed through validation, refactoring, Jest fixes, lint fixes, and a final approval step before being marked complete.
• Brute Force & Smart Prompting: Instead of perfect prompt engineering, success came from retrying steps multiple times and feeding the LLM richer context—including 50 related files, good test examples, and component-specific guidelines.
• Rapid Migration Success: 75% of the 3.5K files were migrated successfully in just 4 hours, and after refining prompts, they reached 97% completion within four days.
• Final Push: The remaining 100 files required manual fixes, but the automated pipeline had already solved the heavy lifting, reducing engineering time dramatically.
• Broader Implications: This success proves LLMs can revolutionize large-scale code migrations, setting the stage for future automation projects at Airbnb and beyond.

As Airbnb notes, “this migration underscores the power of LLMs for large-scale code transformation.” Their next move? Expanding and refining LLM-driven development tools to push the boundaries of automation. If you’re an engineer who loves solving big problems at scale, well… Airbnb’s hiring. 🚀
Link to Article