The long-held vision of Big Tech CEOs for autonomous AI agents capable of seamlessly interacting with software applications to accomplish human tasks is currently facing significant limitations. Modern consumer AI agents, such as OpenAI’s ChatGPT Agent and Perplexity’s Comet, still demonstrate considerable restrictions in their capabilities, suggesting that a new generation of techniques is required to foster more robust AI agent development. Among these techniques, the careful simulation of workspaces for training agents on multi-step tasks, known as reinforcement learning (RL) environments, is emerging as a crucial element, akin to how labeled datasets propelled the previous wave of AI.

At their core, RL environments serve as sophisticated training grounds that mimic an AI agent’s interactions within a real software application. One founder aptly described building these environments as "creating a very boring video game." For instance, an environment could simulate a Chrome browser, tasking an AI agent with purchasing a specific item like a pair of socks on Amazon. The agent's performance is then graded, and it receives a reward signal upon successful completion of the task. While such a task appears straightforward, AI agents can encounter numerous challenges, from navigating complex web page menus to making incorrect purchasing decisions. The inherent unpredictability of an agent’s potential missteps necessitates that the environment itself be robust enough to capture any unexpected behavior while still providing valuable feedback. This requirement makes the construction of RL environments considerably more complex than simply curating static datasets. Some environments are highly elaborate, enabling agents to utilize tools, access the internet, or integrate various software applications for task completion, while others are more specialized, focusing on specific enterprise software functions.

The concept of using RL environments is not entirely new; historical precedents include OpenAI’s "RL Gyms" from 2016 and Google DeepMind’s AlphaGo, which famously beat a world champion at Go using RL within a simulated environment. However, what distinguishes today's endeavors is the focus on building computer-using AI agents with large transformer models. Unlike the specialized, closed-environment systems of the past, contemporary AI agents are being trained for more general capabilities. While researchers today benefit from a stronger technological starting point, their ambitious goal presents a more intricate challenge with greater potential for errors.

The burgeoning demand for RL environments has created a crowded and dynamic field within the AI industry. AI researchers, founders, and investors confirm that leading AI labs are actively pursuing in-house development of these environments, yet they are also keenly seeking third-party vendors capable of supplying high-quality environments and evaluations. This shift has galvanized established AI data labeling companies and birthed a new class of startups.

Major data labeling entities like Surge, Mercor, and Scale AI are actively adapting to this evolving landscape. Surge, reportedly generating significant revenue from collaborations with major AI labs such as OpenAI, Google, Anthropic, and Meta, has observed a "significant increase" in demand for RL environments and has established a dedicated internal organization for their development. Mercor, valued at $10 billion, is also working with prominent labs and is focusing its efforts on building domain-specific RL environments for areas like coding, healthcare, and law. Despite facing increased competition and past losses of major clients, Scale AI is demonstrating its ability to rapidly adapt, investing in new frontier spaces including agents and environments, drawing on its history of successful pivots from autonomous vehicles to the chatbot era.

Alongside these established players, a new wave of startups is focusing exclusively on RL environments. Mechanize, a relatively new firm, aims to "automate all jobs" but has strategically begun by developing robust RL environments specifically for AI coding agents, reportedly working with Anthropic and offering highly competitive salaries to attract top engineering talent. Prime Intellect, backed by notable investors, is taking a different approach by targeting smaller developers, launching an RL environments hub designed to democratize access to resources typically available only to large AI labs. This platform, envisioned as a "Hugging Face for RL environments," also offers access to computational resources, acknowledging the increased GPU demand for training generally capable agents.

Despite the widespread enthusiasm, a critical question remains regarding the scalability of RL environments compared to prior AI training methods. Reinforcement learning has undeniably driven significant AI advancements, including models like OpenAI’s o1 and Anthropic’s Claude Opus 4, especially as traditional methods show diminishing returns. Environments offer a promising avenue by allowing agents to interact with tools and computers in simulations, moving beyond simple text-based rewards. However, this approach is also considerably more resource-intensive. Skepticism exists, with concerns about "reward hacking"—where AI models exploit loopholes to gain rewards without truly completing tasks—and the inherent difficulty in scaling environments effectively, as highlighted by former Meta AI research lead Ross Taylor. Sherwin Wu, OpenAI’s Head of Engineering for its API business, expressed caution regarding RL environment startups due to intense competition and the rapid pace of AI research. Even Andrej Karpathy, an investor in Prime Intellect who sees environments as a potential breakthrough, has voiced broader reservations about the extent of future progress attainable specifically from reinforcement learning, stating he is "bearish on reinforcement learning specifically" but "bullish on environments and agentic interactions." The future of RL environments, while promising, is subject to ongoing innovation and the resolution of these significant challenges.