AI Agents Face New Gatekeepers: Websites Erect Digital Barriers

A new era of consumer AI is emerging with personal AI agents like Meta’s Muse, Instinct, and ChatGPT’s Dots, which are designed to go beyond simple queries to actively complete tasks on users' behalf. These sophisticated agents can book flights, make dinner reservations, and order groceries, providing a seamless experience without requiring users to navigate complex technical setups. However, the widespread adoption of these agents is encountering significant friction as websites frequently block them from completing requested actions, leading to considerable user frustration.

The problem manifests in various forms. Most notably, Amazon recently began blocking Meta’s Muse AI agent from its retail site, preventing it from browsing products or making purchases. This issue is not isolated; user complaints across social media indicate that agents often encounter similar blocks across numerous platforms. These blocks can be intentional, like Amazon's, or unintentional, resulting from websites’ existing anti-bot measures designed to protect against spam and malicious activities. The lack of clarity for end-users on whether a block is accidental or deliberate exacerbates their frustration with both the service provider/retailer and the AI agent.

For instance, TechCrunch reported complaints regarding Muse's inability to complete purchases on Walmart’s retail website. While NBC also noted Walmart had faced this issue during its own tests of AI agents, a spokesperson for Walmart clarified that these blocks were not intentional. In fact, Walmart is a partner with Muse, as announced at Meta’s Connect developer conference, aiming to be accessible wherever its customers are, including through AI agent experiences. The specific problem often encountered by Walmart customers involves the website presenting a human verification button, which, if interrupted, can cause the check to fail and the agent to be blocked. This highlights a critical incompatibility between current human verification technologies and the evolving agent-first web.

Beyond retail, other sectors are also experiencing challenges. Flight booking, a prime use case for AI agents, has seen numerous reports of airlines rejecting agent requests. Delta, for example, stated it is taking steps to protect customers from unauthorized automated activity and would only open up to AI agents with security and customer experience carefully considered. United's Terms of Use explicitly prohibit the use of robots or automatic devices for unauthorized purposes. Early adopters have cited brands like Yelp, eBay, Zillow, Pizza Hut, and Adidas as reportedly blocking AI agents, with some users even experiencing account suspensions on eBay due to agentic AI use. Google reportedly blocked Instinct while it was managing a user's Gmail inbox.

Companies like Yelp have clear policies: non-human traffic is not permitted unless the agent has paid for data access through its licensing program. This means agents attempting to get quotes, join waitlists, or book tables on Yelp will likely fail without a formal partnership. eBay, while not prohibiting all third-party shopping agents, restricts unauthorized agents and actions such as automated scraping and model training. Adidas, Zillow, and Pizza Hut did not provide comments on their policies.

Content delivery networks (CDNs) like Cloudflare are also implicated. Cloudflare, which offers protection against AI agents that might train on site data, may be disrupting Muse traffic. Changes to crawler defaults on September 15 allowed site owners to block AI training while permitting other bot types, potentially shifting existing sites to block AI agents on ad-bearing pages for security reasons. Although Cloudflare did not share specific data, its public data hub, Cloudflare Radar, shows an increase in bot activity without differentiating between 'good' and 'bad' bots. Cloudflare also has a vested interest in mitigating AI traffic as it operates a marketplace where AI bots pay for data access and has been testing a web browser for AI agents.

To address these pervasive issues, industry partners including Meta, Walmart, Stripe, Sierra, Genesys, Rocket, NiCE, and Decagon have begun collaborating on an open standard. This standard will dictate how AI agents communicate with businesses, differentiating legitimate bots acting on behalf of users from malicious ones. Meta emphasizes that this protocol will specifically focus on agent-to-agent communication for online commerce. Meta’s strategy of forming specific brand partnerships for Muse access, coupled with its focus on new standards, appears to be a viable path forward. This approach assures customers that its agent is welcome at partner sites, transforming access issues into reportable bugs. Meta also maintains a growing list of