The AI That Needs a Human
When the human is the product and the AI is the marketing.
The AI That Needs a Human
The Evolving Mindset
A drive-thru company told the market its AI was taking orders. More than seven in ten of those orders needed a human to step in. The gap between those two facts is the most important number in enterprise AI right now, and almost no buyer can see it.
The company was Presto Automation, a publicly traded restaurant-technology firm. It marketed its flagship product, Presto Voice, as an AI system that automated drive-thru order-taking and removed the human from the transaction. In January 2025, the Securities and Exchange Commission found that the claim did not match the operation. During the relevant period, more than 70 percent of orders processed by the in-house system required human intervention. At certain locations, the figure was 100 percent. The SEC also found Presto had failed to disclose that the AI itself was owned and operated by a third party. The product the market was told ran itself was, in substantial part, being run by people.
This is the failure mode the industry does not like to name, and it is now common enough to have drawn regulators. It is worth naming plainly. A large share of what is sold as artificial intelligence is performance. The capability is asserted in the marketing, demonstrated in the controlled demo, and quietly backstopped by humans, by narrower function, or by nothing at all when the deployment meets reality. The buyer pays for the asserted capability. The buyer inherits the gap.
Presto is not an isolated case. It is a category, and the category now has an enforcement record.
Consider Workado. The company sold an AI Content Detector advertised as 98 percent accurate at identifying AI-generated text. In 2025, the Federal Trade Commission found that the tool’s accuracy on general-purpose content was closer to 53 percent. The model had effectively been trained and tested on a narrow band of material, academic abstracts and ChatGPT output, and then sold as a general detector. Fifty-three percent is not a detector. It is a coin flip with a confidence interval attached. The FTC required the company to stop making the accuracy claim and to hold competent, reliable evidence for any efficacy claim going forward. The point is not that the number was off by a few points. The point is that the headline capability and the actual capability were different products, and only one of them was for sale.
Consider Cox Media Group. The company marketed a service called Active Listening, which it claimed used AI to listen to consumers’ real-time conversations through the microphones on their devices in order to target advertising. In May 2026, the FTC found the service did not do what it advertised. It did not listen. It used no voice data. What it actually did, according to the Commission, was resell email lists from data brokers at a markup, wrapped in a story about AI that listens. The AI capability was not understated or overstated. It was absent. The performance was the entire product.
These three are not the same offense. Presto had a real product that needed far more human help than it claimed. Workado had a real product that worked far less well than it claimed. Cox had a claim with no product underneath it. But they share a single structural feature, and that feature is the subject worth your attention: in every case, the buyer had no practical way to verify the capability before relying on it. The claim was checkable only by the regulator, only after the fact, only because someone complained.
That is the real exposure, and it does not sit with the vendor. It sits with the organization that bought the claim.
When a vendor overstates what its AI does, the immediate damage lands on the vendor in the form of an eventual enforcement action or a refund. But the operational damage lands on the customer, and it lands earlier and lasts longer. The restaurant that deployed Presto Voice told its own operations the drive-thru was automated and staffed accordingly. The firm that bought a 98 percent detector made decisions, about students, about hiring, about content, on a tool that was right about half the time. The advertiser that paid for Active Listening built a campaign on a capability that never existed. In each case the vendor made the claim and the customer made the decision. Accountability for the decision does not transfer back to the vendor because the vendor exaggerated. It stays with the organization that acted.
This is the through line from a piece earlier this week. The deployed system has an owner whether or not the organization ever decided who that owner is. When the system underperforms its claims, the question is not only whether the vendor lied. The question is who in the buyer’s organization was responsible for confirming the capability was real before the business depended on it. In most organizations, the answer is the same as it is for AI governance generally. No one specific. The vendor’s marketing was treated as the verification step. It is not one.
The defense the industry offers is human oversight. The vendor will say a human is in the loop. The buyer will say a person reviews the output. This is supposed to be the reassurance that makes the capability gap survivable. Presto is the precise reason it is not.
Presto is, on paper, the human-in-the-loop model working. Humans intervened on more than 70 percent of orders. The humans were there. But the humans were not the safeguard the phrase implies. They were the undisclosed mechanism that made a falling-short product look like a working one. “Human in the loop” was not a control that caught the AI’s failures. It was the labor that hid them. The presence of a human in the workflow told the buyer nothing about whether the AI worked, because the human was doing the work the AI was credited for. Oversight that exists to cover a capability gap is not oversight. It is the gap, staffed.
So the operative question for any organization buying AI capability is not whether there is a human in the loop. Of course there is. The question is what that human is actually doing. Are they exercising judgment over the AI’s output, with the authority and the time to reject it when it is wrong? Or are they silently completing the work the AI cannot, while the organization reports the function as automated? Those are opposite conditions that look identical on an org chart and in a vendor deck. The first is governance. The second is Presto.
Regulators have started to close on the vendors, and the trajectory matters. The SEC brought its first AI-washing cases in March 2024 against two investment advisers for overstating their use of AI, with penalties totaling 400,000 dollars. It reached its first public-company AI-washing case with Presto in January 2025. The FTC ran Operation AI Comply in September 2024, a coordinated sweep against deceptive AI claims, and has continued through Workado in 2025 and Cox Media Group in 2026. The enforcement is real and it is accelerating. But enforcement arrives late, lands on the vendor, and does nothing to unwind the decisions the buyer already made on a claim that turned out to be performance. By the time the SEC documents that 70 percent of the orders needed a human, the customer has already run the operation as if they did not.
The enforcement record should not be read as reassurance that the system is self-correcting. It should be read as a published list of capability claims that turned out to be false, assembled by the only parties with subpoena power, because no one downstream could check the claims on their own. For every case that reaches an enforcement action, the more relevant population is the deployments where the gap exists, no one complained, and the buyer is still operating on the asserted capability rather than the real one.
There is a discipline that closes this, and it is unglamorous. Before an organization relies on an AI capability, someone inside it has to own the question of whether the capability is real, in this organization’s actual conditions, not in the vendor’s demo. That means a defined owner for vendor capability verification, distinct from procurement and distinct from IT security. It means treating the vendor’s accuracy claim as a hypothesis to be tested against your own data before deployment, not a specification to be accepted. It means knowing, specifically, what the humans in your AI workflows are doing: governing the output, or quietly producing it. None of that is exotic. It is the same accountability discipline that the rest of enterprise risk already takes for granted, applied to a vendor category that has so far been allowed to grade its own capability.
The vendors will keep performing capability, because the market keeps paying for the performance and only rarely tests the substance. That will not change on the vendor’s side until testing the substance becomes the buyer’s default. The organizations that build that discipline now are the ones that will not appear, eighteen months from now, in the next enforcement release, explaining to a regulator why they ran the business on a capability that needed a human all along.
The Evolving Mindset publishes weekly on AI governance and organizational accountability. If you are relying on a vendor’s AI capability claim you have not independently tested, that is a specific and answerable exposure. Reach out through the link in the profile.
Sources and notes
Presto Automation: SEC administrative proceeding, settled January 14, 2025 (Release 33-11352). Findings include that more than 70 percent of orders required human intervention, 100 percent at certain locations, and that the AI was owned and operated by a third party. Cease-and-desist; no civil penalty, citing the company’s financial condition and cooperation.
Workado, LLC (formerly Content at Scale AI): FTC final order approved August 2025. Advertised 98 percent accuracy on its AI Content Detector; FTC found real-world accuracy on general content of approximately 53 percent. Order requires competent and reliable evidence for efficacy claims.
Cox Media Group “Active Listening”: FTC settlement announced May 2026; total monetary relief of approximately 930,000 dollars across Cox Media Group and two related firms. FTC found the service did not listen to conversations and used no voice data.
SEC first AI-washing actions: Delphia (USA) Inc. and Global Predictions Inc., settled March 2024, penalties totaling 400,000 dollars (SEC release 2024-36).
FTC Operation AI Comply announced September 25, 2024.
All figures verified against primary or top-tier sources during research. Nothing in this piece constitutes legal or investment advice.