When you buy an AI liability policy, the underwriter has to price it. To price it, they have to know what AI you run, where it touches a consequential decision, and what stands between the model and the outcome. So before they quote you, they inventory your AI and score its risk. They ask which systems are in production, which ones face customers or regulators, what the human review step is, what gets logged, and who is accountable when it is wrong. Then they turn that into a number and put capital behind it.

Read that sequence again, because there is something uncomfortable in the order of operations. The first complete inventory of your AI exposure, the first honest assessment of the gap between what your systems do and what your governance controls, may not be produced by your risk team, your CISO, or your board. It may be produced by an outside party that is betting money against you, and keeping the map.

That is the move worth naming. Not that insurance is bad. That in buying it, most companies will outsource the first act of governance to the one party whose interests are not aligned with theirs.

The inventory is not paperwork. It is the governance.

There is a habit of treating an AI inventory as a compliance chore, a spreadsheet someone fills in before an audit. That gets the causality backwards. You cannot govern what you have not named. The act of listing every place AI touches a decision, and writing down who owns each one, is not preparation for governance. It is governance, in its first and most load-bearing form. Everything downstream, the review steps, the escalation paths, the logging, depends on that list existing and being owned inside the company.

When the underwriter builds that list for you, you get a version of the artifact without the thing that made it valuable. You learn your number. You do not build the capability. You hold a map drawn by someone who will use it to price your failure, not to prevent it. And you still do not have, inside your own walls, a named person who can stand in front of a regulator and say: I own this decision, here is how it is controlled, here is the record.

That last sentence is the whole game, and it is the thing no policy contains.

Subscribe now

What coverage settles, and what it cannot

A policy settles a loss. When the AI-driven decision goes wrong and someone is harmed, the policy, if the claim is covered and not excluded, pays money. That is real value and I am not waving it away. But walk the failure forward past the payment. The regulator opens an inquiry and asks who approved the output that broke the rule. The board asks who was accountable for the system. The customer, and eventually a court, holds your company to what your AI did, exactly as a tribunal held Air Canada to its chatbot. At none of those tables does the check from your carrier answer the question being asked. The question is about ownership of a decision. Insurance is about ownership of a cost. They are different objects, and only one of them can be handed to someone else.

This is the error the premium invites you to make. It is designed, honestly, to feel like resolution. You had an exposure, you paid, the exposure is handled. But you handled the accounting for the exposure. The exposure itself, the ungoverned decision running in production, is exactly where it was before you wrote the check. You are now insured against its cost and no better governed against its occurrence.

The premium is a score. Read it as one.

Treat it as a governance score issued by a party that cannot be spun. Your vendor wants to sell you capability. Your team wants to report progress. The underwriter wants to be paid correctly for real risk, which means their assessment of your AI governance is the least flattering and most reliable one you will get. A high premium is not only a cost to negotiate down. It is a reading. It says the distance between the AI you run and the control you have around it is large, and here is what that distance is worth in dollars.

The companies that get this right treat the quote as a mirror and go do, internally and before the renewal, the work the underwriter was about to do for them. Inventory the systems. Name an owner for each consequential decision. Put a human review step where the output is high-consequence. Record what the system did and why. Not mainly to lower the premium, though it will, but to hold the thing the premium cannot: the accountability the claim will eventually demand a name for.

Because the sequence only runs one of two ways. Either you inventory your AI and decide who owns each decision it makes, or an insurer does it first, prices your gap, and hands you a policy that pays for the failure while leaving the ownership exactly where it always was, unassigned and now running at scale.

The underwriter is not your governance function. They are a signal that you need one, arriving with a number attached and a map you should have drawn yourself.

AI operates. You own the decision. Including the decision to let your underwriter be the first to find out where your AI actually lives.

Share

Fellowship Intelligence defines and installs the governance this piece describes: named ownership for every consequential AI decision, in place before the underwriter or the regulator prices its absence. The entry point is a Diagnostic that maps where AI touches decisions across your organization and who is accountable for each one. It is $500, credited in full toward a Decision Brief signed within 30 days.

Sources:

1. McKinsey, State of AI (Nov 2025) — the 88% / 78% figure: https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai

2. ISO / Verisk exclusions (CG 40 47/40 48/35 08) — Verisk issuer note · Gallagher explainer · CG 40 48 form PDF

3. Carriers filing exclusions — Financial Times · InsuranceNewsNet

4. Mayflower & Hadron program — Business Wire · Reinsurance News

5. Moffatt v. Air Canada, 2024 BCCRT 149 — CanLII full decision

6. Walters v. OpenAI (GA, May 19, 2025) — Loeb & Loeb · Cleary Gottlieb

The question is simple to state and expensive to ignore: when AI inside your operation produces a bad outcome, whose loss is it?

For two years the honest answer inside most organizations has been a shrug spread across a committee. IT owns the tool. Legal owns the contract. A working group owns the policy document. No one owns the decision the AI actually makes. That was survivable while AI was a pilot. It stopped being survivable once 88% of organizations reported using AI in at least one function, up from 78% a year earlier. The pilots became infrastructure. The ownership question did not get answered. It got deployed around.

The insurance market just answered it for you, in two moves.

Subscribe now

Move one: your existing policy stopped covering AI

In January, Verisk’s ISO, the body that writes the standard forms most commercial insurers build on, released new endorsements, CG 40 47 chief among them, that let carriers exclude generative AI from commercial general liability. This is not a fringe product. By spring, carriers including W.R. Berkley, Chubb, Travelers, Berkshire Hathaway, and Cincinnati Financial had filed to adopt AI exclusions, and regulators approved the large majority. The coverage many companies assume protects their AI use is being removed by endorsement, quietly, at renewal.

Sit with what an exclusion is. It is an insurer looking at a category of risk and deciding it is too uncertain, too correlated, or too large to keep inside a general policy at the current price. That is not indifference to AI. It is a judgment that the risk is real, material, and separable enough to name and carve out.

Move two: a market opened to price it directly

In June, Mayflower and Hadron launched what they describe as the first dedicated affirmative AI liability program in the US: explicit coverage across directors and officers, employment practices, and errors and omissions, written for enterprise AI. The launch is not the story. The underwriting method is. They price the policy by scoring the deployer’s AI for specific failure modes, bias, drift, and hallucination, against NIST and ISO frameworks.

Read that plainly. An insurer has built an actuarial model of your AI risk. Not the model maker’s. Yours, the deployer’s. They are pricing the gap between the AI you run and the governance you have wrapped around it, and they are confident enough in that price to put reinsurance behind it.

Put the two moves together and the logic is unambiguous. The market pulled AI out of the pool where risk is shared and vague and moved it into one where risk is named and individually priced. That only happens to a risk with a clear owner. You cannot price what you cannot attribute. The underwriters attributed it. To you.

What the underwriter actually asks

To get that price, you fill out a questionnaire. Strip away the formatting and it asks a handful of questions, in substance:

Where is AI used in your operation, and which of those uses touch a customer, an employee, or a regulated decision?

For each one, who is the named person accountable when it is wrong?

What is the human review step before the output leaves the building?

What do you log, and could you reconstruct why the system did what it did?

Not one of those is a technical question about the model. Every one is a governance question about you. The underwriter is not curious how the model works. It is establishing whether anyone is holding the wheel, because that is what decides whether it pays out, and how often.

Which is why two companies running the identical models will not get the identical quote.

Firm A and Firm B

Firm A and Firm B use the same three vendors and the same foundation models. Firm A can name an owner for each consequential AI use, can show the review step, and can produce a log. Firm B connected the tools, saw the productivity, and never assigned the accountability, because it felt administrative and no one made it anyone’s job.

The underwriter cannot see intentions. It can see controls, or their absence. Firm A prices a small gap. Firm B prices a large one, or cannot complete the questionnaire and does not qualify. Same models, same industry, different number, because the risk was never really about the model. It was about who owns the decision the model makes.

So the quote is a diagnostic you did not order. An outside party, with money at stake and no incentive to flatter you, has assessed your AI governance and returned a number. If it comes back high, that is not the insurer being cautious. It is the insurer telling you what your ungoverned surface is worth.

The courts drew the line first

None of this is the insurance industry inventing a position. It is the industry catching up to one the law has been sketching for a while.

When Air Canada’s customer-service chatbot told a passenger he could claim a bereavement fare after the fact, and the airline argued it should not be bound by its own bot’s error, the British Columbia Civil Resolution Tribunal rejected the argument outright. Air Canada was responsible for the information on its website, whether it came from a static page or a chatbot. The bot was not a third party. It was the company. That is the cleanest statement of the principle: when your AI speaks to a customer, you own what it says.

A separate case cuts the other way, and it pays to state it precisely, because it is narrower than it first looks. When Mark Walters sued OpenAI over a false statement ChatGPT generated about him, a Georgia court granted OpenAI summary judgment on several independent grounds, one of them that OpenAI’s repeated, prominent disclaimers meant no reasonable reader would take the output as a statement of fact. That is a trial-level defamation holding, not a rule about who owns AI liability. Read for direction rather than doctrine, the two cases point the same way: the maker that discloses its limits steps back, and the party that puts the output to work answers for it. Insurance is now priced on that same line.

What the premium is actually measuring

Here is the part to carry into your next renewal conversation.

Insurance is a fine thing to buy. It turns a volatile loss into a fixed cost, and for AI exposure that is prudent. But be precise about what it does, because the marketing will blur it. A policy transfers the cost of a failure. It does not transfer the ownership of the decision that caused it.

Walk the failure past the payment. The regulator opens an inquiry and asks who approved the output. The board asks who was accountable. The customer, and eventually a court, holds your company, not your carrier, to what your system did. At none of those tables does the check answer the question being asked. Coverage answers the invoice. It does not answer the question.

The market has done you an inadvertent favor. It has put a dollar figure on a thing your organization kept treating as unpriceable, and therefore ignorable. The exposure was always yours. Now it has a number next to it, and the number moves for exactly one reason.

The only move that lowers it is the one no policy can make for you: decide, before the system runs, who owns the decision it makes.

AI operates. You own the decision. The underwriter already assumed you did, and priced it.

Schedule a call

Thursday’s piece goes to the sharp edge of this, and it is subscriber-only: how the underwriter ends up doing your AI inventory for you, and why the check that pays the claim still cannot answer for the decision. Subscribe and it will reach you.

Sources

• “88% use AI, up from 78%”: McKinsey, The State of AI (Nov 2025)

• ISO exclusions CG 40 47 / 40 48 / 35 08 (eff. Jan 1, 2026): Verisk / ISO general liability filing

• Carriers excluding AI (W.R. Berkley, Chubb, Travelers, Berkshire Hathaway, Cincinnati Financial): Financial Times

• Mayflower & Hadron AI liability program (June 24, 2026): Business Wire

• Moffatt v. Air Canada, 2024 BCCRT 149: CanLII, full decision

• Walters v. OpenAI (Gwinnett County, GA, May 19, 2025): Loeb & Loeb

Notes: carrier-level specifics are press-sourced (primary record is each carrier’s SERFF state filing). Walters is a trial-level defamation ruling, read for direction, not a liability-allocation holding.

For most of corporate history, business-critical technology arrived through an institution. Procurement bought it. Legal reviewed the contract. Someone signed off and owned it. The controls came attached, because the path it traveled had checkpoints built in.

AI broke that path. It arrives from an app store. A browser tab. A free tier. A tool someone found on a Tuesday. No procurement step, no contract review, no owner. The technology shows up without the institution that used to govern it.

So when people ask what to put in their AI vendor contracts, the honest answer is that the contract was rarely the problem. A competent legal team handles standard terms. The problem is that the decision to use the tool never reached legal in the first place.

That’s the gap. Not the contract. The missing institution around it.

The fix isn’t a better agreement. It’s deciding, on purpose, what data is allowed into which tools and who signs off on a new use case before it’s running. That used to happen on the way in. Now it only happens if someone decides to make it happen.

First shared on LinkedIn

That same day, an email landed in my inbox. Not as someone who reads policy papers, but as a Google Workspace administrator. It announced a new feature in Google Vids: personal avatars, built from a user’s own face and voice. Rolling out July 14, 2026. On by default for every eligible user. I could turn it off, but only if I knew to look, and only if I understood why it mattered.

Illinois and Texas have laws that govern exactly this kind of data collection. Illinois’ Biometric Information Privacy Act and Texas’ Capture or Use of Biometric Identifier statute both require affirmative, informed consent before collecting biometric identifiers, which include face geometry and voiceprints. Default-on with an administrative opt-out does not satisfy either statute. Consent must be prior and affirmative. (Note: this is not legal advice. Organizations with users in Illinois or Texas should consult counsel before July 14.)

Now hold Google’s own standard against Google’s own product. The paper says if something is illegal without AI, it is illegal with AI. Collecting face geometry and voiceprints without prior, affirmative consent is illegal without AI in Illinois and Texas. Switching it on by default does not become lawful because the feature is powered by AI. By the paper’s own rule, the policy argument and the product notice, published the same day, point in opposite directions.

Before setting that contradiction aside, it is worth examining the frontier solution Google is proposing. The paper calls for a Frontier AI Regulatory Organization, modeled on FINRA and NERC. The governance design is in the paper’s own language: “The FARO board would include a combination of independent directors and industry representatives.” Companies with frontier models would become FARO members. FARO would draw its standards from bodies like the Frontier Model Forum, which Google co-founded alongside Microsoft, OpenAI, and Anthropic. The baseline framework FARO would audit against is explicitly modeled on Google DeepMind’s own Frontier Safety Framework. Audit reports would be submitted confidentially to FARO, not released publicly.

Read the structure. Google proposes the body, Google funds it as a member, Google sits on its board as an industry representative. Google co-founded the standards body whose guidelines FARO would apply. Google’s own safety framework is the explicit template. Audit results stay inside the structure.

This is not a regulatory design that holds frontier labs accountable to an external standard. It is a regulatory design that seats frontier labs inside the accountability structure itself. The “independent” part of the board is balanced, by design, against industry voices who have an explicit interest in speed to market. That phrase appears verbatim in the board composition paragraph.

The body that would govern the most powerful AI systems would be funded by those systems’ developers, staffed from the same talent pool, governed by frameworks those developers wrote, and answerable to a board those developers help populate.

This is not primarily a story about Google. It is a story about the governance gap.

Every enterprise Google Workspace administrator who received that email has until July 14 to act. Most won’t recognize the legal exposure. Not because they are negligent, but because they don’t have an internal system that connects a product rollout notice to a biometric consent obligation. That connection requires someone who understands both the product and the law simultaneously, knows which states your employees are in, and has authority to act before the default kicks in. Most organizations don’t have that person, that process, or that infrastructure.

Google’s paper proposes solving the governance problem at the federal level, through a frontier regulatory body focused on the most advanced AI models. What arrived in my inbox is not a frontier problem. It is an organizational problem. The risk arrived as a routine admin notice, with a two-week window to act.

The governance framework that would have caught this doesn’t live in Washington. It lives inside the organization, in the layer between the tool and the people responsible for using it responsibly. It is a policy layer, a workflow control, an ownership structure, an escalation path. It is the system that asks, when a vendor pushes a new feature: what does this collect, where does it go, what law governs it, and who in this organization is accountable for the answer?

Most organizations don’t have that system. Google’s paper doesn’t propose building it. And a frontier AI regulatory organization, if it ever materializes, won’t require it.

The governance gap isn’t at the frontier. It’s in the inbox.

Subscribe now

Sources

Google, “A Pragmatic Approach to AI Governance in America,” June 25, 2026. The paper argues existing law is sufficient for widely-deployed AI: “the federal government does not need new regulatory regimes” (p.13), and “if something is illegal to do without AI, it’s illegal to do with AI” (p.13). It proposes a Frontier AI Regulatory Organization (FARO) modeled on FINRA and NERC, industry-funded, with a board combining independent directors and industry representatives. https://blog.google/company-news/outreach-and-initiatives/public-policy/white-paper-ai-regulation/

Google Workspace admin notice, received June 25, 2026 (archived .eml). Google Vids personal avatar feature, built from a user’s face and voice, default-on July 14, 2026. Public corroboration: Google Workspace Updates. https://workspaceupdates.googleblog.com/2026/06/enhanced-ai-avatars-vids.html

Illinois Biometric Information Privacy Act (740 ILCS 14) and Texas Capture or Use of Biometric Identifier Act (Tex. Bus. & Com. Code 503.001). Both require informed, affirmative consent before collecting biometric identifiers, including face geometry and voiceprints.

When a vendor pushes a new feature into the tools you already run, who in your organization connects that notice to the law that governs it, and has the authority to act before the default kicks in? Tell me in the comments.

This is the question my work begins with. The governance gap is rarely at the frontier. It is in the layer between the tool and the people accountable for using it. That gap is what I help close.

Schedule a conversation

I gave a talk on this at Tech Alley Henderson, a practical health check on putting AI to work inside a business without giving up control, accountability, or your data. It covers where the real liability sits, including the Air Canada chatbot case, where a Canadian tribunal held the company responsible for what its bot told a customer. It covers Shadow AI, the unapproved tools already running inside most organizations. And it covers what your free, paid, and enterprise AI accounts actually protect, because the difference is in the contract, not the interface.

If you are responsible for decisions, standards, or outcomes in an organization that uses AI, this is the orientation.

A note on what this is. This is the foundational overview, the wide version. The Evolving Mindset goes the other way each week. Examining the structural problems, in writing, for the people who own the decision. If the talk is the map, the writing is the turns you actually have to take.

Subscribe now

No noise, no hype, no generic AI content. Publishing several times a week.

Nothing about her qualifications changed. Nothing about the role changed. The only thing that moved was a number that, by law, is not allowed to decide whether she gets a chance. The software had been configured to reject women over 55 and men over 60, and it did so silently, to more than two hundred applicants.

That company was iTutorGroup. In August 2023 it paid $365,000 to settle the EEOC’s first-ever AI discrimination case. And the resubmitted application is the most important fact in it, because it shows this was never a mysterious emergent bias buried in a black box. It was a rule. A person chose the threshold. The software just enforced it at scale.

Subscribe now

The move has a name

What iTutorGroup did when challenged is a pattern now common enough to name: decision-laundering. An organization runs a consequential decision through an AI, and when the decision turns out to be illegal or indefensible, it points at the system. The algorithm did it. As if the software, and not the company that bought it, configured it, and switched it on, had made the call.

Decision-laundering is attractive for the same reason money-laundering is. It puts distance between an actor and a consequence. The model feels objective, so the bias reads as math rather than choice. Responsibility diffuses: the quants own the model, not the board; HR treats the tool as a vendor product, not as a hiring decision; everyone can point somewhere else. The decision still gets made. It just arrives with no fingerprints.

The problem is that regulators do not accept the laundering. They look straight through the model to the outcome, and then they look for the owner.

How the laundering actually happens

To see why “the algorithm did it” is not just legally weak but factually false, look at how the bias gets in. It is almost never the model’s invention. It is the organization’s own, passed through and given a clean face.

It enters two ways. The first is inheritance. A model trained on a company’s past decisions learns the patterns in those decisions, including the discriminatory ones. Feed it years of who you hired and who you rejected, who you approved and who you turned away, and it will reproduce that history with statistical confidence and present it as prediction. The Massachusetts Attorney General said exactly this about the lender Earnest: its models were trained, in the AG’s account, on arbitrary and discretionary past human decisions. Human bias went in, was encoded as statistics, and came back out wearing the authority of an algorithm.

The second is proxies. A model does not need the protected trait to discriminate on it. It finds correlates, a zip code, a school, a cohort-default-rate variable, an immigration-status flag, that stand in for age or race or national origin. This is why “we never told it to consider age” is not a defense. The system found age anyway, because age was latent in the data it was given. Removing the label does not remove the pattern.

Then framing finishes the job. Treat the system as a neutral tool rather than a decision-maker, and no one feels responsible for auditing it. The output looks like math, so it escapes the scrutiny a human manager making the same call by hand would attract. iTutorGroup’s screening was treated as a tool. It was a decision, made two hundred times.

The second case makes the remedy explicit

That Earnest matter is worth reading in full, because of what the regulator demanded. In July 2025 the Massachusetts Attorney General reached a $2.5 million settlement with the student lender over AI underwriting alleged to produce disparate harm to Black, Hispanic, and non-citizen applicants, including through an immigration-status knockout rule, with the company faulted for never testing its models for disparate impact.

Now look past the fine. The AG ordered Earnest to implement a detailed corporate governance structure, develop and maintain written policies for responsible AI use, and report on its compliance on an ongoing basis. The state’s remedy for a laundered decision was to force the company to install an owner for it. That is the entire argument of modern AI governance, ordered by a law-enforcement office as a consent term. The regulator did not just punish the bias. It prescribed accountability.

Why “the algorithm did it” has no standing

Put the two cases together and the rule is plain. In both, an AI made a consequential decision about a human being. In both, the decision was unlawful. And in both, the fact that software executed it changed nothing about who was responsible. The deployer owned the outcome, fully, exactly as if a manager had made the same call by hand.

This should not surprise anyone, because in both cases a human did make the call. Someone set the age threshold. Someone wrote the knockout rule. Someone chose the training data and decided not to test it. The AI was the instrument, not the author. “The algorithm did it” is the corporate cousin of “I was only following orders,” and it carries the same weight, which is none.

What the model actually buys an organization is not objectivity. It is speed and distance: the same biased decision, made faster, against more people, with a story attached about why no one is to blame. That is not a mitigating factor. It is an aggravating one, because the harm is now systematic, and the people running the system were more confident in it precisely because it looked technical.

This is also why the defense is being closed across domains, not just one. Employment regulators, state attorneys general in lending, and securities regulators on AI disclosures are converging on the same position: the entity that deploys the system answers for what it decides. There is no agency where “a model did it” has found purchase.

The line between automation and laundering

None of this is an argument against automating decisions. Plenty of consequential decisions can and should be assisted or made by machines. The argument is about who answers for them.

Automation is legitimate when a named human owns the rule the system applies, can explain it, and can defend the outcome to a regulator, a customer, or a court. It becomes laundering the moment no one can. The dividing line is not how advanced the model is or how much data trained it. It is whether a person stands behind what it decides. One is delegation, which is normal and fine. The other is evasion wearing the costume of efficiency.

The governance question, again

The instinct after reading these cases is to ask whether the model was accurate, or fair, or well-tested. Those are real questions, but they are not the first one. The first question is who owns the decision the model makes.

Because the failure in both cases was not fundamentally a modeling failure. It was an ownership vacuum. No named human was accountable for the outcomes the system produced, empowered to see the rule it applied and answer for it. Into that vacuum the model became the de facto decision-maker, and “the algorithm” became the answer to every question about why.

So the test for any organization is concrete. For every consequential decision your AI touches, hiring, lending, pricing, eligibility, termination, name the human accountable for the outcome and ask whether they can explain the rule the system is applying. Not the architecture. The rule. Why this applicant and not that one. If the answer is “the model handles that,” you have not automated a decision. You have laundered one. And as iTutorGroup and Earnest both learned, you own it anyway.

The algorithm did not do it. Someone chose to let it, and chose not to look. That someone is you, and a regulator will find them even if your org chart cannot.

AI operates. You own the decision, including the one you handed to a machine.

For every consequential decision your AI touches, can you name the human who owns the outcome and explain the rule? Tell me where the gaps are in the comments.

This is the question my work begins with. When AI makes consequential decisions and no named human owns them, that gap is what we help close.

Schedule a conversation

Sources

U.S. Equal Employment Opportunity Commission, “iTutorGroup to Pay $365,000 to Settle EEOC Discriminatory Hiring Suit,” August 9, 2023. The EEOC’s first AI-discrimination settlement. iTutorGroup’s hiring software automatically rejected female applicants over 55 and male applicants over 60, violating the Age Discrimination in Employment Act. The charging party was rejected, then offered an interview after resubmitting with only a more recent date of birth. https://www.eeoc.gov/newsroom/itutorgroup-pay-365000-settle-eeoc-discriminatory-hiring-suit

Office of the Massachusetts Attorney General, “AG Campbell Announces $2.5 Million Settlement With Student Loan Lender For Unlawful Practices Through AI Use,” July 2025. Earnest Operations’ AI underwriting models allegedly produced disparate harm to Black, Hispanic, and non-citizen applicants, including via an immigration-status knockout rule, with no disparate-impact testing and models trained on prior discretionary human decisions. The settlement requires a formal AI governance structure, written policies, and ongoing compliance reporting. https://www.mass.gov/news/ag-campbell-announces-25-million-settlement-with-student-loan-lender-for-unlawful-practices-through-ai-use-other-consumer-protection-violations

The code runs. It demos well. It shipped. And no one on the team can tell me what it actually does at the level that matters, because an AI wrote it and an AI was the only thing that checked it. The pattern is consistent enough now that I can name the moment it goes wrong, and it is not a moment of malice or even carelessness. It is a sentence said with confidence: I let the AI write it and the AI review it, so I am good.

That sentence is the failure. Not the AI, not the speed, not the founder moving fast. The sentence.

Because it describes a closed loop. The author and the reviewer are the same machine. A system grading its own output is not a review, it is a second opinion from the same witness. And the human who was supposed to sit outside that loop has quietly stepped out of it, reassured by the one thing that should reassure no one: it runs.

Subscribe now

Running tells you almost nothing. We now have the data to say that plainly. Veracode tested AI-generated code from more than a hundred models across four languages and found that 45% of it shipped with security flaws from the standard OWASP list. In Java the failure rate was 72%. And the most important finding is the one that gets the least attention: as the models improved, they got better at writing code that works and no better at writing code that is secure. The two skills came apart. So “it runs” and “it is sound” are now separate questions, and the loop only ever answers the first.

The human cost compounds it. Stanford researchers found that developers using AI assistants wrote less secure code than those working by hand, and were more confident in its security while doing it. Sit with that. The tool does not just let the review lapse. It produces the feeling that no review was necessary. Confidence goes up while correctness goes down, which is the exact condition under which no one thinks to look.

And often, by the time something goes wrong, the person who shipped it could not look if they wanted to. They cannot read the headers on the code blocks to see what each step actually instructs. This is not a competence insult. It is what the default manufactures. When the loop is closed tightly enough, the human is removed from it so completely that they lose the literacy to re-enter. You cannot own what you cannot read.

Here is where this stops being the builder’s problem and becomes yours.

None of this stays with the shop that built it. The artifact leaves. It ships as a product you license, or as a vendor implementation you approved, and it enters your environment, frequently a regulated one, carrying every gap no human ever caught. And when it fails, the liability does not travel back to the builder. It lands on the deployer. The principle is already settled in plain terms: a company is accountable for what its AI produces, regardless of who produced it. When Air Canada’s chatbot invented a refund policy, a tribunal held the airline to it. The defense that a system, not a person, made the error did not survive. No regulator and no court is going to accept “the AI wrote that part” as a reason the obligation does not apply to you.

So consider the chain honestly, because no one in it did anything wrong. The vendor shipped capability to the customers who contracted for it. The builder kept the product moving. The buyer enabled what they bought. Each act is reasonable. None of them is a review. The review is a separate thing, and it belongs to a named human who read the code and can answer for it. If no one in that chain did it, it did not happen, and now it is running inside your stack.

This is the same question I asked earlier this week, arriving from the other direction. Inside your own operation, the question was who decides what is critical. Through your supply chain, it is narrower and harder: who read the code you are about to run, and can they explain it? One is about the decisions your systems make. This is about the decisions someone else’s system already made, that you are about to inherit.

The inventory question has not changed, it has only moved upstream. Where is AI-built code operating inside what you deploy, and who is the named human accountable for having actually read it? “It runs” is not an answer. “The AI reviewed it” is not an answer. The only answer that holds up, when the thing fails and the question is who owns it, is a name.

AI operates. You own the decision. Including the decision to run code no one ever read.

Where is AI-built code running inside what you deploy, and who actually read it? If you cannot name the person, that is the gap. Tell me what you find in the comments.

This is the question my work begins with. When AI-built code reaches a regulated environment and no named human owns it, that gap is what I help close.

Schedule a conversation

Sources:

Veracode 2025 GenAI Code Security Report. Veracode tested AI-generated code from more than 100 models across four languages and found that 45% shipped with OWASP-class security flaws, rising to 72% in Java. The report’s key finding: as models improved, they got better at writing code that works and no better at writing code that is secure.
https://www.veracode.com/resources/analyst-reports/2025-genai-code-security-report/

Perry, Srivastava, Kumar, and Boneh, “Do Users Write More Insecure Code with AI Assistants?” (Stanford, ACM CCS 2023). The first large-scale user study on the question. Developers with an AI assistant wrote measurably less secure code, and were more likely to believe their code was secure than those working without one.
https://doi.org/10.1145/3576915.3623157

Moffatt v. Air Canada, 2024 BCCRT 149. A Canadian tribunal held Air Canada liable for a refund policy its chatbot invented, rejecting the argument that the company was not responsible for what its own AI told a customer. The clearest statement yet that the deployer owns the output.
https://www.canlii.org/en/bc/bccrt/doc/2024/2024bccrt149/2024bccrt149.html

Set aside the national-security politics, because the politics are not the lesson. The lesson is structural, and it is uncomfortable. The off switch on the AI your business runs on was never in your hands. It sits with the vendor, and now, we have learned, with the government standing behind the vendor. You are a tenant in a building someone else can lock.

Subscribe now

For most organizations, that is not a manageable risk, because they cannot even see it coming.

The blind spot, measured

On June 17, IBM’s Institute for Business Value, with Oxford Economics, published a study of 1,000 senior executives across 16 countries and 17 industries. The headline number is the one that should keep operators up at night: 91% say they do not fully understand their organization’s dependencies across AI vendors, models, and infrastructure.

The supporting figures turn that from an awareness gap into a continuity problem. 71% say switching their primary AI vendor or model would be difficult. Companies reported an average of six AI-related disruptions over the past two years, most of them traced to vendor services. And 81% say a seven-day outage of an AI vendor would cause severe or critical disruption, the kind that effectively stops operations.

Hold those four facts next to the shutdown. A model can be switched off by someone other than you. Most companies cannot see which models they depend on. They cannot easily switch off the ones they can see. And the great majority would be in crisis within a week. The June shutoff was not an anomaly that proves the system works. It was a live demonstration of a risk that the data says almost no one is positioned to absorb.

This is concentration risk, and you already know how to manage it

Strip away the word “AI” and this is a category every competent operator already governs. No serious business runs on a single supplier with no second source, no continuity plan, and no clear inventory of what it depends on. Procurement does not allow it. Finance does not allow it. IT continuity does not allow it. You would treat “we are not sure what we rely on, and we could not survive a week without it” as an audit finding requiring immediate remediation.

Yet that is precisely the posture the IBM data describes for the models now sitting under underwriting, claims, customer support, research, and forecasting. The discipline that is routine everywhere else in the enterprise has not reached the AI layer. Not because the risk is exotic, but because the adoption outran the governance. The models arrived faster than anyone inventoried them.

Why the dependency hides

The reason 91% cannot see their dependencies is not negligence. It is the shape of how AI enters a company.

AI rarely arrives as a logged decision with an owner attached. It shows up as a feature inside a tool you already license, switched on through a product update you did not evaluate as an AI adoption. It gets embedded three layers deep in a workflow by a team trying to move faster. The model under your support queue, your document review, your risk scoring, was often never selected in a meeting. It accumulated.

And what accumulates without a decision has no owner. No owner means no inventory, because no one was assigned to keep one. No fallback plan, because no one was named to write it. No continuity drill, because no one is accountable for the outcome. The dependency is invisible for the same reason it is ungoverned: it belongs to no one.

The tempting wrong answer

The instinct, once a leadership team sees this, is to buy something. A control plane, an AI gateway, an observability layer. The market is flooded with them right now, and they are not useless. But notice what they do not do. A tool that watches your models does not reduce your dependence on them. A dashboard that shows your usage does not give you a second source. Adding a layer of software to manage the problem leaves the concentration exactly where it was, and adds one more vendor you now also depend on.

Visibility tooling can support governance. It cannot be governance, because the thing missing is not a screen. It is a decision, and a person to own it.

What governance actually requires here

Treat the AI your operations depend on as critical infrastructure, and govern it the way you govern every other system you could not afford to lose. In practice, that is three things, owned by named humans, in writing.

An inventory. Where is AI operating, and which vendor and model sits beneath each process that matters. This is the 91% problem, and it is solvable in weeks, not years.

A continuity plan for each dependency that counts. What happens if this model goes dark or goes wrong. Is there a fallback, a second source, a manual path, and who executes it. This is the direct answer to the seven-day-outage question, and to the shutdown.

An owner for each. Not a committee. A person with the authority to act when the model fails or disappears, who keeps the inventory and the plan current as both change.

There is a number in the IBM study that makes the business case for all of this. Organizations with the most advanced AI control capabilities protect more than half of their operating profit from AI-driven disruptions. Governance here is not an insurance cost. It is a margin you keep when the off switch gets pressed.

The point

The June shutdown will not be the last one. Between regulators, vendors, security incidents, and ordinary outages, you have already averaged six disruptions in two years. There will be a seventh, and an eighth. That is not a forecast. It is the run rate.

So the question the IBM study really puts to every operator is not whether your AI will fail or vanish. It will. The question is whether, on the day it does, you can name what you will lose, point to the plan, and point to the person who owns it. Right now, nine in ten cannot.

Sovereignty over your own operations does not mean building your own models. It means never depending on something you cannot see, cannot replace, and have not decided you can live without. The off switch may not be in your hands. The plan for when it gets pressed has to be.

AI operates. You own the decision, including the decision to depend on it.

Where is AI operating inside your business, and who owns the plan for the day it goes dark? Tell me in the comments.

This is the question my work begins with. When AI becomes infrastructure and no named human owns the dependency, that gap is what I help close.

Schedule a conversation

Sources

IBM Institute for Business Value with Oxford Economics, “The Calculus of AI Sovereignty,” June 17, 2026. Survey of 1,000 senior executives across 16 countries and 17 industries. Found that 91% do not fully understand their AI vendor, model, and infrastructure dependencies; 71% say switching their primary model would be difficult; respondents averaged six AI-related disruptions in two years; and 81% say a seven-day vendor outage would cause severe or critical disruption. https://newsroom.ibm.com/2026-06-17-ibm-study-limited-control-and-rising-dependencies-leave-enterprises-exposed-in-the-age-of-ai

U.S. Commerce Department export-control suspension of Fable 5 and Mythos 5, reported June 13, 2026. A national-security export-control directive barred foreign-national access to two frontier models, including the vendor’s own non-citizen employees, forcing a global suspension because users could not be separated cleanly in real time.

https://fortune.com/2026/06/13/anthropic-disables-fable-mythos-export-controls-national-security-threat/

https://www.bloomberg.com/news/articles/2026-06-13/anthropic-says-us-limits-foreign-access-to-fable-5-mythos-5

That is the number I keep coming back to. It sat on the command dashboard of an autonomous business platform I was reviewing this week. 147 tasks completed today. 3 awaiting your approval. The banner above it read, with no apparent irony, total visibility.

I look at systems like this for a living. Operators bring me the agentic platform they are about to switch on, and I trace one thing through it: what happens to human judgment once the system is running. The marketing is always about speed and headcount. The thing worth examining is always the same, and it is rarely on the slide.

This particular product is built to run a company. Ten specialist agents, one for finance, one for legal, one for sales, one even labeled CEO and strategy, coordinated by an overseer that, in the company’s own words, translates your goals into operating plans, calls the tools, files the paperwork, and sends the reports. It runs around the clock. The promise is that you can run an entire company while you sleep, and stop managing people so you can start directing outcomes.

I am not here to dunk on the vendor. The product is well made and the category is real. I am here for the safety net, because it is identical across nearly every autonomous system I see, and it does not hold the weight buyers put on it.

The safety net is this: the system escalates to you only when it matters, and surfaces only the decisions that genuinely need you. Anything consequential pauses at a checkpoint. One tap to approve, redirect, or take over. You stay at the helm.

That is the line that sells. It is also the line that should stop you.

The variable nobody inspects

“A human in the loop for critical decisions” carries a hidden term, and it is the only one that matters: who defines critical?

Read the promise again and the answer is already sitting inside it. The system escalates what matters. The system surfaces what genuinely needs you. The system decides. You are not reviewing the decisions your business made today. You are reviewing the ones it chose to hand up. Everything below that line already happened, in your name, while the dashboard assured you that you had total visibility.

Go back to 147 and 3. The business took 147 actions and asked you about three. The other 144 were not hidden from you, exactly. They were classified as beneath you. Somewhere in that feed, the quarterly pricing review was marked complete. The item that did earn a checkpoint was a vendor contract renewal. Maybe that is the right split. The point is that you did not make it. The overseer made it, and it will make the same kind of call 147 times tomorrow, and a few thousand times by the end of the quarter.

That is the quiet move. Control was the word in the brochure. What I was actually looking at was a feed.

Subscribe now

You consented once, in the abstract

The timing compounds it. You brief the system once. You set goals and guardrails in plain language and configure an auto, ask, or block policy for each tool. Then the agents run.

Which means your judgment was collected up front, as categories, before a single real situation existed. That is consent, not oversight. Judgment is not a preference you select in advance. It is what you do when a specific decision is in front of you, in its context, with what you know in that moment. A policy written at setup cannot do that. And the checkpoint queue that follows is mostly the work of clearing items the system pre-sorted, at the pace the system sets, framed by the information the system chose to attach. The button is real. The deliberation it implies left at brief time.

The layer underneath the checkpoint

There is a second problem that no queue can reach. Ten agents coordinated by an overseer do not only act, they interact. Finance feeds pricing, pricing shapes outreach, outreach moves pipeline, pipeline loops back to finance. Outcomes emerge from the coordination that no single agent decides and no checkpoint is built to catch, because a checkpoint sits on one action, not on the relationship between many. This is not a sci-fi warning. It is the ordinary behavior of coupled systems, and it lives exactly where the human is not looking.

And consider what the overseer is doing while you sleep. Its own description says it reviews all output. An AI grades the work of the other agents and then decides what reaches you. So the checkpoint you were handed sits outside a loop in which AI is already checking AI. You are not in the loop. You are downstream of it, holding the three items it elected to show you.

Watch what the premium tiers sell

The most honest page on the site is the pricing page. The upper plans advertise compliance autopilot and advanced approval automation. Sit with that second phrase. The product is now automating the approval step itself, the one human function the entire safety net was built around. When approval is automated, the checkpoint stops being a control. It becomes a notification, and notifications can be silenced.

The price for all of it runs from fifteen to forty dollars a month, marketed to solo founders and small operators. The people most drawn to full autonomy at streaming-subscription prices are precisely the ones least likely to have any governance of their own. Cheap autonomy, expensive accountability, sold to the buyers least equipped to tell which one they are paying for.

What you are actually buying

The selling logic treats management as overhead, a drag that automation removes. That is the wrong frame, and it is the costly one. You did not eliminate the management burden. You relocated it. It used to live with people, who are visible and correctable. Now it lives inside a system, which is opaque and compounding. The work of managing did not shrink. It moved to where you cannot see it, and then a banner told you that you could see everything.

So the founder who bought “stop managing people” did not buy a scaling solution. They have purchased a control problem in a scaling solution’s packaging, and they will not feel the weight of it until something authorized and wrong has already traveled through a few hundred actions nobody read.

The one question that turns a notification back into a control

A human in the loop becomes real governance under exactly one condition: a named human, not the vendor’s overseer, owns the threshold for what escalates. That person decides what counts as critical, can see what runs beneath the line, and can move the line when it is wrong.

So before you switch any of these systems on, ask the question the demo will not volunteer: who defines critical, and can I change it? If the definition belongs to the system, you do not have a checkpoint. You have a feed with a nicer font.

The trophy in this cycle is supposed to be ten million dollars in revenue with three people. The real achievement is ten million with three people and a control layer you actually understand. One is a headline. The other is a company that can survive its own automation.

AI operates. You own the decision. As long as you are still the one who decides which decisions are yours.

Who defines critical in your systems, and can you change it? Tell me in the comments.

The Evolving Mindset goes out every week on where AI actually breaks and who is left holding it. Subscribe and the next one finds you.

This is the question my work begins with. When an organization switches these systems on and no named human owns the threshold, that gap is what I help close.

Schedule a conversation

This is not the easy complaint that AI customer service is bad. Mostly it is not. The point is narrower and worse: an accountability word has been bolted onto a system that no longer performs the accountable function, and the word is now doing the work the system stopped doing.

Subscribe now

Two versions of the same failure, both generalizable.

The first has no money in it, only a wrong answer with a confident face. A customer emails a vendor with a simple compliance question: does the data processing agreement need to be signed before going live? A reply lands within the minute from a bot that signs itself “AI Support and Compliance Agent.” It returns a clean five-step process. Send your legal entity name. Send your signer. Watch for the agreement to arrive for e-signature. It even appends a “Sources” line. The customer follows the steps. Only later does the truth surface: the agreement was already in force, automatically, from day one. There was no five-step process. The agent named for compliance had invented a compliance procedure for its own company and dressed it in citations. The customer asks for a human. The bot promises one. None arrives.

The second has money in it, and that is what turns an annoyance into a governance story. A customer upgrades a subscription. The charge posts. Three days later the account silently drops back to the old tier while the new price keeps billing. The customer opens a ticket. The bot answers a question no one asked, explaining that it cannot offer compensation for technical issues. The customer never wanted compensation. The customer wanted the service already paid for, and said so. The bot replies that a human will follow up by email. Then it stops answering. For the rest of the billing month the account is metered at the old tier’s limits and charged at the new tier’s price, and the overage piles up. On the renewal date the system quietly resets itself and access returns. The defect lasted exactly one billing cycle and was cured by the next invoice. The fix arrived. The human never did.

There is a coda, and it is the worst part. When the customer goes back for the transcript, the support platform’s own links no longer resolve. The system of record has lost the record. The only surviving copy is the one the customer exported. Ask the payment platform for help and its automation points back to the developer. The developer’s bot had already pointed to the payment platform. Each machine points at the other, and there is no human anywhere in the circle.

Three layers run through both, and they generalize to any company putting these systems in front of customers.

The label is doing work the system does not. Compliance agent. Support agent. Human escalation. These are accountability words, and bolted onto a router with no one behind it, they borrow trust the company is no longer earning. It is a kind of optimization: the name is chosen for the confidence it projects, the way answer-engine optimization chases the look of an authoritative answer rather than the substance of one. FINRA warned in its 2026 report that generative AI can hallucinate, producing “misrepresentation or incorrect interpretation of rules, regulations or policies.” A bot wearing a compliance title and doing exactly that is the warning come to life.

Share

This is not hypothetical. A Canadian tribunal has already held a company liable when its support chatbot invented a refund policy that did not exist, ruling that the business owned the representation regardless of the fact that a bot made it.

The escalation exists only as a sentence. The bot says the words. The process never produces a human. From inside the company the failure is invisible: tickets closed, conversations answered, response times excellent. The gap lives only on the customer’s side of the glass, which is the one place no dashboard looks.

The record survives only if the customer keeps it. When the platform’s own links die, the customer’s export becomes the sole proof the case ever existed. Every governance argument reduces to one test: when the question comes, can you produce the record, the owner, the decision? More and more, the customer can, and the company cannot.

Be clear about what this is not. It is not a story about bad people or bad companies. The organizations doing this are competent, shipping products their customers respect and will keep paying for. That is what should bother you. The failure is not malice and not incompetence. It is structure. A customer-facing AI agent is a workflow, and every workflow either has a named human owner or it has nobody. “A human will follow up” is an escalation path. An escalation path that ends in silence is not a path. It is a sentence the bot was trained to say, and the sentence has become the product.

If you deploy AI agents in front of customers, this is your preview. Somewhere in your queue right now is a customer holding a promise your bot made and your org chart cannot keep. The following questions are the ones worth asking about any consequential system. Who owns the human handoff, by name? When the bot promises escalation, what happens next, and who checks that it happened? How long do you keep the record, and can the customer get it back? If any answer is a shrug, your escalation path is also just a sentence, and you will learn it the way everyone does: one customer, one cycle, one receipt at a time.

AI agents are coming to every support queue, and most of the time that will be fine. The companies that get it right will not be the ones with the smartest bots. They will be the ones where the bot’s promises are still owned by a person, where “a human will contact you” carries a name rather than a training weight. The technology can carry the conversation. It cannot carry the accountability.

The bot says a human is coming. The record, when anyone finally checks, says otherwise. The companies worth trusting will be the ones who keep the record themselves.

If those three questions, who owns the human handoff by name, who verifies the escalation happened, who keeps the record, are ones your organization cannot answer yet, that is the gap this piece is about. Defining that owner before the question arrives is the work Fellowship Intelligence does with risk-sensitive teams. If you cannot name that person yet, it is worth a conversation.

Book a conversation

Sources

FINRA, 2026 Annual Regulatory Oversight Report, Generative AI section. The quoted language, that AI can produce “misrepresentation or incorrect interpretation of rules, regulations or policies,” appears in FINRA’s discussion of generative-AI hallucinations. https://www.finra.org/rules-guidance/guidance/reports/2026-finra-annual-regulatory-oversight-report/gen-ai

Moffatt v. Air Canada, 2024 BCCRT 149 (British Columbia Civil Resolution Tribunal, February 14, 2024). The tribunal held Air Canada liable after its support chatbot described a refund policy that did not exist, ruling that a company is responsible for the representations its chatbot makes and rejecting the argument that the AI was a separate entity. https://www.americanbar.org/groups/business_law/resources/business-law-today/2024-february/bc-tribunal-confirms-companies-remain-liable-information-provided-ai-chatbot/

The AI safety market is organized as a stack. When something goes wrong, the reflex is to add a layer: a new filter, a tighter guardrail, a better gateway, a more granular permission. The instinct is understandable, because the stack is where the visible work happens and where the budget lives. But the layer that decides whether your organization survives an AI failure is not on the stack, and no amount of spending will put it there.

Share

It is worth walking the stack honestly, because every layer on it is real and most of it is worth buying.

At the bottom is the input. You can sanitize prompts, screen for injection, and strip the obvious attempts to talk a model into something it should not do. Above that sit the guardrails, the filters on the output that catch responses you have ruled off limits. Above that is identity and access, the work of deciding what an agent is and which systems it may touch. Above that is capability, least privilege applied to software that can act, so the agent simply does not possess the ability to take an action it was never granted. Above that is the network, the transport boundary that can stop an unverified state from crossing the wire at all. And wrapped around all of it, monitoring watches what the agent does in production, and evaluation red-teams what it might do before you ship it.

These are not marketing categories. They are genuine engineering, and an organization running AI without them is exposed in ways it should not be. Buy them. The point that follows is not that the stack is wrong. It is that the stack is incomplete in a way no further layer can fix.

Notice what every one of those layers does. Each one either limits what the agent can do or watches what the agent did. The input, capability, and network layers draw boundaries around its range of action. Monitoring and evaluation observe and test that action, before and after. Add any layer the vendors have not shipped yet, including a human dropped into the loop to approve the output, and it still falls into one of those two buckets. It limits, or it watches.

Not one of them governs whether the action the agent took was the right one.

That distinction is the whole argument, so be precise about it. There are two kinds of failure. The first is the agent doing something it was never permitted to do, and the stack is built for exactly that. A capability the agent does not have cannot be invoked by any prompt, however clever. A vault door blocks the robber regardless of intent. The second kind of failure is the agent doing something it was fully permitted to do that turned out to be wrong. No layer of the stack catches that one, because to the stack it does not look like a failure. It looks like permitted activity.

Make it concrete. A bank deploys an agent to handle servicing decisions inside a tightly scoped envelope: read the customer record, apply the policy, approve or decline within set limits. Every layer is in place. The prompts are clean, the permissions are minimal, the actions are logged, nothing crosses a boundary it should not. Then a customer in genuine hardship calls, and the agent applies the policy exactly as written and declines a forbearance that any experienced officer would have granted, because the situation was the kind of exception the policy never anticipated. Nothing broke. No control failed. The agent did precisely what it was authorized to do, and it was the wrong call. There is no layer in the stack that flags this, because no rule was violated. The only thing that catches it is a person who can look at the decision and say, that one is wrong, reverse it.

And this is not a temporary gap that a smarter layer will eventually close. The space of consequential decisions is not enumerable. You cannot write, in advance, the complete list of which permitted actions will turn out to be wrong, because that depends on context the rules cannot see. This is not a matter of opinion. NIST recently published a mathematical proof, applying Gödel’s incompleteness logic to AI guardrails, establishing that no finite set of rules is ever complete against a system operating in an open world. For any fixed set of controls, a case that defeats them exists. You can push the control down the stack as far as you like, from the words to the structure to the capability to the wire, and at every level you will have governed what the agent can do and never whether it should have. The stack has no layer for judgment, and by that proof, it never will.

Subscribe now

This leaves an uncomfortable piece of math for anyone who built an AI program by buying layers. You have most likely spent the most on the layers that protect you least against the failure that actually reaches a customer, a regulator, or a headline. The breach you can wall off is rarely the one that ends up in front of a board. The authorized, plausible, in-policy decision that was simply wrong is the one that does, and it is the one the stack was never built to catch.

The gap, then, is not a missing control. It is a missing owner.

Here is why the owner is missing by default, and not through anyone’s negligence. Every layer of the stack has a clear owner. Security owns the controls. Engineering owns the integration. Compliance owns the policy. The business owns the use case. Ask any of them who owns the controls and you get a fast, confident answer. Ask who owns the decision the agent makes, the actual call, accountable for the outcome, and the answers come from four directions at once, which is the same as no answer at all. Ownership of the decision falls into the space between the seats. Everyone is responsible for a layer, and no one is accountable for the call. That gap is invisible during normal operation and the only thing anyone can see the moment something goes wrong.

Closing it requires being specific about what an owner is, because the word is easy to say and easy to fake. An owner is not a committee that meets quarterly to oversee a system that acts every second, and it is not an acceptable-use policy in a binder. An owner is a named individual with the authority to override the system, who does not default to its output, and who can be asked, later, why. This is not the human dropped into the loop to approve and move on. It is the one with the standing, and the obligation, to say no. Each of those properties is load-bearing. Authority to override is the one organizations quietly withhold, because overriding a system under operational pressure means stopping something the business wants to keep moving, and that authority carries a real cost. An owner without it is not an owner, only a name on the incident report. And the role cannot be set once and forgotten, because the named owner moves on, the role shifts, and a year later the accountability points at someone who is no longer there. Ownership has to be a live assignment, not an entry in a document.

A practical objection is scale. If the agent is making thousands of decisions a day, no named owner can review each one, and no sensible governance model requires it. But this does not dissolve the ownership question, it clarifies it. At volume, the owner’s job is not to review every output. It is to govern the design of the decision envelope: which decisions the agent handles autonomously, where the escalation threshold sits, and how the exception path is structured when a case falls outside what the rules anticipated. Those are consequential design choices, and they carry real accountability. An owner who has drawn that boundary with intention, who has thought carefully about what the system should handle alone and what it should surface, has discharged the obligation. One who set those parameters by convenience and then stepped back has not. The volume does not change who owns the call. It changes what owning it requires.

It is also the one thing a better stack can’t give you. IT has always joked that the real problem was never in the seven layers of the stack. Layer 8 was the user, the ID-10-T error, the fault between the keyboard and the chair. Layer 9 was the organization itself, the budget and the politics and the accountability, the part no tool ever fixed. AI just changes the cast. The agent now sits in layer 8, the seat the user used to hold, taking the actions. So the thing that has to govern it is layer 9, the same accountable human layer that always sat above the user. And layer 9 has never been a product. There is no product to buy for the person who owns the decision.

If that sounds like a soft organizational point next to the hard engineering of the stack, consider that the law is now reaching for exactly the same thing. When Colorado rewrote its AI statute this spring, it did not mandate another control. It defined, in the text of the law, what meaningful human review has to be: a designated individual with the authority to approve, modify, or override the decision, who is trained, who considers the actual evidence, and who, in the statute’s own words, does not default to the system output. A legislature, working independently of any vendor’s roadmap, wrote the named owner into law, because it arrived where the proof does. What makes an AI decision defensible is not the controls around it. It is the person accountable for it.

So the question to carry out of this is not whether your stack is good. Assume it is. Assume you bought well. The question is the one the stack cannot answer, and the one that surfaces the day an AI decision goes wrong: who owned the call. Not which layer failed. Who owned the call. In most organizations running AI today the honest answer is that no one did, that ownership diffused until it belonged to no one, and that the absence stays invisible right up until the moment it is the only thing anyone wants to know.

No one asks how many layers you bought after an AI decision goes wrong. They ask who owned the call. The work is making sure that question has a name attached to it before it is asked, not after.

Naming that owner is harder than buying another layer, and it is the work Fellowship Intelligence does with risk-sensitive organizations: defining who decides, who can override the system, and who answers when a decision is wrong. If your organization cannot name that person yet, it is worth a conversation.

Schedule a conversation

On the limits of AI guardrails: Apostol Vassilev, a senior scientist at the National Institute of Standards and Technology (NIST), applied Gödel’s incompleteness theorems to AI guardrails and showed that no finite set of guardrails is universally robust against adversarial prompts; for any fixed rule set, a prompt that defeats it exists. Published by NIST and peer-reviewed in IEEE Security & Privacy.

https://web.archive.org/web/20260613121124/https://www.nist.gov/news-events/news/2026/06/nist-mathematical-proof-supports-transition-continuous-monitor-and-update

On meaningful human review: Colorado Senate Bill 26-189, “Automated Decision-Making Technology,” signed into law May 14, 2026, repealing and replacing the 2024 Colorado AI Act (SB 24-205) and effective January 1, 2027. The quoted language is the act’s definition of “meaningful human review,” which requires a designated individual with the authority to approve, modify, or override a consequential decision, who is trained, considers the available evidence, and does not default to the system output. Colorado General Assembly.

https://web.archive.org/web/20260609152522/https://leg.colorado.gov/bills/sb26-189

Subscribe now

I want to be precise about what is true here, because the easy version of this argument is wrong, and the accurate version is more useful.

It is true that AI-mediated discovery matters, and it matters more every quarter. McKinsey, in an October 2025 analysis, put roughly half of consumers already using AI-powered search, projected $750 billion in US revenue flowing through it by 2028, and warned of a 20 to 50 percent decline in traditional search traffic for unprepared brands. That is not hype. That is a real shift in how people find and choose. And a brand that ignores it is making a mistake.

This is not an argument that AI search is a fad, or that you cannot influence it. You can. The argument is narrower and more important: you cannot control it, and the category is being sold as if you can. The word optimization carries a promise it cannot keep.

Here is the distinction that the marketing glosses over. Traditional SEO was, at its core, a control game played against a knowable interface. There was an index. There were ranking signals. You could improve your position against those signals with enough technical work, content volume, authority building. The relationship between effort and outcome, while never perfect, was real and repeatable. That is what optimization means. A lever you pull that moves a result in a predictable direction.

AI-mediated discovery does not work that way, and the people building the tools say so plainly. The system does not retrieve your page and rank it. It synthesizes an answer from a wide and shifting field of sources, then compresses that into a response that varies by model, by phrasing, by the user’s context, and over time. You are not optimizing a position. You are one input among many into a process whose output you do not hold.

Share The Evolving Mindset

The numbers make the point concrete. By McKinsey’s own measure, a brand’s own website typically accounts for only 5 to 10 percent of the sources AI search draws on. The other 90 percent and up is third-party material, publishers, reviews, forums, user-generated content, affiliates, none of which you own and most of which you cannot direct. In some categories, McKinsey found, publishers, user-generated content, and affiliate sites make up more than 65 percent of the sources. The plain consequence: “Visibility is not guaranteed,” in McKinsey’s own words, even for established market leaders. Traditional brand strength does not carry over. The lever SEO offered does not exist here, because the thing you would pull on is mostly not yours.

The honest players in this space already concede the point, and the clearest example is one of the biggest. HubSpot now ships an AEO product, and its own documentation describes, accurately, something much weaker than the word optimization implies. It tells users that AI “responses change over time,” that “each answer engine may generate different responses” to the same prompt, and that after you act on its recommendations, improvements “may not appear immediately” and should be reviewed “across multiple analysis cycles.” Read the recommendations the tool actually produces and they are not a control panel. They create content, engage relevant Reddit threads, post on LinkedIn and YouTube, and reach out to third-party publishers. That is content production, public relations, and reputation work. It is genuinely useful. It is also, by the tool’s own account, measurement and influence with no guaranteed outcome.

That is the gap between the documentation and the label. HubSpot’s help pages describe influence. The category name promises optimization. Those are not the same word, and the difference is the whole problem. A buyer who reads “optimization” hears SEO, hears a lever, hears control. What they are actually buying is the disciplined management of evidence they mostly do not own, feeding a system whose output no one can dictate.

Which brings me back to the names. A field confident enough to charge for outcomes has not yet agreed on what to call itself. McKinsey says GEO. HubSpot says AEO. Others say AI engine optimization. When the most credible voices in a category cannot settle its vocabulary, that is not a branding quirk. It is a sign the practice is younger and less defined than the confidence of its sales pitch suggests.

So if optimization is the wrong frame, what is the right one?

The right frame is governance, not marketing. The problem an organization actually faces in AI-mediated discovery is not a visibility problem to be optimized. It is a risk problem to be governed: the risk that the public evidence an AI system retrieves about you is inconsistent, outdated, unsupported, or contradictory, and that the system, doing exactly what it is designed to do, synthesizes those contradictions into an answer you would never have authorized.

That is not a marketing trick. It is reputation risk created by an ungoverned evidence layer, and the discipline that addresses it is the same discipline any governance problem requires. The principles are not exotic. They are the ones that govern any system of record. Define the standard: what the organization’s authoritative claims actually are, and which claims are not defensible and should never be made. Establish a single source of truth, so the public record states those claims consistently rather than five different ways across five channels. Assign ownership, a named person accountable for the coherence of that record, not a responsibility scattered across marketing, PR, and whoever updated the website last. And monitor, because the answer layer is not stable and drift is the normal condition, not the exception.

Notice what that discipline does and does not promise. It does not promise to make the model say what you want. Nothing can promise that and a vendor who does is selling you the one thing the technology does not offer. What it does is reduce the number of reasons an AI system has to get you wrong. You cannot control the output. You can govern the inputs you own and influence the ones you do not, and you can know, continuously, what the systems are actually saying so that you are managing a current picture rather than an assumed one. That is the difference between governing your evidence and optimizing your ranking. One is honest about the limits of control. The other sells a lever that was unplugged the moment discovery moved from the index to the synthesis.

The organizations that will navigate this well are not the ones that buy the most aggressive optimization promise. They are the ones that treat their public evidence as a governed asset: consistent, owned, monitored, and defensible. That is unglamorous, it does not come with a guarantee of where you land in tomorrow’s answer, and it is the only version of this work that survives contact with how the systems actually behave.

AEO is selling control. The honest product underneath it is governance. Buy the second, and be skeptical of anyone charging you for the first.

The Evolving Mindset publishes weekly on AI governance and organizational accountability. If a vendor has promised to control what AI systems say about your organization, that is a claim worth testing before you buy it. Reach out through the link below.

Schedule a conversation

Sources and notes

• AI-search adoption and economics: McKinsey & Company, “New front door to the internet: Winning in the age of AI search,” October 16, 2025. Cited figures: roughly 50 percent of consumers already use AI-powered search; projected $750 billion in US revenue flowing through AI-powered search by 2028; 20 to 50 percent potential decline in traditional search traffic for unprepared brands; a brand’s own sites typically comprise only 5 to 10 percent of the sources AI search references; in some categories publishers, user-generated content, and affiliate sites exceed 65 percent of sources; “Visibility is not guaranteed”; just 16 percent of brands systematically track AI search performance. Note: this is a pro-GEO analysis; it is cited here for its data, not its recommendation.

• Output variability and the nature of the service: HubSpot, “Set up and analyze AEO” (product documentation, last updated May 21, 2026). Cited language: AI “responses change over time”; “each answer engine may generate different responses”; improvements “may not appear immediately” and should be reviewed across multiple analysis cycles. The tool’s recommendations consist of creating content and engaging earned, social, and third-party channels.

• Terminology: the category is referred to variously as GEO (generative engine optimization), AEO (answer engine optimization), and AI engine optimization, with no settled industry taxonomy as of this writing.

All figures verified against primary or top-tier sources prior to publication. Nothing in this piece constitutes legal, financial, or marketing advice.

Subscribe now

It is a reasonable sounding sentence. It has the cadence of prudence. And it rests on a premise that is now demonstrably false: that somewhere ahead is a rule, a date, a starting gun, after which AI governance becomes mandatory and before which it is optional. The record says otherwise. The regulation did not arrive as a rule with an effective date. It arrived as a posture, stated in writing, that the rules firms already operate under never stopped applying. The wait is not prudence. The wait is exposure.

Start with what the SEC actually did, because it is widely misread. On June 12, 2025, the Commission formally withdrew fourteen pending rule proposals, among them the conflicts-of-interest proposal on predictive data analytics, proposed in August 2023 and the closest thing to an AI-specific rule the SEC had in motion. It stated that it does not intend to issue final rules on the withdrawn proposals. In many firms, compliance logged that as relief: the AI rule died, pressure off.

That reading gets the mechanics right and the meaning backwards. A withdrawn proposal means there will be no finalization date, no implementation window, no compliance deadline to plan against. Each of those would have been a future moment when a firm could say “now it begins.” Withdrawal removes the future moment. What remains is the present one: the antifraud provisions, the fiduciary standards, the supervision and books-and-records rules that already reach AI-assisted conduct, with no on-ramp, because they have applied the entire time. The Commission’s AI-washing enforcement, brought under existing antifraud provisions, makes the same point from the other direction: no new rule was needed to charge the conduct.

FINRA, for its part, stopped implying and started itemizing. The arc is worth seeing whole. In June 2020, it published a report on artificial intelligence in the securities industry. In 2024, it issued Regulatory Notice 24-09, reminding members that their regulatory obligations apply when using generative AI and large language models. And in December 2025, its 2026 Annual Regulatory Oversight Report arrived carrying a standalone GenAI section, marked new for this cycle. Each step says the same thing more loudly. That is not a regulator waiting to make up its mind. That is a regulator documenting expectations in ascending detail.

The section’s first move is doctrinal. FINRA states that its rules are “intended to be technologically neutral” and that they “continue to apply when firms use GenAI or similar technologies in the course of their businesses, just as they apply when firms use any other technology or tool.”

Technological neutrality is the quiet death of the regulatory-lag myth. A technology neutral rule cannot lag the technology, by construction. Rule 3110’s requirement of a reasonably designed supervisory system covered the telephone, then email, then chat applications, and covers generative AI now, without a single amendment. The lag firms perceive was never in the rulebook. It was inside the firm: the distance between what employees adopted and what leadership wrote down.

Then the report itemizes. It describes the expected shape of governance: a supervision, governance or model risk management framework with “clear policies and procedures to develop, implement, use and monitor GenAI, while maintaining comprehensive documentation throughout.” And it describes the documentation with unusual specificity: “storing prompt and output logs for accountability and troubleshooting; tracking which model version was used and when; and validation and human-in-the-loop review of model outputs.”

That specificity deserves a pause. Prompt logs. Output logs. Model version histories. Named human checkpoints. None of these is an aspiration. Each is an artifact: a thing a firm can either produce on request or cannot. When an oversight body shifts from describing principles to describing artifacts, it is telling you what the request will look like when it comes.

The same report contains FINRA’s first dedicated guidance on AI agents, systems capable of autonomously performing tasks on a user’s behalf, planning and acting without predefined rules. Read the risk list closely, because none of the entries is about capability. They are about accountability. Agents acting without human validation. Agents acting beyond “the user’s actual or intended scope and authority.” Multi-step agent reasoning that is “difficult to trace or explain, complicating auditability.” And the suggested responses are accountability mechanics: tracking agent actions and decisions, placing human-in-the-loop oversight, establishing guardrails that limit agent behavior. The structure is the message. As autonomy increases, the regulator’s attention moves from what the system can do to who answers for what it did. Statements like that tend to run a year or two ahead of the enforcement that follows them.

Beneath the federal posture, the state layer has begun arriving on statutory schedules rather than discussion schedules. California’s AB 2013 took effect January 1, 2026: developers of generative AI systems made available in the state must publicly post summaries of their training data, with no user threshold. The California AI Transparency Act, SB 942, becomes operative August 2, 2026 following amendment, requiring large providers to label AI-generated content and offer free public detection tools, with civil penalties of five thousand dollars per violation per day. Precision matters here: those statutes aim at AI developers and large platforms, not at broker-dealers or advisers. They are cited not as obligations on financial firms but as evidence of direction and tempo. Transparency obligations now arrive with effective dates attached. Meanwhile, inside most firms, the foundational question, where is AI being used and who owns each use, still has no written answer.

Share

Put the layers together and the popular framing inverts completely. There is no regulatory vacuum. There is a regulatory perimeter, already drawn, around a space most firms have never mapped internally.

Which is why the real policy question is not the one firms keep asking. The question is not “what should our AI policy say when we eventually write one.” Every firm already has an AI policy in force today. If leadership did not author it, the policy is the sum of what employees are actually doing: which tools they signed up for with corporate email, what client information they paste into which windows, which outputs leave the building under the firm’s name unreviewed. That is a real policy. It governs real conduct, every day. It simply has no author, no review, no documentation, and no defense. A default is just governance without a named author.

The first act of replacing the default is not philosophical. It is an inventory. Where AI is used, by whom, for what, approved by whom, owned by whom, by name. You cannot supervise what you have not inventoried. You cannot assign ownership to uses you do not know exist. You cannot log what you have not located. Every control the FINRA report describes presumes the inventory exists, and most firms have not built it, because it feels administrative and because the first draft is always embarrassing. It is also the cheapest control in the stack, and the one all the others stand on.

From there, the discipline is the same one that governs any consequential system, and none of it is exotic. A written policy that states what is permitted, what is prohibited, and for whom. Workflow controls that make the policy operative rather than aspirational. Ownership: a named human answerable for each use. Approval and escalation paths, so new uses and exceptional moments route to people instead of defaults. And monitoring, because usage drifts, and a policy never checked against reality is a document, not a control. None of this waits on a rule. All of it is what the existing rules, by the regulators’ own account, already expect.

The firms reading this moment correctly are not the ones waiting. They are the ones who noticed that the waiting period was cancelled, quietly and in writing, and that the question, when it comes, will be answered with an artifact or an absence. The firms running on defaults are not ungoverned. They are governed by policies no one wrote, and they will be examined on them all the same.

The default is the policy. The exam assumes you are the author. Become one before the assumption gets tested.

The Evolving Mindset publishes weekly on AI governance and organizational accountability. If your organization could not produce its AI inventory, its named owners, or its logs on request, that gap is the work. Reach out through the link in the profile.

Share The Evolving Mindset

Sources and notes

• SEC: formal withdrawal of fourteen notices of proposed rulemaking, June 12, 2025, including Conflicts of Interest Associated with the Use of Predictive Data Analytics by Broker-Dealers and Investment Advisers (File No. S7-12-23, proposed August 2023). The Commission stated it does not intend to issue final rules with respect to the withdrawn proposals. AI-washing enforcement actions referenced were brought under existing antifraud provisions; cited as posture, not individual case analysis.

• FINRA: 2026 Annual Regulatory Oversight Report, published December 2025; GenAI section marked “NEW FOR 2026.” Quoted language verified against the report page on finra.org: “intended to be technologically neutral”; “continue to apply when firms use GenAI or similar technologies in the course of their businesses, just as they apply when firms use any other technology or tool”; “clear policies and procedures to develop, implement, use and monitor GenAI, while maintaining comprehensive documentation throughout”; “storing prompt and output logs for accountability and troubleshooting; tracking which model version was used and when; and validation and human-in-the-loop review of model outputs”; AI agents risks including action beyond “the user’s actual or intended scope and authority” and reasoning “difficult to trace or explain, complicating auditability.” Escalation arc: Artificial Intelligence in the Securities Industry report (June 2020); Regulatory Notice 24-09 (2024); 2026 report (December 2025). Top member-firm GenAI use case per FINRA: summarization and information extraction.

• California: AB 2013 (generative AI training data transparency), effective January 1, 2026; applies to developers of generative AI systems made available in California, without a user threshold. SB 942 (California AI Transparency Act), operative August 2, 2026, as amended by AB 853 (signed October 13, 2025); requires covered providers (over one million monthly users) to label AI-generated content and provide a free public detection tool; civil penalties of $5,000 per violation per day. Both statutes target developers and large providers rather than financial firms and are cited as direction and tempo of the state layer only.

• Nothing in this piece constitutes legal advice. Firms should consult qualified counsel on their specific regulatory obligations.

The Evolving Mindset

A drive-thru company told the market its AI was taking orders. More than seven in ten of those orders needed a human to step in. The gap between those two facts is the most important number in enterprise AI right now, and almost no buyer can see it.

Share

The company was Presto Automation, a publicly traded restaurant-technology firm. It marketed its flagship product, Presto Voice, as an AI system that automated drive-thru order-taking and removed the human from the transaction. In January 2025, the Securities and Exchange Commission found that the claim did not match the operation. During the relevant period, more than 70 percent of orders processed by the in-house system required human intervention. At certain locations, the figure was 100 percent. The SEC also found Presto had failed to disclose that the AI itself was owned and operated by a third party. The product the market was told ran itself was, in substantial part, being run by people.

This is the failure mode the industry does not like to name, and it is now common enough to have drawn regulators. It is worth naming plainly. A large share of what is sold as artificial intelligence is performance. The capability is asserted in the marketing, demonstrated in the controlled demo, and quietly backstopped by humans, by narrower function, or by nothing at all when the deployment meets reality. The buyer pays for the asserted capability. The buyer inherits the gap.

Presto is not an isolated case. It is a category, and the category now has an enforcement record.

Consider Workado. The company sold an AI Content Detector advertised as 98 percent accurate at identifying AI-generated text. In 2025, the Federal Trade Commission found that the tool’s accuracy on general-purpose content was closer to 53 percent. The model had effectively been trained and tested on a narrow band of material, academic abstracts and ChatGPT output, and then sold as a general detector. Fifty-three percent is not a detector. It is a coin flip with a confidence interval attached. The FTC required the company to stop making the accuracy claim and to hold competent, reliable evidence for any efficacy claim going forward. The point is not that the number was off by a few points. The point is that the headline capability and the actual capability were different products, and only one of them was for sale.

Consider Cox Media Group. The company marketed a service called Active Listening, which it claimed used AI to listen to consumers’ real-time conversations through the microphones on their devices in order to target advertising. In May 2026, the FTC found the service did not do what it advertised. It did not listen. It used no voice data. What it actually did, according to the Commission, was resell email lists from data brokers at a markup, wrapped in a story about AI that listens. The AI capability was not understated or overstated. It was absent. The performance was the entire product.

These three are not the same offense. Presto had a real product that needed far more human help than it claimed. Workado had a real product that worked far less well than it claimed. Cox had a claim with no product underneath it. But they share a single structural feature, and that feature is the subject worth your attention: in every case, the buyer had no practical way to verify the capability before relying on it. The claim was checkable only by the regulator, only after the fact, only because someone complained.

That is the real exposure, and it does not sit with the vendor. It sits with the organization that bought the claim.

Share The Evolving Mindset

When a vendor overstates what its AI does, the immediate damage lands on the vendor in the form of an eventual enforcement action or a refund. But the operational damage lands on the customer, and it lands earlier and lasts longer. The restaurant that deployed Presto Voice told its own operations the drive-thru was automated and staffed accordingly. The firm that bought a 98 percent detector made decisions, about students, about hiring, about content, on a tool that was right about half the time. The advertiser that paid for Active Listening built a campaign on a capability that never existed. In each case the vendor made the claim and the customer made the decision. Accountability for the decision does not transfer back to the vendor because the vendor exaggerated. It stays with the organization that acted.

This is the through line from a piece earlier this week. The deployed system has an owner whether or not the organization ever decided who that owner is. When the system underperforms its claims, the question is not only whether the vendor lied. The question is who in the buyer’s organization was responsible for confirming the capability was real before the business depended on it. In most organizations, the answer is the same as it is for AI governance generally. No one specific. The vendor’s marketing was treated as the verification step. It is not one.

The defense the industry offers is human oversight. The vendor will say a human is in the loop. The buyer will say a person reviews the output. This is supposed to be the reassurance that makes the capability gap survivable. Presto is the precise reason it is not.

Presto is, on paper, the human-in-the-loop model working. Humans intervened on more than 70 percent of orders. The humans were there. But the humans were not the safeguard the phrase implies. They were the undisclosed mechanism that made a falling-short product look like a working one. “Human in the loop” was not a control that caught the AI’s failures. It was the labor that hid them. The presence of a human in the workflow told the buyer nothing about whether the AI worked, because the human was doing the work the AI was credited for. Oversight that exists to cover a capability gap is not oversight. It is the gap, staffed.

So the operative question for any organization buying AI capability is not whether there is a human in the loop. Of course there is. The question is what that human is actually doing. Are they exercising judgment over the AI’s output, with the authority and the time to reject it when it is wrong? Or are they silently completing the work the AI cannot, while the organization reports the function as automated? Those are opposite conditions that look identical on an org chart and in a vendor deck. The first is governance. The second is Presto.

Regulators have started to close on the vendors, and the trajectory matters. The SEC brought its first AI-washing cases in March 2024 against two investment advisers for overstating their use of AI, with penalties totaling 400,000 dollars. It reached its first public-company AI-washing case with Presto in January 2025. The FTC ran Operation AI Comply in September 2024, a coordinated sweep against deceptive AI claims, and has continued through Workado in 2025 and Cox Media Group in 2026. The enforcement is real and it is accelerating. But enforcement arrives late, lands on the vendor, and does nothing to unwind the decisions the buyer already made on a claim that turned out to be performance. By the time the SEC documents that 70 percent of the orders needed a human, the customer has already run the operation as if they did not.

The enforcement record should not be read as reassurance that the system is self-correcting. It should be read as a published list of capability claims that turned out to be false, assembled by the only parties with subpoena power, because no one downstream could check the claims on their own. For every case that reaches an enforcement action, the more relevant population is the deployments where the gap exists, no one complained, and the buyer is still operating on the asserted capability rather than the real one.

There is a discipline that closes this, and it is unglamorous. Before an organization relies on an AI capability, someone inside it has to own the question of whether the capability is real, in this organization’s actual conditions, not in the vendor’s demo. That means a defined owner for vendor capability verification, distinct from procurement and distinct from IT security. It means treating the vendor’s accuracy claim as a hypothesis to be tested against your own data before deployment, not a specification to be accepted. It means knowing, specifically, what the humans in your AI workflows are doing: governing the output, or quietly producing it. None of that is exotic. It is the same accountability discipline that the rest of enterprise risk already takes for granted, applied to a vendor category that has so far been allowed to grade its own capability.

The vendors will keep performing capability, because the market keeps paying for the performance and only rarely tests the substance. That will not change on the vendor’s side until testing the substance becomes the buyer’s default. The organizations that build that discipline now are the ones that will not appear, eighteen months from now, in the next enforcement release, explaining to a regulator why they ran the business on a capability that needed a human all along.

Subscribe now

The Evolving Mindset publishes weekly on AI governance and organizational accountability. If you are relying on a vendor’s AI capability claim you have not independently tested, that is a specific and answerable exposure. Reach out through the link in the profile.

Sources and notes

• Presto Automation: SEC administrative proceeding, settled January 14, 2025 (Release 33-11352). Findings include that more than 70 percent of orders required human intervention, 100 percent at certain locations, and that the AI was owned and operated by a third party. Cease-and-desist; no civil penalty, citing the company’s financial condition and cooperation.

• Workado, LLC (formerly Content at Scale AI): FTC final order approved August 2025. Advertised 98 percent accuracy on its AI Content Detector; FTC found real-world accuracy on general content of approximately 53 percent. Order requires competent and reliable evidence for efficacy claims.

• Cox Media Group “Active Listening”: FTC settlement announced May 2026; total monetary relief of approximately 930,000 dollars across Cox Media Group and two related firms. FTC found the service did not listen to conversations and used no voice data.

• SEC first AI-washing actions: Delphia (USA) Inc. and Global Predictions Inc., settled March 2024, penalties totaling 400,000 dollars (SEC release 2024-36).

• FTC Operation AI Comply announced September 25, 2024.

All figures verified against primary or top-tier sources during research. Nothing in this piece constitutes legal or investment advice.

Most organizations that believe they govern their AI have built the performance of governance, not the fact of it. The two are nearly indistinguishable from the outside, and often from the inside as well. They diverge at exactly one moment: when a deployed system produces an outcome someone has to answer for. At that moment, the published principles, the ethics committee, and the policy on file do none of the work. What does the work is a named owner, a record of the decision, and the authority to have stopped it. Most organizations have invested in the first set and assumed it produced the second. It does not.

This is worth naming precisely, because the naming is the part that has been missing. Governance theater is governance optimized for the appearance of control rather than the fact of it. It is not fraud. It is rarely even cynical. It is the predictable result of asking an organization to demonstrate governance on a timeline faster than governance can actually be installed, in a market that rewards the demonstration and does not yet test the substance. The artifacts that signal governance are cheap and fast to produce. The architecture that exercises governance is slow, expensive, and invisible until it is needed. Under those incentives, organizations build what they are rewarded for building.

The pattern is not anecdotal. It shows up in the academic record, in the corporate record, in the survey data, and in the case law, and it shows up the same way in each.

Share

The principle is the easy part

In 2019, researchers at ETH Zurich published a study in Nature Machine Intelligence that catalogued the global landscape of AI ethics guidelines. They found 84 separate sets of principles issued by companies, governments, and institutions worldwide. The striking finding was not the volume. It was the shape of the agreement. The documents converged on a small set of high-level principles, transparency, justice and fairness, non-maleficence, responsibility, and privacy, with remarkable consistency. And they diverged, sharply, on the question that determines whether a principle governs anything at all: how it should be implemented, by whom, and with what consequence for noncompliance. The field had reached consensus on what to value and no consensus on how to enforce it.

That same year, Brent Mittelstadt of the Oxford Internet Institute published a companion critique, also in Nature Machine Intelligence, arguing that principles alone cannot guarantee ethical AI. His reasoning is the load-bearing point for everything that follows. Medicine has principles that work, he noted, because medicine also has the infrastructure that makes principles bind: fiduciary duties, professional norms developed over generations, proven methods for translating principle into practice, and robust legal and professional accountability mechanisms. AI development has the principles and almost none of the infrastructure. Borrowing medicine’s four principles without medicine’s enforcement apparatus produces the vocabulary of governance without its function.

There is direct empirical support for the proposition that a principle without enforcement changes nothing. A controlled study presented at a major software engineering conference in 2018 tested whether showing practitioners a formal code of ethics altered their decisions. The researchers ran 63 students and 105 professional developers through a series of real-world ethical vignettes. Half were explicitly directed to consider the relevant code of ethics; half were not. The result was no statistically significant difference between the two groups, for any vignette, for either population. The code was present. The behavior did not move. The authors were careful about the limits of a single study, and the caution is warranted, but the finding is a clean illustration of the mechanism: the existence of the document and the change in the outcome are two different things, and the first does not produce the second on its own.

The corporate record tracks the same divide

If principles are the cheap part and enforcement is the expensive part, you would expect that under cost pressure, organizations cut the expensive part and keep the cheap one. That is what the record shows.

In March 2023, Microsoft eliminated its Ethics and Society team during a broader round of layoffs. The team’s specific function was to translate the company’s published AI principles into product-level practice, the connective tissue between the values statement and the shipped feature, and it was cut in the same period the company was accelerating the integration of generative AI across its products. Microsoft’s published principles remained in place. Its Office of Responsible AI and other governance bodies remained in place, and the company has stated, accurately, that it continues to invest substantially in responsible AI. That response is true, and it is also the cleanest possible illustration of the thesis. When the organization had to choose, the principles survived and the team that operationalized them was the line item that did not. The performance layer is more durable under pressure than the function layer, because the performance layer is what the outside world can see.

This is not unique to one company, and it is not evidence of bad actors. The broader transparency picture has moved in the same direction. Stanford’s Foundation Model Transparency Index, which scores major AI developers on disclosure, found persistent and systemic opacity precisely in the areas that matter most for governance: the effectiveness of guardrails, the handling of data, and downstream impact. The areas where companies disclose least are the areas where governance would actually be tested.

The survey data shows the same gap at scale

The 2025 McKinsey State of AI survey provides the clearest quantitative picture. Seventy-eight percent of organizations reported using AI. Against that near-universal adoption, the governance figures are strikingly thin. Twenty-eight percent reported that their CEO was involved in overseeing AI governance. Seventeen percent reported board-level oversight. Twenty-seven percent said all generative-AI output was reviewed before use. McKinsey’s own analysis found that CEO-level oversight of governance was the factor most correlated with bottom-line impact from AI, which means the element most predictive of value is also among the least common.

Read those numbers together and the shape is unmistakable. Adoption is everywhere. Senior ownership of the consequences is rare. The same survey found that roughly one percent of leaders described their AI rollouts as mature. The governance that exists is, overwhelmingly, the part that can be documented and published. The part that requires a named human to own a consequential decision and answer for it is the part that most organizations have not built.

None of this means the principles are worthless or the ethics statements are a con. Principles are necessary. They set direction, they signal intent, and they give the eventual enforcement architecture something to enforce. The error is not in having them. The error is in treating their existence as the finished state, when they are the first and easiest step of a much longer build.

The gap becomes concrete the moment a system fails

Theater holds right up until something tests it. Then the missing layer becomes the only layer that matters.

In February 2024, the British Columbia Civil Resolution Tribunal decided Moffatt v. Air Canada. The airline’s customer-service chatbot had given a passenger false information about bereavement fares, and the airline refused to honor it. Air Canada’s defense is the detail worth remembering: it argued that the chatbot was, in effect, a separate legal entity responsible for its own actions. The tribunal rejected that argument completely and held the airline responsible for everything its website told a customer, whether the words came from a static page or an AI. The monetary award was small, just over 800 Canadian dollars. The principle it established is not small. “The system did it” is not a defense. The output has an owner whether or not the organization ever decided who that owner is.

The same logic is now arriving through regulators, aimed at the claims themselves. In March 2024, the U.S. Securities and Exchange Commission settled its first enforcement actions for “AI washing,” charging two investment advisers with overstating their use of artificial intelligence and imposing penalties totaling 400,000 dollars. In September 2024, the Federal Trade Commission announced Operation AI Comply, a coordinated sweep against companies making deceptive AI claims. The through line is that the performance of AI capability is no longer costless. Regulators have begun testing claims against reality, and the performance of governance sits in the same category of claim.

The test that separates the two

Because governance theater is built from real artifacts, you cannot detect it by looking at the artifacts. A published principle looks like a published principle whether or not anyone is accountable to it. The only reliable test is to run a real case through the system and see what comes out.

Take one consequential decision your organization made in the last quarter in which an AI output played a part. Then try to answer three questions from a record rather than from memory. Who owned that decision. What AI output influenced it, and was that output validated before it became action. And who held the authority to stop it before it shipped, if stopping it had been correct. If those answers exist as a record, you have governance. If they exist only as a policy asserting that such answers ought to exist somewhere, you have documentation. The distance between those two states is not a documentation gap that better paperwork closes. It is the entire exposure, and it does not close on its own.

This is the failure mode that should concern competent, well-run organizations the most, precisely because it does not feel like failure while it is happening. Nothing breaks during the performance. The committee meets, the principles are published, the policy is cited in the audit response, and every external signal reports that AI is governed here. The failure surfaces only at the one moment the structure exists for: when an output causes harm and the organization reaches for the owner, the record, and the authority to have intervened, and discovers it built the parts that demonstrate governance and skipped the parts that exercise it.

The work is not to publish a better principle. The principle is already there. The work is to install the layer underneath it that the principle has been standing in for: defined ownership, a decision record, and real escalation authority, calibrated to the consequence of the decision rather than to the appearance of control. That layer is unglamorous, it does not photograph well in a board deck, and it is the only part of AI governance that does anything when it is finally needed.

The Evolving Mindset publishes weekly on AI governance and organizational accountability. If this raised a question about your own organization, the question of whether you have governance or its appearance is a specific and answerable one. Reach out through the link in the profile.

Share The Evolving Mindset

Sources and notes

• AI ethics guidelines convergence on principles and divergence on implementation: Jobin, Ienca and Vayena, “The global landscape of AI ethics guidelines,” Nature Machine Intelligence, 2019 (84 documents identified).

• Principles cannot guarantee ethical AI without enforcement infrastructure: Mittelstadt, “Principles alone cannot guarantee ethical AI,” Nature Machine Intelligence, 2019.

• Code of ethics produced no measurable behavioral change: McNamara, Smith and Murphy-Hill, “Does ACM’s Code of Ethics Change Ethical Decision Making in Software Development?”, ESEC/FSE 2018 (63 students, 105 professionals).

• Microsoft Ethics and Society team eliminated March 2023: first reported by Platformer; widely corroborated (TechCrunch, The Register, Washington Post). Microsoft’s continued responsible-AI investment via the Office of Responsible AI is per the company’s own statements.

• Foundation model transparency gaps: Stanford HAI/CRFM Foundation Model Transparency Index (2023 and 2024 editions).

• Adoption and governance-ownership figures (78% AI use; 28% CEO oversight; 17% board oversight; 27% review all gen-AI output; ~1% mature rollouts): McKinsey, “The State of AI,” 2025.

• Moffatt v. Air Canada, 2024 BCCRT 149, decided February 14, 2024 (total award CA$812.02; “separate legal entity” defense rejected).

• SEC AI-washing settlements (Delphia and Global Predictions, penalties totaling $400,000), March 2024; SEC primary release 2024-36.

• FTC Operation AI Comply announced September 25, 2024.

All figures above were verified against primary or top-tier sources during research. Where reporting traces to a single original outlet, that origin is noted. Nothing in this piece constitutes legal advice.

For every person you refer who subscribes, you’ll receive something from Fellowship Intelligence’s library - documents I don’t distribute publicly.

How it works

Use your referral link below. Any new subscriber who signs up through it counts toward your total.

Refer a friend

What you earn

• 1 referral: The Consigliere - how to build an AI thinking partner that knows your actual position

• 5 referrals: The Translation Protocol - a working tool for communicating across the gaps between you and the people you need to move

• 25 referrals: A companion document built for your specific industry. Reply with your sector and I’ll send the right one.

Visit the leaderboard

To learn more, check out Substack’s FAQ.

Thank you for helping get the word out about The Evolving Mindset!

The investment community is debating whether the AI boom is real. That debate is missing the more consequential question: what happens to your organization if the vendors you depend on are less durable than they appear?

There are two pieces of intelligence circulating this week that most people are reading in isolation. They should be read together.

The first is a well-sourced analysis of where institutional capital is actually flowing in the AI era: PE firms quietly buying discounted vertical SaaS while venture capital concentrates almost entirely into AI-native companies at loss-making valuations. The data is credible. The framing is that smart money reads gaps between narrative and fundamentals and positions accordingly.

The second is a more aggressive claim: that the AI investment boom is partially built on circular accounting. Cloud credits flow from Big Tech to AI startups. Startups spend those credits back on the same cloud infrastructure. The originating companies book the usage as revenue. Meanwhile, unrealized markups on AI startup stakes are appearing as reported profit in ways that materially inflate headline earnings numbers.

Neither piece draws the conclusion that matters most for operating organizations. The investment debate will resolve itself through market forces. The governance gap will resolve itself through failures.

The question worth asking now is which one you want to be ready for.

The Circular Capital Problem, Precisely Stated

Before engaging the implications, it is worth being precise about what the circular accounting concern actually is, and what it is not.

The mechanism is real. Microsoft, Google, Amazon, and Oracle have all structured large AI investments partly as cloud compute commitments rather than straight cash. Amazon’s own 10-Q confirms a $38 billion existing AWS arrangement with OpenAI, expanded by a further $100 billion commitment announced in Q1 2026. Anthropic’s situation is structurally similar. Amazon’s filings confirm it has invested $8.0 billion in Anthropic convertible notes and holds nonvoting preferred stock now valued at $32 billion on its balance sheet, with all of that compute running on AWS infrastructure. The capital flow is circular in the sense that investment and revenue are intertwined in ways that standard financial reporting does not cleanly separate.

The paper profit problem is also real, and it is quantifiable in public filings. Alphabet and Amazon both hold significant Anthropic equity positions. When Anthropic’s valuation increases, as it has repeatedly across successive funding rounds, both companies mark up those positions and recognize the gain in reported income. In Q1 2026, Alphabet’s reported net income of $62.6 billion included $36.9 billion in net gains on equity securities (primarily unrealized gains on non-marketable equity securities), representing 59% of reported profit. The figure comes directly from the company’s own MD&A disclosure. Amazon’s $30.3 billion in reported net income included approximately $16.8 billion in Anthropic-related gains ($12.3 billion in upward adjustments on Anthropic nonvoting preferred stock plus $4.5 billion in reclassification gains on converted notes, per Amazon's 10-Q MD&A), representing 55% of reported profit, confirmed explicitly in Amazon’s 10-Q MD&A as arising from upward adjustments on Anthropic preferred stock and gains on Anthropic convertible notes converted to equity during the quarter. None of it has been received in cash. All of it appears in earnings reports that move stock prices, fund buybacks, and get cited as evidence of AI-era strength. Some circulating analyses have understated the scale; the primary source numbers are more dramatic than the summaries suggest.

The concentration risk is perhaps the most underappreciated dimension of this problem. Microsoft’s commercial remaining performance obligations, future contracted revenue not yet recognized, stand at $627 billion, up 99% year-over-year. Oracle’s equivalent figure is $552.6 billion, a number that barely existed twelve months prior: Oracle’s RPO was $130.2 billion as of February 2025. Both companies describe this growth as driven by “significant cloud contracts.” Neither discloses customer-level concentration in their public filings. Specific figures circulating in market commentary, attributing roughly half of both companies’ backlog to a single AI counterparty, cannot be verified from primary SEC filings and should be treated with caution.

What can be said with confidence: these numbers are large, they grew extraordinarily fast, and their growth is explicitly tied to AI-era cloud commitments whose counterparties are not named. That structure itself is the governance signal. When the durability of half a trillion dollars of contracted future revenue depends on an opaque concentration of AI relationships, and neither the companies nor their auditors are required to disclose the composition, the accountability gap is structural, not incidental.

What the more sensationalist versions of this argument overstate: the Qwest and Global Crossing comparison. Those companies swapped identical fiber capacity back and forth to manufacture revenue from nothing. AI cloud commits involve actual compute services being rendered: real infrastructure, real model training, real inference workloads. The circularity is in the funding mechanism and the accounting treatment, not in the underlying economic activity. That distinction matters for the legal and fraud questions. It matters somewhat less for the durability question, which is the one operating organizations should care about.

What This Looks Like From an Operational Perspective

Here is where both pieces of intelligence miss their most important implication.

Whether the AI investment boom is real or an accounting artifact is a question about returns. It will be answered by markets over time. Investors can hedge, diversify, or exit. The calculus is financial.

For operating organizations, the calculus is different. You are not holding a position in an AI company. You are building operational dependency on one. The distinction carries consequences that do not appear in a portfolio.

Consider what enterprise AI adoption actually looks like in 2026. Organizations are embedding AI into workflows, approvals, customer interactions, and core processes. The AI vendors supplying these capabilities, including model providers, infrastructure layers, and application platforms, are running at operating margins that would be terminal in any conventional business. OpenAI posted a negative 181% operating margin in the first half of 2025, per The Information’s reporting on financial disclosures shared with investors. It remains solvent because capital continues to flow. Capital continues to flow partly because the circular mechanism described above keeps valuations elevated. Valuations staying elevated depends on continued investment rounds at higher prices, which depends on the story holding.

That is a chain with real links. Each one is load-bearing.

The capital expenditure numbers make the constraint visible. Amazon’s Q1 2026 capex was $44.2 billion in a single quarter. Its trailing twelve-month free cash flow, after that capex, was $1.2 billion: a 95% decline from the prior year’s $25.9 billion. This is not a signal of distress in a company Amazon’s size, but it illustrates the scale of infrastructure commitment required to sustain current AI vendor relationships. The investment is real. The returns are, as yet, substantially paper.

The governance question is not whether OpenAI or Anthropic will fail. It is whether your organization has ever formally assessed that dependency as a risk, assigned ownership to that assessment, and built a contingency against it. In most organizations, the answer is no. Not because the people involved are negligent, but because the frameworks for evaluating AI vendor risk were built for a different era, one where the vendors were large, profitable, and financially opaque in conventional ways, not loss-making at scale and financially opaque in novel ones.

The Questions Nobody Is Asking Internally

The investment community has developed new due diligence frameworks for this environment. Bain’s AI stress test asks whether AI will change a target’s value proposition, cost structure, and competitive basis. According to Bain’s 2026 M&A Report (based on a survey of more than 300 M&A executives), one in five strategic dealmakers walked away from a deal in 2025 specifically because of the anticipated impact of AI on the target’s business.

That rigor is being applied to acquisitions. It is not being applied to vendor relationships.

The questions that should be on the table in any governance review of AI infrastructure:

Financial durability. What is the operating margin of your primary AI vendor? What funding conditions sustain their current pricing and service levels? What happens to your contracted terms if they are acquired, restructured, or face a liquidity event?

Concentration exposure. Does your vendor’s revenue depend heavily on a small number of counterparties whose own positions are themselves dependent on continued funding cycles? If the top of that chain experiences stress, what is the transmission mechanism to your service?

Continuity planning. If your primary AI vendor became unavailable in 90 days, what breaks first? Who owns that assessment? Has it been stress-tested?

Accountability structure. Who in your organization is responsible for vendor financial durability as a risk category distinct from vendor performance and security? In most governance structures, this is nobody. It falls between procurement, IT, legal, and finance without clear ownership.

Data and switching costs. If you needed to migrate, how portable is your data, your fine-tuning, your embedded workflow logic? What is the realistic switching cost in time, capital, and operational disruption?

These are not speculative questions. They are the questions that should accompany any material operational dependency on a vendor category characterized by extreme valuation dispersion, circular capital structures, and winner-take-all dynamics that have not yet resolved.

The Framing Problem

There is a reason these questions are not being asked systematically.

AI is being treated as a technology decision. Technology decisions go through IT, security review, and increasingly an AI policy layer focused on data privacy and output risk. That review is necessary but insufficient. It evaluates AI as a product. It does not evaluate AI vendors as counterparties with their own financial fragility and structural dependencies.

The analogy that may be useful: in the aftermath of 2008, organizations discovered that their exposure to financial counterparty risk was far more interconnected than their risk frameworks had captured. The risks were visible in the data: leverage ratios, funding structures, concentration. But they were not in anyone’s formal accountability structure until they became someone’s crisis.

This is not a prediction that an AI vendor will fail the way Lehman failed: suddenly, binary, with no off-ramp. The more plausible failure modes are slower and less dramatic: pricing changes under funding pressure, degraded service levels, capability redirection as a vendor pursues different markets, term renegotiation following an acquisition. The governance lesson from 2008 is not about the speed of the collapse. It is about the gap between visible risk and formal accountability. That gap is identical.

The current AI vendor landscape has analogous characteristics. The financial fragility is visible in public filings for anyone willing to look. The circular capital structures are described, if obliquely, in earnings disclosures. The concentration risk is in backlog numbers. None of this requires insider access. It requires treating AI vendors as counterparties rather than products.

There is a second 2008 parallel worth drawing precisely, because it captures something the counterparty framing alone does not. Before 2008, bond insurers (Ambac, MBIA, and a small number of others) had guaranteed trillions in structured credit obligations. Each institution that purchased that insurance made a reasonable individual risk assessment. Their own exposure was measurable. Their own models were defensible. What no single institution’s model captured was the aggregate: that the same two or three guarantors stood behind nearly everyone’s exposure simultaneously. When those guarantors came under stress, the tail risk did not manifest as one firm’s problem. It manifested as a sector’s problem: multiple organizations experiencing the same disruption simultaneously, competing for the same alternatives, with the same gaps in their contingency plans.

The AI vendor landscape has the same structural property. Enterprise adoption has concentrated rapidly into a small number of model providers and infrastructure layers. Each organization making that dependency is making a reasonable individual assessment of its own exposure. What is not being assessed, formally by anyone, is the aggregate. If a primary model provider experiences a funding disruption, a capability redirection, or an acquisition that changes its service terms, the organizations affected are not one firm working through an isolated vendor problem. They are a sector absorbing the same disruption simultaneously, with the same switching costs, competing for the same migration paths. The concentration has created a dependency that individual risk assessments are structurally unable to capture.

Most organizations are not doing that yet. The ones that build that discipline now will be positioned differently when the investment debate resolves, whichever direction it goes.

What Adequate Governance Actually Requires

This is not an argument for slowing AI adoption. It is an argument for building the control layer that makes AI adoption durable.

Adequate governance in this environment means:

A vendor financial durability assessment that treats AI providers as counterparties, evaluated on operating model sustainability, funding dependency, and concentration exposure, and updated on a regular cadence as conditions change.

A continuity framework that identifies which AI dependencies are mission-critical, documents the failure scenarios, and assigns clear ownership for response. This is not disaster recovery in the traditional sense. It is counterparty contingency planning applied to a new vendor category.

A clear escalation structure that determines when AI vendor risk crosses a threshold requiring board-level awareness. The standard should not be “when something goes wrong.” It should be defined in advance, based on the materiality of dependency relative to the organization’s risk tolerance.

An accountability assignment that places vendor financial risk somewhere specific in the organizational structure, not distributed across procurement, IT, and finance with no single owner.

None of this is exotic. It is the application of governance principles that exist in financial services, regulated industries, and mature enterprise risk functions to a vendor category that has so far escaped that discipline.

The investment community is asking the right questions about where returns will come from. The operating community has a different and more immediate question to answer: if those questions turn out badly, who in your organization saw it coming, and what had you built to absorb the impact?

The gap between narrative and fundamentals is where sophisticated investors position themselves. The gap between vendor exposure and governance structure is where organizations get surprised. The uncomfortable truth is that both gaps exist right now, in the same market, at the same time.

Sources: Alphabet, Amazon, Oracle, and Microsoft financial figures are drawn from each company’s Q1 2026 or Q3 FY2026 SEC 10-Q filings and verified against primary MD&A disclosures prior to publication. OpenAI figures are sourced from The Information’s October 2025 reporting on financial disclosures shared with investors; as a private company, OpenAI’s financials are not independently verifiable from public filings. The Bain M&A statistic is from “Looking Back at M&A in 2025: Behind the Great Rebound,” published in Bain’s 2026 M&A Report series, December 2025, authored by Suzanne Kumar, Dale Stafford, Kai Grass, David Harding, and Kristen Stikeleather. Nothing in this piece constitutes investment advice. Thomas Tornatore is the founder of Fellowship Intelligence, an AI governance and strategy-layer advisory firm. He is the author of The Wrong Default: How Absence Becomes a Decision, and Who Pays the Cost.

Book a governance conversation

It starts with a business owner telling me they haven’t adopted AI yet. They’re being cautious, they say. Waiting until they understand it better. They’ve seen what’s happening with other companies and they want to do it right when the time comes.

Then I ask what software their team uses.

Microsoft 365, usually. Sometimes Google Workspace. A CRM. A project management tool. Video conferencing. Maybe an AI phone agent someone purchased because a vendor told them it would save time on calls.

I ask whether Copilot is enabled in their Microsoft environment.

Most of the time, they don’t know.

I ask whether their team is using Gemini features in Google Docs or Gmail.

Usually, they think so. Maybe. The team seems to like some of the new features.

I ask who decided that client communications could flow through those systems.

The conversation gets quiet.

Here is what I have come to understand from these conversations. Most business owners are not avoiding AI. They are running AI they did not choose, on data they did not intend to expose, under terms of service they did not read, through software updates they clicked through without knowing what changed.

They believe they are being cautious because they have not purchased a dedicated AI platform or made a formal AI adoption decision. What they have actually done is let the adoption happen by default, one software subscription at a time.

This is not a criticism. It is the reality of how AI has entered the market. The major software vendors made a business decision to embed AI capabilities into existing products rather than sell them separately. That decision made AI ubiquitous faster than any dedicated AI platform could have. It also meant that billions of people and millions of businesses started using AI without making a choice to do so.

The choice was made for them.

What concerns me about this is not the technology. The technology works. What concerns me is the accountability gap it creates. When an AI system produces an output that influences a decision, and that decision produces a consequence, someone is accountable for it. The software vendor is not accountable. The terms of service are not accountable. The accountability lands on the organization, on the person whose name is on the client relationship or the professional obligation.

If that person did not know AI was involved, they cannot explain the decision. If they cannot explain the decision, they cannot defend it. And in an environment where AI-influenced decisions are beginning to attract regulatory attention, plaintiff interest, and client scrutiny, the inability to explain a decision is not a minor operational gap. It is the exposure.

I write about this because I do not see anyone else writing about it at the level where it actually matters. The AI conversation in most media is either about enterprise deployments at companies most business owners will never work for, or about the existential possibilities of a technology that most people are still trying to understand on a basic level. Neither of those conversations reaches the financial advisor with eight employees who upgraded to Microsoft 365 Business Premium and had no idea Copilot started summarizing their client meetings.

That person is not careless. They are running a business. They trusted that the software they were paying for was working in their interest. They did not know that the update changed what the software was doing, who had access to the outputs, or what the terms now authorized.

They are the person I am writing for.

If you are reading this and you are not certain what AI is running inside your business software right now, that uncertainty is the place to start. Not because something has necessarily gone wrong. But because the first time it matters, you will want to have made the decision yourself rather than discover that someone else made it for you.

That is the conversation I keep having. I write it down here because the people who need it most are not always the ones I get to talk to directly.

If you found this useful, the newsletter goes deeper every week. And if you want to talk through what AI is actually running in your business, schedule an appointment at book.fellowshipintelligence.com/#/thomas

Discuss the topic

Seat volume tells you how many people have access. It tells you almost nothing about who shaped the conditions of that access, whose risk assessment governed what was permitted, or who will determine whether deployment scales or stalls twelve months from now.

That story belongs to a different class of people — one the market has largely chosen to ignore.

Inside most risk-sensitive organizations, AI adoption decisions don’t originate with end users and don’t terminate with IT procurement. They move through an intermediate layer: legal counsel evaluating liability exposure, compliance officers setting acceptable use boundaries, governance advisors building the policy structure that makes deployment defensible, CISOs determining what data cannot connect to which systems, fractional executives and board advisors framing the decision for leadership before the vendor ever presents.

This is the advisor class. They are not in the seat count. They are rarely in the CRM. And in most enterprise AI distribution strategies, they are treated as friction to be managed rather than infrastructure to be built.

That is a significant miscalculation — and the market has already produced direct evidence of how deep it runs.

Consider how most major AI vendors handle data protection agreements. In many cases, a DPA — the contractual instrument governing how a vendor handles your data, whether your inputs are used for training, what retention controls exist — is reserved for commercial accounts. Often behind a minimum seat threshold.

I’ve encountered this directly. To obtain contractual protection for my own intellectual property, I am required to purchase a minimum number of seats — not because my organization needs them, but because the vendor’s pricing model has no mechanism for protecting individual IP outside of a commercial structure. The protection is not scoped to what I’m putting into the system. It is scoped to how many people I’m paying to use it.

The implication extends well beyond my situation. Writers are feeding unpublished ideas into these tools. Authors are working through argument structures that represent years of intellectual development. Researchers, strategists, and independent advisors are processing the frameworks that constitute their professional IP. And the vendors processing that material have, in many cases, made basic data protection unavailable to them — not because they assessed the risk and declined, but because their pricing architecture never contemplated that a single user with significant intellectual assets would require protection.

This is not a niche edge case. It is a structural design failure — and it maps precisely onto the market’s broader blindness to the advisor class.

The same professionals who shape organizational AI adoption decisions are often personally unprotected by the systems they’re evaluating. A governance advisor helping an organization design its AI policy framework may have no enforceable data handling agreement with the AI vendor they’re working within to build that framework. That is not just a personal risk. It is a credibility problem for the organizations trusting their guidance.

The market has seen this pattern before. Healthcare took decades to recognize that physician influencer networks — not hospital procurement alone — shaped pharmaceutical adoption at scale. Enterprise security vendors eventually learned that a CISO’s informal veto could collapse a deal that cleared every other gate. The same dynamic is now structuring enterprise AI, and most of the market is still optimizing for access metrics while the actual adoption decisions are being made one layer up.

What correcting for it requires is not complex. At the vendor level: decouple data protection from seat minimums. IP protection is a function of what you’re putting into the system, not how many people are using it. At the GTM level: treat the advisor class as a distribution channel — build the frameworks they can put their names behind, the governance architecture that makes organizational adoption defensible, and the trust that generates recommendation rather than resistance.

The organizations that understand this shift have a clear path: bring the advisor class into the adoption decision as a structured input, not an afterthought. The advisors that understand it have leverage they are currently underusing. The vendors that understand it first will be embedded in the influence layer before their competitors realize the influence layer exists.

The enterprise AI market is selling access. The next wave of deployment will be governed by permission. Those are not the same decision, and they are not controlled by the same people.

If your organization is navigating where the real adoption decisions are being made — and who should be making them — that conversation starts at consult.fellowshipintelligence.com.

Share

Business leaders are waiting.

Waiting for the government to set the rules. Waiting for the technology companies to figure it out. Waiting for a clearer picture before they commit to anything.

The waiting is understandable. But it is a mistake.

WHO ACTUALLY OWNS AI RESPONSIBILITY?

Someone on a business message board recently asked a question worth sitting with: who is ultimately responsible for AI?

The honest answer is all of the above — government, technology companies, communities, and business leaders — but not equally and not in the same way.

Government sets the outer boundaries. Technology companies design the systems. Communities shape expectations. But company leaders make the operational decisions that determine how AI actually gets used inside an organization.

Which data goes in. Which use cases get approved. Which outputs get reviewed before they affect a real decision. Who is accountable when something goes wrong.

That is the accountability layer that matters most right now. And it is the one most organizations are leaving unstructured.

THE SOUTH AFRICA LESSON

Earlier this year, South Africa’s government withdrew a draft national AI policy after it was discovered that the document contained fake citations — references to sources that do not exist, generated by AI.

This is not an isolated incident. It is a pattern.

The problem was not that AI hallucinated. That is expected behavior — known, documented, widely understood. The problem was that the output moved through a formal government process without enough verification. No one with authority caught it before it became an official document.

For businesses, the parallel is uncomfortable. How many AI-generated outputs are moving through your workflows right now with no formal review process? How many decisions are being shaped by AI-produced analysis that no one has independently verified?

The risk is not that AI is dangerous in the abstract. The risk is that AI is being used quietly, without clear ownership, without accountability structure, and without anyone formally responsible for the outcome.

AI RELOCATES ACCOUNTABILITY. IT DOES NOT ELIMINATE IT.

This is the point most business leaders are missing.

When your organization uses AI to make or influence a serious decision, the accountability does not transfer to the technology. It stays with you. The AI changes where the question sits — not whether the question exists.

Who reviewed the output?

Who approved the use case?

Who checked the source material?

Who decided the risk was acceptable?

Who is accountable when the system is wrong?

If you cannot answer these questions for your current AI use cases, you do not have a technology problem. You have a governance gap.

Governance is not a regulation you wait for. It is a decision you make.

WHERE TO START

You do not need to build a complete AI governance program this quarter. You need to answer three questions for your organization:

What AI tools are currently in use inside your organization — formally or informally?

Which decisions or outputs are being influenced by those tools?

Who is accountable for reviewing those outputs before they affect a real outcome?

If you do not have clear answers, that is your starting point. Not a technology audit. A governance conversation.

AI will continue to improve. The capabilities will expand. But the accountability question will not go away — it will get more complex as the tools become more embedded in how decisions get made.

Organizations that build governance structures now will be better positioned to use AI with confidence, not just with caution.

That is the difference between organizations that adopt AI well and organizations that end up in the headlines for the wrong reasons.

Fellowship Intelligence is an AI governance and structural advisory firm. If this raised questions about your organization’s AI governance posture, a discovery call is the right next step.

https://book.fellowshipintelligence.com/#/discovery

Schedule a Conversation