AI Wrote It. AI Checked It. You Bought It.
The fastest-growing source of governance risk is not the AI your team adopted. It is the code no human ever read, arriving inside what you buy.
When I review an AI-powered product or a vendor’s implementation plan, I have a habit. Before I look at what it does, I look for the person who can explain how it does it. Lately, on a certain kind of project, that person is not there.
The code runs. It demos well. It shipped. And no one on the team can tell me what it actually does at the level that matters, because an AI wrote it and an AI was the only thing that checked it. The pattern is consistent enough now that I can name the moment it goes wrong, and it is not a moment of malice or even carelessness. It is a sentence said with confidence: I let the AI write it and the AI review it, so I am good.
That sentence is the failure. Not the AI, not the speed, not the founder moving fast. The sentence.
Because it describes a closed loop. The author and the reviewer are the same machine. A system grading its own output is not a review, it is a second opinion from the same witness. And the human who was supposed to sit outside that loop has quietly stepped out of it, reassured by the one thing that should reassure no one: it runs.
Running tells you almost nothing. We now have the data to say that plainly. Veracode tested AI-generated code from more than a hundred models across four languages and found that 45% of it shipped with security flaws from the standard OWASP list. In Java the failure rate was 72%. And the most important finding is the one that gets the least attention: as the models improved, they got better at writing code that works and no better at writing code that is secure. The two skills came apart. So “it runs” and “it is sound” are now separate questions, and the loop only ever answers the first.
The human cost compounds it. Stanford researchers found that developers using AI assistants wrote less secure code than those working by hand, and were more confident in its security while doing it. Sit with that. The tool does not just let the review lapse. It produces the feeling that no review was necessary. Confidence goes up while correctness goes down, which is the exact condition under which no one thinks to look.
And often, by the time something goes wrong, the person who shipped it could not look if they wanted to. They cannot read the headers on the code blocks to see what each step actually instructs. This is not a competence insult. It is what the default manufactures. When the loop is closed tightly enough, the human is removed from it so completely that they lose the literacy to re-enter. You cannot own what you cannot read.
Here is where this stops being the builder’s problem and becomes yours.
None of this stays with the shop that built it. The artifact leaves. It ships as a product you license, or as a vendor implementation you approved, and it enters your environment, frequently a regulated one, carrying every gap no human ever caught. And when it fails, the liability does not travel back to the builder. It lands on the deployer. The principle is already settled in plain terms: a company is accountable for what its AI produces, regardless of who produced it. When Air Canada’s chatbot invented a refund policy, a tribunal held the airline to it. The defense that a system, not a person, made the error did not survive. No regulator and no court is going to accept “the AI wrote that part” as a reason the obligation does not apply to you.
So consider the chain honestly, because no one in it did anything wrong. The vendor shipped capability to the customers who contracted for it. The builder kept the product moving. The buyer enabled what they bought. Each act is reasonable. None of them is a review. The review is a separate thing, and it belongs to a named human who read the code and can answer for it. If no one in that chain did it, it did not happen, and now it is running inside your stack.
This is the same question I asked earlier this week, arriving from the other direction. Inside your own operation, the question was who decides what is critical. Through your supply chain, it is narrower and harder: who read the code you are about to run, and can they explain it? One is about the decisions your systems make. This is about the decisions someone else’s system already made, that you are about to inherit.
The inventory question has not changed, it has only moved upstream. Where is AI-built code operating inside what you deploy, and who is the named human accountable for having actually read it? “It runs” is not an answer. “The AI reviewed it” is not an answer. The only answer that holds up, when the thing fails and the question is who owns it, is a name.
AI operates. You own the decision. Including the decision to run code no one ever read.
Where is AI-built code running inside what you deploy, and who actually read it? If you cannot name the person, that is the gap. Tell me what you find in the comments.
This is the question my work begins with. When AI-built code reaches a regulated environment and no named human owns it, that gap is what I help close.
Sources:
Veracode 2025 GenAI Code Security Report. Veracode tested AI-generated code from more than 100 models across four languages and found that 45% shipped with OWASP-class security flaws, rising to 72% in Java. The report’s key finding: as models improved, they got better at writing code that works and no better at writing code that is secure.
https://www.veracode.com/resources/analyst-reports/2025-genai-code-security-report/
Perry, Srivastava, Kumar, and Boneh, “Do Users Write More Insecure Code with AI Assistants?” (Stanford, ACM CCS 2023). The first large-scale user study on the question. Developers with an AI assistant wrote measurably less secure code, and were more likely to believe their code was secure than those working without one.
https://doi.org/10.1145/3576915.3623157
Moffatt v. Air Canada, 2024 BCCRT 149. A Canadian tribunal held Air Canada liable for a refund policy its chatbot invented, rejecting the argument that the company was not responsible for what its own AI told a customer. The clearest statement yet that the deployer owns the output.
https://www.canlii.org/en/bc/bccrt/doc/2024/2024bccrt149/2024bccrt149.html