AI Mistakes in Drug Development: Who Is Actually Liable?
If an AI model produces data that ends up in a regulatory submission and the data is wrong, the company that submitted it carries the responsibility. That is how Ryan Cawood, Co-Founder and CEO of Lab Thread, reads the position, and he states it plainly.
the FDA has been very clear that a company that submits data to the FDA is 100% legally responsible for that data regardless of where it comes from
On his reading, that resolves most of the ambiguity people carry into AI adoption in drug development. There is no procurement clause, no disclaimer, and no model card that moves the liability somewhere else. If it is in your submission, it is yours. What remains genuinely open is the second question: inside the company, who personally answers for it.
Who personally answers when the model is wrong?
Raphaël Ognar, President, CEO, Co-Founder and Chairman of NKILT Therapeutics, moves the question from the entity to the individual. Corporate liability is a balance-sheet event. The people carrying the regulatory and clinical responsibility carry something else.
the chief medical officers the physicians they are liable for everything that comes out and that affects patients
Ognar spent years running drug development programs inside big pharma and carried two small molecules through to approval before founding his own company, and his framing is not that this makes AI too risky to use. It is the opposite. Because a named human being answers for what reaches a patient, the human review layer is the part that cannot be delegated to the tool. The model can draft, summarize, check and search. It cannot hold the responsibility.
What happens if you cannot show where an answer came from?
This is the question that tends to worry people most, and Ognar's answer is more uncomfortable than the one most teams expect. He does not treat it as a novel AI problem at all.
It should have been spotted earlier. ... if a CMO doesn't find this or if his team doesn't find this we have a serious problem of competence
His argument is that a result nobody can trace should have failed a review long before it reached a submission, and that the same failure is possible with a careless human analyst and no AI in the room. In a small organization he expects the chief medical officer to be close enough to the work to catch it.
in a small organization when you have even less people the CMO has to be very much hands-on
Read that way, provenance is not a new compliance burden invented by machine learning. It is the existing standard, applied to a faster and more confident source of error.
Why is a confident wrong answer harder to catch than a refusal?
The reason this liability question has teeth is that the failure does not announce itself. Both guests describe models that do not decline when they are out of depth. They produce something fluent, in the exact register of a correct answer. Cawood's rule of thumb is to extend to software the skepticism you already extend to people.
it really does need to be treated like a colleague. You wouldn't trust everything that a colleague told you as gospel all the time
The gap he is pointing at is a behavioral one. Most scientists already verify a colleague's surprising claim by reflex. Very few apply that reflex to a well-formatted paragraph returned in two seconds, which is precisely where an unverified result enters the record.
Does the FDA have a position on AI in regulatory submissions?
It does, and it is worth reading rather than inferring. The agency has published draft guidance titled Considerations for the Use of Artificial Intelligence To Support Regulatory Decision-Making for Drug and Biological Products, which sets out how the agency thinks about credibility of AI-derived evidence. The underlying obligation it sits on top of is older and blunter: under 21 CFR 312.50, sponsors carry general responsibility for the conduct of their investigations, including selecting qualified investigators and ensuring the work follows the protocols in the IND.
Neither document tells you which model to use. Both make the same structural point Cawood makes from the operator's side, which is that the accountability is fixed at the sponsor and the tooling choice does not move it.
What does this mean for a small biotech in practice?
Ognar is direct about the asymmetry between a company that can absorb a regulatory setback and one that cannot. A large pharmaceutical company has the balance sheet to survive a delay. For a small biotech, the same delay is one fewer chance of reaching the finish line, which we look at in more detail in our companion piece on why AI errors in biotech surface months too late. The same asymmetry runs through the cost of the tooling itself, where the price a buyer sees today is being underwritten by the provider rather than earned, as we cover in why AI is so expensive to run.
The practical implication is not to avoid AI. Both of these founders use it. It is that the review layer has to be designed in rather than assumed, and it is the same failure that sinks institutional AI programs for reasons that have nothing to do with the model, as the FDA’s first chief information officer described in why government AI projects fail. The company that treats a model output as a draft requiring verification is in a very different regulatory position from the company that treats it as a result. Designing that layer in starts at the request itself, by asking for the work in stages you can inspect, which we set out in how to prompt AI for scientific research.
The convener's read
Neither guest argues about whether AI belongs in drug development. Both have already decided that it does. Their disagreement is narrower and more useful: where the human review sits, who is named on it, and how quickly an unverified answer can travel before someone checks it. What neither of them does is advertise the fact, which is a separate decision one of them is currently testing on investors in should startups put AI in their pitch deck.
The full conversation is on the episode page, along with more from Ryan Cawood and Raphaël Ognar. If your organization wants to reach the people making these calls, partner with Open Door Salon.
Methodology: every quotation above is drawn verbatim from the recorded, on-the-record conversation between Ryan Cawood, Raphaël Ognar and host Lori Ellis on Open Door Salon, and was checked against the episode transcript before publication.