This is the twelfth part in a series on how AI is crossing out of software and into the physical world. This time, Jan Bosch looks at what it takes to establish that something is real when anything can be synthesized.
Let me dispose of detection first, because it’s where most of the money and nearly all of the public conversation has gone, and it’s the wrong problem. Detection asks: Is this artefact fake? That’s an adversarial classification task in which your opponent improves continuously, knows your classifier’s behavior and can iterate against it for free. Worse, a detector must generalize to generators that don’t exist yet. You can win this for eighteen months at a time; you can’t win it structurally. Every serious person in the field worked this out some years ago, which is why the industry pivoted to the question you can actually answer: not whether something is fake, but where it came from.
That pivot is correct, and the infrastructure is further along than most people realize. The C2PA standard, Content Credentials, is now shipping in hardware. Cameras from Leica, Sony, Nikon, Canon and Samsung sign images at the moment of capture, with Apple and Google announced for later this year. Adobe’s tools read, preserve and write credentials, Google Search surfaces them, Meta reads them on upload, Tiktok labels with them, Linkedin preserves the chain and X added display in March (for premium users). The Content Authenticity Initiative has over two hundred members, and the BBC, New York Times, Reuters, AP and Washington Post are signing their output.
So: a real standard, real silicon, real platform support. And then you meet the holes: Email doesn’t preserve credentials. Messaging apps strip metadata. Most content management systems can’t carry the manifest. And, in the understated phrasing of the adoption trackers, the screenshot problem remains unsolved. Which is to say that a cryptographic chain of custody, signed in hardware at the sensor, authenticated through an editing pipeline, is comprehensively defeated by pressing three keys.
The other leg of the strategy is watermarking: marks embedded in the output itself, which should survive the journey that metadata doesn’t. Here, the research is genuinely sobering. A July 2026 empirical evaluation by Tamim and Khan tested three text watermarking schemes: KGW, Unigram and the open-source SynthID-Text implementation, and asked a simple question: Would this stand up as forensic evidence? Before any attack at all, missed detection rates ran from 70 to 83 percent. Then they applied a meaning-preserving paraphrase with a small open model. Among texts the detectors had initially flagged, the watermark was removed in 100, 100 and 98.3 percent of cases, respectively. The authors conclude that none of the schemes meet the evidentiary standard courts require.
The caveats are real: one attack type, short texts, small samples and the open-source rather than production SynthID. But the direction isn’t ambiguous. A watermark that a 1.5-billion-parameter model can wash out with a paraphrase isn’t a chain of custody.
Which produces a collision worth sitting with. Since 2 August 2026, Article 50 of the EU AI Act has been in force. Providers of generative AI must embed machine-readable markings in synthetic audio, image, video and text so that it’s detectable as artificially generated. Deployers must disclose deepfakes and AI-generated text on matters of public interest. Systems already on the market have until 2 December to comply. Penalties reach 15 million euros or 3 percent of worldwide turnover.
I want to be careful here, because this isn’t an argument against the regulation. Requiring marking is reasonable, it raises the cost of casual deception, and the alternative, requiring nothing, is worse. But compliance and efficacy have come apart. Providers can be fully, demonstrably compliant with Article 50 while the markings they embed don’t survive a paraphrase. We should be honest that we’ve mandated a measure whose robustness the literature doesn’t support, and that a compliance checkbox will be mistaken for a solved problem.
And now the thing I actually want to argue, which survives all of the above being fixed. Suppose provenance works perfectly. Every camera signs. Every platform preserves. The screenshot problem is solved by some mechanism we haven’t yet invented. What have we got? We’ve established that a particular file came from a particular device at a particular time and wasn’t altered afterward.
Although genuinely valuable, it isn’t what we needed. Because the hard cases were never synthetic. The most effective disinformation in history has used entirely authentic images with false captions: a real photograph from a different conflict, a genuine video from three years earlier, an unedited clip of someone saying something true in a context that inverts its meaning. Every one of those passes provenance verification perfectly. The manifest confirms the pixels. It has nothing to say about the claim.
We’ve built excellent infrastructure for the question “Where did this come from?” and we’re deploying it against the question “Is this true?” Those are different questions, and the gap between them is where the next decade of deception is going to live. Not in synthetic media that fails verification, but in authentic media that passes it while meaning something else entirely.
I should say plainly that this is also the boundary of the argument I’ve been making throughout this series. I’ve maintained that when you can’t verify an artefact directly, you verify the contract and the chain of evidence around it. Provenance is exactly that idea applied to media: evidence about process rather than about content. It’s the right architecture, and here, its limit becomes visible. A chain of custody tells you a thing was handled correctly; it can’t tell you what the thing means.
For startups, that gap is the opportunity, and it isn’t the one most of the funding is chasing. The layer below, such as signing, manifests and verification plumbing, is becoming standardized infrastructure, which is the correct outcome and a poor business. The layer above is wide open: corroboration rather than authentication. Does this image’s claimed location match its shadows and weather? Has this clip appeared before under a different caption? Do three independent sources place this event where it says it is? That’s a reasoning and cross-referencing product, not a cryptography product, and nobody has built a good one.
There’s also a straightforward enterprise market in verification as a control. The fraud pattern of the moment is a convincing voice on a call authorizing a payment, and the defense isn’t better detection but process: out-of-band confirmation, signed instructions, authority limits. That’s the argument of the previous part in this series, reversibility and gating, arriving in the finance function.
Platforms are in the hardest position, and their exposure is subtler than it looks. They’re where the trust judgment actually happens, so they’re being compelled to own authenticity whether they want to or not. But the moment you display a credential, you’ve made an implicit statement about everything you didn’t flag. A label that appears on verified content tells users something about unlabelled content too, and that inference will be wrong constantly: for screenshots, for email attachments, for anything that passed through a messaging app.
For society, the problem I find most troubling isn’t deepfakes; it’s the inversion that successful provenance creates. Once signed content is the norm, unsigned content becomes suspect. That seems fine until you ask who can’t sign. The witness filming with a seven-year-old phone. The journalist in a country where no vendor sells signing hardware. The whistleblower who must strip metadata precisely because metadata identifies them. The activist whose footage reaches the world as a screenshot of a screenshot because that was the only way to get it out.
Provenance infrastructure protects institutions considerably better than it protects individuals, because institutions have procurement budgets and compliant equipment and no need to be anonymous. And historically, the evidence that most needed to be believed has come from precisely the margins that will now struggle to produce a manifest. We’re constructing a system in which the powerful can prove their content is real and the powerless increasingly can’t.
That’s a design choice, not a law of nature, and it’s still early enough to design against it. Provenance schemes that allow after-the-fact attestation by trusted witnesses, archival signing by news organizations on behalf of sources and verification that degrades gracefully rather than failing abruptly are all technically possible. They’re simply not what gets built when the requirement comes from a camera vendor and a compliance department.
Hannah Arendt saw the structure of this problem in 1967, in an essay called “Truth and politics” that she published in The New Yorker. She was writing about political lying, long before any of this was mechanizable, and she set out the condition precisely: “Freedom of opinion is a farce unless factual information is guaranteed and the facts themselves are not in dispute.”
Note that she names two requirements, not one. Factual information must be guaranteed and the facts themselves mustn’t be in dispute. C2PA, watermarking, hardware signing, Article 50: All of it is work on the first clause. Guaranteeing that a file is what it claims to be is a solvable engineering problem, and we’re solving it.
The second clause isn’t an engineering problem, and no camera is going to sign it for us. Deciding which facts are settled is something a society does, through institutions that take time to build and are quick to lose. We’re busy perfecting the guarantee while the dispute spreads, and the uncomfortable possibility is that we finish the first job just in time to discover it was the easier half.
