
Photo by engin akyurt on Unsplash
A supplement to my ProductTank Global talk on managing AI agents as a product leader.
I worked on my first AI product in 2018, when I moved to London. I started using ChatGPT at the end of 2022, when I was working at StepStone. We had to experiment with it for professional reasons, and the products I was working on were ambitious and LLM-forward. My personal use was in fits and starts. While I was enabling others, I was more sceptical of it for my own daily use.
Later, at Sainsbury's, I was researching how to use AI, how to understand it, and what it would mean for large enterprises and their data. The volume of information on AI and data was overwhelming. So much of it was hype. I was trying to sift through it and find some signal amid the noise.
The quote that wasn't there
I started pulling quotes that looked pretty good, and I didn't think anything of it. They came from what felt like small, easy jobs, so I didn't double-check them the way I should have.
Then I was getting ready to publish. In my final edit I realised I had built the entire piece around a quote that wasn't in the source material.
That's when I knew I had a problem.
My first reaction was to assume the AI was always wrong. That made research difficult. Checking every citation myself and running the work again left me exasperated, and I still couldn't get the results I wanted.
It took me a while to see the real problem. Prompt instructions aren't the same as code.
Asking harder wasn't enough
In 2025 I moved to Claude. I tried chat first, with prompts and separate projects for different stages of the work, then Claude Code, and I've learned a lot since. My response to the quote problem was to put more into the instructions. Don't lie. Double-check. Keep the stages separate.
They were still only prompts. The more I squeezed, the less reliable they got.
There is research on part of that problem. The RECAST paper examines the difficulty language models have following instructions with many simultaneous requirements. Its authors also describe training methods that improve that performance. It gives me a useful way to think about the limits of asking a model to hold a growing set of requirements at once.
The idea that extra restrictions caused my agents to invent citations remains a working theory about my own experience. RECAST measures instruction adherence; it doesn't test that explanation.
What I was missing was the difference between a prompt and a check. A prompt asks the model to do something, such as only use accurate quotes. The model weighs it alongside everything else it has been asked. A check runs after the work is done. It is code that tests the result against something fixed, such as the source text, and it gives the same answer every time it runs. That property is called determinism, and it is what a prompt can't give you. A checklist written into the prompt doesn't change that. The model still tries to satisfy it, and nothing outside the model confirms the result.
A quote gives you something specific to check
In August 2026, research became a concrete place to work on that difference. I started using byte-identical verification for quotes and links: an exact, character-for-character match. Compare the words being presented with the source text. Check whether the link being carried through is the same link.
Those are narrow questions. A matching quote can still be taken out of context. A matching link can still point to a weak source. Neither tells me whether the research is worth using. But they give me something specific to test, instead of another instruction asking the agent to be careful.
If a check compares text, I can use it to answer a question about text matching. I still need to decide whether the quotation supports the point I'm making and whether that point belongs in the piece at all.
That's where a second layer of checks helps. The code check runs first and answers the narrow question: is this quote in the source? A second layer, a model reviewing the work against a short list of questions, can look at what code can't, such as whether the quote has been taken out of context. It isn't deterministic. It adds to the first check rather than replacing it. Each layer catches different mistakes, and by the time a draft reaches me, what's left is mostly the work that needs my judgement.
Assuming every output is wrong doesn't help with that split. It leaves me with the whole output to inspect.
The work still needs a leader
Before any work starts, I decide the outcome and name the evidence that would show it worked. That's the Impact Chain. Unlike OKRs, it happens per item, not per quarter, and it works with or without AI.
Research makes that concrete for me. A citation I can inspect is part of the evidence. Choosing what deserves to be cited still needs my judgement. A system that passes a text comparison hasn't decided what I should say.
My first guards were too strict. They made more work for me, not less. Full autonomy didn't work either. Now the guards stop the worst cases, and drafts still reach me.
I'm trying to get useful work I can respond to, while keeping the decisions that need to stay with me. More instructions didn't get me there on their own. Code-assisted checks have given me another way to work on it.
Take one question back to your team
If you've used chat AI but haven't yet handed work to agents, start small. Pick one piece of work you ask AI to do. Where are you asking it to be careful, and what could you check directly?
For research, quotes and links gave me somewhere to start. Your starting point may be different. Name exactly what the check would establish, then name what you would still need to judge after it passed.
The talk and resources include the Impact Chain and the useful links. You can also try Dualtrack, my product team of agents for Claude Code.
Where does checking AI's work take most of your attention today?
Free giveaway: the Impact Chain
A one-page tool for teams shaping AI product strategy. Ten discussion points can help you facilitate hard conversations and build a credible AI strategy in any team. Use it to plan something new or to find the weak link in something that's stuck.
Open the Impact Chain (PDF, CC BY 4.0)
Thank you to my friends Mind the Product for having me, and to Louron Pratt and Valeria Kholkova for making it happen.
Thanks for reading.
Intentionaut is a weekly letter from Saielle DaSilva on design, product, data and AI. What's working, what isn't, and how to keep your bearings.
If this was useful, please forward it to one person who'd get something from it. That's how it grows.
Saielle DaSilva
Putting the soft back in software