Confidence is not correctness
Welcome to the first issue of The Agentic Architect: one pattern, one case study and the AI news worth your time.
Pattern: check confidence from the outside
Plenty of engineers gate an agent's actions on the model's own confidence score: 0.95 goes through, 0.40 goes to a human. That number measures how smoothly the tokens line up, not whether the answer is right, and after RLHF it drifts away from accuracy.
Check certainty with something the model does not control: a ground-truth set it never saw, deterministic tests in code and thresholds calibrated on your own data. Use confidence at most to rank answers that get verified anyway. A test for today: do your highest confidence scores line up with correct answers in the logs?
Case study (proof of concept): the model judges, code scores
A global consumer brand wanted to know which press articles support its messaging and which undermine it. In the proof of concept I built, the LLM only says how an article relates to each strategy pillar: support, neutral or contradict. Pure Python computes the score and the verdict, and every quote behind a verdict is checked word for word. The model never sets the score.
AI news from the last two weeks
- OpenAI DevDay 2026: dots (ChatGPT's always-on agents), GPT-6.1 Sol, the Agents API and Codex cloud, with prices and what works in the EU.
- Gemini 4 Argon: answers of up to 1M tokens, Google says, at $2 per million input and $10 per million output tokens to start, then $4 and $20. Google has given no date yet for wider availability, and nothing for the EU.
Also this week
New on the blog
- What is a forward deployed engineer? One of three new posts on the role.
- Web3, AI, robotics: what remains when the hype fades, my answer from the Polish podcast "I to się liczy", expanded, with sources.
Free practice exams for all four Claude certifications
Four mock exams, one per certification, each with the question count from Anthropic's exam guide, the 120-minute limit and an explanation for every answer. No account needed. The 60 Architect Foundations questions are also a free PDF with the answer key, no email required. Unofficial practice material, not authorized or endorsed by Anthropic.
For consultants
Want to sit the Claude exams without a partner employer? The Certified AI Consultant Program is taking applications. Admission is by selection. Applying is free; membership is paid, and only once you are accepted.