1
Skills — reusable, verifiable capabilities
Packaged once, they work everywhere; in my prod they run as emodul, gh-tasks, infra-nginx, deep-research — a unit of reuse, not a prompt.
Deterministic control over non-deterministic AI — from the trenches, not the slides
28 entries: Building 6 · Orchestration 6 · Closing the loop & trust 8 · Ops & HITL 8
Hover a dot or a legend row to see the rationale and source. Full list below.
28 entries: Building 6 · Orchestration 6 · Closing the loop & trust 8 · Ops & HITL 8
Packaged once, they work everywhere; in my prod they run as emodul, gh-tasks, infra-nginx, deep-research — a unit of reuse, not a prompt.
An enforced schema beats text parsing; the model retries at the tool-call level. CAVEAT: schema-valid ≠ semantically correct — shape, not quality; critical fields still go through verification.
Promoted from Pilot: 'Context Rot' shows up in every model, and degradation by token count makes selecting/compressing context daily bread in prod.
Source: Chroma Context Rot + Context Engineering Reliability Playbook 2026.
A narrow subagent with its own window hands the parent 1-2k tokens of essence — it beats a generalist and guards against degradation, but the coordination cost means you scale it deliberately.
Source: Anthropic multi-agent research system (2026).
Rarely worth it in 2026 — context engineering + skills + prompt caching deliver cheaper and without freezing you onto an old checkpoint.
A monolith fails LOUD in the demo, but the real silent failure is a regression after you touch a single fragment: a change in section A quietly breaks section C — without a golden dataset you never see it.
Fan-out / pipeline / loop-until on my side, cognition on the model's side — repeatable, cheap to debug, resistant to drift.
A human checkpoint at the critical point is the cheapest safety catch against drift — and the first thing regulated industries actually buy.
State saved at every step to Postgres/SQLite lets you resume a multi-hour run instead of starting from scratch — that's what makes long-running agents safe.
Source: LangGraph 2026: checkpointer + HITL primitives as standard; time-travel rollback mostly dev/debug.
Shorter wall-clock than a parallel barrier; ship to prod only where stages are genuinely independent and can be closed off separately.
Tempting for open-ended tasks, but unpredictable when the stakes are high — I keep it on a short leash with a hard budget and a gate.
Several agents converge on a coherent-sounding wrong conclusion — and that agreement READS like verification while it isn't. Without an external yardstick, consensus waves through a hallucination that looks confirmed.
Source: 'sycophantic agreement' / collusion risk between agents (Augment Code 2026).
Keep-if-better / else revert → the model can't fool itself. Proof: OffBall 0.83 Spearman, whitepaper after 3 review rounds.
A single LLM judge picks slot A 10-15 points more often regardless of content; a panel + position swap neutralizes it. Proof: double-diamond 219→2.
Source: futureagi / Adaline 2026: position/verbosity/self-preference bias.
Catches silent failures (status=ok / output=garbage). Proof: a cron-watcher on Haiku every hour + JSONL audit.
'Anything you can check deterministically (schema, test, fact lookup) does NOT go to the judge' — the prod consensus of 2026. You reserve the judge for subjective dimensions.
Source: Vadim.blog / labelyourdata 2026.
Verifying that a tool-call's answer/argument actually follows from the retrieved source (hallucinated tool-arg, retrieval faithfulness). A risk distinct from judge bias and injection; the metrics are still maturing.
Source: RAGAS / hallucinated-tool-arg checks 2026.
Evals run like tests before merge against a fixed reference set, blocking regressions before prod. Less widespread than the verify-before-judge principle itself — hence Pilot.
The judge grades its own work and waves itself through (self-preference bias) — without an external yardstick that's quality theater, not quality.
Confidence ≠ correctness; the model is often most confident exactly when it's lying hardest — calibration drift is well documented.
Every run leaves a trail; without it you can't prove what happened or catch a regression after the fact — and you won't pass an audit in a regulated industry.
MCP is the integration standard (handed to the Agentic AI Foundation under the Linux Foundation, Dec 2025), but to prod ONLY behind a gateway with auth and an allowlist — that's access control (who/what may be called), not a defense against content.
Source: Stacklok 2026: 41% of orgs run MCP in prod; security = top blocker.
The gateway guards ACCESS, not content — a fetched page/document/tool response can carry a prompt injection. Separate data from instructions (content fencing), don't execute commands from untrusted content.
Source: OWASP LLM01 + the 'lethal trifecta' (Anthropic/Willison) 2026.
Working memory in the window + facts in an external store (SQL/graph) is the prod workhorse, but 'memory staleness' (a stale, high-relevance fact) is an open risk — I scale it with an expiry policy.
Source: mem0 State of AI Agent Memory 2026.
A limit saves you from a runaway loop and a surprise invoice; a 90%-cheaper cache read is pure win — as long as you cache the static prefix, not dynamic tool results.
Source: Anthropic 2026: prompt caching, 90% savings on cache reads.
Already a 'production category', but OSWorld shows ~27% first-attempt failures (early-2026 models) — to prod only behind a hard HITL gate and in a narrow sandbox.
Source: OSWorld 2026: Claude ~73%, Operator lower.
A fallback masks the failure — the system says 'ok', quality dropped, nobody knows; in March 2026 ~1/3 of LLM-span errors were rate limits that silent retries sweep under the rug.
Source: Datadog State of AI Engineering 2026.
The provider swaps the model underneath, or someone touches the prompt → a silent regression; without prompt/eval versions you can't roll back or detect that the 'upgrade' broke production.
The radar shows what works in production. Find out where you sit on it — or read the full Q2 2026 report.