A global consumer brand
auditable verdict classes, each backed by an evidence quote
Situation
A global brand's communications team receives hundreds of press articles a month and needs to know which reinforce vs. undermine its strategic messaging. Doing it by hand is slow, subjective, and impossible to audit — and naive "does the LLM like it" scoring hallucinates verdicts and can't cite why.
Task
Build a proof-of-concept agent pipeline that pulls brand coverage from media monitoring on its own, scores each article against the brand's own strategy documents, and produces stakeholder-ready reports with quote-backed verdicts instead of opaque scores.
Action
I designed a deterministic, provider-agnostic pipeline (LangGraph) that deliberately separates the LLM's judgement from the scoring, and wrapped it in a production-shaped service with QA gates so every verdict is traceable.
- A RAG "brand playbook" extractor: a dedicated agent reads the strategy docs (PDF/DOCX) and distills them into 3–7 structured pillars (thesis, keywords, do/don't rules, weights) — that becomes the scoring rubric.
- Hallucination-proof scoring: the LLM only emits per-pillar relations (support / neutral / contradict), while a pure-Python engine computes the integer index and maps it to the verdict locally — the model never sets the score.
- A self-correcting QA loop: verbatim-quote verification, label/score consistency, pillar coverage — with one automatic, stricter re-run whenever validation fails.
- Production shape: a model layer swappable across 3 providers (OpenAI / Anthropic / Google), FastAPI + Celery + Postgres/Redis, encrypted secrets, token-cost tracking, scheduled daily and monthly reports.
Result
verdicts without an evidence quote
computes the score, not the LLM (no verdict hallucination)
enforced test-coverage gate (71 test files)
swappable LLM providers