An agentic LLM pipeline that turns speech into structured decisions
Audio in, structured output out: fine-tuned transcription feeding a RAG-grounded discussion between specialist agents, with a schema that makes the result safe for downstream code.
- Role
- Technical Specialist · architecture and delivery, one direct report
- Result
- In internal testing
The problem
A domain expert listens to a recording, forms a judgement, and writes it up. The judgement is reliable and the write-up is slow, inconsistent between people, and impossible to audit later. The question was whether a language model could do the first draft without inventing anything.
That last clause is the entire difficulty. A summariser that is right most of the time is worse than no summariser, because the failure is invisible: fluent, confident, and wrong.
The constraints
Three of them shaped every decision.
The audio is full of domain-specific terminology that general transcription models mangle, and a wrong term early poisons everything downstream.
The output had to be checkable. A reviewer needs to see which source supports which claim, or they end up redoing the work and the system has negative value.
And it had to be honest about not knowing. Silence is an acceptable output. Confident invention is not.
What I decided
Fine-tune the transcription rather than post-correct it. Fixing terminology after the fact means guessing at what was meant. Fine-tuning Whisper on the vocabulary moved the error to where it could actually be fixed, at the point of hearing.
Ground the reasoning in retrieval, not in the model’s memory. A RAG layer supplies the reference material, so claims trace to a source rather than to whatever the model absorbed in pretraining.
Use several specialist agents instead of one prompt. A single prompt asked to do everything produces averaged, hedged output. Splitting the work so different agents examine different aspects, then reconciling their views, surfaces genuine disagreement instead of hiding it in one confident paragraph.
Make the output a schema, not prose. The pipeline emits a validated structure. If the model cannot fill a required field it fails loudly and gets retried, rather than emitting something plausible that downstream code will happily consume.
What I would do differently
The orchestration is a chain, and the discussion loop has a real termination condition. Today I would build that as an explicit state graph, so the loop, the retries, and the point where a human approves are all inspectable rather than implicit in the control flow. The retrieval and the schema would stay exactly as they are.
Outcome
The pipeline is in internal testing. The part I would defend hardest is not the model choice, it is the decision that every claim carries a source and that the system is allowed to say it does not know.