AI at scale, with every claim checked

A specialist academic publisher with a large archive of recorded lessons by a scholar on a classical text, alongside reference works in more than one language. The publisher wanted the final 37 chapters of that text covered by a written commentary in English, each chapter following the same editorial structure: an introduction, a verse-by-verse analysis covering vocabulary, grammar, rhetoric and meaning, and a study of the chapter as a whole.

The raw material was transcripts, scanned reference works, files in several formats and specialised terminology. Turning that into consistent, publishable prose at that scale is very difficult to manage by hand. Early experiments with generative AI showed the opposite problem: the drafts read well and were quietly wrong. They left out parts of the teaching, added plausible claims no source supported and cited things that did not exist. Good prose, it turned out, is no evidence of good scholarship.

So the real question was how to produce consistent drafts at scale while making omissions, unsupported claims and bad citations detectable, and keeping the scholar’s teaching as the backbone of the work.

We built a controlled editorial pipeline rather than a writing tool.

First we organised the source material so the system could find the right evidence for each passage. Transcripts and reference works were cleaned, split into passages and tagged by chapter, verse and kind of material (grammar, history, rhetoric, reflection and so on, more than 50 labels in all), so that a search could find the right kind of evidence for a given verse rather than any passage with the same words. Uncertain classifications were routed to a person for checking.

For the publishing run itself the sources were deliberately narrowed to the scholar’s lessons, two agreed reference works and the original text, each with a defined role. The original text is never regenerated by AI. A protected reference copy is checked automatically for any change, and the finished documents are compared against it.

Each chapter follows a fixed template that the system controls; the AI writes content under the headings, never the structure. Before drafting, the pipeline inventories every distinctive point in the source lectures for that chapter. Separate passes then draft the sections, verify them against the sources and the inventory, audit coverage to find what was missed, check every citation against a written policy, read the whole document for continuity and plain readability, and compare the finished commentary with the full lecture one last time. Findings come back as specific corrections.

The publisher’s editorial feedback, such as preserving the detail of an argument, explaining what a grammatical feature does rather than naming it, and letting prose flow instead of compressing it into notes, was written into the rules and checks, so a lesson learned on one chapter applied to the next 36. The output is formatted Word documents to an agreed specification, with the audit reports kept alongside each one.

All 37 chapters have been produced to the agreed structure and handed over, each with its verification, coverage and citation reports. The publisher’s scholars and editors remain in charge of interpretation and of the final published text. The pipeline gives them drafts whose provenance they can check rather than prose they have to take on trust.

Along the way the project produced research infrastructure the publisher can keep using: a tagged, searchable body of source material and a way of asking precise questions of it. The lasting lesson is about how to use AI for serious work. It earns its place when it is built around the standards and judgement of the experts, with its claims traceable and its errors findable.