Why document AI invents details and how to check it

An AI assistant can be genuinely helpful with a long document. It can point you to themes, draft a summary, list apparent obligations and help you decide where to read first. The problem is that the same assistant can also produce a neat, confident detail that is not in the file.

This post is not another how-to for summarising a confidential PDF. That workflow is covered in summarise a long PDF you cannot upload. It is also not the privacy comparison covered in chat with a PDF without uploading it. The narrower question here is why invented details happen and how to work with an assistant without treating it as a source of truth.

A language model predicts, it does not look facts up by default

A language model generates text by predicting plausible continuations. Given your question and its context, it produces the next likely word, then the next, and so on. That mechanism is why it can explain, rephrase and summarise in a smooth voice. It is also why a wrong answer can cost it no more effort than a right one.

If you ask for the termination date in a contract, the model has learned what termination-date answers usually look like. If the date is present and the right context is available, it may answer correctly. If the date is absent, hidden in a table, or split away from its condition, the model may still produce the shape of a useful answer. The shape can be persuasive even when the fact is not there.

Questions about absent information are the most dangerous. Humans often answer "the document does not say" after checking the relevant places. A model has no natural stopping point that forces that answer. Unless the system is designed to be conservative, it may fill the gap with the nearest plausible continuation.

Retrieval narrows the problem, but does not solve it

Document assistants usually add retrieval. The file is split into passages, indexed, and searched when you ask a question. The model then answers using the retrieved passages rather than relying only on general training patterns. This is a major improvement, and tools such as PDF summarisation and chat with PDF are most useful when they keep the answer close to the source text.

Retrieval still has failure modes. The search step may return a passage that is nearly relevant but not actually the answer. A question about renewal fees might retrieve the fees section rather than the renewal clause. A question about a party's liability cap might retrieve an indemnity clause because both passages share legal vocabulary. The model then reasons from the wrong evidence.

Chunking adds another risk. Long documents are divided into pieces so they fit within the model's context. A table can be split across chunks. A definition may sit in one chunk while the obligation that uses it sits in another. A contract condition can be separated from the exception that changes its meaning. The assistant may answer from the fragment it sees, not from the whole page.

Confident tone is not proof

AI answers often sound calm because they were trained on text where answers are written that way. Confidence is a property of the style, not evidence that a fact was checked. The model can use the same polished voice for a correct quotation, a rough inference and a complete invention.

This is especially awkward with documents because the answer may be partly true. An assistant might identify the correct clause but attach the wrong deadline. It might name the right customer but copy the amount from a neighbouring row. It might correctly say a policy excludes something while missing the exception that brings it back into scope.

Do not use tone as a confidence meter. Use page evidence.

Where document hallucinations hurt most

Some document details are more fragile than others:

• Figures, totals, rates and percentages, because one digit changes the meaning.
• Dates and deadlines, because nearby dates often refer to different events.
• Names, entities and addresses, because similar labels repeat across files.
• Negations, such as "not", "unless", "except" and "without prior consent".
• Clause conditions in contracts, where the answer depends on several sentences read together.
• Tables, where row and column alignment carries meaning that plain text may lose.

These are exactly the details people most want an assistant to extract. They are also the details you should verify manually before acting on them.

Work with claims, not vibes

A safer practice is to treat every extracted fact as a claim. Ask the assistant for the supporting quotation and page reference. Then read that quotation in the PDF, not just in the chat transcript. If the quotation does not contain the answer, the answer is an inference. If the quotation is missing, treat the answer as unverified.

Prefer questions whose answers you can check: "Which page mentions the renewal notice period?" is safer than "What should I do next?" A question that asks for a list of candidate passages gives you a reading map. A question that asks for a final judgement may hide the chain of reasoning.

When the file already contains selectable text, the text extraction tool can also help you inspect the raw material the assistant may be using. If the extracted text has broken columns, missing symbols or scrambled table order, the assistant inherits that weakness.

A short verification checklist

• Ask for the exact supporting quotation, not just a summary.
• Open the page and compare the answer with the document yourself.
• Check numbers, names, dates and identifiers character by character.
• Read the sentences before and after any quoted clause.
• Treat "the document does not say" as a valid and often valuable answer.