Turn rough extraction output into text you can trust

The first extraction is rarely the final text. It may contain the right words, but in the wrong order, with broken line endings, page furniture, strange characters, and tables flattened into number soup. That is annoying if you only wanted a quotation. It is dangerous if you are building research notes, a small dataset, or context for a model, because the mess becomes part of the evidence.

This is a different problem from basic extraction and a different problem from OCR. The text extraction tool gets the available text out of the PDF. The OCR tool creates text when the page is only an image. Cleanup starts after that, when you decide whether the output is faithful enough to use and what normalisation it needs.

Sample before you process the whole file

Do not judge a document by page one. Front matter, title pages, and abstracts are often simple; the damage starts in tables, footnotes, appendices, and two-column sections. Pick a small sample before you commit to a workflow:

• one ordinary body page;
• one page with a table or figure;
• one page with footnotes;
• one page from the middle of a chapter or article;
• one page near the end, where appendices and references often change layout.

Extract those pages and read them beside the PDF. You are looking for structural problems, not just typos. Are paragraphs in order? Do columns interleave? Are headers repeated inside sentences? Are numbers still attached to the labels that explain them? If the sample fails in several different ways, write down the patterns before cleaning anything.

Choose native extraction or re-OCR

Native extraction is usually preferable when the embedded text layer is sound. It reads stored characters rather than guessing from pixels, so it avoids recognition errors and is fast. But the embedded layer is not always good. Some PDFs contain invisible text from old OCR, broken encodings, or drawing orders that bear little relationship to human reading order.

Re-OCR is worth testing when you see these signs:

• copied words contain nonsense characters even though the page looks clear;
• search finds a word visually present on the page but the extracted text is garbled around it;
• two-column pages are so interleaved that paragraph reconstruction is slower than recognition;
• the file has an old, hidden OCR layer behind a cleaner scan;
• ligatures, symbols, or accented characters repeatedly map to the wrong code.

Run OCR on a copy of the same sample pages and compare the result. OCR may introduce its own errors, especially in small type and tables, but a fresh text layer with better reading order can be more useful than a native layer that is technically exact and practically unusable.

Fix line and paragraph damage first

Most extracted text keeps visual line breaks. A paragraph becomes twenty short lines because that is how it appeared on the page. If you feed that directly into notes or a model, sentence boundaries become noisy.

Start with hyphenated line breaks. A word split at the right margin may extract as inter- followed by national on the next line. Rejoin these only when the second part starts with a lower-case letter and the combined word makes sense. Do not blindly remove every hyphen at a line ending: compound terms, part numbers, and legal references may be intentionally hyphenated.

Then rebuild paragraphs. Lines that do not end with sentence punctuation usually belong with the next line. Keep list items, headings, poetry, code, and table rows separate. This is where a repeatable script or editor macro helps, but keep a manual review pass because layout exceptions are common.

Remove repeated page furniture

Headers, footers, running titles, confidentiality labels, and page numbers often land inside the extracted text once per page. They are harmless to a human reader and poisonous to downstream processing. A repeated journal title every 800 words can distort keyword counts; a page number in the middle of a sentence can break a quotation.

Look for lines that repeat with small changes. Examples include the document title, chapter name, file code, date, page number, and copyright notice. Remove them with rules that are narrow enough not to delete real content. For page numbers, match isolated numbers or patterns such as Page 12 of 48, not every number that appears at a line boundary.

Footnote markers need separate handling. Superscript references often arrive as plain digits in the middle of a sentence. Decide whether to keep them as markers, move footnotes to the end of the paragraph, or strip them for a reading copy. Do not leave a raw marker where it changes meaning, as in a dosage, equation, or legal clause.

Normalise characters deliberately

PDFs can encode text in surprising ways. Ligatures such as fi, fl, and ffi may appear as single Unicode characters. Smart quotes, en dashes, non-breaking spaces, soft hyphens, and mathematical symbols may survive correctly in one page and fail on the next.

For research notes, it is usually safe to normalise quotes, spaces, and common ligatures. For datasets, decide first what downstream tools expect. Replacing an en dash with a hyphen can be helpful in a plain-text search index and harmful in a diplomatic transcription. Keep a short log of replacements so you can explain how the text was prepared.

Treat tables as evidence, not prose

Tables are the point at which plain extraction often stops being enough. Columns may collapse into runs of numbers, row labels may detach from values, and units may appear only once at the top of a column. A flattened table can look tidy while being wrong.

If the table matters, check it against the page. For small tables, rebuild it as Markdown or CSV by hand. For large tables, split the work: extract the surrounding prose as text, then handle the table with a table-aware workflow rather than pretending a stream of values is reliable. If you later use PDF summarisation, keep the cleaned table or source page available so claims about quantities can be checked.

Build a repeatable cleanup pass

A good cleanup pass is boring and documented:

1. Extract or OCR representative pages.
2. Compare the sample against the visible PDF.
3. Choose native extraction or re-OCR for the full file.
4. Rejoin safe hyphenated line breaks.
5. Rebuild paragraphs while preserving lists and headings.
6. Remove repeated headers, footers, page numbers, and labels.
7. Normalise ligatures, spaces, quotes, and dashes according to the target use.
8. Isolate tables for separate checking.
9. Spot-check another sample after cleanup, not just before it.

Keep the raw extraction, the cleaned text, and the rules you applied. If someone questions a note, dataset row, or model answer later, you need to be able to walk back to the page that supported it.

A short checklist

• Sample difficult pages before processing the whole document.
• Prefer native extraction when the text layer is good; test re-OCR when it is garbled or badly ordered.
• Clean hyphens, line breaks, page furniture, footnote markers, and odd characters in a repeatable order.
• Rebuild important tables instead of trusting flattened numbers.
• Spot-check the cleaned result against the PDF before using it as evidence.