Prompt Engineering for Document Analysis

Why Analysis Quality Depends on More Than the Model

The same model, run on the same document, can produce two fundamentally different results depending on how the task is phrased. This isn't randomness in generation. A loose prompt like "analyze this document" doesn't specify the goal, the criteria, the format, or what to do with ambiguous passages, and the model ends up filling in those gaps on its own.

Formal measurements back this up. Research into how sensitive models are to prompt phrasing shows that changing the format of an instruction, with the exact same model weights, can shift accuracy by dozens of percentage points, and for some models the spread between semantically equivalent phrasings of the same task runs into several dozen points of accuracy. For document analysis this isn't abstract theory: context, the goal of the analysis, constraints, and output format matter as much as the model itself.

The general breakdown of working with sources already covers three working prompts as examples. What follows isn't more of the same list, it's the mechanics behind a good prompt in the first place: a good prompt turns a loose request into a formalized analysis procedure, instead of a phrase that happened to work once by luck.

What Kinds of Tasks Prompts Can Handle

The range of tasks a document-analysis prompt can handle is wider than "summarize this": extracting facts and specific details, structuring unstructured text, condensing into a short summary, classifying documents and fragments, finding contradictions, comparing multiple documents, flagging risks and issues, analyzing argumentation, pulling out key claims, converting content into structured data.

Each class of task needs its own framing, and that's not a formality. An extraction task ("find every date and dollar amount in this text") sets a narrow, literal mode: quote, don't interpret. An argumentation-analysis task ("assess how well the author's data actually supports their conclusion") requires the opposite, the model has to reason, not just pattern-match. A prompt written for one class of task usually performs poorly on another, even when both get called "document analysis."

What a Good Document-Analysis Prompt Is Made Of

A working prompt is usually built from eight components: the model's role, the task, context, analysis criteria, constraints, requirements for reliability, output format, and rules for handling missing information. Skipping any one of them doesn't always break the result outright, but it always leaves a decision to the model in a spot where the decision should be yours.

One distinction is easy to miss here: a prompt has to describe not just what to find, but how to interpret what's found. "Find the liability clauses in this contract" is an extraction task. "Find the liability clauses in this contract and assess whether they're imbalanced in one party's favor" adds interpretation on top of extraction. Without that second instruction, the model either quietly adds its own interpretation or stops at a bare list, and either outcome might not be what was actually needed.

Defining the Goal of the Analysis

The most common problem at this stage is a request that's too general, something like "analyze this document." An abstract verb like "analyze" or "break down" doesn't specify a concrete operation, so the model picks its own starting and ending point, usually defaulting to a neutral recap instead of whatever result was actually needed.

A working phrasing turns an abstract task into a concrete analytical operation with a legible output: not "analyze the risks," but "flag every clause in this contract that creates financial risk for our side, and rate each one on a scale of 1 to 5." It's worth distinguishing five different request types that easily get blurred into one phrasing: recap, analysis, evaluation, classification, and extraction. Each implies a different degree of the model's intervention in the source text, from none at all for literal extraction to a lot for evaluation involving its own judgment. Mixing them in one prompt usually produces an answer to two questions at once, and neither one properly.

Managing Context

Just handing the model a document and asking it to analyze it isn't enough. Without explicit instructions, it's unclear which part of the document to weigh most heavily, which details take priority, what outside knowledge is fair game, what to do with ambiguous passages, and where the boundary of interpretation actually sits.

One of the most practically significant effects here: models are noticeably worse at retaining and analyzing information placed in the middle of a long context than at the beginning or the end. Liu and colleagues' 2023 study, "Lost in the Middle," showed this directly: accuracy at retrieving a fact drops sharply when that fact sits somewhere in the middle of a long document, even when the model is formally capable of processing the entire text in one pass. The practical takeaway: important instructions and criteria belong at the start or the end of a prompt, not somewhere you're trusting the model to read with equal attention throughout.

It's also worth explicitly separating the document's content from the instructions for the model, instead of blending them into one continuous block of text. Clear delimiters, XML tags like <document> and <instructions>, for instance, cut the risk of the model mistaking part of the document for part of the instruction, especially if the document itself contains phrasing that reads like a command.

Working With Long Documents

A model's context window is physically limited, and even where the whole document technically fits, processing it in one request isn't always the best strategy. The longer the document, the less evenly the model spreads its attention across it, and the lost-in-the-middle effect from the previous section only gets worse as volume grows.

Working approaches: splitting the document into fragments ahead of time, analyzing each fragment separately, keeping intermediate results, aggregating those results afterward, hierarchical analysis where the document itself has a nested structure, and re-checking the most important conclusions with a separate follow-up query. In practice this usually comes down to one of two patterns: map-reduce, where fragments get analyzed in parallel and independently and the results get reconciled in a separate step, or hierarchical summarization, where small parts get condensed first, then summaries of those summaries, all the way down to one final result.

Prompts for Extracting Structured Information

Moving from free text to a predefined output structure is one of the most practically valuable techniques, because structured output can go straight into the next stage of a pipeline without manual parsing.

A working prompt specifies a fixed set of fields, a data type for each field, explicitly marks which values are required and which aren't, and separately covers what to do when data is missing: output an explicit marker like null or "not specified" instead of guessing at a value. Modern models support this at the mechanism level, not just through a text instruction. Strict output schemas, JSON mode, function calling, structured decoding against a defined schema, cut the rate of syntactically invalid responses down to close to zero compared with free text, where a model occasionally garbles the format, skips a field, or drops a stray comment in front of the data.

How to Cut Down on Hallucinations

Unreliable conclusions in document analysis usually don't come from the model "lying." They come from a prompt that doesn't forbid it from filling gaps with general knowledge instead of the document's actual content.

Working techniques: an explicit requirement to rely only on the document's content, a direct ban on inventing missing details, mandatory flagging of uncertainty wherever the text doesn't have an answer, treating facts and interpretations as two separate output categories, tying every conclusion to a specific fragment of the document, and running an extra check on the most critical claims as a separate step. Chain-of-Verification, where the model drafts an initial answer, then writes its own verification questions about that draft and answers them independently, cuts the number of hallucinated facts by 50 to 70% compared to a plain direct answer on the same tasks, according to the published results.

A zero-tolerance instruction on invented content works better than a vague "be accurate." "If the text doesn't have a direct answer to this question, output the status 'not stated in the document.' Using knowledge from outside the document is not allowed" is a concrete rule instead of a general wish.

Citations and Making Conclusions Verifiable

A separate task: making the model show its work for every conclusion instead of just handing over an answer that sounds convincing. The working technique: require an exact quote, a page or section number, or a document-part identifier next to every substantive claim.

The practical difference is noticeable. The prompt "list the risks in this contract" and the prompt "list the risks in this contract, and for each one give the exact quote and clause number" produce formally similar-looking lists. Only the second one can actually be checked quickly, without rereading the whole document. This is especially valuable for automation: if the model has to return pairs of "claim plus exact quote," a separate script can verify whether that quote actually exists in the source document and filter out any conclusion that fails the check before a person ever sees the result.

Analyzing a Document in Multiple Stages

A sequential pipeline of several prompts usually gives a more reliable result than one universal "fully analyze this document" request. The logic might run: identify the document's structure, extract relevant information, classify what was found, run the main analysis, check for contradictions and gaps, produce the final result.

The reasoning behind splitting it up is simple: each individual step asks the model for something far narrower than "do everything at once." A prompt that simultaneously asks the model to extract facts, classify them, assess risk, and formulate recommendations competes with itself for the model's attention. Usually one of those subtasks suffers for the sake of the others, often invisibly to whoever reads the finished result.

Comparing Multiple Documents

Prompts for comparing several sources run into problems that don't exist when analyzing one document: different document structures, different terminology for the same concepts, direct contradictions between sources, information missing from one of them, and the need to match up the same entities under different names.

The working technique for that last problem: the first step of any comparison prompt is unifying the vocabulary, ahead of the actual comparison. The model first builds a single glossary of terms and their correspondences across every document being compared, and only after that moves to substantive comparison. Skipping this step almost guarantees false "contradictions" that are really just different terminology for the same thing. Comparison criteria are also worth setting explicitly ahead of time, instead of leaving the model to decide on its own what's actually worth comparing.

Analyzing Contradictions and Inconsistencies

This is its own class of task, where the model isn't just extracting information, it's cross-checking different claims against each other: internal contradictions within one document, contradictions between multiple documents, mismatched values for the same figure, logical inconsistencies in the author's reasoning, and potential errors in the underlying data.

Detecting hidden contradictions works more reliably when the prompt explicitly breaks the reasoning into steps instead of one request to "find the contradictions": first list every claim on a specific question, then check them pairwise for compatibility, and only at the third step formulate a conclusion. Models simply asked to find inconsistencies routinely miss implicit logical collisions that aren't spelled out as literally conflicting sentences. It's also worth keeping the detected contradiction separate from the interpretation of its cause: the fact that two places in a contract list different deadlines is a fact, whether that's a typo or an intentional condition is already interpretation, and conflating those two levels in one conclusion isn't a good idea.

How to Specify the Output Format

The response format has to be part of the prompt, not something the model decides by default. Structuring can go several levels deep: a short summary, thematic blocks, a table, a list of individual claims, a set of structured fields, a machine-readable format like JSON.

The format is worth choosing based on what happens to the result next. A result someone will visually scan for specific numbers is usually easier to work with as a table or a list, finding a specific value among rows and columns is faster than in a paragraph where the same number is buried mid-sentence. A result headed further into automated processing has to be machine-readable, no matter how much more approachable a plain-text recap might look to a person.

Bad and Good Prompts: Quality Criteria

Instead of concrete examples, it's more useful to hold onto criteria. A good prompt unambiguously defines the task, sets relevant context, bounds the scope of analysis, defines evaluation criteria, controls for unreliable conclusions, specifies the expected format, and lets the result be checked regardless of who's reading it.

A bad prompt is recognizable by its symptoms, not by any specific wording: a vague goal, no criteria, a final result that depends heavily on however the model happens to decide to interpret an ambiguous request. A simple test: if two separate runs of the same prompt on the same document produce structurally different answers, not just different words carrying the same substance, the prompt probably isn't constraining the task enough.

How to Test Prompts

A prompt is worth treating as an object of iterative optimization, not a phrasing you came up with once and never touch again. The working process: assemble a set of test documents, define upfront what a correct result should look like, run the prompt across different document types, log the typical failure modes, change one group of instructions at a time, and compare results across versions of the prompt against each other.

It's also worth testing a prompt against awkward documents, not just representative ones: scans with bad OCR, documents with nonstandard structure, edge cases where the right answer is actually "the information isn't there." One working way to automate this step is LLM-as-a-judge: a separate model scores the quality of an answer against a predefined rubric, which makes it possible to run dozens of prompt versions against dozens of documents without manually checking every single result.

A personal example: for a long stretch I tested document-analysis prompts on the same two or three examples, ones I'd picked precisely because they processed well. A prompt that handled that narrow set beautifully regularly broke on real user documents with a different structure. A test set curated to fit an already-working prompt says nothing about how that prompt behaves outside the set.

Common Mistakes When Writing Prompts

The same problems come up again and again: a task framed too generally, instructions that run too long, mixing several independent tasks into one request, no quality criteria, no rules for undefined data, no requirement to source conclusions, trying to solve a complex task in one request instead of a pipeline, and no check on the result after getting an answer.

One mistake on this list breaks a prompt especially reliably: too many rules and constraints, some of which end up contradicting each other. In that situation the model follows whichever rule its own internal logic lands on, not yours, and the result turns unpredictable exactly where the prompt was trying hardest to plan for everything in advance.

Prompts as Part of Automated Document Analysis

Inside a software pipeline, a prompt stops being a one-off user request and becomes part of the architecture: document upload, text extraction, segmentation, identifying relevant parts, LLM analysis, structuring the results, storage, and later search and use of the data.

At this level, a prompt is better thought of as configurable code. It gets versioned, tested for regressions when the model changes, and modified under the same rules as the rest of the system's code, not patched by eye whenever something goes wrong.

How to Choose an Analysis Strategy

There's no universal prompt for every case. A practical way to pick an approach runs off the specific task's parameters: a short document with a simple task usually needs one direct prompt, a long document needs splitting and aggregation, several documents need vocabulary unification before comparison, extracting structured data needs a strict output schema, an analytical task with judgment calls needs explicit evaluation criteria, a task with high standards for verifiability needs mandatory citation and checking, a recurring automated analysis needs a versioned prompt inside a pipeline.

Conclusion

A few principles everything above comes down to: define the analytical task first, then the context it needs, set criteria and constraints explicitly, keep extraction separate from interpretation, require verifiable conclusions, use multi-step analysis for complex documents, test prompts on real data instead of convenient data, and treat a prompt as part of the analytical process, not a phrasing you stumbled into that happened to work once.