Extract structured information

Extraction should preserve the source, not improve it

OpenAI’s prompt guidance recommends clear, specific instructions and context, which are essential when a model must map source text to predefined fields.

NIST’s AI RMF emphasizes measurement and documentation, supporting validation rather than assuming a structured output is accurate because it looks orderly.

Define whether each field is verbatim, normalized or inferred. A verbatim quotation should keep the source wording. A normalized date changes form but not meaning. An inferred sentiment is an interpretation and should be labeled separately.

Design a schema with missing-value behavior

For each field, write its name, definition, allowed values, whether multiple values are permitted and what to do when information is absent. Add source location or evidence text for high-value fields. Test the schema on a small sample before running a large batch.

1. 1

   Choose the downstream use.

2. 2

   Define every field.

3. 3

   Mark fields as verbatim, normalized or inferred.

4. 4

   Set allowed values and formats.

5. 5

   Define null or unknown behavior.

6. 6

   Add an evidence field.

Extract in stages and validate

Start with five diverse documents. Ask for a table using only the provided text. Review false positives, missing values and ambiguous mappings. Revise field definitions before scaling. For large or consequential datasets, sample rows after every batch and compare them with source text.

1. 1

   Provide the schema and source text.

2. 2

   Require unknown when evidence is absent.

3. 3

   Extract a small pilot.

4. 4

   Compare each row with the source.

5. 5

   Record error types.

6. 6

   Revise definitions and examples.

7. 7

   Run the full batch in manageable groups.

8. 8

   Sample and audit the final output.

Measure field-level accuracy

A single overall accuracy number can hide a dangerous field. Names may be accurate while consent status or monetary values are not. Track false positives, false negatives and ambiguity for the fields that matter. Escalate records that need interpretation rather than forcing them into a category.

Extraction quality depends on a field dictionary. Define what each column means, the allowed format, how missing values are represented and whether one source item can produce multiple rows. Include at least one edge case in the example. After extraction, sample both populated and blank fields against the original document; a tidy table can still be wrong if the model silently inferred values that were never present.

Common mistakes to avoid

• Letting the model fill blanks.
• Mixing verbatim extraction with interpretation.
• Scaling before testing diverse examples.
• Reporting one accuracy score for all fields.