diff --git a/README.md b/README.md index 49fe82a..550d928 100644 --- a/README.md +++ b/README.md @@ -1,81 +1,84 @@

- document-extractor + document-extractor by Modelshifts

-

Turn PDF invoices and contracts into structured fields, each linked to its source.

+

A PDF goes in. Structured fields come out, each one pointing back to the words on the page.

+ CI Status: v1 License: MIT Python 3.11+ Backends: API or Hugging Face

---- +

+ Web demo on an invoice: nine fields are extracted (invoice number NWP-2026-0412, total 5,496.68 and seven others), each with a numbered source box on the page, and the question “When is payment due?” is answered with a cited passage. +

+

Web demo on an invoice (MP4). The recording replays results computed by Qwen2.5-1.5B-Instruct.

+ +

+ Evaluation highlights: 60 of 61 fields correct (98%); all 60 correct values sourced; 9 to 22 seconds per document. +

+ +## The problem + +Imagine a finance operations desk that closes the week by retyping 400 invoices. Each one is a PDF: a number, a due date, a total, two company names. Someone opens the file, finds those nine fields, and keys them into a spreadsheet. The work is careful and slow, and a transposed digit is easy to miss until a payment comes back. + +The useful result is not a model that can “read PDFs.” It is a list of fields a person can audit, because every value still knows the place on the page it came from. -## Features +## How it works -All five features below work in v1. See [Limitations](#limitations) for what they do not cover yet. +The pipeline is five steps, and a sixth if you ask a question. -- **Field extraction to JSON**: define the fields you want (invoice number, totals, parties, dates) and get typed JSON back. -- **Source citation per value**: every value carries the page and bounding box it came from, so it can be checked in one click. -- **Q&A over the document (RAG)**: ask a question and get an answer grounded in cited passages. -- **Batch mode**: point it at a folder and get one CSV or JSON file per run. -- **Web demo**: pick a sample or upload a PDF, see fields highlighted on the page. +1. A PDF or a scan comes in. +2. The page is read into words with coordinates. Scanned pages go through Florence-2. +3. A YAML schema lists the fields you want — invoice number, totals, parties, dates. No extraction code to write. +4. Each returned value is matched back to the printed words on the page. +5. You get JSON, and every value carries its page, bounding box and source text. + +Ask a question (“When is payment due?”) and a retriever — bge-m3, or BM25 — picks passages; the model answers with citations. Point it at a folder and it writes one CSV or JSON for the run.

- Web demo: a synthetic invoice is selected, the extracted fields appear with their source boxes on the page, and a question is answered with a cited passage + Pipeline: a PDF or scan is read into words with page boxes; a YAML schema drives field extraction; each value is grounded back to the printed words, producing JSON with page, bounding box and source text. A parallel path answers questions from retrieved passages with citations. The same pipeline runs on an OpenAI-compatible API or on open models from Hugging Face.

-

Web demo on a synthetic invoice (MP4). Results in the recording were computed earlier by Qwen2.5-1.5B-Instruct on CPU and loaded from the demo's result cache.

- -## Architecture - -```mermaid -%%{init: {"theme":"base","flowchart":{"curve":"basis","wrappingWidth":260},"themeVariables":{"fontFamily":"Georgia, Times New Roman, serif","fontSize":"14px","primaryColor":"#f3efe6","primaryTextColor":"#1c1a17","primaryBorderColor":"#1c1a17","lineColor":"#1c1a17","clusterBkg":"#faf7f0","clusterBorder":"#8a3a24","titleColor":"#8a3a24","edgeLabelBackground":"#f3efe6"}}}%% -flowchart TB - P["PDF / scan"]:::ink --> L["Layout and OCR
pages, blocks, boxes"]:::ink - L --> X["Field extractor
schema to JSON"]:::ink - L --> C["Chunk and embed"]:::ink - C --> Q["Q&A (RAG)
retrieve, answer, cite"]:::ink - X --> G["Grounding
match each value to page words"]:::ink - G --> O["JSON + source links
page, box, snippet"]:::ink - Q --> O - subgraph BK["Backend · pick one"] - direction LR - BKA["(a) Your API key
OpenAI-compatible
LLM + embeddings"]:::rust - BKH["(b) Hugging Face
Qwen2.5-7B-Instruct
bge-m3 · Florence-2"]:::rust - end - BK -.-> X - BK -.-> Q - BK -.-> C - - classDef cl fill:#faf7f0,stroke:#8a3a24,stroke-dasharray:4 3,color:#8a3a24; - classDef ink fill:#f3efe6,stroke:#1c1a17,color:#1c1a17,stroke-width:1px; - classDef rust fill:#f3efe6,stroke:#8a3a24,color:#8a3a24,stroke-width:1.5px; -``` +

The same five steps, whether the backend is your API or open models on your own machine.

-### Two interchangeable backends +One of the samples, before and after: -Pick one with `MS_BACKEND` in `.env`. The rest of the pipeline does not change. +

+ Left: tax invoice invoice-northwind-classic.pdf, from Northwind Parts Pte. Ltd. to Example Logistics Pte. Ltd., invoice NWP-2026-0412, total due SGD 5,496.68. Right: the nine extracted fields, all correct, each sourced to matching text on page 1 — invoice_number NWP-2026-0412, invoice_date 2026-03-14 (from “14 March 2026”), due_date 2026-04-13, seller and buyer names, currency SGD, subtotal 5,042.83, tax 453.85, total 5,496.68. +

+

invoice-northwind-classic.pdf beside the extract. All nine fields correct, all nine sourced. scripts/evaluate.py.

-| | (a) Bring your own API key | (b) Open-source on Hugging Face | -|---|---|---| -| **What** | Any OpenAI-compatible endpoint | Open models, run locally with `transformers` | -| **Set** | `MS_BACKEND=api`, `MS_API_BASE`, `MS_API_KEY`, `MS_MODEL` | `MS_BACKEND=hf`, optionally `MS_HF_MODEL` | -| **Scanned pages** | `MS_VISION_MODEL` (a vision model on the same endpoint) | Florence-2 | -| **Q&A retrieval** | `MS_EMBED_MODEL` if set, otherwise BM25 | bge-m3 | -| **Good for** | Fastest start, strongest hosted models | Data stays on your machines, no per-call cost | +Think of each field as a claim, and the bounding box as the exhibit attached to it. -**Default open-source models** +## Results -| Model | Used for | License | -|---|---|---| -| [`Qwen/Qwen2.5-7B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) | field extraction and answers | Apache-2.0 | -| [`BAAI/bge-m3`](https://huggingface.co/BAAI/bge-m3) | embeddings for retrieval | MIT | -| [`florence-community/Florence-2-large`](https://huggingface.co/florence-community/Florence-2-large) | OCR for scanned pages (`transformers`-native copy of [`microsoft/Florence-2-large`](https://huggingface.co/microsoft/Florence-2-large)) | MIT | +On seven PDFs — four invoice layouts, two contracts, and one image-only scan — the Hugging Face backend with Qwen2.5-1.5B-Instruct got **60 of 61 fields correct (98%)**. Every correct value also carried a source box whose text matches the value (**60/60**). + +The one miss is an OCR slip on the scanned invoice: `buyer_name` came back as `Northwind Parts Pte., Ltd.` (an extra comma) instead of `Northwind Parts Pte. Ltd.` + +| Sample | Correct | Sourced | Time (s) | Notes | +|---|---|---|---|---| +| contract-master-services.pdf | 8/8 | 8/8 | 18.7 | — | +| contract-supply.pdf | 8/8 | 8/8 | 9.0 | — | +| invoice-acme-scanned.pdf | 8/9 | 8/8 | 22.1 | `buyer_name`: extra OCR comma | +| invoice-demo-freight-compact.pdf | 9/9 | 9/9 | 10.5 | — | +| invoice-example-logistics-banner.pdf | 9/9 | 9/9 | 9.7 | — | +| invoice-northwind-classic.pdf | 9/9 | 9/9 | 10.9 | — | +| invoice-placeholder-two-page.pdf | 9/9 | 9/9 | 13.6 | — | +| **All** | **60/61 (98%)** | **60/60** | **9.0–22.1** | | + +**Methodology.** Seven PDFs (four invoice layouts, two contracts, one image-only scan), 61 fields in total. A field counts as correct when the typed value matches the ground truth in [`samples/*.truth.json`](samples/) (strings compared case-insensitively, extra spaces and trailing punctuation ignored; numbers within 0.005). “Sourced” counts correct values that also carry a source box whose text matches the value. [`python scripts/evaluate.py`](scripts/evaluate.py). Hugging Face backend, Qwen2.5-1.5B-Instruct, Florence-2 for the scanned page, 12 November 2026. Wall-clock for all seven samples was 96 seconds, including model load. Per-document times were 9.0–22.1 seconds. The raw table is in [`docs/eval.md`](docs/eval.md). + +The API backend against a local llama.cpp server (Qwen2.5-1.5B-Instruct, Q4_K_M) got 52 of 61 fields correct: all 52 fields on the six text-layer PDFs, in 3.9–12.1 seconds each. The scanned invoice is scored 0/9 there, because that backend has no OCR unless a vision model is configured. + +## Why it matters -On a machine without a GPU, set `MS_HF_MODEL=Qwen/Qwen2.5-1.5B-Instruct`. The 7B model needs a GPU or a lot of memory. -`MS_RETRIEVER=bm25` skips the bge-m3 download and uses keyword retrieval instead. +A total without a source is a rumour. A total with a bounding box is a claim you can audit — from `5,496.68` back to the printed “SGD 5,496.68” on the page, and reject it if the box is wrong. That is the difference between an extract you retype to be safe, and an extract you can actually use. + +It fits anywhere a team already files PDFs and maintains a spreadsheet: accounts payable, contract intake, a records desk. The schema is a YAML file, so the fields can change without changing the code. The backend can be a hosted API or open models on your own machine; the rest of the pipeline stays the same. ## Quickstart @@ -105,60 +108,49 @@ document-extractor batch samples --glob "invoice-*.pdf" --schema invoice --out r document-extractor serve ``` -### Using the API backend +A schema is a YAML file that lists fields and their types (`string`, `date`, `number`, `integer`). See [`schemas/invoice.yaml`](schemas/invoice.yaml) and [`schemas/contract.yaml`](schemas/contract.yaml). Dates come back as `YYYY-MM-DD` and amounts as numbers. -Any OpenAI-compatible endpoint works. In `.env`: +`samples/` holds seven PDFs (four invoice layouts, two contracts, one image-only scan) with a ground-truth file for each. Tests use a mocked model and need no downloads: `pip install -e ".[dev]" && pytest`. -```bash -MS_BACKEND=api -MS_API_BASE=https://your-provider.example/v1 -MS_API_KEY=your-api-key -MS_MODEL=your-chat-model -``` +## Backends -To try it without a paid key, run an open model behind a local OpenAI-compatible server, for example [llama-cpp-python](https://github.com/abetlen/llama-cpp-python): +
+Two interchangeable backends — API key or open models. The rest of the pipeline does not change. -```bash -pip install "llama-cpp-python[server]" huggingface_hub \ - --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu # prebuilt CPU wheels, no compiler needed -hf download Qwen/Qwen2.5-1.5B-Instruct-GGUF qwen2.5-1.5b-instruct-q4_k_m.gguf --local-dir models -python -m llama_cpp.server --model models/qwen2.5-1.5b-instruct-q4_k_m.gguf --n_ctx 8192 --port 8001 -# then: MS_API_BASE=http://127.0.0.1:8001/v1 MS_API_KEY=local MS_MODEL=qwen2.5-1.5b-instruct-q4_k_m -``` +Pick one with `MS_BACKEND` in `.env`. -### Schemas +| | (a) Bring your own API key | (b) Open-source on Hugging Face | +|---|---|---| +| **What** | Any OpenAI-compatible endpoint | Open models, run locally with `transformers` | +| **Set** | `MS_BACKEND=api`, `MS_API_BASE`, `MS_API_KEY`, `MS_MODEL` | `MS_BACKEND=hf`, optionally `MS_HF_MODEL` | +| **Scanned pages** | `MS_VISION_MODEL` (a vision model on the same endpoint) | Florence-2 | +| **Q&A retrieval** | `MS_EMBED_MODEL` if set, otherwise BM25 | bge-m3 | +| **Good for** | Fastest start, hosted models | Data stays on your machines, no per-call cost | + +**Default open-source models** -A schema is a YAML file that lists fields and their types (`string`, `date`, `number`, `integer`). See [`schemas/invoice.yaml`](schemas/invoice.yaml) and [`schemas/contract.yaml`](schemas/contract.yaml). Dates come back as `YYYY-MM-DD` and amounts as numbers. The model is asked to copy each value as printed, and the value is then matched back to the words on the page to find its box. +| Model | Used for | License | +|---|---|---| +| [`Qwen/Qwen2.5-7B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) | field extraction and answers | Apache-2.0 | +| [`BAAI/bge-m3`](https://huggingface.co/BAAI/bge-m3) | embeddings for retrieval | MIT | +| [`florence-community/Florence-2-large`](https://huggingface.co/florence-community/Florence-2-large) | OCR for scanned pages (`transformers`-native copy of [`microsoft/Florence-2-large`](https://huggingface.co/microsoft/Florence-2-large)) | MIT | -### Sample documents and tests +On a machine without a GPU, set `MS_HF_MODEL=Qwen/Qwen2.5-1.5B-Instruct`. The 7B model needs a GPU or a lot of memory. `MS_RETRIEVER=bm25` skips the bge-m3 download and uses keyword retrieval instead. -`samples/` holds seven synthetic PDFs (four invoice layouts, two contracts, one image-only scan) with a ground-truth file for each. All names and figures in them are fictional. +To try the API backend without a paid key, run an open model behind a local OpenAI-compatible server, for example [llama-cpp-python](https://github.com/abetlen/llama-cpp-python): ```bash -pip install -e ".[dev]" -python scripts/make_samples.py # regenerate samples/ -pytest # uses a mocked model, no downloads -python scripts/evaluate.py # compare the configured backend with the ground truth +pip install "llama-cpp-python[server]" huggingface_hub \ + --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu # prebuilt CPU wheels, no compiler needed +hf download Qwen/Qwen2.5-1.5B-Instruct-GGUF qwen2.5-1.5b-instruct-q4_k_m.gguf --local-dir models +python -m llama_cpp.server --model models/qwen2.5-1.5b-instruct-q4_k_m.gguf --n_ctx 8192 --port 8001 +# then: MS_API_BASE=http://127.0.0.1:8001/v1 MS_API_KEY=local MS_MODEL=qwen2.5-1.5b-instruct-q4_k_m ``` -## Limitations - -- The samples are synthetic and have simple layouts. The project has not been measured on real-world documents. -- On a CPU, extraction with the open models takes tens of seconds per document, and loading Qwen, bge-m3 and Florence-2 together needs several GB of memory. The 7B default needs a GPU. -- Florence-2 reads a scanned page in one pass, so small or dense text can be misread. On the API backend, OCR through `MS_VISION_MODEL` is implemented but has only been tested with a mocked model. -- Source boxes come from matching the value against the page text. If a value is printed more than once, the copy closest to a matching label is used, which is not always the one the model read. -- Numeric dates such as `03.02.2026` are read day-first. -- The web demo saves results under `.cache/results` and reuses them for the same file, schema and model. +
## Roadmap -- [x] Repository scaffold and README -- [x] PDF parsing with page and box coordinates -- [x] Schema-driven field extraction -- [x] Source citation per value -- [x] Q&A over the document (RAG) -- [x] Batch mode -- [x] Web demo - [ ] Reranking retrieved passages with `BAAI/bge-reranker-v2-m3` - [ ] Scanned-page OCR tested with a real vision model on the API backend - [ ] Hosted demo @@ -167,4 +159,4 @@ python scripts/evaluate.py # compare the configured backend with the groun [MIT](LICENSE) © Modelshifts Technology Pte. Ltd. -

Part of the Modelshifts showcase.

+

Part of the Modelshifts showcase. November 2026, v1.

diff --git a/assets/before-after.png b/assets/before-after.png new file mode 100644 index 0000000..3018328 Binary files /dev/null and b/assets/before-after.png differ diff --git a/assets/og.png b/assets/og.png new file mode 100644 index 0000000..c140277 Binary files /dev/null and b/assets/og.png differ diff --git a/assets/pipeline.svg b/assets/pipeline.svg new file mode 100644 index 0000000..66e8aaa --- /dev/null +++ b/assets/pipeline.svg @@ -0,0 +1,103 @@ + + How document-extractor works + A PDF or scan is read into words with page boxes (Florence-2 OCR for scans). A YAML schema drives field extraction into typed values. Each value is grounded back to the words on the page, giving JSON with page, bounding box and source text. A second path answers questions from retrieved passages with citations. + + + PIPELINE · ONE COMMAND, FIVE STEPS + + + + + + + + + + + + + + + + + + + 01 INPUT + PDF or scan + invoice, contract + + + + + + + + + 02 READ + Words + boxes + OCR for scanned pages + + + + + + total: type: numberdue_date: date + + 03 EXTRACT + Schema → values + a YAML file, no code + + + + + + + + p.1 + + 04 GROUND + Value → page + matched to printed words + + + + + + "total": 5496.68, + "source": {"page": 1, + "bbox": [...]} + + 05 OUTPUT + JSON + citation + one click to check + + + + + + + + ASK + "When is payment due?" → retrieve passages (bge-m3 or BM25) → answer with [passage] citations + + Backend: your OpenAI-compatible API, or open models from Hugging Face (Qwen2.5, bge-m3, Florence-2). Same pipeline either way. + diff --git a/assets/results.svg b/assets/results.svg new file mode 100644 index 0000000..146d43f --- /dev/null +++ b/assets/results.svg @@ -0,0 +1,19 @@ + + Evaluation results + 60 of 61 fields correct (98%). All 60 correct values sourced. 9.0 to 22.1 seconds per document. + + + 60 / 61 + FIELDS CORRECT · 98% + + 60 / 60 + CORRECT VALUES SOURCED + + 9–22 s + PER DOCUMENT + diff --git a/assets/social.png b/assets/social.png new file mode 100644 index 0000000..b0516f1 Binary files /dev/null and b/assets/social.png differ diff --git a/docs/eval-api.md b/docs/eval-api.md new file mode 100644 index 0000000..7130952 --- /dev/null +++ b/docs/eval-api.md @@ -0,0 +1,13 @@ + +Backend: api, model: qwen2.5-1.5b-instruct-q4_k_m + +| Sample | Correct | Sourced | Time (s) | Misses | +|---|---|---|---|---| +| contract-master-services.pdf | 8/8 | 8/8 | 7.0 | - | +| contract-supply.pdf | 8/8 | 8/8 | 3.9 | - | +| invoice-acme-scanned.pdf | 0/9 | 0/0 | 0.0 | error: invoice-acme-scanned.pdf, page 1: this page has no text layer (it looks like a scan) and no OCR engine is configured. Use MS_BACKEND=hf (Florence-2 OCR) or set MS_VISION_MODEL for MS_BACKEND=api. | +| invoice-demo-freight-compact.pdf | 9/9 | 9/9 | 6.4 | - | +| invoice-example-logistics-banner.pdf | 9/9 | 9/9 | 4.7 | - | +| invoice-northwind-classic.pdf | 9/9 | 9/9 | 5.9 | - | +| invoice-placeholder-two-page.pdf | 9/9 | 9/9 | 12.1 | - | +| **All** | **52/61 (85%)** | 52/52 | | | diff --git a/docs/eval-hf.json b/docs/eval-hf.json new file mode 100644 index 0000000..50c5452 --- /dev/null +++ b/docs/eval-hf.json @@ -0,0 +1,912 @@ +[ + { + "file": "contract-master-services.pdf", + "schema": "contract", + "backend": "hf", + "model": "Qwen/Qwen2.5-1.5B-Instruct", + "fields": { + "contract_title": { + "value": "Master Services Agreement", + "raw": "Master Services Agreement", + "source": { + "page": 1, + "bbox": [ + 81.29, + 101.78, + 197.93, + 112.28 + ], + "text": "Master Services Agreement" + } + }, + "party_a": { + "value": "Northwind Parts Pte. Ltd.", + "raw": "Northwind Parts Pte. Ltd.", + "source": { + "page": 1, + "bbox": [ + 423.92, + 101.78, + 531.54, + 112.28 + ], + "text": "Northwind Parts Pte. Ltd." + } + }, + "party_b": { + "value": "Example Logistics Pte. Ltd.", + "raw": "Example Logistics Pte. Ltd.", + "source": { + "page": 1, + "bbox": [ + 143.7, + 115.78, + 260.66, + 126.28 + ], + "text": "Example Logistics Pte. Ltd." + } + }, + "effective_date": { + "value": "2026-07-01", + "raw": "2026-07-01", + "source": { + "page": 1, + "bbox": [ + 248.67, + 101.78, + 297.68, + 112.28 + ], + "text": "1 July 2026" + } + }, + "term_months": { + "value": 24, + "raw": "24", + "source": { + "page": 1, + "bbox": [ + 371.14, + 201.78, + 381.64, + 212.28 + ], + "text": "24" + } + }, + "notice_period_days": { + "value": 60, + "raw": "60", + "source": { + "page": 1, + "bbox": [ + 433.29, + 387.78, + 450.78, + 398.28 + ], + "text": "(60)" + } + }, + "payment_terms_days": { + "value": 30, + "raw": "30", + "source": { + "page": 1, + "bbox": [ + 292.73, + 251.78, + 303.23, + 262.28 + ], + "text": "30" + } + }, + "governing_law": { + "value": "Singapore", + "raw": "Singapore", + "source": { + "page": 1, + "bbox": [ + 243.72, + 437.78, + 288.92, + 448.28 + ], + "text": "Singapore." + } + } + } + }, + { + "file": "contract-supply.pdf", + "schema": "contract", + "backend": "hf", + "model": "Qwen/Qwen2.5-1.5B-Instruct", + "fields": { + "contract_title": { + "value": "Supply Agreement", + "raw": "Supply Agreement", + "source": { + "page": 1, + "bbox": [ + 81.29, + 101.78, + 160.33, + 112.28 + ], + "text": "Supply Agreement" + } + }, + "party_a": { + "value": "Sample Robotics Ltd.", + "raw": "Sample Robotics Ltd.", + "source": { + "page": 1, + "bbox": [ + 400.62, + 101.78, + 491.92, + 112.28 + ], + "text": "Sample Robotics Ltd." + } + }, + "party_b": { + "value": "Demo Freight GmbH", + "raw": "Demo Freight GmbH", + "source": { + "page": 1, + "bbox": [ + 109.57, + 115.78, + 199.4, + 126.28 + ], + "text": "Demo Freight GmbH" + } + }, + "effective_date": { + "value": "2026-09-15", + "raw": "2026-09-15", + "source": { + "page": 1, + "bbox": [ + 247.81, + 101.78, + 329.47, + 112.28 + ], + "text": "15 September 2026" + } + }, + "term_months": { + "value": 12, + "raw": "12", + "source": { + "page": 1, + "bbox": [ + 265.29, + 187.78, + 282.78, + 198.28 + ], + "text": "(12)" + } + }, + "notice_period_days": { + "value": 90, + "raw": "90", + "source": { + "page": 1, + "bbox": [ + 289.52, + 309.78, + 307.01, + 320.28 + ], + "text": "(90)" + } + }, + "payment_terms_days": { + "value": 45, + "raw": "45", + "source": { + "page": 1, + "bbox": [ + 393.0, + 237.78, + 403.5, + 248.28 + ], + "text": "45" + } + }, + "governing_law": { + "value": "England and Wales", + "raw": "England and Wales", + "source": { + "page": 1, + "bbox": [ + 412.25, + 359.78, + 496.53, + 370.28 + ], + "text": "England and Wales." + } + } + } + }, + { + "file": "invoice-acme-scanned.pdf", + "schema": "invoice", + "backend": "hf", + "model": "Qwen/Qwen2.5-1.5B-Instruct", + "fields": { + "invoice_number": { + "value": "AES-5120", + "raw": "AES-5120", + "source": { + "page": 1, + "bbox": [ + 175.0, + 149.0, + 232.5, + 162.5 + ], + "text": "AES-5120" + } + }, + "invoice_date": { + "value": "2026-06-08", + "raw": "2026-06-08", + "source": { + "page": 1, + "bbox": [ + 175.0, + 169.5, + 244.0, + 182.0 + ], + "text": "8 June 2026" + } + }, + "due_date": { + "value": "2026-07-08", + "raw": "2026-07-08", + "source": { + "page": 1, + "bbox": [ + 176.5, + 190.5, + 239.0, + 203.0 + ], + "text": "8 July 2026" + } + }, + "seller_name": { + "value": "Acme Example Services Pte. Ltd.", + "raw": "Acme Example Services Pte. Ltd.", + "source": { + "page": 1, + "bbox": [ + 54.0, + 61.5, + 370.5, + 86.0 + ], + "text": "Acme Example Services Pte. Ltd." + } + }, + "buyer_name": { + "value": "Northwind Parts Pte., Ltd.", + "raw": "Northwind Parts Pte., Ltd.", + "source": { + "page": 1, + "bbox": [ + 176.5, + 209.0, + 311.0, + 222.5 + ], + "text": "Northwind Parts Pte ., Ltd." + } + }, + "currency": { + "value": "SGD", + "raw": "SGD", + "source": { + "page": 1, + "bbox": [ + 498.29, + 247.5, + 533.5, + 261.0 + ], + "text": "(SGD)" + } + }, + "subtotal": { + "value": 5219.37, + "raw": "5219.37", + "source": { + "page": 1, + "bbox": [ + 486.0, + 353.0, + 535.0, + 365.5 + ], + "text": "5,219.37" + } + }, + "tax": { + "value": 469.74, + "raw": "469.74", + "source": { + "page": 1, + "bbox": [ + 496.5, + 375.0, + 536.0, + 386.5 + ], + "text": "469.74" + } + }, + "total": { + "value": 5689.11, + "raw": "5689.11", + "source": { + "page": 1, + "bbox": [ + 486.0, + 397.0, + 535.0, + 409.5 + ], + "text": "5,689.11" + } + } + } + }, + { + "file": "invoice-demo-freight-compact.pdf", + "schema": "invoice", + "backend": "hf", + "model": "Qwen/Qwen2.5-1.5B-Instruct", + "fields": { + "invoice_number": { + "value": "DF-88213", + "raw": "DF-88213", + "source": { + "page": 1, + "bbox": [ + 504.17, + 122.17, + 545.28, + 132.17 + ], + "text": "DF-88213" + } + }, + "invoice_date": { + "value": "2026-02-03", + "raw": "03.02.2026", + "source": { + "page": 1, + "bbox": [ + 500.28, + 137.17, + 545.28, + 147.17 + ], + "text": "03.02.2026" + } + }, + "due_date": { + "value": "2026-03-05", + "raw": "05.03.2026", + "source": { + "page": 1, + "bbox": [ + 500.28, + 152.17, + 545.28, + 162.17 + ], + "text": "05.03.2026" + } + }, + "seller_name": { + "value": "Demo Freight GmbH", + "raw": "Demo Freight GmbH", + "source": { + "page": 1, + "bbox": [ + 50.0, + 47.47, + 195.76, + 63.47 + ], + "text": "Demo Freight GmbH" + } + }, + "buyer_name": { + "value": "Northwind Parts Pte. Ltd.", + "raw": "Northwind Parts Pte. Ltd.", + "source": { + "page": 1, + "bbox": [ + 50.0, + 136.17, + 152.5, + 146.17 + ], + "text": "Northwind Parts Pte. Ltd." + } + }, + "currency": { + "value": "EUR", + "raw": "EUR", + "source": { + "page": 1, + "bbox": [ + 525.28, + 167.17, + 545.28, + 177.17 + ], + "text": "EUR" + } + }, + "subtotal": { + "value": 3151.06, + "raw": "3151.06", + "source": { + "page": 1, + "bbox": [ + 487.78, + 296.17, + 522.78, + 306.17 + ], + "text": "3,151.06" + } + }, + "tax": { + "value": 598.7, + "raw": "598.7", + "source": { + "page": 1, + "bbox": [ + 495.28, + 311.17, + 522.78, + 321.17 + ], + "text": "598.70" + } + }, + "total": { + "value": 3749.76, + "raw": "3749.76", + "source": { + "page": 1, + "bbox": [ + 480.8, + 328.39, + 519.3, + 339.39 + ], + "text": "3,749.76" + } + } + } + }, + { + "file": "invoice-example-logistics-banner.pdf", + "schema": "invoice", + "backend": "hf", + "model": "Qwen/Qwen2.5-1.5B-Instruct", + "fields": { + "invoice_number": { + "value": "EL/INV/7731", + "raw": "EL/INV/7731", + "source": { + "page": 1, + "bbox": [ + 515.3, + 52.07, + 572.0, + 62.07 + ], + "text": "EL/INV/7731" + } + }, + "invoice_date": { + "value": "2026-05-02", + "raw": "2026-05-02", + "source": { + "page": 1, + "bbox": [ + 75.02, + 172.07, + 126.16, + 182.07 + ], + "text": "2026-05-02" + } + }, + "due_date": { + "value": "2026-06-01", + "raw": "2026-06-01", + "source": { + "page": 1, + "bbox": [ + 264.48, + 172.07, + 315.62, + 182.07 + ], + "text": "2026-06-01" + } + }, + "seller_name": { + "value": "Example Logistics Inc.", + "raw": "Example Logistics Inc.", + "source": { + "page": 1, + "bbox": [ + 40.0, + 127.07, + 139.48, + 137.07 + ], + "text": "Example Logistics Inc." + } + }, + "buyer_name": { + "value": "Sample Robotics Ltd.", + "raw": "Sample Robotics Ltd.", + "source": { + "page": 1, + "bbox": [ + 306.0, + 127.07, + 401.04, + 137.07 + ], + "text": "Sample Robotics Ltd." + } + }, + "currency": { + "value": "USD", + "raw": "USD", + "source": { + "page": 1, + "bbox": [ + 426.12, + 172.07, + 447.23, + 182.07 + ], + "text": "USD" + } + }, + "subtotal": { + "value": 12594.58, + "raw": "12594.58", + "source": { + "page": 1, + "bbox": [ + 517.52, + 332.07, + 562.0, + 342.07 + ], + "text": "12,594.58" + } + }, + "tax": { + "value": 881.62, + "raw": "881.62", + "source": { + "page": 1, + "bbox": [ + 531.42, + 350.07, + 562.0, + 360.07 + ], + "text": "881.62" + } + }, + "total": { + "value": 13476.2, + "raw": "13476.2", + "source": { + "page": 1, + "bbox": [ + 508.62, + 374.48, + 562.0, + 386.48 + ], + "text": "13,476.20" + } + } + } + }, + { + "file": "invoice-northwind-classic.pdf", + "schema": "invoice", + "backend": "hf", + "model": "Qwen/Qwen2.5-1.5B-Instruct", + "fields": { + "invoice_number": { + "value": "NWP-2026-0412", + "raw": "NWP-2026-0412", + "source": { + "page": 1, + "bbox": [ + 470.81, + 84.07, + 545.28, + 94.07 + ], + "text": "NWP-2026-0412" + } + }, + "invoice_date": { + "value": "2026-03-14", + "raw": "2026-03-14", + "source": { + "page": 1, + "bbox": [ + 478.58, + 98.07, + 545.28, + 108.07 + ], + "text": "14 March 2026" + } + }, + "due_date": { + "value": "2026-04-13", + "raw": "2026-04-13", + "source": { + "page": 1, + "bbox": [ + 486.36, + 112.07, + 545.28, + 122.07 + ], + "text": "13 April 2026" + } + }, + "seller_name": { + "value": "Northwind Parts Pte. Ltd.", + "raw": "Northwind Parts Pte. Ltd.", + "source": { + "page": 1, + "bbox": [ + 50.0, + 55.73, + 265.03, + 73.73 + ], + "text": "Northwind Parts Pte. Ltd." + } + }, + "buyer_name": { + "value": "Example Logistics Pte. Ltd.", + "raw": "Example Logistics Pte. Ltd.", + "source": { + "page": 1, + "bbox": [ + 50.0, + 156.07, + 170.61, + 166.07 + ], + "text": "Example Logistics Pte. Ltd." + } + }, + "currency": { + "value": "SGD", + "raw": "SGD", + "source": { + "page": 1, + "bbox": [ + 490.25, + 362.07, + 511.92, + 372.07 + ], + "text": "SGD" + } + }, + "subtotal": { + "value": 5042.83, + "raw": "5042.83", + "source": { + "page": 1, + "bbox": [ + 506.36, + 344.07, + 545.28, + 354.07 + ], + "text": "5,042.83" + } + }, + "tax": { + "value": 453.85, + "raw": "453.85", + "source": { + "page": 1, + "bbox": [ + 514.7, + 362.07, + 545.28, + 372.07 + ], + "text": "453.85" + } + }, + "total": { + "value": 5496.68, + "raw": "5496.68", + "source": { + "page": 1, + "bbox": [ + 502.46, + 383.28, + 545.28, + 394.28 + ], + "text": "5,496.68" + } + } + } + }, + { + "file": "invoice-placeholder-two-page.pdf", + "schema": "invoice", + "backend": "hf", + "model": "Qwen/Qwen2.5-1.5B-Instruct", + "fields": { + "invoice_number": { + "value": "PS-INV-30027", + "raw": "PS-INV-30027", + "source": { + "page": 1, + "bbox": [ + 487.25, + 64.86, + 545.28, + 73.86 + ], + "text": "PS-INV-30027" + } + }, + "invoice_date": { + "value": "2026-01-20", + "raw": "20/01/2026", + "source": { + "page": 1, + "bbox": [ + 113.92, + 106.07, + 163.96, + 116.07 + ], + "text": "20/01/2026" + } + }, + "due_date": { + "value": "2026-02-19", + "raw": "19/02/2026", + "source": { + "page": 1, + "bbox": [ + 138.37, + 120.07, + 188.41, + 130.07 + ], + "text": "19/02/2026" + } + }, + "seller_name": { + "value": "Placeholder Supplies Pte. Ltd.", + "raw": "Placeholder Supplies Pte. Ltd.", + "source": { + "page": 1, + "bbox": [ + 50.0, + 48.9, + 250.73, + 62.9 + ], + "text": "Placeholder Supplies Pte. Ltd." + } + }, + "buyer_name": { + "value": "Sample Robotics Ltd.", + "raw": "Sample Robotics Ltd.", + "source": { + "page": 1, + "bbox": [ + 98.36, + 92.07, + 193.4, + 102.07 + ], + "text": "Sample Robotics Ltd." + } + }, + "currency": { + "value": "SGD", + "raw": "SGD", + "source": { + "page": 1, + "bbox": [ + 519.78, + 152.86, + 545.28, + 161.86 + ], + "text": "(SGD)" + } + }, + "subtotal": { + "value": 6204.22, + "raw": "6204.22", + "source": { + "page": 2, + "bbox": [ + 506.36, + 268.07, + 545.28, + 278.07 + ], + "text": "6,204.22" + } + }, + "tax": { + "value": 558.38, + "raw": "558.38", + "source": { + "page": 2, + "bbox": [ + 514.7, + 284.07, + 545.28, + 294.07 + ], + "text": "558.38" + } + }, + "total": { + "value": 6762.6, + "raw": "6762.60", + "source": { + "page": 2, + "bbox": [ + 502.46, + 303.28, + 545.28, + 314.28 + ], + "text": "6,762.60" + } + } + } + } +] diff --git a/docs/eval-hf.md b/docs/eval-hf.md new file mode 100644 index 0000000..3b81b36 --- /dev/null +++ b/docs/eval-hf.md @@ -0,0 +1,13 @@ + +Backend: hf, model: Qwen/Qwen2.5-1.5B-Instruct + +| Sample | Correct | Sourced | Time (s) | Misses | +|---|---|---|---|---| +| contract-master-services.pdf | 8/8 | 8/8 | 18.7 | - | +| contract-supply.pdf | 8/8 | 8/8 | 9.0 | - | +| invoice-acme-scanned.pdf | 8/9 | 8/8 | 22.1 | buyer_name='Northwind Parts Pte., Ltd.' (want 'Northwind Parts Pte. Ltd.') | +| invoice-demo-freight-compact.pdf | 9/9 | 9/9 | 10.5 | - | +| invoice-example-logistics-banner.pdf | 9/9 | 9/9 | 9.7 | - | +| invoice-northwind-classic.pdf | 9/9 | 9/9 | 10.9 | - | +| invoice-placeholder-two-page.pdf | 9/9 | 9/9 | 13.6 | - | +| **All** | **60/61 (98%)** | 60/60 | | | diff --git a/docs/eval.md b/docs/eval.md new file mode 100644 index 0000000..0ad9c91 --- /dev/null +++ b/docs/eval.md @@ -0,0 +1,48 @@ +# Evaluation record + +Numbers used in the README and on the case page. Re-run with: + +```bash +python scripts/evaluate.py +``` + +A field counts as correct when the typed value matches the ground truth in `samples/*.truth.json` (strings compared case-insensitively, extra spaces and trailing punctuation ignored; numbers within 0.005). “Sourced” counts correct values that also carry a source box whose text matches the value. The logic is in `scripts/evaluate.py`. All seven PDFs are synthetic; all names and figures are fictional. + +## Hugging Face backend — 12 November 2026 + +- Backend: `hf` +- Model: `Qwen/Qwen2.5-1.5B-Instruct` +- OCR (scanned page): Florence-2 (`florence-community/Florence-2-large`) +- Machine: 8-core CPU (Intel Xeon), 15 GB RAM, no GPU, PyTorch 2.15.0+cpu, 8 threads +- Wall-clock for all seven samples: 96 seconds, including model load +- Source of this table: a quiet re-run of `scripts/evaluate.py` on that machine, 12 November 2026 (Asia/Singapore) + +| Sample | Correct | Sourced | Time (s) | Notes | +|---|---|---|---|---| +| contract-master-services.pdf | 8/8 | 8/8 | 18.7 | — | +| contract-supply.pdf | 8/8 | 8/8 | 9.0 | — | +| invoice-acme-scanned.pdf | 8/9 | 8/8 | 22.1 | `buyer_name='Northwind Parts Pte., Ltd.'` (want `'Northwind Parts Pte. Ltd.'`) | +| invoice-demo-freight-compact.pdf | 9/9 | 9/9 | 10.5 | — | +| invoice-example-logistics-banner.pdf | 9/9 | 9/9 | 9.7 | — | +| invoice-northwind-classic.pdf | 9/9 | 9/9 | 10.9 | — | +| invoice-placeholder-two-page.pdf | 9/9 | 9/9 | 13.6 | — | +| **All** | **60/61 (98%)** | **60/60** | **9.0–22.1** | | + +The first document’s time includes some model-load cost. The same accuracy (60/61, 60/60 sourced, the same OCR miss) was also obtained on an earlier run on this machine, with per-document times of 9.3–24.9 seconds. + +## API backend — November 2026 + +Recorded on the same 8-core CPU, against a local llama.cpp server running Qwen2.5-1.5B-Instruct (Q4_K_M GGUF). Source: the evaluation captured for v1 (`eval-api.md`). + +| Sample | Correct | Sourced | Time (s) | Notes | +|---|---|---|---|---| +| contract-master-services.pdf | 8/8 | 8/8 | 7.0 | — | +| contract-supply.pdf | 8/8 | 8/8 | 3.9 | — | +| invoice-acme-scanned.pdf | 0/9 | 0/0 | 0.0 | no text layer, no OCR configured; clean error | +| invoice-demo-freight-compact.pdf | 9/9 | 9/9 | 6.4 | — | +| invoice-example-logistics-banner.pdf | 9/9 | 9/9 | 4.7 | — | +| invoice-northwind-classic.pdf | 9/9 | 9/9 | 5.9 | — | +| invoice-placeholder-two-page.pdf | 9/9 | 9/9 | 12.1 | — | +| **All** | **52/61 (85%)** | **52/52** | **3.9–12.1** | | + +The 7B default (`Qwen/Qwen2.5-7B-Instruct`) has not been measured.