PAWLs Format Specification¶

Overview¶

PAWLs (Page-Aware Word-Level Segmentation) is OpenContracts' format for representing document structure with precise token positioning. Each page in a document has tokens (text or image) with bounding box coordinates that enable:

• Precise text selection and annotation
• Image region identification and annotation
• Spatial queries for finding tokens in regions
• Frontend rendering with accurate positioning

Format Structure¶

A PAWLs file is a JSON array of page objects:

[
  {
    "page": {
      "width": 612.0,
      "height": 792.0,
      "index": 0
    },
    "tokens": [
      {"x": 100, "y": 100, "width": 50, "height": 12, "text": "Hello"},
      {"x": 160, "y": 100, "width": 60, "height": 12, "text": "World"}
    ]
  },
  {
    "page": {"width": 612.0, "height": 792.0, "index": 1},
    "tokens": [...]
  }
]

Page Object¶

Token Object¶

Tokens represent either text or images. The is_image field distinguishes between them.

Common Fields (All Tokens)¶

Image Token Fields¶

When is_image is true, the token represents an image:

*Either image_path (preferred) or base64_data should be present.

Text Token Example¶

{
  "x": 100.5,
  "y": 150.25,
  "width": 45.0,
  "height": 12.0,
  "text": "Revenue"
}

Image Token Example¶

{
  "x": 50.0,
  "y": 200.0,
  "width": 300.0,
  "height": 200.0,
  "text": "",
  "is_image": true,
  "image_path": "documents/123/images/page_0_img_0.jpg",
  "format": "jpeg",
  "content_hash": "a1b2c3d4e5f6...",
  "original_width": 800,
  "original_height": 533,
  "image_type": "embedded"
}

Coordinate System¶

• Origin: Top-left corner of the page
• Units: PDF points (1 point = 1/72 inch)
• X-axis: Increases left to right
• Y-axis: Increases top to bottom
• Standard page size: Letter is 612 x 792 points

Token References¶

Annotations reference tokens using TokenIdPythonType:

{
  "pageIndex": 0,
  "tokenIndex": 5
}

This format works for both text and image tokens since they're in the same array.

Annotation Integration¶

Single Modality Annotation (Text Only)¶

{
  "tokens_jsons": [
    {"pageIndex": 0, "tokenIndex": 0},
    {"pageIndex": 0, "tokenIndex": 1}
  ],
  "content_modalities": ["TEXT"]
}

Single Modality Annotation (Image Only)¶

{
  "tokens_jsons": [
    {"pageIndex": 0, "tokenIndex": 15}
  ],
  "content_modalities": ["IMAGE"]
}

Mixed Modality Annotation (Image + Caption)¶

{
  "tokens_jsons": [
    {"pageIndex": 0, "tokenIndex": 15},
    {"pageIndex": 0, "tokenIndex": 16},
    {"pageIndex": 0, "tokenIndex": 17}
  ],
  "content_modalities": ["IMAGE", "TEXT"]
}

Image Storage¶

Images are stored separately from the PAWLs file to avoid bloat:

1. During parsing: Images are extracted and saved to Django storage (S3, GCS, or filesystem)
2. In PAWLs: Only the image_path reference is stored
3. On retrieval: Image tools load from storage and return base64 data

Storage Path Convention¶

documents/{document_id}/images/page_{page_idx}_img_{img_idx}.{format}

Example: documents/123/images/page_0_img_0.jpg

Content Modalities¶

The content_modalities field on Annotation tracks what types of content are present:

This enables embedders to efficiently filter annotations they can process.

Parser Responsibilities¶

When generating PAWLs data, parsers should:

1. Extract text tokens with accurate bounding boxes
2. Extract images and save to storage
3. Create image tokens in the tokens[] array with is_image: true
4. For structural annotations (figures, charts):
5. Reference image tokens via tokens_jsons
6. Set content_modalities: ["IMAGE"]

Frontend Handling¶

The frontend should:

1. Check token.is_image to identify image tokens
2. Render image tokens with different visual treatment (e.g., border instead of text highlight)
3. Allow selection of both text and image tokens
4. Display mixed annotations spanning both types

v1 vs v2: Compact PAWLs Format¶

Motivation¶

PAWLs files can be large — a typical 9-page PDF produces ~549 KB of v1 JSON. Since every document stores a PAWLs file (in S3, GCS, or filesystem via the pawls_parse_file field on Document), the aggregate storage cost is significant. The v2 compact format reduces this by ~67% (549 KB → 180 KB in measured benchmarks).

v1 Format (Legacy)¶

The original format, documented above. A JSON array of page objects with verbose, human-readable keys:

[
  {
    "page": {"width": 612.0, "height": 792.0, "index": 0},
    "tokens": [
      {"x": 72.0, "y": 720.0, "width": 41.0, "height": 12.0, "text": "Hello"},
      {"x": 120.5, "y": 720.0, "width": 35.2, "height": 12.0, "text": "world"}
    ]
  }
]

Per text token overhead: ~105 characters (JSON key names dominate).

v2 Format (Compact)¶

A JSON dict with a version marker. Tokens become positional arrays; keys are shortened:

{
  "v": 2,
  "p": [
    {
      "w": 612.0,
      "h": 792.0,
      "t": [
        [72.0, 720.0, 41.0, 12.0, "Hello"],
        [120.5, 720.0, 35.2, 12.0, "world"]
      ]
    }
  ]
}

Per text token overhead: ~37 characters (~65% savings per token).

Image tokens carry a 6^th element with compact metadata:

[0.0, 100.0, 200.0, 300.0, "", {"p": "documents/123/images/page_0_img_0.jpg", "f": "jpeg", "ch": "a1b2c3..."}]

Note: The presence of a 6^th element (the metadata dict) is what distinguishes image tokens from text tokens in v2. On decode, expand_pawls_pages() reconstructs the is_image: true field from this — v2 does not store is_image explicitly.

Five Compression Techniques¶

Image Metadata Key Mapping¶

Format Detection¶

The two formats are distinguishable by shape:

• v1: Top-level value is a JSON array ([{…}, …])
• v2: Top-level value is a JSON dict with "v": 2 and "p" keys

from opencontractserver.utils.compact_pawls import is_compact_pawls_format

is_compact_pawls_format([...])          # False (v1)
is_compact_pawls_format({"v": 2, "p": [...]})  # True (v2)

The Accessor Layer: Why We Didn't Replace v1 Throughout¶

Rather than rewriting every consumer to understand v2, we use a format-agnostic accessor layer. All code reads PAWLs through expand_pawls_pages(), which transparently normalizes either format to v1 shape:

from opencontractserver.utils.compact_pawls import expand_pawls_pages

# Always returns list[PawlsPagePythonType] regardless of input format
pages = expand_pawls_pages(raw_json)

This design was chosen over a full v1 replacement for several architectural reasons:

1. Storage is file-based, not column-based¶

PAWLs data lives in Django FileField storage (S3/GCS/filesystem), not in a database column. There is no single SQL migration that can convert all existing files. A backfill job would need to download, re-encode, and re-upload every file — risky and unnecessary when the accessor layer handles both formats.

2. Backward compatibility with zero consumer changes¶

Dozens of consumers read PAWLs data: LLM tools, PDF redaction, annotation import/export, the frontend REST layer, and more. Rewriting all of them to use v2 arrays would be a large, error-prone change with no functional benefit — they all need the same v1-shaped data structures internally.

3. Incremental adoption without a "big bang" migration¶

New documents are automatically stored in v2 (all write paths call compact_pawls_pages()). Old v1 documents continue to work as-is. Over time, as documents are re-parsed or replaced, the corpus naturally migrates to v2 — no coordinated migration event required.

4. Graceful fallback for edge cases¶

If a page has more than 100,000 tokens (pathological input), compact_pawls_pages() falls back to storing v1 format rather than producing a potentially broken compact file. The read path handles both.

5. Frontend only needs the decoder¶

The frontend never writes PAWLs — it only fetches and renders. So it only needs expandPawlsPages() (the v2 → v1 decoder), which lives in frontend/src/utils/compactPawls.ts. The entire frontend codebase continues to work with v1 types (PageTokens[], Token).

Write Paths (Where v2 Encoding Happens)¶

Primary entry points that persist PAWLs files automatically compact to v2. If any page exceeds MAX_TOKENS_PER_PAGE (100,000), the entire document falls back to v1 format.

from opencontractserver.utils.compact_pawls import compact_pawls_pages

compact_data = compact_pawls_pages(v1_pages)  # v2 dict (or v1 fallback)
pawls_string = json.dumps(compact_data)

Read Paths (Where v2 Expansion Happens)¶

Key consumers read through expand_pawls_pages(). Run grep -r expand_pawls_pages for the full list (~15 files).

Constants¶

Defined in opencontractserver/constants/pawls.py:

Implementation Files¶

Comparison with Annotation Compact Format¶

A similar v2 compression strategy exists for annotation JSON payloads in opencontractserver/annotations/compact_json.py. It uses the same design principles (version marker, format-agnostic accessor, auto-compact on write) but applies range-encoding for token indices instead of array-based tokens. The annotation format achieves ~75% storage reduction.

Migration Notes¶

If processing older documents without image tokens:

• Documents parsed before image support have only text tokens
• is_image field will be absent (falsy) for all tokens
• Re-parsing with current parsers will add image tokens

If processing older documents with v1 PAWLs format:

• v1 files on disk are NOT automatically converted — they stay as-is until re-parsed
• All read paths handle both formats transparently via expand_pawls_pages()
• New documents are always stored in v2 format automatically

Related Documentation¶

• PAWLs Token Format Walkthrough
• Image Token Implementation Plan
• Pipeline Overview