Skip to content

PDF Data Processing Architecture

Overview

OpenContracts uses a modern, pluggable document processing pipeline that has evolved from the original PAWLs/Grobid approach to support multiple advanced parsing backends. The system now leverages state-of-the-art machine learning models while maintaining backward compatibility with the PAWLs data format.

Current Architecture

Parser Pipeline System

OpenContracts implements a modular pipeline architecture with four main parser options:

  1. Docling Parser (Primary) - IBM's advanced ML-based parser running as a REST microservice
  2. Source: opencontractserver/pipeline/parsers/docling_parser_rest.py
  3. Superior layout understanding and table extraction
  4. Intelligent OCR with automatic detection
  5. Hierarchical document structure extraction

  6. Group relationship detection for contract clauses

  7. LlamaParse Parser - Cloud-based parser using LlamaIndex API

  8. Source: opencontractserver/pipeline/parsers/llamaparse_parser.py
  9. High-quality layout extraction
  10. Automatic OCR support

  11. Good for complex document structures

  12. Text Parser - Simple parser for plain text and markdown files

  13. Source: opencontractserver/pipeline/parsers/oc_text_parser.py
  14. Direct text extraction
  15. Minimal processing overhead
  16. Preserves original formatting

Data Layers

OpenContracts maintains a multi-layered data architecture that provides a consistent interface regardless of the parsing backend used:

1. PAWLs Layer (JSON)

The PAWLs (PDF Annotation With Labels) layer remains the core data format, storing: - Individual tokens (words) with precise bounding box coordinates - Page dimensions and layout information - Token-level positional data enabling pixel-perfect annotation overlay - Hierarchical structure information (headers, paragraphs, lists)

{
  "pawls_file_content": [
    {
      "page": {"width": 612, "height": 792, "index": 0},
      "tokens": [
        {
          "text": "Contract",
          "bbox": {"x": 100, "y": 50, "width": 80, "height": 15}
        }
      ]
    }
  ]
}

2. Text Layer

A pure text extraction built from the PAWLs layer that: - Preserves reading order - Maintains paragraph and section boundaries - Enables full-text search and NLP processing - Provides character-level position mapping back to PAWLs tokens

3. Annotation Layer

Structural and semantic annotations including: - Document structure (headers, sections, paragraphs) - Detected entities and labels - User-created annotations - Relationships between document elements

4. Relationship Layer (New)

Advanced parsers like Docling can detect relationships between document elements: - Parent-child hierarchies (section → subsection) - Cross-references between clauses - Grouped elements (related paragraphs, list items) - Table cell relationships

Processing Pipeline

```mermaid graph LR A[PDF Upload] → B{Parser Selection} B → C[Docling REST API] B → D[LlamaParse API] B → E[Text Parser] B → P[LlamaParse API]

C --> F[PAWLs Generation]
D --> F
E --> F
P --> F

F --> G[Text Extraction]
F --> H[Annotation Creation]
F --> I[Relationship Mapping]

G --> J[Searchable Text Layer]
H --> K[Visual Annotations]
I --> L[Document Graph]

```

Evolution from Original PAWLs

The original OpenContracts implementation used: - Grobid for layout analysis - Tesseract for OCR - Re-OCR of every document for consistency

The current system has evolved to: - Use modern ML models for better accuracy - Support multiple parsing backends - Preserve embedded text when appropriate - Only apply OCR when needed (configurable) - Extract richer structural information

Key Improvements

  1. Better Accuracy: ML-based parsers provide superior layout understanding
  2. Flexibility: Choose the right parser for your document types
  3. Performance: Microservice architecture enables better scaling
  4. Rich Structure: Extract hierarchies and relationships, not just text
  5. Selective OCR: Only OCR when needed, preserving original text quality

Maintaining Compatibility

Despite the architectural evolution, OpenContracts maintains full compatibility: - PAWLs format remains the standard interface - All parsers output to the same data structure - Existing annotations and tools continue to work - Text-to-position mapping preserved

Configuration

Parsers are configured in Django settings. See the base settings file for current defaults.

Available Parser Classes: - opencontractserver.pipeline.parsers.docling_parser_rest.DoclingParser - Primary ML-based parser - opencontractserver.pipeline.parsers.oc_text_parser.TxtParser - Plain text parser - opencontractserver.pipeline.parsers.llamaparse_parser.LlamaParseParser - LlamaIndex cloud parser

Example Configuration: 

PREFERRED_PARSERS = {
    "application/pdf": "opencontractserver.pipeline.parsers.docling_parser_rest.DoclingParser",
    "text/plain": "opencontractserver.pipeline.parsers.oc_text_parser.TxtParser",
}

# Parser-specific settings
DOCLING_PARSER_SERVICE_URL = "http://docling-parser:8000/parse/"
DOCLING_PARSER_TIMEOUT = 300

Limitations and Trade-offs

Current Limitations

  1. OCR Quality: While improved, OCR can still make errors (O vs 0, I vs 1)
  2. Processing Time: ML-based parsers are slower than simple text extraction
  3. Resource Usage: Advanced parsers require more memory and CPU
  4. Format Support: Currently limited to PDF and text formats

Design Trade-offs

  1. Accuracy vs Speed: ML parsers are more accurate but slower
  2. Flexibility vs Complexity: Multiple parsers add configuration complexity
  3. Consistency vs Fidelity: Standardizing to PAWLs format may lose some format-specific features

Future Directions

  1. Additional Format Support: DOCX, XLSX, HTML parsing
  2. Streaming Processing: Handle very large documents efficiently
  3. Custom Parser Plugins: Easy integration of domain-specific parsers
  4. Enhanced Relationships: More sophisticated document graph analysis
  5. Hybrid Processing: Combine multiple parsers for optimal results

See Also