Bulk ZIP Import¶

The bulk ZIP import allows uploading many documents at once, preserving folder structure from the ZIP archive. It also supports optional metadata and relationship CSV files to set document titles/descriptions and create inter-document relationships automatically.

Overview¶

Upload a ZIP file containing documents organized in folders. The folder structure is preserved in the target corpus. Two optional CSV files at the ZIP root provide metadata overrides and document-to-document relationships.

GraphQL Mutation: ImportZipToCorpus

The import runs asynchronously via Celery and returns a jobId for progress tracking.

Bulk-loading a very large local collection (100k+ files)? Use the resumable CLI driver at scripts/bulk_import/oc_bulk_import.py. It walks a local directory tree, packs the files into right-sized ZIPs that feed this same zip-to-corpus endpoint (mirroring the folder structure), paces itself against the corpus parse backlog, and records progress in a SQLite ledger so the job is crash-resumable. See scripts/bulk_import/README.md for the full operational runbook.

ZIP File Structure¶

A typical ZIP for import:

my-import.zip
+-- contracts/
|   +-- legal/
|   |   +-- agreement.pdf
|   +-- financial/
|       +-- report.pdf
+-- docs/
|   +-- amendment.pdf
+-- readme.txt
+-- meta.csv             (optional)
+-- relationships.csv    (optional)

Supported Document Types in ZIP¶

All file types registered in the pipeline are accepted. Currently:

• PDF (.pdf)
• Word (.docx)
• Plain Text (.txt)

Skipped Files¶

The following are automatically skipped:

• Hidden files (names starting with .)
• macOS metadata (__MACOSX/, .DS_Store)
• Files exceeding size limits
• Unsupported file types
• The meta.csv and relationships.csv control files themselves

Metadata File (meta.csv)¶

An optional CSV file at the ZIP root that overrides document titles and descriptions. Accepted filenames (priority order): meta.csv, META.csv, metadata.csv, METADATA.csv.

Schema¶

Example¶

source_path,title,description,external_id
contracts/legal/agreement.pdf,Master Services Agreement,The main services contract,
contracts/financial/report.pdf,Q4 Financial Report,Quarterly financial summary,
rulings/H022844.txt,Plastic serving trays; classification,CROSS HQ ruling,cross:H022844

Behavior¶

• Empty cells are ignored (defaults are used instead)
• Not all documents need metadata entries -- partial coverage is fine
• If a titlePrefix is provided in the mutation, it is prepended to the metadata title (e.g., "2024 - Master Services Agreement")
• external_id values should be namespaced by the producer (e.g. cross:H022844); unlike the display title, they survive later renames. Consumers resolve through the namespace — the enrichment engine treats a cross:-namespaced id as a document's canonical ruling number, ahead of its path or title (opencontractserver/enrichment/resolver.py::document_identity_candidates). Overlong values are rejected per-row with an import warning, never truncated.

Relationships File (relationships.csv)¶

An optional CSV file at the ZIP root that creates document-to-document relationships. Must be named relationships.csv or RELATIONSHIPS.csv.

Schema¶

Example¶

source_path,relationship_label,target_path,notes
contracts/legal/agreement.pdf,AMENDS,docs/amendment.pdf,Amendment to main contract
docs/amendment.pdf,AMENDED_BY,contracts/legal/agreement.pdf,
contracts/legal/agreement.pdf,REFERENCES,contracts/financial/report.pdf,

Path Normalization¶

Paths in CSV files are normalized to a canonical form:

• Backslashes converted to forward slashes
• Leading ./ removed
• Leading slashes normalized
• Duplicate slashes collapsed

All of these reference the same document:

contracts/agreement.pdf
/contracts/agreement.pdf
./contracts/agreement.pdf
contracts\agreement.pdf

Path traversal (..) is rejected for security.

Cross-batch Relationships¶

An endpoint that does not match a file inside the current ZIP is resolved against documents already in the corpus (matched by their path within the corpus, which equals the ZIP-relative path when no target folder is used). This lets a relationships.csv connect documents that were imported in separate batches — you can upload a ZIP containing only a relationships.csv to wire together documents already present in the corpus.

Resolution order for each endpoint:

1. A document imported in the current ZIP (same archive), then
2. A current, non-deleted document already in the corpus that you can access.

Only documents you can already see in the corpus are eligible, so cross-batch resolution never exposes a document you lacked access to. An endpoint that matches neither is skipped (and reported), exactly as before.

Annotation Sidecars¶

A document may ship a co-located JSON sidecar (same name, .json extension — e.g. agreement.pdf + agreement.json) describing producer annotations to attach. Each annotation is a dumb anchor: a label + the rawText to find, plus a location hint (PDF page + bbox, or text start/end). The document is parsed normally and the annotations are re-anchored onto the freshly-produced text/PAWLs layer after ingest, so the anchors survive re-parsing.

Sidecar Schema¶

{
  "annotations": [
    {
      "id": 1,
      "label": "OC_SECTION",
      "rawText": "Article 1 — Definitions",
      "page": 0,
      "bbox": {"left": 50, "top": 50, "right": 300, "bottom": 70},
      "link_url": "https://example.com/ref",
      "data": {"canonical_name": "France", "lat": 46.0, "lng": 2.0}
    },
    {
      "id": 2,
      "label": "OC_CLAUSE",
      "rawText": "indemnification obligations",
      "start": 1234,
      "end": 1261
    }
  ],
  "doc_labels": ["Contract"],
  "relationships": [
    {
      "id": "r1",
      "relationshipLabel": "OC_PARENT_CHILD",
      "source_annotation_ids": [1],
      "target_annotation_ids": [2]
    }
  ]
}

Annotation-to-annotation Relationships¶

The optional relationships list declares edges between annotations in the same sidecar. Each entry carries a relationshipLabel (auto-created as a RELATIONSHIP_LABEL if it doesn't exist — e.g. OC_PARENT_CHILD, OC_SUBTREE_GROUP) and source_annotation_ids / target_annotation_ids that reference the sidecar annotations' own ids. Relationships are wired after the annotations are re-anchored, using the producer-id → new-annotation map. An edge whose endpoints did not survive anchoring is dropped and recorded on the import report, never silently lost.

Security Constraints¶

All limits are configurable via Django settings.

Import Phases¶

The import proceeds through four phases:

Phase 1: Validation¶

The ZIP is scanned for security violations (size limits, zip bombs, path traversal) before any files are extracted. A ZipManifest is produced listing valid files, skipped files, and any errors.

Phase 2: Folder Creation¶

Folder structure from the ZIP is created in the corpus in a single transaction. Existing folders with matching names are reused, not duplicated.

Phase 3: Document Import¶

Each valid file is extracted and created as a document. If a document path already exists in the corpus, a new version is created (upversioning) rather than failing.

Phase 4: Relationship Creation¶

If a relationships.csv is present, document relationships are created. Annotation labels are created or reused via the corpus's label set. Missing source/target documents are logged and skipped -- a malformed relationships file does not fail the overall import.

Flat ZIP Upload (Without Corpus)¶

There is also a simpler UploadDocumentsZip mutation that uploads a ZIP of documents without targeting a specific corpus and without preserving folder structure. Documents are created as standalone items. The same security constraints apply.

Error Handling¶

The import uses graceful degradation:

• Validation failures (zip bomb, too many files) block the entire import
• Individual file errors (unsupported type, too large) skip that file and continue
• Relationship errors (missing source/target document) skip that relationship and continue
• The import result includes detailed counts of processed, skipped, errored, and upversioned files

For implementation details (code snippets, task result schema, file locations, and testing commands), see the Bulk Import Architecture.