Worker Uploads (REST API)¶
The worker upload system provides a REST API for external pipelines and scripts to push pre-processed documents -- with annotations, labels, and embeddings already attached -- directly into a corpus. Unlike the other upload methods, worker uploads bypass OpenContracts' built-in parsing pipeline entirely because the documents have already been processed externally.
When to Use Worker Uploads¶
- External processing pipelines -- your organization has a custom NLP or document processing pipeline that produces structured output
- Bulk ingestion -- migrating thousands of pre-annotated documents from another system
- Pre-computed embeddings -- documents where text extraction, layout analysis, and embedding generation have already been performed
How It Works¶
Worker uploads use a token-scoped REST API with a database-backed processing queue:
- An admin creates a WorkerAccount (a service account with no web login)
- A corpus owner creates a CorpusAccessToken scoped to a specific corpus
- The external pipeline sends documents via
POST /api/worker-uploads/documents/with the token in theAuthorization: WorkerKey <token>header - Uploads are staged in a database queue and processed asynchronously by a Celery batch processor
- Each upload becomes a Document with annotations, labels, and embeddings in the target corpus
Key Differences from Other Upload Methods¶
| Feature | Worker Uploads | Other Methods |
|---|---|---|
| Authentication | Scoped API token (WorkerKey) | Session/Auth0 |
| Processing pipeline | Skipped (data pre-processed) | Runs parser, thumbnailer, embedder |
| Annotations | Included in upload metadata | Created during parsing or added manually |
| Embeddings | Can be pre-computed and included | Generated by the embedder pipeline stage |
| Folder assignment | Via target_folder_path in metadata | Via UI or mutation parameter |
| Document ownership | Corpus creator (not the worker) | Uploading user |
REST API Endpoints¶
| Method | Endpoint | Description |
|---|---|---|
| POST | /api/worker-uploads/documents/ | Upload a document with metadata |
| GET | /api/worker-uploads/documents/<upload-id>/ | Check status of a single upload |
| GET | /api/worker-uploads/documents/list/ | List uploads for this token (paginated) |
All endpoints require the Authorization: WorkerKey <token> header.
Upload Request¶
The upload is a multipart/form-data POST with two fields:
| Field | Type | Description |
|---|---|---|
file | File | The document file (PDF, DOCX, TXT, etc.) |
metadata | String | JSON string with document text, annotations, labels, and embeddings |
Required Metadata Fields¶
| Field | Type | Description |
|---|---|---|
title | string | Document title |
content | string | Full extracted text |
page_count | integer | Number of pages |
pawls_file_content | array | PAWLs token data (one entry per page with page dimensions and tokens) |
Optional Metadata Fields¶
| Field | Type | Description |
|---|---|---|
description | string | Document description |
file_type | string | MIME type (defaults to application/pdf) |
target_folder_path | string | Folder path to place the document in (auto-created if missing) |
doc_labels | array of strings | Document-level label names to apply |
doc_labels_definitions | object | Label definitions for document-level labels |
text_labels | object | Label definitions for text annotations |
labelled_text | array | Text annotations (same format as corpus export annotations) |
relationships | array | Annotation-to-annotation relationships |
embeddings | object | Pre-computed vectors with embedder_path, optional document_embedding, and annotation_embeddings map |
Upload Lifecycle¶
PENDING --> PROCESSING --> COMPLETED
|
+--------> FAILED
- PENDING -- staged, waiting for the batch processor
- PROCESSING -- batch processor has claimed this upload
- COMPLETED -- document created and added to the corpus
- FAILED -- check the
error_messagefield for details
A Celery worker must drain the queue
The batch processor (process_pending_uploads) runs on the worker_uploads Celery queue, and the post-upload ingest stages (thumbnail, parse, unlock) run on the doc_parse queue. The target deployment must run a Celery worker consuming all three queues (the stock images start it with -Q celery,worker_uploads,doc_parse) plus Celery Beat. With no worker draining worker_uploads, uploads are accepted (HTTP 202) but stay PENDING indefinitely and no documents are created -- a silent failure for self-hosted/custom deployments. See Celery Configuration for Worker Uploads for the full operations reference (queues, Beat, scaling, verification).
Uploads stuck in PROCESSING for longer than WORKER_UPLOAD_STALE_MINUTES (default: 15 minutes) are automatically reset to PENDING by a recovery task (recover_stalled_uploads, scheduled by Celery Beat).
Security Model¶
- Token storage: Only a SHA-256 hash is stored; the plaintext is shown exactly once at creation
- Scope: Each token is scoped to exactly one corpus
- Worker accounts: Django users with unusable passwords -- cannot log in via web UI
- Ownership: Documents are owned by the corpus creator, not the worker account
- Revocation: Revoking a token immediately prevents authentication; existing queued uploads continue processing
Rate Limiting¶
Rate limits are set per token via rate_limit_per_minute (0 = unlimited). When exceeded, the API returns HTTP 429 with a Retry-After: 60 header. The check is best-effort; for strict enforcement, use a reverse proxy.
Configuration¶
| Setting | Default | Description |
|---|---|---|
MAX_WORKER_UPLOAD_SIZE_BYTES | 256 MiB | Max file size per upload |
MAX_WORKER_METADATA_SIZE_BYTES | 500 MiB | Max metadata JSON size |
WORKER_UPLOAD_BATCH_SIZE | 50 | Uploads claimed per batch processor run |
WORKER_UPLOAD_STALE_MINUTES | 15 | Minutes before stalled uploads reset to PENDING |
Managing Worker Accounts and Tokens¶
Worker accounts are created by superusers. Corpus access tokens can be created by superusers or corpus owners. Both are manageable through the UI (Admin Settings for accounts, Corpus Settings for tokens) and via GraphQL mutations.
For the complete setup walkthrough with examples, UI screenshots, and the full metadata schema reference, see the Worker Upload System Walkthrough.
Higher-Level Driver: The Remote Ingest Worker¶
You usually do not call this REST API by hand. The Remote Ingest Worker (scripts/remote_ingest) is a turnkey, resumable CLI + docker-compose bundle that runs the real Docling parser and embedder on your own hardware, optionally calculates extra metadata and annotations, and streams the finished documents to this endpoint -- so the upload metadata above is produced faithfully for you rather than assembled by hand. Also mints tokens with one command: python manage.py mint_worker_token.