datalab-labcup-plugin

datalab-labcup-plugin imports LabCup Excel exports into DataLab starting_materials entries.

It is designed as a standalone command-line tool that can be run manually, scheduled, or triggered by datalab_scraper through a generic command hook.

Quick Start

datalab-labcup-import validate Chemicals_all.xlsx
datalab-labcup-import import Chemicals_all.xlsx --dry-run
datalab-labcup-import import Chemicals_all.xlsx --dry-run --report-json dry-run-report.json
datalab-labcup-import import Chemicals_all.xlsx --state-file .labcup-state.json --archive-dir archive

Connection settings are read from environment variables:

• DATALAB_API_URL
• DATALAB_API_KEY
• DATALAB_VERIFY_SSL (true by default; set false for trusted self-signed deployments)

If a .env file exists in the current working directory, these values are loaded automatically before CLI argument defaults are resolved. Existing process environment variables still take precedence over .env values.

Behavior

• Uses LabCup Barcode as the DataLab item_id.
• Applies DataLab-compatible item_id normalization when needed:
  • trims leading/trailing punctuation,
  • normalizes invalid characters,
  • deterministically shortens IDs longer than 40 characters with a hash suffix.
• Skips rows with missing barcodes and reports them.
• Creates missing starting_materials entries.
• Updates existing entries only when configured update fields differ.
• Supports --dry-run for safe review.
• Persists import state by file hash when --state-file is supplied.
• Blocks older exports by default after a newer successful import.
• Optionally archives successfully imported source files.

Output and Reports

Import commands print progress to stderr while the run is active, then print a compact summary to stdout when complete.

Use --report-json PATH to save the full machine-readable report without flooding the terminal. Use --details-json if you explicitly want the full JSON report printed to stdout.

Reports include a transformed_item_ids section listing any rows where LabCup Barcode was transformed before upload.

Use --quiet to suppress progress messages while keeping the final summary/report behavior.

Docker

Build the single-shot importer image:

docker build -t datalab-labcup-plugin .

Run a dry run against a mounted export directory:

docker run --rm \
  --env DATALAB_API_URL="https://sce-chem-c01894.chem.ed.ac.uk/api" \
  --env DATALAB_API_KEY \
  --env DATALAB_VERIFY_SSL="false" \
  -v "/path/to/labcup_exports:/data" \
  datalab-labcup-plugin \
  import /data/Chemicals_all_03_08_2026.xlsx \
  --dry-run \
  --state-file /data/.labcup-state.json \
  --lock-file /data/.labcup-state.json.lock \
  --report-json /data/reports/Chemicals_all_03_08_2026.report.json

Use a host-mounted directory for source exports, reports, state, locks, and archives. The container is intended to import one file and exit.

DataLab Scraper Integration

Example datalab_scraper command target:

watch_targets:
  - directory: "/watch/labcup"
    glob: "Chemicals*.xlsx"
    type: "labcup"
    command: "datalab-labcup-import"
    command_args:
      - "import"
      - "{file_path}"
      - "--state-file"
      - "/watch/labcup/.labcup-state.json"
      - "--lock-file"
      - "/watch/labcup/.labcup-state.json.lock"
      - "--archive-dir"
      - "/watch/labcup/archive"
      - "--report-json"
      - "/watch/labcup/reports/{file_stem}.report.json"
    command_mode: "background"
    max_concurrent_commands: 1