ntrs-dl

Download reports and metadata from the NASA Technical Reports Server (NTRS) for offline archives.

For each citation, ntrs-dl saves:

• Each file NTRS lists for it (usually one PDF), under its NTRS name
• The plaintext NTRS extracted from each file, when there is one
• NTRS's own citation record (ntrs.json), verbatim
• Four sidecar metadata files: metadata.md, metadata.yaml, metadata.json, metadata.bib

Only citations NTRS marks as public (distribution: PUBLIC) with files to download are archived.

NASA's terms: read before using

Using the NTRS API means accepting NASA's STI OpenAPI terms. In particular:

When a document or record is redistributed and becomes unavailable, it is expected that any copies be removed and the data be resynced.

NASA expects you to check the Redistributions endpoint (GET https://ntrs.nasa.gov/api/citations/redistributions) weekly at minimum, and to remove local copies of documents that become unavailable. ntrs-dl does not do this. It builds an offline archive and never deletes anything. Keeping your archive in line with NASA's expectation is up to you.

NASA also asks harvesters to cite NASA STI Compliance and Distribution Services as the source, and to share the citation URL (https://ntrs.nasa.gov/citations/<id>) rather than copies where possible. ntrs-dl writes both into every metadata.md.

The API allows 500 requests per 15 minutes. ntrs-dl's default of one request every 3 seconds stays under that.

Installation

gem install ntrs-dl

CLI usage

ntrs-dl <NTRS_ID_OR_URL> [<NTRS_ID_OR_URL>...]

Accepted input forms:

Form	Example
NTRS ID	19700020471
Citation URL	https://ntrs.nasa.gov/citations/19700020471
API URL	https://ntrs.nasa.gov/api/citations/19700020471
Download URL	https://ntrs.nasa.gov/api/citations/19700020471/downloads/19700020471.pdf

Flags

Flag	Description
-i FILE, --input FILE	Read IDs/URLs from FILE, one per line (- for stdin, blanks and # skipped)
-p PATH, --path PATH	Root download directory
--rate-limit SECONDS	Seconds between HTTP requests (0 disables throttling)
-v, --verbose	Print step lines and per-request URL/byte logs to stdout
-q, --quiet	Print nothing to stdout. Errors still go to stderr.
--version	Print the gem version and exit
-h, --help	Print help and exit

-v and -q are mutually exclusive.

Environment variables

Variable	Effect
NTRS_DOWNLOAD_PATH	Root download directory (default: $HOME/Downloads/NTRS_Papers)
NTRS_RATE_LIMIT	Seconds between HTTP requests (default: 3, 0 disables)

Precedence: CLI flag, then ENV var, then default.

Errors and exit status

A target that fails (unrecognized ID, no such citation, no public files, HTTP error, network failure) is reported on stderr as <target>: <message>, and the remaining targets still download. Exit status is 0 when every target succeeds and 1 when any fails.

Output layout

$NTRS_DOWNLOAD_PATH/                    # default: $HOME/Downloads/NTRS_Papers
  YYYY/MM/DD/<subject>/<ntrs-id>-<slug>/
    <ntrs-id>.pdf                       # each file NTRS lists, under its NTRS name
    <ntrs-id>.txt                       # NTRS's plaintext of it
    ntrs.json                           # NTRS's citation record, verbatim
    metadata.md                         # YAML frontmatter + Markdown body
    metadata.yaml
    metadata.json
    metadata.bib                        # synthesized @techreport

YYYY/MM/DD is the publication date, or the submission date when there is none. <subject> is the first NTRS subject category as a slug (space-sciences). <slug> is derived from the title.

Each citation downloads into a sibling .partial folder and is renamed into place only when every file succeeded. Re-running skips citations already archived.

Library usage

require 'ntrs/downloader'

identifier = NTRS::Downloader::Identifier.new 'https://ntrs.nasa.gov/citations/19700020471'
client     = NTRS::Downloader::Client.new                 # 3-second rate limit by default
path       = NTRS::Downloader::Archive.new(identifier, root: '/tmp/papers', client: client).run

Development

script/setup    # install dependencies
script/test     # run specs and rubocop
script/console  # interactive prompt

Specs run offline against recorded fixtures in spec/fixtures/http/. The PDF fixture is the first 4 KB of the real file. To check the citation fixtures against the live NTRS API, run:

NTRS_LIVE=1 script/test

License

MIT. See LICENSE.md.

Code of Conduct

This project follows the Contributor Covenant 3.0. See CODE_OF_CONDUCT.md.