A comprehensive toolkit for transforming OpenAPI specifications into StackQL providers. This library streamlines the process of parsing, mapping, validating, testing, and generating documentation for StackQL providers.
- Prerequisites
- Installation
- Directory Structure
- Provider Development Workflow
providerdev.split- Divide a large OpenAPI specification into smaller, service-specific filesproviderdev.normalize- Reshape split specs for relational consumption (flatten allOf, rename oneOf/anyOf, strip misplaced keywords, lower opaque objects)providerdev.analyze- Examine split API specifications to generate mapping recommendationsproviderdev.generate- Create StackQL provider extensions from specifications and mappingsdocgen.generateDocs- Generate comprehensive documentation for StackQL providers
- Contributing
- License
- Support
- Node.js >= 20
npmoryarnstackqlfor testing
Add @stackql/provider-utils to your package.json:
npm install @stackql/provider-utils
# or
yarn add @stackql/provider-utilsThe package ships two CLI entry points: provider-dev-utils (for split / normalize / analyze / generate) and docgen-utils (for generate-docs / generate-docs-v2). Wrap them as npm scripts in your provider repo's package.json so every step of the workflow is a single npm run <step> invocation:
{
"scripts": {
"split": "node node_modules/@stackql/provider-utils/bin/provider-dev-utils.mjs split",
"normalize": "node node_modules/@stackql/provider-utils/bin/provider-dev-utils.mjs normalize",
"generate-mappings": "node node_modules/@stackql/provider-utils/bin/provider-dev-utils.mjs analyze",
"generate-provider": "node node_modules/@stackql/provider-utils/bin/provider-dev-utils.mjs generate",
"generate-docs": "node node_modules/@stackql/provider-utils/bin/docgen-utils.mjs generate-docs"
}
}Call the bins through node (rather than relying on node_modules/.bin/ shims) because npm does not always link .bin shims for .mjs files on Windows. Pass flags through with npm's -- separator, e.g.:
npm run normalize -- --api-dir provider-dev/source --verboseJSON-valued flags (--servers, --provider-config, --service-config, --skip-files, --svc-name-overrides) accept either an inline JSON string or a path to a JSON file.
npm run split -- \
--provider-name github \
--api-doc provider-dev/downloaded/api.github.com.json \
--svc-discriminator tag \
--output-dir provider-dev/source \
--overwrite
npm run normalize -- \
--api-dir provider-dev/source \
--verbose
npm run generate-mappings -- \
--input-dir provider-dev/source \
--output-dir provider-dev/config
npm run generate-provider -- \
--provider-name github \
--input-dir provider-dev/source \
--output-dir provider-dev/openapi/src/github \
--config-path provider-dev/config/all_services.csv \
--servers '[{"url": "https://api.github.com"}]' \
--provider-config '{"auth": {"type": "basic", "username_var": "STACKQL_GITHUB_USERNAME", "password_var": "STACKQL_GITHUB_PASSWORD"}}' \
--overwrite
npm run generate-docs -- \
--provider-name github \
--provider-dir ./provider-dev/openapi/src/github/v00.00.00000 \
--output-dir ./website \
--provider-data-dir ./provider-dev/docgen/provider-dataOn generate-provider, --provider-name is accepted as an alias for --provider-id so the flag name is consistent with the other commands.
A typical project structure for the development of a stackql provider would be...
.
├── bin # convinience scripts
│ ├── ...
├── provider-dev
│ ├── config
│ │ └── all_services.csv # mappings generated or updated by the `providerdev.analyze` function, used by `providerdev.generate`
│ ├── docgen
│ │ └── provider-data # provider metadata used by `docgen.generateDocs`
│ │ ├── headerContent1.txt
│ │ └── headerContent2.txt
│ ├── downloaded # used to store the original spec for the provider
│ │ └── management-minimal.yaml
│ ├── openapi # output from `providerdev.generate`, this is the stackql provider
│ │ └── src
│ │ └── okta
│ │ └── v00.00.00000
│ │ ├── provider.yaml
│ │ └── services
│ │ ├── agentpools.yaml
│ │ ├── ...
│ ├── scripts # optional scripts for pre or post processing if required
│ │ └── post_processing.sh
│ └── source # output from `providerdev.split` if used, this is the source used with the mappings to generate the provider
│ ├── agentpools.yaml
│ ├── ...
└── website # docusaurus site
├── docs # output from `docgen.generateDocs`
│ ├── ...see stackql-provider-okta for a working example.
The library provides a streamlined workflow for creating StackQL providers from OpenAPI specifications:
providerdev.split- Divide a large OpenAPI specification into smaller, service-specific filesproviderdev.normalize- Reshape split specs for relational consumption (flatten allOf, rename oneOf/anyOf, strip misplaced keywords, lower opaque objects)providerdev.analyze- Examine split API specifications to generate mapping recommendationsproviderdev.generate- Create StackQL provider extensions from specifications and mappingsdocgen.generateDocs- Generate comprehensive documentation for StackQL providers
Contributions are welcome!
Unit tests are written with Jest and live in the tests/ directory. This package uses native ES modules, so the test script runs Jest under node --experimental-vm-modules.
npm ci
npm testTo run a single test suite:
npm test -- tests/providerdev/normalize.test.jsTests run automatically in CI via GitHub Actions (.github/workflows/test.yml) on pushes and pull requests to main. Publishing to npm is deliberately not automated - it is a manual step performed by a maintainer as it requires 2FA.
MIT