Sky Port makes it easy to consume cloud resources by user software.
This repository contains sources of the workload manager (core component of the Sky Port project). It supports a full workload lifecycle in cloud:
- job submission,
- job scheduling,
- cloud cluster creation,
- ports forwarding,
- job data uploading,
- job container pulling and starting
- job monitoring,
- job results and logs downloading,
- the cloud resources removal.
Terminology:
- Remote Site: remote computing system where user jobs run. Currently it is Microsoft Azure.
- swm (Sky Port Workload Manager): a core service (daemon) of Sky Port that orchestrate the workload.
- Terminal: any user software that connects to swm-core, submits jobs, retrieves current jobs status, displays the information. From user prospective the terminal is an interface to remote sites where jobs run. It uses swm-core as a transportanion layer to run its workload.
- Gate: a service that is used by swm-core, and which is in charge of all communications with one or several remote site.
Sky Port consists of 3 main components:
- Core (this repository).
- Gate. See the default cloud gate as a gate example.
- Terminal. See the JupyterLab terminal as a terminal example.
See openworkload.org for details.
How-to guides in this repository:
| Guide | Description |
|---|---|
| HOWTO/INSTALL.md | Install Sky Port for production and development |
| HOWTO/BUILD.md | Build Sky Port from source |
| HOWTO/SECURITY.md | CA, mTLS, SSH/SFTP, and cluster trust layout |
| HOWTO/JOBS.md | Write job scripts and #SWM directives |
| HOWTO/CONTAINERS.md | Run jobs in rootless Podman containers |
| HOWTO/ACCOUNTING.md | Job resource metrics and Prometheus export |
| HOWTO/CHECKPOINTS.md | Checkpoint and restart MPI jobs with DMTCP/MANA |
podman pull openworkload/skyport:latestmake start-release-containerThis command starts a Podman pod skyport-pod with two containers:
skyport (swm-core under supervisord) and skyport-gate (cloud gate under
supervisord). If spool directory ($HOME/.swm/spool/) is missing or empty, the
core container runs the prompt script that asks a few questions, bootstraps
certificates and configuration, then stops. If spool is already ready, core
starts swm-core and the gate container runs the gate.
If the spool is created and the core container is stopped then the user needs to ensure that Azure cloud provider is configured correctly, see AZURE.md. When the configuration is completed, then start the pod again:
podman pod start skyport-pod
# or: make start-release-containerThis starts swm-core and gate in background if the spool is (still) ready.
If you prefer to build the Sky Port container image from scratch, see HOWTO/BUILD.md.
For day-to-day development, use the debug container and run swm-core from the repository checkout. See HOWTO/INSTALL.md for the full procedure.
- Build the development container image (once) and start a shell in it:
make build-debug-container
make cr- Ensure
/opt/swmexists and is owned by your user (required byscripts/swm.env; the host/optdirectory is mounted into the container):
sudo mkdir -p /opt/swm
sudo chown $USER:$USER /opt/swm- From the swm-core directory inside the container, build and bootstrap:
make gen
make format
make compile porter
make release
./scripts/setup-skyport-dev.shmake release is required before the first ./scripts/setup-skyport-dev.sh
run so the setup script can assemble the worker distribution archive.
- Run and verify:
make run-skyport # foreground
# or: scripts/run-in-shell.sh -x -b # background
scripts/swm-ping localhost 10001 # expect: Pong: idleTo build a new release container image run the following commands:
make
make release
make worker
make build-release-containerThe release container image will include swm-core and swm-cloud-gate, and ready to run. But during a development it is useful to have a container where Sky Port components are used from sources. To build such a development container image run this command:
make build-debug-containerThe development container can be started with this command:
make crWe appreciate all contributions. If you are planning to contribute back bug-fixes, please do so without any further discussion. If you plan to contribute new features, utility functions or extensions, please first open an issue and discuss the feature with us. Our code of conduct can be found here.
We use a shared copyright model that enables all contributors to maintain the copyright on their contributions.
This software is licensed under the BSD-3-Clause license. See the LICENSE file for details.