Skip to content
openworkloadPublic

About

Sky Port makes it easy to run user workload in cloud

Resources

Code of conduct

Stars

11 stars

Watchers

2 watching

Forks

Repository files navigation

Sky Port

Introduction

Sky Port makes it easy to consume cloud resources by user software.

This repository contains sources of the workload manager (core component of the Sky Port project). It supports a full workload lifecycle in cloud:

  • job submission,
  • job scheduling,
  • cloud cluster creation,
  • ports forwarding,
  • job data uploading,
  • job container pulling and starting
  • job monitoring,
  • job results and logs downloading,
  • the cloud resources removal.

Design

Terminology:

  • Remote Site: remote computing system where user jobs run. Currently it is Microsoft Azure.
  • swm (Sky Port Workload Manager): a core service (daemon) of Sky Port that orchestrate the workload.
  • Terminal: any user software that connects to swm-core, submits jobs, retrieves current jobs status, displays the information. From user prospective the terminal is an interface to remote sites where jobs run. It uses swm-core as a transportanion layer to run its workload.
  • Gate: a service that is used by swm-core, and which is in charge of all communications with one or several remote site.

Sky Port consists of 3 main components:

See openworkload.org for details.

Documentation

How-to guides in this repository:

Guide Description
HOWTO/INSTALL.md Install Sky Port for production and development
HOWTO/BUILD.md Build Sky Port from source
HOWTO/SECURITY.md CA, mTLS, SSH/SFTP, and cluster trust layout
HOWTO/JOBS.md Write job scripts and #SWM directives
HOWTO/CONTAINERS.md Run jobs in rootless Podman containers
HOWTO/ACCOUNTING.md Job resource metrics and Prometheus export
HOWTO/CHECKPOINTS.md Checkpoint and restart MPI jobs with DMTCP/MANA

How to run

Pull skyport container

podman pull openworkload/skyport:latest

Start skyport container (as a regular user)

make start-release-container

This command starts a Podman pod skyport-pod with two containers: skyport (swm-core under supervisord) and skyport-gate (cloud gate under supervisord). If spool directory ($HOME/.swm/spool/) is missing or empty, the core container runs the prompt script that asks a few questions, bootstraps certificates and configuration, then stops. If spool is already ready, core starts swm-core and the gate container runs the gate.

If the spool is created and the core container is stopped then the user needs to ensure that Azure cloud provider is configured correctly, see AZURE.md. When the configuration is completed, then start the pod again:

podman pod start skyport-pod
# or: make start-release-container

This starts swm-core and gate in background if the spool is (still) ready.

If you prefer to build the Sky Port container image from scratch, see HOWTO/BUILD.md.

Run from sources (development)

For day-to-day development, use the debug container and run swm-core from the repository checkout. See HOWTO/INSTALL.md for the full procedure.

  1. Build the development container image (once) and start a shell in it:
make build-debug-container
make cr
  1. Ensure /opt/swm exists and is owned by your user (required by scripts/swm.env; the host /opt directory is mounted into the container):
sudo mkdir -p /opt/swm
sudo chown $USER:$USER /opt/swm
  1. From the swm-core directory inside the container, build and bootstrap:
make gen
make format
make compile porter
make release
./scripts/setup-skyport-dev.sh

make release is required before the first ./scripts/setup-skyport-dev.sh run so the setup script can assemble the worker distribution archive.

  1. Run and verify:
make run-skyport                  # foreground
# or: scripts/run-in-shell.sh -x -b   # background

scripts/swm-ping localhost 10001  # expect: Pong: idle

How to build

Build skyport container

To build a new release container image run the following commands:

make
make release
make worker
make build-release-container

The release container image will include swm-core and swm-cloud-gate, and ready to run. But during a development it is useful to have a container where Sky Port components are used from sources. To build such a development container image run this command:

make build-debug-container

The development container can be started with this command:

make cr

Contributing

We appreciate all contributions. If you are planning to contribute back bug-fixes, please do so without any further discussion. If you plan to contribute new features, utility functions or extensions, please first open an issue and discuss the feature with us. Our code of conduct can be found here.

License

We use a shared copyright model that enables all contributors to maintain the copyright on their contributions.

This software is licensed under the BSD-3-Clause license. See the LICENSE file for details.

About

Sky Port makes it easy to run user workload in cloud

Resources

Code of conduct

Stars

11 stars

Watchers

2 watching

Forks

Used by

Contributors

Languages