Allow LAVA jobs to pass artifacts back to the runner

[tintou]

This is a rebase of #116 which is a rework of #17

We'd like to potentially add artifacts created by LAVA jobs to the archive stored by Gitlab. To achieve this, we run a cut-down web server on the lava-gitlab-runner that is able to respond to POST requests at

   http://<host>:<port>/artifacts/<key>/<path>

The LAVA job is able to know the upload URL because we introduce a new namespace for templated variables runner in the lava-gitlab-runner and add ARTIFACT_UPLOAD_URL to it.

It's still relatively complicated to get variables into LAVA tests; the pattern I used in my test repository

https://gitlab.collabora.com/tintou/callback-tests

is to create a parameter called CALLBACK_URL in the test itself, and then in the job we can use a stanza like:

   test: foo
     parameters:
       CALLBACK_URL: {{ '{{ runner.ARTIFACT_UPLOAD_URL }}' }}

to make it available to the test. Bear in mind, if you use this that LAVA does not automatically export parameters from environment variables so you will need to export it inside your steps: in your test if you want to use it in scripts.

The key part of the upload URL is a long random string. It's generated uniquely per Gitlab job, not per LAVA job, although this detail is not important unless you are performing multi-node tests.

Because of the dynamic nature of the key, and the runner's port and IP, artifact upload is only possible for submit jobs. For monitor jobs, there's simply no way to communicate the necessary URL to them.

Backing the webserver, there is a shared UploadServer that stores the uploaded artifacts, and bridges between the web server thread and the job thread. It stores a JobArtifacts for each active job, which the ArtifactStore can query when we come to upload files. Uploaded artifacts are placed in the archive under exactly the relative <path> given in the upload URL (not automatically prefixed with the job ID, unlike the log/JUnit uploads), alongside the log/JUnit output, once the LAVA job finishes. Note however that it will require some significant reworking to support distinct directories for multi-node jobs. That's because we do not know how many nodes there are in the job until after we submit, at which point it's too late to create new keys for the other jobs. We could speculatively create a surplus, for example, but we couldn't then tie them to job IDs anyway.

Uploads are disabled unless LAVA_ARTIFACT_UPLOAD_BASE_URL is set on the runner; this is the externally reachable base URL of the upload endpoint (e.g. https://my-runner-host:8443/artifacts), for example for an appropriate reverse proxy. Optionally, LAVA_ARTIFACT_UPLOAD_LISTEN_ADDR can be set to control the local address the upload server binds to (e.g. 0.0.0.0:8443); it defaults to 0.0.0.0:0 (a random port). Uploads are capped at 1 GB per job.

[tintou]

I did test that the artifacts are being published with https://gitlab.collabora.com/tintou/callback-tests
This job has a test_data.txt that is an artifact from the job directly.

[Copilot]

Pull request overview

Adds an HTTP upload service so submitted LAVA jobs can return files for inclusion in GitLab artifact archives.

Changes:

• Adds per-job upload URLs and artifact storage.
• Integrates uploaded files into runner artifacts.
• Documents configuration and adds axum dependencies.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/upload.rs	Implements upload storage and quotas.
src/main.rs	Integrates server, templates, and artifacts.
README.md	Documents artifact uploads.
Cargo.toml	Adds axum and Tokio features.
Cargo.lock	Locks new dependencies.

Suppressed comments (1)

README.md:137

• curl -T sends a PUT request, while the router only registers POST, so this documented example receives 405 Method Not Allowed. Use a POST-producing curl option (or explicitly support PUT on the route).

              - curl -T my-large-log.txt "{{ runner.ARTIFACT_UPLOAD_URL }}my-large-log.txt"

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[sjoerdsimons]

Can you also test this documention in your test repo :) (basically for inline jobs)

[sjoerdsimons]

Can you make your test repo public? No reason for it to be private :)

I'm confused by this comment: "Also the upload URL will always be https, even though the service itself does not have TLS support" ; From a quick look i don't see https being enforced and really it shouldn't be. If i have a purely local network setup i should have the freedom to use http instead.

For documentation going via the secrets mechanism makes a lot of sense even if lava secrets aren't as secretive as they should be.

[sjoerdsimons]

Also your MR talks about LAVA_GITLAB_RUNNER_ROUTABLE_HOST but i don't see that in the code? Fwiw you indeed don't want a host and port but a base url which it looks like the code got right, so this is more about ensure your MR descriptoin actually matches your code