Skip to content

feat(file-downloads): add possibility to offload to EOS - #2193

Merged
zzacharo merged 2 commits into
CERNDocumentServer:mainfrom
zzacharo:eos-offload
Oct 29, 2026
Merged

zzacharo merged 2 commits into
CERNDocumentServer:mainfrom
zzacharo:eos-offload

Conversation

@zzacharo

Copy link
Copy Markdown
Contributor

No description provided.

@zzacharo
zzacharo force-pushed the eos-offload branch 4 times, most recently from 3d84f42 to 2075694 Compare October 29, 2026 09:47
@zzacharo
zzacharo merged commit 5b3084b into CERNDocumentServer:main Oct 29, 2026
1 check passed
Comment thread cds/config.py
CDS_EOS_OFFLOAD_REDIRECT_BASE_PATH = "/eos_storage"
"""Internal nginx location prefix used for the ``X-Accel-Redirect``."""

CDS_EOS_OFFLOAD_ENDPOINTS = ["invenio_files_rest.object_api"]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are there any other endpoints that might be used for downloads?

silly question because I didn't investigate in depth - what about streaming connections, are they also causing issues?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

streaming connections on Videos are handled by the wowza streaming server so they are completely independent. For now, the direct file downloads are the ones causing the problem.

Comment thread cds/config.py
CDS_EOS_OFFLOAD_AUTH_X509 = True
"""Authenticate to the EOS gateway with X.509 (otherwise Kerberos)."""

CDS_EOS_OFFLOAD_X509_CERT_PATH = "/etc/certificates/cert.pem"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would leave this empty here

Comment thread cds/config.py
# served by a separate streaming server and is unaffected.
#
# The single on/off toggle is ``CDS_EOS_OFFLOAD_ENABLED``.
FILES_REST_XSENDFILE_ENABLED = False

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this variable True in files rest by default? if not I would maybe not expose it here since it opens the door for modifications, unless you leave it here because you want to state clearly that it should not be changed.

Comment thread cds/config.py
CDS_EOS_OFFLOAD_ENABLED = False
"""Enable offloading of ``/api/files`` downloads to EOS (the single toggle)."""

CDS_EOS_OFFLOAD_HTTPHOST = "https://eosmedia.cern.ch:8444"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would leave this empty here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants