Repository navigation
fix: site name/path split, schema text, module glob - #29
Merged
Merged
Conversation
A run exported the site name as EB_PIPELINE_SITE, which is also the variable that --site falls back to. An eb-pipeline that a build, a route or a cache command started without --site read the name as a path, found no such file and exited 2. The commands of a run now get the site file in EB_PIPELINE_SITE, made absolute so that it names the same file from any directory, and the name in EB_PIPELINE_SITE_NAME. With EB_PIPELINE_TARGET, EB_PIPELINE_BUILD_TYPE, EB_PIPELINE_RUN_ID and EB_PIPELINE_LOG_STAMP, which already meant to the commands what they mean to the command line, an eb-pipeline call inside a run sees the settings of the run. The name cannot stay in a variable that must hold a path, so the fragment names the break: a command that read the name from EB_PIPELINE_SITE reads EB_PIPELINE_SITE_NAME. Nothing in the templates, examples and tests of this repository, nor in the sibling tools, read it. resolve::explain takes the site file, which Scope now carries, so that explain shows the environment that a run sets.
The schema text of [targets.cpu_hardware] said that every command exports a hardware class, while only the source prefetch and build commands of a line do: hardware_env is called from prefetch_cmd and build_cmd alone. The code is right for site operations, and the text now says what it does. Those two commands work for one unit on nodes of one class, so a hook or wrapper that chooses by CPU, such as an override of CPU detection for prerequisites built on other nodes, needs the class there. The shard, admit, keep-failed, cache and rotate commands work for a whole list, a reservation, a log or a target, and no class is theirs to carry. The text of [targets.hardware] also says that it is exported only when cpu_hardware is set, which the site file reference now says too.
The schema text of targets.route.shares_ro named {phase_dir}, {log_dir}
and {work_dir}, and shares_rw pointed to it, while check accepts
{target} and {phase} in a share as well and a run replaces them:
validate_route checks the shares against ROUTE_BASIC, and render_basic,
which renders the shares as it renders the route's args, replaces all
five.
{target} and {phase} are not accidental. They are the placeholders of
args that make sense in a path, and a share such as a per-target
source tree or a per-phase scratch directory needs them. Narrowing the
code would refuse site files that check accepts today, so the text
now lists all five.
A unit test reads the placeholders that the texts of args and
shares_ro name and compares them with ROUTE_ALL and ROUTE_BASIC, so
the text and check cannot drift apart again.
The wrapper loops over the unquoted ${EB_PIPELINE_MODULES:-} to split
it at spaces, and the shell expanded each word as a pathname pattern
as well: a name with a *, ? or [ that matched files, in the directory
the command started in or under an absolute path, reached module load
as the names of those files.
Quoting the expansion would stop the split too, so the loop now runs
with set -f, which turns pathname expansion off in any POSIX shell.
The word list of a for loop is expanded once, before the first pass,
so set +f at the top of the body puts expansion back for module load
itself, whose module function may eval code that relies on it, and the
set +f after the loop does the same when the list is empty.
The test runs a build through the wrapper in bash and in sh with an
init name that matches a file, and a module function that records its
arguments. The wrapper already needed a POSIX shell, so the schema text
and the reference now say that of modules.shell, and the reference
says that each name is loaded as written.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A run exported the site name as
EB_PIPELINE_SITE, which--sitealso falls back to, so aneb-pipelinestarted from a build, route or cache command without--siteread the name as a path and exited 2. The commands of a run now get the site file, as an absolute path, inEB_PIPELINE_SITE, and the name inEB_PIPELINE_SITE_NAME. A command that read the name fromEB_PIPELINE_SITEmust readEB_PIPELINE_SITE_NAME.The schema text of
[targets.cpu_hardware]now says that only the source prefetch and build commands of a line get the hardware class, as the code always did. The schema text oftargets.route.shares_roandshares_rwnow lists all five placeholders thatcheckaccepts and a run replaces:{phase_dir},{log_dir},{work_dir},{target}and{phase}.The module wrapper now runs its split of
EB_PIPELINE_MODULESunderset -f, so each name is loaded as written, including a name with*,?or[.Three findings are left as follow-ups:
*in Org verbatim, such as=*.log=on the log-rotation explanation page, is written as\*.login the built site. A filter indocs/export.elwould also fix eessi-arch-guard.checkaccepts anymodules.shell. The wrapper needs a POSIX shell; zsh leaves an unquoted list unsplit, and fish and csh cannot run the script.modules.initentry with white space is loaded as two names.checkcould refuse such an entry.