feat: end-to-end TFC migration pipeline with guided init and import - #16
Draft
hllvc wants to merge 30 commits into
Draft
feat: end-to-end TFC migration pipeline with guided init and import#16hllvc wants to merge 30 commits into
hllvc wants to merge 30 commits into
Conversation
- pull workspace state from the TFC API instead of per-workspace init
- write migration-summary.{md,json}; replace workspace.tmpl with summary.tmpl
- map TFC vcs settings to SG VCSTriggers (push, PR speculative, file filters)
- add SGDefault*/workspaceOverrides for vcs auth, source kind and triggers
Validate generated payloads against schema/sg-payload.schema.json via yajsv.
- sg-migrate.sh + Dockerfile run the full pipeline inside a container - scripts/migrate.sh: init/apply/enrich/convert/validate/import/triggers/all - merge TFC variable sets, parallel convert/import, register VCS triggers - move convert_hcl_to_json.sh into scripts/
Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
sg-cli exits 0 even when individual workflows fail, so the importer now parses its output. Workflows rejected because their pinned Terraform version is above SG's managed ceiling (1.5.7, last MPL/FOSS release) are re-imported with SGDefaultTerraformVersion, the payload is patched in place, and each case is logged to export/terraform-version-fallbacks.log with a printed notice. Any other per-workflow failure now fails the run. Also: sg_retry captured the wrong exit status (always 0), definitive HTTP 4xx responses are no longer retried, the trigger pass skips workflows that were never created, and failure output no longer echoes the API token. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
'sg-migrate.sh completion bash|zsh' prints a completion script for the current shell session; 'init' prints the source line for the user's shell. Runs natively like 'clean'. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
Running sg-migrate.sh without a command used to start the whole pipeline; it now prints the help menu, locally, without touching Docker. Usage and hints show the name the user invoked (./sg-migrate.sh) instead of the in-container script path. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
Without TFE_TOKEN or a terraform login credentials file, apply used to warn and then fail deep inside terraform with "Invalid provider configuration". Both the host entrypoint and migrate.sh now stop with a clear message before anything runs. Add an explicit provider "tfe" block so tfHostname also applies to the provider, not just the state export. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
An expired 'terraform login' session (or a bad TFE_TOKEN) used to fail deep inside terraform. Resolve the token the tfe provider will use, in its own precedence, and check it against /api/v2/account/details first; stop with a clear message on 401/403 or when the host is unreachable. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
Every workflow was hardcoded to shared runners unless overridden per workspace. Add SGDefaultRunnerConstraints (default shared) so all workflows can be put behind a private runner group in one place; workspaceOverrides[name].RunnerConstraints still wins per workspace. Validated: type is shared|private, and private requires names. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
Move the TFC/SG API helpers out of migrate.sh into scripts/lib/ and add the building blocks the upcoming init wizard, preflight and checklist need: interactive prompt helpers (tty or scripted answers), cached tfvars reads, paginated TFC listing (orgs/projects/workspaces) and SG lookups for integrations, runner groups, workflows and secrets. The enrich script now shares the same token resolution as the provider. Also stop turning curl connection failures into "000000" status codes. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
'init' now walks through the configuration instead of copying the example file: it lists the TFC organisations, projects and workspaces the token can see, the SG VCS and cloud connectors and runner groups, and derives the source kind and repo prefix from the chosen connector. Every step falls back to free text when a token is missing or an API call fails, and non-interactive runs keep the template behaviour. The generated terraform.tfvars keeps the example's order and comments; re-running the wizard prefills the current values and keeps a .bak. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
Verify up front what used to fail minutes later inside terraform or as API 400s: TFC token and org, that the workspace selection matches anything, SG token and org, every connector / secret / runner group referenced in terraform.tfvars (defaults and workspaceOverrides), SGDefaultSourceConfigDestKind and SGDefaultTerraformVersion format, the exportPath vs export-dir mismatch, and override keys that match no workspace. Runs automatically for apply, import and all; also available as 'preflight'; --skip-preflight bypasses it. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
'all' now records each phase's inputs in .sg/state.json and skips phases whose inputs have not changed, so a failed run resumes instead of re-running terraform apply. Import records per payload file which workflows landed and which failed; unchanged, fully imported files are skipped and files with failures are re-imported (sg-cli updates existing workflows in place). --fresh discards the state. --project and --workspace restrict every phase to a subset; apply maps --workspace to the workspacenames variable. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
apply now prints a condensed migration summary (workflows per project, sensitive variables skipped, unpinned Terraform versions, non-remote execution modes, renames, failed state exports) instead of leaving it in a file nobody opens. import shows a per-workflow table before the confirmation prompt: create/update, Terraform version with the 1.5.7 fallback marked, runner, triggers, variable and secret counts. 'import --dry-run' stops after the table without touching anything. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
Map the API messages users actually hit (unknown connector, missing runner group, repository not reachable, bad approvers, invalid names or VCS kind, rejected token, Terraform ceiling) to a one-line hint naming the terraform.tfvars field to fix. Shown once per distinct hint, both for direct API calls and for sg-cli's per-workflow failures; unknown messages still show the raw response. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
After import (or via 'checklist'), write export/post-import-checklist.md
listing everything the migration could not do by itself: secrets to
fill in, workflows that failed to import, Terraform version fallbacks
to verify, VCS triggers that failed, state that could not be exported,
non-remote execution modes and renamed workflows, with deep links into
the SG UI (SG_UI_URL). For every sensitive variable TFC would not
expose, create an SG secret with the value CHANGE_ME, reference it from
the workflow (env var or IaC input, ${secret::<name>}) and patch the
payload to match; --no-secret-stubs opts out. Trigger registration
results are now recorded in the run state.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
The integrations list has no ResourceType; the kind (GITHUB_COM, AWS_RBAC, ...) is Settings.kind. Map it from there, make the type filters null-safe, and only treat msg/data as the item list when it is actually an array. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
_w_csv_json emitted "[]" twice when the list was empty (grep -v on empty input trips the fallback under pipefail), so the generated file was not valid HCL and preflight reported every field as missing. Validate the file right after writing it, report an unparsable tfvars as such in preflight, remember the SG org / API host chosen in the wizard so a new shell does not need SG_ORG, pretty-print objects in the generated file and separate menu values from their descriptions. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
The hint used $SHELL, which is the login shell and was wrong for a bash login shell running zsh, so the bash script got sourced into zsh and broke on the first Tab. sg-migrate.sh now detects the parent shell and 'completion' with no argument prints the matching script; each script also hands off to the other shell's version when sourced by mistake. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
The per-workflow plan passed whatever sg_list_workflows returned to jq --argjson and crashed when the response was not a plain array. List helpers now always yield an array (also unwrapping data.Workflows) and the plan guards the value. The wizard no longer asks for the profile name (not required), the repo URL prefix, approvers or the fallback Terraform version — they take sensible defaults and are edited in terraform.tfvars. sg-migrate.sh says when it runs in Docker. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
When terraform.tfvars is missing, all runs the wizard and then asks whether to continue right away or edit the file first; only the non-interactive template copy still stops, since it holds placeholders. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
Wrap argument parsing and dispatch in main so bash reads the whole script up front. The repo is bind-mounted into the container, and a file edited during a long run was read half-old, half-new, ending in "unexpected EOF while looking for matching quote" after the import had already succeeded. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
TFC_* / TFE_* variables (TFC_WORKSPACE_NAME, the TFC_AWS_* dynamic credential settings, ...) only mean something inside Terraform Cloud. A new ignoreVarPatterns input (default ["^TFC_", "^TFE_"]) drops matching terraform and env variables from the payloads, from workspaces and from variable sets, lists them in the migration summary and keeps them out of the sensitive-variable list so no secret stubs are created for them. The init wizard asks whether to strip them. Also fix the secrets link in the checklist (orgs/<org>?tab=secrets). Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
The "^(AWS|AZURE|GCP)_" filter also matched AZURE_DEVOPS, so an Azure DevOps VCS connector could be chosen as the cloud connector and only schema validation caught it. Both pickers now use the exact kind lists (VCS also accepts AZURE_DEVOPS_SP and GITLAB_OAUTH_SSH and maps them to the source kind), are sorted by name, and preflight rejects a DeploymentPlatformConfig kind outside the schema enum. The validator no longer prints each failure twice. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01YKBeQrpJgR9DVFtR83myqX
Presets: SGTerraformVersionSource (carry|preset) plus nullable SGDefaultTerraformVersion and SGDefaultRunnerConstraints. Keys left out of the payload are filled from the org's execution preset by the SG API at import. init offers the preset with the current value shown, preflight reports it, the plan marks 'preset (...)' cells, the checklist explains what changed, and a pin rejected above the 1.5.7 ceiling falls back to the preset when the fallback is null. Flow: numbered phases with timings, live progress lines for terraform and the parallel phases, plain-language preflight with connector/kind/prefix consistency checks, readable wizard review, per-file convert/validate lines, a condensed post-import checklist with group links, and a neutral message when the import is declined. Fixes: init kept the previous repo URL prefix when the connector switched provider (the prefix now comes from the TFC repositories); enrich re-ran on every 'all' because convert changed its recorded hash; quotes in connector names broke the generated tfvars; a RunnerConstraints override with names failed HCL type unification.
hllvc
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Turns the migrator from a hand-driven
terraform applyplus manual editing into one guided pipeline:./sg-migrate.sh allconfigures, preflights, extracts and transforms the workspaces, merges Variable Sets, converts and validates the payloads, and bulk-imports them into per-project workflow groups. The transformer gains per-project output, a migration summary, API-based state export, VCS trigger mapping and execution-preset support.Suggested review order:
transformer/terraform-cloud/locals.tf(the TFC to SG mapping, where payload shape is decided), thenscripts/migrate.sh(phase orchestration, import parsing, resume state), thenscripts/lib/(wizard, preflight, checklist). The rest is glue and docs.Changes
sg-payload.<project>.jsonper TFC project withWorkflowGroup.name = tfc-<project>;migration-summary.{md,json}; state pulled from the TFC API instead of a per-workspaceterraform init; TFC VCS settings mapped toVCSTriggers; sensitive variables skipped and reported;ignoreVarPatternsstripsTFC_*/TFE_*variables.SGDefault*inputs plus typedworkspaceOverrides.SGTerraformVersionSourceand nullableSGDefaultTerraformVersion/SGDefaultRunnerConstraintsleave keys out of the payload so the SG API fills them from the org's execution preset.sg-migrate.shrunsscripts/migrate.shin the Docker image (native fallback) withinit|preflight|apply|enrich|convert|validate|import|triggers|checklist|all|clean|completion. Phases resume via.sg/state.json, filter with--project/--workspace;import --dry-runstops at the plan.tfc-<project>groups, parses sg-cli output (it exits 0 on per-workflow failures), retries a pin above SG's 1.5.7 Terraform ceiling with the fallback or preset, maps known API errors to a hint naming the tfvars field, and writesexport/post-import-checklist.mdwithCHANGE_MEsecret stubs for sensitive variables.schema/sg-payload.schema.jsonwithscripts/validate_payload.sh; README rewritten for the orchestrated flow.Testing
./sg-migrate.sh allagainst a multi-project TFC org: wizard, preflight, one payload per project, groups created, workflows imported, checklist written.allafter a failed import: unchanged phases are skipped, only files with failures are re-imported.export/terraform-version-fallbacks.log; withSGDefaultTerraformVersion = nullthe preset applies.terraform fmt -checkon the transformer and./scripts/validate_payload.sh export/sg-payload.*.jsonpass.Risks
sg-payload.jsonbecame one file per project andWorkflowGroup.nameistfc-<project>rather than the TFC project ID.--no-create-groupsand--no-secret-stubsopt out.terraformVersionorRunnerConstraintsomitted depend on the org's execution preset being set as intended.Deployment Notes
SG_API_TOKEN,SG_ORG, optionalSG_BASE_URL, andTFE_TOKEN(preferred overterraform login); tuning knobs are in the README.--buildto rebuild); nothing is pushed to a registry.