diff --git a/CHANGES/2474.feature b/CHANGES/2474.feature new file mode 100644 index 000000000..6e644930d --- /dev/null +++ b/CHANGES/2474.feature @@ -0,0 +1 @@ +Optimize sync performance for repositories with specific tag lists by bypassing expensive /tags/list pagination when include_tags contains only non-wildcard references. Reduces sync time from minutes to seconds for deep repositories (50K+ tags) when syncing specific digests. Cosign companion tags are discovered via HEAD probing. diff --git a/ansible/OCP-PRODUCT-GUIDE.md b/ansible/OCP-PRODUCT-GUIDE.md new file mode 100644 index 000000000..3e26e22f1 --- /dev/null +++ b/ansible/OCP-PRODUCT-GUIDE.md @@ -0,0 +1,323 @@ +# OCP Container Product Management for Satellite + +This guide covers generating OCP container product definitions from OpenShift +release metadata and pushing them to a Satellite server. + +## Overview + +The workflow has two stages: + +1. **Generate** — fetch an OCP release.txt, parse container image digests, + and produce Satellite-ready configuration files +2. **Push** — create the product and repositories on Satellite using the + `redhat.satellite` collection + +These can run independently. Generate once, push to multiple Satellites. +Or generate and push in sequence for a single target. + +## Prerequisites + +- Ansible with the `redhat.satellite` collection installed +- An OpenShift pull secret from +- Quay.io credentials extracted into a vault file (see [PULL-SECRET-SETUP.md](PULL-SECRET-SETUP.md)) + +### Install the Satellite collection + +```bash +ansible-galaxy collection install redhat.satellite +``` + +Or use the provided requirements file: + +```bash +ansible-galaxy install -r requirements.yml +``` + +### One-time credential setup + +```bash +./extract-pull-secret.sh pull-secret.txt +``` + +This creates encrypted vault files in `group_vars/` with the `quay_registry_*` +and `redhat_registry_*` variables. + +## Vault Variables + +The playbooks rely on two sets of vaulted credentials stored in `group_vars/`. +These are created by the `extract-pull-secret.sh` script. + +### `group_vars/vault_quay_credentials.yml` + +```yaml +--- +quay_registry_username_vault: "" +quay_registry_password_vault: "" +``` + +Used as `upstream_username` and `upstream_password` for quay.io container +repositories (OCP release and component images). + +### `group_vars/vault_redhat_registry_credentials.yml` + +```yaml +--- +redhat_registry_username_vault: "" +redhat_registry_password_vault: "" +``` + +Used for Red Hat certified container image repositories. Not required for +OCP images (those come from quay.io) but available for other products. + +### Vault password + +Pass `--ask-vault-pass` on the command line, or configure a vault password +file in `ansible.cfg`: + +```ini +[defaults] +vault_password_file = ~/.vault_pass +``` + +## Inventory + +The `push-ocp-product.yml` playbook runs against `localhost` and connects +to Satellite via its API, so no SSH inventory is needed. The Satellite +connection details are passed as extra variables: + +| Variable | Example | Description | +|---|---|---| +| `satellite_url` | `https://satellite1.example.ca` | Satellite server URL | +| `satellite_username` | `admin` | Satellite admin user | +| `satellite_password` | `changeme` | Satellite admin password | +| `satellite_organization` | `Default Organization` | Target organization | + +Alternatively, these can be set in a variables file to avoid repeating them: + +### `group_vars/satellite_connection.yml` (example) + +```yaml +--- +satellite_url: "https://satellite1.example.ca" +satellite_username: "admin" +satellite_organization: "Default Organization" +satellite_validate_certs: true +``` + +Then pass only the password at runtime: + +```bash +ansible-playbook push-ocp-product.yml \ + -e ocp_minor=4.20 \ + -e satellite_password=changeme \ + --ask-vault-pass +``` + +For GSSAPI authentication (Kerberos), no username or password is needed: + +```bash +ansible-playbook push-ocp-product.yml \ + -e ocp_minor=4.20 \ + -e satellite_url=https://satellite1.example.ca \ + -e satellite_use_gssapi=true \ + --ask-vault-pass +``` + +The `--ask-vault-pass` is still needed to decrypt the quay.io credentials. + +## Stage 1: Generate Product Definition + +The `generate-ocp-product.yml` playbook fetches the OCP release.txt for a +given version, parses all container image digests, and produces three files. + +### Usage + +```bash +ansible-playbook generate-ocp-product.yml -e ocp_version=4.20.32 +``` + +For environments with non-blocking IO issues (e.g. running from Claude Code): + +```bash +script -qc "ansible-playbook generate-ocp-product.yml -e ocp_version=4.20.32" /dev/null +``` + +### Parameters + +| Variable | Required | Description | +|---|---|---| +| `ocp_version` | Yes | Full version in X.Y.Z format (e.g. `4.20.32`) | + +### Output Files + +All output goes to the `generated/` directory: + +| File | Purpose | +|---|---| +| `ocp-4.20-custom-product.yml` | Product entry to merge into `custom_products.yml` | +| `ocp_420_tags.yml` | Tag lists as YAML arrays (one digest per line) | +| `ocp-4.20-repositories.yml` | Repository entries to append to `repositories.yml` | +| `ocp-4.20-versions.json` | Version tracking data for z-stream accumulation | + +### Accumulating Z-stream Releases + +Run the playbook multiple times with different z-stream versions to accumulate +digests into a single product per OCP minor version: + +```bash +ansible-playbook generate-ocp-product.yml -e ocp_version=4.20.32 +ansible-playbook generate-ocp-product.yml -e ocp_version=4.20.33 +ansible-playbook generate-ocp-product.yml -e ocp_version=4.20.34 +``` + +Each run merges new digests with existing ones. The `versions.json` file +tracks which z-streams have been included. Use content views in Satellite +to create point-in-time snapshots for specific z-stream releases. + +### Using with rhis-builder-satellite + +To use the generated files with rhis-builder-satellite: + +1. Copy the tags file to the host_vars directory: + ```bash + cp generated/ocp_420_tags.yml \ + /host_vars// + ``` + +2. Merge the product entry from `ocp-4.20-custom-product.yml` into + `custom_products.yml` — the `include_tags` fields reference the tag + list variables via `join(',')`, keeping the file lint-friendly. + +3. Append the repository entries from `ocp-4.20-repositories.yml` to + `repositories.yml` to set download policy and mirroring policy. + +## Stage 2: Push to Satellite + +The `push-ocp-product.yml` playbook creates the product and repositories +directly on a Satellite server using the `redhat.satellite` collection. +This is the standalone alternative to running rhis-builder-satellite. + +### Usage + +Basic — create product and repositories: + +```bash +ansible-playbook push-ocp-product.yml \ + -e ocp_minor=4.20 \ + -e satellite_url=https://satellite1.example.ca \ + -e satellite_username=admin \ + -e satellite_password=changeme \ + --ask-vault-pass +``` + +Create and immediately trigger a sync: + +```bash +ansible-playbook push-ocp-product.yml \ + -e ocp_minor=4.20 \ + -e satellite_url=https://satellite1.example.ca \ + -e satellite_username=admin \ + -e satellite_password=changeme \ + -e sync_after_create=true \ + --ask-vault-pass +``` + +With GSSAPI authentication (no username/password needed): + +```bash +ansible-playbook push-ocp-product.yml \ + -e ocp_minor=4.20 \ + -e satellite_url=https://satellite1.example.ca \ + -e satellite_use_gssapi=true \ + --ask-vault-pass +``` + +### Parameters + +| Variable | Required | Default | Description | +|---|---|---|---| +| `ocp_minor` | Yes | — | OCP minor version (e.g. `4.20`) | +| `satellite_url` | Yes | — | Satellite server URL | +| `satellite_username` | * | — | Satellite admin username | +| `satellite_password` | * | — | Satellite admin password | +| `satellite_use_gssapi` | * | `false` | Use GSSAPI instead of username/password | +| `satellite_organization` | No | `Default Organization` | Satellite organization | +| `satellite_validate_certs` | No | `true` | Validate Satellite TLS certificate | +| `sync_after_create` | No | `false` | Trigger repository sync after creation | + +\* Either `satellite_username`/`satellite_password` or `satellite_use_gssapi=true` +is required. + +### What It Creates + +| Resource | Details | +|---|---| +| **Product** | `OpenShift Container Platform 4.20` | +| **Repository: ocp-release** | Release images from `quay.io/openshift-release-dev/ocp-release` | +| **Repository: ocp-v4.0-art-dev** | Component images from `quay.io/openshift-release-dev/ocp-v4.0-art-dev` | + +Both repositories are configured with: +- `download_policy: immediate` +- `mirroring_policy: additive` (required to preserve the bypass sync optimization) +- `include_tags` set to the sha256 digests from the generated tag file +- `exclude_tags: *-source` + +## End-to-End Example + +Generate the product definition for OCP 4.22.9 and push it to Satellite: + +```bash +# Step 1: Generate +ansible-playbook generate-ocp-product.yml -e ocp_version=4.22.9 + +# Step 2: Push and sync +ansible-playbook push-ocp-product.yml \ + -e ocp_minor=4.22 \ + -e satellite_url=https://satellite1.example.ca \ + -e satellite_username=admin \ + -e satellite_password=changeme \ + -e sync_after_create=true \ + --ask-vault-pass +``` + +## Important Notes + +### Mirroring Policy + +The repositories **must** use `additive` mirroring policy. Any mirror mode +(`mirror_content_only` or `mirror_complete`) will cause the sync bypass +optimization to be skipped, resulting in significantly longer sync times. + +### Pull Secret Expiration + +The OpenShift pull secret from console.redhat.com does not expire. +However, regenerating it on the console revokes the previous token. + +### Digest-Only Tags + +The `include_tags` values are sha256 digests, not human-readable tag names. +This is intentional — the bypass sync optimization only activates for +digest-based tags, providing a 249x speedup over tag-based syncs. + +## File Reference + +``` +ansible/ +├── generate-ocp-product.yml # Stage 1: parse release.txt, generate config +├── push-ocp-product.yml # Stage 2: push product to Satellite via API +├── extract-pull-secret.sh # Extract registry creds from pull-secret.txt +├── templates/ +│ ├── ocp-custom-product.yml.j2 # Product entry template (join references) +│ ├── ocp-tags.yml.j2 # Tag list template (one digest per line) +│ └── ocp-repositories.yml.j2 # Repository config template +├── generated/ # Output directory (git-ignored) +│ ├── ocp-4.20-custom-product.yml +│ ├── ocp_420_tags.yml +│ ├── ocp-4.20-repositories.yml +│ └── ocp-4.20-versions.json +├── group_vars/ +│ ├── vault_quay_credentials.yml # Vaulted quay.io creds +│ └── vault_redhat_registry_credentials.yml +├── OCP-PRODUCT-GUIDE.md # This file +└── PULL-SECRET-SETUP.md # Pull secret retrieval guide +``` diff --git a/ansible/PULL-SECRET-SETUP.md b/ansible/PULL-SECRET-SETUP.md new file mode 100644 index 000000000..9f450ad0c --- /dev/null +++ b/ansible/PULL-SECRET-SETUP.md @@ -0,0 +1,111 @@ +# Setting Up Registry Pull Secrets for OCP Container Sync + +Syncing OCP container images from quay.io requires credentials from your +Red Hat OpenShift pull secret. This document covers retrieving the pull secret +and extracting the registry credentials into Ansible vault files. + +## Prerequisites + +- A Red Hat account with an active OpenShift subscription +- `ansible-vault` installed +- `jq` and `base64` (standard on RHEL) + +## Step 1: Download Your Pull Secret + +1. Go to +2. Click **Download pull secret** to save `pull-secret.txt` +3. Copy the file to this directory (or any accessible path) + +The pull secret is a JSON file containing base64-encoded credentials for +multiple registries (quay.io, registry.redhat.io, registry.connect.redhat.com, +etc.). + +**Important:** The pull secret does not expire. However, if you regenerate it +on console.redhat.com, the previous token is revoked. Keep it in a secure +location and never commit it to version control. + +## Step 2: Extract Credentials + +Run the extraction script to parse the pull secret and create vault-encrypted +variable files: + +```bash +# Extract quay.io credentials (for OCP container syncs) +./extract-pull-secret.sh pull-secret.txt + +# You will be prompted for your Ansible vault password +``` + +This creates: +- `group_vars/vault_quay_credentials.yml` — encrypted vault file with + `quay_registry_username_vault` and `quay_registry_password_vault` +- `group_vars/vault_redhat_registry_credentials.yml` — encrypted vault file + with `redhat_registry_username_vault` and `redhat_registry_password_vault` + +## Step 3: Verify + +```bash +ansible-vault view group_vars/vault_quay_credentials.yml +``` + +You should see: +```yaml +quay_registry_username_vault: "" +quay_registry_password_vault: "" +``` + +## Pull Secret JSON Structure + +The pull secret follows the Docker/OCI auth config format: + +```json +{ + "auths": { + "quay.io": { + "auth": "", + "email": "user@example.com" + }, + "registry.redhat.io": { + "auth": "" + } + } +} +``` + +The `auth` field is a base64-encoded string of `username:password`. The +extraction script decodes this and splits on the first colon to separate +the username from the password/token. + +## Using Credentials in rhis-builder-satellite + +The vault variables are referenced in `custom_products.yml` as: + +```yaml +upstream_username: "{{ quay_registry_username_vault }}" +upstream_password: "{{ quay_registry_password_vault }}" +``` + +When deploying via rhis-provisioner, pass `--ask-vault-pass` or configure +a vault password file. + +## Registries in the Pull Secret + +| Registry | Variable Prefix | Used For | +|---|---|---| +| quay.io | `quay_registry_*` | OCP release and component images | +| registry.redhat.io | `redhat_registry_*` | Red Hat certified container images | +| registry.connect.redhat.com | (same as registry.redhat.io) | Partner/ISV containers | + +## Troubleshooting + +**Sync fails with 401 Unauthorized:** +Re-download the pull secret from console.redhat.com. If you regenerated +it recently, the old token is revoked. + +**Username looks like an email address:** +This is normal for quay.io — the username from the pull secret is typically +a service account identifier or email, not a human-readable name. + +**Token works with podman but not Satellite:** +Satellite needs the decoded username and password separately, not the +base64-encoded `auth` string. The extraction script handles this decoding. diff --git a/ansible/QUICKSTART.md b/ansible/QUICKSTART.md new file mode 100644 index 000000000..4d8575d45 --- /dev/null +++ b/ansible/QUICKSTART.md @@ -0,0 +1,229 @@ +# Quick Start Guide - Ansible Deployment + +Deploy the pulp_container tag list bypass optimization to Satellite 6.19.3 in 5 minutes. + +## Step 0: Install Prerequisites (30 seconds) + +```bash +cd /home/ansiblerunner/foreman/pulp_container/ansible + +# Install required Ansible collections +ansible-galaxy collection install -r requirements.yml +``` + +## Step 1: Edit Inventory (1 minute) + +```bash +cd /home/ansiblerunner/foreman/pulp_container/ansible +vi inventory.yml +``` + +Update these values: +```yaml +ansible_host: satellite1.parmstrong.ca # Your Satellite hostname +ansible_user: root # SSH user +``` + +Save and exit (`:wq`) + +## Step 2: Test Prerequisites (1 minute) + +```bash +# With password authentication +ansible-playbook -i inventory.yml test-prerequisites.yml --ask-pass + +# Or with SSH keys (if configured) +# ansible-playbook -i inventory.yml test-prerequisites.yml +``` + +Look for the final message: +- ✓ **"ALL PREREQUISITES MET"** → Continue to Step 3 +- ✗ **"PREREQUISITES NOT MET"** → Fix the issues shown, then retry + +## Step 3: Deploy (2 minutes) + +```bash +# Dry-run first (recommended) +ansible-playbook -i inventory.yml deploy-optimization.yml --check --ask-pass + +# Actual deployment +ansible-playbook -i inventory.yml deploy-optimization.yml --ask-pass +``` + +The playbook will: +1. Backup everything +2. Deploy the optimization +3. Run migrations +4. Restart services +5. Verify success + +**Auto-rollback**: If anything fails, it automatically rolls back. + +## Step 4: Verify (30 seconds) + +```bash +ansible-playbook -i inventory.yml verify-deployment.yml --ask-pass +``` + +Look for: **"Optimization is DEPLOYED"** + +## Step 5: Test the Optimization (Optional) + +SSH to your Satellite and run a quick sync test: + +```bash +ssh root@satellite1.parmstrong.ca + +# Watch for bypass activation in logs (in background) +journalctl -u pulpcore-worker* -f | grep -i bypass & + +# Create test repo with specific tags +hammer repository create \ + --name "ocp-test" \ + --product "Test" \ + --content-type "docker" \ + --url "https://quay.io" \ + --docker-upstream-name "openshift-release-dev/ocp-v4.0-art-dev" + +# Configure specific tags (no wildcards) +REMOTE_ID=$(hammer --output json repository info --name "ocp-test" | jq -r '.["Remote ID"]') + +curl -X PATCH -u admin:password -k \ + -H "Content-Type: application/json" \ + https://satellite1.parmstrong.ca/pulp/api/v3/remotes/container/container/$REMOTE_ID/ \ + -d '{"includes": ["4.12.0-x86_64"]}' + +# Sync and time it +time hammer repository synchronize --name "ocp-test" +``` + +**Expected results**: +- Sync completes in ~3-5 seconds (vs minutes) +- Logs show: `"Bypassing /tags/list enumeration - syncing 1 explicit references directly"` + +--- + +## Rollback (If Needed) + +If you need to undo the deployment: + +```bash +# Rollback to latest backup +ansible-playbook -i inventory.yml rollback-optimization.yml + +# Verify rollback +ansible-playbook -i inventory.yml verify-deployment.yml +``` + +Look for: **"Optimization is NOT DEPLOYED"** + +--- + +## Troubleshooting + +### Connection Issues +```bash +# Test SSH connection +ansible -i inventory.yml satellite_servers -m ping + +# If fails, check: +# - Can you SSH manually? ssh root@satellite1.parmstrong.ca +# - Is the hostname correct in inventory.yml? +# - Are SSH keys set up? ssh-copy-id root@satellite1.parmstrong.ca +``` + +### Prerequisites Not Met +```bash +# Re-run the prerequisites check to see specific issues +ansible-playbook -i inventory.yml test-prerequisites.yml + +# Common fixes: +# - PostgreSQL not running: systemctl start postgresql +# - Missing Ansible collection: ansible-galaxy collection install community.postgresql +# - Wrong source path: update pulp_container_repo_path in inventory.yml +``` + +### Deployment Failed +```bash +# The playbook auto-rolls back on failure +# Check the deployment log for details: +ls -lt /var/log/pulp-container-optimization-deploy-*.log | head -1 + +# View the log: +cat + +# List available backups: +ansible-playbook -i inventory.yml list-backups.yml + +# Manual rollback to specific backup if needed: +ansible-playbook -i inventory.yml rollback-optimization.yml \ + -e backup_path=/var/lib/pulp-backups/ +``` + +--- + +## What Happens Behind the Scenes + +### Deployment creates: +- **Backup**: `/var/lib/pulp-backups/YYYYMMDDTHHMMSS/` + - All modified Python files + - Complete migrations directory + - Database table backup + - Migration state snapshot + +### Files modified: +- `models.py` - Adds `auto_discover_cosign` field +- `sync_stages.py` - Adds bypass logic +- `synchronize.py` - Passes mirror parameter +- Migration: `0051_containerremote_auto_discover_cosign.py` + +### Database changes: +- Adds column: `container_containerremote.auto_discover_cosign BOOLEAN DEFAULT TRUE` + +### Services managed: +- `pulpcore-api` +- `pulpcore-content` +- `pulpcore-worker@1` through `pulpcore-worker@4` + +--- + +## Next Steps After Deployment + +1. **Monitor syncs** - Watch for the bypass activation in logs +2. **Measure performance** - Compare sync times before/after +3. **Read the full documentation** - See `README.md` for advanced usage +4. **Test thoroughly** - Try different sync scenarios +5. **Plan for production** - Document your testing results + +--- + +## Getting Help + +1. **Check README.md** - Comprehensive documentation +2. **Run verify playbook** - `ansible-playbook -i inventory.yml verify-deployment.yml` +3. **Check logs** - `/var/log/pulp-container-optimization-*.log` +4. **Satellite health** - `satellite-maintain health check` + +--- + +## Summary of Commands + +```bash +# Setup +cd /home/ansiblerunner/foreman/pulp_container/ansible +vi inventory.yml # Edit your Satellite hostname + +# Deploy +ansible-playbook -i inventory.yml test-prerequisites.yml +ansible-playbook -i inventory.yml deploy-optimization.yml +ansible-playbook -i inventory.yml verify-deployment.yml + +# Rollback (if needed) +ansible-playbook -i inventory.yml rollback-optimization.yml + +# Utilities +ansible-playbook -i inventory.yml list-backups.yml +ansible-playbook -i inventory.yml cleanup-backups.yml +``` + +That's it! The optimization is deployed and ready to use. diff --git a/ansible/README.md b/ansible/README.md new file mode 100644 index 000000000..6ada23e54 --- /dev/null +++ b/ansible/README.md @@ -0,0 +1,571 @@ +# Ansible Automation for pulp_container Tag List Bypass Optimization + +This directory contains Ansible playbooks for safely deploying and managing the pulp_container tag list bypass optimization on Red Hat Satellite 6.19.3 servers. + +## Overview + +The tag list bypass optimization dramatically improves sync performance (50-100x faster) for container repositories with specific tag lists. These playbooks handle: + +- ✅ Automatic backups before deployment +- ✅ Safe service management +- ✅ Database migration +- ✅ Verification and health checks +- ✅ Automatic rollback on failure +- ✅ Manual rollback capability + +## Prerequisites + +### On the Control Node (where you run ansible) + +```bash +# Install Ansible +pip3 install ansible + +# Install required Ansible collections +ansible-galaxy collection install -r requirements.yml + +# Or install manually: +# ansible-galaxy collection install community.postgresql community.general +``` + +### On the Satellite Server + +- Red Hat Satellite 6.19.3 installed +- PostgreSQL accessible +- Root SSH access or sudo privileges +- pulp_container already installed + +### SSH Authentication + +You have two options for SSH authentication: + +**Option 1: Password Authentication** (simpler, requires password each run) +```bash +# Requires sshpass to be installed +sudo dnf install sshpass # RHEL/CentOS +# or +sudo apt install sshpass # Debian/Ubuntu + +# Method 1a: Interactive password prompt (recommended) +ansible-playbook -i inventory.yml --ask-pass + +# Method 1b: Use a password file (for automation) +# Create password file (already gitignored) +echo "your_password" > .claude.password.txt +chmod 600 .claude.password.txt + +# Run with password file +ansible-playbook -i inventory.yml --extra-vars "ansible_password=$(cat .claude.password.txt)" +``` + +**⚠️ Security Note:** Password files are automatically excluded from git via `.gitignore`. Never commit passwords to version control! + +**Option 2: SSH Key Authentication** (recommended for automation) +```bash +# Generate SSH key +ssh-keygen -t ed25519 -C "ansible-automation" + +# Copy to Satellite +ssh-copy-id ansiblerunner@satellite1.parmstrong.ca + +# Run playbooks without --ask-pass +ansible-playbook -i inventory.yml +``` + +### Source Code + +Ensure the optimized pulp_container code is available at the path specified in `inventory.yml`: + +```yaml +pulp_container_repo_path: /home/ansiblerunner/foreman/pulp_container +pulp_container_branch: feature/bypass-taglist-sync-optimization +``` + +## Quick Start + +### 1. Configure Inventory + +Edit `inventory.yml` with your Satellite server details: + +```yaml +satellite1: + ansible_host: satellite1.parmstrong.ca + ansible_user: root +``` + +### 2. Test Connection + +```bash +ansible -i inventory.yml satellite_servers -m ping +``` + +### 3. Verify Current State + +```bash +ansible-playbook -i inventory.yml verify-deployment.yml +``` + +This shows whether the optimization is already deployed. + +### 4. Deploy the Optimization + +```bash +# Dry-run first (recommended) +ansible-playbook -i inventory.yml deploy-optimization.yml --check + +# Actual deployment +ansible-playbook -i inventory.yml deploy-optimization.yml +``` + +The deployment will: +1. ✅ Backup all files and database state +2. ✅ Stop Pulp services +3. ✅ Deploy optimized files +4. ✅ Run database migration +5. ✅ Restart services +6. ✅ Verify deployment +7. ✅ Automatically rollback if anything fails + +### 5. Verify Deployment + +```bash +ansible-playbook -i inventory.yml verify-deployment.yml +``` + +## Playbooks Reference + +### deploy-optimization.yml + +**Purpose**: Deploy the tag list bypass optimization + +**Usage**: +```bash +# Full deployment +ansible-playbook -i inventory.yml deploy-optimization.yml + +# Dry-run +ansible-playbook -i inventory.yml deploy-optimization.yml --check + +# Only create backups +ansible-playbook -i inventory.yml deploy-optimization.yml --tags backup + +# Only verify (post-deployment) +ansible-playbook -i inventory.yml deploy-optimization.yml --tags verify +``` + +**What it does**: +- Pre-flight checks (Satellite version, services, database) +- Creates timestamped backup in `/var/lib/pulp-backups/` +- Stops Pulp services gracefully +- Deploys optimized Python files +- Runs database migration (adds `auto_discover_cosign` field) +- Restarts services +- Verifies deployment success +- Auto-rollback on any failure + +**Output**: +- Backup location: `/var/lib/pulp-backups/YYYYMMDDTHHMMSS/` +- Deployment log: `/var/log/pulp-container-optimization-deploy-TIMESTAMP.log` + +--- + +### rollback-optimization.yml + +**Purpose**: Rollback the optimization to previous state + +**Usage**: +```bash +# Rollback to latest backup +ansible-playbook -i inventory.yml rollback-optimization.yml + +# Rollback to specific backup +ansible-playbook -i inventory.yml rollback-optimization.yml \ + -e backup_path=/var/lib/pulp-backups/20260814T120000 + +# Auto-confirm (skip prompt) +ansible-playbook -i inventory.yml rollback-optimization.yml \ + -e auto_confirm=true +``` + +**What it does**: +- Validates backup exists +- Shows confirmation prompt +- Stops Pulp services +- Restores files from backup +- Rolls back database migration +- Removes `auto_discover_cosign` field +- Restarts services +- Verifies rollback success + +**Output**: +- Rollback log: `/var/log/pulp-container-optimization-rollback-TIMESTAMP.log` +- Success marker: `/ROLLBACK_SUCCESS.txt` + +--- + +### verify-deployment.yml + +**Purpose**: Check current deployment status + +**Usage**: +```bash +ansible-playbook -i inventory.yml verify-deployment.yml +``` + +**What it shows**: +- Database column existence +- Migration status +- Service status +- API health +- Sample remote configurations + +--- + +### list-backups.yml + +**Purpose**: List all available backups + +**Usage**: +```bash +ansible-playbook -i inventory.yml list-backups.yml +``` + +**What it shows**: +- All backup directories +- Creation timestamps +- Deployment/rollback status +- Backup manifests +- Latest backup symlink + +--- + +### cleanup-backups.yml + +**Purpose**: Remove old backups + +**Usage**: +```bash +# Default: remove backups older than 30 days +ansible-playbook -i inventory.yml cleanup-backups.yml + +# Custom retention +ansible-playbook -i inventory.yml cleanup-backups.yml \ + -e backup_retention_days=7 +``` + +--- + +## Configuration Files + +### inventory.yml + +Main inventory file with Satellite server details. + +**Key variables**: +- `ansible_host`: Satellite server hostname/IP +- `ansible_user`: SSH user (usually root) +- `backup_base_dir`: Where backups are stored (default: `/var/lib/pulp-backups`) +- `backup_retention_days`: How long to keep backups (default: 30) +- `pulp_services`: List of Pulp systemd services + +### group_vars/satellite_servers.yml + +Group variables for all Satellite servers. + +**Key variables**: +- `pulp_container_python_path`: Python package location +- `pulp_container_files_to_backup`: Files to backup +- `migration_file`: Migration filename + +## Directory Structure + +``` +ansible/ +├── README.md # This file +├── inventory.yml # Inventory configuration +├── group_vars/ +│ └── satellite_servers.yml # Group variables +├── deploy-optimization.yml # Main deployment playbook +├── rollback-optimization.yml # Rollback playbook +├── rollback-tasks.yml # Shared rollback tasks +├── verify-deployment.yml # Verification playbook +├── list-backups.yml # List backups playbook +└── cleanup-backups.yml # Cleanup old backups +``` + +## Backup Structure + +Each deployment creates a timestamped backup: + +``` +/var/lib/pulp-backups/ +├── 20260814T120000/ # Timestamp-based directory +│ ├── BACKUP_MANIFEST.txt # Backup metadata +│ ├── DEPLOYMENT_SUCCESS.txt # Deployment success marker (if deployed) +│ ├── ROLLBACK_SUCCESS.txt # Rollback marker (if rolled back) +│ ├── models.py # Backed up Python files +│ ├── sync_stages.py +│ ├── synchronize.py +│ ├── migrations/ # Full migrations directory +│ ├── migration_state.json # Migration state snapshot +│ └── container_containerremote.sql # Database table backup +└── latest -> 20260814T120000/ # Symlink to most recent backup +``` + +## Common Workflows + +### Initial Deployment + +```bash +# 1. Check current state +ansible-playbook -i inventory.yml verify-deployment.yml + +# 2. Dry-run deployment +ansible-playbook -i inventory.yml deploy-optimization.yml --check + +# 3. Deploy +ansible-playbook -i inventory.yml deploy-optimization.yml + +# 4. Verify +ansible-playbook -i inventory.yml verify-deployment.yml +``` + +### Testing and Rollback + +```bash +# 1. Deploy +ansible-playbook -i inventory.yml deploy-optimization.yml + +# 2. Test (perform manual testing on Satellite) + +# 3. If issues found, rollback +ansible-playbook -i inventory.yml rollback-optimization.yml + +# 4. Verify rollback +ansible-playbook -i inventory.yml verify-deployment.yml +``` + +### Backup Management + +```bash +# List all backups +ansible-playbook -i inventory.yml list-backups.yml + +# Clean up old backups +ansible-playbook -i inventory.yml cleanup-backups.yml +``` + +## Safety Features + +### Automatic Rollback + +If deployment fails at any step, the playbook automatically: +1. Stops the deployment +2. Restores files from backup +3. Rolls back the database migration +4. Restarts services +5. Preserves the backup for investigation + +### Pre-flight Checks + +Before deploying, the playbook verifies: +- ✅ Running as root +- ✅ Satellite version detected +- ✅ pulp_container is installed +- ✅ PostgreSQL is accessible +- ✅ Database is responsive +- ✅ Services are available + +### Backup Verification + +Before rollback, the playbook: +- ✅ Confirms backup exists +- ✅ Shows backup manifest +- ✅ Prompts for confirmation +- ✅ Validates restored files + +## Troubleshooting + +### Deployment Fails + +1. **Check the deployment log**: + ```bash + ls -lt /var/log/pulp-container-optimization-deploy-*.log | head -1 + cat + ``` + +2. **Verify services**: + ```bash + satellite-maintain health check + systemctl status pulpcore-worker@1 + ``` + +3. **Check backup location**: + ```bash + ansible-playbook -i inventory.yml list-backups.yml + ``` + +4. **Manual rollback if needed**: + ```bash + ansible-playbook -i inventory.yml rollback-optimization.yml \ + -e backup_path=/var/lib/pulp-backups/ + ``` + +### Services Won't Start + +```bash +# Check service logs +journalctl -u pulpcore-api -n 50 +journalctl -u pulpcore-worker@1 -n 50 + +# Check for Python errors +python3 -m py_compile /usr/lib/python3.11/site-packages/pulp_container/app/models.py + +# Verify database +sudo -u postgres psql -d pulpcore -c "\d container_containerremote" +``` + +### Migration Fails + +```bash +# Check migration status +sudo -u pulp pulpcore-manager showmigrations container + +# Check for pending migrations +sudo -u pulp pulpcore-manager migrate --plan + +# Manual migration rollback if needed +sudo -u pulp pulpcore-manager migrate container 0050_alter_containernamespace_options +``` + +### Rollback Fails + +If automated rollback fails: + +```bash +# Stop services +systemctl stop pulpcore-api pulpcore-content pulpcore-worker@* + +# Manually restore files +BACKUP_PATH=/var/lib/pulp-backups/latest +cp $BACKUP_PATH/models.py /usr/lib/python3.11/site-packages/pulp_container/app/ +cp $BACKUP_PATH/sync_stages.py /usr/lib/python3.11/site-packages/pulp_container/app/tasks/ +cp $BACKUP_PATH/synchronize.py /usr/lib/python3.11/site-packages/pulp_container/app/tasks/ + +# Restore migrations +rm -rf /usr/lib/python3.11/site-packages/pulp_container/app/migrations +cp -r $BACKUP_PATH/migrations /usr/lib/python3.11/site-packages/pulp_container/app/ + +# Rollback migration +sudo -u pulp pulpcore-manager migrate container 0050_alter_containernamespace_options + +# Restore database table +sudo -u postgres psql -d pulpcore -f $BACKUP_PATH/container_containerremote.sql + +# Start services +systemctl start pulpcore-api pulpcore-content pulpcore-worker@* +``` + +## Testing the Optimization + +After successful deployment, test the optimization: + +```bash +# SSH to Satellite +ssh root@satellite1.parmstrong.ca + +# Create test repository with specific tags (triggers bypass) +hammer repository create \ + --name "test-bypass" \ + --product "Test" \ + --content-type "docker" \ + --url "https://quay.io" \ + --docker-upstream-name "openshift-release-dev/ocp-v4.0-art-dev" \ + --organization "Default Organization" + +# Get remote ID and configure includes +REMOTE_ID=$(hammer --output json repository info --name "test-bypass" | jq -r '.["Remote ID"]') + +curl -X PATCH -u admin:password -k \ + -H "Content-Type: application/json" \ + https://satellite1.parmstrong.ca/pulp/api/v3/remotes/container/container/$REMOTE_ID/ \ + -d '{"includes": ["4.12.0-x86_64", "4.12.1-x86_64"]}' + +# Sync and watch logs +hammer repository synchronize --name "test-bypass" --async + +# In another terminal, watch for bypass activation +journalctl -u pulpcore-worker* -f | grep -i "bypass\|cosign" +``` + +**Expected log output**: +``` +INFO: Bypassing /tags/list enumeration - syncing 2 explicit references directly +INFO: Auto-discovering cosign companion tags via HEAD probing +INFO: Found N cosign companion tag(s) for synced manifests +``` + +## Advanced Usage + +### Multiple Satellite Servers + +Add multiple hosts to `inventory.yml`: + +```yaml +satellite_servers: + hosts: + satellite1: + ansible_host: satellite1.parmstrong.ca + satellite2: + ansible_host: satellite2.parmstrong.ca + satellite3: + ansible_host: satellite3.parmstrong.ca +``` + +Deploy to all: +```bash +ansible-playbook -i inventory.yml deploy-optimization.yml +``` + +Deploy to specific host: +```bash +ansible-playbook -i inventory.yml deploy-optimization.yml --limit satellite1 +``` + +### Custom Backup Location + +```bash +ansible-playbook -i inventory.yml deploy-optimization.yml \ + -e backup_base_dir=/custom/backup/path +``` + +### Skip Confirmation Prompts + +```bash +ansible-playbook -i inventory.yml rollback-optimization.yml \ + -e auto_confirm=true +``` + +### Verbose Output + +```bash +ansible-playbook -i inventory.yml deploy-optimization.yml -vvv +``` + +## Support + +For issues or questions: + +1. Check this README +2. Review deployment/rollback logs +3. Run verification playbook +4. Check Satellite health: `satellite-maintain health check` + +## License + +Same as pulp_container project + +## Contributors + +- Paul Armstrong (parmstro) +- Claude (Anthropic) - Initial automation development diff --git a/ansible/ansible-run.sh b/ansible/ansible-run.sh new file mode 100755 index 000000000..57631e960 --- /dev/null +++ b/ansible/ansible-run.sh @@ -0,0 +1,14 @@ +#!/bin/bash +# Ansible playbook wrapper that uses the password file if available + +SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" +PASSWORD_FILE="$SCRIPT_DIR/.claude.password.txt" + +if [ -f "$PASSWORD_FILE" ]; then + # Use password file + export ANSIBLE_PASSWORD=$(cat "$PASSWORD_FILE") + ansible-playbook "$@" --extra-vars "ansible_ssh_pass=$ANSIBLE_PASSWORD" +else + # Fall back to interactive password prompt + ansible-playbook "$@" --ask-pass +fi diff --git a/ansible/cleanup-backups.yml b/ansible/cleanup-backups.yml new file mode 100644 index 000000000..7211f338f --- /dev/null +++ b/ansible/cleanup-backups.yml @@ -0,0 +1,56 @@ +--- +# Ansible Playbook: Cleanup Old Backups +# +# Removes backups older than retention period +# +# Usage: +# # Use default retention (30 days) +# ansible-playbook -i inventory.yml cleanup-backups.yml +# +# # Custom retention period +# ansible-playbook -i inventory.yml cleanup-backups.yml -e backup_retention_days=7 +# + +- name: Cleanup old pulp_container backups + hosts: satellite_servers + gather_facts: yes + + tasks: + - name: Find old backups + find: + paths: "{{ backup_base_dir }}" + file_type: directory + age: "{{ backup_retention_days }}d" + recurse: no + register: old_backups + + - name: Display backups to be removed + debug: + msg: | + Found {{ old_backups.matched }} backup(s) older than {{ backup_retention_days }} days: + {% for item in old_backups.files %} + - {{ item.path }} ({{ item.path | basename }}) + {% endfor %} + + - name: Confirm deletion + pause: + prompt: | + + About to delete {{ old_backups.matched }} old backup(s). + Press Ctrl+C then 'A' to abort. + Press Enter to continue. + + when: old_backups.matched > 0 and not (auto_confirm | default(false)) + + - name: Remove old backups + file: + path: "{{ item.path }}" + state: absent + loop: "{{ old_backups.files }}" + when: old_backups.matched > 0 + + - name: Summary + debug: + msg: | + Removed {{ old_backups.matched }} old backup(s). + Retention policy: {{ backup_retention_days }} days diff --git a/ansible/deploy-optimization.yml b/ansible/deploy-optimization.yml new file mode 100644 index 000000000..0289ae12d --- /dev/null +++ b/ansible/deploy-optimization.yml @@ -0,0 +1,571 @@ +--- +# Ansible Playbook: Deploy Tag List Bypass Optimization to Satellite 6.19.3 +# +# This playbook safely deploys the pulp_container optimization with: +# - Pre-deployment validation +# - Automatic backups +# - Service management +# - Database migration +# - Post-deployment verification +# - Automatic rollback on failure +# +# Usage: +# ansible-playbook -i inventory.yml deploy-optimization.yml +# +# Options: +# --check : Dry-run mode +# --tags backup : Only create backups +# --tags deploy : Only deploy (skip backups - not recommended) +# --tags verify : Only run verification +# + +- name: Deploy pulp_container Tag List Bypass Optimization + hosts: satellite_servers + gather_facts: yes + vars: + deployment_log: "/var/log/pulp-container-optimization-deploy-{{ backup_timestamp }}.log" + + tasks: + - name: Main deployment with automatic rollback on failure + tags: [always] + block: + # ==================================================================== + # PRE-FLIGHT CHECKS + # ==================================================================== + + - name: Create deployment log + tags: [preflight] + copy: + content: | + Pulp Container Optimization Deployment + Started: {{ ansible_date_time.iso8601 }} + Host: {{ ansible_hostname }} + User: {{ ansible_user_id }} + dest: "{{ deployment_log }}" + mode: '0644' + + - name: Check if running as root + tags: [preflight] + fail: + msg: "This playbook must be run as root (use --become)" + when: ansible_user_id != "root" + + - name: Check Satellite version + tags: [preflight] + shell: | + satellite-maintain packages list 2>/dev/null | grep -E 'satellite-[0-9]' | head -1 + register: satellite_version + changed_when: false + failed_when: false + + - name: Display Satellite version + tags: [preflight] + debug: + msg: "Detected Satellite version: {{ satellite_version.stdout | default('Unable to detect') }}" + + - name: Search for pulp_container installation + tags: [preflight] + shell: | + # Try multiple search methods + if [ -d /usr/lib/python3.12/site-packages/pulp_container ]; then + echo "/usr/lib/python3.12/site-packages/pulp_container" + elif [ -d /usr/lib/python3.11/site-packages/pulp_container ]; then + echo "/usr/lib/python3.11/site-packages/pulp_container" + else + find /usr -name "pulp_container" -type d 2>/dev/null | grep site-packages | head -1 + fi + register: pulp_container_search + changed_when: false + + - name: Fail if pulp_container not found + tags: [preflight] + fail: + msg: "pulp_container installation not found! Searched in /usr/lib/python3*/site-packages. Is Satellite 6.19.3 installed?" + when: pulp_container_search.stdout == "" + + - name: Set pulp_container paths + tags: [preflight] + set_fact: + pulp_container_python_path: "{{ pulp_container_search.stdout }}" + pulp_container_app_path: "{{ pulp_container_search.stdout }}/app" + + - name: Display discovered pulp_container location + tags: [preflight] + debug: + msg: "Found pulp_container at {{ pulp_container_python_path }} ({{ pulp_container_python_path | regex_search('python3\\.[0-9]+') }})" + + - name: Verify app directory exists + tags: [preflight] + stat: + path: "{{ pulp_container_app_path }}" + register: pulp_app_check + + - name: Fail if app directory not found + tags: [preflight] + fail: + msg: "pulp_container/app directory not found at {{ pulp_container_app_path }}" + when: not pulp_app_check.stat.exists + + - name: Check PostgreSQL status + tags: [preflight] + systemd: + name: postgresql + state: started + check_mode: yes + register: postgres_check + + - name: Verify database accessibility + tags: [preflight] + become_user: postgres + postgresql_query: + db: "{{ pulp_db_name }}" + query: "SELECT 1" + register: db_check + failed_when: false + + - name: Fail if database not accessible + tags: [preflight] + fail: + msg: "Cannot access PostgreSQL database {{ pulp_db_name }}" + when: db_check is failed + + - name: Check pulp services status + tags: [preflight] + systemd: + name: "{{ item }}" + register: service_check + loop: "{{ pulp_services }}" + failed_when: false + + - name: Display service status + tags: [preflight] + debug: + msg: "Service {{ item.item }}: {{ item.status.ActiveState | default('unknown') }}" + loop: "{{ service_check.results }}" + loop_control: + label: "{{ item.item }}" + + # ==================================================================== + # BACKUP EXISTING FILES + # ==================================================================== + + - name: Create backup directory + tags: [backup] + file: + path: "{{ backup_base_dir }}/{{ backup_timestamp }}" + state: directory + mode: '0777' # Allow postgres user to write SQL dumps + + - name: Create backup manifest file + tags: [backup] + copy: + content: | + Backup created: {{ ansible_date_time.iso8601 }} + Host: {{ ansible_hostname }} + Backup directory: {{ backup_base_dir }}/{{ backup_timestamp }} + Reason: pulp_container optimization deployment + dest: "{{ backup_base_dir }}/{{ backup_timestamp }}/BACKUP_MANIFEST.txt" + mode: '0644' + + - name: Backup Python files + tags: [backup] + copy: + src: "{{ item.path }}" + dest: "{{ backup_base_dir }}/{{ backup_timestamp }}/{{ item.name }}" + remote_src: yes + mode: preserve + loop: "{{ pulp_container_files_to_backup }}" + when: not (item.is_directory | default(false)) + + - name: Backup migrations directory + tags: [backup] + synchronize: + src: "{{ item.path }}/" + dest: "{{ backup_base_dir }}/{{ backup_timestamp }}/{{ item.name }}/" + archive: yes + delegate_to: "{{ inventory_hostname }}" + loop: "{{ pulp_container_files_to_backup }}" + when: item.is_directory | default(false) + + - name: Backup database schema (migrations state) + tags: [backup] + become_user: postgres + postgresql_query: + db: "{{ pulp_db_name }}" + query: | + SELECT app, name, applied + FROM django_migrations + WHERE app = '{{ migration_app }}' + ORDER BY applied DESC + register: migration_state + + - name: Save migration state to file + tags: [backup] + copy: + content: "{{ migration_state.query_result | to_nice_json }}" + dest: "{{ backup_base_dir }}/{{ backup_timestamp }}/migration_state.json" + mode: '0644' + + - name: Create backup of container_containerremote table + tags: [backup] + become_user: postgres + postgresql_db: + name: "{{ pulp_db_name }}" + state: dump + target: "{{ backup_base_dir }}/{{ backup_timestamp }}/container_containerremote.sql" + target_opts: "--table=container_containerremote --clean --if-exists" + + - name: Record backup location + tags: [backup] + set_fact: + current_backup_path: "{{ backup_base_dir }}/{{ backup_timestamp }}" + + - name: Create latest backup symlink + tags: [backup] + file: + src: "{{ backup_base_dir }}/{{ backup_timestamp }}" + dest: "{{ backup_base_dir }}/latest" + state: link + force: yes + + - name: Display backup location + tags: [backup] + debug: + msg: "Backup created at: {{ current_backup_path }}" + + # ==================================================================== + # WAIT FOR RUNNING TASKS TO DRAIN + # ==================================================================== + + - name: Check for running pulp tasks + tags: [deploy] + become_user: pulp + shell: | + pulpcore-manager shell -c " + from pulpcore.app.models import Task + running = Task.objects.filter(state__in=['running', 'waiting', 'canceling']) + for t in running[:20]: + print('{0} {1} {2}'.format(t.pulp_id, t.state.ljust(10), t.name)) + print('TOTAL: {0}'.format(running.count())) + " + environment: + PULP_SETTINGS: /etc/pulp/settings.py + DJANGO_SETTINGS_MODULE: pulpcore.app.settings + register: running_tasks + changed_when: false + + - name: Display running tasks + tags: [deploy] + debug: + msg: "{{ running_tasks.stdout_lines }}" + + - name: Wait for running tasks to drain + tags: [deploy] + become_user: pulp + shell: | + pulpcore-manager shell -c " + from pulpcore.app.models import Task + count = Task.objects.filter(state__in=['running', 'waiting', 'canceling']).count() + print(count) + " + environment: + PULP_SETTINGS: /etc/pulp/settings.py + DJANGO_SETTINGS_MODULE: pulpcore.app.settings + register: task_count + until: task_count.stdout | trim | int == 0 + retries: 60 + delay: 30 + changed_when: false + when: "running_tasks.stdout is search('TOTAL: [1-9]')" + + - name: Confirm all tasks drained + tags: [deploy] + debug: + msg: "All pulp tasks have completed. Safe to proceed with deployment." + + # ==================================================================== + # STOP PULP SERVICES + # ==================================================================== + + - name: Stop all pulp services + tags: [deploy] + systemd: + name: "{{ item }}" + state: stopped + loop: "{{ pulp_services }}" + + - name: Wait for services to stop + tags: [deploy] + wait_for: + timeout: 30 + + - name: Verify services are stopped + tags: [deploy] + systemd: + name: "{{ item }}" + register: service_verify + failed_when: service_verify.status.ActiveState == "active" + loop: "{{ pulp_services }}" + + # ==================================================================== + # DEPLOY NEW FILES + # ==================================================================== + + - name: Copy optimized models.py + tags: [deploy] + copy: + src: "{{ pulp_container_repo_path }}/pulp_container/app/models.py" + dest: "{{ pulp_container_app_path }}/models.py" + owner: root + group: root + mode: '0644' + + - name: Copy optimized sync_stages.py + tags: [deploy] + copy: + src: "{{ pulp_container_repo_path }}/pulp_container/app/tasks/sync_stages.py" + dest: "{{ pulp_container_app_path }}/tasks/sync_stages.py" + owner: root + group: root + mode: '0644' + + - name: Copy optimized synchronize.py + tags: [deploy] + copy: + src: "{{ pulp_container_repo_path }}/pulp_container/app/tasks/synchronize.py" + dest: "{{ pulp_container_app_path }}/tasks/synchronize.py" + owner: root + group: root + mode: '0644' + + - name: Copy migration file + tags: [deploy] + copy: + src: "{{ pulp_container_repo_path }}/pulp_container/app/migrations/{{ migration_file }}" + dest: "{{ pulp_container_app_path }}/migrations/{{ migration_file }}" + owner: root + group: root + mode: '0644' + + - name: Compile Python files + tags: [deploy] + shell: | + python3 -m py_compile {{ pulp_container_app_path }}/models.py + python3 -m py_compile {{ pulp_container_app_path }}/tasks/sync_stages.py + python3 -m py_compile {{ pulp_container_app_path }}/tasks/synchronize.py + register: compile_result + failed_when: compile_result.rc != 0 + + # ==================================================================== + # RUN DATABASE MIGRATION + # ==================================================================== + + - name: Check pending migrations + tags: [deploy, migrate] + become_user: pulp + shell: | + pulpcore-manager showmigrations {{ migration_app }} | grep -E '\[ \]' | wc -l + environment: + PULP_SETTINGS: /etc/pulp/settings.py + DJANGO_SETTINGS_MODULE: pulpcore.app.settings + register: pending_migrations + changed_when: false + + - name: Display pending migrations count + tags: [deploy, migrate] + debug: + msg: "Pending migrations: {{ pending_migrations.stdout }}" + + - name: Run Django migrations + tags: [deploy, migrate] + become_user: pulp + shell: | + pulpcore-manager migrate {{ migration_app }} + environment: + PULP_SETTINGS: /etc/pulp/settings.py + DJANGO_SETTINGS_MODULE: pulpcore.app.settings + register: migration_result + + - name: Display migration output + tags: [deploy, migrate] + debug: + var: migration_result.stdout_lines + + - name: Verify migration applied + tags: [deploy, migrate] + become_user: "{{ postgres_user }}" + postgresql_query: + db: "{{ pulp_db_name }}" + query: | + SELECT name FROM django_migrations + WHERE app = '{{ migration_app }}' + AND name = '{{ migration_file.replace('.py', '') }}' + register: migration_verify + + - name: Fail if migration not applied + tags: [deploy, migrate] + fail: + msg: "Migration {{ migration_file }} was not applied successfully" + when: migration_verify.rowcount == 0 + + - name: Verify auto_discover_cosign column exists + tags: [deploy, migrate] + become_user: "{{ postgres_user }}" + postgresql_query: + db: "{{ pulp_db_name }}" + query: | + SELECT column_name, data_type, column_default + FROM information_schema.columns + WHERE table_name='container_containerremote' + AND column_name='auto_discover_cosign' + register: column_check + + - name: Display column info + tags: [deploy, migrate] + debug: + var: column_check.query_result + + - name: Fail if column not found + tags: [deploy, migrate] + fail: + msg: "Column auto_discover_cosign not found in container_containerremote" + when: column_check.rowcount == 0 + + # ==================================================================== + # RESTART PULP SERVICES + # ==================================================================== + + - name: Start all pulp services + tags: [deploy] + systemd: + name: "{{ item }}" + state: started + loop: "{{ pulp_services }}" + + - name: Wait for services to start + tags: [deploy] + wait_for: + timeout: 30 + + - name: Verify services are running + tags: [deploy] + systemd: + name: "{{ item }}" + register: service_running + failed_when: service_running.status.ActiveState != "active" + loop: "{{ pulp_services }}" + + # ==================================================================== + # POST-DEPLOYMENT VERIFICATION + # ==================================================================== + + - name: Wait for Pulp API to be responsive + tags: [verify] + shell: curl -k -s -o /dev/null -w "%{http_code}" https://{{ ansible_host }}/pulp/api/v3/status/ + register: api_check + until: api_check.stdout == "200" + retries: 12 + delay: 5 + changed_when: false + + - name: Check Satellite health + tags: [verify] + shell: satellite-maintain health check --assumeyes + register: health_check + failed_when: false + + - name: Display health check results + tags: [verify] + debug: + var: health_check.stdout_lines + + - name: Verify all remotes have auto_discover_cosign field + tags: [verify] + become_user: "{{ postgres_user }}" + postgresql_query: + db: "{{ pulp_db_name }}" + query: | + SELECT upstream_name, auto_discover_cosign + FROM container_containerremote + LIMIT 5 + register: remote_check + + - name: Display remote auto_discover_cosign values + tags: [verify] + debug: + var: remote_check.query_result + + - name: Create deployment success marker + tags: [verify] + copy: + content: | + Deployment successful + Completed: {{ ansible_date_time.iso8601 }} + Backup location: {{ current_backup_path }} + Migration applied: {{ migration_file }} + dest: "{{ backup_base_dir }}/{{ backup_timestamp }}/DEPLOYMENT_SUCCESS.txt" + mode: '0644' + + # ==================================================================== + # FINAL REPORT + # ==================================================================== + + - name: Deployment completed successfully + pause: + seconds: 1 + prompt: | + + ======================================== + DEPLOYMENT COMPLETED SUCCESSFULLY + ======================================== + Timestamp: {{ ansible_date_time.iso8601 }} + Host: {{ ansible_hostname }} + Backup location: {{ current_backup_path }} + Migration applied: {{ migration_file }} + + Services status: All running + API status: Healthy + Database: Migration successful + + Next steps: + 1. Review deployment log: {{ deployment_log }} + 2. Test the optimization with a sample sync + 3. Monitor logs: journalctl -u pulpcore-worker* -f + + To rollback: + ansible-playbook -i inventory.yml rollback-optimization.yml -e backup_path={{ current_backup_path }} + ======================================== + + rescue: + # ==================================================================== + # AUTOMATIC ROLLBACK ON FAILURE + # ==================================================================== + + - name: Deployment failed - initiating automatic rollback + debug: + msg: "Deployment failed. Starting automatic rollback..." + + - name: Determine backup path for rollback + set_fact: + rollback_backup_path: "{{ current_backup_path | default(backup_base_dir + '/latest') }}" + + - name: Check if backup was created + stat: + path: "{{ rollback_backup_path }}" + register: backup_exists + + - name: Include rollback tasks + include_tasks: rollback-tasks.yml + vars: + rollback_reason: "Automatic rollback due to deployment failure" + backup_path: "{{ rollback_backup_path }}" + when: backup_exists.stat.exists + + - name: Warn if no backup available + debug: + msg: "WARNING: Deployment failed before backup was created. Manual recovery may be needed." + when: not backup_exists.stat.exists + + - name: Final failure message + fail: + msg: "Deployment failed{{ ' and has been rolled back' if backup_exists.stat.exists else ' (no backup to rollback)' }}. {{ 'Backup at ' + rollback_backup_path if backup_exists.stat.exists else 'Manual recovery needed' }}. Check logs {{ deployment_log }}" diff --git a/ansible/extract-pull-secret.sh b/ansible/extract-pull-secret.sh new file mode 100755 index 000000000..519262acd --- /dev/null +++ b/ansible/extract-pull-secret.sh @@ -0,0 +1,116 @@ +#!/bin/bash +# +# Extract registry credentials from an OpenShift pull secret and write +# them to Ansible vault-encrypted variable files. +# +# Usage: ./extract-pull-secret.sh +# + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +GROUP_VARS_DIR="${SCRIPT_DIR}/group_vars" + +if [[ $# -lt 1 ]]; then + echo "Usage: $0 " + echo "" + echo "Download your pull secret from:" + echo " https://console.redhat.com/openshift/install/pull-secret" + exit 1 +fi + +PULL_SECRET="$1" + +if [[ ! -f "$PULL_SECRET" ]]; then + echo "ERROR: File not found: $PULL_SECRET" + exit 1 +fi + +# Validate JSON structure +if ! jq -e '.auths' "$PULL_SECRET" > /dev/null 2>&1; then + echo "ERROR: Invalid pull secret format — expected JSON with .auths key" + exit 1 +fi + +echo "Parsing pull secret: $PULL_SECRET" +echo "" + +# List available registries +echo "Available registries:" +jq -r '.auths | keys[]' "$PULL_SECRET" | while read -r reg; do + echo " - $reg" +done +echo "" + +extract_credentials() { + local registry="$1" + local username_var="$2" + local password_var="$3" + local vault_file="$4" + + local auth_b64 + auth_b64=$(jq -r ".auths[\"${registry}\"].auth // empty" "$PULL_SECRET") + + if [[ -z "$auth_b64" ]]; then + echo "WARNING: No credentials found for ${registry} — skipping" + return 1 + fi + + local decoded + decoded=$(echo "$auth_b64" | base64 -d 2>/dev/null) + + # Split on first colon — password may contain colons + local username="${decoded%%:*}" + local password="${decoded#*:}" + + if [[ -z "$username" || -z "$password" ]]; then + echo "ERROR: Failed to decode credentials for ${registry}" + return 1 + fi + + echo "Extracted ${registry} credentials:" + echo " Username: ${username}" + echo " Password: ${password:0:8}...$(echo -n "$password" | tail -c 4) (${#password} chars)" + + # Write to a temp file, then vault-encrypt it + local tmpfile + tmpfile=$(mktemp) + cat > "$tmpfile" << VARS +--- +${username_var}: "${username}" +${password_var}: "${password}" +VARS + + # Encrypt with ansible-vault + if [[ -f "$vault_file" ]]; then + echo " Existing vault file found — overwriting: $vault_file" + fi + + ansible-vault encrypt "$tmpfile" --output="$vault_file" + rm -f "$tmpfile" + + echo " Wrote vault file: $vault_file" + echo "" +} + +mkdir -p "$GROUP_VARS_DIR" + +# Extract quay.io credentials +extract_credentials \ + "quay.io" \ + "quay_registry_username_vault" \ + "quay_registry_password_vault" \ + "${GROUP_VARS_DIR}/vault_quay_credentials.yml" + +# Extract registry.redhat.io credentials +extract_credentials \ + "registry.redhat.io" \ + "redhat_registry_username_vault" \ + "redhat_registry_password_vault" \ + "${GROUP_VARS_DIR}/vault_redhat_registry_credentials.yml" + +echo "========================================" +echo "Done. Verify with:" +echo " ansible-vault view ${GROUP_VARS_DIR}/vault_quay_credentials.yml" +echo " ansible-vault view ${GROUP_VARS_DIR}/vault_redhat_registry_credentials.yml" +echo "========================================" diff --git a/ansible/generate-ocp-product.yml b/ansible/generate-ocp-product.yml new file mode 100644 index 000000000..d8fda0dde --- /dev/null +++ b/ansible/generate-ocp-product.yml @@ -0,0 +1,181 @@ +--- +# Playbook: Generate OCP Container Product Definition for rhis-builder-satellite +# +# Fetches an OpenShift release.txt, parses the container image digests, and +# generates a custom_products YAML entry compatible with rhis-builder-satellite. +# +# Accumulates digests across z-stream releases into a single product per OCP +# minor version. Use content views to snapshot specific z-stream releases. +# +# Usage: +# ansible-playbook generate-ocp-product.yml -e ocp_version=4.20.32 +# ansible-playbook generate-ocp-product.yml -e ocp_version=4.20.33 # merges with existing +# +# Output: +# generated/ocp-4.20-custom-product.yml (product entry for custom_products.yml) +# generated/ocp_420_tags.yml (tag lists for host_vars/) +# generated/ocp-4.20-repositories.yml (repository entries for repositories.yml) +# + +- name: Generate OCP container product definition + hosts: localhost + connection: local + gather_facts: no + + vars: + ocp_minor: "{{ ocp_version | regex_replace('^([0-9]+\\.[0-9]+)\\.[0-9]+$', '\\1') }}" + release_base_url: "https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp" + release_url: "{{ release_base_url }}/{{ ocp_version }}/release.txt" + output_dir: "{{ playbook_dir }}/generated" + output_file: "{{ output_dir }}/ocp-{{ ocp_minor }}-custom-product.yml" + tags_file: "{{ output_dir }}/ocp_{{ ocp_minor | replace('.', '') }}_tags.yml" + repos_file: "{{ output_dir }}/ocp-{{ ocp_minor }}-repositories.yml" + versions_file: "{{ output_dir }}/ocp-{{ ocp_minor }}-versions.json" + + tasks: + - name: Validate ocp_version format + assert: + that: + - ocp_version is defined + - "ocp_version is match('^[0-9]+\\.[0-9]+\\.[0-9]+$')" + fail_msg: "Provide ocp_version in X.Y.Z format, e.g. -e ocp_version=4.20.32" + + - name: Display configuration + debug: + msg: | + OCP Version: {{ ocp_version }} + Minor: {{ ocp_minor }} + Release URL: {{ release_url }} + Output: {{ output_file }} + + # ================================================================== + # FETCH AND PARSE RELEASE.TXT + # ================================================================== + + - name: Fetch release.txt + uri: + url: "{{ release_url }}" + return_content: yes + status_code: 200 + register: release_txt + + - name: Parse release digest + set_fact: + release_digest: "{{ release_txt.content | regex_search('Pull From:.*@(sha256:[a-f0-9]+)', '\\1') | first }}" + + - name: Parse component image lines + set_fact: + component_lines: "{{ release_txt.content.split('\n') | select('match', '^ *\\S+\\s+quay\\.io/openshift-release-dev/ocp-v4\\.0-art-dev@sha256:') | list }}" + + - name: Extract component digests + set_fact: + art_dev_digests: "{{ component_lines | map('regex_search', '(sha256:[a-f0-9]+)') | select | unique | sort | list }}" + + - name: Display parse results + debug: + msg: | + Release digest: {{ release_digest }} + Component images: {{ art_dev_digests | length }} + + # ================================================================== + # MERGE WITH EXISTING (accumulate across z-streams) + # ================================================================== + + - name: Create output directory + file: + path: "{{ output_dir }}" + state: directory + mode: '0755' + + - name: Check for existing versions file + stat: + path: "{{ versions_file }}" + register: versions_stat + + - name: Load existing versions + set_fact: + existing_versions: "{{ lookup('file', versions_file) | from_json }}" + when: versions_stat.stat.exists + + - name: Initialize versions tracking + set_fact: + existing_versions: + included_versions: [] + release_tags: [] + art_dev_tags: [] + when: not versions_stat.stat.exists + + - name: Merge release tags + set_fact: + merged_release_tags: "{{ (existing_versions.release_tags | default([])) | union([release_digest]) | sort }}" + + - name: Merge component tags + set_fact: + merged_art_dev_tags: "{{ (existing_versions.art_dev_tags | default([])) | union(art_dev_digests) | sort }}" + + - name: Update included versions list + set_fact: + included_versions: "{{ (existing_versions.included_versions | default([])) | union([ocp_version]) | sort }}" + + - name: Display merge results + debug: + msg: | + Z-stream releases: {{ included_versions | join(', ') }} + Total release digests: {{ merged_release_tags | length }} + Total component digests: {{ merged_art_dev_tags | length }} + New component digests added: {{ merged_art_dev_tags | length - (existing_versions.art_dev_tags | default([]) | length) }} + + # ================================================================== + # GENERATE OUTPUT FILES + # ================================================================== + + - name: Save versions tracking data + copy: + content: "{{ versions_data | to_nice_json }}" + dest: "{{ versions_file }}" + mode: '0644' + vars: + versions_data: + ocp_minor: "{{ ocp_minor }}" + included_versions: "{{ included_versions }}" + release_tags: "{{ merged_release_tags }}" + art_dev_tags: "{{ merged_art_dev_tags }}" + last_updated: "{{ lookup('pipe', 'date -u +%Y-%m-%dT%H:%M:%SZ') }}" + + - name: Generate custom product YAML + template: + src: templates/ocp-custom-product.yml.j2 + dest: "{{ output_file }}" + mode: '0644' + + - name: Generate tags file + template: + src: templates/ocp-tags.yml.j2 + dest: "{{ tags_file }}" + mode: '0644' + + - name: Generate repositories file + template: + src: templates/ocp-repositories.yml.j2 + dest: "{{ repos_file }}" + mode: '0644' + + - name: Summary + debug: + msg: | + ======================================== + PRODUCT DEFINITION GENERATED + ======================================== + Output: {{ output_file }} + Product: OpenShift Container Platform {{ ocp_minor }} + Z-streams: {{ included_versions | join(', ') }} + Release digests: {{ merged_release_tags | length }} + Component digests: {{ merged_art_dev_tags | length }} + + Next steps: + 1. Review: cat {{ output_file }} + 2. Copy tags file to host_vars: cp {{ tags_file }} /host_vars// + 3. Merge product entry into custom_products.yml (uses join references) + 4. Append repository entries from {{ repos_file }} into repositories.yml + 5. Run rhis-builder-satellite to apply + ======================================== diff --git a/ansible/generated/.gitignore b/ansible/generated/.gitignore new file mode 100644 index 000000000..d6b7ef32c --- /dev/null +++ b/ansible/generated/.gitignore @@ -0,0 +1,2 @@ +* +!.gitignore diff --git a/ansible/group_vars/satellite_servers.yml b/ansible/group_vars/satellite_servers.yml new file mode 100644 index 000000000..133509ba2 --- /dev/null +++ b/ansible/group_vars/satellite_servers.yml @@ -0,0 +1,27 @@ +--- +# Group variables for Satellite servers + +# Note: pulp_container_python_path and pulp_container_app_path are discovered +# dynamically during playbook execution by searching /usr/lib/python3*/site-packages + +# Files to backup (relative paths, will be prefixed with discovered path) +pulp_container_files_to_backup: + - path: "{{ pulp_container_app_path }}/models.py" + name: models.py + - path: "{{ pulp_container_app_path }}/tasks/sync_stages.py" + name: sync_stages.py + - path: "{{ pulp_container_app_path }}/tasks/synchronize.py" + name: synchronize.py + - path: "{{ pulp_container_app_path }}/migrations" + name: migrations + is_directory: true + +# Migration file to check +migration_file: "0046_containerremote_auto_discover_cosign.py" +migration_app: "container" + +# Health check endpoints +satellite_health_endpoint: "https://{{ ansible_host }}/api/status" + +# Backup timestamp format +backup_timestamp: "{{ ansible_date_time.iso8601_basic_short }}" diff --git a/ansible/install-requirements.sh b/ansible/install-requirements.sh new file mode 100755 index 000000000..5ce563872 --- /dev/null +++ b/ansible/install-requirements.sh @@ -0,0 +1,38 @@ +#!/bin/bash +# Install Ansible Galaxy requirements for pulp_container deployment playbooks + +set -e + +echo "=========================================" +echo "Installing Ansible Galaxy Requirements" +echo "=========================================" +echo "" + +# Check if ansible-galaxy is available +if ! command -v ansible-galaxy &> /dev/null; then + echo "ERROR: ansible-galaxy not found!" + echo "" + echo "Please install Ansible first:" + echo " pip3 install ansible" + echo "" + exit 1 +fi + +# Get the directory of this script +SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" + +# Install collections +echo "Installing collections from requirements.yml..." +ansible-galaxy collection install -r "${SCRIPT_DIR}/requirements.yml" + +echo "" +echo "=========================================" +echo "Installation Complete!" +echo "=========================================" +echo "" +echo "Installed collections:" +ansible-galaxy collection list | grep -E "community\.(postgresql|general)" +echo "" +echo "You can now run the deployment playbook:" +echo " ansible-playbook -i inventory.yml deploy-optimization.yml" +echo "" diff --git a/ansible/inventory.yml b/ansible/inventory.yml new file mode 100644 index 000000000..2657cdbe2 --- /dev/null +++ b/ansible/inventory.yml @@ -0,0 +1,40 @@ +--- +# Ansible Inventory for Satellite 6.19.3 Deployment +# Update this file with your Satellite instance details + +all: + children: + satellite_servers: + hosts: + satellite1: + ansible_host: satellite1.parmstrong.ca + satellite1-dev: + ansible_host: satellite1.example.ca + ansible_user: ansiblerunner + ansible_become: yes + ansible_python_interpreter: /usr/bin/python3.12 + + # Satellite-specific variables + satellite_admin_user: admin + satellite_organization: "Default Organization" + + # Backup configuration + backup_base_dir: /var/lib/pulp-backups + backup_retention_days: 30 + + # Pulp service names + pulp_services: + - pulpcore-api + - pulpcore-content + - pulpcore-worker@1 + - pulpcore-worker@2 + - pulpcore-worker@3 + - pulpcore-worker@4 + + # Database configuration + postgres_user: postgres + pulp_db_name: pulpcore + + # Source repository (where the optimized code is) + pulp_container_repo_path: /home/ansiblerunner/foreman/pulp_container + pulp_container_branch: feature/bypass-taglist-sync-optimization diff --git a/ansible/list-backups.yml b/ansible/list-backups.yml new file mode 100644 index 000000000..0f6652291 --- /dev/null +++ b/ansible/list-backups.yml @@ -0,0 +1,104 @@ +--- +# Ansible Playbook: List Available Backups +# +# Shows all available backups that can be used for rollback +# +# Usage: +# ansible-playbook -i inventory.yml list-backups.yml +# + +- name: List available pulp_container backups + hosts: satellite_servers + gather_facts: yes + + tasks: + - name: Check if backup directory exists + stat: + path: "{{ backup_base_dir }}" + register: backup_dir + + - name: Find all backups + find: + paths: "{{ backup_base_dir }}" + file_type: directory + recurse: no + register: backup_dirs + when: backup_dir.stat.exists + + - name: Get backup manifests + slurp: + src: "{{ item.path }}/BACKUP_MANIFEST.txt" + register: manifests + loop: "{{ backup_dirs.files }}" + when: backup_dir.stat.exists + failed_when: false + + - name: Check for deployment success markers + stat: + path: "{{ item.path }}/DEPLOYMENT_SUCCESS.txt" + register: success_markers + loop: "{{ backup_dirs.files }}" + when: backup_dir.stat.exists + + - name: Check for rollback markers + stat: + path: "{{ item.path }}/ROLLBACK_SUCCESS.txt" + register: rollback_markers + loop: "{{ backup_dirs.files }}" + when: backup_dir.stat.exists + + - name: Display backup information + debug: + msg: | + ======================================== + AVAILABLE BACKUPS + ======================================== + {% if not backup_dir.stat.exists %} + No backup directory found at {{ backup_base_dir }} + {% elif backup_dirs.matched == 0 %} + No backups found in {{ backup_base_dir }} + {% else %} + {% for item in backup_dirs.files %} + + Backup: {{ item.path }} + Created: {{ item.path | basename }} + Size: {{ (item.size / 1024 / 1024) | round(2) }} MB + {% if success_markers.results[loop.index0].stat.exists %} + Status: ✓ Deployment successful + {% elif rollback_markers.results[loop.index0].stat.exists %} + Status: ↺ Rolled back + {% else %} + Status: ⚠ Unknown (check manually) + {% endif %} + + Manifest: + {% if manifests.results[loop.index0].content is defined %} + {{ manifests.results[loop.index0].content | b64decode | indent(2) }} + {% else %} + (No manifest found) + {% endif %} + {% endfor %} + + Latest symlink points to: + {% if (backup_base_dir + '/latest') is link %} + {{ lookup('file', backup_base_dir + '/latest') | default('(broken symlink)') }} + {% else %} + (No latest symlink) + {% endif %} + {% endif %} + + ======================================== + + To rollback to a specific backup: + ansible-playbook -i inventory.yml rollback-optimization.yml \ + -e backup_path=/path/to/backup + + To rollback to latest: + ansible-playbook -i inventory.yml rollback-optimization.yml + ======================================== + + - name: Cleanup old backups (optional) + debug: + msg: | + To remove backups older than {{ backup_retention_days }} days: + ansible-playbook -i inventory.yml cleanup-backups.yml diff --git a/ansible/push-ocp-product.yml b/ansible/push-ocp-product.yml new file mode 100644 index 000000000..2e9680b77 --- /dev/null +++ b/ansible/push-ocp-product.yml @@ -0,0 +1,192 @@ +--- +# Playbook: Push OCP Product Definition to Satellite +# +# Standalone playbook that uses the redhat.satellite collection to create +# or update an OCP container product and its repositories on a Satellite +# server. Uses the same logic as the rhis-builder-satellite custom_products +# role but operates independently. +# +# Prerequisites: +# - redhat.satellite collection installed +# - Generated tag files in generated/ (run generate-ocp-product.yml first) +# - Vault file with quay.io credentials (run extract-pull-secret.sh first) +# +# Usage: +# ansible-playbook push-ocp-product.yml \ +# -e ocp_minor=4.20 \ +# -e satellite_url=https://satellite1.example.ca \ +# -e satellite_username=admin \ +# -e satellite_password= \ +# --ask-vault-pass +# +# # Or with GSSAPI authentication: +# ansible-playbook push-ocp-product.yml \ +# -e ocp_minor=4.20 \ +# -e satellite_url=https://satellite1.example.ca \ +# -e satellite_use_gssapi=true \ +# --ask-vault-pass +# +# # Optionally trigger a sync after creating: +# ansible-playbook push-ocp-product.yml \ +# -e ocp_minor=4.20 \ +# -e satellite_url=https://satellite1.example.ca \ +# -e satellite_username=admin \ +# -e satellite_password= \ +# -e sync_after_create=true \ +# --ask-vault-pass +# + +- name: Push OCP product to Satellite + hosts: localhost + connection: local + gather_facts: no + + vars: + satellite_organization: "Default Organization" + satellite_validate_certs: true + sync_after_create: false + tags_file: "{{ playbook_dir }}/generated/ocp_{{ ocp_minor | replace('.', '') }}_tags.yml" + product_name: "OpenShift Container Platform {{ ocp_minor }}" + product_label: "ocp-{{ ocp_minor | replace('.', '-') }}" + + vars_files: + - "group_vars/vault_quay_credentials.yml" + + pre_tasks: + - name: Validate required variables + assert: + that: + - ocp_minor is defined + - satellite_url is defined + - (satellite_username is defined and satellite_password is defined) or satellite_use_gssapi | default(false) + fail_msg: > + Required variables: ocp_minor, satellite_url, and either + satellite_username/satellite_password or satellite_use_gssapi=true + + - name: Validate tags file exists + stat: + path: "{{ tags_file }}" + register: tags_stat + failed_when: not tags_stat.stat.exists + + - name: Load OCP tag lists + include_vars: + file: "{{ tags_file }}" + + tasks: + # ================================================================== + # ENSURE PRODUCT + # ================================================================== + + - name: "Ensure custom product - {{ product_name }}" + redhat.satellite.product: + username: "{{ satellite_username | default(omit) }}" + password: "{{ satellite_password | default(omit) }}" + use_gssapi: "{{ satellite_use_gssapi | default(omit) }}" + server_url: "{{ satellite_url }}" + organization: "{{ satellite_organization }}" + validate_certs: "{{ satellite_validate_certs }}" + name: "{{ product_name }}" + label: "{{ product_label }}" + description: "OCP {{ ocp_minor }} release and component container images" + state: present + + # ================================================================== + # ENSURE REPOSITORIES + # ================================================================== + + - name: "Ensure repository - ocp-release" + redhat.satellite.repository: + username: "{{ satellite_username | default(omit) }}" + password: "{{ satellite_password | default(omit) }}" + use_gssapi: "{{ satellite_use_gssapi | default(omit) }}" + server_url: "{{ satellite_url }}" + organization: "{{ satellite_organization }}" + validate_certs: "{{ satellite_validate_certs }}" + product: "{{ product_name }}" + name: "ocp-release" + label: "ocp-release" + description: "OCP {{ ocp_minor }} release images" + content_type: "docker" + docker_upstream_name: "openshift-release-dev/ocp-release" + url: "https://quay.io" + upstream_username: "{{ quay_registry_username_vault }}" + upstream_password: "{{ quay_registry_password_vault }}" + download_policy: "immediate" + mirroring_policy: "additive" + include_tags: "{{ vars['ocp_' + ocp_minor | replace('.', '') + '_release_tags'] | join(',') }}" + exclude_tags: "*-source" + state: present + register: repo_release + + - name: "Ensure repository - ocp-v4.0-art-dev" + redhat.satellite.repository: + username: "{{ satellite_username | default(omit) }}" + password: "{{ satellite_password | default(omit) }}" + use_gssapi: "{{ satellite_use_gssapi | default(omit) }}" + server_url: "{{ satellite_url }}" + organization: "{{ satellite_organization }}" + validate_certs: "{{ satellite_validate_certs }}" + product: "{{ product_name }}" + name: "ocp-v4.0-art-dev" + label: "ocp-v4_0-art-dev" + description: "OCP {{ ocp_minor }} component images" + content_type: "docker" + docker_upstream_name: "openshift-release-dev/ocp-v4.0-art-dev" + url: "https://quay.io" + upstream_username: "{{ quay_registry_username_vault }}" + upstream_password: "{{ quay_registry_password_vault }}" + download_policy: "immediate" + mirroring_policy: "additive" + include_tags: "{{ vars['ocp_' + ocp_minor | replace('.', '') + '_art_dev_tags'] | join(',') }}" + exclude_tags: "*-source" + state: present + register: repo_art_dev + + # ================================================================== + # OPTIONAL SYNC + # ================================================================== + + - name: "Sync ocp-release repository" + redhat.satellite.repository_sync: + username: "{{ satellite_username | default(omit) }}" + password: "{{ satellite_password | default(omit) }}" + use_gssapi: "{{ satellite_use_gssapi | default(omit) }}" + server_url: "{{ satellite_url }}" + organization: "{{ satellite_organization }}" + validate_certs: "{{ satellite_validate_certs }}" + product: "{{ product_name }}" + repository: "ocp-release" + when: sync_after_create | bool + + - name: "Sync ocp-v4.0-art-dev repository" + redhat.satellite.repository_sync: + username: "{{ satellite_username | default(omit) }}" + password: "{{ satellite_password | default(omit) }}" + use_gssapi: "{{ satellite_use_gssapi | default(omit) }}" + server_url: "{{ satellite_url }}" + organization: "{{ satellite_organization }}" + validate_certs: "{{ satellite_validate_certs }}" + product: "{{ product_name }}" + repository: "ocp-v4.0-art-dev" + when: sync_after_create | bool + + # ================================================================== + # SUMMARY + # ================================================================== + + - name: Summary + debug: + msg: | + ======================================== + OCP PRODUCT PUSHED TO SATELLITE + ======================================== + Satellite: {{ satellite_url }} + Product: {{ product_name }} + Repositories: + - ocp-release ({{ vars['ocp_' + ocp_minor | replace('.', '') + '_release_tags'] | length }} release digests) + - ocp-v4.0-art-dev ({{ vars['ocp_' + ocp_minor | replace('.', '') + '_art_dev_tags'] | length }} component digests) + Download policy: immediate + Mirroring policy: additive + Sync triggered: {{ sync_after_create | bool }} + ======================================== diff --git a/ansible/requirements.yml b/ansible/requirements.yml new file mode 100644 index 000000000..2e4e49db3 --- /dev/null +++ b/ansible/requirements.yml @@ -0,0 +1,16 @@ +--- +# Ansible Galaxy Requirements +# Install with: ansible-galaxy collection install -r requirements.yml + +collections: + # PostgreSQL collection for database operations + # Used for: postgresql_query, postgresql_db modules + - name: community.postgresql + version: ">=3.0.0" + source: https://galaxy.ansible.com + + # General utilities collection + # Used for: various utility modules + - name: community.general + version: ">=8.0.0" + source: https://galaxy.ansible.com diff --git a/ansible/rollback-optimization.yml b/ansible/rollback-optimization.yml new file mode 100644 index 000000000..07a7fc5aa --- /dev/null +++ b/ansible/rollback-optimization.yml @@ -0,0 +1,224 @@ +--- +# Ansible Playbook: Rollback Tag List Bypass Optimization +# +# This playbook safely rolls back the pulp_container optimization by: +# - Restoring files from backup +# - Rolling back database migration +# - Restarting services +# - Verifying system health +# +# Usage: +# # Rollback from latest backup +# ansible-playbook -i inventory.yml rollback-optimization.yml +# +# # Rollback from specific backup +# ansible-playbook -i inventory.yml rollback-optimization.yml \ +# -e backup_path=/var/lib/pulp-backups/20260814T120000 +# +# # Dry-run +# ansible-playbook -i inventory.yml rollback-optimization.yml --check +# + +- name: Rollback pulp_container Tag List Bypass Optimization + hosts: satellite_servers + gather_facts: yes + vars: + rollback_log: "/var/log/pulp-container-optimization-rollback-{{ ansible_date_time.iso8601_basic_short }}.log" + # Use latest backup if not specified + backup_path: "{{ backup_path | default(backup_base_dir + '/latest') }}" + + tasks: + # ======================================================================== + # PRE-ROLLBACK VALIDATION + # ======================================================================== + + - name: Pre-rollback validation + tags: [always] + block: + - name: Search for pulp_container installation + shell: | + # Try multiple search methods + if [ -d /usr/lib/python3.12/site-packages/pulp_container ]; then + echo "/usr/lib/python3.12/site-packages/pulp_container" + elif [ -d /usr/lib/python3.11/site-packages/pulp_container ]; then + echo "/usr/lib/python3.11/site-packages/pulp_container" + else + find /usr -name "pulp_container" -type d 2>/dev/null | grep site-packages | head -1 + fi + register: pulp_container_search + changed_when: false + + - name: Fail if pulp_container not found + fail: + msg: "pulp_container installation not found! Cannot perform rollback." + when: pulp_container_search.stdout == "" + + - name: Set pulp_container paths + set_fact: + pulp_container_python_path: "{{ pulp_container_search.stdout }}" + pulp_container_app_path: "{{ pulp_container_search.stdout }}/app" + + - name: Display discovered pulp_container location + debug: + msg: "Found pulp_container at: {{ pulp_container_python_path }}" + + - name: Create rollback log + copy: + content: | + Pulp Container Optimization Rollback + Started: {{ ansible_date_time.iso8601 }} + Host: {{ ansible_hostname }} + User: {{ ansible_user_id }} + Backup path: {{ backup_path }} + dest: "{{ rollback_log }}" + mode: '0644' + + - name: Resolve latest backup symlink if needed + stat: + path: "{{ backup_path }}" + register: backup_stat + + - name: Update backup_path if it's a symlink + set_fact: + backup_path: "{{ backup_stat.stat.lnk_source }}" + when: backup_stat.stat.islnk + + - name: Display backup path being used + debug: + msg: "Rolling back from: {{ backup_path }}" + + - name: Check if backup exists + stat: + path: "{{ backup_path }}/BACKUP_MANIFEST.txt" + register: backup_check + + - name: Fail if backup not found + fail: + msg: "Backup not found at {{ backup_path }}. Available backups in {{ backup_base_dir }}/. List with: ansible-playbook -i inventory.yml list-backups.yml" + when: not backup_check.stat.exists + + - name: Display backup manifest + slurp: + src: "{{ backup_path }}/BACKUP_MANIFEST.txt" + register: manifest + + - name: Show backup details + debug: + msg: "{{ manifest.content | b64decode }}" + + - name: Confirm rollback + pause: + prompt: | + + ======================================== + ROLLBACK CONFIRMATION + ======================================== + You are about to rollback the pulp_container optimization. + + This will: + 1. Restore Python files from backup + 2. Rollback database migration + 3. Restart Pulp services + 4. Remove the auto_discover_cosign feature + + Backup location: {{ backup_path }} + + Press Ctrl+C and then 'A' to abort. + Press Enter to continue with rollback. + ======================================== + + when: not (auto_confirm | default(false)) + + # ======================================================================== + # PERFORM ROLLBACK + # ======================================================================== + + - name: Execute rollback + tags: [rollback] + block: + - name: Include rollback tasks + include_tasks: rollback-tasks.yml + vars: + rollback_reason: "Manual rollback requested by {{ ansible_user_id }}" + + # ======================================================================== + # POST-ROLLBACK VERIFICATION + # ======================================================================== + + - name: Post-rollback verification + tags: [always, verify] + block: + - name: Check Satellite health + shell: satellite-maintain health check --assumeyes + register: health_check + failed_when: false + + - name: Display health check results + debug: + var: health_check.stdout_lines + + - name: Verify optimization is removed + become_user: "{{ postgres_user }}" + postgresql_query: + db: "{{ pulp_db_name }}" + query: | + SELECT column_name + FROM information_schema.columns + WHERE table_name='container_containerremote' + AND column_name='auto_discover_cosign' + register: feature_check + + - name: Confirm feature removed + assert: + that: feature_check.rowcount == 0 + fail_msg: "Feature not properly removed - auto_discover_cosign column still exists" + success_msg: "Feature successfully removed" + + - name: Check Python file restoration + stat: + path: "{{ item.path }}" + register: file_check + loop: "{{ pulp_container_files_to_backup }}" + when: not (item.is_directory | default(false)) + + - name: Verify all files restored + assert: + that: item.stat.exists + fail_msg: "File {{ item.item.path }} was not restored" + loop: "{{ file_check.results }}" + when: not item.skipped | default(false) + + # ======================================================================== + # FINAL REPORT + # ======================================================================== + + - name: Rollback completed successfully + tags: [always] + pause: + seconds: 1 + prompt: | + + ======================================== + ROLLBACK COMPLETED SUCCESSFULLY + ======================================== + Timestamp: {{ ansible_date_time.iso8601 }} + Host: {{ ansible_hostname }} + Restored from: {{ backup_path }} + + Services status: All running + API status: Healthy + Database: Migration rolled back + Feature removed: auto_discover_cosign + + The system has been restored to its previous state. + + Review rollback log: {{ rollback_log }} + + The backup at {{ backup_path }} has been preserved + and can be removed manually if no longer needed. + ======================================== + + rescue: + - name: Rollback failed + fail: + msg: "ROLLBACK FAILED! System may be inconsistent. Check logs at {{ rollback_log }}. Backup intact at {{ backup_path }}" diff --git a/ansible/rollback-tasks.yml b/ansible/rollback-tasks.yml new file mode 100644 index 000000000..ad1314ce4 --- /dev/null +++ b/ansible/rollback-tasks.yml @@ -0,0 +1,127 @@ +--- +# Rollback tasks - included by both rollback playbook and deploy rescue block +# This file contains the actual rollback logic + +- name: Search for pulp_container installation (for rollback) + shell: | + # Try multiple search methods + if [ -d /usr/lib/python3.12/site-packages/pulp_container ]; then + echo "/usr/lib/python3.12/site-packages/pulp_container" + elif [ -d /usr/lib/python3.11/site-packages/pulp_container ]; then + echo "/usr/lib/python3.11/site-packages/pulp_container" + else + find /usr -name "pulp_container" -type d 2>/dev/null | grep site-packages | head -1 + fi + register: pulp_container_search_rollback + changed_when: false + when: pulp_container_app_path is not defined + +- name: Set pulp_container paths (for rollback) + set_fact: + pulp_container_python_path: "{{ pulp_container_search_rollback.stdout }}" + pulp_container_app_path: "{{ pulp_container_search_rollback.stdout }}/app" + when: pulp_container_app_path is not defined and pulp_container_search_rollback.stdout != "" + +- name: Stop pulp services for rollback + systemd: + name: "{{ item }}" + state: stopped + loop: "{{ pulp_services }}" + ignore_errors: yes + +- name: Restore Python files from backup + copy: + src: "{{ backup_path }}/{{ item.name }}" + dest: "{{ item.path }}" + remote_src: yes + mode: preserve + loop: "{{ pulp_container_files_to_backup }}" + when: not (item.is_directory | default(false)) + +- name: Restore migrations directory from backup + synchronize: + src: "{{ backup_path }}/{{ item.name }}/" + dest: "{{ item.path }}/" + archive: yes + delete: yes + delegate_to: "{{ inventory_hostname }}" + loop: "{{ pulp_container_files_to_backup }}" + when: item.is_directory | default(false) + +- name: Remove migration file if it exists + file: + path: "{{ pulp_container_app_path }}/migrations/{{ migration_file }}" + state: absent + +- name: Rollback database migration + become_user: pulp + shell: | + pulpcore-manager migrate {{ migration_app }} 0045_alter_manifest_compressed_image_size + environment: + PULP_SETTINGS: /etc/pulp/settings.py + DJANGO_SETTINGS_MODULE: pulpcore.app.settings + register: migration_rollback + failed_when: false + +- name: Display migration rollback output + debug: + var: migration_rollback.stdout_lines + when: migration_rollback.stdout_lines is defined + +- name: Restore database table from backup + become_user: "{{ postgres_user }}" + postgresql_db: + name: "{{ pulp_db_name }}" + state: restore + target: "{{ backup_path }}/container_containerremote.sql" + when: migration_rollback is failed + +- name: Verify migration rollback + become_user: "{{ postgres_user }}" + postgresql_query: + db: "{{ pulp_db_name }}" + query: | + SELECT column_name + FROM information_schema.columns + WHERE table_name='container_containerremote' + AND column_name='auto_discover_cosign' + register: column_verify + +- name: Confirm column removed + debug: + msg: "Column auto_discover_cosign removed: {{ column_verify.rowcount == 0 }}" + +- name: Restart pulp services after rollback + systemd: + name: "{{ item }}" + state: restarted + loop: "{{ pulp_services }}" + +- name: Wait for services to stabilize + wait_for: + timeout: 30 + +- name: Verify services are running + systemd: + name: "{{ item }}" + register: service_verify + failed_when: service_verify.status.ActiveState != "active" + loop: "{{ pulp_services }}" + +- name: Wait for Pulp API to be responsive + shell: curl -k -s -o /dev/null -w "%{http_code}" https://{{ ansible_host }}/pulp/api/v3/status/ + register: api_check + until: api_check.stdout == "200" + retries: 12 + delay: 5 + changed_when: false + +- name: Create rollback success marker + copy: + content: | + Rollback successful + Completed: {{ ansible_date_time.iso8601 }} + Reason: {{ rollback_reason | default('Manual rollback requested') }} + Restored from: {{ backup_path }} + dest: "{{ backup_path }}/ROLLBACK_SUCCESS.txt" + mode: '0644' diff --git a/ansible/templates/ocp-custom-product.yml.j2 b/ansible/templates/ocp-custom-product.yml.j2 new file mode 100644 index 000000000..457ccaae8 --- /dev/null +++ b/ansible/templates/ocp-custom-product.yml.j2 @@ -0,0 +1,35 @@ +--- +# Generated by generate-ocp-product.yml +# OCP minor version: {{ ocp_minor }} +# Z-stream releases included: {{ included_versions | join(', ') }} +# Generated: {{ lookup('pipe', 'date -u +%Y-%m-%dT%H:%M:%SZ') }} +# +# Append or merge this into your custom_products.yml for rhis-builder-satellite + + - name: "OpenShift Container Platform {{ ocp_minor }}" + desc: "OCP {{ ocp_minor }} release and component container images" + org: "{{ '{{' }} satellite_organization {{ '}}' }}" + label: "ocp-{{ ocp_minor | replace('.', '-') }}" + repositories: + - name: "ocp-release" + label: "ocp-release" + description: "OCP {{ ocp_minor }} release images" + docker_upstream_name: "openshift-release-dev/ocp-release" + content_type: "docker" + url: "https://quay.io" + upstream_username: "{{ '{{' }} quay_registry_username_vault {{ '}}' }}" + upstream_password: "{{ '{{' }} quay_registry_password_vault {{ '}}' }}" + download_policy: "on_demand" + include_tags: "{{ '{{' }} ocp_{{ ocp_minor | replace('.', '') }}_release_tags | join(',') {{ '}}' }}" + exclude_tags: "*-source" + - name: "ocp-v4.0-art-dev" + label: "ocp-v4_0-art-dev" + description: "OCP {{ ocp_minor }} component images" + docker_upstream_name: "openshift-release-dev/ocp-v4.0-art-dev" + content_type: "docker" + url: "https://quay.io" + upstream_username: "{{ '{{' }} quay_registry_username_vault {{ '}}' }}" + upstream_password: "{{ '{{' }} quay_registry_password_vault {{ '}}' }}" + download_policy: "on_demand" + include_tags: "{{ '{{' }} ocp_{{ ocp_minor | replace('.', '') }}_art_dev_tags | join(',') {{ '}}' }}" + exclude_tags: "*-source" diff --git a/ansible/templates/ocp-repositories.yml.j2 b/ansible/templates/ocp-repositories.yml.j2 new file mode 100644 index 000000000..e931fc517 --- /dev/null +++ b/ansible/templates/ocp-repositories.yml.j2 @@ -0,0 +1,14 @@ + +############## OpenShift Container Platform {{ ocp_minor }} ################## + + - name: "ocp-release" + product: "OpenShift Container Platform {{ ocp_minor }}" + content_type: "docker" + download_policy: "immediate" + mirroring_policy: "additive" + + - name: "ocp-v4.0-art-dev" + product: "OpenShift Container Platform {{ ocp_minor }}" + content_type: "docker" + download_policy: "immediate" + mirroring_policy: "additive" diff --git a/ansible/templates/ocp-tags.yml.j2 b/ansible/templates/ocp-tags.yml.j2 new file mode 100644 index 000000000..2082e55e7 --- /dev/null +++ b/ansible/templates/ocp-tags.yml.j2 @@ -0,0 +1,15 @@ +--- +# OCP {{ ocp_minor }} container image digest tags +# Generated by generate-ocp-product.yml +# Z-stream releases included: {{ included_versions | join(', ') }} +# Generated: {{ lookup('pipe', 'date -u +%Y-%m-%dT%H:%M:%SZ') }} + +ocp_{{ ocp_minor | replace('.', '') }}_release_tags: +{% for tag in merged_release_tags | sort %} + - "{{ tag }}" +{% endfor %} + +ocp_{{ ocp_minor | replace('.', '') }}_art_dev_tags: +{% for tag in merged_art_dev_tags | sort %} + - "{{ tag }}" +{% endfor %} diff --git a/ansible/test-prerequisites.yml b/ansible/test-prerequisites.yml new file mode 100644 index 000000000..3128ed3e2 --- /dev/null +++ b/ansible/test-prerequisites.yml @@ -0,0 +1,233 @@ +--- +# Ansible Playbook: Test Prerequisites +# +# Validates that all prerequisites are met before deployment +# +# Usage: +# ansible-playbook -i inventory.yml test-prerequisites.yml +# + +- name: Test deployment prerequisites + hosts: satellite_servers + gather_facts: yes + + tasks: + - name: Test SSH connectivity + ping: + register: ping_result + + - name: Check user privileges + command: whoami + register: whoami_result + changed_when: false + + - name: Check if running as root + set_fact: + is_root: "{{ whoami_result.stdout == 'root' }}" + + - name: Test sudo access (if not root) + command: sudo -n whoami + register: sudo_test + changed_when: false + failed_when: false + when: not is_root + + - name: Check Satellite installation + stat: + path: /usr/bin/satellite-maintain + register: satellite_cmd + + - name: Get Satellite version + command: satellite-maintain packages list + register: satellite_packages + changed_when: false + failed_when: false + when: satellite_cmd.stat.exists + + - name: Check PostgreSQL + systemd: + name: postgresql + register: postgres_service + + - name: Test database connection + become_user: postgres + postgresql_query: + db: "{{ pulp_db_name }}" + query: "SELECT version()" + register: db_version + failed_when: false + + - name: Search for pulp_container installation + shell: | + # Try multiple search methods + if [ -d /usr/lib/python3.12/site-packages/pulp_container ]; then + echo "/usr/lib/python3.12/site-packages/pulp_container" + elif [ -d /usr/lib/python3.11/site-packages/pulp_container ]; then + echo "/usr/lib/python3.11/site-packages/pulp_container" + else + find /usr -name "pulp_container" -type d 2>/dev/null | grep site-packages | head -1 + fi + register: pulp_container_search + changed_when: false + + - name: Set pulp_container paths + set_fact: + pulp_container_python_path: "{{ pulp_container_search.stdout }}" + pulp_container_app_path: "{{ pulp_container_search.stdout }}/app" + when: pulp_container_search.stdout != "" + + - name: Check pulp_container installation + stat: + path: "{{ pulp_container_app_path }}" + register: pulp_path + when: pulp_container_search.stdout != "" + + - name: Find pulp_container version + shell: | + pip3 list | grep pulp-container || rpm -qa | grep pulp_container + register: pulp_version + changed_when: false + failed_when: false + + - name: Check source repository + stat: + path: "{{ pulp_container_repo_path }}" + register: source_repo + delegate_to: localhost + + - name: Check source branch + command: git -C {{ pulp_container_repo_path }} branch --show-current + register: git_branch + delegate_to: localhost + changed_when: false + failed_when: false + when: source_repo.stat.exists + + - name: Check required files in source + stat: + path: "{{ pulp_container_repo_path }}/{{ item }}" + register: source_files + delegate_to: localhost + loop: + - pulp_container/app/models.py + - pulp_container/app/tasks/sync_stages.py + - pulp_container/app/tasks/synchronize.py + - pulp_container/app/migrations/0051_containerremote_auto_discover_cosign.py + when: source_repo.stat.exists + + - name: Check Ansible collections + command: ansible-galaxy collection list community.postgresql + register: ansible_collections + delegate_to: localhost + changed_when: false + failed_when: false + + - name: Check for existing backups + find: + paths: "{{ backup_base_dir }}" + file_type: directory + register: existing_backups + failed_when: false + + - name: Check disk space + shell: df -h {{ backup_base_dir | dirname }} | tail -1 | awk '{print $4}' + register: disk_space + changed_when: false + + # ======================================================================== + # RESULTS SUMMARY + # ======================================================================== + + - name: Prerequisites check header + debug: + msg: "======================================== PREREQUISITES CHECK RESULTS ========================================" + + - name: Check connectivity + debug: + msg: "CONNECTIVITY: {{ '✓ SSH successful' if ping_result is success else '✗ SSH failed' }} | {{ '✓ Running as root' if is_root else ('⚠ Running as ' + whoami_result.stdout + ' (sudo available)' if sudo_test is success else '✗ Not root, no sudo') }}" + + - name: Check Satellite + debug: + msg: "SATELLITE: {{ '✓ Satellite tools found' if satellite_cmd.stat.exists else '✗ Satellite not found' }}" + + - name: Check PostgreSQL + debug: + msg: "DATABASE: {{ '✓ PostgreSQL running' if postgres_service.status.ActiveState == 'active' else '✗ PostgreSQL not running' }} | {{ '✓ Database accessible' if db_version is success else '✗ Database not accessible' }}" + + - name: Check pulp_container + debug: + msg: "PULP_CONTAINER: {{ '✓ Installed at ' + pulp_container_python_path + ' (' + (pulp_container_python_path | regex_search('python3\\.[0-9]+')) + ')' if pulp_container_search.stdout != '' and pulp_path.stat.exists else '✗ Not found in /usr/lib/python3*/site-packages' }}" + + - name: Check source repository + debug: + msg: "SOURCE CODE: {{ '✓ Found at ' + pulp_container_repo_path + ' (branch: ' + git_branch.stdout + ')' if source_repo.stat.exists else '✗ Not found at ' + pulp_container_repo_path }}" + + - name: Check required source files + debug: + msg: " {{ '✓' if item.stat.exists else '✗' }} {{ item.item }}" + loop: "{{ source_files.results }}" + when: source_repo.stat.exists and source_files.results is defined + loop_control: + label: "{{ item.item }}" + + - name: Check Ansible collections + debug: + msg: "ANSIBLE: {{ '✓ community.postgresql installed' if ansible_collections.rc == 0 else '✗ community.postgresql NOT installed - run: ansible-galaxy collection install -r requirements.yml' }}" + + - name: Check backups + debug: + msg: "BACKUPS: Directory {{ backup_base_dir }} | Existing: {{ existing_backups.matched | default(0) }} | Disk space: {{ disk_space.stdout | default('unknown') }}" + + - name: Prerequisites footer + debug: + msg: "========================================================================================================" + + - name: Set overall readiness + set_fact: + ready_to_deploy: >- + {{ ping_result is success and + (is_root or sudo_test is success) and + satellite_cmd.stat.exists and + postgres_service.status.ActiveState == 'active' and + db_version is success and + pulp_path.stat.exists and + source_repo.stat.exists and + source_files.results | selectattr('stat.exists') | list | length == 4 }} + + - name: Prerequisites met - ready to deploy + pause: + seconds: 1 + prompt: | + + ======================================== + ✓ ALL PREREQUISITES MET + ======================================== + + Your system is ready for deployment! + + Next steps: + 1. Review the deployment plan: cat ansible/README.md + 2. Run a dry-run: ansible-playbook -i inventory.yml deploy-optimization.yml --check + 3. Deploy: ansible-playbook -i inventory.yml deploy-optimization.yml + + when: ready_to_deploy + + - name: Prerequisites not met + pause: + seconds: 1 + prompt: | + + ======================================== + ✗ PREREQUISITES NOT MET + ======================================== + + Please resolve the issues marked with ✗ above before deploying. + + Common fixes: + - SSH connectivity: Check firewall, SSH keys, ansible_user in inventory + - Root/sudo: Use --become flag or update ansible_user to root + - Database: Start PostgreSQL: systemctl start postgresql + - Source code: Check pulp_container_repo_path in inventory.yml + - Ansible collections: ansible-galaxy collection install -r requirements.yml + + when: not ready_to_deploy diff --git a/ansible/verify-deployment.yml b/ansible/verify-deployment.yml new file mode 100644 index 000000000..9c1506b59 --- /dev/null +++ b/ansible/verify-deployment.yml @@ -0,0 +1,151 @@ +--- +# Ansible Playbook: Verify Deployment Status +# +# Quick verification playbook to check if the optimization is deployed +# +# Usage: +# ansible-playbook -i inventory.yml verify-deployment.yml +# + +- name: Verify pulp_container optimization deployment status + hosts: satellite_servers + gather_facts: yes + + tasks: + - name: Search for pulp_container installation + shell: | + # Try multiple search methods + if [ -d /usr/lib/python3.12/site-packages/pulp_container ]; then + echo "/usr/lib/python3.12/site-packages/pulp_container" + elif [ -d /usr/lib/python3.11/site-packages/pulp_container ]; then + echo "/usr/lib/python3.11/site-packages/pulp_container" + else + find /usr -name "pulp_container" -type d 2>/dev/null | grep site-packages | head -1 + fi + register: pulp_container_search + changed_when: false + + - name: Set pulp_container paths + set_fact: + pulp_container_python_path: "{{ pulp_container_search.stdout }}" + pulp_container_app_path: "{{ pulp_container_search.stdout }}/app" + when: pulp_container_search.stdout != "" + + - name: Display pulp_container location + debug: + msg: "Found pulp_container at: {{ pulp_container_python_path | default('NOT FOUND') }}" + + - name: Check if auto_discover_cosign column exists + become_user: "{{ postgres_user }}" + postgresql_query: + db: "{{ pulp_db_name }}" + query: | + SELECT column_name, data_type, column_default + FROM information_schema.columns + WHERE table_name='container_containerremote' + AND column_name='auto_discover_cosign' + register: column_check + + - name: Check migration status + become_user: pulp + shell: | + pulpcore-manager showmigrations {{ migration_app }} | grep "0046_containerremote_auto_discover_cosign" + environment: + PULP_SETTINGS: /etc/pulp/settings.py + DJANGO_SETTINGS_MODULE: pulpcore.app.settings + register: migration_check + changed_when: false + failed_when: false + + - name: Check service status + systemd: + name: "{{ item }}" + register: service_status + loop: "{{ pulp_services }}" + + - name: Query sample remotes + become_user: "{{ postgres_user }}" + postgresql_query: + db: "{{ pulp_db_name }}" + query: | + SELECT name, includes, excludes, auto_discover_cosign + FROM container_containerremote + LIMIT 5 + register: remote_sample + + - name: Check API status + uri: + url: "https://{{ ansible_host }}/pulp/api/v3/status/" + validate_certs: no + status_code: 200 + register: api_status + failed_when: false + + - name: Display deployment status header + debug: + msg: + - "========================================" + - "DEPLOYMENT STATUS" + - "========================================" + + - name: Display database column status + debug: + msg: "Database Column: {{ 'DEPLOYED' if column_check.rowcount > 0 else 'NOT DEPLOYED' }}" + + - name: Display column details + debug: + var: column_check.query_result + when: column_check.rowcount > 0 + + - name: Display migration status + debug: + msg: "Migration Status: {{ migration_check.stdout if migration_check.rc == 0 else 'Migration not found' }}" + + - name: Display services status + debug: + msg: "{{ item.item }}: {{ item.status.ActiveState }}" + loop: "{{ service_status.results }}" + loop_control: + label: "{{ item.item }}" + + - name: Display API status + debug: + msg: "API Status: {{ 'Healthy (HTTP ' + (api_status.status | string) + ')' if api_status.status == 200 else 'Unhealthy' }}" + + - name: Display sample remotes + debug: + msg: "Sample Remotes:" + + - name: Show remote details + debug: + var: remote_sample.query_result + + - name: Status footer + debug: + msg: "========================================" + + - name: Set deployment fact + set_fact: + is_deployed: "{{ column_check.rowcount > 0 }}" + + - name: Summary + debug: + msg: "========== Optimization is {{ 'DEPLOYED' if is_deployed else 'NOT DEPLOYED' }} ==========" + + - name: Display deployed instructions + pause: + seconds: 1 + prompt: | + + The tag list bypass optimization is ACTIVE on this system. + All container remotes have auto_discover_cosign enabled by default. + + To test: Create a remote with specific includes (no wildcards) + and sync - it should bypass /tags/list. + + when: is_deployed + + - name: Display not deployed instructions + debug: + msg: "Optimization NOT DEPLOYED. To deploy: ansible-playbook -i inventory.yml deploy-optimization.yml" + when: not is_deployed diff --git a/pulp_container/app/migrations/0046_containerremote_auto_discover_cosign.py b/pulp_container/app/migrations/0046_containerremote_auto_discover_cosign.py new file mode 100644 index 000000000..2791d5daa --- /dev/null +++ b/pulp_container/app/migrations/0046_containerremote_auto_discover_cosign.py @@ -0,0 +1,16 @@ +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ("container", "0045_alter_manifest_compressed_image_size"), + ] + + operations = [ + migrations.AddField( + model_name="containerremote", + name="auto_discover_cosign", + field=models.BooleanField(default=True), + ), + ] diff --git a/pulp_container/app/models.py b/pulp_container/app/models.py index d42f74a33..44033ce70 100644 --- a/pulp_container/app/models.py +++ b/pulp_container/app/models.py @@ -490,6 +490,7 @@ class ContainerRemote(Remote, AutoAddObjPermsMixin): include_tags = fields.ArrayField(models.TextField(null=True), null=True) exclude_tags = fields.ArrayField(models.TextField(null=True), null=True) sigstore = models.TextField(null=True) + auto_discover_cosign = models.BooleanField(default=True) TYPE = "container" diff --git a/pulp_container/app/tasks/sync_stages.py b/pulp_container/app/tasks/sync_stages.py index efd2be7fc..0be24c6b8 100644 --- a/pulp_container/app/tasks/sync_stages.py +++ b/pulp_container/app/tasks/sync_stages.py @@ -52,12 +52,13 @@ class ContainerFirstStage(Stage): """ - def __init__(self, remote, signed_only): + def __init__(self, remote, signed_only, mirror=False): """Initialize the stage.""" super().__init__() self.remote = remote self.deferred_download = self.remote.policy != Remote.IMMEDIATE self.signed_only = signed_only + self.mirror = mirror self.tag_dcs = [] self.manifest_list_dcs = [] @@ -111,12 +112,90 @@ async def _check_for_existing_manifest(self, download_tag): return content_data, raw_text_data, response + def _can_bypass_taglist(self): + """ + Check if we can safely bypass /tags/list enumeration. + + Returns True only if: + - include_tags contains ONLY sha256 digests (not tag names) + - exclude_tags is empty OR won't match any includes (harmless) + - No wildcards in include_tags (need exact refs, not patterns) + - Not in mirror mode (would need full list to detect removals) + + IMPORTANT: We can only bypass when syncing digests, not tag names. + When syncing tag names (e.g., "manifest_a"), we need /tags/list to + get their digests for cosign companion discovery. + """ + include_tags = self.remote.include_tags or [] + exclude_tags = self.remote.exclude_tags or [] + + if not include_tags: + return False + + # CRITICAL: Only bypass if ALL includes are sha256 digests + # Tag names need /tags/list to resolve their digests + if not all(tag.startswith("sha256:") for tag in include_tags): + return False + + # If excludes exist, check if they're harmless (won't match our includes) + if exclude_tags: + # Satellite often adds '*-source' exclude which doesn't match sha256 digests + harmless_excludes = all( + exclude.endswith("-source") + or ( + exclude.startswith("*") + and not any(tag.endswith(exclude.lstrip("*")) for tag in include_tags) + ) + for exclude in exclude_tags + ) + if not harmless_excludes: + return False + + if self.mirror: + return False + + wildcard_chars = ["*", "?", "["] + includes_str = "".join(include_tags) + if any(char in includes_str for char in wildcard_chars): + return False + + return True + async def run(self): """ ContainerFirstStage. """ signature_source = await self.get_signature_source() + # Optimization: if syncing specific refs (no wildcards, no excludes), + # skip expensive /tags/list enumeration and sync them directly + if self._can_bypass_taglist(): + log.info( + "Bypassing /tags/list enumeration - syncing %d explicit references directly", + len(self.remote.include_tags), + ) + await self._process_tags( + self.remote.include_tags, signature_source, msg="Processing Manifests" + ) + + # Auto-discover cosign companion tags if enabled + if getattr(self.remote, "auto_discover_cosign", True): + log.info("Auto-discovering cosign companion tags via HEAD probing") + companion_tags = await self._discover_cosign_companions_without_taglist( + self._synced_digests + ) + if companion_tags: + log.info( + "Found %d cosign companion tag(s) for synced manifests", + len(companion_tags), + ) + await self._process_tags( + companion_tags, + signature_source, + msg="Processing Cosign Companion Tags", + ) + return + async with ProgressReport( message="Downloading tag list", code="sync.downloading.tag_list", total=1 ) as pb: @@ -150,6 +229,62 @@ async def run(self): companion_tags, signature_source, msg="Processing Cosign Companion Tags" ) + async def _tag_exists(self, tag_name): + """Check if a tag exists via lightweight HEAD request.""" + from urllib.parse import urljoin + + relative_url = "/v2/{name}/manifests/{tag}".format( + name=self.remote.namespaced_upstream_name, tag=tag_name + ) + manifest_url = urljoin(self.remote.url, relative_url) + downloader = self.remote.get_downloader(url=manifest_url) + + try: + await downloader.run(extra_data={"headers": V2_ACCEPT_HEADERS, "http_method": "HEAD"}) + return True + except Exception: + return False + + async def _discover_cosign_companions_without_taglist(self, synced_digests): + """ + Discover cosign companion tags by probing expected patterns via HEAD requests. + + Used when bypassing /tags/list to avoid expensive enumeration. + Probes for known cosign patterns: + - V2: sha256-.sig, sha256-.att, sha256-.sbom + - V3: sha256- (71 chars, verified via manifest check) + """ + companion_tags = [] + semaphore = asyncio.Semaphore(20) # Limit concurrent probes + + async def probe_tag(tag): + async with semaphore: + if await self._tag_exists(tag): + return tag + return None + + # Build list of potential cosign tags to probe + candidates = [] + for digest in synced_digests: + # digest format: "sha256:abc123..." + if not digest.startswith("sha256:"): + continue + digest_hex = digest.split(":", 1)[1] + + # V2 cosign patterns + candidates.append(f"sha256-{digest_hex}.sig") + candidates.append(f"sha256-{digest_hex}.att") + candidates.append(f"sha256-{digest_hex}.sbom") + + # V3 cosign pattern (71 chars total) + candidates.append(f"sha256-{digest_hex}") + + # Probe all candidates concurrently + results = await asyncio.gather(*[probe_tag(tag) for tag in candidates]) + companion_tags = [tag for tag in results if tag] + + return companion_tags + def _find_cosign_companion_tags(self): """Find cosign companion tags for synced digests.""" companion_tags = [] diff --git a/pulp_container/app/tasks/synchronize.py b/pulp_container/app/tasks/synchronize.py index 26f2560dc..fee8a9bc0 100644 --- a/pulp_container/app/tasks/synchronize.py +++ b/pulp_container/app/tasks/synchronize.py @@ -36,7 +36,7 @@ def synchronize(remote_pk, repository_pk, mirror, signed_only): remote = ContainerRemote.objects.get(pk=remote_pk) repository = ContainerRepository.objects.get(pk=repository_pk) log.info("Synchronizing: repository={r} remote={p}".format(r=repository.name, p=remote.name)) - first_stage = ContainerFirstStage(remote, signed_only) + first_stage = ContainerFirstStage(remote, signed_only, mirror=mirror) dv = ContainerDeclarativeVersion(first_stage, repository, mirror) return dv.create() diff --git a/pulp_container/tests/unit/test_sync_stages.py b/pulp_container/tests/unit/test_sync_stages.py index 835347cb7..95199889b 100644 --- a/pulp_container/tests/unit/test_sync_stages.py +++ b/pulp_container/tests/unit/test_sync_stages.py @@ -152,5 +152,125 @@ async def test_has_cosign_signature_false_when_no_cosign_tags(self): self.assertFalse(await self.stage._has_cosign_signature(digest)) +class TestBypassTaglistOptimization(unittest.IsolatedAsyncioTestCase): + """Test bypass logic for skipping /tags/list enumeration.""" + + def setUp(self): + remote = MagicMock() + remote.policy = MagicMock() + remote.namespaced_upstream_name = "library/test" + remote.url = "https://registry.example/" + remote.get_downloader = MagicMock() + remote.include_tags = None + remote.exclude_tags = None + remote.auto_discover_cosign = True + + self.stage = ContainerFirstStage(remote=remote, signed_only=False, mirror=False) + + def test_can_bypass_with_specific_digests_only(self): + """Bypass activates when include_tags contains only sha256 digests.""" + self.stage.remote.include_tags = [ + "sha256:abc123", + "sha256:def456", + ] + self.stage.remote.exclude_tags = None + self.assertTrue(self.stage._can_bypass_taglist()) + + def test_cannot_bypass_without_includes(self): + """Bypass does not activate when include_tags is empty.""" + self.stage.remote.include_tags = None + self.stage.remote.exclude_tags = None + self.assertFalse(self.stage._can_bypass_taglist()) + + def test_cannot_bypass_with_tag_names(self): + """Bypass does not activate when include_tags contains tag names (not digests).""" + self.stage.remote.include_tags = ["manifest_a", "latest"] + self.stage.remote.exclude_tags = None + self.assertFalse(self.stage._can_bypass_taglist()) + + def test_cannot_bypass_with_mixed_digests_and_tag_names(self): + """Bypass does not activate when include_tags mixes digests and tag names.""" + self.stage.remote.include_tags = ["sha256:abc123", "manifest_a"] + self.stage.remote.exclude_tags = None + self.assertFalse(self.stage._can_bypass_taglist()) + + def test_cannot_bypass_with_wildcards(self): + """Bypass does not activate when include_tags contains wildcards.""" + self.stage.remote.include_tags = ["v4.0*", "sha256:abc123"] + self.stage.remote.exclude_tags = None + self.assertFalse(self.stage._can_bypass_taglist()) + + def test_cannot_bypass_in_mirror_mode(self): + """Bypass does not activate in mirror mode.""" + self.stage.mirror = True + self.stage.remote.include_tags = ["sha256:abc123"] + self.stage.remote.exclude_tags = None + self.assertFalse(self.stage._can_bypass_taglist()) + + def test_can_bypass_with_harmless_excludes(self): + """Bypass activates when excludes won't match sha256 includes.""" + self.stage.remote.include_tags = ["sha256:abc123", "sha256:def456"] + self.stage.remote.exclude_tags = ["*-source"] + self.assertTrue(self.stage._can_bypass_taglist()) + + def test_cannot_bypass_with_harmful_excludes(self): + """Bypass does not activate when excludes could match digest includes.""" + self.stage.remote.include_tags = ["sha256:abc123", "sha256:def456"] + self.stage.remote.exclude_tags = ["sha256:*"] + self.assertFalse(self.stage._can_bypass_taglist()) + + def test_cannot_bypass_with_overlapping_excludes(self): + """Bypass does not activate when exclude pattern matches an include.""" + self.stage.remote.include_tags = ["sha256:abc123", "v4.0-source"] + self.stage.remote.exclude_tags = ["*-source"] + self.assertFalse(self.stage._can_bypass_taglist()) + + async def test_tag_exists_returns_true_on_200(self): + """_tag_exists returns True when HEAD request succeeds.""" + downloader = MagicMock() + mock_result = AsyncMock() + mock_result.status_code = 200 + downloader.run = AsyncMock(return_value=mock_result) + self.stage.remote.get_downloader.return_value = downloader + + result = await self.stage._tag_exists("test-tag") + self.assertTrue(result) + + async def test_tag_exists_returns_false_on_404(self): + """_tag_exists returns False when tag doesn't exist.""" + downloader = MagicMock() + downloader.run = AsyncMock(side_effect=Exception("404")) + self.stage.remote.get_downloader.return_value = downloader + + result = await self.stage._tag_exists("missing-tag") + self.assertFalse(result) + + async def test_discover_cosign_companions_probes_variants(self): + """Discover cosign companions probes .sig, .att, .sbom variants.""" + synced_digests = {"sha256:abc123"} + self.stage._synced_digests = synced_digests + + # Mock _tag_exists to return True for .sig variant only + async def mock_tag_exists(tag): + return tag == "sha256-abc123.sig" + + self.stage._tag_exists = AsyncMock(side_effect=mock_tag_exists) + + companions = await self.stage._discover_cosign_companions_without_taglist(synced_digests) + + self.assertEqual(len(companions), 1) + self.assertEqual(companions[0], "sha256-abc123.sig") + + async def test_discover_cosign_companions_returns_empty_when_none_exist(self): + """Discover returns empty list when no companion tags exist.""" + synced_digests = {"sha256:abc123"} + self.stage._synced_digests = synced_digests + self.stage._tag_exists = AsyncMock(return_value=False) + + companions = await self.stage._discover_cosign_companions_without_taglist(synced_digests) + + self.assertEqual(len(companions), 0) + + if __name__ == "__main__": unittest.main()