CICDRepairEnv: The Professional Manual

CICDRepairEnv is a rigorous, industrial-grade Reinforcement Learning (RL) environment designed to evaluate autonomous agents in the complex domain of CI/CD pipeline repair. It provides a standardized benchmark for diagnosing failures from unstructured logs and executing sequential remediation actions.

1. Motivation & Real-World Utility

In modern software engineering, CI/CD pipelines are the backbone of rapid delivery. However, failures occur frequently due to dependency drifts, cache corruptions, and environment mismatches. Manually diagnosing these from thousands of lines of logs is time-consuming. CICDRepairEnv models these genuine challenges, offering a platform to train agents that can:

• Parse unstructured multi-line stack traces.
• Execute interdependent repair actions in strict order.
• Exploit external memory banks for specialized patches.
• Maintain stability by avoiding destructive state mutations.

---

2. Technical Specification

2.1 Interface & Compliance

The environment strictly follows the OpenEnv v2.0.0 specification, providing a standard step() / reset() / state() API with Pydantic v2 typed models.

2.2 Action Space

Agents interact via 8 discrete remediation actions:

ID	Action	Description	Destructive
0	restart_step	Restarts the current failing pipeline stage.	No
1	install_dependency	Corrects ModuleNotFoundError by installing missing packages.	No
2	change_version	Resolves version conflicts by pinning specific releases.	No
3	set_env_variable	Configures environment flags (e.g., ABI/Compiler flags).	No
4	clear_cache	Purges stale build or package caches.	No
5	rollback	Reverts the last state mutation.	Yes (Penalty)
6	use_memory_fix	Applies a patch retrieved from the memory bank.	No
7	ignore_continue	bypasses the error (risky).	Yes (Penalty)

2.3 Observation Space

At each timestep, the environment emits a structured observation:

• pipeline_stage: Active execution phase.
• failure_log: Raw stdout/stderr traces (can be procedurally generated).
• error_type: High-level error classification.
• available_actions: List of available action IDs (always [0-7]).
• memory_hints: Relevant context from the memory bank.
• progress_pct: Current completion scalar [0.0, 1.0].
• task_score: Final normalized score [0.01, 0.99] (emitted on done=True).

---

3. Benchmark Tiers & Results

3.1 Tier Definitions

Tier	Difficulty	Failure Vector	Optimal Steps
Tier 1	Easy	ModuleNotFoundError	1
Tier 2	Medium	Cache version conflict	2
Tier 3	Hard	GCC ABI/Linker mismatch	3

3.2 Verified Baseline Results

The following scores were produced by the rule-based baseline_agent using deterministic evaluation:

Tier	Final Normalized Score
Tier 1 (Easy)	0.9704
Tier 2 (Medium)	0.9655
Tier 3 (Hard)	0.9606
Average Score	0.9655

All scores are normalized to the OpenEnv standard range of [0.01, 0.99].

---

4. Advanced Operational Modes

4.1 Stochastic Mode

Controlled via StochasticConfig(sigma=...). At sigma > 0, the environment introduces:

• Intermittent Failures: Correct actions may transiently fail (simulates network blips). The step is not consumed — the agent can retry without penalty.
• Action Corruption: Flaky runners may execute a different action than requested.
• Log Noise: Irrelevant warning lines are injected to challenge the agent's parsing.

4.2 Procedural Log Generation

All logs are procedurally generated by default. The engine randomizes package names, versions, and paths while preserving the semantic markers of the error, preventing agents from overfitting to static strings. When sigma=0, a fixed seed of 42 is used for deterministic reproducibility.

---

5. Installation & Usage

Standard Setup

git clone https://github.com/your-repo/cicd_repair_env.git
cd cicd_repair_env
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Running the Baseline

Verify your environment setup by running the baseline agent:

python run_baseline.py

Starting the Server (Gradio + API)

python server/app.py

Access the interactive UI at http://localhost:7860 or use the Headless API at /reset, /step, and /state.

Containerized Execution

docker build -t cicd-repair-env .
docker run --rm -p 7860:7860 cicd-repair-env

---

6. Project Structure

• env/: Core environment logic, models, and task definitions.
• server/: FastAPI + Gradio deployment layer.
• tests/: Comprehensive unit and integration testing suite.
• scripts/: Submission validation tooling.
• inference.py: Standard LLM agent evaluation script.
• grader.py: Programmatic evaluation harness.
• run_baseline.py: Rule-based baseline agent.
• openenv.yaml: OpenEnv metadata and task specifications.