docs: add operator runbooks and update climate-ref to v0.18.1 - #60
Conversation
📝 WalkthroughWalkthroughChangesDeployment operations
Merge Risk: 🟡 Moderate · up to Following the runbooks can produce an incomplete deployment or send operators down incorrect troubleshooting paths. These operational instructions should be corrected before merge. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 1 files. (12 skipped: 12 unsupported.)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
lewisjared
changed the title
Fixes the prose defects in the new operator docs. - Restores the "Who writes what" heading, because "Volumes" nested under "Required volumes" said the same thing twice. - Rewrites the sentence about `REF_CONFIGURATION`, which had lost a word and carried a typo. - Replaces the truncated obs4MIPs note with what to download and where. - Uses the `ref-orch` alias for the health check rather than repeating the full `kubectl exec` line.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Team
Run ID: f37c11ba-3be7-4e8c-a204-2ba617929359
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (13)
README.mdchangelog/60.docs.mdchangelog/60.improvement.mddocker/docker-compose.yamldocs/runbooks/bootstrap-a-deployment.mddocs/runbooks/run-and-triage-a-solve.mdhelm/Chart.yamlhelm/README.mdhelm/examples/small-values.yamlhelm/values.yamlpyproject.tomltests/test_helm_render.pyversions.toml
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.
| After the install every pod should be Running, and the API answers its health check: | ||
|
|
||
| ```bash | ||
| ref-orch python3 -c "import urllib.request as u; print(u.urlopen('http://$RELEASE-climate-ref-aft-api/api/v1/utils/health-check/').read())" |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🔎 Supported by static analysis
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- target excerpt ---'
cat -n docs/runbooks/bootstrap-a-deployment.md | sed -n '60,84p'
printf '%s\n' '--- relevant urllib usage in the runbook ---'
rg -n -C 2 'urlopen|health-check|bootstrap' docs/runbooks/bootstrap-a-deployment.mdRepository: Climate-REF/climate-ref-aft
Length of output: 2134
🤖 get_repo_knowledge executed:
get_repo_knowledge Climate-REF/climate-ref-aft /tmp/coderabbit-repo-knowledge/climate-ref-climate-ref-aft-76c847f7/architecture
Length of output: 23616
Bound the health-check wait.
Pass timeout=10 to urllib.request.urlopen on line 74. Without it, an unresponsive API can block bootstrap indefinitely.
| ref-orch sh -c 'setsid nohup ref datasets fetch-data --registry obs4ref --output-directory /ref/data/obs4ref > /ref/log/fetch-obs4ref.log 2>&1 < /dev/null &' | ||
| ref-orch ref datasets ingest --source-type obs4ref /ref/data/obs4ref |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Wait for fetch-data to complete before ingestion.
Line 114 backgrounds fetch-data and returns immediately. Line 115 then walks the same directory without checking fetch completion. The ingest can record only the files copied so far, which can leave the deployment without required observation datasets. Run the fetch in the foreground, or add a durable completion and polling step before starting ingestion.
| A queue that stays non-zero while its worker shows nothing active is the picture to recognise. | ||
| The worker is not consuming. |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🔎 Supported by static analysis
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- runbook excerpt ---'
sed -n '60,75p' docs/runbooks/run-and-triage-a-solve.md
printf '%s\n' '--- broker_state.py location ---'
fd -t f 'broker_state\.py$' .
printf '%s\n' '--- broker_state.py excerpt ---'
file=$(fd -t f 'broker_state\.py$' . | head -n 1)
sed -n '1,240p' "$file"
printf '%s\n' '--- related usage ---'
rg -n -C 3 'broker_state|reserved|active' docs scripts tests 2>/dev/null | head -n 240Repository: Climate-REF/climate-ref-aft
Length of output: 5655
🤖 get_repo_knowledge executed:
get_repo_knowledge Climate-REF/climate-ref-aft /tmp/coderabbit-repo-knowledge/climate-ref-climate-ref-aft-76c847f7/architecture /tmp/coderabbit-repo-knowledge/climate-ref-climate-ref-aft-76c847f7/learnings
Length of output: 25410
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- Celery dependency pins ---'
rg -n -C 2 'celery|climate-ref-celery' pyproject.toml uv.lock versions.toml 2>/dev/null | head -n 160
printf '%s\n' '--- app binding ---'
rg -n -C 4 'from .* import app|app\s*=|def celery_inspect|inspect\(' scripts pyproject.toml 2>/dev/null | head -n 200Repository: Climate-REF/climate-ref-aft
Length of output: 3975
🌐 Web query:
Celery documentation inspect active reserved tasks prefetch reserved method
💡 Result:
In Celery, the inspect tool and its associated methods are used to query the status and state of active workers without altering them [1]. Inspecting Tasks To inspect worker tasks, use the celery inspect command line utility [2][3]. Active Tasks: Use the celery inspect active command to list tasks currently being executed by the workers [2][4]. Programmatically, this corresponds to the inspect.active method [1]. Reserved Tasks: Use the celery inspect reserved command to list tasks that have been received (prefetched) by the worker but are still waiting in the queue to be executed [3][5]. This command does not include active tasks or tasks with an ETA/countdown [6][3]. Programmatically, this corresponds to the inspect.reserved method [1][7]. Querying Specific Tasks: Use the celery inspect query_task ... command to retrieve specific information about tasks by their unique identifiers [2][4][3]. Prefetching Concepts Prefetching is the mechanism by which a worker reserves tasks (messages) from the broker before they are explicitly requested for execution [8][9]. Prefetch Limit: This defines how many unacknowledged tasks a worker is allowed to hold in memory [8]. It is controlled by the worker_prefetch_multiplier setting (default is the multiplier multiplied by the concurrency slot count) [8][9]. Custom Inspection/Control: While Celery provides built-in inspect commands, you can extend these by writing custom inspect_command or control_command functions [5]. For example, developers often implement custom commands to inspect the current prefetch count (e.g., via state.consumer.qos.value) or to increment it dynamically [5]. Key Distinctions Active vs. Reserved: Active tasks are currently running. Reserved tasks are prefetched and waiting to run [6][3][5]. Inspect vs. Control: Inspect commands have no side effects and only return information [1]. Control commands perform actions that may change worker behavior (e.g., enabling events or modifying prefetch settings) [1][5].
Citations:
- 1: https://docs.celeryq.dev/en/main/reference/celery.app.control.html
- 2: https://docs.celeryq.dev/en/stable/userguide/monitoring.html
- 3: https://docs.celeryq.dev/en/v5.4.0/userguide/monitoring.html
- 4: https://docs.celeryq.dev/en/latest/userguide/monitoring.html
- 5: https://docs.celeryq.dev/en/stable/userguide/workers.html
- 6: https://docs.celeryq.dev/en/stable/%5Fmodules/celery/worker/control.html
- 7: https://docs.celeryq.dev/en/latest/%5Fmodules/celery/app/control.html
- 8: https://docs.celeryq.dev/en/stable/userguide/optimizing.html
- 9: https://docs.celeryq.dev/en/latest/userguide/optimizing.html?highlight=prefetch
Check reserved tasks before declaring a worker stalled.
If active is empty, inspect reserved before declaring that the worker is not consuming. Celery reports prefetched tasks as reserved while they wait for execution.
| kubectl -n $NS logs deploy/$RELEASE-climate-ref-aft-esmvaltool --tail=200 | ||
| ``` |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Select the log for the failed provider.
This command always reads the ESMValTool deployment. A PMP or ILAMB failure therefore sends the operator to an unrelated worker log. Use the provider from the execution record.
Proposed fix
-kubectl -n $NS logs deploy/$RELEASE-climate-ref-aft-esmvaltool --tail=200
+PROVIDER=esmvaltool # Set this to esmvaltool, pmp, or ilamb.
+kubectl -n $NS logs deploy/$RELEASE-climate-ref-aft-$PROVIDER --tail=200📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| kubectl -n $NS logs deploy/$RELEASE-climate-ref-aft-esmvaltool --tail=200 | |
| ``` | |
| PROVIDER=esmvaltool # Set this to esmvaltool, pmp, or ilamb. | |
| kubectl -n $NS logs deploy/$RELEASE-climate-ref-aft-$PROVIDER --tail=200 |
| A queue with a task and no consumer is an orphan. | ||
| That happens when `celeryRoutes` names a queue no instance lists under `queues`. |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Delay orphan classification during KEDA scale-up. When KEDA is enabled, a provider can run with zero replicas and remain without a consumer until the next polling cycle starts a pod. Check the ScaledObject status and wait for the configured pollingInterval before classifying the queue as orphaned.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| A queue with a task and no consumer is an orphan. | |
| That happens when `celeryRoutes` names a queue no instance lists under `queues`. | |
| During KEDA scale-up, a queued task can have no consumer temporarily. | |
| Treat it as orphaned only after the polling interval, or when | |
| `celeryRoutes` names a queue no instance lists under `queues`. |
1 participant
Adds two operator runbooks under
docs/runbooks/and a small example values file they are written against, and moves the pinned climate-ref components to v0.18.1.bootstrap-a-deployment.mdtakes the chart fromhelm installto a smoke solve.run-and-triage-a-solve.mdcovers running a solve, watching it, and the stuck, wedged, OOM and orphan-queue cases.helm/examples/small-values.yamlis a single-node, SQLite, one-worker-per-provider starting point. The chart README now points at it instead of carrying its own copy, and the render tests cover it.Both runbooks were rehearsed step by step on a fresh release on
research-ovh-gra-1(namespaceclimate-ref-aft-test).Summary by CodeRabbit
New Features
Improvements
obs4refsource type.Documentation