[pigeonligh]

⚠️ Breaking upgrade notice — Task data structure is not compatible

This change redesigns the Task contract and several other API contracts in a backward-incompatible way. Do not upgrade an existing environment in place without confirming your upgrade path first. Existing Task/Job/Workflow objects written with the old schema will not validate or reconcile against the new API.

• Recommended path: stand up a new environment (new cluster / new installation) on this version and recreate workloads from the new manifests. Treat the old environment as read-only until migration is verified.
• If you must upgrade in place: confirm the preconditions below, back up and export all existing Task/Job/Workflow objects and archived data, and rewrite every manifest to the new Task spec before the first reconcile. Expect old Task objects to fail closed (they will not be silently adopted or converted).
• Rollback: because the new schema removes old fields, a downgrade is not lossless. Snapshot the persistence backend and CRDs before upgrading.

Description

This change consolidates a period of internal iteration. The headline themes:

1. Runtime-neutral Task execution architecture (data plane)
A new internal framework (apps/rlark/pkg/agent/execution/**) splits one Task reconcile into orthogonal, testable layers: host (claim/finalizer, one-Task serialization, the only writer of Agent-owned Task status), coordination/coordinator (selection, compile→bind→observe→diff→apply), workload (runtime-neutral Adapter contracts), binding, environment (declaration-vs-effect, allow-listed mutation targets, trusted handler ordering, a restricted secret-material lease), runtime (Provider, ownership, typed reconcile profiles), plan, trigger, evidence (conservative fast-observation gate), observability, fault, and gRPC transports (protocol v2). Concrete Kubernetes implementations live in apps/rlark/pkg/agent/controllers/task/**; the framework never imports them.

2. Task execution contract redesign (BREAKING)
TaskSpec is rewritten:

• Added: spec.runtime{kind,name}, spec.workload{kind,version,parameters,carrier{type,version,parameters},template}, spec.placement.nodeSelector, spec.environments[]{type,version,parameters}, spec.suspended; status.observedGeneration.
• Removed: spec.agentType, top-level spec.nodeSelector, spec.domain, spec.kubernetes.*, spec.docker.*, spec.raw.*, spec.tensorBoardDir, spec.prepareScript, spec.runScript, spec.sshPublicKey, and the deprecated pvcStorageMap/pvcSizeGbMap. Ray becomes workload rlinf.io/ray@v1 with a Kubernetes carrier; SSH/network/TensorBoard become environments.
• Also breaking: Job.spec.stopped → suspended, Workflow.spec.stopped → suspended.
• Runtime/Workload/Carrier selectors are frozen at claim time; changing them is a non-destructive WaitingForChange/Blocked rather than an in-place migration (delete and recreate the Task).

3. Unified Phase/Condition/ObservedGeneration contract
New api/rlark.io/v1alpha1/conditions.go and consts.go; status.conditions (map-keyed) added across Job/Workflow/Task/Domain/Addon/Node/Cluster. ObservedGeneration added where a Spec-derived verdict exists and is documented as a controller contract (not an API-server guarantee). Controllers and the Agent now write conditions through meta.SetStatusCondition with stable reasons.

4. Cluster as a first-class resource
New cluster-scoped Cluster CRD with spec.agentNamespace as the authoritative membership scope, durable cordoned/draining intent, and a heartbeat Lease-derived phase (Online/Offline/Downgraded/Cordoned/Draining) plus a Ready condition. Node aggregation in the Gateway is now namespace-membership based; Cluster write paths are removed from the Gateway.

5. Tenant-aware business resources and server-side enforcement
Tenant identity is carried in metadata (rlark.io/tenant-id, rlark.io/created-by) with inheritance Workflow→Job→Task→Pod; the Gateway stamps identity on create, preserves it on update, strips it on patch, and enforces scope on every entry point (list narrowing, 403 for out-of-scope filters, 404 for other tenants' named objects, no-assignment refusal). Read-only users/tenants routes added. Write-path quotas for Job tags enforced in the Gateway (deliberately not in the CRD schema).

6. Gateway/DB hardening
Unified ErrorResponse {code,message,reason,requestId,details} with a closed machine-readable code set and X-Request-Id middleware; unified list filter/pagination semantics with strict validation (400 instead of silent coercion), maxListLimit=500, deterministic ordering, correct != missing-key semantics (IS DISTINCT FROM), parameterized JSON-path SQL, and removal of synthesized UIDs (missing metadata.uid now stays empty). A new swagger consistency gate ties routes, error-code enums, and $refs together.

7. Job archive and persistence
Finished Jobs are persisted and then archived (deleted from the management API after ArchiveTTL, default 5m) with a new POST /jobs/{name}/start to restore them (new UID, same name/spec). Pluggable persistence backends (PostgreSQL / Secret). Removed the GET /jobs/{name}/metrics stub route.

8. StorageClass moved to the distribution model
StorageClasses are now management-plane declarations distributed to target clusters (declaration vs. observed delivery), rather than live per-agent query.

9. UI typed CRD models
TypeScript models are generated from the CRD bases (scripts/generate-crd-models.mjs) with a kind-set contract check and committed output under src/generated/**; hand-written CRD interfaces were removed. UI follows the new Task/Cluster/Job contracts (suspended, /start, new cluster phases).

10. Proto toolchain pin, execution protocol, SDKs
Pinned protoc/plugins/buf via proto/toolchain.mk with protoc-check, buf lint, and buf breaking gates; new proto/execution (protocol v2) and sdks/execution-go; embodied-runtime stubs regenerated; Python SDK floors raised (grpcio>=1.84.0, protobuf>=7.35.1,<8, Python ≥3.10).

11. e2e framework
A product-neutral, configurable Kind-based e2e framework (test/e2e/**) with a provider abstraction, scenario runner, ownership-proven cleanup, and an RLark smoke scenario (make test-e2e-rlark).

12. Documentation
New/updated EN+zh pages: design-principles, api-conventions, runtime-provider, tenancy, e2e-testing, project-layout, agent admin guide, concepts, architecture, storage-api, clusters-nodes, jobs, ssh-keys, configuration, CRD reference; new verify-generated and proto CI gates.

Motivation and Context

The previous Task model collapsed runtime, workload, and environment concerns into agentType-specific fields, hard-coded Ray/Kubernetes lifecycle into shell scripts, derived cluster state ad hoc from Node records, and exposed tenant identity/authorization only in the UI. As more runtimes, workloads, environments, tenants, and nodes are added, this prevented clean extension and made correctness depend on event ordering and implicit ownership.

This change moves the code toward the documented Project Design Principles: reconcile from state, separate intent/observation/operations/history, prove ownership, evolve contracts conservatively, enforce tenancy at trusted boundaries, and test failure and recovery. It also makes the Task structure explicitly runtime-neutral so Docker/Raw and additional workloads can be added without redesigning the core contract — at the cost of a hard, intentional break with the old Task/Job schema.

How has this been tested?

• make test-go / make test-web and make lint-go / make lint-web for the repository suites.
• Focused execution-framework tests, including -race: go test ./pkg/agent/execution/... and go test -race ./pkg/agent/execution/....
• New failure/recovery tests for duplicate/lost events, conflict/stale updates, restart reconstruction, partial external success, delayed/absent deletion, ownership conflicts, cache immutability, cancellation/timeout, tenant/authorization denial, and drift self-healing.
• Gateway: swagger consistency tests (route↔spec, error-code enum, $ref resolution), list/selector validation and != semantics tests, migration ordering/reversibility tests, and deterministic-ordering tests.
• Controller: offline-node stopped-Job convergence and adopted-Task backfill tests; condition/ObservedGeneration behavior tests.
• Generated-artifact gates: make verify-generated (CRDs, clients, docs, UI models) and make verify-proto-generated.
• Proto gates: make lint-proto and make proto-breaking.
• E2E: make test-e2e-rlark Kind smoke scenario (install, discovery, deployment availability, uninstall).
• Upgrade validation: manifests were rewritten to the new Task spec; old-schema Task objects were verified to fail closed rather than be silently adopted.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Documentation update
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have added tests to cover my changes.
• All new and existing tests passed.