diff --git a/.github/.copilot/breadcrumbs/2026-08-25-1758-gep-4894-backend-evaluation.md b/.github/.copilot/breadcrumbs/2026-08-25-1758-gep-4894-backend-evaluation.md new file mode 100644 index 00000000..40d367bb --- /dev/null +++ b/.github/.copilot/breadcrumbs/2026-08-25-1758-gep-4894-backend-evaluation.md @@ -0,0 +1,128 @@ +# GEP-4894 Backend Evaluation + +## Requirements + +- Evaluate GEP-4894's proposed Gateway API `Backend` resource and Kubernetes YAML shape. +- Compare it with `rchinchani/afd-global-ingress-rfc` at `bb6ed4e6ba859d4895c9647568e792905ff70038`. +- Compare it with `rchinchani/gep-1748-gateway-api` at `7bf9918ce41b10ae268a9acfdd21193cd92411ad`. +- Determine whether GEP-4894 can accomplish the same global ingress, multi-cluster backend, Azure Front Door, public origin, and Private Link requirements. +- Identify required API, controller, status, migration, conformance, and security changes. + +## Additional comments from user + +- Create any changes on a new PR branch under `rchinchani/*`. +- Working branch: `rchinchani/gep-4894-backend-evaluation`. +- Make the case for binding Fleet to the current GEP-4894 `selectorRef` + proposal versus not binding to it now. + +## Plan + +### Phase 1: Establish the API baseline + +- [x] **Task 1.1: Analyze GEP-4894.** Record the `Backend` types, namespace rules, inline TLS model, status contract, conformance level, and active upstream schema changes. + - Success criteria: The evaluation distinguishes merged Experimental behavior from pending proposals. +- [x] **Task 1.2: Analyze the existing RFC.** Extract the global ingress goals, origin providers, connectivity modes, traffic controls, and provider-specific policy requirements. + - Success criteria: Every material RFC requirement has a comparison target. +- [x] **Task 1.3: Analyze the GEP-1748 prototype.** Trace the `HTTPRoute` to Fleet `ServiceImport` to member-origin model and current controller foundation. + - Success criteria: Existing public API and normalized model responsibilities are documented. + +### Phase 2: Build the compatibility assessment + +- [x] **Task 2.1: Map portable concepts.** Compare `Backend` with `ServiceImport`, `FleetBackendPolicy`, and the internal normalized backend model. + - Success criteria: The assessment identifies direct fits, adapters, and semantic mismatches. +- [x] **Task 2.2: Evaluate end-to-end scenarios.** Test the design conceptually against public direct Service, private direct Service with PLS, shared cluster Gateway, active-active, active-passive, and cross-namespace scenarios. + - Success criteria: Each scenario has a clear feasible, conditional, or unsupported result. +- [x] **Task 2.3: Evaluate operational contracts.** Compare status, ownership, migration, security, conformance, and failure behavior. + - Success criteria: No product requirement is treated as solved solely by a compatible YAML shape. + +### Phase 3: Document the recommendation + +- [x] **Task 3.1: Update the applicable design document.** Add a concise GEP-4894 evaluation and recommended target architecture without prematurely committing to an unstable upstream field. + - Success criteria: The RFC clearly states what GEP-4894 replaces, what remains Fleet-specific, and the adoption gates. +- [x] **Task 3.2: Update related implementation guidance if needed.** Align the GEP-1748 implementation direction with an additive `Backend` adapter path. + - Success criteria: Existing `ServiceImport` behavior remains compatible and migration avoids dual programming. +- [x] **Task 3.3: Complete the breadcrumb.** Record decisions, implementation details, changed files, before/after comparison, references, and any course corrections. + - Success criteria: The breadcrumb is an accurate review trail for the resulting PR. + +### Detailed checklist + +- [x] Phase 1 / Task 1.1 complete. +- [x] Phase 1 / Task 1.2 complete. +- [x] Phase 1 / Task 1.3 complete. +- [x] Phase 2 / Task 2.1 complete. +- [x] Phase 2 / Task 2.2 complete. +- [x] Phase 2 / Task 2.3 complete. +- [x] Phase 3 / Task 3.1 complete. +- [x] Phase 3 / Task 3.2 complete; the standalone evaluation records the + required additive adapter direction without modifying either unmerged source + branch. +- [x] Phase 3 / Task 3.3 complete. + +### Overall success criteria + +- The recommendation answers whether GEP-4894 can meet the same goals and under which conditions. +- The design preserves Fleet multi-cluster endpoint aggregation and Azure Front Door-specific behavior. +- The proposal avoids depending on an unmerged `selectorRef` contract. +- Existing GEP-1748 `ServiceImport` routes have a safe, explicit compatibility and migration path. + +## Decisions + +- Treat GEP-4894 as an additive consumer-side connection resource, not as a replacement for MCS `ServiceImport`. +- Preserve `ServiceImport` as the logical multi-cluster endpoint aggregation contract. +- Preserve Fleet/Azure configuration for origin provider, public versus Private Link connectivity, placement, per-cluster priority and weight, AFD SKU, WAF, diagnostics, and Azure ownership. +- Do not ship a user-facing dependency on `Backend.spec.endpointSelector.selectorRef` while Gateway API PR 5158 proposes removing it and the referenced upstream `EndpointSelector` API does not exist. +- Present early binding as a viable option rather than dismissing it: it can + accelerate a coherent Backend-first API, generate upstream implementation + feedback, and reduce later user migration if the shape survives. +- Evaluate the decision by API compatibility, implementation cost, upstream + influence, conformance truthfulness, migration risk, and delivery schedule. +- Keep the existing provider-neutral normalized model as the convergence point for direct `ServiceImport` references and any future `Backend` references. +- Document the evaluation in a standalone design note because both source design documents live on separate unmerged branches. + +## Implementation Details + +- GEP-4894 `Backend` directly covers consumer-side protocol and backend TLS configuration. +- `ExternalHostname` covers one external FQDN but not a dynamic, weighted set of Fleet member origins. +- `EndpointSelector` currently lacks a stable binding that can represent a Fleet `ServiceImport` on the hub. +- GEP-4894 requires a `Backend` to share a namespace with its Route, while GEP-1748 permits direct cross-namespace `ServiceImport` references through `ReferenceGrant`. +- A future integration requires either a stable upstream endpoint-selection resource derived from `ServiceImport`, or a separately standardized extension point. Fleet must not reinterpret `selectorRef` as a direct `ServiceImport` reference. +- The decision record distinguishes an experimental incubation implementation + from a customer-facing API commitment. Fleet may implement the current field + behind a disabled-by-default gate to generate upstream evidence while + retaining direct `ServiceImport` references as the supported path. + +## Changes Made + +- Created this breadcrumb on `rchinchani/gep-4894-backend-evaluation`. +- Added `docs/design/gep-4894-backend-evaluation.md`. +- Documented requirement mapping, scenario feasibility, target architecture, + migration, status, security, and adoption gates. +- Added a balanced decision analysis for binding to the current `selectorRef` + versus deferring, with a comparative scorecard and bounded-incubation option. + +## Before/After Comparison + +Before this evaluation, the AFD RFC treated Gateway API and GEP-1748 as a +future option, while the GEP-1748 branch directly referenced Fleet +`ServiceImport`. Neither branch evaluated GEP-4894. + +After this evaluation, the proposed architecture has explicit responsibility +boundaries: + +- `Backend` owns consumer protocol and origin TLS. +- `ServiceImport` owns multi-cluster endpoint aggregation. +- Fleet/Azure policy owns AFD origin topology and provider lifecycle. +- Both current and future API paths converge on one normalized model. +- The current `selectorRef` can be incubated experimentally but is not + recommended as a production Fleet API commitment. + +## References + +- GEP-4894: (Experimental) +- GEP-1748: (Experimental) +- Gateway API PR 5158: (open; proposes deferring `selectorRef`) +- `rchinchani/afd-global-ingress-rfc:docs/design/afd-global-ingress-rfc.md` +- `rchinchani/gep-1748-gateway-api:docs/design/gep-1748-gateway-api.md` +- `rchinchani/gep-1748-gateway-api:docs/design/gep-1748-implementation-plan.md` +- `rchinchani/gep-1748-gateway-api:docs/howtos/gateway-api-afd-configuration.md` +- `rchinchani/gep-1748-gateway-api:pkg/controllers/hub/gatewaymodel/model.go` diff --git a/.github/.copilot/breadcrumbs/2026-08-25-1847-afd-public-private-origin-proposal.md b/.github/.copilot/breadcrumbs/2026-08-25-1847-afd-public-private-origin-proposal.md new file mode 100644 index 00000000..87c5d983 --- /dev/null +++ b/.github/.copilot/breadcrumbs/2026-08-25-1847-afd-public-private-origin-proposal.md @@ -0,0 +1,401 @@ +# AFD Public and Private Origin Proposal + +## Requirements + +- Write a detailed proposal for Azure Front Door plus WAF using public + LoadBalancer Services in Fleet member clusters. +- Write a detailed proposal for Azure Front Door plus WAF plus Azure Private + Link Service using private LoadBalancer Services in Fleet member clusters. +- Assume Gateway API PR 5158 merges and the initial GEP-4894 + `EndpointSelector` supports namespace-local pod selection without + `selectorRef`. +- Preserve Fleet `ServiceImport` as the hub-side multi-cluster endpoint + aggregation contract. +- Define Kubernetes YAML, Azure resource mapping, controller interactions, + security, status, failure handling, lifecycle, migration, testing, and + rollout. +- Produce a phased implementation plan with test-first tasks and measurable + success criteria. +- Explain why AFD, rather than Traffic Manager, is required for the SFI + Application DDoS control and carry that constraint into the implementation + plan. + +## Additional comments from user + +- Create changes on a new PR branch under `rchinchani/*`. +- Current branch: `rchinchani/afd-public-private-origin-proposal`. +- User approved proceeding from the GEP-4894 evaluation to a detailed public + and private AFD origin proposal. +- User approved this implementation plan before documentation changes. +- User requested that the feature-branch documentation and commit message explicitly + identify its GEP alignment: preserve GEP-1748 `HTTPRoute -> ServiceImport` routing and + avoid a production dependency on GEP-4894's evolving `selectorRef`. +- User requested a small AFD-versus-Traffic Manager SFI DDoS rationale in the + relevant design documents. +- User chose to continue with the recommended attachment-scoped backend model + after reviewing direct `ServiceImport` policy attachment versus an explicit + Gateway-to-`ServiceImport` attachment. +- User approved implementing the API and controller foundation on this branch. +- The approved branch scope is API contracts plus a feature-gated, read-only + controller foundation; Azure resource writes remain in later pull requests. +- The initial API supports controller-managed AFD profiles only. +- Public origin bypass conformance will be reported by a member-cluster + condition rather than asserted by a hub-side acknowledgement or probe. + +## Plan + +### Phase 1: Confirm repository and API baselines + +- [x] **Task 1.1: Inspect Fleet service aggregation APIs.** Trace + `ServiceExport`, `InternalServiceExport`, and `ServiceImport` fields and + controller ownership. + - Success criteria: The proposal uses existing public fields correctly and + identifies any required internal-only transport extensions. +- [x] **Task 1.2: Inspect existing Azure global-routing patterns.** Review + Traffic Manager reconciliation, ownership, status, and Azure identity + conventions that can be reused. + - Success criteria: The design reuses repository patterns where applicable + and explicitly explains deviations for AFD. +- [x] **Task 1.3: Reconcile the two source RFC branches.** Use the evaluated AFD + RFC and GEP-1748 controller design as inputs while accounting for PR 5158. + - Success criteria: The proposal has one coherent target API and does not + depend on the removed `selectorRef`. + +### Phase 2: Design the public-origin topology + +- [x] **Task 2.1: Define public member resources and readiness.** Specify + LoadBalancer Service, ServiceExport, endpoint eligibility, bypass + protection, and health. + - Success criteria: A member becomes an AFD origin only when every required + public readiness condition is satisfied. +- [x] **Task 2.2: Define hub and Azure resources.** Specify Gateway, + HTTPRoute, ServiceImport policy, AFD profile/endpoint, custom domain, WAF + security policy, origin group, and origins. + - Success criteria: Every Kubernetes field has a deterministic Azure mapping + and ownership rule. +- [x] **Task 2.3: Define public reconciliation and failure behavior.** + - Success criteria: Partial member failures, all-or-nothing validation, + draining, health failures, and Azure errors have explicit behavior. + +### Phase 3: Design the private-origin topology + +- [x] **Task 3.1: Define private member resources and PLS discovery.** Specify + internal LoadBalancer Service, PLS creation, approval, regional metadata, + and internal transport. + - Success criteria: Public fallback is impossible when Private Link is + explicitly required. +- [x] **Task 3.2: Define AFD Premium private origins.** Specify managed private + endpoints, origin grouping, region constraints, approval state, and WAF. + - Success criteria: The design covers asynchronous provisioning and approval + without reporting false readiness. +- [x] **Task 3.3: Define private reconciliation and failure behavior.** + - Success criteria: Missing PLS, stale resource IDs, rejected connections, + mixed connectivity, and regional failures have deterministic outcomes. + +### Phase 4: Define shared contracts + +- [x] **Task 4.1: Define API and validation.** Separate portable Gateway API + fields from Fleet/Azure-specific policy and define defaults and immutability. + - Success criteria: The YAML shape is schema-validatable and has no secret + values in annotations or status. +- [x] **Task 4.2: Define traffic, health, and status semantics.** Preserve + logical backend weights separately from member-origin priority and weight. + - Success criteria: `Programmed`, `Ready`, origin health, and regional + availability cannot be confused. +- [x] **Task 4.3: Define security and ownership.** Cover WAF, origin bypass, + identity, RBAC, ReferenceGrant, Azure RBAC, finalizers, drift, and deletion. + - Success criteria: The proposal has explicit tenant and Azure resource + boundaries for both topologies. + +### Phase 5: Write the implementation plan + +- [x] **Task 5.1: Plan test infrastructure and API work first.** + - Success criteria: Unit, integration, envtest, fake Azure, and live Azure + tests precede or accompany their implementation tasks. +- [x] **Task 5.2: Plan controllers and Azure provider delivery.** + - Success criteria: Tasks identify packages, dependencies, watches, indexes, + reconciliation order, retry behavior, and status writers. +- [x] **Task 5.3: Plan staged rollout and production readiness.** + - Success criteria: Public and private capabilities can ship independently + behind feature gates with upgrade and rollback tests. + +### Phase 6: Complete documentation + +- [x] **Task 6.1: Write the detailed design proposal.** + - Success criteria: A reviewer can evaluate both complete data paths, + resource mappings, API contracts, security, operations, and alternatives. +- [x] **Task 6.2: Write the detailed implementation plan.** + - Success criteria: Each task has prerequisites, deliverables, validation, + and completion criteria. +- [x] **Task 6.3: Complete this breadcrumb.** + - Success criteria: Decisions, changed files, comparisons, and references + accurately reflect the final documents. +- [x] **Task 6.4: Document the SFI Application DDoS product boundary.** + - Success criteria: The proposal explains why DNS-based Traffic Manager is + not the required Layer-7 enforcement plane, and the implementation plan + makes the AFD Premium, WAF, Bot Manager, rate-limit, and origin-bypass + controls explicit. +- [x] **Task 6.5: Adopt the Gateway-to-ServiceImport attachment API.** + - Success criteria: The proposal and implementation plan replace the direct + backend policy with an attachment identified by Gateway UID, + ServiceImport UID, and port; define conflict, namespace, status, and + ownership behavior; and close implementation-plan Task 0.1. + +### Phase 7: Implement the API and controller foundation + +- [x] **Task 7.1: Add API and validation tests first.** Cover defaults, + immutable references, enum/range constraints, managed-profile placement, + WAF requirements, status bounds, and scheme registration. + - Success criteria: tests constrain every new API default and validation + rule before generated CRDs are accepted. +- [x] **Task 7.2: Implement and generate the provider APIs.** Add + `AzureFrontDoorGatewayPolicy`, `AzureFrontDoorBackendAttachment`, shared + status types, deep copies, CRDs, and RBAC. + - Success criteria: generated artifacts contain no manual drift and reject + invalid combinations at the API server boundary. +- [x] **Task 7.3: Add read-only attachment controller tests first.** Cover + missing references, port validation, deterministic duplicate precedence, + generation-aware conditions, dependency-triggered reconciliation, and no + finalizer or Azure writes. + - Success criteria: tests define deterministic status behavior for every + dependency and conflict state. +- [x] **Task 7.4: Implement the feature-gated controller foundation.** Add + repository-consistent structured logging, events, field indexes, watches, + status patching, and manager wiring behind a default-off flag. + - Success criteria: disabled installations start no AFD controller; enabled + installations reconcile attachment status without changing Azure or + Gateway resources. +- [x] **Task 7.5: Validate the foundation.** Run generation, formatting, + focused tests, `go vet`, linting, and manifest drift checks. + - Success criteria: all checks pass and the breadcrumb records any + repository baseline failures separately from change-induced failures. + +### Phase 8: Final API hardening + +- [ ] **Task 8.1: Preserve nested defaults for omitted YAML objects.** Ensure + omitting `healthProbe` and `traffic` still materializes their documented + child defaults, and cover the manifest-shaped admission path. + - Success criteria: an API-server round trip of an object without either + parent returns all documented defaults. +- [ ] **Task 8.2: Reject non-TCP ServiceImport origins.** Resolve the selected + service port's protocol and reject UDP or SCTP because AFD HTTP/S origins + require TCP. + - Success criteria: controller tests accept TCP and reject UDP/SCTP ports. +- [ ] **Task 8.3: Regenerate and revalidate.** Regenerate deep copies and CRDs, + then rerun focused tests, compile checks, vet, lint, and manifest rendering. + - Success criteria: generated artifacts are stable and all applicable + checks pass. + +### Detailed checklist + +- [x] Phase 1 / Task 1.1 complete. +- [x] Phase 1 / Task 1.2 complete. +- [x] Phase 1 / Task 1.3 complete. +- [x] Phase 2 / Task 2.1 complete. +- [x] Phase 2 / Task 2.2 complete. +- [x] Phase 2 / Task 2.3 complete. +- [x] Phase 3 / Task 3.1 complete. +- [x] Phase 3 / Task 3.2 complete. +- [x] Phase 3 / Task 3.3 complete. +- [x] Phase 4 / Task 4.1 complete. +- [x] Phase 4 / Task 4.2 complete. +- [x] Phase 4 / Task 4.3 complete. +- [x] Phase 5 / Task 5.1 complete. +- [x] Phase 5 / Task 5.2 complete. +- [x] Phase 5 / Task 5.3 complete. +- [x] Phase 6 / Task 6.1 complete. +- [x] Phase 6 / Task 6.2 complete. +- [x] Phase 6 / Task 6.3 complete. +- [x] Phase 6 / Task 6.4 complete. +- [x] Phase 6 / Task 6.5 complete. +- [x] Phase 7 / Task 7.1 complete. +- [x] Phase 7 / Task 7.2 complete. +- [x] Phase 7 / Task 7.3 complete. +- [x] Phase 7 / Task 7.4 complete. +- [x] Phase 7 / Task 7.5 complete. +- [ ] Phase 8 / Task 8.1 complete. +- [ ] Phase 8 / Task 8.2 complete. +- [ ] Phase 8 / Task 8.3 complete. + +### Overall success criteria + +- Public and private origin topologies are independently implementable. +- WAF is consistently enforced at the AFD edge in both topologies. +- Private mode never silently downgrades to public connectivity. +- Fleet membership changes produce safe origin add, drain, and removal. +- Existing `ServiceImport` consumers remain compatible. +- One `ServiceImport` can be consumed by multiple Gateways with independent, + deterministic AFD connectivity contracts. +- The design does not bind Fleet to GEP-4894 `selectorRef`. +- The implementation plan is test-first, phased, and suitable for a sequence + of reviewable pull requests. + +## Decisions + +- Keep direct `HTTPRoute` to Fleet `ServiceImport` references for the initial + global ingress backend path. Do not bind production behavior to GEP-4894's + namespace-local Pod selector. +- Use typed `AzureFrontDoorGatewayPolicy` and + `AzureFrontDoorBackendAttachment` candidate APIs instead of annotations as + the durable provider contract. +- Scope each backend attachment to a Gateway UID, `ServiceImport` UID, and + service port. Keep references same-namespace initially and require + `ReferenceGrant` before adding cross-namespace backend references. +- Resolve duplicate attachment tuples without disrupting traffic: the oldest + attachment by creation timestamp, with UID as tie-breaker, remains accepted + and later duplicates report `Accepted=False`, reason `Conflicted`. +- Program a route backend only when its parent Gateway, + `ServiceImport`, and port match an accepted attachment. Multiple routes + share that attachment and origin group; an unused attachment performs no + Azure writes. +- Keep Azure endpoint and PLS metadata internal to + `InternalServiceExport`; do not add Azure-specific fields to public + `ServiceImport.status`. +- Use one homogeneous AFD origin group per logical backend. Public and Private + Link origins cannot be mixed. +- Require AFD Premium for Private Link and for managed WAF rule sets. +- Reference an existing security-owned WAF policy initially; Fleet owns the + AFD security-policy association. +- Require both `AzureFrontDoor.Backend` network filtering and exact + `X-Azure-FDID` validation for public origins. +- Use restrictive, RBAC-only PLS visibility by default and manual private + endpoint approval initially. Automatic approval is deferred until exact + request correlation is proven safe. +- Distinguish desired configuration programming from AFD probe health and + Private Link connection readiness. +- Deliver public origins first, then WAF enforcement, then Private Link, each + behind feature gates. +- Use AFD, not Traffic Manager, as the SFI Application DDoS enforcement plane + for internet-facing HTTP/S. Traffic Manager remains suitable only where a + DNS steering product without Layer-7 enforcement is intentionally required. +- Limit this branch to the API and read-only controller foundation. Do not add + an Azure SDK client, finalizer, or Azure write until a later implementation + pull request. +- Support managed AFD profiles only in the initial API. +- Require member-cluster status to attest exact `X-Azure-FDID` conformance + before a later public-origin controller may report the origin programmed. + +## Implementation Details + +- The proposal includes complete public and private request paths, example + member/hub YAML, candidate CRD fields, validation, origin eligibility, + traffic/drain behavior, WAF/TLS semantics, status, reconciliation, + ownership, observability, scale, failures, migration, and alternatives. +- The implementation plan defines nine reviewable capability PRs and a + test-first task sequence covering API, member transport, normalized model, + Azure provider abstractions, public origins, WAF, Private Link, operations, + and e2e rollout. +- Existing Traffic Manager patterns are reused for ServiceImport expansion, + indexes, finalizer timing, ownership-aware deletion, status/events, and + Azure error classification. +- The proposal now records the SFI product boundary, and Phase 6 includes + tests and exit criteria for AFD Premium, WAF association, Bot Manager, rate + limiting, diagnostics, and public/private origin-bypass controls. +- The direct `ServiceImport` backend policy was replaced with an explicit + Gateway-to-`ServiceImport` attachment. Attachment status and finalizers now + map to one AFD origin group consumption context. +- Phase 1 and normalized-model tests cover attachment reuse, duplicate tuples, + missing matches, multiple routes sharing one origin group, and unused + attachments producing no desired Azure resources. +- The initial provider API is implemented in `api/v1alpha1` with managed + profile placement, existing WAF references, public and Private Link + attachment contracts, health/traffic defaults, immutable references, + bounded status conditions, and generated CRDs/deep copies. +- `pkg/controllers/hub/azurefrontdoorbackendattachment` resolves same-namespace + Gateway, Gateway policy, ServiceImport, and service-port dependencies. It + selects duplicate attachments by creation timestamp and UID, publishes + generation-aware conditions and resolved UIDs, and deliberately adds no + finalizer or Azure client. +- The controller uses the repository's klog `InfoS`/`ErrorS`, Kubernetes + events, API-server error wrappers, status patching, field indexes, and + dependency watches. +- The hub manager and Helm chart expose + `--enable-azure-front-door-gateway-api`, defaulting to false. Conditional + RBAC grants Gateway and provider-resource reads plus attachment status + updates only. +- Gateway API `v1.2.1` is reused from the earlier GEP-1748 branch and is + compatible with the repository's Kubernetes `v0.31.1` dependencies. + +## Changes Made + +- Created this breadcrumb and completed its approved plan. +- Added `docs/design/afd-public-private-origin-proposal.md`. +- Added `docs/design/afd-public-private-origin-implementation-plan.md`. +- Added the AFD-versus-Traffic Manager SFI Application DDoS rationale and its + implementation gates to both design documents. +- Replaced the direct backend-policy candidate API with + `AzureFrontDoorBackendAttachment` and closed implementation-plan Task 0.1. +- Added the two AFD `v1alpha1` API types, shared status types, unit and API + integration tests, generated deep copies, and generated CRDs. +- Added the read-only backend attachment controller and table-driven + reconciliation tests. +- Added hub-manager scheme registration, CRD discovery checks, feature-gate + wiring, generated RBAC, conditional Helm RBAC, values, and chart + documentation. +- Updated the CRD installer test so hub installations include the new AFD + CRDs while member installations remain unchanged. +- Added and tidied the Gateway API `v1.2.1` dependency. +- Validation completed: + - focused API, controller, and hub-manager tests passed; + - all 60 `v1alpha1` API integration specs passed; + - the integration command reported only the existing Windows envtest + teardown limitation (`not supported by windows`) after all specs passed; + - repository-wide compile-only tests and `go vet ./...` passed; + - repository-pinned golangci-lint `v1.64.7`, rebuilt with Go `1.25.12`, + passed for branch changes; + - CRD-installer tests and disabled/enabled Helm rendering checks passed; + - generated CRDs have no unrelated version drift and `git diff --check` + passed. + +## Before/After Comparison + +Before this work, the repository had an AFD global ingress direction and a +GEP-4894 compatibility analysis, but no end-to-end contract for public and +private member-cluster origins. After this work, both topologies have explicit +Kubernetes and Azure resource models, security/readiness state machines, and a +test-first delivery sequence that does not depend on unstable `selectorRef` +behavior. A `ServiceImport` can also be reused by multiple Gateways without +forcing all AFD profiles to share one origin contract or one ambiguous status. +The branch now also contains an installable, default-off API and read-only +controller foundation that validates those attachment identities without +claiming Azure resources are programmed. + +## References + +- `docs/design/gep-4894-backend-evaluation.md` +- `rchinchani/afd-global-ingress-rfc:docs/design/afd-global-ingress-rfc.md` +- `rchinchani/gep-1748-gateway-api:docs/design/gep-1748-gateway-api.md` +- `rchinchani/gep-1748-gateway-api:docs/design/gep-1748-implementation-plan.md` +- GEP-1748: +- GEP-4894: +- `api/v1alpha1/azurefrontdoorgatewaypolicy_types.go` +- `api/v1alpha1/azurefrontdoorbackendattachment_types.go` +- `pkg/controllers/hub/azurefrontdoorbackendattachment/controller.go` +- Gateway API `v1.2.1`: +- Gateway API PR 5158: + +- Azure Front Door Private Link: + +- Azure WAF on Front Door: + +- Azure Front Door origin security: + +- Azure Front Door origins and origin groups: + +- SFI NS 2.5.3 KPI: + +- Application DDoS Standard: + +- AFD to internal load balancer with Private Link: + +- AKS internal load balancer and PLS: + +- AKS public Standard Load Balancer: + +- `api/v1alpha1/serviceimport_types.go` +- `api/v1alpha1/internalserviceexport_types.go` +- `pkg/controllers/member/serviceexport/controller.go` +- `pkg/controllers/hub/serviceimport/controller.go` +- `pkg/controllers/hub/trafficmanagerbackend/controller.go` +- `pkg/controllers/hub/trafficmanagerprofile/controller.go` +- `api/v1beta1/trafficmanagerbackend_types.go` diff --git a/api/v1alpha1/azurefrontdoor_status_types.go b/api/v1alpha1/azurefrontdoor_status_types.go new file mode 100644 index 00000000..dbc6dcd4 --- /dev/null +++ b/api/v1alpha1/azurefrontdoor_status_types.go @@ -0,0 +1,46 @@ +/* +Copyright (c) Microsoft Corporation. +Licensed under the MIT license. +*/ + +package v1alpha1 + +import ( + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" +) + +const ( + AzureFrontDoorConditionAccepted = "Accepted" + AzureFrontDoorConditionResolvedRefs = "ResolvedRefs" + AzureFrontDoorConditionProgrammed = "Programmed" + AzureFrontDoorReasonAccepted = "Accepted" + AzureFrontDoorReasonConflicted = "Conflicted" + AzureFrontDoorReasonInvalid = "Invalid" + AzureFrontDoorReasonInvalidPort = "InvalidPort" + AzureFrontDoorReasonPending = "Pending" + AzureFrontDoorReasonRefNotFound = "RefNotFound" + AzureFrontDoorReasonUnsupportedRef = "UnsupportedRef" + AzureFrontDoorReasonUnsupportedConfig = "UnsupportedConfiguration" +) + +// AzureFrontDoorResolvedReference records the identity used to resolve an immutable API reference. +type AzureFrontDoorResolvedReference struct { + // Name is the referenced object's name. + Name string `json:"name"` + + // UID distinguishes a recreated object from the object originally resolved. + UID types.UID `json:"uid"` +} + +// AzureFrontDoorResourceStatus is shared status for provider resources. +type AzureFrontDoorResourceStatus struct { + // Conditions describe the current reconciliation state. + // +optional + // +patchMergeKey=type + // +patchStrategy=merge + // +listType=map + // +listMapKey=type + // +kubebuilder:validation:MaxItems=8 + Conditions []metav1.Condition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type"` +} diff --git a/api/v1alpha1/azurefrontdoor_types_test.go b/api/v1alpha1/azurefrontdoor_types_test.go new file mode 100644 index 00000000..7b050920 --- /dev/null +++ b/api/v1alpha1/azurefrontdoor_types_test.go @@ -0,0 +1,64 @@ +/* +Copyright (c) Microsoft Corporation. +Licensed under the MIT license. +*/ + +package v1alpha1 + +import ( + "testing" + + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + gatewayv1 "sigs.k8s.io/gateway-api/apis/v1" +) + +func TestAzureFrontDoorTypesAreRegistered(t *testing.T) { + scheme := runtime.NewScheme() + if err := AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() error = %v", err) + } + + for _, object := range []runtime.Object{ + &AzureFrontDoorGatewayPolicy{}, + &AzureFrontDoorGatewayPolicyList{}, + &AzureFrontDoorBackendAttachment{}, + &AzureFrontDoorBackendAttachmentList{}, + } { + gvks, _, err := scheme.ObjectKinds(object) + if err != nil { + t.Fatalf("ObjectKinds(%T) error = %v", object, err) + } + if len(gvks) != 1 || gvks[0].GroupVersion() != GroupVersion { + t.Errorf("ObjectKinds(%T) = %v, want one %s GVK", object, gvks, GroupVersion) + } + } +} + +func TestAzureFrontDoorBackendAttachmentDeepCopy(t *testing.T) { + attachment := &AzureFrontDoorBackendAttachment{ + Spec: AzureFrontDoorBackendAttachmentSpec{ + GatewayRef: gatewayv1.LocalObjectReference{Name: "global"}, + Connectivity: AzureFrontDoorConnectivitySpec{ + Mode: AzureFrontDoorConnectivityModePrivateLink, + PrivateLink: &AzureFrontDoorPrivateLinkSpec{Approval: "Manual"}, + }, + }, + Status: AzureFrontDoorBackendAttachmentStatus{ + AzureFrontDoorResourceStatus: AzureFrontDoorResourceStatus{ + Conditions: []metav1.Condition{{Type: AzureFrontDoorConditionAccepted}}, + }, + }, + } + + copied := attachment.DeepCopy() + copied.Spec.Connectivity.PrivateLink.Approval = "Changed" + copied.Status.Conditions[0].Type = AzureFrontDoorConditionResolvedRefs + + if attachment.Spec.Connectivity.PrivateLink.Approval != "Manual" { + t.Error("DeepCopy() aliased private link configuration") + } + if attachment.Status.Conditions[0].Type != AzureFrontDoorConditionAccepted { + t.Error("DeepCopy() aliased status conditions") + } +} diff --git a/api/v1alpha1/azurefrontdoorbackendattachment_types.go b/api/v1alpha1/azurefrontdoorbackendattachment_types.go new file mode 100644 index 00000000..1559edb8 --- /dev/null +++ b/api/v1alpha1/azurefrontdoorbackendattachment_types.go @@ -0,0 +1,239 @@ +/* +Copyright (c) Microsoft Corporation. +Licensed under the MIT license. +*/ + +package v1alpha1 + +import ( + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + gatewayv1 "sigs.k8s.io/gateway-api/apis/v1" +) + +const ( + AzureFrontDoorBackendAttachmentKind = "AzureFrontDoorBackendAttachment" +) + +// AzureFrontDoorConnectivityMode selects public or Private Link origins. +// +kubebuilder:validation:Enum=Public;PrivateLink +type AzureFrontDoorConnectivityMode string + +const ( + AzureFrontDoorConnectivityModePublic AzureFrontDoorConnectivityMode = "Public" + AzureFrontDoorConnectivityModePrivateLink AzureFrontDoorConnectivityMode = "PrivateLink" +) + +// AzureFrontDoorOriginProtocol is the protocol used from AFD to an origin. +// +kubebuilder:validation:Enum=HTTP;HTTPS +type AzureFrontDoorOriginProtocol string + +const ( + AzureFrontDoorOriginProtocolHTTP AzureFrontDoorOriginProtocol = "HTTP" + AzureFrontDoorOriginProtocolHTTPS AzureFrontDoorOriginProtocol = "HTTPS" +) + +// AzureFrontDoorHealthProbeMethod is an AFD health probe method. +// +kubebuilder:validation:Enum=GET;HEAD +type AzureFrontDoorHealthProbeMethod string + +const ( + AzureFrontDoorHealthProbeMethodGET AzureFrontDoorHealthProbeMethod = "GET" + AzureFrontDoorHealthProbeMethodHEAD AzureFrontDoorHealthProbeMethod = "HEAD" +) + +// AzureFrontDoorMemberFailurePolicy controls whether one invalid member blocks the backend. +// +kubebuilder:validation:Enum=All;Partial +type AzureFrontDoorMemberFailurePolicy string + +const ( + AzureFrontDoorMemberFailurePolicyAll AzureFrontDoorMemberFailurePolicy = "All" + AzureFrontDoorMemberFailurePolicyPartial AzureFrontDoorMemberFailurePolicy = "Partial" +) + +// AzureFrontDoorBackendReference identifies a ServiceImport port in the attachment namespace. +type AzureFrontDoorBackendReference struct { + gatewayv1.LocalObjectReference `json:",inline"` + + // Port is the ServiceImport service port consumed by the Gateway. + // +kubebuilder:validation:Minimum=1 + // +kubebuilder:validation:Maximum=65535 + Port int32 `json:"port"` +} + +// AzureFrontDoorBackendAttachmentSpec configures one Gateway consumption of one ServiceImport port. +type AzureFrontDoorBackendAttachmentSpec struct { + // GatewayRef identifies the Gateway that consumes this backend. + // +required + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="spec.gatewayRef is immutable" + GatewayRef gatewayv1.LocalObjectReference `json:"gatewayRef"` + + // BackendRef identifies a ServiceImport port in the attachment namespace. + // +required + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="spec.backendRef is immutable" + BackendRef AzureFrontDoorBackendReference `json:"backendRef"` + + // Connectivity selects public or Private Link origins. + // +required + Connectivity AzureFrontDoorConnectivitySpec `json:"connectivity"` + + // Origin configures AFD-to-origin connections. + // +required + Origin AzureFrontDoorOriginSpec `json:"origin"` + + // HealthProbe configures AFD origin health evaluation. + // +kubebuilder:default={"protocol":"HTTPS","method":"HEAD","path":"/healthz","intervalSeconds":30,"sampleSize":4,"successfulSamplesRequired":3} + // +optional + HealthProbe AzureFrontDoorHealthProbeSpec `json:"healthProbe,omitempty"` + + // Traffic configures defaults for member-cluster origins. + // +kubebuilder:default={"defaultPriority":1,"defaultWeight":1000} + // +optional + Traffic AzureFrontDoorTrafficSpec `json:"traffic,omitempty"` + + // MemberFailurePolicy controls whether invalid members block all origins. + // +kubebuilder:default=Partial + MemberFailurePolicy AzureFrontDoorMemberFailurePolicy `json:"memberFailurePolicy,omitempty"` +} + +// AzureFrontDoorConnectivitySpec selects the origin connectivity contract. +type AzureFrontDoorConnectivitySpec struct { + // Mode selects public or Private Link origins. + // +required + Mode AzureFrontDoorConnectivityMode `json:"mode"` + + // PrivateLink configures Private Link origin behavior. + // +optional + PrivateLink *AzureFrontDoorPrivateLinkSpec `json:"privateLink,omitempty"` +} + +// AzureFrontDoorPrivateLinkSpec configures the initial manual approval workflow. +type AzureFrontDoorPrivateLinkSpec struct { + // Approval is Manual in the initial API. + // +kubebuilder:validation:Enum=Manual + // +kubebuilder:default=Manual + Approval string `json:"approval,omitempty"` + + // RegionSelection determines how the managed private endpoint region is selected. + // +kubebuilder:validation:Enum=MemberRegion;ClosestSupported + // +kubebuilder:default=ClosestSupported + RegionSelection string `json:"regionSelection,omitempty"` +} + +// AzureFrontDoorOriginSpec configures connections from AFD to every selected member origin. +type AzureFrontDoorOriginSpec struct { + // Protocol is the origin connection protocol. + // +kubebuilder:default=HTTPS + Protocol AzureFrontDoorOriginProtocol `json:"protocol,omitempty"` + + // HostHeader is the HTTP Host header sent to the origin. + // +kubebuilder:validation:MinLength=1 + // +kubebuilder:validation:MaxLength=253 + HostHeader string `json:"hostHeader"` + + // CertificateSubjectNameCheck enables origin certificate name validation. + // +kubebuilder:default=true + CertificateSubjectNameCheck *bool `json:"certificateSubjectNameCheck,omitempty"` +} + +// AzureFrontDoorHealthProbeSpec configures one origin group's health probe. +// +kubebuilder:validation:XValidation:rule="self.successfulSamplesRequired <= self.sampleSize",message="successfulSamplesRequired cannot exceed sampleSize" +type AzureFrontDoorHealthProbeSpec struct { + // Protocol is the probe protocol. + // +kubebuilder:default=HTTPS + Protocol AzureFrontDoorOriginProtocol `json:"protocol,omitempty"` + + // Method is the probe method. + // +kubebuilder:default=HEAD + Method AzureFrontDoorHealthProbeMethod `json:"method,omitempty"` + + // Path is an absolute HTTP path. + // +kubebuilder:default="/healthz" + // +kubebuilder:validation:Pattern=`^/` + // +kubebuilder:validation:MaxLength=1024 + Path string `json:"path,omitempty"` + + // IntervalSeconds is the interval between probes. + // +kubebuilder:validation:Enum=30;60;120;180;240 + // +kubebuilder:default=30 + IntervalSeconds int32 `json:"intervalSeconds,omitempty"` + + // SampleSize is the number of recent samples used for health evaluation. + // +kubebuilder:validation:Minimum=1 + // +kubebuilder:validation:Maximum=255 + // +kubebuilder:default=4 + SampleSize int32 `json:"sampleSize,omitempty"` + + // SuccessfulSamplesRequired is the minimum successful sample count. + // +kubebuilder:validation:Minimum=1 + // +kubebuilder:validation:Maximum=255 + // +kubebuilder:default=3 + SuccessfulSamplesRequired int32 `json:"successfulSamplesRequired,omitempty"` +} + +// AzureFrontDoorTrafficSpec configures member origin defaults. +type AzureFrontDoorTrafficSpec struct { + // DefaultPriority is the origin priority. + // +kubebuilder:validation:Minimum=1 + // +kubebuilder:validation:Maximum=5 + // +kubebuilder:default=1 + DefaultPriority int32 `json:"defaultPriority,omitempty"` + + // DefaultWeight is the origin weight. + // +kubebuilder:validation:Minimum=1 + // +kubebuilder:validation:Maximum=1000 + // +kubebuilder:default=1000 + DefaultWeight int32 `json:"defaultWeight,omitempty"` +} + +// AzureFrontDoorBackendAttachmentStatus describes reference resolution and acceptance. +type AzureFrontDoorBackendAttachmentStatus struct { + AzureFrontDoorResourceStatus `json:",inline"` + + // Gateway records the resolved Gateway identity. + // +optional + Gateway *AzureFrontDoorResolvedReference `json:"gateway,omitempty"` + + // Backend records the resolved ServiceImport identity. + // +optional + Backend *AzureFrontDoorResolvedReference `json:"backend,omitempty"` +} + +// +kubebuilder:object:root=true +// +kubebuilder:resource:scope=Namespaced,categories={fleet-networking},shortName=afdba +// +kubebuilder:subresource:status +// +kubebuilder:printcolumn:JSONPath=`.spec.gatewayRef.name`,name="Gateway",type=string +// +kubebuilder:printcolumn:JSONPath=`.spec.backendRef.name`,name="Backend",type=string +// +kubebuilder:printcolumn:JSONPath=`.spec.backendRef.port`,name="Port",type=integer +// +kubebuilder:printcolumn:JSONPath=`.status.conditions[?(@.type=='Accepted')].status`,name="Accepted",type=string +// +kubebuilder:printcolumn:JSONPath=`.metadata.creationTimestamp`,name="Age",type=date +// +kubebuilder:validation:XValidation:rule="self.spec.gatewayRef.group == 'gateway.networking.k8s.io' && self.spec.gatewayRef.kind == 'Gateway'",message="spec.gatewayRef must reference a Gateway" +// +kubebuilder:validation:XValidation:rule="self.spec.backendRef.group == 'networking.fleet.azure.com' && self.spec.backendRef.kind == 'ServiceImport'",message="spec.backendRef must reference a ServiceImport" +// +kubebuilder:validation:XValidation:rule="self.spec.connectivity.mode == 'PrivateLink' ? has(self.spec.connectivity.privateLink) : !has(self.spec.connectivity.privateLink)",message="privateLink must be set only when connectivity mode is PrivateLink" +// +kubebuilder:validation:XValidation:rule="self.spec.connectivity.mode != 'PrivateLink' || self.spec.origin.protocol != 'HTTPS' || self.spec.origin.certificateSubjectNameCheck == true",message="Private Link HTTPS origins require certificate subject-name validation" + +// AzureFrontDoorBackendAttachment binds one Gateway to one ServiceImport port. +type AzureFrontDoorBackendAttachment struct { + metav1.TypeMeta `json:",inline"` + // +optional + metav1.ObjectMeta `json:"metadata,omitempty"` + + Spec AzureFrontDoorBackendAttachmentSpec `json:"spec"` + + // +optional + Status AzureFrontDoorBackendAttachmentStatus `json:"status,omitempty"` +} + +// +kubebuilder:object:root=true + +// AzureFrontDoorBackendAttachmentList contains AzureFrontDoorBackendAttachment objects. +type AzureFrontDoorBackendAttachmentList struct { + metav1.TypeMeta `json:",inline"` + // +optional + metav1.ListMeta `json:"metadata,omitempty"` + // +listType=set + Items []AzureFrontDoorBackendAttachment `json:"items"` +} + +func init() { + SchemeBuilder.Register(&AzureFrontDoorBackendAttachment{}, &AzureFrontDoorBackendAttachmentList{}) +} diff --git a/api/v1alpha1/azurefrontdoorgatewaypolicy_types.go b/api/v1alpha1/azurefrontdoorgatewaypolicy_types.go new file mode 100644 index 00000000..b43dd3d1 --- /dev/null +++ b/api/v1alpha1/azurefrontdoorgatewaypolicy_types.go @@ -0,0 +1,148 @@ +/* +Copyright (c) Microsoft Corporation. +Licensed under the MIT license. +*/ + +package v1alpha1 + +import ( + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + gatewayv1 "sigs.k8s.io/gateway-api/apis/v1" +) + +const ( + AzureFrontDoorGatewayPolicyKind = "AzureFrontDoorGatewayPolicy" +) + +// AzureFrontDoorProfileMode identifies how Fleet obtains the AFD profile. +// +kubebuilder:validation:Enum=Managed +type AzureFrontDoorProfileMode string + +const ( + AzureFrontDoorProfileModeManaged AzureFrontDoorProfileMode = "Managed" +) + +// AzureFrontDoorProfileSKU is the Azure Front Door Standard or Premium SKU. +// +kubebuilder:validation:Enum=Standard_AzureFrontDoor;Premium_AzureFrontDoor +type AzureFrontDoorProfileSKU string + +const ( + AzureFrontDoorProfileSKUStandard AzureFrontDoorProfileSKU = "Standard_AzureFrontDoor" + AzureFrontDoorProfileSKUPremium AzureFrontDoorProfileSKU = "Premium_AzureFrontDoor" +) + +// AzureFrontDoorGatewayPolicySpec configures the AFD profile used by one Gateway. +type AzureFrontDoorGatewayPolicySpec struct { + // TargetRef identifies a Gateway in the policy namespace. + // +required + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="spec.targetRef is immutable" + TargetRef gatewayv1.LocalObjectReference `json:"targetRef"` + + // Profile configures the controller-managed AFD profile. + // +required + Profile AzureFrontDoorProfileSpec `json:"profile"` + + // WAF references the security-owned WAF policy associated with AFD routes. + // +required + WAF AzureFrontDoorWAFSpec `json:"waf"` + + // Diagnostics configures an existing Azure diagnostic destination. + // +optional + Diagnostics *AzureFrontDoorDiagnosticsSpec `json:"diagnostics,omitempty"` +} + +// AzureFrontDoorProfileSpec configures a controller-managed AFD profile. +type AzureFrontDoorProfileSpec struct { + // Mode is Managed in the initial API. + // +kubebuilder:default=Managed + Mode AzureFrontDoorProfileMode `json:"mode,omitempty"` + + // SKU is the AFD profile SKU. + // +required + SKU AzureFrontDoorProfileSKU `json:"sku"` + + // ResourceGroup is the Azure resource group containing the profile. + // +kubebuilder:validation:MinLength=1 + // +kubebuilder:validation:MaxLength=90 + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="spec.profile.resourceGroup is immutable" + ResourceGroup string `json:"resourceGroup"` + + // Name is the Azure AFD profile name. + // +kubebuilder:validation:MinLength=1 + // +kubebuilder:validation:MaxLength=260 + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="spec.profile.name is immutable" + Name string `json:"name"` +} + +// AzureFrontDoorWAFSpec references an existing WAF policy. +type AzureFrontDoorWAFSpec struct { + // Required must remain true for internet-facing AFD Gateways. + // +kubebuilder:default=true + Required *bool `json:"required,omitempty"` + + // PolicyResourceID is the complete Azure resource ID of an existing WAF policy. + // +kubebuilder:validation:MinLength=1 + // +kubebuilder:validation:MaxLength=2048 + PolicyResourceID string `json:"policyResourceID"` +} + +// AzureFrontDoorDiagnosticsSpec references an existing Azure diagnostic destination. +type AzureFrontDoorDiagnosticsSpec struct { + // Enabled controls diagnostic settings for resources managed by this policy. + // +kubebuilder:default=true + Enabled *bool `json:"enabled,omitempty"` + + // DestinationResourceID is the complete Azure resource ID of the destination. + // +kubebuilder:validation:MinLength=1 + // +kubebuilder:validation:MaxLength=2048 + DestinationResourceID string `json:"destinationResourceID"` +} + +// AzureFrontDoorGatewayPolicyStatus describes the resolved Gateway and policy state. +type AzureFrontDoorGatewayPolicyStatus struct { + AzureFrontDoorResourceStatus `json:",inline"` + + // Gateway records the resolved target identity. + // +optional + Gateway *AzureFrontDoorResolvedReference `json:"gateway,omitempty"` + + // ProfileResourceID is populated only after a later Azure-writing controller creates the profile. + // +optional + ProfileResourceID string `json:"profileResourceID,omitempty"` +} + +// +kubebuilder:object:root=true +// +kubebuilder:resource:scope=Namespaced,categories={fleet-networking},shortName=afdgp +// +kubebuilder:subresource:status +// +kubebuilder:printcolumn:JSONPath=`.spec.targetRef.name`,name="Gateway",type=string +// +kubebuilder:printcolumn:JSONPath=`.status.conditions[?(@.type=='Accepted')].status`,name="Accepted",type=string +// +kubebuilder:printcolumn:JSONPath=`.metadata.creationTimestamp`,name="Age",type=date +// +kubebuilder:validation:XValidation:rule="self.spec.targetRef.group == 'gateway.networking.k8s.io' && self.spec.targetRef.kind == 'Gateway'",message="spec.targetRef must reference a Gateway" +// +kubebuilder:validation:XValidation:rule="self.spec.waf.required == true",message="spec.waf.required must be true" + +// AzureFrontDoorGatewayPolicy configures the AFD infrastructure used by a Gateway. +type AzureFrontDoorGatewayPolicy struct { + metav1.TypeMeta `json:",inline"` + // +optional + metav1.ObjectMeta `json:"metadata,omitempty"` + + Spec AzureFrontDoorGatewayPolicySpec `json:"spec"` + + // +optional + Status AzureFrontDoorGatewayPolicyStatus `json:"status,omitempty"` +} + +// +kubebuilder:object:root=true + +// AzureFrontDoorGatewayPolicyList contains AzureFrontDoorGatewayPolicy objects. +type AzureFrontDoorGatewayPolicyList struct { + metav1.TypeMeta `json:",inline"` + // +optional + metav1.ListMeta `json:"metadata,omitempty"` + // +listType=set + Items []AzureFrontDoorGatewayPolicy `json:"items"` +} + +func init() { + SchemeBuilder.Register(&AzureFrontDoorGatewayPolicy{}, &AzureFrontDoorGatewayPolicyList{}) +} diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go index c5c9c1ed..9c04c8b5 100644 --- a/api/v1alpha1/zz_generated.deepcopy.go +++ b/api/v1alpha1/zz_generated.deepcopy.go @@ -11,11 +11,413 @@ package v1alpha1 import ( corev1 "k8s.io/api/core/v1" - v1 "k8s.io/api/discovery/v1" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + discoveryv1 "k8s.io/api/discovery/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1" runtime "k8s.io/apimachinery/pkg/runtime" ) +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AzureFrontDoorBackendAttachment) DeepCopyInto(out *AzureFrontDoorBackendAttachment) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) + in.Spec.DeepCopyInto(&out.Spec) + in.Status.DeepCopyInto(&out.Status) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AzureFrontDoorBackendAttachment. +func (in *AzureFrontDoorBackendAttachment) DeepCopy() *AzureFrontDoorBackendAttachment { + if in == nil { + return nil + } + out := new(AzureFrontDoorBackendAttachment) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *AzureFrontDoorBackendAttachment) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AzureFrontDoorBackendAttachmentList) DeepCopyInto(out *AzureFrontDoorBackendAttachmentList) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ListMeta.DeepCopyInto(&out.ListMeta) + if in.Items != nil { + in, out := &in.Items, &out.Items + *out = make([]AzureFrontDoorBackendAttachment, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AzureFrontDoorBackendAttachmentList. +func (in *AzureFrontDoorBackendAttachmentList) DeepCopy() *AzureFrontDoorBackendAttachmentList { + if in == nil { + return nil + } + out := new(AzureFrontDoorBackendAttachmentList) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *AzureFrontDoorBackendAttachmentList) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AzureFrontDoorBackendAttachmentSpec) DeepCopyInto(out *AzureFrontDoorBackendAttachmentSpec) { + *out = *in + out.GatewayRef = in.GatewayRef + out.BackendRef = in.BackendRef + in.Connectivity.DeepCopyInto(&out.Connectivity) + in.Origin.DeepCopyInto(&out.Origin) + out.HealthProbe = in.HealthProbe + out.Traffic = in.Traffic +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AzureFrontDoorBackendAttachmentSpec. +func (in *AzureFrontDoorBackendAttachmentSpec) DeepCopy() *AzureFrontDoorBackendAttachmentSpec { + if in == nil { + return nil + } + out := new(AzureFrontDoorBackendAttachmentSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AzureFrontDoorBackendAttachmentStatus) DeepCopyInto(out *AzureFrontDoorBackendAttachmentStatus) { + *out = *in + in.AzureFrontDoorResourceStatus.DeepCopyInto(&out.AzureFrontDoorResourceStatus) + if in.Gateway != nil { + in, out := &in.Gateway, &out.Gateway + *out = new(AzureFrontDoorResolvedReference) + **out = **in + } + if in.Backend != nil { + in, out := &in.Backend, &out.Backend + *out = new(AzureFrontDoorResolvedReference) + **out = **in + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AzureFrontDoorBackendAttachmentStatus. +func (in *AzureFrontDoorBackendAttachmentStatus) DeepCopy() *AzureFrontDoorBackendAttachmentStatus { + if in == nil { + return nil + } + out := new(AzureFrontDoorBackendAttachmentStatus) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AzureFrontDoorBackendReference) DeepCopyInto(out *AzureFrontDoorBackendReference) { + *out = *in + out.LocalObjectReference = in.LocalObjectReference +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AzureFrontDoorBackendReference. +func (in *AzureFrontDoorBackendReference) DeepCopy() *AzureFrontDoorBackendReference { + if in == nil { + return nil + } + out := new(AzureFrontDoorBackendReference) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AzureFrontDoorConnectivitySpec) DeepCopyInto(out *AzureFrontDoorConnectivitySpec) { + *out = *in + if in.PrivateLink != nil { + in, out := &in.PrivateLink, &out.PrivateLink + *out = new(AzureFrontDoorPrivateLinkSpec) + **out = **in + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AzureFrontDoorConnectivitySpec. +func (in *AzureFrontDoorConnectivitySpec) DeepCopy() *AzureFrontDoorConnectivitySpec { + if in == nil { + return nil + } + out := new(AzureFrontDoorConnectivitySpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AzureFrontDoorDiagnosticsSpec) DeepCopyInto(out *AzureFrontDoorDiagnosticsSpec) { + *out = *in + if in.Enabled != nil { + in, out := &in.Enabled, &out.Enabled + *out = new(bool) + **out = **in + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AzureFrontDoorDiagnosticsSpec. +func (in *AzureFrontDoorDiagnosticsSpec) DeepCopy() *AzureFrontDoorDiagnosticsSpec { + if in == nil { + return nil + } + out := new(AzureFrontDoorDiagnosticsSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AzureFrontDoorGatewayPolicy) DeepCopyInto(out *AzureFrontDoorGatewayPolicy) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) + in.Spec.DeepCopyInto(&out.Spec) + in.Status.DeepCopyInto(&out.Status) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AzureFrontDoorGatewayPolicy. +func (in *AzureFrontDoorGatewayPolicy) DeepCopy() *AzureFrontDoorGatewayPolicy { + if in == nil { + return nil + } + out := new(AzureFrontDoorGatewayPolicy) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *AzureFrontDoorGatewayPolicy) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AzureFrontDoorGatewayPolicyList) DeepCopyInto(out *AzureFrontDoorGatewayPolicyList) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ListMeta.DeepCopyInto(&out.ListMeta) + if in.Items != nil { + in, out := &in.Items, &out.Items + *out = make([]AzureFrontDoorGatewayPolicy, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AzureFrontDoorGatewayPolicyList. +func (in *AzureFrontDoorGatewayPolicyList) DeepCopy() *AzureFrontDoorGatewayPolicyList { + if in == nil { + return nil + } + out := new(AzureFrontDoorGatewayPolicyList) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *AzureFrontDoorGatewayPolicyList) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AzureFrontDoorGatewayPolicySpec) DeepCopyInto(out *AzureFrontDoorGatewayPolicySpec) { + *out = *in + out.TargetRef = in.TargetRef + out.Profile = in.Profile + in.WAF.DeepCopyInto(&out.WAF) + if in.Diagnostics != nil { + in, out := &in.Diagnostics, &out.Diagnostics + *out = new(AzureFrontDoorDiagnosticsSpec) + (*in).DeepCopyInto(*out) + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AzureFrontDoorGatewayPolicySpec. +func (in *AzureFrontDoorGatewayPolicySpec) DeepCopy() *AzureFrontDoorGatewayPolicySpec { + if in == nil { + return nil + } + out := new(AzureFrontDoorGatewayPolicySpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AzureFrontDoorGatewayPolicyStatus) DeepCopyInto(out *AzureFrontDoorGatewayPolicyStatus) { + *out = *in + in.AzureFrontDoorResourceStatus.DeepCopyInto(&out.AzureFrontDoorResourceStatus) + if in.Gateway != nil { + in, out := &in.Gateway, &out.Gateway + *out = new(AzureFrontDoorResolvedReference) + **out = **in + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AzureFrontDoorGatewayPolicyStatus. +func (in *AzureFrontDoorGatewayPolicyStatus) DeepCopy() *AzureFrontDoorGatewayPolicyStatus { + if in == nil { + return nil + } + out := new(AzureFrontDoorGatewayPolicyStatus) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AzureFrontDoorHealthProbeSpec) DeepCopyInto(out *AzureFrontDoorHealthProbeSpec) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AzureFrontDoorHealthProbeSpec. +func (in *AzureFrontDoorHealthProbeSpec) DeepCopy() *AzureFrontDoorHealthProbeSpec { + if in == nil { + return nil + } + out := new(AzureFrontDoorHealthProbeSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AzureFrontDoorOriginSpec) DeepCopyInto(out *AzureFrontDoorOriginSpec) { + *out = *in + if in.CertificateSubjectNameCheck != nil { + in, out := &in.CertificateSubjectNameCheck, &out.CertificateSubjectNameCheck + *out = new(bool) + **out = **in + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AzureFrontDoorOriginSpec. +func (in *AzureFrontDoorOriginSpec) DeepCopy() *AzureFrontDoorOriginSpec { + if in == nil { + return nil + } + out := new(AzureFrontDoorOriginSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AzureFrontDoorPrivateLinkSpec) DeepCopyInto(out *AzureFrontDoorPrivateLinkSpec) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AzureFrontDoorPrivateLinkSpec. +func (in *AzureFrontDoorPrivateLinkSpec) DeepCopy() *AzureFrontDoorPrivateLinkSpec { + if in == nil { + return nil + } + out := new(AzureFrontDoorPrivateLinkSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AzureFrontDoorProfileSpec) DeepCopyInto(out *AzureFrontDoorProfileSpec) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AzureFrontDoorProfileSpec. +func (in *AzureFrontDoorProfileSpec) DeepCopy() *AzureFrontDoorProfileSpec { + if in == nil { + return nil + } + out := new(AzureFrontDoorProfileSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AzureFrontDoorResolvedReference) DeepCopyInto(out *AzureFrontDoorResolvedReference) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AzureFrontDoorResolvedReference. +func (in *AzureFrontDoorResolvedReference) DeepCopy() *AzureFrontDoorResolvedReference { + if in == nil { + return nil + } + out := new(AzureFrontDoorResolvedReference) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AzureFrontDoorResourceStatus) DeepCopyInto(out *AzureFrontDoorResourceStatus) { + *out = *in + if in.Conditions != nil { + in, out := &in.Conditions, &out.Conditions + *out = make([]v1.Condition, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AzureFrontDoorResourceStatus. +func (in *AzureFrontDoorResourceStatus) DeepCopy() *AzureFrontDoorResourceStatus { + if in == nil { + return nil + } + out := new(AzureFrontDoorResourceStatus) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AzureFrontDoorTrafficSpec) DeepCopyInto(out *AzureFrontDoorTrafficSpec) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AzureFrontDoorTrafficSpec. +func (in *AzureFrontDoorTrafficSpec) DeepCopy() *AzureFrontDoorTrafficSpec { + if in == nil { + return nil + } + out := new(AzureFrontDoorTrafficSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AzureFrontDoorWAFSpec) DeepCopyInto(out *AzureFrontDoorWAFSpec) { + *out = *in + if in.Required != nil { + in, out := &in.Required, &out.Required + *out = new(bool) + **out = **in + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AzureFrontDoorWAFSpec. +func (in *AzureFrontDoorWAFSpec) DeepCopy() *AzureFrontDoorWAFSpec { + if in == nil { + return nil + } + out := new(AzureFrontDoorWAFSpec) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *ClusterStatus) DeepCopyInto(out *ClusterStatus) { *out = *in @@ -121,7 +523,7 @@ func (in *EndpointSliceExportSpec) DeepCopyInto(out *EndpointSliceExportSpec) { } if in.Ports != nil { in, out := &in.Ports, &out.Ports - *out = make([]v1.EndpointPort, len(*in)) + *out = make([]discoveryv1.EndpointPort, len(*in)) for i := range *in { (*in)[i].DeepCopyInto(&(*out)[i]) } @@ -332,7 +734,7 @@ func (in *InternalServiceExportStatus) DeepCopyInto(out *InternalServiceExportSt *out = *in if in.Conditions != nil { in, out := &in.Conditions, &out.Conditions - *out = make([]metav1.Condition, len(*in)) + *out = make([]v1.Condition, len(*in)) for i := range *in { (*in)[i].DeepCopyInto(&(*out)[i]) } @@ -570,7 +972,7 @@ func (in *MultiClusterServiceStatus) DeepCopyInto(out *MultiClusterServiceStatus in.LoadBalancer.DeepCopyInto(&out.LoadBalancer) if in.Conditions != nil { in, out := &in.Conditions, &out.Conditions - *out = make([]metav1.Condition, len(*in)) + *out = make([]v1.Condition, len(*in)) for i := range *in { (*in)[i].DeepCopyInto(&(*out)[i]) } @@ -665,7 +1067,7 @@ func (in *ServiceExportStatus) DeepCopyInto(out *ServiceExportStatus) { *out = *in if in.Conditions != nil { in, out := &in.Conditions, &out.Conditions - *out = make([]metav1.Condition, len(*in)) + *out = make([]v1.Condition, len(*in)) for i := range *in { (*in)[i].DeepCopyInto(&(*out)[i]) } @@ -943,7 +1345,7 @@ func (in *TrafficManagerBackendStatus) DeepCopyInto(out *TrafficManagerBackendSt } if in.Conditions != nil { in, out := &in.Conditions, &out.Conditions - *out = make([]metav1.Condition, len(*in)) + *out = make([]v1.Condition, len(*in)) for i := range *in { (*in)[i].DeepCopyInto(&(*out)[i]) } @@ -1094,7 +1496,7 @@ func (in *TrafficManagerProfileStatus) DeepCopyInto(out *TrafficManagerProfileSt } if in.Conditions != nil { in, out := &in.Conditions, &out.Conditions - *out = make([]metav1.Condition, len(*in)) + *out = make([]v1.Condition, len(*in)) for i := range *in { (*in)[i].DeepCopyInto(&(*out)[i]) } diff --git a/api/v1beta1/zz_generated.deepcopy.go b/api/v1beta1/zz_generated.deepcopy.go index 0f0facb8..8d0093ee 100644 --- a/api/v1beta1/zz_generated.deepcopy.go +++ b/api/v1beta1/zz_generated.deepcopy.go @@ -10,7 +10,7 @@ Licensed under the MIT license. package v1beta1 import ( - v1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1" runtime "k8s.io/apimachinery/pkg/runtime" ) diff --git a/charts/hub-net-controller-manager/README.md b/charts/hub-net-controller-manager/README.md index ac73f907..63eda4c9 100644 --- a/charts/hub-net-controller-manager/README.md +++ b/charts/hub-net-controller-manager/README.md @@ -36,6 +36,7 @@ helm upgrade hub-net-controller-manager ./charts/hub-net-controller-manager/ | leaderElectionNamespace | The namespace in which the leader election resource will be created. | `fleet-system` | | fleetSystemNamespace | The namespace that this Helm chart is installed on and reserved by fleet. | `fleet-system` | | enableTrafficManagerFeature | Set to true to enable the Azure Traffic Manager feature. | `false` | +| enableAzureFrontDoorGatewayAPI | Enable the read-only Azure Front Door Gateway API attachment controller. Requires the Gateway API and Fleet AFD CRDs. | `false` | | resources | The resource request/limits for the container image | limits: 500m CPU, 1Gi, requests: 100m CPU, 128Mi | | podAnnotations | Pod Annotations | `{}` | | affinity | The node affinity to use for pod scheduling | `{}` | diff --git a/charts/hub-net-controller-manager/templates/deployment.yaml b/charts/hub-net-controller-manager/templates/deployment.yaml index ba602ae8..7b99e362 100644 --- a/charts/hub-net-controller-manager/templates/deployment.yaml +++ b/charts/hub-net-controller-manager/templates/deployment.yaml @@ -40,6 +40,7 @@ spec: - --add_dir_header - --force-delete-wait-time={{ .Values.forceDeleteWaitTime }} - --enable-traffic-manager-feature={{ .Values.enableTrafficManagerFeature }} + - --enable-azure-front-door-gateway-api={{ .Values.enableAzureFrontDoorGatewayAPI }} {{- if .Values.enableTrafficManagerFeature }} - --cloud-config=/etc/kubernetes/provider/azure.json {{- end }} diff --git a/charts/hub-net-controller-manager/templates/rbac.yaml b/charts/hub-net-controller-manager/templates/rbac.yaml index e5debe2b..91ec2bb1 100644 --- a/charts/hub-net-controller-manager/templates/rbac.yaml +++ b/charts/hub-net-controller-manager/templates/rbac.yaml @@ -202,6 +202,33 @@ rules: - patch - update {{- end }} +{{- if .Values.enableAzureFrontDoorGatewayAPI }} +- apiGroups: + - gateway.networking.k8s.io + resources: + - gateways + verbs: + - get + - list + - watch +- apiGroups: + - networking.fleet.azure.com + resources: + - azurefrontdoorbackendattachments + - azurefrontdoorgatewaypolicies + verbs: + - get + - list + - watch +- apiGroups: + - networking.fleet.azure.com + resources: + - azurefrontdoorbackendattachments/status + verbs: + - get + - patch + - update +{{- end }} --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 diff --git a/charts/hub-net-controller-manager/values.yaml b/charts/hub-net-controller-manager/values.yaml index 89c252e5..494a2c68 100644 --- a/charts/hub-net-controller-manager/values.yaml +++ b/charts/hub-net-controller-manager/values.yaml @@ -27,6 +27,7 @@ leaderElectionNamespace: fleet-system fleetSystemNamespace: fleet-system forceDeleteWaitTime: 2m0s enableTrafficManagerFeature: false +enableAzureFrontDoorGatewayAPI: false resources: limits: diff --git a/cmd/hub-net-controller-manager/main.go b/cmd/hub-net-controller-manager/main.go index 487900a3..21e361ab 100644 --- a/cmd/hub-net-controller-manager/main.go +++ b/cmd/hub-net-controller-manager/main.go @@ -31,6 +31,7 @@ import ( "sigs.k8s.io/controller-runtime/pkg/log/zap" metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server" "sigs.k8s.io/controller-runtime/pkg/webhook" + gatewayv1 "sigs.k8s.io/gateway-api/apis/v1" //+kubebuilder:scaffold:imports clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" @@ -39,6 +40,7 @@ import ( fleetnetv1alpha1 "go.goms.io/fleet-networking/api/v1alpha1" fleetnetv1beta1 "go.goms.io/fleet-networking/api/v1beta1" + "go.goms.io/fleet-networking/pkg/controllers/hub/azurefrontdoorbackendattachment" "go.goms.io/fleet-networking/pkg/controllers/hub/endpointsliceexport" "go.goms.io/fleet-networking/pkg/controllers/hub/internalserviceexport" "go.goms.io/fleet-networking/pkg/controllers/hub/internalserviceimport" @@ -66,7 +68,12 @@ var ( enableV1Beta1APIs = flag.Bool("enable-v1beta1-apis", true, "If set, the agents will watch for the v1beta1 APIs.") - enableTrafficManagerFeature = flag.Bool("enable-traffic-manager-feature", true, "If set, the traffic manager feature will be enabled.") + enableTrafficManagerFeature = flag.Bool("enable-traffic-manager-feature", true, "If set, the traffic manager feature will be enabled.") + enableAzureFrontDoorGatewayAPI = flag.Bool( + "enable-azure-front-door-gateway-api", + false, + "If set, the read-only Azure Front Door Gateway API controller will be enabled.", + ) cloudConfigFile = flag.String("cloud-config", "/etc/kubernetes/provider/azure.json", "The path to the cloud config file which will be used to access the Azure resource.") ) @@ -76,6 +83,11 @@ var ( fleetnetv1beta1.GroupVersion.WithKind(fleetnetv1beta1.TrafficManagerProfileKind), fleetnetv1beta1.GroupVersion.WithKind(fleetnetv1beta1.TrafficManagerBackendKind), } + azureFrontDoorGatewayAPIRequiredGVKs = []schema.GroupVersionKind{ + {Group: gatewayv1.GroupName, Version: "v1", Kind: "Gateway"}, + fleetnetv1alpha1.GroupVersion.WithKind(fleetnetv1alpha1.AzureFrontDoorGatewayPolicyKind), + fleetnetv1alpha1.GroupVersion.WithKind(fleetnetv1alpha1.AzureFrontDoorBackendAttachmentKind), + } ) func init() { @@ -83,6 +95,7 @@ func init() { utilruntime.Must(fleetnetv1alpha1.AddToScheme(scheme)) utilruntime.Must(fleetnetv1beta1.AddToScheme(scheme)) utilruntime.Must(clusterv1beta1.AddToScheme(scheme)) + utilruntime.Must(gatewayv1.Install(scheme)) klog.InitFlags(nil) //+kubebuilder:scaffold:scheme } @@ -236,6 +249,24 @@ func main() { exitWithErrorFunc() } } + if *enableAzureFrontDoorGatewayAPI { + klog.V(1).InfoS("Azure Front Door Gateway API feature is enabled, checking the required CRDs") + for _, gvk := range azureFrontDoorGatewayAPIRequiredGVKs { + if err = utils.CheckCRDInstalled(discoverClient, gvk); err != nil { + klog.ErrorS(err, "Unable to find the required CRD", "GVK", gvk) + exitWithErrorFunc() + } + } + + klog.V(1).InfoS("Start to setup AzureFrontDoorBackendAttachment controller") + if err := (&azurefrontdoorbackendattachment.Reconciler{ + Client: mgr.GetClient(), + Recorder: mgr.GetEventRecorderFor(azurefrontdoorbackendattachment.ControllerName), + }).SetupWithManager(ctx, mgr); err != nil { + klog.ErrorS(err, "Unable to create AzureFrontDoorBackendAttachment controller") + exitWithErrorFunc() + } + } klog.V(1).InfoS("Starting ServiceExportImport controller manager") if err := mgr.Start(ctx); err != nil { diff --git a/cmd/net-crd-installer/utils/util_test.go b/cmd/net-crd-installer/utils/util_test.go index 6f684f3b..65ab82a5 100644 --- a/cmd/net-crd-installer/utils/util_test.go +++ b/cmd/net-crd-installer/utils/util_test.go @@ -53,9 +53,11 @@ func runTest(t *testing.T, crdPath string) { wantError bool }{ { - name: "hub mode excludes MultiClusterService CRD", + name: "hub mode includes all Fleet networking CRDs", mode: "hub", wantedCRDNames: []string{ + "azurefrontdoorbackendattachments.networking.fleet.azure.com", + "azurefrontdoorgatewaypolicies.networking.fleet.azure.com", "endpointsliceexports.networking.fleet.azure.com", "endpointsliceimports.networking.fleet.azure.com", "internalserviceexports.networking.fleet.azure.com", diff --git a/config/crd/bases/networking.fleet.azure.com_azurefrontdoorbackendattachments.yaml b/config/crd/bases/networking.fleet.azure.com_azurefrontdoorbackendattachments.yaml new file mode 100644 index 00000000..07e3cd12 --- /dev/null +++ b/config/crd/bases/networking.fleet.azure.com_azurefrontdoorbackendattachments.yaml @@ -0,0 +1,396 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.20.0 + name: azurefrontdoorbackendattachments.networking.fleet.azure.com +spec: + group: networking.fleet.azure.com + names: + categories: + - fleet-networking + kind: AzureFrontDoorBackendAttachment + listKind: AzureFrontDoorBackendAttachmentList + plural: azurefrontdoorbackendattachments + shortNames: + - afdba + singular: azurefrontdoorbackendattachment + scope: Namespaced + versions: + - additionalPrinterColumns: + - jsonPath: .spec.gatewayRef.name + name: Gateway + type: string + - jsonPath: .spec.backendRef.name + name: Backend + type: string + - jsonPath: .spec.backendRef.port + name: Port + type: integer + - jsonPath: .status.conditions[?(@.type=='Accepted')].status + name: Accepted + type: string + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + name: v1alpha1 + schema: + openAPIV3Schema: + description: AzureFrontDoorBackendAttachment binds one Gateway to one ServiceImport + port. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: AzureFrontDoorBackendAttachmentSpec configures one Gateway + consumption of one ServiceImport port. + properties: + backendRef: + description: BackendRef identifies a ServiceImport port in the attachment + namespace. + properties: + group: + description: |- + Group is the group of the referent. For example, "gateway.networking.k8s.io". + When unspecified or empty string, core API group is inferred. + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + description: Kind is kind of the referent. For example "HTTPRoute" + or "Service". + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: Name is the name of the referent. + maxLength: 253 + minLength: 1 + type: string + port: + description: Port is the ServiceImport service port consumed by + the Gateway. + format: int32 + maximum: 65535 + minimum: 1 + type: integer + required: + - group + - kind + - name + - port + type: object + x-kubernetes-validations: + - message: spec.backendRef is immutable + rule: self == oldSelf + connectivity: + description: Connectivity selects public or Private Link origins. + properties: + mode: + description: Mode selects public or Private Link origins. + enum: + - Public + - PrivateLink + type: string + privateLink: + description: PrivateLink configures Private Link origin behavior. + properties: + approval: + default: Manual + description: Approval is Manual in the initial API. + enum: + - Manual + type: string + regionSelection: + default: ClosestSupported + description: RegionSelection determines how the managed private + endpoint region is selected. + enum: + - MemberRegion + - ClosestSupported + type: string + type: object + required: + - mode + type: object + gatewayRef: + description: GatewayRef identifies the Gateway that consumes this + backend. + properties: + group: + description: |- + Group is the group of the referent. For example, "gateway.networking.k8s.io". + When unspecified or empty string, core API group is inferred. + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + description: Kind is kind of the referent. For example "HTTPRoute" + or "Service". + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: Name is the name of the referent. + maxLength: 253 + minLength: 1 + type: string + required: + - group + - kind + - name + type: object + x-kubernetes-validations: + - message: spec.gatewayRef is immutable + rule: self == oldSelf + healthProbe: + default: + intervalSeconds: 30 + method: HEAD + path: /healthz + protocol: HTTPS + sampleSize: 4 + successfulSamplesRequired: 3 + description: HealthProbe configures AFD origin health evaluation. + properties: + intervalSeconds: + default: 30 + description: IntervalSeconds is the interval between probes. + enum: + - 30 + - 60 + - 120 + - 180 + - 240 + format: int32 + type: integer + method: + default: HEAD + description: Method is the probe method. + enum: + - GET + - HEAD + type: string + path: + default: /healthz + description: Path is an absolute HTTP path. + maxLength: 1024 + pattern: ^/ + type: string + protocol: + default: HTTPS + description: Protocol is the probe protocol. + enum: + - HTTP + - HTTPS + type: string + sampleSize: + default: 4 + description: SampleSize is the number of recent samples used for + health evaluation. + format: int32 + maximum: 255 + minimum: 1 + type: integer + successfulSamplesRequired: + default: 3 + description: SuccessfulSamplesRequired is the minimum successful + sample count. + format: int32 + maximum: 255 + minimum: 1 + type: integer + type: object + x-kubernetes-validations: + - message: successfulSamplesRequired cannot exceed sampleSize + rule: self.successfulSamplesRequired <= self.sampleSize + memberFailurePolicy: + default: Partial + description: MemberFailurePolicy controls whether invalid members + block all origins. + enum: + - All + - Partial + type: string + origin: + description: Origin configures AFD-to-origin connections. + properties: + certificateSubjectNameCheck: + default: true + description: CertificateSubjectNameCheck enables origin certificate + name validation. + type: boolean + hostHeader: + description: HostHeader is the HTTP Host header sent to the origin. + maxLength: 253 + minLength: 1 + type: string + protocol: + default: HTTPS + description: Protocol is the origin connection protocol. + enum: + - HTTP + - HTTPS + type: string + required: + - hostHeader + type: object + traffic: + default: + defaultPriority: 1 + defaultWeight: 1000 + description: Traffic configures defaults for member-cluster origins. + properties: + defaultPriority: + default: 1 + description: DefaultPriority is the origin priority. + format: int32 + maximum: 5 + minimum: 1 + type: integer + defaultWeight: + default: 1000 + description: DefaultWeight is the origin weight. + format: int32 + maximum: 1000 + minimum: 1 + type: integer + type: object + required: + - backendRef + - connectivity + - gatewayRef + - origin + type: object + status: + description: AzureFrontDoorBackendAttachmentStatus describes reference + resolution and acceptance. + properties: + backend: + description: Backend records the resolved ServiceImport identity. + properties: + name: + description: Name is the referenced object's name. + type: string + uid: + description: UID distinguishes a recreated object from the object + originally resolved. + type: string + required: + - name + - uid + type: object + conditions: + description: Conditions describe the current reconciliation state. + items: + description: Condition contains details for one aspect of the current + state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + maxItems: 8 + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + gateway: + description: Gateway records the resolved Gateway identity. + properties: + name: + description: Name is the referenced object's name. + type: string + uid: + description: UID distinguishes a recreated object from the object + originally resolved. + type: string + required: + - name + - uid + type: object + type: object + required: + - spec + type: object + x-kubernetes-validations: + - message: spec.gatewayRef must reference a Gateway + rule: self.spec.gatewayRef.group == 'gateway.networking.k8s.io' && self.spec.gatewayRef.kind + == 'Gateway' + - message: spec.backendRef must reference a ServiceImport + rule: self.spec.backendRef.group == 'networking.fleet.azure.com' && self.spec.backendRef.kind + == 'ServiceImport' + - message: privateLink must be set only when connectivity mode is PrivateLink + rule: 'self.spec.connectivity.mode == ''PrivateLink'' ? has(self.spec.connectivity.privateLink) + : !has(self.spec.connectivity.privateLink)' + - message: Private Link HTTPS origins require certificate subject-name validation + rule: self.spec.connectivity.mode != 'PrivateLink' || self.spec.origin.protocol + != 'HTTPS' || self.spec.origin.certificateSubjectNameCheck == true + served: true + storage: true + subresources: + status: {} diff --git a/config/crd/bases/networking.fleet.azure.com_azurefrontdoorgatewaypolicies.yaml b/config/crd/bases/networking.fleet.azure.com_azurefrontdoorgatewaypolicies.yaml new file mode 100644 index 00000000..93f3b091 --- /dev/null +++ b/config/crd/bases/networking.fleet.azure.com_azurefrontdoorgatewaypolicies.yaml @@ -0,0 +1,261 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.20.0 + name: azurefrontdoorgatewaypolicies.networking.fleet.azure.com +spec: + group: networking.fleet.azure.com + names: + categories: + - fleet-networking + kind: AzureFrontDoorGatewayPolicy + listKind: AzureFrontDoorGatewayPolicyList + plural: azurefrontdoorgatewaypolicies + shortNames: + - afdgp + singular: azurefrontdoorgatewaypolicy + scope: Namespaced + versions: + - additionalPrinterColumns: + - jsonPath: .spec.targetRef.name + name: Gateway + type: string + - jsonPath: .status.conditions[?(@.type=='Accepted')].status + name: Accepted + type: string + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + name: v1alpha1 + schema: + openAPIV3Schema: + description: AzureFrontDoorGatewayPolicy configures the AFD infrastructure + used by a Gateway. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: AzureFrontDoorGatewayPolicySpec configures the AFD profile + used by one Gateway. + properties: + diagnostics: + description: Diagnostics configures an existing Azure diagnostic destination. + properties: + destinationResourceID: + description: DestinationResourceID is the complete Azure resource + ID of the destination. + maxLength: 2048 + minLength: 1 + type: string + enabled: + default: true + description: Enabled controls diagnostic settings for resources + managed by this policy. + type: boolean + required: + - destinationResourceID + type: object + profile: + description: Profile configures the controller-managed AFD profile. + properties: + mode: + default: Managed + description: Mode is Managed in the initial API. + enum: + - Managed + type: string + name: + description: Name is the Azure AFD profile name. + maxLength: 260 + minLength: 1 + type: string + x-kubernetes-validations: + - message: spec.profile.name is immutable + rule: self == oldSelf + resourceGroup: + description: ResourceGroup is the Azure resource group containing + the profile. + maxLength: 90 + minLength: 1 + type: string + x-kubernetes-validations: + - message: spec.profile.resourceGroup is immutable + rule: self == oldSelf + sku: + description: SKU is the AFD profile SKU. + enum: + - Standard_AzureFrontDoor + - Premium_AzureFrontDoor + type: string + required: + - name + - resourceGroup + - sku + type: object + targetRef: + description: TargetRef identifies a Gateway in the policy namespace. + properties: + group: + description: |- + Group is the group of the referent. For example, "gateway.networking.k8s.io". + When unspecified or empty string, core API group is inferred. + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + description: Kind is kind of the referent. For example "HTTPRoute" + or "Service". + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: Name is the name of the referent. + maxLength: 253 + minLength: 1 + type: string + required: + - group + - kind + - name + type: object + x-kubernetes-validations: + - message: spec.targetRef is immutable + rule: self == oldSelf + waf: + description: WAF references the security-owned WAF policy associated + with AFD routes. + properties: + policyResourceID: + description: PolicyResourceID is the complete Azure resource ID + of an existing WAF policy. + maxLength: 2048 + minLength: 1 + type: string + required: + default: true + description: Required must remain true for internet-facing AFD + Gateways. + type: boolean + required: + - policyResourceID + type: object + required: + - profile + - targetRef + - waf + type: object + status: + description: AzureFrontDoorGatewayPolicyStatus describes the resolved + Gateway and policy state. + properties: + conditions: + description: Conditions describe the current reconciliation state. + items: + description: Condition contains details for one aspect of the current + state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + maxItems: 8 + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + gateway: + description: Gateway records the resolved target identity. + properties: + name: + description: Name is the referenced object's name. + type: string + uid: + description: UID distinguishes a recreated object from the object + originally resolved. + type: string + required: + - name + - uid + type: object + profileResourceID: + description: ProfileResourceID is populated only after a later Azure-writing + controller creates the profile. + type: string + type: object + required: + - spec + type: object + x-kubernetes-validations: + - message: spec.targetRef must reference a Gateway + rule: self.spec.targetRef.group == 'gateway.networking.k8s.io' && self.spec.targetRef.kind + == 'Gateway' + - message: spec.waf.required must be true + rule: self.spec.waf.required == true + served: true + storage: true + subresources: + status: {} diff --git a/config/rbac/role.yaml b/config/rbac/role.yaml index d606b24e..36029624 100644 --- a/config/rbac/role.yaml +++ b/config/rbac/role.yaml @@ -51,6 +51,37 @@ rules: - patch - update - watch +- apiGroups: + - gateway.networking.k8s.io + resources: + - gateways + verbs: + - get + - list + - watch +- apiGroups: + - networking.fleet.azure.com + resources: + - azurefrontdoorbackendattachments + - azurefrontdoorgatewaypolicies + verbs: + - get + - list + - watch +- apiGroups: + - networking.fleet.azure.com + resources: + - azurefrontdoorbackendattachments/status + - internalserviceexports/status + - multiclusterservices/status + - serviceexports/status + - serviceimports/status + - trafficmanagerbackends/status + - trafficmanagerprofiles/status + verbs: + - get + - patch + - update - apiGroups: - networking.fleet.azure.com resources: @@ -78,19 +109,6 @@ rules: - serviceexports/finalizers verbs: - update -- apiGroups: - - networking.fleet.azure.com - resources: - - internalserviceexports/status - - multiclusterservices/status - - serviceexports/status - - serviceimports/status - - trafficmanagerbackends/status - - trafficmanagerprofiles/status - verbs: - - get - - patch - - update - apiGroups: - networking.fleet.azure.com resources: diff --git a/docs/design/afd-public-private-origin-implementation-plan.md b/docs/design/afd-public-private-origin-implementation-plan.md new file mode 100644 index 00000000..5b29851f --- /dev/null +++ b/docs/design/afd-public-private-origin-implementation-plan.md @@ -0,0 +1,770 @@ +# Azure Front Door Public and Private Origins Implementation Plan + +## Purpose + +This plan implements the architecture in +`docs/design/afd-public-private-origin-proposal.md` as a sequence of reviewable pull +requests. It deliberately delivers public origins before Private Link while keeping the +API and normalized model capable of both topologies from the start. + +All new behavior is disabled by default until its rollout gate is met. Tests are written +before or in the same change as the implementation they constrain. + +## Gateway API GEP Alignment + +- GEP-1748 supplies the portable routing contract: + `HTTPRoute.backendRefs -> ServiceImport`. +- The provider-specific `AzureFrontDoorBackendAttachment` augments that route with an + explicit `(Gateway, ServiceImport, port)` AFD connectivity contract. +- GEP-4894 remains an evaluated future integration point. This plan does not bind Fleet + to its evolving namespace-local Pod `selectorRef`, because a hub selector cannot + represent member-cluster endpoints. +- A future GEP-4894 migration must preserve `ServiceImport` identity, multi-cluster + aggregation, deterministic attachment ownership, and status compatibility before it + can replace the explicit attachment. + +## Guiding Rules + +- Keep `ServiceImport` backward compatible and status-only. +- Put Azure transport observations only in `InternalServiceExport`. +- Keep Gateway API fields portable; use typed provider CRDs for Azure behavior. +- Do not use GEP-4894's Pod selector for Fleet multi-cluster backends. +- Do not dual-write legacy annotations and typed provider resources. +- Never mix public and private origins in one AFD origin group. +- Never fall back from Private Link to a public origin. +- Treat AFD, not Traffic Manager, as the SFI Application DDoS enforcement plane for + internet-facing HTTP/S. Traffic Manager is DNS-only and cannot satisfy the required + Layer-7 proxy, WAF, bot-management, rate-limit, or origin-bypass controls. +- Add a finalizer only immediately before the first owned Azure write. +- Preserve the last known good Azure configuration when new desired state is invalid or a + dependency read fails transiently. +- Run `goimports`, `go vet`, package tests, and generated-manifest verification for every + Go/API pull request. Run `go mod tidy` only when dependencies change. + +## Delivery Overview + +| PR | Capability | Default state | +| --- | --- | --- | +| 1 | API contracts, CRDs, validation, generated code | No controller | +| 2 | Member transport discovery for public and PLS metadata | Feature-gated | +| 3 | Normalized Gateway/Fleet model and read-only status | Feature-gated | +| 4 | Azure provider interfaces, fakes, ownership, retry model | No production writes | +| 5 | Managed AFD profile, endpoint, public origin group, public origins | Feature-gated | +| 6 | Routes, custom domains/certificates, public draining and lifecycle | Feature-gated | +| 7 | Required WAF association, diagnostics, public bypass checks | Feature-gated | +| 8 | Private Link origin creation and manual approval workflow | Feature-gated | +| 9 | Production hardening, charts, scale, upgrade, and e2e | Candidate for opt-in | + +## Phase 0: Resolve API Review Questions + +### Task 0.1: Record the approved backend attachment scope + +**Decision** + +- Use `AzureFrontDoorBackendAttachment` with explicit `gatewayRef` and `backendRef`. +- Identify one attachment by Gateway UID, `ServiceImport` UID, and service port. +- Keep all three resources in one namespace initially. Require `ReferenceGrant` before + later enabling cross-namespace `backendRef`. + +**Work** + +- Reject direct `AzureFrontDoorBackendPolicy.targetRef -> ServiceImport` because AFD + connectivity, probes, traffic settings, readiness, and Azure ownership are scoped to a + particular Gateway/profile consumption. +- Admission rejects observable duplicates. Under concurrent creation, the oldest + attachment by creation timestamp, with UID as tie-breaker, remains accepted; later + duplicates report `Accepted=False`, reason `Conflicted`, without disrupting traffic. + +**Tests** + +- API examples for one ServiceImport reused by two Gateways. +- Conflict tests for duplicate Gateway/ServiceImport/port tuples. +- Same ServiceImport with distinct Gateways or ports. +- Same-namespace enforcement and future `ReferenceGrant` behavior. + +**Exit criteria** + +- Attachment reuse, conflicts, authorization, status, and Azure ownership are + deterministic. + +**Status:** Resolved for the candidate `v1alpha1` API. + +### Task 0.2: Approve profile and WAF ownership + +**Work** + +- Decide whether the first release supports both `Managed` and `Existing` profiles or + only `Managed`. +- Confirm that WAF policies are referenced, not created. +- Define required Azure ownership tags and hub cluster identity source. + +**Exit criteria** + +- Creation, adoption, update, and deletion ownership are explicit for every Azure + resource type. + +**Decision** + +- The initial API supports `Managed` profiles only. +- WAF policies are existing, security-owned resources referenced by resource ID. +- Existing-profile adoption is deferred until ownership and deletion behavior can be + proven independently. + +**Status:** Resolved for the candidate `v1alpha1` API. + +### Task 0.3: Approve public bypass conformance contract + +**Work** + +- Decide how the controller proves exact `X-Azure-FDID` enforcement when the application + owns that rule. +- Select one initial mechanism: + - a typed acknowledgement on the backend attachment; + - a separate conformance condition published by a member agent; + - a probe that expects rejection for a request with an invalid ID. +- Keep `AzureFrontDoor.Backend` Service annotation validation mandatory. + +**Exit criteria** + +- A public origin cannot become `Programmed=True` merely because it has a public IP. + +**Decision** + +- A member-cluster condition reports exact `X-Azure-FDID` conformance. +- The hub controller must not infer conformance from a user acknowledgement. +- Negative hub-side probing may be added as defense in depth later, but is not the + authoritative initial signal. + +**Status:** Resolved for the candidate `v1alpha1` API. + +## Phase 1: API Types, CRDs, and Validation + +### Task 1.1: Add API unit tests first + +**Candidate files** + +- `api/v1alpha1/azurefrontdoorgatewaypolicy_types_test.go` +- `api/v1alpha1/azurefrontdoorbackendattachment_types_test.go` +- `api/v1alpha1/validation/azurefrontdoor_test.go` +- `test/apis/azurefrontdoor_integration_test.go` + +**Test cases** + +- defaults for probe method, interval, priority, weight, failure policy, and certificate + validation; +- invalid profile SKU and Private Link combinations; +- invalid probe sample counts; +- invalid origin ports/protocols; +- missing WAF policy when required; +- duplicate Gateway/ServiceImport/port attachment tuples; +- one ServiceImport reused by distinct Gateways and service ports; +- multiple routes sharing one attachment and one origin group; +- route backend without a matching attachment; +- accepted but unused attachment causing no Azure writes; +- same-namespace attachment enforcement; +- immutable placement fields after Azure resources exist; +- status list bounds and condition generation. + +**Exit criteria** + +- Tests fail because types and validation do not yet exist. + +### Task 1.2: Implement provider APIs + +**Candidate files** + +- `api/v1alpha1/azurefrontdoorgatewaypolicy_types.go` +- `api/v1alpha1/azurefrontdoorbackendattachment_types.go` +- `api/v1alpha1/azurefrontdoor_status_types.go` +- `api/v1alpha1/groupversion_info.go` + +**Work** + +- Add kubebuilder validation markers and defaults. +- Use Gateway API `LocalPolicyTargetReference` conventions for the Gateway policy where + compatible; define explicit Gateway and backend references for the attachment. +- Make attachment conflict precedence, status ownership, and immutable identity fields + explicit. +- Define conditions and reasons as constants. +- Avoid storing secrets or raw Azure error bodies. +- Keep per-member status bounded by the selected member count. + +**Validation** + +- Run unit and API integration tests. +- Regenerate deep copies and CRDs. +- Verify generated files have no manual drift. + +**Exit criteria** + +- CRDs reject all invalid combinations identified in Task 1.1. + +### Task 1.3: Add feature gates and RBAC + +**Candidate files** + +- controller option/feature-gate packages used by existing managers; +- `charts/hub-net-controller-manager/templates/`; +- generated role manifests under `config/`. + +**Work** + +- Add `AzureFrontDoorGatewayAPI` feature gate, default `false`. +- Add gateway-policy and backend-attachment read/write/status RBAC. +- Do not add Azure credentials to chart values. + +**Tests** + +- Helm rendering with feature gate off and on. +- RBAC test proving the controller cannot modify unrelated Gateway objects. + +**Exit criteria** + +- Installing the CRDs does not start reconciliation or create Azure resources. + +## Phase 2: Internal Member Transport + +### Task 2.1: Write transport discovery tests + +**Candidate files** + +- `pkg/controllers/member/serviceexport/controller_test.go` +- Azure provider fake tests near the existing member ServiceExport provider. + +**Test cases** + +- public Service reports public IP resource ID, FQDN, location, and bypass annotation; +- public IP exists without DNS settings; +- internal Service reports load balancer address and PLS resource ID/alias/location; +- PLS provisioning, failed, and deleted states; +- multiple PLS resources do not produce nondeterministic selection; +- stale Azure resource ID is removed from internal transport; +- Service changes from public to private and private to public; +- visibility and auto-approval observations are sanitized; +- Azure authorization, not-found, conflict, and throttling errors. + +**Exit criteria** + +- Tests define exactly when public and private transport is ready. + +### Task 2.2: Extend InternalServiceExport + +**Candidate files** + +- `api/v1alpha1/internalserviceexport_types.go` +- generated CRDs and deep-copy files. + +**Work** + +- Add an optional Azure transport structure. +- Preserve existing fields during the transition. +- Store Azure location, public endpoint metadata, load balancer classification, and PLS + metadata. +- Do not copy PLS details into `ServiceImport.status`. + +**Compatibility** + +- Older member agents omit new fields and remain readable by the hub. +- New member agents continue writing fields required by existing Traffic Manager + reconciliation. + +**Exit criteria** + +- Existing ServiceImport behavior is unchanged. + +### Task 2.3: Implement member discovery + +**Candidate files** + +- `pkg/controllers/member/serviceexport/controller.go` +- existing Azure client interface and fake implementations. + +**Work** + +- Resolve the Service load balancer frontend deterministically. +- Resolve public IP DNS settings and location. +- Resolve PLS by the load balancer frontend configuration/resource relationship, not name + guessing alone. +- Classify errors using repository-standard retry behavior. +- Watch/requeue on Service and ServiceExport changes; Azure state remains polled with + bounded backoff. + +**Validation** + +- Run modified package tests and `go vet`. +- Add envtest for public/private transition and deletion. + +**Exit criteria** + +- The hub receives sufficient observed metadata without querying member clusters. + +## Phase 3: Normalized Model and Read-only Reconciliation + +### Task 3.1: Write model tests + +**Candidate package** + +- `pkg/controllers/hub/gatewaymodel/` + +**Test cases** + +- `HTTPRoute` resolves direct `ServiceImport` references; +- route backend weight remains separate from member origin weight; +- `ServiceImport` expands into deterministic sorted member origins; +- all-public and all-private classifications; +- mixed public/private rejection; +- missing attachment, duplicate attachment tuple, and invalid port; +- one ServiceImport attached to distinct Gateways with independent connectivity; +- multiple routes sharing one Gateway/ServiceImport/port attachment; +- route backend without an accepted attachment; +- unused attachment omitted from the desired Azure model; +- `All` versus `Partial` member failure policy; +- zero eligible origins; +- same PLS/resource/region with different port rejection; +- Private Link on Standard SKU rejection; +- namespace and `ReferenceGrant` handling for supported references. + +**Exit criteria** + +- Model behavior is fully testable without Azure clients. + +### Task 3.2: Implement model builder and indexes + +**Work** + +- Add field indexes: + - gateway policy by target UID/name; + - backend attachment by Gateway and ServiceImport/port tuple; + - route by parent Gateway; + - route by ServiceImport backend; + - InternalServiceExport by ServiceImport identity; + - ServiceImport by selected cluster. +- Return immutable desired model objects with deterministic Azure names. +- Include a stable hash of desired Azure-owned fields for drift/status. +- Never materialize member Pods or EndpointSlices on the hub. + +**Exit criteria** + +- One event queues only the affected Gateway/profile reconciliations. + +### Task 3.3: Add read-only status controller + +**Work** + +- Reconcile gateway policies, backend attachments, and Gateway API references without + Azure writes. +- Set `Accepted`, `ResolvedRefs`, transport readiness, and topology validation conditions. +- Emit transition events for missing public FQDN, missing PLS, mixed topology, and + attachment conflicts. + +**Tests** + +- Envtest with Gateway, HTTPRoute, ServiceImport, gateway policies, backend attachments, + and InternalServiceExports. +- Confirm feature gate off produces no status mutation. + +**Exit criteria** + +- Operators can validate YAML and member readiness before granting Azure permissions. + +## Phase 4: Azure Provider Layer + +### Task 4.1: Define provider interfaces and fakes + +**Candidate package** + +- `pkg/azure/frontdoor/` + +**Interfaces** + +- profile; +- endpoint; +- custom domain and certificate; +- origin group; +- origin; +- route; +- security policy; +- diagnostics; +- PLS/private endpoint connection reader. + +Each operation accepts context and returns typed result/error information. Interfaces are +resource-oriented and narrow enough to fake without reproducing the entire Azure SDK. + +**Tests** + +- compile-time interface conformance; +- fake operation recording; +- context cancellation; +- typed not-found, conflict, throttling, authorization, and asynchronous provisioning. + +### Task 4.2: Implement error classification and retry + +**Work** + +- Reuse existing Traffic Manager Azure error classification where possible. +- Honor `Retry-After`. +- Distinguish: + - retryable control-plane errors; + - invalid desired state; + - authorization errors requiring operator action; + - ownership conflicts; + - asynchronous provisioning. +- Add per-profile mutation serialization and bounded concurrency. + +**Exit criteria** + +- Unit tests prove no hot loop for pending Private Link approval or Azure throttling. + +### Task 4.3: Implement ownership and cleanup primitives + +**Work** + +- Generate stable resource names from namespace/name/UID and member cluster identity. +- Apply ownership tags. +- Require exact tags before update/delete. +- List owned children for cleanup. +- Provide disable/drain/delete operations. + +**Tests** + +- Existing untagged resource collision; +- matching ownership update; +- mismatched UID after Kubernetes object recreation; +- partial deletion retry; +- referenced WAF/PLS/public IP never deleted. + +**Exit criteria** + +- Destructive tests prove the controller cannot delete unowned Azure resources. + +## Phase 5: Public Origin Vertical Slice + +### Task 5.1: Write public reconciler tests + +**Candidate files** + +- `pkg/controllers/hub/azurefrontdoorgateway/controller_test.go` +- `pkg/controllers/hub/azurefrontdoorbackend/controller_test.go` + +**Test sequence** + +1. no finalizer before valid model; +2. add finalizer immediately before profile creation; +3. create profile and endpoint; +4. create origin group with probe/load-balancing settings; +5. create one origin per eligible member; +6. wait for asynchronous Azure provisioning; +7. set provider status; +8. update only changed fields; +9. disable/drain/delete removed member; +10. clean up owned resources on attachment/Gateway deletion. + +**Failure cases** + +- no public FQDN; +- internal member in a public attachment; +- bypass annotation absent; +- origin-host/certificate mismatch observed through Azure; +- one member invalid under `Partial` and `All`; +- Azure 429/409/403; +- controller restart during deletion. + +### Task 5.2: Implement profile and public origin reconciliation + +**Work** + +- Start with managed profiles unless Phase 0 approves existing profiles. +- Create one origin group per normalized logical backend/configuration. +- Use member FQDN as origin hostname. +- Keep certificate subject-name validation enabled for HTTPS. +- Create disabled origins first; enable only after required dependencies exist. +- Normalize ServiceExport weights to Azure origin weights. + +**Exit criteria** + +- A fake-Azure integration test programs two public member origins and survives one member + removal without traffic configuration loss. + +### Task 5.3: Implement Gateway API route reconciliation + +**Work** + +- Map listeners/domains/routes to the AFD endpoint. +- Support the agreed subset of `HTTPRoute` matches, filters, and backend references. +- Report unsupported Gateway API fields with `UnsupportedProtocol` or + `UnsupportedValue`-style standard reasons. +- Enable the route only after origin group and frontend dependencies are ready. + +**Tests** + +- route attach/detach; +- hostname conflict; +- unsupported filters; +- two logical backend weights; +- zero ready origins; +- deterministic reconciliation after restart. + +**Exit criteria** + +- Public traffic can flow in a disposable test subscription with WAF still gated off. + +## Phase 6: WAF, TLS, and Public Bypass Enforcement + +### Task 6.1: Write WAF association tests + +**Test cases** + +- referenced policy exists and matches profile tier; +- policy missing, unauthorized, wrong tier, or disabled; +- every route domain is associated; +- domain moves between policies without an unprotected enabled interval; +- WAF required versus explicitly optional development mode; +- security policy drift. + +### Task 6.2: Implement security-policy association + +**Work** + +- Read the existing WAF policy. +- Create/update `Microsoft.Cdn/profiles/securityPolicies`. +- Associate only domains owned by the target Gateway. +- Apply WAF policy changes without recreating origins. +- Fail closed when `waf.required` is true. + +**Exit criteria** + +- No AFD route is enabled before its domain is WAF-associated. + +### Task 6.3: Implement certificate and bypass gates + +**Work** + +- Integrate the reviewed `AzureFrontDoorCertificate` contract. +- Report frontend certificate provisioning separately from origin TLS. +- Expose AFD profile ID in policy status. +- Validate `AzureFrontDoor.Backend` observation and the approved exact-FDID conformance + mechanism. +- Add documentation and example regional ingress rules. + +**Live validation** + +- Direct request to public origin without valid FDID is denied. +- Request through AFD succeeds. +- WAF detection and prevention logs reach diagnostics destination. + +**Exit criteria** + +- Public vertical slice is production-security complete, not merely routable. + +### Task 6.4: Enforce the SFI Application DDoS baseline + +**Tests** + +- reject production configuration that is not AFD Premium; +- reject a missing WAF association or disabled Bot Manager managed rule set; +- reject configuration with no enabled rate-limit custom rule; +- verify public origin bypass denial and private origin public unreachability; +- verify diagnostics expose AFD access, health-probe, and WAF events. + +**Exit criteria** + +- Every production HTTP/S route is demonstrably behind the AFD enforcement plane before + it is enabled; a Traffic Manager endpoint is never accepted as an equivalent control. + +## Phase 7: Private Link Vertical Slice + +### Task 7.1: Write Private Link state-machine tests + +**Test cases** + +- PLS absent, provisioning, ready, failed, and replaced; +- origin creation yields pending AFD-managed private endpoint request; +- manual approval pending does not hot loop or report an error; +- approved but not established; +- established and origin enabled; +- rejected/disconnected connection; +- no public fallback; +- wrong SKU; +- unsupported/member region mapping; +- same PLS tuple and port conflict; +- private endpoint reuse across origins; +- public member in a private attachment; +- deletion while approval is pending. + +### Task 7.2: Implement PLS origin reconciliation + +**Work** + +- Require Premium profile. +- Use observed PLS resource ID. +- Select same or nearest supported Private Link region using a versioned mapping. +- Set request message with stable Kubernetes/profile identity. +- Create origin disabled until connection is established. +- Read and publish connection state. +- Leave approval manual. +- Ensure AFD probes use the private path. + +**Exit criteria** + +- A two-member private topology reaches `OriginsProgrammed=True` only after both PLS + connections satisfy the configured member failure policy. + +### Task 7.3: Implement private lifecycle and resilience + +**Work** + +- Handle PLS resource replacement as a new connection requiring approval. +- Drain old origin before removing the old private endpoint relationship. +- Preserve other origins when one regional connection fails. +- Serialize topology transitions so Azure never sees a mixed origin group. +- Add alerts for prolonged pending/rejected/disconnected states. + +**Live validation** + +- Member internal load balancer has no public route. +- AFD request succeeds after approval. +- Direct internet request cannot reach the origin. +- One private member/region outage shifts traffic to another healthy origin. + +**Exit criteria** + +- Private topology passes security and failover tests with no public endpoint. + +## Phase 8: Operations, Scale, and Productization + +### Task 8.1: Add metrics, events, and diagnostics + +**Work** + +- Add reconciler, origin state, Private Link state-age, WAF, Azure retry, and programming + latency metrics. +- Emit bounded transition events. +- Include object UID, Azure resource name, member cluster, and correlation ID in logs. +- Add sample alerts and dashboards. + +**Tests** + +- Metric label cardinality test. +- Event deduplication test. +- Sanitization test for Azure errors. + +### Task 8.2: Add quota and scale protection + +**Work** + +- Preflight known AFD resource counts. +- Limit concurrent Azure writes per profile/subscription. +- Cache/deduplicate Azure reads during one reconciliation. +- Add load tests for large numbers of routes, ServiceImports, and member origins. +- Verify current Azure service limits before publishing supported scale. + +**Success criteria** + +- At target scale, reconciliation meets the agreed latency SLO without Azure throttling + storms or unbounded hub memory. + +### Task 8.3: Complete charts, docs, and supportability + +**Work** + +- Chart values for feature gates, Azure identity/client configuration, concurrency, and + diagnostics. +- Example YAML for public and private topologies. +- Troubleshooting guide for WAF, certificates, public bypass, PLS approval, and probes. +- Upgrade/downgrade guide and CRD compatibility policy. +- Document Azure permissions separately for profile management and optional future PLS + approval. + +**Exit criteria** + +- A new operator can deploy both topologies using only published examples and diagnose + every failure-matrix state from status/events. + +## Phase 9: End-to-End Validation and Rollout + +### Task 9.1: Automated test matrix + +| Layer | Required coverage | +| --- | --- | +| Unit | defaults, validation, model, naming, weight normalization, state machines, error classification | +| Envtest | watches/indexes, policy/attachment conflicts, conditions, finalizers, deletion, feature gates | +| Fake Azure integration | complete create/update/drain/delete flows and injected Azure errors | +| Live Azure integration | profile/origin/route/WAF/PLS API shapes and asynchronous states | +| E2E public | two clusters, WAF, bypass denial, health failover, member removal | +| E2E private | two clusters/regions, manual approvals, no public reachability, failover | +| Upgrade | gate off/on, controller restart, CRD upgrade, rollback with existing Azure resources | + +Tests must use unique resource names, explicit cleanup, and ownership assertions. Live tests +must fail rather than silently skip cleanup errors. + +### Task 9.2: Rollout gates + +1. **Developer preview:** APIs and read-only status only. +2. **Private preview - public:** managed profile, public origins, WAF required. +3. **Private preview - PLS:** Premium only, manual approval. +4. **Public preview:** scale and multi-region failure tests complete. +5. **General availability:** upgrade compatibility, SLOs, support documentation, and + security review complete. + +Each gate defines: + +- supported Gateway API fields; +- supported Azure clouds/regions; +- maximum tested scale; +- upgrade paths; +- known limitations; +- rollback procedure. + +### Task 9.3: Rollback + +Feature-gate rollback stops new reconciliation but must not orphan finalizers. Provide a +safe suspend mode: + +- no create/update writes; +- status reports `ReconciliationSuspended`; +- deletion cleanup remains available through an explicit controller mode or documented + operator procedure. + +For production traffic rollback: + +1. disable affected AFD route; +2. restore the last known good route/origin-group configuration; +3. verify WAF association; +4. drain newly introduced origins; +5. remove only owned resources; +6. retain referenced WAF, PLS, load balancer, and public IP resources. + +## Pull Request Validation Checklist + +Every implementation PR must complete the applicable items: + +- [ ] Tests were added before or with behavior. +- [ ] Modified package tests pass. +- [ ] `goimports` made no further changes. +- [ ] `go vet` passes for modified packages. +- [ ] Generated deep copies and CRDs are current. +- [ ] Helm templates render with feature gates on and off. +- [ ] No dependency changed without `go mod tidy`. +- [ ] No secret, token, certificate content, or unbounded Azure error is stored in API + spec/status. +- [ ] Azure writes are ownership checked and idempotent. +- [ ] Deletion and controller restart are tested. +- [ ] Conditions include `observedGeneration` and stable reasons. +- [ ] Public/private topology cannot silently change. +- [ ] Documentation and examples match the implemented API. + +## Completion Criteria + +The implementation is complete when: + +- public and private topologies pass their two-member, multi-region e2e suites; +- public origin bypass attempts are denied while AFD traffic succeeds; +- private origins have no public path and never receive public fallback; +- WAF is associated before routes are enabled; +- member add, failure, drain, removal, and Fleet leave are safe and observable; +- PLS pending, approved, established, rejected, and disconnected states are distinguishable; +- origin programming and probe health are separate status signals; +- unowned Azure resources survive all update and deletion tests; +- tested scale and Azure quotas are documented; +- feature-gate rollback and controller restart do not strand finalizers or traffic; +- `ServiceImport` API compatibility and existing Traffic Manager tests remain intact. diff --git a/docs/design/afd-public-private-origin-proposal.md b/docs/design/afd-public-private-origin-proposal.md new file mode 100644 index 00000000..f1902ea3 --- /dev/null +++ b/docs/design/afd-public-private-origin-proposal.md @@ -0,0 +1,989 @@ +# Azure Front Door Public and Private Fleet Origins + +## Status + +Proposed; the default-off API and read-only attachment-controller foundation is +implemented on `rchinchani/afd-public-private-origin-proposal`. + +This document defines a target architecture and candidate Kubernetes API for exposing a +multi-cluster Fleet application through Azure Front Door (AFD) and Azure Web Application +Firewall (WAF). It covers: + +1. AFD + WAF with public `LoadBalancer` Services in member clusters. +2. AFD + WAF + Azure Private Link Service (PLS) with internal `LoadBalancer` Services in + member clusters. + +The API names are introduced as `v1alpha1` contracts and remain subject to API review +before Azure resource programming is enabled. + +## Decision Summary + +Fleet should keep `ServiceImport` as the multi-cluster service identity and create one AFD +origin per eligible `InternalServiceExport`. Gateway API expresses listeners, hostnames, +routes, and route-to-backend weights. Two typed, Azure-specific provider resources express +AFD profile/WAF configuration and each Gateway-to-`ServiceImport` origin attachment. + +The controller must not depend on GEP-4894's current Pod selector. Even if Gateway API +PR 5158 merges, a namespace-local Pod selector on the hub cannot select Pods in member +clusters. The initial backend path remains: + +```text +HTTPRoute + -> Fleet ServiceImport + -> InternalServiceExport per member cluster + -> public load balancer FQDN, or + -> internal load balancer through Private Link Service +``` + +### Gateway API GEP Alignment + +This design preserves the GEP-1748 multi-cluster backend model: +`HTTPRoute.backendRefs` continues to reference Fleet `ServiceImport`, and Fleet remains +responsible for aggregating the member-cluster endpoints behind that identity. The +Azure-specific `AzureFrontDoorBackendAttachment` adds placement and connectivity policy +for a `(Gateway, ServiceImport, port)` tuple without changing the portable route. + +GEP-4894 is treated as a compatible future direction, not the current Fleet binding +contract. Its evolving, namespace-local Pod `selectorRef` cannot select member-cluster +Pods from a hub cluster. The implementation therefore does not reinterpret +`selectorRef`, create proxy Pods, or make production behavior depend on an experimental +field. The explicit attachment can be revisited if GEP-4894 later standardizes a backend +reference model that preserves Fleet's cross-cluster semantics. + +Public and private origins must never share an AFD origin group. Private origins require +AFD Premium. A route must fail closed when its required WAF policy is absent, when origin +connectivity is ambiguous, or when a private origin is not approved and established. +There is no automatic private-to-public fallback. + +### Why AFD, Not Traffic Manager, for SFI Application DDoS + +The SFI Application DDoS standard requires untrusted internet HTTP/S traffic to pass +through an approved, globally distributed Layer-7 proxy before it consumes +service-owned regional capacity. Traffic Manager cannot provide that enforcement +boundary: it returns an origin through DNS and the client then connects directly to the +selected regional endpoint. It cannot inspect HTTP requests, apply WAF or Bot Manager +rules, enforce per-client rate limits, or prevent clients from bypassing the protected +path to reach an origin. + +AFD is in the request data path and supplies the required shared global proxy capacity. +For the NS 2.5.3 baseline, the production design therefore uses AFD Premium with an +associated WAF policy, enabled `Microsoft_BotManagerRuleSet`, at least one enabled +rate-limit custom rule, and origin bypass prevention. Public origins require both +`AzureFrontDoor.Backend` filtering and exact `X-Azure-FDID` validation; private origins +remove the public path by using Private Link. Traffic Manager may remain useful for +non-HTTP protocols, but it is not an equivalent SFI Application DDoS control for this +HTTP/S ingress architecture. + +## Goals + +- Present one global HTTPS endpoint for an application exported from multiple member + clusters. +- Apply WAF at the AFD edge before traffic reaches a member cluster. +- Use AFD health and latency signals to steer traffic only to healthy origins. +- Preserve per-cluster traffic weight and priority independently from `HTTPRoute` + backend weights. +- Protect public origins from direct WAF bypass. +- Keep private member-cluster origins off the public internet by using PLS. +- Expose precise Kubernetes status for Azure provisioning, Private Link approval, and + origin programming. +- Reuse Fleet's existing `ServiceExport` -> `InternalServiceExport` -> `ServiceImport` + data flow and its established Azure ownership/finalizer patterns. + +## Non-goals + +- Replacing `ServiceImport` with GEP-4894 `Backend`. +- Selecting member-cluster Pods from the hub. +- Creating fake hub Pods to satisfy a Gateway API selector. +- Configuring application-level authorization or WAF rules from `HTTPRoute`. +- Providing AFD-to-origin mutual TLS. AFD does not support mTLS for public or Private + Link origins. +- Mixing public and private origins in one origin group. +- Silently falling back from private to public connectivity. +- Managing arbitrary Azure resources referenced by ID unless policy explicitly selects + `Managed` ownership. +- Solving non-Azure member-cluster connectivity in the first release. + +## Personas and Responsibility Boundaries + +| Persona | Owns | +| --- | --- | +| Platform operator | `GatewayClass`, controller deployment, Azure identity, profile placement, diagnostics defaults | +| Security operator | WAF policy lifecycle and mode, managed/custom rules, origin bypass controls | +| Application operator | `Gateway`, `HTTPRoute`, `ServiceExport`, health endpoint, origin host/certificate | +| Fleet member agent | Observing member Service and Azure transport resources; publishing internal metadata | +| Fleet hub controller | Resolving routes and provider configuration; reconciling AFD resources; reporting status | +| Azure cloud provider | Creating public/internal load balancers and PLS from Service annotations | + +Gateway API remains portable. Azure-specific intent is isolated in typed provider +resources instead of an expanding set of annotations on `Gateway` and `HTTPRoute`. + +## Shared Architecture + +### Kubernetes objects + +The member cluster contains: + +- application Pods; +- a `Service` of type `LoadBalancer`; +- a `ServiceExport`. + +The hub contains: + +- an internal `InternalServiceExport` for each exported member Service; +- a public, status-only `ServiceImport`; +- a `GatewayClass` controlled by Fleet AFD; +- a `Gateway`; +- one or more `HTTPRoute` objects; +- an `AzureFrontDoorGatewayPolicy` targeting the `Gateway`; +- an `AzureFrontDoorBackendAttachment` for each Gateway, `ServiceImport`, and service + port tuple consumed through AFD. + +### Azure objects + +The controller maps the Kubernetes model to: + +```text +Microsoft.Cdn/profiles + Microsoft.Cdn/profiles/afdEndpoints + Microsoft.Cdn/profiles/afdEndpoints/routes + Microsoft.Cdn/profiles/originGroups + Microsoft.Cdn/profiles/originGroups/origins + Microsoft.Cdn/profiles/securityPolicies + +Microsoft.Network/FrontDoorWebApplicationFirewallPolicies +``` + +For private origins, each member cluster also owns: + +```text +Microsoft.Network/loadBalancers (internal frontend) +Microsoft.Network/privateLinkServices + privateEndpointConnections requested by AFD +``` + +The member Service and its cloud provider own the load balancer and PLS. The hub AFD +controller consumes their resource IDs but does not adopt or delete them. + +### Resource ownership + +The first implementation should support two AFD profile modes: + +- `Managed`: Fleet creates and owns the AFD profile and all child resources it names. +- `Existing`: Fleet references an operator-owned profile and owns only child resources + carrying Fleet ownership tags. + +WAF policy lifecycle is independent: + +- `Existing` is required initially. The policy contains an Azure resource ID and the + controller creates the AFD security-policy association. +- Creating and editing WAF rule sets is a later capability because it crosses a security + administration boundary. + +Every created Azure resource must carry stable ownership tags containing the hub cluster +identity and the owning Kubernetes UID. Deletion must remove only resources with matching +ownership. + +## Candidate API + +### AzureFrontDoorGatewayPolicy + +This policy targets one `Gateway` and controls profile-wide and frontend settings. + +```yaml +apiVersion: networking.fleet.azure.com/v1alpha1 +kind: AzureFrontDoorGatewayPolicy +metadata: + name: global-ingress + namespace: app +spec: + targetRef: + group: gateway.networking.k8s.io + kind: Gateway + name: global + profile: + mode: Managed + sku: Premium_AzureFrontDoor + resourceGroup: fleet-global + name: fleet-global + waf: + required: true + policyResourceID: /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/security/providers/Microsoft.Network/frontDoorWebApplicationFirewallPolicies/fleet-waf + diagnostics: + enabled: true + destinationResourceID: /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/observability/providers/Microsoft.OperationalInsights/workspaces/fleet +``` + +Proposed fields: + +| Field | Meaning | +| --- | --- | +| `targetRef` | Same-namespace `Gateway` controlled by the Fleet AFD GatewayClass | +| `profile.mode` | `Managed` or `Existing` | +| `profile.sku` | `Standard_AzureFrontDoor` or `Premium_AzureFrontDoor` | +| `profile.resourceGroup/name` | Stable Azure placement and identity | +| `waf.required` | Fail route programming if no valid WAF association exists | +| `waf.policyResourceID` | Existing WAF policy to associate with route domains | +| `diagnostics` | Access, health-probe, and WAF diagnostic destination | + +The policy must not contain credentials. The controller uses workload identity. + +### AzureFrontDoorBackendAttachment + +This attachment defines how one Gateway consumes one `ServiceImport` port through its +AFD profile. The attachment, rather than the `ServiceImport`, owns the resulting origin +group and member origins. This allows the same logical service to use different public or +Private Link, TLS, probe, and traffic settings in different Gateways without overloading +the portable `ServiceImport`. + +```yaml +apiVersion: networking.fleet.azure.com/v1alpha1 +kind: AzureFrontDoorBackendAttachment +metadata: + name: global-store-https + namespace: app +spec: + gatewayRef: + group: gateway.networking.k8s.io + kind: Gateway + name: global + backendRef: + group: networking.fleet.azure.com + kind: ServiceImport + name: store + port: 443 + connectivity: + mode: PrivateLink + privateLink: + approval: Manual + regionSelection: ClosestSupported + origin: + protocol: HTTPS + hostHeader: store.internal.contoso.example + certificateSubjectNameCheck: true + healthProbe: + protocol: HTTPS + method: HEAD + path: /healthz + intervalSeconds: 30 + sampleSize: 4 + successfulSamplesRequired: 3 + traffic: + defaultPriority: 1 + defaultWeight: 1000 + memberFailurePolicy: Partial +``` + +Proposed fields: + +| Field | Meaning | +| --- | --- | +| `gatewayRef` | Same-namespace `Gateway` whose gateway policy resolves the AFD profile | +| `backendRef` | `ServiceImport` and required service port consumed by this Gateway | +| `connectivity.mode` | `Public` or `PrivateLink` | +| `privateLink.approval` | `Manual`; `Controller` is deferred until safe correlation is proven | +| `privateLink.regionSelection` | `MemberRegion` when supported, otherwise `ClosestSupported` | +| `origin.protocol` | AFD-to-origin connection protocol for `backendRef.port` | +| `origin.hostHeader` | HTTP Host header sent to every member origin | +| `certificateSubjectNameCheck` | Must be `true` for Private Link and defaults to `true` for public HTTPS | +| `healthProbe` | AFD origin-group health settings | +| `traffic.defaultPriority/defaultWeight` | Defaults before per-member overrides | +| `memberFailurePolicy` | `All` or `Partial` origin eligibility | + +The attachment identity is the tuple of Gateway UID, `ServiceImport` UID, and service +port. Only one attachment may be accepted for a tuple. Admission rejects a duplicate +when it can observe the incumbent. Reconciliation remains deterministic under concurrent +creation: the oldest attachment by creation timestamp, with UID as the tie-breaker, +remains accepted and later duplicates report `Accepted=False` with reason `Conflicted`. +A new duplicate must not disrupt a programmed attachment. + +An `HTTPRoute` backend is programmed only when its parent Gateway and +`ServiceImport`/port reference match an accepted attachment. Multiple routes may share +that attachment and its origin group. An accepted but unused attachment does not create +Azure resources. If one Gateway needs two different origin contracts for the same +`ServiceImport` port, the initial API requires separate Gateways rather than ambiguous +per-route overrides. + +The initial API requires the attachment, Gateway, and `ServiceImport` to share a +namespace. A later cross-namespace `backendRef` may be enabled only with an applicable +Gateway API `ReferenceGrant` in the `ServiceImport` namespace. + +### Provider API validation + +Admission and reconciliation must enforce: + +- `PrivateLink` requires `Premium_AzureFrontDoor`. +- One attachment cannot mix public and private member origins in its AFD origin group. +- `gatewayRef`, `backendRef`, and `backendRef.port` must resolve. +- `gatewayRef`, `backendRef`, and `backendRef.port` are immutable. +- A route backend must have an accepted attachment matching its parent Gateway, + `ServiceImport`, and port before the route can be programmed. +- HTTPS Private Link requires certificate subject-name validation. +- `backendRef.port` must exist in `ServiceImport.status.ports`. +- HTTP and HTTPS are the only AFD origin protocols. +- Health probes use HTTP or HTTPS and `GET` or `HEAD`. +- `sampleSize >= successfulSamplesRequired > 0`. +- An origin weight and priority fit Azure limits. +- A WAF policy tier matches the AFD profile tier. +- `waf.required: true` rejects an empty or inaccessible policy resource ID. +- No controller-managed resource can collide with an existing resource lacking matching + ownership tags. + +## Member-to-Hub Transport Contract + +`ServiceImport.status` remains portable and must not expose Azure resource IDs or private +endpoint connection state. `InternalServiceExportSpec` is the internal transport seam. + +The member controller should publish an observed endpoint structure similar to: + +```yaml +spec: + serviceReference: + namespace: app + name: store + serviceType: LoadBalancer + azure: + location: eastus2 + loadBalancer: + internal: true + frontendAddress: 10.20.0.10 + privateLinkService: + resourceID: /subscriptions/.../privateLinkServices/pls-store + alias: pls-store....azure.privatelinkservice + provisioningState: Succeeded +``` + +For public Services it should publish: + +- public IP resource ID; +- public FQDN derived from the public IP DNS settings; +- public IP provisioning readiness; +- whether `AzureFrontDoor.Backend` is allowed by the Service's Azure service-tag + annotation; +- member Azure region. + +For private Services it should publish: + +- internal load balancer address; +- PLS resource ID and alias; +- PLS Azure region; +- PLS provisioning state; +- observed visibility configuration; +- observed private endpoint connections relevant to the managed AFD profile. + +The internal fields are observed state, not a second source of desired configuration. +The member Service annotations remain authoritative for load balancer and PLS creation. + +## Public Origin Topology + +### Member Service + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: store + namespace: app + annotations: + service.beta.kubernetes.io/azure-dns-label-name: store-eastus2 + service.beta.kubernetes.io/azure-allowed-service-tags: AzureFrontDoor.Backend +spec: + type: LoadBalancer + selector: + app: store + ports: + - name: https + protocol: TCP + port: 443 + targetPort: 8443 +--- +apiVersion: networking.fleet.azure.com/v1beta1 +kind: ServiceExport +metadata: + name: store + namespace: app +``` + +`azure-allowed-service-tags` and `loadBalancerSourceRanges` must not be used together. +AKS rejects that combination. The public origin must also validate `X-Azure-FDID` at the +application or regional ingress layer. The service tag blocks non-AFD source ranges; the +header binds requests to this specific AFD profile because AFD backend addresses are +shared across customers. + +The header value is an output of the managed AFD profile. The controller reports it in +Gateway policy status; an application delivery mechanism outside this proposal injects +it into the regional ingress policy. `Programmed=True` must not be reported for a public +backend until the operator confirms bypass protection, unless an explicit unsafe +development override is enabled. + +### Public request path + +```text +Client + -> AFD edge listener + -> WAF security policy + -> AFD route + -> public origin group + -> member public Standard Load Balancer + -> Service endpoints + -> application Pods +``` + +AFD uses the member FQDN as the origin hostname. For HTTPS, the member endpoint must +present a certificate whose subject matches that hostname. The controller does not +disable subject-name validation. + +### Public origin eligibility + +A member becomes eligible only when: + +- the `InternalServiceExport` is valid and selected by the `ServiceImport`; +- the Service is a public `LoadBalancer`; +- the exported port exists; +- public IP and FQDN discovery succeeded; +- bypass protection is observed or acknowledged; +- the origin certificate and host contract are configured; +- the member is not deleting or explicitly disabled. + +An AFD health probe decides runtime health after programming. Kubernetes +`Programmed=True` means the Azure desired configuration exists; it does not mean the +origin is healthy. + +## Private Origin Topology + +### Member Service + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: store + namespace: app + annotations: + service.beta.kubernetes.io/azure-load-balancer-internal: "true" + service.beta.kubernetes.io/azure-pls-create: "true" + service.beta.kubernetes.io/azure-pls-name: pls-store +spec: + type: LoadBalancer + selector: + app: store + ports: + - name: https + protocol: TCP + port: 443 + targetPort: 8443 +--- +apiVersion: networking.fleet.azure.com/v1beta1 +kind: ServiceExport +metadata: + name: store + namespace: app +``` + +Omitting PLS visibility keeps the service at its most restrictive, RBAC-only default. +Broad visibility (`"*"`) is not an acceptable controller default. Auto-approval must not +be configured unless its subscription list is a subset of visibility and the security +owner explicitly opts in. + +Member prerequisites: + +- AKS Standard Load Balancer; +- `nodeIPConfiguration` backend-pool type; +- IPv4 and TCP for AFD HTTP/HTTPS traffic; +- a PLS NAT subnet with sufficient addresses; +- if `externalTrafficPolicy: Local`, the PLS subnet differs from the Pod subnet; +- PROXY protocol remains disabled unless the backend and health-probe behavior support it. + +### Private request path + +```text +Client + -> AFD Premium edge listener + -> WAF security policy + -> AFD route + -> private origin group + -> AFD-managed regional private endpoint + -> member Private Link Service + -> member internal Standard Load Balancer + -> Service endpoints + -> application Pods +``` + +The AFD origin references the PLS resource ID and chooses the same supported Private Link +region as the member, or the nearest supported region. The origin hostname is used for +SNI and must match the server certificate. The origin host header is independently +configurable. + +### Private Link state machine + +```text +PLSNotFound + -> PLSProvisioning + -> PLSReady + -> AFDRequestCreated + -> ApprovalPending + -> Approved + -> ConnectionEstablished + -> OriginProgrammed + -> ProbeHealthy +``` + +Terminal or degraded branches include: + +- `PLSFailed`; +- `ApprovalRejected`; +- `ConnectionDisconnected`; +- `OriginProgrammingFailed`; +- `ProbeUnhealthy`. + +In the initial release, approval is manual. The controller emits an event containing the +PLS resource ID, expected AFD profile, and request message, and requeues without treating +the pending state as an error. A later `Controller` mode may approve requests only after +the Azure API exposes enough stable identity to correlate exactly one pending request to +the owning AFD origin. It must never approve an arbitrary new PLS connection. + +AFD traffic and health probes both use the private path. The controller must not create a +temporary public origin while approval is pending. + +### Private endpoint reuse and ports + +Within one AFD profile, origins with the same PLS resource ID, group ID, and Private Link +region share one AFD-managed private endpoint and therefore one approval. A change to any +of those values creates another request. + +AFD documents a routing limitation when identical Private Link resource/group/region +tuples are used with different origin ports. Validation must reject that configuration +within one profile. + +For regional resilience, distinct member origins should use distinct Private Link +regions. This avoids concentrating all private traffic through one AFD regional cluster. + +## Hub Example + +The same hub routing YAML works for either topology; only the backend attachment +connectivity changes. + +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: global + namespace: app +spec: + gatewayClassName: fleet-azure-front-door + listeners: + - name: https + protocol: HTTPS + port: 443 + hostname: store.contoso.com + tls: + mode: Terminate + certificateRefs: + - group: networking.fleet.azure.com + kind: AzureFrontDoorCertificate + name: store +--- +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: store + namespace: app +spec: + parentRefs: + - name: global + hostnames: + - store.contoso.com + rules: + - backendRefs: + - group: networking.fleet.azure.com + kind: ServiceImport + name: store + port: 443 + weight: 100 +--- +apiVersion: networking.fleet.azure.com/v1alpha1 +kind: AzureFrontDoorBackendAttachment +metadata: + name: global-store-https + namespace: app +spec: + gatewayRef: + group: gateway.networking.k8s.io + kind: Gateway + name: global + backendRef: + group: networking.fleet.azure.com + kind: ServiceImport + name: store + port: 443 + connectivity: + mode: PrivateLink + privateLink: + approval: Manual + regionSelection: ClosestSupported + origin: + protocol: HTTPS + hostHeader: store.internal.contoso.example + certificateSubjectNameCheck: true + healthProbe: + protocol: HTTPS + method: HEAD + path: /healthz + intervalSeconds: 30 + sampleSize: 4 + successfulSamplesRequired: 3 + traffic: + defaultPriority: 1 + defaultWeight: 1000 + memberFailurePolicy: Partial +``` + +`HTTPRoute.backendRefs[].weight` divides traffic between logical backends in one route. +It does not replace the per-member origin weight inside the selected `ServiceImport`. + +## Traffic Policy + +Per-member priority and weight are resolved in this order: + +1. an explicit per-member override in the backend attachment, if added after API review; +2. the existing `ServiceExport` weight annotation; +3. backend-attachment defaults. + +Weights are normalized into AFD's accepted range while preserving relative ratios. +Priority takes precedence over weight: only healthy origins at the highest available +priority receive traffic, and weight divides traffic within that priority. + +Member draining is two phase: + +1. disable the AFD origin and wait at least the configured drain interval; +2. delete the origin after active traffic has quiesced. + +Deleting a `ServiceExport`, leaving the Fleet, or losing endpoint readiness enters the +same drain path. A transient read failure does not immediately delete an origin. + +`memberFailurePolicy` controls desired-state eligibility: + +- `All`: any invalid member prevents origin-group updates and keeps the last known good + configuration. +- `Partial`: valid members are programmed and invalid members appear in status. + +`Partial` is recommended for availability, but the route is not accepted if zero origins +are eligible. + +## WAF and Frontend Security + +AFD WAF protection is attached through an AFD security policy that associates the WAF +policy with the domains used by routes. A WAF policy may be in: + +- `Detection` mode for staged rollout; +- `Prevention` mode for enforcement. + +Managed rule sets require AFD Premium. Standard supports custom rules but not managed rule +sets. Because the private topology already requires Premium, the recommended production +default for both topologies is Premium with managed rules in prevention mode after an +observed detection period. + +The controller verifies that every programmed custom domain requiring WAF is present in +the security-policy association. It does not report the Gateway `Programmed=True` while +the association is missing. + +Public origin security requires both: + +- NSG filtering through `AzureFrontDoor.Backend`; +- exact `X-Azure-FDID` validation. + +Private origin security relies on a non-public internal load balancer plus PLS. NSGs may +further restrict the VNet, but the member Service is not internet routable. + +## TLS Semantics + +There are two independent TLS hops: + +1. client to AFD listener, configured by the `Gateway` listener certificate reference; +2. AFD to member origin, configured by `AzureFrontDoorBackendAttachment.origin`. + +The controller must never infer the origin protocol from the listener protocol. HTTPS at +the edge may use HTTP or HTTPS to the origin, although HTTPS is recommended. + +For origin HTTPS: + +- use a DNS hostname, not an ephemeral IP, when possible; +- the origin hostname drives SNI; +- keep certificate subject-name validation enabled; +- report a clear condition if Fleet cannot determine a stable hostname; +- do not claim support for client certificates or AFD backend mTLS. + +## Status and Conditions + +### Gateway policy status + +```yaml +status: + observedGeneration: 3 + profile: + resourceID: /subscriptions/.../profiles/fleet-global + frontDoorID: 11111111-1111-1111-1111-111111111111 + conditions: + - type: Accepted + status: "True" + reason: Valid + - type: ProfileReady + status: "True" + reason: Succeeded + - type: WAFReady + status: "True" + reason: Associated + - type: DiagnosticsReady + status: "True" + reason: Configured +``` + +### Backend attachment status + +```yaml +status: + observedGeneration: 2 + conditions: + - type: Accepted + status: "True" + reason: Valid + - type: ResolvedRefs + status: "True" + reason: Resolved + - type: OriginsProgrammed + status: "False" + reason: PrivateLinkApprovalPending + members: + - clusterName: member-east + originName: store-member-east + connectivity: PrivateLink + privateLink: + serviceResourceID: /subscriptions/.../privateLinkServices/pls-store + connectionState: Pending + conditions: + - type: TransportReady + status: "True" + reason: PLSProvisioned + - type: ConnectionReady + status: "False" + reason: ApprovalPending +``` + +Condition rules: + +- conditions include `observedGeneration`; +- expected asynchronous states use `False` with specific reasons, not generic errors; +- Azure errors retain a sanitized error code and correlation ID; +- status is bounded and does not copy unbounded Azure response text; +- probe health is reported separately from configuration programming; +- events are emitted on transitions, not every retry. + +`Gateway` and `HTTPRoute` standard conditions are updated only by their owning +reconcilers. Gateway-policy and backend-attachment status supply Azure-specific detail. + +## Reconciliation Model + +The hub controller builds an immutable normalized model from Gateway API, Fleet, and +provider configuration objects. Azure reconcilers consume only that model. + +Reconciliation order: + +1. validate GatewayClass, Gateway, gateway policy, backend attachments, listeners, and + route references, including an exact attachment match for every AFD route backend; +2. resolve each accepted attachment's `ServiceImport` to current + `InternalServiceExport` objects; +3. classify every member as public, private, invalid, or pending; +4. reject mixed connectivity for one backend; +5. ensure the profile, endpoint, domains, and certificates; +6. ensure the WAF security-policy association; +7. ensure homogeneous origin groups; +8. ensure origins, including Private Link request properties; +9. ensure routes only after required origins and WAF associations exist; +10. update provider and Gateway API status. + +The finalizer is added immediately before the first owned Azure resource is created. +Deletion lists Azure resources by ownership tags, disables routes/origins, drains traffic, +deletes owned child resources, then removes the finalizer. A referenced PLS, load balancer, +public IP, or existing WAF policy is never deleted. + +Drift policy: + +- owned mutable fields are reconciled to desired state; +- operator changes to non-owned fields are preserved; +- ownership-tag loss or resource replacement is reported as a conflict, not adopted; +- the last known good route remains when a transient dependency read fails; +- invalid desired changes do not destructively replace a working origin group. + +## Identity and RBAC + +Kubernetes RBAC requires read/watch access to Gateway API objects, gateway policies, +backend attachments, `ServiceImport`, and `InternalServiceExport`, plus +status/finalizer updates only for owned types. The initial same-namespace attachment rule +keeps authorization explicit. Future cross-namespace backend references require a +`ReferenceGrant` from the `ServiceImport` namespace before the attachment is accepted. + +The hub workload identity requires least-privilege Azure actions for: + +- AFD profiles and owned child resources; +- reading referenced WAF policies and writing AFD security-policy associations; +- diagnostic settings when enabled; +- reading public IPs and PLS resources; +- reading private endpoint connection state. + +Approval permissions on member PLS resources are not required for the initial manual +approval mode. If controller approval is added, it receives a separate opt-in identity or +role assignment so profile management does not imply permission to approve private +network access. + +## Observability + +Metrics should include: + +- reconcile count, latency, result, and Azure error class; +- eligible, pending, invalid, programmed, and deleting origins; +- Private Link requests by state and state age; +- WAF association readiness; +- Azure API throttling and retry-after duration; +- configuration-to-programmed latency. + +Azure diagnostics should enable AFD access logs, health-probe logs, and WAF logs. Logs and +events correlate: + +- Kubernetes object UID; +- AFD profile, endpoint, origin group, and origin names; +- Fleet member cluster name; +- Azure operation/correlation ID. + +Alerts should cover zero healthy origins, prolonged approval pending, WAF association +loss, repeated authorization failures, quota exhaustion, and high configuration latency. + +## Failure Matrix + +| Failure | Controller behavior | Traffic behavior | +| --- | --- | --- | +| Public IP/FQDN pending | Keep member pending | Other healthy origins continue | +| Missing public bypass control | Reject member | No direct origin programming | +| PLS provisioning | Requeue with pending condition | No public fallback | +| Private endpoint approval pending | Emit transition event; poll slowly | Other established origins continue | +| Private endpoint rejected | Mark member invalid until intervention | Origin remains disabled | +| Origin certificate mismatch | Report TLS contract failure | AFD marks origin unhealthy | +| WAF policy missing/inaccessible | Do not enable route | Fail closed | +| One invalid member with `Partial` | Program valid members | Reduced capacity | +| One invalid member with `All` | Preserve last known good config | Existing traffic continues | +| Azure throttling | Honor `Retry-After`, exponential backoff | Existing config continues | +| Ownership conflict | Refuse adoption or deletion | Existing Azure resource unchanged | +| ServiceImport deleted | Disable, drain, delete owned origins/group | Route ceases after drain | + +## Scale and Quotas + +Before programming, the controller estimates profile resource consumption for origins, +origin groups, routes, domains, private endpoints, and WAF associations. It reports a +`QuotaExceeded` condition before partial creation where possible. + +Private Link has an additional Front Door regional-cluster protection limit. Workloads +expecting high request rates should use multiple origins in different Private Link +regions and validate current Azure limits during capacity planning. + +Controller implementation must: + +- use shared informers and indexes instead of namespace scans; +- deduplicate Azure reads within a reconciliation; +- limit concurrent Azure mutations per profile; +- serialize changes that would temporarily mix public and private origins; +- use deterministic Azure names with hashes to stay within service limits. + +## Rollout and Migration + +1. Introduce APIs, internal transport fields, normalized model, and feature gates without + Azure writes. +2. Enable read-only validation/status in test environments. +3. Implement public origin groups and routes against new profiles. +4. Add mandatory WAF association and public bypass conformance checks. +5. Implement private PLS discovery and manual approval workflow. +6. Run public and private canaries in isolated AFD profiles. +7. Add opt-in adoption of compatible prototype annotations through a one-time conversion + tool; do not dual-write annotations and typed resources indefinitely. +8. Promote the APIs only after multi-region failure, deletion, and quota tests pass. + +Migration uses dual-read, single-write reconciliation. An AFD origin group may be owned +by either legacy annotations or the typed backend attachment, never both. Duplicate +ownership is a hard conflict. + +## Alternatives Considered + +### Use annotations for all provider configuration + +Rejected as the durable API. Annotations are expedient but lack schema, structured status, +field-level validation, discoverability, and conflict semantics. They may remain only as a +temporary compatibility input. + +### Point GEP-4894 selectorRef at ServiceImport + +Rejected. This would be a Fleet-specific reinterpretation, not upstream conformance. The +current selector is Pod-oriented and namespace-local. + +### Materialize member Pods or EndpointSlices on the hub + +Rejected. It expands hub cardinality, creates stale identity and readiness risks, and +misrepresents endpoints that AFD cannot reach directly. + +### One origin for the logical ServiceImport + +Rejected for direct Service mode. AFD needs independently addressable member origins for +health, latency, weight, priority, drain, and Private Link approval. + +### Attach one backend policy directly to ServiceImport + +Rejected as the initial API because origin groups, probes, connectivity, and traffic +settings are scoped to a particular AFD profile. A direct policy would force every +Gateway consuming the service to share one contract, make per-Gateway readiness status +ambiguous, and create many-to-many Azure cleanup ownership. The explicit attachment +keeps `ServiceImport` portable and permits deliberate reuse across profiles. + +### Mix public and private members + +Rejected within a backend/origin group because AFD prohibits the topology. An application +that requires both must use separate Gateways/routes/origin groups and an explicit traffic +migration procedure. + +### Automatically approve every AFD Private Link request + +Rejected. Approval is a network access grant. Automation must prove exact request +ownership before it can be safely enabled. + +## Open Questions + +- Should Gateway policy support only managed profiles in the first release, reducing + ownership ambiguity? +- What stable Azure metadata identifies an AFD private endpoint request strongly enough + for safe automatic approval? +- Should the member agent only observe PLS configuration, or validate security defaults + such as restrictive visibility? +- Which resource reports AFD probe health when Azure exposes it asynchronously? +- How should custom-domain certificate ownership integrate with the existing + `AzureFrontDoorCertificate` design? +- Should public bypass validation be mandatory admission, observed status, or an external + conformance check when `X-Azure-FDID` enforcement is application-owned? +- What is the exact API for per-member priority overrides without overloading the existing + ServiceExport weight annotation? + +## References + +- [GEP-4894: Backend API for Gateway API](https://gateway-api.sigs.k8s.io/geps/gep-4894/) +- [Gateway API PR 5158](https://github.com/kubernetes-sigs/gateway-api/pull/5158) +- [Secure traffic to Azure Front Door origins](https://learn.microsoft.com/azure/frontdoor/origin-security) +- [Origins and origin groups in Azure Front Door](https://learn.microsoft.com/azure/frontdoor/origin) +- [Secure your origin with Private Link](https://learn.microsoft.com/azure/frontdoor/private-link) +- [Connect AFD to an internal load balancer with Private Link](https://learn.microsoft.com/azure/frontdoor/standard-premium/how-to-enable-private-link-internal-load-balancer) +- [Use an internal load balancer with AKS](https://learn.microsoft.com/azure/aks/internal-lb) +- [Configure a public Standard Load Balancer in AKS](https://learn.microsoft.com/azure/aks/configure-load-balancer-standard) +- [WAF on Azure Front Door](https://learn.microsoft.com/azure/web-application-firewall/afds/afds-overview) +- [Azure Front Door Manager and security policies](https://learn.microsoft.com/azure/frontdoor/manager) +- [SFI NS 2.5.3 KPI](https://eng.ms/docs/initiatives/project-standard/standards-categories/sc-networking/ddos/sfi-ns/sfi-ns253-kpi) +- [Application DDoS Standard](https://eng.ms/docs/initiatives/project-standard/standards-categories/sc-networking/ddos/ads/index) +- `docs/design/gep-4894-backend-evaluation.md` +- `api/v1alpha1/serviceimport_types.go` +- `api/v1alpha1/internalserviceexport_types.go` +- `pkg/controllers/member/serviceexport/controller.go` +- `pkg/controllers/hub/serviceimport/controller.go` +- `pkg/controllers/hub/trafficmanagerbackend/controller.go` +- `pkg/controllers/hub/trafficmanagerprofile/controller.go` diff --git a/docs/design/gep-4894-backend-evaluation.md b/docs/design/gep-4894-backend-evaluation.md new file mode 100644 index 00000000..b5df6048 --- /dev/null +++ b/docs/design/gep-4894-backend-evaluation.md @@ -0,0 +1,700 @@ +# GEP-4894 Backend evaluation for Fleet global ingress + +## Document status + +| Field | Value | +|---|---| +| Status | Evaluation | +| Date | 2026-08-25 | +| GEP-4894 status | Experimental | +| AFD RFC revision | `bb6ed4e6ba859d4895c9647568e792905ff70038` | +| GEP-1748 branch revision | `7bf9918ce41b10ae268a9acfdd21193cd92411ad` | + +## Executive recommendation + +GEP-4894 can contribute to the Fleet global ingress API, but it cannot +currently replace either GEP-1748 `ServiceImport` integration or the +Fleet/Azure backend policy described in the AFD global ingress RFC. + +The resources have different responsibilities: + +- Gateway API `Backend` describes how one Gateway client connects to a + destination. +- Fleet `ServiceImport` describes one logical Service whose endpoints span + member clusters. +- Fleet/Azure configuration describes how those member endpoints become AFD + origins, including public or Private Link connectivity, health probes, + placement, priority, weight, and provider-specific lifecycle. + +The recommended target is an additive model: + +```text +HTTPRoute + -> Backend consumer connection contract + -> stable endpoint binding future upstream integration point + -> Fleet ServiceImport logical multi-cluster Service + -> member export state public endpoint or PLS per cluster + -> AFD origin group provider reconciliation +``` + +Fleet should retain direct `HTTPRoute` references to `ServiceImport` until +Gateway API defines a stable endpoint binding that can represent a +multi-cluster endpoint producer. Both the current path and a future `Backend` +path should normalize into the existing provider-neutral internal model. + +## Sources evaluated + +This evaluation compares: + +- GEP-4894, which introduces the Experimental `Backend` resource. +- GEP-1748, which defines Experimental and Extended support for using MCS + `ServiceImport` as a Gateway API backend. +- `rchinchani/afd-global-ingress-rfc`, which defines the product requirements, + origin models, security model, and candidate Fleet/Azure policy APIs. +- `rchinchani/gep-1748-gateway-api`, which defines and partially implements a + Fleet AFD Gateway controller based on direct `ServiceImport` references. +- Gateway API PR 5158, which proposes removing + `Backend.spec.endpointSelector.selectorRef` for now because its upstream + `EndpointSelector` dependency does not exist. + +GEP-4894 is not a stable API contract. In particular, its endpoint-selection +shape is changing while this evaluation is being written. Fleet must treat the +GEP as design input rather than a dependency baseline. + +## What GEP-4894 provides + +GEP-4894 proposes a namespace-scoped, consumer-owned `Backend` with two +destination types: + +| Type | Purpose | Conformance | +|---|---|---| +| `EndpointSelector` | Select internal endpoints and attach connection configuration | Core | +| `ExternalHostname` | Represent one external FQDN without an `ExternalName` Service | Extended | + +The resource also provides: + +- an explicit backend port; +- protocol metadata; +- inline server TLS validation; +- an optional client certificate for mutual TLS; +- parent-scoped status; and +- a future attachment point for retries, session persistence, timeouts, load + balancing, and health checks. + +GEP-4894 requires the `Backend` and referring Route to be in the same +namespace. The current draft allows an `EndpointSelector` backend to refer to a +producer-side endpoint selector across namespaces, but that field is the +subject of active upstream revision. + +## Existing Fleet global ingress model + +The evaluated GEP-1748 design uses: + +```text +Gateway + -> HTTPRoute + -> Fleet ServiceImport + -> InternalServiceExport records + -> one origin per eligible member cluster +``` + +The `ServiceImport` backend maps to an AFD origin group. Each contributing +member export maps to an AFD origin. The current normalized model preserves two +different traffic controls: + +- `HTTPRoute.backendRefs[*].weight` distributes traffic between logical + backends. +- Fleet `ServiceExport` weight distributes traffic between member-cluster + origins inside one logical backend. + +The AFD RFC additionally requires: + +- `DirectService` and `ClusterGateway` origin providers; +- public and Private Link connectivity; +- active-active and active-passive placement; +- per-cluster priority and weight overrides; +- health probe configuration; +- AFD Standard and Premium selection; +- WAF and diagnostics; +- certificate and custom-domain lifecycle; +- Azure ownership, deletion, and drift rules; +- PLS discovery, region selection, and approval state; and +- detailed origin programming and health status. + +These requirements are destination topology and provider lifecycle concerns. +They are not equivalent to client protocol or TLS configuration. + +## Compatibility assessment + +| Requirement | GEP-4894 fit | Result | +|---|---|---| +| Route references an explicit backend object | Direct | Stronger discoverability than policy applied to `ServiceImport` | +| Consumer-specific backend TLS | Direct | Inline `Backend.spec.tls` is a good fit | +| Backend protocol metadata | Direct | `Backend.spec.protocol` is a good fit | +| One external FQDN | Direct | `ExternalHostname` is a good fit | +| Existing cluster-local Service decoration | Intended, but endpoint binding is unstable | Conditional | +| Fleet `ServiceImport` endpoint aggregation | Not defined | Gap | +| Dynamic member-cluster origins | Not defined | Gap | +| Public versus Private Link origin connectivity | Not defined | Fleet/Azure extension remains required | +| Direct Service versus shared cluster Gateway origin | Not defined | Fleet/Azure extension remains required | +| Per-cluster priority and weight | Not defined | Fleet placement policy remains required | +| AFD health probe contract | Only identified as a future Backend field | Fleet/Azure extension remains required | +| AFD SKU, WAF, diagnostics, resource ownership | Out of scope | Gateway-level Azure policy remains required | +| Frontend listener certificate | Different TLS direction | Gateway listener and AFD certificate design remain required | +| Parent-scoped backend status | Partial | Useful for connection readiness, insufficient for member-origin detail | +| GEP-1748 Extended conformance | Independent | Supporting `Backend` does not provide multi-cluster conformance | + +## Important semantic distinctions + +### Backend is not ServiceImport + +`Backend` is consumer-side connection intent. `ServiceImport` is +multi-cluster service discovery and endpoint aggregation. Replacing +`ServiceImport` with `Backend` would remove the source of: + +- contributing member-cluster identity; +- member eligibility; +- exported ports; +- per-cluster endpoint readiness; +- per-cluster weight; +- public load balancer state; and +- PLS resource ID, location, and readiness. + +Fleet still needs `ServiceImport` and internal export state even if users +eventually reference a `Backend` from `HTTPRoute`. + +### Inline backend TLS is not listener TLS + +GEP-4894 TLS controls the connection from AFD to an origin. Gateway listener +TLS controls the connection from the client to AFD. The latter still requires +AFD custom-domain validation and an AFD-managed or Key Vault certificate +lifecycle. + +The two directions must remain independently configurable: + +```text +client -- listener TLS --> AFD -- Backend TLS --> member origin +``` + +### ExternalHostname is not a multi-cluster origin set + +`ExternalHostname` represents one destination hostname. A Fleet backend is a +logical destination containing a dynamic set of member-cluster origins, each +with independent health, priority, weight, connectivity, and lifecycle. + +Creating one `ExternalHostname` Backend per cluster would expose cluster +membership in application Routes, force Route churn when membership changes, +and conflate route-level weights with per-cluster placement. It would also not +represent Private Link resource identity. This is not a recommended mapping. + +## Kubernetes YAML shape + +### Current GEP-1748 Fleet shape + +The current design is implementable with the existing Fleet APIs: + +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: api + namespace: contoso +spec: + parentRefs: + - name: contoso-global + rules: + - backendRefs: + - group: networking.fleet.azure.com + kind: ServiceImport + name: api + port: 443 +``` + +Fleet resolves `ServiceImport/contoso/api` into member origins and the AFD +provider receives one normalized logical backend. + +### GEP-4894 shape for a single external destination + +The `ExternalHostname` case is independently useful and does not require +multi-cluster discovery: + +```yaml +apiVersion: gateway.networking.k8s.io/v1alpha1 +kind: Backend +metadata: + name: partner-api + namespace: contoso +spec: + type: ExternalHostname + externalHostname: + hostname: api.partner.example + port: 443 + protocol: HTTP2 + tls: + mode: ServerOnly + validation: + wellKnownCACertificates: System +--- +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: partner-api + namespace: contoso +spec: + parentRefs: + - name: contoso-global + rules: + - backendRefs: + - group: gateway.networking.k8s.io + kind: Backend + name: partner-api + port: 443 +``` + +The exact Experimental fields may change. This example must not be treated as +a stable Fleet API commitment. + +### Future multi-cluster shape + +The desired user experience is: + +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: api + namespace: contoso +spec: + parentRefs: + - name: contoso-global + rules: + - backendRefs: + - group: gateway.networking.k8s.io + kind: Backend + name: api + port: 443 +--- +apiVersion: gateway.networking.k8s.io/v1alpha1 +kind: Backend +metadata: + name: api + namespace: contoso +spec: + type: EndpointSelector + port: 443 + protocol: HTTP + tls: + mode: ServerOnly + endpointSelector: + selectorRef: + name: api + namespace: contoso-backends +``` + +This shape is only viable if a stable upstream endpoint-selection resource can +represent endpoints derived from Fleet `ServiceImport/contoso-backends/api`. +The Fleet controller would own the translation from `ServiceImport` and +internal export state to that endpoint resource. + +It is not viable today because: + +1. the referenced upstream `EndpointSelector` API does not exist; +2. Gateway API PR 5158 proposes deferring `selectorRef`; +3. the proposed embedded pod selector cannot select member workloads from the + Fleet hub; and +4. a direct `selectorRef` to Fleet `ServiceImport` would violate the current + GEP semantics. + +Fleet must not reinterpret the flexible object-reference syntax as permission +to put `kind: ServiceImport` in `selectorRef`. + +## Scenario evaluation + +| Scenario | Feasibility with GEP-4894 | Required design | +|---|---|---| +| Direct public Service in one cluster | Conditional | Stable Service or endpoint binding plus AFD provider mapping | +| Direct public Services across Fleet members | Conditional | Retain `ServiceImport` aggregation behind a stable endpoint binding | +| Direct private Services with PLS | Conditional | Retain Fleet PLS discovery and Azure connectivity policy | +| Shared public cluster Gateway | Conditional | Retain `ClusterGateway` provider selection and local Gateway orchestration | +| Shared private cluster Gateway with PLS | Conditional | Same as shared public plus PLS lifecycle | +| One external Internet hostname | Feasible | Implement `ExternalHostname` Extended support | +| Active-active multi-region | Conditional | Retain Fleet placement, priority, weight, and health semantics | +| Active-passive multi-region | Conditional | Retain explicit origin priority and failover semantics | +| Cross-namespace producer backend | Blocked on stable endpoint binding | Route-local `Backend`; producer-side endpoint object with authorization | +| Existing direct ServiceImport Route | Feasible now | Keep GEP-1748 path during migration | + +No required global ingress scenario becomes impossible because of GEP-4894. +However, only the single external-hostname and consumer TLS/protocol scenarios +are directly solved by it. The core Fleet multi-cluster scenarios remain +conditional on a separate endpoint aggregation and provider policy layer. + +## Recommended target architecture + +### Public resources + +Use portable Gateway API resources for portable intent: + +- `GatewayClass` selects the Fleet AFD implementation. +- `Gateway` defines listeners, hostnames, and route delegation. +- `HTTPRoute` defines HTTP matching, filters, and logical backend weights. +- `Backend`, after its API is stable, defines consumer protocol and origin TLS. +- `ServiceExport` and `ServiceImport` define multi-cluster service discovery. + +Use Fleet/Azure APIs only for non-portable behavior: + +- AFD SKU, WAF, diagnostics, Azure placement, and ownership; +- origin provider and public or Private Link connectivity; +- PLS region and approval behavior; +- member placement, priority, and weight overrides; and +- detailed member-origin programming and health status. + +### Controller layers + +```text +Gateway API reconcilers + - validate Gateway, HTTPRoute, Backend, and references + - produce portable connection intent + +Fleet backend resolver + - resolve ServiceImport to eligible member exports + - resolve public endpoint or PLS metadata + - apply placement, priority, and weight + +Normalized gateway model + - preserve route weight separately from member-origin weight + - carry protocol and backend TLS + - carry provider connectivity and health settings + +AFD provider + - reconcile profile, endpoint, route, origin group, origins, Private Link, + WAF, domains, certificates, and diagnostics +``` + +The normalized model on `rchinchani/gep-1748-gateway-api` is the correct +convergence boundary, but it must eventually add explicit fields for backend +protocol, backend TLS, host header, health probe details, priority, and +provider-specific origin metadata. + +## Migration strategy + +### Phase 1: Preserve the implementable path + +- Continue supporting direct Fleet `ServiceImport` backend references. +- Complete public and Private Link origin resolution. +- Keep provider configuration on the current typed annotation or candidate + Fleet policy surface. +- Do not claim GEP-4894 conformance. + +### Phase 2: Add Backend where it is independent + +- Add Experimental `ExternalHostname` support behind a feature gate. +- Add protocol and backend TLS fields to the normalized model. +- Publish the exact supported GEP-4894 revision and conformance tier. +- Reject unsupported fields rather than silently ignoring them. + +### Phase 3: Add multi-cluster Backend binding + +Proceed only after an upstream endpoint-selection contract can represent +controller-produced multi-cluster endpoints. + +- Materialize or manage the stable upstream endpoint resource from + `ServiceImport`. +- Require the Route and `Backend` to share a namespace. +- Authorize producer-side cross-namespace endpoint references with the + upstream mechanism. +- Resolve both direct `ServiceImport` and `Backend` paths into the same + normalized identity. + +### Phase 4: Migrate with dual-read, single-write + +- Accept both direct `ServiceImport` and new `Backend` references. +- Detect and reject configurations where both API paths claim the same AFD + route/backend attachment. +- Preserve Azure resource identity and ownership tags. +- Program Azure from only one selected source. +- Report migration status before removing the old reference. +- Retain rollback until status, traffic behavior, and Azure ownership are + verified. + +## Decision analysis: bind to the current `selectorRef` or defer + +This decision is not simply "upstream is Experimental, therefore wait." +Binding now has meaningful product and upstream advantages. The choice depends +on whether Fleet is willing to treat the YAML as a disposable preview contract. + +### Option A: bind Fleet to the current `selectorRef` + +Under this option, Fleet adopts the current GEP-4894 shape in a preview: + +```yaml +spec: + type: EndpointSelector + endpointSelector: + selectorRef: + group: networking.fleet.azure.com + kind: ServiceImport + name: api + port: 443 +``` + +Because the current GEP text defines `selectorRef` as a reference to an +`EndpointSelector`, using `ServiceImport` would be an explicit Fleet extension. +Fleet could alternatively introduce an adapter resource that implements the +expected endpoint-selection role and is populated from `ServiceImport`. + +#### Case for binding now + +1. **The conceptual boundary is right for Fleet** + + A Route-local `Backend` describes the consumer's connection contract while + a producer-side reference supplies endpoints. That separation matches + Fleet's need to keep protocol and TLS distinct from multi-cluster endpoint + aggregation. + +2. **Fleet supplies a real implementation test that upstream lacks** + + Fleet has dynamic endpoints, cross-namespace consumers, multiple regions, + and provider-specific origin metadata. Implementing the proposal would + expose whether `selectorRef`, status ownership, and authorization work for + more than a cluster-local Service. + +3. **Early adoption can influence the standard** + + A working implementation and conformance proposal carry more weight than a + design-only request. Fleet could use concrete findings to influence the + future EndpointSelector KEP and GEP-4894 rather than adapting after those + decisions are closed. + +4. **Users get the intended Backend-first experience sooner** + + Protocol and TLS live next to the destination, and Routes consistently + reference `Backend` instead of mixing `ServiceImport` and `Backend` kinds. + If the shape survives, Fleet avoids a later user migration from direct + `ServiceImport` references. + +5. **The controller architecture already has the right seam** + + The GEP-1748 prototype normalizes Kubernetes resources before programming + AFD. A `Backend` resolver can be added as another input without rewriting + the AFD provider. + +6. **An alpha API is allowed to learn** + + If the feature is explicitly experimental, feature-gated, disabled by + default, and excluded from compatibility guarantees, changing or removing + `selectorRef` is an acceptable preview cost. + +#### Requirements for responsibly binding now + +Binding now is defensible only with all of these constraints: + +- use the upstream Experimental API version or a clearly named Fleet + experimental API, never a stable Fleet API version; +- place the feature behind a disabled-by-default gate; +- publish the exact Gateway API commit implemented; +- state that `selectorRef` to `ServiceImport` is a Fleet extension and does not + provide GEP-4894 conformance; +- keep direct `ServiceImport` references supported as the stable path; +- normalize both paths into the same internal backend identity; +- reject simultaneous claims from both paths instead of programming twice; +- add conversion or migration tooling before changing the preview schema; +- prohibit automatic field pruning during a Gateway API CRD upgrade; +- define authorization and status ownership locally rather than leaving them + implicit; and +- accept that preview objects may require user action to migrate. + +With these controls, early binding is a calculated upstream incubation +investment, not a production API commitment. + +### Option B: do not bind Fleet to `selectorRef` now + +Under this option, Fleet continues to expose direct `ServiceImport` +`backendRef`s and prepares the internal model for a future `Backend` adapter. +Fleet may implement the independent `ExternalHostname`, protocol, or TLS +features only after choosing a pinned Experimental revision. + +#### Case for deferring + +1. **Upstream is actively removing the field** + + Gateway API PR 5158 proposes deferring `selectorRef`, retaining only an + embedded namespace-local pod selector, and adding `selectorRef` later when + the upstream EndpointSelector resource exists. This is a direct signal that + the current field has not reached design consensus. + +2. **The referenced resource does not exist** + + The current GEP describes `selectorRef` as pointing to an upstream + `EndpointSelector`, but that API is still being pursued through KEP-6116. + The pending GEP update states Kubernetes 1.38 is the earliest target and + notes Gateway API's GA+5 dependency policy. Fleet would be binding to a + relationship whose target contract is unknown. + +3. **`ServiceImport` is not an EndpointSelector** + + A `ServiceImport` carries logical multi-cluster Service semantics, ports, + cluster membership, and aggregation status. Treating it as an + EndpointSelector because `selectorRef` uses a generic object reference + would be syntax-compatible but semantically incompatible with the current + GEP. + +4. **The pending replacement cannot serve Fleet** + + PR 5158's embedded label selector selects pods in the Backend namespace. + Fleet's hub does not contain the member-cluster pods, so the proposed + replacement cannot express Fleet's destination. Adopting it would create an + API dead end rather than an incremental path. + +5. **Authorization is unresolved** + + GEP-4894 currently requires the Route and Backend to share a namespace, but + cross-namespace `selectorRef` authorization remains an open question. Fleet + cannot safely infer that existing `ReferenceGrant` behavior applies. + +6. **The abstraction does not reduce the hard implementation work** + + AFD still needs Fleet to resolve member exports, public endpoints, PLS + metadata, eligibility, priority, weight, and health. Adding an unstable + Backend-to-selector layer does not remove any of those control loops. + +7. **CRD churn has operational cost** + + If upstream removes or changes `selectorRef`, upgrading the Experimental CRD + can reject or prune stored fields. Fleet would need conversion, status + migration, rollback, and user communication before it has delivered any + additional AFD capability. + +8. **Conformance cannot justify the cost** + + GEP-4894 and GEP-1748 are both Experimental, Fleet uses an + implementation-specific `ServiceImport` API group, and there are no + conformance tests that make a Fleet `selectorRef` extension portable. + +Deferral protects the durable public API while leaving the controller design +ready to adopt the eventual standard. + +### Comparative scorecard + +| Criterion | Bind current `selectorRef` | Do not bind now | +|---|---|---| +| Backend-first user experience | Strong immediately | Delayed | +| Upstream implementation feedback | Strong | Limited to design feedback | +| Chance to influence the standard | Higher | Lower | +| Current semantic correctness | Low for direct `ServiceImport`; medium with an adapter | High | +| API stability | Low | High | +| Cross-namespace authorization clarity | Low | Existing GEP-1748 path is clear | +| Conformance value | Low | Neutral | +| New AFD capability unlocked | Little by itself | No loss | +| Migration burden | High if upstream changes | Lower | +| Time to production-ready public/private origins | Slower if coupled to adoption | Faster | +| Reversibility | Acceptable only as gated preview | High | + +### Recommendation + +Do not bind the production Fleet API to the current `selectorRef`. + +The decisive point is not merely field instability: the current reference is +defined for an endpoint resource that does not exist, while a direct +`ServiceImport` target would be a Fleet-specific semantic fork. It does not +unlock public origins, Private Link, placement, or AFD programming, so the +migration cost is not justified on the critical delivery path. + +Fleet should nevertheless pursue a **bounded incubation implementation** if +upstream influence is a priority: + +1. keep the stable user path as direct `HTTPRoute` to `ServiceImport`; +2. add a disabled-by-default experimental resolver for the current + `selectorRef`; +3. pin it to a Gateway API commit and make no compatibility promise; +4. use it to test status, authorization, and multi-cluster endpoint semantics; +5. upstream the findings and conformance cases; and +6. delete or migrate the experiment when GEP-4894 and EndpointSelector settle. + +This separates two decisions that should not be conflated: + +- **Should Fleet help validate `selectorRef` now?** Yes, potentially. +- **Should Fleet make the current shape its customer API now?** No. + +## Status model + +Standard Gateway API conditions remain authoritative where applicable: + +- `GatewayClass Accepted`; +- `Gateway Accepted` and `Programmed`; +- listener conditions; +- `HTTPRoute Accepted` and `ResolvedRefs`; and +- GEP-4894 Backend parent conditions for connection configuration. + +Fleet-specific status remains necessary for: + +- each member cluster and region; +- origin eligibility and drain state; +- public or Private Link connectivity; +- PLS approval; +- Azure origin programming; +- origin health; +- active-active or active-passive readiness; and +- minimum healthy-origin and regional availability policy. + +`Backend Available=True` or `Programmed=True` must not imply that every member +origin is healthy. The condition reason and Fleet origin summary must preserve +that distinction. + +## Security and namespace implications + +- A Route-local `Backend` improves consumer ownership and avoids ambiguous + producer policy. +- Inline client certificate references must remain namespace-bound and must + not expose secret material in status. +- `ExternalHostname` requires DNS trust, admission guardrails, egress network + controls, and protection from confused-deputy targets. +- Fleet Private Link remains the preferred private-origin topology for + SFI-NS253 requirements. +- A Backend must not let a namespace bypass `ServiceImport` authorization, + Fleet membership, Azure policy, WAF requirements, or allowed origin + connectivity. +- Cross-namespace direct `ServiceImport` references remain governed by + `ReferenceGrant` while that API path exists. +- A future Route-local Backend to producer endpoint binding must use the + standard upstream authorization model; Fleet must not invent implicit + cross-namespace access. + +## Adoption gates + +Fleet should not make GEP-4894 part of its durable public contract until: + +1. the `EndpointSelector` shape and Service binding are merged and versioned; +2. the binding can represent controller-produced endpoints from + `ServiceImport`; +3. namespace authorization semantics are testable; +4. Gateway API publishes usable CRDs and conformance tests; +5. backend TLS and protocol semantics are sufficiently stable for AFD; +6. status ownership between the Backend controller and AFD Gateway controller + is defined; +7. direct `ServiceImport` migration and rollback are tested; and +8. Fleet can publish which Core and Extended features it supports without + implying upstream MCS conformance for its implementation-specific API + group. + +## Final decision + +Adopt the GEP-4894 concepts, but do not replace the current GEP-1748 +`ServiceImport` path now. + +The long-term API should use `Backend` as the explicit consumer connection +object only after its endpoint binding is stable. `ServiceImport` remains the +multi-cluster destination source, and Fleet/Azure policy remains the source of +global origin topology and provider behavior. This layered design can satisfy +the same product goals while improving backend discoverability and per-consumer +TLS, without forcing unstable upstream YAML into the first implementation. + +## References + +- [GEP-4894: Backend Resource](https://gateway-api.sigs.k8s.io/geps/gep-4894/) +- [GEP-1748: Gateway API Interaction with Multi-Cluster Services](https://gateway-api.sigs.k8s.io/geps/gep-1748/) +- [Gateway API PR 4488: Experimental Backend Resource](https://github.com/kubernetes-sigs/gateway-api/pull/4488) +- [Gateway API PR 5158: Defer selectorRef](https://github.com/kubernetes-sigs/gateway-api/pull/5158) +- `rchinchani/afd-global-ingress-rfc:docs/design/afd-global-ingress-rfc.md` +- `rchinchani/gep-1748-gateway-api:docs/design/gep-1748-gateway-api.md` +- `rchinchani/gep-1748-gateway-api:docs/design/gep-1748-implementation-plan.md` +- `rchinchani/gep-1748-gateway-api:pkg/controllers/hub/gatewaymodel/model.go` diff --git a/go.mod b/go.mod index c698ae16..2bc0afb9 100644 --- a/go.mod +++ b/go.mod @@ -4,6 +4,8 @@ go 1.25.12 require go.goms.io/fleet v0.14.0 +require sigs.k8s.io/gateway-api v1.2.1 + require ( github.com/Azure/azure-sdk-for-go v68.0.0+incompatible // indirect github.com/Azure/azure-sdk-for-go/sdk/azcore v1.18.0 diff --git a/go.sum b/go.sum index a4654e6b..72d0435f 100644 --- a/go.sum +++ b/go.sum @@ -315,6 +315,8 @@ sigs.k8s.io/cloud-provider-azure/pkg/azclient v0.0.50 h1:l9igMANNptVwYmZrqGS51oW sigs.k8s.io/cloud-provider-azure/pkg/azclient v0.0.50/go.mod h1:1M90A+akyTabHVnveSKlvIO/Kk9kEr1LjRx+08twKVU= sigs.k8s.io/controller-runtime v0.19.0 h1:nWVM7aq+Il2ABxwiCizrVDSlmDcshi9llbaFbC0ji/Q= sigs.k8s.io/controller-runtime v0.19.0/go.mod h1:iRmWllt8IlaLjvTTDLhRBXIEtkCK6hwVBJJsYS9Ajf4= +sigs.k8s.io/gateway-api v1.2.1 h1:fZZ/+RyRb+Y5tGkwxFKuYuSRQHu9dZtbjenblleOLHM= +sigs.k8s.io/gateway-api v1.2.1/go.mod h1:EpNfEXNjiYfUJypf0eZ0P5iXA9ekSGWaS1WgPaM42X0= sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd h1:EDPBXCAspyGV4jQlpZSudPeMmr1bNJefnuqLsRAsHZo= sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd/go.mod h1:B8JuhiUyNFVKdsE8h686QcCxMaH6HrOAZj4vswFpcB0= sigs.k8s.io/structured-merge-diff/v4 v4.4.1 h1:150L+0vs/8DA78h1u02ooW1/fFq/Lwr+sGiqlzvrtq4= diff --git a/pkg/controllers/hub/azurefrontdoorbackendattachment/controller.go b/pkg/controllers/hub/azurefrontdoorbackendattachment/controller.go new file mode 100644 index 00000000..82f51215 --- /dev/null +++ b/pkg/controllers/hub/azurefrontdoorbackendattachment/controller.go @@ -0,0 +1,402 @@ +/* +Copyright (c) Microsoft Corporation. +Licensed under the MIT license. +*/ + +// Package azurefrontdoorbackendattachment validates AFD backend attachments without writing Azure resources. +package azurefrontdoorbackendattachment + +import ( + "context" + "fmt" + "sort" + "time" + + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/equality" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/tools/record" + "k8s.io/klog/v2" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/builder" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/handler" + "sigs.k8s.io/controller-runtime/pkg/predicate" + "sigs.k8s.io/controller-runtime/pkg/reconcile" + gatewayv1 "sigs.k8s.io/gateway-api/apis/v1" + + "go.goms.io/fleet/pkg/utils/controller" + + fleetnetv1alpha1 "go.goms.io/fleet-networking/api/v1alpha1" +) + +const ( + // ControllerName is the name of the AFD backend attachment controller. + ControllerName = "azurefrontdoorbackendattachment-controller" + + gatewayRefField = ".spec.gatewayRef.name" + backendRefField = ".spec.backendRef.name" + policyTargetField = ".spec.targetRef.name" + + eventReasonAccepted = "Accepted" + eventReasonRejected = "Rejected" +) + +// Reconciler validates AzureFrontDoorBackendAttachment dependencies and publishes status. +type Reconciler struct { + client.Client + Recorder record.EventRecorder +} + +//+kubebuilder:rbac:groups=networking.fleet.azure.com,resources=azurefrontdoorbackendattachments,verbs=get;list;watch +//+kubebuilder:rbac:groups=networking.fleet.azure.com,resources=azurefrontdoorbackendattachments/status,verbs=get;update;patch +//+kubebuilder:rbac:groups=networking.fleet.azure.com,resources=azurefrontdoorgatewaypolicies,verbs=get;list;watch +//+kubebuilder:rbac:groups=networking.fleet.azure.com,resources=serviceimports,verbs=get;list;watch +//+kubebuilder:rbac:groups=gateway.networking.k8s.io,resources=gateways,verbs=get;list;watch +//+kubebuilder:rbac:groups="",resources=events,verbs=create;patch + +// Reconcile resolves an attachment and records whether a later Azure-writing controller may consume it. +func (r *Reconciler) Reconcile(ctx context.Context, req reconcile.Request) (reconcile.Result, error) { + startTime := time.Now() + attachmentRef := klog.KRef(req.Namespace, req.Name) + klog.V(2).InfoS("Reconciliation starts", "azureFrontDoorBackendAttachment", attachmentRef) + defer func() { + klog.V(2).InfoS("Reconciliation ends", "azureFrontDoorBackendAttachment", attachmentRef, "latency", time.Since(startTime).Milliseconds()) + }() + + attachment := &fleetnetv1alpha1.AzureFrontDoorBackendAttachment{} + if err := r.Get(ctx, req.NamespacedName, attachment); err != nil { + if apierrors.IsNotFound(err) { + klog.V(2).InfoS("Ignoring NotFound AzureFrontDoorBackendAttachment", "azureFrontDoorBackendAttachment", attachmentRef) + return ctrl.Result{}, nil + } + klog.ErrorS(err, "Failed to get AzureFrontDoorBackendAttachment", "azureFrontDoorBackendAttachment", attachmentRef) + return ctrl.Result{}, controller.NewAPIServerError(true, err) + } + + original := attachment.DeepCopy() + if err := r.resolve(ctx, attachment); err != nil { + return ctrl.Result{}, err + } + return ctrl.Result{}, r.updateStatus(ctx, original, attachment) +} + +func (r *Reconciler) resolve(ctx context.Context, attachment *fleetnetv1alpha1.AzureFrontDoorBackendAttachment) error { + attachment.Status.Gateway = nil + attachment.Status.Backend = nil + + if !hasSupportedReferences(attachment) { + setAttachmentConditions(attachment, metav1.ConditionFalse, fleetnetv1alpha1.AzureFrontDoorReasonUnsupportedRef, + "GatewayRef must reference a Gateway and backendRef must reference a ServiceImport", false) + return nil + } + + gateway := &gatewayv1.Gateway{} + gatewayKey := types.NamespacedName{Namespace: attachment.Namespace, Name: string(attachment.Spec.GatewayRef.Name)} + if err := r.Get(ctx, gatewayKey, gateway); err != nil { + if apierrors.IsNotFound(err) { + setAttachmentConditions(attachment, metav1.ConditionFalse, fleetnetv1alpha1.AzureFrontDoorReasonRefNotFound, + fmt.Sprintf("Gateway %q was not found", gatewayKey.Name), false) + return nil + } + klog.ErrorS(err, "Failed to get Gateway", "azureFrontDoorBackendAttachment", klog.KObj(attachment), "gateway", gatewayKey) + return controller.NewAPIServerError(true, err) + } + attachment.Status.Gateway = resolvedReference(gateway) + + serviceImport := &fleetnetv1alpha1.ServiceImport{} + backendKey := types.NamespacedName{Namespace: attachment.Namespace, Name: string(attachment.Spec.BackendRef.Name)} + if err := r.Get(ctx, backendKey, serviceImport); err != nil { + if apierrors.IsNotFound(err) { + setAttachmentConditions(attachment, metav1.ConditionFalse, fleetnetv1alpha1.AzureFrontDoorReasonRefNotFound, + fmt.Sprintf("ServiceImport %q was not found", backendKey.Name), false) + return nil + } + klog.ErrorS(err, "Failed to get ServiceImport", "azureFrontDoorBackendAttachment", klog.KObj(attachment), "serviceImport", backendKey) + return controller.NewAPIServerError(true, err) + } + attachment.Status.Backend = resolvedReference(serviceImport) + + if !serviceImportHasTCPPort(serviceImport, attachment.Spec.BackendRef.Port) { + setAttachmentConditions(attachment, metav1.ConditionFalse, fleetnetv1alpha1.AzureFrontDoorReasonInvalidPort, + fmt.Sprintf("ServiceImport %q does not expose TCP port %d", backendKey.Name, attachment.Spec.BackendRef.Port), false) + return nil + } + + winner, err := r.attachmentWinner(ctx, attachment) + if err != nil { + return err + } + if winner.UID != attachment.UID { + setAttachmentConditions(attachment, metav1.ConditionFalse, fleetnetv1alpha1.AzureFrontDoorReasonConflicted, + fmt.Sprintf("Attachment %q has precedence for this Gateway, ServiceImport, and port", winner.Name), true) + return nil + } + + policy, reason, message, err := r.resolveGatewayPolicy(ctx, attachment) + if err != nil { + return err + } + if policy == nil { + setAttachmentConditions(attachment, metav1.ConditionFalse, reason, message, true) + return nil + } + if attachment.Spec.Connectivity.Mode == fleetnetv1alpha1.AzureFrontDoorConnectivityModePrivateLink && + policy.Spec.Profile.SKU != fleetnetv1alpha1.AzureFrontDoorProfileSKUPremium { + setAttachmentConditions(attachment, metav1.ConditionFalse, fleetnetv1alpha1.AzureFrontDoorReasonUnsupportedConfig, + "Private Link requires a Premium_AzureFrontDoor profile", true) + return nil + } + + setAttachmentConditions(attachment, metav1.ConditionTrue, fleetnetv1alpha1.AzureFrontDoorReasonAccepted, + "Attachment references and configuration are valid", true) + return nil +} + +func hasSupportedReferences(attachment *fleetnetv1alpha1.AzureFrontDoorBackendAttachment) bool { + return attachment.Spec.GatewayRef.Group == gatewayv1.Group(gatewayv1.GroupName) && + attachment.Spec.GatewayRef.Kind == gatewayv1.Kind("Gateway") && + string(attachment.Spec.BackendRef.Group) == fleetnetv1alpha1.GroupVersion.Group && + attachment.Spec.BackendRef.Kind == gatewayv1.Kind("ServiceImport") +} + +func resolvedReference(object client.Object) *fleetnetv1alpha1.AzureFrontDoorResolvedReference { + return &fleetnetv1alpha1.AzureFrontDoorResolvedReference{ + Name: object.GetName(), + UID: object.GetUID(), + } +} + +func serviceImportHasTCPPort(serviceImport *fleetnetv1alpha1.ServiceImport, port int32) bool { + for i := range serviceImport.Status.Ports { + servicePort := serviceImport.Status.Ports[i] + if servicePort.Port == port && (servicePort.Protocol == "" || servicePort.Protocol == corev1.ProtocolTCP) { + return true + } + } + return false +} + +func (r *Reconciler) attachmentWinner(ctx context.Context, attachment *fleetnetv1alpha1.AzureFrontDoorBackendAttachment) (*fleetnetv1alpha1.AzureFrontDoorBackendAttachment, error) { + attachments := &fleetnetv1alpha1.AzureFrontDoorBackendAttachmentList{} + if err := r.List(ctx, attachments, client.InNamespace(attachment.Namespace)); err != nil { + klog.ErrorS(err, "Failed to list AzureFrontDoorBackendAttachments", "azureFrontDoorBackendAttachment", klog.KObj(attachment)) + return nil, controller.NewAPIServerError(true, err) + } + + candidates := make([]*fleetnetv1alpha1.AzureFrontDoorBackendAttachment, 0, len(attachments.Items)) + for i := range attachments.Items { + candidate := &attachments.Items[i] + if candidate.Spec.GatewayRef == attachment.Spec.GatewayRef && + candidate.Spec.BackendRef == attachment.Spec.BackendRef { + candidates = append(candidates, candidate) + } + } + sort.Slice(candidates, func(i, j int) bool { + left, right := candidates[i], candidates[j] + if !left.CreationTimestamp.Time.Equal(right.CreationTimestamp.Time) { + return left.CreationTimestamp.Time.Before(right.CreationTimestamp.Time) + } + if left.UID != right.UID { + return string(left.UID) < string(right.UID) + } + return left.Name < right.Name + }) + if len(candidates) == 0 { + return nil, controller.NewUnexpectedBehaviorError(fmt.Errorf("attachment %s was absent from its namespace list", klog.KObj(attachment))) + } + return candidates[0], nil +} + +func (r *Reconciler) resolveGatewayPolicy( + ctx context.Context, + attachment *fleetnetv1alpha1.AzureFrontDoorBackendAttachment, +) (*fleetnetv1alpha1.AzureFrontDoorGatewayPolicy, string, string, error) { + policies := &fleetnetv1alpha1.AzureFrontDoorGatewayPolicyList{} + if err := r.List(ctx, policies, client.InNamespace(attachment.Namespace)); err != nil { + klog.ErrorS(err, "Failed to list AzureFrontDoorGatewayPolicies", "azureFrontDoorBackendAttachment", klog.KObj(attachment)) + return nil, "", "", controller.NewAPIServerError(true, err) + } + + matches := make([]*fleetnetv1alpha1.AzureFrontDoorGatewayPolicy, 0, 1) + for i := range policies.Items { + policy := &policies.Items[i] + if policy.Spec.TargetRef.Group == gatewayv1.Group(gatewayv1.GroupName) && + policy.Spec.TargetRef.Kind == gatewayv1.Kind("Gateway") && + policy.Spec.TargetRef.Name == attachment.Spec.GatewayRef.Name { + matches = append(matches, policy) + } + } + if len(matches) == 0 { + return nil, fleetnetv1alpha1.AzureFrontDoorReasonRefNotFound, + fmt.Sprintf("No AzureFrontDoorGatewayPolicy targets Gateway %q", attachment.Spec.GatewayRef.Name), nil + } + if len(matches) > 1 { + return nil, fleetnetv1alpha1.AzureFrontDoorReasonConflicted, + fmt.Sprintf("Multiple AzureFrontDoorGatewayPolicies target Gateway %q", attachment.Spec.GatewayRef.Name), nil + } + return matches[0], "", "", nil +} + +func setAttachmentConditions( + attachment *fleetnetv1alpha1.AzureFrontDoorBackendAttachment, + acceptedStatus metav1.ConditionStatus, + acceptedReason, message string, + resolved bool, +) { + resolvedStatus := metav1.ConditionFalse + resolvedReason := acceptedReason + resolvedMessage := message + if resolved { + resolvedStatus = metav1.ConditionTrue + resolvedReason = fleetnetv1alpha1.AzureFrontDoorReasonAccepted + resolvedMessage = "Gateway and ServiceImport references are valid" + } + meta.SetStatusCondition(&attachment.Status.Conditions, metav1.Condition{ + Type: fleetnetv1alpha1.AzureFrontDoorConditionResolvedRefs, + Status: resolvedStatus, + ObservedGeneration: attachment.Generation, + Reason: resolvedReason, + Message: resolvedMessage, + }) + meta.SetStatusCondition(&attachment.Status.Conditions, metav1.Condition{ + Type: fleetnetv1alpha1.AzureFrontDoorConditionAccepted, + Status: acceptedStatus, + ObservedGeneration: attachment.Generation, + Reason: acceptedReason, + Message: message, + }) + meta.SetStatusCondition(&attachment.Status.Conditions, metav1.Condition{ + Type: fleetnetv1alpha1.AzureFrontDoorConditionProgrammed, + Status: metav1.ConditionUnknown, + ObservedGeneration: attachment.Generation, + Reason: fleetnetv1alpha1.AzureFrontDoorReasonPending, + Message: "Azure resource programming is not enabled in the foundation release", + }) +} + +func (r *Reconciler) updateStatus( + ctx context.Context, + original, attachment *fleetnetv1alpha1.AzureFrontDoorBackendAttachment, +) error { + if equality.Semantic.DeepEqual(original.Status, attachment.Status) { + return nil + } + if err := r.Status().Patch(ctx, attachment, client.MergeFrom(original)); err != nil { + klog.ErrorS(err, "Failed to update AzureFrontDoorBackendAttachment status", "azureFrontDoorBackendAttachment", klog.KObj(attachment)) + return controller.NewUpdateIgnoreConflictError(err) + } + + accepted := meta.FindStatusCondition(attachment.Status.Conditions, fleetnetv1alpha1.AzureFrontDoorConditionAccepted) + if accepted != nil && accepted.Status == metav1.ConditionTrue { + r.Recorder.Event(attachment, corev1.EventTypeNormal, eventReasonAccepted, accepted.Message) + klog.V(2).InfoS("Accepted AzureFrontDoorBackendAttachment", "azureFrontDoorBackendAttachment", klog.KObj(attachment)) + } else if accepted != nil { + r.Recorder.Event(attachment, corev1.EventTypeWarning, eventReasonRejected, accepted.Message) + klog.V(2).InfoS("Rejected AzureFrontDoorBackendAttachment", "azureFrontDoorBackendAttachment", klog.KObj(attachment), "reason", accepted.Reason) + } + return nil +} + +// SetupWithManager registers indexes and watches for every dependency that can change attachment validity. +func (r *Reconciler) SetupWithManager(ctx context.Context, mgr ctrl.Manager) error { + indexes := []struct { + object client.Object + field string + extractor client.IndexerFunc + }{ + { + object: &fleetnetv1alpha1.AzureFrontDoorBackendAttachment{}, + field: gatewayRefField, + extractor: func(object client.Object) []string { + attachment := object.(*fleetnetv1alpha1.AzureFrontDoorBackendAttachment) + return []string{string(attachment.Spec.GatewayRef.Name)} + }, + }, + { + object: &fleetnetv1alpha1.AzureFrontDoorBackendAttachment{}, + field: backendRefField, + extractor: func(object client.Object) []string { + attachment := object.(*fleetnetv1alpha1.AzureFrontDoorBackendAttachment) + return []string{string(attachment.Spec.BackendRef.Name)} + }, + }, + { + object: &fleetnetv1alpha1.AzureFrontDoorGatewayPolicy{}, + field: policyTargetField, + extractor: func(object client.Object) []string { + policy := object.(*fleetnetv1alpha1.AzureFrontDoorGatewayPolicy) + return []string{string(policy.Spec.TargetRef.Name)} + }, + }, + } + for _, index := range indexes { + if err := mgr.GetFieldIndexer().IndexField(ctx, index.object, index.field, index.extractor); err != nil { + klog.ErrorS(err, "Failed to create AFD field index", "field", index.field) + return err + } + } + + generationChanged := builder.WithPredicates(predicate.GenerationChangedPredicate{}) + return ctrl.NewControllerManagedBy(mgr). + For(&fleetnetv1alpha1.AzureFrontDoorBackendAttachment{}, generationChanged). + Watches(&gatewayv1.Gateway{}, handler.EnqueueRequestsFromMapFunc(r.attachmentsForGateway), generationChanged). + Watches(&fleetnetv1alpha1.ServiceImport{}, handler.EnqueueRequestsFromMapFunc(r.attachmentsForBackend)). + Watches(&fleetnetv1alpha1.AzureFrontDoorGatewayPolicy{}, handler.EnqueueRequestsFromMapFunc(r.attachmentsForPolicy), generationChanged). + Watches(&fleetnetv1alpha1.AzureFrontDoorBackendAttachment{}, handler.EnqueueRequestsFromMapFunc(r.attachmentsForPeer), generationChanged). + Complete(r) +} + +func (r *Reconciler) attachmentsForGateway(ctx context.Context, object client.Object) []reconcile.Request { + return r.listAttachmentRequests(ctx, object.GetNamespace(), gatewayRefField, object.GetName()) +} + +func (r *Reconciler) attachmentsForBackend(ctx context.Context, object client.Object) []reconcile.Request { + return r.listAttachmentRequests(ctx, object.GetNamespace(), backendRefField, object.GetName()) +} + +func (r *Reconciler) attachmentsForPolicy(ctx context.Context, object client.Object) []reconcile.Request { + policy, ok := object.(*fleetnetv1alpha1.AzureFrontDoorGatewayPolicy) + if !ok { + return nil + } + return r.listAttachmentRequests(ctx, policy.Namespace, gatewayRefField, string(policy.Spec.TargetRef.Name)) +} + +func (r *Reconciler) attachmentsForPeer(ctx context.Context, object client.Object) []reconcile.Request { + attachment, ok := object.(*fleetnetv1alpha1.AzureFrontDoorBackendAttachment) + if !ok { + return nil + } + requests := r.listAttachmentRequests(ctx, attachment.Namespace, gatewayRefField, string(attachment.Spec.GatewayRef.Name)) + filtered := requests[:0] + for _, request := range requests { + candidate := &fleetnetv1alpha1.AzureFrontDoorBackendAttachment{} + if err := r.Get(ctx, request.NamespacedName, candidate); err != nil { + if !apierrors.IsNotFound(err) { + klog.ErrorS(err, "Failed to get peer AzureFrontDoorBackendAttachment", "azureFrontDoorBackendAttachment", request.NamespacedName) + } + continue + } + if candidate.Spec.BackendRef == attachment.Spec.BackendRef { + filtered = append(filtered, request) + } + } + return filtered +} + +func (r *Reconciler) listAttachmentRequests(ctx context.Context, namespace, field, value string) []reconcile.Request { + attachments := &fleetnetv1alpha1.AzureFrontDoorBackendAttachmentList{} + if err := r.List(ctx, attachments, client.InNamespace(namespace), client.MatchingFields{field: value}); err != nil { + klog.ErrorS(err, "Failed to list AzureFrontDoorBackendAttachments for dependency event", "namespace", namespace, "field", field, "value", value) + return nil + } + requests := make([]reconcile.Request, 0, len(attachments.Items)) + for i := range attachments.Items { + requests = append(requests, reconcile.Request{NamespacedName: client.ObjectKeyFromObject(&attachments.Items[i])}) + } + return requests +} diff --git a/pkg/controllers/hub/azurefrontdoorbackendattachment/controller_test.go b/pkg/controllers/hub/azurefrontdoorbackendattachment/controller_test.go new file mode 100644 index 00000000..404d0f4e --- /dev/null +++ b/pkg/controllers/hub/azurefrontdoorbackendattachment/controller_test.go @@ -0,0 +1,333 @@ +/* +Copyright (c) Microsoft Corporation. +Licensed under the MIT license. +*/ + +package azurefrontdoorbackendattachment + +import ( + "context" + "testing" + "time" + + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/tools/record" + "k8s.io/utils/ptr" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/fake" + gatewayv1 "sigs.k8s.io/gateway-api/apis/v1" + + fleetnetv1alpha1 "go.goms.io/fleet-networking/api/v1alpha1" +) + +func TestReconcile(t *testing.T) { + now := metav1.NewTime(time.Now()) + later := metav1.NewTime(now.Add(time.Minute)) + + tests := []struct { + name string + objects []client.Object + attachmentName string + wantAccepted metav1.ConditionStatus + wantReason string + wantResolvedRefs metav1.ConditionStatus + }{ + { + name: "accept valid attachment", + objects: validObjects( + newAttachment("attachment", "attachment-uid", now), + ), + attachmentName: "attachment", + wantAccepted: metav1.ConditionTrue, + wantReason: fleetnetv1alpha1.AzureFrontDoorReasonAccepted, + wantResolvedRefs: metav1.ConditionTrue, + }, + { + name: "accept explicit TCP service port", + objects: []client.Object{ + newAttachment("attachment", "attachment-uid", now), + newGateway(), + newServiceImportWithProtocol(443, corev1.ProtocolTCP), + newGatewayPolicy(fleetnetv1alpha1.AzureFrontDoorProfileSKUPremium), + }, + attachmentName: "attachment", + wantAccepted: metav1.ConditionTrue, + wantReason: fleetnetv1alpha1.AzureFrontDoorReasonAccepted, + wantResolvedRefs: metav1.ConditionTrue, + }, + { + name: "reject attachment when Gateway is missing", + objects: []client.Object{ + newAttachment("attachment", "attachment-uid", now), + newServiceImport(), + newGatewayPolicy(fleetnetv1alpha1.AzureFrontDoorProfileSKUPremium), + }, + attachmentName: "attachment", + wantAccepted: metav1.ConditionFalse, + wantReason: fleetnetv1alpha1.AzureFrontDoorReasonRefNotFound, + wantResolvedRefs: metav1.ConditionFalse, + }, + { + name: "reject attachment when service port is missing", + objects: []client.Object{ + newAttachment("attachment", "attachment-uid", now), + newGateway(), + newServiceImport(), + newGatewayPolicy(fleetnetv1alpha1.AzureFrontDoorProfileSKUPremium), + }, + attachmentName: "attachment", + wantAccepted: metav1.ConditionFalse, + wantReason: fleetnetv1alpha1.AzureFrontDoorReasonInvalidPort, + wantResolvedRefs: metav1.ConditionFalse, + }, + { + name: "reject UDP service port", + objects: []client.Object{ + newAttachment("attachment", "attachment-uid", now), + newGateway(), + newServiceImportWithProtocol(443, corev1.ProtocolUDP), + newGatewayPolicy(fleetnetv1alpha1.AzureFrontDoorProfileSKUPremium), + }, + attachmentName: "attachment", + wantAccepted: metav1.ConditionFalse, + wantReason: fleetnetv1alpha1.AzureFrontDoorReasonInvalidPort, + wantResolvedRefs: metav1.ConditionFalse, + }, + { + name: "reject SCTP service port", + objects: []client.Object{ + newAttachment("attachment", "attachment-uid", now), + newGateway(), + newServiceImportWithProtocol(443, corev1.ProtocolSCTP), + newGatewayPolicy(fleetnetv1alpha1.AzureFrontDoorProfileSKUPremium), + }, + attachmentName: "attachment", + wantAccepted: metav1.ConditionFalse, + wantReason: fleetnetv1alpha1.AzureFrontDoorReasonInvalidPort, + wantResolvedRefs: metav1.ConditionFalse, + }, + { + name: "reject newer duplicate", + objects: validObjects( + newAttachment("winner", "winner-uid", now), + newAttachment("attachment", "attachment-uid", later), + ), + attachmentName: "attachment", + wantAccepted: metav1.ConditionFalse, + wantReason: fleetnetv1alpha1.AzureFrontDoorReasonConflicted, + wantResolvedRefs: metav1.ConditionTrue, + }, + { + name: "keep the existing attachment accepted when a duplicate appears", + objects: validObjects( + newAttachment("attachment", "winner-uid", now), + newAttachment("duplicate", "duplicate-uid", later), + ), + attachmentName: "attachment", + wantAccepted: metav1.ConditionTrue, + wantReason: fleetnetv1alpha1.AzureFrontDoorReasonAccepted, + wantResolvedRefs: metav1.ConditionTrue, + }, + { + name: "reject Private Link with Standard profile", + objects: []client.Object{ + newPrivateLinkAttachment("attachment", "attachment-uid", now), + newGateway(), + newServiceImport(443), + newGatewayPolicy(fleetnetv1alpha1.AzureFrontDoorProfileSKUStandard), + }, + attachmentName: "attachment", + wantAccepted: metav1.ConditionFalse, + wantReason: fleetnetv1alpha1.AzureFrontDoorReasonUnsupportedConfig, + wantResolvedRefs: metav1.ConditionTrue, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + scheme := testScheme(t) + fakeClient := fake.NewClientBuilder(). + WithScheme(scheme). + WithStatusSubresource(&fleetnetv1alpha1.AzureFrontDoorBackendAttachment{}). + WithObjects(tt.objects...). + Build() + reconciler := &Reconciler{ + Client: fakeClient, + Recorder: record.NewFakeRecorder(10), + } + + _, err := reconciler.Reconcile(context.Background(), ctrl.Request{ + NamespacedName: types.NamespacedName{Namespace: testNamespace, Name: tt.attachmentName}, + }) + if err != nil { + t.Fatalf("Reconcile() error = %v", err) + } + + attachment := &fleetnetv1alpha1.AzureFrontDoorBackendAttachment{} + if err := fakeClient.Get(context.Background(), client.ObjectKey{Namespace: testNamespace, Name: tt.attachmentName}, attachment); err != nil { + t.Fatalf("Get() error = %v", err) + } + accepted := meta.FindStatusCondition(attachment.Status.Conditions, fleetnetv1alpha1.AzureFrontDoorConditionAccepted) + if accepted == nil { + t.Fatal("Accepted condition is missing") + } + if accepted.Status != tt.wantAccepted || accepted.Reason != tt.wantReason { + t.Errorf("Accepted condition = (%s, %s), want (%s, %s)", accepted.Status, accepted.Reason, tt.wantAccepted, tt.wantReason) + } + resolvedRefs := meta.FindStatusCondition(attachment.Status.Conditions, fleetnetv1alpha1.AzureFrontDoorConditionResolvedRefs) + if resolvedRefs == nil || resolvedRefs.Status != tt.wantResolvedRefs { + t.Errorf("ResolvedRefs condition = %#v, want status %s", resolvedRefs, tt.wantResolvedRefs) + } + if len(attachment.Finalizers) != 0 { + t.Errorf("finalizers = %v, want none for read-only controller", attachment.Finalizers) + } + }) + } +} + +func validObjects(attachments ...*fleetnetv1alpha1.AzureFrontDoorBackendAttachment) []client.Object { + objects := []client.Object{ + newGateway(), + newServiceImport(443), + newGatewayPolicy(fleetnetv1alpha1.AzureFrontDoorProfileSKUPremium), + } + for _, attachment := range attachments { + objects = append(objects, attachment) + } + return objects +} + +const testNamespace = "app" + +func newGateway() *gatewayv1.Gateway { + return &gatewayv1.Gateway{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: testNamespace, + Name: "global", + UID: "gateway-uid", + }, + } +} + +func newServiceImport(ports ...int32) *fleetnetv1alpha1.ServiceImport { + servicePorts := make([]fleetnetv1alpha1.ServicePort, 0, len(ports)) + for _, port := range ports { + servicePorts = append(servicePorts, fleetnetv1alpha1.ServicePort{Port: port}) + } + return &fleetnetv1alpha1.ServiceImport{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: testNamespace, + Name: "store", + UID: "service-import-uid", + }, + Status: fleetnetv1alpha1.ServiceImportStatus{Ports: servicePorts}, + } +} + +func newServiceImportWithProtocol(port int32, protocol corev1.Protocol) *fleetnetv1alpha1.ServiceImport { + serviceImport := newServiceImport(port) + serviceImport.Status.Ports[0].Protocol = protocol + return serviceImport +} + +func newGatewayPolicy(sku fleetnetv1alpha1.AzureFrontDoorProfileSKU) *fleetnetv1alpha1.AzureFrontDoorGatewayPolicy { + return &fleetnetv1alpha1.AzureFrontDoorGatewayPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: testNamespace, + Name: "global", + }, + Spec: fleetnetv1alpha1.AzureFrontDoorGatewayPolicySpec{ + TargetRef: gatewayv1.LocalObjectReference{ + Group: gatewayv1.Group(gatewayv1.GroupName), + Kind: gatewayv1.Kind("Gateway"), + Name: "global", + }, + Profile: fleetnetv1alpha1.AzureFrontDoorProfileSpec{ + Mode: fleetnetv1alpha1.AzureFrontDoorProfileModeManaged, + SKU: sku, + ResourceGroup: "fleet-global", + Name: "fleet-global", + }, + WAF: fleetnetv1alpha1.AzureFrontDoorWAFSpec{ + Required: ptr.To(true), + PolicyResourceID: "/subscriptions/sub/resourceGroups/security/providers/Microsoft.Network/frontDoorWebApplicationFirewallPolicies/fleet-waf", + }, + }, + } +} + +func newAttachment(name string, uid types.UID, creationTime metav1.Time) *fleetnetv1alpha1.AzureFrontDoorBackendAttachment { + return &fleetnetv1alpha1.AzureFrontDoorBackendAttachment{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: testNamespace, + Name: name, + UID: uid, + CreationTimestamp: creationTime, + }, + Spec: fleetnetv1alpha1.AzureFrontDoorBackendAttachmentSpec{ + GatewayRef: gatewayv1.LocalObjectReference{ + Group: gatewayv1.Group(gatewayv1.GroupName), + Kind: gatewayv1.Kind("Gateway"), + Name: "global", + }, + BackendRef: fleetnetv1alpha1.AzureFrontDoorBackendReference{ + LocalObjectReference: gatewayv1.LocalObjectReference{ + Group: gatewayv1.Group(fleetnetv1alpha1.GroupVersion.Group), + Kind: "ServiceImport", + Name: "store", + }, + Port: 443, + }, + Connectivity: fleetnetv1alpha1.AzureFrontDoorConnectivitySpec{ + Mode: fleetnetv1alpha1.AzureFrontDoorConnectivityModePublic, + }, + Origin: fleetnetv1alpha1.AzureFrontDoorOriginSpec{ + Protocol: fleetnetv1alpha1.AzureFrontDoorOriginProtocolHTTPS, + HostHeader: "store.internal.contoso.example", + CertificateSubjectNameCheck: ptr.To(true), + }, + HealthProbe: fleetnetv1alpha1.AzureFrontDoorHealthProbeSpec{ + Protocol: fleetnetv1alpha1.AzureFrontDoorOriginProtocolHTTPS, + Method: fleetnetv1alpha1.AzureFrontDoorHealthProbeMethodHEAD, + Path: "/healthz", + IntervalSeconds: 30, + SampleSize: 4, + SuccessfulSamplesRequired: 3, + }, + Traffic: fleetnetv1alpha1.AzureFrontDoorTrafficSpec{ + DefaultPriority: 1, + DefaultWeight: 1000, + }, + MemberFailurePolicy: fleetnetv1alpha1.AzureFrontDoorMemberFailurePolicyPartial, + }, + } +} + +func newPrivateLinkAttachment(name string, uid types.UID, creationTime metav1.Time) *fleetnetv1alpha1.AzureFrontDoorBackendAttachment { + attachment := newAttachment(name, uid, creationTime) + attachment.Spec.Connectivity = fleetnetv1alpha1.AzureFrontDoorConnectivitySpec{ + Mode: fleetnetv1alpha1.AzureFrontDoorConnectivityModePrivateLink, + PrivateLink: &fleetnetv1alpha1.AzureFrontDoorPrivateLinkSpec{ + Approval: "Manual", + RegionSelection: "ClosestSupported", + }, + } + return attachment +} + +func testScheme(t *testing.T) *runtime.Scheme { + t.Helper() + scheme := runtime.NewScheme() + if err := gatewayv1.Install(scheme); err != nil { + t.Fatalf("gatewayv1.Install() error = %v", err) + } + if err := fleetnetv1alpha1.AddToScheme(scheme); err != nil { + t.Fatalf("fleetnetv1alpha1.AddToScheme() error = %v", err) + } + return scheme +} diff --git a/test/apis/v1alpha1/azurefrontdoor_validation_integration_test.go b/test/apis/v1alpha1/azurefrontdoor_validation_integration_test.go new file mode 100644 index 00000000..cf9e7d8e --- /dev/null +++ b/test/apis/v1alpha1/azurefrontdoor_validation_integration_test.go @@ -0,0 +1,170 @@ +/* +Copyright (c) Microsoft Corporation. +Licensed under the MIT license. +*/ + +package v1alpha1 + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/utils/ptr" + gatewayv1 "sigs.k8s.io/gateway-api/apis/v1" + + fleetnetv1alpha1 "go.goms.io/fleet-networking/api/v1alpha1" +) + +var _ = Describe("Azure Front Door API validation", func() { + It("defaults a valid managed Gateway policy", func() { + policy := validAFDGatewayPolicy() + Expect(hubClient.Create(ctx, policy)).To(Succeed()) + + Expect(policy.Spec.Profile.Mode).To(Equal(fleetnetv1alpha1.AzureFrontDoorProfileModeManaged)) + Expect(policy.Spec.WAF.Required).NotTo(BeNil()) + Expect(*policy.Spec.WAF.Required).To(BeTrue()) + }) + + It("rejects a Gateway policy targeting another kind", func() { + policy := validAFDGatewayPolicy() + policy.Spec.TargetRef.Kind = "Service" + + err := hubClient.Create(ctx, policy) + Expect(apierrors.IsInvalid(err)).To(BeTrue(), "error: %v", err) + Expect(err.Error()).To(ContainSubstring("must reference a Gateway")) + }) + + It("requires WAF enforcement", func() { + policy := validAFDGatewayPolicy() + policy.Spec.WAF.Required = ptr.To(false) + + err := hubClient.Create(ctx, policy) + Expect(apierrors.IsInvalid(err)).To(BeTrue(), "error: %v", err) + Expect(err.Error()).To(ContainSubstring("required must be true")) + }) + + It("defaults a valid public backend attachment", func() { + attachment := validAFDBackendAttachment() + Expect(hubClient.Create(ctx, attachment)).To(Succeed()) + + Expect(attachment.Spec.Origin.Protocol).To(Equal(fleetnetv1alpha1.AzureFrontDoorOriginProtocolHTTPS)) + Expect(attachment.Spec.Origin.CertificateSubjectNameCheck).NotTo(BeNil()) + Expect(*attachment.Spec.Origin.CertificateSubjectNameCheck).To(BeTrue()) + Expect(attachment.Spec.HealthProbe.Method).To(Equal(fleetnetv1alpha1.AzureFrontDoorHealthProbeMethodHEAD)) + Expect(attachment.Spec.HealthProbe.Path).To(Equal("/healthz")) + Expect(attachment.Spec.HealthProbe.IntervalSeconds).To(Equal(int32(30))) + Expect(attachment.Spec.HealthProbe.SampleSize).To(Equal(int32(4))) + Expect(attachment.Spec.HealthProbe.SuccessfulSamplesRequired).To(Equal(int32(3))) + Expect(attachment.Spec.Traffic.DefaultPriority).To(Equal(int32(1))) + Expect(attachment.Spec.Traffic.DefaultWeight).To(Equal(int32(1000))) + Expect(attachment.Spec.MemberFailurePolicy).To(Equal(fleetnetv1alpha1.AzureFrontDoorMemberFailurePolicyPartial)) + }) + + It("defaults omitted health probe and traffic objects", func() { + object, err := runtime.DefaultUnstructuredConverter.ToUnstructured(validAFDBackendAttachment()) + Expect(err).NotTo(HaveOccurred()) + unstructured.RemoveNestedField(object, "spec", "healthProbe") + unstructured.RemoveNestedField(object, "spec", "traffic") + + attachment := &unstructured.Unstructured{Object: object} + attachment.SetGroupVersionKind(fleetnetv1alpha1.GroupVersion.WithKind("AzureFrontDoorBackendAttachment")) + Expect(hubClient.Create(ctx, attachment)).To(Succeed()) + + defaulted := &fleetnetv1alpha1.AzureFrontDoorBackendAttachment{} + Expect(runtime.DefaultUnstructuredConverter.FromUnstructured(attachment.Object, defaulted)).To(Succeed()) + Expect(defaulted.Spec.HealthProbe.Protocol).To(Equal(fleetnetv1alpha1.AzureFrontDoorOriginProtocolHTTPS)) + Expect(defaulted.Spec.HealthProbe.Method).To(Equal(fleetnetv1alpha1.AzureFrontDoorHealthProbeMethodHEAD)) + Expect(defaulted.Spec.HealthProbe.Path).To(Equal("/healthz")) + Expect(defaulted.Spec.HealthProbe.IntervalSeconds).To(Equal(int32(30))) + Expect(defaulted.Spec.HealthProbe.SampleSize).To(Equal(int32(4))) + Expect(defaulted.Spec.HealthProbe.SuccessfulSamplesRequired).To(Equal(int32(3))) + Expect(defaulted.Spec.Traffic.DefaultPriority).To(Equal(int32(1))) + Expect(defaulted.Spec.Traffic.DefaultWeight).To(Equal(int32(1000))) + }) + + It("rejects Private Link mode without Private Link configuration", func() { + attachment := validAFDBackendAttachment() + attachment.Spec.Connectivity.Mode = fleetnetv1alpha1.AzureFrontDoorConnectivityModePrivateLink + + err := hubClient.Create(ctx, attachment) + Expect(apierrors.IsInvalid(err)).To(BeTrue(), "error: %v", err) + Expect(err.Error()).To(ContainSubstring("privateLink must be set")) + }) + + It("rejects an invalid health sample count", func() { + attachment := validAFDBackendAttachment() + attachment.Spec.HealthProbe.SampleSize = 2 + attachment.Spec.HealthProbe.SuccessfulSamplesRequired = 3 + + err := hubClient.Create(ctx, attachment) + Expect(apierrors.IsInvalid(err)).To(BeTrue(), "error: %v", err) + Expect(err.Error()).To(ContainSubstring("cannot exceed sampleSize")) + }) + + It("keeps attachment identity immutable", func() { + attachment := validAFDBackendAttachment() + Expect(hubClient.Create(ctx, attachment)).To(Succeed()) + + attachment.Spec.BackendRef.Port = 8443 + err := hubClient.Update(ctx, attachment) + Expect(apierrors.IsInvalid(err)).To(BeTrue(), "error: %v", err) + Expect(err.Error()).To(ContainSubstring("backendRef is immutable")) + }) +}) + +func validAFDGatewayPolicy() *fleetnetv1alpha1.AzureFrontDoorGatewayPolicy { + return &fleetnetv1alpha1.AzureFrontDoorGatewayPolicy{ + ObjectMeta: metav1.ObjectMeta{ + GenerateName: "afd-policy-", + Namespace: testNamespace, + }, + Spec: fleetnetv1alpha1.AzureFrontDoorGatewayPolicySpec{ + TargetRef: gatewayv1.LocalObjectReference{ + Group: gatewayv1.Group(gatewayv1.GroupName), + Kind: "Gateway", + Name: "global", + }, + Profile: fleetnetv1alpha1.AzureFrontDoorProfileSpec{ + SKU: fleetnetv1alpha1.AzureFrontDoorProfileSKUPremium, + ResourceGroup: "fleet-global", + Name: "fleet-global", + }, + WAF: fleetnetv1alpha1.AzureFrontDoorWAFSpec{ + PolicyResourceID: "/subscriptions/sub/resourceGroups/security/providers/Microsoft.Network/frontDoorWebApplicationFirewallPolicies/fleet-waf", + }, + }, + } +} + +func validAFDBackendAttachment() *fleetnetv1alpha1.AzureFrontDoorBackendAttachment { + return &fleetnetv1alpha1.AzureFrontDoorBackendAttachment{ + ObjectMeta: metav1.ObjectMeta{ + GenerateName: "afd-attachment-", + Namespace: testNamespace, + }, + Spec: fleetnetv1alpha1.AzureFrontDoorBackendAttachmentSpec{ + GatewayRef: gatewayv1.LocalObjectReference{ + Group: gatewayv1.Group(gatewayv1.GroupName), + Kind: "Gateway", + Name: "global", + }, + BackendRef: fleetnetv1alpha1.AzureFrontDoorBackendReference{ + LocalObjectReference: gatewayv1.LocalObjectReference{ + Group: gatewayv1.Group(fleetnetv1alpha1.GroupVersion.Group), + Kind: "ServiceImport", + Name: "store", + }, + Port: 443, + }, + Connectivity: fleetnetv1alpha1.AzureFrontDoorConnectivitySpec{ + Mode: fleetnetv1alpha1.AzureFrontDoorConnectivityModePublic, + }, + Origin: fleetnetv1alpha1.AzureFrontDoorOriginSpec{ + HostHeader: "store.internal.contoso.example", + }, + }, + } +}