feat(site): prove safe bootstrap and adoption before retiring the legacy site writer #3

Open
opened 2026-10-02 11:51:17 +00:00 by Andreas · 8 comments
Owner

Current boundary

Production Omada 6.3.0.45 is healthy; Terraform omada in namespace omada is Ready / TerraformPlannedNoChanges. Controller digest: sha256:310ddbcad764337014dbf630a79771ce3cb423920002fe23a37391c41fd5058f. Legacy emanuelbesliu/tplink-omada 2.1.1 owns the site, SSIDs and VLANs; wncservices/omada 0.13.0 owns switch profiles/ports. Root ./iac/omada auto-applies reviewed main commits. No production ownership transfer or site mutation occurred during this investigation.

Research findings (not controller proof)

Tohaker provider commit cf65c3c3ae4bb68ae7fab8af740c790772ad5ca8 requires device-account credentials; its password is Sensitive, not WriteOnly, and therefore stored in state. Metadata Update also unconditionally writes the device account. Delete does not inspect the controller response errorCode. Do not introduce this third provider unchanged.
github.com/Tohaker/terraform-provider-omada@cf65c3c3ae/internal/service/site/resource.go
SDK commit 8dd07f8005e4490cd6e5d4150448ec1cd28b3716 defines POST /openapi/v1/{omadacId}/sites, Global Dashboard Manager Modify permission, and required deviceAccountSetting/name/region/timeZone/scenario. Metadata PUT and device-account PUT are separate endpoints.
github.com/Tohaker/omada-go-sdk@8dd07f8005/omada/api_site.go

Proposed implementation

Add a narrowly scoped site resource in this fork, with HTTP logic in internal/omada and framework mapping in internal/provider. Passwords must be write-only ephemeral inputs, with a separate explicit revision trigger for intentional credential rotation. Imported sites must not require reading or resupplying passwords to update metadata. Never implicitly write device credentials during metadata changes; preserve unmodeled fields.

Required proof and cutover gates

  • Disposable isolated controller at the exact version: create/read/import/no-op/update/delete a unique empty site. No devices, adoption, discovery, LAN exposure, host networking or production credentials. Bound CPU/memory/time and arrange cleanup.
  • Verify actual permissions/response shapes, duplicate-name and errorCode failures, timeout recovery, and cleanup. Never duplicate an ambiguously created site. Generated SDK code alone is not a live proof.
  • Mock acceptance tests on both engines: stable identity, no-op import, metadata-only writes, no device-account calls, preserved fields, unknown/omitted config, no secrets anywhere in saved plan/state, failure retention, safe delete.
  • prevent_destroy in production; refuse primary/non-empty site deletion. Adopt existing site by ID without replacement.
  • Reviewed immutable fork delivery and provider lock checksums: no production dev_overrides or expiring unsigned Actions artifacts.
  • Verified controller/state restore point and exact Flux/app revisions; pause auto-apply during ownership transfer; one writer per object and a no-change migration plan.
  • Reconfirm direct LAN/cluster fallback; verify AP/switch adoption, access, controller health and final no-drift reconciliation before retiring legacy references.

Keep the old site writer isolated and pinned only until these gates pass; this is a temporary compatibility boundary, not completion of the full migration. PSK rotation is addressed in PR #1 / upstream PR #79; fork CI/packaging in PR #2.

## Current boundary Production Omada 6.3.0.45 is healthy; Terraform omada in namespace omada is Ready / TerraformPlannedNoChanges. Controller digest: sha256:310ddbcad764337014dbf630a79771ce3cb423920002fe23a37391c41fd5058f. Legacy emanuelbesliu/tplink-omada 2.1.1 owns the site, SSIDs and VLANs; wncservices/omada 0.13.0 owns switch profiles/ports. Root ./iac/omada auto-applies reviewed main commits. No production ownership transfer or site mutation occurred during this investigation. ## Research findings (not controller proof) Tohaker provider commit cf65c3c3ae4bb68ae7fab8af740c790772ad5ca8 requires device-account credentials; its password is Sensitive, not WriteOnly, and therefore stored in state. Metadata Update also unconditionally writes the device account. Delete does not inspect the controller response errorCode. Do not introduce this third provider unchanged. https://github.com/Tohaker/terraform-provider-omada/blob/cf65c3c3ae4bb68ae7fab8af740c790772ad5ca8/internal/service/site/resource.go SDK commit 8dd07f8005e4490cd6e5d4150448ec1cd28b3716 defines POST /openapi/v1/{omadacId}/sites, Global Dashboard Manager Modify permission, and required deviceAccountSetting/name/region/timeZone/scenario. Metadata PUT and device-account PUT are separate endpoints. https://github.com/Tohaker/omada-go-sdk/blob/8dd07f8005e4490cd6e5d4150448ec1cd28b3716/omada/api_site.go ## Proposed implementation Add a narrowly scoped site resource in this fork, with HTTP logic in internal/omada and framework mapping in internal/provider. Passwords must be write-only ephemeral inputs, with a separate explicit revision trigger for intentional credential rotation. Imported sites must not require reading or resupplying passwords to update metadata. Never implicitly write device credentials during metadata changes; preserve unmodeled fields. ## Required proof and cutover gates - Disposable isolated controller at the exact version: create/read/import/no-op/update/delete a unique empty site. No devices, adoption, discovery, LAN exposure, host networking or production credentials. Bound CPU/memory/time and arrange cleanup. - Verify actual permissions/response shapes, duplicate-name and errorCode failures, timeout recovery, and cleanup. Never duplicate an ambiguously created site. Generated SDK code alone is not a live proof. - Mock acceptance tests on both engines: stable identity, no-op import, metadata-only writes, no device-account calls, preserved fields, unknown/omitted config, no secrets anywhere in saved plan/state, failure retention, safe delete. - prevent_destroy in production; refuse primary/non-empty site deletion. Adopt existing site by ID without replacement. - Reviewed immutable fork delivery and provider lock checksums: no production dev_overrides or expiring unsigned Actions artifacts. - Verified controller/state restore point and exact Flux/app revisions; pause auto-apply during ownership transfer; one writer per object and a no-change migration plan. - Reconfirm direct LAN/cluster fallback; verify AP/switch adoption, access, controller health and final no-drift reconciliation before retiring legacy references. Keep the old site writer isolated and pinned only until these gates pass; this is a temporary compatibility boundary, not completion of the full migration. PSK rotation is addressed in PR #1 / upstream PR #79; fork CI/packaging in PR #2.
Author
Owner

The exact Omada 6.3.0.45 image has now been exercised in a rootless Podman network-none environment, using fresh volumes and synthetic credentials only. Site create/read/rename/delete passed. POST /sites returns {siteId}; duplicate names fail -33101. Metadata writes use PATCH /sites/{id}/setting with {site:{...}}, not PATCH /sites/{id} (which fails -1600). Metadata rename preserved identity, device credentials and switch-support flags, with no device-account write. Device inventory is a bare array; the proof requires it empty before deletion. Repeatable harness and redacted findings are staged in the site-bootstrap-proof PR. No production mutations or ownership transfer. Terraform import/no-op, least privilege, rotation/error recovery, safe-delete regressions and reviewed delivery/cutover gates remain.

The exact Omada 6.3.0.45 image has now been exercised in a rootless Podman network-none environment, using fresh volumes and synthetic credentials only. Site create/read/rename/delete passed. POST /sites returns {siteId}; duplicate names fail -33101. Metadata writes use PATCH /sites/{id}/setting with {site:{...}}, not PATCH /sites/{id} (which fails -1600). Metadata rename preserved identity, device credentials and switch-support flags, with no device-account write. Device inventory is a bare array; the proof requires it empty before deletion. Repeatable harness and redacted findings are staged in the site-bootstrap-proof PR. No production mutations or ownership transfer. Terraform import/no-op, least privilege, rotation/error recovery, safe-delete regressions and reviewed delivery/cutover gates remain.
Author
Owner

PR #4 is merged after green hosted CI (run #9, 3m50s) and a successful final committed fresh-controller harness (exit 0, controller and anonymous-volume cleanup passed). SITE_BOOTSTRAP_PROOF.md records exact endpoints, evidence, and remaining limitations. The single-fork site implementation is now grounded in real exact-version controller behavior; production still has no ownership/configuration change.

PR #4 is merged after green hosted CI (run #9, 3m50s) and a successful final committed fresh-controller harness (exit 0, controller and anonymous-volume cleanup passed). SITE_BOOTSTRAP_PROOF.md records exact endpoints, evidence, and remaining limitations. The single-fork site implementation is now grounded in real exact-version controller behavior; production still has no ownership/configuration change.
Author
Owner

Milestone: PR #5 is merged after hosted CI run #11 passed (3m56s). omada_site now implements basic-site creation with write-only bootstrap credentials, import by ID, metadata-only updates preserving unmodeled fields, strict bounded inventory reads/error retention, and guarded deletion with disappearance verification. Candidate mirror is version 0.13.1-homelab.2; the previous published prerelease is unchanged.

Both Terraform 1.16.4 and exact deployed OpenTofu 1.12.1 passed full mock validation and real isolated controller 6.3.0.45 import/no-op/in-place rename/no-op/destroy. Saved plan/state credential checks, device-account preservation and unmodeled metadata preservation passed. Disposable container and volumes were removed. Production read-only checks: controller 1/1; Terraform Ready=True / TerraformPlannedNoChanges. No production mutation or ownership transfer.

Explicit scope adjustment: this first resource uses bootstrap credentials only at creation. Device-account rotation is intentionally unsupported until its contract/revision semantics are independently proved. Still open: real provider-driven creation (real fixture used typed client), least-privilege permissions, ambiguous-create recovery proof, immutable artifact trust/delivery and lock checksums, verified rollback, and read-only no-change/single-writer ownership transfer. Do not retire the legacy writer yet.

Milestone: PR #5 is merged after hosted CI run #11 passed (3m56s). omada_site now implements basic-site creation with write-only bootstrap credentials, import by ID, metadata-only updates preserving unmodeled fields, strict bounded inventory reads/error retention, and guarded deletion with disappearance verification. Candidate mirror is version 0.13.1-homelab.2; the previous published prerelease is unchanged. Both Terraform 1.16.4 and exact deployed OpenTofu 1.12.1 passed full mock validation and real isolated controller 6.3.0.45 import/no-op/in-place rename/no-op/destroy. Saved plan/state credential checks, device-account preservation and unmodeled metadata preservation passed. Disposable container and volumes were removed. Production read-only checks: controller 1/1; Terraform Ready=True / TerraformPlannedNoChanges. No production mutation or ownership transfer. Explicit scope adjustment: this first resource uses bootstrap credentials only at creation. Device-account rotation is intentionally unsupported until its contract/revision semantics are independently proved. Still open: real provider-driven creation (real fixture used typed client), least-privilege permissions, ambiguous-create recovery proof, immutable artifact trust/delivery and lock checksums, verified rollback, and read-only no-change/single-writer ownership transfer. Do not retire the legacy writer yet.
Author
Owner

PR #6 merged after hosted CI run #13 passed (3m55s). The real provider Create gap is now closed for both Terraform 1.16.4 and exact deployed OpenTofu 1.12.1 on isolated controller 6.3.0.45. Each engine passed saved create-plan/state credential absence, intended bootstrap account verification, bootstrap omission/no-op, duplicate -33101 rejection with one original site and no implicit state adoption, explicit ID reimport/no-op, destroy and disappearance verification. Final source was rerun successfully; controller and anonymous volumes removed.

New injected-transport regressions cover timeout/EOF after simulated commit: one POST only, actionable inventory/import guidance, discoverable committed identity, and no secret diagnostics. Null/malformed/duplicate inventories fail closed and echoed controller text is suppressed. This is not a real-controller packet-loss experiment.

Remaining gates: least-privilege service-account permission proof; actual-controller ambiguous-response recovery if required; immutable artifact trust/delivery and lock checksums; verified controller/state rollback; read-only production no-change comparison and paused/single-writer ownership transfer. Device credential rotation remains intentionally unsupported. Production is unchanged: controller 1/1 and Terraform Ready / TerraformPlannedNoChanges. Legacy provider must remain until the applicable transfer gates pass.

PR #6 merged after hosted CI run #13 passed (3m55s). The real provider Create gap is now closed for both Terraform 1.16.4 and exact deployed OpenTofu 1.12.1 on isolated controller 6.3.0.45. Each engine passed saved create-plan/state credential absence, intended bootstrap account verification, bootstrap omission/no-op, duplicate -33101 rejection with one original site and no implicit state adoption, explicit ID reimport/no-op, destroy and disappearance verification. Final source was rerun successfully; controller and anonymous volumes removed. New injected-transport regressions cover timeout/EOF after simulated commit: one POST only, actionable inventory/import guidance, discoverable committed identity, and no secret diagnostics. Null/malformed/duplicate inventories fail closed and echoed controller text is suppressed. This is not a real-controller packet-loss experiment. Remaining gates: least-privilege service-account permission proof; actual-controller ambiguous-response recovery if required; immutable artifact trust/delivery and lock checksums; verified controller/state rollback; read-only production no-change comparison and paused/single-writer ownership transfer. Device credential rotation remains intentionally unsupported. Production is unchanged: controller 1/1 and Terraform Ready / TerraformPlannedNoChanges. Legacy provider must remain until the applicable transfer gates pass.
Author
Owner

Permission milestone merged in PR #7 after hosted CI run #15 passed (3m53s), source c82a916784. Candidate version is now 0.13.1-homelab.3, offline installation passed; no production publication or adoption.

Exact isolated 6.3.0.45 proof established a narrow custom role: globalDashboard=2, siteNetwork=2, devices=1; every other returned privilege checked zero. User/role inventory is denied (-1007). Both Terraform 1.16.4 and exact deployed OpenTofu 1.12.1 passed complete provider lifecycle under this account, with credential/unmodeled-field preservation and secret-free plan/state checks. View-only role cannot create/update/delete; remote identity/name stay unchanged.

Important discovered/fixed bug: site-scoped inventory hides a real existing site, which the previous code classified as ErrSiteNotFound. Absence now requires /users/current privilege.all=true; scoped, unknown and denied profile scope retains state with an error. Both-engine mock acceptance verifies restoration of the same binding with no additional create. Real scope-hidden fixture proof now fails closed.

All synthetic roles/accounts, fixture data and state stayed in the network-none disposable controller; container/volumes removed. Production still controller 1/1, Terraform Ready / TerraformPlannedNoChanges, unchanged two-provider ownership.

Remaining: durable reviewed artifact delivery/trust and lock checksums; production-account/all-root permission coverage including switch/Open API; verified rollback and read-only no-change ownership-transfer proof before retiring legacy writer. This establishes a tested narrow site role, not a mathematically minimal permission set. Device-account rotation remains intentionally unsupported.

Permission milestone merged in PR #7 after hosted CI run #15 passed (3m53s), source c82a916784acea66fbda8c52af9bbb32f125ed37. Candidate version is now 0.13.1-homelab.3, offline installation passed; no production publication or adoption. Exact isolated 6.3.0.45 proof established a narrow custom role: globalDashboard=2, siteNetwork=2, devices=1; every other returned privilege checked zero. User/role inventory is denied (-1007). Both Terraform 1.16.4 and exact deployed OpenTofu 1.12.1 passed complete provider lifecycle under this account, with credential/unmodeled-field preservation and secret-free plan/state checks. View-only role cannot create/update/delete; remote identity/name stay unchanged. Important discovered/fixed bug: site-scoped inventory hides a real existing site, which the previous code classified as ErrSiteNotFound. Absence now requires /users/current privilege.all=true; scoped, unknown and denied profile scope retains state with an error. Both-engine mock acceptance verifies restoration of the same binding with no additional create. Real scope-hidden fixture proof now fails closed. All synthetic roles/accounts, fixture data and state stayed in the network-none disposable controller; container/volumes removed. Production still controller 1/1, Terraform Ready / TerraformPlannedNoChanges, unchanged two-provider ownership. Remaining: durable reviewed artifact delivery/trust and lock checksums; production-account/all-root permission coverage including switch/Open API; verified rollback and read-only no-change ownership-transfer proof before retiring legacy writer. This establishes a tested narrow site role, not a mathematically minimal permission set. Device-account rotation remains intentionally unsupported.
Author
Owner

Trusted delivery milestone: PR #8 merged after green provider CI run #18 (3m46s). Main run #19 succeeded (4m55s), publishing registry.ancofazi.de/homelab/omada-tf-runner@sha256:425b9c94d87e4554ed087f286a530325da72592a5e9501a91f04de760056e58c from source revision 2284d0f85a. Shared publisher signs and verifies the OCI image and source provenance and attaches an SBOM. Both local and hosted image proofs passed: exact OpenTofu 1.12.1, non-root user and original entrypoint, network-none backend-free filesystem-mirror installation, read-only lock reuse, required write-only schemas, and corrupted-package checksum rejection. Fixed proof cleanup for both rootful CI and rootless local Podman. Production controller/runner/provider ownership is unchanged and remains healthy/no-change. Remaining gate: independent agent-side signature/source-revision verification could not authenticate to private Harbor (UNAUTHORIZED); no suitable Harbor entry or local registry login is available, and CI credentials were not extracted. Obtain an approved read-only pull credential, verify the immutable digest and exact revision, then perform rollback and read-only no-change ownership-transfer preflight in deploy-omada. Do not infer that publication authorizes a state or runner cutover.

Trusted delivery milestone: PR #8 merged after green provider CI run #18 (3m46s). Main run #19 succeeded (4m55s), publishing registry.ancofazi.de/homelab/omada-tf-runner@sha256:425b9c94d87e4554ed087f286a530325da72592a5e9501a91f04de760056e58c from source revision 2284d0f85a6eb5341b7c294b4a88c295322267a3. Shared publisher signs and verifies the OCI image and source provenance and attaches an SBOM. Both local and hosted image proofs passed: exact OpenTofu 1.12.1, non-root user and original entrypoint, network-none backend-free filesystem-mirror installation, read-only lock reuse, required write-only schemas, and corrupted-package checksum rejection. Fixed proof cleanup for both rootful CI and rootless local Podman. Production controller/runner/provider ownership is unchanged and remains healthy/no-change. Remaining gate: independent agent-side signature/source-revision verification could not authenticate to private Harbor (UNAUTHORIZED); no suitable Harbor entry or local registry login is available, and CI credentials were not extracted. Obtain an approved read-only pull credential, verify the immutable digest and exact revision, then perform rollback and read-only no-change ownership-transfer preflight in deploy-omada. Do not infer that publication authorizes a state or runner cutover.
Author
Owner

Independent delivery verification gate cleared. With approval, used the existing kyverno/registry-verification-credentials Docker-config credential only in a private temporary runtime directory. Confirmed publisher verification key equals platform-infrastructure canonical admission-trust key. Cosign independently verified image signature and transparency-log claims for registry.ancofazi.de/homelab/omada-tf-runner@sha256:425b9c94d87e4554ed087f286a530325da72592a5e9501a91f04de760056e58c. Verified signed SLSA provenance binds that exact digest to git+https://git.ancofazi.de/Andreas/terraform-provider-omada and commit 2284d0f85a. Temporary credential file/directory removed after verification; no credentials displayed, persisted, changed or rotated. Production remains unchanged. Next gate remains rollback/recovery preflight and read-only no-change ownership-transfer proof before adopting the fork.

Independent delivery verification gate cleared. With approval, used the existing kyverno/registry-verification-credentials Docker-config credential only in a private temporary runtime directory. Confirmed publisher verification key equals platform-infrastructure canonical admission-trust key. Cosign independently verified image signature and transparency-log claims for registry.ancofazi.de/homelab/omada-tf-runner@sha256:425b9c94d87e4554ed087f286a530325da72592a5e9501a91f04de760056e58c. Verified signed SLSA provenance binds that exact digest to git+https://git.ancofazi.de/Andreas/terraform-provider-omada and commit 2284d0f85a6eb5341b7c294b4a88c295322267a3. Temporary credential file/directory removed after verification; no credentials displayed, persisted, changed or rotated. Production remains unchanged. Next gate remains rollback/recovery preflight and read-only no-change ownership-transfer proof before adopting the fork.
Author
Owner

Live signed-fork import milestone passed and evidence merged in deploy-omada PR #23 (app main 8a721d1d0004b10d5d32948ed05cef4c5c5f4730; chart and deploy checks green). Read-only isolated plan on current controller image 6.3.0.45@sha256:47f1c6f40754a4bbf417930e81a71bb1a95c6cd2595a164989dddea829d825e9: engine 1.12.1; seven expected site/VLAN/WLAN/SSID imports, seven no-ops, zero updates/creates/deletes. Authoritative state only streamed through strict ID extraction; no state dump saved or published, no backend/apply/PSK/bootstrap/OpenAPI write, no production ownership changes. Removed proof Pod/ID Secret/ConfigMap and private ID file; production remains healthy/no-drift. Added negative plan/identity and helper-boundary regressions. Repaired Renovate-mutated assessment engine constraint and held operator-proof constraints only. This is not full ownership-transfer approval: still need authoritative plan with forget/import only and all fifteen switch objects unchanged, plus verified matching off-host encrypted state/controller recovery checkpoint. Read-only rollback preflight observed Pascal kubernetes-backup-generation.service failed (exit 1, last exit 2026-10-02 17:27:50 CEST), while USB/cloud last executions reported success. Root cause and checkpoint freshness are NOT established by these service statuses; do not treat upload success alone as the required recovery proof. No backup restart or repair performed.

Live signed-fork import milestone passed and evidence merged in deploy-omada PR #23 (app main 8a721d1d0004b10d5d32948ed05cef4c5c5f4730; chart and deploy checks green). Read-only isolated plan on current controller image 6.3.0.45@sha256:47f1c6f40754a4bbf417930e81a71bb1a95c6cd2595a164989dddea829d825e9: engine 1.12.1; seven expected site/VLAN/WLAN/SSID imports, seven no-ops, zero updates/creates/deletes. Authoritative state only streamed through strict ID extraction; no state dump saved or published, no backend/apply/PSK/bootstrap/OpenAPI write, no production ownership changes. Removed proof Pod/ID Secret/ConfigMap and private ID file; production remains healthy/no-drift. Added negative plan/identity and helper-boundary regressions. Repaired Renovate-mutated assessment engine constraint and held operator-proof constraints only. This is not full ownership-transfer approval: still need authoritative plan with forget/import only and all fifteen switch objects unchanged, plus verified matching off-host encrypted state/controller recovery checkpoint. Read-only rollback preflight observed Pascal kubernetes-backup-generation.service failed (exit 1, last exit 2026-10-02 17:27:50 CEST), while USB/cloud last executions reported success. Root cause and checkpoint freshness are NOT established by these service statuses; do not treat upload success alone as the required recovery proof. No backup restart or repair performed.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
Andreas/terraform-provider-omada#3
No description provided.