fix: get CI building on the aarch64 runner again #195
No reviewers
Labels
No labels
needs/docs
needs/research
needs/upstream
type/bug
type/external
type/feature
type/refactor
urgency
high
urgency
immediate
urgency
low
urgency
medium
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
ScottyLabs/infrastructure!195
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/ci-semgrep-aarch64"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
CI on
mainand on every PR has failed since8842761(Oct 21), the last green run being 174. All three jobs fail building the devenv shell:Cause
8842761moved the devenv nixpkgs pin from 55bfb4d (semgrep 1.179.0) to 02c087f (semgrep 1.180.0). nixpkgs repackages semgrep-core from the PyPI wheel, and its only build step is onepatchelf --set-interpreter --set-rpathon the binary. On the aarch64-linux wheel (the runner on infra-03 is aarch64), patchelf 0.19.2 from that same nixpkgs aborts:I reproduced this outside CI with the exact wheel (its hash matches nixpkgs,
sha256-y6BvZLEOXshqCzqXV8/f3zzmwrvbXph2KqKksKKW+Mo=), so it isn't a nixbuild.net problem, and nixbuild just keeps returning the cached failure. nixpkgs master and both unstable channels still have 1.180.0 with the same build. governance passes because its devenv nixpkgs is older (semgrep 1.172.0), while kennel's own CI fails too.Fix
Turn off kennel's
scottylabs.security.semgrepon aarch64-linux only. That removes semgrep-core from the shell and drops thesecurity:semgreptask there, while x86_64 and darwin shells keep it. CI stops running semgrep until this is reverted. It was not running before this PR either, since the shell couldn't build.The proper fix belongs upstream: either nixpkgs' semgrep-core avoids patchelf on aarch64 (for example, launching the binary through the dynamic loader), or a newer semgrep wheel stops triggering the bug. Once a fixed nixpkgs is in
devenv.lock, this line can go.CI on this PR will show whether anything else is broken behind semgrep.
Second failure behind it
With the shell building,
check-1passed, andbuildreached its next step and failed:The build job builds each kennel site for the runner's own system, and
flake.nixonly declaredx86_64-linux. The second commit addsaarch64-linuxtosystems. All 25 per-system packages (the docs site and every terranix configuration) evaluate for aarch64. Nothing else in the flake readssystems, and host configurations are unaffected. They pick their ownsystemfromhosts.nix/systems.nix.The
::error::Command failed: devenv shell rustc -vVline in the logs is harmless: kennel's prepare action probes for a Rust toolchain, which this repo doesn't have, and the step carries on.Kennel Deployments
Last updated for commit
34078d5.fix: skip semgrep in the devenv shell on aarch64-linuxto fix: get CI building on the aarch64 runner againRan Plan for 22 projects:
occdir:.workspace:occlitellmdir:.workspace:litellmgarage-webadmindir:.workspace:garage-webadminobservabilitydir:.workspace:observabilitynixbotdir:.workspace:nixbotatlantisdir:.workspace:atlantisheadscaledir:.workspace:headscalegovernancedir:.workspace:governancehostsdir:.workspace:hostsmaildir:.workspace:mailcloudflare-tokensdir:.workspace:cloudflare-tokensposthogdir:.workspace:posthogvaultwardendir:.workspace:vaultwardenkenneldir:.workspace:kennelsearch-consoledir:.workspace:search-consolematrixdir:.workspace:matrixsnotdir:.workspace:snotricochetdir:.workspace:ricochetwebhookdir:.workspace:webhookforgejodir:.workspace:forgejoopenbaodir:.workspace:openbaokeycloakdir:.workspace:keycloak1. project:
occdir:.workspace:occPlan Failed: This project is currently locked by an unapplied plan from pull #193. To continue, delete the lock from #193 or apply that plan and merge the pull request.
Once the lock is released, comment
atlantis planhere to re-plan.2. project:
litellmdir:.workspace:litellmPlan Failed: This project is currently locked by an unapplied plan from pull #193. To continue, delete the lock from #193 or apply that plan and merge the pull request.
Once the lock is released, comment
atlantis planhere to re-plan.3. project:
garage-webadmindir:.workspace:garage-webadminPlan Failed: This project is currently locked by an unapplied plan from pull #193. To continue, delete the lock from #193 or apply that plan and merge the pull request.
Once the lock is released, comment
atlantis planhere to re-plan.4. project:
observabilitydir:.workspace:observabilityPlan Failed: This project is currently locked by an unapplied plan from pull #193. To continue, delete the lock from #193 or apply that plan and merge the pull request.
Once the lock is released, comment
atlantis planhere to re-plan.5. project:
nixbotdir:.workspace:nixbotPlan Failed: This project is currently locked by an unapplied plan from pull #193. To continue, delete the lock from #193 or apply that plan and merge the pull request.
Once the lock is released, comment
atlantis planhere to re-plan.6. project:
atlantisdir:.workspace:atlantisPlan Failed: This project is currently locked by an unapplied plan from pull #193. To continue, delete the lock from #193 or apply that plan and merge the pull request.
Once the lock is released, comment
atlantis planhere to re-plan.7. project:
headscaledir:.workspace:headscalePlan Failed: This project is currently locked by an unapplied plan from pull #193. To continue, delete the lock from #193 or apply that plan and merge the pull request.
Once the lock is released, comment
atlantis planhere to re-plan.8. project:
governancedir:.workspace:governancePlan Failed: This project is currently locked by an unapplied plan from pull #193. To continue, delete the lock from #193 or apply that plan and merge the pull request.
Once the lock is released, comment
atlantis planhere to re-plan.9. project:
hostsdir:.workspace:hostsPlan Failed: This project is currently locked by an unapplied plan from pull #193. To continue, delete the lock from #193 or apply that plan and merge the pull request.
Once the lock is released, comment
atlantis planhere to re-plan.10. project:
maildir:.workspace:mailPlan Failed: This project is currently locked by an unapplied plan from pull #193. To continue, delete the lock from #193 or apply that plan and merge the pull request.
Once the lock is released, comment
atlantis planhere to re-plan.11. project:
cloudflare-tokensdir:.workspace:cloudflare-tokensPlan Failed: This project is currently locked by an unapplied plan from pull #193. To continue, delete the lock from #193 or apply that plan and merge the pull request.
Once the lock is released, comment
atlantis planhere to re-plan.12. project:
posthogdir:.workspace:posthogPlan Failed: This project is currently locked by an unapplied plan from pull #193. To continue, delete the lock from #193 or apply that plan and merge the pull request.
Once the lock is released, comment
atlantis planhere to re-plan.13. project:
vaultwardendir:.workspace:vaultwardenPlan Failed: This project is currently locked by an unapplied plan from pull #193. To continue, delete the lock from #193 or apply that plan and merge the pull request.
Once the lock is released, comment
atlantis planhere to re-plan.14. project:
kenneldir:.workspace:kennelPlan Failed: This project is currently locked by an unapplied plan from pull #193. To continue, delete the lock from #193 or apply that plan and merge the pull request.
Once the lock is released, comment
atlantis planhere to re-plan.15. project:
search-consoledir:.workspace:search-consolePlan Failed: This project is currently locked by an unapplied plan from pull #193. To continue, delete the lock from #193 or apply that plan and merge the pull request.
Once the lock is released, comment
atlantis planhere to re-plan.16. project:
matrixdir:.workspace:matrixPlan Failed: This project is currently locked by an unapplied plan from pull #193. To continue, delete the lock from #193 or apply that plan and merge the pull request.
Once the lock is released, comment
atlantis planhere to re-plan.17. project:
snotdir:.workspace:snotPlan Failed: This project is currently locked by an unapplied plan from pull #193. To continue, delete the lock from #193 or apply that plan and merge the pull request.
Once the lock is released, comment
atlantis planhere to re-plan.18. project:
ricochetdir:.workspace:ricochetPlan Failed: This project is currently locked by an unapplied plan from pull #193. To continue, delete the lock from #193 or apply that plan and merge the pull request.
Once the lock is released, comment
atlantis planhere to re-plan.19. project:
webhookdir:.workspace:webhookPlan Failed: This project is currently locked by an unapplied plan from pull #193. To continue, delete the lock from #193 or apply that plan and merge the pull request.
Once the lock is released, comment
atlantis planhere to re-plan.20. project:
forgejodir:.workspace:forgejoPlan Failed: This project is currently locked by an unapplied plan from pull #193. To continue, delete the lock from #193 or apply that plan and merge the pull request.
Once the lock is released, comment
atlantis planhere to re-plan.21. project:
openbaodir:.workspace:openbaoPlan Failed: This project is currently locked by an unapplied plan from pull #193. To continue, delete the lock from #193 or apply that plan and merge the pull request.
Once the lock is released, comment
atlantis planhere to re-plan.22. project:
keycloakdir:.workspace:keycloakPlan Failed: This project is currently locked by an unapplied plan from pull #193. To continue, delete the lock from #193 or apply that plan and merge the pull request.
Once the lock is released, comment
atlantis planhere to re-plan.Plan Summary
22 projects, 0 with changes, 0 with no changes, 22 failed
atlantis plan
Ran Plan for 22 projects:
maildir:.workspace:mailforgejodir:.workspace:forgejogovernancedir:.workspace:governanceoccdir:.workspace:occmatrixdir:.workspace:matrixnixbotdir:.workspace:nixbotgarage-webadmindir:.workspace:garage-webadminobservabilitydir:.workspace:observabilityatlantisdir:.workspace:atlantisheadscaledir:.workspace:headscalekeycloakdir:.workspace:keycloakcloudflare-tokensdir:.workspace:cloudflare-tokenskenneldir:.workspace:kennellitellmdir:.workspace:litellmhostsdir:.workspace:hostsopenbaodir:.workspace:openbaoposthogdir:.workspace:posthogricochetdir:.workspace:ricochetsnotdir:.workspace:snotwebhookdir:.workspace:webhookvaultwardendir:.workspace:vaultwardensearch-consoledir:.workspace:search-console1. project:
maildir:.workspace:mailShow Output
No changes. Your infrastructure matches the configuration.
2. project:
forgejodir:.workspace:forgejoShow Output
No changes. Your infrastructure matches the configuration.
3. project:
governancedir:.workspace:governanceShow Output
No changes. Your infrastructure matches the configuration.
4. project:
occdir:.workspace:occShow Output
No changes. Your infrastructure matches the configuration.
5. project:
matrixdir:.workspace:matrixShow Output
No changes. Your infrastructure matches the configuration.
6. project:
nixbotdir:.workspace:nixbotShow Output
No changes. Your infrastructure matches the configuration.
7. project:
garage-webadmindir:.workspace:garage-webadminShow Output
No changes. Your infrastructure matches the configuration.
8. project:
observabilitydir:.workspace:observabilityShow Output
No changes. Your infrastructure matches the configuration.
9. project:
atlantisdir:.workspace:atlantisShow Output
No changes. Your infrastructure matches the configuration.
10. project:
headscaledir:.workspace:headscaleShow Output
No changes. Your infrastructure matches the configuration.
11. project:
keycloakdir:.workspace:keycloakShow Output
No changes. Your infrastructure matches the configuration.
12. project:
cloudflare-tokensdir:.workspace:cloudflare-tokensShow Output
No changes. Your infrastructure matches the configuration.
13. project:
kenneldir:.workspace:kennelShow Output
No changes. Your infrastructure matches the configuration.
14. project:
litellmdir:.workspace:litellmShow Output
No changes. Your infrastructure matches the configuration.
15. project:
hostsdir:.workspace:hostsShow Output
No changes. Your infrastructure matches the configuration.
16. project:
openbaodir:.workspace:openbaoShow Output
No changes. Your infrastructure matches the configuration.
17. project:
posthogdir:.workspace:posthogShow Output
No changes. Your infrastructure matches the configuration.
18. project:
ricochetdir:.workspace:ricochetShow Output
No changes. Your infrastructure matches the configuration.
19. project:
snotdir:.workspace:snotShow Output
No changes. Your infrastructure matches the configuration.
20. project:
webhookdir:.workspace:webhookShow Output
No changes. Your infrastructure matches the configuration.
21. project:
vaultwardendir:.workspace:vaultwardenShow Output
No changes. Your infrastructure matches the configuration.
22. project:
search-consoledir:.workspace:search-consoleShow Output
No changes. Your infrastructure matches the configuration.
Plan Summary
22 projects, 0 with changes, 22 with no changes, 0 failed
@ -10,1 +10,4 @@kennel.sites.docs.customDomain = "infra.scottylabs.org";# CI runs on aarch64-linux, where patchelf aborts on semgrep-core 1.180.0's binarysecurity.semgrep.enable = !(pkgs.stdenv.hostPlatform.isLinux && pkgs.stdenv.hostPlatform.isAarch64);is there a tracking issue for this being fixed upstream?
the convention for patches like this across the infrastructure repo is:
with no other context
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.