ScottyLabs/infrastructure
Watch
4
0
Fork
You've already forked infrastructure
 2
Code Issues 45 Pull requests 3 Projects Releases Packages Wiki Activity Actions

Auto-delete orphaned Grafana alert rules on deploy #103

New issue
Open
opened 2026-07-26 04:50:26 +00:00 by anish · 0 comments
anish commented 2026-07-26 04:50:26 +00:00 (Migrated from codeberg.org)
Copy link

Grafana file provisioning never deletes alert rules dropped from config, so removed rules keep evaluating from the DB and firing (see this issue's old title)

Add a post-grafana.service oneshot that reconciles via the Provisioning API with X-Disable-Provenance, deleting any provisioned rule whose uid isn't in the current scottylabs.observability set

This would probably need a provisioned Grafana service-account token

Grafana file provisioning never deletes alert rules dropped from config, so removed rules keep evaluating from the DB and firing (see this issue's old title) Add a post-`grafana.service` oneshot that reconciles via the Provisioning API with `X-Disable-Provenance`, deleting any provisioned rule whose uid isn't in the current `scottylabs.observability` set This would probably need a provisioned Grafana service-account token
anish changed title from Remove dead comin alert from observability to Auto-delete orphaned Grafana alert rules on deploy 2026-08-05 19:15:24 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
renovate/scottylabs-devenv-digest
crawler-archive-disk
alert-earlier
triage/triage-issue-108-replace-org-level-ci-secrets-with-20260728-201750
triage/triage-issue-109-explore--nixos-unified--as-an-alt-20260728-202048
vault-iam
renovate/lock-file-maintenance
fix/postgres-public-schema-permissions
docs-devops-subdomain
openbao-auto-unseal
icon-url
atlantis
fix-secrets-dns-apply
revert-bus-sign
slai-backend
mcp-flake
voting-app-flake
tofu-s3-bucket
retry-add-thesuperRL
No results found.
Labels
Clear labels
  
needs/docs
Must have documentation written

  
needs/research
Discuss the ideal implementation with the tech lead

  
needs/upstream
Needs a contribution to an upstream project

  
type/bug
Something not working as intended

  
type/external
Lives outside of code

  
type/feature
New functionality

  
type/refactor
Improving existing code

  
urgency
high
 This is currently causing issues

  
urgency
immediate
 This is making a project or feature unusable

  
urgency
low
 This can happen whenever

  
urgency
medium
 This is expected to cause issues soon

No labels
needs/docs
needs/research
needs/upstream
type/bug
type/external
type/feature
type/refactor
urgency
high
urgency
immediate
urgency
low
urgency
medium
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
DevOps TODO
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
ScottyLabs/infrastructure#103
No description provided.