Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

admin: add alert auto-close feature #2660

Merged
merged 20 commits into from
Nov 10, 2022

Conversation

Lakshmi2107
Copy link
Contributor

Description:
GoAlert sends alerts to various communication methods, once the alert is created it will be in unacknowledged state. there are some alerts that are in unack state forever. Any alert created in the goalert especially for monitoring production systems should have some lifecycle and close beyond some time say 24 hrs, otherwise it won't be meaningful.

Which issue(s) this PR fixes:
Fixes #2658

Screenshots:
Screenshot 2022-10-14 at 6 39 06 PM

Additional Info:
Need to add a few rows in config_limits table and add values to the config limit enum
alter type enum_limit_type add value 'maximum_time_to_auto_close_alert';

Rows to be inserted in table config_limits
Max time to auto-close an alert ( in hours )
Example:
"maximum_time_to_auto_close_alert" 3

Lakshmi V added 2 commits October 14, 2022 18:15
# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.
Copy link
Member

@mastercactapus mastercactapus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We settled on a design for this: #2658 (comment)

@mastercactapus
Copy link
Member

@Lakshmi2107 let us know if you'd like any help implementing the changes, thanks for the PR!

@sivtechrepo
Copy link

thank you @mastercactapus. We are good for now. We will update the implementation once done. @Lakshmi2107

@Lakshmi2107
Copy link
Contributor Author

@mastercactapus Updated PR with auto-close changes. Please review.

@Lakshmi2107
Copy link
Contributor Author

@mastercactapus Thanks for the comments. I have pushed all suggested changes. Please review.

@Lakshmi2107
Copy link
Contributor Author

@mastercactapus Addressed all review comments. Please check

Copy link
Member

@mastercactapus mastercactapus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!!!

I pushed a couple of tweaks to wrap this up:

  • re-ordered things to be AlertCleanup, then AlertAutoClose, everywhere
  • Added a small smoke test to validate the functionality
  • added a coalesce to the EP notice query, as it caused a NULL error when there were no escalation policy steps

@mastercactapus mastercactapus changed the title Feature/auto close alert admin: add alert auto-close feature Nov 8, 2022
@Lakshmi2107
Copy link
Contributor Author

Thanks @mastercactapus

@mastercactapus mastercactapus merged commit a6ddae7 into target:master Nov 10, 2022
@sivtechrepo
Copy link

thank you so much @mastercactapus @Lakshmi2107

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

alerts: auto-close unacked/stale alerts
4 participants