Splunk On-Call is experiencing disruptions for Rules Engine functionality and Incident Timeline Data

Incident Report for Splunk On-Call

Resolved

This incident has been resolved.

We apologize for this inconvenience. Please reach out to On-Call Support with any questions:
https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

Posted May 21, 2024 - 14:18 MDT

Monitoring

The issue has been identified as upstream from the On-Call platform but impacting our CPU utilization. Mitigation is underway and we are now monitoring the results. Our engineering teams continue to work on this issue with urgency. We will provide updates as soon as we learn more information.

Please Note: No disruption to Incident creation and notifications out to on-call end-users have been detected.

Stay tuned for further updates as we work towards a complete resolution.

We apologize for this inconvenience. Please reach out to On-Call Support with any questions:
https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

Posted May 17, 2024 - 13:59 MDT

Update

Please Note: No disruption to Incident creation and notifications out to on-call end-users have been detected.

Impacted Functionality:
-Increased RegEx Rules Engine rule timeouts - Disabled Rules Engine Rules may be reenabled to resume functionality.

-UI and Mobile App Timeline data (incident cards) - Alert details may still be found via the On-Call API

-UI and Mobile App empty Incident alert details - Alert details may still be found via the On-Call API or our Reporting features: Incident Frequency, On-Call Review, and Response Metrics.

Please be advised that our engineering teams are working urgently on this issue. We will provide updates as soon as we learn more information.

We apologize for this inconvenience. Please reach out to On-Call Support with any questions:
https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

Posted May 16, 2024 - 16:35 MDT

Update

We continue to investigate this issue. Please note that only a subset of instances are affected by this issue and no disruption to incident creation and notifications out to on-call end-users have been detected.

Apologies for any inconvenience.

Please reach out to On-Call Support with any questions:
https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

Posted May 15, 2024 - 16:03 MDT

Update

We are continuing to investigate this issue.

Posted May 14, 2024 - 14:57 MDT

Investigating

Please Note: No disruption to Incident creation and notifications out to on-call end-users have been detected.

Impacted Functionality:
-Increased RegEx Rules Engine rule timeouts - Disabled Rules Engine Rules may be reenabled to resume functionality.

-UI and Mobile App Timeline data (incident cards) - Alert details may still be found via the On-Call API

-UI and Mobile App empty Incident alert details - Alert details may still be found via the On-Call API or our Reporting features: Incident Frequency, On-Call Review, and Response Metrics.

Please be advised that our engineering teams are working on this issue with urgency. We will provide updates as soon as we learn more information.

We apologize for this inconvenience. Please reach out to On-Call Support with any questions:
https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

Posted May 14, 2024 - 14:56 MDT

This incident affected: Clients (Web client (portal), Android client, iOS client) and Alert Processing & Integrations (Alert Rules Engine).