Postmortem -
Read details
Feb 26, 16:22 MST
Resolved -
The issue has been resolved. All message delivery types (Phone, SMS, Push, and Email) and associated notifications are fully functional.
We will be providing additional updates on our status page upon the completion of our internal review.
If you have any immediate questions, please reach out to the Splunk On-Call Support team:
victorops-support@splunk.comWe sincerely apologize for any unintended inconvenience this issue may have caused.
Feb 26, 09:38 MST
Update -
We are continuing to investigate this issue.
Feb 26, 09:34 MST
Update -
SMS message delivery is improving significantly (to) fully functional.
Phone messaging is still delayed.
We are continuing to investigate the issue. Updates to follow.
Feb 26, 08:58 MST
Update -
We are continuing to investigate this issue.
Feb 26, 08:56 MST
Update -
As we continue to investigate this issue, we recommend leveraging PUSH Notification in your Splunk On-Call personal paging policy settings. For guidance:
https://help.victorops.com/knowledge-base/paging-policy-setup/The service disruption we are continuing to troubleshoot is primarily affecting Phone and SMS messaging.
Updates to follow.
Feb 26, 08:38 MST
Update -
We are continuing to investigate this issue.
Feb 26, 08:30 MST
Update -
Please be advised that message delivery and alerting associated with the Splunk On-Call platform are significantly delayed as a result of this issue.
As we continue to investigate and work this issue to ground, we recommend that you directly evaluate any monitoring solutions you may have integrated with Splunk On-Call to better determine the health and status of your associated systems, applications, and/or platform.
Updates to follow.
Feb 26, 07:55 MST
Update -
Update: The service affecting issues that we are actively investigating appear to be associated with a service disruption being experienced by one of our communications partners, Twilio:
https://status.twilio.com/We are continuing to pursue the investigation and resolution of this issue with the highest levels of urgency.
Updates to follow.
Feb 26, 07:46 MST
Update -
We are continuing to investigate this issue.
Feb 26, 07:33 MST
Investigating -
We are actively investigating a service affecting issue with the message delivery and alerting associated to our platform. We are approaching this issue with the highest level of urgency and the appropriate parties are actively engaged in troubleshooting the issue.
Updates to follow as soon as they are available.
f you have any immediate questions, please contact the Splunk On-Call Support team:
victorops-support@splunk.com
Feb 26, 07:33 MST