Data Distribution Network: Investigating Reported Service Disruptions
Incident Report for Splunk On-Call
Resolved
Incident Synopsis: One of our Data Distribution Network providers experienced tangible technical challenges in the following points of presence (PoPs) beginning at 07:00 UTC yesterday:­ San Jose, CA; Los Angeles, CA; Chicago, IL; Hong Kong; Taipei, Taiwan; and Paris, France.

Impact: This issue caused severe degradation, and, in some cases, total unavailability of DNS resolution in the above listed data centers between 07:00 UTC and 09:25 UTC yesterday.

Resolution: The provider in-question identified the technical symptom that caused this incident and they are taking defined actions to remedy the situation going forward. Correspondingly, VictorOps will continue to stay in real time communication with the technical teams at said provider throughout the overall life cycle of this incident. Additional internal measures have been implemented here at VictorOps to ensure a much more seamless overall handling of such an incident if and when it may occur again.

In closing, we sincerely apologize for any unintended inconvenience you and your colleagues may have experienced during the above incident. If you have any additional questions or concerns in relation to this incident, please do not hesitate to contact the VictorOps Support Team at support@victorops.com
Posted Sep 15, 2016 - 10:25 MDT
Monitoring
We are continuing to actively work with the Data Distribution Network provider in-question in response to the incidents reported earlier today. All indications are that the earlier service disruptions have been addressed and that our end-users should not be experiencing any related technical challenges. Nonetheless, VictorOps feels that it is prudent to leave the status of this incident as, "monitoring", until further notice.

We will update our status page with any applicable updates. In the interim, you are welcome and encouraged to contact the VictorOps Support team with any immediate questions or concerns via support@victorops.com

Thank you for your continued patience.
Posted Sep 14, 2016 - 11:08 MDT
Investigating
We are actively investigating reported service disruptions in association to one of our Data Distribution Network providers. We sincerely apologize for any unintended inconvenience. Please stay tuned to the VictorOps status page for additional information along with more detailed technical updates.

If you have any immediate questions or concerns, please contact the VictorOps Support Team at support@victorops.com

Again, our apologies for any unintended inconvenience. Updates to follow.
Posted Sep 14, 2016 - 07:56 MDT
This incident affected: Alert Processing & Integrations (Alert Ingestion).