Phone Notification Delivery Issues to +86 (China)
Incident Report for Splunk On-Call
Resolved
RESOLVED: Successful phone call routing to carriers in China (country code +86) has resumed.

The changes Splunk On-Call has implemented to avoid zero-second duration calls are outlined below and in our updated Knowledge Base article, found here: https://help.victorops.com/knowledge-base/mobile-app-getting-started/#phone-notifications-to-china-country-code-86

IMPORTANT NOTE: In response to this situation, On-Call took technical measures to modify call handling for end-users within the +86/China Country Code. These measures were implemented in an effort to prevent/avoid telecommunications restrictions and zero-second duration calls associated with the +86/China Country Code:

**After three attempts to notify via PHONE CALL to +86, we will cease any further attempts to contact the end user for a period of two hours. After two hours, we will make another three attempts, beginning the cycle again.
. . .

We sincerely appreciate your continued patience as we worked to resolve this issue.
Posted Aug 29, 2022 - 08:59 MDT
Monitoring
Although we have not received formal notification back from our telecommunications provider on this front, we are currently reviewing our logs and seeing successful call delivery into the +86/China Country Code.

IMPORTANT NOTE: In response to this situation, On-Call took technical measures to modify call handling for end-users within the +86/China Country Code. These measures were implemented in an effort to prevent/avoid telecommunications restrictions and zero-second duration calls associated with the +86/China Country Code:

**After three attempts to notify via PHONE CALL to +86, we will cease any further attempts to contact the end user for a period of two hours. After two hours, we will make another three attempts, beginning the cycle again.
. . .

We sincerely appreciate your continued patience as we (continue) to work this issue to ground.
Posted Aug 26, 2022 - 09:17 MDT
Identified
Working in collaboration with our current telecommunications provider for Phone notification service into +86/China, we're taking steps this week to significantly reduce (or) eliminate "zero-duration" calls from our inbound +86/China call volume. In doing so, we believe we can improve the quality of our inbound call volume into +86/China in specific response to their stringent requirements around inbound Phone calls.

NOTE: The above effort will require code development, testing, and production deployment coordination and is not a guarantee that the current Phone notification issues to end-users in the +86/China Country Code will be resolved. Nonetheless, we're working with urgency to have these updates deployed into production later this week.

In the interim, we strongly encourage all Splunk On-Call end-users to diversify their Personal Paging policies to include PUSH, EMAIL, and SMS notification methods.

We will continue to update our Status Page as relevant information arrives. We sincerely apologize for the ongoing challenges.
Posted Aug 22, 2022 - 09:29 MDT
Update
Thank you for your patience while we continue to work with our telecommunication provider to investigate phone notification service disruption to the (+86) China region.

As mentioned, this disruption is limited to Phone notifications to the (+86) region. We strongly encourage all On-Call end users in this region to diversify their personal paging policies to include Email, SMS and Push notification delivery as alternatives to Phone.

Our apologies for any inconvenience this situation may be causing you.
Posted Aug 19, 2022 - 16:05 MDT
Update
As we continue to investigate the +86 (China) Phone notification issue with our telecommunications providers, we strongly encourage Splunk On-Call end-users to diversify their Personal Paging Policies to include Push, Email, and SMS as notification methods.
Posted Aug 12, 2022 - 15:37 MDT
Investigating
We are actively researching a geographically isolated issue related to Phone Notification delivery challenges into the +86 (China) country code.

We have escalated this issue to our telecommunications providers and will continue to stay actively engaged in all efforts to resolve this situation. Relevant updates will be provided to this Status Page as they arrive.

As we continue to research this issue, we recommend that Splunk On-Call end-users located in China diversify their Splunk On-Call Personal Paging Policy to include Email, SMS, and Push notification delivery as alternatives to Phone.

Our apologies for any inconvenience this situation may be causing you and your end-users.

Updates to follow.
Posted Aug 08, 2022 - 21:46 MDT