tag:status.victorops.com,2005:/historySplunk On-Call Status - Incident History2024-03-28T23:23:30-06:00Splunk On-Calltag:status.victorops.com,2005:Incident/202162772024-03-21T23:30:34-06:002024-03-21T23:30:34-06:00Scheduled Splunk On-Call Reporting Maintenance - March 21st, 2024 from 9:00 PM to 1:00 AM (Mountain Time)<p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>23:30</var> MDT</small><br><strong>Completed</strong> - Scheduled maintenance has been completed, reporting tools are available again and all systems are fully operational.</p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>21:15</var> MDT</small><br><strong>Update</strong> - Scheduled Maintenance is currently ongoing. During this time all reports will be inaccessible. This includes creation of a Post-Incident Review, Response Metrics, On-Call Review, and Incident Frequency reports. We thank you for your patience as we perform this maintenance and will let you know when these reports are accessible again.</p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>21:00</var> MDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>11:33</var> MDT</small><br><strong>Update</strong> - UPDATE: Estimated Reporting unavailability time during scheduled maintenance has been extended to FOUR HOURS.</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>09:42</var> MDT</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>09:36</var> MDT</small><br><strong>Scheduled</strong> - Please pardon our dust! Splunk On-Call Reporting will be unavailable during time of maintenance: Estimated Reporting unavailability time: TWO HOURS. <br /><br />After initial maintenance, we expect delays in Reporting data. Estimated Reporting data delay: FOUR to SIX hours. No data loss is expected during or after maintenance, only delays to Incident Frequency Reporting (IFR), Response Metrics (MTTA/R), On-Call Reporting (OCR), and Post Incident Review (PIR).<br /><br />Please reach out to Splunk On-Call Support with any questions or concerns!<br />https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/</p>tag:status.victorops.com,2005:Incident/201369082024-03-01T12:32:21-07:002024-03-01T12:32:21-07:00Splunk On-Call MS Teams integration disruption<p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>12:32</var> MST</small><br><strong>Resolved</strong> - The issue has been identified and resolved! Apologies for any inconvenience!<br /><br />Please direct any questions regarding this incident to On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>11:52</var> MST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>11:51</var> MST</small><br><strong>Investigating</strong> - We are currently investigating a problem regarding the performance of our MS Team integration. Please bear with us while we troubleshoot this issue — updates to follow.</p>tag:status.victorops.com,2005:Incident/200954602024-02-28T16:46:46-07:002024-02-28T16:46:47-07:00Experiencing Degradation with Reports<p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>16:46</var> MST</small><br><strong>Resolved</strong> - This incident has been resolved. Reporting is fully operational. We apologize again for any inconvenience. <br /><br />Please direct any questions regarding this incident to On-Call Support:<br />https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>14:08</var> MST</small><br><strong>Update</strong> - UPDATE: Reporting back to Operational status. <br />We continue to monitor our Reporting data and will be keeping this incident open for the time being.<br /><br />Thanks again for your patience while we work to completely resolve this incident.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>12:24</var> MST</small><br><strong>Monitoring</strong> - UPDATE: Remediation complete; watching the latency decrease. <br />Thanks again for your patience while we work this issue to resolution.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>09:28</var> MST</small><br><strong>Update</strong> - Apologies for the Reporting inconvenience! <br /><br />Current Status:<br /> - Data is up to date as of 6:00 AM (UTC)<br /> - No reporting data lost <br /> - Data replication is in progress and will be 100% available once complete<br /><br />Please direct any questions to Splunk On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>01:58</var> MST</small><br><strong>Identified</strong> - We have identified the issue and are working towards a resolution.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>01:57</var> MST</small><br><strong>Investigating</strong> - We are currently experiencing lag in our report tooling. New Incidents will not show-up in any reports until the incident is resolved.</p>tag:status.victorops.com,2005:Incident/198201002024-01-24T11:47:08-07:002024-01-24T11:47:08-07:00Splunk On-Call is experiencing Isolated challenges related to loading the Incident Page for large instances<p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>11:47</var> MST</small><br><strong>Resolved</strong> - This incident has been resolved. All systems are operational. If you're still experiencing an issue with loading large lists of on-call users in the Incident tab under "Team: All Teams" please refresh the page and give it a few seconds. If the problem continues, please reach out to On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>10:57</var> MST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented. Please stay tuned for further updates.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>10:31</var> MST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>10:30</var> MST</small><br><strong>Investigating</strong> - Splunk On-Call is currently experiencing an isolated disruption to the Incident Page for large instances. We are actively working to resolve this issue as quickly as possible.<br /><br />Please use the Subscribe to Updates option or follow @VOSupport on Twitter for updates. If you have any immediate questions or concerns, you may contact our support team as outlined here: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/<br /><br />Thank you for your patience while we work this issue to ground.</p>tag:status.victorops.com,2005:Incident/194553252023-12-18T17:19:00-07:002023-12-18T17:19:01-07:00On-Call SSO iOS Mobile App log-in disruption<p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>17:19</var> MST</small><br><strong>Resolved</strong> - This incident has been resolved. All On-Call systems are operational. Thank you for your patience while we completed our testing.<br />For questions regarding this incident, please contact Splunk On-Call Support: <br />https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/</p><p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>12:08</var> MST</small><br><strong>Update</strong> - We are still monitoring the status of this incident. Testing continues. All systems are operational.</p><p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>02:13</var> MST</small><br><strong>Update</strong> - We continue to monitor the issue but have moved our iOS SSO Mobile login back to operational status. Thanks again for your patience while we continue to work the root cause of this issue to ground.</p><p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>02:10</var> MST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>01:45</var> MST</small><br><strong>Monitoring</strong> - We've identified the issue and have implemented a fix. Moving this incident to a monitoring status. iOS SSO mobile login should now be available. Updates to follow.<br /><br />Apologies again for any inconvenience.</p><p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>01:02</var> MST</small><br><strong>Investigating</strong> - We are currently investigating an issue with iOS mobile SSO log-in. For immediate access to Splunk On-Call please use the Web Client until we can identify the root cause of this issue. Again, this is only affecting iOS mobile. Thanks your your patience while we troubleshoot this issue.</p>tag:status.victorops.com,2005:Incident/184408352023-09-29T10:30:08-06:002023-09-29T10:30:08-06:00DEPRECATION OF SERVICE: Splunk On-Call Voice Notifications to China (+86)<p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>10:30</var> MDT</small><br><strong>Completed</strong> - Maintenance completed. Voice notification service to China (+86) is deprecated.<br /><br />Affected end-users are encouraged to configure personal paging policies to diversify their Splunk On-Call notification methods (i.e., utilize SMS, Push, and/or Email notification methods). <br /><br />Please note that the Splunk On-Call product continues to be supported globally and all functionality will be available to customers with responders outside of the +86 country code (China). <br /><br />We sincerely apologize for any inconvenience this may cause</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>10:26</var> MDT</small><br><strong>Verifying</strong> - We are verifying maintenance on deprecating Splunk On-Call voice call notification +86 country code (China).</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>10:06</var> MDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>16:39</var> MDT</small><br><strong>Scheduled</strong> - As you may know, we are all experiencing increased restrictions and regulations between the U.S. and China. As a company, Splunk has made the decision to no longer provide Voice Call notifications to responders in +86 country code (China) effective on September 29th, 2023.<br /><br />In response, affected end-users are encouraged to configure personal paging policies to diversify their Splunk On-Call notification methods (i.e., utilize SMS, Push, and/or Email notification methods). <br /><br />Please note that the Splunk On-Call product continues to be supported globally and all functionality will be available to customers with responders outside of the +86 country code (China). <br /><br />Again, please be advised that we will be deprecating Splunk On-Call voice call notification service for +86 country code (China) effective on September 29th, 2023. <br /><br />We sincerely apologize for any inconvenience this may cause.</p>tag:status.victorops.com,2005:Incident/185032062023-09-14T15:04:16-06:002023-09-14T15:04:16-06:00Splunk On-Call Knowledge Base temporarily unavailable<p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>15:04</var> MDT</small><br><strong>Resolved</strong> - Host Update - "The original issue has been successfully resolved." Accordingly, we will be resolving this incident. <br /><br />Please reach out Splunk On-Call Support with any questions or concerns regarding this incident!<br /><br />https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/</p><p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>13:29</var> MDT</small><br><strong>Update</strong> - Host Update - "Site availability is continuing to improve. We are actively monitoring and continue to see a full recovery in progress.<br />Sep 14, 2023 - 12:12 PDT" ... Splunk On-Call will resolve this Statuspage incident when our host gives the all clear. For all intents and purposes, our Knowledge Base is back to fully operational.</p><p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>12:54</var> MDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>12:52</var> MDT</small><br><strong>Monitoring</strong> - A fix has been implemented by our host and the Knowledge Base (https://help.victorops.com/) is once again available. If you are still experiencing loading issues please clear your cache and refresh the page. Thank you!</p><p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>12:42</var> MDT</small><br><strong>Identified</strong> - The issue has been identified by our KB host and they are working as quickly as possible towards resolution. Thanks for your patience while we/they restore functionality.</p><p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>11:53</var> MDT</small><br><strong>Investigating</strong> - We are currently investigating an issue with our Knowledge Base hosting service. The ability to contact Support is unaffected and fully operational. <br />We will provide updates as soon as they're available.</p>tag:status.victorops.com,2005:Incident/174716762023-06-07T12:41:20-06:002023-06-07T12:41:21-06:00UPDATE: Rescheduled Splunk On-Call Platform Maintenance - June 7th, 2023 from 10:00 AM to 2:00 PM (Mountain)<p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>12:41</var> MDT</small><br><strong>Completed</strong> - Scheduled Maintenance has been completed. Apologies for the extended window and disruption. All Systems Operational.<br /><br />For questions or concerns regarding this maintenance please contact Splunk On-Call (formerly VictorOps) Support. Please review the linked Knowledge Base article: How to Contact Splunk On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>12:38</var> MDT</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>12:08</var> MDT</small><br><strong>Update</strong> - UPDATE: Platform has returned to operational status.<br />We will continue testing features and components.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>11:57</var> MDT</small><br><strong>Update</strong> - UPDATE: Please reference the On-Call Mobile App for access to alerting and notifications. <br />The Web Client is currently impacted.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>11:53</var> MDT</small><br><strong>Update</strong> - Update: The On-Call platform is still in a degraded state. Extending maintenance window to 2:00 PM Mountain. Thank you for your patience while we work these issues to resolution.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>11:34</var> MDT</small><br><strong>Update</strong> - Update: Deploy complications are impacting our downtime. Please bear with us while we work towards resolution.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>11:15</var> MDT</small><br><strong>Update</strong> - Scheduled Maintenance is still in progress. Entering a bit of platform degradation and approximately 5 mins of pause. Please bear with us and thanks for your patience.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>10:00</var> MDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>12:05</var> MDT</small><br><strong>Scheduled</strong> - **UPDATED / RESCHEDULED MAINTENANCE WINDOW**<br /><br />Please note, the Splunk On-Call Platform will be undergoing scheduled maintenance for approximately two hours on June 7th from 10:00 AM to 12:00 PM (Mountain time). <br /><br />*We expect zero interruptions to alert ingestion. No dropped alerts.<br />*We expect some platform disruption for up to an hour.<br />*We expect delays in paging/notifications of up to ten minutes.<br />*We expect that the Public API, SMS replies for Ack and Resolve, the Web Client, and Mobile Clients will be unavailable for up to ten minutes.<br /><br />To contact Splunk On-Call (formerly VictorOps) Support during this scheduled maintenance window, please review the linked Knowledge Base article: How to Contact Splunk On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/<br /><br />Thank you for your patience while we proceed with maintenance.</p>tag:status.victorops.com,2005:Incident/172941802023-05-18T13:09:13-06:002023-05-18T13:09:15-06:00On-Call Team Incidents Dashboard in the Incidents Tab is experiencing issues loading<p><small>May <var data-var='date'>18</var>, <var data-var='time'>13:09</var> MDT</small><br><strong>Resolved</strong> - This incident has been resolved. Apologies for the inconvenience -- Team Incidents Dashboard is back to operational.</p><p><small>May <var data-var='date'>18</var>, <var data-var='time'>12:55</var> MDT</small><br><strong>Investigating</strong> - Apologies for the inconvenience. We are investigating a loading issue on the Incidents Tab affecting the Team Incidents Dashboard. Please refer to the People Pane in the Timeline tab for this information. <br /><br />Updates to follow.</p>tag:status.victorops.com,2005:Incident/172081452023-05-10T15:21:02-06:002023-05-10T15:21:02-06:00Splunk On-Call Timeline experiencing loading issues.<p><small>May <var data-var='date'>10</var>, <var data-var='time'>15:21</var> MDT</small><br><strong>Resolved</strong> - Testing has been completed and this incident has been resolved.<br /><br />Duration of Impact: (109 min) 7:49 AM to 9:38 AM (MST)<br />- This event was a visual impact on the Timeline in the UI.<br />- No Alerts were dropped during this time<br />- Detailed Alert information in the affected time range may be found via the Incident Frequency Report in our in-product reporting features.<br /><br />Again, apologies for this disruption and thank you for your patience while we worked the issue to the ground.<br /><br />Please reach out to Splunk On-Call Support with any questions regarding this event.<br />https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>13:20</var> MDT</small><br><strong>Update</strong> - Current Status: We remain in a "monitoring" state for this issue. <br /><br />- Impacted customer Timelines have returned to fully operational<br />- System testing continues<br /><br />We'll provide a more detailed update here as soon as the issue is fully resolved. Apologies again for this inconvenience and thank you for your patience while we work this issue towards resolution.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>13:15</var> MDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>09:58</var> MDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results. Timelines are back to operational. Thank you for your patience while we continue to work on this issue. Please stay tuned for further updates.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>09:43</var> MDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>09:19</var> MDT</small><br><strong>Investigating</strong> - We are currently investigating Timeline and Mobile App loading issues. Please bear with us as we investigate this issue. Please stay tuned for updates.</p>tag:status.victorops.com,2005:Incident/171382652023-05-03T20:23:23-06:002023-05-03T20:23:24-06:00Single Sign On Login Disabled Platform wide<p><small>May <var data-var='date'> 3</var>, <var data-var='time'>20:23</var> MDT</small><br><strong>Resolved</strong> - This incident has been resolved. All systems are operational. Apologies again for this disruption.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>20:20</var> MDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>19:59</var> MDT</small><br><strong>Update</strong> - We are working with our provider to regain access. Please bear with us while we work this issue to resolution. Our apologies for this inconvenience.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>19:27</var> MDT</small><br><strong>Identified</strong> - Currently experiencing issues with SSO login for all providers. Please bear with us while we work toward a resolution.</p>tag:status.victorops.com,2005:Incident/163588192023-03-08T11:04:54-07:002023-03-08T11:04:56-07:00Scheduled Splunk On-Call Integration Maintenance (MS Teams and Jira Cloud) - March 8th, 2023 from 10:00 to 11:30 am (Mountain)<p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>11:04</var> MST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed. Thanks for patience during our work. <br /><br />Please reach out to On-Call Support with any questions concerns!</p><p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>10:47</var> MST</small><br><strong>Update</strong> - We are continuing to verify the maintenance items.</p><p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>10:46</var> MST</small><br><strong>Verifying</strong> - Verification is currently underway for the MSTeams and Jira Cloud maintenance.</p><p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>10:31</var> MST</small><br><strong>Update</strong> - Jira Cloud back to operational. Moving on to MSTeams.</p><p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>10:00</var> MST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>12:33</var> MST</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>12:31</var> MST</small><br><strong>Scheduled</strong> - Please pardon our dust, we will be undergoing scheduled maintenance to our MS Teams and Jira Cloud integration databases during this time.<br /><br />*We expect total downtime to be less than 10 min. <br /><br />To contact Splunk On-Call (formerly VictorOps) Support during this scheduled maintenance window, please review the linked Knowledge Base article: How to Contact Splunk On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/<br /><br />Thank you for your patience while we proceed with maintenance.</p>tag:status.victorops.com,2005:Incident/160683162023-02-18T10:57:31-07:002023-02-18T10:57:33-07:00Scheduled Splunk On-Call Platform Maintenance - Feb 18th, 2023 from 9:00 to 11:30 am (Mountain)<p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>10:57</var> MST</small><br><strong>Completed</strong> - The Scheduled Maintenance has been completed. <br /><br />To contact Splunk On-Call (formerly VictorOps) Support for any questions regarding this scheduled maintenance window, please reach out to On-Call Support via the linked Knowledge Base article: How to Contact Splunk On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>10:54</var> MST</small><br><strong>Update</strong> - Scheduled maintenance is still in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>10:15</var> MST</small><br><strong>Update</strong> - Scheduled Maintenance is still in progress. Thank you for your patience while we complete our work.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>09:00</var> MST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>15:11</var> MST</small><br><strong>Scheduled</strong> - Please note, the Splunk On-Call Platform will be undergoing scheduled maintenance for approximately one hour on Feb 18th from 9:00 to 10:00 am (Mountain time). <br /><br />*We expect zero interruptions to alert ingestion. No dropped alerts.<br />*We expect some platform disruption for up to an hour.<br />*We expect delays in paging/notifications of up to ten minutes.<br />*We expect that the Public API, SMS replies for Ack and Resolve, the Web Client, and Mobile Clients will be unavailable for up to ten minutes.<br /><br />To contact Splunk On-Call (formerly VictorOps) Support during this scheduled maintenance window, please review the linked Knowledge Base article: How to Contact Splunk On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/<br /><br />Thank you for your patience while we proceed with maintenance.</p>tag:status.victorops.com,2005:Incident/158452662023-01-17T15:07:21-07:002023-01-17T15:07:21-07:00On-Call Timeline is experiencing issues with ACK and RESOLVE actions<p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>15:07</var> MST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>14:50</var> MST</small><br><strong>Update</strong> - MS Teams Add-On Maintenance finished. Thanks for your patience while we completed this work</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>14:29</var> MST</small><br><strong>Update</strong> - Update: Related to the Timeline issues experienced over the weekend, the MS Teams integration is in temporary maintenance. Please bear with us while we work towards a permanent fix.</p><p><small>Jan <var data-var='date'>15</var>, <var data-var='time'>12:06</var> MST</small><br><strong>Monitoring</strong> - The MS Teams integration is back to operational status. We continue to monitor Timeline performance and are working towards a resolution. Please reach out to Support via in-product Chat, the Support Portal, and our hotline (1-855) SPLUNK-S or (1-855) 775-8657 if you are experiencing issues with On-Call Timeline performance. Updates to follow.</p><p><small>Jan <var data-var='date'>15</var>, <var data-var='time'>12:02</var> MST</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Jan <var data-var='date'>15</var>, <var data-var='time'>11:35</var> MST</small><br><strong>Identified</strong> - This issue has been identified. MS Teams is impacted. Please bear with us while we implement a fix.</p><p><small>Jan <var data-var='date'>14</var>, <var data-var='time'>16:04</var> MST</small><br><strong>Investigating</strong> - We are currently investigating disconnects and loading issues to the On-Call timeline that may affect the ability to ACK and RESOLVE active incidents.<br /><br />If you are experiencing any issues with incident actions in the Timeline, please reach out to On-Call Support via https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/<br /><br />Our Support hotline may be reached here: (1-855) SPLUNK-S or (1-855) 775-8657<br /><br />Thank you for your patience while we work this issue to ground. Updates to follow.</p>tag:status.victorops.com,2005:Incident/141099122022-12-10T11:45:26-07:002022-12-10T11:45:27-07:00Planned Maintenance: Identity Service (Okta SCIM) on Saturday, December 10th between 8:00 am to 11:00 am PST.<p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>11:45</var> MST</small><br><strong>Completed</strong> - The scheduled maintenance is complete! Thanks for patience while we finished up the work. On-Call Okta SCIM provisioning and de-provisioning has returned to fully operational.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>10:58</var> MST</small><br><strong>Update</strong> - Scheduled Maintenance is still in progress. We're extending the maintenance window by another hour, to 11:00 am PST. Thanks for your patience while we continue our work.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>09:00</var> MST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>09:00</var> MST</small><br><strong>Scheduled</strong> - UPDATED: Planned Maintenance: Identity Service on Saturday, December 10th between 8:00 am to 10:00 am PST (UTC -8). <br /><br />**This maintenance will only affect customers using Okta SCIM**<br /><br />The planned maintenance window will be 2 hours. During this time, SSO logins should be unaffected. However, add and subtract user operations will be impacted. <br /><br />During the maintenance window, Okta SCIM users will be unable to add new users or change passwords. <br /><br />All other functionality, such as authentication, alerting, incident response, etc. will remain operational. <br /><br />To contact Splunk On-Call (formerly VictorOps) Support during this scheduled maintenance window, please review the linked Knowledge Base article: How to Contact Splunk On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/</p>tag:status.victorops.com,2005:Incident/144358702022-12-06T08:30:04-07:002022-12-06T08:30:04-07:00Support Access via In-Product Live Chat<p><small>Dec <var data-var='date'> 6</var>, <var data-var='time'>08:30</var> MST</small><br><strong>Resolved</strong> - Thanks for your patience while we worked this issue to the ground. Chat has returned to fully operational status.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>13:15</var> MST</small><br><strong>Investigating</strong> - Apologies for this inconvenience! Currently, our in-product Live Chat feature is experiencing issues. We are investigating. <br /><br />For immediate Support, please use the Splunk Support Portal (https://login.splunk.com/) or the Call Center: https://www.splunk.com/en_us/about-splunk/contact-us.html#customer-support</p>tag:status.victorops.com,2005:Incident/113029962022-09-28T15:14:26-06:002022-09-28T15:14:26-06:00Elevated error rates and latencies for invokes on API Gateway endpoints in the US-WEST-2 Region.<p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>15:14</var> MDT</small><br><strong>Resolved</strong> - This incident has been resolved and the API service is operating normally. For further information on this incident please visit: For more information please visit: https://health.aws.amazon.com/health/status<br /><br />Apologies for this inconvenience.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>12:14</var> MDT</small><br><strong>Identified</strong> - Please be aware: we elevated error rates and latencies for invokes on API Gateway endpoints in the US-WEST-2 Region.<br />For more information please visit: https://health.aws.amazon.com/health/status<br /><br />Updates to follow.</p>tag:status.victorops.com,2005:Incident/111192232022-09-13T16:21:26-06:002022-09-13T16:21:26-06:00Phone Notifications to China (+86)<p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>16:21</var> MDT</small><br><strong>Resolved</strong> - This incident has been resolved. Apologies for the inconvenience.</p><p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>07:11</var> MDT</small><br><strong>Identified</strong> - We are actively researching a geographically isolated issue related to Phone Notification delivery challenges to the +86 (China) country code. <br /><br />We have escalated this issue to our telecommunications providers and will continue to stay actively engaged in all efforts to resolve this situation. Relevant updates will be provided to this Status Page as they arrive. <br /><br />As we continue to research this issue, we recommend that Splunk On-Call end-users located in China diversify their Splunk On-Call Personal Paging Policy to include Email, SMS, and Push notification delivery as alternatives to Phone.<br /><br />Our apologies for any inconvenience this situation may be causing you and your end-users. <br /><br />Updates to follow.</p>tag:status.victorops.com,2005:Incident/107871672022-08-29T08:59:21-06:002022-08-29T08:59:22-06:00Phone Notification Delivery Issues to +86 (China)<p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>08:59</var> MDT</small><br><strong>Resolved</strong> - RESOLVED: Successful phone call routing to carriers in China (country code +86) has resumed. <br /><br />The changes Splunk On-Call has implemented to avoid zero-second duration calls are outlined below and in our updated Knowledge Base article, found here: https://help.victorops.com/knowledge-base/mobile-app-getting-started/#phone-notifications-to-china-country-code-86<br /><br />IMPORTANT NOTE: In response to this situation, On-Call took technical measures to modify call handling for end-users within the +86/China Country Code. These measures were implemented in an effort to prevent/avoid telecommunications restrictions and zero-second duration calls associated with the +86/China Country Code:<br /><br />**After three attempts to notify via PHONE CALL to +86, we will cease any further attempts to contact the end user for a period of two hours. After two hours, we will make another three attempts, beginning the cycle again.<br />. . . <br /><br />We sincerely appreciate your continued patience as we worked to resolve this issue.</p><p><small>Aug <var data-var='date'>26</var>, <var data-var='time'>09:17</var> MDT</small><br><strong>Monitoring</strong> - Although we have not received formal notification back from our telecommunications provider on this front, we are currently reviewing our logs and seeing successful call delivery into the +86/China Country Code. <br /><br />IMPORTANT NOTE: In response to this situation, On-Call took technical measures to modify call handling for end-users within the +86/China Country Code. These measures were implemented in an effort to prevent/avoid telecommunications restrictions and zero-second duration calls associated with the +86/China Country Code:<br /><br />**After three attempts to notify via PHONE CALL to +86, we will cease any further attempts to contact the end user for a period of two hours. After two hours, we will make another three attempts, beginning the cycle again.<br />. . . <br /><br />We sincerely appreciate your continued patience as we (continue) to work this issue to ground.</p><p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>09:29</var> MDT</small><br><strong>Identified</strong> - Working in collaboration with our current telecommunications provider for Phone notification service into +86/China, we're taking steps this week to significantly reduce (or) eliminate "zero-duration" calls from our inbound +86/China call volume. In doing so, we believe we can improve the quality of our inbound call volume into +86/China in specific response to their stringent requirements around inbound Phone calls.<br /><br />NOTE: The above effort will require code development, testing, and production deployment coordination and is not a guarantee that the current Phone notification issues to end-users in the +86/China Country Code will be resolved. Nonetheless, we're working with urgency to have these updates deployed into production later this week.<br /><br />In the interim, we strongly encourage all Splunk On-Call end-users to diversify their Personal Paging policies to include PUSH, EMAIL, and SMS notification methods.<br /><br />We will continue to update our Status Page as relevant information arrives. We sincerely apologize for the ongoing challenges.</p><p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>16:05</var> MDT</small><br><strong>Update</strong> - Thank you for your patience while we continue to work with our telecommunication provider to investigate phone notification service disruption to the (+86) China region.<br /><br />As mentioned, this disruption is limited to Phone notifications to the (+86) region. We strongly encourage all On-Call end users in this region to diversify their personal paging policies to include Email, SMS and Push notification delivery as alternatives to Phone.<br /><br />Our apologies for any inconvenience this situation may be causing you.</p><p><small>Aug <var data-var='date'>12</var>, <var data-var='time'>15:37</var> MDT</small><br><strong>Update</strong> - As we continue to investigate the +86 (China) Phone notification issue with our telecommunications providers, we strongly encourage Splunk On-Call end-users to diversify their Personal Paging Policies to include Push, Email, and SMS as notification methods.</p><p><small>Aug <var data-var='date'> 8</var>, <var data-var='time'>21:46</var> MDT</small><br><strong>Investigating</strong> - We are actively researching a geographically isolated issue related to Phone Notification delivery challenges into the +86 (China) country code. <br /><br />We have escalated this issue to our telecommunications providers and will continue to stay actively engaged in all efforts to resolve this situation. Relevant updates will be provided to this Status Page as they arrive. <br /><br />As we continue to research this issue, we recommend that Splunk On-Call end-users located in China diversify their Splunk On-Call Personal Paging Policy to include Email, SMS, and Push notification delivery as alternatives to Phone.<br /><br />Our apologies for any inconvenience this situation may be causing you and your end-users. <br /><br />Updates to follow.</p>tag:status.victorops.com,2005:Incident/88557482021-12-15T09:45:11-07:002021-12-15T09:45:12-07:00AWS US-WEST-2 Region experiencing Internet connectivity issues<p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>09:45</var> MST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>09:27</var> MST</small><br><strong>Update</strong> - AWS has posted that they have resolved the issue affecting Internet connectivity to the US-WEST-1 Region. Connectivity within the region was not affected by this event. The issue has been resolved and the service is operating normally.</p><p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>09:12</var> MST</small><br><strong>Monitoring</strong> - AWS has updated their StatusPage to state that they have identified the root cause of the Internet connectivity to the US-WEST-1 Region and have taken steps to restore connectivity. They have seen some improvement to Internet connectivity in the last few minutes but will continue to work towards full recovery.</p><p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>09:01</var> MST</small><br><strong>Identified</strong> - The AWS US-WEST-2 Region is currently experiencing Internet connectivity issues and are investigating at the moment. These connectivity issues are impacting our services but we will continue to monitor their status and update ours accordingly.</p>tag:status.victorops.com,2005:Incident/84646712021-11-11T10:00:37-07:002021-11-11T10:00:37-07:00Courtesy Reminder: Splunk On-Call Support Changes on November 11th<p><small>Nov <var data-var='date'>11</var>, <var data-var='time'>10:00</var> MST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>20:05</var> MST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>19:22</var> MST</small><br><strong>Scheduled</strong> - This is a courtesy reminder that the Splunk On-Call (formerly VictorOps) contact page is moving to join the Splunk Support Portal. Please know that you’ll continue to receive the same world-class support from your Splunk On-Call Support Engineers. <br /><br />NOTE: This move will NOT affect your login and access to your Splunk On-Call (formerly VictorOps) instance.<br /><br />On November 11th, end-users can contact Splunk On-Call Support via the following methods:<br /><br />1. Live Chat: <br /><br />If you are logged into your Splunk On-Call instance, you will have the ability to Live Chat with the Splunk On-Call Support team.<br /><br />2. Splunk Support Portal: <br /><br />You can open a Splunk On-Call support case in the Splunk Support Portal: <br /><br />https://login.splunk.com/<br /><br />For more information surrounding this Support migration, please click on the link below:<br /><br />https://help.victorops.com/knowledge-base/important-splunk-on-call-support-changes-coming-nov-11th/</p>tag:status.victorops.com,2005:Incident/81316232021-10-01T14:09:45-06:002021-10-01T14:09:46-06:00Splunk On-Call Email Ingestion Failures<p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>14:09</var> MDT</small><br><strong>Resolved</strong> - We are moving our Splunk On-Call status page to a Resolved state, in relation to this issue. We'll continue to monitor this situation.<br /><br />If you have any immediate questions in reference to your Splunk On-Call instance, please don't hesitate to reach out to our Support team at spoc-support@splunk.com.<br /><br />We're here to help.<br /><br />Thanks for your patience in relation to this issue and apologies for any inconvenience.</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>13:05</var> MDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>12:47</var> MDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring results.</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>12:16</var> MDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is in progress</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>11:24</var> MDT</small><br><strong>Investigating</strong> - We’re currently investigating an issue around email ingestion.<br /><br />We will be providing additional updates on our status page.<br /><br />Apologies for this inconvenience! If you have any immediate questions, please reach out to the Splunk On-Call Support team:<br />spoc-support@splunk.com</p>tag:status.victorops.com,2005:Incident/73179512021-06-23T23:04:38-06:002021-06-23T23:04:38-06:00Splunk On-Call Reporting Data Temporarily Delayed - UI and API<p><small>Jun <var data-var='date'>23</var>, <var data-var='time'>23:04</var> MDT</small><br><strong>Resolved</strong> - This incident has been resolved. Reporting UI and Reporting API are operational. Thanks for your patience while we worked this issue to resolution.</p><p><small>Jun <var data-var='date'>23</var>, <var data-var='time'>23:00</var> MDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are now monitoring.</p><p><small>Jun <var data-var='date'>23</var>, <var data-var='time'>21:17</var> MDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented. Apologies for the inconvenience and thank you for your patience.</p><p><small>Jun <var data-var='date'>23</var>, <var data-var='time'>20:39</var> MDT</small><br><strong>Investigating</strong> - Alert ingestion, incident creation, and notifications are unaffected and operating normally. The availability of newly occurring data in our reporting user interfaces as well as our reporting API is currently delayed beyond the expected amount. Recent events may not be available for viewing in reports or accessible via API.<br /><br />More precise details will be shared in subsequent updates. We are actively working to resolve this issue as quickly as possible.<br /><br />Use the "Subscribe to Updates" option or follow @VOSupport on Twitter for updates. If you have any immediate questions or concerns, our support team is standing by to respond via email victorops-support@splunk.com or submit the form on our Contact Support page (https://victorops.com/contact-support/).</p>tag:status.victorops.com,2005:Incident/67344202021-04-15T10:02:48-06:002021-04-15T10:02:48-06:00Courtesy Notification: Cloudflare Updated IP Ranges<p><small>Apr <var data-var='date'>15</var>, <var data-var='time'>10:02</var> MDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>12</var>, <var data-var='time'>09:19</var> MDT</small><br><strong>Scheduled</strong> - Cloudflare has updated their IP ranges. If you need to add egress rules to send information into Splunk On-Call please update those rules to accommodate the updated Cloudflare IP ranges. The updates are as follows:<br />104.16.0.0/12 removed from ips-v4<br />104.16.0.0/13 added to ips-v4<br />104.24.0.0/14 added to ips-v4<br />The complete list of ranges can also be found here: https://www.cloudflare.com/ips/</p>tag:status.victorops.com,2005:Incident/65170292021-03-11T10:30:00-07:002021-03-12T13:00:52-07:00Splunk On-Call Service Disruption with Notifications<p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>10:30</var> MST</small><br><strong>Resolved</strong> - We’re currently monitoring an issue that has affected less than 1% of outbound notifications (all message delivery types: Phone, SMS, Push, and Email) during an isolated period of time. A fix is in progress and should be resolved soon.<br /><br />We will be providing additional updates on our status page upon the completion of our internal review.<br /><br />Apologies for this inconvenience! If you have any immediate questions, please reach out to the Splunk On-Call Support team:<br /><br />victorops-support@splunk.com</p>