System Status
Service Metrics
Below you will find the postmortem for the TeleSign incident alerted Thursday, May 30th, 2019.
Original Reported Subject: TeleSign API Intermittent Availability
Date: Thursday, May 30th, 2019
Start Time: 5:12 AM PDT
End Time: 5:37 AM PDT
Summary:
From 5:12 to 5:17 AM PDT on Thursday May 30th, a large Internet provider had route failures in the California area which impacted some customers' connections to TeleSign's Los Angeles datacenter. TeleSign's Los Angeles datacenter itself was functioning normally with no issues during this time. Once the Internet paths to the Los Angeles datacenter became stable, however, there was a spike in traffic volume which caused delays in SMS delivery.
Cause Analysis:
TeleSign experienced an issue with inbound and outbound connections in one datacenter. This issue affected some customers that were reaching us over problematic Internet paths. TeleSign's Internet service provider confirmed the issue was on their side and was due to a routing hardware failure on one of their core switches, causing multiple route failures in the California region.
The Internet provider confirmed they shut off access to the failed routing hardware and all circuits were restored by 6:10 AM PDT. By that time TeleSign traffic had already been moved away from the affected region. The ISP confirmed the failed routing hardware has subsequently been replaced.
As a result of the ISP connection issues described above, at 5:17 AM PDT TeleSign had an increase of traffic across all products. For SMS API and SMS Verify services, the increase of traffic caused a queue in one datacenter. TeleSign moved traffic out of the affected datacenter by 5:35 AM PDT to manage the queue. Customers may have received service unavailability or timeout responses for transactions processed between 5:17 AM to 5:37 AM. As of 5:37 AM PDT all traffic was being processed normally.
Preventive Measures:
To minimize risk of this issue recurring, TeleSign’s Network Operations and Development Teams will create additional Uptrends network level monitoring.
We apologize for the inconvenience this may have caused you. Should you have any questions, please don’t hesitate to contact us at support@telesign.com.
Further information will be provided here as it becomes available.
We apologize for any inconvenience, and should you have any questions or want to report particular issues related to this incident please contact support@telesign.com
Start Time: 05:17 AM PDT
End Time: 05:37 AM PDT
We have traced the issue to a particular data center and redirected all traffic from that data center to alternatives. All traffic is being processed normally from 05:37 AM. Right now we believe SMS services only were affected and we continue to investigate the cause of the issue and monitor active data centers.
We are currently experiencing a service disruption.
Our Network Engineers team is working to identify the root cause and implement a solution.
Once we have more information, we will make sure to update you.