System Status

This page offers API uptime information for TeleSign services, in addition to incident history reports and service issues alerts.
TeleSign API Intermittent Availability
Incident Report for TeleSign System Status
Postmortem

Below you will find the postmortem for the TeleSign incident alerted Thursday, May 30th, 2019.

Original Reported Subject: TeleSign API Intermittent Availability

Date: Thursday, May 30th, 2019

Start Time: 5:12 AM PDT

End Time: 5:37 AM PDT

Summary:

From 5:12 to 5:17 AM PDT on Thursday May 30th, a large Internet provider had route failures in the California area which impacted some customers' connections to TeleSign's Los Angeles datacenter. TeleSign's Los Angeles datacenter itself was functioning normally with no issues during this time. Once the Internet paths to the Los Angeles datacenter became stable, however, there was a spike in traffic volume which caused delays in SMS delivery.

Cause Analysis:

TeleSign experienced an issue with inbound and outbound connections in one datacenter. This issue affected some customers that were reaching us over problematic Internet paths. TeleSign's Internet service provider confirmed the issue was on their side and was due to a routing hardware failure on one of their core switches, causing multiple route failures in the California region.

The Internet provider confirmed they shut off access to the failed routing hardware and all circuits were restored by 6:10 AM PDT. By that time TeleSign traffic had already been moved away from the affected region. The ISP confirmed the failed routing hardware has subsequently been replaced.

As a result of the ISP connection issues described above, at 5:17 AM PDT TeleSign had an increase of traffic across all products. For SMS API and SMS Verify services, the increase of traffic caused a queue in one datacenter. TeleSign moved traffic out of the affected datacenter by 5:35 AM PDT to manage the queue. Customers may have received service unavailability or timeout responses for transactions processed between 5:17 AM to 5:37 AM. As of 5:37 AM PDT all traffic was being processed normally.

Preventive Measures:

To minimize risk of this issue recurring, TeleSign’s Network Operations and Development Teams will create additional Uptrends network level monitoring.

We apologize for the inconvenience this may have caused you. Should you have any questions, please don’t hesitate to contact us at support@telesign.com.

Posted 12 days ago. Jun 04, 2019 - 07:04 PDT

Resolved
As no further issues are apparent we are marking this issue as resolved. While SMS products were affected the issue may have been expressed as service unavailability or timeouts during the timeframes provided. As of 05:37 AM PDT all traffic is being processed normally.

Further information will be provided here as it becomes available.

We apologize for any inconvenience, and should you have any questions or want to report particular issues related to this incident please contact support@telesign.com
Posted 17 days ago. May 30, 2019 - 07:45 PDT
Update
We are continuing to monitor for any further issues.
Posted 17 days ago. May 30, 2019 - 06:11 PDT
Monitoring
Date: 30th May 2019
Start Time: 05:17 AM PDT
End Time: 05:37 AM PDT

We have traced the issue to a particular data center and redirected all traffic from that data center to alternatives. All traffic is being processed normally from 05:37 AM. Right now we believe SMS services only were affected and we continue to investigate the cause of the issue and monitor active data centers.
Posted 17 days ago. May 30, 2019 - 06:10 PDT
Investigating
Dear Customers and Partners,

We are currently experiencing a service disruption.

Our Network Engineers team is working to identify the root cause and implement a solution.

Once we have more information, we will make sure to update you.
Posted 17 days ago. May 30, 2019 - 05:39 PDT
This incident affected: Enterprise Services (SMS Verify, Voice Verify, Smart Verify, Push Verify, PhoneID Standard, PhoneID Contact, PhoneID Live, Score) and Standard Services (SMS, Voice, PhoneID, Score, App Verify).