System Status
Service Metrics
Below you will find the Root Cause Analysis for the incident that occurred on Tuesday, October 15th, 2019.
Original Reported Subject: Delivery Delay
Date: Tuesday, October 15th, 2019
Start Time: 2:01AM PDT
End Time: 2:14 AM PDT
Summary:
On Tuesday, October 15th TeleSign’s internal monitoring detected delayed delivery for all SMS traffic starting at 2:01 AM PDT. TeleSign’s SMS Verify and SMS API were available throughout this reported incident, however SMS processing time was delayed.
Root Cause Analysis:
A load balancer failover at 2:01 AM PDT in one data center caused multiple virtual IPs hosted on the load balancer to become unavailable. SMS transactions then started to queue until the load balancer stabilized at 2:12 AM PDT. At 2:14 AM PDT TeleSign’s network team confirmed all queues were cleared and no further delays were observed.
Preventive Measures:
To minimize the risk of, and/or prevent this issue from recurring in the future, TeleSign’s Tech OPS team has taken the following actions:
· Vendor confirmed presence of bug and worked with Telesign on recommended workaround
· Additionally, a configuration audit regarding failovers in TeleSign’s data centers was also performed.
We apologize for the inconvenience this may have caused you. Should you have any questions, please don’t hesitate to contact us at support@telesign.com.
Previously reported delivery delay has now been restored as of 2:10 AM PDT.
We are currently reviewing root cause and will provide additional information once we have completed the investigation.
Our internal monitoring system shows intermittent delivery delays on ALL traffic starting at 2 AM PDT
Our Network Operations team is currently investigating and working on a speedy resolution.
We will inform you once the issue has been remedied.