Resolved -
All services at our Great Britain data center have been fully restored and with that, the impact of this outage to Infobip services across all our data centers has been successfully mitigated.
During the specified timeframe, customers attempting to reach services (Customer Portal, HTTP API, SMPP, SIP trunk, SMTP) for message processing hosted at any of our data centers globally could have experienced rejected requests, delayed traffic processing, and unexpected behavior from the Customer Portal and its modules (applications). Responses from Infobip Support (support@infobip.com) could have been answered with a delay.
The root cause of the outage happened in CrowdStrike’s defect found in a single content update for Windows hosts which affected our services supplied with the faulty version, however, it is important to note that this is not a security incident.
Start (UTC): 19 July 04:58
End (UTC): 19 July 16:00
Impact: Rejected/delayed traffic processing, unexpected errors from the Customer Portal/APIs
Jul 19, 16:24 UTC
Update -
The service has been fully restored on Hong Kong data center.
We have restored most of our services and data centers. Our teams are finalizing the process of retrieving all data centers to functional state.
Jul 19, 13:56 UTC
Update -
The service has been fully restored on the United Arab Emirates data center.
Jul 19, 13:38 UTC
Update -
The service has been fully restored on Ukraine, Colombia, Sri Lanka and Bangladesh data centers.
Jul 19, 11:36 UTC
Update -
We are continuing to work on a fix for this issue.
Jul 19, 11:11 UTC
Update -
The service has been fully restored on Malaysia, EU Region-locked (Germany), South Africa and Vietnam data centers. The team is investing efforts to bring back the services to the rest of data centers.
Jul 19, 11:08 UTC
Update -
The services have been fully restored on Indonesia, India, Pakistan, Singapore, Brazil, Turkey and Saudi Arabia data centers. The team is working on bringing back the services to the rest of data centers.
Jul 19, 10:50 UTC
Update -
We are continuing to work on restoring the services on other data centers. We have all our hands on deck and this is currently top priority for us.
The issue was caused by CrowdStrike’s defect found in a single content update for Windows hosts.
It is important to note that this is not a security incident nor any kind of cyberattack.
Jul 19, 10:34 UTC
Identified -
The services on Global (Germany) and United States data centers should be fully restored now. The team is focusing on restoring the services on other data centers as well. Updates are incoming soon.
Jul 19, 09:20 UTC
Update -
Our teams continue working on restoring the traffic and services. More updates will follow.
Jul 19, 08:36 UTC
Update -
The team has partially restored traffic on Global (Germany) and United States data centers and it should slowly recover. Other data centers are still in progress. Updates will follow.
Jul 19, 07:47 UTC
Update -
Preliminary investigation results are indicating the issue is connected with the global incident of our vendor. Infobip services should still misbehave due to this incident. Also, response delays from our Customer Support (support@infobip.com) department are still expected.
Joint task force is continuously trying to restore all Infobip services. We will keep sharing the updates.
Jul 19, 07:21 UTC
Update -
Latest findings show that you might experience issues with submitting traffic toward our platform, as well as errors and delays in processing the already submitted traffic across all channels.
Customer Portal and its modules might misbehave and present unexpected errors on attempts to utilize the services.
Delays are expected in responses from our Customer Support (support@infobip.com) department as well.
Our teams are working on pinpointing and mitigating the issue with the highest priority and more updates will follow.
Jul 19, 06:22 UTC
Investigating -
We are currently observing issues with several of our services. The scope of the issue is currently under investigation and more information will follow in the update.
Jul 19, 05:11 UTC