Resolved
All services at our Great Britain data center have been fully restored and with that, the impact of this outage to Infobip services across all our data centers has been successfully mitigated.
During the specified timeframe, customers attempting to reach services (Customer Portal, HTTP API, SMPP, SIP trunk, SMTP) for message processing hosted at any of our data centers globally could have experienced rejected requests, delayed traffic processing, and unexpected behavior from the Customer Portal and its modules (applications). Responses from Infobip Support (support@infobip.com) could have been answered with a delay.
The root cause of the outage happened in CrowdStrike’s defect found in a single content update for Windows hosts which affected our services supplied with the faulty version, however, it is important to note that this is not a security incident.
Start (UTC): 19 July 04:58
End (UTC): 19 July 16:00
Impact: Rejected/delayed traffic processing, unexpected errors from the Customer Portal/APIs
Posted Jul 19, 2024 - 16:24 UTC
Update
The service has been fully restored on Hong Kong data center.
We have restored most of our services and data centers. Our teams are finalizing the process of retrieving all data centers to functional state.
Posted Jul 19, 2024 - 13:56 UTC
Update
The service has been fully restored on the United Arab Emirates data center.
Posted Jul 19, 2024 - 13:38 UTC
Update
The service has been fully restored on Ukraine, Colombia, Sri Lanka and Bangladesh data centers.
Posted Jul 19, 2024 - 11:36 UTC
Update
We are continuing to work on a fix for this issue.
Posted Jul 19, 2024 - 11:11 UTC
Update
The service has been fully restored on Malaysia, EU Region-locked (Germany), South Africa and Vietnam data centers. The team is investing efforts to bring back the services to the rest of data centers.
Posted Jul 19, 2024 - 11:08 UTC
Update
The services have been fully restored on Indonesia, India, Pakistan, Singapore, Brazil, Turkey and Saudi Arabia data centers. The team is working on bringing back the services to the rest of data centers.
Posted Jul 19, 2024 - 10:50 UTC
Update
We are continuing to work on restoring the services on other data centers. We have all our hands on deck and this is currently top priority for us.
The issue was caused by CrowdStrike’s defect found in a single content update for Windows hosts.
It is important to note that this is not a security incident nor any kind of cyberattack.
Posted Jul 19, 2024 - 10:34 UTC
Identified
The services on Global (Germany) and United States data centers should be fully restored now. The team is focusing on restoring the services on other data centers as well. Updates are incoming soon.
Posted Jul 19, 2024 - 09:20 UTC
Update
Our teams continue working on restoring the traffic and services. More updates will follow.
Posted Jul 19, 2024 - 08:36 UTC
Update
The team has partially restored traffic on Global (Germany) and United States data centers and it should slowly recover. Other data centers are still in progress. Updates will follow.
Posted Jul 19, 2024 - 07:47 UTC
Update
Preliminary investigation results are indicating the issue is connected with the global incident of our vendor. Infobip services should still misbehave due to this incident. Also, response delays from our Customer Support (support@infobip.com) department are still expected.
Joint task force is continuously trying to restore all Infobip services. We will keep sharing the updates.
Posted Jul 19, 2024 - 07:21 UTC
Update
Latest findings show that you might experience issues with submitting traffic toward our platform, as well as errors and delays in processing the already submitted traffic across all channels.
Customer Portal and its modules might misbehave and present unexpected errors on attempts to utilize the services.
Delays are expected in responses from our Customer Support (support@infobip.com) department as well.
Our teams are working on pinpointing and mitigating the issue with the highest priority and more updates will follow.
Posted Jul 19, 2024 - 06:22 UTC
Investigating
We are currently observing issues with several of our services. The scope of the issue is currently under investigation and more information will follow in the update.
Posted Jul 19, 2024 - 05:11 UTC
This incident affected: Customer Portal Login/Homepage (Global (Germany), EU Region-locked (Germany), Great Britain, India, Indonesia, Pakistan, Saudi Arabia, Turkey, United Arab Emirates, United States), People (Global (Germany), EU Region-locked (Germany), Great Britain, India, Indonesia, Pakistan, Saudi Arabia, Turkey, United Arab Emirates, United States), Moments (Global (Germany), EU Region-locked (Germany), Great Britain, India, Indonesia, Pakistan, Saudi Arabia, Turkey, United Arab Emirates, United States), Analyze (Global (Germany), EU Region-locked (Germany), Great Britain, India, Indonesia, Pakistan, Saudi Arabia, Turkey, United Arab Emirates, United States), Conversations (Global (Germany), EU Region-locked (Germany), Great Britain, India, Indonesia, Pakistan, Saudi Arabia, Turkey, United Arab Emirates, United States), Answers (Global (Germany), EU Region-locked (Germany), Great Britain, India, Indonesia, Pakistan, Saudi Arabia, Turkey, United Arab Emirates, United States), Channels and Numbers (Global (Germany), EU Region-locked (Germany), Great Britain, India, Indonesia, Pakistan, Saudi Arabia, Turkey, United Arab Emirates, United States), Other Portal Apps (Global (Germany), EU Region-locked (Germany), Great Britain, India, Indonesia, Pakistan, Saudi Arabia, Turkey, United Arab Emirates, United States), SMS (Brazil, EU Region-locked (Germany), Global (Germany), Great Britain, Hong Kong, India, Indonesia, Malaysia, Pakistan, Saudi Arabia, Singapore, South Africa, Turkey, United Arab Emirates, United States, Vietnam), Number Lookup (Brazil, EU Region-locked (Germany), Global (Germany), Great Britain, Hong Kong, India, Indonesia, Malaysia, Pakistan, Saudi Arabia, Singapore, South Africa, Turkey, United Arab Emirates, United States, Vietnam), Voice (Bangladesh, Brazil, Colombia, EU Region-locked (Germany), Global (Germany), Hong Kong, India, Indonesia, Malaysia, Pakistan, Saudi Arabia, Singapore, South Africa, Sri Lanka, Turkey, Ukraine, United Arab Emirates, United States), Email (Brazil, EU Region-locked (Germany), Global (Germany), India, Indonesia, Pakistan, Singapore, United Arab Emirates, United States), RCS (EU Region-locked (Germany), Global (Germany), Great Britain, India, Indonesia, Pakistan, Saudi Arabia, Turkey, United Arab Emirates, United States), MMS (Global (Germany), Pakistan, United States), WhatsApp Business (Brazil, EU Region-locked (Germany), Global (Germany), Great Britain, Hong Kong, India, Indonesia, Malaysia, Pakistan, Saudi Arabia, Singapore, United Arab Emirates, United States), ChatApps (Brazil, EU Region-locked (Germany), Global (Germany), Great Britain, Hong Kong, India, Indonesia, Pakistan, Saudi Arabia, Turkey, United Arab Emirates, United States), Live Chat (EU Region-locked (Germany), Global (Germany), Great Britain, India, Indonesia, Pakistan, Saudi Arabia, Turkey, United Arab Emirates, United States), Mobile Push Notifications (Global (Germany), Great Britain, India, Indonesia, Pakistan, Saudi Arabia, Turkey, United Arab Emirates, United States), USSD (Global (Germany)), Mobile Identity (Brazil, Global (Germany), India), Integrations (Global (Germany), Pakistan), SMPP (Brazil, Global (Germany), Great Britain, Hong Kong, India, Indonesia, Malaysia, Pakistan, Saudi Arabia, Singapore, South Africa, Turkey, United Arab Emirates, United States, Vietnam), HTTP API (Brazil, EU Region-locked (Germany), Global (Germany), Great Britain, Hong Kong, India, Indonesia, Malaysia, Pakistan, Saudi Arabia, Singapore, South Africa, Turkey, United Arab Emirates, United States, Vietnam), SIP Trunk (Bangladesh, Brazil, Colombia, EU Region-locked (Germany), Global (Germany), Hong Kong, India, Indonesia, Malaysia, Pakistan, Saudi Arabia, Singapore, South Africa, Sri Lanka, Turkey, Ukraine, United Arab Emirates, United States), and SMTP (Brazil, EU Region-locked (Germany), Global (Germany), India, Indonesia, United Arab Emirates, United States).