We would like to inform you, about a significant incident that occurred last night. Due to a server outage, the EU Nobi cloud and app were not reachable. The incident exclusively affected EU Nobi cloud services, while other regions remained fully operational. As of 6:30 AM UTC, all servers are back online and fully operational.
Impact
- During the server outage, escalations were not reported to care staff via email or phone calls.
- Were unaffected: sites with an on-premise call system (Televic, Essec, Aperi, etc.).
Cause
A nightly maintenance job created an excessive number of connections to our database server, causing it to reach 400% of its normal capacity and resulting in an outage. This sudden increase in database connections is highly unusual and should not occur under normal circumstances. We are currently investigating the root cause to prevent this from happening again.
Future Preventative Measures
- In addition to our internal incident monitoring system, we will implement an external monitoring service to immediately flag any (potential) outages.
- We will deploy multiple database servers to balance the load and prevent single points of failure.
We apologize for any inconvenience this incident may have caused and appreciate your understanding and cooperation as we work to prevent future occurrences.
If you have any questions or require further details, please do not hesitate to contact our service desk.