Dear Customers,
On Saturday 17. August from 10:15 to 10:32 there was an unexpected power outage in our data center which was caused by a defect on the data center operator's side and which unfortunately touched most of our services. According to the data center operator's observations, the function of one of the UPS circuit breakers, which was not properly configured at the time of the work, occurred during planned electrical installation maintenance. This specific error has caused that, at other times, reliably functioning measures for continuous power supply (diesel aggregates, UPS, power supply using two independent branches) have failed. Our engineers started to work immediately after the restoration of the electricity supply, half of the services were in operation immediately, the commissioning of other services required manual intervention and therefore proceeded gradually. Shortly before 13:00, the operation was resumed from more than 95%, the operation was fully restored shortly after 15:00. Some parts of our platform did not boot properly after an unexpected hard reboot or did not boot up at all and needed to be repaired or replaced by a backup HW directly on site. For example, in the SAN infrastructure, the connection between the servers and the data store has suddenly been interrupted. After such a situation, there was a need for gradual manual intervention on the individual systems so that the services were hired without damaging the stored data. Before 16:00 we removed the complication in retrieving data into the customer center. The time evolution of the restoration of the operation of our services after this power outage in the data center documents the graph of our data traffic.
A sudden power outage also caused a hardware defect on our telephone panel and therefore we were forced to provide a makeshift dial-up connection to our support team. On Sunday, 18. 8. In 11:00 the panel was fully restored. Our services managed to stabilize technicians in a relatively short time and from Saturday afternoon they are again available in the usual quality. Although the cause of the outage was not an error on the ACTIVE 24 side, we do not take this experience lightly. We have thoroughly analysed the progress of the outage, its impact on customers and the possibilities of prevention in the future, and we have adopted a set of measures that would better protect our customers in similar situations. We immediately informed about the downtime on this site, which is intended for similar situations and which we have previously informed you. If the ACTIVE 24 standard communication channels are not available, we recommend that you monitor this website for the most up-to-date information. We publish more information in our Customer centre and on the Facebook page. Dear customers, once again we apologize for the inconvenience and thank you for your understanding.