At 17:30, the loading time of our app increased significantly, and some merchants couldn't access this. We quickly found the root cause but it took our system much longer than expected to recuperate with the busy load of a Friday night. At 20:30, our systems fully recovered. The root cause: At 6:00 on Friday the 6th, some of our team performed a data migration that had a critical human error. With little system activity at the start of the day, we did not notice the heavy load this brought. It was only until later in the afternoon when business became busy that we noticed the strain on the Tebi system.
This was a rare domino effect caused by errors, communication, and system capacity. We are now dedicating our teams to put in safeguards to avoid this in the future. |
