All systems are stable since more than 24hrs now and there are no indicators that the core problem might reoccur. We've already started replaying data from our backup systems for the messages that got lost during the outages of the broker. This is ongoing and we will keep you updated once this task is finished. Estimated timeline is by beginning of next week. Reach out to us if you got further questions or remarks: consulting@azeti.net
A comprehensive post mortem will be added in the next weeks, this is ongoing and requires additional time and resources.
Kind regards azeti Engineering Team
Posted Sep 18, 2019 - 20:51 CEST
Update
As, we are doing a data recovery, that might delay for the regular data may occur.
Posted Sep 17, 2019 - 12:01 CEST
Monitoring
Our MQTT Broker is operational since 18:10 CET and stable. Still it is under intense monitoring and we're providing further details after an incubation time of at least 48 hrs.
Posted Sep 16, 2019 - 18:55 CEST
Update
Messages are enqueued again since 13:00 for most sites but still some performance delays are to be expected. Investigation is ongoing. Thanks for your understaning.
Posted Sep 16, 2019 - 15:00 CEST
Update
We are continuing the investigation, unfortunately the incoming MQTT connections are not enqueued properly at the moment and are therefore not inserted into the database. You might see "gaps" inside your graphs for certain sites. Your local SiteController will cache the data until our ActiveMQ broker is online again and we are working with all our resources on resolving the issue. Thanks for you understanding. Feel free to reach out to us via consulting@azeti.net
Posted Sep 16, 2019 - 11:44 CEST
Investigating
We found some, abnormalities in our cloud today morning. So, we have immediately try to give a quick fix and we will be further investigating it.
Posted Sep 16, 2019 - 08:51 CEST
This incident affected: Backend, MQTT Broker, and APIs.