This incident has been resolved. The incident was caused by a relative drop in performance of the application during a burst of callbacks, which in turn triggered an error/retries effect saturating its capacity.
Corrections have been carried out to fix weaknesses identified at the origin of the performance drop.
Additional capacity has been provisioned to add further margin against similar performance issue and against impact of escalating load due to a snowball effect.
Our alerting system has been further strengthened to accelerate our responsiveness.
Posted Nov 24, 2021 - 18:15 CET
Monitoring
All services are back to nominal operation. A fix has been implementing and we are monitoring the results.
Posted Nov 24, 2021 - 04:24 CET
Update
We continue to investigate this issue. At the same time, we have implemented initial fixes that are improving the callback success rate. Our teams are still working to resolve the incident as soon as possible.
Posted Nov 24, 2021 - 03:52 CET
Investigating
Please be advised there is an incident on Sigfox network Analysis of the root cause is still on-going and will be communicated to you as soon as possible.
Incident reference CORP-34512
Severity S1
Service Impact Delays on Uplink messages capture Uplink callbacks dispatch is partially down Downlink sequences is failing
Start Date and Time 2021-11-24 01:00 UTC
End Date and Time 2021-11-24 03:10 UTC
Root cause analysis The incident was caused by a relative drop in performance of the application during a burst of callbacks, which in turn triggered an error/retries effect saturating its capacity.