We're currently having issues

Resolved
Resolved

We've now resolved the incident. Thank you for your patience.

All services are available and all metrics are green. We are double-checking the nightly loads and will then progress to close the incident.

Our current status is all services are available. We are still monitoring closely.
We see still a small backlog of Data Jobs being processed by the platform. This may delay single Data Jobs for a little bit more than usual.

Monitoring is continuing. All services are looking healthy.

All services are looking good. We'll keep monitoring for some time more.

We keep on monitoring the services. All services are looking good.

The services are fully available again. We are closely monitoring the infrastructure.

We are currently bringing the remaining frontend services back into their target state. Therefore the frontend may show slower performance or some requests failing.

The system is fully available. We are closely observing the processing of the backlog for the next hours.

The system is fully available and working through the backlog. We are closely monitoring and will be back soon with another update.

Recovering

Based on our monitoring, error rates and latency have now recovered. We will continue to observe the overall platform health and keep you updated.

Avatar for

We are still investigating closely all the issues and share another status update soon.

The performance of the application may still be degraded for some services. We are continuing our investigation of the issues.

Investigating

We are again seeing increased failure rates in one of the service. We will share another update once we have more details

We have adjusted database pool size parameters and are continuing monitoring. We are still seeing occasional long-running requests which we are investigating.

Avatar for

We have successfully scaled the components and are continuing to monitor the application metrics. We will provide another update in roughly one hour.

Avatar for
Recovering

We are currently scaling up various components throughout the platform. Some customers might experience slightly elevated error rates during this operation.

Avatar for

We have rolled out changes to the database connectivity layer for multiple core services. We will monitor whether this has the desired effect and will update you as soon as we have results.

Avatar for

We are continuing our investigation of the issues.

Avatar for

We have just restored the Process Analytics service. We are in contact with the cloud provider to further investigate the root cause of the database issues we encountered today. We will continue to update you as we make progress.

Avatar for

We don't have any further updates to share so far. We are still working on Process Analytics.

Action Engine is available again. We are still working on restoring Process Analytics.

We have resolved the overall performance issues and are continuing work on Process Analytics and Action Engine.

Avatar for

We are performing emergency maintenance on the Process Analytics and Action Engine services and have therefore taken them offline temporarily. We will provide another update in about 30 minutes.

Avatar for
Identified

We are still working on resolving the issues. We will provide another update in 30 minutes.

Avatar for
Recovering

We have applied another mitigation and are closely monitoring recovery.

Avatar for

We are still investigating the root cause of the issues.

Avatar for
Investigating

Unfortunately, the issues have resurfaced and we are continuing our investigation

Avatar for
Recovering

We have mitigated the underlying issue and are waiting for everything to fully recover.

Avatar for
Identified

We have identified a potential root cause and are applying mitigations to improve the performance and reduce the error rate.

Avatar for
Investigating

We are currently investigating degraded performance on the EU-3 realm.

Avatar for
Began at:

Affected components
  • EU-3