Post-Mortem

At 9:17 AM Central on August 25, 2022, we began an investigation of the service outage on the London data center after our monitoring tools triggered an alert. An incident was posted at 9:36 AM to accompany the service outage (at https://status.modxcloud.com).

We ultimately determined that the MySQL service was deadlocked. We shut down the service, and a consistency scan was started to ensure data integrity.

Service was fully restored at 9:57 AM with no data loss.

The rare bug that triggered this deadlock does not affect the stack underpinning our new Gen2 platforms, which use a different version of MySQL (MariaDB). Unfortunately, this fix has not been ported to the version we use on our legacy platforms, including London.

There are new Gen2 platforms available in the US and EU. We will add a UK location by the end of 2022.

Migrations to a new data center are straightforward, happen without downtime, and can be triggered at any time from the MODX Cloud Dashboard (on the Cloud view page). Click the list of Clouds, then click the name of the Cloud you'd like to migrate. The right actions column has a “Migrate… (new)” menu option at the top of the list. Click that to initiate a migration.

This will immediately trigger a clone of your site to the new data center, which typically only takes a few minutes to complete. While all upgrades and configurations are cloned, cron tasks do require manual migration at this time. You can then review the site for functionality in its new location, ask for help with anything that’s not working as expected, or let the Cloud support team know you’re ready to proceed.

Migrations between Gen2 platforms will be even more automated and faster. If you would like to temporarily move from London to the EU Gen2 platform, moving back to a new Gen2 London platform will be just a button press and DNS update away (or just a button press-only if you use our WAF and/or CDN).

We apologize for the outage in London and know how important uptime is for all websites. Our goal is always to ensure reliable service, and unfortunately, we missed the mark with this incident. Should you have any questions or concerns, please open a ticket in the MODX Cloud Dashboard.

Posted: Mon, 29 Aug 2022 23:23:00 +0000

Resolved

Service has been restored. A post mortem will be posted soon.

Posted: Thu, 25 Aug 2022 14:53:00 +0000

Update

We are in the process of restoring service to the London data center

Posted: Thu, 25 Aug 2022 14:51:00 +0000