MODX Cloud

Infrastructure Uptime

Systems are currently experiencing a major outage.

Return to Statuspage

Event History

January 2018

Platform Outage for Amsterdam 2 (paas2.ams.modxcloud.com)

Post-Mortem - The complete post mortem is available for review here: https://docs.google.com/document/d/1HOjyYoyIMKEEjHuKGLuCgf7zqV6lwxTXXtks-QZtTY8/edit?usp=sharing
Posted: Wed, 24 Jan 2018 20:02:00 +0000

Resolved - All Clouds should now be restored to their last backups. Please open a ticket from the MODX Cloud Dashboard if you require assistance with anything related to the paas2.ams platform. We will release a postmortem on this incident early next week and be reaching out to all customers with instances on the paas2.ams platform.
Posted: Sun, 21 Jan 2018 00:22:00 +0000

Update - At this time, production sites on this platform should be back online. If they are not, please contact support. We will continue restoring development clouds next.
Posted: Sat, 20 Jan 2018 10:23:00 +0000

Update - At this point, partition copy is at 95%.
Posted: Sat, 20 Jan 2018 09:52:00 +0000

Update - At this time, it's about 90% complete on the transfer.
Posted: Sat, 20 Jan 2018 09:38:00 +0000

Update - We apologize for a lack of update. Due to a misconfiguration, we had to take the services offline to migrate from one partition to another. At this time, it's about 85% completed.
Posted: Sat, 20 Jan 2018 09:16:00 +0000

Update - All production Clouds should now be restored; non-production Clouds without domains assigned are now being restored. We expect this to continue for the next ~18 hours. If you have any questions about sites that are not working as expected, please let us know by opening a support ticket in the MODX Cloud Dashboard. We will continue monitoring progress through the weekend.
Posted: Sat, 20 Jan 2018 00:29:00 +0000

Update - Production sites with domains are being restored from the most recent backups to paas2.ams. We expect this to continue for the next few hours when sites without domains will start restoring. Please open a ticket from the Dashboard if you have any questions or require assistance.
Posted: Fri, 19 Jan 2018 19:49:00 +0000

Update - Over the last 8 hours, IBM Cloud data center engineers performed multiple rounds of hardware replacement including all drives and a RAID controller (which is supposed to handle redundant local storage). Due to the severity of the failures, our team had to reinstall the operating system and configure it as new. The server is now back online and we have begun the process of restoring all customer sites from backups. Sites with custom domains will be restored first, followed by development Clouds (with no custom domains activated). We sincerely apologize for the unusual nature of the downtime. If you have any questions, please use the Help button from the lower right of the MODX Cloud dashboard to ask for assistance.
Posted: Fri, 19 Jan 2018 16:49:00 +0000

Update - We have worked with data center technicians overnight to replace all affected hardware including all disks and the RAID controller. We are preparing the platform for recovering sites and site functionality. At this time we do not have an estimated time of completion. To get your site back online quickly, we recommend you create a new Cloud at another location such as London, Frankfurt or Amsterdam 1, and restore your most recent backup or Snapshot into the newly created Cloud. Then add any custom domains and point your A Record to direct traffic to the new location. Once that is done you can install a free SSL certificate if needed and copy over your original web rules. We sincerely apologize for the extended outage and will continue to update this incident as we have more information.
Posted: Fri, 19 Jan 2018 11:39:00 +0000

In Progress - We continue to run disk recovery operations with IBM. Due to the nature of this process, we do not have an ETA at this point. To get your site back online quickly, we recommend you create a new Cloud at another location, and restore your most recent backup or Snapshot into the newly created Cloud. Then add any custom domains and point your A Record to direct traffic to the new location. Once that is done you can install a free SSL certificate if needed and copy over your original web rules. We sincerely apologize for this outage and will continue to update this incident as we have more information. We sincerely apologize for this outage and will continue to update this incident as we have more information.
Posted: Thu, 18 Jan 2018 20:24:00 +0000

Update - We're still working to restore service to Amsterdam 2 Platform with IBM Cloud engineers. At this time we expect the outage to continue for some time as we are working to confirm the integrity of hardware and data and identify the cause.
Posted: Thu, 18 Jan 2018 18:47:00 +0000

Update - We are working with our infrastructure partner, IBM Cloud, to restore service to our Amsterdam 2 Platform, as soon as possible.
Posted: Thu, 18 Jan 2018 17:51:00 +0000

Update - We continue to work to bring the Amsterdam 2 Platform back online.
Posted: Thu, 18 Jan 2018 17:31:00 +0000

Investigating - We're currently investigating the cause of an outage that's occurring on our Amsterdam 2 platform that will affect sites containing the Cloud URL of paas2.ams.modxcloud.com. We're working with our upstream partner, IBM Cloud to identify the source of the issue and recover normal operations.
Posted: Thu, 18 Jan 2018 17:04:00 +0000

Subscribe to Updates