» Published on Thu, 18 Jan 2018 00:00:00 +0000

• Post-Mortem

  The complete post mortem is available for review here: https://docs.google.com/document/d/1HOjyYoyIMKEEjHuKGLuCgf7zqV6lwxTXXtks-QZtTY8/edit?usp=sharing

  » Updated Wed, 24 Jan 2018 20:02:00 +0000

• Resolved

  All Clouds should now be restored to their last backups. Please open a ticket from the MODX Cloud Dashboard if you require assistance with anything related to the paas2.ams platform. We will release a postmortem on this incident early next week and be reaching out to all customers with instances on the paas2.ams platform.

  » Updated Sun, 21 Jan 2018 00:22:00 +0000

• Update

  At this time, production sites on this platform should be back online. If they are not, please contact support. We will continue restoring development clouds next.

  » Updated Sat, 20 Jan 2018 10:23:00 +0000

• Update

  At this point, partition copy is at 95%.

  » Updated Sat, 20 Jan 2018 09:52:00 +0000

• Update

  At this time, it's about 90% complete on the transfer.

  » Updated Sat, 20 Jan 2018 09:38:00 +0000

• Update

  We apologize for a lack of update. Due to a misconfiguration, we had to take the services offline to migrate from one partition to another. At this time, it's about 85% completed.

  » Updated Sat, 20 Jan 2018 09:16:00 +0000

• Update

  All production Clouds should now be restored; non-production Clouds without domains assigned are now being restored. We expect this to continue for the next ~18 hours. If you have any questions about sites that are not working as expected, please let us know by opening a support ticket in the MODX Cloud Dashboard. We will continue monitoring progress through the weekend.

  » Updated Sat, 20 Jan 2018 00:29:00 +0000

• Update

  Production sites with domains are being restored from the most recent backups to paas2.ams. We expect this to continue for the next few hours when sites without domains will start restoring. Please open a ticket from the Dashboard if you have any questions or require assistance.

  » Updated Fri, 19 Jan 2018 19:49:00 +0000

• Update

  Over the last 8 hours, IBM Cloud data center engineers performed multiple rounds of hardware replacement including all drives and a RAID controller (which is supposed to handle redundant local storage). Due to the severity of the failures, our team had to reinstall the operating system and configure it as new.

  The server is now back online and we have begun the process of restoring all customer sites from backups.

  Sites with custom domains will be restored first, followed by development Clouds (with no custom domains activated).

  We sincerely apologize for the unusual nature of the downtime. If you have any questions, please use the Help button from the lower right of the MODX Cloud dashboard to ask for assistance.

  » Updated Fri, 19 Jan 2018 16:49:00 +0000

• Update

  We have worked with data center technicians overnight to replace all affected hardware including all disks and the RAID controller. We are preparing the platform for recovering sites and site functionality.

  At this time we do not have an estimated time of completion.

  To get your site back online quickly, we recommend you create a new Cloud at another location such as London, Frankfurt or Amsterdam 1, and restore your most recent backup or Snapshot into the newly created Cloud. Then add any custom domains and point your A Record to direct traffic to the new location. Once that is done you can install a free SSL certificate if needed and copy over your original web rules.

  We sincerely apologize for the extended outage and will continue to update this incident as we have more information.

  » Updated Fri, 19 Jan 2018 11:39:00 +0000

• In Progress

  We continue to run disk recovery operations with IBM. Due to the nature of this process, we do not have an ETA at this point.

  To get your site back online quickly, we recommend you create a new Cloud at another location, and restore your most recent backup or Snapshot into the newly created Cloud. Then add any custom domains and point your A Record to direct traffic to the new location. Once that is done you can install a free SSL certificate if needed and copy over your original web rules.

  We sincerely apologize for this outage and will continue to update this incident as we have more information.

  We sincerely apologize for this outage and will continue to update this incident as we have more information.

  » Updated Thu, 18 Jan 2018 20:24:00 +0000

• Update

  We're still working to restore service to Amsterdam 2 Platform with IBM Cloud engineers. At this time we expect the outage to continue for some time as we are working to confirm the integrity of hardware and data and identify the cause.

  » Updated Thu, 18 Jan 2018 18:47:00 +0000

• Update

  We are working with our infrastructure partner, IBM Cloud, to restore service to our Amsterdam 2 Platform, as soon as possible.

  » Updated Thu, 18 Jan 2018 17:51:00 +0000

• Update

  We continue to work to bring the Amsterdam 2 Platform back online.

  » Updated Thu, 18 Jan 2018 17:31:00 +0000

• Investigating

  We're currently investigating the cause of an outage that's occurring on our Amsterdam 2 platform that will affect sites containing the Cloud URL of paas2.ams.modxcloud.com.

  We're working with our upstream partner, IBM Cloud to identify the source of the issue and recover normal operations.

  » Updated Thu, 18 Jan 2018 17:04:00 +0000