» Published on Tue, 22 Jan 2019 18:51:00 +0000

• Post-Mortem

  The Texas 1 platform is now fully operational. On Tuesday when datacenter engineers reviewed the health of the drives, they determined that they were intact but in a degraded condition and needed to be replaced.

  Over the last 48 hours, 3 drives were replaced and integrated back into the full, redundant state.

  In order for us to ensure sites stayed up for visitors while the rebuild of the drives was in progress, we suspended the automatic nightly backups. We are resuming automatic, nightly backups today.

  At MODX Cloud, we understand how important uptime is and we take every measure to ensure sites stay online. With little exception, we exceed 99.99% uptime on our platforms. We use redundant disk arrays (RAID 1+0) to prevent data loss in case of drive failure. With all sites backed up nightly off-site, we also have a further fallback in case of catastrophic equipment failure.

  For many customers downtime is frustrating and inconvenient, for others, it may mean significant financial or reputational losses. At MODX Cloud, we have been helping customers with high-availability needs for several years. In addition, we recently started to roll out an affordable, managed failover configuration for select customers. Contact help@modx.com for more information.

  » Updated Thu, 24 Jan 2019 18:36:00 +0000

• Update

  Sites remain online and fully functional. Response times seem to be normal for all sites.

  Datacenter engineers are in the process of replacing the degraded hardware. This operation should complete sometime in the next 4-6 hours.

  » Updated Wed, 23 Jan 2019 15:33:00 +0000

• Update

  Sites are back online. We will continue monitoring platform health.

  » Updated Tue, 22 Jan 2019 22:56:00 +0000

• Update

  Engineers are performing integrity checks on the platform hardware. We'll continue to keep you posted of changes.

  » Updated Tue, 22 Jan 2019 20:30:00 +0000

• Update

  Engineers are reviewing the platform hardware. We'll keep you posted of changes.

  » Updated Tue, 22 Jan 2019 20:29:00 +0000

• Update

  At this time we're investigating a possible hardware issue on the platform. We are working with our infrastructure partner engineers in determining the situation and will provide updates as we have them.

  » Updated Tue, 22 Jan 2019 19:26:00 +0000

• In Progress

  Our monitors detected degraded performance on the Texas 1 Platform affecting Clouds that have Cloud addresses of paas1.tx.modxcloud.com.

  We have suspended the web server in the process of reviewing the services to preserve the integrity of the system. We're actively working on restoring function and investigating the cause.

  We'll provide updates as we have them.

  » Updated Tue, 22 Jan 2019 18:51:00 +0000