Outage on Texas 1 Platform (paas1.tx.modxcloud.com)
»
View Event Details
| Created Tue, 22 Jan 2019 18:51:00 +0000
Post-Mortem
The Texas 1 platform is now fully operational. On Tuesday when datacenter engineers reviewed the health of the drives, they determined that they were intact but in a degraded condition and needed to be replaced.
Over the last 48 hours, 3 drives were replaced and integrated back into the full, redundant state.
In order for us to ensure sites stayed up for visitors while the rebuild of the drives was in progress, we suspended the automatic nightly backups. We are resuming automatic, nightly backups today.
At MODX Cloud, we understand how important uptime is and we take every measure to ensure sites stay online. With little exception, we exceed 99.99% uptime on our platforms. We use redundant disk arrays (RAID 1+0) to prevent data loss in case of drive failure. With all sites backed up nightly off-site, we also have a further fallback in case of catastrophic equipment failure.
For many customers downtime is frustrating and inconvenient, for others, it may mean significant financial or reputational losses. At MODX Cloud, we have been helping customers with high-availability needs for several years. In addition, we recently started to roll out an affordable, managed failover configuration for select customers. Contact help@modx.com for more information.
Posted: Thu, 24 Jan 2019 18:36:00 +0000
Update
Sites remain online and fully functional. Response times seem to be normal for all sites.
Datacenter engineers are in the process of replacing the degraded hardware. This operation should complete sometime in the next 4-6 hours.
Posted: Wed, 23 Jan 2019 15:33:00 +0000
Update
Sites are back online. We will continue monitoring platform health.
Posted: Tue, 22 Jan 2019 22:56:00 +0000
Update
Engineers are performing integrity checks on the platform hardware. We'll continue to keep you posted of changes.
Posted: Tue, 22 Jan 2019 20:30:00 +0000
Update
Engineers are reviewing the platform hardware. We'll keep you posted of changes.
Posted: Tue, 22 Jan 2019 20:29:00 +0000
Update
At this time we're investigating a possible hardware issue on the platform. We are working with our infrastructure partner engineers in determining the situation and will provide updates as we have them.
Posted: Tue, 22 Jan 2019 19:26:00 +0000
In Progress
Our monitors detected degraded performance on the Texas 1 Platform affecting Clouds that have Cloud addresses of paas1.tx.modxcloud.com.
We have suspended the web server in the process of reviewing the services to preserve the integrity of the system. We're actively working on restoring function and investigating the cause.
We'll provide updates as we have them.
Posted: Tue, 22 Jan 2019 18:51:00 +0000