Texas 1 and 2 Network Issues

» View Event Details | Created Fri, 07 Jun 2019 01:11:00 +0000

Resolved

Network issues seem to have resolved and websites are now working as they should be. We will coordinate with our upstream provider on the cause of the issue and report the update once we have it.

Posted: Fri, 07 Jun 2019 02:37:00 +0000

Update

We have taken additional steps to restart the database server for Texas 2, and Cloud instances that were experiencing 502s are now working as they should be.

We're still investigating network issues related to Texas 1

Posted: Fri, 07 Jun 2019 01:58:00 +0000

Update

Issues seem to be specifically related to the internal and outbound network. This should not affect most sites, however, sites that reference external services to function with no timeout will be seeing 502 errors. Some other sites may see degraded performance. We hope to have further updates from the upstream engineers soon.

Posted: Fri, 07 Jun 2019 01:56:00 +0000

In Progress

Customers with sites on paas1.tx and paas2.tx may be experiencing 502 errors for sites at this time. The outage appears to be a result of a maintenance operation that was being performed in the data center by our upstream provider, IBM Cloud. There was not meant to be any disruption during this operation and this event was unexpected. Their engineers are working to resolve the issue and return network connectivity that caused MySQL to become disconnected. We will provide updates until resolved.

Posted: Fri, 07 Jun 2019 01:11:00 +0000

Major Service Disruption: All Services

» View Event Details | Created Fri, 07 Jun 2019 14:47:00 +0000

Post-Mortem

Reason For Outage Report (Post Mortem)

On Jun 7 at 9:49 AM Central, MODX Cloud experienced a systemwide outage.

The cause was determined to be the result of an error at our infrastructure vendor, which resulted in the public network ports being blocked.

Service began restoring at 10:19 AM. However, some servers and services required manual intervention to clear stuck processes and to restart NGINX and MySQL. By 11:16 AM, we had completed all manual checks and verifications for all public-facing infrastructure. The Dashboard control panel for viewing all platforms and instances was returned to service at 11:31 AM and the back end Agent platform for new instance provisioning and other automation was confirmed fully functional across all platforms at 12:30 PM.

While no data was lost, this service outage is tremendously frustrating and disappointing. We failed in our promise for service uptime. We sincerely apologize for this.

We are currently working with IBM to ensure that this can never occur again. If you have any further questions, please do not hesitate to contact us via the support tab in the MODX Cloud Dashboard.

Again, we sincerely apologize for this event and thank you for being MODX Cloud customers.

Posted: Mon, 10 Jun 2019 21:38:00 +0000

Resolved

At this time, all services are online and the MODX Cloud Dashboard is fully functional. We will have our full reason for outage later today.

Posted: Fri, 07 Jun 2019 16:31:00 +0000

Update

As we bring services back online the MODX Cloud Dashboard will not function fully until we complete our resolution of all sub-services.

Posted: Fri, 07 Jun 2019 15:56:00 +0000

Update

We have determined that private servers did not come online automatically. We're working now to restore access to those sites.

Posted: Fri, 07 Jun 2019 15:34:00 +0000

Update

It appears that sites are coming online more slowly than we hoped. Some sites are still experiencing 502 errors. We're working to resolve these issues.

Posted: Fri, 07 Jun 2019 15:23:00 +0000

Resolved

We have worked with IBM Cloud to resolve the outage and sites affected will be coming online at this time. We will post a complete reason for the outage statement in the next business day.

Posted: Fri, 07 Jun 2019 15:21:00 +0000

Update

We're in communication with our upstream provider to bring back service as soon as possible. More news soon.

Posted: Fri, 07 Jun 2019 14:59:00 +0000

In Progress

We are experiencing a major outage affecting all customers. We're working with our upstream provider to resolve this issue as quickly as possible.

Posted: Fri, 07 Jun 2019 14:47:00 +0000

Disruption to Dashboard

» View Event Details | Created Mon, 10 Jun 2019 20:56:00 +0000

Resolved

Dashboard logins are now operating correctly. We are sorry for the short disruption.

Posted: Mon, 10 Jun 2019 21:01:00 +0000

Investigating

We are currently aware of an issue with the MODX Cloud Dashboard logins. More information will be made available shortly.

Posted: Mon, 10 Jun 2019 20:56:00 +0000