» Published on Fri, 07 Jun 2019 14:47:00 +0000

• Post-Mortem

  Reason For Outage Report (Post Mortem)

  On Jun 7 at 9:49 AM Central, MODX Cloud experienced a systemwide outage.

  The cause was determined to be the result of an error at our infrastructure vendor, which resulted in the public network ports being blocked.

  Service began restoring at 10:19 AM. However, some servers and services required manual intervention to clear stuck processes and to restart NGINX and MySQL. By 11:16 AM, we had completed all manual checks and verifications for all public-facing infrastructure. The Dashboard control panel for viewing all platforms and instances was returned to service at 11:31 AM and the back end Agent platform for new instance provisioning and other automation was confirmed fully functional across all platforms at 12:30 PM.

  While no data was lost, this service outage is tremendously frustrating and disappointing. We failed in our promise for service uptime. We sincerely apologize for this.

  We are currently working with IBM to ensure that this can never occur again. If you have any further questions, please do not hesitate to contact us via the support tab in the MODX Cloud Dashboard.

  Again, we sincerely apologize for this event and thank you for being MODX Cloud customers.

  » Updated Mon, 10 Jun 2019 21:38:00 +0000

• Resolved

  At this time, all services are online and the MODX Cloud Dashboard is fully functional. We will have our full reason for outage later today.

  » Updated Fri, 07 Jun 2019 16:31:00 +0000

• Update

  As we bring services back online the MODX Cloud Dashboard will not function fully until we complete our resolution of all sub-services.

  » Updated Fri, 07 Jun 2019 15:56:00 +0000

• Update

  We have determined that private servers did not come online automatically. We're working now to restore access to those sites.

  » Updated Fri, 07 Jun 2019 15:34:00 +0000

• Update

  It appears that sites are coming online more slowly than we hoped. Some sites are still experiencing 502 errors. We're working to resolve these issues.

  » Updated Fri, 07 Jun 2019 15:23:00 +0000

• Resolved

  We have worked with IBM Cloud to resolve the outage and sites affected will be coming online at this time. We will post a complete reason for the outage statement in the next business day.

  » Updated Fri, 07 Jun 2019 15:21:00 +0000

• Update

  We're in communication with our upstream provider to bring back service as soon as possible. More news soon.

  » Updated Fri, 07 Jun 2019 14:59:00 +0000

• In Progress

  We are experiencing a major outage affecting all customers. We're working with our upstream provider to resolve this issue as quickly as possible.

  » Updated Fri, 07 Jun 2019 14:47:00 +0000