February 2017

Upgrading SSL infrastructure
The maintenance is completed.
Feb 24, 13:31-13:38 CST

January 2017

Help Desk and User Guide issues at provider (ZenDesk)
Zendesk seems to be returning to normal. Our support tools seem to be working as expected. The user guide is back up and running normally as well.
Jan 12, 07:21-08:34 CST

December 2016

paas1.ams.modxcloud.com Outage
At this point in time paas1.ams.modxcloud.com has been fully restored. If you notice anything off, try clearing your MODX cache via ssh by removing /www/core/cache/*. If you issues persist, please contact us at support@modxcloud.com.
Dec 25, 20:39-21:58 CST

November 2016

[Scheduled] SSL Certificate Service Interruption
The scheduled maintenance has been completed.
Nov 20, 01:30-03:00 CST
[Scheduled] Security Upgrade Amsterdam Datacenter Affecting Dashboard Operations
This maintenance has been completed.
Nov 19, 17:00-17:57 CST

October 2016

Support Desk Outage
Service has been restored to the help desk. We'll continue monitoring the situation as a precaution.
Oct 21, 12:10-17:11 CST
Support Desk Outage
It appears that Zendesk has resolved the outage which affected our Support Desk for MODX and MODX Cloud.
Oct 13, 13:57-14:20 CST

September 2016

[Scheduled] [MODX Cloud] Scheduled Maintenance: US Public Platforms with Downtime
Work on paas2.tx is complete. All Texas platforms have been completed. If you notice anything out of place, please email support@modxcloud.com. Thanks!
Sep 16, 22:39-22:44 CST
[Scheduled] [MODX Cloud] Scheduled Maintenance: Europe Public Platforms with Downtime
The scheduled maintenance has been completed.
Sep 16, 15:12-15:12 CST
Minor DNS issue at upstream provider affecting some MODX Cloud Web Addresses
Our upstream provider has resolved the issue and we can confirm that web addresses are now being added as expected. The web addresses for clouds created since yesterday have been corrected. Continuity of service is very important to us, and we appreciate your patience during this event.
Sep 14, 14:06-20:21 CST
Cloud Creation, New Domains and Web Rules Updates Issues on Amsterdam 1 and 2
We have determined this issue has been corrected.
Sep 14, 08:35-13:44 CST

August 2016

Texas 1 Platform (paas1.tx) not Responding
Everything is working normally at this time. We'll continue to monitor the server's health and ensure it runs as expected. If you're experiencing any errors with your site(s), please submit a support ticket from the MODX Cloud Dashboard. We understand the critical nature of uptime for all MODX Cloud customers and we sincerely apologize for the inconvenience. Thanks for choosing MODX Cloud.
Aug 28, 04:14-11:01 CST
Amsterdam Platform 1 Outage
At this time, service has been restored to this platform.
Aug 7, 23:20-23:46 CST
Emergency hard drive replacement on Amsterdam 2 (paas2.ams)
RAID rebuild was successfully completed without data loss over night.
Aug 3, 20:47 - Aug 4, 08:06 CST

July 2016

[Scheduled] MODX Cloud Dashboard patch and restart
The scheduled maintenance has been completed.
Jul 25, 02:46-03:31 CST
[Scheduled] Amsterdam Data Center Security Patch and Restart
The scheduled maintenance has been completed.
Jul 23, 17:01-19:01 CST

June 2016

Possible Inbound Support Ticket Email Degradation
A combination of overzealous anti-spam measures and mistyped emails proved to be the cause of the issue.
Jun 17, 12:17-17:22 CST

May 2016

No incidents reported for this month.

April 2016

Downtime in paas2.tx
The platform has been confirmed stable and fully operational. We apologize for any inconvenience this outage has caused. If you have questions or require assistance, please do not hesitate to contact support@modxcloud.com
Apr 20, 11:38-14:40 CST
[Scheduled] MODX Cloud Dashboard April 7 2016
Dashboard Database maintenance has completed without a hitch in 10 minutes. If you're noticing anything out of the ordinary, please contact support@modxcloud.com. Thanks!
Apr 8, 18:00-18:11 CST

March 2016

Maintenance for paas2.ams.modxcloud.com
Maintenance has been completed.
Mar 3, 15:37-15:58 CST

February 2016

No incidents reported for this month.

January 2016

[Scheduled] Emergency Network Maintenance Affecting Dashboard Operations for Amsterdam (AMS) Based Clouds
The scheduled maintenance has been completed.
Jan 30, 17:00-19:28 CST
Issue creating or deleting Clouds for some customers
We have resolved the problem with various aspects of MODX Cloud dashboard functions. We apologize for the inconvenience and appreciate your patience.
Jan 12, 13:30-23:31 CST

December 2015

Zendesk Support Outage
Zendesk service is back in action and we're working tickets again.
Dec 23, 09:00-15:10 CST
Outage on paas2.tx server
We have restored functionality to the Texas 2 platform in MODX Cloud. Sites should be responding normally now. We're sincerely sorry that this happened. We will provide a complete post mortem later today.
Dec 17, 09:56-11:35 CST
Degraded Performance of Queued Operations (Snapshots, Cloud create, etc.)
We have restored functionality to the queued operations on all platforms. There was a maintenance operation overnight that restarted the machines, however, manual intervention was required to bring them back online fully. No Clouds or sites were affected by this issue.
Dec 13, 07:26-09:03 CST
Partial Outage on Amsterdam 2 Affecting Small Number of Customers
At approximately 4:00am UTC -6 (10:00 UTC 0) December 9, 2015, a small number of customers with sites on the Amsterdam 2 platform (paas2.ams.modxcloud.com) were impacted by 502 errors, rendering their sites unavailable for some visitors. We want to sincerely apologize for this and fully understand the importance of site uptime. We determined the cause to be a large traffic spike which overloaded the database. For some sites that rely on fully dynamic content (uncached Snippet calls and extensive database queries), some of the queries executed slowly and filled up their site's worker pools. To resolve this, we had to manually reset the worker pools. Due to the nature of how this event occurred, and it not affecting the vast majority of the platform, our monitoring systems did not alert on this event, which we do not consider acceptable. Moving forward, we pledge to do better and to improve our monitoring and alerting to ensure this doesn't occur again: 1. Add more MySQL related "change alerts" to our monitoring system to catch this type of event. 2. Research modifying the behavior of UNIX Sockets to make them more resilient against "queue" limits in the pool. 3. Give user the ability to reset their PHP worker pool manually from the MODX Cloud Dashboard without needing support intervention. 4. Research additional emergency triage alerts based on the frequency of ticket keywords such as "site down", "outage", "502" as a preventative measure for other potential issues that are currently not monitored adequately. We are confident this plan of action will further reduce the risk of downtime and significantly reduce the duration. Again, we are extremely sorry for any stress we created for you and your clients. If you have any specific questions or would like further information, please email us at support@modxcloud.com.
Dec 9, 10:34 CST

November 2015

PHP on Amsterdam 2
We have restored service to all affected sites on Amsterdam 2 Platform (paas2.ams) and are continuing our analysis for cause.
Nov 20, 13:13-13:35 CST

October 2015

Multiple server reboots to address Xen vulnerability.
A clarification on an earlier update: Frankfurt does not need rebooting, this therefore completes our reboots due to the undisclosed Xen vulnerabilities.
Oct 24, 11:23-12:09 CST

September 2015

Outage on paas1.tx.modxcloud.com
The issue at hand is resolved.
Sep 16, 03:01-05:16 CST
New Snapshots not being created
New Snapshots are working. We apologize for the inconvenience.
Sep 6, 10:23 - Sep 7, 10:12 CST
Dashboard Outage
This issue has been resolved.
Sep 1, 09:06-10:07 CST

August 2015

Outage for pass1.tx.modxcloud.com
The issue has been resolved, further monitoring will be employed to further help SoftLayer if more issues occur.
Aug 20, 11:44-13:38 CST

July 2015

[Scheduled] Platform Upgrade to PHP 5.6
All servers are now running PHP 5.6.11. If you've noticed something acting odd on your site, open a support ticket by emailing support@modxcloud.com. Thanks!
Jul 20, 20:00-20:57 CST
Partial Outage Texas 1
We have determined the cause of the outage and are ensuring all sites are running as expected. If your site hosting on the paas1.tx.modxcloud.com server is not running as expected, please contact support via the Help button at the right of the MODX Cloud Dashboard.
Jul 13, 12:04-12:25 CST
[Scheduled] Amsterdam Network Maintenance with Downtime (AMS1 & AMS2)
The scheduled maintenance has been completed.
Jul 1, 17:00-22:00 CST

June 2015

No incidents reported for this month.

May 2015

No incidents reported for this month.

April 2015

No incidents reported for this month.

March 2015

Dashboard Limited Functionality in Amsterdam
As of this moment, all caches have been updated and functionality is restored.
Mar 23, 11:07-12:37 CST
Emergency server reboot: paas1.tx
We have resolved this issue and will publish a post mortem shortly. Thanks for your patience and thanks for using MODX Cloud.
Mar 19, 18:47 - Mar 20, 08:28 CST
[Scheduled] Dallas Data Center Security Patch and Restart
The scheduled maintenance has been completed.
Mar 8, 00:01-04:24 CST
[Scheduled] Amsterdam Data Center Security Patch and Restart
Maintenance has been completed in Amsterdam.
Mar 4, 17:01-22:39 CST

February 2015

Errors on Texas 2 (paas2.tx)
We were alerted to errors causing php to fail on specific sites on Dallas 2. We restarted PHP and all sites are now back online. We will continue our investigation until permanent resolution to this issue has been found. If you have any questions or concerns please contact support [at] modxcloud [dot] com.
Feb 25, 06:36 CST
Investigating timeouts and errors on some sites on paas2.tx (Texas 2)
Due to the isolated nature of this event we are not able to determine the root cause at this time. We believe that several sites with high MySQL demand failed during the start of automatic backup cycle. This initially returned 504 errors to those sites as PHP was queuing process, and eventually, 502 errors as PHP stopped taking requests for the affected sites. We will continue to monitor the affected sites (along with the platform as a whole) and further isolate and resolve quickly if this event recurs.
Feb 23, 07:35-10:56 CST
Degraded Performance of Queued Operations (Snapshots, Cloud create, etc.)
We have established that the fixes we put in place have completely resolved the issues with queued operations in MODX Cloud. We sincerely appreciate your patience with this.
Feb 3, 12:42-14:01 CST

January 2015

Major glibc vulnerability
All servers are patched and affected services have been restarted.
Jan 27, 12:56 - Jan 28, 13:11 CST
Emergency Maintenance for paas2.ams.modxcloud.com
The situation is now resolved, and the new hard drives are operating at optimal levels.
Jan 18, 22:39 - Jan 19, 11:34 CST

December 2014

No incidents reported for this month.

November 2014

paas1.ams server outage
The AMS 1 server has been stable since rebooting 24 hours ago. We have received multiple confirmations from data center engineers that the power supply for this server is 'all green', meaning that in their estimation, there is no need to replace the hardware in question.
Nov 29, 20:41 - Nov 30, 20:50 CST
[Scheduled] System Upgrades at All Datacenters (AMS, TX)
Maintenance is completed, all systems operational.
Nov 13, 20:43-22:15 CST
Issues with Dashboard/API
Pending Clouds have now finished being created. We apologize again for the inconvenience, and thank you for your patience.
Nov 7, 08:58-11:51 CST

October 2014

[Scheduled] Major Network Maintenance Affecting Dashboard Operations (AMS,TX) and All Clouds in Dallas
The scheduled maintenance has been completed.
Oct 16, 00:00-05:04 CST
[Scheduled] Brief Network Maintenance Affecting Dashboard Operations (AMS)
The scheduled maintenance has been completed.
Oct 8, 17:00-19:14 CST
[Scheduled] Data Center Security Patch and Restart
All tests are completing successfully. At this time, all functionality should be restored. Please open a support ticket with us if you have any problems operating the dashboard and we will look into it. Thank you for your patience. - The MODX Cloud team
Oct 1, 22:10-23:57 CST

September 2014

Delays in deploying instances to paas1.ams
We've resolved the issues with messaging between the dashboard and paas1.ams. All systems are green. Thanks, -The MODX Cloud Team
Sep 3, 13:34-16:42 CST

August 2014

No incidents reported for this month.

July 2014

MySQL outages on AMS servers
This incident also later affected the AMS2 server, and service was restored there as well. We are continuing to explore the root causes, and to see if the issues on both servers are related.
Jul 17, 15:34-20:55 CST
Brief Outage Amsterdam 1 & 2
During a routine server procedure there was a brief outage on Amsterdam 1 and 2 servers that causes some sites to 502 for a brief period. As this issue was related to an operation we were able to quickly resolve and restore normal operation. We apologize for any inconvenience this may have caused.
Jul 14, 15:03 CST

June 2014

Amsterdam 1 Platform Outage
It appears that most, if not all instances on paas1.ams are now up and running at this time and we continue to monitor the platform to ensure it's continued operation and availability.
Jun 21, 05:36-06:31 CST
Blank Sites and Managers on Amsterdam 2
On Tuesday, June 17, at 02:50 U.S. Central time, our paas2.ams (Amsterdam 2) server at the Amsterdam data center began experiencing issues with blank sites and managers and reports of disk-full errors. The cause has been found to be disk space issues related to the automated back-ups on the platform. This is not related to another partial outage related to MySQL (Percona) database from Sunday June 15 on the same platform. This issue left a number of customers with sites unavailable and or their MODX Managers inaccessible as a result of corrupted MODX cache files. We are actively working to replace the automatic backup handling, work that was started in the past two weeks and hope to be testing and rolling it into place to ensure such an issue doesn't recur. We sincerely apologize for any downtime or loss of productivity resulting from this incident.
Jun 17, 15:51 CST
Reported Issues on Amsterdam 2
This incident has been resolved.
Jun 16, 08:55-13:19 CST
brief outage on AMS 2 server - resolved, investigating
This morning's brief interruption in service was found to be due to way the PHP process manager (PHP-FPM) reloads it's configuration files during certain events on our system. This occurs only rarely however we will continue to look at ways to improve this part of our infrastructure to minimize or eliminate downtime.
Jun 11, 09:54-13:15 CST

May 2014

AMS datacenter degradation
AMS service is restored. If your site still experiences issues, try clearing the MODX cache first. If they persist, use the MODX Cloud Dashboard help tab to open a ticket with support. We sincerely apologize for this outage and will post a root cause analysis on our blog when we have all the facts from our service provider.
May 19, 08:41-09:53 CST
Dashboard performance degraded
Dashboard operations are back to normal and remain stable as we've removed a performance bottleneck from the equation. Thanks for your patience while we make MODX Cloud even better!
May 15, 12:50-14:57 CST
[Scheduled] PHP/nginx Upgrades and Dashboard Platform Update
The scheduled maintenance has been completed.
May 9, 20:00-23:00 CST

April 2014

[Scheduled] System Updates, PHP, MySQL, and HD Checks
MySQL updated to latest version 5.6.16 on all platforms. This concludes the maintenance window. Please open a ticket if you are experiencing issues with any of your instances and we will be happy to assist you. Thank you for your patience.
Apr 19, 22:30-23:19 CST
Dashboard unavailable
This incident has been resolved.
Apr 9, 08:12-09:11 CST

March 2014

Dashboard maintenance
Thanks for your patience - the maintenance is complete and dashboard functionality is fully restored.
Mar 28, 22:27 - Mar 29, 01:24 CST
Dashboard restore and reboot
This incident has been resolved.
Mar 14, 10:23 - Mar 15, 08:03 CST

February 2014

paas2.ams (Amsterdam) investigating outage of some MODX instances
This incident has been resolved.
Feb 28, 08:37-11:28 CST
[Scheduled] Emergency Maintenance on Amsterdam 1
We have completed our post-maintenance checks and verified that all functionality is working as it should. This completes the maintenance period. Please submit a ticket if you encounter issues in the wake of this maintenance.
Feb 14, 20:47-21:06 CST
[Scheduled] Upgrading PHP to 5.4.25
Due to scheduling conflicts, the PHP 5.4.25 upgrade is postponed for 24 hours.
Feb 8, 23:00 - Feb 9, 02:28 CST
[Scheduled] Upgrading Dashboard
The backend upgrade process is now complete, and the dashboard is now operating significantly faster. Thank you for your patience.
Feb 8, 15:41-16:12 CST
Cloud Controller outage
This incident has been resolved.
Feb 4, 08:32-09:48 CST

January 2014

Paas1.tx emergency reboot
Agent on Paas1.tx is working, and the platform is once again fully operational. Thank you for your patience.
Jan 17, 18:06-18:33 CST
Errors on Platform: Texas 1 (paas1.tx)
Our monitoring showed and we received reports of issues with Cloud sites on Texas 1 (paas1.tx) timing out. We identified the issue and have restarted the server. This issue is now resolved.
Jan 6, 12:32 CST

December 2013

[Scheduled] Upgrade Clouds to PHP 5.4
We have tested creation and destruction of instances on all platforms and have not noted any other errors beyond the initial report. All maintenance objectives are now complete, and the Cloud platform is fully green.
Dec 22, 02:33-02:45 CST
Downtime on paas1.ams.modxcloud.com
We have investigated the incident on Amsterdam 1 and concluded that it was related to a change that was part of an upcoming feature rollout which, unfortunately, included an undetected misconfiguration. This caused all sites on the platform to have their URLs routed to a single site. Visitors to the sites would have not arrived at their destination appropriately. We were alerted to the issue at 04:02 CST (10:02 UTC) and corrected this configuration issue by 04:12 CST (10:12 UTC). We will provide complete details to all affected customers.
Dec 12, 04:15-06:28 CST

November 2013

Downtime on AMS1 server
We recorded an incident of downtime on the paas1.ams server between 22:48 and 23:21 on Sunday, November 24, 2013. No potential causes for this have been revealed in our initial exploration. Our investigation is ongoing, and we will update this incident with any findings and or fixes. Our sincere apologies for this incident.
Nov 25, 10:47 CST
Snapshots failing on paas2.ams and paas2.tx platforms.
Snapshots are again behaving properly on paas2.ams and paas2.tx platforms, following the restart of the cloud workers on those platforms. The exact cause of the issue is as yet unknown, as the workers did appear to be up and issuing commands as expected. Investigation will be ongoing. We apologize for the inconvenience and thank you for your patience.
Nov 20, 08:47-20:49 CST
paas1.ams intermittent connectivity
This incident has been resolved.
Nov 19, 08:49-23:02 CST
Cloud controller offline
This incident has been resolved.
Nov 13, 09:26 - Nov 14, 08:00 CST

October 2013

pingdom uptime checks
Our status page has again started receiving Pingdom response time and uptime feeds.
Oct 30, 14:31 - Oct 31, 02:21 CST
Dashboard outage
On Monday, October 28, the MODX Cloud dashboard experienced a brief outage around 8:50 AM U.S. Central time, lasting about 3 minutes. The problem (miscommunication between the web server and the php process manager) has been corrected.
Oct 28, 09:15 CST
PHP upgrade
At 9 AM (U.S. Central time), October 25, 2013, we upgraded PHP on the paas1.tx.modxcloud.com web server to 5.4.21. No downtime resulted from this upgrade. Websites on this server now enjoy the benefits of PHP 5.4 and the security and increased performance that comes with this latest version. This upgrade will be rolled out to the remaining servers over the next few days.
Oct 25, 14:15 CST
We have restarted the controller on paas2.tx, and the issue is now resolved. Thank you for your patience.
Oct 23, 15:14-15:29 CST
[Scheduled] Upgrade to PHP 5.4
Following some preliminary testing results, we have rescheduled this upgrade in order to provide the highest continuation of service. The rescheduled upgrade window will be announced here on our status page.
Oct 17, 23:00 - Oct 18, 07:32 CST
PHP outage on AMS 1
We have experienced a brief outage of the PHP service on our AMS 1 (paas1.ams) server which is now resolved. This led to roughly 5 minutes of downtime for sites hosted on that server, from about 1:44PM to 1:49PM U.S. Central time. We apologize for the incident and are investigating further.
Oct 8, 14:05 CST

September 2013

[Scheduled] [6410382] Texas Datacenter: Firmware Upgrades - Multiple Routers
Router maintenance was completed. No downtime on the paas1.tx or paas2.tx servers, which host our customer's websites, was recorded. The availability of the MODX Cloud dashboard was, however, affected. This has since been rectified.
Sep 26, 05:00-08:36 CST
Dashboard outage
On completion of router maintenance this morning (as described here: http://status.modxcloud.com/incidents/mhvs4hs8bcdz), shortly after 3 AM U.S. Central time, the MODX Cloud dashboard server failed to respond to web requests. This was resolved at 8 AM U.S. Central time. Our apologies for the incident. We're investigating further and intend to put measures in place to minimize this kind of disruption to our service in the future.
Sep 26, 08:30 CST
[Scheduled] [6410382] Texas Datacenter: Hardware Component Replacements - Multiple Routers
This router hardware maintenance has been successfully completed.
Sep 25, 00:00-08:00 CST
paas2.ams outage
We have monitored an outage on the paas2.ams server that began at 7:10 PM CST (Wednesday, September 18, 2013) and was resolved at 10 PM CST. We identified the issue as a problem with the PHP process manager, and corrected it. Additionally we have noted that our uptime notifications via Pingdom were not correctly reporting the outage into our monitoring systems. Investigation into the cause, and steps to both mitigate and react to this type of outage, are ongoing.
Sep 18, 23:13 CST
Intermittent results for Pingdom Uptimes
Pingdom has resolved scheduling issues on their side that resulted in our intermittent monitoring.
Sep 17, 12:19 - Sep 18, 08:55 CST
paas1.tx not responding to Cloud messages
This has been resolved. Total impact time was about 30 minutes. Web sites hosted on this platform were not affected.
Sep 17, 12:17-14:44 CST

August 2013

Amsterdam Firewall Maintenance
August 28, 2013 Outage :: Firewall hardware affecting the Amsterdam datacenter underwent maintenance on August 28, 2013. Initially this hardware was not identified as affecting routes to our servers and therefore we did not communicate the maintenance window to our customers. The scheduled maintenance was planned to cause no more than a 10 minute interruption, but ended up taking 49 minutes to complete, and our Amsterdam servers ended up being unreachable during this window. We sincerely apologies for the outage and will endeavor to better schedule and communicate planned maintenance in the future.
Aug 29, 13:00 CST

July 2013

No incidents reported for this month.

June 2013

No incidents reported for this month.

May 2013

No incidents reported for this month.

April 2013

No incidents reported for this month.

March 2013

No incidents reported for this month.