We have identified the two simultaneous events that we believe are the cause of the database issues this morning. We have established new procedures to ensure this issue doesn't happen again. At this time, we are successfully bringing all components of the database layer back online.
Posted Nov 16, 2016 - 16:25 CST
Update
The database is running at reduced capacity while we continue to bring the entire system back online.
Posted Nov 16, 2016 - 14:47 CST
Update
After a routine application of database software updates to address new CVEs, an error was noticed in the logs. While attempting to identify the source of the error, the database cluster crashed. Bringing the database back online after a crash is unfortunately, quite time consuming. We anticipate that the database will be operational within the next hour.
Posted Nov 16, 2016 - 12:36 CST
Identified
A security update to a portion of our database solution has caused a major issue. We are actively working on bringing the database back up.