Final Note: Senior staffer at our co-location provider gave us the details that the power issues were caused by a lightning strike. The San Francisco Bay area recorded upwards of 1,200 ground lightning hits in thunderstorms yesterday and looks like we were impacted by one of those unfortunately.
While waiting for a full post-mortem we have learned it was an at least partial power outage in the data center which took down sections of their core networking, and did impact a few of our legacy systems, one of which is our support ticket system.
Staff will be monitoring and answering support emails directly in email until our support ticket system returns to normal operation.
All client systems are online and operational, internal monitoring systems are showing green for systems and service processes like apache, mysql, etc. If you are still having any issues, please let our support team know, just be aware you will not see a automated ticket reply until the support system is operational again.
Thank you.
Posted Sep 11, 2017 - 21:55 CDT
Update
We may have updated too soon. Seems that they are getting things stabilized but still seeing up and down monitoring reports from global test locations against our servers in the US West data center.
We will update again once we get further details.
Posted Sep 11, 2017 - 21:17 CDT
Monitoring
Fortunately the issues has been resolved by our data center/co-location provider in Fremont, CA.
Unfortunately this was a much longer outage that we had hoped for and will be getting a post-mortem from our data center provider in Fremont, CA as the the cause and work they will do to prevent such issues from happening again in the future. We have been in this data center since early 2001 and we have not suffered an outage there in more than 10 years.
Thank you for your patience and please let our support teams know if you have any questions or are experiencing any further issues.
We will be monitoring this closely for a bit and will close the incident in approx. 30 minutes.
Posted Sep 11, 2017 - 21:13 CDT
Investigating
We are getting remote monitoring systems showing routing issues through specific paths to systems in our Fremont CA data center. We will post more once we get details from our upstream provider there.