What happened:

At approximately 5:00AM PST ServerCanyon was alerted to severe performance degradation. The cause as to why is still unknown, but for transparency here is a timeline of events:

  • 5:30AM, We rebooted our primary server
  • 5:45AM, After realizing a severe problem was preventing the server from booting, the failover server kicked into gear and began to download client backups
  •  6:45AM, 80% of affected clients had been activated on the backup server
  • 7:30AM, We got on the phone with our domain registrar, as the changes to our name servers “A” record were about 45min overdue.
  • 7:40AM, Data recovery began on the old server.
  • 8:00AM, All accounts had been brought back online, changes since the last offsite backup began to sync.

 

What we are doing to prevent this from happening in the future:

  • We increased our in-datacenter backup provision by 50%
  • All accounts will now be backed up daily (5 day retention) and weekly (4 week retention)
  • Over the course of the next several weeks we will be moving clients to a pool of IP addresses that can be routed to new servers
  • We are moving to a more reliable domain registrar

 

If any customer discovers any issues with their website, please contact support immediately!

1 view