Unscheduled Downtime ~3 hours 18:45-21:45 PST :(

During an routine update we encountered an error with the database. No worries, all seems well now.
Details: A previous version of discourse created a duplicate entry for a user or tag that got its capitalization changed, and failed to delete the previous entry. The index was updated and worked fine but the entry was still there. This totaled ~10 tags and 2 users.
I simply added a “1” to the duplicate entries and split them into two distinct entities.

Normally when there is an issue updating I would see the fail, and not transfer from staging to production. I fat fingered updating our whole stack. To remedy this I will remove the ability to update from the web interface. When the crash occurred I had the option to restore back to 1500pst or attempt a recovery of current DB. Restore would have been faster but manual fix saved all data.

Lesson learned!

Help threads:

11 Likes

I cried just a little when the website went offline. :cry:

10 Likes

Didn’t take long to get bored with “social media”…
But I did learn what a Dyson Sphere is…

8 Likes

Was like taking a book from a person that likes to read

they’ll say the r/haters crashed the update

3 Likes

So bored I had to post about it. :rofl:

3 Likes

Friendly reminder for the slow times

5 Likes

Awesome rundown! Thanks for always sharing the process!

3 Likes

14 Likes

13 Likes

@sidco gave us a taste of what it’s like to be super banned. No one make dad angry again! lol but that was a fun few moments am I right?

2 Likes

4 Likes

I laughed way too hard!

1 Like