Monday morning downtime 3 years ago

We effectively have two catch alls when it comes to problems here at Twitter; database and load balancer.  This morning, our “load balancer” was “overloaded” and we had a few “hung” machines.  Honestly, we’re amazed the Internet stays up as it does considering we can’t get a Linux box to function for more than 8 hours without it seizing up.  Ya gotta wonder how Google does it sometimes.

Our best guess is that the Intertubes we have connected to the load balancer were flooded with all kinds of pornography from the weekend.  Its taken us most of the morning to flush those tubes out and we’re still at a loss as to why the Internets are so slow.  On the line with our co-lo now.

Update: 11:35am - phew, we’re still “working through this problem”.  Ha.  Just kidding.  Its been a Mimosa kind of a morning.  Please bear with us.