I left yesterday to visit my girlfriend and we spent quite a while chatting, watching tv, eating, etc. I visited her since we haven’t seen each other for about a week already, so I didn’t care what I left on our server that day. What I last configured in our server was the email notification for the online support team. I set their email to our default email support email address.
I went back home last night, plugged the computer and do my routine… I checked the server and the pending online support tickets. Hmmm… I thought the the ticket was responded by someone already so I smiled, but I was terrified when I clicked the ticket and viewed a long list of reply history, in loop! I don’t know what happened but it’s probably because of my email configuration. It triggered a system bug inside our support center that made the CPU load in our server quite high this evening. Well… I left the email looping inside our “piped” email support system for and 4 hours, it resulted to 2.8 Gigabytes of MySQL database (overhead) size from a 2 kb email.
If you don’t know Piping is a technology that immediately query an email right inside the database for storage. In other words, it “capture” the email for instant trouble ticket issue. So to simplify that, in every loop, the email was stored repeatedly in the database.
From 2 kb of email, it was able to accumulate its size to 2.8 Gigabytes inside the database. In fact I’m quite amazed that our Linux server managed this kind of load! It was able to hold 100% CPU utilization for 4 hours, and still serve our customers fast. It simply means that we are powerful enough to host in heavy loads without system failure.
This is an accident but it also tested how powerful our server is. 
Luckily we have the hardware to support this kind of abuse.
Recent Comments