GIH servers suffer sustained DoS attack

After spending a couple of days off-line, I found out on Sunday 30th August that our servers in London had been hit by a denial of service (DoS) attack through mail-bombing from different sources.

This happens every now and then. The last time, it took place when I was at the ICANN conference in Cairo, and another European participant's systems had been hit at the same time. Thankfully the GIH servers in London survived then, and thankfully, they survived (barely) this time.
The strength of the attack this time round was higher than before since every time there is an attack, I find out what the weak point is in our network and upgrade it. I'm glad that the servers actually manage to recover each time, but it is still very disruptive indeed.

So this time round, the main mail hub got attacked with 360 emails per second from zombie computers all around the world, for a very sustained amount of time, starting at 8:00pm on saturday night. As a result, the server ran out of memory+swap (500Mb + 500Mb) and froze by going into some kind of panic recovery mode, thus sending the attack to our backup route, and this then sent the emails to our back-end machine via another path, in a more controlled manner (aka - via UUCP over IPv6, the new with the old working very well together). As a result, the back-end stored 28 000 emails, all but 200 being spam sent to wrong addresses.

I managed to reboot the main mail-server remotely in a short window of time that it allowed me to log in. It took 30 minutes to shut down, so clogged were its processes, running at a load of 58. I spent sunday afternoon trying to find out how to process the backlog of emails, bearing in mind, more was coming in. It felt like dealing with a flood.

Finally, I commissioned a third computer running Linux, which I had kept running for the past 4 months as standby and which has now taken the load off the front and back end machines and is shifting through the now 24 000 remaining emails. All in all, the GIH computer systems in London have filtered 98 000+ spams in 24h. That is a lot more than at any time before. If this is a taste of things to come in the near future, we're going to have to beef those servers more with a lot more processing power.
When I remember that the first computer system for GIH.COM was a 20Mhz 80386 running linux & 4 Mb memory, downloading emails via UUCP and sending them via SLIP/PPP through a half hourly telephone call using a 9 600 baud modem, and it was running smoothly, even when downloading a few USENET newsgroups that I enjoyed reading...

...I just wonder where this spam is leading us to.

...sigh...