#define ICTS Information and Communication Technology Services, University of Cape Town
About a week ago there was an issue with UCT's mail servers which created a massive backlog. After "resolving" the issue, this email was sent out explaining the cause:
If the above is cropped, check the original here.Dear ICTS Customer,
Yesterday's delays with external mail delivery have been resolved and
mail delivery is back to normal. The delays were caused by an
individual on campus who set up a rule to forward mail to an off-campus
mail address. The rule was incorrectly set up as it forwarded messages
to a mail account that did not exist. This caused a mail loop. A mail
loop is created when the message is forwarded (as per the rule) and then
bounces back. The bounced message is in turn forwarded and bounces
back. This grows exponentially as each message (and its bounced
version) repeats the loop. This loop prevents the mail gateway from
processing other mail.
ICTS blocked the offending messages, but by that stage there was a
backlog of other messages that the mail gateway had to process.
To prevent this from happening, when setting up a mail forwarding rule,
please double-check the email addresses used.
The first thing that came to my mind is: Why haven't they resolved the issue? All they've done is identified the cause of the problem and told us as users how to prevent this from happening again. When you think about it though, this email will only reach a very limited number of users and therefore most of them will be completely unaware of the issue. And even then, why leave it up to the users? Humans are known to be prone to errors, so even if I double-check I might still make a mistake.
The more crucial problem is the fact that they announced a vulnerability and explained exactly how to take advantage of it. They are in essence advertising to anyone wanting to have a bit of fun attacking their mail servers a very simple way to cause havoc. They should have kept the cause to themselves while attempting to close the hole. But instead, they have done little to resolve the issue and I haven't heard of any attempts to do so. And then there's another small issue that could cause greater headaches - the mail archives are public! Why?
Another less important question I have in mind is how do they come to the conclusion that the growth is exponential? For each outgoing mail there is one incoming bounce mail, causing one outgoing mail and so on. So then why is it not linear? It makes me wonder why they couldn't manage load.
ICTS have a history of screwing things up. Just a couple months ago they started forcing the UCT network onto dynamic IP's, which has caused so many problems it has resulted in many administrators retiring. I also heard once that they actually celebrated logging their 100,000th call or some number. Who celebrates receiving such a large number of complaints?