#define ICTS Information and Communication Technology Services, University of Cape Town
About a week ago there was an issue with UCT's mail servers which created a massive backlog. After "resolving" the issue, this email was sent out explaining the cause:
If the above is cropped, check the original here.Dear ICTS Customer,
Yesterday's delays with external mail delivery have been resolved and
mail delivery is back to normal. The delays were caused by an
individual on campus who set up a rule to forward mail to an off-campus
mail address. The rule was incorrectly set up as it forwarded messages
to a mail account that did not exist. This caused a mail loop. A mail
loop is created when the message is forwarded (as per the rule) and then
bounces back. The bounced message is in turn forwarded and bounces
back. This grows exponentially as each message (and its bounced
version) repeats the loop. This loop prevents the mail gateway from
processing other mail.
ICTS blocked the offending messages, but by that stage there was a
backlog of other messages that the mail gateway had to process.
To prevent this from happening, when setting up a mail forwarding rule,
please double-check the email addresses used.
Regards
ICTS Communications
The first thing that came to my mind is: Why haven't they resolved the issue? All they've done is identified the cause of the problem and told us as users how to prevent this from happening again. When you think about it though, this email will only reach a very limited number of users and therefore most of them will be completely unaware of the issue. And even then, why leave it up to the users? Humans are known to be prone to errors, so even if I double-check I might still make a mistake.
The more crucial problem is the fact that they announced a vulnerability and explained exactly how to take advantage of it. They are in essence advertising to anyone wanting to have a bit of fun attacking their mail servers a very simple way to cause havoc. They should have kept the cause to themselves while attempting to close the hole. But instead, they have done little to resolve the issue and I haven't heard of any attempts to do so. And then there's another small issue that could cause greater headaches - the mail archives are public! Why?
Another less important question I have in mind is how do they come to the conclusion that the growth is exponential? For each outgoing mail there is one incoming bounce mail, causing one outgoing mail and so on. So then why is it not linear? It makes me wonder why they couldn't manage load.
ICTS have a history of screwing things up. Just a couple months ago they started forcing the UCT network onto dynamic IP's, which has caused so many problems it has resulted in many administrators retiring. I also heard once that they actually celebrated logging their 100,000th call or some number. Who celebrates receiving such a large number of complaints?
I do it the other way round.. forward gmail to UCT.. but I got an error message from ICTS mail server -to- my gmail adrdress saying my gmail address was invalid.. i mean 'wtf'!!!, why send the error message to the flipping invalid address?
ReplyDeleteI'm not great with mail servers, but doesn't one implement some sort of basic loop prevention when you set up a mail server???!!! This is ridiculous.
ReplyDeleteUCT is a great University, but its downfall is going to be the patheticness of ICTS. It's just not that hard to do things right.
I bet if UCT employ'd some CS students as a "New ICTS", UCT would have an almost flawless network within 3 months.
Oh, and Marco, you had me literally LOL'ing at the 100 000 call celebration! :)
I must admit that this is the most passionately upset I have ever been at ICTS - Not just upset, but disappointed. Usually I just tell myself: "Everybody makes mistakes..." and "Many people don't ever make them right...". But this time, I cannot help but wonder: "WTF!"
ReplyDelete