ICTS Can Fix Things

Yes, there are those rare few who know what they're doing at ICTS. And Adrian Frith is one of those people who knows those particular people and what they're good at.

With Adrian's help, last week the issues with the LCC were solved. The solution ended up being: "Oh, let's just disable all the smart stuff in the smart switch!" Problem I find in that is, what was the point of a new switch then? Apart from the nice upgrade to a Gbit network, which doesn't really improve the situation for ICTS and them wanting to remotely block ports. It's fixed though and nothing has gone backwards (thankfully) so I'm happy.

Then, another issue we had with them was with the Olympiad server. This is the server we use for the SACO. Now, the team are supposed to be training for the upcoming IOI and half of their training is done off this server. The 3rd round of the SACO is also approaching and again, they use this server to train on. The issue was that it could only be accessed on campus via the IP address. The DNS wouldn't resolve and the firewall was blocking access from off campus.

This problem started about 3-4 weeks ago. We've been in contact with ICTS for at least two weeks. They kept on closing the call, telling us the problem was resolved. Three of us (myself, Bruce and Carl) all tried desperately to get them to sort this out as it was really slowing other things down. On Friday I tell Adrian about the problem and this morning it is solved! Wow, how it helps knowing the right people at ICTS.

So that's two of three issues solved for now. The last one which is only getting worse is affecting everyone. Carl has blogged about this problem, so I won't say much more other than it appears to be related to the issues they're having with multi-casting. They've pulled down the network about five times now to get it working and they're doing it again this evening.

My word, how disruptive a small group of people can be!

ICTS Driving Me Mad

I think everyone I know at UCT has at least one story about a nasty experience with ICTS. I tell you, I have many. One of the reasons is that I co-administrate the IBM Linux Competency Center (LCC). If you tried clicking on that link and it didn't work that either means you aren't on the UCT network (blame ICTS) or the problem I am about to describe has yet to be resolved.

A very brief background on the LCC. It is a lab with a rack of IBM servers ranging from dual core blades through to a server with 8 cores. I took over admining the lab almost exactly a year ago from now together with Jason Brownbridge who has since left UCT and Adrian Frith has taken over his duties. Running on our own subnet we often have to deal with ICTS, especially since our CS admin, Matthew West, left last month.

During this year ICTS has been gradually "upgrading" the UCT network, claiming the end user will benefit, although all I've heard that will be new is that they will have more control over the network such as being able to disable machines remotely. On Friday 29 June, the PD Hahn building in which the LCC is situated underwent the upgrade. This is when all the troubles began. To give you an idea, when the CS building underwent the upgrade it indirectly caused one of our sys admins to retire.

First problem was the IP adresses were all changed. We appeared to resolve that issue pretty quickly, although more on that later. Then they replaced the switch with a nice new Gbit Cisco switch. This is where the real problems start. The blade center could not connect to the switch. Three people from ICTS checked it out on seperate occasions - one of them checking it twice - and every time they've told us it's an issue on our end. So, we decided to put in our own switch to be sure. Guess what? It worked!! We're still following this up though, as it would be nice to get the Gbit switch back.

The other problem we're still experiencing started on Monday. All of a sudden after working on Sunday none of the nodes could connect to the external network. First one ICTS person told us they had been having issues deploying multicasting services on Friday and that it has been spreading the PD Hahn building, affecting the various subnets. However, today another ICTS person tells us that he knows of no network issues in PD Hahn. Tell me about miscommunication! He tells us that we're using the incorrect IP address. HOW?

I wll end there before I explode. Another show of ICTS' brilliance.

ICTS and The Mail Loop

#define ICTS Information and Communication Technology Services, University of Cape Town

About a week ago there was an issue with UCT's mail servers which created a massive backlog. After "resolving" the issue, this email was sent out explaining the cause:

Dear ICTS Customer,

Yesterday's delays with external mail delivery have been resolved and
mail delivery is back to normal.  The delays were caused by an
individual on campus who set up a rule to forward mail to an off-campus
mail address.  The rule was incorrectly set up as it forwarded messages
to a mail account that did not exist.  This caused a mail loop.  A mail
loop is created when the message is forwarded (as per the rule) and then
bounces back.  The bounced message is in turn forwarded and bounces
back.  This grows exponentially as each message (and its bounced
version) repeats the loop.  This loop prevents the mail gateway from
processing other mail.

ICTS blocked the offending messages, but by that stage there was a
backlog of other messages that the mail gateway had to process.

To prevent this from happening, when setting up a mail forwarding rule,
please double-check the email addresses used.

Regards
ICTS Communications

If the above is cropped, check the original here.

The first thing that came to my mind is: Why haven't they resolved the issue? All they've done is identified the cause of the problem and told us as users how to prevent this from happening again. When you think about it though, this email will only reach a very limited number of users and therefore most of them will be completely unaware of the issue. And even then, why leave it up to the users? Humans are known to be prone to errors, so even if I double-check I might still make a mistake.

The more crucial problem is the fact that they announced a vulnerability and explained exactly how to take advantage of it. They are in essence advertising to anyone wanting to have a bit of fun attacking their mail servers a very simple way to cause havoc. They should have kept the cause to themselves while attempting to close the hole. But instead, they have done little to resolve the issue and I haven't heard of any attempts to do so. And then there's another small issue that could cause greater headaches - the mail archives are public! Why?

Another less important question I have in mind is how do they come to the conclusion that the growth is exponential? For each outgoing mail there is one incoming bounce mail, causing one outgoing mail and so on. So then why is it not linear? It makes me wonder why they couldn't manage load.

ICTS have a history of screwing things up. Just a couple months ago they started forcing the UCT network onto dynamic IP's, which has caused so many problems it has resulted in many administrators retiring. I also heard once that they actually celebrated logging their 100,000th call or some number. Who celebrates receiving such a large number of complaints?