The University is analyzing network problems that plagued campus from Feb. 24 until last week to learn how to better handle and communicate technology problems, UH officials said.
A single failed hardware component disrupted the campus wide network and affected access to systems ranging from e-mail and PeopleSoft to voicemail and the credit card system, said Mary Dickerson, interim executive director of Information Technology Security.’
The connectivity problem affected network users inconsistently, which made the problem difficult to track, she said.
‘We take a little comfort in the fact it was very difficult to detect,’ Dickerson said. ‘It was such an obscure problem, it was so difficult to find.’
Dickerson said one of the hardware vendors’ top engineers needed 24 hours to find the problem.’
‘It wasn’t something that we just missed,’ she said.
Information technology officials said although users experienced connectivity issues that limited their ability to use the network, no systems or data were damaged.
‘Anything people were having trouble getting to, it was probably because of this (network problem),’ said Jim Bradley, interim executive director of Technology Support Services. ‘We moved heaven and earth to get this fixed.’
Bradley said any problems that have occurred since Tuesday are unrelated and should be addressed separately.
Problems first appeared on Feb. 24 when the University’s backup system began having difficulty moving large files.’
By Feb. 27, ‘The majority of campus started having significant problems,’ said Dickerson.
IT staff initially dealt with problems as they came up, but as more systems were affected, they realized they were dealing with the symptoms of a single underlying issue.
After an estimated 40 to 50 IT staff members put in hundreds of hours of manpower over the course of the week, the failing component was identified, and on March 3 the system was re-routed to avoid it.
‘For about five days we were all hands on deck,’ Bradley said.’
Some staff members worked 18-20 hour days and ate and slept on campus as they tested systems and worked to identify and correct the problem, Dickerson said.
Since the correction, Bradley said work hasn’t stopped examining the IT Department’s response to the issue and he has communicated with personnel or federation partners who work with teachers and staff in each department.
Dickerson said the department hopes specifically to analyze how it could have identified the problem sooner.
‘One of the things we’re looking at is what happened, and how it happened,’ Dickerson said. ‘It’s been 11 years since we’ve had an event that affected this many systems.’
Richard Willson, professor of chemical and bio-molecular engineering, said faculty members understand technology goes down sometimes, but he said he wants to see improvements in the future.
‘It’s just frustrating,’ Willson said. ‘It’s kind of a miracle to me this kind of stuff works at all, just the complexity of what happens, but still, nobody likes for their computer stuff to crash.’
The network issues were especially frustrating in wasted time for faculty, students and staff, he said.
‘We need to do better,’ Willson said. ‘Every IT Service has outages. Look at Ebay, look at Gmail – nobody’s perfect, but we can get better.’
Bradley said he’s had mostly positive feedback from the campus regarding how IT handled the situation. The department communicated with the campus by e-mail, the Web site and paper notices for those who could not access online resources.
‘We were keeping people aware through multiple levels. In the future, I’d like to have more targeted information.’
Overall, Bradley said, the IT Department is pleased with how it handled and tracked the issues, but there’s always room for improvement.
‘At the end of the day, technology does fail.’
