| Submit your comments on this article |
| Science & Technology |
| NSA Facing Electrical Overload - Cannot Upgrade Computers |
| 2006-08-07 |
| The National Security Agency is running out of juice. The demand for electricity to operate its expanding intelligence systems has left the high-tech eavesdropping agency on the verge of exceeding its power supply, the lifeblood of its sprawling 350-acre Fort Meade headquarters, according to current and former intelligence officials. Agency officials anticipated the problem nearly a decade ago as they looked ahead at the technology needs of the agency, sources said, but it was never made a priority, and now the agency's ability to keep its operations going is threatened. The NSA is already unable to install some costly and sophisticated new equipment, including two new supercomputers, for fear of blowing out the electrical infrastructure, they said. At minimum, the problem could produce disruptions leading to outages and power surges at the Fort Meade headquarters, hampering the work of intelligence analysts and damaging equipment, they said. At worst, it could force a virtual shutdown of the agency, paralyzing the intelligence operation, erasing crucial intelligence data and causing irreparable damage to computer systems -- all detrimental to the fight against terrorism. Estimates on how long the agency has to stave off such an overload vary from just two months to less than two years. NSA officials "claim they will not be able to operate more than a month or two longer unless something is done," said a former senior NSA official familiar with the problem, who spoke on condition of anonymity. Agency leaders, meanwhile, are scrambling for stopgap measures to buy time while they develop a sustainable plan. Limitations of the electrical infrastructure in the main NSA complex and the substation serving the agency, along with growing demand in the region, prevent an immediate fix, according to current and former government officials. "If there's a major power failure out there, any backup systems would be inadequate to power the whole facility," said Michael Jacobs, who headed the NSA's information assurance division until 2002. "It's obviously worrisome, particularly on days like today," he said in an interview during last week's barrage of triple-digit temperatures. William Nolte, a former NSA executive who spent decades with the agency, said power disruptions would severely hamper the agency. "You've got an awfully big computer plant and a lot of precision equipment, and I don't think they would handle power surges and the like really well," he said. "Even re-calibrating equipment would be really time consuming -- with lost opportunities and lost up-time." Power surges can also wipe out analysts' hard drives, said Matthew Aid, a former NSA analyst who is writing a multivolume history of the agency. The information on those hard drives is so valuable that many NSA employees remove them from their computers and lock them in a safe when they leave each day, he said. A half-dozen current and former government officials knowledgeable about the energy problem discussed it with The Sun on condition of anonymity because of the sensitivity of the issue. NSA spokesman Don Weber declined to comment on specifics about the NSA's power needs or what is being done to address them, saying that even private companies consider such information proprietary. In a statement to The Sun, he said that "as new technologies become available, the demand for power increases and NSA must determine the best and most economical way to use our existing power and bring on additional capacity." Biggest BGE customer The NSA is Baltimore Gas & Electric's largest customer, using as much electricity as the city of Annapolis, according to James Bamford, an intelligence expert and author of two comprehensive books on the agency. BGE spokeswoman Linda Foy acknowledged a power company project to deal with the rising energy demand at the NSA, but she referred questions about it to the NSA. The agency got a taste of the potential for trouble Jan. 24, 2000, when an information overload, rather than a power shortage, caused the NSA's first-ever network crash. It took the agency 3 1/2 days to resume operations, but with a power outage it could take considerably longer to get the NSA humming again. The 2000 shutdown rendered the agency's headquarters "brain-dead," as then-NSA Director Gen. Michael V. Hayden told CBS's 60 Minutes in 2002. "I don't want to trivialize this. This was really bad," Hayden said. "We were dark. Our ability to process information was gone." As an immediate fallback measure, the NSA sent its incoming data to its counterpart in Great Britain, which stepped up efforts to process the NSA's information along with its own, said Bamford. The agency came under intense criticism from members of Congress after the crash, and the incident rapidly accelerated efforts to modernize the agency. One former NSA official familiar with the electricity problem noted a sense of deja vu six years later. "To think that this was not a priority probably tells you more about the extent to which NSA has actually transformed," the former official said. "In the end, if you don't have power, you can't do [anything]." Already some equipment is not being sufficiently cooled, and agency leaders have forgone plugging in some new machinery, current and former government officials said. The power shortage will also delay the installation of two new, multimillion-dollar supercomputers, they said. To begin to alleviate pressure on the electrical grid, the NSA is considering buying additional generators and shutting down so-called "legacy" computer systems that are decades old and not considered crucial to the agency's operations, said three current and former government officials familiar with the situation. "It's a temporary fix," one former senior NSA official said. On Wednesday, the same day that The Sun inquired about the power issue with the NSA's public affairs office, the agency sent word to Capitol Hill about its energy conservation efforts. "They have told us they have been shutting down all non-essential uses of power to help out BG&E," said one congressional aide, adding that the NSA is also raising the temperature in its buildings two degrees to conserve. The information was presented in the context that the NSA was making these changes "to be a good corporate citizen," the aide said. Contractors on at least one high-priority, power-intensive NSA project that is located off the headquarters campus, have upgraded their electrical infrastructure to ensure power for their project, according to two former agency officials. That lone upgrade, a fraction of the agency's total demand, took four months. Longer-term solutions being considered would move some operations to off-campus facilities with more electrical capacity, current and former officials said. Adding more capacity to the substation feeding NSA is an obvious answer, but constraints on that particular facility make an expansion difficult, they said. BGE's Foy declined to discuss specifics about the substation. She said it takes 1 1/2 to 2 1/2 years to design, procure equipment, obtain permits, and build a new one. Post-9/11 needs Since the 2001 terrorist attacks, the NSA has ramped up its operations, and the electricity needed to sustain major projects -- such as the warrantless surveillance program and technology modernization programs -- has increased sharply. The computer systems supporting these programs demand far more wattage per square foot than their predecessors and still more energy to cool them. Area development like the Arundel Mills Mall has contributed to the problem by putting additional strain on the local electrical grid, according to two sources familiar with the issue. Joe Bunch, BGE's director of strategic customer engineering, said, however, that the mall's demand "was fairly easily accommodated." Demand in the Baltimore-Washington region has been growing, and the regional operator for Maryland and 12 other states has been studying the installation of up to $10 billion in new power lines to deliver more and cheaper electricity to this region. "We've seen a lot of growth in Anne Arundel County as a whole but particularly in the north and northwest area of the county," said Bunch, who agreed to talk about trends in the area but not the NSA's specific demand. Much of that growth is because of the surge of high-tech jobs in the area from the NSA and government contractors, he said. He said BGE is working to meet the demand by building new substations in the area. One was built about a year ago, and another is scheduled to be built in two to three years, he said. "We have adequate capacity" now, he said, but upgrades like the new substation are being planned to stave off future strains on the electrical grid. The NSA's problem was identified in the late 1990s and could have been fixed by now -- and for much less money -- had keeping the lights on been a priority, current and former officials said. "It fits into a long, long pattern of crisis-of-the-day management as opposed to investing in the future," said one former government official familiar with the NSA's electricity shortfall. Electrical infrastructure maintenance and upgrades have been a casualty of the fight against terrorism, according to unclassified budget documents. Upgrades delayed Even as the NSA's budget has ballooned after 9/11, the agency has put off basic utility upgrades such as a $4 million computer system to manage the allocation of power at the NSA -- a sliver of the NSA's estimated $8 billion budget. "Due to budget constraint [sic] and other development [sic] in the fight against terrorism," a 2007 budget document reads, the system was never fully implemented. Without this system, the document stated, the NSA "may experience difficulties in meeting its power requirement to support critical war fighting missions." Neglect of infrastructure at the NSA has been a chronic problem, often fraught with bureaucratic politics, former agency officials said. Fort Meade is not the only NSA outpost facing limitations on its ability to upgrade electrical infrastructure. Listening posts around the world, such as Menwith Hill in Britain and Bad Aibling in Germany, are ailing. The NSA's largest listening station, Menwith Hill, has an "aging infrastructure that cannot support the people or equipment" there, according to a budget document for 2007. It is faced with "concrete foundations that are crumbling," an "electrical infrastructure that is not in compliance with current codes," and a weakened infrastructure that poses a safety hazard, the document said. Identical language appeared in the previous year's budget documents. With agency operations facing an imminent threat, facilities issues are front and center. "It's a big deal," said one former senior NSA official. "They're all talking about it, anyway. That's progress." |
| Posted by:Anonymoose |
| #20 We use active online-ups(es) backed up by generators. None of the switching ups can hang in our environment. Essentialy, all power is fed through the ups(es) which condition it and clean it up. We have AC aluminum welders and what not, very harsh environment for systems. Have to do all we can for clean and reliable power. |
| Posted by: bombay 2006-08-07 22:12 |
| #19 I know that with phone equipment, the grid power is rectified and floated on a battery bank. Then that bank is inverted and fed to the phone equipment. That way, power is always stable. |
| Posted by: Alaska Paul 2006-08-07 22:05 |
| #18 Yeah, vectors and their supporting equipment are very picky. And boy do they need cooling! |
| Posted by: bombay 2006-08-07 21:11 |
| #17 As an interesting sidenote: When installing a Cray II, you first brought in three phase 480VAC and then connected that to another independent electrical generator which was able to provide much more stable power delivery than the utilities could supply. Very finnicky power regulation requirements were needed to ensure peak performance. |
| Posted by: Zenster 2006-08-07 20:38 |
| #16 The trend in CPUs, even the new intel big boys, is more power consumption not less. The reason the new intels appear less consumptive is because of multi-cpu comparisons to n-core comparison - not an apple to apple comparison. But my friends, it is not just the ICs that are burning the juice, it is the cooling for all of these massive heat generators (CPUs, Memory, BIOSes, disks, etc) running ... that is where the real drain is, cooling these massive computing centers. |
| Posted by: bombay 2006-08-07 20:24 |
| #15 You dont wipe secret hard drives. You crush them and burn them. As one computer expert said, "The only way to prevent someone from recovering any data off of your hard drive is to remove the the disk and hammer it repeatedly." Sounds like the NSA needs to import a submarine or carrier reactor to meet its ampacity shortfall. |
| Posted by: Zenster 2006-08-07 20:17 |
| #14 Turn out tyhe lights and turn off the computer when you leave the room, guys. It's absolutely amazing how much energy that frees up. ;-) |
| Posted by: trailing wife 2006-08-07 19:47 |
| #13 Easily solved. Cut all power to Senate and House. Divert to Ft. Meade. Not only will we never miss them, we'll only wonder why we didn't do it sooner. |
| Posted by: SOP35/Rat 2006-08-07 19:07 |
| #12 Its not just the harddrives. In a server there are lots of parts that have NVRAM that *can* have info written to it if someone really knows what they are doing. Some of these parts can cost upwards of 20K and the big boys refuse to return them when broken unless the server venders can guarantee the NVRAM is wiped absolutely clean. Its kind of eye-opening to see the level of secrecy and realize that its probably not simple paranoia. Makes those movies where they use usb thumb drives to get data out of the CIA look niave. |
| Posted by: rjschwarz 2006-08-07 18:32 |
| #11 DeGauss them. |
| Posted by: Nimble Spemble 2006-08-07 17:56 |
| #10 You dont wipe secret hard drives. You crush them and burn them. Donate them for elementary teacher workstations. Feather + Anvil = KERBOOM! |
| Posted by: 6 2006-08-07 17:55 |
| #9 They should just move everything next to Hoover Dam. |
| Posted by: Nimble Spemble 2006-08-07 17:50 |
| #8 NSA is moving some of their operations to a ormer Sony plant in San Antonio. |
| Posted by: Brett 2006-08-07 17:27 |
| #7 You send them off to be shreaded and melted down. |
| Posted by: DarthVader 2006-08-07 17:27 |
| #6 You dont wipe secret hard drives. You crush them and burn them. |
| Posted by: 3dc 2006-08-07 16:58 |
| #5 They can set up a remote site if they want to and run jobs from there. And now that Intel has the new big-boy on the block that takes a lot less power to do a lot more work than anything else out there. Out with the old, in with the new! Don't forget to wipe the hard drives before you auction them off, though! |
| Posted by: gorb 2006-08-07 14:15 |
| #4 The BRAC is also relocation approximately a bazillion jobs to Ft. Meade in the next five years...they better get their grid powered up ASAP. |
| Posted by: Seafarious 2006-08-07 13:46 |
| #3 Nah, it is all the bit-torrent sites that people are running there. |
| Posted by: DarthVader 2006-08-07 13:45 |
| #2 Well - they are mothballing NORAD's Cheyenne Mountain so -- they have lots of un-interuptable power... move some of the operation there until Fort Meade is fixed. No question of it being secure! |
| Posted by: 3dc 2006-08-07 13:44 |
| #1 Lets wait till it's a crisis and get congress to fund it rather than do it right. |
| Posted by: 49 Pan 2006-08-07 13:36 |