When you enter the computer room on the second floor of Tokyo Institute of Technology’s computer building, you’re not immediately struck by the size of Japan’s second-fastest supercomputer. You can’t see the Tsubame computer for the industrial air conditioning units that are standing in your way, but this in itself is telling. With more than 30,000 processing cores buzzing away, the machine consumes a megawatt of power and needs to be kept cool.
Tsubame was ranked 29th-fastest supercomputer in the world in the latest Top 500 ranking with a speed of 77.48T Flops (floating point operations per second) on the industry-standard Linpack benchmark.
While its position is relatively good, that’s not what makes it so special. The interesting thing about Tsubame is that it doesn’t rely on the raw processing power of CPUs (central processing units) alone to get its work done. Tsubame includes hundreds of graphics processors of the same type used in consumer PCs, working alongside CPUs in a mixed environment that some say is a model for future supercomputers serving disciplines like material chemistry.
Graphics processors (GPUs) are very good at quickly performing the same computation on large amounts of data, so they can make short work of some problems in areas such as molecular dynamics, physics simulations and image processing.
“I think in the vast majority of the interesting problems in the future, the problems that affect humanity where the impact comes from nature … requires the ability to manipulate and compute on a very large data set,” said Jen-Hsun Huang, CEO of Nvidia, who spoke at the university this week. Tsubame uses 680 of Nvidia’s Tesla graphics cards.
Just how much of a difference do the GPUs make? Takayuki Aoki, a professor of material chemistry at the university, said that simulations that used to take three months now take 10 hours on Tsubame.
Tsubame itself – once you move past the air-conditioners – is split across several rooms in two floors of the building and is largely made up of rack-mounted Sun x4600 systems. There are 655 of these in all, each of which has 16 AMD Opteron CPU cores inside it, and Clearspeed CSX600 accelerator boards.
The graphics chips are contained in 170 Nvidia Tesla S1070 rack-mount units that have been slotted in between the Sun systems. Each of the 1U Nvidia systems has four GPUs inside, each of which has 240 processing cores for a total of 960 cores per system.
The Tesla systems were added to Tsubame over the course of about a week while the computer was operating.
“People thought we were crazy,” said Satoshi Matsuoka, director of the Global Scientific Information and Computing Center at the university. “This is a ¥1 billion (US$11 million) supercomputer consuming a megawatt of power, but we proved technically that it was possible.”
The result is what university staff call version 1.2 of the Tsubame supercomputer.
“I think we should have been able to achieve 85 [T Flops], but we ran out of time so it was 77 [T Flops],” said Matsuoka of the benchmarks performed on the system. At 85T Flops it would have risen a couple of places in the Top 500 and been ranked fastest in Japan.
There’s always next time: A new Top 500 list is due out in June 2009, and Tokyo Institute of Technology is also looking further ahead.
“This is not the end of Tsubame, it’s just the beginning of GPU acceleration becoming mainstream,” said Matsuoka. “We believe that in the world there will be supercomputers registering several petaflops in the years to come, and we would like to follow suit.”
Tsubame 2.0, as he dubbed the next upgrade, should be here within the next two years and will boast a sustained performance of at least a petaflop (a petaflop is 1,000 teraflops), he said. The basic design for the machine is still not finalized but it will continue the heterogeneous computing base of mixing CPUs and GPUs, he said.