With memory, as with real estate, location matters. A group of researchers from Advanced Micro Devices (AMD) and the Department of Energy’s Los Alamos National Laboratory have found that the altitude at which SRAM (static random access memory) resides can influence how many random errors the memory produces.
In a field study of two high-performance computers, the researchers found that L2 and L3 caches had more transient errors on the supercomputer located at a higher altitude, compared with the one closer to sea level. They attributed the disparity largely to lower air pressure and higher cosmic ray-induced neutron strikes.
Strangely, higher elevation even led to more errors within a rack of servers, the researchers found. Their tests showed that memory modules on the top of a server rack had 20 percent more transient errors than those closer to the bottom of the rack. However, it’s not clear what causes this smaller-scale effect.
Vilas Sridharan, an AMD technical staff member, presented the findings Thursday at the SC13 supercomputing conference, being held this week in the mile-high city of Denver.
Using the error logs of two large high-performance computers, the study examined the characteristics of transient memory errors, in which a memory module may store a 1 as a 0, or vice versa.
Transient errors are different from permanent or even intermittent errors, which are usually caused by hardware failure, Sridharan said. Transient errors appear more randomly and are not usually the fault of machinery. They are relatively rare, but depending on where they occur, they can cause a cascade of additional system errors.
The group studied the monthly transient fault rates of SRAM—the L2 and L3 caches within processors—in two large Cray supercomputers, each running thousands of AMD processors.
One supercomputer was the Jaguar system at Oak Ridge National Laboratory in Oak Ridge, Tennessee, which is approximately 817 feet (249 meters) above sea level, according to an online altitude finder.
The other system under study was the Cielo supercomputer at the Los Alamos National Laboratory in Los Alamos, New Mexico, which is about 7,058 feet (2,151 meters) above sea level.
The group had found that, when all other possible confounding issues were factored out, Cielo’s SRAM had a “significantly higher rate of SRAM faults,” compared with Jaguar’s SRAM, Sridharan said.
For example, with L3 caches, Cielo was bedeviled by 735 transient faults for every 219 that Jaguar endured. L2 transient faults across the two machines showed a similar relationship.
The findings were not a surprise, according to Sridharan. It has long been theorized that transient memory errors can come from the high-energy impact of neutrons from cosmic rays, which is more pronounced at higher elevations. Other factors related to elevation, such as air pressure, may also play a role.
“This is theoretically well-known, but it is nice to see the data,” Sridharan said.
Another effect was slightly more mysterious: The SRAM at the top of server racks had a significantly greater number of transient errors than that at the middle or the bottom of the same racks, within both Jaguar and Cielo.
“There is a trend towards a higher rate of SRAM faults as you go up the rack,” Sridharan said. “This is something we don’t really have a good explanation for.”
SRAM on the server on the top of the rack had 20 percent more transient errors than the SRAM on the servers on the lower levels. “This is not a huge effect, but it is a consistent one,” Sridharan said.
The difference probably could not be attributed solely to cosmic rays, Sridharan said. He briefly speculated on a number of possible causes. For example, because heat rises, the servers at the top of a rack are hotter than those on the bottom. Heat is a well-known culprit in equipment failure.
A low-cost solution, such as installing heat shielding on server racks, may be worth investigating, Sridharan said.
In the study, the group also looked at the DRAM memory faults. They examined memory from three different vendors and found that the fault rate of one vendor was four times the rate of another vendor. The group did not release the names of the vendors but did alert the vendor with the leading error rate about the comparatively high rate of faults for its products.