In 2010, the Air Force Research Laboratory (AFRL) unveiled Condor, a roughly $2 million supercomputer that combined more than 1,700 PlayStation 3 consoles with Intel servers and Nvidia graphics processors. The consoles stayed intact, sitting on wire shelves in Rome, New York, running Linux and sharing research calculations across a network.
The attraction was inexpensive computing power for carefully chosen tasks. Making that bargain work meant dividing problems into small pieces, moving data between machines and working around the PS3’s limited memory.
In the announcement for Condor’s December 1, 2010 ribbon-cutting, Mark Barnell, who directed AFRL’s high-performance computing work, said its capability exceeded that of any other interactive supercomputer then used by the Defense Department. That announcement cited 1,760 PS3s, but later Air Force reporting and the engineers’ own papers describe 1,716, arranged in 78 groups of 22. The sources we reviewed do not resolve the discrepancy.
Condor’s advertised capability of 500 teraflops, or 500 trillion floating-point operations per second, also needs context. It described the whole system on paper, not the consoles alone. A September 2011 AFRL briefing credits about 263 teraflops to the PS3s and about 230 to the Intel servers and their Nvidia Tesla graphics cards. The same briefing separately lists sustained-throughput benchmark results for each group of processors, totaling 271.6 teraflops. It does not present that total as a single application running across the entire cluster.
Why a PlayStation 3, and not a PC?
The answer is the chip inside. The PS3 used the Cell Broadband Engine, developed by Sony, Toshiba and IBM. IBM’s research describes it as a nine-core processor: one general-purpose PowerPC core plus eight Synergistic Processing Elements (SPEs), small vector cores built to perform the same arithmetic on several numbers at once. Under Linux on a PS3, programs could use six of those SPEs.
IBM sold the same processor in its Cell blade servers. Sony sold it in a games console that cost far less, and the original model could also run Linux. As IBM researcher Rodric Rabbah put it in a 2007 tutorial, the PS3 “is not just a gaming console.”
The Air Force was testing this idea well before Condor. In a 2008 study, researchers from AFRL’s Sensors Directorate and contractor SET Corporation ran a radar imaging algorithm on a single PS3 and on a server with two quad-core Intel Xeons. Performance was very similar, with the Xeons slightly ahead. They chose the PS3 as their Cell test platform “because of its low cost and ready availability,” noting that Cell blade servers were “considerably more expensive.”
The attraction was computing performance for the money and power consumed. In a radar test described in the engineers’ longer 2012 paper, one PS3 took about 59% longer than an Intel Xeon server node to complete the same optimized workload. But the console cost $380 against $3,000 and drew 111 watts against 513. For that workload, the authors calculated, the PS3’s hardware cost per gigaflop of throughput was about one-fifth of the tested Xeon node’s, and its power use per unit of throughput was about one-third. Barnell told Federal News Network that similar computer architecture cost $7,000 to $10,000, while a PlayStation cost a few hundred dollars.
The bargain came with limits. Each console had only 256 MB of memory. The 2012 paper notes that the PS3 performs single-precision calculations about 10 times faster than double-precision ones, so code had to be rewritten to use less precise numbers wherever accuracy allowed. Those constraints shaped everything that ran on Condor.
What “connecting them” actually meant
Linking 1,716 consoles doesn’t create one giant PlayStation. It creates 1,716 small computers that each have to be told what to do, fed data and asked for their answers. That job fell to the servers.
Researchers logged in through six gateway servers and reserved the parts of the cluster they needed. Each of the 78 compute servers ran a group of 22 consoles, connected through Ethernet switches. The servers also shared a faster InfiniBand network for jobs that only needed the Intel chips and graphics cards. The PS3s ran ordinary Linux distributions, Fedora 9 or Yellow Dog Linux, with wake-on-LAN enabled so idle consoles could sleep, cutting each one’s draw from 95 watts to 5.
Any job on a cluster like this follows the same three steps, however much hardware is involved:
- Divide. Break the problem into pieces that can be worked on independently.
- Move. Ship each piece, plus the data it needs, to a worker.
- Combine. Collect the results and stitch them back together.
Adding consoles can shorten the parallel calculation stage, but communication and coordination overhead limit the overall speedup. Nextgov reported in early 2010 that console-to-console connections were relatively slow, which restricted the programs the system could run efficiently. With 256 MB per console, data also had to arrive in small portions.
The team’s text-recognition work shows the problem in practice. According to the 2012 paper, a program that reconstructed damaged scanned text ran on all 132 SPEs in a group of 22 PS3s at once. With poor-quality input, the time threads spent waiting to synchronize with one another limited how much extra threads could help; larger buffers eased it. With “reasonable efforts in performance optimization,” the system processed about 16 to 20 scanned pages per second.
One real job: turning radar echoes into pictures
Condor’s main applied project, according to the HPEC paper, was synthetic aperture radar (SAR): generating radar images for “persistent surveillance of city-sized areas” with a new image every second, effectively radar video. The paper also lists space situational awareness, video target tracking and large-scale “computational intelligence” models among the cluster’s workloads.
An airborne SAR fires radar pulses rapidly as it flies and records the echoes. A technique called backprojection then builds the image one pixel at a time. For every pixel, the computer works out how far that patch of ground was from the aircraft at the moment of each pulse, then adds up the matching slice of every echo. In its simplest form, the 2008 AFRL study notes, the work grows with the cube of the image width, so doubling the width makes the job roughly eight times bigger.
That makes it a good fit for a cluster. Each pixel can be calculated without knowing anything about its neighbors, so the image can be cut into tiles and spread across consoles. Condor’s engineers divided it into 32-by-32-pixel blocks, small enough that each SPE only had to hold the block it was working on, while the Intel servers prepared the radar data and sent it out. They also trimmed the arithmetic, exploiting the fact that the distance to a given pixel changes almost linearly from one pulse to the next to cut the operations in the innermost loop by about a quarter.
The software catch
Condor depended on a feature Sony was in the middle of removing. The original PS3 had an “Install Other OS” option that let owners run Linux. When Sony introduced the cheaper PS3 Slim in 2009, it dropped the option. A Sony representative explained that supporting it “did not fit with the wider objective to offer a lower cost PS3.”
In March 2010, Sony announced that firmware update 3.21, released April 1, would remove Other OS from older models too, citing security concerns. Owners could keep Linux by declining the update, but they would lose access to the PlayStation Network and to games and Blu-ray discs that required the newer firmware.
For a research cluster that never needed the PlayStation Network, declining was easy. Replacing hardware was the problem. In May 2010, an AFRL representative told Ars Technica that the lab would “have to continue to use the systems we already have in hand,” and that the change made it “difficult to replace systems that break or fail.” At Condor’s launch, HPCwire reported that the consoles couldn’t be sent to Sony for repair, since they would come back with the update installed, and that the cluster used older PS3s rather than Slims.
The firmware change did not shut Condor down; the full cluster was unveiled eight months after the update. But it changed the nature of the bet. Running the consoles the lab already owned was fine. Replacing one that failed meant finding an older, Linux-capable PS3 that Sony no longer made.
A bargain with conditions
Condor worked because AFRL chose problems that fit the PS3’s strengths: tiled image formation, text recognition split across many small cores, jobs where the same arithmetic runs on many independent pieces. Barnell told Federal News Network that Condor was not a general-purpose supercomputer and did not replace others, but worked well for specific applications.
Its bargain depended on three things working together: affordable hardware, access to the software and workloads that could be divided efficiently. The PS3 supplied the first two, and AFRL’s engineers supplied the third. Sony’s decision to withdraw Linux support showed that the second was never fully in the lab’s control.