Life with Ed: A Case Study of a LinuxBIOS/BProc Cluster

In this paper, we describe experiences with our 127-no de/161-processor Alpha cluster testbed, Ed. Ed is unique for two distinct reasons. First, we have replaced the standard BIOS on the cluster nodes with the Lin-uxBIOS which loads Linux directly from non-volatile memory (Flash RAM). Second, the operating system provides a single-system image of the entire cluster, much like a traditional supercomputer. We will discuss the advantages of such a cluster, including time to boot (101 seconds for 100 nodes), upgrade (same as time to boot), and start processes (2.4 seconds for 15,000 pro-cesses). Additionally, we have discovered that certain predictions about the nature of terascale clusters, such as the need for hierarchical structure,are false. Finally, we argue that to achieve true scalability, terascale clusters must be built in the way of Ed.