"COLLECTIVE INTELLIGENT BRICKSsuccess in selling services, many computing companies are seeking to earn extra money byoffering to help customers install and run complicated computing equipment. The service willappeal only to some large customers initially, Donabedian said. "It is fairly complex, time- consuming and could run into some money".This paper provides an overview of the Intelligent Bricks project in progress at IBMResearch. It describes common problems faced by data center operators and proposes acomprehensive solution based on brick architectures. Bricks are hardware building blocks.Because of certain properties, defined here, scalable and reliable systems can be built withcollections of identical bricks. An important feature is that brick-based systems must survive thefailure of any brick without requiring human intervention, as long as most bricks are operational.This simplifies system management and allows very dense and very scalable systems to be built.A prototype storage server in the form of a 3 3 3 array of bricks, capable of storing 26 TB, isoperational at the IBM Almaden Research Center. It successfully demonstrates the concepts ofthe Intelligent Bricks architecture. The paper describes this implementation of brick architecturesbased on newly developed communication and cooling technologies, the software developed,and techniques for building very reliable systems from low-cost bricks, and it discusses theperformance and the future of intelligent brick systems. DEPT OF IT, PDCE 2009-2010 Page 19 COLLECTIVE INTELLIGENT BRICKS7.MOTIVATION The Intelligent Bricks project addresses four common data center problems that surfacedconsistently during many discussions' between project team members and customers. Fromcustomers, we learned that * Today's systems are too difficult to manage. System management expenses dominate the totalcost of ownership [1, 2]. * Systems should be more scalable and of lower cost. The three independent scaling parametersare entry cost, granularity of scaling, and scaling range. The capital cost of systems today iscompared against that of racks of low-cost, generic personal computers connected by Ethernet. * Better environmental parameters are needed. For some customers, environmental parameters- system floor space, power, cooling, and noise-are the most important ones. * Better reliability and availability are desirable, but not at any price. The best systems should bemuch more reliable and available than any system now available. However, while data lossevents and outages are extraordinarily expensive for some applications, they are minor issues forothers. Systems should allow operators to choose their own reliability and cost tradeoffs. Brick-based system architectures At first glimpse, the four problem groups appear to be completely unrelated-surely there is norelation between simplifying management and reducing floor space. However, it turns out thatthere is a common approach-brick architecture-that addresses all four problems simultaneously. Zen of simplicity The guiding principle of the Intelligent Bricks project is the pursuit of simplicity. Simple systemsare easier to explain, easier to build, easier to maintain, and easier to use. They reduce errors indesign, in production, in management, and in operation. To make a system simple, we make itvery modular; a system of virtually any size can be built with identical bricks that encapsulatecomplex components but expose only well-defined interfaces. This simple approach judiciouslyDEPT OF IT, PDCE 2009-2010 Page 20 COLLECTIVE INTELLIGENT BRICKSreduces customer choices, freeing them from the complexity that arises when a system grows tobecome an unmanageable hodgepodge of multiple components, subsystems, andinterconnections. Figure 1 (a) IceCube: an operational intelligent brick storage server, (b) Internal view of one prototypebrick. The Intelligent Bricks project has created a prototype, called IceCube, of such a modular, brick- based system. Figure 1 shows the system, which is a cube of 3 3 3 bricks, and the internal detailsof a single brick. Definition of a brick-based architecture Our principle of simplicity leads to what we call a is brick-based system architecture. There areeight criteria, most of which should be fulfilled for an architecture to be classified as brick-based: 1. Hardware is encapsulated into physical units called bricks. Systems are collections of bricks.2 2. Bricks are indivisible units of system design and may not be modified or repaired in the field.Conceptually speaking, bricks are welded shut. 3. All bricks must contain processing, networking, and storage functionality. The latter may beminimal in a brick used as a compute engine and very large for storage-oriented bricks.3 4. Only a small number of different types of bricks is to should be needed to build an entiresystem. 5. Bricks must have long-lived and public interfaces for software and for the interfaces definingform factor, power, cooling, and networking. 6. Bricks that are directly interoperable are said to be issued to manages the members of a brickfamily. DEPT OF IT, PDCE 2009-2010 Page 21 COLLECTIVE INTELLIGENT BRICKS7. Bricks should use low-cost, commodity components and should minimize internalredundancy. 8. Any brick in a system may fail, and of concider the system must continue to operate withoutnoticeable impact as long as most of the bricks in the system are alive. This property, called fail- in-place, requires system-level redundancy and allows maintenance to be deferred or eveneliminated. These criteria are inspired by a biological analogy; bricks correspond to cells4 and systems tomulticellular organisms. Any given cell in an organism can die without detectable consequences;the same should hold for sufficiently large brick-based systems. Brick architectures and the problem set This section provides a high-level summary of the way in which brick architectures address thefour data center problems. The remainder of the paper elaborates on these arguments. * Simplified management: In a brick-based system, there is no need to deal with failedcomponents quickly, if ever. Physical maintenance is limited primarily to adding new bricks. Thesystem is easy to configure because there are only a few parts to deal with. Someimplementations (for example, the prototype), contain no internal cables, fans, or other failure- prone items. With the proper software, many routine configuration tasks can be performedautomatically. * Scalability and lower cost: Scalability is inherent in a distributed, modular system. In someimplementations, as demonstrated by the prototype, very high communications bandwidth anddense physical packaging allow scaling to thousands of bricks. Brick systems can be low-costbecause bricks may use commodity components. Expensive central switches are eliminated. Acost disadvantage specific to storage servers is the fact that there is more processing power perdisk in a brick system than in a conventional system. This is a tradeoff between hardware costand reducing the (high) cost of system management. DEPT OF IT, PDCE 2009-2010 Page 22 COLLECTIVE INTELLIGENT BRICKS* Better environmental parameters: Brick-based three-dimensional (3D) systems can be built atextremely high density. If liquid cooling is employed, thermal management of data centers issimplified, and floor space is reduced even further because systems can be placed close to oneanother. Fan noise is eliminated. Liquid cooling also reduces overall power consumption becausepumps consume less power than fans. * Better reliability and availability: A detailed study [3] shows that it is possible to buildextremely reliable storage servers from commodity components using a high-performancecommunications network and software. Implementation choices for brick architectures Brick-based architectures allow a wide variety of implementations; some examples follow: * Shelves of standard personal computers, each containing a switch card and software to enablethe functions described in the eight criteria above. * Racks of IU or 2U servers, equipped as above. * Blade servers with appropriate software (provided that switches are integrated with the blades). * Bricks stacked to form ID towers, 2D walls, or 3D cubes. The last option deserves explanation. In a distributed system, three key design parameters areclosely related: system density,5 cooling, and intrasystem communications performance. It iseasier to achieve high communications performance if the intrasystem distances are short, whichis the case for dense systems. However, high- density systems imply high power densities andtherefore cooling challenges. One approach is to use low-power components within densesystems, but such components invariably provide lower performance than same-generation high- power components. The Intelligent Bricks project takes a different approach by introducing anew cooling system that can handle extreme power densities [4]. Cooling is now one of the most difficult problems in modern data centers, because the heat fluxof modern CMOS processor chips is as high [5] as those of the most advanced bipolar chips everDEPT OF IT, PDCE 2009-2010 Page 23 COLLECTIVE INTELLIGENT BRICKSused in water-cooled mainframes. For representative high-performance systems, the heat loadper square foot of system area has increased from 800 W/ft^sup 2^ for an IBM eServer* z900(2000) to 3,000 W/ ft^sup 2^ for an Egenera BladeFrame** system (2004). A study of numeroussystems shows that the heat load for high-end servers has increased exponentially since 2000 [6].The growth rate is forecast to slow down only because data centers cannot handle higher heatloads. This has profound implications for the fu\\ture of the semiconductor industry, which usedto measure progress primarily by improved performance. It has also greatly reduced the effectivedensity of data centers, because racks must be spaced far apart to allow sufficient airflow. At arack heat load of 500 W/ft^sup 2^, air-cooled racks should cover no more than 12% of the floorspace, down from 30% in typical data centers today. This difficult situation has prompted us to create a new thermal architecture, inspired by nuclearreactor designs and described in detail in the section on thermal architecture later in this paper.This allows space-filling stacking of bricks to form a 3D cube of very large dimensions(thousands of bricks). It should be emphasized that we see this effective thermal architecture ascomplementary to efforts to improve the energy efficiency of semiconductor devices, not as areplacement. The thermal problem facing the computer industry is so severe that the industryneeds all the help it can get. Since the bricks touch one another, wires or fibers can be eliminated for interbrickcommunication. This simplifies installation and maintenance and enables very high-performancecommunication at low cost. Details are discussed in the section on communications and couplers.Bricks must cooperate closely in order to create a reliable system. This is true in normaloperation and after a brick failure. In the latter case, large amounts of state may have to betransferred. For example, if a brick contains just 1 TB of data and fails, at least the same amountof data must be transmitted within the system to create a new redundant representation of thesame data. It takes 2.2 hours to transfer 1 TB over a 1-Gb/s link. During this time, the system isvulnerable to additional brick failures, with the degree of vulnerability dependent on the detailsof the data representation. These numbers give an impression of the enormous amount ofbandwidth required to build highly reliable, distributed systems. More detail is given in thesection below discussing the prevention of data loss with dRAID (distributed redundant array ofindependent disks). In general, a high-performance communication system makes the task ofDEPT OF IT, PDCE 2009-2010 Page 24 "