Solution Brief:

Life SciencesOpening the pipe to faster research, discovery, computation and resource sharing

Abstract

Advances in Information Technology (IT) are significantly improving the speed at which organizations

dedicated to improving the lives of many through research and development conduct business today. In

order to dramatically improve the efficiency of their computing infrastructure, life sciences organizations

and educational institutions helping with scientific research, computational chemistry, systems biology, and

chemical mixing and analysis are turning to Voltaire’s InfiniBand-based solutions.

The Life Sciences Challenge

There are many focus areas for research and development in the life sciences. Life science IT managers are

challenged to provide the right solutions for computational chemistry, systems biology, and chemical mixing

and analysis. These areas share several challenges:

1. Conducting more simulations per day

2. Alleviating storage bottlenecks associated with exponential data growth

3. Implementing cost effective solutions

4. Allocating compute and storage resources dynamically to meet researchers’ needs

Complex simulations can take days or weeks to run. When simulations take longer, scientific discoveries are

delayed, slowing commercialization and increasing competitive threats. With faster simulations, more complex

models can be analyzed, additional assumptions can be tested and further modifications become possible

leading to more efficient development cycles.

Accelerating simulations has clear benefits. The key to accelerating life science application performance is

to select high performance computing systems that eliminate bottlenecks associated with Inter-Processor

Communications (IPC) and storage connectivity.

Large symmetric multi-processing-machines (SMPs) have been used in the past as an answer for generating

massive compute power in data centers. However, these proprietary, expensive systems have given way to

cluster and grid architectures built from lower-cost commodity elements that offer incredible performance at a

significantly lower cost.

Because of the ready-availability of Ethernet, many of today’s clusters and grids are built with Ethernet as the

interconnect. While Gigabit Ethernet-based clustering is less expensive than SMP-based architectures, it tends

to be very inefficient. For applications that rely on bandwidth or memory sharing, or require large amounts of

data transfer, InfiniBand-based interconnects significantly improve the efficiency of clusters by providing high

bandwidths and low latency without increasing CPU utilization.

Contents

The Life Sciences Challenge . . . . . . . 1

Today’s Solutions . . . . . . . . . . . . . . . . . 2

Computational Chemistry . . . . . . . . 2

Systems Biology . . . . . . . . . . . . . . . 2

Chemical Mixing and Material Analysis . . . . . . . . . . . . . . . . . . . . . . . 2

A Better Way . . . . . . . . . . . . . . . . . . . . . 3

Building High Performance Clusters . . . . . . . . . . . . . . . . . . . . . . . . . . 4

High-performance InfiniBand Switches . . . . . . . . . . . . . . . . . . . . . . 4

Fast Storage Access . . . . . . . . . . . . 7

Visualization Solutions . . . . . . . . . . 8

Putting It All Together . . . . . . . . . . . . . 8

Key Features & Benefits . . . . . . . . . 8

Tested & Certified with Leading Applications . . . . . . . . . . . . . . . . . . . 9

Customer Success Stories . . . . . . . . . 9

Swiss Institute of BioInformatics (SIB) . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Tokyo Institute of Technology (TiTech) . . . . . . . . . . . . . . . . . . . . . . 11

About Voltaire . . . . . . . . . . . . . . . . . . . 12

2

Solution Brief: Higher Education and Research

When clusters are built using high performance servers, storage and interconnects, organizations can

experience drastically faster simulations and modeling while continuing to decrease the cost of providing

computing infrastructure.

Today’s Solutions

Many life sciences organizations employ last generation, less-efficient platforms that use proprietary- or

Ethernet-based server interconnects. This approach does not provide the necessary bandwidth for complex

simulations or data transfer that are so common with life sciences applications today.

In the past, applications were “embarrassingly parallel” and were thought to not require a high speed

inteconnect such as InfiniBand. The latest commerical, open source and university-created applications have

been optimized to take advantage of InfiniBand’s improved performance and data transfer speeds.

Computational ChemistryLife science organizations in the area of computational chemistry conduct drug research and discovery as well

as biochemical analysis and modeling. These areas share five primary challenges:

Growing size of formulas and problems �

Need to accelerate development of new life-saving drugs �

More comprehensive safety and drug interaction identification requirements �

Applications that are not optimized for today’s powerful CPUs �

Severe price sensitivity and cost pressure from various groups �

Simulations and modeling for chemical analysis involves huge formulas and constant number crunching. Until

recently, most solutions used Ethernet as an interconnect. Because of the high CPU overhead related to the

handling of communication requests, servers spend more cycles managing inter-processor communications

than actually solving computational tasks.

Systems BiologyLife science organizations conducting analyses and modeling in many different chemical and molecular areas

share three primary challenges:

Simulations of larger, more complex systems and modeling of cell behaviors require lots of �

bandwidth, low latency, and fast storage access

File size and quantity is growing exponentially leading to I/O bottlenecks �

Grids are used to share data, discoveries and models �

Systems biology involves many researchers that constantly retrieve, change, and replace extremely large

files. This causes a tremendous amount of bottlenecks in the network. Because so much data is shared across

organizations, life sciences organizations use large-scale grids.

Chemical Mixing and Material AnalysisLife science organizations conducting research and discovery in chemical analysis, material, adhesive, flavors

and scent modeling share three primary challenges:

Growing size of formulas and problems cause I/O bottlenecks �

The need to accelerate product development �

Safety and quality issues need to be identified sooner in the process �

3

Similar to computational chemistry, the size of formulas can be quite large and cause huge bottlenecks. If these

bottlenecks can be resolved, more analyses can be done faster, accelerating products to market. Additionally,

issues and safety hazzards can be caught sooner in the product cycle, saving organizations thousands of

dollars.

A Better Way

To improve the speed of life sciences applications, engineers need to optimize the design of high performance

computing systems. IT managers spend a lot of time determining the server CPU to be used in clusters and

grids, but the interconnect that is deployed to transport information between the servers is often ignored. This

is a missed opportunity as less efficient interconnects cause significant degradation in application performance.

Voltaire’s InfiniBand solutions accelerate application performance. Voltaire offers high-performance, low-

latency solutions that enable applications to reach their full performance potential. Benchmark testing has

found that Voltaire interconnect solutions reduce runtime by as much as 50-300 percent. Voltaire offers high-

performance (10, 20 and 40 Gbps), low-latency (< 2 microseconds) interconnect solutions used in the world’s

highest performance supercomputers and data centers.

InfiniBand is an industry-standard interconnect for high-performance computing (HPC) and enterprise

applications. The combination of high bandwidth, low latency, and scalability with high performance storage

makes InfiniBand the interconnectof choice to power many of the world’s largest and fastest computer systems

and commercial data centers. Voltaire solutions support most major server vendors, operating systems, storage

solutions and chip manufacturers.

1 Gb Ethernet 10 Gb Ethernet Myrinet InfiniBand

Bandwidth 1 Gb/sec 10 Gb/sec 2.5 Gb/sec 10, 20 & 40 Gb/sec

Latency ~10 us 2.5 - 5.5 us < 2 us

Average Efficiency 53% No Entries 68% 74%

Price Per Gig/Port ~$350.00 >~$700.00 ~$225.00 <$100.00

Table 1: Price/performance advantages for InfiniBand

In addition, Voltaire works with leading storage and application vendors to optimize their solutions to alleviate

IPC and file I/O bottlenecks. By combining leading storage technologies with InfiniBand and Voltaire’s Grid

Director™ family of switch products, life sciences organizations can conduct research faster and more

efficiently to gain a clear competitive advantage.

Benchmark

testing has found

that Voltaire

interconnect

solutions reduce

runtime by as

much as 50 – 300%

and increase

application and

file-system

performance 10x .

4

Solution Brief: Higher Education and Research

\

70

60

50

40

30

21

10

00 20 40 60 7010 30 50

# Cores

46.5975% Efficiency

33.0051% Efficiency

Para

llel S

peed

up

Parallel Speedup

Linear Scaling GbE InfiniBand

Figure 1. 75% Parallel Speed Up with Voltaire vs. only 50% with GbE

Building High Performance Clusters

Voltaire offers complete end-to-end server interconnect solutions for speeding life sciences applications. The

three major elements of the solution include:

High-speed, low latency InfiniBand switches �

Fast Storage Access and Scalable File Systems �

Visualization Solutions �

High-performance InfiniBand SwitchesVoltaire’s InfiniBand-based solutions deliver high performance and scalability to compute clusters. Voltaire

offers a complete portfolio of products including a scalable line of InfiniBand switches, high performance I/O

gateways (for seamless connectivity to Ethernet and Fibre Channel networks) and fabric management software.

Voltaire solutions use the Open Fabric Alliance’s OFED drivers and the Open MPI (Message Passing Interface)

libraries to optimize application performance for both MPI-based and socket-based applications.

Figure 2. Voltaire Grid Director 9024 for small-to-medium sized clusters ranging from 16 to 24 nodes

5

For small-to-medium sized clusters, Voltaire offers the Voltaire Grid Director™ 9024. It is a 1U device with

twenty-four 10 Gbps (SDR) or 20 Gbps (DDR) InfiniBand ports. The switch is a high performance, low latency,

fully non-blocking edge or leaf-switch with a throughput of 480 Gbps.

The Grid Director 9024 is well-suited for small InfiniBand fabrics with up to 24 nodes because it includes all of

the necessary management capabilities to function as a stand-alone switch. The Grid Director 9024 is internally

managed and offers comprehensive device and fabric management capabilities. Designed for high-availability

(high MBTF) and easy maintenance, the switch is simple to install and features straightforward initialization.

The solution is scalable as additional switches can be added to support additional nodes.

Figure 3. Voltaire Grid Director 2004 for scalable clusters ranging from 24-96 compute nodes.

For larger clusters ranging from 25–288 compute nodes, Voltaire offers the Grid Director™ 2004 and 2012

multi-service switch—the industry’s highest performing multi-service switches for medium-to-large clusters

and grids. The switch enables high performance non-blocking configurations and features an enterprise-level,

high availability design. The Grid Director 2004 supports up to 96 InfiniBand 4X ports (20 Gbps) and the Grid

Director 2012 supports up to 288 InfiniBand 4X ports (20 Gbps). Voltaire Grid Director switches are scalable

through the use of modular line boards and they feature 10 GbE and Fibre Channel capabilities so the solution

can provide high-performance, integrated SAN and LAN connectivity.

Voltaire has also defined scalable units for deploying larger, scalable clusters. Scalable units are ideal for

constructing large clusters that deliver unparalleled performance to applications. Scalable units combine

compute, interconnect and storage capabilities with scalable file systems.

At the heart of the solution is the Voltaire Grid Director 2012 multi-service switch. Voltaire’s director-class,

multi-service switches offer integrated InfiniBand, GbE and Fibre Channel connectivity in a single chassis. This

enables MPI and storage traffic to run on the same network, a capability that is not available with Ethernet

or proprietary fabrics. By enabling IPC and high performance storage on a single network, Voltaire solutions

enable far greater scalability.

6

Solution Brief: Higher Education and Research

Figure 3. A scalable unit of 200 nodes powered by a Voltaire Grid Director 2012

Scaling out further is made easy by using Voltaire Grid Director switches as core switches to interconnect

multiple scalable units. Such connectivity can be implemented as fully non-blocking or as partially blocking

depending on application requirements or budget constraints.

sFU-8

SMB

SMB

sFB-12sFB-12

sFB-12sFB-12

sFU-8

SMB

SMB

sFB-12sFB-12

sFB-12sFB-12

sFU-8

SMB

SMB

sFB-12sFB-12

sFB-12sFB-12

sFU-8

SMB

SMB

sFB-12sFB-12

sFB-12sFB-12

Figure 4. Multiple scalable units interconnected using a Voltaire Grid Director 2012

7

Fast Storage AccessFor companies looking to incorporate storage into their InfiniBand cluster, Voltaire offers fast I/O capabilities

for storage. Voltaire solutions combine scalable compute and storage capabilities with parallel file systems. By

using InfiniBand with parallel file systems, the server’s CPU overhead is reduced, freeing up CPU cycles for your

application.

ComputeServer

CPUUtilization

Before Now

Software

Network

- Low performance- High overhead on CPU- No scalability

- High performance- CPU available for applications- Scalability to thousands of nodes

IPC Storage

Compute

MPI NFS

ProprietaryInterconnect GbE

IPC Storage

MPI Parallel FileSystem

InfiniBand

At the heart of the solution is the Voltaire Grid Director 2004 multi-service switch (described above). Voltaire’s

director-class, multi- service switches offer seamless InfiniBand, GbE and Fibre Channel connectivity. This

enables MPI and storage traffic to run on the same network, a capability Ethernet and proprietary fabrics do not

offer.

By enabling IPC and high-performance storage on a single network, Voltaire solutions allow companies to leave

behind the limitations of network file systems (NFS) and move to parallel file systems over InfiniBand. This

provides far-greater scalability. Applications can now have effective file I/O rates of 350MB/s compared with

the 50MB/s previously available by using NFS. Additionally, the size of compute clusters is no longer limited by

the limitations imposed by NFS.

Scalable File SystemsRunning scalable file systems over Voltaire InfiniBand solutions creates the most scalable solution in the

industry with more than 1,000 nodes on a single name space, and delivers high performance connectivity for

the storage and client nodes.

Voltaire has significant experience and expertise in enabling large-scale parallel file system deployments.

Such solutions include: Lustre, HP SFS, IBM GPFS, Panasas and PVFS. These solutions, when combined with

InfiniBand, solve two critical problems that NFS creates: limited throughput and limited scalability.

The diagram (Figure 6) below outlines a Voltaire deployment with HP SFS (Lustre) with 1,100 nodes (2,200

cores) all accessing a single file system.

8

Solution Brief: Higher Education and Research

VoltaireISR 9288

Visualization Nodes

Lustre Clients

Compute Nodes Storage Nodes

sCTRL

Voltaire Grid

Multi-PanelDisplays Lustre Servers (OSS, MDS)

DOD T1-05

OSS Sustained FS Performance 900 MB/sClient Side Performance 350 MB/s

Figure 6: Voltaire’s TI-05 installation at the D.O.D – 1,100 nodes on a single file system –

InfiniBand is used for both MPI and Lustre over the same wires.

Visualization SolutionsCustomers in a variety of industry sectors including life sciences, energy, automotive, aerospace, government

and military use Voltaire solutions for visualization on clusters and grids ranging from dozens to hundreds of

nodes.

Visualization clusters typically require very high bandwidth and Voltaire’s InfiniBand solutions with up to

20Gbps per host link are ideal for this. In addition, the low overhead on the CPU frees it to carry out the image

processing faster. The result is a powerful solution that delivers higher levels of resolutions and faster image

processing.

Putting It All Together

Key Features & BenefitsVoltaire solutions for life sciences offer many compelling benefits to users:

High Bandwidth: � Voltaire solutions provide bandwidth of 20 Gbps to allow for faster and more

frequent analysis by life sciences applications.

Lower Latency: � Voltaire solutions provide latency as low as 1.3 microseconds. Moreover, Voltaire’s

InfiniBand-based solutions leverage Remote Direct Memory Access (RDMA) with CPU and OS

bypass-technologies that greatly reduce memory-copy overheads and associated CPU utilization.

Standards-Based: � Voltaire solutions are based on InfiniBand: the only industry-standard, high-

performance interconnect

Flexibility: � Grids and clusters that use Voltaire solutions can be built as a fully non-blocking 20

Gbps fabric or as a lower-bandwidth fabric based on the needs of the application. Moreover, Voltaire

switches are upgradeable in a non-disruptive, hot-pluggable manner.

Figure 7: 3D visualization is

commonly used in

Life Sciences.

9

Fast I/O for Storage: � Voltaire solutions enable parallel file systems over InfiniBand, which offers

far-greater performance and scalability than NFS solutions.

Tested & Certified with Leading ApplicationsBy working closely with leading server and software vendors on integration and testing, Voltaire offers the

fastest and most efficient high-speed interconnect solutions for the life sciences market.

OS Support Linux Enterprise Edition from Novell (SUSE SLES) and Red Hat

(EL/AS)

Supported Parallel File Systems Lustre, HP SFS, IBM GPFS, Panasas, IBRIX, TerraScale’s TeraGrid

Applications Accelrys, Gaussian, AMBER, BLAST, FASTA, GlimmerM, Wise2,

ACT, ClustalW, EMBOSS, HMMER, Image, T-Coffee, Artermis,

CHARMm, Cn3D, GAMESS, GROMACS, RasMol, ReadSeq,

TribeMCL, NAMD, NMRView

Systems & Platform Partners HP, IBM, SUN, NEC, SGI, NEC, Intel, AMD

Customer Success Stories

Customers in a variety of industries that rely on modeling, simulation and analysis leverage Voltaire solutions

for their cluster interconnects. Companies include consumer-product manufacturers, research orgnizations,

pharmaceutical companies, and university development labs throughout the world.

Swiss Institute of BioInformatics (SIB)The Swiss Institute of BioInformatics (SIB) needed a system that could help them speed up discovery of

research for their humanitarian efforts around diseases like mad cow and breast cancer research. They turned

to HP and Voltaire to supply a system that could grow with their needs, but also help speed up the life science

applications running on the cluster.

The Swiss Institute of BioInformatics, through its VitaI IT Computing Institution, created a joint venture

between Oracle, HP and Intel using Voltaire Grid Director™ switches to enable the modeling and analysis.

SIB first built the Vital-IT Computer Center in 2003 using a configuration relying on SAN storage and NFS

services, using gigabit Ethernet for communication among servers. But 18 months after the initial deployment,

SIB realized two major problems.

“With SAN storage, a limited number of servers can be connected to the SAN,” said Dr. Victor Jongeneel of

SIB. “Our servers send and receive data from the compute nodes using NFS, and this turned out to be a major

performance bottleneck. With 64 clients running significant I/O, we needed a faster way than gigabit Ethernet

for all of our compute nodes to share common file space.”

Voltaire solutions support

many leading life sciences

applications.

10

Solution Brief: Higher Education and Research

Storage HP SFS

Test Cluster #1

(8 nodes)

Test Cluster #2

(4 nodes)

Production Cluster

(68 nodes)

Voltaire Grid Director Switch

sFU-8

SMB

SMB

sFB-12sFB-12

sFB-12sFB-12

Figure 8: The system configuration as deployed by Swiss Institute of BioInformatics (SIB)

SIB also has many computing jobs that read large amounts of data into memory, many of which are I/O

bound. Because of the load on the file servers, they would sometimes crash, and jobs could be aborted before

completion. “Because of this, we could not run as many jobs per unit time as we wanted,” Jongeneel said.

The Voltaire interconnect between compute nodes provides much better performance both in terms of

bandwidth and latency,” Jongeneel said. “Computing jobs using HP-MPI, Voltaire MPI and LAM MPI now

run much faster than they used to, and more importantly, our server I/O capacity is no longer a bottleneck for

running any size job.”

SIB selected a configuration of one large cluster consisting of 80 HP Servers with a mix of Itanium2 and

EM64T processors—connected by the Voltaire Grid Director switch. The solution leverages the multi-service

capabilities of Voltaire’s Grid Director switches to enable Fast I/O with storage connectivity. The clusters run

multiple life science applications such as Blast, Gromacs, CHARM and Platform Computing’s scheduler LSF. This

customer is extremely satisfied with the solution because the combination of the Voltaire Grid Director switch,

HP servers, MPI software and File I/O for larger jobs makes for easier file manipulation at high performance.

11

Tokyo Institute of Technology (TiTech)The Tokyo Institute of Technology (TiTech), located in Tokyo, Japan, is one of the leading technical universities

in the world. TiTech has a long history as a world leader in high performance and GRID computing, and houses

one of the world’s largest supercomputers.

Storage

Core

Server

sCTRL sCTRL sCTRL

sCTRL sCTRL

sCTRL sCTRL

sCTRL

The TiTech system serves as an example of how scalable and flexible the Voltaire family of products is. Used

by a wide range of researchers across the university and collaborators across Japan and around the globe, the

solution delivers more than 40 trillion floating point operations per second (TFlops).

The solution tackles computationally difficult problems ranging from:

Analysis of how Avian Flu mutates and is transmitted from birds to humans �

Structural analysis of new materials �

Making buildings more resistant to earthquakes �

More accurate prediction of the earth’s climate �

Visit www.top500.org for more information on this solution.

Contact Voltaire to Learn More

1.800.865.8247 info@voltaire.com www.voltaire.com

©2008 Voltaire Inc. All rights reserved. Voltaire and the Voltaire logo are registered trademarks of Voltaire Inc. Grid Director is a trademark of Voltaire Inc. Other company, product, or service names are the property of their respective owners.

About Voltaire

Voltaire (NASDAQ: VOLT) designs and develops server and storage switching and software solutions that

enable high-performance grid computing within the data center. Voltaire refers to its server and storage

switching and software solutions as the Voltaire Grid Backbone™. Voltaire’s products leverage InfiniBand

technology and include director-class switches, multi-service switches, fixed-port configuration switches,

Ethernet and Fibre Channel routers and standards-based driver and management software. Voltaire’s

solutions have been sold to a wide range of end customers including governmental, research and educational

organizations, as well as market-leading enterprises in the manufacturing, oil and gas, entertainment, life

sciences and financial services industries. More information about Voltaire is available at www.voltaire.com or

by calling 1-800-865-8247.

NoticeReproduction of this publication in any form without prior written permission is not allowed. The information in this publication is subject to change without notice and is provided “AS IS” WITHOUT WARRANTY OF ANY KIND. THE ENTIRE RISK ARISING OUT OF THE USE OR INTERPRETATIONS OF THIS INFORMATION REMAINS WITH RECIPIENT. IN NO EVENT SHALL VOLTAIRE BE LIABLE FOR ANY DIRECT, SPECIAL, PUNITIVE OR OTHER DAMAGES.

Performance results will vary based upon a number of system factors. Some of these include: server configuration of the processor, chip set, memory size, firmware and driver release versions, MPI version and OS kernel version. The configuration or configurations tested or described may or may not be the only available solution. These tests are not a determination of product quality or correctness, nor does it ensure compliance with any federal state or local requirements.

Product names mentioned herein may be trademarks and/or registered trademarks of their respective companies.