Scalable Strategies for Computing with Massive Data
-
Upload
nguyenquynh -
Category
Documents
-
view
222 -
download
4
Transcript of Scalable Strategies for Computing with Massive Data
Scalable Strategies for Computing with Massive Data
Michael J. Kane, John W. Emerson, and Stephen B. Weston
Department of Statistics
Yale University, New Haven, CT 06520
January 28, 2011
1
Author’s Footnote:
Michael J. Kane ([email protected]) is Associate Research Scientist, Yale Center for An-
alytical Sciences, Yale University; 300 George Street Suite 555, New Haven, CT 06520-8290.
John W. Emerson ([email protected]) is Associate Professor, Department of Statis-
tics, Yale University; P.O. Box 208290, New Haven, CT 06520-8290. Stephen B. Weston
([email protected]) is an independent consultant; 89 Carleton Street, Hamden,
CT 06517. The authors thank Dirk Eddelbuettel, Bryan Lewis, David Pollard, and Martin
Schultz for advice and guidance on many aspects of these projects.
2
Abstract
The analysis of very large data sets has recently become an active area of research in
statistics and machine learning. Many new computational challenges arise when managing,
exploring, and analyzing such data. These challenges effectively put the data beyond the
reach of researchers who lack specialized software development skills or expensive hardware.
This paper presents a scalable, extensible framework for statistical computing that addresses
these challenges with new ways of working with very large sets of data: memory- and file-
mapped data structures, which provide access to arbitrarily large data while retaining a look
and feel that is familiar to statisticians; and data structures that are shared across processor
cores in order to support efficient parallel computing techniques. This paper also presents
a new framework for portable parallel computing that benefits from the use of shared data
structures. All of the contributions are available in R extension packages, but they also
provide a roadmap for the evolution of statistical computing environments.
Keywords: concurrent programming, memory-mapping, parallel programming, shared
memory, statistical computing
3
1. INTRODUCTION
On October 2, 2006, the DVD-rental service Netflix released the Netflix Prize data (Bennet
and Lanning 2007), consisting of 99,072,112 ratings from 480,189 users for 17,770 movies.
The competition’s goal was to improve upon the Netflix algorithm for predicting user ratings
by at least 10%; a one million dollar prize was offered. On September 18, 2009 the prize was
awarded to the “BellKor’s Pragmatic Chaos” team, which achieved 10.05% improvement.
Statisticians have worked for decades developing analyses for these types of problems.
Ironically, only two of the seven members of the winning team were statisticians. In fact,
it is difficult to find teams with statisticians on the Netflix Leaderboard (http://www.
netflixprize.com/leaderboard/); leading teams generally consisted of computer scien-
tists and engineers. Given the nature of the challenge, it seems natural to ask, “Why wasn’t
this problem, which is statistical in nature, fully engaged by statisticians?”
The major stumbling block to exploring the Netflix Prize data set is its size. At about
two gigabytes (GB), the data set is large enough that it is not easily managed or analyzed
with readily available hardware and software. Teams appearing on the leaderboard gener-
ally had both access to expensive hardware along with the professional computer science
and programming expertise necessary for implementing highly specialized algorithms. Most
statisticians were left behind because their two most widely-used software packages, SAS R©
(SAS Institute Inc. 2011) and R (R Development Core Team 2010b), were ill-equipped to
handle this class of problem. SAS handles large data problems with an impressive number
of standard methods, but the Netflix Prize Competition required the development of new
methodology. On the other hand, R is very well-suited for the development of new method-
ology but does not seamlessly handle massive data sets. These barriers to entry meant that
few statisticians were able to engage the challenge.
The root of the problem is the current inability of modern high-level programming en-
vironments like R to exploit standard hardware capabilities. This paper bridges this gap
by presenting new approaches for scalable statistical computing. Low-level operating sys-
4
tem features are leveraged to provide data structures capable of supporting massive data,
potentially larger than random access memory (RAM). Unlike database solutions and other
alternative approaches, these data structures are compatible with basic linear algebra subrou-
tines (BLAS) and algorithms which rely upon column-major matrices. The data structures
are available in shared memory for use by multiple processes in concurrent programming
applications. These advances address two interrelated challenges in computing with massive
data, data management and statistical analysis. Section 2 describes the challenges and solu-
tions for managing and exploring massive data. Section 4 considers a broad class of statistical
analyses well-suited for parallel programming solutions. It presents a technology-agnostic
framework for implementing parallel algorithms which can exploit the shared memory capa-
bilities of the proposed data structures.
The solutions proposed in this paper are implemented in a collection of packages extend-
ing the R programming environment: bigmemory (Kane and Emerson 2010a), bigalgebra
(Kane, Lewis and Emerson 2010), bigtabulate (Kane and Emerson 2010b), biganalytics
(Emerson and Kane 2010), synchronicity (Kane 2010), and foreach (Weston and REvolu-
tion Computing 2009c). Together they provide a software framework for computing with
massive data (demonstrated in Section 3) that includes shared memory parallel computing
capabilities (demonstrated in Section 5). More importantly, they identify critical advances
that must be incorporated into the next generation of statistical computing environments in
order to provide a truly seamless environment for statistical computing with massive data.
2. A FRAMEWORK FOR MANAGING MASSIVE DATA
High-level programming environments such as R and MATLAB R© (The MathWorks, Inc.
2011) are flexible and convenient, allowing statisticians to import, visualize, manipulate,
and model data as well as develop new techniques. However, this convenience comes at
a cost because even simple analyses can incur significant memory overhead. Lower-level
programming languages sacrifice this convenience but often can reduce the overhead by
referencing existing data structures instead of creating unnecessary temporary copies. When
5
the data are small, the overhead of creating copies in high-level environments is negligible
and generally goes unnoticed. However, as data grow in size the memory overhead can
become prohibitively expensive. Programming environments like R and Matlab have some
referencing capabilities, but their existing functions generally don’t take advantage of them.
Uninformed use of these features can lead to unanticipated results.
According to the R Installation and Administration guide (R Development Core Team
2010a), R is not well-suited for working with data structures larger than about 10-20% of
a computer’s RAM. Data exceeding 50% of available RAM are essentially unusable because
the overhead of all but the simplest of calculations quickly consumes available RAM. Based
on these guidelines, we consider a data set large if it exceeds 20% of the RAM on a given
machine and massive if it exceeds 50%. Although the notion of size varies with hardware
capability, the challenges and solutions of computing with massive data scale to the statis-
tician’s problem of interest and computing resources. We are particular interested in cases
where massive data or the memory overhead of an analysis exceed the limits of available
RAM. In such cases, computing with massive data typically requires use of fixed storage
(disk space) in combination with RAM.
Historically, size limitations of high-level programming languages have resulted in the use
of database management systems. A database can provide convenient access to subsets of
large and massive data structures and are particularly efficient for certain types of summaries
and tabulations. However, reliance on a database has several drawbacks. First, it can
be relatively slow in executing many of the numerical calculations upon which statistical
algorithms rely. Second, calculations not supported by the database require copying subsets
of the data into objects of the high-level language stored in RAM. This copying can be
slow, and the subsequent analysis may require customized implementations of algorithms for
cases where the overhead of standard implementations (used for smaller data) exceeds the
capacity of RAM. Finally, the use of a database requires the installation and maintenance
of a software package separate from the high-level programming environment and may not
be available to all researchers.
6
Customized extraction of subsets of data from files resident on disk provides an alternative
to databases. This approach suffers from two drawbacks. First, it has to be done manually,
requiring extra time to develop specialized code and, as a result, proportionally more time
for debugging. Second, custom extractions are often coupled to specific calculations and
cannot be implemented as part of a general solution. For example, the calculation of the
mean of a column of a matrix requires an extraction scheme that loads only elements from
the column of interest. However, a column-wise extraction will not work well for calculating
matrix row means. The modes of extraction are specific to the chosen file format and data
structures. As a result, different algorithms may need to be implemented over the course of
a data exploration, increasing development and debugging time. In the extreme case, some
calculations may require sophisticated extraction schemes that may be prohibitively difficult
to implement; in such cases the statistician is effectively precluded from performing these
types of calculations.
Both the database and custom extraction approaches are limited because their data
structures on disk are not numerical objects that can be used directly in the implementation
of a statistical analysis in the high-level language. They require loading small portions of
the data from disk into data structures of the high-level language in RAM, completing some
partial analysis, and then moving on to other subsets of the data. As a result, existing
code designed to work on entire data structures native to the language and within RAM
is generally incompatible with analyses relying upon database or customized extractions of
data from files. Fundamentally, these approaches are limited by their reliance on technologies
that evolved decades ago and lack the flexibility provided by modern computing systems and
languages. Some algorithms have been designed specifically for use with massive data. For
example, the R extension package biglm (Lumley 2009) implements an iterative algorithm
for linear regression (Miller 1992) that processes the data in chunks, avoiding the memory
overhead of R’s native lm function for fitting linear models. However, such solutions are
not always possible. The prospect of implementing different algorithms for a certain type of
analysis simply to support different data sizes seems grossly inefficient.
7
Modern operating systems allow files resident on disk to be memory-mapped, associating
a segment of virtual memory in a one-to-one correspondence with contents of a file. The
function mmap is available for this purpose on POSIX-compliant operating systems (including
UNIX, Linux, and Mac OS X, in particular); Microsoft Windows offers similar functionality.
The Boost C++ Libraries (Boost 2010) provide an application programming interface for
the use of memory-mapped files which allows portable solutions across Microsoft Windows
and POSIX-compliant systems. The access of memory-mapped files is faster than standard
read and write operations for a variety of reasons beyond the scope of this paper. Most
importantly, the task of moving data between disk and RAM (called caching) is handled at
the operating-system level and avoids the inevitable costs associated with an intermediary
such as a database or a customized extraction solution. Interested readers should consult
one of the many web pages describing memory-mapping, such as Kath (1993), for example.
Memory-mapping is the cornerstone of a scalable solution for working with massive data.
Size limitations become associated with available file resources (disk space) rather than RAM.
From the perspective of both the developer and end-user, only one type of data structure is
needed to support data sets of all sizes, from trivial to massive. The resulting programming
environment is thus both efficient and scalable, allowing single implementations of algorithms
for statistical analyses to be used regardless of the size of the problem. When an algorithm
requires data not yet analyzed, the operating system automatically caches the data in RAM.
This caching happens at a speed faster than any general data management alternative and
only slower than customized solutions designed for very specific purposes. Once in RAM,
calculations proceed at standard in-memory speeds. Once memory is exhausted, the operat-
ing system handles caching of new data and displacing older data (which are written to disk
if modified). The end-user is insulated from the details of this mechanism, which is certainly
not the case with either database or customized extraction approaches.
We offer a family of extension packages to the R statistical programming environment
which provides a first step towards a scalable solution for computing with massive data.
The main contribution is a new matrix data structure called a big.matrix which exploits
8
memory-mapping for several purposes. First, the use of a memory-mapped file (called a
filebacking) allows matrices to exceed available RAM in size, up to the limitations of available
filesystem resources. Second, the matrices support the use of shared memory for efficiencies
in parallel computing both on single machines or in cluster environments. The support for
shared memory and a new framework for portable concurrent programming will be discussed
in Section 4. Third, the data structure provides reference behavior, potentially avoiding the
need for unnecessary temporary copies of massive objects. Finally, the underlying matrix
data structure is in standard column-major format and is thus compatible with existing
BLAS libraries as well as other legacy code for statistical analysis (primarily implemented
in C, C++, and Fortran). Together, these features provide a foundation for the development
of new statistical methods for analyzing massive data.
These extension packages are immediately useful in R, supporting the basic exploration
and analysis of massive data in a manner that is accessible to non-expert R users. Some of
these features can be used independently of R by expert developers in other environments
having a C++ interface. Typical uses involve the extraction of subsets of data into native R
objects in RAM rather than the application of existing functions (such as lm) for analysis of
the entire data set. Although existing algorithms can be modified specifically for use with
big.matrix objects, this opens a Pandora’s box of recoding which is not a long-term solution
for scalable statistical analyses. Instead, we support the position taken by Ihaka and Temple
Lang (2008) in planning for the next-generation statistical programming environment (likely
a new major release of R). At the most basic level, a new environment could provide seam-
less scalability through filebacked memory-mapping for large data objects within the native
memory allocator. This would help avoid the need for specialized tools for managing massive
data, allowing statisticians and developers to focus on new methodology and algorithms for
analyses.
9
3. APPLICATION: MASSIVE DATA EXPLORATIONS
Data analysis usually begins with the importation of data into native data structures of a
statistical computing environment. In R, this is generally accomplished with functions such
as read.table, which have three limitations. First, the resulting data structures are limited
to available RAM. Second, R’s use of signed 32-bit integers for indexing limits vectors and
matrices to no more than 231−1 elements. Finally, these function can incur significant mem-
ory overhead. Together, these limitations have precluded many statisticians from exploring
large data using R’s native data structures.
Listing 1: Importing the airline data into a filebacked big.matrix object.
> library(bigmemory)
> x <- read.big.matrix("airline.csv", header=TRUE ,
+ backingfile="airline.bin",
+ descriptorfile="airline.desc",
+ type="integer")
> dim(x)
[1] 123534969 29
The bigmemory package offers an alternative approach, the read.big.matrix function.
The example in Listing 1 creates a filebacking for the Airline On-time Performance data
(http://stat-computing.org/dataexpo/2009/). This data set contains 29 records on each
of approximately 120 million commercial domestic flights in the United States from October
1987 to April 2008, and consumes approximately 12 GB of memory. The creation of the
filebacking avoids significant memory overhead and, as discussed in Section 2, the resulting
big.matrix may be larger than available RAM. Double-precision floating point values are
used for indexing, so a big.matrix may contain up to 252 elements (about 16 petabytes of
4-byte integer data, for example). The creation of the filebacking for the Airline On-time
Performance data takes about 15 minutes. In contrast, the use of R’s native read.csv
would require about 32 GB of RAM, beyond the resources of common hardware circa 2010
(all examples in this paper were run in 2010 using a quad-core (3.00 GHz) desktop personal
computer with four GB of RAM running Ubuntu Linux). The creation of a SQLite database
10
(SQLite database 2010) takes more than an hour and requires the availability and use of
separate database software.
Listing 2: Attaching to the airline data filebacking in a fresh R session.
> library(bigmemory)
> x <- attach.big.matrix("airline.desc")
> dim(x)
[1] 123534969 29
At the most basic level, the big.matrix is used in exactly the same way as a native
matrix via the bracket operators ("[" and "[<-") for extractions and assignments. One
notable difference is that the filebacking only needs to be created once (as shown in Listing
1). A subsequent R session may instantly reference, or attach to, this filebacking (as shown
in Listing 2). Subsets of data may be extracted and used with native R functions. In List-
ing 3, x[,"DepDelay"] creates an R vector of integer departure delays of length 123,534,969,
consuming about 600 megabytes (MB). Though non-trivial, this is handled on most modern
hardware without difficulty, leaving the researcher working with familiar native R commands
on such extracted subsets.
Listing 3: Exploring airline departure delays.
> summary(x[,"DepDelay"])
Min. 1st Qu. Median Mean
-1410.000 -2.000 0.000 8.171
3rd Qu. Max. NA’s
6.000 2601.0 2302136.0
A second example illustrates some important features relating to the performance of
the underlying mmap technology. Consider the task of first calculating the minimum value
of a variable, year, followed by the extraction of that same column from the data set.
R can ask a database to return the minimum value of year from an airline database
(called airline): from db("select min(year) from airline"). This takes about two
minutes. If, immediately afterwards, the entire year column is extracted into an R vector,
a <- from db("select year from airline"), this extraction also takes about two min-
11
utes. In contrast, colmin(x, "year") takes about eight seconds and a subsequent extrac-
tion, b <- x[,"year"], takes a mere two seconds. This example illustrates two important
points about the memory management solution implemented in bigmemory. First, the low-
level caching performance of bigmemory is an order of magnitude better than the database.
Second, the caching benefits from an awareness of the presence of the year data in RAM
following the calculation of the column minimum. The subsequent extraction takes place
entirely in RAM without any disk access.
4. FUNDAMENTAL CHALLENGES OF COMPUTING WITH MASSIVE DATA
When confronted with a fascinating real-data problem, statisticians quickly move beyond
basic summaries to more sophisticated explorations and formal analyses. Many such explo-
rations involve repeated analysis of subsets of data. This section discusses challenges of such
explorations with massive data. It also describes a new, flexible, easy-to-use framework for
concurrent programming which is useful for many typical applications and also has critical
advantages when working with massive data. Section 5 provides examples illustrating the
points developed here.
Initial massive data explorations often lead to formal analyses using well-established
methodology, but they could also lead to the development of new methodology. Some exist-
ing methods may be easily re-implemented with filebacked memory-mappings for use with
massive data. For example, the algorithm for k-means cluster analysis involves straightfor-
ward matrix calculations that are easily adapted to the data structures discussed earlier.
Other methods may demand substantively new algorithms or modifications to aspects of the
analysis. Miller (1992), for example, offers an iterative algorithm for linear and generalized
linear modeling that is suitable for the analysis of massive data. Similarly, Baglama and
Reichel (2005) presents a new algorithm for a truncated singular value decomposition which
may provide a sufficient alternative to a complete principal component analysis in many
applications. Our framework provides an effective platform for further development of tools
for the analysis of massive data.
12
There are several interrelated topics central to the exploration of massive data and the
development of new methodology. We begin by considering the general problem of con-
ducting repeated calculations on subsets of data, common in statistical explorations and
analyses. Many analyses can be accomplished by partitioning data (called the split), per-
forming a single calculation on each partition (the apply), and returning the results in a
specified format (the combine). The term “split-apply-combine” was coined by Wickham
(2009) but the approach has been supported on a number of different computing environ-
ments for some time under different names. SAS supports the approach through its by
statement. Google’s MapReduce (Dean and Ghemawat 2008) and Apache’s Hadoop (The
Apache Software Foundation 2011) software packages implement split-apply-combine opera-
tions for distributed systems. The design for a split-apply-combine framework has also been
explored for relational databases as described in Chen, Therber, Hsu, Zeller, Zhang and Wu
(2009). Each of these approaches attempts to simultaneously support data management and
analysis in a framework that lends itself to parallel computing. However, other computing
environments lack R’s statistical capabilities and their support for concurrent programming
is neither portable nor easy to exploit.
There are several benefits to the split-apply-combine approach that may not be obvious.
First, split-apply-combine is computationally efficient. It only requires two passes through
the data: one to create the partition and one to perform the analysis. Admittedly, storing
the partition groups from the split step requires extra memory. However, this overhead is
usually manageable. In contrast, a naive approach makes a costly pass through the data for
each group, adding an order of magnitude to the computational complexity of the analysis.
Second, the apply step is an ideal candidate for parallel computing. Because the split
defines a partition on the rows of a data set, calculations for each group are guaranteed to
be on disjoint subsets. Depending on the analysis, the apply step may be time consuming,
either because the number of groups is large or because the required calculation is intensive.
When the calculation is intensive, parallel computing techniques can dramatically reduce
the execution time.
13
Although the split-apply-combine approach provides an attractive opportunity for paral-
lel computing, the parallelization of algorithms has historically been cumbersome and suffers
from two serious drawbacks. First, it requires that the statistician re-implement existing
code, repeating the process of development, testing, and debugging. Second, there are a
plethora of different parallel mechanisms (also called a parallel backends), each with its own
unique interface. As a result, a different version of the parallel code is needed for each of the
parallel backends.
We introduce the foreach package to solve both of these historic difficulties with parallel
(or concurrent) programming. The goal is to allow the statistician to create concurrent
loops which can be used independently of the choice of a particular parallel backend. A
foreach loop can be run sequentially, in parallel on a single machine, or in parallel on a
cluster of machines without requiring any re-implementation or code modification of the
algorithm itself. Listing 4 illustrates this point, where G represents a partition of row indices
of data X into disjoint groups, f is some function implementing an analysis, and Y represents
(optional) auxiliary data. The example uses package doSNOW (Weston and REvolution
Computing 2009b) to enable the foreach loop to use the snow (Tierney, Rossini, Li and
Sevcikova 2006) parallel backend. The use of an alternate parallel backend requires no
changes to the algorithm designated as the “real work.”
Listing 4: Foreach framework.
# Optional declaration of a parallel backend here , such as
# the following for a 2-worker SNOW parallel environment:
library(foreach)
library(doSNOW)
cl <- makeCluster (2)
registerDoSNOW(cl)
# The real work:
ans <- foreach(g=G) %dopar% {
f(X[g,], Y)
}
14
Parallelizing the apply step comes with its own challenges because there is often con-
tention between data size and the number of parallel processes that can be utilized. If the
amount of data required by a parallel process is large, then extra time is usually required to
copy the data to the worker. In addition, data copies received by the worker consume valu-
able RAM. This limits the number of parallel workers that can be employed on a machine
as data copies eventually consume available RAM.
We illustrate these challenges with a simple example. We consider the calculation of 50%,
90%, and 99% quantiles of departure delays for each day of the week for only the 1995 flights
in the airline data set. Although this example only requires two of the 29 variables and is
thus somewhat contrived, many big-data problems would make use of the complete data and
the principle illustrated here provides a scalable solution. We use the split-apply-combine
approach and examine the consequences of parallel programming. First, the rows of the
data are split into groups by the day of the week. Next, quantiles are calculated for each
day. A data structure holding the 1995 data set consumes over 500 MB of RAM. Over two
GB of RAM is required when performing this calculation using four parallel processes. If
this exhausts available RAM, the operating system would try to compensate by “swapping”
inactively-used RAM to disk. However, swapping is inefficient and results in much longer
execution times. Thus, the memory overhead of a parallel worker effectively dictates the
total number of parallel processes that can be employed.
We address these challenges by providing the use of shared memory for the data struc-
tures described in the previous sections. This avoids both the time and memory overhead
of copying data to worker processes. Instead, a parallel worker receives a descriptor of a
big.matrix which allows for immediate access to the data via shared memory. Hence, the
framework implemented by bigmemory and foreach dramatically increases the ease of de-
velopment and efficiency of execution (both in terms of speed and memory consumption) for
parallel problems working on massive sets of data.
Figure 1 examines the memory usage of an algorithm for calculating the 1995 delay
quantiles, described above, when using a native R matrix and a shared big.matrix. A
15
copy of the data is required by each process with the native matrix, but this overhead is
completely avoided with the shared-memory data structure. Similarly, Figure 2 compares the
execution times of the quantile calculations. Parallel processing with traditional non-shared
matrices takes longer as the number of processors increases. In contrast, the execution time
is close to constant when additional processors are added using shared memory. In this
simple problem, the fixed communication overhead (independently of data size) is too large
for parallel programming to be practically useful. However, the comparison with respect to
the use of shared or non-shared data structures is illustrative. The next section shows a
more impressive example of gains from parallel programming when using shared memory.
[Figure 1 about here.]
[Figure 2 about here.]
5. APPLICATION: SPLIT-APPLY-COMBINE IN PRACTICE
We present a detailed example of the split-apply-combine approach for the problem of finding
quantiles of flight delays using all years of the airline data set. Even if the entire data set
were available as a 12 GB native R matrix in RAM (beyond typical capabilities in 2010), the
memory overhead of providing copies of the data to parallel processes would be prohibitively
expensive. Instead, we provide a scalable parallel solution making use of shared memory.
The function shown in Listing 5 calculates the desired quantiles for a subset of data
given by row indices rows. Listing 6 shows the sequence of steps in this scalable solution.
First, the filebacked airline data is attached (as described in Section 2 and illustrated in
Section 3). Second, a parallel environment is defined. This step could be omitted, in which
case the calculations would be executed sequentially on a single processor. Alternative
parallel backends are available and discussed below. Third, the split step of the split-apply-
combine approach divides the row indices into seven groups, one for each day of the week.
Finally, foreach is used to define the calculations required of each worker process. Most
significantly, neither the worker function GetDepQuantiles nor the body of the foreach
16
construct requires modification for sequential computation or use with alternative parallel
computing environments. The result of this toy exploration appears in Table 1, showing that
Tuesdays and Saturdays had less severe departure delays.
The snow package is not the only backend compatible with foreach; foreach provides
a framework for supporting new or alternative parallel backends. There are currently five
open-source packages that provide compatibility between various parallel environments and
foreach: doMC (Weston and REvolution Computing 2009a), doMPI (Weston 2010), doRe-
dis (Lewis 2010), doSMP (Weston, Wing and Revolution Analytics 2011), and doSNOW.
These implementations are freely available for use with the associated parallel environments
and other parallel backends exist outside of the open-source community.
Listing 5: The worker function for calculating the departure delay quantiles for a subset ofthe data.
GetDepQuantiles <- function(rows , data) {
return(quantile(data[rows , "DepDelay"],
probs=c(0.5, 0.9, 0.99),
na.rm=TRUE))
}
17
Listing 6: Calculating departure delay quantiles for the entire airline data set in parallelwith shared memory.
# Attach the filebacked airline data set:
library(bigmemory)
x <- attach.big.matrix("airline.desc")
# Optional declaration of a parallel backend here , such as
# the following for a 4-worker SNOW parallel environment:
library(foreach)
library(doSNOW)
cl <- makeSOCKcluster (4)
registerDoSNOW(cl)
# Obtain a "descriptor" of the shared -memory airline data , x:
xdesc <- describe(x)
# The "split" of row indices into seven groups ,
# one for each day:
G <- split (1: nrow(x), x[,"DayOfWeek"])
# The real work:
qs <- foreach(g=G) %dopar% {
# Provide access to shared -memory airline data:
require(bigmemory)
x <- attach.big.matrix(xdesc)
# Obtain the departure quantiles for the day indexed by g:
GetDepQuantiles(g, x)
}
[Table 1 about here.]
6. DISCUSSION
This paper presents a range of contributions in computing with massive data, shared memory,
and parallel programming. It focuses on the core contributions of R packages bigmemory and
foreach; other packages referenced in Section 2 offer additional features. Package biganalyt-
ics offers k-means cluster analysis as well as linear and generalized linear model support for
use with the massive data structures, for example. Package synchronicity supports advanced
18
process synchronization which is unnecessay for most parallel programming challenges. How-
ever, package NMF (Gaujoux 2010) relies upon synchronicity for conducting a non-negative
matrix factorization with a non-trivial parallel algorithm. Package bigalgebra is still in de-
velopment, but has been used to implement a massive partial singular value decomposition
as described in Baglama and Reichel (2005) and implemented in package irlba (Baglama
and Reichel 2009).
The Netflix Prize competition showed that the analysis of large and massive data is a
potientially fertile and largely unexplored area of statistics. Efron (2005) notes, “We have
entered an era of massive scientific data collection, with a demand for answers to large-scale
inference problems that lie beyond the scope of classical statistics.” We believe that “classical
statistics” should include “mainstream computational statistics.” Researchers were able to
engage the Netflix data challenge on modern hardware and make advances in predicting
ratings. However, the fact that many statisticians struggle to engage massive data research
opportunities shows that new computational techniques are essential to exploring this new
area of statistics. One contribution of our work is to level the playing field, making it more
likely that statisticians with diverse backgrounds and resources can all participate.
Ihaka and Temple Lang (2008) argue that although R has had a tremendous impact on
statistics, it should not be assumed that significant future development of the language is
a given. Major changes risk disrupting the user community, slowing the evolution of the
environment. At the same time, the nature of applied research in the field of statistics
continues to increase demand for advanced statistical computing capabilities, including the
analysis of massive data and flexible parallel programming. The R packages complementing
this paper offer a short-term solution to current size limitations of the R environment and
a simple, elegant framework for portable parallel computing. However, these solutions also
point towards a general design for scalable statistical computing that could be used in future
statistical computing environments or a future major revision of R itself.
19
REFERENCES
Baglama, J., and Reichel, L. (2005), “Augmented Implicitly Restarted Lanczos Bidiagonal-
ization Methods,” SIAM Journal of Scientific Computing, 27, 19–42.
Baglama, J., and Reichel, L. (2009), irlba: Fast partial SVD by implicitly-restarted Lanczos
bidiagonalization. R package version 0.1.1, implemented by Bryan W. Lewis, available
at http://www.rforge.net/irlba/.
Bennet, J., and Lanning, S. (2007). “The Netflix Prize,” Proceedings of KDD Cup
and Workshop, San Jose, California. Available at http://www.cs.uic.edu/~liub/
KDD-cup-2007/NetflixPrize-description.pdf.
Boost (2010). The Boost C++ Libraries. Available at http://www.boost.org.
Chen, Q., Therber, A., Hsu, M., Zeller, H., Zhang, B., and Wu, R. (2009), “Efficiently
support MapReduce-like computation models inside parallel DBMS,” Proceedings of
the 2009 International Database Engineering and Applications Symposium, pp. 43–53.
ISBN 978-1-60558-402-7.
Dean, J., and Ghemawat, J. (2008), “MapReduce: simplified data processing on large clus-
ters,” Communications of the ACM - 50th anniversary issue, 51(1).
Efron, B. (2005), “Bayesians, Frequentists, and Scientists,” Journal of the American Stistical
Association, 100, 1–5.
Emerson, J. W., and Kane, M. J. (2010), biganalytics: A library of utilities for big.matrix
objects of package bigmemory. R package version 1.0.14, available at http://cran.
r-project.org/package=biganalytics.
Gaujoux, R. (2010), NMF: Algorithms and framework for Nonnegative Matrix Factorization
(NMF). R package version 0.4.8, available at http://CRAN.R-project.org/package=
NMF.
20
Ihaka, R., and Temple Lang, D. (2008), “Back to the Future: Lisp as a Base for a Statistical
Computing System,” Proceedings in Computational Statistics, pp. 21–33. ISBN 978-3-
7908-2083-6.
Kane, M. J. (2010), synchronicity: Boost mutex functionality for R. R package version 1.0.9,
available at http://cran.r-project.org/package=synchronicity.
Kane, M. J., and Emerson, J. W. (2010a), bigmemory: Manage massive matrices with shared
memory and memory-mapped files. R package version 4.2.3, available at http://cran.
r-project.org/package=bigmemory.
Kane, M. J., and Emerson, J. W. (2010b), bigtabulate: table-, tapply-, and split-like func-
tionality for matrix and big.matrix objects. R package version 1.0.13, available at
http://cran.r-project.org/package=bigtabulate.
Kane, M. J., Lewis, B. W., and Emerson, J. W. (2010), bigalgebra: BLAS and LAPACK
routines for native R matrices and big.matrix objects. R package version 0.6, available
at https://r-forge.r-project.org/R/?group_id=556.
Kath, R. (1993). Managing Memory-Mapped Files. Available at http://msdn.microsoft.
com/en-us/library/ms810613.aspx.
Lewis, B. W. (2010), doRedis: Foreach parallel adapter for the rredis package. R package
version 1.0.1, available at http://CRAN.R-project.org/package=doRedis.
Lumley, T. (2009), biglm: bounded memory linear and generalized linear models. R package
version 0.7, available at http://cran.r-project.org/package=biglm.
Miller, A. J. (1992), “Least Squares Routines to Supplement those of Gentlemen,” Applied
Statistics, 41(2), 458–478. Algorithm AS 274, available at http://www.jstor.org/
stable/2347583.
21
R Development Core Team (2010a), “R Installation and Administration,”. ISBN 3-900051-
09-7, available at http://cran.r-project.org/doc/manuals/R-admin.html.
R Development Core Team (2010b), R: A Language and Environment for Statistical Com-
puting, Vienna, Austria: R Foundation for Statistical Computing. ISBN 3-900051-07-0,
available at http://www.R-project.org.
SAS Institute Inc. (2011). SAS R© software, Cary, NC. http://www.sas.com.
SQLite database (2010). The SQLite database engine, available at http://www.sqlite.org.
The Apache Software Foundation (2011). Apache Hadoop, available at http://hadoop.
apache.org.
The MathWorks, Inc. (2011). MATLAB R© software. http://www.mathwords.com.
Tierney, L., Rossini, A. J., Li, N., and Sevcikova, H. (2006), snow: Simple Network of
Workstations. R package version 0.3-3, available at cran.r-project.org/package=
snow.
Weston, S. B. (2010), doMPI: Foreach parallel adaptor for the Rmpi package. R package
version 0.1-5, available at http://CRAN.R-project.org/package=doMPI.
Weston, S. B., and REvolution Computing (2009a), doMC: Foreach parallel adaptor for the
multicore package. R package version 1.2.0, available at http://CRAN.R-project.org/
package=doMC.
Weston, S. B., and REvolution Computing (2009b), doSNOW: Foreach parallel adaptor for
the snow package. R package version 1.0.3, available at http://CRAN.R-project.org/
package=doSNOW.
Weston, S. B., and REvolution Computing (2009c), foreach: Foreach looping construct for R.
R package version 1.3.0, available at http://CRAN.R-project.org/package=foreach.
22
Weston, S. B., Wing, J., and Revolution Analytics (2011), doSMP: Foreach parallel adaptor
for the revoIPC package. R package version 1.0-1, available at https://r-forge.
r-project.org/projects/dosmp/.
Wickham, H. (2009). “The split-apply-combine strategy for data analysis,” submitted to
the Journal of Computation and Graphical Statistics, available at http://had.co.nz/
plyr/plyr-intro-090510.pdf.
23
List of Figures
1 The memory usage in the delay quantile calculation using a native R matrix
compared to a big.matrix. . . . . . . . . . . . . . . . . . . . . . . . . . . . 252 The execution time for the delay quantile calculation using a native R matrix
compared to a big.matrix. . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
24
Figure 1: The memory usage in the delay quantile calculation using a native R matrix
compared to a big.matrix.
25
Figure 2: The execution time for the delay quantile calculation using a native R matrix
compared to a big.matrix.
26
List of Tables
1 Quantiles of departure delays. . . . . . . . . . . . . . . . . . . . . . . . . . . 28
27
Table 1: Quantiles of departure delays.
Percentile Mon Tues Wed Thu Fri Sat Sun50% 0 0 0 0 0 0 090% 25 23 25 30 33 23 2799% 127 119 125 136 141 116 130
28