Parallel Computing with Mathematica UVACSE Short Course...Parallel Computing with Mathematica UVACSE...

Parallel Computing with MathematicaUVACSE Short Course

E Hall1

1University of Virginia Alliance for Computational Science and [email protected]

October 8, 2014

(UVACSE) October 8, 2014 1 / 46

Outline

1 NX Client for Remote Linux Desktop

2 Parallel Computing with Mathematica

3 Parallel Computing on the Linux Cluster

4 References


NX Client for Remote Linux Desktop

Installing and Configuring NX Client

The NX client provides a Gnome Linux desktop interface to the loginnode of the fir.itc Linux cluster.

http://uvacse.virginia.edu/resources/itc-linux-cluster-overview/the-nx-client/






Starting and Configuring NX Client

Once logged into fir.itc.virginia.edu through NX

Open a terminal from Applications/System Tools/Terminal menuSelect and right-click on Terminal to add to launcher

Create a mathematica directory with mkdir commandStart web browser from icon at top of desktop



Download Short Course Examples

Download the short-course materials fromhttp://www.uvacse.virginia.edu/software/mathematica-at-uva/

Follow the links,→Parallel Mathematica→Parallel Mathematica Short Courseand download 3 files to mathematica directory you create withmkdir command

ClassExamples_Fa14.zipmathematica-parallel_Fa14.pdf


http://www.uvacse.virginia.edu/software/mathematica-at-uva/

Parallel Computing with Mathematica

Solving Big Technical Problems

Computationally intensive, long-running codes

Run tasks in parallel on many processorsTask parallel

Large Data sets

Load data across multiple machines that work in parallelData parallel




Parallel computing in Mathematica is based on launching andcontrolling multiple Mathematica kernel (worker) processes from withina single master Mathematica, providing a distributed-memoryenvironment for parallel programming.

Tested on Unix, Linux, Windows, and Macintosh platforms and are wellsuited to working locally on a multi-core machine or in a cluster ofmachines,

Parallel computing is now provided as a standard part of Mathematica.




To perform computations in parallel, you need to be able to perform thefollowing tasks:

start processes and wait for processes to finishschedule processes on available processorsexchange data between processes and synchronize access tocommon resources

In the Mathematica environment, the term processor refers to arunning Mathematica kernel, whereas a process is an expression to beevaluated.



Connection Methods

Mathematica can run parallel workers locally on the same machine orremotely on a compute cluster controlled by a resource managementapplication, e.g. PBSPro.

Local KernelsThe Local Kernels connection method is used to run parallelworkers on the same computer as the master Mathematica. It issuitable for a multi-core environment, and is the easiest way to getup and running with parallel computation.

Cluster IntegrationThe Cluster Integration connection method is used to run parallelworkers on a compute cluster from the master Mathematicaprocess. It integrates with the PBSPro cluster managementsoftware.



Parallel Computing Functions

mathematica/guide/ParallelComputing.html


http://reference.wolfram.com/mathematica/guide/ParallelComputing.html


Parallel Computing Features

Mathematica supports both task and data parallelism.

The main features of parallel computing in Mathematica are:

distributed memory, master/slave parallelismwritten in Mathematica, machine independentMathLink communication with remote kernelsexchange of symbolic expressions and programs with remotekernels, not only numbers and arraysvirtual process scheduling or explicit process distribution toavailable processorsvirtual shared memory, synchronization, lockingparallel functional programming and automatic parallelizationsupport



Programming Parallel Applications in Mathematica

To demonstrate demonstrate parallel processing in Mathematica, weuse ParallelEvaluate. If this is the first parallel computation, it willlaunch the configured parallel kernels.



Parallel Kernels Status Monitor

You can open the Parallel Kernels Status monitor through theEvaluation > Parallel Kernel Status menu selection.



Parallel Preferences

The default settings of Mathematica automatically configure a numberof parallel kernels to use for parallel computation, as seen through theEvaluation > Parallel Kernel Configuration menu selection.



Parallel Preferences



Launching and Connecting

Mathematica launches parallel kernels automatically as they areneeded, but you can also launch kernels manually with the commandLaunchKernels. This is be useful if you were running in a batchmode.

Local Kernels

The Local Kernels connection method supports launching localkernels directly from LaunchKernels by passing it an integer(setting the number of kernels)




Cluster Integration configuration done through Parallel Preferences




Cluster Integration

The Cluster Integration connection method is used to run parallelworkers on the compute nodes of a cluster from the masterMathematica.

The Cluster Integration connection method supports launchingkernels directly from LaunchKernels. To do so you must firstload the ClusterIntegration‘ package, this is shown below.

To launch on a particular cluster you have to pass the name forthat cluster into LaunchKernels.




Cluster Integration



Sending Commands to Remote Kernels

Values of variables defined on the local master kernel are usually notavailable to remote kernels.

If you need variable values and definitions carried over to the remotekernels, use DistributeDefinitions or shared variables.




Recall that connections to remote kernels, as opened byLaunchKernels, are represented as kernel objects. The commandsbelow take parallel kernels as arguments and use them to carry outcomputations.



Remote Definitions

Mathematica contains a command DistributeDefinitions thatmakes it easy to transport local variables and function definitions to allparallel kernels.

Higher-level parallel commands, such as Parallelize,ParallelTable, ParallelSum, ... will automatically distributedefinitions of symbols occurring in their arguments.



Automatically Parallelizing Existing Serial Expressions

Use Parallelize to have Mathematica decide how to distribute workacross multiple kernels.



Automatically Parallelizing Existing Serial Expressions

There is a natural trade-off in parallelization between controlling theoverhead of splitting a problem or keeping all the cores busy.



Sending a Command to Multiple Kernels

Use ParallelEvaluate to send commands to multiple kernels andwait for completion. Use With to bind locally defined variables beforedistribution.



Implementing Task-Parallel algorithms



Implementing Data-Parallel algorithms

ParallelMap is a natural way to introduce parallelism using afunctional programming style. When you have a computationallyexpensive function to execute of a large data set, Mathematica canexecute the operations in parallel by splitting the mapping amongmultiple kernels.



Decomposing a Problem in Parallel Data Sets

Estimating Pi Using Monte Carlo Simulation: Serial and Parallel.




Graphic of work distribution among the 4 kernels.




You can generate multiple data sets in parallel, then plot or processthem further.



Choosing an Appropriate Distribution Method

The parallel primitives Parallelize, ParallelMap,ParallelTable, ParallelDo, ParallelSum, andParallelCombine support an option called Method

It allows you to specify the granularity of the subdivisions used todistribute the computation across kernels.

Use Method → "FinestGrained" when the completion time ofeach atomic unit of computation is expected to vary widely.

Use Method → "CoarseGrained" when the completion time ofeach atomic unit of computation is expected to be uniform.



Virtual Shared Memory

Virtual shared memory is a programming model that allows processorson a distributed-memory machine to be programmed as if they hadshared memory. A software layer takes care of the necessarycommunication in a transparent way.

Mathematica provides functions that implement virtual shared memoryfor these remote kernels.

The drawback of a shared variable is that every access for read orwrite requires communication over the network, so it is slower thanaccess to a local unshared variable.

Declaring shared variables and functions.



Mathematica Parallel Workflow

The toolbox enables application prototyping on the desktop with up to16 local workers (left), and with the Mathematica Cluster Integrationpackage(right), applications can be scaled to multiple computers on acluster (subsitute Mathematica for Matlab in figure below).


Parallel Computing on the Linux Cluster

Scaling Up from the Desktop

Mathematica’s parallel computing provides the ability to use up to 16local kernels on a multicore or multiprocessor computer.

When used together with Cluster Integration package, you can scaleup your application to use any number of kernels running on anynumber of computers.

ITS Linux cluster allows for 128 kernels.

Alternatively, you can run up to 16 kernels on a single multi-corecompute node of the cluster.



Running Mathematica on Cluster Front-end Node

Mathematica Parallel Computing jobs can be submitted to the ITCLinux cluster by first logging onto the cluster front-end nodefir.itc.virginia.edu using the NX client.

Start up Mathematica from a Linux desktop terminal window.

Parallel Mathematica jobs can be submitted from with the Mathematicanotebook interface as well as using PBS command files and theexample scripts show how to setup and submit the jobs

Documentation: Submitting Mathematica Parallel Jobs


https://collab.itc.virginia.edu/access/content/group/3a39a42b-2e4b-4cd2-b59f-6185ddc04136/Parallel%20Mathematica/Parallel%20Mathematica%20Short%20Course


Example Mathematica Scripts


References

References

1 Parallel Computing Tools User Guidereference.wolfram.com/mathematica/ParallelTools/tutorial/Overview.html

2 Parallel Computing: Mathematica Documentationreference.wolfram.com/mathematica/guide/ParallelComputing.html

3 Mathematica Cookbook, by Sal Mangano, O’Reilly Press.

Need further help? Email [email protected].


http://reference.wolfram.com/mathematica/ParallelTools/tutorial/Overview.html





Parallel Computing with Mathematica UVACSE Short Course...Parallel Computing with Mathematica UVACSE...

Documents

Transcript of Parallel Computing with Mathematica UVACSE Short Course...Parallel Computing with Mathematica UVACSE...