Parallel Processing

LAB NO 1

Parallel Processing

• Ability to carry out multiple operations/tasks at the same time using more than one processors or processor cores

• Differs from multitasking or multiprogramming in which a single CPU executes several programs at the same time

Distributed Computing

• Multiple autonomous computers that communicate through a computer network to achieve a common goal

• In parallel computing, all processors have access to shared memory, which can be used to exchange information between processors

• In distributed computing, each processor has its own private memory (distributed memory); info can be exchanged by passing messages between processors

Cluster Computing

• Computer cluster is a set of loosely connected computers that work together so that they can be viewed as a single, highly available system

• Used for either load balancing or high availability

Grid Computing

• Applying resources of many computers (from multiple administrative domains) in a network to a single problem at the same time

• Although a grid can be dedicated to a specialized application, it is more common to use a single grid for a variety of different purposes

Cluster vs Grid

• Homogeneous system vs heterogeneous system• Tight coupling vs loose coupling• Grids are inherently distributed and spread over LANs, Man

or WAN• Single system image vs independent, autonoumous nodes• In a cluster, entire system behaves like a single system with

resources managed by a centralized resource manager• In a grid, every node is autonomous, has its own resource

manager and behaves like an independent entity.

Parallel Computing MemoryArchitectures

• Shared Memory• Can be simultaneously accessed by multiple processes• Programs may run on single or multiple processors• In multi-processor systems, large block of RAM can be

accessed by several CPUs• Distributed Memory• Each processor in a multi-processor computer system

has its own private memory• MPI or socket programming are main tools used for

distributed memory environment

MPI- Message PassingInterface

• A library of functions or an API• Allows communication between processors in

a distributed memory architecture• Allows portability• Uses either a C or a FORTRAN compiler

MPI Data TypesMPI datatype handle C datatype

MPI_INT int

MPI_SHORT short

MPI_LONG long

MPI_FLOAT float

MPI_DOUBLE double

MPI_CHAR char

MPI_BYTE unsigned char

MPI Program

# include <stdio.h># include <mpi.h>main(int argc, char **argv) {//codingMPI_Init(&argc, &argv);//codingMPI_Finalize();//coding}

MPI Init

NetworkNetwork

OO44

33 22

11

MPI Finalize

NetworkNetwork

OO44

33 22

11

Compiling and running

• mpicc –o <executable file name> <source file name>

• mpirun <executable file name>• mpirun –np <no> <executable file name>