Region-Based Software Distributed Shared Memory

Song Li, Yu Lin, and Michael Walker

CS 656 -- Operating Systems

May 1, 2000

Outline

• Motivation

• Problems

• DSM Models

• Proposed Solution

• Conclusion

Motivation

• Share distributed resources to increase computing power– First Solution: Tightly-coupled system– Problem: Central memory bus– Alternative: Loosely-coupled system– New Problem: Need IPC in distributed systems

Motivation: RPC

• Solution: Remote Procedure Calls (RPC)– Snd/Rcv protocol– Pass-by-value– OS Assignment 1

• Problems with RPC– Explicit awareness of communication– Marshalling complex data structures is hard

Motivation: DSM

• Solution: Distributed Shared Memory (DSM) [Li86--Yale]– Idea: Nonresident pages are fetched from network

– Illusion of tightly-coupled system in a loosely-coupled one

– Abstraction on top of message-passing model– Programmers use memory-accessing paradigm

DSM: Potential Benefits

• Shared access to memory

• Avoids Von-Neumann bottleneck

• Familiar abstraction

• Pass-by-reference

• Spreads communication load over time

• Provides more memory than is locally available

Problems

• Who handles remote access?

• What should be shared?

• Cache, cache, cache…but how?

• Page Replacement & Thrashing

• More than one way to solve the problems...

DSM Models: Decisions

• Hardware: MMU controls message passing, data migration, and caching

• Software: Control managed by OS or library

• Page-based shared memory versus shared variable versus Object

• Cache consistency models

Proposed Solution: Overview

• Software-based model using libraries

• Variable-size regions -- similar to segments– Every granularity has its advantages/drawbacks– Make the region size flexible, provide

reasonable defaults (sound familiar?)

• Multiple reader, single writer (MRSW) caching support

Solution: User Interface

• r_handle sm_malloc(size);• int sm_regionat(r_handle, attrib);• int sm_read(r_handle, offset, buf, size);

• int sm_write(r_handle, offset, buf, size);

• int sm_regiondt(r_handle);• int sm_free(r_handle);

Solution: Mode of Operation

...

...Provider 1

Shared Memory

ManagerMaster Table

Client 1

Cache

Client Table

Client 2

Cache

Client Table

Client n

Cache

Client Table

Provider 2

Shared Memory

Provider n

Shared Memory

Solution: Mode of Operation

...

...Provider 1

Shared Memory

ManagerMaster Table

Client 1

Cache

Client Table

Client 2

Cache

Client Table

Client n

Cache

Client Table

Provider 2

Shared Memory

Provider n

Shared Memory

(1) Allocate memory

Solution: Mode of Operation

...

...Provider 1

Shared Memory

ManagerMaster Table

Client 1

Cache

Client Table

Client 2

Cache

Client Table

Client n

Cache

Client Table

Provider 2

Shared Memory

Provider n

Shared Memory

(2) Query table & attach region

Solution: Mode of Operation

...

...Provider 1

Shared Memory

ManagerMaster Table

Client 1

Cache

Client Table

Client 2

Cache

Client Table

Client n

Cache

Client Table

Provider 2

Shared Memory

Provider n

Shared Memory

(3) Send region info to client

Solution: Mode of Operation

...

...Provider 1

Shared Memory

ManagerMaster Table

Client 1

Cache

Client Table

Client 2

Cache

Client Table

Client n

Cache

Client Table

Provider 2

Shared Memory

Provider n

Shared Memory

(4) Client communicates with provider

Solution: Mode of Operation

...

...Provider 1

Shared Memory

ManagerMaster Table

Client 1

Cache

Client Table

Client 2

Cache

Client Table

Client n

Cache

Client Table

Provider 2

Shared Memory

Provider n

Shared Memory

(5) Client uses cached copy

Solution: Mode of Operation

...

...Provider 1

Shared Memory

ManagerMaster Table

Client 1

Cache

Client Table

Client 2

Cache

Client Table

Client n

Cache

Client Table

Provider 2

Shared Memory

Provider n

Shared Memory

(6) Client detaches and frees memory

Solution: Mode of Operation

...

...Provider 1

Shared Memory

ManagerMaster Table

Client 1

Cache

Client Table

Client 2

Cache

Client Table

Client n

Cache

Client Table

Provider 2

Shared Memory

Provider n

Shared Memory

(7) Memory is freed

Solution: MRSW Caching Model

• Replication is good for reads, so allow MR– No communication overhead

• Writer requires cache invalidation, so restrict to SW– MW is too complicated

Solution: MRSW Caching Model

...

...Provider 1

Shared Memory

ManagerMaster Table

Client 1

Cache

Client Table

Client 2

Cache

Client Table

Client n

Cache

Client Table

Provider 2

Shared Memory

Provider n

Shared Memory

Readers

(1) Multiple readers

Solution: MRSW Caching Model

...

...Provider 1

Shared Memory

ManagerMaster Table

Client 1

Cache

Client Table

Client 2

Cache

Client Table

Client n

Cache

Client Table

Provider 2

Shared Memory

Provider n

Shared Memory

Readers Writer

?

(2) Write request

Solution: MRSW Caching Model

...

...Provider 1

Shared Memory

ManagerMaster Table

Client 1

Cache

Client Table

Client 2

Cache

Client Table

Client n

Cache

Client Table

Provider 2

Shared Memory

Provider n

Shared Memory

Readers Writer

(3) Cache invalidation to selected clients only

Solution: MRSW Caching Model

...

...Provider 1

Shared Memory

ManagerMaster Table

Client 1

Cache

Client Table

Client 2

Cache

Client Table

Client n

Cache

Client Table

Provider 2

Shared Memory

Provider n

Shared Memory

Readers Writer

(4) Single writer, multiple readers

Conclusion: Current Status

• Basic client/manager model is implemented

Conclusion: Remaining Work

• Page replacement strategy: LRU

• Region protection mechanism

• Replicated managers for better performance and fault tolerance

• Evaluation...

Conclusion: Evaluation

• Find distributed apps to use DSM model

• Performance evaluation:– Test apps with DSM model and message-

passing model– Desired results: DSM performance is better