Gpars workshop
-
Upload
vaclav-pech -
Category
Documents
-
view
722 -
download
0
description
Transcript of Gpars workshop
![Page 1: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/1.jpg)
GPars workshop
Václav Pechhttp://jroller.com/vaclav
Russel Winderhttp://www.russel.org.uk
![Page 2: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/2.jpg)
Agenda
● Threads, thread pools, tasks● Agents● Fork/Join● Parallel collections● Dataflow● Actors
http://gpars.org/GPars_workshop.zip
![Page 3: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/3.jpg)
We’re all in the parallel computing business!
![Page 4: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/4.jpg)
Multithreaded programs today work mostly by accident!
![Page 5: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/5.jpg)
Map/ReduceMap/Reduce
Fork/JoinFork/Join
ActorsActors
STMSTM
DataflowDataflow
AgentAgent
Can we do better?
![Page 6: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/6.jpg)
Part 1
Thread management
![Page 7: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/7.jpg)
Asynchronous invocation
Future f = threadPool.submit(calculation);
…
System.out.println(“Result: “ + f.get());
![Page 8: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/8.jpg)
Async the Groovy way
def group = new NonDaemonPGroup(10)
group.task {
calculation.process()
}
![Page 9: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/9.jpg)
Part 2
Agents
![Page 10: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/10.jpg)
Shared Mutable State
Misused most of the time
When really needed, use Agents Software Transactional Memory Locks
![Page 11: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/11.jpg)
Agent inside
Double IncrementAdd 25
Message Queue
36 thread
![Page 12: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/12.jpg)
Agent List
def jugMembers = new Agent(['Me']) //add Me
task { jugMembers.send {it.add 'Joe'} //add Joe}
task { jugMembers << {it.add 'Dave'} //add Dave jugMembers << {it.add 'Alice'} //add Alice}...println jugMembers.val
![Page 13: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/13.jpg)
Part 3
Fork/Join
![Page 14: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/14.jpg)
Thread Pool
Tasks
Worker threads
Queue
![Page 15: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/15.jpg)
Thread Pool
Contention!
![Page 16: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/16.jpg)
Fork/Join Thread Pool
![Page 17: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/17.jpg)
Fork/Join Thread Pool
Work stealing
![Page 18: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/18.jpg)
Fork/Join
Solve hierarchical problems
Divide and conquer
Merge sort, Quick sort Tree traversal File scan / search …
[a, b, c, d, e, f, g, h]
[a, b, c, d] [e, f, g, h]
[a, b] [c, d] [e, f] [g, h]
![Page 19: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/19.jpg)
Fork/Join (GPars)
runForkJoin(new File(“./src”)) {currentDir -> long count = 0; currentDir.eachFile { if (it.isDirectory()) { forkOffChild it } else { count++ } } return count + childrenResults.sum(0) }
Waits for children without blocking the thread!
![Page 20: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/20.jpg)
Part 4
Parallel collections
![Page 21: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/21.jpg)
Parallel collections
images.eachParallel {it.process()}
documents.sumParallel()
candidates.maxParallel {it.salary}.marry()
![Page 22: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/22.jpg)
Parallel collections
registrations = submissions
.collectParallel { form -> form.process()}
.findAllParallel { it.valid }
registrations = submissions.parallel
.map { form -> form.process()}
.filter { it.valid }.collection
![Page 23: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/23.jpg)
GPU
Java watch list:https://github.com/pcpratts/rootbeer1/
http://openjdk.java.net/projects/sumatra/
![Page 24: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/24.jpg)
Part 5
Dataflow
![Page 25: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/25.jpg)
Composing async functions
int hash1 = hash(download('http://www.gpars.org'))
int hash2 = hash(loadFile('/gpars/website/index.html'))
boolean result = compare(hash1, hash2)
println result
![Page 26: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/26.jpg)
Composing async functions
@AsyncFun hash = oldHash
@AsyncFun compare = oldCompare
@AsyncFun download = oldDownload
@AsyncFun loadFile = oldLoadFile
def hash1 = hash(download('http://www.gpars.org'))
def hash2 = hash(loadFile('/gpars/website/index.html'))
def result = compare(hash1, hash2)
println result.get()
![Page 27: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/27.jpg)
Composing async functions
@AsyncFun hash = oldHash
@AsyncFun(blocking = true) compare = oldCompare
@AsyncFun download = oldDownload
@AsyncFun loadFile = oldLoadFile
def hash1 = hash(download('http://www.gpars.org'))
def hash2 = hash(loadFile('/gpars/website/index.html'))
boolean result = compare(hash1, hash2)
println result
![Page 28: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/28.jpg)
int hash(String text) {…}
Promise<int> hash(Promise<String> | String text)
![Page 29: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/29.jpg)
int hash(String text) {…}
Promise<int> hash(Promise<String> | String text)
compare(
hash( download() ),
hash( loadFile() )
)
![Page 30: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/30.jpg)
int hash(String text) {…}
Promise<int> hash(Promise<String> | String text) {
1. Return a Promise for the result
2. Wait (non-blocking) for the text param
3. Call the original hash()
4. Bind the result
}
![Page 31: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/31.jpg)
Composing async functions
Combine functions as usual
Parallelism is detected automatically
![Page 32: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/32.jpg)
Composing async functions
@AsyncFun hash = oldHash
@AsyncFun compare = oldCompare
@AsyncFun download = oldDownload
@AsyncFun loadFile = oldLoadFile
def hash1 = hash(download('http://www.gpars.org'))
def hash2 = hash(loadFile('/gpars/website/index.html'))
def result = compare(hash1, hash2)
println result.get()
![Page 33: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/33.jpg)
Promise chaining
@AsyncFun download = oldDownload
@AsyncFun loadFile = oldLoadFile
def h1 = download('http://www.gpars.org') then hash
def h2 = loadFile('/gpars/website/index.html') then hash
whenAllBound([h1, h2], compare) then {println it}
![Page 34: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/34.jpg)
Promise chaining
@AsyncFun download = oldDownload
@AsyncFun loadFile = oldLoadFile
whenAllBound ([
download('http://www.gpars.org') >> hash,
loadFile('/gpars/website/index.html') >> hash
], compare) >> {println it}
![Page 35: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/35.jpg)
Dataflow Concurrency
No race-conditions No live-locks Deterministic deadlocks
Completely deterministic programs
BEAUTIFUL code (Jonas Bonér)
![Page 36: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/36.jpg)
Dataflow Variables / Promisesmain task2 task3
x
yz
task1
![Page 37: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/37.jpg)
Dataflow Variables / Promisestask2
x
task1
![Page 38: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/38.jpg)
Promises to exchange data
task { z << x.val + y.val }
task { x << 10 }
task {
println ”I am task 3”
y << 5
}
assert 15 == z.val
![Page 39: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/39.jpg)
Dataflow Variables / Promisestask2
x
task1
![Page 40: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/40.jpg)
Dataflow Channelstask2
x|y|z
task1
![Page 41: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/41.jpg)
Synchronous Channelstask2
x|y|z
task1
![Page 42: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/42.jpg)
operator(inputs: [headers, bodies, footers],
outputs: [articles, summaries])
{header, body, footer ->
def article = buildArticle(header, body, footer)
bindOutput(0, article)
bindOutput(1, buildSummary(article))
} *
+
<>
Dataflow Operators
![Page 43: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/43.jpg)
Url resolverUrl resolver
Downloader
Groovy scanner Scala scanner
Url resolverUrl resolver
Reporter
Speculator
Evaluator
Splitter
Confirm
Cache updates
Approvals
Dataflow Operators
![Page 44: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/44.jpg)
Part 6
Actors
![Page 45: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/45.jpg)
Actors
Processes with a mail-box
Share no data
Communicate by sending messages
Use a thread-pool
![Page 46: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/46.jpg)
Actors
Isolated Communicating
Immutable messages Active
Pooled shared threads Activities
Create a new actorSend a messageReceive a message
ActorActorActorActorActorActorActor
TTT
Thread pool
![Page 47: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/47.jpg)
Actors use
GateKeeper
Form
HTTP
FingerPrints
AddressCheck
EmailCheck
Process
FraudDetect
Response
SOAP
SMTP
![Page 48: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/48.jpg)
Sending messages
buddy.send 10.eur
buddy << new Book(title:’Groovy Recipes’, author:’Scott Davis’)
def canChat = buddy.sendAndWait ‘Got time?’
buddy.sendAndContinue ‘Need money!’, {cash-> pocket.add cash}
![Page 49: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/49.jpg)
Stateless Actors (pure Java)class MyActor extends DynamicDispatchActor { Account account = ... public void onMessage(String msg) { String encripted = encrypt(msg); reply(encripted); } public void onMessage(Integer number) { reply(2 * number); }
public void onMessage(Money cash) { System.out.println("Received a donation " + cash); account.deposit(cash); }}
![Page 50: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/50.jpg)
Stateful Actors
class MyActor extends DefaultActor {
void act() {
def buddy = new YourActor()
buddy << ‘Hi man, how\’re things?’
def response = receive()
}
}
![Page 51: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/51.jpg)
Implicit State in Actors
val me = actor { react {msg1 -> switch (msg1) { case Work: reply “I don't work so early” ; stop(); case Breakfast: msg1.eat() react {msg2 → switch (msg2) { case Work: reply “OK, time to work”; msg2.do() case Lunch: ...} } } }
![Page 52: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/52.jpg)
Continuation Style
loop { … react { … react {/*schedule the block; throw CONTINUE*/ … } //Never reached } //Never reached}//Never reached
![Page 53: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/53.jpg)
Active Objects
@ActiveObject
class MyCounter {
private int counter = 0
@ActiveMethod
def incrementBy(int value) {
println "Received an integer: $value"
this.counter += value
}
}
![Page 54: Gpars workshop](https://reader034.fdocuments.net/reader034/viewer/2022051609/547c5bb6b379596a2b8b4ff2/html5/thumbnails/54.jpg)
'coz concurrency is Groovy