Facilitando a Programação concorrente com o Fork/Join

FACILITANDO A PROGRAMAÇÃO

CONCORRENTE COM O FORK/JOIN

Mário Amaral@mario_fts

Thursday, December 6, 12

QUE MÁRIO?


QUE MÁRIO?

•Desenvolvedor Java desde 2004


QUE MÁRIO?


• Instrutor na Caelum


QUE MÁRIO?



• Caelum-Stella


QUE MÁRIO?



• Caelum-Stella

• Personagem de games desde 1981


VAMOS FALAR DE PARALELISMO


POR QUE PARALELIZAR

• Ganhar tempo


POR QUE PARALELIZAR

• Aproveitar recursos ociosos


O QUE PARALELIZAR?


O QUE PARALELIZAR?

• Tarefas Independentes com processamento pesado


O QUE PARALELIZAR?


• Processar mensagens de uma fila


O QUE PARALELIZAR?



• Conversão de dados


O QUE PARALELIZAR?




• Aplicar filtros em imagens


O PROBLEMA


O PROBLEMA


•Dados de vendas


O PROBLEMA


•Dados de vendas

• CSV => JSON


OS DADOS


OS DADOS • 1;1;04122012;fichario preto;9;33.55


OS DADOS • 1;1;04122012;fichario preto;9;33.55

{ "number": 1, "terminal": 1, "description": "fichario preto", "price": 33.55, "date": "04122012"}


PASSO A PASSO

• Ler os dados do CSV e converter para objetos.

• Para cada objeto, converter em Json.

• Adicionar o Json em uma lista de String.

• Gravar o conteúdo da lista em um arquivo.


SINGLE THREAD

• Processar tudo sequencialmente em um loop

• Implementação mais simples


CÓDIGO....

public List<String> toJson(List<?> list) { List<String> lines = new ArrayList<>(); for (Object value : list) { lines.add(new JsonObject(value)

.toString()); } return lines; }


TESTANDO

• 1 milhão de registros

• Executar 10 vezes seguidas cada método.

•Desconsiderar o menor e o maior tempo.

•Obter o tempo médio.


TEMPO

Método Tempo Médio (ms)

Single 5562


MULTI THREAD


MULTI THREAD

• Cada conversão é independente da outra.


MULTI THREAD


• Múltiplos processadores


MULTI THREAD


• Múltiplos processadores

•Dividir e distribuir!


threads.add(createANewThread(sales, jsons, beginIndex,finalIndex));


threads.add(createANewThread(sales, jsons, beginIndex,finalIndex)); beginIndex = finalIndex; finalIndex = finalIndex + amount >= size ?

size: finalIndex + amount;


while (true) { threads.add(createANewThread(sales, jsons, beginIndex,finalIndex)); beginIndex = finalIndex; finalIndex = finalIndex + amount >= size ?


}




if (beginIndex == size) { break; }}


protected List<String> convertToJson(List<Sale> sales){...



if (beginIndex == size) { break; }}

}


protected List<String> convertToJson(List<Sale> sales){

int size = sales.size(); int amount = size / NUMBER_OF_THREADS; //4 int beginIndex = 0; int finalIndex = amount;

List<Thread> threads = new ArrayList<>(); List<String> jsons = Collections.synchronizedList(

new ArrayList<String>());

while (true) { //Cria uma nova thread para cada parte

}



... for (Thread thread : threads) { thread.start(); }

return jsons;

}



... for (Thread thread : threads) { thread.start(); }

for (Thread thread : threads) { thread.join(); }

return jsons;

}


@Override protected List<String> convertToJson(List<Sale> vendas) throws Exception {

System.out.println("Convert to json....");

int size = vendas.size(); int amount = size / NUMBER_OF_THREADS; //4 int beginIndex = 0; int finalIndex = amount;

List<Thread> threads = new ArrayList<>(); List<String> jsons = Collections.synchronizedList(new ArrayList<String>());

while (true) { threads.add(createANewThread(vendas, jsons, beginIndex, finalIndex)); beginIndex = finalIndex; finalIndex = finalIndex + amount >= size ? size: finalIndex + amount;

if (beginIndex == size) { break; } }

for (Thread thread : threads) { thread.start(); }

for (Thread thread : threads) { thread.join(); }

System.out.println("Finished!\n"); return jsons;

}Thursday, December 6, 12

TEMPO

Método Tempo Médio (ms)

Single 5562

Multi Thread 3739


COMPARATIVO

0

2000

4000

6000

Tempo

37395562

Single Multi Thread


DIFICULDADES


DIFICULDADES

•Decidir a quantidade de Threads


DIFICULDADES


•Dividir o trabalho


DIFICULDADES


•Dividir o trabalho

• Juntar tudo no final



int size = sales.size(); int amount = size / NUMBER_OF_THREADS; //4 int beginIndex = 0; int finalIndex = amount;

List<Thread> threads = new ArrayList<>(); List<String> jsons = Collections.synchronizedList(

new ArrayList<String>());

while (true) { //Cria uma nova thread para cada parte

}


FORK / JOIN


FORK / JOIN

• Java 7


FORK / JOIN

• Java 7

•Dividir tarefas entre múltiplos processadores


FORK / JOIN

• Java 7

•Dividir tarefas entre múltiplos processadores

• Tarefas que podem ser quebradas recursivamente


class ExemploForkJoin extends RecursiveTask<Long>{

@Override protected Long compute() {

if(qtdeDeDados < LIMITE){ // execute a tarefa }else{ //Divida o trabalho entre subtarefas //Execute as subtarefas e espere o resultado } }}

COMO FUNCIONA


COMO FUNCIONA


COMO FUNCIONA

fork()


COMO FUNCIONA

fork() fork()


COMO FUNCIONA

fork() fork()

join()


COMO FUNCIONA

fork() fork()

join() join()


COMO FUNCIONA

fork()

join()

fork()

join() join()


COMO FUNCIONA

fork()

join()

fork()

join()

join() join()


IMPLEMENTANDOpublic class JsonTask extends RecursiveTask<List<String>>{

private final int THRESHOLD = 10000; private List<Sale> list; //constructor...

@Override protected List<String> compute() { }}


@Overrideprotected List<String> compute() { List<String> jsons = new ArrayList<>(); int size = list.size(); if ( size < THRESHOLD){ jsons.addAll(new JsonMapper().toJson(list)); }else{ ... } return jsons;}


}else{

}


}else{ JsonTask left = new JsonTask(list.subList(0, size / 2));

}


}else{ JsonTask left = new JsonTask(list.subList(0, size / 2)); JsonTask right = new JsonTask(list.subList(size / 2, size));

}


}else{ JsonTask left = new JsonTask(list.subList(0, size / 2)); JsonTask right = new JsonTask(list.subList(size / 2, size)); left.fork();

}


}else{ JsonTask left = new JsonTask(list.subList(0, size / 2)); JsonTask right = new JsonTask(list.subList(size / 2, size)); left.fork(); right.fork();

}



jsons.addAll(right.join()); jsons.addAll(left.join());}


P


P

L

fork()

R

fork()


P

L

fork()

join()

R

fork()

join()



jsons.addAll(right.join()); jsons.addAll(left.join());}


P

Poolde Threads


P

Poolde Threads P


P

L

fork()

R

fork()

Poolde Threads P


P

L

fork()

R

fork()

Poolde Threads P L R


}else{ JsonTask jsonTaskLeft = new JsonTask(list.subList(0, size / 2)); JsonTask jsonTaskRight = new JsonTask(list.subList(size / 2, size)); jsonTaskLeft.fork(); jsonTaskRight.fork();

jsons.addAll(jsonTaskRight.join()); jsons.addAll(jsonTaskLeft.join());}


P

L

fork()

R

fork()

Poolde Threads P L R


RECURSO OCIOSO


}else{ JsonTask jsonTaskLeft = new JsonTask(list.subList(0, size / 2)); JsonTask jsonTaskRight = new JsonTask(list.subList(size / 2, size)); jsonTaskLeft.fork(); jsonTaskRight.fork();



}else{ JsonTask jsonTaskLeft = new JsonTask(list.subList(0, size / 2)); JsonTask jsonTaskRight = new JsonTask(list.subList(size / 2, size)); jsonTaskLeft.fork();




jsons.addAll(jsonTaskRight.compute()); jsons.addAll(jsonTaskLeft.join());}


P

Poolde Threads

L R

P


P

L R

Poolde Threads

LL

P


P

L

fork()

R

Poolde Threads

LL

P


P

L

fork()

R

Poolde Threads

L

P

L


P

L

fork()

R

Poolde Threads

L

P L


P

L

fork()

R

Poolde Threads

R

LP


P

L

fork()

R

compute()

Poolde Threads

R

LP


P

L

fork()

R

compute()

Poolde Threads R L


P

L

fork()

R

Poolde Threads

compute()

LR

P


P

L

fork()

R

Poolde Threads

compute()

return

LR

P


P

L

fork()

R

Poolde Threads

compute()

return

LP



jsons.addAll(jsonTaskRight.compute()); jsons.addAll(jsonTaskLeft.join());}


P

L

fork()

R

Poolde Threads P L

compute()

return


P

L

fork()

R

Poolde Threads P L

compute()

returnjoin()


P

L

fork()

R

Poolde Threads P

compute()

returnjoin()


EXECUTANDO public class JsonForkJoinConverter {protected List<String> convertToJson(List<Sale> sales){

ForkJoinPool forkJoinPool = new ForkJoinPool(); return forkJoinPool.invoke(new JsonTask(sales)); }}


TEMPO

MétodoTempo

Médio (ms)

Single 5562

Multi Thread 3739

Fork/Join 3461


COMPARATIVO

0

2000

4000

6000

Tempo

346137395562

Single Multi Thread Fork/join


@Override protected List<String> compute() { List<String> jsons = new ArrayList<>(); int size = list.size(); if ( size < THRESHOLD){ jsons.addAll(new JsonMapper().toJson(list)); } else{ JsonTask jsonTaskFirst = new JsonTask(list.subList(0, size / 2)); jsonTaskFirst.fork(); JsonTask jsonTaskSecond = new JsonTask(list.subList(size / 2, size)); jsons.addAll(jsonTaskSecond.compute()); jsons.addAll(jsonTaskFirst.join()); } return jsons; }


PROBLEMS?


A ORDEM DAS CHAMADAS

•Ordem das chamadas é importante.

• Perde todo o paralelismo


left.fork();long rightAns = right.compute();long leftAns = left.join();return leftAns + rightAns;

P

P

L R



P

P

L R

fork()

L



P

R

L R

fork()

L

compute()



P

P

L R

fork()

L

compute()

return



P

P

L R

fork() compute()

returnjoin()


left.fork();

return leftAns + rightAns;

long rightAns = right.compute();long leftAns = left.join();


left.fork();

return leftAns + rightAns;long rightAns = right.compute();long leftAns = left.join();


P

P

L R

left.fork();long leftAns = left.join();long rightAns = right.compute();return leftAns + rightAns;


P

P

L R

fork()

L



P

P

L R

fork()

L


Bloqueia a thread!


P

P

L R

fork()

L


join()


P

R

L R

fork()


join()

compute()


P

R

L R

fork() compute()

return


join()


P

P

L R

fork() compute()

returnjoin()



long rightAns = right.compute();left.fork();long leftAns = left.join();return leftAns + rightAns;


P

P

L R



P

P

L R


compute()


P

R

L R


compute()


P

R

L R


compute()

return


P

P

L R


compute()

return


P

P

L R


compute()

return

fork()

L


P

P

L R

fork() compute()

returnjoin()



P

P

L R

fork() compute()

returnjoin()


Executa Sequencialmente


• Código aqui:

• github.com/mariofts/javaOneBR-2012

• Links / Refs:

• http://homes.cs.washington.edu/~djg/teachingMaterials/grossmanSPAC_forkJoinFramework.html

• http://www.oracle.com/technetwork/articles/java/fork-join-422606.html

• http://docs.oracle.com/javase/tutorial/essential/concurrency/forkjoin.html


https://github.com/mariofts/javaOneBR-2012

https://github.com/mariofts/javaOneBR-2012

http://homes.cs.washington.edu/~djg/teachingMaterials/grossmanSPAC_forkJoinFramework.html




http://www.oracle.com/technetwork/articles/java/fork-join-422606.html

http://www.oracle.com/technetwork/articles/java/fork-join-422606.html

http://docs.oracle.com/javase/tutorial/essential/concurrency/forkjoin.html

http://docs.oracle.com/javase/tutorial/essential/concurrency/forkjoin.html

OBRIGADO!

github.com/mariofts

@mario_fts

[email protected]


mailto:[email protected]

mailto:[email protected]

Facilitando a Programação concorrente com o Fork/Join

Technology

Transcript of Facilitando a Programação concorrente com o Fork/Join