실시간 이벤트 처리

문병원byeongweon.moon@redduck.com

레드덕

실시간 이벤트 처리

솔루션

EsperApache S4Twitter StormHStreaming

비슷함

Event Stream Pro-cessing

Complex Event Processing

Event Stream Process-ing  은 이벤트 스트림의 빠른 처리와 단순한 산술 계산에 초첨을 두고 있다 .

Complex Event Pro-cessing 은 패턴 분석에 주안점을 둔 데이터 분석을 포함하고 있다

Esper

• ESP/CEP with expressive Event Pro-cessing Language: continuous queries, aggregation, joins, causality and missing events, joins to historical data, output flow control...

• High throughput, low latency• Standalone or embedded, open API• Multi platform support

Computing Rates Per Feed

insert into TicksPerSecond select feed, rate(10) as cnt from MarketDataEvent group by feed

Detecting a Fall-off

select feed, avg(cnt) as avgCnt, cnt as feedCnt from TicksPerSecond.win:time(10 seconds) group by feed having cnt < avg(cnt) * 0.75

Storm

Stream processingContinuous computationDistributed RPC

Storm do

• Guaranteed message process-ing• Robust process management• Fault detection and automatic

reassignment• Efficient message passing• Local mode and distributed

mode

TopologyBuilder builder = new TopologyBuilder();builder.setSpout(1, new KestrelSpout("kestrel.backtype.com",                                     22133,                                     "sentence_queue",                                     new StringScheme()));builder.setBolt(2, new SplitSentence(), 10)        .shuffleGrouping(1);builder.setBolt(3, new WordCount(), 20)        .fieldsGrouping(2, new Fields("word"));

public class SplitSentence implements IBasicBolt {    public void execute(Tuple tuple, BasicOutputCollector collector) {        String sentence = tuple.getString(0);        for(String word: sentence.split(" ")) {            collector.emit(new Values(word));        }    }

    public void declareOutputFields(OutputFieldsDeclarer declarer) {        declarer.declare(new Fields("word"));    } }

public class WordCount implements IBasicBolt {    private Map<String, Integer> _counts = new HashMap<String, Integer>();

    public void execute(Tuple tuple, BasicOutputCollector collector) {        String word = tuple.getString(0);        int count;        if(_counts.containsKey(word)) {            count = _counts.get(word);        } else {            count = 0;        }        count++;        _counts.put(word, count);        collector.emit(new Values(word, count));    }

    public void declareOutputFields(OutputFieldsDeclarer declarer) {        declarer.declare(new Fields("word", "count"));    } }