© 2015 Continuum Analytics- Confidential & Proprietary

Memex: Mining the Deep Web

Katrina Riehl, PhDData ScientistContinuum Analytics

July 25, 2015

© 2015 Continuum Analytics- Confidential & Proprietary

THE DEEP WEBExplaining

When you ask the internet a question, who is answering?

© 2015 Continuum Analytics- Confidential & Proprietary

DARPA MEMEXAn introduction to

What is MEMEX?• Today's web searches use a centralized, one-size-fits-all approach that searches the Internet

with the same set of tools for all queries. While that model has been wildly successful commercially, it does not work well for many government use cases.

• DARPA launched the Memex program in September, 2014. • Memex seeks to develop software that advances online search capabilities• Creation of a new domain-specific indexing and search paradigm

• content discovery • information extraction• information retrieval• user collaboration

• Extension of current search capabilities • deep web • dark web • nontraditional (e.g. multimedia) content.

Memex Search Domains• Human/Labor Trafficking• Weapons• Material Research Science• Financial Fraud• Counterfeit Electronics• Patent Trolling• Child Exploitation

http://opencatalog.darpa.mil

© 2015 Continuum Analytics- Confidential & Proprietary

LARGE SCALE DATA ANALYTICSAn Overview of the Ecosystem

BI - DB DM/Stats/ML

Scientific Computing

Distributed Systems

Numba

bcolz

RHadoop

© 2015 Continuum Analytics- Confidential & Proprietary

THE ANALYTICS PIPELINE

Analytics Pipeline• Web Crawlers & Scrapers• Entity Extractors• Indexers• Visual Analytics

Memex Explorer• Pluggable Framework for Crawling & Data Discovery• Django Web Application• Elasticsearch Index• Bokeh Visualizations for Crawling Stats• Kibana Dashboards for Initial Data Exploration• Apache Nutch Crawler• NYU ACHE Crawler• NYU Domain Discovery Tool

Data Storage

Abstract expressions

Computational backend

csv

HDF5bcolz

DataFrame HDFS

selectionfilter

group by

join

column wise

Pandas

Streaming Python

Spark

MongoDB

SQLAlchemy

json

DATA ANALYSIS

Topic Modeling

Topic Modeling

Topic Modeling

QUESTIONS?Thank you!!