DrupalCamp Ottawa 2015

Taming Open Data with Solr and SarniaJosé de Leon24 July 2015

About José de Leon

● Currently Web Developer, Askari Systems● Working on buyandsell.gc.ca at PWGSC● Interest: Teaching and coaching, especially

about technology● Maintain Sarnia module: http://drupal.

org/project/sarnia● jose_de_leon@hotmail.com

About Sarnia Module

● Drupal Module that exposes Apache Solr documents as Drupal Entities to Drupal Views

● A way around the “10,000 Nodes” problem to display external data in Drupal -- reduce need to import or migrate large data sets into Drupal

Sarnia Advantages

● As a Search API module, you can also search and facet external data in Drupal with Sarnia○ provide a consistent search interface

between Drupal content and non-Drupal data○ create “mini-search apps” on non-Drupal

data● Leverage existing Solr knowledge and

infrastructure already used to provide robust search in Drupal

Sarnia Examples

● Sarnia powers advanced search on:○ Standards Council Of Canada (scc.ca)○ Canadian Museum Of Human Rights

(humanrights.ca)

Related Tools

● Sarnia leverages Search API and Search API Solr modules

● Other tools:○ Solr: DataImportHandler (DIH), CSV-JDBC○ Drupal: Panels, Views, Custom Search,

Facet API

Module Architecture

Search API

Sarnia Sarnia Drupal Index

Views plugins

Drupal Display Layer (Panels)

Drupal

Non Drupal Data

Solr

Views

Non Drupal Data

Solr

Facet API

Custom Search

Module List To Download

● Drupal Modules to install for demo:○ sarnia○ search_api○ search_api_solr○ facetapi○ views○ entity○ ctools○ custom_search○ panels○ search_api_db

Module List To Activate

● Drupal Modules to enable for demo:○ Views (views), Views UI (views_ui), Entity API (entity),

Chaos tools (ctools), Page manager (page_manager), Views content panes (views_content), Custom content panes (ctools_custom_content), Panels (panels), Mini panels (panels_mini)

○ Sarnia (sarnia), Search API (search_api), Search views (search_api_views), Search facets (search_api_facetapi), Facet API (facetapi), Solr search (search_api_solr), Database search (search_api_db), Custom Search (custom_search), Custom Search Blocks (custom_search_blocks)

Demo

● Show Solr example with DIH and CSV-JDBC○ DataImportHandler: Solr Import tool, works

primarily with JDBC and XML sources○ use “content” field to enable full-text search (use

schema CopyField)● Show Drupal configuration

○ Search API (Sarnia server, entity)○ Sarnia Index○ Views○ Panels○ Facets

Open Data Demo

● Use Contract History CSV File from buyandsell.gc.cahttp://open.canada.ca/data/en/dataset/53753f06-8b28-42d7-89f7-04cd014323b0

● Demo build steps:○ Build Solr core (DIH, JDBC, XML) with “content” field○ Create Sarnia Server, Define Sarnia Entity○ Build Sarnia Views (View Panes for Panels)○ Set Panels on Pages○ Add Facets

Thanks and Questions

● http://drupal.org/project/sarnia● https://wiki.apache.org/solr/DataImportHandler● http://csvjdbc.sourceforge.net/● https://github.com/jmdeleon/dih-csv-jdbc● http://ouvert.canada.ca/, http://open.canada.ca/● jose_de_leon@hotmail.com