DrupalCamp Ottawa 2015
Taming Open Data with Solr and SarniaJosé de Leon24 July 2015
About José de Leon
● Currently Web Developer, Askari Systems● Working on buyandsell.gc.ca at PWGSC● Interest: Teaching and coaching, especially
about technology● Maintain Sarnia module: http://drupal.
org/project/sarnia● [email protected]
About Sarnia Module
● Drupal Module that exposes Apache Solr documents as Drupal Entities to Drupal Views
● A way around the “10,000 Nodes” problem to display external data in Drupal -- reduce need to import or migrate large data sets into Drupal
Sarnia Advantages
● As a Search API module, you can also search and facet external data in Drupal with Sarnia○ provide a consistent search interface
between Drupal content and non-Drupal data○ create “mini-search apps” on non-Drupal
data● Leverage existing Solr knowledge and
infrastructure already used to provide robust search in Drupal
Sarnia Examples
● Sarnia powers advanced search on:○ Standards Council Of Canada (scc.ca)○ Canadian Museum Of Human Rights
(humanrights.ca)
Related Tools
● Sarnia leverages Search API and Search API Solr modules
● Other tools:○ Solr: DataImportHandler (DIH), CSV-JDBC○ Drupal: Panels, Views, Custom Search,
Facet API
Module Architecture
Search API
Sarnia Sarnia Drupal Index
Views plugins
Drupal Display Layer (Panels)
Drupal
Non Drupal Data
Solr
Views
Non Drupal Data
Solr
Facet API
Custom Search
Module List To Download
● Drupal Modules to install for demo:○ sarnia○ search_api○ search_api_solr○ facetapi○ views○ entity○ ctools○ custom_search○ panels○ search_api_db
Module List To Activate
● Drupal Modules to enable for demo:○ Views (views), Views UI (views_ui), Entity API (entity),
Chaos tools (ctools), Page manager (page_manager), Views content panes (views_content), Custom content panes (ctools_custom_content), Panels (panels), Mini panels (panels_mini)
○ Sarnia (sarnia), Search API (search_api), Search views (search_api_views), Search facets (search_api_facetapi), Facet API (facetapi), Solr search (search_api_solr), Database search (search_api_db), Custom Search (custom_search), Custom Search Blocks (custom_search_blocks)
Demo
● Show Solr example with DIH and CSV-JDBC○ DataImportHandler: Solr Import tool, works
primarily with JDBC and XML sources○ use “content” field to enable full-text search (use
schema CopyField)● Show Drupal configuration
○ Search API (Sarnia server, entity)○ Sarnia Index○ Views○ Panels○ Facets
Open Data Demo
● Use Contract History CSV File from buyandsell.gc.cahttp://open.canada.ca/data/en/dataset/53753f06-8b28-42d7-89f7-04cd014323b0
● Demo build steps:○ Build Solr core (DIH, JDBC, XML) with “content” field○ Create Sarnia Server, Define Sarnia Entity○ Build Sarnia Views (View Panes for Panels)○ Set Panels on Pages○ Add Facets
Thanks and Questions
● http://drupal.org/project/sarnia● https://wiki.apache.org/solr/DataImportHandler● http://csvjdbc.sourceforge.net/● https://github.com/jmdeleon/dih-csv-jdbc● http://ouvert.canada.ca/, http://open.canada.ca/● [email protected]
Top Related