groonga with PostgreSQL

groonga with PostgreSQL

2011/11/29 groongaを囲む夕べ #2

板垣 & 奥野(@choplin)

フォルシア株式会社

1

Agenda

• Extensibility of PostgreSQL

• How to work with groonga

• textsearch_groonga

• groonga_fdw

2

Extensibility ofPostgreSQL

3

http://www.slideshare.net/HitoshiHarada/extensions-on-postgresql

©2011 FORCIA, Inc.

Extensibility of PostgreSQL

Index

Table

GiST/GIN,textsearch_senna, textsearch_groonga

pl/R, pl/v8js, pl/Ruby, pl/Lua, …

cube, ltree, citext, hstore, json, …

Executor

Optimizer

Data TypesFunctions PL

Parser

dblink, adminpack, pgcrypto, ...

auto_explain

pgadviser

SQL/MED (9.1!)

c.f. external tools: pgpool, slony, pg_bulkload…

4



PGXN

• PostgreSQL Extension Network

• Central distribution system for PostgreSQL Extension

• Inspired by CPAN for Perl library

• Provides:

• Manager, API, Search, Client

• under development ...

5

PGXN

6

Only three steps to install an extension

$ pgxn install tinyint

$ psql

# CREATE EXTENSION tinyint;--That’s all, yay!

Coming soon

7

How to workwith groonga

8


©2011 FORCIA, Inc.


Index

Table




Executor

Optimizer


Parser


auto_explain

pgadviser

SQL/MED (9.1!)


9




©2011 FORCIA, Inc.


Index

Table




Executor

Optimizer


Parser


auto_explain

pgadviser

SQL/MED (9.1!)


As Index

9




©2011 FORCIA, Inc.


Index

Table




Executor

Optimizer


Parser


auto_explain

pgadviser

SQL/MED (9.1!)


As IndexAs FDW

groonga_fdw

9



textsearch_groonga

10

Full text searchin PostgreSQL

https://docs.google.com/present/view?id=dhk2smpg_3h44cnrdv

Builtin Extension

morphological analysis

Tsearch (8.3-)+

textsearch_ja

N-gram contrib/pg_trgm(9.1-)

Ludiatextsearch_senna

textsearch_groonga

11



Motivationtextsearch_groonga䛾㛤Ⓨືᶵ

• textsearch_ja (༢ㄒ)

ᅛ᭷ྡモ䜔䜹䝍䜹䝘㐃ㄒ䛾᳨⣴₃䜜䛜ᛧ䛔

• textsearch_senna (N-gram)

©2010 FORCIA, Inc.

PostgreSQL 8.3䛾HOT᭦᪂䛸┦ᛶ䛜ᝏ䛔䛣䛸䛜ุSenna䜒䝯䞁䝔䝘䞁䝇䞉䝣䜵䞊䝈䛻ධ䜛䜏䛯䛔䛰䛧…


テキスト

http://www.slideshare.net/ItagakiTakahiro/textsearch-groonga-v0112



Indexing䛔᪉ :䜲䞁䝕䝑䜽䝇సᡂ

• CREATE TABLE tbl(id serial, doc text);• COPY tbl(doc) FROM '...'; --䝕䞊䝍ᢞධ• CREATE INDEX idx ON tbl USING groonga (doc);

CTID ᩥ᭩ᩥ᭩ᩥ᭩ᩥ᭩ 䛭䛾䛭䛾䛭䛾䛭䛾 _key ᩥ᭩ᩥ᭩ᩥ᭩ᩥ᭩ CTID ༢ㄒ༢ㄒ༢ㄒ༢ㄒ _key

©2010 FORCIA, Inc.

CTID ᩥ᭩ᩥ᭩ᩥ᭩ᩥ᭩ 䛭䛾䛭䛾䛭䛾䛭䛾

(1, 0) foo 1.5

(1, 1) bar 2.0

(2, 0) baz 4.5

PostgreSQL䝔䞊䝤䝹

CTID =≀⌮⨨

_key ᩥ᭩ᩥ᭩ᩥ᭩ᩥ᭩ CTID

1 bar (1, 1)

2 baz (2, 0)

3 foo (1,0)

groonga䝔䞊䝤䝹

༢ㄒ༢ㄒ༢ㄒ༢ㄒ _key

ar 1

az 2

ba 1, 2

fo 3

oo 3

groonga㌿⨨⾲




Querying


䛔᪉ : ᳨⣴

• SELECT * FROM tbl WHERE …–༢⣧᳨⣴

• doc = '䜻䞊䝽䞊䝗' –ẚ㍑₇⟬Ꮚ䛻䜘䜛᳨⣴– <, <=, =, >=, >, <>

• ᩥ᭩䛸ᩘ್䜢」ᩘิ䜲䞁䝕䝑䜽䝇䛻䛧䛶䛔䜛ሙྜ➼–༢ㄒ᳨⣴

©2010 FORCIA, Inc.

–༢ㄒ᳨⣴• doc %% '䜻䞊䝽䞊䝗' – N-gram䛻䜘䜛᳨⣴• 䛂䜻䞊䝽䞊䝗䜢ྵ䜐䛃᳨⣴

– grn_expr᳨⣴• doc @@ groonga.query('䜻䞊䝽䞊䝗', 'ิ䛸㔜䜏')• ⮬⏤䛺᳨⣴

14



groonga_fdw

15

•Management of External Data

• SQL Standard (SQL:2003)

• Consists:

• Foreign Table

• DATALINK

SQL/MED

16

•Management of External Data

• SQL Standard (SQL:2003)

• Consists:

• Foreign Table

• DATALINK

SQL/MED

Implemented as FDWin Postgres

16

FDW• Foreign Data Wrapper

• External data as Table of PostgreSQL

• Available from 9.1

• Only “SELECT” statement is available now

• Very simple API interface

• Only 6 callbacks are required :)

17

FDW

Parser Planner Executer InternalStorage

18

FDW

Parser Planner Executer InternalStorage

ExternalData

FDW18

FDW• Example:

• Oracle

• MySQL

• Redis

• Amazon S3

• Twitter

• and more... (see http://pgxn.org/tag/fdw/ )19

http://pgxn.org/tag/fdw/

http://pgxn.org/tag/fdw/

groonga_fdw

• Foreign Data Wrapper for groonga

• You can query groonga via postgres with usual SQL

• Under Development

20

How to use# CREATE EXTENTION groonga_fdw;

# CREATE SERVER groonga_server FOREIGN DATA WRAPPER groonga_fdw OPTIONS ( dbpath ‘ );

# CREATE FOREIGN TABLE groonga_fdw_tutorial ( _key text title text ) SERVER groonga_server OPTIONS ( ... );

Install

Connect

Table

21

How to use

# SELECT * FROM groonga_fdw_tutorial;Query

22

Compare


• <-> Tritton

• groonga_fdw

• <-> mroonga

23

Summary

• PostgreSQL is quite extensible

• Two ways to work with groonga


• groonga_fdw

• PostgreSQL and groonga are good friend :)

24

Thanks!

25

groonga with PostgreSQL

Technology

Transcript of groonga with PostgreSQL