Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical...
Transcript of Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical...
![Page 1: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/1.jpg)
Invenio TechnologySelected Practical Software Development Lessons
From A Large Digital Library System
Tibor Šimko<[email protected]>
Department of Information TechnologyCERN
August 2013 / openlab talk
Tibor Šimko (CERN) Invenio Technology openlab 2013 1 / 62
![Page 2: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/2.jpg)
Outline
1 IntroductionDigital LibraryInvenio
2 Case StudiesEpisode 1: PythonEpisode 2: GitEpisode 3: TestingEpisode 4: Building Efficient IndexesEpisode 5: NIHEpisode 6: Scalability
3 Conclusions
Tibor Šimko (CERN) Invenio Technology openlab 2013 2 / 62
![Page 3: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/3.jpg)
Outline
1 IntroductionDigital LibraryInvenio
2 Case StudiesEpisode 1: PythonEpisode 2: GitEpisode 3: TestingEpisode 4: Building Efficient IndexesEpisode 5: NIHEpisode 6: Scalability
3 Conclusions
Tibor Šimko (CERN) Invenio Technology openlab 2013 3 / 62
![Page 4: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/4.jpg)
Outline
1 IntroductionDigital LibraryInvenio
2 Case StudiesEpisode 1: PythonEpisode 2: GitEpisode 3: TestingEpisode 4: Building Efficient IndexesEpisode 5: NIHEpisode 6: Scalability
3 Conclusions
Tibor Šimko (CERN) Invenio Technology openlab 2013 4 / 62
![Page 5: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/5.jpg)
What is Digital Library?
“library in which collections are stored in digital formats (asopposed to print, microform, or other media) and accessible bycomputers”(1) institutional document repositories(2) world-wide subject-based information systems
Example #1: CERN Document Server
managing CERN and selected non-CERN high-energy physicsand related documents since ∼1993more than 1,000,000 recordsarticles, books, theses, photos, videos, and morepowered by Invenio since 2002http://cdsweb.cern.ch/
Tibor Šimko (CERN) Invenio Technology openlab 2013 5 / 62
![Page 6: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/6.jpg)
What is Digital Library?
“library in which collections are stored in digital formats (asopposed to print, microform, or other media) and accessible bycomputers”(1) institutional document repositories(2) world-wide subject-based information systems
Example #1: CERN Document Server
managing CERN and selected non-CERN high-energy physicsand related documents since ∼1993more than 1,000,000 recordsarticles, books, theses, photos, videos, and morepowered by Invenio since 2002http://cdsweb.cern.ch/
Tibor Šimko (CERN) Invenio Technology openlab 2013 5 / 62
![Page 7: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/7.jpg)
CDS: Collection Tree
Tibor Šimko (CERN) Invenio Technology openlab 2013 6 / 62
![Page 8: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/8.jpg)
CDS: Search for Books
Tibor Šimko (CERN) Invenio Technology openlab 2013 7 / 62
![Page 9: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/9.jpg)
CDS: Search for Photos
Tibor Šimko (CERN) Invenio Technology openlab 2013 8 / 62
![Page 10: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/10.jpg)
CDS Features: Commenting
Tibor Šimko (CERN) Invenio Technology openlab 2013 9 / 62
![Page 11: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/11.jpg)
Invenio Features: Reviewing
Tibor Šimko (CERN) Invenio Technology openlab 2013 10 / 62
![Page 12: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/12.jpg)
CDS: Create Personal Alert
Tibor Šimko (CERN) Invenio Technology openlab 2013 11 / 62
![Page 13: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/13.jpg)
CDS: Add to Personal Basket
Tibor Šimko (CERN) Invenio Technology openlab 2013 12 / 62
![Page 14: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/14.jpg)
CDS: Display Personal Basket
Tibor Šimko (CERN) Invenio Technology openlab 2013 13 / 62
![Page 15: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/15.jpg)
CDS: Organize and Share Your Baskets
Tibor Šimko (CERN) Invenio Technology openlab 2013 14 / 62
![Page 16: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/16.jpg)
CDS: Journals and Bulletins
Tibor Šimko (CERN) Invenio Technology openlab 2013 15 / 62
![Page 17: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/17.jpg)
What is digital library?
Example #2: INSPIRE
world-wide high-energy physics information systemrun by CERN, DESY, FNAL, SLACmetadata curation since 1960s, Invenio technology since 2007citation analysis, author/affiliation analysisclose partnership with arXiv and ADShttp://inspirehep.net/
Tibor Šimko (CERN) Invenio Technology openlab 2013 16 / 62
![Page 18: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/18.jpg)
INSPIRE: full-text search
Tibor Šimko (CERN) Invenio Technology openlab 2013 17 / 62
![Page 19: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/19.jpg)
INSPIRE: cite summary
Tibor Šimko (CERN) Invenio Technology openlab 2013 18 / 62
![Page 20: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/20.jpg)
INSPIRE: citation history
Tibor Šimko (CERN) Invenio Technology openlab 2013 19 / 62
![Page 21: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/21.jpg)
INSPIRE: author pages
Tibor Šimko (CERN) Invenio Technology openlab 2013 20 / 62
![Page 22: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/22.jpg)
Outline
1 IntroductionDigital LibraryInvenio
2 Case StudiesEpisode 1: PythonEpisode 2: GitEpisode 3: TestingEpisode 4: Building Efficient IndexesEpisode 5: NIHEpisode 6: Scalability
3 Conclusions
Tibor Šimko (CERN) Invenio Technology openlab 2013 21 / 62
![Page 23: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/23.jpg)
Invenio Key Features
navigable collection tree (regular, virtual)powerful search engine
Google-like speed for up to 5M recordscombined metadata, reference and fulltext search
flexible metadata (MARC, OA)handling any kind of document (multimedia)customizable input, formatting and linking
personalization and collaborative features:alerts, baskets, groups, reviews, commentsinternationalisation (28 languages)
open source, GNU General Public Licenseco-developed by CERN (2002–), EPFL (2004–), DESY/FNAL/SLAC(2008–), CfA (2009–), Cornell (2011–)installed at 30+ institutions world-wide
Tibor Šimko (CERN) Invenio Technology openlab 2013 22 / 62
![Page 24: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/24.jpg)
Invenio Modules: Overview
Author
Tibor Šimko (CERN) Invenio Technology openlab 2013 23 / 62
![Page 25: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/25.jpg)
Invenio Modules: Overview
Author
Sources
Tibor Šimko (CERN) Invenio Technology openlab 2013 23 / 62
![Page 26: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/26.jpg)
Invenio Modules: Overview
Author
Sources Librarian
Tibor Šimko (CERN) Invenio Technology openlab 2013 23 / 62
![Page 27: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/27.jpg)
Invenio Modules: Overview
Author
Sources Librarian
User
Tibor Šimko (CERN) Invenio Technology openlab 2013 23 / 62
![Page 28: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/28.jpg)
Invenio Modules: Overview
Author
Sources Librarian
User
Database
Tibor Šimko (CERN) Invenio Technology openlab 2013 23 / 62
![Page 29: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/29.jpg)
Invenio Modules: Overview
Author
Sources Librarian
User
Database
Ingestion Modules
Tibor Šimko (CERN) Invenio Technology openlab 2013 23 / 62
![Page 30: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/30.jpg)
Invenio Modules: Overview
Author
Sources Librarian
User
Database
Ingestion Modules
Tibor Šimko (CERN) Invenio Technology openlab 2013 23 / 62
![Page 31: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/31.jpg)
Invenio Modules: Overview
Author
Sources Librarian
User
Database
Ingestion Modules
Processing Modules
Tibor Šimko (CERN) Invenio Technology openlab 2013 23 / 62
![Page 32: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/32.jpg)
Invenio Modules: Overview
Author
Sources Librarian
User
Database
Ingestion Modules
Processing Modules
Dissemination Modules
Tibor Šimko (CERN) Invenio Technology openlab 2013 23 / 62
![Page 33: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/33.jpg)
Invenio Modules: Overview
Author
Sources Librarian
User
Database
Ingestion Modules
Processing Modules
Dissemination Modules
Tibor Šimko (CERN) Invenio Technology openlab 2013 23 / 62
![Page 34: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/34.jpg)
Invenio Modules: Overview
Author
Sources Librarian
User
Database
Ingestion Modules
Processing Modules
Dissemination Modules
Curation Modules
Tibor Šimko (CERN) Invenio Technology openlab 2013 23 / 62
![Page 35: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/35.jpg)
Invenio Modules: Overview
Author
Sources Librarian
User
Database
Ingestion Modules
Processing Modules
Dissemination Modules
Curation Modules
Tibor Šimko (CERN) Invenio Technology openlab 2013 23 / 62
![Page 36: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/36.jpg)
Invenio Modules: Ingestion
Author
Tibor Šimko (CERN) Invenio Technology openlab 2013 24 / 62
![Page 37: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/37.jpg)
Invenio Modules: Ingestion
Author
WebSubmit
WebSession, WebAccess
Tibor Šimko (CERN) Invenio Technology openlab 2013 24 / 62
![Page 38: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/38.jpg)
Invenio Modules: Ingestion
Author
WebSubmit
WebSession, WebAccess
Metadata Full-text
full-text documentmetadata
Tibor Šimko (CERN) Invenio Technology openlab 2013 24 / 62
![Page 39: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/39.jpg)
Invenio Modules: Ingestion
Author
WebSubmit
WebSession, WebAccess
Metadata Full-text
full-text document
BibUpload
BibSched
BibConvert
metadata
MARCXML
Tibor Šimko (CERN) Invenio Technology openlab 2013 24 / 62
![Page 40: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/40.jpg)
Invenio Modules: Ingestion
Author
WebSubmit
WebSession, WebAccess
Metadata Full-text
full-text document
BibUpload
BibSched
BibConvert
metadata
MARCXML
BibHarvest
OAI Data Source
Tibor Šimko (CERN) Invenio Technology openlab 2013 24 / 62
![Page 41: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/41.jpg)
Invenio Modules: Ingestion
Author
WebSubmit
WebSession, WebAccess
Metadata Full-text
full-text document
BibUpload
BibSched
BibConvert
metadata
MARCXML
BibHarvest
OAI Data Source
ElmSubmit
Non-OAI Data Source
Tibor Šimko (CERN) Invenio Technology openlab 2013 24 / 62
![Page 42: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/42.jpg)
Invenio Modules: Processing
Metadata Full-text
Tibor Šimko (CERN) Invenio Technology openlab 2013 25 / 62
![Page 43: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/43.jpg)
Invenio Modules: Processing
Metadata Full-textRefExtract
Tibor Šimko (CERN) Invenio Technology openlab 2013 25 / 62
![Page 44: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/44.jpg)
Invenio Modules: Processing
Metadata Full-textRefExtract
BibClassify
Tibor Šimko (CERN) Invenio Technology openlab 2013 25 / 62
![Page 45: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/45.jpg)
Invenio Modules: Processing
Metadata Full-textRefExtract
BibClassify
BibDocFile
Tibor Šimko (CERN) Invenio Technology openlab 2013 25 / 62
![Page 46: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/46.jpg)
Invenio Modules: Processing
Metadata Full-textRefExtract
BibClassify
BibDocFile
BibEncode
Tibor Šimko (CERN) Invenio Technology openlab 2013 25 / 62
![Page 47: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/47.jpg)
Invenio Modules: Processing
Metadata Full-textRefExtract
BibClassify
BibDocFile
BibEncode
Clusters BibIndex
Tibor Šimko (CERN) Invenio Technology openlab 2013 25 / 62
![Page 48: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/48.jpg)
Invenio Modules: Processing
Metadata Full-textRefExtract
BibClassify
BibDocFile
BibEncode
Clusters BibIndex
WebColl
Tibor Šimko (CERN) Invenio Technology openlab 2013 25 / 62
![Page 49: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/49.jpg)
Invenio Modules: Processing
Metadata Full-textRefExtract
BibClassify
BibDocFile
BibEncode
Clusters BibIndex
WebColl
BibRank
Tibor Šimko (CERN) Invenio Technology openlab 2013 25 / 62
![Page 50: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/50.jpg)
Invenio Modules: Processing
Metadata Full-textRefExtract
BibClassify
BibDocFile
BibEncode
Clusters BibIndex
WebColl
BibRank
BibFormat
Tibor Šimko (CERN) Invenio Technology openlab 2013 25 / 62
![Page 51: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/51.jpg)
Invenio Modules: Processing
Metadata Full-textRefExtract
BibClassify
BibDocFile
BibEncode
Clusters BibIndex
WebColl
BibRank
BibFormat
BibSort
Tibor Šimko (CERN) Invenio Technology openlab 2013 25 / 62
![Page 52: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/52.jpg)
Invenio Modules: Processing
Metadata Full-textRefExtract
BibClassify
BibDocFile
BibEncode
Clusters BibIndex
WebColl
BibRank
BibFormat
BibSort
BibAuthorID
Tibor Šimko (CERN) Invenio Technology openlab 2013 25 / 62
![Page 53: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/53.jpg)
Invenio Modules: Dissemination
Metadata Full-textClusters
Tibor Šimko (CERN) Invenio Technology openlab 2013 26 / 62
![Page 54: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/54.jpg)
Invenio Modules: Dissemination
Metadata Full-textClusters
WebSearch
User
Tibor Šimko (CERN) Invenio Technology openlab 2013 26 / 62
![Page 55: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/55.jpg)
Invenio Modules: Dissemination
Metadata Full-textClusters
WebSearch
User
WebBasket
Tibor Šimko (CERN) Invenio Technology openlab 2013 26 / 62
![Page 56: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/56.jpg)
Invenio Modules: Dissemination
Metadata Full-textClusters
WebSearch
User
WebBasketBibAuthorID
Tibor Šimko (CERN) Invenio Technology openlab 2013 26 / 62
![Page 57: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/57.jpg)
Invenio Modules: Dissemination
Metadata Full-textClusters
WebSearch
User
WebBasketBibAuthorID WebAlert
Tibor Šimko (CERN) Invenio Technology openlab 2013 26 / 62
![Page 58: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/58.jpg)
Invenio Modules: Dissemination
Metadata Full-textClusters
WebSearch
User
WebBasketBibAuthorID WebAlert BibHarvest
OAI Harvester
Tibor Šimko (CERN) Invenio Technology openlab 2013 26 / 62
![Page 59: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/59.jpg)
Invenio Modules: Dissemination
Metadata Full-textClusters
WebSearch
User
WebBasketBibAuthorID WebAlert BibHarvest
OAI HarvesterWebComment
Tibor Šimko (CERN) Invenio Technology openlab 2013 26 / 62
![Page 60: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/60.jpg)
Invenio Modules: Dissemination
Metadata Full-textClusters
WebSearch
User
WebBasketBibAuthorID WebAlert BibHarvest
OAI HarvesterWebComment
WebMessage
Tibor Šimko (CERN) Invenio Technology openlab 2013 26 / 62
![Page 61: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/61.jpg)
Invenio Modules: Dissemination
Metadata Full-textClusters
WebSearch
User
WebBasketBibAuthorID WebAlert BibHarvest
OAI HarvesterWebComment
WebMessageWebJournal
Tibor Šimko (CERN) Invenio Technology openlab 2013 26 / 62
![Page 62: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/62.jpg)
Invenio Modules: Dissemination
Metadata Full-textClusters
WebSearch
User
WebBasketBibAuthorID WebAlert BibHarvest
OAI HarvesterWebComment
WebMessageWebJournal BibCirculation
Tibor Šimko (CERN) Invenio Technology openlab 2013 26 / 62
![Page 63: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/63.jpg)
Invenio Modules: Dissemination
Metadata Full-textClusters
WebSearch
User
WebBasketBibAuthorID WebAlert BibHarvest
OAI HarvesterWebComment
WebMessageWebJournal BibCirculation
WebStat
Tibor Šimko (CERN) Invenio Technology openlab 2013 26 / 62
![Page 64: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/64.jpg)
Invenio Modules: Dissemination
Metadata Full-textClusters
WebSearch
User
WebBasketBibAuthorID WebAlert BibHarvest
OAI HarvesterWebComment
WebMessageWebJournal BibCirculation
WebStat WebHelp
Tibor Šimko (CERN) Invenio Technology openlab 2013 26 / 62
![Page 65: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/65.jpg)
Invenio Modules: Curation
Metadata Librarian Full-text
Tibor Šimko (CERN) Invenio Technology openlab 2013 27 / 62
![Page 66: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/66.jpg)
Invenio Modules: Curation
Metadata Librarian Full-textBibEdit
Tibor Šimko (CERN) Invenio Technology openlab 2013 27 / 62
![Page 67: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/67.jpg)
Invenio Modules: Curation
Metadata Librarian Full-textBibEdit
MultiEdit
Tibor Šimko (CERN) Invenio Technology openlab 2013 27 / 62
![Page 68: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/68.jpg)
Invenio Modules: Curation
Metadata Librarian Full-textBibEdit
MultiEdit
BatchUploader
Tibor Šimko (CERN) Invenio Technology openlab 2013 27 / 62
![Page 69: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/69.jpg)
Invenio Modules: Curation
Metadata Librarian Full-textBibEdit
MultiEdit
BatchUploader
BibCheck
Tibor Šimko (CERN) Invenio Technology openlab 2013 27 / 62
![Page 70: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/70.jpg)
Invenio Modules: Curation
Metadata Librarian Full-textBibEdit
MultiEdit
BatchUploader
BibCheck
BibCirculation
Tibor Šimko (CERN) Invenio Technology openlab 2013 27 / 62
![Page 71: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/71.jpg)
Invenio Modules: Curation
Metadata Librarian Full-textBibEdit
MultiEdit
BatchUploader
BibCheck
BibCirculation
BibDocFile
Tibor Šimko (CERN) Invenio Technology openlab 2013 27 / 62
![Page 72: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/72.jpg)
Invenio Modules: Curation
Metadata Librarian Full-textBibEdit
MultiEdit
BatchUploader
BibCheck
BibCirculation
BibDocFile
BibClassify
Tibor Šimko (CERN) Invenio Technology openlab 2013 27 / 62
![Page 73: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/73.jpg)
Invenio Modules: Curation
Metadata Librarian Full-textBibEdit
MultiEdit
BatchUploader
BibCheck
BibCirculation
BibDocFile
BibClassify
RefExtract
Tibor Šimko (CERN) Invenio Technology openlab 2013 27 / 62
![Page 74: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/74.jpg)
Invenio Modules: Curation
Metadata Librarian Full-textBibEdit
MultiEdit
BatchUploader
BibCheck
BibCirculation
BibDocFile
BibClassify
RefExtract
Tasks
BibCatalog
Tibor Šimko (CERN) Invenio Technology openlab 2013 27 / 62
![Page 75: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/75.jpg)
Invenio Modules: Curation
Metadata Librarian Full-textBibEdit
MultiEdit
BatchUploader
BibCheck
BibCirculation
BibDocFile
BibClassify
RefExtract
Tasks
BibCatalog
Knowledge Bases
BibKnowledge
Tibor Šimko (CERN) Invenio Technology openlab 2013 27 / 62
![Page 76: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/76.jpg)
Invenio Modules: Curation
Metadata Librarian Full-textBibEdit
MultiEdit
BatchUploader
BibCheck
BibCirculation
BibDocFile
BibClassify
RefExtract
Tasks
BibCatalog
Knowledge Bases
BibKnowledge
BibExport
Tibor Šimko (CERN) Invenio Technology openlab 2013 27 / 62
![Page 77: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/77.jpg)
Invenio Modules: Curation
Metadata Librarian Full-textBibEdit
MultiEdit
BatchUploader
BibCheck
BibCirculation
BibDocFile
BibClassify
RefExtract
Tasks
BibCatalog
Knowledge Bases
BibKnowledge
BibExportBibMatch
Tibor Šimko (CERN) Invenio Technology openlab 2013 27 / 62
![Page 78: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/78.jpg)
Invenio Modules: Curation
Metadata Librarian Full-textBibEdit
MultiEdit
BatchUploader
BibCheck
BibCirculation
BibDocFile
BibClassify
RefExtract
Tasks
BibCatalog
Knowledge Bases
BibKnowledge
BibExportBibMatch
BibMerge
Tibor Šimko (CERN) Invenio Technology openlab 2013 27 / 62
![Page 79: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/79.jpg)
Invenio Modules: Summary
∼40 modulescodebase
∼350,000 lines of Python code∼15,000 lines of JavaScript code∼7,000 lines of XSL code∼8,000 lines of autotools code
∼120 authors and contributors since 2002∼48 authors and contributors in 2012 (18 new)many short-term students, importance of informal coding standards
∼10 years of developmentstarted at CERN, first release in 2002now co-developed world-wide (EU, US)
lego programming... but no silver bullet
Tibor Šimko (CERN) Invenio Technology openlab 2013 28 / 62
![Page 80: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/80.jpg)
Developer Community
I n v e n i oEPFLCDS AUTH UAB ...
2002– 2004–2006+ 2002–
Tibor Šimko (CERN) Invenio Technology openlab 2013 29 / 62
![Page 81: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/81.jpg)
Developer Community
I n v e n i oEPFLCDS AUTH UAB ...
2002– 2004–2006+ 2002–
Tibor Šimko (CERN) Invenio Technology openlab 2013 29 / 62
![Page 82: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/82.jpg)
Developer Community
I n v e n i oEPFLCDS AUTH UAB ...
2002– 2004–2006+ 2002–
Tibor Šimko (CERN) Invenio Technology openlab 2013 29 / 62
![Page 83: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/83.jpg)
Developer Community
I n v e n i oEPFLCDS AUTH UAB ...
2002– 2004–2006+ 2002–
I n v e n i oADSINSPIRE arXiv
2008– 2009– 2011–
Tibor Šimko (CERN) Invenio Technology openlab 2013 29 / 62
![Page 84: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/84.jpg)
Developer Community
I n v e n i oEPFLCDS AUTH UAB ...
2002– 2004–2006+ 2002–
I n v e n i oADSINSPIRE arXiv
2008– 2009– 2011–
Tibor Šimko (CERN) Invenio Technology openlab 2013 29 / 62
![Page 85: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/85.jpg)
Developer Community
I n v e n i oEPFLCDS AUTH UAB ...
2002– 2004–2006+ 2002–
I n v e n i oADSINSPIRE arXiv
2008– 2009– 2011–
Tibor Šimko (CERN) Invenio Technology openlab 2013 29 / 62
![Page 86: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/86.jpg)
Developer Community
I n v e n i oEPFLCDS AUTH UAB ...
2002– 2004–2006+ 2002–
I n v e n i oADSINSPIRE arXiv
2008– 2009– 2011–
I n v e n i oBlogForever CRISP OpenAIRE M9
2011– 2012– 2009– 2012–
Tibor Šimko (CERN) Invenio Technology openlab 2013 29 / 62
![Page 87: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/87.jpg)
Developer Community
I n v e n i oEPFLCDS AUTH UAB ...
2002– 2004–2006+ 2002–
I n v e n i oADSINSPIRE arXiv
2008– 2009– 2011–
I n v e n i oBlogForever CRISP OpenAIRE M9
2011– 2012– 2009– 2012–
Tibor Šimko (CERN) Invenio Technology openlab 2013 29 / 62
![Page 88: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/88.jpg)
Developer Community
I n v e n i oEPFLCDS AUTH UAB ...
2002– 2004–2006+ 2002–
I n v e n i oADSINSPIRE arXiv
2008– 2009– 2011–
I n v e n i oBlogForever CRISP OpenAIRE M9
2011– 2012– 2009– 2012–
330k LOC - Invenio core sources
10k LOC - INSPIRE overlay sources
Tibor Šimko (CERN) Invenio Technology openlab 2013 29 / 62
![Page 89: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/89.jpg)
Outline
1 IntroductionDigital LibraryInvenio
2 Case StudiesEpisode 1: PythonEpisode 2: GitEpisode 3: TestingEpisode 4: Building Efficient IndexesEpisode 5: NIHEpisode 6: Scalability
3 Conclusions
Tibor Šimko (CERN) Invenio Technology openlab 2013 30 / 62
![Page 90: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/90.jpg)
Outline
1 IntroductionDigital LibraryInvenio
2 Case StudiesEpisode 1: PythonEpisode 2: GitEpisode 3: TestingEpisode 4: Building Efficient IndexesEpisode 5: NIHEpisode 6: Scalability
3 Conclusions
Tibor Šimko (CERN) Invenio Technology openlab 2013 31 / 62
![Page 91: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/91.jpg)
Why Python?
easy to read and understand(good for many temporary developers)suitable for rapid prototyping(good for organic-growth software development model)write code to throw it away
Tibor Šimko (CERN) Invenio Technology openlab 2013 32 / 62
![Page 92: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/92.jpg)
Art of Ikebana
Ikebana, “giving life to flowers”Japanese art of flowerarrangement, “way of flowers”“disciplined art form in whichnature and humanity arebrought together”natural shapes, graceful linesminimalism
Tibor Šimko (CERN) Invenio Technology openlab 2013 33 / 62
![Page 93: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/93.jpg)
Art of Ikebana Programming
example of anonymous functions
Java?
new Callable() {public Object call(Object x) {
return x.times(k)}
}
Python!
lambda x: k * x
Tibor Šimko (CERN) Invenio Technology openlab 2013 34 / 62
![Page 94: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/94.jpg)
Art of Ikebana Programming
example of anonymous functions
Java?
new Callable() {public Object call(Object x) {
return x.times(k)}
}
Python!
lambda x: k * x
Tibor Šimko (CERN) Invenio Technology openlab 2013 34 / 62
![Page 95: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/95.jpg)
Speeding Up Python
bytecode interpreted language: what about speed?Cython permits to write C extensions easilycombining efficiency of C with high-levelness of Python
Example: intbitset.pyx
ctypedef unsigned long long int word_t
ctypedef struct IntBitSet:int sizeint allocatedword_t trailing_bitsint totword_t *bitset
Tibor Šimko (CERN) Invenio Technology openlab 2013 35 / 62
![Page 96: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/96.jpg)
Outline
1 IntroductionDigital LibraryInvenio
2 Case StudiesEpisode 1: PythonEpisode 2: GitEpisode 3: TestingEpisode 4: Building Efficient IndexesEpisode 5: NIHEpisode 6: Scalability
3 Conclusions
Tibor Šimko (CERN) Invenio Technology openlab 2013 36 / 62
![Page 97: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/97.jpg)
Why Git?
good for distributed teamsgood for offline developmentpowerful branching/merging, first class citizenshipcommit early, commit often(to private repositories)rebase and clean when ready(before pushing for public consumption)using pull-on-demand collaboration model(as opposed to shared-push collaboration model)
Tibor Šimko (CERN) Invenio Technology openlab 2013 37 / 62
![Page 98: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/98.jpg)
Git Collaboration
pull-on-demand collaboration modelinherent code review and QA processes before integrationmodules maintainers aka “integration lieutenants”
Tibor Šimko (CERN) Invenio Technology openlab 2013 38 / 62
![Page 99: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/99.jpg)
Git Branches
C1 master
Tibor Šimko (CERN) Invenio Technology openlab 2013 39 / 62
![Page 100: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/100.jpg)
Git Branches
C1 masterC2
Tibor Šimko (CERN) Invenio Technology openlab 2013 39 / 62
![Page 101: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/101.jpg)
Git Branches
C1 masterC2 C3
v1.0.0
Tibor Šimko (CERN) Invenio Technology openlab 2013 39 / 62
![Page 102: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/102.jpg)
Git Branches
C1 masterC2 C3
v1.0.0
C4
Tibor Šimko (CERN) Invenio Technology openlab 2013 39 / 62
![Page 103: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/103.jpg)
Git Branches
C1 masterC2 C3
v1.0.0
C4
M1 maint-1.0
Tibor Šimko (CERN) Invenio Technology openlab 2013 39 / 62
![Page 104: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/104.jpg)
Git Branches
C1 masterC2 C3
v1.0.0
C4
M1 maint-1.0M2
Tibor Šimko (CERN) Invenio Technology openlab 2013 39 / 62
![Page 105: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/105.jpg)
Git Branches
C1 masterC2 C3
v1.0.0
C4
M1 maint-1.0M2
C5
Tibor Šimko (CERN) Invenio Technology openlab 2013 39 / 62
![Page 106: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/106.jpg)
Git Branches
C1 masterC2 C3
v1.0.0
C4
M1 maint-1.0M2
C5
M3
v1.0.1
Tibor Šimko (CERN) Invenio Technology openlab 2013 39 / 62
![Page 107: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/107.jpg)
Git Branches
C1 masterC2 C3
v1.0.0
C4
M1 maint-1.0M2
C5
M3
v1.0.1M4
Tibor Šimko (CERN) Invenio Technology openlab 2013 39 / 62
![Page 108: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/108.jpg)
Git Branches
C1 masterC2 C3
v1.0.0
C4
M1 maint-1.0M2
C5
M3
v1.0.1M4
N1 next
Tibor Šimko (CERN) Invenio Technology openlab 2013 39 / 62
![Page 109: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/109.jpg)
Git Branches
C1 masterC2 C3
v1.0.0
C4
M1 maint-1.0M2
C5
M3
v1.0.1M4
N1 next
C6
Tibor Šimko (CERN) Invenio Technology openlab 2013 39 / 62
![Page 110: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/110.jpg)
Git Branches
C1 masterC2 C3
v1.0.0
C4
M1 maint-1.0M2
C5
M3
v1.0.1M4
N1 next
C6
N2
Tibor Šimko (CERN) Invenio Technology openlab 2013 39 / 62
![Page 111: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/111.jpg)
Git Branches
C1 masterC2 C3
v1.0.0
C4
M1 maint-1.0M2
C5
M3
v1.0.1M4
N1 next
C6
N2
C7
v1.1.0
Tibor Šimko (CERN) Invenio Technology openlab 2013 39 / 62
![Page 112: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/112.jpg)
Git Branches
C1 masterC2 C3
v1.0.0
C4
M1 maint-1.0M2
C5
M3
v1.0.1M4
N1 next
C6
N2
C7
v1.1.0
M5
v1.0.2
Tibor Šimko (CERN) Invenio Technology openlab 2013 39 / 62
![Page 113: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/113.jpg)
Git Branches
C1 masterC2 C3
v1.0.0
C4
M1 maint-1.0M2
C5
M3
v1.0.1M4
N1 next
C6
N2
C7
v1.1.0
M5
v1.0.2
maint-X.Y — release maintenance branches
master — new feature branch
next — things not yet release-ready
Tibor Šimko (CERN) Invenio Technology openlab 2013 39 / 62
![Page 114: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/114.jpg)
Git Development
C1
M1
N1
master
maint-1.0
next
Tibor Šimko (CERN) Invenio Technology openlab 2013 40 / 62
![Page 115: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/115.jpg)
Git Development
C1
M1
N1
master
maint-1.0
next
B1 some-bugfix
Tibor Šimko (CERN) Invenio Technology openlab 2013 40 / 62
![Page 116: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/116.jpg)
Git Development
C1
M1
N1
master
maint-1.0
next
B1 some-bugfix
C2
M2
N2
Tibor Šimko (CERN) Invenio Technology openlab 2013 40 / 62
![Page 117: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/117.jpg)
Git Development
C1
M1
N1
master
maint-1.0
next
B1 some-bugfix
C2
M2
N2
B2
Tibor Šimko (CERN) Invenio Technology openlab 2013 40 / 62
![Page 118: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/118.jpg)
Git Development
C1
M1
N1
master
maint-1.0
next
B1 some-bugfix
C2
M2
N2
B2
M3
merge
Tibor Šimko (CERN) Invenio Technology openlab 2013 40 / 62
![Page 119: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/119.jpg)
Git Development
C1
M1
N1
master
maint-1.0
next
B1 some-bugfix
C2
M2
N2
B2
M3
merge
C3
merge
Tibor Šimko (CERN) Invenio Technology openlab 2013 40 / 62
![Page 120: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/120.jpg)
Git Development
C1
M1
N1
master
maint-1.0
next
B1 some-bugfix
C2
M2
N2
B2
M3
merge
C3
merge
F1 some-new-feature
Tibor Šimko (CERN) Invenio Technology openlab 2013 40 / 62
![Page 121: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/121.jpg)
Git Development
C1
M1
N1
master
maint-1.0
next
B1 some-bugfix
C2
M2
N2
B2
M3
merge
C3
merge
F1 some-new-featureF2
Tibor Šimko (CERN) Invenio Technology openlab 2013 40 / 62
![Page 122: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/122.jpg)
Git Development
C1
M1
N1
master
maint-1.0
next
B1 some-bugfix
C2
M2
N2
B2
M3
merge
C3
merge
F1 some-new-featureF2
C4
merge
Tibor Šimko (CERN) Invenio Technology openlab 2013 40 / 62
![Page 123: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/123.jpg)
Git Development
C1
M1
N1
master
maint-1.0
next
B1 some-bugfix
C2
M2
N2
B2
M3
merge
C3
merge
F1 some-new-featureF2
C4
merge
N3
merge
Tibor Šimko (CERN) Invenio Technology openlab 2013 40 / 62
![Page 124: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/124.jpg)
Git Development
C1
M1
N1
master
maint-1.0
next
B1 some-bugfix
C2
M2
N2
B2
M3
merge
C3
merge
F1 some-new-featureF2
C4
merge
N3
merge
E1 some-experimental-feature
Tibor Šimko (CERN) Invenio Technology openlab 2013 40 / 62
![Page 125: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/125.jpg)
Git Development
C1
M1
N1
master
maint-1.0
next
B1 some-bugfix
C2
M2
N2
B2
M3
merge
C3
merge
F1 some-new-featureF2
C4
merge
N3
merge
E1 some-experimental-featureE2
Tibor Šimko (CERN) Invenio Technology openlab 2013 40 / 62
![Page 126: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/126.jpg)
Git Development
C1
M1
N1
master
maint-1.0
next
B1 some-bugfix
C2
M2
N2
B2
M3
merge
C3
merge
F1 some-new-featureF2
C4
merge
N3
merge
E1 some-experimental-featureE2
N4
merge
Tibor Šimko (CERN) Invenio Technology openlab 2013 40 / 62
![Page 127: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/127.jpg)
Continuous integration
Tibor Šimko (CERN) Invenio Technology openlab 2013 41 / 62
![Page 128: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/128.jpg)
Outline
1 IntroductionDigital LibraryInvenio
2 Case StudiesEpisode 1: PythonEpisode 2: GitEpisode 3: TestingEpisode 4: Building Efficient IndexesEpisode 5: NIHEpisode 6: Scalability
3 Conclusions
Tibor Šimko (CERN) Invenio Technology openlab 2013 42 / 62
![Page 129: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/129.jpg)
Unit testing
test-driven development when appropriatee.g. before/while developing strip_accents(), write:
Example: search_engine_tests.py
class TestStripAccents(unittest.TestCase):"""Test for handling of UTF-8 accents."""
def test_strip_accents(self):"""search engine - stripping of accented letters"""self.assertEqual("memememe",
search_engine.strip_accents('mémêmëmè'))self.assertEqual("MEMEMEME",
search_engine.strip_accents('MÉMÊMËMÈ'))
Tibor Šimko (CERN) Invenio Technology openlab 2013 43 / 62
![Page 130: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/130.jpg)
Functional testing
functional/acceptance/regression testingtestbed site (Atlantis of Institute Fictive Science)e.g. Python mechanize module to emulate browser
Example: websearch_regression_tests.py
class WebSearchSearchEnginePythonAPITest(unittest.TestCase):
"Check typical search engine Python API calls on the demo data."
def test_search_engine_python_api_for_failed_query(self):
"websearch - search engine Python API for failed query"
self.assertEqual([],
perform_request_search(p='aoeuidhtns'))
def test_search_engine_python_api_for_successful_query(self):
"websearch - search engine Python API for successful query"
self.assertEqual([8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
18, 47],
perform_request_search(p='ellis'))
Tibor Šimko (CERN) Invenio Technology openlab 2013 44 / 62
![Page 131: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/131.jpg)
Web testing
sometimes we need to run tests in real browser– e.g. pages with heavy JavaScript
using Selenium extension for Firefox– record and replay browser actions– test for text existence or non-existence on pages– test for link labels and targets
Example: websearch_web_tests.py
class InvenioWebSearchWebTests(InvenioWebTestCase):
def test_search_ellis(self):
"""websearch - web test search for ellis"""
self.browser.get(CFG_SITE_URL)
p = self.browser.find_element_by_name("p")
p.send_keys("ellis")
p.submit()
self.page_source_test(expected_text=[
'Thermal conductivity of dense quark matter ' + \
'and cooling of stars'])
Tibor Šimko (CERN) Invenio Technology openlab 2013 45 / 62
![Page 132: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/132.jpg)
Outline
1 IntroductionDigital LibraryInvenio
2 Case StudiesEpisode 1: PythonEpisode 2: GitEpisode 3: TestingEpisode 4: Building Efficient IndexesEpisode 5: NIHEpisode 6: Scalability
3 Conclusions
Tibor Šimko (CERN) Invenio Technology openlab 2013 46 / 62
![Page 133: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/133.jpg)
Designing A Search Engine
performance-driven design assumptions:– high number of selects, low number of updates– fast searching, slow indexation– cache everything cacheable
search functionality:– search for words, phrases, regular expressions– search in any field, authors, titles, etc
index design:– forward indexes: word1 −→ [rec1, rec2, . . . ]
word2 −→ [rec2, rec7, . . . ]– reverse indexes: rec1 −→ [word1, word8, . . . ]
rec2 −→ [word1, word2, . . . ]Zipf’s law on word frequency:
– few words occur very often (e.g. the)– most words are infrequent (even e.g. boson)
Tibor Šimko (CERN) Invenio Technology openlab 2013 47 / 62
![Page 134: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/134.jpg)
Search Engine Under Cover
Tibor Šimko (CERN) Invenio Technology openlab 2013 48 / 62
![Page 135: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/135.jpg)
Measuring the Performance
three important speed factors to consider:– speed of finding sets (DB Server)– speed of demarshaling sets (DB↔Web App Server)– speed of intersecting sets (Web App Server)
Example: speed of various parts (2002, before optimization)
action / query: "CERN 2002" "of the this"
-----------------------------------------------
fetching 0.28 sec 0.34 sec
demarshaling 0.78 sec 1.10 sec
adding colls 0.37 sec 0.63 sec
intersecting 0.64 sec 1.19 sec
-----------------------------------------------
total search time 2.07 sec 3.22 sec
Tibor Šimko (CERN) Invenio Technology openlab 2013 49 / 62
![Page 136: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/136.jpg)
Optimizing Data Structures
data structures tested:– ‘sorted’ (lists, Patricia trees)– ‘unsorted’ (hashed sets, binary vectors)
fast prototyping: (Python, Lisp in 2002)– throw-away coding to test ideas
Example: lists vs dicts, 350K sets in 800K universe
marshaling lists ..... 532616+532571 bytes in 1.33 sec
demarshaling lists ... 350000+350000 items in 0.10 sec
merging lists ........ 546965 items in 0.34 sec
intersecting lists ... 153035 items in 0.35 sec
marshaling dicts ..... 576491+576450 bytes in 0.87 sec
demarshaling dicts ... 350000+350000 items in 0.36 sec
merging dicts ........ 546965 items in 0.09 sec
intersecting dicts ... 153035 items in 0.15 sec
Tibor Šimko (CERN) Invenio Technology openlab 2013 50 / 62
![Page 137: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/137.jpg)
. . . and the winner is:
binary vectors found the best compromise!using Numeric Python module (in 2002)typical search time gain: 4.0 sec→ 0.2 sec (in 2002)typical indexing time loss: 7 hours→ 4 days (in 2002)mostly spare data modelled via mostly dense data structure?free your mind, think critically
further optimisation:Numeric module not addressing real bits, only bytesso home-made intbitset C extension (2007)
– addressing real bits, saving factor of 8 already– saving space, saving (indexing) time
use of external information retrieval tools (2011)– Solr, Xapian
Tibor Šimko (CERN) Invenio Technology openlab 2013 51 / 62
![Page 138: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/138.jpg)
Outline
1 IntroductionDigital LibraryInvenio
2 Case StudiesEpisode 1: PythonEpisode 2: GitEpisode 3: TestingEpisode 4: Building Efficient IndexesEpisode 5: NIHEpisode 6: Scalability
3 Conclusions
Tibor Šimko (CERN) Invenio Technology openlab 2013 52 / 62
![Page 139: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/139.jpg)
Not Invented Here
technology overview:– load balancing: HAProxy– web application: Apache, WSGI, Python, Flask, Jinja, Cython– database: SQLAlchemy, MySQL/PostgreSQL/SQLite, MongoDB– indexing: Solr, Xapian– caching: Memcached, Redis– UI: Twitter Bootstrap, jQuery– mobile app: Apache Cordova– tools: Git, Trac, Jenkins, Selenium
Tibor Šimko (CERN) Invenio Technology openlab 2013 53 / 62
![Page 140: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/140.jpg)
Example: Invenio “next” branch UI
Tibor Šimko (CERN) Invenio Technology openlab 2013 54 / 62
![Page 141: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/141.jpg)
Example: ZENODO
Tibor Šimko (CERN) Invenio Technology openlab 2013 55 / 62
![Page 142: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/142.jpg)
Outline
1 IntroductionDigital LibraryInvenio
2 Case StudiesEpisode 1: PythonEpisode 2: GitEpisode 3: TestingEpisode 4: Building Efficient IndexesEpisode 5: NIHEpisode 6: Scalability
3 Conclusions
Tibor Šimko (CERN) Invenio Technology openlab 2013 56 / 62
![Page 143: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/143.jpg)
Splitting Web App Server and DB Server
load of CDS Web and DB servers at the split time:
split leads to efficient use of OS resources by lone, non-competingWeb and DB daemon processes
Tibor Šimko (CERN) Invenio Technology openlab 2013 57 / 62
![Page 144: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/144.jpg)
Multi-Node Architecture
800 hits per sec on CDS during Higgs seminar July 4th
LoadBalancer
Apache 1
Apache 2
Worker 1/2
Worker 1/1
Worker 1/3
Worker 2/2
Worker 2/1
Worker 2/3NODE 2
DB Master
DB Slave
Solr
shared fs
Redis
HTTP
HTTP
WSGI
WSGI
SQL R/W
SQL R/O
query
key,val
files
replication
Tibor Šimko (CERN) Invenio Technology openlab 2013 58 / 62
![Page 145: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/145.jpg)
Measuring Scalability
using siege and ab to simulate concurrent users and to measurethroughput on a sample of typical URLs
Example: inspirehep.net under gentle siege
$ siege -d 1 -c 20 -t 1m -f inspirehep_urls.txtTransactions: 1329 hitsAvailability: 100.00 %Elapsed time: 60.23 secsData transferred: 37.12 MBResponse time: 0.41 secsTransaction rate: 22.07 trans/secThroughput: 0.62 MB/secConcurrency: 8.96Successful transactions: 1329Failed transactions: 0Longest transaction: 3.05Shortest transaction: 0.01
Tibor Šimko (CERN) Invenio Technology openlab 2013 59 / 62
![Page 146: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/146.jpg)
Measuring Scalability: “ab” on top of “siege”
Tibor Šimko (CERN) Invenio Technology openlab 2013 60 / 62
![Page 147: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/147.jpg)
Outline
1 IntroductionDigital LibraryInvenio
2 Case StudiesEpisode 1: PythonEpisode 2: GitEpisode 3: TestingEpisode 4: Building Efficient IndexesEpisode 5: NIHEpisode 6: Scalability
3 Conclusions
Tibor Šimko (CERN) Invenio Technology openlab 2013 61 / 62
![Page 148: Invenio Technology - Selected Practical Software ...€¦ · Invenio Technology Selected Practical Software Development Lessons From A Large Digital Library System Tibor Šimko](https://reader033.fdocuments.net/reader033/viewer/2022052011/6026c5143e75a82c841ad873/html5/thumbnails/148.jpg)
Conclusions
selected lessons from building a digital library system– 350,000+ LOC from 110+ authors over 10+ years
selected technology:– load balancing: HAProxy– web application: Apache, WSGI, Python, Flask, Jinja– database: SQLAlchemy, MySQL/PostgreSQL/SQLite, MongoDB– caching: Memcached, Redis– UI: Twitter Bootstrap, jQuery– project tools: Git, Trac, Jenkins, Selenium
morale from selected anecdotes?– value of rapid prototyping– value of organic-growth software development model– value of coding aesthetics and minimalism– “Never Lose A Holy Curiosity” (A. Einstein)
Tibor Šimko (CERN) Invenio Technology openlab 2013 62 / 62