Improved Index Compression Techniques for Versioned Document Collections
SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases
Transcript of SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases
![Page 1: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/1.jpg)
Visualization of Inferred Versioned Schemasfrom NoSQL Databases
Alberto Hernández Chillón Diego Sevilla Ruiz Jesús García [email protected] [email protected] [email protected]
Cátedra SAES-UMUUniversity of Murcia
Faculty of Computer ScienceUniversity of Murcia
Faculty of Computer ScienceUniversity of Murcia
November 15, 2016
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 1 / 41
![Page 2: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/2.jpg)
About the authors
Member of the Cátedra SAES team since 2014
M.Sc. in Computer Science from the UM
MDE, automatic code generation, NoSQL databases
Associate professor at the Faculty of Computer Science of the UM
M.Sc and Ph.D. in Computer Science from the UM
NoSQL databases, distributed systems, testing
Professor at the Faculty of Computer Science of the UM since 1984
Head of the Modelum Group
MDE, DSL, Software modernization, reverse engineering
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 2 / 41
![Page 3: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/3.jpg)
Index
1 Introduction and context
2 Inference process
3 Schema visualization
4 Conclusions and future work
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 3 / 41
![Page 4: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/4.jpg)
Index
1 Introduction and context
2 Inference process
3 Schema visualization
4 Conclusions and future work
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 4 / 41
![Page 5: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/5.jpg)
Introduction and context
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 5 / 41
![Page 6: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/6.jpg)
NoSQL systems
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 6 / 41
![Page 7: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/7.jpg)
Usefulness of an explicit schema
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 7 / 41
![Page 8: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/8.jpg)
What kind of NoSQL systems? - Schemaless
{" person_id ": "123" ,"type": " Person "," lastName ": "Rush"," firstName ": " Christopher "," address ": "C/Gran Via, 13, Madrid "
},{
" person_id ": "456" ,"type": " Person "," lastName ": " England "," firstName ": " Wayne "," address ": {
" street ": "Av. Pinos , 24","city": " Murcia " }
},{
" person_id ": "789" ,"type": " Person "," lastName ": " Hoover "," firstName ": " Quinton "," address ": " Ronda Norte , 15, Murcia ","age": 35
}
NoSQLDatabase
Data non-uniformityDifferent versions for thesame data
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 8 / 41
![Page 9: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/9.jpg)
What kind of NoSQL systems? - Aggregations
{"_id": "2","type": "book"," title ": " Writing and Querying MapReduce
Views in CouchDB "," publisher_id ": "928672" ," author ":{
"_id": "101" ,"type": " author ","name": " Bradley Holt"," company ": {
"_id": "324" ,"type": " company ","name": "IBM Cloudant "," country ": "USA"}
},},{
"_id": "928672" ,"type": " publisher ","name": "O’ Reilly Media ","city": " Newton "
}
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 9 / 41
![Page 10: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/10.jpg)
Proposed objective
A DataVersity report (2015) indicated essential functionalityrequired in the near future:
Model visualizationCode generation from schemasMetadata management
Our goal is to design and implement a tool which will allow usto visualize NoSQL schemas:
Taking into account concepts such as versions......having in mind the underlying inference process......among other things
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 10 / 41
![Page 11: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/11.jpg)
Index
1 Introduction and context
2 Inference process
3 Schema visualization
4 Conclusions and future work
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 11 / 41
![Page 12: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/12.jpg)
Process overview
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 12 / 41
![Page 13: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/13.jpg)
Inference process (I)
NoSQLDatabase MapReduce
ObjectVersions(JSON)
JSONInjection
JSONModel
JSONMetamodel
SchemaReverse Eng
SchemaModel
ApplicationGeneration
SchemaViewer/
DataValidator/MigrationAssistant
Applications SchemaMetamodel
inst
ance
instance
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 13 / 41
![Page 14: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/14.jpg)
Initial metamodel
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 14 / 41
![Page 15: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/15.jpg)
Polished metamodel (I)
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 15 / 41
![Page 16: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/16.jpg)
Polished metamodel (II)
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 16 / 41
![Page 17: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/17.jpg)
Metamodel elements
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 17 / 41
![Page 18: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/18.jpg)
Schema versions (I)
Schema version for Book_1
...{
"_id": "53","type": "Book"," title ": " Book_1 "," hasContent ":[{
"_id": "64","type": " Content "," chapters ": 3," pages ": 17
}]," hasAuthors ":[{
"_id": "155" ,"type": " Author ","name": " Author_155 "," hasCompany ":[{
"_id": "123" ,"type": " Company ","name": " Company_123 "," country ": " Country_1 "
}]}]
}...
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 18 / 41
![Page 19: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/19.jpg)
Schema versions (II)
Schema version for Book_2...{
"_id": "89","type": "Book"," title ": " Book_2 "," hasAuthors ":[{
"_id": "77","type": " Author ","name": " Author_77 "," hasCompany ":[{
"_id": "61","type": " Company ","name": " Company_61 "," country ": " Country_2 "
}]}]
}...
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 19 / 41
![Page 20: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/20.jpg)
Schema versions (III)
Schema version for Author_2...{
"_id": "99","type": " Author ","name": " Author_2 "," company ": " Company_3 "," country ": " Country_3 "
}...
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 20 / 41
![Page 21: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/21.jpg)
Index
1 Introduction and context
2 Inference process
3 Schema visualization
4 Conclusions and future work
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 21 / 41
![Page 22: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/22.jpg)
Schema visualization
Why using Sirius?Views we need:
General tree viewGlobal schema viewSchema version viewsEntity detail view
Ability to create different viewpoints and navigabilitymechanisms between themAutomatic generation of an embedded editor in the EclipseenvironmentExtensibility
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 22 / 41
![Page 23: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/23.jpg)
Design process
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 23 / 41
![Page 24: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/24.jpg)
Schema visualization - Input model
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 24 / 41
![Page 25: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/25.jpg)
Schema visualization - Tree viewpoint (I)Schema versions grouped by Entity
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 25 / 41
![Page 26: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/26.jpg)
Schema visualization - Tree viewpoint (II)Schema versions grouped by Version
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 26 / 41
![Page 27: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/27.jpg)
Schema visualization - Tree viewpoint (III)List of Entities, Versions and Properties
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 27 / 41
![Page 28: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/28.jpg)
Schema visualization - Global schema (I)Diagram of Entities, Versions and Properties
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 28 / 41
![Page 29: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/29.jpg)
Schema visualization - Global schema (II)Implementation of the Global schema
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 29 / 41
![Page 30: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/30.jpg)
Schema visualization - Schema version (I)Schema version root and its associations
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 30 / 41
![Page 31: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/31.jpg)
Schema visualization - Schema version (II)Implementation of the Schema view
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 31 / 41
![Page 32: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/32.jpg)
Schema visualization - Schema version (III)Variation with embedded direct associations
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 32 / 41
![Page 33: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/33.jpg)
Schema visualization - Schema version (IV)Implementation of the Schema view
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 33 / 41
![Page 34: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/34.jpg)
Schema visualization - Entity detailEntity detail with its Versions associations
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 34 / 41
![Page 35: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/35.jpg)
Navigation between views
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 35 / 41
![Page 36: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/36.jpg)
Index
1 Introduction and context
2 Inference process
3 Schema visualization
4 Conclusions and future work
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 36 / 41
![Page 37: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/37.jpg)
To sum up (I)
About Sirius...4 Built in the Eclipse Modeling Framework4 Easily deployable along with the metamodel into plugins4 Different viewpoints, layers and customization options4 Less development time and easier to extend8 AQL might be quite tricky sometimes when defining actions8 Learning curve for creating complex examples may be too harsh
? Language/model maintenance and evolution? Managing large input models
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 37 / 41
![Page 38: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/38.jpg)
To sum up (II)
About the tool...We developed one of the first approaches for NoSQLvisualization considering versionsTool to visualize NoSQL database schemas and schema versionswith Sirius
Tree viewpoint and global schema viewpointSchema version and entity detail viewpoint
...but there is still a lot of work to do:Polish the NoSQL_Schema metamodelImprove the Sirius viewpoint definitionsImplement the editor functionality
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 38 / 41
![Page 39: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/39.jpg)
Future work (I) - DB alterations
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 39 / 41
![Page 40: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/40.jpg)
Future work (II) - DB migration
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 40 / 41
![Page 41: SiriusCon2016 - Visualization of Inferred Versioned Schemas from NoSQL Databases](https://reader031.fdocuments.net/reader031/viewer/2022030317/5870ec311a28abcf288b6f41/html5/thumbnails/41.jpg)
[email protected] https://github.com/Soltari https://linkedin.com/in/albertohc
Alberto H.C. & Diego S.R. & Jesús G.M. NoSQL Visualization November 15, 2016 41 / 41