1. CONNECT. TRANSFORM. AUTOMATE. Big Data Meets FME

2. Agenda What is Big Data Big Data Challenges FME and Big Data Big Data Technologies DynamoDB Workflow MarkLogic Workflow 3. Big Data and Cloud Big Data needs big resources Big data stores Big processing power Big bandwidth Cloud technology gives you this for fraction of traditional cost! 4. Big Data and FME Big Data is a new data classification for FME. Big Data is no different than other data to FME FME Cloud is a natural fit for data in the Cloud FME makes it easy to leverage the power of Big Data 5. Big Data and FME Support Amazon S3 Limitless internet based storage Amazon RDS See blog article on Amazon RDS (PostGIS/SQLServer/Oracle) Amazon DynamoDB NoSQL limitless database service Amazon RedShift Petabyte scale database warehouse service. Google BigQuery Superfast append only tables MarkLogic Large XML based NoSQL database 6. Big Data Challenges Loading Data Lacks Spatial Support Big Data Analysis Querying and Exporting Data 7. Why Demo FME with MarkLogic and DynamoDB? Different from other databases supported by FME 8. Demo #1 Limitless Spatial Database 9. DynamoDB NoSQL SSD-based database service No limit on size of database Specify the needed performance Autoscale thru Dynamic DynamoDB Amazon EMR (Hadoop) integration 10. Dynamodb Big Data Demo Spatially locate and store anything in DynamoDB! 11. Dynamodb Demo Index Strategy Generate GeoHash Index for each feature and Write to GeoHashSpatialIndex 12. DynamoDB Demo Storing Vector, Raster, Lidar Write small features to DynamoDB Write large features to Amazon S3, link to DynamoDB 13. DynamoDB Demo Storing Geocoded Images Generate Geohash record of picture location Write Image to S3, link to DynamoDB 14. DynamoDB Demo Spatially Locate and Store Any document or Web Resource Generate Geohash index Write Document to S3 and Link to DynamoDB location 15. DynamoDB Demo Retrieve any stored document Write URI Link to DynamoDB Generate Geohash index location 16. What is ? NoSQL database XML optimized Powerful search and analysis Native spatial support XML based data model (GML, XML, etc.) Deploy on Hadoop HDFS 17. FME and MarkLogic A Natural Fit Convert data to XML/GML* Easily load XML into MarkLogic with FME Process and convert XML results FME 2014: New schema based GML Writer 18. Demo #1a - Loading MarkLogic Convert GIS / CAD data to GML (XML) Compose REST request to PUT to MarkLogic database 19. 1.Convert GIS / CAD data into Valid GML 2.Generate Key Fields 3.Build insert message 4.Execute PUT REST call MarkLogic accepts any valid XML just PUT it! Loading GIS to MarkLogic 20. Loading GIS to MarkLogic with FME 21. Demo #1b Exporting from MarkLogic GET Query to find URIs for features of interest GET Query using URIs to get feature XML/GML, then Conversion to format of choice (CAD, GIS ) /WFS 22. Exporting XML from MarkLogic 1. Query database via GET request 2. Parse search result and compose GET feature request 3. Extract attributes and geometry from result 4. Validate and write XML Result 23. Exporting XML from MarkLogic Search GET request to find URI based on query: http://localhost:8003/v1/keyvalue?element=comment&value=AIXM.Chicago Document Retrieval GET request based on URI: http://localhost:8003/v1/documents?uri=/docs/myXML_653c46c3-fdfb-4837-ae1c- 49735dd29356.xml 24. AIXM from MarkLogic via FMEServer http://UHURA/fmedatastreaming/Demos/QueryMarkLogicDB. fmw?Element=airportCode&Value=CYVR /AIXM 25. AIXM from MarkLogic via FMEServer 26. Summary Big Data = big new opportunities FME great for working with Big Data Cloud model is a natural fit for Big Data This is just the beginning - more to come! 27. Thank You! Questions? For more information: info@safe.com www.safe.com