Version Your Cloud: Using Perforce to Manage Your Object Storage
-
Upload
perforce -
Category
Technology
-
view
1.364 -
download
2
description
Transcript of Version Your Cloud: Using Perforce to Manage Your Object Storage
![Page 1: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.fdocuments.net/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/1.jpg)
#
Daniel AnolikSolutions Architect, Jeppesen
Version Your Cloud
![Page 2: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.fdocuments.net/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/2.jpg)
#
Daniel AnolikSolutions Architect, Airborne Navigation SystemsJeppesen (A Boeing Company)• 8 years with Jeppesen
• 15 years of direct SCM experience, 8 with Perforce
• Software developer, solutions architect, Scrum master, bluegrass mandolin picker, wilderness survival enthusiast…
![Page 3: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.fdocuments.net/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/3.jpg)
#
Daniel AnolikSolutions Architect, Airborne Navigation SystemsJeppesen (A Boeing Company)• 8 years with Jeppesen
• 15 years of direct SCM experience, 8 with Perforce
• Software developer, solutions architect, Scrum master, bluegrass mandolin picker, wilderness survival enthusiast,
and lucky father of this baby girl:
![Page 4: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.fdocuments.net/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/4.jpg)
#
Jeppesen at a Glance• 80-year strong company founded in aviation safety• Wide array of software supporting:
– airborne navigation systems– flight planning services– pilot training – airline operations management
![Page 5: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.fdocuments.net/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/5.jpg)
#
Supplier of Critical Aviation Data
![Page 6: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.fdocuments.net/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/6.jpg)
#
• Distribute many data types to many applications running on many platforms
• Support frequent data updates for hundreds of thousands of customers around the world
• Transfer hundreds of terabytes of data each month• Don’t be a bottleneck!
The Challenge
![Page 7: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.fdocuments.net/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/7.jpg)
#
Initial Plan
Data Brokering Magic
Jeppesen DataProduction
Magic
JeppesenInfrastructure CDM
Cloud Object
Storage
Workflow Magic
Data PublishingInterface
Boring File Server
![Page 8: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.fdocuments.net/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/8.jpg)
#
The system must support:• Fine read/write access controls within storage tree
• Atomic change-sets for publishing files
• Detailed history tracking
• Individual file restores
• Data lifecycle management
Hmm…It’s starting to sound like we need a real CMS system in there
Pesky Implementation Constraints
![Page 9: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.fdocuments.net/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/9.jpg)
#
Improved Plan
Data Brokering Magic
Jeppesen DataProduction
Magic
JeppesenInfrastructure CDM
Cloud Object
Storage
Workflow Magic
Data PublishingInterface
![Page 10: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.fdocuments.net/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/10.jpg)
#
• Supported all our implementation constraints with minimal customization
• Leverage existing in-house expertise
• P4D had very stable track record at Jeppesen
• Efficient, scalable blob storage
Why Perforce vs Other CMS?
![Page 11: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.fdocuments.net/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/11.jpg)
#
• Keeping changes atomic:– File sets are submitted to P4 as single changelist– Data is processed by the changelist– Data is published by the changelist
• Files are never ‘available’ to our applications until all files in a given changelist are available
Leveraging Perforce: Changelists
![Page 12: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.fdocuments.net/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/12.jpg)
#
• Changelists tell us:– What files were published,– when,– and by whom
• Spec depots!• Soft deletes of files minimize ‘clutter’ while still
supporting data retention requirements
Leveraging Perforce: History
![Page 13: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.fdocuments.net/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/13.jpg)
#
• Custom counters– are defined for each data type
– track last successful changelist processed
– provide a central, durable mechanism for state tracking
across different work nodes in the system
Leveraging Perforce: Counters
![Page 14: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.fdocuments.net/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/14.jpg)
#
• Job queue are simple files in p4
• In process jobs are ‘locked’
• Completed jobs are ‘deleted’
• Full job history is retained
Leveraging Perforce: Locks
![Page 15: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.fdocuments.net/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/15.jpg)
#
Leveraging Perforce: Branching
Private “Staged” Data
Public “Promoted” Data
p4 integrate
Depot paths represent the promotional model of the data
![Page 16: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.fdocuments.net/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/16.jpg)
#
• P4 protections restrict access by:– depot path/data library– pattern matching/file extensions – IP addresses
• Service accounts granted read and write access to only what they need
Leveraging Perforce: Security
![Page 17: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.fdocuments.net/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/17.jpg)
#
• 100% of interaction with Perforce fully automated through the P4Java API
• Custom utilities used to:– Read and write data in depot
– Manage branch integrations
– Read and update counters
– Read changelist details
– ‘Expire’ content
Leveraging Perforce: Java API
![Page 18: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.fdocuments.net/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/18.jpg)
#
All the great capabilities of cloud based object storage, plus true configuration management:
– Detailed history tracking and extreme auditability
– Reproducibility: roll back any part of your ‘cloud’ to any point in time
– Atomic changelist management of file sets
The Result: Object Storage++
![Page 19: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.fdocuments.net/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/19.jpg)
#
Any Questions?
#