Who Watches the Watchers?
-
Upload
perforce -
Category
Technology
-
view
162 -
download
0
description
Transcript of Who Watches the Watchers?
![Page 1: Who Watches the Watchers?](https://reader033.fdocuments.net/reader033/viewer/2022061118/5469b9d2af7959cb768b53a6/html5/thumbnails/1.jpg)
#
Mark HarrisonTech Lead, Asset ManagementPixar
Who Watches the Watchers?
![Page 2: Who Watches the Watchers?](https://reader033.fdocuments.net/reader033/viewer/2022061118/5469b9d2af7959cb768b53a6/html5/thumbnails/2.jpg)
#
Who Can You Trust?
![Page 3: Who Watches the Watchers?](https://reader033.fdocuments.net/reader033/viewer/2022061118/5469b9d2af7959cb768b53a6/html5/thumbnails/3.jpg)
#
![Page 4: Who Watches the Watchers?](https://reader033.fdocuments.net/reader033/viewer/2022061118/5469b9d2af7959cb768b53a6/html5/thumbnails/4.jpg)
#
![Page 5: Who Watches the Watchers?](https://reader033.fdocuments.net/reader033/viewer/2022061118/5469b9d2af7959cb768b53a6/html5/thumbnails/5.jpg)
#
• Data• Metadata (file sizes, checksums, owners, etc)
Who Can You Trust(With Your Perforce Assets)?
![Page 6: Who Watches the Watchers?](https://reader033.fdocuments.net/reader033/viewer/2022061118/5469b9d2af7959cb768b53a6/html5/thumbnails/6.jpg)
#
• Very Good Track Record• P4 verify• P4 checkpoint• No Known Undetected Failures!
Trust Perforce!
![Page 7: Who Watches the Watchers?](https://reader033.fdocuments.net/reader033/viewer/2022061118/5469b9d2af7959cb768b53a6/html5/thumbnails/7.jpg)
#
• What if it does mess up?– Undetected error?– Metadata goes odd?
• It’s your job on the line!
Don’t Trust Perforce!
![Page 8: Who Watches the Watchers?](https://reader033.fdocuments.net/reader033/viewer/2022061118/5469b9d2af7959cb768b53a6/html5/thumbnails/8.jpg)
#
• People are Great!• People hardly ever make mistakes!• People are always looking out for your best
interests!
Trust People!
![Page 9: Who Watches the Watchers?](https://reader033.fdocuments.net/reader033/viewer/2022061118/5469b9d2af7959cb768b53a6/html5/thumbnails/9.jpg)
#
• Some People are Bad!• Some, just incompetent!• Some, both!• How can you tell? It doesn’t matter!• Intentional Data/Metadata Corruption
– Bad guy made a src code mod• Example, modded code, blame on msundy
– Crazy man put his picture in a film
Don’t Trust People!
![Page 10: Who Watches the Watchers?](https://reader033.fdocuments.net/reader033/viewer/2022061118/5469b9d2af7959cb768b53a6/html5/thumbnails/10.jpg)
#
• P4 Verify?– Good but not good enough– Intentional hacking– Plain old Bug
Trust, But Verify!
![Page 11: Who Watches the Watchers?](https://reader033.fdocuments.net/reader033/viewer/2022061118/5469b9d2af7959cb768b53a6/html5/thumbnails/11.jpg)
#
• “Who shall guard the self-same guardians?”• Need to have two systems that cross-check and
validate• Each system serves as a check on the other
– Catches (we hope!!) bugs and other oddities– Catches (we hope!!) intentional data modification
• Is two enough?– Down that path lies insanity
Quis custodiet ipsos custodes?
![Page 12: Who Watches the Watchers?](https://reader033.fdocuments.net/reader033/viewer/2022061118/5469b9d2af7959cb768b53a6/html5/thumbnails/12.jpg)
#
• We mirror all Perforce metadata (including checksums, sizes) in the Templar database
• Perforce is still the “system of record”• But, we can see if the system of record ever
changes• A Bad Guy will have to subvert both Perforce and
Templar to do his evil deeds
The Templar Approach
![Page 13: Who Watches the Watchers?](https://reader033.fdocuments.net/reader033/viewer/2022061118/5469b9d2af7959cb768b53a6/html5/thumbnails/13.jpg)
#
• Templar database initially mirrored Perforce data and metadata for “offline verify”
• Basically a “p4 verify” that bypassed the P4D– Didn’t cause a slowdown
• Takeaway:– Have a good abstraction, it can be used in multiple
places
Side Effect of Previous Work
![Page 14: Who Watches the Watchers?](https://reader033.fdocuments.net/reader033/viewer/2022061118/5469b9d2af7959cb768b53a6/html5/thumbnails/14.jpg)
#
SQL> desc p4_files
• ID generic asset id (234363343)
• P4PATH p4 path ("//ts3/myfile.jpg")
• HEADVER the head revision (3)
• REPOS repository name ("ts3")
SQL> desc p4_versions
• ID generic asset id (234363343)
• VERSIONOF this revision is a version of file x
• P4PATH p4 path ("//ts3/myfile.jpg")
• VERSION version number ("3")
• ISDELETED true if deleted version
• REPOS repository name ("ts3”)
Database Structure
![Page 15: Who Watches the Watchers?](https://reader033.fdocuments.net/reader033/viewer/2022061118/5469b9d2af7959cb768b53a6/html5/thumbnails/15.jpg)
#
SQL> desc asset_metadata
• ID generic asset id (234363343)
• md5sum checksum as per P4
• filesize filesize as per P4
• etc…
• For each asset ID, we store the metadata we care about
• Not just P4, but other asset types as well• All keyed by asset ID• Updates are by P4 checkin trigger
Database Structure (cont.)
![Page 16: Who Watches the Watchers?](https://reader033.fdocuments.net/reader033/viewer/2022061118/5469b9d2af7959cb768b53a6/html5/thumbnails/16.jpg)
#
• So far, things are good• Never detected a P4 error• Never detected an intentional modification• (Maybe the bad guys are just that good!)• But we can correlate our results with the Backup
System Metadata• All Systems Agree!!
Results
![Page 17: Who Watches the Watchers?](https://reader033.fdocuments.net/reader033/viewer/2022061118/5469b9d2af7959cb768b53a6/html5/thumbnails/17.jpg)
#
• Good to know that our systems are working• Good to know that our colleagues seem
trustworthy.• We don’t have to trust each other to like each
other.
Conclusion
![Page 19: Who Watches the Watchers?](https://reader033.fdocuments.net/reader033/viewer/2022061118/5469b9d2af7959cb768b53a6/html5/thumbnails/19.jpg)
#
Mark HarrisonTech Lead, Data ManagementPixar Animation StudiosMark is in charge of Pixar’s Data Management Group where he has a 50 year charter to store all data and metadata related to the Studio's feature films and related work.
Prior to that he lived in China and was the Chief Software Architect of the China Internet. His software supported the growth from 200K users to the current base of 350M users.
He studied Computer Science, worked in Texas, and wrote a couple of computer books.
(Speaker photo here)
![Page 20: Who Watches the Watchers?](https://reader033.fdocuments.net/reader033/viewer/2022061118/5469b9d2af7959cb768b53a6/html5/thumbnails/20.jpg)
#
RESOURCESCD Report: info.perforce.com/cd-report
White Paper: perforce.com/white-paper-link
Template: perforce.com/template