ChannelStream: The challenges of continuously delivering
-
Upload
jfitzell -
Category
Technology
-
view
717 -
download
3
description
Transcript of ChannelStream: The challenges of continuously delivering
The Challenge of Continuously Delivering
Julian Fitzell, CincomESUG, EdinburghAugust 24, 2011
Image © Petr Vins: http://www.sxc.hu/photo/1054507
ChannelStream
Document Output Management
Control
Enrichment
OMR
Marketing
Barcode Datamatrix
Output
Why?
• Postal contract savings
• Saves licking envelopes
• Personalization and cross-marketing
• Avoid IT/supplier for customization
• Enhance legacy output
ChallengesImage © Tim Regan: http://www.flickr.com/photos/dumbledad/3225255407/
Slow development
Manual process
Test environment not
easily reproducible
External dependencies
fragile
Test data unversioned
Configuration files unversioned
False positives
Unclear separation between
unit/functionalTests not always run
Tests not always updated
Building images slow +
error-prone
Tests slow/hard to run
Infrequent integration
One developer integrating
Reduced confidence from automated tests
STRESS!
+Frequent releases
Integration is risky/hard
Difficulty tracking versions
So...
• Make building images easier
• Make running tests easier
• Make integration easier
Jez Humble, http://continuousdelivery.com/2010/08/continuous-delivery-vs-continuous-deployment/
con·tin·u·ous de·liv·er·y• putting the release schedule in the
hands of the business• ensuring your software is always
production ready throughout its entire lifecycle
• making any build potentially releasable to users at the touch of a button
Goals
• Reduce cycle time between an idea and its availability to users
• Deliver high-quality software reliably and efficiently
Frequent, automated releases:
• Decrease stress• Eliminate errors and improve
auditability• Make feedback more immediate• Protect against reliance on individual
expertise• Reduce wasted time on repetitive tasks
Make releases boring
• Automate (almost) everything
• Version everything
• Deploy everywhere the same way
• If it hurts, do it more often
• Everyone is responsible for delivery
Development+
Integration
Problems
• Contention over “head”
• One person doing most of the integration
• Delayed integration => harder integration
• Poor visibility of changes
• Hard to predict/analyze failure
Integration
Issue tracker + emailIntegration
Task/merge toolsIntegration
Integrate often
• Hide functionality (feature toggles)
• Incremental, releasable changes
• “Branch by abstraction” for large-scale changes
• Use components to decouple parts
Integration
Image © Lars Sundstrom: http://www.sxc.hu/photo/1082530
Remaining Issues
• Restructurings can still be tricky when branches are outstanding
• “Integration token” still just informal conversation
Integration
Building
Problems
• Manual builds on developer machines
• Wasted developer time
• “Here, take my image”
• Taking shortcuts
• “Works on my machine”
Building
Helper scripts + docsBuilding
Fully automated scripts
Builder
-filein foo-wrapper.ws
foo.ws
foo-wrapper.ws
Target
Building
Builds run (at least)dailyBuilding
Archive built imagesBuilding
Image © 2009 by Urbancode
Run automatically on ALL code changes
Avoid versioning split between SVN + Store
Rationalize build/version numbering
!"#$%&'()*+,%&-+.+/011+23&45(+67)%3)+8359)+:00+.+;4)7)4$%(*+<=5'+11>>?+.+@A+/>:-B?B-C000++.+DDD-3"#$%&'()-&'E+.+F+/00C+#G+!"#$%&'()+
&'EE'%H4$&)+)%'3I=+9=$9+"'445%I+59+'39+%'+4'%I)"+")J35")K+9=)+)LL'"9+9'+J3$45LG+59+$K+!"#$"%-+M$K5&+;,+5K+%'D+)$KG+)%'3I=+9'+#)+$%+!"&%'(%)*$&%+4)7)4+$&95759G-+
!"#$%#&'()*+,$-.#&'()/+01#&')2&%)3+,-41#&')N59=+'3"+E'()4+'L+E$93"59G+5%+H4$&)+D)+$")+")$(G+9'+$((")KK+'3"+9'H5&K+5%+2%9)"H"5K)+;'%95%3'3K+,%9)I"$95'%A+M354(5%I*+O)H4'G5%I*+P)K95%I+$%(+Q)H'"95%I-+P=)K)+9'H5&K+&'7)"+9=)+)KK)%95$4+)4)E)%9K+'L+9=)+)%(R9'R)%(+M354(+S5L)&G&4)*+9=)+T'3"%)G+L"'E+K'3"&)+&'()+9'+K'L9D$")+5%+H"'(3&95'%-+
!"#$%#&')U$"()%R7$"5)9G*+()7)4'H)"R&)%9"5&+;,+5K+$#'39+L$K9+L))(#$&V+L"'E+#354(5%I+9=)+K'L9D$")-+N=)%+;,+E))9K+)%9)"H"5K)+&'%&)"%K*+#354(+E$%$I)E)%9*+()H)%()%&5)K+#)9D))%+H"'T)&9K*+$%(+$+&'%9"'4+'7)"+9=)+#354(+H"'&)KK+#)&'E)+&"595&$4+)4)E)%9K-+
W'K9+%)D+H"'T)&9K+K9$"9+'39+D59=+#354(K+H)"L'"E)(+'%+()7)4'H)"+E$&=5%)K+$%(+D59='39+$+K9$%($"(+H"'&)KK-+<%)+()7)4'H)"+#354(K+5%+9=)5"+,O2+D=54)+$%'9=)"+3K)K+$+#354(+K&"5H9-+P=)+4)$K9+E$93")+9)$EK+3K)+9=)K)+#354(K+L'"+9)K95%I+'"+)7)%+L'"+")4)$K)K+9'+H"'(3&95'%-+P=)+H"'#4)EK+")K3495%I+L"'E+9=5K+4$&V+'L+&'%9"'4+$")+J35&V4G+$HH$")%9+9'+E'K9+9)$EK*+$%(+9=3K+K9$"9+$+K)$"&=+L'"+#)99)"+$49)"%$957)K-+
+
+
P=)+L5"K9+K9)HK+L'"+E$93"5%I+9=)+#354(+$")+K9$%($"(5X5%I+9=)+#354(+H"'&)KK+$%(+H)"L'"E5%I+'LL5&5$4+#354(K+R()7)4'H)"+E$&=5%)+L'"+9=)+#354(+E)$%K+&=$%I)K+5%+'%)+
5K+")9"5)7)(+L"'E+K'3"&)+&'%9"'4+9'+3K)+5%+#354(K-+P=5K+E$G+#)+9=)+4$9)K9+L"'E+9=)+95H+'L+$+#"$%&=*+'"+4$#)4)(+K'3"&)+&'()*+'"+K'E)9=5%I+)4K)-+P=)+5EH'"9$%9+H$"9+5K+9=$9+9=)+&='K)%+&'%7)%95'%+#)+$HH45)(+&'%K5K9)%94G-+N59=+9=)K)+H"$&95&)K+5%+H4$&)*+9=)+9)$E+=$K+")$&=)(+$%+!"&'+),-&+'.+4)7)4+'L+#354(+&'EH)9)%&)-+
6%G+9)$E+3K5%I+K9$%($"(+;'%95%3'3K+,%9)I"$95'%+D544+9$V)+9=5K+$+K9)H+L3"9=)"+$%(+$39'E$9)+)Y)&395'%+'L+9=)+#354(+K9)HK-+P=)+#354(+K)"7)"+D544+(5&9$9)+E$&=5%)K*+K'3"&)R&'()+"34)K*+$%(+"3%+9='K)+#354(+K9)HK*+
Building
http://www.urbancode.com/html/resources/white-papers/
Testing
Problems• Tests took up to 16 hours to run
• Test environment hard to create
• Failing tests sometimes pass when re-run
• Not everyone could run tests or be sure of results
• High cost of test maintenance
• Sometimes quicker to test manually than fix the test environment
Testing
Done
• Optimize tests (now run in ~ 2 hours)
• Separate unit/regression from functional/integration tests
• Store test data in a central location
Testing
Run tests automaticallyTesting
Image © 2009 by Urbancode
Run regression and smoke tests as part of build and notify of errors by email
Run functional tests after build
Version control test data
Automate creation of testing environment!"#$%&'()*+,%&-+.+/011+23&45(+67)%3)+8359)+:00+.+;4)7)4$%(*+<=5'+11>>?+.+@A+/>:-B?B-C000++.+DDD-3"#$%&'()-&'E+.+F+/00C+#G+!"#$%&'()+
H)$EI+$9+9=)+!"#$"%+4)7)4+J"$&95&)+&'%95%3'3I+()J4'GE)%9*+$39'E$95&$44G+()J4'G5%K+9'+J"'(3&95'%+D59='39+$%G+E$%3$4+5%9)"7)%95'%A+6+#354(+5I+&")$9)(+9=)%+$39'E$95&$44G+()J4'G)(+9="'3K=+9=)+9)I95%K+)%75"'%E)%9I-+6I+59+E'7)I+9="'3K=+9=)+J5J)45%)*+9=)+#354(+5I+I3#L)&9)(+9'+$39'E$9)(+9)I95%K+$9+)7)"G+I9$K)*+$%(+'%&)+59+=$I+J$II)(+$44+9=)+$JJ"'J"5$9)+M3$459G+K$9)I*+59+5I+5EE)(5$9)4G+()J4'G)(+9'+J"'(3&95'%-+8'E)+I)"5'3I+('9N&'E+$JJ45&$95'%I+")4)$I)+&=$%K)I+D59=5%+$%+='3"+'O+59+")$&=5%K+I'3"&)+&'%9"'4-+<#75'3I4G*+9=)+$39'E$9)(+9)I95%K+E3I9+#)+7)"G+E$93")*+$4'%K+D59=+"'44N#$&P+$%(+$JJ45&$95'%+E'%59'"5%K-++Q39+5%+$+O$I9NJ$&)(+)%75"'%E)%9*+)R9")E)4G+O$I9+()J4'GE)%9I+'O+%)D+O3%&95'%$459G+$")+$+P)G+&'EJ)95957)+$(7$%9$K)*+$%(+E595K$9)+9=)+"5IP+'O+#5KN#$%K+O3%&95'%$459G+&=$%K)I-+
!"#$%&'(;'%95%3'3I+,%9)K"$95'%+=$I+4'%K+#))%+$II'&5$9)(+D59=+I'E)+4)7)4+'O+$39'E$9)(+9)I95%K-+H=5I+5I+9"3)+#'9=+5%+9=)+I)E5%$4+$"95&4)+#G+S$"95%+T'D4)"+'"+5%+9=)+'4()"+J"$&95&)+()I&"5#)(+#G+89)7)+S&;'%%)44+'O+9=)+U$54G+Q354(+$%(+8E'P)+H)I9-+V59=5%+9=)+I&'J)+'O+2%9)"J"5I)+;'%95%3'3I+,%9)K"$95'%*+E3495J4)+9GJ)I+'O+$39'E$9)(+9)I9I+$%(+E$%3$4+9)I95%K+$")+&'%I5()")(-++
U)IJ59)+9=5I*+E$%G+9)$EI+$")+D)$P+$9+9)I95%K-+H=)G+J)"O'"E+$+#354(*+J39+59+9="'3K=+I'E)+#$I5&+)R)"&5I)I+E$%3$44G+$%(+9=)%+")4)$I)-+H=)+I$E)+J$"9I+'O+9=)+$JJ45&$95'%+#")$P+")J)$9)(4G+$%(+%)D+O3%&95'%$459G+5I+45K=94G+9)I9)(-+6I+9)$EI+E$93")+5%+9)I95%K+9=)G+()9)&9+J"'#4)EI+I''%)"*+5%&")$I5%K+#'9=+J"'(3&95759G+$%(+&'%O5()%&)-+
+
+
S'I9+9)$EI+=$7)+I'E)+O'"E+'O+$39'E$9)(+9)I95%K+5%+J4$&)-+@)"=$JI+$+IE$44+3%59+9)I9+I359)+'"+I'E)+I&"5J9)(+9)I9I+)%I3")+9=$9+9=)+#$I5&+O3%&95'%$459G+'O+9=)+$JJ45&$95'%+D'"PI-+H=)I)+#$I5&+$39'E$9)(+")K")II5'%+9)I9I+J"'75()+)$IG*+)$"4G+()9)&95'%+'O+O3%($E)%9$4+J"'#4)EI-+H)$EI+$9+9=)+!"&'()*+&(',+I9$K)I+'O+2%9)"J"5I)+;'%95%3'3I+,%9)K"$95'%+$")+K)%)"$44G+L3I9+#)&'E5%K+$&&45E$9)(+9'+$39'E$9)(+9)I95%K-+
H'+")$&=+9=)+-(./+%+4)7)4+'O+E$93"59G*+9=)")+I='34(+#)+$+I)9+'O+O$I9+9)I9I+9=$9+$")+"3%+D59=+)7)"G+#354(-+H=)I)+9)I9I+J"'75()+9=)+9)$E+&'%O5()%&)+9=$9+9=)+$JJ45&$95'%+D544+#$I5&$44G+D'"P+75"93$44G+$44+'O+9=)+95E)-+
Image © 2009 by Urbancode
Testing
Releasing
Problems
• One person is:
• best person for build
• best person for integration
(so, one can delay the other)
• Forced to branch when close to release
Releasing
Done
• Helper scripts
Releasing
Image © 2009 by Urbancode
Fully automate runtime image build
Automate installer creation
Automate installation into test environments
Changelog generation
!"#$%&'()*+,%&-+.+/011+23&45(+67)%3)+8359)+:00+.+;4)7)4$%(*+<=5'+11>>?+.+@A+/>:-B?B-C000++.+DDD-3"#$%&'()-&'E+.+F+/00C+#G+!"#$%&'()+
H"'75(5%I+45994)+$JJ3"$%&)+9=$9+$+J3&&)JJK34+()H4'GE)%9+5%+'%)+)%75"'%E)%9+5%(5&$9)J+9=$9+9=)+()H4'GE)%9+D544+#)+J3&&)JJK34+5%+9=)+%)L9+)%75"'%E)%9-+
+
+
6+9)$E+9=$9+=$J+E'7)(+$D$G+K"'E+$+K344G+E$%3$4+H"'&)JJ+9'+3J5%I+$+%3E#)"+'K+=)4H)"+J&"5H9J*+'"+$+K344G+J&"5H9)(+H"'&)JJ*+=$J+E$()+5EH'"9$%9+5EH"'7)E)%9J-+6&"'JJ+9=)+5%(3J9"G*+E'J9+9)$EJ+=$7)+J'E)+=)4H)"+J&"5H9J+#39+K$44+J='"9+'K+K344G+J&"5H9)(+()H4'GE)%9J+ +)JH)&5$44G+9'+&'%9"'44)(+)%75"'%E)%9J-+
!"#$%&$'()#$+9)$EJ+$")+I''(+$9+()H4'G5%I+K'"+9)J95%I+H3"H'J)J-+M=)G+D544+=$7)+H39+9=)5"+K344G+J&"5H9)(+()H4'GE)%9J+#)=5%(+9''45%I+9=$9+$44'DJ+K'"+H3J=N#399'%+()H4'GE)%9J+9'+J'E)+'"+$44+'K+9=)5"+9)J9+)%75"'%E)%9J-+M=5J+9$O)J+$+I")$9+()$4+'K+4'$(+'KK+'K+9=)+()H4'GE)%9+)%I5%))"J+D=54)+")(3&5%I+9=)+('D%95E)+'K+9)J9+9)$EJ+D$595%I+K'"+$+()H4'GE)%9-+P3J9+$J+&'%95%3'3J+#354(J+$")+$+&=$"$&9)"5J95&+'K+!"#$%&$'()#$+#354(+9)$EJ*+$39'E$95&+()H4'GE)%9+9'+9=)+K5"J9+9)J9+)%75"'%E)%9+5J+$+=$44E$"O+'K+!"#$%&$'()#$+E$93"59G+5%+()H4'GE)%9-++Q)H)%(5%I+'%+9)$E+(G%$E5&J*+9=5J+J='34(+=$HH)%+$9+9=)+)%(+'K+$%G+J3&&)JJK34+&'%95%3'3J+#354(+'"+$9+")I34$"+5%9)"7$4J+9="'3I=+9=)+($G+D=)%+9)J9)"J+$")+3%45O)4G+9'+#)+5%9)""3H9)(-+
M=)+4$J9+E$"O)"+'K+$%+!"#$%&$'()#$+9)$E+5J+9=$9+9=)G+=$7)+E$()+I''(+H"'I")JJ+5%+J9$%($"(5R5%I+9=)5"+()H4'GE)%9J+$&"'JJ+)%75"'%E)%9J-+M=)")+E$G+J9544+#)+J'E)+)%75"'%E)%9$4+7$"5$95'%*+'"+9D'+9GH)J+'K+
4$9)"+()H4'GE)%9J-+S)$&=5%I+9=5J+4)7)4+'K+E$93"59G+5J+$+I''(+9$"I)9+K'"+E$%G+9)$EJ-+
*'+)",$'+9)$EJ+93"%+9=)5"+$99)%95'%+9'+&'%9"'44)(+'"+H"'(3&95'%+)%75"'%E)%9J-+Q)H4'GE)%9J+9'+H"'(3&95'%+T")4)$J)JU+$")+H3J=N#399'%+$%(+J3&&)JJK34+H"'(3&95'%+")4)$J)J+$39'E$95&$44G+9"5II)"+E$9&=5%I+")4)$J)J+9'+(5J$J9)"+")&'7)"G-+6%G+9)$E+9=$9+()H4'GJ+9'+H"'(3&95'%+'%+5%9)"%$4+5%K"$J9"3&93")+J='34(+&'%J5()"+9$"I)95%I+9=)+*'+)",$'+4)7)4A+=$75%I+$+&'%J5J9)%9+()H4'GE)%9+H"'&)JJ+$&"'JJ+$44+)%75"'%E)%9J+("$E$95&$44G+")(3&)J+9=)+45O)45=''(+'K+4$J9NE5%39)+K$543")J+D=)%+E'75%I+9'+H"'(3&95'%-+*'+)",$'+9)$EJ+$")+$4J'+&=$"$&9)"5R)(+#G+K344G+$39'E$9)(+()H4'GE)%9J+9'+J'E)+9)J95%I+)%75"'%E)%9J+#$J)(+H$JJ5%I+V3$459G+I$9)J-+W'"+)L$EH4)*+$%+$HH"'7$4+K"'E+$+9)J9+E$%$I)"+J9$I)J+$+#354(+9'+#)+$39'E$95&$44G+5%J9$44)(+5%+9=)+J9")JJN9)J9+)%75"'%E)%9-+
Releasing
CINCOM and the Quadrant Logo are registered trademarks of Cincom Systems, Inc.All other trademarks belong to their respective companies.
© 2011 Cincom Systems, Inc.All rights reserved
Julian [email protected]
Twitter: @jfitzell