DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ......
Transcript of DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ......
DIABLO VS HAVANAHOW OPENSTACK HAS MATURED
Joe Gordon
ABOUT ME Developer @
Full time upstream OpenStack developernova-coreATC since EssexInterested in making OpenStack scale and workcontact information
jog0 on freenodegithub.com/jogolaunchpad.net/~jogo
OPENSTACK FROM DIABLO TO HAVANAA BRIEF HISTORY OF THE INTEGRATED RELEASE
DIABLOSwiftNova
Nova-volumeNova-networkIdentity
GlanceKeystone*
ESSEXSwiftNova
Nova-volumeNova-networkIdentity
GlanceKeystoneHorizon
FOLSOMSwiftNova
Nova-volumeNova-network
GlanceKeystoneHorizonQuantum (Neutron)Cinder
GRIZZLYSwiftNova
Nova-networkGlanceKeystoneHorizonQuantum (Neutron)Cinder
Total number of OpenStack services: 32
HAVANASwift (4)Nova (6)
Nova-networkGlance (2)Keystone (1)Horizon (1)Neutron (6)Cinder (4)Ceilometer (4)Heat (4)
HAVANA FEATURES SUPPORTED BY DIABLOSwiftNova
Nova-networkGlanceKeystoneHorizonNeutron (Quantum)CinderCeilometerHeat
HAVANA FEATURES SUPPORTED BY DIABLOSwiftNova
Nova-networkGlanceKeystoneCinder
OPENSTACK HAS GROWN
GROWN OUTNEW SERVICESHeatCeilometerMore in pipeline
IcehouseTrove
IncubationIronicMarconiSavanna
GROWN UPExpand existing APIs
Split out projectsExtensions
DIABLO NOVA EXTENSIONS| Hosts | Host administration| Volumes | Volumes support| Quotas | Quotas management support| SimpleTenantUsage | Simple tenant usage extension| FlavorExtraData | Provide additional data for flavors| SecurityGroups | Security group support| FlavorExtraSpecs | Instance type (flavor) extra specs| Createserverext | Extended support to the Create Server v1.1 API| Multinic | Multiple network support| VSAs | Virtual Storage Arrays support| VolumeTypes | Volume types support| Keypairs | Keypair Support| Rescue | Instance rescue mode| VirtualInterfaces | Virtual interface support| Floating_ips | Floating IPs support
POST DIABLO NOVA EXTENSIONS AdminActions Agents Aggregates AssistedVolumeSnapshots AttachInterfaces AvailabilityZone BareMetalExtStatus BareMetalNodes BlockDeviceMappingV2Boot CellCapacities Cells Certificates Cloudpipe CloudpipeUpdate ConfigDrive ConsoleOutput Consoles Coverage DeferredDelete DiskConfig Evacuate ExtendedAvailabilityZone ExtendedFloatingIps ExtendedIps ExtendedIpsMac ExtendedQuotas ExtendedServerAttributes ExtendedServices ExtendedStatus ExtendedVIFNet ExtendedVolumes FixedIPs FlavorAccess FlavorDisabled FlavorManage FlavorRxtx FlavorSwap FloatingIpDns FloatingIpPools FloatingIps FloatingIpsBulk Fping HideServerAddresses Hypervisors ImageSize InstanceActions Migrations MultipleCreate NetworkAssociationSupport Networks OSInstanceUsageAuditLog OSTenantNetworks QuotaClasses SchedulerHints SecurityGroupDefaultRules ServerDiagnostics ServerPassword ServerStartStop ServerUsage Services Shelve UsedLimits UsedLimitsForAdmin UserData UserQuotas VolumeAttachmentUpdate
SO ONCE WE NORMALIZEFOR THE GROWTH IN
OPENSTACK WHAT ELSEHAVE WE DONE IN THEPAST FOUR RELEASES?
CHANGESDriversTesting and bugsPerformance and scalabilityForward facing internal changesProcess
MORE DRIVERSNova: baremetal, docker, hyperv, powervm
Cinder: coraid, emc, glusterfs, gpfs, nexenta, rbd, sheepdog,soldfire, ...
BETTER TESTING, FEWER BUGSUnit testsIntegration tests (Tempest)Lots of fixed bugs
UNIT TESTSPERCENT UNIT TESTS BY LOC
Project Diablo HavanaNova 43% 56%Glance 62% 62%Swift 65% 71%Keystone 41% 45%
TEMPESTTHE OPENSTACK INTEGRATION TEST SUITE
DIABLO TEMPESTOrignally named 'openstack-integration-tests'Just 107 tests*
extensionsflavorsfloating ipsimage metadataimageskeypairsserversvolumes
2,655 lines of python codeFirst release of Tempest
HAVANA TEMPEST1275* tests
1150 api testscompute, image, identity, network, object_storage,orchestration, volume102 test files
92 cli tests11 scenario tests18 thirdparty (boto) tests4 other
Parallel20-30 minutes to run in gate24,837 lines of python code
HAVANA API TESTS785 compute216 identity24 image17 network50 object_storage54 volume
HAVANA TEMPEST VS OPENSTACK DIABLOSame APIs as Diablo plus new onesDiablo has been 'end of life'edMany tests cover new features. Ignore thoseTempest not designed to be version independentGot 622 tests to run and 154 (27%) to pass.*
110 of which were in compute
HAVANA TEMPEST VS OPENSTACK DIABLORESULTS
Diablo has poor parameter validationInvalid key namesMin/max count for multiple creates
Changed return codes for negative casesReturn empty list and 200 instead of 400 when invalid filtervalue in nova list serversWe shouldn't have done this
Poor pagination in DiabloJSON only tests in Diablo, Havana tests XML tooMany false negatives in results due to slight changes in nowAPIs and metadata are used
CRITICAL BUGSProject Diablo Essex Folsom Grizzly HavanaNova 14 22 30 31 33Glance 5 15 30 13 8Keystone 31 11 9 8Cinder 10 3 12Neutron 27 14 24
CategoriesNon-gated featuresDependenciesRace conditionsPerformance
PERFORMANCE ANDSCALABILITY
PERFORMANCE AND SCALABILITYNova-scheduler
Active-activeNo RPC fanout
DBIndicesSmarter queriesFewer queries
service groupHeartbeat to DB every 10 seconds1k DB calls a second if have 10K nodes.Adjustable intervalAlternatives memcache and zookeeper
PERFORMANCE AND SCALABILITYMore efficient image work flow
Glance, Nova, CinderMove large files around less
PKI TokensFewer round trips to keystone
PERFORMANCE AND SCALABILITYCASE STUDY FROM HP PUBLIC CLOUD
Diablo40,836 LOC changed to 239 files in nova2,265 LOC to manage DB+100s of hours of DB tuning to make it work with morethen just a few nodes
HavanaDB just worked1,167 LOC changed in 35 filesMost of that is in patches in review today
Thanks to Thom Leggett for these statistics
FORWARD FACING INTERNAL CHANGESSplit out projectsOsloRoad to live and continuous upgradesIterate on APIs
PROCESS
DRIVING PRINCIPLESNever break trunk
Master branch is always greenDevelopers are never blocked
TransparencyAutomate everythingEgalitarianBe strict. Reduce burden on reviewers
DIABLOAutomated Gate87 Contributors in September 2011 (release of Diablo)No GateNo Git or GerritBzr and Launchpad
HAVANA346 Contributors in October 2013 (release of Havana)
Scale development processIntegrated Gate
9 Projects30+ separate services1 Merge pipeline
New tools for an asynchronous worldTests not pass fail, rather classified with a race percentageFix that race condition we hit 0.5% of the timeDebugging is hardelastic-recheck
HOW WILL OPENSTACK KEEP MATURINGWHAT CHALLENGES ARE WE FACING TODAY?
More testingLive rolling upgradesLimiting factor to the whole system is reviewers (and not justcore)Improving quality while growing the project as a wholeScalability