Syncing with-upstream

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Syncing with UpstreamDarragh Bailey/Tokyo/Oct, 2015

© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.2

Bio – who am I?•Work for HP since 2005•Mostly focused on source control/CI/release over the 10

years• Involved in HP Public cloud and recently Helion 2.0 (just

released)

•OpenStack projects mostly involved with:– jenkins-job-builder, python-jenkins, git-upstream

• IRC– electrofelix (was pelix) in #openstack-infra on freenode

•GitHub– electrofelix (recent work around vagrant plugins)


What does ‘Syncing with upstream’ mean?•Direct consumer & contributor to OpenStack This talk

– Managing local backports/fixes where/when required– Encouraging upstreaming of fixes and automatic discarding

of backports– Workflow used by development of HP Helion to sync with

upstream

•Consuming distribution packages?– Still mention a few interesting tools that might be useful


Why?Why do we need to align local patches with upstream?And why do we need git-upstream to handle this?


Why would anyone want a tool to align patches?• Syncing daily• Manually it could take 5-30

minutes (ave 15)

• For 25+ projects and growing?

• Worth spending ~90 engineer days per year to have this take <1 minute on average

• Currently just works >90%

https://xkcd.com/1205/


Ideally “Contribute upstream and consume”, sounds great but is that possible?

Why are we carrying local patches?

Assumptions:• All bugs can be debugged and

fixed within a virtual environment• Acceptance upstream is always

quick• There is no risk of new bugs being

added• Issues never impact a production

environment or stop teams from working– The only customers are end users

Get latest

upstream

Find issues(stress test)

Develop Fix

Submit upstream

Release


Probably more likely…

Get latest upstream

Develop Fix

Submit upstream

Release*

Apply/Drop local fixes Apply

locally


None found (ideal)

Bugs


Types of changes

• Packaging files (including pinning against older python packages)

• Compatibility hacks for older libraries/applications for migration/upgrades

• Backports where can’t roll forward, but need a recent change

• Dumb fixes breaks for other environments or configurations but allows progress for local configurations until proper solution

• Scale testing proving a fix really works; can’t always test upstream at sufficient scale

Anything that needs to be used in an environment besides your personal devstack area.


Where git-upstream fits

Get latest upstream

Develop Fix

Submit upstream

Release

Apply/Drop local fixes Apply

locally

Try to automate apply/drop of

fixes in an submit upstream

compatible manner



Is there a benefit to upstream if we carry?•Verify change actually solves the problem•Run debug patches on a live system

– Provide data for discussion on fix upstream•Allow time for review and improvements

– Avoid unnecessary pressure for changes to be accepted


So we wrote a tool to support a work-flow

•To clone: https://github.com/openstack/git-upstream•PyPI: https://pypi.python.org/pypi/git-upstream• IRC: #git-upstream

•Basic description:– git-upstream is an open source Python application that can

be used to keep in sync with upstream open source projects, mainly OpenStack.

https://github.com/openstack/git-upstream

https://pypi.python.org/pypi/git-upstream


How?How do we go about staying in sync?


Two phases to the work-flow

Syncing

• Fetch upstream changes• Push to namespace in

local repo

• master upstream/master

JJB example: contrib/jjb/mirror.yaml

Importing/Re-aligning

• Clone local project repo• Run ‘git upstream

import’….• Profit

Tends to be a little more…


The basics

Git-upstream expects to find upstream refs under ‘upstream/*’, i.e. upstream/stable/kilo, upstream/master

To sync all patches on your master branch against upstream/master:• git upstream importMore complete version• git upstream import --into master upstream/master

Followed by upload to internal gerrit


Quiet

By default git-upstream is quiet:


Verbose

But you can increase the verbosity (spam debug messages to console)


Separate logging

Or request a separate log file for debug:


git-upstream commands available


How does this work?Under the hood.


Two main approaches

Merge requests• Proven for Linux (kernel)• Well supported by

GitHub/GitLab

• Requires main maintainer in house for each project repo

• Conflicts likely unavoidable unless patch accepted upstream identical

• Doesn’t map well to OpenStack review workflow???

Patches• Proven for OpenStack• Well supported by Gerrit

• Possibly needs patch owner to help resolve conflicts

• Duplicate changes will cause conflicts

• Adding a set of patches to a repo as separate files will not be fun to review


Tools in this area already

•Patch queue• guilt (quilt in git)

• Branches• stgit• topgit

•Packaging• git-dpm

None appeared to play well with Gerrit workflow

https://xkcd.com/927/


So to the requirements we worked out…

•Wanted all developers to use same workflow as upstream (local Gerrit)

•Avoid reviewing commits containing patches (diff of a diff)•Automated discard of patches landed upstream

•A workflow that encourages us to upstream any local changes


Git is already great at managing patches as commitsWhat did we want?

previous import

next desired import

So can we get from old to new state without causing issues?

current

desired

landed upstream so drop next time


But how do we avoid this?


The secret sauce git merge -s ours --no-commit <desired>git read-tree -u --reset <desired>git commit --no-edit

• We use ‘ours’ to merge from desired to current and then replace the contents to make things easier for devs to follow, not because it’s necessary. Could use ‘ours’ and merge from current into desired.

landed upstream so drop next time

previous import

next desired import

current

desired

Outstanding fixes can be merged by Gerrit or cleanly rebased if conflicting with upstream


Usage in HP Helion and status


Example drift

Currently worked out by a custom script, like to have git-upstream report the raw information.


Current state of affairsGit-upstream is still in a alpha/beta phase

Lack of reporting• Difficulty in comparing between upstream and local• Difficulty in understanding what changes are being carriedManual interaction is poor• Conflicts require manual finishing after resolving• No resume + checking of locally carried changesLack of syncing• Notes used for supersede must be explicitly synced using git

commands• No ‘git-upstream status’ to let user know


Git-upstream roadmap

List of main changes to be targeted• https://etherpad.openstack.org/p/git-upstream

Project available on stackforge• https://github.com/openstack/git-upstream• https://review.openstack.org/#/q/status:open+project:openstack/git

-upstream,n,z

Any help greatly appreciated!

https://etherpad.openstack.org/p/git-upstream

https://etherpad.openstack.org/p/git-upstream

https://github.com/openstack/git-upstream

https://review.openstack.org/#/q/status:open+project:openstack/git-upstream,n,z

https://review.openstack.org/#/q/status:open+project:openstack/git-upstream,n,z


FinTo clone: https://github.com/openstack/git-upstreamPyPI: https://pypi.python.org/pypi/git-upstreamIRC: #git-upstream (look for electrofelix/pelix)

Syncing with-upstream

Engineering

Transcript of Syncing with-upstream