Managing Python Packages with Pulp - FOSDEM 2020 · Create New Repositories Create a new repository...
Transcript of Managing Python Packages with Pulp - FOSDEM 2020 · Create New Repositories Create a new repository...
![Page 1: Managing Python Packages with Pulp - FOSDEM 2020 · Create New Repositories Create a new repository in Pulp Sync content from a remote repo, manually or on a schedule Upload your](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7e414b522d310243ab3f0/html5/thumbnails/1.jpg)
Managing Python Packageswith Pulp
Presented by:
Michael HrivnakPrincipal Software Engineer
@michael_hrivnak
http://www.pulpproject.org
![Page 2: Managing Python Packages with Pulp - FOSDEM 2020 · Create New Repositories Create a new repository in Pulp Sync content from a remote repo, manually or on a schedule Upload your](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7e414b522d310243ab3f0/html5/thumbnails/2.jpg)
Distributing software is a messy business.
![Page 3: Managing Python Packages with Pulp - FOSDEM 2020 · Create New Repositories Create a new repository in Pulp Sync content from a remote repo, manually or on a schedule Upload your](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7e414b522d310243ab3f0/html5/thumbnails/3.jpg)
What is Pulp?
Manage repositories of content
Supports many content types
Pull-through cache
OSS under GPL2
https://github.com/pulp
Python web app
![Page 4: Managing Python Packages with Pulp - FOSDEM 2020 · Create New Repositories Create a new repository in Pulp Sync content from a remote repo, manually or on a schedule Upload your](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7e414b522d310243ab3f0/html5/thumbnails/4.jpg)
Create New Repositories
Create a new repository in Pulp
Sync content from a remote repo, manually or on a schedule
Upload your own content
Copies are cheap
![Page 5: Managing Python Packages with Pulp - FOSDEM 2020 · Create New Repositories Create a new repository in Pulp Sync content from a remote repo, manually or on a schedule Upload your](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7e414b522d310243ab3f0/html5/thumbnails/5.jpg)
Publish Your Repository
Mix and match content, then publish
Publish can mean many things
![Page 6: Managing Python Packages with Pulp - FOSDEM 2020 · Create New Repositories Create a new repository in Pulp Sync content from a remote repo, manually or on a schedule Upload your](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7e414b522d310243ab3f0/html5/thumbnails/6.jpg)
$ pulp-admin python repo create --repo-id pypi --feed https://pypi.python.org/ --package-names numpy,scipy
Repository [pypi] successfully created
![Page 7: Managing Python Packages with Pulp - FOSDEM 2020 · Create New Repositories Create a new repository in Pulp Sync content from a remote repo, manually or on a schedule Upload your](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7e414b522d310243ab3f0/html5/thumbnails/7.jpg)
$ pulp-admin python repo sync run --repo-id pypi+----------------------------------------------------------------+ Synchronizing Repository [pypi]+----------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Downloading and processing metadata.[-]... completed
Downloading and processing Python packages.[==================================================] 100%30 of 30 items... completed
Task Succeeded
![Page 8: Managing Python Packages with Pulp - FOSDEM 2020 · Create New Repositories Create a new repository in Pulp Sync content from a remote repo, manually or on a schedule Upload your](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7e414b522d310243ab3f0/html5/thumbnails/8.jpg)
$ pip install -i http://pulp.example.com/pulp/python/web/my_own_pypi/simple/ pulp-python-pluginsDownloading/unpacking pulp-python-plugins Downloading pulp-python-plugins-0.0.0.tar.gz Running setup.py egg_info for package pulp-python-plugins
Installing collected packages: pulp-python-plugins Running setup.py install for pulp-python-plugins
Successfully installed pulp-python-pluginsCleaning up...
![Page 9: Managing Python Packages with Pulp - FOSDEM 2020 · Create New Repositories Create a new repository in Pulp Sync content from a remote repo, manually or on a schedule Upload your](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7e414b522d310243ab3f0/html5/thumbnails/9.jpg)
Content Types
RPM family
Docker Images
Puppet Modules
Python Packages
OSTree
Regular Files
Debian Packages (community)
NPM?
![Page 10: Managing Python Packages with Pulp - FOSDEM 2020 · Create New Repositories Create a new repository in Pulp Sync content from a remote repo, manually or on a schedule Upload your](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7e414b522d310243ab3f0/html5/thumbnails/10.jpg)
Who uses Pulp?
Red Hat Release Engineering
Public Clouds
Katello / Red Hat Satellite 6
Community
![Page 11: Managing Python Packages with Pulp - FOSDEM 2020 · Create New Repositories Create a new repository in Pulp Sync content from a remote repo, manually or on a schedule Upload your](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7e414b522d310243ab3f0/html5/thumbnails/11.jpg)
Use Case: Mirror Python Packages
Sync packages from PyPI to a local repository
Add or remove packages as desired
Retain old versions
Control which versions appear together
![Page 12: Managing Python Packages with Pulp - FOSDEM 2020 · Create New Repositories Create a new repository in Pulp Sync content from a remote repo, manually or on a schedule Upload your](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7e414b522d310243ab3f0/html5/thumbnails/12.jpg)
Use Case: Dev / Test / Production
Sync content into a development repository
“Promote” by copying to a Test repository, and then to a Production repository
Useful for testing upstream content, such as new RHEL point releases
Dev ProdTest
![Page 13: Managing Python Packages with Pulp - FOSDEM 2020 · Create New Repositories Create a new repository in Pulp Sync content from a remote repo, manually or on a schedule Upload your](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7e414b522d310243ab3f0/html5/thumbnails/13.jpg)
Distributed Application
REST API
Content served via HTTP
Worker Processes for Async Jobs
![Page 14: Managing Python Packages with Pulp - FOSDEM 2020 · Create New Repositories Create a new repository in Pulp Sync content from a remote repo, manually or on a schedule Upload your](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7e414b522d310243ab3f0/html5/thumbnails/14.jpg)
HTTPD
Workers
DB
AMQP
Publish!
![Page 15: Managing Python Packages with Pulp - FOSDEM 2020 · Create New Repositories Create a new repository in Pulp Sync content from a remote repo, manually or on a schedule Upload your](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7e414b522d310243ab3f0/html5/thumbnails/15.jpg)
Extensible
Standard ways to support new content types
How does content flow in?
How does content flow out?
![Page 16: Managing Python Packages with Pulp - FOSDEM 2020 · Create New Repositories Create a new repository in Pulp Sync content from a remote repo, manually or on a schedule Upload your](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7e414b522d310243ab3f0/html5/thumbnails/16.jpg)
Plugins
Type Definition
Model Class (using mongoengine)
Importer
Used to pull in upstream content
One per repository
Distributor
Used to publish content
Many per repository
core
importer
distributor
![Page 17: Managing Python Packages with Pulp - FOSDEM 2020 · Create New Repositories Create a new repository in Pulp Sync content from a remote repo, manually or on a schedule Upload your](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7e414b522d310243ab3f0/html5/thumbnails/17.jpg)
Integration
REST API
Events published to AMQP topic exchange
HTTP callbacks
![Page 18: Managing Python Packages with Pulp - FOSDEM 2020 · Create New Repositories Create a new repository in Pulp Sync content from a remote repo, manually or on a schedule Upload your](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7e414b522d310243ab3f0/html5/thumbnails/18.jpg)
Pull-through Cache
Available in 2.8.0 Beta
Adds metadata to database without downloading files
Retrieves files on-demand or in the background
yum pulp squid streamer remoterepo
filesystem
![Page 19: Managing Python Packages with Pulp - FOSDEM 2020 · Create New Repositories Create a new repository in Pulp Sync content from a remote repo, manually or on a schedule Upload your](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7e414b522d310243ab3f0/html5/thumbnails/19.jpg)
Documentation
http://www.pulpproject.org/docs/
![Page 20: Managing Python Packages with Pulp - FOSDEM 2020 · Create New Repositories Create a new repository in Pulp Sync content from a remote repo, manually or on a schedule Upload your](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7e414b522d310243ab3f0/html5/thumbnails/20.jpg)
Stickers!