Import this, that, and the other thing: custom importers
-
Upload
zoom-quiet -
Category
Technology
-
view
1.380 -
download
0
description
Transcript of Import this, that, and the other thing: custom importers
![Page 1: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/1.jpg)
If you do not know what __path__ is,
this talk is NOT for you.
Sorry.
![Page 2: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/2.jpg)
import this, that, and the other thingCustom importers in Python
Brett Cannonwww.DrBrett.ca
Slides are sparse, so do listen to what I say.
![Page 3: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/3.jpg)
Thanks ...
• Python Software Foundation• PyCon Financial Aid committee
• Nasuni• Jesse Noller
![Page 4: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/4.jpg)
What the heck is an importer?
Relevant since Python 2.3
![Page 5: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/5.jpg)
importer=
finder + loader
![Page 6: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/6.jpg)
A finder finds modules.
![Page 7: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/7.jpg)
A loader loads modules.
![Page 8: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/8.jpg)
“Why do I want one?”
Customization/control, easier to work w/ than __import__
![Page 9: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/9.jpg)
How are custom importers used by
import?
Simplified view; ignoring implicit importers
![Page 10: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/10.jpg)
Meta pathsys.meta_path
![Page 11: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/11.jpg)
Start
for finder in sys.meta_path:
loader = finder.find_module(name, path)
return loader.load_module(name)
...
True
False
What ‘path’ arg is
![Page 12: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/12.jpg)
Pathsys.path or __path__, sys.path_hooks,
& sys.path_importer_cache
![Page 13: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/13.jpg)
... Parent module
has __path__
search = parent's __path__
search = sys.path
search
True
False
![Page 14: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/14.jpg)
for entry in search:
finder = sys.path_importer_cache[entry]
loader = finder.find_module(name)
return loader.load_module(name)
Search
path hookFalse
True
False finder
raise ImportError
True
![Page 15: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/15.jpg)
for hook in sys.path_hooks:
finder = hook(entry)
sys.path_importer_cache[entry] = finder
sys.path_importer_cache[entry] = dummy
path hook
finder
False
True
True/False = ImportError (not) raised
![Page 16: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/16.jpg)
how do I write my own
importer?Only masochists need apply.
![Page 17: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/17.jpg)
Option 1:Painfully from
scratchRead PEP 302 for the gory details.
![Page 18: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/18.jpg)
Option 2:Use importlib
Available since Python 3.1.I have suffered so you don’t have to.
![Page 19: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/19.jpg)
Option 3: importers
http://packages.python.org/importers/
File path abstraction on top of importlib.Treating as purgatory for importlib
inclusion.
If a lesson here, then it is to use option 2 or 3 depending on your needs.Rest of talk is about lessons that led to ‘importers’.
![Page 20: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/20.jpg)
Using azipfile importeras an example
Assuming use of importlib.Talking from perspective of using an archive.
![Page 21: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/21.jpg)
we need a hookFor sys.path_hooks.
![Page 22: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/22.jpg)
Refresher:Hooks look for a finder for a path
Path either from sys.path or __path__
![Page 23: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/23.jpg)
Hooks can get funky paths
E.g. /path/to/file/code.zip/some/pkg
Search backwards looking for a file; find a directory then you have gone too far.
![Page 24: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/24.jpg)
Consider caching archive file objects
No need to keep 3 connection objects open for the same sqlite3 file
![Page 25: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/25.jpg)
Pass your finder the “location”:
1)the path/object &2) the package path
Import assumes you are looking in a part of a package.
![Page 26: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/26.jpg)
Raise ImportError if you got nuthin’
![Page 27: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/27.jpg)
Have finder, will look for code
![Page 28: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/28.jpg)
Don’t treat modules as code but as files
Just trust me. Too many people/code make this assumption already for stuff like __file__, __path__, etc.
![Page 29: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/29.jpg)
You did remember where in the
package you are looking, RIGHT?!?
Needed because of __path__ manipulation by user code.
![Page 30: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/30.jpg)
fullname.rpartition(‘.’)[-1]
![Page 31: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/31.jpg)
Need to care about packages &
modulessome/pkg/name/__init__.py
andsome/pkg/name.py
Care about bytecode if you want.Notice how many stat calls this takes?
![Page 32: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/32.jpg)
Avoid caching within a finder
Blame sys.path_importer_cache
![Page 33: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/33.jpg)
Tell the loader if package &path to code
Don’t Repeat Yourself ... within reason.
![Page 34: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/34.jpg)
Nuthin’?Give back None
![Page 35: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/35.jpg)
Now it gets tricky
Writing a loader.
![Page 36: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/36.jpg)
Are you still thinking in terms of
file paths?
![Page 37: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/37.jpg)
importlib.abc.PyLoader
• source_path()• Might be changing...
• is_package()• get_data()
Everything in terms of exactly what it takes to import source
![Page 38: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/38.jpg)
importlib.abc.PyPycLoader
• source_path()• is_package()• get_data()• source_mtime()• bytecode_path()
• Might be changing...
This is what is needed to get source w/ bytecode right
![Page 39: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/39.jpg)
Reasons to ignore .pyc
• Jython, IronPython couldn’t care less.• Safe to support, though.
• Another thing to code up.• Bytecode is just an optimization.• If you only ship .pyc for code
protection, stop it.
![Page 40: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/40.jpg)
What to do when using
importlib ABCs
![Page 41: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/41.jpg)
Require anchor point for paths
somewhere/mod.py is too ambiguous
Too hazy as to where a relative path is anchored; archive? Package location?
![Page 42: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/42.jpg)
Consider cachingstat calls
Only for stand-alone loaders!Also consider caching if package or not.
Consider whether storage is read-only, append-only, or read-write.
![Page 43: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/43.jpg)
Don’t overdo error checking
EAFP is your friend.
![Page 44: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/44.jpg)
Perk of importers is the abstraction
![Page 45: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/45.jpg)
Lazy loader mix-inwritten in19 lines
![Page 46: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/46.jpg)
class Module(types.ModuleType): pass
class Mixin: def load_module(self, name): if name in sys.modules: return super().load_module(name) # Create a lazy module that will type check. module = LazyModule(name) # Set the loader on the module as ModuleType will not. module.__loader__ = self # Insert the module into sys.modules. sys.modules[name] = module return module
class LazyModule(types.ModuleType): def __getattribute__(self, attr): # Remove this __getattribute__ method by re-assigning. self.__class__ = Module # Fetch the real loader. self.__loader__ = super(Mixin, self.__loader__) # Actually load the module. self.__loader__.load_module(self.__name__) # Return the requested attribute. return getattr(self, attr)
![Page 47: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/47.jpg)
... or you could use the importers package
http://packages.python.org/importers/
![Page 48: Import this, that, and the other thing: custom importers](https://reader035.fdocuments.net/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/48.jpg)
Fin