1

Reviewing CPAN modules

Neil Bowers

2

• Apparently there are 1,900+ Torch/Flashlight apps…

3

4

What is CPAN?

• Something like a university library?• Lots of useful items, nicely catalogued, easily found

A toolchest full of precision instruments that good programmers can wield to make great things

- chromatic

Holy crap, I forgot about [that module that I wrote].- Andy Lester

• I think more like Rube Goldberg's workshop:• A bunch of useful things, some half-finished ideas, practical jokes, stuff

that no longer works, items that shouldn't be there, forgotten things in the corner

5

Review #1Generating passwordsModules that can be used to automatically generate passwords, either random sequences of characters or pronounceable pseudo words

6

Modules for generating passwords

7

Generating random char strings

• 100,000 passwords

8

Structure of reviews

9

After doing the review

• I'm now using Crypt::YAPassGen• v0.02 released 2004-05-30

• App::Genpass is good• I've submitted changes to add features I liked in other modules

• I've ended up using Data::Random elsewhere• Because I was aware of it having done this review

• Hoping to finally release fix for Crypt::RandPasswd• I was granted co-maint yesterday (23rd Nov 2012)

10

Review #2Looking up location of an IP addressModules that can map an IP address to ISO country code,and possibly return additional info like language, county/state, etc

11

Modules for getting IP locationco-maint to release fixes and doc updates

12

Notes

• A number of these are 'commercial'• Pay for full data set, or to get unthrottled access to backend service

• Performance & coverage tests took the most time• In particular it's hard to test for correctness

• A number of good options• Competition keeps them on their toes?• Which one you use might depend on what other info you want for an IP

• Ended up not having the need, so not using any• But have used them occasionally, now I know about them

13

Review #3Spelling out a number in EnglishModules for converting 327 to "three hundred and twenty-seven",-3.25 to "minus three point two five", etc

14

Modules for spelling out numbers

15

Notes on review #3

• Fewer modules meant more time to 'play'• I had fun writing a script to automatically compare results between

modules for 1 .. N

• First time I hit a module that just doesn't work

• Lingua::EN::Numbers• The first module where I remember looking at the code and thinking

"hey, nice"• I took over maintenance from Sean Burke to release fixes• And have a todo-list, adding ideas that came up during the review

16

My rules of engagement

• Treat authors and their code with respect• This might be someone's first tentative steps at code sharing

• Report all bugs I find• If I can see how to fix them, then submit a fix

• Fix and improve documentation• Especially if I had to look at the code to work out how to use it

• If appropriate, try and get co-maint to release fixes• Because CPAN is forever• This can be to just mark a module as deprecated & update SEE ALSO

• If there are four or more modules for something• It goes onto my backlog of potential reviews

17

Observations based on the rules

• What encourages me to submit fixes?• Github• Makefile.PL (or more accurately: not dist.ini)

• Getting co-maint is relatively costly• People disagreed when I said this at LPW 2011 ("You're doing it

wrong")• I still think this (getting co-maint to fix doc is relatively costly)

• I'm toying with writing a tool to manage this• I might submit bugs or send email on 10 different modules in a review

18

Review #4Parsing User-Agent stringsModules for extracting and inferring information from HTTP User-Agent strings.

Eg to tell how many of your site's users are using IE vs Chrome vs Firefox

19

Modules for parsing User-Agent

20

Review structure: module section

21

ComparisonCoverage (test corpus)

Speed

Review-driven development!

22

Curation

• Requirements for these modules• Continuing good coverage of (versions of) browsers, bots, libraries, etc• Good enough performance (processing large log files)• Identifying other things (is it a mobile? what's the screen size?)

• Working with Olaf Alders on "complete solution"• Currently at the R&D stage

23

Review #5Defining constantsModules for defining constants and immutable variables

24

Modules for defining constants

25

Some firsts with this review

• I was prompted to look at the Perl source• To make sure I understood SvREADONLY

• Emailed an author for help understanding his code

• Enough modules to prompt me to map them out

26

Observations

• I had been using Readonly• But I clearly hadn't read all the documentation before

• Now I use Const::Fast• Latency: all those modules using Readonly …

• Surprising variety of designs / approaches• I learned a lot looking at the code for all these modules

• But quite a few experiments• "I wonder if you can …"

27

Review #6Getting module dependency infoFinding out what modules (and dists) are used by a module or modules

28

Tried a new approach

• Previously I only published after a lot of work• But I still missed plenty of modules

• Decided to experiment with a more lean approach• Found 6 modules, none did what I wanted, so wrote my own• Wrote a quick sketch review and published it• More modules came out of the woodwork every time I updated

• I was looking at dependencies for my next review• But ended up pushing this one out sooner

29

Modules for getting dependencies

30

Module Author

• But META.json says:

• I had a "bug" reported – author is listed as:

• End result was PAUSE::Permissions

31

Notes

• The 23rd module nearly matched mine!

• Very much still in progress

• Curation needed:• There are 5 or so distinct module types (out of 26 so far)• But lots of different variations on each theme

32

Why so many modules?

• Whipupitude

• Low barrier to entry for CPAN

• Hard to find (all) appropriate modules

• Easier to upload than contribute or take over

• Playing with Perl is fun (eg constants, dependencies)

33

Review #7Making HTTP requestsModules for making HTTP requests (GET, POST, PUT, DELETE, etc)Excluding higher-level modules like WWW::Mechanize

34

Modules for making HTTP requests

The only module I've given up on

35

Performance on GET requests

36

Runtime dependencies

Require a C library

Requires a C library and Moose!

37

What module to use?

38

Review #8LUHN checkModules for checking credit card numbers

39

Modules for checking CC numbers

40

Performance of LUHN modules

Moose?

41

Review #9Getting a module's pathModules for finding where a module has been installed locally,preferably without having to load the module

42

Modules for getting a module's path

43

Comparison

Nearly three orders of magnitude

44

Feature matrix

45

Features vs Dependencies

46

Appropriate # of dependencies

47

Observations

And some mild venting

48

What do people use CPAN for?

• Sharing code with others!

• Configuration management

• A cloud-based store for configuration (Task::BeLike::*)

• A transport mechanism

• A todo list (eg Business::BankCard)

• A blog / dumping ground for ideas

• None of the above (ACME::*)

49

Version numbers

• Version numbers should be designed for humans• PAUSE just wants them to be monotonically increasing

• All modules in a dist should have• A version number• The same version number

• Don't be afraid of 1.x• 73% of dists on CPAN are version 0.*

• our $VERSION = '0.01';

0.010.004

1.1216600.0.200.6.1

50.000

0.001005v0.0.3

2.46632

2011121001

51

How can you tell a good module?• Version number

• But see Net::HTTP::Tiny v0.01

• Recently updated• Some modules are 'finished'

• Volume and/or quality of documentation• See Crypt::RandPasswd

• Number of bugs• Some big hitters have a long list though…

• Number of dependent modules• from different authors

• Ratings / +1's

• Perl::Critic

• Author

• All of the above, and more

52

How have I changed as a result?• I've learned new things, & new ways to do things, & …

• I do developer releases & pay a lot more attention to CPAN Testers

• I'm more paranoid

• I benchmark and profile

• I use IRC, and have had only positive experiences

• How I export functions (thank-you Zefram)

• How I think about dependencies

• Moose / Mouse / Moo

• I'm a lot more circumspect about releasing code to CPAN

• Beginner's mind

54

My review process

• Quickly skim and find initial set of modules

• Start writing sections for each module• Write generic SYNOPSIS style example, and evolve it as I go along

• Report bugs and documentation issues as I go

• First sketch at benchmarking & comparing

• More thorough search for modules to include

• Email the authors, give them a chance to comment

• Publish first version• Iterate as people point out modules that I've missed

55

It's easy – why not give it a go?% mkdir json% cd json

% cat > module-list.txtJSONJSON::DWIWJSON::JOMJSON::ParseJSON::SyckJSON::TinyJSON::UtilJSON::XSJSON::YAJLJSON::Streaming::ReaderJSON::Streaming::WriterMojo::JSONPegex::JSONTest::JSON^D

% cpan-review

56

What do I want you to do?

• Stop and search CPAN before writing code• Ok, I accept it's hard to find (all of the) relevant modules for a task

• Just because you wrote it, you don't have to upload it• Only release if it's needed for another module, or another user• Put it on github• Write an article about it on blogs.perl.org

• Add links to similar modules in SEE ALSO• MetaCPAN will include those in search results• It's helpful for users• You'll make my life easier

• Write a helpful ABSTRACT in your NAME section• "a perl interface to foolib"• "the hardest working two letters in Perl"

57

Review-related todo list

• Related to existing reviews• Fix Crypt::RandPasswd• Configurability for Lingua::EN::Numbers• User-Agent module with Olaf• New constant module: Exporter::Constants• Dependencies: +5 modules to review, my module, curation• HTTP requests: Mojo::UserAgent, async request modules• Changes to Business::CCCheck

• New reviews in progress• Hash objects• JSON• Module loading

58

Finding modules is hard

• Thinking about trying automatic clustering

• Help identify groups of modules for review

• Possible contribution to MetaCPAN

• Automatic tagging of modules• Use keywords field in metadata