Open Legislation Spring 2011 Talk 1
-
Upload
graylinkim -
Category
Technology
-
view
215 -
download
3
Transcript of Open Legislation Spring 2011 Talk 1
![Page 1: Open Legislation Spring 2011 Talk 1](https://reader031.fdocuments.net/reader031/viewer/2022020306/554dc525b4c905bd488b5038/html5/thumbnails/1.jpg)
Open LegislationSpring 2011
![Page 2: Open Legislation Spring 2011 Talk 1](https://reader031.fdocuments.net/reader031/viewer/2022020306/554dc525b4c905bd488b5038/html5/thumbnails/2.jpg)
Open Data(Government)
![Page 3: Open Legislation Spring 2011 Talk 1](https://reader031.fdocuments.net/reader031/viewer/2022020306/554dc525b4c905bd488b5038/html5/thumbnails/3.jpg)
Secondary Sources are nice
● OpenCongress● GovTrack.US● OpenStates● FedSpending.org
● Many more
![Page 4: Open Legislation Spring 2011 Talk 1](https://reader031.fdocuments.net/reader031/viewer/2022020306/554dc525b4c905bd488b5038/html5/thumbnails/4.jpg)
Primary Sources are better
● Data.gov● USAspending.gov● California● Oregon● Washington
● Many more
![Page 5: Open Legislation Spring 2011 Talk 1](https://reader031.fdocuments.net/reader031/viewer/2022020306/554dc525b4c905bd488b5038/html5/thumbnails/5.jpg)
Sometimes though...Open Data is not Enough.
We need Platforms.
![Page 6: Open Legislation Spring 2011 Talk 1](https://reader031.fdocuments.net/reader031/viewer/2022020306/554dc525b4c905bd488b5038/html5/thumbnails/6.jpg)
A Different Breed of Open
● Making data accessible:● Built-in search● Permanent URIs● Standardized Feeds● Real-time Alerts
● REST Architecture with Feed Publishing● RSS/Atom => Pubsubhubbub => Alerts
![Page 7: Open Legislation Spring 2011 Talk 1](https://reader031.fdocuments.net/reader031/viewer/2022020306/554dc525b4c905bd488b5038/html5/thumbnails/7.jpg)
So back toOpen Legislation
![Page 9: Open Legislation Spring 2011 Talk 1](https://reader031.fdocuments.net/reader031/viewer/2022020306/554dc525b4c905bd488b5038/html5/thumbnails/9.jpg)
Its not a Service;Its an Open Platform
![Page 10: Open Legislation Spring 2011 Talk 1](https://reader031.fdocuments.net/reader031/viewer/2022020306/554dc525b4c905bd488b5038/html5/thumbnails/10.jpg)
1 Year Re-cap
● Open Sourced It (for real)● Improved the API (xml/json)● Decreased Load Times● Restructured the Back-end● Basic Documentation● Wrapped into a build system
![Page 11: Open Legislation Spring 2011 Talk 1](https://reader031.fdocuments.net/reader031/viewer/2022020306/554dc525b4c905bd488b5038/html5/thumbnails/11.jpg)
The next year
● In general..● Data Quality and Documentation● Usage Tracking and Statistics● User Interface Improvements● Further separation of the Platform and Service
● Right now● Data Quality, Data Quality, Data Quality● And a little bit of documentation
![Page 12: Open Legislation Spring 2011 Talk 1](https://reader031.fdocuments.net/reader031/viewer/2022020306/554dc525b4c905bd488b5038/html5/thumbnails/12.jpg)
The Senate has LegislativeData Quality issues?
![Page 13: Open Legislation Spring 2011 Talk 1](https://reader031.fdocuments.net/reader031/viewer/2022020306/554dc525b4c905bd488b5038/html5/thumbnails/13.jpg)
Well, not exactly
● Legislative Research Service has the data● Big, ancient mainframe to boot
● They FTP us updates every 5 minutes● In SOBI formats (what?)● With some XML mixed in
● We parse it back into XML/JSON/SQL structure
![Page 14: Open Legislation Spring 2011 Talk 1](https://reader031.fdocuments.net/reader031/viewer/2022020306/554dc525b4c905bd488b5038/html5/thumbnails/14.jpg)
Reasons for Difficulty
● Poorly Documented SOBI behavior
● Formatted as a change log (sometimes)● Finding sources of error can be hard
● LRS is not co-operative
![Page 15: Open Legislation Spring 2011 Talk 1](https://reader031.fdocuments.net/reader031/viewer/2022020306/554dc525b4c905bd488b5038/html5/thumbnails/15.jpg)
Solutions
● Version Control● Write objects to JSON/XML files● With Git, commit each new version
– Commit message points to the source SOBI● Use git to trace data errors back to SOBI files
● Unit Test known corner cases
● Periodically do a scrape check?
![Page 16: Open Legislation Spring 2011 Talk 1](https://reader031.fdocuments.net/reader031/viewer/2022020306/554dc525b4c905bd488b5038/html5/thumbnails/16.jpg)
Progress
✔ Parsing has been overhauled✔ Objects are written to file✔ Bugs have been found and fixed✔ Periodic Scrapes are approved
![Page 17: Open Legislation Spring 2011 Talk 1](https://reader031.fdocuments.net/reader031/viewer/2022020306/554dc525b4c905bd488b5038/html5/thumbnails/17.jpg)
A short task list
✗ Integrate git into the parsing system.✗ Document expected behavoir✗ Write a small test suite✗ Try to avoid having to scrape.
![Page 18: Open Legislation Spring 2011 Talk 1](https://reader031.fdocuments.net/reader031/viewer/2022020306/554dc525b4c905bd488b5038/html5/thumbnails/18.jpg)
HFOSS Symposium 2011
● Bryan Sivak – Civic Commons● Mark Prutalis – Sahana Foundation● Many universities, Mozilla, Google
● David, Moorthy, Brian, and Myself!● 1 Hour and a few 3' x 4' posters.