The Gory Details Of Datajournalism: what went wrong and who was responsible by Henk van Ess

Data Reality CheckHenk van Ess henk@vaness.nl

chairman WWW.VVOJ.NL

Sunday, March 25, 12

Andrew Lehren, NYT

Age of Data by Geoff McGhee Paul Bradshaw

Digital Diggers ProjectData Driven Stories Book

http://www.vvoj.nl

This year: a new book & site about data journalism

Let’s cut the crap

Who was in charge and what went wrong?

The CoderThe ReporterThe Designer

Showing what is possible, not what is sensible

Assembly line syndrom: no place for new workflows

Do you use tools like:

Do you use tools like:before

List of dangerous chemical companies

before

Adobe Captivate

before

Adobe Captivate

before

There are many new dedicated tools

Timeline the bad way

Timeline the bad way Timeline with Timeline 3D

This was done in 30 minutes

Mind boggling presentations.. leaving your mind boggled

Showing problems, not explaining them

when no one is in charge (anymore)

empty pages

broken databases empty pages

Datajournalism is

Finding information

Datajournalism is

Finding information

Processing information

Datajournalism is

Finding information

Presenting information

Datajournalism is

Put like that, it seems the job description for any reporter

Finding information

Presenting information

Datajournalism is

To be more specific, you need:

A fine researcher

A fine researcherA smooth talker

A fine researcherA smooth talkerA good number cruncher

A fine researcherA smooth talkerA good number cruncherA great story teller

A fine researcherA smooth talkerA good number cruncherA great story tellerA brilliant designer/ GUI specialist

A fine researcherA smooth talkerA good number cruncherA great story tellerA brilliant designer/ GUI specialistFantastic communicator

“overlapping set of competencies drawn from disparate fields” http://bit.ly/nLlbYn

You offer:“overlapping set of competencies drawn from disparate fields” http://bit.ly/nLlbYn

Your problem

Data journalism is a “baggy monster of a field”5 tips for getting started in data journalism

http://www.poynter.org/how-tos/digital-strategies/147734/5-tips-for-getting-started-in-data-journalism/

We can learn from Nokia, for once :

Go for teamwork!

The gory details

The problem

Reporters bored with Wikileaks

Most explosive documents were already published

Finding new leads in new documents was hard

Colleagues told me: ‘It isn’t worth our time’

Me, being shocked

I said:

Me, being shocked

I said:

If you don’t check this, who does?????Me, being shocked

How we started:

No inhouse coders

How we started:

No inhouse codersNo designers

How we started:

No inhouse codersNo designersNo funding

How we started:

No inhouse codersNo designersNo fundingNo technical infrastructure

How we started:

How it ended:

http://cablesearch.org/?page_id=46

So what happened here?

www.cablesearch.org

How it started

http://screencast.com/t/1scMTaBgZytF

Why did this succeed?

It solved a problem

For news beats

It solved a problem

For news beats

It solved a problem

For news beats

It solved a problem

For news beats

By tags, for specialist & background stories

It solved a problem

For news beats

By tags, for specialist & background stories

It solved a problemSharing with public gave us great new ideas

It solved a real problemCrowdsource!The Business plan paid for infrastructure

Even more gory details

Volkskrant:

• Newspaper demanded tons of data about the state of elderly homes in The Netherlands

The response

Volkskrant

Volkskrant• Pushed over and over again to get the

The response

Volkskrant• Invoked the freedom of information act

The response

Volkskrant• They kept pushing

After one year

After one year Finally. Succes!

A cd full of data

• 270.000 questions• 320.000 answers• Devided in 9 categories • More then 12 subcategories

Overwhelmed by data:• 270.000 questions• 320.000 answers• Devided in 9 categories • More then 12 subcategories

Minor detail: all 4500 forms are PDF

So what now?

It’s 2012. Don’t despair

Anatomy of a projectI posted the pitch for the project:

“Need coder for converting 4500 PDF’s”

One minute later

The Whole World Is Bidding

And Bidding

The result was:

The result was:• Over 60 programmers wanted to convert the

data into Excel and Web (MySql) for a price of $100 -$3000

• Choose the ones that are in the top 300 (out of 200.000 coders) to avoid surprises

Topcoder for $500

Out of 9 projects

• 8 were a success• 1 was a miserable failure • Total number of publications: 7, 1 pending

Don’t forget

Always 2x coders for the same jobPerfect Pitch is ParamountDataminded reporter should do the pitch

Project lessonsHire coders, don’t become oneDo get the concepts behind data mining so you can instruct the coder Outsource smaller projects can be done

Back to the Volkskrant

The first job was done in 5 days

The first job was done in 5 days• PDF converted to normal text

Other steps:

• Text converted to Excel• Excel converted to MySql• Search engine for reporters enabled

them to ‘Google’ data of the homes - later used as public search

FINALLY SOME ANSWERS

• How many homes have financial troubles?• How many aggressive people caused

trouble?• How many actions of non-medical

employees were not backed up by a doctor?

• Which elderly home is the best/worst?

FINALLY SOME ANSWERS

Unexpected questions:

• Why are homes with a high mortality rate?• Why are homes with a sickness absence

of 15-20% ?

Results

• Two lead news stories on the front page• A special section about the best/worst

homes• ‘Top 300’ on the web for consumers:

where do I put daddy away• A lot of goodwill

The lessons:

• 1.Datamining is the start of the project, not the end

• 2 Reporters write the story• 3 Public comments on story - new stories

School dataThe 2x 500 euro Solution

What ifYou want to report on a regular basis on the same

subject?

Spidering or custom programming ?

Roughly 7500 records in two spreadsheets

How it was done

Brin codes identify school

http://functiemix.minocw.nl/resultaten.aspx?criteria=1&type=1&brin=00AH

How it was done: securing school codes

List of BRIN codes

ScreenScraper

BuildURL

Req.Page

GetPage

Build Report

How it was done: Building URL (from list)

List of BRIN codes

ScreenScraper

BuildURL

Req.Page

GetPage

Build Report

How it was done: Request web page

List of BRIN codes

ScreenScraper

BuildURL

Req.Page

GetPage

Build Report

How it was done: Get & parse web page

List of BRIN codes

ScreenScraper

BuildURL

Req.Page

GetPage

Build Report

How it was done: Build report

List of BRIN codes

ScreenScraper

BuildURL

Req.Page

GetPage

Build Report

How it was done: REPEAT until done

List of BRIN codes

ScreenScraper

BuildURL

Req.Page

GetPage

Build Report

Things to consider

Things to consider• Creates competitive advantage

Things to consider• Creates competitive advantage• - Saves time

Things to consider• Creates competitive advantage• - Saves time• - More accurate

Things to consider• Creates competitive advantage• - Saves time• - More accurate• - Not tedious, no insult to intelligence

Things to consider• Creates competitive advantage• - Saves time• - More accurate• - Not tedious, no insult to intelligence• Legal ramifications

– Always be respectful of your source

– Always be respectful of your source – You can’t copyright facts

– Always be respectful of your source – You can’t copyright facts– You can copyright how facts are presented

– Always be respectful of your source – You can’t copyright facts– You can copyright how facts are presented– Don’t make yourself conspicuous

LessonsBuild once, use often

Generic & important: zip code / geodata / feedback

The 300 euro project

11/20/08 12:51 Henk van Ess

digital digging: converting data

Source: Funda

Technique: ‘datascraping’

Makelaars manipuleren woningaanbod ROTTERDAM, 3 JAN. NVM-makelaars en huizenverkopers manipuleren met te koop staande huizen, door ze een paar dagen van de markt te halen en daarna als nieuw aan te prijzen. De afgelopen drie maanden is dat over heel Nederland verspreid meer dan 2.700 keer gebeurd

• Script was written to download the data of more then 300.000 houses during three

months

• The results were put in an Excel file.

• Reporters browsed manually for anomalies

Results

Results• Two frontpage stories• 1 Appendix• Higher circulation

Nice, but..

Can’t you do it cheaper?

The 0 euro-projects

european databases

Creative thinking

• What databases are there online in Europe?

• Hmm, there is a register of all people who attended meetings, a presence register.

11/20/08 12:51 Henk van Ess

So what??• What can you do with such a register?• Study which members are there and which not• Study which countries are more absent than others• How big is the effect if certain members of countries

aren’t voting at all?• In reality some countries have more influence then

you would expect from the actual number of seats

11/20/08 12:51 Henk van Ess

CAR: do try this at home! The Netherlands has

more power

Source: EU: database, available in 20

languages

Elaborate work..

Healthcare regulators

11/20/08 12:51 Henk van Ess11/20/08 12:51 Henk van Ess henk@vaness.nl

Tuchtcollege voor de Gezondheidszorg

Verdicts against doctors• Uitspraken van het CTG• <http://www.tuchtcollege-gezondheidszorg.nl/recent_files/recent.htm>De meest

recente uitspraken<http://www.tuchtcollege-gezondheidszorg.nl/recent_files/recent.htm>

• <http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/huisarts.htm>Uitspraken over huisartsen en basisartsen<http://www.tuchtcollege-

gezondheidszorg.nl/huisarts_files/huisarts.htm>• <http://www.tuchtcollege-gezondheidszorg.nl/specialisten_files/

SPECIALISTEN.HTM>Uitspraken over medische specialisten<http://www.tuchtcollege-gezondheidszorg.nl/specialisten_files/SPECIALISTEN.HTM>

• Uitspraken over psychiaters<http://www.tuchtcollege-gezondheidszorg.nl/psychiater.htm>

• <http://www.tuchtcollege-gezondheidszorg.nl/tandarts_files/Tandartsen.htm>Uitspraken over tandartsen<http://www.tuchtcollege-

gezondheidszorg.nl/tandarts_files/Tandartsen.htm>• <http://www.tuchtcollege-gezondheidszorg.nl/verpleegkundigen.htm>Uitspraken

over verpleegkundigen<http://www.tuchtcollege-gezondheidszorg.nl/verpleegkundigen.htm>

• <http://www.tuchtcollege-gezondheidszorg.nl/verloskundigen.htm>Uitspraken over verloskundigen<http://www.tuchtcollege-gezondheidszorg.nl/verloskundigen.htm>

• <http://www.tuchtcollege-gezondheidszorg.nl/apotheker.htm>Uitspraken over apothekers<http://www.tuchtcollege-gezondheidszorg.nl/apotheker.htm>

Total chaos•

2005.009 en 046<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.009%2520en%2520046.htm>2005.012<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.012.htm>2005.013<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.013.htm>2005.014<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.014.htm>2005.015<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.015.htm>2005.016<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.016.htm>2005.017<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.017.htm>2005.018<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.018.htm>2005.019<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.019.htm>2005.020<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.020.htm>2005.032<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.032.htm>2005.033<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.033.htm>2005.034<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.034.htm>2005.055<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.055.htm>2005.109<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.109.htm>2005.110<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.110.htm>2005.151<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.151.htm>2005.152<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.152.htm>2005.155<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.155.htm>2005.178<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.178.htm>2005.211<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.211.htm>2005.223<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.223.htm>2005.224<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.224.htm> 2004.005<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2004.005.htm>2004.012<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2004.012.htm>2004.014<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2004.014.htm>2004.021<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2004.021.htm>2004.022<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2004.022.htm>2004.023<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2004.023.htm>2004.024<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2004.024.htm>2004.025<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2004.025.htm>2004.026<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2004.026.htm>2004.027<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2004.027.htm>2004.028<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/

Step 1: download the site

Step 2

Step 2• Download a desktop

search tool

Step 3

Step 3 Start typing questions as you do in Google

Like this:

DIFFERENCES BETWEEN COURT RULINGS

Coverstory

This presentation:

HENK VAN ESSwww.searchbistro.com

henk@vaness.nl@henkvaness

http://www.slideshare.net/searchbistro

The Gory Details Of Datajournalism: what went wrong and who was responsible by Henk van Ess

Education

Transcript of The Gory Details Of Datajournalism: what went wrong and who was responsible by Henk van Ess

Kocie Gory (nr 3)

Agata Maj - Gory Fogaraskie

Final TR Data Journalism TK PR - European Data Portal · Open!Data!Fuelling!DataJournalism! ePSIplatform Topic Report No: 2011 / 2 October 2011 Page 2 Keywords)! PSI!Directive,!datajournalism,!dataAdriven!journalism

Aline datajournalism 2_giugno2012

Gory cucine - Elisee

Henk Gianotten - Untitled

OKfestival Datajournalism hack

Cos’è il datajournalism e come può nascere un team

Datajournalism café datas #2

The Gory Details

Dispensa Datajournalism | Maggio 2014 | school.dataninja.it

Technolowho? - Henk Zeijts

Datajournalism i Norden 2013 - Páll Hilmarsson

DataJournalism: How To get data and process them?

Henk Keesenberg 0651926723 henk@wkonderwijsadvies.nl

The Gory Details of Debian packages

Gory Swietokrz

JSP – Java Server Pages: The Gory Details

Intro datajournalism - 14-15/06/2017

Kocie Gory (nr 2)