Tim Willoughby

ISA24th October

Big Data Big Costs?Open Data

Gartner Top 10 for 2012 -

• Released today by Forbes

3

What has Cloud ever done for us?

Apart from Scale, Speed, Agility, Low Cost, Enterprise Adoption, Enterprise Mapping, Open Data, Standards, Google, API’s, etcOpen Street Maps, Map Servers,

GIS - More than just Location, Spatial Analysis and wider adoption now possible

How Much Data do you create?

How much data to you create?

Retailers..• Assign each shopper a unique code - keeps tabs on

everything they buy. – If you use a credit card or a coupon, – or fill out a survey, – or mail in a refund, – or call the customer help line, – or open an e-mail we’ve sent you – or visit our Web site, – we’ll record it and link it to your ID,”

• “We want to know everything we can.”• “Cool is one click away from Creepy” (Fjord)

Your Demographic They Have or can get• You

o your age, o whether you are married, or got divorcedo and have kids, Your ethnicity,

• Where you live , the year you bought (or lost) your house, whether you’ve moved recently, • The number of cars you own. • Distance to nearest Outlet,

• Worth• Your estimated salary, if you’ve ever declared bankruptcy, Charity Giving• What credit cards you carry in your wallet

• Education, where you went to college, • Career - job history, • Online -what Web sites you visit., the magazines you read, reading habits,

• The topics you talk about online, • Likes- certain brands of coffee, cereal etc, political leanings,

Big Data goes with…More Devices

More Access

More AppsMore Data…

Social

Mobile

DATA

Cloud

DATA

4 Global Themes

Social has overtaken mobile ….

Mobile Browsing has overtaken desktop…

Radical Change…

Change…With or Without the Owners / Shareholders

So far ICT has not fundamentally changed government

• 1990s: lCT expected to make government more transparent, efficient and user oriented

• 2005+: disillusion as bureaucracy still in existence

• Can Mobile / Data /Cloud Help?

Jane E. Fountain – Gov 1.0 – Just Replicating the Silos on the Internet

Goverment dont always Understand What the people Want?

What governments often Deliver

Open Government?

“If people don’t know what you’re doing, they don’t know what you’re doing wrong”

“Jim Hacker” Open Government (I980)

The World has changed…

Every day - you come across multiple…

• - Devices• - applications• - access points• - Networks• - Locations• - Servers• - Data types• - Data caches• - Service providers• - Operating systems• - Storage locations• - Encryption / security protocols..

My Smart Device is…. Very smart

Source : Peter Cochrane

It knows more about me than my partner…

Source : Peter Cochrane

Security is changing

• Security has to be appropriate• Security has to be measured • Can have things so secure that they are

unusable.

Citizen Expectation Changed?

Social

Media

Mobi

le

Feeds

Web

Publicati

on

Integratio

n of Web

data sets

Mashup applications

The More data we have… the less we (have time to) Understand it

Government as a Platform?

Mobile Development Spend is rapidly overtaking PC Development Spend

BIG DATA?

Massive network of services: water, sewage, drains … Need to know asset location for planning and maintenance Many databases, varying accuracy and provenance Context

Ongoing street openings p.a. Safety!

Typical Individual effort

many hours, one map

OpenStreetMap, 2012

200,000 contributors, one map

Our Lives are Different…

• We communicate on — Facebook, Chat, Twitter, MMS• We Research Information — Blogs, eNews, Wiki,

YouTube• We Buy — eBay, Amazon, Dell, Deal Done, etc…• We Travel - Tripadvisor, Flickr, RyanAir, Hotels, • We Meet / Retate - Linkedln, FaceBook, Friends• We Play – xBox, Playstation, Online games, ¡TV• We expect the Government to fit into these

paradigms too…

What is Big Data?• Size beyond the ability of

commonly used software tools

• Data growing challenges• volume increasing• velocity (speed of i/o)• variety (data type/source)

• Examples: computer tracking of shipments, sales, suppliers, customers, email, web traffic, social network

From McKinsey Report

Why Now?. What is the Catalyst?

• People– Web sites with 300+ million unique visitors/month• Facebook• Google• YouTube

• Access– 1.2 Billion active mobile broadband subscriptions

• Commodity (community) Hardware – possibility for cost effective processing

Facts!• “Data is a vital raw material of the information

economy, much as coal and iron ore were in the Industrial Revolution.”

• “Mining and analyzing these big new data sets can open the door to a new wave of innovation, accelerating productivity and economic growth.”

• We can exploit Internet-scale data sets to uncover new businesses and predict consumer behavior and market shifts.

location data sales business data social network data

Think…

• NY Traffic– They have mountains of data– More Daily– Don’t Understand it, can’t use it– What is the True Cost of Data Mining– Systems, Software, Computers, Analysts, STORAGE– Are we collecting, collating, etc the Right Data– People being removed daily, replaced by machines– Need to Stand Back….

How to deal with Lots of data

• 7TB a Day for Twitter…• 10,000 CD’s• 5m Floppy• 225GB during this talk…• How do you Store that…• HD Write Speed- 80mb/s… • 24.3 Hrs to write 7TB…. • Government Thinking is not in this space.. Yet..

Irish Government

• Decision – “All Government Invoices Published”– Election Manifesto– Election Promises– Programme for Government

• Action– None…– Fear, Bad Reactions, Data Quality,

Stakeholders

• Private citizens• Activists• Coders• Entrepreneurs• NGO’s• Academics• Civil Servants• Gov Institutions

Which Apps?

• Government – Do not try to second guess what are useful applications!

• No one can predict what data to make open based on potential Apps.

• There are a huge number of apps available for smartphones. Impossible to guess a few years back.

• Put Data out there. Data as a platform for others to build on, as infrastructure.

• Apps you will like and hate, but it’s not up to you.

Applications?

Where is Open Data delivering?

• Participation / empowerment• Transparency, democratic control• New and better Government products and services• Impact measurement• Efficient and Effective Government • Data Journalism (Guardian Services)• New knowledge / real innovation

The Internet of Things Cars, Planes, Cameras,

Roads, Pipes, Public Cameras, Traffic Lights, Car Parks, Pharma and Farms – over 1 Trillion

Smart Local Government?

Mass increase in new net devices

• Move to digital controls• Network standards– TCP/IP

communications–Wireless

communications

Wireless integration

60

Or Wizards

GIS is now Mainstream

Water Meters

• Read the Meter• View or Pay the Bill• Compare the Usage to the

Local or National Averages

• Look for areas of Savings

Roadworks

• Link to System – Online Road Works Control (OLRWC)

• License for Road Opening

• Cost of Reinstatement (Local Authority Usage)

• License for any Street usage.

Planning Applications

• View Applications• Get notified of Changes• Make Comments• Make a Submission• Notify Local Authority

of Issues?

Sports

• What is available now• How do I get there• What if I have to Cancel• Pay for it now

Parking

• Select County• Select Zone• Select Price willing to

pay?• Where is the Cheapest

open source / open data / cloudthe fix your street model

The Reality of FixYourStreet

But all of these are static…..

But… would you cross the road based on 5 min old information

Why Open Data

• Because the Data already belongs to everyone!• Because there is Law, Inspire Directive - PSI• It helps change the incentives, from Fail Big, Fail Slow to

Fail Small, Fail Fast, and Fail Forward (Innovation – More eyes)• Open Data will allow issues/problems to be seen, making them

less likely to be repeated (learning from other people’s failures)• Open Data use can allow Incremental Innovation: - stand on

the shoulders of giants. Don’t reinvent the wheel — use or improve someone else’s

• Open Data quality can be increased over time — use and reuse by other people and combining with other data identifies (and helps solve) quality issues far more quickly and cheaply than internal checking

How do we measure Success?

• Usage and Apps• Critical Mass of Companies• Search Results (Bing and Google Interested)• Business Committed to OD• Laws (Prison for breaking them)• If Open Data doesn’t cause difficulty for the

Publisher to seek excellence then it isn’t working

Where next - Augmented Reality

Bring together Big Data, Visualisation, GIS, GPS - Bringing Cloud, Social, Data and Mobile together…

What is Augmented Reality?

Water Meter

Water Meter

Understand what are the Driving Forces…

• Government Focus– Understanding, Efficiency, Accountability

• Technology Innovation Focus– Data as a Platform, Semantic Web

• Reward Focus– Profit, Recognition

• Digitising Government Focus– Computerisation / Technology Drive

• Problem Solving Focus– New Skills needed to work on new Challenge

• Social / Public Sector / Enterprise– More Focussed Services

Conclusions

• Change is constant• Cloud is the Future• Social is a Reality• Mobile is already taking over• Data has to keep up…

References

• We live in “Flat Land” – There is danger in making representations more seductive than the truth

• Envisioning Information - Tufte