Introduction - data.qld.gov.au · Web viewCCO suitability?? Public?? Data sharing policy aligning...
Transcript of Introduction - data.qld.gov.au · Web viewCCO suitability?? Public?? Data sharing policy aligning...
ODIQ Open Policy Engagement Final Report
ODIQ Queensland October 2016
1. IntroductionThe Queensland Government and ODI Queensland have engaged with the broader community to help inform development of a new Open Data Policy for Queensland.
Open Data is a growing global movement that assists organisations to share data freely and openly and strives for transparency and innovation.
The engagement program, ‘Open Data, Open Conversations – inspiring a more informed Queensland’, sought input from a variety of stakeholders across multiple industries.
Articulous Communications facilitated four Deep Dive sessions with representatives across a number of Queensland sectors: Business and Start-Ups, Public Service, Spatial, Industry Organisations, Not-for-profits and Research, Academics and Students.
This report provides a summary of the findings from the Deep Dive sessions, including detailed information from users of Open Data that will assist Government to develop an Open Data policy that best meets the needs of the community.
The Deep Dives were supported by other engagement methods including:1. A preliminary survey conducted by the QLD Government from 21 July to 28
September through its open data portal to understand what types of open data were being used.
2. Social mapping on Social Pinpoint of opportunities and challenges around key themes relating to accessing, using, updating, sharing and reporting on Open Data.
3. Online community forum for discussion around key themes for the Open Data policy including managing data, licensing and attribution, engaging with re-users, quality of data, finding and accessing data, suitability and data stories.
Key issues to emerge from Deep Dive sessions: Need for agreed standards and consistency across departments Provide visualisation of data Develop a centralised catalogue Establish consistent metadata standards Ensure greater usability – provide both raw and aggregate data Provide more clarity around common licensing Promote achievements, undertake more engagement Refine the operational aspects of policy
From these key issues, some similarities and differences were evident across different sectors.
1. Start-ups and business were more concerned with ease of access to data and greater visualisation, and the need to be more engaged.
2. Government was concerned about the quality of the data, sharing of information across agencies, greater consistency, improved standards, and creation of a single catalogue.
3. Spatial, industry and associations were concerned with licensing and being clear about what can be published, metadata and consistency with standards, use of visualisation, and celebrating successes.
4. Research, academia and students were concerned with user interface and ease of access, ensuring both fine grain and aggregate data are available, better search facilities and standardisation of terms and formats, and improved feedback and more engagement.
2. Process: Deep Dive SessionsFour Deep Dive sessions were held with different stakeholder groups to discuss challenges, opportunities, and needs for Open Data in Queensland.
Live polling was conducted at the beginning of each session to get the group thinking more deeply about some of the issues around accessing, using, and managing open data.
Other activities encouraged groups to draw on their experience and knowledge of real life scenarios and possible solutions that would assist to develop an Open Data Policy.
The Deep Dives created an environment of knowledge-sharing and reflected the open data values of ODIQ.
All raw data from the live polling, and attendees input, are available as attachments to this report.
In total there were 82 participants over the four Deep Dive sessions.
This included representatives from 42 non-government organisations
Total non-Queensland government organisations represented in the Deep Dives included:
Session Total organisations
Business and Start-ups 22
Spatial, industry and non profit 12
Research Academia Students 8
The number of non-Queensland Government organisations providing feedback across all engagement was 79 – this included 42 for the forums, 20 for the online forum, and 17 for the social mapping.
At least 3 organisations from each sector provided feedback during the engagement.
Wednesday morning session (businesses and start-ups) had 16 participants with average online polling engagement at 77%.
Wednesday afternoon session (government) had 32 participants with average online polling engagement at 78%.
Thursday morning session (spatial, industry and associations) had 22 participants with average online polling engagement at 83%
Thursday afternoon session (research, academia, students) had 18 participants with average online polling engagement at 81%
2.1 Deep Dive Sessions Agenda
Session Details
Welcome and introductions
Welcome to participants.
The Project Overview of the Open Data Open Conversations Project including: What the Project involves Why We’re Doing it What We Hope to Discover What We hope to Design How We’ll Use the Information
YOUR EXPERIENCES:Live Polling
Live Polling to gather information on the group. Questions …1. How have you ever used open data? Yes, No2. What did you use it for? Open-ended3. What type of information would you be most likely to access with open data?
Health, transport, education, housing, business, employment, economic development, spatial, environment, justice and legal, local government, community services, recreation, emergency services, other.
4. What areas do you believe are the most important when it comes to using open data? Business improvement, increased knowledge, better service delivery, greater transparency, keeping up with global movements, being an industry leader, innovation, data management and quality, ongoing access.
5. How do you deal with multiple data sources and should there be a single source of truth? Open-ended
6. Given that globally more data has been created in the past two years than in the rest of human history, how can we maintain the currency of data? Open-ended
7. How can we best enable government agencies to release and update relevant data regularly? Open-ended
8. How concerned are you that open data can be used for purposes that are unintended or in a way we don’t like? Scale
9. Why did you answer this way? Open-ended10. How can we address the concern of re-identifying anonymised data? Open-data11. Is there a way to track and monitor what business and the community needs in
terms of data? Open-ended12. If we believe in open data should we decide that some data should not be
released because there is no demand? Open-ended13. Who is the owner of the data? (The government, The public, Those who use the
data)14. What’s one thing you would change to make open data easier to find, access and
use? Open-ended15. What is the greatest challenge when it comes to developing an Open Data Policy
for Queensland? Pin-drop on image16. What are the most important outcomes you’d like for your sector?
YOUR EXPERIENCES: Share your stories
Shared experiences of Open Data (small groups) Discuss:
Range of experiences What worked well What didn’t work well
1-minute reviews. Choose a few tables to share their experiences with the larger group.
Morning Tea break Video snippets of people taken.“One word” for open data
YOUR NEEDS: Challenges and Opportunities
Mapping the challenges and opportunities along an Open Data Journey using the following themes.
Finding and accessing data Data quality Suitability Licensing and attribution Managing data Engaging with re-users Implementation
YOUR NEEDS:Common Challenges and Opportunities
Each group to present key findings.
YOUR SOLUTIONS: Map solutions
Map out solutions to those challenges and opportunities that could be incorporated into the Open Data Policy.
SCENARIO TESTING
Each group to come up with an existing or future project they’d like to access open data for, or one they’d discussed earlier. Then take another group’s solution and test their scenario against it. Scribe up new ideas on the solution template.
Group debrief.
YOUR OUTCOMES
Open discussion
What are the most important outcomes you’d like for your sector?
Live poll (open text) or group discussion.
Next Steps Describe how the input will be considered
3. Live PollingLive polling was conducted at the beginning of each session and showed varying results across the four groups. Findings for the close ended questions are listed below.Key: Stand-out differences, stand-out similarities
Question Business and Startups Wed AM
Government Wed PM
Spatial, industry and non profit Thur AM
Research, Academia, Students Thur PM
What type of information would you be most likely to access using open data?
Top 3:Business 24%Local Government 12%Health 10%Transport 10%
Top 3:Transport 20%Spatial 15%Emergency Services 11%
Top 3: Spatial 29%Environment 17%Transport 15%
Top 3:Education 17%Health 15%Economic Development 13%
What areas do you believe are the most important when it comes to using open data?
Top 3:Innovation 27%Business Improvement 17%Greater transparency 15%
Top 3:Greater Transparency 22%Better Service Delivery 16%Data management and quality 14%
Top 3:Data management and quality 19%Ongoing access 15%Greater transparency 14%
Top 3:Increased Knowledge 29%Better service delivery 18%Greater transparency 16%
How concerned are you that open data can be used for purposes that are unintended or in a way we don’t like?
1. Not at all concerned (36%)
2. I’m neutral (29%)
3. Very concerned (21%)
4. Concerned (14%)
5. Slightly Concerned (0%)
1. Not at all concerned (39%)
2. I’m neutral (25%)
3. Slightly concerned (21%)
4. Very concerned (14%)
5. Concerned (0%)
1. Concerned (40%)
2. I’m neutral (25%)Not at all concerned (25%)
3. Slightly Concerned (10%)
4. Very concerned (0%)
1. Not at all concerned (38%)
2. Slightly concerned (31%)
3. I’m neutral (19%)
4. Concerned (13%)
5. Very Concerned (0%)
If we believe in open data should we decide that some data should not be released because there is no demand?
1. All data should be released (57%)
2. It depends on the type of data and the level of demand (21%)
3. Not all data should be released (14%)
4. Not sure (7%)
1. All data should be released (50%)
2. It depends on the type of data and level of demand (36%)
3. Not all data should be released (11%)
4. Not sure (4%)
1. All data should be released (42%)
2. It depends on the type of data and level of demand (32%)
3. Not all data should be released (26%)
4. Not sure (0%)
1. All data should be released (75%)
2. It depends on the type of data and the level of demand (13%)
3. Not all data should be released (6%)
4. Not sure (6%)
Who is the owner of open data?
1. The public (71%)
2. The Government (21%)
3. Those who use the data (7%)
1. The public (75%)
2. The Government (25%)
3. Those who use data (0%)
1. The public (79%)
2. The Government (21%)
3. Those who use the data (0%)
1. The public (73%)
2. The Government (20%)
3. Those who use the data (7%)
4. Key Findings
4.1 Deep Dive AM Session One: Wednesday 05 October, 2016: Business and Start Ups16 Attendees
4.1.1 Key Themes from Across the Session:
Easier access and user interface/user experience that better matches to how we find information online
Visualisation of data Need to engage in an ongoing way with the community Open by default Open data as a means for innovation Need for standards and consistency of open data Need for visualisation of data
4.1.2 Stakeholder Reflection:
Given that globally more data has been created in the past two years than in the rest of human history, how can we maintain the currency of data?
“Open access...the community keeps it current if it's valuable, Metadata and usable governance controls, Automated collection and transformation”
“Faster move towards digital govt, need insights specialists inside govt like corporates, govt consume its own data!”
“The future is in realtime data, Simplify access, Reliable IT systems”
4.2 Deep Dive PM Session Two: Wednesday 05 October, 2016: Government32 Attendees
4.2.1 Key Themes from Across the Session:
Quality of data and how to ensure it is accurate Resourcing of open data Interagency sharing of information and how this relates to procurement of information,
consistent formatting, custodians Revenue impact of releasing government information for free that used to be sold Need for standards and consistency of open data Establish performance indicators for open data publishing for DGs and Senior
Executives, to incorporate this into strategic planning
4.2.2 Stakeholder Reflection:
Is there a way to track and monitor what business and the community needs in term of data?
“Socially engaging digital infrastructure, Profiling to understand needs”
“One point in and out rather than having to deal with separate depts, Use less jargon”
“Single point of contact. Visualisation of requests, Need to move beyond the spreadsheet”
4.3 Deep Dive AM Session Three: Thursday 06 October, 2016: Spatial, Industry Associations and Not-for-Profits16 Attendees
4.3.1 Key Themes from Across the Session:
Potential risks associated with unintended uses Licensing of open data, and the need to be clear what is open data and what is
generally published information Need for visualisation of data Mapping of existing standards Celebrate successes and case studies of usage Metadata (use of metadata, type of metadata included and education of users to
understand what metadata is and how it informs them of the constraints of the open data available)
Federation of data standards / catalogues Measure the value of open data (including the financial benefits to business, plus the
opportunity cost of not maintaining open data) Better articulate and communicate the “why” of open data
4.3.2 Stakeholder Reflection:
How can we address the concern of re-identifying anonymised data?
“De-aggregate to the point where this is not possible”
“There will always be someone who can put two and two together. Difficult to prevent”
” Can't avoid it... so own it, Machine learning can already identify and de-anonymise data”
4.4 Deep Dive PM Session Four: Thursday 06 October, 2016: Research, Academia and Students18 Attendees
4.4.1 Key Themes from Across the Session:
Access of open data which is easy to use Granularity. Need to access fine grain data, as well as aggregate data Need for semantic technologies to assist searchability by incorporating semantic (as
well as instructional) attributes to data. This might include the use of synonyms on terms, relationships between columns.
Standardised format for open data Common licensing Use a participatory approach to Open Data. Users to contribute and provide feedback
on what Open Data is available and in what format Need for raw data to be available and not just aggregate data. And that government
should be responsible for publishing raw data while users are responsible for cleaning data
Consider open data from the perspective of producer to user. Promote the value and benefits of Open Data through user feedback and success
stories Important to define the scope of Open Data and what needs to be an exception Aggregation of data needs to recognise individual privacy (maximum value vs maximum
protection) User experience / interface
4.4.2 Stakeholder Reflection:
What areas do you believe are the most important when it comes to using Open Data?
“Increased knowledge, Better service delivery, Being an industry leader”
“Better service delivery, Greater Transparency, Increased knowledge”
“Innovation, Ongoing Access, Data management and quality”
A number of attendees from across the sessions all identified Standards, Quality, Consistency and Clarity as their desired outcomes of the Open Data Policy.
5. Raw results Wednesday AM Group – Businesses and Start-upsOpen Data Template Responses: Challenges and Opportunities Activity
***Note some of the table fields have been left blank to indicate no response from participants***
Group 1
WED AM Final Challenges Opportunities
Finding and accessing data
AvailabilityFormalSilos of open data
Aggregating data and federating searchStandardsIncrease value awareness
Data quality CompletenessCurrencyReliabilityIntegrity
Open data standardseducation
Suitability Collected for specific purpose-not alwaysSuitable for other purposesLack of disclaimersBalance of sensitivities e.g. Flooding data impact real estate
Metadata (minimum)Disclaimers for data-why was it collected, how, when, refresh rateExtensible frameworks
Licensing and attribution
Lack of awareness of open data licenses and attribution
Consistent use of open data licensingFree data
Managing data Formats not machine readableCurrencyQuality controls
Open data standardsMetadataEducation in data management
Engaging with re-users
Feedback loopsNo need to know ‘who’ is using data but how and whyChampion data use across all levels of govt
Profiling user groupsCommunity of practicesEnabling feedback loops and data requests
Implementation Lack of expertise and knowledgeLimited open source options
Digital infrastructure with intuitive workflow/curation
WED AM Solutions
Finding and accessing data Standardisation of open data formatsFederating portal search engines to facilitate discoveryOpen APIs to facilitate searching portals
Data quality Minimum requirements for quality – reliability, currency, timeliness, accuracy, completeness, disclaimerData quality levels need to be identified against an agreed
standardLow data quality must not delay publishing
Suitability Frequency of data updates must reflect/support the data being collected and resources used to be allocated accordingly – i.e. real time data cannot be constrained/published daily/weekly/monthly because of resource constraints
Licensing and attribution Consistent data attribution using format or metadataData shouldn’t be open if it isn’t licensed
Managing data
Engaging with re-users
Implementation Identify, support and promote implementation expertise through communities of practice and authoritative sources
Group 2
WED AM final Challenges Opportunities
Finding and accessing data
Don’t know what is thereLittle context on a data setNo business problem-taggingGoogle search to find related data sets
*con-tags and meta data
Data quality ConsistencySource of truth
providence
Suitability No metatags-discernibilityData structure-mashing
Choice in download-cube wizardDatabase query tool on portal
Licensing and attribution
Knowing what it means EducationRevenue streamDigital object identifier
Managing data Different data sets in different formats-consistencyDifferent data aggregations “by year/by month”Down to human created data sets-various skill/knowledge
Engaging with re-users
Feedback from data consumers
Implementation CK on limitations Wizard
***No response from Group 2 for solutions
WED AM Solutions
Finding and accessing data
Data quality
Suitability
Licensing and attribution
Managing data
Engaging with re-users
Implementation
Group 3
WED AM Final Challenges Opportunities
Finding and accessing data
Narrowing down data sourcesWhat data exists? (traffic, population, crime rates, development)Availability of local government bylaws, costs
Google, business portals/enterprise centres, chambers of commerce
Data quality Currency, consistency, reliability Establishing standards for accuracy, rating of quality (e.g. gold, silver bronze), ability to edit data sets
Suitability Publisher and custodian knowledge (depending upon user)
Reuse existing data sources and processes
Licensing and attribution
Managing data Data Hard to integrate LGA’s /SA’s
Able to edit data (editable)
Engaging with re-users
Dealing with the government – finding who has data, whether government is sensitive to
Feedback/contact pointUser survey
requirements of those who need to use data
Implementation Automate the dataDependency on it
Small business data service – portable to access, with all data small businesses may need – as portal/mobile app
WED AM Solutions
Finding and accessing data
Federated searches ‘Data.gov.au’
Data quality
Suitability
Licensing and attribution
Managing data 1.
Engaging with re-users
“communication”-just talk. Only solution to overcome ‘use of data’.Establishing processes -“social aspect”. This is seen as critical. Online forum for each data set.
Implementation
5.1 Attendees Storytelling Activity: What worked and what didn’t
5.1.1 Storytelling:Group 1: Validate procurement spend Human Services SpendGroup 2: Medical Research Institute Medical issues for returned servicemenGroup 3: Companies above a certain size across Brisbane
5.1.2 What worked:Group 1: Most reliable source Open Data Portal-Department of Communities Easy to find/UseGroup 2: Process Problem identification goodGroup 3: Quick responses. No data available (i.e. Brisbane City Council) ABS has data- not good
5.1.3 What didn’t:Group 1: Standards-hard to compare apples to apples
Group 2: Poor quality Different formats Difficult to mash together Time ConsumingGroup 3: Couldn’t find Not available for free Lack of knowledge
5.1.4 Attendees Feedback: What is the most important element/What should go in the Policy?
Problems are the same therefore we can apply same solutions Look at what’s worked inside companies and apply them to OD – continual evolution The users of data want to directly communicate with custodians “Standardised formats” – How do we choose & improve standards? Standardisation great but hard to achieve State Government doesn’t need to solve but looks at what others are doing Red Bull-sub sets, create a community, vertical partners and tags for research Many businesses are unaware how much data is available Accessing data for businesses, etc. can be made much easier through government
opening up to open data What measures do you put in place to make data effective? How do you allow open
data to evolve? Beta B Beta C /Channels of communication.
6. Raw Results Wednesday PM Group - GovernmentOpen Data Template Responses: Challenges and Opportunities Activity
Group 1
WED PM Challenges Opportunities
Finding and accessing data
Inconsistent formatLack of a catalogueFinding the right data sets ‘the attributes’Need exists for subject matter exerts to advise consumers of data
Publishing services
Data quality Lack of a catalogueFinding the right data sets ‘the attributes’Need exists for subject matter exerts to advise consumers of data
Different levels Custodian identified eg. State, localCloud toolsets
Suitability Non-existentDifferent data agreementsEffort to make consumable
Good metadataEnd user requirements
Licensing and attribution
Intended use of dataPoorly understood
Defined roles and licensing
Managing data Manual interventionIe. HoursMassage into DB
VisualisationBetter data management
Engaging with re-users
Accepting end-user limitations Multiple re-distributionModel-value add and connectivity of datasets
Implementation All of govt-no STD platform Universal platform (common)Web enabledMachine, DB services, cloud, OData
WED PM Solutions
Finding and accessing data
Mandatory to publish to open data-policy formulation
Data quality Filtering and cleansing toolsRatings (user review site)
Suitability Cyclical Shift Data-Info-knowledgePolicy to recognise this
Licensing and attribution
Awareness campaign
Managing data Vocabulary, synonyms, dictionary- similar to BIM, spatial data, infrastructure
Engaging with re-users
Not shared services arrangementOpen knowledge GP to roadtest/open data hub community to provide support and expert knowledge
Implementation
Group 2
WED PM Challenges Opportunities
Finding and accessing data
Lack of a (federated) catalogue (discovery)Metatags too broad
Data quality Annual report datasets-individual agenciesData capture governance
Explanatory notesImprove/refine data capture process
Suitability
Licensing and attribution
Managing data No authority-top down-resistance OD policy-governance, legislation, open by default.
Engaging with re-users
How do we identify re-usersHow do we connect with re-users
Survey on portalAnalytics, stats, user reports-hits, downloads etc.Close the loop-meaningful metrics-measure outcomes-business case.
Implementation Shared data- date, naming, title, standards.Government use open data-increase usage
Data-INB management strategy
WED PM Solutions
Finding and accessing data
Implement standard classifications
Data quality Concise data descriptionsClearly defined standards in the open data policy
Suitability
Licensing and attribution
Managing data Open data policy that opens by default
Engaging with re-users
Feedback survey on OD portal
Implementation
Group 3
WED PM Challenges Opportunities
Finding and accessing data
Poor metadata (not intuitive/plain English)
Design a good search engine
Data quality Data is not great qualityPoor metadata
To not worry so much about itStandardise data qualityShared data testing lab
Suitability Difficulty in obtaining feedback, user information
Treat data as an asset – not administrative burdenGet communities to drive demandTailor data sets to specific outcomes
Licensing and attribution
Managing data Strategic level inclusion Top Down management
Centrally mandated information standards Expansion beyond ABS
Engaging with re-users
Go beyond the ‘app’Explore shared data/use shared data as a testing ground
Implementation Policy statement 1-2 pages easy to digestBreak down once and for all/silos
WED PM Solutions
Finding and accessing data
Create a good search engine
Data quality Standardised data qualityEstablish WOG data dictionary
Suitability Reopen education/promotion of open data Research (ask them what it is they want to do)
Licensing and attribution
Keep it free at all times
Managing data Top down driven strategy
Engaging with re-users
Crowd sourcing/have specific release time and possible data source lsiting
Implementation Live feed of collected data directly to the data repositoryPolicy needs to be concise and easily understoodClear direction with principles
Group 4
WED PM Challenges Opportunities
Finding and accessing data
Who owns cross department data? What data does gov have?Procurement of data from outside of agencyRed tape to release data
Use previously collated open data registers.Reducing gov cost-joint fund data collection across jurisdictions. WOG clarity around who owns what.
Data quality How do you measure quality? Quality is different for each data setIncreased media scrutiny over lack of data quality
Increased public scrutiny, benefiting govtBetter public serviceImproving resourcing ability
Suitability How to take government internal government data sets and publish in a way which is suitable for public consumption
Intermediaries (organisations in business of wrangling data) to convert govt published data for better consumption
Licensing and attribution
Lack of knowledge and understanding of licensing & attribution
Managing data
Engaging with re-users
131GOV- it doesn’t work
Implementation Lack of understanding Increase knowledge
WED PM Solutions
Finding and accessing data
Improve portal-not user friendly, very boring. IS26 (CUE) locks down what gov can do with website
Data quality At a policy level-guidelines around data governance expectations
Suitability
Licensing and attribution Training Gov staff in open licensing and creative comms
Managing data Classification and standards to be applied to all datasets to drive ‘better’ metadata
Engaging with re-users
Meaningful granular mechanism for consumers to (connect directly)with custodians in a timely fashion. NOT 131 GOV
Implementation
Group 5
WED PM Challenges Opportunities
Finding and accessing data
No consistent taxonomyMany portals
Federated portalsCommunity led taxonomy development/open source Centralised tips & toolkits, SDKs
Data quality Staff hesitation (owner of data) exposurePoor governance BR data captureAccess to data custodian
Community scoringCommunity feedback
Suitability Lack of metadata (explain suitability, context)
Metadata standards
Licensing and attribution
Difficult attributing derived data Lack of confidence in CCNot understood
Education, awareness of license without attribution/regulations
Managing data Updates, automation, creation of API (register)} source, origin, renewalFormats, availability
Improve data quality in org.Embed OD in procurement
Engaging with re-users
Lack of resources, change management to protect end users of data, schedulingTrack usage/communication
Cross-community/jurisdictional collaborationConnecting with communityCommercialisation for startups
Implementation Ownership/resourcing/availability/marketingVisibility, prioritization, communicationEngagement/owners & publishers
More engagement with business for data publishing/sharing Identify and promote outcomes
WED PM Solutions
Finding and accessing data
Grow standardisation approach from small (e.g. SEQ) to large (e.g. state) bottom up Start with scan of existing standards and then settle on one (then top down declaration) bottom up & top down
Data quality Data set centric crowd support/forum and user scoring both public and gov participation
Suitability State data quality risk as part of metadataEnsure minimum metadata (OD certificate)
Licensing and attribution
Baden 2.0
Managing data Ensure business process relies on the data
Engaging with re-users
SLA’s and use service management source desk tools for OD
Implementation
6.1 Wednesday PM Attendees Storytelling Activity: What worked and what didn’t
6.1.1 Storytelling:Group 1: Publishing? Managing? Using? Flood data: releasing map-releasing to publicGroup 2: Capital works program/budget Status-heat maps Level of progressGroup 3: Crime data/usageGroup 4: Open data general (revenue) How will changes affect revenue? Ie. land tax clearance searches If we open up, we won’t sell any more Lost revenue Warranty on access to data Business needs to be confidentGroup 5: Accuracy of data Agencies need policy in place to release data confidently Issues: relevance, budget and qualityGroup 6: Wildlife database Massive increase in requests for data
6.1.2 What worked:Group 1: Nil ResponseGroup 2: Visualisation (ie heat maps)Group 3: Visualisation (heat maps) Use of controlsGroup 4: Purchase a warranty (Gov sells) on search access/access
Group 5: Education will assist Need strong policyGroup 6: Publishing-making it open
6.1.3 What didn’t:Group 1: Can’t release to public Methodology not release Inter-agencies checking other data Collating initially-not everyone using the same gauge Nested copyrightGroup 2: Couldn’t publish, slowed money spent in certain areas and not others. Bottom line figures Scared to publish Transparency-political interference How do we separate?Group 3: Nil ResponseGroup 4: Quality Data Capture Need for definitions for usage-people may not understand Make definitions at deliveryGroup 5: Out of date data Liability-who is responsible Don’t know who is using itGroup 6: Maintaining system takes resources Limited funding People making money off it-but how do you support it?
6.1.4 Attendees Feedback: What is the most important element/What should go in the Policy? Open data by default – everything assessed for publishing How can state gov policy benefit Central place which covers questions from users and prescribe standards Predefined list of data sets to be published by local and state government Tie together with QLD Government Information Security Classification Framework &/or
state government archives Responsibilities of the data publishers Supporting material (e.g. guidelines, fact sheets, FAQs) Both IT and Business level Open resources (publications), not just data In some cases, we may not need to require attribution, in the interest of efficiency
(under status quo, you require attribution) Consider the connotations surrounding the quality of data when attributing data to
certain sources Whole of government API Key Ability to consume data without interference Guiding principles around quality (machine readable, etc) Methodology: Who, How What When Why? Releasing users’ algorithms Right for community to form projects (community insurance)
7. Thursday AM Group – Spatial, Industry Organisations and NFP’sOpen Data Template Responses: Challenges and Opportunities Activity
Group 1
THURS AM Challenges Opportunities
Finding and accessing data
Difficult to find info from gov departments and authorities
Single access portalSingle query interface
Data quality Metadata standards-must publish planimetric/vertical accuracyData SourceGeometric errorsPlanimetric accuracy unknown, data source unknown
Suitability Geometric errors that prevent network analysisAttribute value rules not restrictive-ie.number fields defined as character
Tighten data modelsSystem to assess geometric data qualitySuitability statement
Licensing and attribution
People don’t understand licensing restrictions.Difficult to know which licenses apply to which data.
Educate public on specific license requirements.
Managing data Letting users know when data is updatedLarge volume of data being created-sensors, IOT, satellite imagery
Some ways to register for data updates.
Engaging with re-users
Risk of annoying usersDifficult to provide info in engaging ways.
Register for data updatesDynamically scaling cloud storageStorage systems scale dynamically based on real-time user behaviour/data set demand
Implementation Different data types.Different/restrictive IT architectures
Portal to access each department’s data-back end infrastructure maintained by gov and front end integrates with it.
THURS AM Solutions
Finding and accessing data
Single point of access to all datasets
Data quality Metadata must include statements on: accuracy, source, assumptions,Capture data
Suitability Could include suitability statement in metadata.Data quality info in metadata should be comprehensive enough for the user to determine suitability.
Licensing and attribution
Single access portal needs to prominently display licensing and Attribution.Single access portal to allow filtering on licensing restrictions.
Managing data All data hosted on data custodian infrastructure
Engaging with re-users
Analytics of user interactions with the system
Implementation Metadata needs to be populated on all contributing datasets.System integrates with all contributing agencies.
Group 2
THURS AM Challenges Opportunities
Finding and accessing data
Need to know who to contact in organisationLots of data sets: finding what you need in a listsearch
Portals like QSpatial provide a possible method. Integration between sites and access points.
Data quality Maintaining accurate meta data.
Feedback mechanisms for custodians, crowdsourcing.Data ratings.
Suitability Making it suitable for external consumption in addition to current internal processes. Communicating limmitations of use
Community engagement
Licensing and attribution
Complex System or process for simplifying the attributionsCreative comms
Managing data Not knowing when new data is released.
Streamlining
Engaging with re-users
No register of users of files/services
Use of comments/RSS to notify usersVoluntary subscriptions for providersSocial media channels
Implementation Difficult to use the file typesLack of transparency of file/service content to determine suitabilityBroken links/changing dataLimited to specialist software
UsabilityOpen communication/dialogue with communityCelebrating success
THURS AM Solutions
Finding and accessing data
Ranked list of results
Data quality Metadata standardsRating schemeTimes usedReview content
Suitability User-focused data collection workshops
Licensing and attribution Automated aggregation of metadata
Managing data Publishers-integration and federation. Min metadata standards. Users: comms channels
Engaging with re-users
Subscriptions to updates/broadcastsSocial media channels
Implementation Min standards of usability
Group 3
THURS AM Challenges Opportunities
Finding and accessing data
Metadata kept to a standard and used
Searchable via google
Data quality Currency of data-don’t expect too much from free data.
Spend money improving dataEducate on metadata to interpret quality.
Suitability Address via metadata Highlight suitability in metadata tags.
Licensing and attribution
Should this be a concern?Note:Lack of awareness surrounding licensing purposes and usesDerived data setsWorking out most efficient way to transfer data between peopleLiability
Can it be captured in metadata?
Managing data Update QA, QC-who is the custodian?
Becoming custodian
Engaging with re-users
Communicating to anonymous users
Further refine product for re-users.Provide discussion boards for datasets.Tick a box to get updates automatically.
Implementation
THURS AM Solutions
Finding and accessing data
Searchable via google.Use google technology to order search results.
Data quality Metadata used
Suitability
Licensing and attribution Share it like a millennial.
Managing data Must be funded and importance placed on filling in metadata.
Engaging with re-users
Implementation
Group 4
THURS AM Challenges Opportunities
Finding and accessing data
Right to publish as open dataDefining usage
Supporting metadata, data dictionary, governance documentation
Data quality Linear data flow from capture to publishing Perceived benefit to improve quality where it’s not beneficial to data owners
Standards appliesPolicy supporting publishing process and linking to standards
Suitability Self-assessmentFit for purpose
Work flows-publishing
Licensing and attribution
Definition of “open data”Understanding legal requirements/responsibilities for use
Clearly communicate licensing obligations
Managing data Time/costArchitectureLineageRisk assessment-approvals
AutomationGovernment policy and support management activitiesPublish the approval process, involves classification, risk assessment and approvals
Engaging with re-users
What is wantedWhat doesn’t work
Collaboration-forumsSocial mediaNotifications-releases, modifications
Implementation Is it in a suitable format?AccessibleFor application as well as analytics/data crunching
THURS AM Solutions
Finding and accessing data
Central portal-visual for ease of use and standard keywords (ANZLIC)
Data quality Meet quality standard before publication
Suitability Engage with end users at all stages
Licensing and attribution
System of symbology for metadata/dd elementsEncourage/educate about licensing
Managing data
Engaging with re-users
Funding incentiveCrowd funding for improvements, to a central body to then provide Grants.
Implementation Legislation/policy drivers to publish and/or use open data-incentivesand enablement.
Group 5
THURS AM Challenges Opportunities
Finding and accessing data
Private data accessible through gov portals. Eg using drone data and aggregating to single usable format
Proper negotiations around federation of metadata (e.g. private data federating with public portal)Making license restricted data still discoverable (licensing can be negotiated and copyright eventually expires
Data quality $ to address issues outside data ISO19157 Geographic information – data quality
Suitability Users not understanding the metadata elements (e.g. scale
Metadata reader eg. Colour code for accuracy level or data capture method
Licensing and attribution
Appropriate levels of licensing-ignoranceGovernment procurement practices/policies
Learn off spatial efforts-GILF
Managing data Data processing and storage capacity
GA data cube
Engaging with re-users
Trusting users-education Users providing feedback-improves data quality.
Implementation Working around the political panic reaction when something unexpected happensData culture-real metadata
THURS AM Solutions
Finding and accessing data Formalise the federation methods to metadata structures
Data quality Establish ISO 19157 in Govt
Suitability ISO 19195 in Govt
Licensing and attribution Government procurement practices/policies-standard licensing and at a Minimum whole of Gov.Any changes is by description only.
Managing data Open data needs to be a part of any department office.
Engaging with re-users
Informative feedback loop to improve data quality
Implementation Fund it
7.1 Attendees Storytelling Activity: What worked and what didn’t
7.1.1 Storytelling:Group 1: Policy rushed
Had to backtrackGroup 2: Connectivity/interactions Interface between government and community
Lack of knowledge on open dataGroup 3: Usage from client endGroup 4: Data access-how to find what you needGroup 5: Heart Foundation: Report that had admissions data related to particular medical conditions. Data supplied by hospitals and Institute of Health and Welfare.
7.1.2 What worked:Group 1: Nil ResponseGroup 2: Nil ResponseGroup 3: QLD spatial and globe ahead of NSW.Group 4: Good pockets of data but hard to find/no frameworks. QLD Globe AURIN-hosts 1000’s of data sets for research purposes. Can find what you need.Group 5: All came together in the end.
7.1.3 What didn’t: Group 1: Metadata not appropriate/complete. Poor data quality/no filter. Put everything up. Group 2: Inconsistency (better definitions needed). Blockage: when people approach-nothing happens-cycle of what to do next. Inconsistency made interaction between globes (eg. QLD, NSW) difficult. Group 3: Struggle with use of good vs evil. Sensitive data concerns (i.e. terrorism, telecommunication networks). How do we safeguard? Group 4: No framework. Vast majority of systems-hard to use/find. Group 5: Unclear about licensing of data and whether it is open or not (ie. Lots of data but what’s open data?).
7.1.4 Attendees Feedback: What is the most important element/What should go in the Policy?
Metadata standards Culture of metadata Awareness Thorough understanding and agreement throughout government and public about why
gov are using open data Why = it is a valuable asset Informed decisions around anything Removing hindrances to use Success measures Acknowledgment that the nature/value of data changes over time Understanding/appreciation of value & cost of the data Opportunity cost Policy communicated really well (e.g. understood by average person on the street) What’s the value of google maps?
8. Thursday PM Group – Research, Academia and StudentsOpen Data Template Responses: Challenges and Opportunities Activity
Group 1
THURS PM Challenges Opportunities
Finding and accessing data
No standard portal/format i.e. disparate portals
Accepting third-party data
Data quality Trust-Multipart Raw Data-user
Suitability Purpose of publication of open data needs to be clearly defined
Feedback, Open forum
Licensing and attribution
Common attribution CC.
Managing data Manage-Damager?? Common
Engaging with re-users
Commonality Participatory
Implementation access Build from existing systems (e.g. AAF) Non-profit, research facilitation.
THURS PM Solutions
Finding and accessing data
No Response
Data quality
Suitability
Licensing and attribution
Managing data
Engaging with re-users
Implementation
Group 2
THURS PM Challenges Opportunities
Finding and accessing data
Time required to find dataAccess due to privacy concernsfear
Collaboration with producers to make data available.
Data quality Coarseness of data-aggregation rounding, formats, metadata, state of dynamic, spatial
Standards and semanticsQuality assuranceData measures
Suitability User engagement /feedback
Licensing and attribution
People don’t do it Simple licensing
Managing data Keeping data up to date and findable
Improves API, search engines, data analytics to find usage patterns
Engaging with re-users
Lack of industry pull User engagement/feedback
Implementation $ Existing infrastructure
THURS PM Solutions
Finding and accessing data
Semantic technology, pattern based, data indexing, visualisation, no SQL, location.
Data quality
Suitability
Licensing and attribution
Managing data Analysis of interactive logs, cloud data and infrastructure.
Engaging with re-users
Implementation
Group 3
THURS PM Challenges Opportunities
Finding and accessing data
Knowing what’s out thereSearchabilityAvailability and access to data-in real time (not by request)Customised search (analytics)
App to make searching easierSearch interface and decision tool to “pull at relevant data”Customised data execution tool
Data quality Metadata-detailed descriptions-deficiencies noted.ConfidencePrecedence in the data (how it’s collected and described)Openness of data (temporal/spatial)
Standardised way for data to be described (percentage)
Suitability Who should decide what data is suitable for release?How is it going to be used?Standards of data-can it be regulated by format, fields etc.
Define the scope of open data through policyTodd set a standard and quality of data in various formats
Licensing and attribution
CC-BY 40 as standard fits outputsCCO suitability?? Public??Data sharing policy aligning with licenses which are actually on the data sets
To set the standardAligning policy and licensing and education.
Managing data
Engaging with re-users
Implementation
THURS PM Solutions
Finding and accessing data
Standard format in which data needs to be input (metadata description,Abstract-providing record, data dictionary). Provide a sample extract.Standard interface which uses a combination of keywords, relevance,and currency (timely)
Data quality Standard format
Suitability The scope of open data to be defined in terms of what expectations are.Standardisation.
Licensing and attribution
A license of at least CC-By 40 on the data
Managing data A federal site to direct users to data sets-with min technical standards toprovide access to the data.
Engaging with re-users
Implementation
Group 4
THURS PM Challenges Opportunities
Finding and accessing data
Determine the level of openness of data varies from one dataset to another.Downloading many files
Batch download of data made easier and more acceptable
Data quality Excuse used by agencies to not release data
Release clean data and describe how it was cleaned
Suitability
Licensing and attribution
Applying AUS-Goal type of licenses uniformly and consistently
Managing data
Engaging with re-users
Implementation Data.gov.au-host local government data or get a metadata feed.
Note on sheet: publishing raw data.
THURS PM Solutions
Finding and accessing data
Indexing other data sets to simulate data in a rich context
Data quality
Suitability
Licensing and attribution
Consistent licensing across data sets. Increase literacy of data publishers on licensing.
Managing data
Engaging with re-users
Link output from re-use back to the source data.Link output from data back to gov policy and actions.
Implementation Co-hosting and catalogue.Consider metadata framework.Get state archives involved.Leadership from the top.Change perception of data as an administrative function to a core functionof each department.
8.1 Attendees Storytelling Activity: What worked and what didn’t
8.1.1 Storytelling:Group 1: ResearchGroup 2: Access to coarse grain dataGroup 3: Access difficult. Coarse grain data difficult to applyGroup 4: Environmental data in fine grain form
8.1.2 What worked:Group 1: If they had confidence in the dataGroup 2: Coarse grain dataGroup 3: Web scrapingGroup 4: Getting access (going through custodian). Accessing data on site, before downloading. One single view of a greater amount of info (QLD Open Data)
8.1.3 What didn’t:Group 1: Lack of accessibilityGroup 2: No fine grain data. No standardised approach to releasing data. Hard to combine. Limited data preparation and reusability. Limited awareness about restricted data sets.Group 3: Difficulty accessing. Purpose of publishing unclear. Anonymised, coarse grain, aggregation-make data difficult to use.
Group 4: In fine grain form but limited file size (too much manual handling required to manage data). Aggregation loses individual meaning of data. Finding and accessing. Having to download prior to picking out important bits of data.
8.1.4 Attendees Feedback: What is the most important element/What should go in the Policy? being clear about what the policy is standardised format being clear about data quality policy not existing in a vacuum (aligning with national and international standards, to
retain value) different phrasing of the policy to be delivered to different audiences Access to the data is more important than what department is providing the data ‘one-stop-shop’ concept balancing personal data protection with the need to get data out to the public
9. Social mapping – Social Pinpoint ResultsA full list of comments is included in Attachment C
Word Cloud
Appendices
Appendix ADeep Dive Sessions Raw Data Results-Live Polling from all four sessions. Plus PDFs of Live Polling results
Appendix BOnline Comments from Online Forum
Appendix COnline Comments from Social Mapping
Appendix D