Responsible AI in Consumer Enterprise · artificial intelligence. Responsible AI in Consumer...

Responsible AI in Consumer EnterpriseA framework to help organizations operationalize ethics, privacy, and security as they apply machine learning and artificial intelligence

Responsible AI in Consumer Enterprise | 2

Acknowledgements

This framework has benefited from input and feedback from a diverse group of enterprise executives, product designers, machine learning scientists, security engineers, lawyers, academics, and community advocates. The first draft was released for peer review at the 2018 RightsCon conference in Toronto.

We’d like to extend a special thanks to the following individuals and institutions for supporting our efforts and providing helpful comments to make the framework as useful and accurate as possible:

ALTIMETER INSTITUTE

Susan Etlinger

CORNELL TECH

Helen Nissenbaum

CORUS MEDIA

Anne Harrop

Jane Harrison

GEORGIAN PARTNERS

Jason Brenier

Nick Chen

Parinaz Sobhani

Jon Prial

GOOGLE

Kevin Swersky

INFORMATION AND PRIVACY COMMISSIONER OF ONTARIO

David Weinkauf

LOBLAW DIGITAL

Richard Downe

MCCARTHY TÉTRAULT LLP

Adam Goldenberg

Carole Piovesan

MICROSOFT

Andree Gagnon

OSLER, HOSKIN & HARCOURT LLP

Adam Kardash

Patricia Kosseim

PRIVACY BY DESIGN CENTRE OF EXCELLENCE, RYERSON UNIVERSITY

Ann Cavoukian

RIGHTSCON

Brett Solomon

Melissa Kim

SCOTIABANK

Mike Henry

Daniel Moore

Michael Zerbs

Dubie Cunninghman

Veeru Ramaswamy

TELUS

Pam Snively

Elena Novas

THE ENGINE ROOM

Alix Dunn

VECTOR INSTITUTE AT THE UNIVERSITY OF TORONTO

Richard Zemel

Cameron Schuler

Frank Rudzicz

Marc-Etienne Brunet

Elliot Creager

Jesse Bettencourt

Will Grathwohl


Data has become a business-critical asset, and organizations across

all sectors are recharacterizing themselves as “data companies.”

There is an infinite opportunity for organizations to effectively

leverage and unlock the value inherent in their data repositories.

Companies that deploy artificial intelligence to derive meaningful

insights from their data holdings will be the successful innovators

of tomorrow. But to achieve true success, organizations must

implement the guardrails needed for responsible data use, as the

long-term sustainability of any enterprise is predicated on trust.

For data companies, the respectful and ethical treatment of data

has become a core feature of any trust model.

ADAM KARDASH

Chair, Privacy and Data

Management, Co-Leader of

AccessPrivacy, by Osler

PATRICIA KOSSEIM

Counsel, Privacy and Data

Management, Co-Leader of

AccessPrivacy, by Osler

The concept of data ethics is still in its formative stages and requires active, informed, and multi-stakeholder discussion. Integrate.ai should be commended for developing this framework, which will help facilitate a structured conversation about the ethical considerations and broader economic and social impacts of AI data initiatives.

Foreword


The field has been around for a long time, but a phase shift has occurred over the past five years thanks to faster computation, smarter algorithms, and, most importantly, exponential growth in data.

The subfield of AI having the greatest impact in the enterprise is machine learning, software systems that learn from data and experience. As Amazon CEO Jeff Bezos said in his 2017 letter to shareholders: “Over the past decades computers have broadly automated tasks that programmers could describe with clear rules and algorithms. Modern machine learning techniques now allow us to do the same for tasks where describing the precise rules is much harder.”

But machine learning does more than automate existing businesses processes: It changes how businesses form and strengthen relationships with customers. Using data and machine learning, businesses can turn every interaction into an opportunity to learn what people want and value. At a macro level, machine learning can optimize margins, directing spend and human resources to those customers where outreach and engagement would lead to the highest return.

There are risks. AI requires that enterprises use customer data in new ways, expanding responsibilities to customers to include appropriate data use. People feel shocked when they learn that their sensitive information was leaked. Suspicious when they sense businesses want to manipulate their behavior. Powerless when an automated system denies them a product without any explanation for why. Trust is not a constant: it is earned over years and lost in an instant.

Executive leadership is ultimately responsible for striking the right balance between business risk (both legal or reputational) and opportunity. Leaders need a clear mental model for what AI can and cannot do and a means to effectively arbitrate between business and

risk stakeholders to make the right decisions for the business. This is increasingly difficult in a world where major technology advances like AI challenge existing decision-making models.

This framework presents the privacy, security, and ethics choices businesses face when using machine learning on consumer data. It breaks things down into the various small decisions teams need to make when building a machine learning system. It is an agile approach to ethics and risk management that aligns with agile software development practices. Businesses waste time if governance or ethics reviews start after systems are built. When done well, accountability quickens rather than slows innovation: business and risk teams need to make contextual calls about what constraints are required when and clearly define desired business outcomes. The scientists’ job is to apply the best algorithms to optimize for these goals. There’s no silver bullet. Contextual judgment calls early on can move mountains.

This framework is neither a regulatory compliance compendium nor an exhaustive list of risk management controls. It is a tool to help businesses applying AI to think about ethics and risk contextually. It provides detailed insights for implementation teams and high-level questions for executive leadership.

Expanding governance to include ethics can change employee mindsets towards governance and compliance. Ethics ignite values and empathy, the things that make us human and motivate us to do good work. Sustainable innovation means incentivizing risk professionals to act for quick business wins and showing business leaders why fairness and transparency are good for business. Building for accountability will force cross-functional teams to empathize with one another and communicate better. This alone will be a win.

Artificial intelligence (AI) may be the biggest and most disruptive technology advance we see in our lifetimes.

Executive Summary


Operationalizing ethics starts with breaking down how machine learning systems are built and how they work. The

framework uses the different steps in the system-development process as its organizing principle and localizes ethics

and governance questions so they can be addressed quickly and in parallel with agile development.

Deciding to cut a project early on because it is too risky or poses ethical concerns frees teams up to focus on other

things (and practice their values in the process). Business, risk, science, and technical teams need to communicate

continuously to ensure scientists optimize for the right set of constraints and goals and business teams understand

what’s possible and what’s not possible. Doing ethics up front can open up the creative potential of your business.

Guiding Principles

The framework starts with our guiding principles, the

intuitions everyone in your business, including executive

management, should internalize to inform risk-based

thinking and ethical decisions.

People, Processes, and Communication

Next come dos and don’ts about people, processes, and

communication recommended to make ethics efforts

successful. Use this as a checklist to think about your

team and organizational structure.

How Machine Learning Systems Work

After is an overview of how machine learning systems

work and some common machine learning applications

in consumer enterprise. Use this like a glossary to align

on definitions and level set expectations.

Framework Summary

A framework summary follows, breaking down

the different steps in the machine learning system

development process and indicating the jobs to be

done and ethical questions to be considered at each

step.1 Rely on this table as your legend, map, and guide.

Some readers may only ever use this table.

Context at Each Phase

The body of the document provides further questions

and additional context at each phase in the machine

learning system development process. Privacy, security,

fairness, explainability, and transparency issues are

considered at each phase, including anecdotes and

examples. The framework is systematic, but if a given

category (e.g., security) is not relevant at a given phase, it

is left out. Comprehensive guidance on security, privacy,

compliance, or legal risk management issues are out of

scope, but footnotes include references to additional

resources. Use this to think deeper about a particular

topic and to guide questions and decisions.

This framework is designed to operationalize ethics in machine learning systems.

1 This is a high-level outline designed to focus attention on ethics and risk. For further information about machine learning workflows, we recommend the Georgian Partners Principles of Applied Artificial Intelligence white paper and the O’Reilly Development Workflows for Data Scientists eBook.

The content in this document does not constitute legal advice. Readers seeking to comply with privacy or data protection regulations are advised to seek specific legal advice from lawyers.

How to Use This Framework


This framework is not a list of procedures and controls. It’s a tool to think critically about privacy, security, and ethics.

It will help you ask the right questions to the right people at the right time, but you will have to assess risks and make

tradeoffs. It’s helpful for everyone in the business to share the same intuitions around what matters.

Responsible AI Principles:• In standard practice, machine learning assumes the future will look like the past. When the past is unfair or biased,

machine learning will propagate these biases and enhance them through feedback loops. If you want the future

to look different from the past, you need to design systems with that in mind. You can’t just let the data guide you.

Executive leadership should decide what future outcomes the business wants to achieve. These include fairness,

not just profit.

• The outcomes businesses want to optimize for are often hard to measure or occur far in the future (e.g., customer

lifetime value). Businesses therefore resort to easier-to-measure proxies that stand in for desired outcomes. Be

clear about what these proxies do and don’t optimize. You may learn they exacerbate bias or have downstream

consequences that conflict with values or goals.

• All of your customers are individuals. Representing them as data points necessarily transforms them from people

into abstractions. When you deal with abstractions and groupings, you run the risk of treating humans unethically.

• Beware of correlations that mask sensitive data behind benign proxies. For example, postal code/zip code is often

a proxy for ethnic background. If your machine learning system uses location in decisions, you may end up treating

different ethnic groups differently.

• Context is key for explainability and transparency. Systems that decide who gets a credit card or loan require

more scrutiny than systems that personalize marketing offers. Business and risk teams should assess context and

communicate required constraints to technology teams.

• Privacy is not just about personal data, notices or consent forms, or a set of controls to minimize data use. It is

about appropriate data flows that conform to social norms and expectations. Map these flows and ask if people

would be surprised to learn how their data has been used.

• Accountability is a marathon, not a sprint. Once in production, machine learning systems often make errors on

populations that are less well represented in training data. Develop a plan to catch and fix these errors. “Govern

the optimizations. Patrol the results.”2

• There is no silver bullet to responsible AI. It takes critical thinking and teamwork. Step outside the walls of your

organization and ask communities and customers what matters to them.

2 Weinberger, David. “Optimization over Explanation: Maximizing the benefits of machine learning without sacrificing its intelligence.” https://medium.com/berkman-klein-center/optimization-over-explanation-41ecb135763d

Guiding Principles


Enterprises don’t change overnight. The only way to build an ethics muscle is to execute a project large enough to

matter (not a research project) and small enough to permit learning without disruption (not a two-year data lake

migration). Practice will reveal what works and what doesn’t work for your culture.

Here are some dos and don’ts for people, processes, and communication to execute ethics in practice:

People, Processes, and Communication

DO

Business and risk teams learn-by-doing through work on a machine learning project

Put a privacy stakeholder on a machine learning project working closely with development teams. Seek vendors or consultants who will push your team to learn and grow their skills.

Engage cross-functional teams in ethics decisions throughout system development

Business goals and ethics checks should guide technical choices; technical feasibility should influence scope and priorities; executives should set the right incentives and arbitrate stalemates.

Use risk-based, contextual thinking to evaluate and prioritize technical controls

You can’t do everything at once. Prioritize time and efforts based on how significant consequences to your business and your customers would be if something went wrong.

Include diverse and external points of view in ethics decisions

You minimize risk and find new opportunities by engaging customers and communities who will be affected by machine learning models.

Insist that technical stakeholders communicate risks in plain language

Not everyone will be a scientist or security expert. Clear communication is required for critical thinking and judgment calls to take place.

DON’T

Wait to recruit talent with expertise in ethics and risk related to machine learning

There are fewer business professionals with expertise in AI than highly-coveted technical researchers. Delaying ethics until you find the right talent will stall innovation.

Delegate an ethics review to a fixed part of the organization at the end of a project

Engaging unbiased reviewers without conflicts of interest is a good idea, but for ethics to live and breathe it must be an ongoing priority for the teams that create the systems as well.

Use one-size-fits-all checklists or rely on vague policies or principles

Privacy and ethics risks vary between applications. Granular analysis and critical thinking are required for ethics in practice.

Appoint an ethics committee that only includes executive leadership

This removes accountability from teams. Leadership can arbitrate decisions and set policies, but decentralized accountability is key.

Grant functional teams the authority to stop progress without explaining why

With ambiguity, it’s easier to say no than yes and use jargon to cover uncertainty. Risks should be tied back to business goals.


Agile ethics is a process to operationalise values, iteratively

identify and address ethical challenges of your innovations

before you send them out in to the world, and adaptively

refine processes so that as the capabilities of technology

evolve, so does your ability to diagnose and prevent harm

before it happens.

Agile ethics is the explicit application of agile methods to ethical

assessment, adaptation, and learning that allows for a team to

mature its practices as it works at the bleeding edge. It employs

agile methods to tackle ethical challenges, and inculcates ethical

approaches within an agile development process.

It is not:

• Delegating ethical analysis to a fixed part of an organisation (compliance, research, corporate social responsibility, or otherwise)

• Prescribing a fixed approach to ethics based on vague values (policies won’t cut it)

• Encouraging your staff to reflect on ethics without designing methods or allocating resources to aid them in the process

• A bandaid for underlying problems with governance and leadership

Agile ethics requires four things:

• Decentralisation of critical, ethical thinking in an organisation or team

• Iterative development of process that supports decentralised consideration of the implications of any given idea

• Dedicated staff time to manage the process

• Inclusion of diverse and (when appropriate) external voices

ALIX DUNN

Executive Director, The Engine Room

The Engine Room is a UK-based firm that helps activists, organisations, and other social change agents make the most of data and technology.


How Machine Learning Systems WorkBefore you start addressing the ethics and risks of machine

learning, it helps if everyone shares a common understanding

of what machine learning systems do and how they work.

This doesn’t mean that everyone needs to become a machine

learning scientist and grasp the nuances of different algorithms.

They just need some grounded intuitions to ask good questions.

Machine learning systems create useful mappings between

inputs and outputs. The mappings, called models, are

mathematical functions, equations of the form y = mx + b,

where x is an input and y is an output (just that the equations

can be much more complicated!). You could use hand-written

rules to define those mappings, but rules take a lot of time to

write and usually don’t handle a lot of cases.3 With machine

learning, computer programmers no longer write and update

the mappings between inputs and outputs: computers learn

these mappings from data. So, in y = mx + b, the computer

learns what value “m” and “b” should be after seeing lots of x’s

and y’s. When the system is presented with new inputs it hasn’t

seen before, it uses the mappings it’s learned to make a useful

guess about the corresponding output. These mappings aren’t

certain, and they don’t always generalize perfectly to new data.

Most machine learning applications boil down to making a

prediction about the future (How likely is it that this individual

will become a profitable customer?) or classifying data

into useful categories (Is this email spam? Is this cell phone

stationary or in transit?). Emerging systems that do things like

schedule meetings, make phone calls, or write emails on our

behalf can output a range of possible outputs rather than just

an output with a clear right or wrong answer (like correctly

saying what object is in an image) or a strict binary decision

(Will this customer churn or not?).

Common machine learning applications in consumer enterprise

Recommendation Systems Compare actions of consumers to infer similar taste or suggest affinity between consumers and products based on attributes and actions

Audience SegmentationSeparate consumers into groups that look like one another in a way that is relevant for marketing or product performance

PersonalizationModify the experience of a product, marketing message, or channel to best resonate with a consumer at a scale too large for human teams to execute

ChatbotsHelp customers answer questions, resolve problems, or identify the right product mix to redirect human resources to higher-value interactions that require judgment

Risk AssessmentsModify offer and pricing on an insurance or banking product according to predicted risk or likelihood to default

Anomaly DetectionIdentify a shift in customer behavior that could signal opportunity for upsell or risk of churn, or a shift in network or system behavior that could signal malicious activity

Anti-money Laundering and ComplianceIdentify suspicious behavior or attributes and automate compliance reporting workflows using natural language generation

Data ProductsUse algorithms to identify useful insights about consumer behavior that are packaged and sold to other businesses for targeted marketing

3 The game of Go, for example, has more than 10⁸⁰ possible game outcomes. That is more possible games than there are atoms in the universe! It would take a prohibitively long amount of time for a computer programmer to encode all the possible combinations by hand; machine learning systems can learn useful game strategies from data of past human players (or, in the recent AlphaGo Zero system, through iterative self-play).


When machine learning is incorporated into a business process, businesses must design how to transform a model output into an action. Feedback loops happen when businesses keep track of the difference between expected versus actual outcomes and use this difference to improve prediction accuracy over time.

Consider the following example. Kanetix.ca, an online insurance aggregator, uses the Integrate.ai platform to guess how likely someone is to purchase an insurance product and then surface incentives in real time to customers who could use a nudge. Their business goal is to focus marketing spend where it will have the most impact: that is, on customers who are decently likely to buy but who aren’t entirely sure.

The input is information about a customer, web behavior, and information the customer enters into forms (like household size, age, kind of car, etc.). The input is sent to the machine learning model, which has been trained on historical data about customer purchases, defining mappings between customer attributes and purchases. The model outputs a score—a number in the range of 0 to 1—of the customer’s likelihood to convert. This score is then resolved into an action: do we surface an incentive or not? The system keeps track of whether the expected outcome (guess that the person will convert) materialized as the actual outcome (data that the person converted) and updates the model with this new information.

Ethics and risk questions arise across the machine learning system workflow. Say your system automates decisions

on granting people housing loans. Does your historical data create a mapping that frequently denies loans to black

people? Could a malicious attacker reverse engineer your model to access sensitive personal data? Can you explain

why you denied someone a loan? If the score changes over time, can you reconstruct old mappings that have been

updated and replaced?

The rest of this framework examines these questions in detail, breaking them down according to the different tasks that

go into building a machine learning system.

Incentives are costly: Optimizing their eectiveness focuses spend on those individuals for whom an incentive will change behavior

The model is retrained based on results

Prediction Did the actual outcome meet the expected

outcome?

BUILD THE MODEL

1

TRANSLATE PREDICTION INTO ACTION

2

PROGRAM RULES INTO API

3

SCORE & LEARN

4

Conversion likelihood.75

Give the individual an incentive2

Do not give the individual an incentive1

Personalized web interface


The Responsible AI Framework


ML system development process

Feedback loop:the model improves once it is in production

Framework SummaryThe responsible AI framework breaks down the steps used to build a machine learning system and highlights privacy,

security, and ethics questions teams should consider at each step. Inspired by Privacy by Design, it is characterized by

proactive rather than reactive measures to privacy and ethics, and embeds critical thinking and controls into the design

and architecture of machine learning systems. View this as an agile process with multiple iterations and decision points,

not a waterfall process that plans everything in advance. You may discover you want to cut a project because you lack

sufficient training data, require greater certainty to foster adoption, or have identified ethical concerns. Learn that

quickly and free up resources to do something else. Remember that cross-functional teams should participate in most

meetings (or at least have regular check-ins) throughout the process, in particular during the scoping phase.


1 Problem Definition & Scope

• Map current business process

• Identify where machine learning system adds value or alters process

• Define inputs, outputs, and what you are optimizing for

• Measure baseline performance and expected lift

• Analyze a user flow to understand how data is collected and where users hesitate on what to input

• Decide whether this will be a fully-automated or human-in-the-loop system

• Interview users and apply human-centric principles to understand their experience

• Design how model outputs will translate into insight or action for internal users/external consumers

• Conduct a data census to identify what data you have and what data you need

• Procure second- and third-party data sets

• Align machine learning training needs with data retention schedule

• Format and process the data to prepare it for machine learning algorithms

• Pair subject matter experts with scientists to help understand data and features that matter for predictions

• Experiment with various algorithms to verify the problem can be solved and select the approach that performs best

• Test model performance on reserved test data set to verify functionality beyond training set

• Integrate model outputs into business process

• Capture data on outcomes and provide feedback back to the system

• Define model retraining frequency (batch or real-time) and how scientists evaluate future model changes

• Monitor system for failures or bugs and update code regularly

• Measure and report on results

• How could your system negatively impact individuals? Who is most vulnerable and why?

• How much error in predictions can your business accept for this use case?

• Will you need to explain which input factors had the greatest influence on outputs?

• Do you need personally identifiable information (PII) or can you provide group-level insights?

• How can you make data collection procedures transparent to consumers?

• Will the formats you use to collect data alienate anyone?

• How will you enable end users to control use of their data?

• Should you make it clear to users when they engage with a system and not a human?

• How will you manage the provenance of third-party data?

• Who are the underrepresented minorities in your data set?

• If a vendor processes your data, have you ensured it has appropriate security controls?

• Have you de-identified your data and taken measures to reduce the probability of re-identification?

• Will socially sensitive features like gender or ethnic background influence outputs?

• Are seemingly harmless features like location hiding proxies for socially sensitive features?

• Does your use case require a more interpretable algorithm?

• Should you be optimizing for a different outcome than accuracy to make your outcomes fairer?

• Is it possible that a malicious actor has compromised training data and created misleading results?

• Can a malicious actor infer information about individuals from your system?

• Are you able to identify anomalous activity on your system that might indicate a security breach?

• Do you have a plan to monitor for poor performance on individuals or subgroups?

• Do you have a plan to log and store historical predictions if a consumer requests access in the future?

• Have you documented model retraining cycles and can you confirm that a subject’s data has been removed from models?

Data Collection &

Retention

Data Processing

Model Prototyping &

QA Testing

Deployment, Monitoring & Maintenance

Design2

3

4

5

6

Step Jobs to Be Done Risk & Ethics Questions


Like any initiative, machine learning projects start with ideation and project evaluation, including assessments of

technical feasibility, scope, desired outcomes, and projected return on investment. Don’t underestimate the importance

of this work: there’s a fallacy in thinking that being data-driven starts with finding insights in data. It starts with the

thinking that goes into defining a rigorous hypothesis that can be explored using mathematical models. Subject matter

experts bring valuable information to the table and can import what they know into system and process design to get

to results faster. Machine learning systems are tools to optimize against a set of defined outcomes; it’s up to humans to

define which outcomes to optimize for.

While an AI ethics assessment may seem like an entirely new process, ethics simply refers to norms of

behavior within a product or service. As a result, AI ethics assessments should focus on the implications of

machine learning on decision making, KPIs, transparency, trust and ultimately the customer experience as

a whole.

Many enterprise machine learning applications will not raise new privacy issues (e.g., automating contract

due diligence with natural language processing). Applications that collect data directly from consumers

should be subject to a privacy review. Technical stakeholders should opine on whether the system needs

granular, personally identifiable information (PII) to function optimally, and how system performance

would be impacted if PII were replaced with aggregates. For example, this might entail a tradeoff

between offers that are personalized to each individual versus offers tailored to consumer segments that

share common attributes.

Conduct a privacy impact assessment (PIA) when you start a project to align on what’s at stake.4 You may

need to revise your PIA template to include risks related to inferred traits about individuals, not just PII

(see section on data processing). Take a risk-based approach to managing PIAs with third-party vendors,

focusing more rigorous review on vendors with higher business risk.

Problem Definition & Scope

PRIVACY

4 Multiple privacy regulators have resources and guidance around privacy impact assessments. We recommend Canadian companies start with resources from the Office of the Privacy Commissioner of Canada: https://www.priv.gc.ca/en/privacy-topics/privacy-impact-assessments.

SUSAN ETLINGER

Industry Analyst, Altimeter

1


Prob

lem

Defi

nitio

n &

Sco

pe

Data

Col

lect

ion

& R

eten

tion

Mod

el P

roto

typin

g &

QA

Tes

ting

Des

ign

Data

Pro

cess

ing

Dep

loym

ent,

Mon

itorin

g &

Main

tena

nce

Prob

lem

Defi

nitio

n &

Sco

pe

When scoping your use case and beginning to design your system, apply the principles of Privacy by Design, a framework developed by Dr. Ann Cavoukian and recognized as an international standard operating in 40 languages. Privacy and Data Protection by Design are the underpinnings of new regulations like GDPR.

Principle 1: Proactive not reactive: preventative not remedial

The Privacy by Design (PbD) framework is characterized by the taking of proactive rather than reactive

measures. It anticipates the risks and prevents privacy invasive events before they occur. PbD does not

wait for privacy risks to materialize, nor does it offer remedies for resolving privacy infractions once

they have occurred—it aims to identify the risks and prevent the harms from arising. In short, Privacy

by Design comes before the fact, not after.

Principle 2: Privacy as the default setting

We can all be certain of one thing—the default rules! Privacy by Design seeks to deliver the maximum

degree of privacy by ensuring that personal data are automatically protected as the default in any given

IT system or business practice. If an individual does nothing, their privacy still remains intact. No action

is required on the part of the individual in order to protect their privacy—it is already built into the

system, by default.

Principle 3: Privacy embedded into design

Privacy measures are embedded into the design and architecture of IT systems and business practices.

These are not bolted on as add-ons, after the fact. The result is that privacy becomes an essential

component of the core functionality being delivered. Privacy is thus integral to the system, without

diminishing functionality.

Principle 4: Full functionality: positive-sum, not zero-sum

Privacy by Design seeks to accommodate all legitimate interests and objectives in a positive-sum “win-

win” manner, not through the dated, zero-sum (either/or) approach, where unnecessary trade-offs

are made. Privacy by Design avoids the pretense of false dichotomies, such as privacy vs. security,

demonstrating that it is indeed possible to have both.

Principle 5: End-to-end security: full lifecycle protection

Privacy by Design, having been embedded into the system prior to the first element of information

being collected, extends securely throughout the entire lifecycle of the data involved — strong security

measures are essential to privacy, from start to finish. This ensures that all data are securely collected,

used, retained, and then securely destroyed at the end of the process, in a timely fashion. Thus, Privacy

by Design ensures cradle to grave, secure lifecycle management of information, end-to-end.

PRIVACY


Prob

lem

Defi

nitio

n &

Sco

pe

Data

Col

lect

ion

& R

eten

tion

Mod

el P

roto

typin

g &

QA

Tes

ting

Des

ign

Data

Pro

cess

ing

Dep

loym

ent,

Mon

itorin

g &

Main

tena

nce

Prob

lem

Defi

nitio

n &

Sco

pe

Principle 6: Visibility and transparency: keep it open

Privacy by Design seeks to assure all stakeholders that whatever the business practice or technology

involved, it is in fact, operating according to the stated promises and objectives, subject to independent

verification. The data subject is made fully aware of the personal data being collected, and for what

purpose(s). All the component parts and operations remain visible and transparent, to users and

providers alike. Remember, trust but verify!

Principle 7: Respect for user privacy: keep it user-centric

Above all, Privacy by Design requires architects and operators to keep the interests of the individual

uppermost by offering such measures as strong privacy defaults, appropriate notice, and empowering

user-friendly options. The goal is to ensure user-centred privacy in an increasingly connected world.

Keep it user centric.

Privacy by Design is a framework that restores personal control over one’s data to the individual to

whom the data pertain. There are two essentials to Privacy by Design: It is a model of prevention –

PbD is predicated on proactively embedding privacy-protective measures into the Design of one’s

operations, in an effort to prevent the privacy harms from arising. It also calls for a rejection of zero-

sum, win/lose models: It calls for privacy AND data utility; privacy AND business interests; privacy

AND AI: positive gains must be obtained on both sides: win/win! That is the essence of Privacy by

Design.

ANN CAVOUKIAN

Distinguished Expert-in-Residence, Ryerson University

PRIVACY

SECURITYSecurity is not an absolute. There will always be some risk. The goal is to reduce risk to an acceptable level

to the business and have a plan to contain and mitigate any incidents that occur. A risk-based approach

to security focuses resources on information or technical assets that are critical to the business first and

analyzes threat, consequence, and vulnerability to prioritize efforts. A variety of mathematical models

are available to calculate risk and to illustrate the impact of increasing protective measures on the risk

equation.5 As with PIAs, vendors working with more sensitive data should be subjected to more rigorous

review and standards than those with lower risk and impact to the business.

5 The ISO/IEC 27000 family of standards on information security management systems is a widely-adopted framework for conducting risk assessments and evaluating holistic controls.


Prob

lem

Defi

nitio

n &

Sco

pe

Data

Col

lect

ion

& R

eten

tion

Mod

el P

roto

typin

g &

QA

Tes

ting

Des

ign

Data

Pro

cess

ing

Dep

loym

ent,

Mon

itorin

g &

Main

tena

nce

There are three ethics issues to consider during the scoping phase.

First, give customers a voice when considering impact (versus focusing success metrics on the bottom

line). Be prepared to address what may later be conflicting metrics. For example, a business may deploy

a machine learning tool with the expected goals to increase customer satisfaction and reduce call

time to service more calls. This masks a hidden assumption that customers want short calls. Short and

sweet conversations may appeal more to busy professionals than to older customers who enjoy friendly

conversation. A diverse stakeholder group should participate in a design session prior to launching a

product to consider a broad set of potential outcomes and customer experiences.

Next, evaluate whether the system will “produce legal effects concerning [individuals impacted by the

system] or similarly significantly affect [individuals].”6 The European General Data Protection Regulation

(GDPR), in effect since May 25, 2018, stipulates that, in use cases with significant impact, “data subjects”

have a right to “obtain an explanation of the decision reached by an automated system and to challenge

the decision.”7 These could be things like receiving a line of credit, receiving a home loan, being recruited

for a job, receiving an insurance quote, etc. Transparency around targeted marketing is covered by the

GDPR but does not fall under the same responsibility requirements. Break down your business process

to identify what your model is doing locally. For example, a retail bank’s end-to-end system to sell credit

cards includes some moles that should be explainable (e.g., which factors determine loan eligibility) and

others may not need be (e.g., which individuals are most likely to purchase a product). Breaking things

down like this can help overcome fears about the black box.

6 The General Data Protection Regulation (GDPR), article 22: https://gdpr-info.eu/art-22-gdpr/. This framework uses GDPR as an example regulation guiding data privacy and data processing. It is the most recent example of legislation on the topic. Citations to GDPR are provided as context to help you shape governance efforts. This framework does not provide legal advice on compliance.

7 GPDR, recital 71 to article 22: https://gdpr-info.eu/recitals/no-71/. Note that this requirement also exists in the European Union Data Protection Directive from 1995.

Prob

lem

Defi

nitio

n &

Sco

pe

Break down explainability into three different levels when evaluating what matters for your business

Explain the intention behind how your system impacts customers

Explain the data sources you use and how you audit outcomes

Explain how inputs in a model lead to outputs in a model

1

2

3

ETHICS


Prob

lem

Defi

nitio

n &

Sco

pe

Data

Col

lect

ion

& R

eten

tion

Mod

el P

roto

typin

g &

QA

Tes

ting

Des

ign

Data

Pro

cess

ing

Dep

loym

ent,

Mon

itorin

g &

Main

tena

nce

Finally, get as clear as possible on what your data and outcomes actually optimize. Quantified machine

learning systems rely on easy-to-measure proxies of complex, real-world events, and ethical pitfalls

arise in the gap between the complexity of real life and the simplifying assumptions a system requires.

Consider the example of the COMPAS recidivism prediction system, intended to guide justice officials

to define an optimal sentence. Users naively interpreted that this system gave them information about

recidivism likelihood as a function of sentence length. But the system actually shows likelihood to be

convicted, given the data it analyzes and the limitations of what it can measure.8 Reframed like this, it’s

evident that the system would expose systematic bias given historic incarceration trends in the United

States.

Clearly identifying what information exists and is lacking from proxy metrics also helps businesses

improve machine learning system performance. The inference gap between a proxy and a real, discrete

outcome signal could equate to millions of lost revenue for the business.

monitori

ng &

main

ten

anc

e

SHORT-TERMProspect conversion

LONG-TERMFrequent use and revenue impact

Feedback loopsOptimization for short-term or long-term outcomes

2018

2050AND BEYOND

8 For a complete review of the system, see https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing. For further analysis of what the system actually optimizes for, see: https://medium.com/@yonatanzunger/asking-the-right-questions-about-ai-7ed2d9820c48.

Prob

lem

Defi

nitio

n &

Sco

pe

ETHICS

Key Questions for Problem Definition & Scope

• Is everyone aligned on what success does and does not look like for the use case?

• How could your system negatively impact individuals? Who is most vulnerable and why?

• How much error in predictions can your business accept for this use case?

• Is it important to explain why a system made a decision about an individual?

• Do you need PII or can you work with aggregate information?

• Have you performed risk-based privacy and security assessments to identify what controls matter most?

• Have you mapped out the end-to-end business process? Are you clear on what the model optimizes?


The focus here is on the system front-end: the tangible interface consumers or internal users touch and use. Machine

learning systems can be completely automated, where a model’s output automatically plugs into a website interface

or app, or have a human in the loop, where the system provides information to an internal user who then uses this

information to help make a decision or sends feedback to help train the system and improve system performance.

Different architectures raise nuanced privacy, security, and ethics issues.

Making privacy transparent to users and providing means for them to control how their data is used is a

critical design question. Designing for consent is not trivial. The legal and academic privacy communities

currently lack consensus regarding the utility of individual consent for personal data: experts like Helen

Nissenbaum (Professor of Information Science, Cornell Tech) and Martin Abrams9 recognize that data

processing has passed beyond the ability of most people to understand and, in turn, provide truly

informed consent on use. The dilemma businesses face cuts into the heart of innovation with machine

learning: should you collect as much as possible to preserve optionality for future innovation or as little

as possible to interpret “minimum use” strictly? Should you manage privacy by explicit permission (you

may do this and only this) or exclusion (you may do anything but that)?

Design

PRIVACY

9 Executive Director of the Information Accountability Foundation, Martin Abrams has recently researched the effectiveness of ethical data assessments on Canadian businesses and upholds the critical importance of neutrality in conducting assessments: http://informationaccountability.org/author/tiaf01/.

API that directly integrates into front-end

API serves model prediction to an internal employee, who provides feedback to help

train the system

Automated architecture

Human-in-the-loop architecture

FRONT-END EXPERIENCE

FRONT-END EXPERIENCE

2


Prob

lem

Defi

nitio

n &

Sco

pe

Data

Col

lect

ion

& R

eten

tion

Mod

el P

roto

typin

g &

QA

Tes

ting

Des

ign

Data

Pro

cess

ing

Dep

loym

ent,

Mon

itorin

g &

Main

tena

nce

As with everything in this framework, there is no silver bullet and the decision is contextual: privacy teams need to analyze elements like what personal information is being collected, with which parties personal information is being shared, for what purposes personal information is collected, used or disclosed, and what the risk of harm and other consequences if misuse were to occur.10

A new challenge machine learning poses is to give users some insight into what companies can infer about them based on their behavior, or what “profiles,” to use the language in GDPR, the company creates about them. Say a user opts out in the future. It’s not enough to remove their data from a customer relationship management (CRM) system or other system of record: some models may need to be retrained to remove inferred profiles about the user also (see the section on deployment, monitoring, and maintenance for further details).

Privacy policies should be easy to find and understand; that means, translated from legalese into words or product features that make this matter. They shouldn’t be tucked away but pushed to consumers at key points of engagement with a system as a gateway to continued use. Policies should be general enough to withstand changes in practice but specific enough to build trust.

The theory of contextual integrity proposes that privacy is about appropriate data flow, flow that

conforms with contextual social norms. It’s not a procedural exercise of providing notice or getting

consent. What kind information is being sent by whom and to whom matters.

PRIVACY

10 The Office of the Privacy Commissioner suggest these as factors for analysis in their 2018 “Guidelines for obtaining meaningful consent”: https://www.priv.gc.ca/en/privacy-topics/collecting-personal-information/consent/gl_omc_201805/#_determining.

HELEN NISSENBAUM

Professor of Information Science, Cornell Tech

The Office of the Privacy Commissioner of Canada suggests the following principles for obtaining meaningful consent:

1. Emphasize key elements when

doing a contextual assessment of

consent requirements

2. Allow individuals to control the

level of detail they get

and when

3. Provide individuals with clear

options to say ‘yes’ or ‘no’

4. Be innovative and creative

5. Consider the consumer’s

perspective

6. Make consent a dynamic and

ongoing process

7. Be accountable: Stand ready to

demonstrate compliance that

is packaged and sold to other

businesses for targeted marketing

Des

ign


Prob

lem

Defi

nitio

n &

Sco

pe

Data

Col

lect

ion

& R

eten

tion

Mod

el P

roto

typin

g &

QA

Tes

ting

Des

ign

Data

Pro

cess

ing

Dep

loym

ent,

Mon

itorin

g &

Main

tena

nce

SECURITY

ETHICSDesign choices on how models structure and collect data can have emotional impact on users and

impact downstream quality.

Data users have to actively input into your system can be collected in drop-down lists, structured fields,

or unstructured fields. Most of the time, designers structure fields to facilitate downstream analysis and

quality. This choice can alienate users as fields reveal implicit cultural assumptions. Consider gender.

Some companies still use male and female; as of 2014, Facebook presented users with 71 gender options.

Individuals who self identify with non-binary categories or resist gender identification will be emotionally

impacted by strict binary choices.11 These representations are only exacerbated when systematized at

scale in products.

As regards data quality, be mindful of questions to which users struggle to provide accurate answers.

Garbage into your system is garbage out of your system. Uncertainty in input data will propagate into

models and impact downstream quality. Nuanced subject questions related to insurance or banking are

examples of information that, when input by a user, will likely be unreliable.

A/B testing can raise ethical questions if it involves users’ emotional response. Consider the 2014

experiment where Facebook tested whether posts with positive or negative sentiment would affect the

happiness levels of 689,003 users (inferred by what they posted). Critics deplored the deliberate attempt

to induce emotional pain for the sake of experimentation.12

Google’s new Duplex system has kindled debates about front-end transparency.13 Duplex makes phone

calls to restaurants and other businesses on an individual’s behalf. The system mimics human speech

patterns naturally enough to fool real humans into thinking it’s a real person (saying things like “Mm-

hmm”). These issues are so new that there is no consensus on when lifelike AI is morally acceptable and

when it’s not. Pragmatists suggest system tweaks to communicate that the system is a machine from the

outset. The flip side of the equation is when users assume a system is an automated AI when in fact

humans are in the loop processing data to inform future system intelligence (as with early versions of the

scheduling agent x.ai or Facebook messenger).14 Consumers may reveal more private information when

they think only an abstract machine is watching.

Security incidents become visible to consumers when services stop working as usual, they are forced

to manage the aftermath of a breach, or malware infects other systems they use. Designers should

collaborate with security early to ensure interfaces for systems deemed high risk from the risk assessment

include best-practice features like two-factor authentication. Anomalous activity that might indicate a

breach should first be analyzed internally and communicated to a user as necessary. To instill additional

trust, products can include features users can consult with health checks on various security measures.

11 https://www.telegraph.co.uk/technology/facebook/10930654/Facebooks-71-gender-options-come-to-UK-users.html12 There are many analyses of the controversy. For example, https://techcrunch.com/2014/06/29/ethics-in-a-data-driven-world/.13 There are many analyses of the controversy. For example, https://www.theguardian.com/technology/2018/may/11/google-duplex-ai-identify-itself-as-robot-during-calls. 14 https://www.theguardian.com/technology/2018/jul/06/artificial-intelligence-ai-humans-bots-tech-companies

Des

ign


Prob

lem

Defi

nitio

n &

Sco

pe

Data

Col

lect

ion

& R

eten

tion

Mod

el P

roto

typin

g &

QA

Tes

ting

Des

ign

Data

Pro

cess

ing

Dep

loym

ent,

Mon

itorin

g &

Main

tena

nce

ETHICSHuman in the loop and decision support

Additional ethical issues should be considered with human-in-the-loop systems.

As with automated systems, designers have to make choices about how to format data collection, be

that in drop-down lists, unstructured feedback, or binary choices like thumbs up/down, to get additional

training data. While these choices architect input from trainers, there is still room for subjective bias to

creep in via choices humans make. Data quality issues can arise from outsourced services, like Mechanical

Turk, where individuals lack subject matter expertise. Finally, trainers risk importing their own biases in

selecting labels and options to train the system. Thought should be put into who is qualified to train a

system and what kinds of biases they should be aware of in providing labels and feedback.

Next, systems can present varying levels of information about their outputs to shape the kind of feedback

provided. One option is to simply present the maximum likely output, i.e., to minimize the information

provided to an internal user. For example, a system could provide a retail bank’s call center agents with a

list of prospects to call without indicating anything about their score. Interfaces can be more informative.

To continue with the example, a system could show a score or provide information about what features

influenced the score (less interpretable models won’t be able to support such clarity, so designers need

to work with scientists to support desired functionality—see the section about model prototyping for

further information). Documentation should be provided to internal users so they understand how to

use the system and what it tells them: remember that machine learning systems output scores based on

the data they are trained on, not absolute probabilities. That someone ranked as 0.78 for propensity to

convert on a scale from 0 to 1 in your system does not mean that they will do what’s predicted 78 percent

of the time. Design choices can help users learn how to interpret model outputs, but education will likely

be required.

Legal liability

Clear legal precedent to define liability for injuries caused by machine learning systems has yet to be

defined. Early analysis, however, predicts that fully automated systems and human-in-the-loop decision

support systems will be subject to different liability analysis.

Suppose a customer has been in some way injured by a fully-automated machine learning system. One

line of reasoning would say that the organization made in breach of a duty of care owed to the person

who has been “injured” by the algorithm’s decision making. What complicates this argument, however,

is that the law of negligence has always assumed human limitations on what is or isn’t “reasonable”:

only “unreasonable” acts or omissions can attract liability. Can an objectively “rational” algorithm be

“unreasonable”? If not, how can an organization that deploys that algorithm be “unreasonable” in so

doing? This paradox suggests that documentation around choices in algorithmic design could become

increasingly important. When working with a software vendor, the organization may then have a claim for

contribution or indemnity against the developer of the algorithm, on a product-liability theory.

Des

ign


Prob

lem

Defi

nitio

n &

Sco

pe

Data

Col

lect

ion

& R

eten

tion

Mod

el P

roto

typin

g &

QA

Tes

ting

Des

ign

Data

Pro

cess

ing

Dep

loym

ent,

Mon

itorin

g &

Main

tena

nce

ETHICSWhere the decision is made by an algorithm with a human-in-the-loop, the liability then sits with the

employee who makes the decision, not the system. The employer may be held vicariously liable on the

basis of respondeat superior—or, what is rather un-woke-ly still known as “master-servant liability”.

Legal theorists have looked to analogize autonomous systems to various areas of law (e.g., servant-

master, animal husbandry, employee-employer, independent person, etc.). These analogies are likely

imperfect given the level of intelligence of the system, the remoteness of human intervention, and the

independence between system/human.15

Engage your legal team when designing a system to address liability.

15 Many thanks to Carole Piovesan and Adam Goldenberg at McCarthy Tétrault LLP for their input on this section.

Key Questions for Design• What type of consent is required for your system given contextual analysis and risk of harm?

• How can you make data collection procedures transparent to consumers?

• Will the formats you use to collect data alienate anyone?

• How will you enable end users to control use of their data?

• Should you make it clear to users when they engage with a system and not a human?

• Are you introducing uncertainty into your system by asking questions that are hard to answer?

• Do internal users understand how to interpret the outputs of your system?

Des

ign

TELUS is passionate about unlocking the promise that responsible AI and data analytics have to offer.

We recognize the enormous social benefit and economic value that machine learning has to offer,

and we’re committed to working with academics, ethicists, data scientists and other thought leaders

to ensure that we can deliver on the promise in a responsible, ethical manner that is respectful of

privacy. To this end, our first priority is to earn and maintain our customers’ and team members’ trust.

We are exploring a variety of techniques and strategies to accomplish this goal, including working

with experts in de-identification to produce useful data sets that cannot be tied back to any individual

person, enhancing our data governance model to enable us to properly identify and assess the

social and economic impacts of any AI initiative, and leveraging our Customers First culture to build

an innovative, agile and, most importantly, responsible AI program. Through responsible AI, we can

make the future friendly.PAMELA SNIVELY

Chief Data & Trust Officer, TELUS


One question you’ll face in applying machine learning is whether you’ll use only first-party data or also include public

or private third-party data in your system. Don’t fall into the trap of viewing this as a PII or no-PII question to satisfy

compliance requirements: as Helen Nissenbaum shows, privacy is contextual, and users get shocked when data you

argue is public shows up in an unexpected context. For example, the Allied Irish Bank recently made headlines for spying

on consumers when they include public social media data in models to determine mortgages.16 Compliance would say

they were onside, but the activities still had reputational risk. It was about appropriate collection and appropriate flow.

Many of the issues attributed to algorithmic bias start with data collection: if you’ve historically engaged with a certain

demographic population, you will have more information about this group than other groups, skewing systems to

perform better on well-represented populations. Solving this starts with the data, not the algorithms. The algorithms

simply learn a mathematical function that does a good job mapping inputs to outputs.

How you collect and store data has privacy implications. Today’s age of cheap data storage and the

internet of things means you can collect massive amounts of information about series of events, be they

what someone posts on Reddit or Twitter, GPS location, internet or set top box viewing data, you name

it. Technical teams can make a choice to either collect all those data points and process them in batches

to train algorithms or treat the data like a stream, only collecting snapshots of trends over time.17 Using

stream techniques, you never capture or store granular data about an individual, only approximations

relevant for machine learning purposes. This lowers model accuracy, but there are techniques to bound

errors to meet requirements of your use case.

Your data governance team has likely viewed data retention as a risk and developed procedures to

delete data over time to protect the business (while respecting legal retention requirements). Machine

learning teams will want historical training data to understand past events for future predictions. For

example, a bank may want to model consumer behavior during a past economic cycle that resembles

current conditions; these needs may conflict with strict retention procedures. Have these discussions

early.

Data Collection & Retention

PRIVACY

16 https://www.independent.ie/business/personal-finance/big-brother-aib-now-spying-on-customers-social-media-accounts-36903323.html17 Example techniques include Bloom filters and cuckoo filters, as explored in this blog post: http://blog.fastforwardlabs.com/2016/11/23/probabilistic-data-structure-showdown-cuckoo.html. For an in-depth technical review, we recommend Micha Gorelick and Ian Ozsvald’s High Performance Python: http://shop.oreilly.com/product/0636920028963.do.

3


Prob

lem

Defi

nitio

n &

Sco

pe

Data

Col

lect

ion

& R

eten

tion

Mod

el P

roto

typin

g &

QA

Tes

ting

Des

ign

Data

Pro

cess

ing

Dep

loym

ent,

Mon

itorin

g &

Main

tena

nce

Next, as the recent Cambridge Analytica scandal showed, there should be clarity about how data

collected directly from a few consenting individuals can be used to make indirect inferences about many

other, non-consenting individuals.18 Cambridge Analytica used connections in Facebook’s networked

graph to make inferences about personality types of 80 million individuals from only 270,000 active

survey participants. Lookalike inferences of this kind are core to many personalization projects: decide

what inferences you’ll allow on unknowing data subjects. Facebook also lacked rigorous methods to

audit and verify that Cambridge Analytica complied with requests to remove information from the

network graph after services were suspended; verification methods should be technical, not good faith.

A final thing to consider is data provenance. Data aggregators collect data from hundreds of sources

and pull them all together to resell demographic information to customers. There are chains of

interdependent liability between all the players in a data supply chain. Review contracts with third-party

vendors and data providers carefully to identify surprising indemnity clauses that may indicate untoward

data collection practices.

PRIVACY

18 https://www.vox.com/policy-and-politics/2018/3/23/17151916/facebook-cambridge-analytica-trump-diagram

Data

Col

lect

ion

& R

eten

tion


Prob

lem

Defi

nitio

n &

Sco

pe

Data

Col

lect

ion

& R

eten

tion

Mod

el P

roto

typin

g &

QA

Tes

ting

Des

ign

Data

Pro

cess

ing

Dep

loym

ent,

Mon

itorin

g &

Main

tena

nce

Modern machine learning toolkits are largely

in the cloud; Amazon Web Services, Microsoft

Azure, and Google Cloud Platform are the three

largest providers. Some enterprises, in particular

in regulated industries, are still hesitant to host

sensitive customer data in the cloud and opt to

build systems internally or work with consultants

that build on-premise systems. This can

negatively impact the business’ ability to scale

and govern machine learning systems.

If you decide to put data in the cloud or work

with a cloud-based software provider, there

are numerous standards to inform vendor risk

management programs, including the ISO/IEC

27018 standard for managing PII in the cloud

or regulations like the United States Health

Information Portability and Accountability Act

(HIPPA) Security Rule, which includes a security

assessment tool largely following ISO/IEC 27001.

Vendors should apply security best practices

internally and constantly educate customers on how

they are strengthening their security posture. We

encourage clients to dive deeper into our controls,

software security, and most importantly, have the

discussions needed to gain and maintain their trust.

That’s being accountable.

SECURITYA few essential security controls to look for in third-party vendors

Encryption

Data should be encrypted at rest

and in transit. Encryption keys should

be updated regularly to ensure that

any vulnerabilities are limited to

what was enciphered during a given

key rotation. Algorithms should be

256 bit or higher. There should be

governance on who can access keys.

Data Access

Data should never end up on the

personal laptops of workstations of

a third-party vendor. Data should

be housed in a clean room and only

accessed on a need-to-know basis

by vendor scientists and developers.

Auditability

The vendor should keep logs of

scientist and engineer access to

computing clusters, databases, and

even rows and fields in databases,

with means to detect anomalies as

needed. Data flows across network

perimeters should be monitored.

Breach notification

The vendor should have processes

to identify a breach, conduct a risk

assessment to understand impact,

and notify impacted parties in

accordance with regulations.

CHRIS NELMS

EVP, Trust & Security, PrecisionLender

Data

Col

lect

ion

& R

eten

tion


Prob

lem

Defi

nitio

n &

Sco

pe

Data

Col

lect

ion

& R

eten

tion

Mod

el P

roto

typin

g &

QA

Tes

ting

Des

ign

Data

Pro

cess

ing

Dep

loym

ent,

Mon

itorin

g &

Main

tena

nce

If a product or service has historically been used by a certain subpopulation, data will be skewed to

accurately represent tastes, attributes, and preferences of this population at the expense of others. In

Automating Inequality, for example, Virginia Eubanks shows how this impacted the performance of an

automated system to predict instances of domestic child abuse. Data came from public health facilities

in the United States, not private facilities. Given the structure of the United States healthcare system, low

income individuals tend to use public facilities while higher income individuals use private facilities. As

such, predictions were skewed towards behavior in lower income families, going on to systematize bias.

Propensities of certain populations to engage with advertising or fill in forms will similarly skew

performance. Analyze if certain communities or populations engage with your business more than

others and work to figure out why.

ETHICS

At Scotiabank, we don’t see data governance as just a compliance exercise or a way to manage

risk. For us, it’s a source of competitive advantage and a way to deliver meaningful value to our

customers, while maintaining their trust. Taking an ethical approach to AI is an essential part of that

work. We see ourselves as custodians of our customers’ data and know that our ability to protect it

is intrinsically linked to the value and promise of our brand.

MIKE HENRY

Chief Data Officer, Scotiabank

Key Questions for Data Collection & Retention• How will you manage the provenance of third-party data?

• Are there any underrepresented minorities in your data set?

• If a vendor processes your data, have you ensured it has appropriate security controls?

• Will your existing data retention schedules and procedures impact model training?

• Do you need to store every data point or is it possible to manage data as a stream?

• Would people be surprised to see their data used in this context?

Data

Col

lect

ion

& R

eten

tion


Data processing is the step where you prepare data for use in algorithms. The core data privacy challenge relates to

protecting privacy beyond PII. Focusing narrowly on PII, fields in databases like first and last names, social insurance

numbers, or email addresses, is not sufficient to guarantee privacy. You have to expand risk to protect the possibility of

a breach even when a data set has been scrubbed of PII. The core ethics issues relate to deciding what types of inferred

features or profiles your organization feels are appropriate and identifying tightly correlated features in data sets that

can hide discriminatory treatment.

Let’s examine both these issues using postal code.

Consider this simplified example. Say your database includes information about an individual’s postal

code and gender, and you combine this with another database that has information about an individual’s

age. You don’t have the name of the individual in either database. Can you identify this individual? With

what likelihood?

As always, it depends. How many people live in the postal code? If it’s a dense urban highrise, there

may be a lot; a rural hamlet, there may be just one person.19 We can continue this kind of analysis on

each variable. Age might depend on the income level typical to a building: a location tailored to young

professionals may have many 35-year olds, whereas a different location may have a more varied age

distribution. A postal code for a retirement home may skew much older. Having birth date versus age

will quickly narrow a set to a few people.

Data Processing

PRIVACY

19 This is why use of postal code is problematic in certain jurisdictions. “The cell size of five rule is the practice of releasing aggregate data about individuals only if the number of individuals counted for each cell of the table is greater than or equal to five.” https://www.ipc.on.ca/wp-content/uploads/2016/08/Deidentification-Guidelines-for-Structured-Data.pdf

INDIVIDUAL RECORD

PRIVACY AGGREGATE

MODELName

Address

Postal code

Email

9wNSY7361nd

8264jSiapq3dn3t07Whs

a87rH3SGaD89qw8

Da63ndS21hHa

CRYP

TOG

RAPH

ICHA

SH

Comparison and feedback lo

op

4


Prob

lem

Defi

nitio

n &

Sco

pe

Data

Col

lect

ion

& R

eten

tion

Mod

el P

roto

typin

g &

QA

Tes

ting

Des

ign

Data

Pro

cess

ing

Dep

loym

ent,

Mon

itorin

g &

Main

tena

nce

Identification risks grow when data is released publicly or third-party data is used to augment first-

party data because third-party data might fill in gaps, leading to an increased ability to reverse

engineer an individual from a group. As such, enterprises should have consistent practices for

sharing data with third parties: if two startups have two different views on people, each of which

are private, but collaborate with one another, they’ll have keys to unlock identity.

This is another area where the current best practice is to think critically and apply a risk-based

approach. The Information and Privacy Commissioner of Ontario recommends the following

process for a risk-based approach to de-identify data:20

PRIVACY

20 The full report includes further guidance on how to implement risk-based de-identification: https://www.ipc.on.ca/wp-content/uploads/2016/08/Deidentification-Guidelines-for-Structured-Data.pdf

Data

Pro

cess

ing1 2

6

7 8 9

3

5 4

Determine the release model

public, semi-public, or nonpublic

Classify variables

direct identifiers and quasi-identifiers that can be used

for re-identification

Determine an acceptable

re-identification risk threshold

impact of invasion of privacy

Calculate the overall risk

data risk x context risk

Measure the context risk

for non-public data, considerthreats and vulnerabilities

Measure the data risk

calculate the probability ofidentification per row

De-identify the data

mask direct identifiers, modify equivalence classes, ensure risk below desired threshold

Assess data utility

consider the impact de-identification will have on

system performance

Document the process

for compliance, trust, and transparency


Prob

lem

Defi

nitio

n &

Sco

pe

Data

Col

lect

ion

& R

eten

tion

Mod

el P

roto

typin

g &

QA

Tes

ting

Des

ign

Data

Pro

cess

ing

Dep

loym

ent,

Mon

itorin

g &

Main

tena

nce

The downside to de-identification is that it is not foolproof: there will be residual re-identification

risk, which is why tolerance needs to be assessed and governed against.

An alternative technique that provides theoretical guarantees is differential privacy, which

modifies the data set in such a way that statistical features that matter for a model are preserved,

but it’s impossible to tell the difference between a distribution that contains and does not contain

an individual. Protections can be added at various points in the machine learning pipeline, with

tradeoffs of model performance and privacy guarantees: as we saw above, the more questions you

ask about an aggregate, the closer you get to an individual. Most differential privacy algorithms

have a “privacy budget,” or number of queries they can support before privacy guarantees

lessen. Product management leaders need to consider these tradeoffs during implementation.

At this time, differential privacy is in production in companies like Google, Facebook, Apple,

and Uber, but has yet to become de facto best practice in startups or the enterprise. It is still

relatively new and difficult to implement effectively. Other privacy techniques include one-way

hash functions, which make a cryptographic mapping of input data that cannot be reversed,

and masking, which removes variables or replaces them with pseudonymous or encrypted

information.

PRIVACY

Data

Pro

cess

ing

Information containing direct and indirect identifiers

Information from which direct identifiers have been eliminatied or transformed, but indirect identifiers remain intact

Direct and known indirect indentifiers have been removed or manipulated to break the linkage to real world identities

Direct and indirect identifiers have been removed or manipulated together with mathematical and technical guarantees to prevent re-identification

DEGREES OF IDENTIFIABILITY

PSEUDONYMOUSDATA

DE-IDENTIFIEDDATA

ANONYMOUSDATA


Prob

lem

Defi

nitio

n &

Sco

pe

Data

Col

lect

ion

& R

eten

tion

Mod

el P

roto

typin

g &

QA

Tes

ting

Des

ign

Data

Pro

cess

ing

Dep

loym

ent,

Mon

itorin

g &

Main

tena

nce

Feature engineering is the process of creating second-order features, or insights, relevant for a

model from raw data. For example, in a model to predict customer churn, a first-order attribute

would be something like gender and a second-order attribute would be something like price

sensitivity, inferred from a sequence of transactions. Each transaction doesn’t have much value,

but the inference drawn from multiple transactions does. Decide what inferences your business

will and will not permit for user segmentation and targeting.

Be mindful of proxy correlations when processing data. Removing a column for gender or

ethnicity won’t guarantee that these factors are now absent from a model, as they can be tightly

correlated to other features. For example, ethnic background is often correlated to postal code

given the tendencies of some ethnic groups to settle in communities with people of similar ethnic

backgrounds.

ETHICS

Data

Pro

cess

ing

Confidential Draft – Do Not Copy, Cite or Redistribute without Permission

Trustworthy AI in Consumer Enterprise 21

>80% White

>80% Hispanic

>80% BlackMajority Hispanic

Majority BlackMajority AsianNo Majority

Majority White

Map of Chicago shows how postal code is often a proxy characteristics for ethnicity

Model Prototyping & Quality AssurancePrivacy Security Ethics

Selecting the best algorithm for system goals and business

More complex models can provide stronger privacy guarantees

Beware of fraudulent training data that compromises system performance

Consider explainability as factor in model selection

Define optimization goals to support future outcomes of ethical policies

Choosing the best model for a particular problem is not only a technical question of identifying the algorithm that performs best for the job. Data and machine learning scientists should also take business, ethical, and regulatory considerations into account to not only select a model that works, but one that the business can put into production. Privacy

MAP OF CHICAGO SHOWS HOW POSTAL CODE IS OFTEN A PROXY

CHARACTERISTIC FOR ETHNICITY


Prob

lem

Defi

nitio

n &

Sco

pe

Data

Col

lect

ion

& R

eten

tion

Mod

el P

roto

typin

g &

QA

Tes

ting

Des

ign

Data

Pro

cess

ing

Dep

loym

ent,

Mon

itorin

g &

Main

tena

nce

Key Questions for Data Processing

• Have you conducted a risk assessment on your data set and made an informed choice on which privacy technique is

best for your use case and maturity level?

• Will socially sensitive features like gender or ethnic background influence outputs?

• Are seemingly harmless features like location hiding proxies for socially sensitive features?

• What psychological or behavioural inferences will your company use or ban for targeting or other predictions?

Data

Pro

cess

ing


In this step, machine learning engineers experiment with different algorithms to find the best algorithm for the job, train

the model, and verify that the chosen model satisfies performance requirements (e.g., how accurate the model needs to

be). Choosing the best model for a particular problem is not only a technical question of identifying the algorithm that

performs best for the job. Data and machine learning scientists should also consider business, ethical, and regulatory

requirements when selecting algorithms.

Sometimes teams turn to synthetic data to train models. The privacy argument is that the synthetic data

can mimic the statistical properties relevant for model performance without using real data that could

compromise privacy. Be careful: expect that the model won’t perform well in the real world—while a

synthetic data set can mimic the statistical properties of interest in a real-world data set, they don’t

overlap exactly, which can create performance issues.

The performance of machine learning systems depends on training data quality. If a malicious actor

compromises training data, they can not only access sensitive information or take down a system, but

lead the system to produce the wrong outputs and behave differently than it was intended to. A benign

example is an application like Spotify changing weekly-recommended songs based on activity from a

different user than normal. A serious example is autonomous vehicles run amok due to a hack into GPS

or visual control systems.

Techniques to hack a machine learning algorithm can be very subtle. Machine learning researcher

Ian Goodfellow has focused on “adversarial examples that directly force models to make erroneous

predictions.”21 An adversarial example is data that is input for a model that has is modified with a small

perturbation imperceptible to the human eye. The algorithm, however, can pick up on the perturbation

and classify it as something else. You think the algorithm is working, but it’s learning the wrong thing.

Audit data scientist workstations for vulnerabilities, standardize tooling across your team, and apply

rules-based access controls to minimize risk.

Model Prototyping & Quality Assurance

PRIVACY

SECURITY

21 See, for example, http://www.cleverhans.io/security/privacy/ml/2017/02/15/why-attacking-machine-learning-is-easier-than-defending-it.html.

5


Prob

lem

Defi

nitio

n &

Sco

pe

Data

Col

lect

ion

& R

eten

tion

Mod

el P

roto

typin

g &

QA

Tes

ting

Des

ign

Data

Pro

cess

ing

Dep

loym

ent,

Mon

itorin

g &

Main

tena

nce

Addressing fairness requires that machine learning engineers make a paradoxical move and optimize

for a different goal than strict accuracy. Recall that accuracy assumes that the future will and should

look like the past; you don’t want to replicate biased historical trends, you need to change what you

optimize for.

Responsible AI in Consumer Enterprise · artificial intelligence. Responsible AI in Consumer...

Documents

Transcript of Responsible AI in Consumer Enterprise · artificial intelligence. Responsible AI in Consumer...