Fairness-Aware Online PersonalizationG Roshan Lal

LinkedIn Corporationrlal@linkedin.com

Sahin Cem Geyik ∗Facebook

scgeyik@fb.com

Krishnaram Kenthapadi ∗Amazon AWS AI

kenthk@amazon.com

ABSTRACTDecision making in crucial applications such as lending, hiring,and college admissions has witnessed increasing use of algorithmicmodels and techniques as a result of con�uence of factors such asubiquitous connectivity, ability to collect, aggregate, and processlarge amounts of �ne-grained data using cloud computing, and easeof access to applying sophisticated machine learning models. Quiteoften, such applications are powered by search and recommen-dation systems, which in turn make use of personalized rankingalgorithms. At the same time, there is increasing awareness aboutthe ethical and legal challenges posed by the use of such data-drivensystems. Researchers and practitioners from di�erent disciplineshave recently highlighted the potential for such systems to discrimi-nate against certain population groups, due to biases in the datasetsutilized for learning their underlying recommendation models.

We present a study of fairness in online personalization settingsinvolving ranking of individuals. Starting from a fair warm-startmachine learned model, we �rst demonstrate that online personal-ization can cause the model to learn to act in an unfair manner, if theuser is biased in his/her responses. For this purpose, we construct astylized model for generating training data with potentially biasedfeatures as well as potentially biased labels, and quantify the extentof bias that is learned by the model when the user responds in abiased manner as in many real-world scenarios. We then formulatethe problem of learning personalized models under fairness con-straints, and present a regularization based approach for mitigatingbiases in machine learning. We demonstrate the e�cacy of ourapproach through extensive simulations with di�erent parametersettings.

Our implementation of data model, online learning, fair regular-ization and relevant simulations can be found here:https://github.com/groshanlal/Fairness-Aware-Online-Personalization

1 INTRODUCTIONAlgorithmic models and techniques are being increasingly usedas part of decision making in crucial applications such as lending,hiring, and college admissions due to factors such as ubiquitousconnectivity, ability to collect, aggregate, and process large amountsof �ne-grained data using cloud computing, and ease of access toapplying sophisticated machine learning models. Quite often, suchapplications are powered by search and recommendation systems.In such applications, users may not always be able to articulate theirinformation need in the form of a well-de�ned query, although they

RecSys 2020, FAccTRec Workshop: Responsible Recommendation, Sept 2020,2020. ACM ISBN 978-x-xxxx-xxxx-x/YY/MM. . . $15.00https://doi.org/10.1145/nn%nnnnn.nnnnnnn∗ Work done while the authors were at LinkedIn.

can usually determine if a displayed result (such as a potential candi-date for hiring or college admission) is relevant to their informationneed or not. Hence, it may be desirable to personalize the results toeach user based on their (explicit or implicit) responses, ideally inan online fashion. At the same time, human responses and decisionscan quite often be in�uenced by conscious or unconscious biases,in addition to the relevancy of the results, causing the personalizedmodels themselves to become biased. In fact, there is increasingawareness about the ethical and legal challenges posed by the useof such data-driven systems. Researchers and practitioners fromdi�erent disciplines have highlighted the potential for such systemsto discriminate against certain population groups, due to biases inthe datasets utilized for learning their underlying recommendationmodels. Several studies have demonstrated that recommendationsand predictions generated by a biased machine learning model canresult in systematic discrimination and reduced visibility for alreadydisadvantaged groups [3, 11, 17, 29]. A key reason is that machinelearned models that are trained on data a�ected by societal biasesmay learn to act in accordance with them.

Motivated by the need for understanding and addressing algo-rithmic bias in personalized ranking mechanisms, we perform astudy of fairness or lack thereof in online personalization settingsinvolving ranking of individuals. Starting from a fair warm-startmachine learned model1, we �rst demonstrate that online person-alization can cause the model to learn to act in an unfair manner, ifthe user is biased in his/her responses. For this purpose, we con-struct a stylized mathematical model for generating training datawith potentially biased features as well as potentially biased labels.We quantify the extent of bias that is learned by the model whenthe user sometimes responds in a biased manner (either intention-ally or unintentionally) as in many real-world settings. We studyboth bias and precision measures under di�erent hyper-parameterchoices for the underlying machine learning model. Our frameworkfor generating training data for simulation purposes could be ofbroader use to evaluate fairness-aware algorithms in other settingsas well. We then formulate the problem of learning personalizedmodels under fairness constraints, and present a regularizationbased approach for mitigating biases in machine learning, speci�-cally for linear models. We demonstrate the e�cacy of our approachthrough extensive simulations with di�erent parameter settings.To summarize, the contributions of our work are as follows:

• A stylized mathematical model for simulating the biasedbehavior of users and generating data that is amenable tolearning a biased model, if labeled by such biased users.

• Demonstration of potential bias in online personalizationsystems for ranking/recommending individuals.

1 We note that although such an unbiased model may not be available in practice, weshow that the personalized model can become highly biased in spite of starting froman unbiased model.

arX

iv:2

007.

1527

0v2

[cs

.AI]

6 S

ep 2

020

• A regularization based strategy to mitigate algorithmicbias during model training.

The rest of the paper is organized as follows. We �rst present amathematical model for studying bias in online personalizationsystems and the associated experimental results (§3). Next, wepropose a regularization based method to mitigate algorithmic biasand demonstrate its e�ectiveness through an empirical study (§4).Finally, we present the related work in §2 and conclude the paperin §5.

2 RELATEDWORKAlgorithmic bias, discrimination, and related topics have been stud-ied across disciplines such as law, policy, and computer science[3, 17]. Two di�erent notions of fairness have been explored inmany recent studies: (1) individual fairness, which requires thatsimilar people be treated similarly [11], and (2) group fairness, whichrequires that the disadvantaged group be treated similarly to theadvantaged group or the entire population [28]. There is extensivework on identifying and measuring the extent of discrimination(e.g., [1, 8, 28]), on mitigation approaches in the form of fairness-aware algorithms (e.g., [7, 9–11, 14, 15, 18–20, 22, 23, 34, 36–38]),and on inherent trade-o�s in achieving di�erent notions of fair-ness [10, 11, 14, 23].

Modifying Model Training Algorithm for Fairness: While severalstudies have focused on data transformation/representation (pre-processing, post-processing, or during training) for ensuring fair-ness (e.g., [12, 13, 24, 38]), there has also been work on modifyingthe model training algorithm itself. Regularization based methodsfor fairness have been proposed in [22, 25, 30]. To meet the inde-pendence constraint of fairness, an approximate estimate of mutualinformation of the classi�er output and the protected attributeis computed and used as a regularizer in [22]. This approach isextended in [25] for continuous protected attributes. Correlationbetween the protected attribute and the classi�er output is furtherused as the regularizer in [30]. While we use a similar correlationbased approach for performing classi�cation, our method di�ersin the following key aspects. We �rst train an auxiliary model forpredicting the protected attribute in terms of other features, andthen guide our main model to be uncorrelated with this auxiliarymodel. Our problem setting is also slightly di�erent. We use onlinelearning wherein the training samples are not picked uniformlyat random, but in a special way where we only show the highestscored candidate to the user for feedback in each iteration. The ideaof auxiliary model has earlier been used in the context of trainingan adversarial neural network in a fair manner [4], but our train-ing method is signi�cantly di�erent. While the auxiliary modelis trained to underperform in [4], we train the auxiliary modelto perform well and use it to steer the main model away from it.Another recent paper [31] penalizes the covariance between theprobability assignment of a user to its correct label, and his/hername’s embedding, without access to protected attributes.

Some recent work [5, 27] have also dealt with achieving fairnessvia regularization utilizing pair-wise comparisons where paireditems are chosen from di�erent protected attribute groups. Both ofthese works in general attempt learn models which have similar

probability to assign a larger score to better items than a worseitem, regardless of the protected attribute.

Finally, a meta-learning approach through regularization is pro-posed in [33], where the authors propose to learn a model thatperforms well and fair across a set of tasks, and use a small sizesample for each task. Within each task they apply a regularizationterm towards getting the fair model which is based on the averageprobability that a class is to be classi�ed as positive given the pro-tected attribute (parity), or given the protected attribute and label(equalized odds or equal opportunity).

Fairness in Ranking: There has been a lot of recent work on fair-ness in ranking [2, 6, 9, 26, 32, 35, 37]. Algorithms for rerankingsubject to fairness constraints, speci�ed as the maximum (or min-imum) number of elements of each class that can appear at anyposition in the ranking, have been proposed in [9, 37]. Fairnessmeasures for ranking have been proposed in [35]. Fairness con-straints are modeled as linear constraints along with a optimizingfor relevance in [9]. Fairness in ranking and feedback from it aremodeled as a stochastic bandit problem in [32]. Achieving fairnessin a dynamic learning to rank setting using a fair controller isdiscussed in [26].

Refer Appendix A for a discussion on legal framework on dis-crimination and relevant notions of fairness popular in machinelearning.

3 STUDY OF FAIRNESS IN ONLINEPERSONALIZATION

We next present a study of fairness in online personalization settingsinvolving ranking of individuals. Starting from a fair warm-startmachine learned model, we �rst demonstrate that online personal-ization can cause the model to learn to act in an unfair manner, ifthe user is biased in his/her responses. For this purpose, we con-struct a stylized model for generating training data with potentiallybiased features as well as potentially biased labels, and quantify theextent of bias that is learned by the model when the user respondsin a biased manner as in many real-world scenarios (§3.1). Wethen present experimental results from our simulation framework,where we consider both bias and precision measures under di�er-ent hyper-parameter choices for the underlying machine learningmodel (§3.2 – §3.4). We chose to use a simulation framework sinceit allows us to model biased users, and further vary their levels ofbias.

3.1 Data ModelWe next present our stylized model for generating training datawhich captures some of the sources of unfairness that is usuallypresent in real-world data, as listed in §A.2. We split our discussioninto two parts. We describe how we generate feature vectors in§3.1.1 and we describe how we further label the data in §3.1.2.

3.1.1 Features. Suppose that our training data consists of n datapoints, where the ith data point xi is a vector consisting ofm seem-ingly harmless attributes: xi = (xi,1 , xi,2 , xi,3 , . . . , xi,m ). Forthe ith data point, let ai denote the corresponding value of theprotected attribute. Of the m attribute values of xi , let the �rst m1attribute values be independent of the protected attribute value ai

2

(ProtectedAttribute)

... ...

Figure 1: Model to simulate feature vectors of training data.

and the remaining m2 attribute values be generated in a fashiondependent on ai (m =m1+m2). We can thus model them attributesof each data point as being generated according to the scheme inFigure 1. As presented in the �gure, for each i , we �rst generatethe protected attribute ai independently using a Bernoulli randomvariable which gives 1 with probability pдroup and 0 with proba-bility 1 − pдroup . We can interpret ai as encoding the membershipto some protected class like belonging to a particular gender, race,or age ≥ 40, with ai = 1 as corresponding to the privileged group.For the other (non-protected) attributes, we choose those that are“harmless” (independent of the protected attribute, i.e., xi,1 throughxi,m1 ) to be drawn either from a uniform distribution between 0and 1, or a normal distribution with their respective means andvariances, independently. For the “proxy attributes” (dependent onthe protected attribute, i.e., xi,m1+1 through xi,m1+m2 ), we drawthem independently from normal distributions, whose means andvariance are functions of the protected attribute value ai . Althoughwe describe the attribute values to be drawn from either a uniformdistribution or a normal distribution for simplicity, we remark thatour model can easily be extended to allow more general distribu-tions, e.g., categorical distributions for categorical attributes.

Current Se�ing: While the above mathematical framework isquite general, in our experiments for the current work, we chosem1 = 1 and m2 = 2 so that m = 3, for simplicity. We thereforeemployed one harmless attribute which we draw uniformly in [0,1],and two proxy attributes which we draw from Normal distributionsdependent on the protected attribute. Finally, we held the varianceconstant for all Normal distributions, although as discussed above,this does not necessarily have to be so.

3.1.2 Labels. As stated earlier, our simulation framework en-ables us to model the behavior of biased users and vary their levels ofbias, which we cannot obtain from real world datasets. Our labelingstrategy for a potentially biased user is presented in Figure 2, and isas follows. Given the feature vector associated with a data point xi ,we need to label xi as either “Accept” (1) or “Reject” (0). First, we de-termine whether the user who labels this data point would behave

Will be biased?

User Model

YES, withprobability

NO, withprobability

Input

label = 1 label = 0 label = 1 label = 0

YES NO YES NO

Figure 2: Model to label simulated feature vectors.

in a biased or unbiased manner, using an independent Bernoullitrial with parameter pbias . This means that the user chooses tobehave in a biased (unfair) manner with probability pbias , and inan unbiased (fair) manner with probability 1 − pbias .

If the user chooses to be fair, he/she takes a �xed linear combina-tion of the attribute values in the feature vector (w0 +

∑mk=1wkxi,k ,

or in short, w0 + wT xi ) and labels the data point according tothe sign of the linear combination (1 if the linear combination isnon-negative, and 0 if it is negative). If the user chooses to be un-fair, he/she labels the data point xi purely based on the protectedattribute value, ai . More precisely, the user labels as “Reject” (0)whenever ai = 0 and “Accept” (1) whenever ai = 1.

The weights [w0,w1, . . . ,wm ] are constants as de�ned by theuser and unknown to the machine learning model which subse-quently uses the data. The user chooses these weights without anyknowledge of how the data is generated and has no knowledgeof which attributes of the feature vector are independent of theprotected attribute. Hence, the way the data is generated, the way itis labeled, and the way it is used for training the machine learningmodel are all blind to each other.

In the above model of simulating feature vectors and labelingthem, we have captured the key aspects of unfairness listed in §A:

• Proxy Attributes: xi,m1+1 , . . . , xi,m1+m2 serve as proxyattributes.

• Quantity of training data for di�erent values of the protectedattribute: This is controlled by the parameter pдroup whilegenerating the feature vectors.

• Quality of training data for di�erent values of the protectedattribute: This pertains to how the proxy attributes aregenerated from the protected attribute. Depending on thefunctions used for mapping the protected attribute valueto the parameters of the distribution used for generatingvalues of the proxy attributes, we may obtain training datawith di�erent quality for di�erent groups. For example, thevariance of the normal distributions used in our settingcould be small for ai = 0 and large for ai = 1.

• Noisy/biased labels in training data: This is controlled by pa-rameter pbias which decides the user’s level of unfairness.When the user behaves in an unfair manner, the resulting

3

labels are biased, with ai = 0 resulting in “Reject” (0) andai = 1 resulting in “Accept” (1).

We remark that our stylized training data generation model could beextended/generalized to incorporate a combination of “objectivity”and (possibly unconscious) biases, which may be closer to be morelikely in practice. For instance, instead of considering the user tobe either fully objective or fully biased when labeling, we couldmodel the user as say, boosting the “objective” score based on theprotected attribute value when determining whether to accept orreject. Another extension to our model would be to allow for theuser to be partially biased: when the user is biased, we can viewthe label assigned by the user as a random variable dependent onthe protected attribute value, and draw the value of the label froma Bernoulli distribution whose parameter could be in�uenced bythe protected attribute value. For simplicity, we limit ourselves tothe stylized model in this paper.

3.2 Experimental SetupNext, we would like to describe our experimental setup to measurehow di�erent choices of hyper-parameters like learning rate andthe number of training rounds a�ect fairness and precision in theoutcomes of the online learning model.

Online Learning: In every iteration, we score all the data pointsin the training data and obtain the next data point for training bychoosing the data point with the maximum score (highest possibilityof being labelled as “Accept” as per the current model) that has notbeen used for training till now. The underlying setting is that wepresent one data point at a time to the user (we select the data pointwith the highest score, amongst those that have not been shownto the user so far), to which the user responds either positively ornegatively. In either case, the online personalization model attemptsto incorporate such user preference, so that a more personalizedresult can be presented next time to the user. We use the featurevectors and label of this new data point for training in the nextround and this is repeated over and over again. The algorithm isstarted using a “warm-start” model in which data from a fair sourceis used to train the models by randomly sampling training datapoints from the fair dataset a number of times. We assume theavailability of such a fair dataset so that we can then compare our�nal biased model against the warm-start baseline.

Our Setup: First, we generate 12000 data samples2 atpдroup = 0.5and label them3 in a fair manner (that is, pbias is chosen to be 0in §3.1.2, so that the labeling is based on the sign of the linearcombination). Our model is a online linear model (perceptron) withthe update equation:

wn+1 = wn + η(yn − sgn(wTn xn ))xn

2 As discussed in §3.1.1, for our experiments, we utilize one “harmless” attribute (xi,1for each xi ), and two “proxy” attributes (xi,2 and xi,3 for each xi ). In our trainingdata generation, for each data point, we choose the second and the third features to bedependent on the protected attribute value, but in opposite directions. For each i , ifai = 0, we draw xi,2 independently from the normal distribution with mean 0.35 andstandard deviation 0.12, and if ai = 1, we draw xi,2 independently from the normaldistribution with mean 0.65 and standard deviation 0.12. The mean values are reversedfor the third attribute xi,3 . xi,1 is drawn from the uniform distribution over [0,1].3 We utilized the following weights for the user label decision model (§3.1.2): w0 =−0.48 (intercept), w1 = 0.35, and w2 = w3 = 0.28. For both feature and labelgeneration, we experimented with di�erent choices of the parameters and observedqualitatively similar results.

where η is the learning rate. We randomly sample 1000 data pointsfrom this set and use these data points to train a perceptron. Wecall this perceptron the “warm-perceptron” from here on. Thedata points that were generated are discarded beyond this point.

Next, we generate 12000 data samples at pдroup = 0.5 and labelthem with the bias parameter set to a speci�c choice of pbias . Westart from the warm-perceptron model. In each round, we scoreall the 12000 data points using perceptron weights. We pick thepoint with highest score (the data point that is farthest from theperceptron on the positive side of the half-plane) that is not yetused for training, obtain its label (following §3.1.2), and then updatethe warm-perceptron with this data point and its label.

Metrics: Our measures for evaluating bias are based on the as-sumption that the distribution of the protected attribute values forthe top ranked candidates (i.e., based on the order with which acandidate gets to be shown to the user in our online personalizationsetting) should ideally re�ect the corresponding distribution overthe set of high quality or relevant candidates.

Our �rst measure, Skewv@k computes the logarithmic ratioof the proportion of candidates having v as the value of the pro-tected attribute among the set of k highest scoring candidates tothe corresponding proportion among the set of relevant candidates:

Skewv@k = loge

(pv@k

pv,quali�ed

)= loge

©­­«∑j 1(aj=v & index(j)<k )

k∑j 1(aj=v & w0+wT x j ≥0)∑

j 1(w0+wT x j ≥0)

ª®®¬ ,(1)

where index(j) denotes the position of candidate j when orderedby decreasing scores, and pv@k and pv,quali�ed denote the fractionof candidates with protected attribute value v among the top-k rec-ommended candidates and the overall quali�ed pool of candidates,respectively. Due to our simulation framework, we have furtherde�ned being “quali�ed” as being able to pass the linear criteria ofthe “fair” user (w0 +wT x j ≥ 0). This is one key advantage of uti-lizing the proposed mathematical framework for our experiments,since understanding which candidate is quali�ed is otherwise anextremely di�cult task.

For each possible value v of the protected attribute, a negativeskew corresponds to lesser representation of candidates with valuev , while a positive skew corresponds to higher representation; askew of zero is ideal.

Our second measure, NDCSv (Normalized Discounted Cumula-tive Skew) is a ranking version of the skew measure, inspired byNormalized Discounted Cumulative Gain [21] and the related biasquanti�cation measures proposed in [35]. We de�ne NDCSv as thecumulative skew over all values of k up to the total number of can-didates shown, where each Skewv@k is weighted by a logarithmicdiscount factor based on k , and the sum is normalized by dividingby the sum of the discounting factors:

NDCSv =1Z

∑j

1log2(j + 1)

Skewv@j , (2)

where,Z =

∑j

1log2(j + 1)

.

This measure is based on the intuition that achieving fairness in topresults is more important than at the bottom, since the user is more

4

0.000 0.005 0.010 0.015 0.020 0.025 0.030 0.035 0.040 0.045Learning Rate

1.2

1.0

0.8

0.6

0.4

0.2

0.0

NDCS

NDCS Vs Learning Ratewarm_startwarm + online with bias 0.0warm + online with bias 0.2warm + online with bias 0.4warm + online with bias 0.6warm + online with bias 0.8warm + online with bias 1.0

(a) Legend

0.00 0.01 0.02 0.03 0.04Learning Rate

0.0

0.2

0.4

0.6

0.8

Skew

@25

Skew@25 Vs Learning Rate

(b) Skewдroup=1@25

0.00 0.01 0.02 0.03 0.04Learning Rate

0.0

0.2

0.4

0.6

0.8

Skew

@10

0

Skew@100 Vs Learning Rate

(c) Skewдroup=1@100

0.00 0.01 0.02 0.03 0.04Learning Rate

0.0

0.2

0.4

0.6

0.8

Skew

@50

0

Skew@500 Vs Learning Rate

(d) Skewдroup=1@500

0.00 0.01 0.02 0.03 0.04Learning Rate

0.0

0.2

0.4

0.6

0.8

Skew

@10

00

Skew@1000 Vs Learning Rate

(e) Skewдroup=1@1000

0.00 0.01 0.02 0.03 0.04Learning Rate

0.0

0.2

0.4

0.6

0.8

NDCS

NDCS Vs Learning Rate

(f) NDCS

Figure 3: Skew and NDCS Results for the Final Model After Online Personalization Updates

likely to see and “Accept” (or even just be shown) the candidates atthe top positions.

3.3 Evaluation of the Final Model After OnlinePersonalization Updates

We �rst present the results of our study after a round of onlinepersonalization update scheme is applied. As discussed in §3.2 (seeOur Setup), the initial model is what we call a warm-perceptronmodel, and we apply the online updates for 1000 candidates. Ateach new candidate, we choose the best candidate for the currentmodel, get the feedback from the user (§3.1.2) and update the cur-rent model. At the end of 1000 rounds (candidates, feedback, andupdates), we have generated a personalized model. In this section,we compare this personalized model (achieved by the end of 1000thupdate) to the warm-perceptron model (which was the initial modelbefore online updates), via an independent ranking of a large poolof candidates generated per §3.1, and checking the e�ect on thefairness and precision metrics.

Since the calculation of the fairness measures like Skew@k (Eq. 1)and NDCS (Eq. 2) requires the knowledge of the protected attributedistribution in the baseline population, we use the same data pointsused for online training and label them using a user model withpbias = 0 and the same weights [w0,w1, . . . ,wm ] for the purposeof calculating the baseline proportions (similar to the computationin Eq. 1). We use these labels from the fair user to get the base-line proportions (the binary protected attribute proportions in thequali�ed candidate set) of the two groups.

In Figure 3, we present Skew@k and NDCS results for the on-line personalization updates if we received the feedback to therecommended candidates from users with di�erent pbias values.We observe that as learning rate increases, the skew value increasesat �rst and quickly saturates with increasing learning rate, for alltypes of users. From Figures 3b-3f, we can see this pattern acrossall k for Skew@k and also in the NDCS metric. Furthermore, wesee that the fairness metrics in warm_start (warm-perceptron) and

warm + online (online personalization updates applied to warm-perceptron) with 0 bias are exactly the same. The reason is thatthe warm-perceptron model is also trained with data having 0 bias(pbias = 0). Thus, the perceptron weights don’t change furtherduring the online rounds, since the weights are already learned inthe warm-start phase. Hence, warm-start model and warm-start+ online with 0 bias correspond to exactly the same perceptronweights. Furthermore, we observe that as the bias in training dataincreases, the Skew/NDCS metrics also increase. This can be ex-plained as the perceptron learning the unfair behavior of the userby adjusting the weights of the attributes x2 and x3 which encodeinformation about the sensitive attribute.

In Figure 4, we examine the precision of the warm-perceptronmodel vs. the subsequently online trained model on the biased userdata at various pbias values, as a function of the learning rate. Weobserve that as learning rate increases, the precision increases at�rst and quickly saturates with increasing learning rate. From Fig-ures 4b-4e, we can see this pattern across all k for Precision@k (i.e.,precision within the �rst k candidates recommended for users withdi�erent pbias ). Furthermore, we see that the warm-start modelhas very low precision while the online trained model has preci-sion close to 1. This is because the warm-start model is trained ondata labeled by a user with pbias = 0 and is not precise enough topredict the response of users with large pbias to the recommendedcandidates. The online model however learns the unfair behaviorof user well enough for top k data points and hence precision@k issigni�cantly improved. This means that online personalization, asexpected, comes with a signi�cant advantage for the “relevance” ofrecommendations, especially because it incorporated the algorith-mic bias. This demonstrates the potential problem with applyingsuch a preference understanding methodology, even when the ini-tial model learned from a large pool of (say, unbiased) users is fairin recommendations.

5

0.000 0.005 0.010 0.015 0.020 0.025 0.030 0.035 0.040 0.045Learning Rate

0.8

0.6

0.4

0.2

0.0

0.2

Prec

ision

@10

00

Precision@1000 Vs Learning Ratewarm_start with bias 0.0warm + online with bias 0.0warm_start with bias 0.2warm + online with bias 0.2warm_start with bias 0.4warm + online with bias 0.4warm_start with bias 0.6warm + online with bias 0.6warm_start with bias 0.8warm + online with bias 0.8warm_start with bias 1.0warm + online with bias 1.0

(a) Legend

0.00 0.01 0.02 0.03 0.04Learning Rate

0.4

0.6

0.8

1.0

Prec

ision

@25

Precision@25 Vs Learning Rate

(b) Precision@25

0.00 0.01 0.02 0.03 0.04Learning Rate

0.4

0.6

0.8

1.0

Prec

ision

@10

0

Precision@100 Vs Learning Rate

(c) Precision@100

0.00 0.01 0.02 0.03 0.04Learning Rate

0.4

0.6

0.8

1.0

Prec

ision

@50

0Precision@500 Vs Learning Rate

(d) Precision@500

0.00 0.01 0.02 0.03 0.04Learning Rate

0.4

0.6

0.8

1.0

Prec

ision

@10

00

Precision@1000 Vs Learning Rate

(e) Precision@1000

Figure 4: Precision Results for the Final Model After Online Personalization Updates

3.4 Study of Personalized Model Evolutionwith Respect to Fairness and Precision

In this experiment, we interrupt the online learning process every25 rounds and measure the fairness metrics for the ranked list ofdata points that has been shown thus far. Note that, semantically,this is di�erent from the experiments presented in the previous sec-tion where we used the perceptron that is the outcome of the onlineupdates over 1000 rounds (each round is presenting a candidate us-ing the updated model, getting feedback, and further updating themodel for the next round), and reranked all the 12000 data pointsto measure Skew@k, NDCS, and Prec@k. However, in this exper-iment, we interrupt the online learning process every 25 roundsand use the recommendations so far to measure the metrics. Thiscan be seen as investigating the algorithmic bias that is being inte-grated into the online personalized model in real-time. Thus, thisexperiment measures the evolution of fairness in recommendationsas we update the model with immediate feedback.

In Figure 5b, we evaluate the e�ect of learning rate and thenumber of recommendations on Skewдroup=1@25 metric for per-sonalizing towards the preference of a single user with pbias = 0.4.Since the model learns the biased preferences of the user more andmore with each subsequent training round, we observe that theskew increases with the number of training rounds. Furthermore, asthe learning rate increases, the skew increases (keeping the numberof rounds �xed), since we are moving the model “faster” towardsthe biased preferences of the user.

In Figure 5d, we look into the e�ect of pbias and the number ofrecommendations on Skewдroup=1@25 metric with a �xed learningrate of 0.01. We observe that the skew increases with the numberof rounds (recommendations), as we learn the biased behavior ofthe user. Also, as pbias increases, the skew grows faster with morenumber of rounds. For the model with pbias = 0, the weights donot change much with more training rounds. Hence, the skew doesnot change signi�cantly with the increase in the number of training

0 200 400 600 800 1000Training Rounds

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

Skew

Skew Vs Training Rounds: warm_start with bias 0.4learning_rate 0.01learning_rate 0.02learning_rate 0.03learning_rate 0.04

(a) Legend (�xedbias)

0 200 400 600 800 1000Training Rounds

0.0

0.2

0.4

0.6

Skew

Skew Vs Training Rounds: warm_start with bias 0.4

(b) Evolution with �xed bias

0 200 400 600 800 1000Training Rounds

1.0

0.8

0.6

0.4

Skew

Skew Vs Training Rounds: learning rate of 0.01warm_start with bias 0.0warm_start with bias 0.2warm_start with bias 0.4warm_start with bias 0.6warm_start with bias 0.8warm_start with bias 1.0

(c) Legend (�xedlearning rate)

0 200 400 600 800 1000Training Rounds

0.0

0.2

0.4

0.6

Skew

Skew Vs Training Rounds: learning rate of 0.01

(d) Evolution with �xed learning rate

Figure 5: Evolution of Skew in Online Personalization

rounds. However, for users with high pbias , the feature weightsincorporate the algorithmic bias more and more as the number oftraining rounds increases and hence the skew increases. Overall,this study shows that for biased users, it is really easy to erase theunbiased/neutral nature of the warm-start model, if the goal is tounderstand and personalize the model to their (biased) preferences.

4 FAIR REGULARIZATIONWe next discuss our method for restricting the learned models tobe fair, followed by an evaluation of this approach in §4.2.

Consider the problem of constructing a fair machine learningmodel. Assume that we have data points in the form of (a featurecolumn vector, scalar protected attribute, scalar label) tuple, (x ,a,y).

6

Let us denote our training data as S ={ (x1,a1,y1), (x2,a2,y2),(x3,a3,y3), . . ., (xN ,aN ,yN ) }.

We would like to learn a model h(x) = wT x such that h(x) anda are mutually independent. Let us de�ne matrix X , Y , A out oftraining data, formed by stacking xi s, yi s, zi s as columns.

X =[x1 x2 x3 . . . xN

]A =

[a1 a2 a3 . . . aN

]Y =

[y1 y2 y3 . . . yN

]We assume that S is given to us and it can be used to select theh ∈ H as per our desired constraints. Such a choice of h onlydepends on S = {(xi ,ai ,yi )|i = 1, 2, 3, . . . ,N } and that data pointsin the form of (x ,a,y) can be sampled, independent and identicallydistributed from D and are especially independent of S . Such datapoints (x ,a,y) would also be independent of the selected h whichsolely depends on S .

Now we present our fair learning problem for linear models as:

Minimizew | |Y −wTX | |2

s.t. the distributions of wT x ⊥ a

To make this problem more tractable, we make two relaxations.

Independence to Correlation: We �rst relax the constraint ofwT x and a being independent to only being uncorrelated. It is well-known that independent variables are uncorrelated. Hence, zerocorrelation is a weaker constraint than independence. In the specialcase of jointly Gaussian variables, being uncorrelated implies inde-pendence. Further, we use empirical correlations computed fromtraining data, in the place of actual correlations in the optimization.

a to Model Estimate of a: Instead of using a in the optimiza-tion, we �rst learn a estimate of a from x using a similar modelused to learn y from x and then use those estimates in the opti-mization. Such estimate a is the best that the model can possiblyinfer from the training data about the protected attribute. We tryto �nd the best �t least square solution to the equation wT

a X = A.Once we learn the best �t (in least squares sense) linear modelto predict A from X , we can approximate the correlation with1N wT

a X (wTX )T = 1N wT

a XXTw = wT

a Cov(X ,X )w = wTa Σxw .

With the above relaxations, learning a fair linear model reducesto solving:

Minimizewa | |A −wTa X | |2

s.t. | |wa | | < ϵaand,

Minimizew | |Y −wTX | |2s.t. | |wT Σxwa | | < ϵ

where ϵa and ϵ are su�ciently small real numbers.The above optimization problem has two parts. We �rst learn a

best �t linear model from X to A. We use the weights learned in the�rst model to optimize for a fair best �t linear model from X to Y .

Minimize | |A −wTa X | |2

s.t. | |wa | | < ϵaNote that the �rst minimization problem has an additional con-straint of small L2 norm on weights wa because the unconstrainedoptimization might be ill-posed. The contribution of the harmless

attribute components in feature vectors’ xi s towards the protectedattribute is the ill-posed part. Hence, to make the contributions ofthe harmless attribute components in feature vectors xi s small, weimpose the additional constraint of small L2 norm on the weightswa .

The second optimization (for fair linear model) can be equiva-lently expressed as:

Minimize | |Y −wTX | |2 + λ | |wTwr eд | |2 ,

where wr eд = Σxwa . This is the Langrange Multiplier version orthe regularized linear regression. We call this kind of regularizationas Fair Regularization from now on. This can be solved exactly as:

(XXT + λwr eдwTreд)w = XYT ,

or it can also be solved iteratively by a gradient update equation:

wn+1 = wn + η(yn − sgn(wTn xn ))xn − λ(wT

nwr eд)wr eд .

Posing the fair learning problem as a regularized model has thebene�t that:

• We need not transform the data to any kind of fair space.Fairness conditions are imposed on the model at train time,and not on training data which can still be used in its rawpotentially unfair form.

• Fairness is imposed in an end-to-end single module com-pared to transforming the data to a fair space and thenusing the transformed data for the learning task at hand.

4.1 ExtensionsExtending to other fairness conditions: Some commonly usedfairness notions in machine learning are described in Appendix A.3.To achieve other kinds of fairness conditions like conditional inde-pendence (C ⊥ A | Y ), our method can be tweaked by replacing theexpectations with conditional expectations in the constraint of thesecond optimization problem. The resulting equation would resultin Σx |y = E[XXT | Y ] in the place of Σx = E[XXT ]. However,within the scope of our paper, we only work with the independenceconstraint.

Extending tomore general hypothesis space: Finally, we wouldlike to describe how our approach can be extended for more generalhypothesis space. For general hypothesis space H , our �rst learningproblem can be extended to learning the best hypothesis ha ∈ Hwhich can predict ai from xi for i = 1, 2, 3, . . . ,N . Then in thesecond learning problem, we extend our regularization term whichgets optimized with the training loss using the following insight:

wTwr eд = wT Σxwa = E[(wT x)(wT

a x)T ] = E[(h(x))(ha (x))T ]

Hence, while training to reduce the loss L(hn (xn ),yn ), we replacethe loss with L(hn (xn ),yn ) + λ (hn (xn ))(ha (xn )T )2 where hn is thehypothesis used and xn is the training data point chosen in thenth training iteration. Such a regularization has the e�ect that itpenalizes the correlation of the model with another model that isthe best �t that can be learned to predict the protected attribute.However, such a regularization is more noisy in general and takesmore training rounds to converge compared to our case of linearmodel where the correlation can be better calculated and used at

7

every iteration. Within the context of this paper, we only work withlinear models, and leave such extensions to future work.

4.2 Evaluation of Mitigation using FairRegularization

We next build upon our experiments in §3.3 with the fair regulariza-tion idea developed in §4. We study the e�ect of the regularizationparameter λ on skew and precision. We follow the same setup asin §3.2 and §3.3 except that we �x a constant learning rate andexperiment with di�erent values of λ.

Figure 6 presents Skew@k and NDCS results as function of λ.As expected, Skew@k and NDCS drop towards 0 for all data setsgenerated with di�erent amount of user bias as λ increases, whichshows the e�cacy of the fair regularization approach. However,we incur a trade-o� in precision of the recommendations, sincethe regularization reduces the ability of the models to learn thebiased preferences of the users. From Figure 7, we can see thatas λ increases, Precision@k of the online updated model drops toPrecision@k of the warm-perceptron model for each type of userwith di�erent pbias . These results suggest an inherent tension be-tween ensuring fairness and personalization. We could think offair regularization also as attempting to address the over-�ttingproblem. Here, our goal is to prevent or minimize over�tting tothe biased behavior of the user by imposing constraints on thelearned weights for the proxy attributes. However, since the mainobjective of personalization is to learn and cater to the preferencesof the given user as closely as possible (hence, for lack of a betterword, “over�t” to the user’s preferences, however biased they maybe), regularization for ensuring fairness could hurt precision as weobserved. The lesson learned is that we need to be careful when de-signing online personalized models and be aware of their potentialto introduce algorithmic bias, even more prominently compared too�ine models which may have been trained on data from a largeset of users and tested for desired fairness properties.

5 CONCLUSIONConsidering the importance of understanding and addressing algo-rithmic bias in personalized ranking mechanisms, we formulatedthe problem of studying fairness in online personalization. Weperformed an investigation of potential biases possible in onlinepersonalization settings involving ranking of individuals. Start-ing from a fair warm-start machine learned model, we empiricallyshowed that online personalization can cause the model to learn toact in an unfair manner, if the user is biased in his/her responses.As a part of this study, we presented a stylized mathematical modelfor generating training data with potentially biased features as wellas potentially biased labels, and quanti�ed the extent of bias that islearned by the model when the user sometimes responds in a biasedmanner as in plausible real-world settings. The ideas/intuition un-derlying our framework for generating training data for simulationpurposes are likely to be of broader interest to the research com-munity for evaluating fairness-aware algorithms in other scenarios,where either suitable datasets may not be available or it may beinfeasible to perform extensive experimentation with real datasets.We then formulated the problem of learning personalized modelsunder fairness constraints, and proposed a regularization based

approach for mitigating biases in machine learning, speci�cally forlinear models. We evaluated our methodology through extensivesimulations with di�erent parameter settings.

ACKNOWLEDGMENTSThe authors would like to thank other members of LinkedIn Careersand Talent Solutions teams for their collaboration for performingour experiments, and Deepak Agarwal, Erik Buchanan, Patrick Che-ung, Patrick Driscoll, Nadia Fawaz, Joshua Hartman, Rachel Kumar,Ram Swaminathan, Hinkmond Wong, Ryan Wu, Lin Yang, LiangZhang, and Yani Zhang for insightful feedback and discussions.

REFERENCES[1] J. Angwin, J. Larson, S. Mattu, and L. Kirchner. Machine bias. ProPublica, 2016.[2] A. Asudeh, H. V. Jagadish, J. Stoyanovich, and G. Das. Designing fair ranking

schemes. arXiv:1712.09752, 2017.[3] S. Barocas and M. Hardt. Fairness in machine learning. In NIPS Tutorial, 2017.[4] A. Beutel, J. Chen, Z. Zhao, and E. H. Chi. Data decisions and theoretical impli-

cations when adversarially learning fair representations. In FAT–ML Workshop,2017.

[5] A. Beutel et al. Fairness in recommendation ranking through pairwise compar-isons. In KDD, 2019.

[6] A. J. Biega, K. P. Gummadi, and G. Weikum. Equity of attention: Amortizingindividual fairness in rankings. In SIGIR, 2018.

[7] T. Calders and S. Verwer. Three naive Bayes approaches for discrimination-freeclassi�cation. Data Mining and Knowledge Discovery, 21(2), 2010.

[8] A. Caliskan, J. J. Bryson, and A. Narayanan. Semantics derived automaticallyfrom language corpora contain human-like biases. Science, 356(6334), 2017.

[9] L. E. Celis and N. K. Vishnoi. Fair personalization. In FATML, 2017.[10] S. Corbett-Davies, E. Pierson, A. Feller, S. Goel, and A. Huq. Algorithmic decision

making and the cost of fairness. In KDD, 2017.[11] C. Dwork, M. Hardt, T. Pitassi, and R. Z. Omer Reingold. Fairness through

awareness. In ITCS, 2012.[12] H. Edwards and A. Storkey. Censoring representations with an adversary. In

ICLR, 2016.[13] M. Feldman, S. A. Friedler, J. Moeller, C. Scheidegger, and S. Venkatasubramanian.

Certifying and removing disparate impact. In KDD, 2015.[14] S. A. Friedler, C. Scheidegger, and S. Venkatasubramanian. On the (im) possibility

of fairness. arXiv:1609.07236, 2016.[15] S. A. Friedler, C. Scheidegger, S. Venkatasubramanian, S. Choudhary, E. P. Hamil-

ton, and D. Roth. A comparative study of fairness-enhancing interventions inmachine learning. arXiv:1802.04422, 2018.

[16] S. C. Geyik, S. Ambler, and K. Kenthapadi. Fairness-aware ranking in search &recommendation systems with application to LinkedIn talent search. In KDD,2019.

[17] S. Hajian, F. Bonchi, and C. Castillo. Algorithmic bias: From discriminationdiscovery to fairness-aware data mining. In KDD Tutorial on Algorithmic Bias,2016.

[18] S. Hajian and J. Domingo-Ferrer. A methodology for direct and indirect discrim-ination prevention in data mining. IEEE TKDE, 25(7), 2013.

[19] M. Hardt, E. Price, and N. Srebro. Equality of opportunity in supervised learning.In NIPS, 2016.

[20] S. Jabbari, M. Joseph, M. Kearns, J. Morgenstern, and A. Roth. Fairness inreinforcement learning. In ICML, 2017.

[21] K. Jarvelin and J. Kekalainen. Cumulated gain-based evaluation of IR techniques.ACM Trans. on Information Systems (TOIS), 2002.

[22] T. Kamishima, S. Akaho, H. Asoh, and J. Sakuma. Fairness-aware classi�er withprejudice remover regularizer. In ECML PKDD, 2012.

[23] J. Kleinberg, S. Mullainathan, and M. Raghavan. Inherent trade-o�s in the fairdetermination of risk scores. In ITCS, 2017.

[24] C. Louizos, K. Swersky, Y. Li, M. Welling, and R. Zemel. The variational fairautoencoder. In ICLR, 2016.

[25] J. Mary, C. Calauzenes, and N. E. Karoui. Fairness-aware learning for continuousattributes and treatments. In ICML, 2019.

[26] M. Morik, A. Singh, J. Hong, and T. Joachims. Controlling fairness and bias indynamic learning-to-rank. In SIGIR, 2020.

[27] H. Narasimhan, A. Cotter, M. Gupta, and S. Wang. Pairwise fairness for rankingand regression. In AAAI, 2020.

[28] D. Pedreschi, S. Ruggieri, and F. Turini. Discrimination-aware data mining. InKDD, 2008.

[29] D. Pedreschi, S. Ruggieri, and F. Turini. Measuring discrimination in socially-sensitive decision records. In SDM, 2009.

8

0 20 40 60 80 100Lambda

1.2

1.0

0.8

0.6

0.4

0.2

0.0

NDCS

NDCS Vs Lambda

warm_startwarm + online with bias 0.0warm + online with bias 0.2warm + online with bias 0.4warm + online with bias 0.6warm + online with bias 0.8warm + online with bias 1.0

(a) Legend

0 20 40 60 80 100Lambda

0.0

0.2

0.4

0.6

0.8

Skew

@25

Skew@25 Vs Lambda

(b) Regularized Skewдroup=1@25

0 20 40 60 80 100Lambda

0.0

0.2

0.4

0.6

0.8

Skew

@10

0

Skew@100 Vs Lambda

(c) Regularized Skewдroup=1@100

0 20 40 60 80 100Lambda

0.0

0.2

0.4

0.6

0.8

Skew

@50

0

Skew@500 Vs Lambda

(d) Regularized Skewдroup=1@500

0 20 40 60 80 100Lambda

0.0

0.2

0.4

0.6

0.8

Skew

@10

00

Skew@1000 Vs Lambda

(e) Regularized Skewдroup=1@1000

0 20 40 60 80 100Lambda

0.0

0.2

0.4

0.6

0.8

NDCS

NDCS Vs Lambda

(f) Regularized NDCS

Figure 6: Regularized Skew and NDCS

0 20 40 60 80 100Lambda

0.8

0.6

0.4

0.2

0.0

0.2

Prec

ision

@10

00

Precision@1000 Vs Lambdawarm_start with bias 0.0warm + online with bias 0.0warm_start with bias 0.2warm + online with bias 0.2warm_start with bias 0.4warm + online with bias 0.4warm_start with bias 0.6warm + online with bias 0.6warm_start with bias 0.8warm + online with bias 0.8warm_start with bias 1.0warm + online with bias 1.0

(a) Legend

0 20 40 60 80 100Lambda

0.4

0.6

0.8

1.0

Prec

ision

@25

Precision@25 Vs Lambda

(b) Regularized Precision@25

0 20 40 60 80 100Lambda

0.4

0.6

0.8

1.0

Prec

ision

@10

0

Precision@100 Vs Lambda

(c) Regularized Precision@100

0 20 40 60 80 100Lambda

0.4

0.6

0.8

1.0

Prec

ision

@50

0

Precision@500 Vs Lambda

(d) Regularized Precision@500

0 20 40 60 80 100Lambda

0.4

0.6

0.8

1.0

Prec

ision

@10

00

Precision@1000 Vs Lambda

(e) Regularized Precision@1000

Figure 7: Regularized Precision

[30] A. Pérez-Suay, V. Laparra, G. Mateo-García, J. Muñoz-Marí, L. Gómez-Chova,and G. Camps-Valls. Fair kernel learning. In ECML PKDD, 2017.

[31] A. Romanov et al. What’s in a name? reducing bias in bios without access toprotected attributes. In NAACL, 2019.

[32] A. Singh and T. Joachims. Fairness of exposure in rankings. In KDD, 2018.[33] D. Slack, S. A. Friedler, and E. Givental. Fairness warnings and fair-MAML:

learning fairly with minimal data. In FAccT, 2020.[34] B. Woodworth, S. Gunasekar, M. I. Ohannessian, and N. Srebro. Learning non-

discriminatory predictors. In COLT, 2017.[35] K. Yang and J. Stoyanovich. Measuring fairness in ranked outputs. In SSDBM,

2017.[36] M. B. Zafar, I. Valera, M. Gomez Rodriguez, and K. P. Gummadi. Fairness beyond

disparate treatment & disparate impact: Learning classi�cation without disparatemistreatment. In WWW, 2017.

[37] M. Zehlike, F. Bonchi, C. Castillo, S. Hajian, M. Megahed, and R. Baeza-Yates.FA*IR: A fair top-k ranking algorithm. In CIKM, 2017.

[38] R. Zemel, Y. Wu, K. Swersky, T. Pitassi, and C. Dwork. Learning fair representa-tions. In ICML, 2013.

A APPENDIXA.1 Legal Frameworks on Bias and

DiscriminationOver the last several decades, legal frameworks around the worldhave prohibited the use of sensitive attributes such as race, gender,sexual orientation, and age for di�erent forms of decision making.We refer to such attributes whose use in certain decision makinghas been prohibited by law as protected attributes. Examples ofattributes declared as protected by law in the United States include:

• Race, Color, Sex, Religion, National origin (Civil Rights Actof 1964; Equal Pay Act of 1963),

• Age (Age Discrimination in Employment Act of 1967),• Disability status (Rehabilitation Act of 1973; Americans

with Disabilities Act of 1990), and9

X

A

Features

Protected Attributes

Input ML Model Output R(X,A)

Output,e.g. via Softmax

Thresholding C(X,A)

Classifier,i.e. Binary Output

Train

Y

Target Labels

Figure 8: Formal Machine Learning Model

• Veteran status (Vietnam Era Veterans’ Readjustment As-sistance Act of 1974).

When legal frameworks prohibit the use of such protected at-tributes in decision making, there are usually two competing ap-proaches on how this is enforced in practice: Disparate Treatment vs.Disparate Impact. Avoiding disparate treatment requires that suchattributes should not be actively used as a criterion for decisionmaking and no group of people should be discriminated againstbecause of their membership in some protected group. Avoidingdisparate impact requires that the end result of any decision makingshould result in equal opportunities for members of all protectedgroups irrespective of how the decision is made. This can oftenmean that certain forms of a�rmative action based on protectedattributes may be needed to encourage or improve opportunities forcertain protected groups to o�set the biases present in the societydue to historical reasons. Our work can be thought of as followingthe approach of avoiding disparate impact.

A.2 Sources of BiasTo prevent a machine learning model from incorporating biases, itis not enough to prohibit the usage of protected attributes. Oftenone or more protected attributes could be correlated with seeminglyharmless attributes (hereafter denoted as “proxy attributes”). Pres-ence of proxy attributes in a machine learning model potentiallyallows it to extract information on protected attributes and furtheruse it to make decisions in an unfair manner.

Some possible reasons for biases in the outcomes of machinelearning models include:

• Presence of proxy attributes in the set of features utilizedfor training,

• Di�erent quality and quantity of samples from di�erentprotected groups in training data, and

• Structured noisy labels in training data with di�erent amountsof noise depending upon protected attributes.

In §3.1, we describe our stylized model of simulating training datawhich embodies most of the above sources of algorithmic bias.

A.3 Di�erent Notions of FairnessConsider the formal setup of a machine learning model shown inFigure 8. Let X be the feature vector that is used by the machinelearning model as input and A be the corresponding protectedattribute, which, for simplicity, is assumed to be a binary variabletaking values 1 and 0 (e.g., whether sex = ‘female’; whether age≥ 40). Let R(X ,A) be the output of the model, which can be thought

of as the result of a softmax procedure, and in the range (0, 1). Athresholding scheme is applied on the output R(X ,A) to convert itinto binary labels C(X ,A). The model is trained such that C(X ,A)follows the binary labels Y in training data. In such a setting, wecould consider multiple approaches to formalize what we mean byfairness, leading to the following de�nitions [3].

(1) C ⊥ A : Independence condition requires that output isindependent of the protected attribute. This is often calledstatistical parity where chances of “Accept” (or “Reject”)conditioned on the value of the protected attribute are thesame for both protected groups, i.e.,

P(C = 1 | A = 1) = P(C = 1 | A = 0) = P(C = 1).Many approximate versions of this are often recommendedby guidelines such as the Uniform Guidelines for EmployeeSelection Procedures, adopted by certain U.S. governmentagencies in 1978. According to these guidelines, adverseor disparate impact is determined by using the “four-�fthsor 80% rule” which requires that the selection rate for anyrace, sex, or ethnic group be at least four-�fths (or 80%) ofthe rate for the group with the highest rate for the processto be considered as not having adverse impact. For the caseof binary protected attribute, this corresponds to

10.8≥ P(C = 1 | A = 1)

P(C = 1 | A = 0) ≥ 0.8.

A potential problem with this criterion is that often trainingdata might not support equal “Accept” chances for all pro-tected groups and hence enforcing such an independenceconstraint might hurt the precision or other performancemeasures of the algorithm.

(2) C ⊥ A | Y : Conditional independence condition requiresthat the model makes equal amounts of mis-classi�cationsfor all protected groups, i.e.,

PA=1(C = 1 | Y = 1) = PA=0(C = 1 | Y = 1),PA=1(C = 1 | Y = 0) = PA=0(C = 1 | Y = 0).

Thus, it translates to enforcing equal true positive rates andfalse positive rates for all protected groups. This criterioncan allow the model training to proceed to �t to the datawith high precision.

(3) Y ⊥ A | C : This condition is similar to the previouscondition, but with the roles ofC and Y reversed. Note thatC is trained to follow Y in training data and hence it makessense to treat them equivalently.

In this paper, we focus on the independence condition, i.e., (1) above,based on which we devised an approach to mitigate bias in §4.

10