LinkedIn Salary: A System for Secure Collection andPresentation of Structured Compensation Insights to Job

Seekers

Krishnaram Kenthapadi Ahsan Chudhary Stuart AmblerLinkedIn Corp

Sunnyvale, CA, USA{kkenthapadi, achudhary, sambler}@linkedin.com

ABSTRACTOnline professional social networks such as LinkedIn haveenhanced the ability of job seekers to discover and assesscareer opportunities, and the ability of job providers to dis-cover and assess potential candidates. For most job seekers,salary (or broadly compensation) is a crucial considerationin choosing a new job. At the same time, job seekers facechallenges in learning the compensation associated with dif-ferent jobs, given the sensitive nature of compensation dataand the dearth of reliable sources containing compensationdata. Towards the goal of helping the world’s profession-als optimize their earning potential through salary trans-parency, we present LinkedIn Salary, a system for collectingcompensation information from LinkedIn members and pro-viding compensation insights to job seekers. We present theoverall design and architecture, and describe the key com-ponents needed for the secure collection, de-identification,and processing of compensation data, focusing on the uniquechallenges associated with privacy and security. We performan experimental study with more than one year of compen-sation submission history data collected from over 1.5 mil-lion LinkedIn members, thereby demonstrating the tradeoffsbetween privacy and modeling needs. We also highlight thelessons learned from the production deployment of this sys-tem at LinkedIn.

1. INTRODUCTIONOnline professional social networks such as LinkedIn have

enhanced the ability of job seekers to discover and assesscareer opportunities, and the ability of job providers to dis-cover and assess potential candidates. Compensation is knownto be a crucial consideration in choosing a new job op-portunity for most job seekers.1 However, job seekers en-counter obstacles in learning the compensation associatedwith different jobs, given the sensitive nature of compen-sation data and the dearth of reliable sources containingcompensation data. The recently launched LinkedIn Salary

1More candidates (74%) would like to see compensationcompared to any other feature in a job posting, accordingto a survey of over 5000 job seekers in US and Canada [5].Compensation is valued the most by job seekers when look-ing for new opportunities, according to a US survey of 2305adults [6].

.

product2 has been designed to help job seekers explore com-pensation along different dimensions, make more informedcareer decisions, and thereby optimize their earning poten-tial through salary transparency.

With over 500 million members (registered users), to-gether with the associated structured information includingthe work experience, educational history, and skills, LinkedInis in a unique position to collect compensation data from itsmembers at scale and provide rich, robust insights coveringdifferent aspects of compensation, while preserving memberprivacy. For instance, we can provide insights on the dis-tribution of base salary, bonus, equity, and other types ofcompensation for a given profession; how they vary basedon factors such as region, experience, education, companysize, and industry; and which regions, industries, or compa-nies pay the most.

Besides helping job seekers understand their economic valuein the marketplace, the compensation data has several othersocial benefits. It can help us better understand the mone-tary dimensions of the Economic Graph [44] (which includescompanies, industries, regions, jobs, skills, educational in-stitutions, etc.). The availability of compensation insightswith respect to dimensions such as gender, ethnicity, andother demographic factors can lead to greater transparency,shedding light on the extent of compensation disparity, andthereby help stakeholders including employers, employees,and policy makers to take steps to address pay inequal-ity. Further, products such as LinkedIn Salary can leadto greater efficiency in the labor marketplace by reducingasymmetry of compensation knowledge, and by serving asmarket-perfecting tools for workers and employers [26]. Fi-nally, such tools have the potential to help students makegood career choices, taking expected compensation into ac-count, and to encourage workers to learn skills that are nec-essary for obtaining well paying jobs, thereby helping narrowthe skills gap.

In this paper, we present LinkedIn Salary, a system forsecurely collecting compensation information from LinkedInmembers and providing structured compensation insights tojob seekers. We highlight the unique challenges associatedwith privacy and security while designing and implementingthe system, and present the overall design and architecturethat incorporates a combination of techniques such as en-cryption, access control, de-identification, aggregation, andthresholding. We also describe the key components neededfor the secure collection, de-identification, and processing of

2https://www.linkedin.com/salary

arX

iv:1

705.

0697

6v2

[cs

.SI]

17

Jul 2

017

compensation data. We empirically investigate the trade-offs between privacy and modeling needs using more thanone year of compensation submission history data collectedfrom over 1.5 million LinkedIn members. We also highlightthe lessons learned in practice from the deployment of thissystem at LinkedIn.

2. PROBLEM SETTINGWe next give a brief overview of the product, followed by

a discussion of the key system design goals, as well the chal-lenges from the privacy, security, and modeling perspectives.We then present the problem statement.

In the publicly launched LinkedIn Salary product [37],users can search for different titles and regions, and explorethe corresponding compensation insights (Figure 1). For agiven title and region, we present the quantiles (10th and90th percentiles, median) and histograms for base salary,bonus, and other types of compensation. We also presentmore granular insights on how the pay varies based on fac-tors such as region, experience, education, company size,and industry, and which regions, industries, or companiespay the most.

The compensation insights shown in the product are basedon more than one year of compensation data that we havebeen collecting from LinkedIn members (Figure 2). We de-signed the data collection process based on a give-to-getmodel. First, we select cohorts (such as User Experience De-signers in San Francisco Bay Area) with a sufficient numberof LinkedIn members. Within each cohort, we send emailsto a random subset of members, requesting them to submittheir compensation data (in return for aggregated compen-sation insights later). Once we collect enough data, we getback to the responding members with the compensation in-sights, and also reach out to the remaining members in thosecohorts, promising corresponding insights immediately uponsubmission of their compensation data. We describe the col-lection mechanism in further detail in §3.3.1.

Figure 2 shows screenshots of the collection interface asseen by a hypothetical user. The first two screens showthe interface for a member to enter the base salary and ad-ditional compensation respectively for his/her current jobposition. The current job position information is automat-ically retrieved from the member’s profile and displayed onthe first screen, with an option for the member to modifyif desired. The member can choose to enter different typesof additional compensation, as shown in the second screen.After the member submits the compensation data and oncethe corresponding insights are available, the member is pre-sented with these insights, along with other useful insightssuch as the compensation for related positions in the sameregion and the compensation for the same position in relatedregions.

2.1 System Requirements and ChallengesPreserving privacy of members: Considering the sensitive

nature of compensation data and the desire for preservingprivacy of users, a key requirement is to design our systemsuch that there is protection against data breach, and anyone individual’s compensation data cannot be inferred byobserving the outputs of the system. We would like protec-tion against external data breach as well as insider attacks.Further, we require the compensation insights to be gener-ated based on only cohort level data containing de-identified

compensation submissions (e.g., salaries for UX Designers inSan Francisco Bay Area), limited to those cohorts having atleast a minimum number of entries.

Engineering requirements: The system needs to be de-signed such that “bootstrapping” can be performed whenneeded. Bootstrapping refers to the ability to regenerate thehistorical output data whenever there are changes in partsof the system. For example, the need for this functionalitycould arise due to software bugs in the system implementa-tion, changes to the various services that are invoked by thesystem, and changes to the product design such as additionof new types of insights.

Modeling on aggregated data: Due to the privacy require-ments stated above, the salary modeling system has accessonly to cohort level data containing aggregated compensa-tion submissions. Each cohort is defined by a combinationof attributes such as title, country, region, company, years ofexperience, and so forth, and contains aggregated compen-sation entries obtained from individuals having the samevalues of those attributes. Within a cohort, each individ-ual entry consists of values for different compensation typessuch as base salary, annual bonus, sign-on bonus, commis-sion, annual monetary value of vested stocks, and tips, and isavailable without associated user name, id, or any attributesother than those that define the cohort. As a result, ourmodeling choices are limited since we have access only tothe aggregated data, and cannot, for instance, train predic-tion models that make use of more discriminating featuresnot available due to de-identification.

Evaluation: We face unique evaluation and data qualitychallenges that are typically not present in several otheruser-facing products such as movie and job recommenda-tions. Since users themselves may not have a good per-ception of the true compensation range, we cannot performonline A/B testing to compare the compensation insightsgenerated by different models. Further, there are very fewreliable and easily available ground truth datasets in thecompensation domain, and even when available (e.g., BLSOES dataset [4]), mapping such datasets to LinkedIn’s tax-onomy is inevitably noisy.

Outlier detection: The quality of the insights depends onthe quality of submitted data, and hence we need to detectand prune potential outlier entries. Such entries could arisedue to either mistakes/misunderstandings during submis-sion, or intentional falsification (such as someone attempt-ing to game the system). We required a solution that wouldwork even during the early stages of data collection, whenthis problem was more challenging, and there may not besufficient data across say, related cohorts.

Robustness and stability: Although some cohorts may eachhave a large sample size, a large number of cohorts typicallycontain very few (e.g., < 20) data points each. Given thedesire to have insights for as many cohorts as possible, weneed to make sure that the compensation insights are robustand stable even when there is data sparsity. In other words,the insights should be reliable, and not too sensitive to theaddition of a new entry for such cohorts. A related challengeis whether we can reliably infer the insights for cohorts withno data at all.

2.2 Problem StatementOur focus in this paper is on the unique challenges asso-

ciated with privacy and security while designing and imple-

Figure 1: LinkedIn Salary Insights Page

Figure 2: LinkedIn Salary Collection Interface. The interfaces for a member to enter the base salary andadditional compensation for his/her job position are shown in the first two screens. The compensation insightsfor the corresponding cohort are shown in the third screen.

menting the system. Our methodology for addressing themodeling challenges is described in [29]. Our problem canthus be stated as follows: How do we design LinkedIn Salarysystem to meet the immediate and future needs of LinkedInSalary and other LinkedIn products? How do we design oursystem taking into account the unique privacy and securitychallenges, while addressing the product requirements? Weaddress these questions in §3, wherein we present the systemarchitecture that incorporates a combination of techniquessuch as encryption, access control, de-identification, aggre-gation, and thresholding.

3. LINKEDIN SALARY SYSTEM DESIGNAND ARCHITECTURE

We describe the overall design and architecture of thedeployed system that powers the LinkedIn Salary product.As part of the description, we present the security and de-identification mechanisms. Our system uses a service ori-ented architecture (see Figure 3), and consists of the follow-ing three key components: a collection and storage compo-nent, a de-identification and grouping component, and aninsights and modeling component. We provide an overviewof the requests and responses of the underlying services as-sociated with a member submitting the compensation datato see the insights. We refer to the step numbers found inFigure 3 as part of this description.

3.1 Collection and StorageThe collection and storage component of our system is

responsible for allowing members to submit their compen-sation information, collecting different member attributes,and securely and privately storing the member attributesas well as the submitted compensation data. The primaryentry point for the collection flow is through email cam-paigns that target LinkedIn members, requesting them tosubmit their compensation information (described in detailin §3.3.1). Upon clicking on the email, the member is firstauthenticated and then taken to the secure web interface(that uses TLS) shown in Figure 2. Once the compensationdata is submitted by the member, this data passes throughmultiple routing layers before reaching the front-end service.All the intermediate layers also use TLS to secure the data(Step 1 of the architecture diagram). The front-end ser-vice is responsible for validation and authentication of therequest (Step 2). The data is then handed over to the sub-mission service for storage. The submission service collectsmember attributes such as title, company, and region, en-crypts them along with compensation data and stores in apersistent data store.Collection APIs: LinkedIn exposes all of its services inter-nally using RESTful APIs [23]. These services are poweredby rest.li, which is a open source REST + JSON frameworkfor building robust, scalable service architectures using type-safe bindings, dynamic discovery, and simple asynchronousAPIs [2]. In particular, compensation collection APIs alsouse the rest.li framework. The submission service is exposedas a rest.li endpoint and the front-end service interacts withthe submission API over TLS (Step 3 in the diagram). Thesubmission service then extracts attributes from the mem-ber profile using relevant data service APIs (Step 4). Thisinformation is extracted during the time of submission sincethe member may update the profile over time and we would

like to obtain the attribute values as of the time of the com-pensation submission. Examples of the extracted memberattributes include the standardized (or canonical) versionsof job title, company, and region, obtained by invoking re-spective LinkedIn standardization services. We also obtainattributes such as the the currency exchange rate at the timeof submission.

Once we obtain the member attributes by invoking variousLinkedIn Data services, we perform 2048-bit RSA based en-cryption of compensation data as well as member attributes,and store in our data store using a write-only API (Step5). We use different sets of public/private keys for en-cryption/decryption of member attributes and compensa-tion data due to reasons explained in §3.4.

Our system also provides a verification service which isused to verify if a member has submitted compensation in-formation in the past (not shown in Figure 3). This serviceis powered by a different data store that only stores whether(and when) a member submitted compensation information,but not the compensation data itself. This store is neededto power the give-to-get model used by our product.

3.2 De-identification and GroupingOnce the data is stored, we need to prepare it for anal-

ysis. We used an approach (inspired by k-Anonymity [36,35, 41]) which allowed us to use existing LinkedIn infras-tructure while preserving the privacy of LinkedIn members.We construct collections of member-submitted compensa-tion information by grouping together members satisfyingdifferent predicates, and call these collections as “cohorts”or “slices”. Each cohort or slice is defined by a combina-tion of attributes such as canonical title, country, region,company, and years of experience, and contains aggregatedcompensation entries obtained from individuals having thesame values of those attributes. Within a cohort, each indi-vidual entry contains values for different compensation typessuch as base salary, annual bonus, sign-on bonus, commis-sion, annual monetary value of vested stocks, and tips, andis available without associated user name, id, or any at-tributes other than those that define the cohort. An exam-ple of a title-country-region slice would be “User ExperienceDesigners in San Francisco Bay Area”, and an example ofa title-company-country-region slice would be “User Experi-ence Designers at Google in San Francisco Bay Area”. In ad-dition to the de-identification, we also require that each co-hort contain at least a minimum number k of entries, beforeit is made available for offline data processing and analysis.Note that prior to the above grouping, we apply LinkedInstandardization software to map a free-form attribute to itscanonical version. For example, an arbitrary title is mappedto one of about 25, 000 LinkedIn standardized (canonical) ti-tles, and similarly, each company and each region also getmapped to their canonical versions. This mapping is analo-gous to the generalization step in k-Anonymity [36, 35, 41].

At first, we wanted to store and process the the encrypted(but sensitive) data containing member attributes and mem-ber compensation in LinkedIn’s Hadoop ecosystem. How-ever, due to the inherent security limitations of Hadoopand HDFS, we decided to limit the amount of data madeavailable in HDFS towards minimizing the chances of re-identification, and pursued the de-identification approach ofslicing and requiring a minimum threshold of entries for pro-cessing.

Figure 3: LinkedIn Salary System Architecture, consisting of components/services pertaining to collection &storage, de-identification & grouping, and insights & modeling.

The slicing service runs asynchronously from the submis-sion service, and is triggered for each new submission by amember. Slicing service makes use of Databus, LinkedIn’sopen-source distributed change-data capture and notifica-tion system [1] (Step 6 in Figure 3). We limit the data thatis stored at the Databus servers to only member attributeinformation and submission identifiers (but not the com-pensation data). This ensures that the compensation datais secure even if the Databus service is breached. We usestored attributes to generate the slices, which are then usedfor analysis and generating insights.

As stated before, we perform analysis on a slice only af-ter it contains a minimum number of submission entries.Our system allows the specification of different minimumthresholds for different slice types (e.g., title-country-regionvs title-company-country-region). The slicing service usesa distributed data platform to keep track of the thresholds(Step 7). The slicing service keeps track of the number ofsubmissions that have been collected so far for each slice,and will decide to make a slice available for processing bysubsequent components only once the number of submis-sions exceeds the minimum threshold. In particular, sincethe slicing service does not access the compensation dataat all, a slice that has not met the threshold will not havethe associated compensation entries. Thus, for example, theslice, “User Experience Designers at Google in San FranciscoBay Area” would get processed once the threshold is met,while the slice, “CEOs at LinkedIn in San Francisco BayArea” would never get processed and hence the associatedcompensation will not be available downstream.

Our system supports a mechanism for modifying the times-tamp associated with the compensation submission so as toprevent timestamp based inference attacks, that is, preventthe ability to obtain the member’s identity by joining withpage view or other logs (discussed in detail in §3.4.1). Themodification can be either in the form of a random delay, orbased on the other submissions in the cohort. Such modifi-cation of the timestamp is performed using the delayed jobqueue (Step 8).

Once the queue task is ready to be processed, it is pickedup by the preparation service (Step 9). As our gatewayservice to the offline system, the preparation service pre-pares the data by fetching the compensation data storedduring collection (Step 10), and associates it with the slicedata. The compensation data corresponding to the sub-mission identifiers in a slice is fetched and decrypted, andafterwards, the submission identifiers are removed from theslice. In this manner, we do not retain any association be-tween the (decrypted) compensation data in a slice and thecorresponding submission identifiers. The prepared data isthen copied to HDFS for offline processing (Step 11). At thisstage, our system supports the ability to perform roundingor generalization of compensation entry values with a goalof minimizing the risk of joining the de-identified data withitself based on the exact compensation value and therebyidentifying multiple attributes associated with each com-pensation entry. We remark that the use of two differentservices, slicing service and preparation service, instead of acombined single service, provides an extra layer of security.This gives us protection against linking together member at-

tribute data and compensation data if any one service getsbreached.

3.3 Compensation Insights and Modeling

3.3.1 Selection and Targeting of MembersWe next describe the data collection process, focusing on

the selection of cohorts and members for sending emails re-questing submission of compensation data. First, cohortswith a sufficient number of LinkedIn members were selected.The members within a cohort were randomly ordered, andemails were sent to a random subset of members (followingthis order), requesting them to submit their compensationdata (in return for aggregated compensation insights later).Once we collected suffficient data, we got back to the re-sponding members with the compensation insights, and alsoreached out to the remaining members in the cohort, promis-ing insights immediately upon submission of their compen-sation data. We partitioned a cohort in this manner sinceproviding compensation insights immediately upon submis-sion results in a higher response rate, as well as a betteruser experience, and hence we wanted to ensure that as fewmembers as possible were requested prior to the availabilityof compensation insights.

We dynamically adapted the above process based on theobserved response rate. For example, by sending emails tothe first r1 members in the random ordering and observingthe number of responses s1, we estimated the response rate,γ as s1/r1. If the desired number of responses is α, we thenreached out to about r2 = α/γ − r1 additional members.Further, we wanted to target a cohort only if we expectedenough responses to be able to get back with the compen-sation insights. For this purpose, we estimated the responserate for a cohort based on other similar cohorts (e.g., con-sidering the response rates for the same title in other regionsin the case of a title-country-region cohort).

We remark that through random sampling (which is aspecial case of probability-based sampling [25]) and targeteddata collection, we minimize any selection bias and ensurethat the chosen set of members is representative of the un-derlying population of LinkedIn members in the cohort. Incontrast, compensation insights obtained based on self-reportedsurveys need not be representative of the underlying popu-lation. Addressing other types of biases (such as responsebias) is an avenue for future work.

3.3.2 Computation and Presentation of Compensa-tion Insights

Whenever compensation insights are requested (say, rightafter the member entering compensation data, or as part ofLinkedIn Salary product), the front-end service queries theverification service via a REST API to determine whetherthe member is eligible to view the insights. Based on theproduct and business needs, the eligibility can be defined interms of criteria such as whether the member has providedhis/her compensation data within the last one year (give-to-get model), whether the member has purchased a LinkedInpremium membership, whether the member satisfies a pred-icate expressed in terms of demographic / profile-based at-tributes (e.g., a middle-skilled worker; a student; based inemerging markets), or whether the member satisfies a pred-icate expressed in terms of activity attributes (e.g., likelyto transition from being an active member to an inactive

member). If yes, the front-end obtains the compensationinsights by querying the salary insight service. For each co-hort, depending on data availability, the compensation in-sight could include the quantiles (10th and 90the percentiles,median), and histograms for base salary and other compen-sation types. In addition, there are also other insights suchas how the pay varies based on factors such as region, in-dustry, experience, education, and company size [29].

The insights are generated using an offline data process-ing workflow that consumes the de-identified compensationdataset (Step 12 in Figure 3) and populates the insightVoldemort data stores [39] (Step 13), which are then usedby the salary insight service (Step 14). Statistical modelingcomponents such as outlier detection and Bayesian hierar-chical smoothing are used in the offline workflow to computerobust compensation insights. See [29] for an in-depth de-scription of the salary insight service, the offline workflow,and the statistical modeling components.

We remark that in addition to helping compute robustcompensation insights for cohorts with very little data, Bayesiansmoothing methodology also helps to achieve privacy in cer-tain cases. It is possible for the salaries to be identical in acohort with very few entries, thereby defeating the benefitof having a minimum threshold after de-identification. Forexample, the cohort “User Experience Designers at Googlein San Francisco Bay Area” may contain submissions fromjust 6 members, each with 110K as the base salary. In suchcases, the smoothing helps to derive a more reliable cohortestimate by “borrowing strength” from the ancestral cohortsthat have sufficient data, and thereby not reveal the empir-ical percentiles (all of which equal 110K) [29]. Further, thesmoothing helps to minimize the privacy risk by not reveal-ing the empirical percentiles based on very few data points,which could be quite sensitive to the addition of new datapoints.

3.4 Security MechanismsWe next describe the security mechanisms in our archi-

tecture.

Encryption (and decryption) of member attributes and com-pensation data using different sets of public (and private)key pairs: Recall that the data collected by the submissionservice corresponding to each member submission consistsof two components: the member attributes and the com-pensation data. Since the combination of different memberattributes could uniquely identify an individual in certaincases [40] (e.g., the attributes, title: “CEO” and company:“LinkedIn” together correspond to one person at a givenpoint of time), we would like to not only store the memberattributes and the compensation data in encrypted form, butalso use different encryption keys for both. As a result, anattacker that has access to only one of the decryption keyscannot infer the association between the member attributesand the corresponding compensation data. Further, our sys-tem is designed in such a way that no service has a need tosimultaneously decrypt both the member attributes and thecompensation data, and hence is not given access to bothdecryption keys as elaborated below. By using asymmetric,public key cryptography, we also ensure that the submissionservice can perform encryption but not decryption. In thevery unlikely event of the submission service being breached,the attacker would still not be able to decrypt the histori-

cal submissions (since it does not have the private keys),let alone retrieve the historical data (since the access to thestore is using a write-only API).

Separation of processing of member attributes and compen-sation data: In our system, the member attributes and thecompensation data are never processed at the same time,or by the same service, after the initial submission. Thisdesign, together with the use of two different encryptionmechanisms, ensures that an attacker would have to breakinto both encryption systems (and hence multiple services)to be able to connect the dots between member attributesand their corresponding compensation data.

Key store security: We achieved key store security by sepa-rating the keys into multiple keystores. Although the key-stores are password protected, we decided to restrict accessto the key resource(s) among different services, and henceseparated the keys into multiple keystores.

Limiting access to keys: Each service only has access to thepublic key(s) needed for encryption, and the private key(s)needed for decryption. We next describe the encryption anddecryption of data through the system workflow to illustratethis design choice. Upon obtaining the member attributesand the compensation data, the submission service encryptsthem using the respective public keys, Mpub and Cpub, andthen stores in the submission data store. Since this servicewould never be required to decrypt these data, it does nothave access to the corresponding private keys. Then, theslicing service which listens to the Databus stream gets trig-gered. Since this service requires member attribute informa-tion to create the slices, it decrypts the member attributesusing the corresponding private key, Mpri. Since the slicingservice does not need to access the compensation data atall, it does not possess the corresponding private key, Cpri.Next, the distributed threshold data store is used to identifywhether a slice has met its threshold requirement. The datain this store is encrypted using the symmetric key, T sym

instead of using asymmetric, public/private keys since thisdata is always encrypted and decrypted by the same service(slicing service). Once the threshold for a certain slice hasbeen met, the data is decrypted and pushed into the delayedjob queue along with the corresponding slice information.When the preparation service consumes the data from thequeue, it fetches the compensation data corresponding tothe submission identifiers in each slice from the submissiondata store and decrypts it using the corresponding privatekey, Cpri, prior to copying the data to HDFS. Note that thisservice does not need to access the member attributes, andhence does not possess the corresponding private key, Mpri.

Key rotation: Our design supports a key rotation mecha-nism to rotate expired or compromised keys and re-encryptthe data if needed, and to perform periodic re-encryptionif needed. This process will be carried out from the ser-vice that has access to the private key, e.g., the slicing ser-vice would rotate keys for the member attributes, while thepreparation service would rotate keys for the compensationdata.

No single point of failure: By appropriate separation of ser-vices and using different sets of encryption/decryption keys

for different data components, our design ensures that thereis no single point of failure. Member privacy would be pre-served if there is any single point of breach to the services orthe data stores, for example, if any one system (e.g., HDFS,submission data store, threshold data store, slicing service,preparation service) were to be breached.

Infrastructure security: In addition to measures for memberdata privacy, we also focus on the security of the under-lying communication and storage infrastructure. The webinterfaces associated with compensation collection and in-sights serve traffic over HTTPS. We also encrypt the com-munication between LinkedIn’s web servers and datacenters(through intermediate layers) using TLS. We also secure thedata storage layers using a combination of encryption (ex-plained above) and authentication (e.g., only allow connec-tions to the data store from trusted services running on (pre-configured) trusted hosts).

3.4.1 Preventing Timestamp Based Inference AttacksWe next describe the mechanisms supported by our sys-

tem for preventing timestamp based inference attacks. Theseattacks rely on the ability to join the de-identified compen-sation data with data containing member identity, on thetimestamp information. For example, page view and otherevent logs could contain information on when a member ac-cessed the page associated with the compensation collectionweb interface. Hence, it is not desirable for the exact times-tamp to be retained with a submitted compensation entry inthe de-identified data. One possible mechanism is to perturbthe timestamp by adding a random time period drawn froma certain distribution. For example, we could add a ran-dom delay up to say, 48 hours to the true timestamp, andmake the data available for processing only after this delay.While simple and elegant to implement, this approach maynot provide sufficient protection when the submissions ar-rive infrequently in a cohort, for example, once every week.Hence, our design also supports another mechanism based onk-Anonymity wherein the modification to the timestamp is afunction of other submissions in a cohort. The key idea is todefine a hierarchy of timestamps (e.g., second → minute →hour → date → week → . . .), and generalize the timestampfor each compensation entry in a cohort to the granularity atwhich there are at least (k − 1) other entries with the sametimestamp value. However, since our approach needs to beincremental in practice, we can achieve this property by pro-cessing entries within a cohort in batches of size k, generaliz-ing to a common timestamp within each batch, and makingadditional data available for offline processing only in suchincremental batches. Processing in batches could also beused to achieve incremental privacy protection, since the in-sight for a cohort would not be sensitive to a single newsubmission but only batches of k or more submissions.

3.4.2 Protecting Against Over-representationThe quality of the compensation insights depends cru-

cially on ensuring that the submissions are representativeof the underlying cohort. Although we reach out to ran-domly sampled members from each cohort as discussed in§3.3.1, we need to also ensure that a member cannot submitthe compensation data multiple times, or too often to avoidover-representation of certain members. For this reason and

Figure 4: The distribution of compensation submis-sions as a function of time lapsed since the emailrequest

also for enabling the give-to-get model, we maintain a datastore containing when a member submitted compensationinformation, and query this store to constrain how oftena member can re-submit the compensation data. We canachieve this goal in different ways: (1) by limiting the fre-quency (e.g., at most once every 12 months), (2) by limitingthe frequency, customized for different industries/functionsby taking into account how often pay changes, how oftenpeople change jobs, etc, or (3) based on a change in a mem-ber’s profile (e.g., whenever the member adds a new jobposition, or updates the job description).

4. EXPERIMENTSWe next present an experimental study of the LinkedIn

Salary system, focusing on the the tradeoffs between privacyand modeling requirements.

4.1 Experimental SetupAs stated earlier, our system has been deployed in pro-

duction at LinkedIn, initially for collection of compensationdata from LinkedIn members and later for powering the pub-licly launched LinkedIn Salary product. We study the dis-tribution of responses over time and investigate the tradeoffsbetween privacy guarantees and data available for comput-ing compensation insights. Our experiments are performedusing more than one year of compensation submission his-tory data collected from over 1.5 million LinkedIn members,across three countries (USA, Canada, and UK). Note thatwe use just the submission history data, and not the com-pensation data itself for these experiments. Since LinkedInSalary uses a give-to-get model, we need to keep track of thesubmission history for each member (as discussed in §3.1).An extensive study and evaluation of the compensation in-sights and the associated statistical modeling components ispresented in [29].

4.2 Studying the Distribution of Compensa-tion Submissions

We first study how soon members respond upon receiv-ing the emails requesting them to provide the compensation

Figure 5: Relative percent of cohorts that are avail-able vs. the minimum threshold

information. Figure 4 shows the cumulative percentage ofresponses received as a function of the time lapsed. In thisfigure, the X-axis denotes the number of days that havelapsed since the email request. For each possible value dof the number of days lapsed, we compute the correspond-ing number of member submission responses, and therebyobtain the cumulative percentage of responses that were re-ceived within d days of email request, plotted in Y-axis. Weobserve that 80% of responses were received within one dayof email request, suggesting that batching together responsesto achieve greater privacy is not likely to lead to significantdelay or withholding of data for modeling purposes. Further,more than 85% of all responses were received within first 5days of email request. The responses took considerable timein certain cases: about 5% of responses were received after135 days!

4.3 Studying the Tradeoffs between Privacyand Modeling Needs

As discussed in §3.2, a cohort needs to contain a mini-mum number k of entries before the cohort and the asso-ciated compensation data is made available for offline dataprocessing. By varying k, we obtain a tradeoff between thelevel of privacy protection and the amount of data availablefor modeling. Figure 5 presents the relative percent of title-country-region cohorts that are available as a function of theminimum threshold, k. Note that we only used cohorts withat least 3 entries for this analysis. We observe that onlyabout 65% of the cohorts will be available with a thresholdof 5 and 38% of the cohorts with a threshold of 10, com-pared to a threshold of 3. In other words, out of the cohortswith at least 3 entries, about one-third have either 3 or 4entries each; slightly less than one-third have between 5 and9 entries each; and the rest have 10 or more entries each.

Next, we studied the effect of processing entries within acohort in batches of size k (in addition to requiring a mini-mum threshold of k), and generalizing to a common times-tamp within each batch. As discussed in §3.4.1, processingin batches helps to protect against timestamp based infer-ence attacks and to also achieve incremental privacy protec-tion. However, this results in some data being delayed or

Figure 6: Percent of data available vs. the batchsize

unavailable for offline processing. For example, if there arecurrently 8 submissions in a cohort and the batch size is 5,then the 6th, 7th, and 8th entries would not be made avail-able. Figure 6 shows the percent of data that is availableas a function of the batch size, k. For each k, we computethe number of entries available due to processing in batches(aggregated across cohorts) divided by the total number ofentries (aggregated across cohorts). Again, we only consid-ered title-country-region cohorts that have at least 3 sub-missions. We see that about 4.5% of the submissions wouldbe withheld with a batch size of k = 3 (compared to none inthe absence of batching, since we only consider cohorts withat least 3 entries), about 13% of the submissions would bewithheld with a batch size of k = 5, and about 24% of thesubmissions would be withheld with a batch size of k = 10.

We also investigated the actual delays that could emergeas a result of processing data in batches. Our results arepresented in Figures 7–11. We calculated these delays asfollows. For each title-country-region cohort, we first com-pute the delays introduced for certain submissions as a resultof processing in batches, limiting to those submissions thatwould get processed. For example, if a cohort has 8 entriessubmitted on days 1, 2, . . . , 8 respectively, and the batch sizeis 5, then we would ignore the last three entries, and asso-ciate a delay of 4 days, 3 days, 2 days, 1 day, and 0 dayrespectively for the first five entries. We then sort the delayvalues within each cohort, and compute different percentiles(for instance, the median delay (50th percentile) would be2 days in the above example). For each batch size and eachpercentage, we aggregate the corresponding delay percentilevalues across all cohorts, and show the distribution usinga box-and-whisker plot. The median across all cohorts isshown as a red horizontal line, with the box boundaries cor-responding to q1 (25th percentile) and q3 (75th percentile)respectively. The ends of the whiskers correspond to thelower limit, computed as max(0, q1 − 1.5(q3 − q1)) and theupper limit, computed as q3 + 1.5(q3− q1) respectively. Thepoints outside this range are plotted as outliers (each with ared plus sign). For example, we can infer from Figure 8 that,for a batch size of 8, the 30th percentile delay values rangefrom 0 to 24 days for three-fourth of the cohorts, with a me-

Figure 7: 10th percentile of the delays introducedby batching

Figure 8: 30th percentile of the delays introducedby batching

dian of 4 days. Similarly, Figure 9 shows that the mediandelay for three-fourth of the cohorts is less than about fiveweeks for batch sizes up to 10, although the median delaycould be as high as 300 to 400 days for a few outlier cohortsthat receive relatively fewer and infrequent submissions.

5. LESSONS LEARNED IN PRACTICEOur architecture and design choices evolved over the course

of the project, by taking into account various practical con-siderations. The product design and user interface has beenevolving over time based on feedback from end users as wellas focus group studies, and consequently, our architecturehad to be designed to enable rapid iterations, for example,through easy creation and modification of cohort types, andflexible design of key-value stores. From time to time, wealso performed “bootstrapping” to regenerate the historicaloutput data due to changes in some system components, e.g.,due to software bugs in the system implementation, changesto the various services that are invoked by the system, and

Figure 9: 50th percentile of the delays introducedby batching

Figure 10: 70th percentile of the delays introducedby batching

Figure 11: 90th percentile of the delays introducedby batching

changes to the product design such as addition of new typesof insights. Our privacy model choices were also influencedby the unique product needs as well as limitations of oursetting, as discussed in §6. As discussed in §3.2, we firstconsidered storing and processing the encrypted (but sensi-tive) data in LinkedIn’s Hadoop ecosystem, but chose thede-identification approach instead due to the security lim-itations of Hadoop and HDFS. During deployment of oursystem, we realized that it is not only necessary to have sep-arate services for the encryption of member attributes andthe compensation data, but also that these services need tobe deployed on different machines. Otherwise, an attackercould get access to a single machine containing both services,thereby being able to decrypt both the member attributedata and the compensation data, which defeats our goals.We used custom deployment tools to ensure that these ser-vices are deployed in different machines.

6. RELATED WORKSalary Information Products: There are several commer-

cial services offering information pertaining to compensationand benefits. For example, Glassdoor [7] offers a compara-ble service, while PayScale [9] collects individual salary sub-missions, offers free reports for detailed matches, and sellscompensation information to companies. The US Bureau ofLabor Statistics [8] publishes a variety of statistics on payand benefits.

Privacy: Preserving user privacy is important when col-lecting compensation data, considering its sensitive nature.There is rich literature in the field of privacy-preserving datamining spanning different research communities (e.g., [35,41, 10, 12, 22, 28, 42, 30, 11, 32, 31]), as well as on the limita-tions of simple anonymization techniques (e.g., [13, 33, 38]).Based on the lessons learned from the privacy literature, wefirst attempted to make use of rigorous privacy techniquessuch as differential privacy [18, 19] in our problem setting.However, we soon realized that these are not applicable inour context for the following reasons: (1) the amount of noiseto be added to the quantiles, histograms, and other insightswould be very large (thereby depriving the compensation in-sights of their reliability and usefulness), since the worst casesensitivity of these functions to any one user’s compensationdata could be large, and (2) the insights need to be providedon a continual basis with the arrival of new data points. Al-though there is theoretical work on applying differential pri-vacy under continual observations [16, 20], we have not comeacross any practical implementations or applications of thesetechniques. We also explored approaches similar to recentwork at Google [21] and Apple [24] on privacy-preservingdata collection at scale that focuses on applications such aslearning statistics about how unwanted software is hijackingusers’ settings in Chrome browser and discovering the us-age patterns of a large number of iOS users for improvingthe touch keyboard respectively. These approaches are (orseem to be) built on the concept of randomized response [43]and require response from typically hundreds of thousands ofusers for the results to be useful. In contrast, even the largerof our cohorts contain only a few thousand data points, andhence these approaches are not applicable in our setting.

Survey Techniques: There is extensive work on traditionalstatistical survey techniques [25, 27], as well on newer areassuch as web survey methodology [15]. See [3] for a survey

of non-response bias challenges, and [14] for an overview ofselection bias.

7. CONCLUSIONS AND FUTURE WORKWe presented the design and architecture of LinkedIn Salary,

a system for securely collecting compensation informationfrom LinkedIn members and providing structured compen-sation insights to job seekers. We highlighted unique chal-lenges associated with privacy and security, and describedhow we addressed them using techniques such as encryption,access control, de-identification, aggregation, and threshold-ing. We demonstrated the tradeoffs between privacy andmodeling needs through an experimental study with morethan one year of compensation submission history data col-lected from over 1.5 million LinkedIn members. We alsodiscussed the design decisions and tradeoffs while buildingour system, and the lessons learned from the production de-ployment of this system at LinkedIn.

An interesting direction to extend this work is to investi-gate approaches for improving the balance between privacyand modeling needs. For instance, we could explore the ap-plicability of provably privacy-preserving machine learningapproaches (e.g., [17, 34]) in our setting. Such explorationswould require a redesign of the system, wherein we performmodeling on a secure cluster of servers with access to produc-tion databases and build richer prediction models that makeuse of more discriminating features beyond those availablepost de-identification. We would also like to incorporate out-lier detection during submission stage by using user profileand behavioral features. Further research directions for im-proving the modeling components and for studying compen-sation data towards improving the efficiency of career mar-ketplace are discussed in [29]. More broadly, by presentingthe practical challenges encountered and the lessons learnedduring the design and deployment of privacy mechanisms foran emerging internet application, we hope to stimulate newresearch directions in privacy-aware computing.

AcknowledgmentsThe authors would like to thank all other members of LinkedInSalary team for their collaboration for deploying our systemas part of the launched product, and Stephanie Chou, TimConverse, Tushar Dalvi, Anthony Duerr, David Freeman,Joseph Florencio, Ashish Gupta, David Hardtke, Parul Jain,Prateek Janardhan, Santosh Kumar Kancha, Rong Rong,Ryan Sandler, Cory Scott, Ganesh Venkataraman, LiangZhang, and Lu Zheng for insightful feedback and discus-sions.

8. REFERENCES[1] Databus. https://github.com/linkedin/databus.

[2] Rest.li. https://github.com/linkedin/rest.li.

[3] Encyclopedia of Social Measurement, volume 2,chapter Non-Response Bias. Academic Press, 2005.

[4] BLS Handbook of Methods, chapter 3, OccupationalEmployment Statistics. U.S. Bureau of LaborStatistics, 2008.http://www.bls.gov/opub/hom/pdf/homch3.pdf.

[5] How to rethink the candidate experience and makebetter hires. CareerBuilder’s Candidate BehaviorStudy, 2016.

[6] Job seeker nation study. Jobvite, 2016.

[7] Glassdoor introduces salary estimates in job listings,February 2017. https://www.glassdoor.com/press/glassdoor-introduces-salary-estimates-job-listings-reveals-unfilled-jobs-272-billion/.

[8] Overview of BLS statistics on pay and benefits,February 2017. https://www.bls.gov/bls/wages.htm.

[9] Payscale data & methodology, February 2017.http://www.payscale.com/docs/default-source/

pdf/data_one_pager.pdf.

[10] N. R. Adam and J. C. Worthmann. Security-controlmethods for statistical databases: A comparativestudy. ACM Computing Surveys (CSUR), 21(4), 1989.

[11] G. Aggarwal, M. Bawa, P. Ganesan,H. Garcia-Molina, K. Kenthapadi, R. Motwani,U. Srivastava, D. Thomas, and Y. Xu. Two can keep asecret: A distributed architecture for secure databaseservices. In CIDR, 2005.

[12] R. Agrawal and R. Srikant. Privacy-preserving datamining. In ACM Sigmod Record, volume 29, 2000.

[13] L. Backstrom, C. Dwork, and J. Kleinberg. Whereforeart thou r3579x?: Anonymized social networks, hiddenpatterns, and structural steganography. In WWW,2007.

[14] J. Bethlehem. Selection bias in web surveys.International Statistical Review, 78(2), 2010.

[15] M. Callegaro, K. L. Manfreda, and V. Vehovar. Websurvey methodology. Sage, 2015.

[16] T.-H. H. Chan, E. Shi, and D. Song. Private andcontinual release of statistics. ACM Transactions onInformation and System Security, 14(3), 2011.

[17] K. Chaudhuri, C. Monteleoni, and A. D. Sarwate.Differentially private empirical risk minimization.Journal of Machine Learning Research, 12, 2011.

[18] C. Dwork, K. Kenthapadi, F. McSherry, I. Mironov,and M. Naor. Our data, ourselves: Privacy viadistributed noise generation. In EUROCRYPT, 2006.

[19] C. Dwork, F. McSherry, K. Nissim, and A. Smith.Calibrating noise to sensitivity in private dataanalysis. In TCC, 2006.

[20] C. Dwork, M. Naor, T. Pitassi, and G. N. Rothblum.Differential privacy under continual observation. InSTOC, 2010.

[21] U. Erlingsson, V. Pihur, and A. Korolova. RAPPOR:Randomized aggregatable privacy-preserving ordinalresponse. In CCS, 2014.

[22] A. Evfimievski, J. Gehrke, and R. Srikant. Limitingprivacy breaches in privacy preserving data mining. InPODS, 2003.

[23] R. T. Fielding. Architectural styles and the design ofnetwork-based software architectures. PhD thesis,University of California, Irvine, 2000.

[24] A. Greenberg. Apple’s ‘differential privacy’ is aboutcollecting your data – but not your data. Wired, June2016.

[25] R. M. Groves, F. J. Fowler Jr, M. P. Couper, J. M.Lepkowski, E. Singer, and R. Tourangeau. Surveymethodology. John Wiley & Sons, 2011.

[26] S. Harris. How to make the job market work like asupermarket. LinkedIn Pulse, 2016.https://www.linkedin.com/pulse/

how-make-job-market-work-like-supermarket-seth-harris.

[27] R. J. Jessen. Statistical survey techniques. John Wiley& Sons, 1978.

[28] M. Kantarcioglu and C. Clifton. Privacy-preservingdistributed mining of association rules on horizontallypartitioned data. IEEE transactions on knowledge anddata engineering, 16(9), 2004.

[29] K. Kenthapadi, S. Ambler, L. Zhang, and D. Agarwal.Bringing salary transparency to the world: Computingrobust compensation insights via LinkedIn Salary.LinkedIn Technical Report, available athttps://arxiv.org/abs/1703.09845, 2017.

[30] K. Kenthapadi, N. Mishra, and K. Nissim.Simulatable auditing. In PODS, 2005.

[31] N. Li, T. Li, and S. Venkatasubramanian. t-closeness:Privacy beyond k-anonymity and l-diversity. In ICDE,2007.

[32] A. Machanavajjhala, D. Kifer, J. Gehrke, andM. Venkitasubramaniam. l-diversity: Privacy beyondk-anonymity. ACM Transactions on KnowledgeDiscovery from Data (TKDD), 1(1), 2007.

[33] A. Narayanan and V. Shmatikov. Robustde-anonymization of large sparse datasets. In IEEESymposium on Security and Privacy, 2008.

[34] N. Papernot, M. Abadi, U. Erlingsson, I. Goodfellow,and K. Talwar. Semi-supervised knowledge transfer fordeep learning from private training data. In ICLR,2017.

[35] P. Samarati. Protecting respondents identities inmicrodata release. IEEE Transactions on Knowledgeand Data Engineering, 13(6), 2001.

[36] P. Samarati and L. Sweeney. Protecting privacy whendisclosing information: k-anonymity and itsenforcement through generalization and suppression.Technical report, Technical report, SRI International,1998.

[37] R. Sandler. Introducing “LinkedIn Salary”: Unlockyour earning potential. LinkedIn Blog, 2016.https://blog.linkedin.com/2016/11/02/

introducing-linkedin-salary-unlock-your-earning-potential.

[38] J. Su, A. Shukla, S. Goel, and A. Narayanan.De-anonymizing web browsing data with socialnetworks, 2017.

[39] R. Sumbaly, J. Kreps, L. Gao, A. Feinberg, C. Soman,and S. Shah. Serving large-scale batch computed datawith project Voldemort. In FAST, 2012.

[40] L. Sweeney. Uniqueness of simple demographics in theUS population. Technical report, Technical report,Carnegie Mellon University, 2000.

[41] L. Sweeney. k-anonymity: A model for protectingprivacy. International Journal of Uncertainty,Fuzziness and Knowledge-Based Systems, 10(05), 2002.

[42] J. Vaidya and C. Clifton. Privacy preservingassociation rule mining in vertically partitioned data.In KDD, 2002.

[43] S. L. Warner. Randomized response: A surveytechnique for eliminating evasive answer bias. Journalof the American Statistical Association, 60(309), 1965.

[44] J. Weiner. The future of LinkedIn and the EconomicGraph. LinkedIn Pulse, 2012.