DaSilvaMoorev.PublicisGroupe287F.R.D.182(S.D.N.Y.2012)

MONIQUEDASILVAMOORE,etal.,Plaintiffs,v.

PUBLICISGROUPE&MSLGROUP,Defendants.No.11Civ.1279(ALC)(AJP).

UnitedStatesDistrictCourt,S.D.NewYork.

February24,2012.

OPINIONANDORDER

ANDREWJ.PECK,MagistrateJudge.

InmyarticleSearch,Forward:Willmanualdocumentreviewandkeywordsearchesbereplacedbycomputer-assistedcoding?,Iwrote:

Tomyknowledge,noreportedcase(federalorstate)hasruledontheuseofcomputer-assistedcoding.Whileanecdotallyitappearsthatsomelawyersareusingpredictivecodingtechnology,italsoappearsthatmanylawyers(andtheirclients)arewaitingforajudicialdecisionapprovingofcomputer-assistedreview.

Perhapstheyarelookingforanopinionconcludingthat:"ItistheopinionofthiscourtthattheuseofpredictivecodingisaproperandacceptablemeansofconductingsearchesundertheFederalRulesofCivilProcedure,andfurthermorethatthesoftwareprovidedforthispurposeby[insertnameofyourfavoritevendor]isthesoftwareofchoiceinthiscourt."Ifso,itwillbealongwait.

....

Untilthereisajudicialopinionapproving(orevencritiquing)theuseofpredictivecoding,counselwilljusthavetorelyonthisarticleasasignofjudicialapproval.Inmyopinion,computer-assistedcodingshouldbeusedinthosecaseswhereitwillhelp"securethejust,speedy,andinexpensive"(Fed.R.Civ.P.1)determinationofcasesinoure-discoveryworld.

AndrewPeck,Search,Forward,L.Tech.News,Oct.2011,at25,29.Thisjudicialopinionnowrecognizesthatcomputer-assistedreviewisanacceptablewaytosearchforrelevantESIinappropriatecases.[1]

CASEBACKGROUND

Inthisaction,fivefemalenamedplaintiffsaresuingdefendantPublicisGroupe,"oneoftheworld's`bigfour'advertisingconglomerates,"anditsUnitedStatespublicrelationssubsidiary,defendantMSLGroup.(SeeDkt.No.4:Am.Compl.1,5,26-32.)Plaintiffsallegethatdefendantshavea"glassceiling"thatlimitswomentoentrylevelpositions,andthatthereis"systemic,company-widegenderdiscriminationagainstfemalePRemployeeslikePlaintiffs."(Am.Compl.4-6,8.)Plaintiffsallegethatthegenderdiscriminationincludes

(a)payingPlaintiffsandotherfemalePRemployeeslessthansimilarly-situatedmaleemployees;(b)failingtopromoteoradvancePlaintiffsandotherfemalePRemployeesatthesamerateassimilarly-situatedmaleemployees;and(c)carryingoutdiscriminatoryterminations,demotionsand/orjobreassignmentsoffemalePRemployeeswhenthecompanyreorganizeditsPRpracticebeginningin2008....

(Am.Compl.8.)

PlaintiffsassertclaimsforgenderdiscriminationunderTitleVII(andundersimilarNewYorkStateandNewYorkCitylaws)(Am.Compl.204-25),pregnancydiscriminationunderTitleVIIandrelatedviolationsoftheFamilyandMedicalLeaveAct(Am.Compl.239-71),aswellasviolationsoftheEqualPayActandFairLaborStandardsAct(andthesimilarNewYorkLaborLaw)(Am.Compl.226-38).

1of20

ThecomplaintseekstobringtheEqualPayAct/FLSAclaimsasa"collectiveaction"(i.e.,opt-in)onbehalfofall"current,former,andfuturefemalePRemployees"employedbydefendantsintheUnitedStates"atanytimeduringtheapplicableliabilityperiod"(Am.Compl.179-80,190-203),andasaclassactiononthegenderandpregnancydiscriminationclaimsandontheNewYorkLaborLawpayclaim(Am.Compl.171-98).Plaintiffs,however,havenotyetmovedforcollectiveactionorclasscertificationatthistime.

DefendantMSLdeniestheallegationsinthecomplaintandhasassertedvariousaffirmativedefenses.(SeegenerallyDkt.No.19:MSLAnswer.)DefendantPublicisischallengingtheCourt'sjurisdictionoverit,andthepartieshaveuntilMarch12,2012toconductjurisdictionaldiscovery.(SeeDkt.No.44:10/12/11Order.)

COMPUTER-ASSISTEDREVIEWEXPLAINED

MySearch,Forwardarticleexplainedmyunderstandingofcomputer-assistedreview,asfollows:

Bycomputer-assistedcoding,Imeantools(differentvendorsusedifferentnames)thatusesophisticatedalgorithmstoenablethecomputertodeterminerelevance,basedoninteractionwith(i.e.,trainingby)ahumanreviewer.

Unlikemanualreview,wherethereviewisdonebythemostjuniorstaff,computer-assistedcodinginvolvesaseniorpartner(or[small]team)whoreviewandcodea"seedset"ofdocuments.Thecomputeridentifiespropertiesofthosedocumentsthatitusestocodeotherdocuments.Astheseniorreviewercontinuestocodemoresampledocuments,thecomputerpredictsthereviewer'scoding.(Or,thecomputercodessomedocumentsandaskstheseniorreviewerforfeedback.)

Whenthesystem'spredictionsandthereviewer'scodingsufficientlycoincide,thesystemhaslearnedenoughtomakeconfidentpredictionsfortheremainingdocuments.Typically,theseniorlawyer(orteam)needstoreviewonlyafewthousanddocumentstotrainthecomputer.

Somesystemsproduceasimpleyes/noastorelevance,whileothersgivearelevancescore(say,ona0to100basis)thatcounselcanusetoprioritizereview.Forexample,ascoreabove50mayproduce97%oftherelevantdocuments,butconstitutesonly20%oftheentiredocumentset.

Counselmaydecide,aftersamplingandqualitycontroltests,thatdocumentswithascoreofbelow15aresohighlylikelytobeirrelevantthatnofurtherhumanreviewisnecessary.Counselcanalsodecidethecost-benefitofmanualreviewofthedocumentswithscoresof15-50.

AndrewPeck,Search,Forward,L.Tech.News,Oct.2011,at25,29.[2]

MyarticlefurtherexplainedmybeliefthatDaubertwouldnotapplytotheresultsofusingpredictivecoding,butthatinanychallengetoitsuse,thisJudgewouldbeinterestedinboththeprocessusedandtheresults:

[I]ftheuseofpredictivecodingischallengedinacasebeforeme,Iwillwanttoknowwhatwasdoneandwhythatproduceddefensibleresults.Imaybelessinterestedinthesciencebehindthe"blackbox"ofthevendor'ssoftwarethaninwhetheritproducedresponsivedocumentswithreasonablyhighrecallandhighprecision.

Thatmaymeanallowingtherequestingpartytoseethedocumentsthatwereusedtotrainthecomputer-assistedcodingsystem.(Counselwouldnotberequiredtoexplainwhytheycodeddocumentsasresponsiveornon-responsive,justwhatthecodingwas.)Proofofavalid"process,"includingqualitycontroltesting,alsowillbeimportant.

....

Ofcourse,thebestapproachtotheuseofcomputer-assistedcodingistofollowtheSedonaCooperationProclamationmodel.Adviseopposingcounselthatyouplantousecomputer-assistedcodingandseekagreement;ifyoucannot,considerwhethertoabandonpredictivecodingforthatcaseorgotothecourtforadvanceapproval.

Id.

THEESIDISPUTESINTHISCASEANDTHEIRRESOLUTION

2of20

AfterseveraldiscoveryconferencesandrulingsbyJudgeSullivan(thethen-assignedDistrictJudge),hereferredthecasetomeforgeneralpretrialsupervision.(Dkt.No.48:11/28/11ReferralOrder.)Atmyfirstdiscoveryconferencewiththeparties,bothparties'counselmentionedthattheyhadbeendiscussingan"electronicdiscoveryprotocol,"andMSL'scounselstatedthatanopenissuewas"plaintiff'sreluctancetoutilizepredictivecodingtotrytoculldownthe"approximatelythreemillionelectronicdocumentsfromtheagreed-uponcustodians.(Dkt.No.51:12/2/11Conf.Tr.at7-8.)[3]Plaintiffs'counselclarifiedthatMSLhad"oversimplified[plaintiffs']stanceonpredictivecoding,"i.e.,thatitwasnotopposedbuthad"multipleconcerns...onthewayinwhich[MSL]plantoemploypredictivecoding"andplaintiffswanted"clarification."(12/2/11Conf.Tr.at21.)

TheCourtdidnotrulebutofferedthepartiesthefollowingadvice:

Now,ifyouwantanymoreadvice,forbetterorforworseontheESIplanandwhetherpredictivecodingshouldbeused,...Iwillsayrightnow,whatshouldnotbeasurprise,IwroteanarticleintheOctoberLawTechnologyNewscalledSearchForward,whichsayspredictivecodingshouldbeusedintheappropriatecase.

Isthistheappropriatecaseforit?Youalltalkaboutitsomemore.Andifyoucan'tfigureitout,youaregoingtogetbackinfrontofme.Keywords,certainlyunlesstheyarewelldoneandtested,arenotoverlyuseful.Keywordsalongwithpredictivecodingandothermethodology,canbeveryinstructive.

I'malsosayingtothedefendantswhomay,fromthecommentbefore,havereadmyarticle.Ifyoudopredictivecoding,youaregoingtohavetogiveyourseedset,includingtheseeddocumentsmarkedasnonresponsivetotheplaintiff'scounselsotheycansay,well,ofcourseyouarenotgettingany[relevant]documents,you'renotappropriatelytrainingthecomputer.

(12/2/11Conf.Tr.at20-21.)TheDecember2,2011conferenceadjournedwiththepartiesagreeingtofurtherdiscusstheESIprotocol.(12/2/11Conf.Tr.at34-35.)

TheESIissuewasnextdiscussedataconferenceonJanuary4,2012.(Dkt.No.71:1/4/12Conf.Tr.)Plaintiffs'ESIconsultantconcededthatplaintiffs"havenottakenissuewiththeuseofpredictivecodingor,frankly,withtheconfidencelevelsthatthey[MSL]haveproposed...."(1/4/12Conf.Tr.at51.)Rather,plaintiffstookissuewithMSL'sproposalthatafterthecomputerwasfullytrainedandtheresultsgenerated,MSLwantedtoonlyreviewandproducethetop40,000documents,whichitestimatedwouldcost$200,000(at$5perdocument).(1/4/12Conf.Tr.at47-48,51.)TheCourtrejectedMSL's40,000documentsproposalasa"piginapoke."(1/4/12Conf.Tr.at51-52.)TheCourtexplainedthat"where[the]linewillbedrawn[astoreviewandproduction]isgoingtodependonwhatthestatisticsshowfortheresults,"since"[p]roportionalityrequiresconsiderationofresultsaswellascosts.Andifstoppingat40,000isgoingtoleaveatremendousnumberoflikelyhighlyresponsivedocumentsunproduced,[MSL'sproposedcutoff]doesn'twork."(1/4/12Conf.Tr.at51-52;seealsoid.at57-58;Dkt.No.88:2/8/12Conf.Tr.at84.)ThepartiesagreedtofurtherdiscussandfinalizetheESIprotocolbylateJanuary2012,withaconferenceheldonFebruary8,2012.(1/4/12Conf.Tr.at60-66;see2/8/12Conf.Tr.)

Custodians

ThefirstissueregardingtheESIprotocolinvolvedtheselectionofwhichcustodians'emailswouldbesearched.MSLagreedtothirtycustodiansfora"firstphase."(Dkt.No.88:2/8/12Conf.Tr.at23-24.)MSL'scustodianlistincludedthepresidentandothermembersofMSL's"executiveteam,"mostofitsHRstaffandanumberofmanagingdirectors.(2/8/12Conf.Tr.at24.)

Plaintiffssoughttoincludeasadditionalcustodianssevenmale"comparators,"explainingthatthecomparators'emailswereneededinordertofindinformationabouttheirjobdutiesandhowtheirdutiescomparedtoplaintiffs'jobduties.(2/8/12Conf.Tr.at25-27.)Plaintiffsgaveanexampleofthemenbeinggivengreater"clientcontact"orhavingbetterjobassignments.(2/8/12Conf.Tr.at28-30.)TheCourtheldthatthesearchofthecomparators'emailswouldbesodifferentfromthatoftheothercustodiansthatthecomparatorsshouldnotbeincludedintheemailssubjectedtopredictivecodingreview.(2/8/12Conf.Tr.at28,30.)Asafallbackposition,plaintiffsproposedto"treatthecomparatorsasaseparatesearch,"buttheCourtfoundthatplaintiffscouldnotdescribeinanymeaningfulwayhowtheywouldsearchthecomparators'emails,evenasaseparatesearch.(2/8/12Conf.Tr.at30-31.)Sincetheplaintiffslikelycoulddeveloptheinformationneededthroughdepositionsofthecomparators,theCourtruledthatthecomparators'emailswouldnotbeincludedinphaseone.(2/8/12Conf.Tr.at31.)

PlaintiffsalsosoughttoincludeMSL'sCEO,OlivierFleuriot,locatedinFranceandwhoseemailsweremostly

3of20

writteninFrench.(2/8/12Conf.Tr.at32-34.)TheCourtconcludedthatbecausehisemailswiththeNewYorkbasedexecutivestaffwouldbegatheredfromthosecustodians,andFleuriot'semailsstoredinFrancelikelywouldbecoveredbytheFrenchprivacyandblockinglaws,[4]Fleuriotshouldnotbeincludedasafirst-phasecustodian.(2/8/12Conf.Tr.at35.)

PlaintiffssoughttoincludecertainmanagingdirectorsfromMSLofficesatwhichnonamedplaintiffworked.(2/8/12Conf.Tr.at36-37.)TheCourtruledthatsinceplaintiffshadnotyetmovedforcollectiveactionstatusorclasscertification,untilthemotionsweremadeandgranted,discoverywouldbelimitedtooffices(andmanagingdirectors)wherethenamedplaintiffshadworked.(2/8/12Conf.Tr.at37-39.)

Thefinalissueraisedbyplaintiffsrelatedtothephasingofcustodiansandthediscoverycutoffdates.MSLproposedfinishingphase-onediscoverycompletelybeforeconsideringwhattodoaboutasecondphase.(See2/8/12Conf.Tr.at36.)Plaintiffsexpressedconcernthattherewouldnotbetimefortwoseparatephases,essentiallyseekingtomovethephase-twocustodiansbackintophaseone.(2/8/12Conf.Tr.at35-36.)TheCourtfoundMSL'sseparatephaseapproachtobemoresensibleandnotedthatifnecessary,theCourtwouldextendthediscoverycutofftoallowthepartiestopursuediscoveryinphases.(2/8/12Conf.Tr.at36,50.)

SourcesofESI

ThepartiesagreedoncertainESIsources,includingthe"EMCSourceOne[Email]Archive,"the"PeopleSoft"humanresourcesinformationmanagementsystemandcertainothersourcesincludingcertainHR"shared"folders.(SeeDkt.No.88:2/8/12Conf.Tr.at44-45,50-51.)Astoother"shared"folders,neithersidewasabletoexplainwhetherthefoldersmerelycontainedformsandtemplatesorcollaborativeworkingdocuments;theCourtthereforeleftthosesharedfoldersforphasetwounlessthepartiespromptlyprovidedinformationaboutlikelycontents.(2/8/12Conf.Tr.at47-48.)

TheCourtnotedthatbecausethenamedplaintiffsworkedforMSL,plaintiffsshouldhavesomeideawhatadditionalESIsources,ifany,likelyhadrelevantinformation;sincetheCourtneededtoconsiderproportionalitypursuanttoRule26(b)(2)(C),plaintiffsneededtoprovidemoreinformationtotheCourtthantheyweredoingiftheywantedtoaddadditionaldatasourcesintophaseone.(2/8/12Conf.Tr.at49-50.)TheCourtalsonotedthatwhereplaintiffsweregettingfactualinformationfromonesource(e.g.,payinformation,promotions,etc.),"therehastobealimittoredundancy"tocomplywithRule26(b)(2)(C).(2/8/12Conf.Tr.at54.)[5]

ThePredictiveCodingProtocol

Thepartiesagreedtousea95%confidencelevel(plusorminustwopercent)tocreatearandomsampleoftheentireemailcollection;thatsampleof2,399documentswillbereviewedtodeterminerelevant(andnotrelevant)documentsfora"seedset"tousetotrainthepredictivecodingsoftware.(Dkt.No.88:2/8/12Conf.Tr.at59-61.)AnareaofdisagreementwasthatMSLreviewedthe2,399documentsbeforethepartiesagreedtoaddtwoadditionalconceptgroups(i.e.,issuetags).(2/8/12Conf.Tr.at62.)MSLsuggestedthatsinceithadagreedtoprovideall2,399documents(andMSL'scodingofthem)toplaintiffsfortheirreview,plaintiffscancodethemforthenewissuetags,andMSLwillincorporatethatcodingintothesystem.(2/8/12Conf.Tr.at64.)Plaintiffs'vendoragreedtothatapproach.(2/8/12Conf.Tr.at64.)

Tofurthercreatetheseedsettotrainthepredictivecodingsoftware,MSLcodedcertaindocumentsthrough"judgmentalsampling."(2/8/12Conf.Tr.at64.)TheremainderoftheseedsetwascreatedbyMSLreviewing"keyword"searcheswithBooleanconnectors(suchas"trainingandDaSilvaMoore,"or"promotionandDaSilvaMoore")andcodingthetopfiftyhitsfromthosesearches.(2/8/12Conf.Tr.at64-66,72.)MSLagreedtoprovideallthosedocuments(exceptprivilegedones)toplaintiffsforplaintiffstoreviewMSL'srelevancecoding.(2/8/12Conf.Tr.at66.)Inaddition,plaintiffsprovidedMSLwithcertainotherkeywords,andMSLusedthesameprocesswithplaintiffs'keywordsaswiththeMSLkeywords,reviewingandcodinganadditional4,000documents.(2/8/12Conf.Tr.at68-69,71.)Allofthisreviewtocreatetheseedsetwasdonebyseniorattorneys(notparalegals,staffattorneysorjuniorassociates).(2/8/12Conf.Tr.at92-93.)MSLreconfirmedthat"[a]llofthedocumentsthatarereviewedasafunctionoftheseedset,whether[they]areultimatelycodedrelevantorirrelevant,asidefromprivilege,willbeturnedoverto"plaintiffs.(2/8/12Conf.Tr.at73.)

Thenextareaofdiscussionwastheiterativeroundstostabilizethetrainingofthesoftware.MSL'svendor'spredictivecodingsoftwareranksdocumentsonascoreof100tozero,i.e.,frommostlikelyrelevanttoleastlikelyrelevant.(2/8/12Conf.Tr.at70.)MSLproposedusingseveniterativerounds;ineachroundtheywouldreviewatleast500documentsfromdifferentconceptclusterstoseeifthecomputerisreturningnewrelevantdocuments.(2/8/12Conf.Tr.at73-74.)Aftertheseventhround,todetermineifthecomputeriswell

4of20

trainedandstable,MSLwouldreviewarandomsample(of2,399documents)fromthediscards(i.e.,documentscodedasnon-relevant)tomakesurethedocumentsdeterminedbythesoftwaretonotberelevantdonot,infact,containhighly-relevantdocuments.(2/8/12Conf.Tr.at74-75.)Foreachofthesevenroundsandthefinalquality-checkrandomsample,MSLagreedthatitwouldshowplaintiffsallthedocumentsitlookedatincludingthosedeemednotrelevant(exceptforprivilegeddocuments).(2/8/12Conf.Tr.at76.)

Plaintiffs'vendornotedthat"wedon'tatthispointagreethatthisisgoingtowork.Thisisnewtechnologyandithastobeprovenout."(2/8/12Conf.Tr.at75.)Plaintiffs'vendoragreed,ingeneral,thatcomputer-assistedreviewworks,andworksbetterthanmostalternatives.(2/8/12Conf.Tr.at76.)Indeed,plaintiffs'vendornotedthat"itisfairtosay[that]wearebigproponentsofit."(2/8/12Conf.Tr.at76.)TheCourtremindedthepartiesthatcomputer-assistedreview"worksbetterthanmostofthealternatives,ifnotallofthe[present]alternatives.Sotheideaisnottomakethisperfect,it'snotgoingtobeperfect.Theideaistomakeitsignificantlybetterthanthealternativeswithoutnearlyasmuchcost."(2/8/12Conf.Tr.at76.)

TheCourtacceptedMSL'sproposalfortheseveniterativereviews,butwiththefollowingcaveat:

Butifyougettotheseventhroundand[plaintiffs]aresayingthatthecomputerisstilldoingweirdthings,it'snotstabilized,etc.,weneedtodoanotherroundortwo,eitheryouwillagreetothatoryouwillbothcomeinwiththeappropriateQCinformationandeverythingelseand[maybeorderedto]doanotherroundortwoorfiveor500orwhateverittakestostabilizethesystem.

(2/8/12Conf.Tr.at76-77;seealsoid.at83-84,88.)

OnFebruary17,2012,thepartiessubmittedtheir"final"ESIProtocolwhichtheCourt"soordered."(Dkt.No.92:2/17/12ESIProtocol&Order.)[6]BecausethisisthefirstOpiniondealingwithpredictivecoding,theCourtannexesheretoasanExhibittheprovisionsoftheESIProtocoldealingwiththepredictivecodingsearchmethodology.

OBSERVATIONSONPLAINTIFF'SOBJECTIONSTOTHECOURT'SRULINGS

OnFebruary22,2012,plaintiffsfiledobjectionstotheCourt'sFebruary8,2012rulings.(Dkt.No.93:Pls.Rule72(a)Objections;seealsoDkt.No.94:NurhusseinAff.;Dkt.No.95:NealeAff.)WhilethoseobjectionsarebeforeDistrictJudgeCarter,afewcommentsareinorder.

Plaintiffs'RelianceonRule26(g)(1)(A)isErroneous

Plaintiffs'objectionstomyFebruary8,2012rulingsassertthatmyacceptanceofMSL'spredictivecodingapproach"providesunlawful`cover'forMSL'scounsel,whohasadutyunderFRCP26(g)to`certify'thattheirclient'sdocumentproductionis`complete'and`correct'asofthetimeitwasmade.FRCP26(g)(1)(A)."(Dkt.No.93:Pls.Rule72(a)Objectionsat8n.7;accord,id.at2.)Inlarge-datacaseslikethis,involvingoverthreemillionemails,nolawyerusinganysearchmethodcouldhonestlycertifythatitsproductionis"complete"—butmoreimportantly,Rule26(g)(1)doesnotrequirethat.PlaintiffssimplymisreadRule26(g)(1).ThecertificationrequiredbyRule26(g)(1)applies"withrespecttoadisclosure."Fed.R.Civ.P.26(g)(1)(A)(emphasisadded).Thatisatermofart,referringtothemandatoryinitialdisclosuresrequiredbyRule26(a)(1).SincetheRule26(a)(1)disclosureisinformation(witnesses,exhibits)that"thedisclosingpartymayusetosupportitsclaimsordefenses,"andfailuretoprovidesuchinformationleadstovirtuallyautomaticpreclusion,seeFed.RCiv.P.37(c)(1),itisappropriatefortheRule26(g)(1)(A)certificationtorequiredisclosuresbe"completeandcorrect."

Rule26(g)(1)(B)istheprovisionthatappliestodiscoveryresponses.Itdoesnotcallforcertificationthatthediscoveryresponseis"complete,"butratherincorporatestheRule26(b)(2)(C)proportionalityprinciple.Thus,Rule26(g)(1)(A)hasabsolutelynothingtodowithMSL'sobligationstorespondtoplaintiffs'discoveryrequests.Plaintiffs'argumentisbasedonamisunderstandingofRule26(g)(1).[7]

Rule702andDaubertAreNotApplicabletoDiscoverySearchMethods

Plaintiffs'objectionsalsoarguethatmyacceptanceofMSL'spredictivecodingprotocol"iscontrarytoFederalRuleofEvidence702"and"violatesthegatekeepingfunctionunderlyingRule702."(Dkt.No.93:Pls.Rule72(a)Objectionsat2-3;accord,id.at10-12.)[8]

FederalRuleofEvidence702andtheSupremeCourt'sDaubertdecision[9]dealwiththetrialcourt'sroleasgatekeepertoexcludeunreliableexperttestimonyfrombeingsubmittedtothejuryattrial.SeealsoAdvisoryComm.NotestoFed.R.Evid.702.Itisaruleforadmissibilityofevidenceattrial.

5of20

IfMSLsoughttohaveitsexperttestifyattrialandintroducetheresultsofitsESIprotocolintoevidence,DaubertandRule702wouldapply.Here,incontrast,thetensofthousandsofemailsthatwillbeproducedindiscoveryarenotbeingofferedintoevidenceattrialastheresultofascientificprocessorotherwise.Theadmissibilityofspecificemailsattrialwilldependuponeachemailitself(forexample,whetheritishearsay,orabusinessrecordorpartyadmission),nothowitwasfoundduringdiscovery.

Rule702andDaubertsimplyarenotapplicabletohowdocumentsaresearchedforandfoundindiscovery.

Plaintiffs'ReliabilityConcernsAre,AtBest,Premature

Finally,plaintiffs'objectionsassertthat"MSL'smethodlacksthenecessarystandardsforassessingwhetheritsresultsareaccurate;inotherwords,thereisnowaytobecertainifMSL'smethodisreliable."(Dkt.No.93:Pls.Rule72(a)Objectionsat13-18.)Plaintiffs'concernsmaybeappropriateforresolutionduringoraftertheprocess(whichtheCourtwillbecloselysupervising),butareprematurenow.Forexample,plaintiffscomplainthat"MSL'smethodfailstoincludeanagreed-uponstandardofrelevancethatistransparentandaccessibletoallparties....Withoutthisstandard,thereisahigh-likelihoodofdelayasthepartiesresolvedisputeswithregardtoindividualdocumentsonacase-by-casebasis."(Id.at14.)Relevanceisdeterminedbyplaintiffs'documentdemands.Asstatisticsshow,perhapsonly5%ofthedisagreementamongreviewerscomesfromclosequestionsofrelevance,asopposedtoreviewererror.(Seepage18n.11below.)TheissueregardingrelevancestandardsmightbesignificantifMSL'sproposalwasnottotallytransparent.Here,however,plaintiffswillseehowMSLhascodedeveryemailusedintheseedset(bothrelevantandnotrelevant),andtheCourtisavailabletoquicklyresolveanyissues.

Plaintiffscomplaintheycannotdetermineif"MSL'smethodactuallyworks"becauseMSLdoesnotdescribehowmanyrelevantdocumentsarepermittedtobelocatedinthefinalrandomsampleofdocumentsthesoftwaredeemedirrelevant.(Pls.Rule72(a)Objectionsat15-16.)Plaintiffsarguethat"withoutanydecisionaboutthismadeinadvance,theCourtissimplykickingthecandowntheroad."(Id.at16.)Inordertodetermineproportionality,itisnecessarytohavemoreinformationthantheparties(ortheCourt)nowhas,includinghowmanyrelevantdocumentswillbeproducedandatwhatcosttoMSL.Willthecaseremainlimitedtothenamedplaintiffs,orwillplaintiffsseekandobtaincollectiveactionand/orclassactioncertification?Inthefinalsampleofdocumentsdeemedirrelevant,areanyrelevantdocumentsfoundthatare"hot,""smokinggun"documents(i.e.,highlyrelevant)?Oraretheonlyrelevantdocumentsmoreofthesamething?Onehotdocumentmayrequirethesoftwaretobere-trained(orsomeothersearchmethodemployed),whileseveraldocumentsthatreallydonotaddanythingtothecasemightnotmatter.Thesetypesofquestionsarebetterdecided"downtheroad,"whenrealinformationisavailabletothepartiesandtheCourt.

FURTHERANALYSISANDLESSONSFORTHEFUTURE

Thedecisiontoallowcomputer-assistedreviewinthiscasewasrelativelyeasy—thepartiesagreedtoitsuse(althoughdisagreedabouthowbesttoimplementsuchreview).TheCourtrecognizesthatcomputer-assistedreviewisnotamagic,Staples-Easy-Button,solutionappropriateforallcases.Thetechnologyexistsandshouldbeusedwhereappropriate,butitisnotacaseofmachinereplacinghumans:itistheprocessusedandtheinteractionofmanandmachinethatthecourtsneedstoexamine.

Theobjectiveofreviewinediscoveryistoidentifyasmanyrelevantdocumentsaspossible,whilereviewingasfewnon-relevantdocumentsaspossible.Recallisthefractionofrelevantdocumentsidentifiedduringareview;precisionisthefractionofidentifieddocumentsthatarerelevant.Thus,recallisameasureofcompleteness,whileprecisionisameasureofaccuracyorcorrectness.Thegoalisforthereviewmethodtoresultinhigherrecallandhigherprecisionthananotherreviewmethod,atacostproportionatetothe"value"ofthecase.See,e.g.,MauraR.Grossman&GordonV.Cormack,Technology-AssistedReviewinE-DiscoveryCanBeMoreEffectiveandMoreEfficientThanExhaustiveManualReview,Rich.J.L.&Tech.,Spring2011,at8-9,availableathttp://jolt.richmond.edu/v17i3/article11.pdf.

Theslightlymoredifficultcasewouldbewheretheproducingpartywantstousecomputer-assistedreviewandtherequestingpartyobjects.[10]Thequestiontoaskinthatsituationiswhatmethodologywouldtherequestingpartysuggestinstead?Linearmanualreviewissimplytooexpensivewhere,ashere,thereareoverthreemillionemailstoreview.Moreover,whilesomelawyersstillconsidermanualreviewtobethe"goldstandard,"thatisamyth,asstatisticsclearlyshowthatcomputerizedsearchesareatleastasaccurate,ifnotmoreso,thanmanualreview.HerbRoitblatt,AnneKershaw,andPatrickOotoftheElectronicDiscoveryInstituteconductedanempiricalassessmentto"answerthequestionofwhethertherewasabenefittoengaginginatraditionalhumanrevieworwhethercomputersystemscouldbereliedontoproducecomparableresults,"andconcludedthat"[o]neverymeasure,theperformanceofthetwocomputersystemswasatleastasaccurate(measuredagainsttheoriginalreview)asthatofhumanre-review."HerbertL.

6of20

Roitblatt,AnneKershaw&PatrickOot,DocumentCategorizationinLegalElectronicDiscovery:ComputerClassificationvs.ManualReview,61J.Am.Soc'yforInfo.Sci.&Tech.70,79(2010).[11]

Likewise,Wachtell,Lipton,Rosen&KatzlitigationcounselMauraGrossmanandUniversityofWaterlooprofessorGordonCormack,studieddatafromtheTextRetrievalConferenceLegalTrack(TREC)andconcludedthat:"[T]hemyththatexhaustivemanualreviewisthemosteffective—andthereforethemostdefensible—approachtodocumentreviewisstronglyrefuted.Technology-assistedreviewcan(anddoes)yieldmoreaccurateresultsthanexhaustivemanualreview,withmuchlowereffort."MauraR.Grossman&GordonV.Cormack,Technology-AssistedReviewinE-DiscoveryCanBeMoreEffectiveandMoreEfficientThanExhaustiveManualReview,Rich.J.L.&Tech.,Spring2011,at48.[12]Thetechnology-assistedreviewsintheGrossman-Cormackarticlealsodemonstratedsignificantcostsavingsovermanualreview:"Thetechnology-assistedreviewsrequire,onaverage,humanreviewofonly1.9%ofthedocuments,afifty-foldsavingsoverexhaustivemanualreview."Id.at43.

BecauseofthevolumeofESI,lawyersfrequentlyhaveturnedtokeywordsearchestocullemail(orotherESI)downtoamoremanageablevolumeforfurthermanualreview.KeywordshaveaplaceinproductionofESI—indeed,thepartieshereusedkeywordsearches(withBooleanconnectors)tofinddocumentsfortheexpandedseedsettotrainthepredictivecodingsoftware.Intoomanycases,however,thewaylawyerschoosekeywordsistheequivalentofthechild'sgameof"GoFish."[13]Therequestingpartyguesseswhichkeywordsmightproduceevidencetosupportitscasewithouthavingmuch,ifany,knowledgeoftherespondingparty's"cards"(i.e.,theterminologyusedbytherespondingparty'scustodians).Indeed,therespondingparty'scounseloftendoesnotknowwhatisinitsownclient's"cards."

Anotherproblemwithkeywordsisthattheyoftenareover-inclusive,thatis,theyfindresponsivedocumentsbutalsolargenumbersofirrelevantdocuments.Inthiscase,forexample,akeywordsearchfor"training"resultedin165,208hits;DaSilvaMoore'snameresultedin201,179hits;"bonus"resultedin40,756hits;"compensation"resultedin55,602hits;and"diversity"resultedin38,315hits.(Dkt.No.92:2/17/12ESIProtocolEx.A.)IfMSLhadtomanuallyreviewallofthekeywordhits,manyofwhichwouldnotberelevant(i.e.,wouldbefalsepositives),itwouldbequitecostly.

Moreover,keywordsearchesusuallyarenotveryeffective.In1985,scholarsDavidBlairandM.Maroncollected40,000documentsfromaBayAreaRapidTransitaccident,andinstructedexperiencedattorneyandparalegalsearcherstousekeywordsandotherreviewtechniquestoretrieveatleast75%ofthedocumentsrelevantto51documentrequests.DavidL.Blair&M.E.Maron,AnEvaluationofRetrievalEffectivenessforaFull-TextDocument-RetrievalSystem,28Comm.ACM289(1985).Searchersbelievedtheymetthegoals,buttheiraveragerecallwasjust20%.Id.ThisresulthasbeenreplicatedintheTRECLegalTrackstudiesoverthepastfewyears.

Judicialdecisionshavecriticizedspecifickeywordsearches.Importantearlydecisionsinthisareacamefromtwooftheleadingjudicialscholarsinediscovery,MagistrateJudgesJohnFacciola(DistrictofColumbia)andPaulGrimm(Maryland).SeeUnitedStatesv.O'Keefe,37F.Supp.2d14,24(D.D.C.2008)(Facciola,M.J.);EquityAnalytics,LLCv.Lundin,248F.R.D.331,333(D.D.C.2008) (Facciola,M.J.);VictorStanley,Inc.v.CreativePipe,Inc.,250F.R.D.251,260,262(D.Md.2008)(Grimm,M.J.).IfollowedtheirleadwithWillaimA.GrossConstructionAssociates,Inc.,whenIwrote:

ThisOpinionshouldserveasawake-upcalltotheBarinthisDistrictabouttheneedforcarefulthought,qualitycontrol,testing,andcooperationwithopposingcounselindesigningsearchtermsor"keywords"tobeusedtoproduceemailsorotherelectronicallystoredinformation("ESI").

....

ElectronicdiscoveryrequirescooperationbetweenopposingcounselandtransparencyinallaspectsofpreservationandproductionofESI.Moreover,wherecounselareusingkeywordsearchesforretrievalofESI,theyataminimummustcarefullycrafttheappropriatekeywords,withinputfromtheESI'scustodiansastothewordsandabbreviationstheyuse,andtheproposedmethodologymustbequalitycontroltestedtoassureaccuracyinretrievalandeliminationof"falsepositives."ItistimethattheBar—eventhoselawyerswhodidnotcomeofageinthecomputerera—understandthis.

WilliamA.GrossConstr.Assocs.,Inc.v.Am.Mfrs.Mut.Ins.Co.,256F.R.D.134,134,136(S.D.N.Y.2009) (Peck,M.J.).

Computer-assistedreviewappearstobebetterthantheavailablealternatives,andthusshouldbeusedinappropriatecases.WhilethisCourtrecognizesthatcomputer-assistedreviewisnotperfect,theFederal

7of20

RulesofCivilProceduredonotrequireperfection.See,e.g.,PensionComm.ofUniv.ofMontrealPensionPlanv.BancofAm.Sec.,685F.Supp.2d456,461(S.D.N.Y.2010).CourtsandlitigantsmustbecognizantoftheaimofRule1,to"securethejust,speedy,andinexpensivedetermination"oflawsuits.Fed.R.Civ.P.1.ThatgoalisfurtherreinforcedbytheproportionalitydoctrinesetforthinRule26(b)(2)(C),whichprovidesthat:

Onmotionoronitsown,thecourtmustlimitthefrequencyorextentofdiscoveryotherwiseallowedbytheserulesorbylocalruleifitdeterminesthat:

(i)thediscoverysoughtisunreasonablycumulativeorduplicative,orcanbeobtainedfromsomeothersourcethatismoreconvenient,lessburdensome,orlessexpensive;

(ii)thepartyseekingdiscoveryhashadampleopportunitytoobtaintheinformationbydiscoveryintheaction;or

(iii)theburdenorexpenseoftheproposeddiscoveryoutweighsitslikelybenefit,consideringtheneedsofthecase,theamountincontroversy,theparties'resources,theimportanceoftheissuesatstakeintheaction,andtheimportanceofthediscoveryinresolvingtheissues.

Fed.R.Civ.P.26(b)(2)(C).

Inthiscase,theCourtdeterminedthattheuseofpredictivecodingwasappropriateconsidering:(1)theparties'agreement,(2)thevastamountofESItobereviewed(overthreemilliondocuments),(3)thesuperiorityofcomputer-assistedreviewtotheavailablealternatives(i.e.,linearmanualrevieworkeywordsearches),(4)theneedforcosteffectivenessandproportionalityunderRule26(b)(2)(C),and(5)thetransparentprocessproposedbyMSL.

ThisCourtwasoneoftheearlysignatoriestoTheSedonaConferenceCooperationProclamation,andhasstatedthat"thebestsolutionintheentireareaofelectronicdiscoveryiscooperationamongcounsel.ThisCourtstronglyendorsesTheSedonaConferenceProclamation(availableatwww.TheSedonaConference.org).";WilliamA.GrossConstr.Assocs.,Inc.v.Am.Mfrs.Mut.Ins.Co.,256F.R.D.at136.Animportantaspectofcooperationistransparencyinthediscoveryprocess.MSL'stransparencyinitsproposedESIsearchprotocolmadeiteasierfortheCourttoapprovetheuseofpredictivecoding.Asdiscussedaboveonpage10,MSLconfirmedthat"[a]llofthedocumentsthatarereviewedasafunctionoftheseedset,whether[they]areultimatelycodedrelevantorirrelevant,asidefromprivilege,willbeturnedoverto"plaintiffs.(Dkt.No.88:2/8/12Conf.Tr.at73;seealso2/17/12ESIProtocolat14:"MSLwillprovidePlaintiffs'counselwithallofthenon-privilegeddocumentsandwillprovide,totheextentapplicable,theissuetag(s)codedforeachdocument....Ifnecessary,counselwillmeetandconfertoattempttoresolveanydisagreementsregardingthecodingappliedtothedocumentsintheseedset.")WhilenotallexperiencedESIcounselbelieveitnecessarytobeastransparentasMSLwaswillingtobe,suchtransparencyallowstheopposingcounsel(andtheCourt)tobemorecomfortablewithcomputer-assistedreview,reducingfearsabouttheso-called"blackbox"ofthetechnology.[14]ThisCourthighlyrecommendsthatcounselinfuturecasesbewillingtoatleastdiscuss,ifnotagreeto,suchtransparencyinthecomputer-assistedreviewprocess.

SeveralotherlessonsforthefuturecanbederivedfromtheCourt'sresolutionoftheESIdiscoverydisputesinthiscase.

First,itisunlikelythatcourtswillbeabletodetermineorapproveaparty'sproposalastowhenreviewandproductioncanstopuntilthecomputer-assistedreviewsoftwarehasbeentrainedandtheresultsarequalitycontrolverified.OnlyatthatpointcanthepartiesandtheCourtseewherethereisacleardropofffromhighlyrelevanttomarginallyrelevanttonotlikelytoberelevantdocuments.WhilecostisafactorunderRule26(b)(2)(C),itcannotbeconsideredinisolationfromtheresultsofthepredictivecodingprocessandtheamountatissueinthelitigation.

Second,stagingofdiscoverybystartingwiththemostlikelytoberelevantsources(includingcustodians),withoutprejudicetotherequestingpartyseekingmoreafterconclusionofthatfirststagereview,isawaytocontroldiscoverycosts.Ifstagingrequiresalongerdiscoveryperiod,mostjudgesshouldbewillingtograntsuchanextension.(ThisJudgerunsaself-proclaimed"rocketdocket,"butinformedthepartieshereoftheCourt'swillingnesstoextendthediscoverycutoffifnecessarytoallowthestagingofcustodiansandotherESIsources.)

Third,inmanycasesrequestingcounsel'sclienthasknowledgeoftheproducingparty'srecords,eitherbecauseofanemploymentrelationshipashereorbecauseofotherdealingsbetweentheparties(e.g.,

8of20

contractualorotherbusinessrelationships).Itissurprisingthatinmanycasescounseldonotappeartohavesoughtandutilizedtheirclient'sknowledgeabouttheopposingparty'scustodiansanddocumentsources.Similarly,counselfortheproducingpartyoftenisnotsufficientlyknowledgeableabouttheirownclient'scustodiansandbusinessterminology.Anotherwaytophrasecooperationis"strategicproactivedisclosureofinformation,"i.e.,ifyouareknowledgeableaboutandtelltheothersidewhoyourkeycustodiansareandhowyouproposetosearchfortherequesteddocuments,opposingcounselandtheCourtaremoreapttoagreetoyourapproach(atleastasphaseonewithoutprejudice).

Fourth,theCourtfounditveryhelpfulthattheparties'ediscoveryvendorswerepresentandspokeatthecourthearingswheretheESIProtocolwasdiscussed.(Atediscoveryprograms,thisissometimesjokinglyreferredtoas"bringyourgeektocourtday.")EvenwhereasherecounselisveryfamiliarwithESIissues,itisveryhelpfultohavetheparties'ediscoveryvendors(orin-houseITpersonnelorin-houseediscoverycounsel)presentatcourtconferenceswhereESIissuesarebeingdiscussed.Italsoisimportantforthevendorsand/orknowledgeablecounseltobeabletoexplaincomplicatedediscoveryconceptsinwaysthatmakeiteasilyunderstandabletojudgeswhomaynotbetech-savvy.

CONCLUSION

ThisOpinionappearstobethefirstinwhichaCourthasapprovedoftheuseofcomputer-assistedreview.Thatdoesnotmeancomputer-assistedreviewmustbeusedinallcases,orthattheexactESIprotocolapprovedherewillbeappropriateinallfuturecasesthatutilizecomputer-assistedreview.NordoesthisOpinionendorseanyvendor(theCourtwasverycarefulnottomentionthenamesoftheparties'vendorsinthebodyofthisOpinion,althoughitisrevealedintheattachedESIProtocol),noranyparticularcomputer-assistedreviewtool.WhattheBarshouldtakeawayfromthisOpinionisthatcomputer-assistedreviewisanavailabletoolandshouldbeseriouslyconsideredforuseinlarge-data-volumecaseswhereitmaysavetheproducingparty(orbothparties)significantamountsoflegalfeesindocumentreview.Counselnolongerhavetoworryaboutbeingthe"first"or"guineapig"forjudicialacceptanceofcomputer-assistedreview.Aswithkeywordsoranyothertechnologicalsolutiontoediscovery,counselmustdesignanappropriateprocess,includinguseofavailabletechnology,withappropriatequalitycontroltesting,toreviewandproducerelevantESIwhileadheringtoRule1andRule26(b)(2)(C)proportionality.Computer-assistedreviewnowcanbeconsideredjudicially-approvedforuseinappropriatecases.

SOORDERED.

EXHIBIT

UNITEDSTATESDISTRICTCOURTSOUTHERNDISTRICTOFNEWYORK

MONIQUEDASILVAMOORE,:MARYELLENO'DONOHUE,LAURIE:MAYERS,HEATHERPIERCE,and:K.ATHERINEWILKINSON,onbehalfof:CaseNo.11-cv-1279(ALC)(AJP)themselvesandallotherssimilarly:situated,::Plaintiffs,::PARTIES'PROPOSEDPROTOCOLvs.:RELATINGTOTHEPRODUCTIONOF:ELECTRONICALLYSTOREDPUBLICISGROUPESAand:INFORMATION("ESI")&ORDERMSLGROUP,:Defendants.:

A.Scope

1.General.Theproceduresandprotocolsoutlinedhereingoverntheproductionofelectronicallystoredinformation("ESI")byMSLGROUPAmericas,Inc.("MSL")duringthependencyofthislitigation.Thepartiestothisprotocolwilltakereasonablestepstocomplywiththisagreed-uponprotocolfortheproductionofdocumentsandinformationexistinginelectronicformat.Nothinginthisprotocolwillbeinterpretedtorequiredisclosureofdocumentsorinformationprotectedfromdisclosurebytheattorney-clientprivilege,work-productproductdoctrineoranyotherapplicableprivilegeorimmunity.ItisPlaintiffs'positionthatnothinginthisprotocolwillbeinterpretedtowaivePlaintiffs'righttoobjecttothisprotocolasportionsofitweremandatedbytheCourtoverPlaintiffs'objections,includingPlaintiffs'objectionstothepredictivecodingmethodologyproposedbyMSL.

2.LimitationsandNo-Waiver.ThisprotocolprovidesageneralframeworkfortheproductionofESIonagoingforwardbasis.ThePartiesandtheirattorneysdonotintendbythisprotocoltowaivetheirrightstotheattorneywork-productprivilege,exceptasspecificallyrequiredherein,andanysuchwaivershallbestrictlyandnarrowlyconstruedandshallnotextendtoothermattersorinformationnotspecificallydescribedherein.AllPartiespreservetheirattorneyclientprivilegesandotherprivilegesandthereisnointentbytheprotocol,ortheproductionofdocumentspursuanttotheprotocol,toinanywaywaiveorweakentheseprivileges.AlldocumentsproducedhereunderarefullyprotectedandcoveredbytheParties'confidentialityandclawbackagreementsandordersoftheCourteffectuatingsame.

9of20

3.RelevantTimePeriod.January1,2008throughFebruary24,2011forallnon-emailESIrelatingtotopicsbesidespaydiscriminationandforalle-mails.January1,2005throughFebruary24,2011forallnon-e-mailESIrelatingtopaydiscriminationforNewYorkPlaintiffs.

B.ESIPreservation

1.MSLhasissuedlitigationnoticestodesignatedemployeesonFebruary10,2010,March14,2011andJune9,2011.

C.Sources

1.ThePartieshaveidentifiedthefollowingsourcesofpotentiallydiscoverableESIatMSL.PhaseIsourceswillbeaddressedfirst,andPhaseIIsourceswillbeaddressedafterPhaseIsourcesearchesarecomplete.Sourcesmarkedas"N/A"willnotbesearchedbytheParties.

aEMCSourceOneArchivingSystemusedtocaptureandstoreallIArchiveincomingandoutbounde-mailsandselectedinstantmessageconversationssavedthroughIBMSametime(seebelow).

bLotusNotesE-mailActivecorporatesystemthatprovidese-mailN/Acommunicationandcalendaringfunctions.

cGroupWiseE-mailLegacycorporatesystemthatprovidede-mailN/Acommunicationandcalendaringfunctions.

dIBMSametimeLotusNotesInstantMessagingandcollaborationN/Aapplication.

eHomeDirectoriesPersonalnetworkstoragelocationsonthefileserver(s)II

10of20

dedicatedtoindividualusers.(Withtheexceptionof2homedirectoriesforwhichMSLwillcollectandanalyzethedatatodeterminethelevelofduplicationascomparedtotheEMCSourceOneArchive.Thepartieswillmeetandconferregardingtheselectionofthetwocustodians.)

fSharedFoldersSharednetworkstoragelocationsonthefileserver(s)IIthatareaccessiblebyindividualusers,groupsofusersorentiredepartments.(WiththeexceptionofthefollowingHumanResourcessharedfolderswhichwillbeinPhaseI:CorporateHR,NorthAmericaHRandNewYorkHR.)gDatabaseServersBackenddatabases(e.g.Oracle,SQL,MySQL)usedtoN/Astoreinformationforfrontendapplicationsorotherpurposes.

hHalogenSoftwarePerformancemanagementprogramprovidedbyIHalogentoconductperformanceevaluations.

iNoovooCorporateIntranetsite.II

jCorporateE-mailaddressesthatemployeesmayutilizetoprovideIFeedbackthecompanywithcomments,suggestionsandoverallfeedback.

kHyperionOracleapplicationthatoffersglobalfinancialN/AFinancialconsolidation,reportingandanalysis.Management("HFM")

lVurv/TaleoTalentrecruitmentsoftware.II

mServiceNowHelpDeskapplicationusedtotrackemployeecomputerN/Arelatedrequests.

nPeopleSoftHumanresourcesinformationmanagementsystem.I

oPRISMPeopleSoftcomponentusedfortimeandbillingImanagement.

pPortalAprojectbasedportalprovidedthroughOracle/BEAIISystems.

qDesktops/LaptopsFixedandportablecomputersprovidedtoemployeestoIIperformworkrelatedactivities.(Withtheexceptionof2desktop/laptopharddrivesforwhichMSLwillcollectandanalyzethedatatodeterminethelevelofduplicationascomparedtotheEMCSourceOneArchive.Thepartieswillmeetandconferregardingtheselectionofthetwocustodians.)rPublicisBenefitsWebbasedsitethatmaintainsinformationaboutIIConnectionemployeebenefitsandrelatedinformation.

sGEARSEmployeeexpensereportingsystem.II

tMS&LCityFormercorporateIntranet.N/A

uAdiumApplicationwhichaggregatesinstantmessages.N/A

vPidginApplicationwhichaggregatesinstantmessage.N/A

wIBMLotusMobiledevicesynchronizationandsecuritysystem.N/A

11of20

TravelerandMobilelron

yMobilePortablePDAs,smartphones,tabletsusedforN/ACommunicationcommunication.Devices

zYammerSocialmediaandcollaborationportal.N/A

aaSalesForce.comWeb-basedcustomerrelationshipmanagementN/Aapplication.

bbRemovablePortablestoragemedia,externalharddrives,thumbN/AStorageDevicesdrives,etc.usedtostorecopiesofworkrelatedESI.

a.EMCSourceOne—MSLusesSourceOne,anEMCe-mailarchivingsystemthatcapturesandstoresalle-mailmessagesthatpassthroughthecorporatee-mailsystem.Inaddition,ifauserchoosestosaveaninstantmessagingchatconversationfromIBMSametime(referencedbelow),thattoowouldbearchivedinSourceOne.DefendantMSLalsoacknowledgesthatcalendaritemsareregularlyingestedintotheSourceOnesystem.ThepartieshaveagreedthatthisdatasourcewillbehandledasoutlinedinsectionEbelow.

b.LotusNotesE-mail—MSLcurrentlymaintainsmultipleLotusNotesDominoserversinvariousdatacentersaroundtheworld.Alle-mailcommunicationandcalendaritemsarejournaledinrealtimetotheEMCSourceOnearchive.Thepartieshaveagreedtonotcollectanyinformationfromthisdatasourceatthistime.

c.GroupWiseE-mail—PriortotheimplementationoftheLotusNotesenvironment,GroupWisewasusedforalle-mailandcalendarfunctionality.BeforethedecommissioningoftheGroupWiseservers,MSLcreatedbackuptapesofallserversthathousedtheGroupWisee-maildatabases.Thepartieshaveagreedtonotcollectanyinformationfromthisdatasourceatthistime.

d.IBMSametime—MSLprovidescustodianswiththeabilitytohaverealtimechatconversationsviatheIBMSametimeapplicationthatispartoftheLotusNotessuiteofproducts.

e.HomeDirectories—CustodianswithcorporatenetworkaccessatMSLalsohaveadedicatedandsecurednetworkstoragelocationwheretheyareabletosavefiles.MSLwillcollectthehomedirectorydatafor2custodiansandanalyzethedatatodeterminethelevelofduplicationofdocumentsinthisdatasourceagainstthedatacontainedintheEMCSourceOnearchiveforthesamecustodians.(Thepartieswillmeetandconferregardingtheselectionofthetwocustodians.)TheresultsoftheanalysiswillbeprovidedtoPlaintiffssothatadeterminationcanbemadebythepartiesastowhetherMSLwillincludethisdatasourceinitsproductionofESItoPlaintiffs.Ifso,thepartieswillattempttoreachanagreementastotheapproachusedtocollect,reviewandproduceresponsiveandnon-privilegeddocuments.

f.SharedFolders—Individualemployees,groupsofemployeesandentiredepartmentsatMSLaregivenaccesstosharednetworkstoragelocationstosaveandsharefiles.AsitrelatestotheHumanResourcesrelatedsharedfolders(i.e.,NorthAmericaHRDrive(10.2OB),CorporateHRDrive(440MB),NYHRDrive(1.9OB),ChicagoHRDrive(1.16OB),BostonHRDrive(43.3MB),andAtlantaHRDrive(6.64OB)),MSLwilljudgmentallyreviewandproduceresponsiveandnon-privilegeddocumentsfromtheNorthAmericaHRDrive,CorporateHRDrive,andNYHRDrive.MSLwillproducetoPlaintiffsgeneralinformationregardingthecontentofotherSharedFolders.ThepartieswillmeetandconferregardingtheinformationgatheredconcerningtheotherSharedFoldersanddiscusswhetheranyadditionalSharedFoldersshouldbemovedtoPhaseI.

g.DatabaseServers—MSLhasindicatedthatitdoesnotutilizeanydatabaseservers,otherthanthosethatpertaintothesourcesoutlinedaboveinC,whicharelikelytocontaininformationrelevanttoPlaintiffs'claims.

h.HalogenSoftware—MSLutilizesathirdpartyproduct,Halogen,forperformancemanagementandemployeeevaluations.Thepartieswillmeetandconferinordertoexchangeadditionalinformationandattempttoreachanagreementastothescopeofdataandtheapproachusedtocollect,reviewandproduceresponsiveandnon-privilegeddocuments.

i.Noovoo—MSLmaintainsacorporateIntranetsitecalled"Noovoo"whereemployeesareabletoaccessCompany-relatedinformation.MSLwillprovidePlaintiffswithanyemployment-relatedpoliciesmaintainedwithinNoovoo.

j.CorporateFeedback—MSLhasmaintainedvariouse-mailaddressesthatemployeesmayutilizetoprovide12of20

thecompanywithcomments,suggestionsandoverallfeedback.Thesee-mailaddressesinclude"powerofone@mslworldwide.com","poweroftheindividual@mslworldwide.com","townhall@mslworldwide.com"and"whatsonyourmind@mslworldwide.com".Thepartieshaveagreedthatallresponsiveandnon-privilegedESIwillbeproducedfromthesee-mailaccountsandanyothere-mailaccountsthatfallunderthiscategoryofinformation.Atpresent,MSLintendstomanuallyreviewthecontentsofeachofthesee-mailaccounts.However,ifaftercollectingthecontentsofeachofthee-mailaccountsMSLdeterminesthatamanualreviewwouldbeimpractical,thepartieswillmeetandconferastotheapproachusedtocollect,reviewandproduceresponsiveandnon-privilegeddocuments.

k.HyperionFinancialManagement("HFM")—MSLusesanOracleapplicationcalledHFMthatoffersglobalfinancialconsolidation,reportingandanalysiscapabilities.

l.Vurv/Taleo—Sinceapproximately2006,MSLusedanapplicationknownasVurvasitstalentrecruitmentsoftware.AsofAugust31,2011,asaresultofVurvbeingpurchasedbyTaleo,MSLhasbeenusingasimilarapplicationbyTaleoasitstalentrecruitmentsoftware.Theapplication,whichisaccessedthroughMSL'spublicwebsite,allowsuserstosearchforopenpositionsaswellasinputinformationaboutthemselves.TotheextentPlaintiffscontendtheyweredeniedanyspecificpositions,theywillidentifysameandthePartieswillmeetandconfertodiscusswhat,ifany,informationexistswithinVurv/Taleoregardingtheidentifiedposition.IfinformationexistsinVurv/Taleooranothersourceregardingthesepositions,MSLwillproducethisinformation,totheextentsuchinformationisdiscoverable.

m.ServiceNow—MSLutilizesServiceNowasitsHelpDeskapplication.Thissystemcoversawidevarietyofrequestsbyemployeesforcomputer-relatedassistance(e.g.,troubleshootincidents,installsoftware,etc.).

n.PeopleSoft—MSLutilizesPeopleSoft,anOracle-basedsoftwareproduct,torecordemployeedatasuchasdateofhire,dateoftermination,promotions,salaryincreases,transfers,etc.MSLhasproduceddatafromthissourceandwillconsiderproducingadditionaldatainresponsetoaspecificinquiryfromPlaintiffs.

o.PRISM—MSLutilizesPRISMfortrackingtimeandbilling.Itisusedprimarilytotrackanemployee'sbillabletime.MSLwillconsiderproducingadditionaldatainresponsetoaspecificinquiryfromPlaintiffs.

p.Portal—MSLmaintainsaportalprovidedthroughOracle/BEASystems.Theportalisweb-basedandisusedforlightworkflowactivities(suchasreviewingdraftdocuments).

q.Desktops/Laptops—MSLprovidedemployeeswithdesktopand/orlaptopcomputerstoassistinworkrelatedactivities.MSLwillcollectthedesktop/laptopharddrivedatafor2custodiansandanalyzethedatatodeterminethelevelofduplicationofdocumentsinthisdatasourceagainstthedatacontainedintheEMCSourceOnearchiveforthesamecustodians.(Thepartieswillmeetandconferregardingtheselectionofthetwocustodians.)TheresultsoftheanalysiswillbeprovidedtoPlaintiffssothatadeterminationcanbemadebythepartiesastowhetherMSLwillincludethisdatasourceinitsproductionofESItoPlaintiffs.Ifso,thePartieswillattempttoreachanagreementastotheapproachusedtocollect,reviewandproduceresponsiveandnon-privilegeddocuments.

r.PublicisBenefitsConnection—PlaintiffsunderstandthatMSLprovidesemployeeswithaccesstoacentralizedwebbasedsitethatprovidesaccesstocorporatebenefitsinformationandotherrelatedcontent.

s.GEARS—MSLmaintainsacentralizedweb-basedexpensetrackingandreportingsystemcalled"GEARS"whereusersareabletoenterexpensesandgeneratereports.

t.MS&LCity—MSLmaintainedacorporateweb-basedIntranetpriortomigratingtoNoovoo.

u.Adium—ThisisafreeandopensourceinstantmessagingclientforMacOSXusers.

v.Pidgin—Pidginisachatprogramwhichletsuserslogintoaccountsonmultiplechatnetworkssimultaneously.However,thedataresideswithathirdpartymessagingprovider(e.g.AIM,Yahoo!,GoogleTalk,MSNMessenger,etc.).

w.IBMLotusTravelerandMobileIron—MSLmaintainsthesesystemsfore-maildevicesyncandsecurityfeaturesforemployees'mobiledevices,includingBlackberrydevices,iPhones,iPads,Androidphones,andAndroidtablets.

x.MobileCommunicationDevices—MSLprovidesmobiledevicesand/orconnectivityincludingBlackberrydevices,iPhones,iPads,Androidphones,andAndroidtabletstodesignatedemployees.

y.Yammer—Thisisaninstantmessagingapplicationhostedexternally,usedforapproximatelyoneyearinoraround2008through2009.

z.SalesForce.com—Thisisaweb-basedcustomerrelationshipmanagementapplicationbutitwasnot

13of20

widelyused.

aa.RemovableStorageDevices—MSLdoesnotrestrictauthorizedemployeesfromusingremovablestoragedevices.

D.Custodians

1.ThePartiesagreethatMSLwillsearchthee-mailaccountsofthefollowingindividualsastheyexistonMSL'sEMCSourceOnearchive.(Exceptwhereadaterangeisnoted,thecustodian'sentiree-mailaccountwascollectedfromthearchive.)

1.Lund,WendyExecutiveVPofGlobalClientandBusinessDevelopment2.Fite,VickiManagingDirector,MSLLosAngeles3.Wilson,ReneePresident,NERegion,ManagingDirectorNY4.Brennan,Nancy(1/1/08to5/31/08)SVP/DirectorCorporateBranding5.Lilien(Lillien,Kashanian),TaraSVP,NorthAmericaHumanResources6.Miller,PeterExecutiveVicePresident,CFO7.Masini,RitaChiefTalentOfficer8.Tsokanos,JimPresidentoftheAmericas9.DaSilvaMoore,MoniqueDirectorHealthcarePractice,Global10.O'Kane,Jeanine(2/8/10to2/24/11)DirectorofHealthcareNorthAmerica11.Perlman,CarolSeniorVP12.Mayers,LaurieSVPMS&LDigital13.Wilkinson,KateAccountExecutive14.Curran,Joel(5/1/08to5/31/10)ManagingDirectorMSLChicago15.Shapiro,MauryNorthAmericanCFO16.Baskin,Rob(1/1/08to12/31/08)ManagingDirector17.Pierce,HeatherVP18.Branam,Jud(1/1/08to1/31/10)ManagingDirector,MS&LDigital19.McDonough,Jenni(1/1/08toVP,DirectorofHumanResources12/31/08)20.Hannaford,Donald(1/1/08to3/1/08)ManagingDirector21.Orr,Bill(1/1/08to2/24/11)ManagingDirector22.Dhillon,Neil(9/8/08to5/31/10)ManagingDirectorMSLWashingtonDC23.Hubbard,ZanetaAccountSupervisor24.Morgan,Valerie(1/1/08to2/24/11)HRDirector25.Daversa,Kristin(1/1/08to2/24/11)HRDirector26.Vosk,Lindsey(1/1/08to2/24/11)HRManager27.Carberry,Joe(1/1/08to2/24/11)President,WesternRegion28.Sheffield,Julie(1/1/08to2/24/11)HR/RecruitingAssociate29.MaryEllenO'DonohueSVP(2010)

30.Hass,MarkCEO(former)31.Morsman,MichaelManagingDirector,AnnArbor(former)

E.SearchMethodology[1]

1.General.ThePartieshavediscussedthemethodologiesorprotocolsforthesearchandreviewofESIcollectedfromtheEMCSourceOnearchiveandthefollowingisasummaryoftheParties'agreementontheuseofPredictiveCoding.ThissectionrelatessolelytotheEMCSourceOnedatasource(hereinafterreferredtoasthe"e-mailcollection").

2.GeneralOverviewofPredictiveCodingProcess.MSLwillutilizetheAxceleratesoftwarebyRecommindtosearchandreviewthee-mailcollectionforproductioninthiscase.

TheprocessbeginswithJacksonLewisattorneysdevelopinganunderstandingoftheentiree-mailcollectionwhileidentifyingasmallnumberofdocuments,theinitialseedset,thatisrepresentativeofthecategoriestobereviewedandcoded(relevance,privilege,issue-relation).Itisthestepwhenthefirstseedsetsaregeneratedwhichisdonebyuseofsearchandanalyticaltools,includingkeyword,Booleanandconceptsearch,conceptgrouping,and,asneeded,uptotheattorneys'identificationofprobativedocumentsforeachcategorytobereviewedandcoded.

Plaintiffs'counselwillbeprovidedwithpreliminaryresultsofMSL'shitcountsusingkeywordsearchesto

14of20

createahighpriorityrelevantseedset,andwillbeinvitedtocontributetheirownproposedkeywords.Thereafter,Plaintiffs'counselwillbeprovidedwiththenon-privilegedkeywordhits—bothfromMSL'skeywordlistandPlaintiffs'keywordlistwhichwerereviewedandcodedbyMSL.Plaintiffs'counselwillreviewthedocumentsproducedandpromptlyprovidedefensecounselwiththeirownevaluationoftheinitialcodingappliedtothedocuments,includingidentificationofanydocumentsitbelieveswereincorrectlycoded.Totheextentthepartiesdisagreeregardingthecodingofaparticulardocument,theywillmeetandconferinanefforttoresolvethedisputepriortocontactingtheCourtforresolution.TheirrelevantdocumentssoproducedshallbepromptlyreturnedafterreviewandanalysisbyPlaintiffs'counseland/orresolutionofanydisputesbytheCourt.

TheseedsetsarethenusedtobeginthePredictiveCodingprocess.Eachseedsetofdocumentsisappliedtoitsrelevantcategoryandstartsthesoftware"training"process.Thesoftwareuseseachseedsettoidentifyandprioritizeallsubstantivelysimilardocumentsoverthecompletecorpusofthee-mailcollection.Theattorneysthenreviewandcodeajudgmentalsampleofatleast500ofthe"computersuggested"documentstoensuretheirpropercategorizationandtofurthercalibratethesystembyrecodingdocumentsintotheirpropercategories.AxceleratelearnsfromthenewcorrectedcodingandthePredictiveCodingprocessisrepeated.

AttorneysrepresentingMSLwillhaveaccesstotheentiree-mailcollectiontobesearchedandwillleadthecomputertraining,buttheywillobtaininputfromPlaintiffs'counselduringtheiterativeseedselectionandqualitycontrolprocessesandwillsharetheinformationusedtocraftthesearchprotocolasfurtherdescribedherein.Allnon-privilegeddocumentsreviewedbyMSLduringeachroundoftheiterativeprocess(i.e.,bothdocumentscodedasrelevantandirrelevant)willbeproducedtoPlaintiffs'counselduringtheiterativeseedsetselectionprocess.Plaintiffs'counselwillreviewthedocumentsproducedandpromptlyprovidedefensecounselwithitsownevaluationoftheinitialcodingappliedtothedocuments,includingidentificationofanydocumentsitbelieveswereincorrectlycoded.TotheextentthePartiesdisagreeregardingthecodingofaparticulardocument,theywillmeetandconferinanefforttoresolvethedisputepriortocontactingtheCourtforresolution.Again,theirrelevantdocumentssoproducedshallbepromptlyreturnedafterreviewandanalysisbyPlaintiffs'counseland/orresolutionofanydisputesbytheCourt.

Attheconclusionoftheiterativereviewprocess,alldocumentpredictedbyAxceleratetoberelevantwillbemanuallyreviewedforproduction.However,dependingonthenumberofdocumentsreturned,therelevancyratingofthosedocuments,andthecostsincurredduringthedevelopmentoftheseedsetanditerativereviews,MSLreservestherighttoseekappropriaterelieffromtheCourtpriortocommencingthefinalmanualreview.

Theaccuracyofthesearchprocesses,boththesystems'functionsandtheattorneyjudgmentstotrainthecomputer,willbetestedandqualitycontrolledbybothjudgmentalandstatisticalsampling.Instatisticalsampling,asmallsetofdocumentsisrandomlyselectedfromthetotalcorpusofthedocumentstobetested.Thesmallsetisthenreviewedandanerrorratecalculatedtherefrom.Theerrorratescanthenbereliablyprojectedonthetotalcorpus,havingamarginoferrordirectlyrelatedtothesamplesize.

3.IssueTags.Thepartiesagreethat,totheextentapplicable,aspartoftheseedsettrainingdescribedabove,aswellasduringtheiterativereviewprocess,alldocumentscategorizedasrelevantandnotprivileged,totheextentapplicable,alsoshallbecodedwithoneormoreofthefollowingagreed-uponissuetags:

a.Reorganization.

b.Promotion/Assignments.

c.Work/LifeBalance.

d.Termination.

e.Compensation.

f.Maternity/Pregnancy.

g.Complaints/HR.

h.PublicisGroupe/Jurisdiction.

15of20

Thisissuecodingwilltakeplaceduringtheinitialrandomsample,creationoftheseedsetandinitialanditerativetraining(seeparagraphs4,5and6below).ThisinputshallbeprovidedtoPlaintiffs'counselalongwiththeinitialdocumentproductions.Plaintiffs'counselshallpromptlyreportanydisagreementsonclassification,andthepartiesshalldiscusstheseissuesingoodfaith,sothattheseedsettrainingmaybeimprovedaccordingly.Thisissue-tagginganddisclosureshalltakeplaceduringthedescribedcollaborativeseedsettrainingprocess.ThedisclosuresheremadebyMSLonitsissuecodingarenotrequiredinthefinalproductionset.

4.InitialRandomSample.UsingtheAxceleratesoftwaretogeneratearandomsampleoftheentirecorpusofdocumentsuploadedtotheAxceleratesearchandreviewplatform,MSL'sattorneyswillconductareviewoftherandomsampleforrelevanceandtodevelopabaselineforcalculatingrecallandprecision.Totheextentapplicable,anyrelevantdocumentsalsowillbecodedwithoneormoreoftheissuetagsreferencedinparagraphE.3above.Therandomsampleconsistsof2,399documents,whichrepresentsa95%confidencelevelwithaconfidenceestimationofplusorminus2%.ThePartiesagreetoutilizetherandomsamplegeneratedpriortothefinalizationofthisprotocol.However,duringPlaintiffs'counsel'sreviewoftherandomsample,theymayadviseastowhethertheybelieveanyofthedocumentsshouldbecodedwithoneormoreofthesubsequentlyaddedissuecodes(i.e.,Complaints/HRandPublicisGroupe/Jurisdiction)andwill,asdiscussedabove,indicateanydisagreementwithMSL'sclassifications.

5.SeedSet.

a.DefendantMSL.Tocreatetheinitialseedsetofdocumentsthatwillbeusedto"train"theAxceleratesoftwareasdescribedgenerallyabove,MSLprimarilyutilizedkeywordslistedonExhibitsAandBtothisprotocol,butalsoutilizedotherjudgmentalanalysisandsearchtechniquesdesignedtolocatehighlyrelevantdocuments,includingtheBoolean,conceptsearchandotherfeaturesofAxcelerate.Giventhevolumeofhitsforeachkeyword(ExhibitA),MSLreviewedasamplingofthehitsandcodedthemforrelevanceaswellasforthefollowingeightpreliminaryissues:(i)Reorganization;(ii)Promotion;(iii)Work/LifeBalance;(iv)Termination;(v)Compensation;and(vi)Maternity.Specifically,exceptforkeywordsthatwerepropernames,MSLperformedseveralsearcheswithineachsetofkeywordhitsandreviewedasampleofthehits.TheAxceleratesoftwarerankedthehitsinorderofrelevancebasedonthesoftware'sanalyticalcapabilitiesandthedocumentswerereviewedindecreasingorderofrelevance(i.e.,eachreviewofthesampleofsupplementalsearchesstartedwiththehighestrankeddocuments).ExhibitBidentifiesthesupplementalsearchesconducted,thenumberofhits,thenumberofdocumentsreviewed,thenumberofdocumentscodedaspotentiallyresponsiveandgeneralcommentsregardingtheresults.Inaddition,totheextentapplicable,documentscodedasresponsivealsowerecodedwithoneormoreissuetags.MSLwillrepeattheprocessoutlinedaboveandwillincludethenewlydefinedissuesandnewlyaddedcustodians.MSLwillprovidePlaintiffs'counselwithallofthenon-privilegeddocumentsandwillprovide,totheextentapplicable,theissuetag(s)codedforeachdocument,asdescribedabove.Plaintiffs'counselshallpromptlyreviewandprovidenoticeastoanydocumentswithwhichtheydisagreewheretheydonotunderstandthecoding.Ifnecessary,counselwillmeetandconfertoattempttoresolveanydisagreementsregardingthecodingappliedtothedocumentsinthisseedset.

b.Plaintiffs.Tohelpcreatetheinitialseedsetofdocumentsthatwillbeusedto"train"theAxceleratesoftware,PlaintiffsprovidedalistofpotentialkeywordstoMSL.MSLprovidedPlaintiffswithahitlistfortheirproposedkeywords.ThisprocesswasrepeatedtwicewiththehitlistforPlaintiffs'mostrecentsetofkeywordsattachedasExhibitC.MSLwillreview4,000randomlysampleddocumentsfromPlaintiffs'supplementallistofkeywordstobecodedforrelevanceandissuetags.MSLwillprovidePlaintiffs'counselwithallnon-privilegeddocumentsandwillprovide,totheextentapplicable,theissuetag(s)codedforeachdocument.Plaintiffs'counselshallpromptlyreviewandprovidenoticeastoanydocumentswithwhichtheydisagreewithorwheretheydonotunderstandthecoding.Ifnecessary,theParties'counselwillmeetandconfertoattempttoresolveanydisagreementsregardingthecodingappliedtothedocumentsinthisseedset.

c.JudgmentalSampling.Inadditiontotheabove,anumberoftargetedsearcheswereconductedbyMSLinanefforttolocatedocumentsresponsivetoseveralofPlaintiffs'specificdiscoveryrequests.Approximately578documentshavealreadybeencodedasresponsiveandproducedtoPlaintiffs.Inaddition,severaljudgmentalsearcheswereconductedwhichresultedinapproximately300documentsinitiallybeingcodedasresponsiveandseveralthousandadditionaldocumentscodedasirrelevant.Thedocumentscodedasrelevantandnon-privilegedalsowillbereviewedbyPlaintiffs'counseland,subjecttotheirfeedback,includedintheseedset.AnexplanationshallbeprovidedbyMSL'sattorneysforthebasisofthebulktaggingofirrelevantdocuments(primarilyelectronicperiodicalsandnewslettersthatwereexcludedinthesamemannerasspamjunkmailisexcluded).Theexplanationshallincludethetypesofdocumentsbulktaggedasirrelevantaswellastheprocessusedtoidentifythosetypesofdocumentsandothersimilardocumentsthat

16of20

werebulktaggedasirrelevant.

6.InitialAndIterativeTraining.Followingthecreationofthefirstseedset,theAxceleratesoftwarewillreviewtheentiredatasettoidentifyotherpotentiallyrelevantdocuments.MSLwillthenreviewandtagajudgmentalbasedsample,consistingofaminimumof500documents,includingalldocumentsrankedashighlyrelevantorhot,ofthenew"ComputerSuggested"documents,whichweresuggestedbytheAxceleratesoftware.MSL'sattorneysshallactinconsultationwiththeAxceleratesoftwareexpertstomakeareasonable,goodfaithefforttoselectdocumentsinthejudgmentalsamplethatwillservetoenhanceandincreasetheaccuracyofthepredictivecodingfunctions.Theresultsofthisfirstiteration,boththedocumentsnewlycodedasrelevantandnotrelevantforparticularissuecodeorcodes,willbeprovidedtoPlaintiffs'counselforreviewandcomment.(Alldocumentsproducedbythepartieshereintoeachother,including,withoutlimitation,thesesmallseedsetdevelopmentproductions,shallbemadeundertheConfidentialityStipulationinthismatteraswellasanyclawbackagreementthatshallbereducedtoanorderacceptabletotheCourt.AnydocumentsmarkedasirrelevantshallbereturnedtocounselforMSLattheconclusionoftheiterativetrainingphase,unlesstherelevancyofanydocumentsaredisputed,inwhichcasetheymaybesubmittedtotheCourtforreview.)

Uponcompletionoftheinitialreview,andanyrelatedmeetandconfersessionsandagreeduponcodingcorrections,theAxceleratesoftwarewillberunagainovertheentiredatasetforsuggestionsonotherpotentiallyrelevantdocumentsfollowingthesameproceduresasthefirstiteration.ThepurposeofthissecondandanysubsequentiterationsofthePredictiveCodingprocesswillbetofurtherrefineandimprovetheaccuracyofthepredictionsonrelevanceandvariousothercodes.TheresultsoftheseconditerationshallbereviewedandnewcodingsharedwithPlaintiffs'counselasdescribedforthefirstiteration.Thisprocessshallberepeatedfivemoretimes,foratotalofseveniterations,unlessthechangeinthetotalnumberofrelevantdocumentspredictedbythesystemasaresultofanewiteration,ascomparedtothelastiteration,islessthanfivepercent(5%),andnonewdocumentsarefoundthatarepredictedtobehot(akahighlyrelevant),atwhichpointMSLshallhavethediscretiontostoptheiterativeprocessandbeginthefinalreviewasnextdescribed.Ifmorethan40,000documentsarereturnedinthefinaliteration,thenMSLreservestherighttoapplytotheCourtforreliefandlimitationsinitsreviewobligationshereunder.Plaintiffsreservetheright,atalltimes,tochallengetheaccuracyandreliabilityofthepredictivecodingprocessandtherighttoapplytotheCourtforareviewoftheprocess.

7.FinalSearchandProduction.AllofthedocumentspredictedtoberelevantinthefinaliterationdescribedinparagraphsixabovewillbereviewedbyMSL,unlessitappliestothecourtforreliefhereunder.AlldocumentsfoundbyMSL'sreviewtoberelevantandnon-privilegeddocumentswillbepromptlyproducedtoPlaintiffs.Ifmorethan40,000documentsareincludedinthefinaliteration,thenMSLreservesitsrighttoseekpaymentfromPlaintiffsforallreasonablecostsandfeesMSLincurredrelatedtotheattorneyreviewandproductionofmore40,000documents.ThisprovisionisnotintendedasawaiverbyMSLtoalsoseekanawardofalldiscoverycostsincurred,includingcostsrelatedtothefirst40,000documents,attheconclusionofthisactionunder28U.S.C.§1920(4)andRule54(d)(1)FederalRulesofCivilProcedure.[PlaintiffsobjecttotheinclusionofMSL'sproposedcost-shiftinglanguageasprematureandargumentative,incontraventionoftheCourt'sJanuary4,2012andFebruary8,2012orders.Plaintiffsbelievecostsshouldbesubjecttoaseparatehearing.]

8.QualityControlbyRandomSampleofIrrelevantDocuments.Inaddition,attheconclusionofthissearchprotocoldevelopmentprocessdescribedabove,andbeforethefinalsearchandproductiondescribedinParagraph7above,MSLwillreviewarandomsampleof2,399documentscontainedintheremainderofthedatabasethatwereexcludedasirrelevant.Theresultsofthisreview,boththedocumentscodedasrelevantandnotrelevant,butnotprivileged,willbeprovidedtoPlaintiffs'counselforreview.(Anydocumentsinitiallycodedas"notrelevant"willbeprovidedsubjecttotheConfidentialityStipulationandanyclawbackagreementsenteredinthismatterwillbereturnedtocounselforMSLwithin60daysoftheirproduction.)Thepurposeforthisreviewistoallowcalculationoftheapproximatedegreeofrecallandprecisionofthesearchandreviewprocessused.IfPlaintiffsobjecttotheproposedreviewbasedontherandomsamplequalitycontrolresults,oranyothervalidobjection,theyshallprovideMSLwithwrittennoticethereofwithinfivedaysofthereceiptoftherandomsample.Thepartiesshallthenmeetandconferingoodfaithtoresolveanydifficulties,andfailingthatshallapplytotheCourtforrelief.MSLshallnotberequiredtoproceedwiththefinalsearchandreviewdescribedinParagraph7aboveunlessanduntilobjectionsraisedbyPlaintiffshavebeenadjudicatedbytheCourtorresolvedbywrittenagreementoftheParties.

F.Costs

1.MSLproposestolimitthecostsofitsfinalreviewandproductionofresponsiveESIfromtheMSLemailcollectiontoanadditional$200,000,aboveandbeyondtheapproximately$350,000ithasalreadypaidoris

17of20

anticipatedtopayine-discoveryrelatedactivitiesaspreviouslydescribedanddisclosedtoPlaintiffs.Specifically,althoughMSLpotentiallywillconductandpayforreviewofmorethat40,000documents,ifthatisrequiredunderthepredictivecodingprocessdescribedinparagraphs6and7above,MSLreservesitsrighttoseekrelieffromtheCourt(e.g.,acostshiftingawardand/orrulingthatMSLneedtoreviewmorethanaspecifiednumberofdocumets)pursuanttotheprinciplesofproportionality.SeeRule1,Rule26(b)(2)(C),Rule26(b)(2)(B),andRule26(g),FederalRulesofCivilProcedure;CommentaryonProportionalityinElectronicDiscovery,11SEDONACONF.J.289(2010);Octo,etal.,MandatingReasonablenessinaReasonableInquiry,DenverUniversityLawReview,87.2,522-559(2010);AlsoseeRule403oftheFederalEvidencecode(inadminissibilityofcomumlativeevidence).[PlaintiffsobjecttotheinclusionofMSL'sproposedcost-shiftinglanguageasprematureandargumentative,incontraventionoftheCourt'sJanuary4,2012.andFebruary8,2012,orders.Plaintiffsbelievecostsshouldbesubjecttoaseparatehearing.

2.PlaintiffsagreetobearallofthecostsassociatedwiththeircompliancewiththetermsofthisprotocolandwiththereceiptandreviewofESIproducedhereunderincludingthecostsassociatedwithitsESIexpertsatDOARLitigationConsultingwhowillbeinvolvedwithPlaintiffsinallaspectsofthisESIprotocol.PlaintiffsproposethatMSLbearallofthecostsassociatedwithitsobligationsunderthetermsofthisprotocolanddonotagreetolimittheamountofinformationsubjecttothereviewandproductionofESIbyMSL.

G.FormatofProductionForDocumentsProducedFromAxcelerate

1.TIFF/NativeFileFormatProduction.Documentswillbeproducedassingle-pageTIFFimageswithcorrespondingmulti-pagetextandnecessaryloadfiles.Theloadfileswillincludeanimageloadfileaswellasametadata(.DAT)filewiththemetadatafieldsidentifiedonExhibitD.DefendantMSLwillproducespreadsheets(.xlsfiles)andPowerPointpresentations(.pptfiles)innativeformaswellasanydocumentsthatcannotbeconvertedtoTIFFformat(e.g.,audioorvideofiles,suchasmp3s,wavs,megs,etc.).Inaddition,foranyredacteddocumentsthatareproduced,thedocuments'metadatafieldswillberedactedwhererequired.FortheproductionofESIfromnon-emailsources,thepartieswillmeetandconfertoattempttoreachanagreementoftheformatofproduction.

2.Appearance.Subjecttoappropriateredaction,eachdocument'selectronicimagewillconveythesameinformationandimageastheoriginaldocument.Documentsthatpresentimagingorformattingproblemswillbepromptlyidentifiedandthepartieswillmeetandconferinanattempttoresolvetheproblems.

3.DocumentNumbering.Eachpageofaproduceddocumentwillhavealegible,uniquepageidentifier"BatesNumber"electronically"burned"ontotheimageatalocationthatdoesnotobliterate,concealorinterferewithanyinformationfromthesourcedocument.TheBatesNumberforeachpageofeachdocumentwillbecreatedsoastoidentifytheproducingpartyandthedocumentnumber.InthecaseofmaterialsredactedinaccordancewithapplicablelaworconfidentialmaterialscontemplatedinanyConfidentialityStipulationenteredintobytheparties,adesignationmaybe"burned"ontothedocument'simageatalocationthatdoesnotobliterateorobscureanyinformationfromthesourcedocument.

4.ProductionMedia.Theproducingpartywillproducedocumentsonreadilyaccessible,computerorelectronicmediaasthepartiesmayhereafteragreeupon,includingCD-ROM,DVD,externalharddrive(withstandardPCcompatibleinterface),(the"ProductionMedia").EachpieceofProductionMediawillbeassignedaproductionnumberorotheruniqueidentifyinglabelcorrespondingtothedateoftheproductionofdocumentsontheProductionMedia(e.g.,"DefendantMSLProductionApril1,2012")aswellasthesequenceofthematerialinthatproduction(e.g."-001","-002").Forexample,iftheproductioncomprisesdocumentimagesonthreeDVDs,theproducingpartymaylabeleachDVDinthefollowingmanner"DefendantMSLProductionApril1,2012","DefendantMSLProductionApril1,2012-002","DefendantMSLProductionApril1,2012-003."AdditionalinformationthatwillbeidentifiedonthephysicalProductionMediaincludes:(1)textreferencingthatitwasproducedindaSilvaMoorev.PublicisGroupeSA,etal.;and(2)theBatesNumberrangeofthematerialscontainedontheProductionMedia.Further,anyreplacementProductionMediawillcross-referencetheoriginalProductionMediaandclearlyidentifythatitisareplacementandcross-referencetheBatesNumberrangethatisbeingreplaced.

5.WriteProtectionandPreservation.Allcomputermediathatiscapableofwrite-protectionshouldbewrite-protectedbeforeproduction.

6.InadvertentDisclosures.ThetermsoftheParties'ClawbackAgreementandCourtOrdershallapplytothisprotocol.

7.DuplicateProductionNotRequired.Apartyproducingdatainelectronicformneednotproducethesamedocumentinpaperformat.

H.Timing.18of20

1.Totheextentatimeframeisnotspecificallyoutlinedherein,thepartieswillusetheirreasonableeffortstoproduceESIinatimelymannerconsistentwiththeCourt'sdiscoveryschedule.

2.ThepartieswillproduceESIonarollingbasis.

I.GeneralProvisions.

1.Anypracticeorproceduresetforthhereinmaybevariedbyagreementoftheparties,andfirstwillbeconfirmedinwriting,wheresuchvarianceisdeemedappropriatetofacilitatethetimelyandeconomicalexchangeofelectronicdata.

2.Shouldanypartysubsequentlydetermineitcannotingoodfaithproceedasrequiredbythisprotocol,thepartieswillmeetandconfertoresolveanydisputebeforeseekingCourtintervention.

3.ThePartiesagreethate-discoverywillbeconductedinphasesand,attheconclusionofthesearchprocessdescribedinSectionEabove,thePartieswillmeetandconferregardingwhetherfurthersearchesofadditionalcustodiansand/orthePhaseIIsourcesiswarrantedand/orreasonable.Ifagreementcannotbereached,eitherpartymayseekrelieffromtheCourt.

J.Plaintiffs'Objection

1.PlaintiffsobjecttothisESIProtocolinitsentirety.PlaintiffssubmittedtheirownproposedESIProtocoltotheCourt,butitwaslargelyrejected.TheCourtthenorderedthepartiestosubmitajointESIProtocolreflectingtheCourt'srulings.Accordingly,PlaintiffsjointlysubmitthisESIProtocolwithMSL,butreservetherighttoobjecttoitsuseinthiscase.

Thisprotocolmaybeexecutedincounterparts.Eachcounterpart,whensoexecuted,willbedeemedandoriginal,andwillconstitutethesameinstrument.

By:______________________By:__________________________JANETTEWIPPER,ESQ.BRETTM.ANDERS,ESQ.DEEPIKABAINS,ESQ.VICTORIAWOODINCHAVEY,ESQ.SIHAMNURHUSSEIN,ESQ.JEFFREYW.BRECHER,ESQ.JACKSONLEWISLLPSANFORDWITTELS&HEISLER,LLPAttorneysforDefendantMSLGROUPAttorneysforPlaintiffsandClass58SouthServiceRoad,Suite410555MontgomeryStreet,Ste.1206Melville,NY11747SanFrancisco,CA94111Telephone:(631)247-0404Telephone:(415)391-6900Date:___________________,2012Date:___________________,2012

SOORDERED.

[1]Tocorrectthemanyblogsaboutthiscase,initiatedbyapressreleasefromplaintiffs'vendor—theCourtdidnotorderthepartiestousepredictivecoding.Thepartieshadagreedtodefendants'useofit,buthaddisputesoverthescopeandimplementation,whichtheCourtruledon,thusacceptingtheuseofcomputer-assistedreviewinthislawsuit.

[2]Fromadifferentperspective,everypersonwhousesemailusespredictivecoding,eveniftheydonotrealizeit.The"spamfilter"isanexampleofpredictivecoding.

[3]Whendefensecounselmentionedthedisagreementaboutpredictivecoding,Istatedthat:"YoumusthavethoughtyoudiedandwenttoHeavenwhenthiswasreferredtome,"towhichMSL'scounselresponded:"Yes,yourHonor.Well,I'mjustthankfulthat,youknow,wehaveapersonfamiliarwiththepredictivecodingconcept."(12/2/11Conf.Tr.at8-9.)

[4]See,e.g.,SocieteNationaleIndustrielleAerospatialev.U.S.Dist.Ct.fortheS.D.ofIowa,482U.S.522,107S.Ct.2542(1987);seealsoTheSedonaConference,InternationalPrinciplesonDiscovery,Disclosure&DataProtection(2011),availableathttp://www.thesedonaconference.org/dltForm?did=IntlPrinciples2011.pdf.

[5]TheCourtalsosuggestedthatthebestwaytoresolveissuesaboutwhatinformationmightbefoundinacertainsourceisforMSLtoshowplaintiffsasampleprintoutfromthatsource.(2/8/12Conf.Tr.at55-56.)

[6]PlaintiffsincludedaparagraphnotingitsobjectiontotheESIProtocol,asfollows:

PlaintiffsobjecttothisESIProtocolinitsentirety.PlaintiffssubmittedtheirownproposedESIProtocoltotheCourt,butitwaslargelyrejected.TheCourtthenorderedthepartiestosubmitajointESIProtocolreflectingtheCourt'srulings.Accordingly,PlaintiffsjointlysubmitthisESIProtocolwithMSL,butreservetherighttoobjecttoitsuseinthiscase.

(ESIProtocolJ.1atp.22.)

[7]Rule26(g)(1)provides:

19of20

©2017eDiscoveryAssistantLLC.NoclaimtooriginalU.S.GovernmentWorks.

(g)SigningDisclosuresandDiscoveryRequests,Responses,andObjections.

(1)SignatureRequired;EffectofSignature.EverydisclosureunderRule26(a)(1)or(a)(3)andeverydiscoveryrequest,response,orobjectionmustbesignedbyatleastoneattorneyofrecordintheattorney'sownname....Bysigning,anattorneyorpartycertifiesthattothebestoftheperson'sknowledge,information,andbeliefformedafterareasonableinquiry:

(A)withrespecttoadisclosure,itiscompleteandcorrectasofthetimeitismade;and

(B)withrespecttoadiscoveryrequest,response,orobjection,itis:

(i)consistentwiththeserulesandwarrantedbyexistinglaworbyanonfrivolousargumentforextending,modifying,orreversingexistinglaw,orforestablishingnewlaw;

(ii)notinterposedforanyimproperpurpose,suchastoharass,causeunnecessarydelay,orneedlesslyincreasethecostoflitigation;and

(iii)neitherunreasonablenorundulyburdensomeorexpensive,consideringtheneedsofthecase,priordiscoveryinthecase,theamountincontroversy,andtheimportanceoftheissuesatstakeintheaction.

Fed.R.Civ.P.26(g)(1)(emphasisadded).

[8]Aspartofthisargument,plaintiffscomplainthatalthoughbothparties'experts(i.e.,vendors)spokeatthediscoveryconferences,theywerenotswornin.(Pls.Rule72(a)Objectionsat12:"Tohiscredit,theMagistrate[Judge]didaskthepartiestobring[totheconference]theESIexpertstheyhadhiredtoadvisethemregardingthecreationofanESIprotocol.Theseexperts,however,wereneverswornin,andthusthestatementstheymadeincourtatthehearingswerenotsworntestimonymadeunderpenaltyofperjury.")PlaintiffsneveraskedtheCourttohavetheexpertstestifytotheirqualificationsorbeswornin.

[9]Daubertv.MerrellDowPharms.,Inc.,509U.S.579,113S.Ct.2786(1993) .

[10]Thetougherquestion,raisedinKleinProds.LLCv.PackagingCorp.ofAm.beforeMagistrateJudgeNanNolaninChicago,iswhethertheCourt,atplaintiffs'request,shouldorderthedefendanttousecomputer-assistedreviewtorespondtoplaintiffs'documentrequests.

[11]TheRoitblatt,Kershaw,Ootarticlenotedthat"[t]helevelofagreementamonghumanreviewersisnotstrikinglyhigh,"around70-75%.Theyidentifytwosourcesforthisvariability:fatigue("Adocumentthatthey[thereviewers]mighthavecategorizedasresponsivewhentheyweremoreattentivemightthenbecategorized[whenthereviewerisdistractedorfatigued]asnon-responsiveorviceversa."),anddifferencesin"strategicjudgment."Id.at77-78.Anotherstudyfoundthatresponsiveness"isfairlywelldefined,andthatdisagreementsamongassessorsarelargelyattributabletohumanerror,"withonly5%ofreviewerdisagreementattributabletoborderlineorquestionableissuesastorelevance.MauraR.Grossman&GordonV.Cormack,InconsistentAssessmentofResponsivenessinE-Discovery:DifferenceofOpinionorHumanError?9(DESIIV:2011ICAILWorkshoponSettingStandardsforSearchingElec.StoredInfo.inDiscovery,ResearchPaper),availableathttp://www.umiacs.umd.edu/~oard/desi4/papers/grossman3.pdf.

[12]GrossmanandCormackalsonotethat"notalltechnology-assistedreviews...arecreatedequal"andthatfuturestudieswillbeneededto"addresswhichtechnology-assistedreviewprocess(es)willimprovemostonmanualreview."Id.

[13]SeeRalphC.Losey,"Child'sGameof`GoFish'isaPoorModelfore-DiscoverySearch,"inAdventuresinElectronicDiscovery209-10(2011).

[14]ItalsoavoidstheGIGOproblem,i.e.,garbagein,garbageout.

[1]AsnotedinParagraphsA(1)andJofthisProtocol,PlaintiffsobjecttothepredictivecodingmethodologyproposedbyMSL.

EndofDocument.

20of20