CS7650:NaturalLanguageProcessing

WeiXu(many slides from Greg Durrett)

Administrivia

‣ Coursewebsite: hAps://cocoxu.github.io/CS7650_fall2021/

‣ PiazzaandGradescope:‣ linksonthecoursewebsite‣ WewilldoourbesttomakesurequesNonsaboutthehomework,etc.getansweredwithin24hours

‣ TAOfficehours:TBA

CourseRequirements

‣ Priorexposuretomachinelearningveryhelpful

‣ Programming/Pythonexperience

‣ Probability

‣ LinearAlgebra

‣ MulNvariableCalculus

There will be a lot of math and programming!

FreeTextbooks!

‣ 2reallyawesomefreetextbooksavailable

‣ Therewillbeassignedreadingsfromboth

‣ Bothfreelyavailableonline

CourseworkPlan

‣ ProblemSet1(mathreview)willbereleasedlaterthisweekonGradeScope.

‣ 3ProgrammingProjects(40%;fairlysubstanNalimplementaNoneffort)

‣ TextclassificaNon

‣ NamedenNtyrecogniNon(BiLSTM-CNN-CRF)

‣ Neuralchatbot(Seq2SeqwithaAenNon)

‣ 2wriAenassignments(20%)+midtermexam(15%)

‣ MostlymathproblemsrelatedtoML/NLP

‣ Finalproject(20%;detailsoncoursewebsite,willdiscusslater)

{subject to change

ProgrammingProjects‣ ModernNLPmethodsrequirenon-trivialcomputaNon

‣ TrainingneuralnetworkswithmanyparameterscantakealongNme(itisaverygoodideatostartworkingontheassignmentsearly!)

‣ MostprogrammingwillbedonewithPyTorchlibrary(canbetrickytodebug)

‣ YouwillwanttouseaGPU

‣ GoogleColab:freeGPUs(somelimitaNons;proaccountfor$10/month)

‣ TheprogrammingprojectsaredesignedwithColabinmind

What’sthegoalofNLP?

‣ Beabletosolveproblemsthatrequiredeepunderstandingoftext

Siri,what’sthemostvaluableAmerican

company?

Apple

recognizemarketCapisthetargetvalue

recognizepredicate

docomputaNon

WhoisitsCEO?

‣ Example:dialoguesystems

resolvereferences

TimCook

AutomaNcSummarizaNon

…

…

OneofNewAmerica’swriterspostedastatementcriNcalofGoogle.EricSchmidt,Google’sCEO,wasdispleased.

Thewriterandhisteamweredismissed.

providemissingcontext

paraphrasetoprovideclarity

compresstext

MachineTranslaNon

TrumpPopefamilywatchahundredyearsayearintheWhiteHousebalcony

People’sDaily,August30,2017

NLPAnalysisPipeline

SyntacNcparses

CoreferenceresoluNon

EnNtydisambiguaNon

Discourseanalysis

Summarize

ExtractinformaNon

AnswerquesNons

IdenNfysenNment

‣ NLPisaboutbuildingthesepieces!Translate

TextAnalysis Applica/onsText Annota/ons

‣ AllofthesecomponentsaremodeledwithstaNsNcal approachestrainedwithmachinelearning

Howdowerepresentlanguage?Labels

Sequences/tags

Trees

Text

themoviewasgood +Beyoncéhadoneofthebestvideosofall6me subjec/ve

TomCruisestarsinthenewMissionImpossiblefilmPERSON MOVIE

Ieatcakewithicing

PPNP

S

NPVP

VBZ NNflightstoMiami

λx.flight(x)∧dest(x)=Miami

HowdoweusetheserepresentaNons?

Labels

Sequences

Trees

TextAnalysisText

‣MainquesNon:WhatrepresentaNonsdoweneedforlanguage?Whatdowewanttoknowaboutit?

‣ Boilsdownto:whatambiguiNesdoweneedtoresolve?

…

Applica/ons

Treetransducers(formachinetranslaNon)

ExtractsyntacNcfeatures

Tree-structuredneuralnetworks

end-to-endmodels …

Whyislanguagehard? (andhowcanwehandlethat?)

LanguageisAmbiguous!

‣ HectorLevesque(2011):“Winogradschemachallenge”(namedarerTerryWinograd,thecreatorofSHRDLU)

Thecitycouncilrefusedthedemonstratorsapermitbecausethey______violence

theyfeared

theyadvocated

‣ Thisissocomplicatedthatit’sanAIchallengeproblem!(AI-complete)

‣ ReferenNal/semanNcambiguity

LanguageisAmbiguous!

‣ AmbiguousNewsHeadlines:

slidecredit:DanKlein

‣ SyntacNc/semanNcambiguity:parsingneededtoresolvethese,butneedcontexttofigureoutwhichparseiscorrect

‣ TeacherStrikesIdleKids‣ HospitalsSuedby7FootDoctors‣ BanonNudeDancingonGovernor’sDesk‣ IraqiHeadSeeksArms

‣ StolenPainNngFoundbyTree‣ KidsMakeNutriNousSnacks‣ LocalHSDropoutsCutinHalf

LanguageisReallyAmbiguous!

‣ Therearen’tjustoneortwopossibiliNeswhichareresolvedpragmaNcally

‣ CombinatoriallymanypossibiliNes,manyyouwon’tevenregisterasambiguiNes,butsystemssNllhavetoresolvethem

Itisreallyniceout

ilfaitvraimentbeau It’sreallyniceTheweatherisbeauNfulItisreallybeauNfuloutsideHemakestrulybeauNful

ItfactactuallyhandsomeHemakestrulyboyfriend

‣ Lotsofdata!

slidecredit:DanKlein

Whatdoweneedtounderstandlanguage?

Whatdoweneedtounderstandlanguage?

‣ Worldknowledge:haveaccesstoinformaNonbeyondthetrainingdata

DOJgreenlightsDisney-Foxmerger

metaphor;“approves”

DepartmentofJus6ce

‣ Whatisagreenlight?Howdoweunderstandwhat“greenlighNng”does?

‣ Grounding:learnwhatfundamentalconceptsactuallymeaninadata-drivenway

McMahanandStone(2015)Gollandetal.(2010)

Whatdoweneedtounderstandlanguage?

‣ LinguisNcstructure‣ …butcomputersprobablywon’tunderstandlanguagethesamewayhumansdo

‣ However,linguisNcstellsuswhatphenomenaweneedtobeabletodealwithandgivesushintsabouthowlanguageworks

CenteringTheoryGroszetal.(1995)

Whatdoweneedtounderstandlanguage?

Whattechniquesdoweuse?(tocombinedata,knowledge,linguisNcs,etc.)

Unsup:topicmodels,grammarinducNon

Collinsvs.Charniakparsers

Abriefhistoryof(modern)NLP

1980 1990 2000 2010 2018

earlieststatMTworkatIBM

“AIwinter”rule-based,expertsystems

Penntreebank

NP VP

S

Ratnaparkhitagger

NNP VBZ

Sup:SVMs,CRFs,NER,SenNment

Neural

Pretraining

Semi-sup,structuredpredicNon

StructuredPredicNon

‣ SupervisedtechniquesworkwellonveryliAledata

annotaNon(twohours!)

unsupervisedlearning

‣ EvenneuralnetscandopreAywell!

“LearningaPart-of-SpeechTaggerfromTwoHoursofAnnotaNon” GarreAeandBaldridge(2013)

beAersystem!

‣ Allofthesetechniquesaredata-driven!Somedataisnaturallyoccurring,butmayneedtolabel

Bahdanauetal.(2014)DeNeroetal.(2008)

LessManualStructure?

Doesmanualstructurehaveaplace?

‣ Neuralnetsdon’talwaysworkoutofdomain!

MoosaviandStrube(2017)

‣ Coreference:rule-basedsystemsaresNllaboutasgoodasdeeplearningout-of-domain

‣ LORELEI:transiNonpointbelowwhichphrase-basedsystemsarebeAer

‣ Whyisthis?InducNvebias!

‣ CanmulN-tasklearninghelp?

Wikipedia

Newswire

TrumpPopefamilywatchahundredyearsayearintheWhiteHousebalcony

‣ Maybemanualstructurewouldhelp…

Doesmanualstructurehaveaplace?

Wherearewe?

‣ NLPconsistsof:analyzingandbuildingrepresentaNonsfortext,solvingproblemsinvolvingtext

‣ Theseproblemsarehardbecauselanguageisambiguous,requiresdrawingondata,knowledge,andlinguisNcstosolve

‣ Knowingwhichtechniquesuserequiresunderstandingdatasetsize,problemcomplexity,andalotoftricks!

‣ NLPencompassesallofthesethings

NLPvs.ComputaNonalLinguisNcs

‣ NLP:buildsystemsthatdealwithlanguagedata

‣ CL:usecomputaNonaltoolstostudylanguage

Hamiltonetal.(2016)

NLPvs.ComputaNonalLinguisNcs

‣ ComputaNonaltoolsforotherpurposes:literarytheory,poliNcalscience…

Bamman,O’Connor,Smith(2013)

CourseGoals

‣ CoverfundamentalmachinelearningtechniquesusedinNLP

‣ Makeyoua“producer”ratherthana“consumer”ofNLPtools

‣ CovermodernNLPproblemsencounteredintheliterature:whataretheacNveresearchtopicsin2021?

‣ Thethreeassignmentsshouldteachyouwhatyouneedtoknowtounderstandnearlyanysystemintheliterature

‣ UnderstandhowtolookatlanguagedataandapproachlinguisNcphenomena

Assignments

‣ 3ProgrammingAssignments(40%grade)

‣ ImplementaNon-oriented

‣ ~2weeksperassignment,3“slipdays”forautomaNcextensions

Theseprojectsrequireunderstandingoftheconcepts,abilitytowriteperformantcode,andabilitytothinkabouthowtodebugcomplexsystems.Theyarechallenging,sostartearly!

FinalProject

‣ Finalproject(20%grade)‣ Groupsof3-4preferred,1ispossible.‣ Goodideatotalktorunyourprojectideabymeinofficehoursoremail.

‣ 4pagereport+finalprojectpresentaNon.

QuesNons?

�33

Piazza — https://piazza.com/class/ksjq7xenrbp3g5