OFC2016 OpticalInternetworkingForum 1

OIFCEI-56GApplicationNoteCommonElectricalInterfaceat56Gb/s

Abstract:TheOIFiswellalonginthedevelopmentoftheCEI-56GsuiteofhighspeedinterconnectImplementationAgreements(IA)s.TheOIFisdevelopingIAsfornine56Gclauses,spanningfivereachesandthreemodulationtechniques.

TheOIFhasalonglegacyofdevelopingIAsforserialelectricalinterfaces,withtheCEIfamilybeingtheprincipalindustrydefinitionforSerDesforthe6Gb/s,11Gb/sand28Gb/sgenerations.InterfacesthatleveragetheCEIfamilyofspecificationshavebeenbuiltinmostofthemajorASICflowsandFPGAdevices.CEIhasbeenadoptedoradaptedintomultipleinterfacesbyIEEE802.3,InfiniBandandFibreChannel.

AbouttheOIF:

TheOIFfacilitatesthedevelopmentanddeploymentofinteroperablenetworkingsolutionsandservices.MemberscollaboratetodriveImplementationAgreements(IAs)andinteroperabilitydemonstrationstoaccelerateandmaximizemarketadoptionofadvancedinternetworkingtechnologies.OIFworkappliestoopticalandelectricalinterconnects,opticalcomponentandnetworkprocessingtechnologies,andtonetworkcontrolandoperationsincludingsoftwaredefinednetworksandnetworkfunctionvirtualization.TheOIFactivelysupportsandextendstheworkofnationalandinternationalstandardsbodies.Launchedin1998,theOIFistheonlyindustrygroupunitingrepresentativesfromacrossthespectrumofnetworking,includingmanyoftheworld’sleadingserviceproviders,systemvendors,componentmanufacturers,softwareandtestingvendors.InformationontheOIFcanbefoundathttp://www.oiforum.com.

Foradditionalinformationcontact:

TheOpticalInternetworkingForum,48377FremontBlvd,Suite117,

Fremont,CA94538USA

510-492-4040info@oiforum.comwww.oiforum.com

OFC2016 OpticalInternetworkingForum 2

TableofContents

Abstract:.......................................................................................................................................................................1

AbouttheOIF:.......................................................................................................................................................1

Glossary†.....................................................................................................................................................................4

TheOIFCEILegacy..................................................................................................................................................7

Introduction................................................................................................................................................................8

MotivationforCEI-56G..........................................................................................................................................9

CEI-56GChallenges...............................................................................................................................................12

DieArea.................................................................................................................................................................12

ChipPowerDissipation..................................................................................................................................12

PackagePowerDissipation...........................................................................................................................12

ChipPowerWiring............................................................................................................................................12

SystemPowerDissipation–Mid-planes.................................................................................................12

SystemPowerDissipation–Mid-boardoptics.....................................................................................13

I/ODensitiesonChipsandConnectors...................................................................................................13

ChannelMaterialsandCharacteristics....................................................................................................13

ChannelReach....................................................................................................................................................14

Cables.....................................................................................................................................................................14

Latency...................................................................................................................................................................14

Summaryofchallenges...................................................................................................................................14

ModulationTechniques.......................................................................................................................................16

OFC2016 OpticalInternetworkingForum 3

NRZ..........................................................................................................................................................................16

ENRZ.......................................................................................................................................................................16

PAM-4.....................................................................................................................................................................17

InterconnectReachesandApplicationSpaces..........................................................................................19

USR-DietoDieInterconnectWithinAPackage.................................................................................20

USR-DietoOpticalEnginewithinaPackage.......................................................................................21

XSR-ChiptoNearbyOpticalEngine.........................................................................................................21

XSR-CPUtoCPU...............................................................................................................................................22

XSR–DSPArrays...............................................................................................................................................22

XSR-CPUtoMemoryStack..........................................................................................................................22

VSR-ChiptoModule.......................................................................................................................................22

MR-ChiptoChipwithinaPCBA................................................................................................................23

LR–ChiptoChipacrossaBackplane/Midplane.................................................................................23

LR–ChiptoChipacrossaCable.................................................................................................................24

InteroperabilityPoints........................................................................................................................................24

DieBump...............................................................................................................................................................24

PackageBall.........................................................................................................................................................24

ModuleInterconnect........................................................................................................................................24

Summary....................................................................................................................................................................25

TableofFigures

Figure1InterconnectApplicationSpaces....................................................................................................8

Figure2ScalingofGates,Bumps,PinsandI/O(Ref.Xilinx,I/OLineadded)............................10

Figure3InterconnectChallenges..................................................................................................................11

OFC2016 OpticalInternetworkingForum 4

Figure4TypicalNRZEyeDiagramandStatisticalEyePlot...............................................................16

Figure5ENRZTypicalStatisticalEyes........................................................................................................17

Figure6TypicalPAM-4Eyes...........................................................................................................................18

Figure7CEI-56GReaches.................................................................................................................................19

Figure8UseofCEI-56GReaches...................................................................................................................20

Figure9USR-DietoDie....................................................................................................................................20

Figure10USR-DietoOpticalEngine..........................................................................................................21

Figure11XSRChiptoOpticalEngine..........................................................................................................21

Figure13MR-ChiptoChip..............................................................................................................................23

Figure14LRChiptoChipacrossabackplane..........................................................................................23

Figure15PackageBumpInteroperabilityPoints...................................................................................24

Figure16ChiptoModuleInteroperabilityPoint....................................................................................25

Glossary†2.5D:Referstoatypeofdie-to-dieintegrationviaasiliconinterposerhavingthrough-siliconvias(TSVs)connectingitstopandbottommetallayers

3D:Referstoathree-dimensional(3D)integrateddeviceinwhichtwoormorelayersofactiveelectroniccomponents(e.g.,integratedcircuitdies)areintegratedverticallyintoasinglecircuitwherethrough-siliconvias(TSVs)arecommonlyusedfordie-to-dieconnection.

ApplicationSpaces:Portionsofequipmentornetworkarchitecturethatcouldbenefitfromhavingadefinedsetofinterconnectionparameters.

ASIC:Anapplication-specificintegratedcircuitisanintegratedcircuit(IC)customizedforaparticularuse,ratherthanintendedforgeneral-purposeuse.

BCH:Bose,Ray-Chaudhuri,Hocquenghemforwarderrorcorrection(FEC)codesareaclassofadvancedcyclicerror-correctingcodesthatareconstructedusingfinitefields.

OFC2016 OpticalInternetworkingForum 5

BER:BitErrorRatioisthenumberofbiterrorsdividedbythetotalnumberoftransferredbitsduringastudiedtimeinterval.

BGA:BallGridArray,apackagetype

CDR:Clockanddatarecovery,ablockinareceiverthatrecreatesaclockorclocksinareceiverbylookingatthestatisticaledgeinformationinthereceiveddataandthenusesthatclockorclockstosamplethereceiveddata.

CEI:CommonElectricalInterface,anOIFImplementationAgreementcontainingclausesdefiningelectricalinterfacespecifications.

ClockForwarding:AclockcanbeforwardedinparallelwiththedatatoallowatransceivertoavoidusingCDRinordertosavepower.ThisistypicallydoneinUSRapplications.

EMB:Effectivemodalbandwidth,seeTIA-492AAAD.

ENRZ:EnsembleNonReturntoZero,amulti-wirecodeinwhich3bitsaremodulatedwiththeHadamardTransformontofourwires.

FEC:Forwarderrorcorrectiongivesareceivertheabilitytocorrecterrorswithoutneedingareversechanneltorequestretransmissionofdata.

FPGA:FieldProgrammableGateArray–areprogrammablelogicdevicethatoftenincludesSerDes.

FR4:Agradedesignationassignedtoglass-reinforcedepoxyprintedcircuitboards(PCB).

Gb/s:Gigabitspersecond.Thestatedthroughputordatarateofaportorpieceofequipment.Gb/sis1x109bitspersecond.

GBd:Thebaudrateistheactualnumberofelectricaltransitionspersecond,alsocalledsymbolrate.OneGigaBaudis1x109symbolspersecond.

IA:ImplementationAgreements,whattheOIFnamestheirdefinedinterfacespecifications.

IC:IntegratedCircuit

I/O:InputOutput,acommonnamefordescribingaportorportsonequipment

ISI:Inter-SymbolInterference,theeyeclosurecausedbytheenergyremainingonthechannelfrompreviousunitintervals,whichtypicallyimpactsthehorizontaleyeopening.

ISI-Ratio:Theratioofthelargestcodeeyetothesmallestcodeeye,whichisasimplemetrictoevaluatetheimpactofISIonhorizontaleyeclosureforagivencode.

OFC2016 OpticalInternetworkingForum 6

LR:LongReach–longenoughtoreachachiplocatedacrossabackplane

MCM:Multichipmodule,aspecializedelectronicpackagewheremultipleintegratedcircuits(ICs),semiconductordiesorotherdiscretecomponentsarepackagedontoaunifyingsubstrate,facilitatingtheiruseasasinglecomponent(asthoughalargerIC).

Mid-boardoptics:anopticaltransceiverthatismountedonaPCBAawayfromthePCBAedge,closetoaswitchASICtoreducetheamountofPCBAtracelossbetweenanASICandtheopticaltransceiver.ThisisincontrasttothecommonpracticetodayoflocatingopticaltransceiversatthePCBAedge.

MUX/DEMUX:Multiplex/demultiplex,amultiplexer(ormux)isadevicethatselectsoneofseveralanalogordigitalinputsignalsandforwardstheselectedinputintoasingleline,Conversely,ademultiplexer(ordemux)isadevicetakingasingleinputsignalandselectingoneofmanydata-output-lines,whichisconnectedtothesingleinput.

MR:MediumReach–longenoughtoreachachiplocatedacrossaPCBAincludingonadaughter-card

NRZ:NonReturntoZero,abinarycodeinwhich1sarerepresentedbyonesignificantcondition(usuallyapositivevoltage)and0sarerepresentedbysomeothersignificantcondition(usuallyanegativevoltage),withnootherneutralorrestcondition.

PAM:Pulseamplitudemodulation,aformofsignalmodulationwherethemessageinformationisencodedintheamplitudeofaseriesofsignalpulses.

PAM-4:Pulseamplitudemodulation-4isatwo-bitmodulationthatwilltaketwobitsatatimeandwillmapthesignalamplitudetooneoffourpossiblelevels.

PCBA:Printedcircuitboard(PCB)assembly,anassemblyofelectricalcomponentsbuiltonarigidglass-reinforcedepoxybasedboard.

RS:ReedSolomonFECcoding,thisisatypeofblockcode.Blockcodesworkonfixed-sizeblocks(packets)ofbitsorsymbolsofpredeterminedsize.Itcandetectandcorrectmultiplerandomandbursterrors.

SerDes:Serializer/Deserializer

Tb/s:Terabitspersecond.Thestatedthroughputordatarateofaportorpieceofequipment.Tb/sis1x1012bitspersecond

USR:Ultra-ShortReach–justlongenoughtoreachanotherdiewithinthesamepackage

VSR:Very-ShortReach–longenoughtoreachatransceiverinafront-panelopticalmodule

OFC2016 OpticalInternetworkingForum 7

XSR:eXtra-ShortReach–longenoughtoreachanearbydeviceormid-boardopticalmodule

†Somedefinitionsincludecontentfromwww.wikipedia.com

TheOIFCEILegacyTheOIFhasalonglegacyofdevelopingIAsforhighspeedelectricalinterfaces.TheCEIfamilyhasbeentheprincipalindustrydefinitionforSerDesforthe6Gb/s,11Gb/sand28Gb/sgenerations.TheOIFSxIinterfacewasCEI’simmediatepredecessorandservedthe3Gb/sgeneration.TheOIFSPI/SFI3and4electricalinterfacesservedthe800Mb/sand1.6Gb/sgenerations.Thislonghistoryof7generationsandsixdoublingsgivestheOIFauniqueviewpointontheneedsoftheindustryfortodayandtomorrow.

InterfacesthatsupporttheCEIfamilyofspecificationshavebeenbuiltinmostofthemajorASICflowsandFPGAdevices.CEIhasbeenadopted,adapted,orinfluencedthedevelopmentofmultipleinterfacesincludingIEEE802.3,InfiniBandandFibreChannel,SATA,SAS,RapidIOandHyperTransportinterfacesaswellasnumerousproprietaryprotocols.

Name Rateperpair Year Adopted,AdaptedorInfluenced

CEI-56G 56Gb/s 2016 Thefutureisbright

CEI-28G 28Gb/s 2011 InfiniBandEDR,32GFC,SATA3.2,100GBASE-KR4and100GBASE-CR4,CAUI4,SAS-4

CEI-11G 11 2008 InfiniBandQDR,10GBASE-KR,10GFC,16GFC,SAS-3,RapidIOv3

CEI-6G 6 2004 4GFC,8GFC,InfiniBandDDR,SATA3.0,SAS-2,RapidIOv2,HyperTransport3.1

OFC2016 OpticalInternetworkingForum 8

SxI5 3.125 2002-3 Interlaken,FC2G,InfiniBandSDR,XAUI,10GBASE-KX4,10GBASE-CX4,SATA2.0,SAS-1,RapidIOv1

SPI4,SFI4 1.6 2001-2 SPI-4.2,HyperTransport1.03

SPI3,SFI3 0.800 2000 (fromPL3)

Introduction

Figure1InterconnectApplicationSpaces

www.oiforum.com 8

2 Introduction

2.1 Purpose The OIF Next Generation Interconnect Framework identifies application spaces for next generation systems and identifies areas for future work by the OIF and other standards bodies. The scope of this document explores Next Generation (NG) Interconnects that are limited to data center or intra-office applications which are generally less than 2km from both an electrical and optical perspective. Virtual Platforms in the cloud is an example of just one of the applications that will take advantage of NG Interconnect technology to achieve higher data bandwidths in a smaller footprint with better energy efficiency.

Figure 1 Interconnect Application Spaces

As shown in Figure 1, interconnection interfaces in a typical system are needed for chip-to-chip within a module, chip to chip within a PCBA (printed circuit board assembly), between two PCBAs over a backplane/midplane, or between two chassis’. These interfaces may be unidirectional or bi-directional, optical or electrical, and may support a range of data rates. For each application space, the IAs that follow from this framework should identify requirements to support interoperability across the various application spaces for optical and electrical links. They may include, but not be limited to:

� Cost Considerations � Link Performance � Power Consumption

OFC2016 OpticalInternetworkingForum 9

AsshowninFigure1,interconnectioninterfacesinatypicalsystemareneededforchip-to-chipwithinamodule,chiptochipwithinaPCBA(printedcircuitboardassembly),betweentwoPCBAsoverabackplane/midplane,orbetweentwochassis’.Theseinterfacesmaybeunidirectionalorbi-directional,opticalorelectrical,andmaysupportarangeofdatarates.

Considerationsfortheselinksmayinclude:

• Cost• LinkPerformance• PowerConsumption• Channellossbudgets• ChoiceofPCBmaterial• Modulationtechniqueandsignallevels• Numberoflanes,channelconfigurationandgeneralcharacteristics• Referenceclockjitter• ForwardedClockorCDR• Latency• Connectorperformance• Reliability• Size• OperatingTemperature

MotivationforCEI-56GNextgenerationsystemsarebeingdrivenbytheneedtohandletheincreasingvolumeofdatatraffic.Atthesametime,thenextgenerationsystemsareconstrainedbylimitsonpowerconsumption,bylimitsonthesizeofasystem,andbytheneedtoprovideacosteffectivesolution.

Theseneedsdrivenextgenerationsystemstoeverincreasingcommunicationportdensities.Theincreaseddensityleadstosmallersurfaceareasavailabletodissipatetheheatgeneratedandthereforerequiresdecreasedpowerconsumptionforaport.

Theindustrycurrentlyhaselectricalinterfacesfor10Gb/s(OIF’sCEI-11G),25Gb/s&28Gb/s(OIF’sCEI-25/28G)inproduction,andworkiscomingalongon56Gb/s(OIF’sCEI-56G).However,channelsproducedwithcopperPCBtracesareseverelybandwidthlimited,andbecauseofthisitisincreasinglydifficulttoachievethesamelinkdistancesusinghighersignalingrates.

ImprovementsinICintegration,whichhasbeenrelentlesslydrivenbyMoore’sLawoverpastdecades,haveenabledhigherdensitiesoflogicgatesatescalatinghigherclockratesto

OFC2016 OpticalInternetworkingForum 10

beusedinICdesigns.Thesetrendshaveallowedtheindustrytodeployincreasinglymorecomplexcommunicationsystemsateachgenerationtomeettheinfrastructureneeds.

However,asonedigsdeeper,thefutureappearstobechallenging.Manydifferenttechnologiesneedtoconvergetoimprovethethroughput.ThesecomplexICshaveincreasinggatecounts,butthepowerdissipationpergateandI/Ospeedhavenolongerscaledatthesamerateasthegatecount.Inaddition,thenumbersofelectricalconnections(bumpsandpackagepins)arealsonotscalingatthesamerate-leadingtoapower,capacityandportcountgapforthenextgenerationinterconnectinterfaces.

Figure2ScalingofGates,Bumps,PinsandI/O(Ref.Xilinx,I/OLineadded)

Thepredominantinterconnectchallengestoovercomearepresentedbelowinasolutionspacediagram,andarediscussedingreaterdetailinsubsequentsub-sections.

OFC2016 OpticalInternetworkingForum 11

Figure3InterconnectChallenges

OFC2016 OpticalInternetworkingForum 12

CEI-56GChallenges

DieArea

SerDesimplementingCEI-56GareexpectedtobelargerthanSerDessupportingCEI-28Ginterfacesbecauseoftheneedformoreadvancedequalization.Additionallythesizeoftheanalogportionoftheinterfacesdonotscaleaswellasthedigitalportions.

Thisareasituationhasrenderedsomepreviouschiparchitecturesunuseable.ThishasledtoatrendofSerDes“out-boarding”whereanUltra-ShortReach(USR)oreXtraShortReach(XSR)interfaceisputonthemainASICandthelongerreachSerDesareputinthesamepackageornearby.

ChipPowerDissipation

SerDesimplementingCEI-56GareexpectedtobehigherpowerthanSerDessupportingCEI-28Ginterfacesbecauseoftheneedformoreadvancedequalization.Additionallythepowerdissipationoftheanalogportionoftheinterfacesdonotscaleaswellasthedigitalportions.

ThishasfurtheredthetrendmentionedaboveofSerDes“out-boarding”whereanUltra-ShortReach(USR)interfaceisputonthemainASICandthelongerreachSerDesareputinthesamepackage.

PackagePowerDissipation

Whenpackagepoweristhekeyconstraint,theuseofanXSRinterfacecanhelpmovesomeofthepowerdissipationoutsideofthemainASICandontootherchipsnearbyonthePCBA.

ChipPowerWiring

Lowervoltagepowersourcesareusedwithsmallertechnologynodes.Theneteffectofhigherintegrationoflowvoltagesemiconductorsisasignificantincreaseintotaldevicecurrent,whichrequireshighcurrentsourcepowersupplieswhichmustbecontrolledwithinseveralmVtolerances,whichinturnrequiresadditionalpowerpinsperdevicetoaccommodatethehighelectricalcurrents.

SystemPowerDissipation–Mid-planes

Becausesystemcoolingcanescalatebeyondphysicallimits,multiplepowersavingstrategiesmaybeneeded.

OFC2016 OpticalInternetworkingForum 13

SincetheI/Opowerisrelatedtothedistancethattheelectricalsignalstravelforagivenchannel’sproperties,reducingthedistancethattheelectricalsignalmustbedrivencanreducepowerdissipation.Onetechniquethathasbeenusedistousemid-planes.

Withthisphysicalarchitecture,aswitchboardismountedhorizontallybehindtheverticallymountedfrontboards.Withthisarchitecture,theelectricallinksareshorterandonlyhavetotraverseoneconnector.

SystemPowerDissipation–Mid-boardoptics

Insomecases,itmaymakesensetointegratetheopticsclosetotheASICpackage,therebyminimizingtheneedtodriveelectricalsignalsfar.Thesearesometimescalledopticalenginesandthetechniqueisreferredtoasusingmid-boardoptics.

I/ODensitiesonChipsandConnectors

ThemaximumnumberofusefulI/Osforhighspeedseriallinksperdeviceisnotonlylimitedbytheavailablepackagetechnologyitself,butalsobytheabilitytoroutethedeviceonthePCBA.

Inordertomaintainsignalintegrityforahighspeedseriallinkdesign,itisrequiredtobeabletorouteadifferentialpairbetweentwopackageballswhenescapingfromtheinnerballrowsofaballgridarray(BGA),anditthereforemaybecomemorecostlytousepackageswithaballpitchbelow1.0mm.

Inaddition,foreveryballrowfromtheedgeofthepackageonwhichdifferentialpairshavebeenplaced,aseparatecircuitpackagelayerhastobeused.Differentialpairsintheouter4rowsoftheBGArequires4signallayersonthePCBA,whilethe6outerrowswouldrequire6layers,andthusthePCBlayerstackgrowswitheveryinnerBGArowtoberouted.

ChannelMaterialsandCharacteristics

Linkapplicationsarecharacterizedbythesupportedlossbudgetandsignalimpairments.Lossisdeterminedbylinklength,boardmaterials,back-drilling,viatypeusage,thenumberofconnectors,connectortypes,andpackagematerials.Increasesinsignalingratecausemorefrequency-dependentlossandthusalowerSignaltoNoiseRatio(SNR).

Thedeviationofthelossfromasmoothcurveisalsoimportantastheripplesinalosscurvearenoteasilyequalizable.ThisiscalledInsertionLossDeviation(ILD).

Ingreenfieldapplications,advancedmaterialsforthechannel(PCBandconnector)canresultinandecreasedlossandcanthussupporteitheranincreasedelectricaldatarateoradecreasedamountofrequiredequalization.

OFC2016 OpticalInternetworkingForum 14

ChannelReach

Incommunicationsystemsusuallythefrontplateareaisoccupiedbypluggablemodulesforinter-systemcommunications,whiletheintra-systemtrafficbetweenthePCBAsisconnectedoverthebackplane/midplane.Tosupportreasonablesystemdimensions,systembackplane/midplaneconnectionstypicallyneedtobridgedistancesofupto70–100cm.Insomemorerecentapplications,alternativechannelarchitecturesreplacethebackplanewithdirectplugorthogonal(DPO)structureswhereI/Olinecardsonthefrontsideoftheequipmentdirectlyplugintoswitchfabriclinecardsontherearsideoftheequipment,thusdirectlyconnectingthelinecardswithoutthelossofthebackplanebetweenthem

Theintroductionofrepeaterdevicesintothedatapathcanincreasetheeffectivereachattheexpenseofpowerandboardarea.

Cables

Twinax,micro-coaxand/orflexcircuitsarealsoanoptiontoreducelossandextendreachforthehighspeedlinks,particularlyinsystemswithfewerhighspeedlinks.Insomecircumstances,thiscanallowthemainPCBAtobebuiltfromlowcostmaterial.InanothercasecableassembliesarenowbeingintegratedontobackplaneconnectorandusedinlieuofPCBbackplanesduetotheirabilitytoreducelossandextendreach.Theuseofcablesbringstheirownshareofchallenges.

Latency

CPU-to-CPUandCPU-to-MemoryapplicationsofCEIareoftenintolerantoflatency.ThisoftenprecludestheuseofheavyweightFECintheseapplications.Thecharacteristicmethodthatisintolerantoflatencyistheuseofcredit-basedflowcontrolwhereareceivergrantsthetransmittertherighttosendacertainamountofdata.Onlyamoderateamountofbufferingistypicallyprovidedinthesesystems,mostofteninmorecostlySRAM.MostofthemajorCPUtoCPUprotocolsusecredit-basedflowcontrol.

Summaryofchallenges

Astimeproceeds,ICswillbecomefasteranddenser.Tocopewiththeissueofinterconnectcapacityanddensityoffuturesystems,photonicinterconnectswillbecomeanevenmoreimportantconnectiontechnology.

Theimplementationof56Gb/sInterconnectstechnologyposesseveralchallengesespeciallyinrelationto:diearea,chippowerdissipation,packagepowerdissipation,systempowerdissipation,limitedI/Odensity,channelloss,channelreach,andlatency.Highlightedweretheside-effectsofsomesolutions,whicharearesultofthecomplexinter-

OFC2016 OpticalInternetworkingForum 15

dependenciesof:higherintegration,complexmodulationschemes,chipbreak-outandrouting,signalconditioning,thermal&powerissues,andpackagefootprint.

OFC2016 OpticalInternetworkingForum 16

ModulationTechniquesAdvancedmodulationtechniquescanbeusedtolowertheNyquistfrequencyrequirementforalinkusingabandwidthlimitedelectricalchannel.ThesetechniquesrequireahigherSNRforequivalentBER.Theprovisionofchannelswithmorecorrelationstomeasureagainstcanalsohelp.Forwarderrorcorrection(FEC)canincreasetheeffectiveSNRandthereforethesupportedlossbudgetattheexpenseofhigherpowerdissipation,complexity,andlatency.Bandwidthlimitationsandincreasesinsignalingrateresultinimpairmentsthatmaybecompensatedbyusingequalizationtechniques,whichthemselvesalsoimpactpowerconsumptionandcomplexity.

NRZ

NRZ(NonReturntoZero)isthemodulationtechniqueusedbymostcurrentspeedlinks.Itrequiresachannelwithtwocorrelatedconductors.Itusestwosignallevelsandconveysonebitperbaud.Tosupport56Gb/sperpair,NRZrunsatasymbolrateof56GBaudandhasaNyquistfrequencyof28GHz.IthasanISI-Ratioofone,meaningtheratioofthelargesteyetothesmallesteyeisalwaysthesame.

At56Gb/s,NRZisusefulinapplicationswherethechannel’sfrequencydependentlossisnotexcessive.Thatlosshasbeenfoundtobeexcessiveinmanylongreachapplications.

Figure4TypicalNRZEyeDiagramandStatisticalEyePlot

ENRZ

EnsembleNRZ(ENRZ)requiresachannelwithfourcorrelatedconductors.Usefulchannelscanbeconstructedwithtwocorrelatedlooselycoupledpairs.Itusesfoursignallevelsandconveysthreebitsperbaudoverthefourconductorsbymodulatingthethreesub-channelscollectivelyusingtheHadamardmatrix.ItTosupporta56Gb/sperpaireffectiverate,ENRZrunsatasymbolrateof

�������� �������� ��������������������������������� � � � � �

Figure 9 shows the LR SERDES implemented in adiscrete retiming device. with b At the 56G data rate thereare proposals for PAM4 and ENRZ modulation.

�+)52( �

�+)52(

NRZ vs PAM4

The OIF made the decision to specify modulationformats of both NRZ and PAM4 for a number of reaches.This decision was based on the lack of clear technicaldata showing that either was a superior solution. Thiswhite paper addresses only a few of the comparisonsbetween NRZ and PAM4. A more detailed analysis is leftto the proponents of each modulation scheme.

Why NRZ?

NRZ has been the proven modulation technique forprinted circuit board electrical interconnects since thebeginning of digital communications. As data rates haveincreased, the use of first linear equalization and thendecision feedback equalization have kept NRZ as a viabletechnology. At both 10G and 25G data rates specificationshave been written for PAM4 based LR/backplane solutionsbut NRZ has been the clear market winner. Figure 10shows a 56G NRZ eye diagram.

�+)52( �� �����#�(9(�'+$)2$.

The use of two level signaling (0 and 1) provides a 9dBSNR advantage over PAM4 (4 levels). This provides ageneral rule of thumb that the insertion loss of a givenchannel measured at the PAM4 nyquist frequency shouldbe 9dB less than the insertion loss at the NRZ Nyquistfrequency (2x the PAM4 Nyquist frequency) before PAM4should be considered. Using this rule of thumb on the 28GVSR insertion loss curves extended to 56G (See Figure X)shows NRZ to be the best choice.

Why PAM4?

As Common Electrical Interfaces have increased indata rates system vendors have always requested supportto the same printed circuit board trace distances. This hasresulted in the use of lower loss PCB materials andimprovements in equalization techniques. At some pointthe additional ’knob’ of higher order modulation needs tobe used to achieve the same trace distance with higherdata rates. PAM4 modulation uses 4 voltage levels toencode 2 bit of data in single baud period. This result in aNyquist frequency that is 1/2 that required for NRZ. FigureZ shows a PAM4 eye diagram.

�������� �������� ��������������������������������� � � � � �

Figure 9 shows the LR SERDES implemented in adiscrete retiming device. with b At the 56G data rate thereare proposals for PAM4 and ENRZ modulation.

�+)52( �

�+)52(

NRZ vs PAM4

The OIF made the decision to specify modulationformats of both NRZ and PAM4 for a number of reaches.This decision was based on the lack of clear technicaldata showing that either was a superior solution. Thiswhite paper addresses only a few of the comparisonsbetween NRZ and PAM4. A more detailed analysis is leftto the proponents of each modulation scheme.

Why NRZ?

NRZ has been the proven modulation technique forprinted circuit board electrical interconnects since thebeginning of digital communications. As data rates haveincreased, the use of first linear equalization and thendecision feedback equalization have kept NRZ as a viabletechnology. At both 10G and 25G data rates specificationshave been written for PAM4 based LR/backplane solutionsbut NRZ has been the clear market winner. Figure 10shows a 56G NRZ eye diagram.

�+)52( �� �����#�(9(�'+$)2$.

The use of two level signaling (0 and 1) provides a 9dBSNR advantage over PAM4 (4 levels). This provides ageneral rule of thumb that the insertion loss of a givenchannel measured at the PAM4 nyquist frequency shouldbe 9dB less than the insertion loss at the NRZ Nyquistfrequency (2x the PAM4 Nyquist frequency) before PAM4should be considered. Using this rule of thumb on the 28GVSR insertion loss curves extended to 56G (See Figure X)shows NRZ to be the best choice.

Why PAM4?

As Common Electrical Interfaces have increased indata rates system vendors have always requested supportto the same printed circuit board trace distances. This hasresulted in the use of lower loss PCB materials andimprovements in equalization techniques. At some pointthe additional ’knob’ of higher order modulation needs tobe used to achieve the same trace distance with higherdata rates. PAM4 modulation uses 4 voltage levels toencode 2 bit of data in single baud period. This result in aNyquist frequency that is 1/2 that required for NRZ. FigureZ shows a PAM4 eye diagram.

�������� �������� ��������������������������������� � � � � �

Figure 9 shows the LR SERDES implemented in adiscrete retiming device. with b At the 56G data rate thereare proposals for PAM4 and ENRZ modulation.

�+)52( �

�+)52(

NRZ vs PAM4

The OIF made the decision to specify modulationformats of both NRZ and PAM4 for a number of reaches.This decision was based on the lack of clear technicaldata showing that either was a superior solution. Thiswhite paper addresses only a few of the comparisonsbetween NRZ and PAM4. A more detailed analysis is leftto the proponents of each modulation scheme.

Why NRZ?

NRZ has been the proven modulation technique forprinted circuit board electrical interconnects since thebeginning of digital communications. As data rates haveincreased, the use of first linear equalization and thendecision feedback equalization have kept NRZ as a viabletechnology. At both 10G and 25G data rates specificationshave been written for PAM4 based LR/backplane solutionsbut NRZ has been the clear market winner. Figure 10shows a 56G NRZ eye diagram.

�+)52( �� �����#�(9(�'+$)2$.

The use of two level signaling (0 and 1) provides a 9dBSNR advantage over PAM4 (4 levels). This provides ageneral rule of thumb that the insertion loss of a givenchannel measured at the PAM4 nyquist frequency shouldbe 9dB less than the insertion loss at the NRZ Nyquistfrequency (2x the PAM4 Nyquist frequency) before PAM4should be considered. Using this rule of thumb on the 28GVSR insertion loss curves extended to 56G (See Figure X)shows NRZ to be the best choice.

Why PAM4?

As Common Electrical Interfaces have increased indata rates system vendors have always requested supportto the same printed circuit board trace distances. This hasresulted in the use of lower loss PCB materials andimprovements in equalization techniques. At some pointthe additional ’knob’ of higher order modulation needs tobe used to achieve the same trace distance with higherdata rates. PAM4 modulation uses 4 voltage levels toencode 2 bit of data in single baud period. This result in aNyquist frequency that is 1/2 that required for NRZ. FigureZ shows a PAM4 eye diagram.

OFC2016 OpticalInternetworkingForum 17

37.3GBaudandhasaNyquistfrequencyof18.6GHz.IthasanISI-Ratioofone,meaningtheratioofthelargesteyetothesmallesteyeisalwaysthesame.ENRZcanoftenbeusedwithoutForwardErrorCorrection,evenonsomelongreachchannels.

Ata56Gb/seffectiverate,ENRZisusefulinlongerreachapplications,particularlywhentheuseofFECisnotdesired.

Figure5ENRZTypicalStatisticalEyes

PAM-4

PAM-4requiresachannelwithtwocorrelatedconductors.Itusesfoursignallevelsandconveystwobitsperbaud.Tosupport56Gb/sperpair,PAM-4runsatasymbolrateof28GBaudandhasaNyquistfrequencyof14GHz.IthasanIntersymbolInterferenceRatio(ISI-Ratio)ofthree,meaningthelargestcodeeyeisthreetimesaslargeasthesmallestcodeeye.Ithasaverticalimpairmentof9dBascomparedtoNRZandhasahorizontalimpairmentduetoitshighISI-Ratio.PAM-4isusuallypairedwithequalizationandForwardErrorCorrection.

At56Gb/sPAM-4isusefulinapplicationswherethechannel’sfrequencydependentlosswouldotherwisebeexcessive.

25

25

•  The chart below graphs the horizontal opening vs. modulation technique over a TE Connectivity Channel with equal throughput per-wire

•  PAM-4 (red) and NRZ (orange) are very similar, ENRZ (purple) is consistently better

Horizontal eye opening analysis Channel for comparison

Source: TE Connectivity

Differential

ENRZPAM-4

Comparison Eye Diagrams

PAM-4 @ 14 GHz NyquistNRZ @ 28 GHz Nyquist ENRZ @ 18.66 GHz Nyquist

ISI-ratio = 1!but frequency high

ISI-ratio = 3!bad ISI performance

ISI-ratio = 1!reasonable frequency!balanced eye shape

Same scale for each!112 Gb/s throughput over 4 wires for each

OFC2016 OpticalInternetworkingForum 18

Figure6TypicalPAM-4Eyes

�������� �������� ��������������������������������� � � � � �

�+)52( �������(9(�'+$)2$.

PAM modulation has been used for many years as thesolution of higher data rates over Category 3 and 5 twistedpair wiring for Ethernet. Figure Y shows an example of a28G VSR compliant channel that would benefit from theuse of PAM4 modulation to double the data rate with nochanges in PCB materials. Note that the loss difference atthe NRZ Nyquist frequency vs the PAM4 Nyquistfrequency is well beyond 9dB.

�+)52( ��������&*$//(-

The CEI 56G Technology ShowcaseIn an effort to highlight the progress of the OIF in the

CEI 56G projects a technology showcase was developedwith the intent of demonstrating technical feasibility of thedifferent reaches. There are 6 demonstrations fromvarious members of the OIF.

Demo 1: CEI-56G VSR NRZ QSFP channel demoThis demonstration shows a working VSR channel

using Test equipment from Tektronix to generate a50Gbps NRZ signal which is sent thru commerciallyavailableQSFP connectors on test boards from Multilaneand Yamaichi. (See Figure ?)The receiver is acommercially available NRZ receiver from Credo. TheBER of the system is shown on a PC using the BER testerin the Credo receiver.

�+)52( �����(.0��

Demo 2: CEI-56G-VSR PAM4 QSFP Complianceboard

The ability to transmit and receive PAM4 signals acrossa commercially available connectors is shown in Figure 14using test equipment from Tektronix, Multilane andAnritsu. One demonstration uses a Tektronix PAM4 signalgenerator to transmit PAM4 formatted data over a QSFPconnector using test boards from Molex and TEConnectivity. The PAM4 receiver is provided by Multilane.An additional demonstration uses an Anritsu PAM4 datagenerator sending signals through a CFP connector usingtest boards from Yamaichi and Multilane. The PAM4receiver is provided by Tektronix.

�������� �������� ��������������������������������� � � � � �

�+)52( �������(9(�'+$)2$.

PAM modulation has been used for many years as thesolution of higher data rates over Category 3 and 5 twistedpair wiring for Ethernet. Figure Y shows an example of a28G VSR compliant channel that would benefit from theuse of PAM4 modulation to double the data rate with nochanges in PCB materials. Note that the loss difference atthe NRZ Nyquist frequency vs the PAM4 Nyquistfrequency is well beyond 9dB.

�+)52( ��������&*$//(-

The CEI 56G Technology ShowcaseIn an effort to highlight the progress of the OIF in the

CEI 56G projects a technology showcase was developedwith the intent of demonstrating technical feasibility of thedifferent reaches. There are 6 demonstrations fromvarious members of the OIF.

Demo 1: CEI-56G VSR NRZ QSFP channel demoThis demonstration shows a working VSR channel

using Test equipment from Tektronix to generate a50Gbps NRZ signal which is sent thru commerciallyavailableQSFP connectors on test boards from Multilaneand Yamaichi. (See Figure ?)The receiver is acommercially available NRZ receiver from Credo. TheBER of the system is shown on a PC using the BER testerin the Credo receiver.

�+)52( �����(.0��

Demo 2: CEI-56G-VSR PAM4 QSFP Complianceboard

The ability to transmit and receive PAM4 signals acrossa commercially available connectors is shown in Figure 14using test equipment from Tektronix, Multilane andAnritsu. One demonstration uses a Tektronix PAM4 signalgenerator to transmit PAM4 formatted data over a QSFPconnector using test boards from Molex and TEConnectivity. The PAM4 receiver is provided by Multilane.An additional demonstration uses an Anritsu PAM4 datagenerator sending signals through a CFP connector usingtest boards from Yamaichi and Multilane. The PAM4receiver is provided by Tektronix.

�������� �������� ��������������������������������� � � � � �

�+)52( �������(9(�'+$)2$.

PAM modulation has been used for many years as thesolution of higher data rates over Category 3 and 5 twistedpair wiring for Ethernet. Figure Y shows an example of a28G VSR compliant channel that would benefit from theuse of PAM4 modulation to double the data rate with nochanges in PCB materials. Note that the loss difference atthe NRZ Nyquist frequency vs the PAM4 Nyquistfrequency is well beyond 9dB.

�+)52( ��������&*$//(-

The CEI 56G Technology ShowcaseIn an effort to highlight the progress of the OIF in the

CEI 56G projects a technology showcase was developedwith the intent of demonstrating technical feasibility of thedifferent reaches. There are 6 demonstrations fromvarious members of the OIF.

Demo 1: CEI-56G VSR NRZ QSFP channel demoThis demonstration shows a working VSR channel

using Test equipment from Tektronix to generate a50Gbps NRZ signal which is sent thru commerciallyavailableQSFP connectors on test boards from Multilaneand Yamaichi. (See Figure ?)The receiver is acommercially available NRZ receiver from Credo. TheBER of the system is shown on a PC using the BER testerin the Credo receiver.

�+)52( �����(.0��

Demo 2: CEI-56G-VSR PAM4 QSFP Complianceboard

The ability to transmit and receive PAM4 signals acrossa commercially available connectors is shown in Figure 14using test equipment from Tektronix, Multilane andAnritsu. One demonstration uses a Tektronix PAM4 signalgenerator to transmit PAM4 formatted data over a QSFPconnector using test boards from Molex and TEConnectivity. The PAM4 receiver is provided by Multilane.An additional demonstration uses an Anritsu PAM4 datagenerator sending signals through a CFP connector usingtest boards from Yamaichi and Multilane. The PAM4receiver is provided by Tektronix.

OFC2016 OpticalInternetworkingForum 19

InterconnectReachesandApplicationSpacesThefiveCEI-56Greachescanbesummarizedinthefollowingtable:

Figure7CEI-56GReaches

Theuseofthesereachesforinterconnectapplicationspacescanbebrokendownasfollows.

OFC2016 OpticalInternetworkingForum 20

Figure8UseofCEI-56GReaches

USR-DietoDieInterconnectWithinAPackage

Figure9USR-DietoDie

Itmaybenecessarytousemultipledieswithinamulti-chipmodule(MCM)toachievethepowerobjectives.Theseco-packagedsolutionscancommunicatewithlesspowersincethesubstrateprovidesahighqualitycommunicationchannel.

Thecommunicationchannelislessthan10mm.Thisshortelectricallinkmayallowforamuchsimplerinterfaceandrequirelesspowerthananexistingstandardelectricalinterface.Forexample,equalizationisunlikelytobeneededanditmaybepossibletoassumesuchshortlinksaresynchronous(singlereferenceclockgoingtoallchips),removingtheneedforafrequencytrackingCDR.

���� ����-2%0&!#%1�&.0���

�

��

�,�)�����'�

���

���

��

������

OFC2016 OpticalInternetworkingForum 21

Atypicaluseforthisistooff-boardtheSerDesfromaswitchASIC.

ClockforwardingcanbeusedintheplaceofCDRforUSRlinks.

USR-DietoOpticalEnginewithinaPackage

Figure10USR-DietoOpticalEngine

Itmaybenecessarytouseadieandanopticalenginewithinamulti-chipmodule(MCM)toachievetheindustry’sobjectives.Theseco-packagedsolutionscancommunicatewithlowpowersincethesubstrateprovidesahighqualitycommunicationchannel.

Thecommunicationchannelwouldtypicallybelessthan10mm.Thisshortelectricallinkmayallowforamuchsimplerinterfaceandrequirelesspowerthananexistingstandardelectricalinterface.

XSR-ChiptoNearbyOpticalEngine

Figure11XSRChiptoOpticalEngine

Itmaybeusefultoplaceanopticalinterfaceveryclosetothehostchip.SomeopticaldevicescannotsitwithinahostMCMduetoheatrestrictionsoftheopticalcomponents.Inthiscase,ashortelectricallinkoflessthan50mmisanticipated.Theshortreachofthischannelallowspowertobesaved.Theseopticalmodulesareoftencalledmid-boardoptics.

OFC2016 OpticalInternetworkingForum 22

XSR-CPUtoCPU

CPUscanbeconnectedviaashortconnectionoverhighrateSerDes.SomeCPUsuseacoherencyprotocoltomakethedifferentprocessorsappeartobeinthesamecoherencydomain.

Itisgenerallynotpossibletouseheavy-weightForwardErrorCorrectioninCPUapplicationsduetothelatencyrequirementsoftheseinterfaces,whichtypicallyusecredit-basedflowcontrol.

XSR–DSPArrays

DSPsaresometimesconnectedinarraystoprocesshighrateinformationsuchasfromaRADARoraLIDAR.Anautonomousvehicleisanexampleofsuchanapplication.Sometimestheinterfacesareunidirectionalandothertimes,bidirectional.BotharetypicallylatencysensitivesimilartoCPUsaseventheunidirectionalapplicationsoftenhavecomplex,timesensitivecontrolflows.

XSR-CPUtoMemoryStack

CPUscanbeconnectedviaashortconnectiontoamemorystack.Aclassicwaytoaccomplishthisistohaveadevicemadewithalogicprocessthatformsthebaselayerdevicefortheactuallymemorydies.

Itisnotpossibletouseheavy-weightForwardErrorCorrectioninthisapplicationduetothelatencyrequirementsofmemoryinterfaces.CPUthreadsorstatemachinesareoftenwaitingfortheresultsofthememoryaccess.Memorycacheshelp,butdonoteliminatethesignificantperformancehitsthatCPUsseewhenlatencyisadded.

VSR-ChiptoModule

Itiscommoninmoderncommunicationsystemstosupportpluggablemodulesatthefrontfaceplateoftheequipment.Thisfacilitateslowcostinitialdeploymentoftheequipmentifsomeportsareleftunpopulated.Apayasyougopolicyisthenuseduntiltheentirefaceplateispopulatedwithpluggablemodules.Theelectricallinkusedtoconnectthesepluggablemodulescanextendtobeyond30cm.Athigherdataratesthischallengestheabilityofthehostchiptodrivetheselongtracelengthswithinthepowerconstraintsoflargeswitchchips.Placingretimingdevicesinsidethepluggablemoduleprovidessupportforlongerhosttracesbuttheinclusionofcomplexequalizationfeaturescanoverburdenthelimitedpowerbudgetsofthepluggablemodule.Advancedmodulationformats(suchasPAMorDMTschemes),ForwardErrorCorrection(FEC)andequalizationfeaturesareallpossiblesolutionsforthechiptomoduleinterconnect.

OFC2016 OpticalInternetworkingForum 23

OnetechniquethathasbeenusedistoengineertheelectricallinkstohavealowernativeBERthanthatoftheopticallink.Inthisway,asingleFECcanbeusedtoprotectaCEI-56Glinkateachendandanopticallinkinthemiddle.

MR-ChiptoChipwithinaPCBA

Figure12MR-ChiptoChip

AninterconnectioninterfacemaybeneededbetweentwochipsonthesamePCBAoronadaughtercardorshortermid-plane.Bydefinition,thisinterfaceisrelativelyshortrangingupto50cm.Thisinterfacecouldincludeasingleconnector.

LR–ChiptoChipacrossaBackplane/Midplane

Figure13LRChiptoChipacrossabackplane

Thisinterfacecommunicatesbetweentwocardsacrossabackplane/midplanewithinachassisandislessthan1mwithupto2connectors.

FECmaybearequirementtomeettheBER–howeverthechoiceoftheFECmustbeconsideredcarefullytoaddressbothlatencyandpowerconcerns.PossibleFECimplementationsincludeRSorBCH.

OFC2016 OpticalInternetworkingForum 24

LR–ChiptoChipacrossaCable

WhilenotanexplicitgoaloftheCEI-56Gproject,cabledversionsofCEI-28Ghavebeenusedtosupportbothbackplaneandchassistochassisinterconnects.

InteroperabilityPointsCEI-56Gdefinesthreedifferentinteroperabilitypointsforthevariousreaches.ThesearethepointsatwhichcompliancethetheOIFIAscanbedeterminedbymeasuringthepropertiesofanimplementationandcomparingthoseresultstotheparametersintheIA.

DieBump

CEI-56G-USRdefinesinteroperabilityatthediebumpofaflip-chipsemiconductordevice

PackageBall

CEI-56G-XSR,CEI-56G-MRandCEI-56G-LRalldefinesinteroperabilityatthepackageball.

Figure14PackageBumpInteroperabilityPoints

OIFchosepackageballsastheinteroperabilitypointsforthesethreespecificationsbecausetheentirechannelistypicallytheresponsibilityofthesamecompany.XSRdoesnothaveaconnectorandthereforenootherinteroperabilitypointisavailable.ForMRandLR,whichdohaveconnectors,thesystemdesignerstillownstheentirechannelandthereforeitdoesnotmakesensetoconstrainhowthesystemdesignerallocatesthelinkbudget.

ModuleInterconnect

CEI-56G-VSRdefinesinteroperabilityatthemoduleinterconnect.

OFC2016 OpticalInternetworkingForum 25

Figure15ChiptoModuleInteroperabilityPoint

Hostcomplianceboards(HCB)andmodulecomplianceboards(MCB)aredefinedtomeasurecompliance.

SummaryTheCEI-56Gfamilyofclausesdefinesasetofinterfacesthatcanbeusedtosolvemosthighspeedinterconnectneeds.Fiveseparatereachesandthreeseparatemodulationtechniquescansupporttheneedsofmostapplications.