12/4/2002 Lab Board A description of the board you will do measurements on.
-
Upload
augustine-white -
Category
Documents
-
view
217 -
download
2
Transcript of 12/4/2002 Lab Board A description of the board you will do measurements on.
12/4/2002
Lab Board
A description of the board you will do measurements on
12/4/2002Introduction
2
Lab Board Description Code name is ‘King’s Canyon’
Referred to hereafter as ‘KC’You will do measurements on this board in the lab
Background for KCLow-end server boardInexpensive board for customers who want servers
ServerComputer which serves data to othersSupports more memory, more processing power, and more I/O (i.e., network bandwidth, hard drives) than desktop computersMore reliable (crashes less and recovers better) than desktop computers
12/4/2002Introduction
3
Server in a network
``
`
Server
Desktop PCPC
PC
Shared Printer
Network Switch
12/4/2002Introduction
4
Inside the Server
Plug in memory boardsGigabytes of memory
2 CPUs
MCH
Hard drive storage
Network (Ethernet)to network switch
Server Circuit Board
12/4/2002
Circuit Board Block Diagram
Front Side Bus (FSB)
Front Side Bus (FSB)
Hub Interface
Hub Interface
SS
SS
SS
SS
SSSS
SS – Source Synchronous
CC- Common Clock
CC
CC
CC
CC
CC
4 PCI-X busses4 PCI-X busses
Low speed 33MHz PCI bus
Low speed 33MHz PCI bus
CPU 0
CPU 1
MCH
Memory
Memory
Two (2) DDR (Double Data Rate) Memory Busses
Two (2) DDR (Double Data Rate) Memory Busses
ICHVideo
P64H2 HI to PCI bridge
P64H2 HI to PCI bridge
Details to Follow ….
12/4/2002
Components on KC
‘the chips’
12/4/2002Introduction
7
CPU CPUs –Central Processing Units
The brains of the computer that ‘run’ your programRuns the Operating System (e.g. windows) and all the softwareSpeed of CPU has large effect on speed of system
Currently measured in GHz (e.g., 3.0 GHz)Consumer pc’s typically have 1 cpu inside the box – called ‘UP’
KC has 2 cpus – this is called ‘DP’, i.e. dual processorSome boards have 4+ cpus, they are called ‘MP’, i.e. multi-processor
KC uses Intel Xeon CPU’sCurrently at 3GHzSimilar to Pentium 4
Bigger cache (superfast memory inside CPU) Can operate with other CPUs on same bus
The Silicon die (2) is under this metal heat spreader (1). The die is significantly smaller than the heat spreader. A ‘Fansink’ mounts to the heat spreader to keep die cool.
12/4/2002Introduction
8
MCH
MCH – Memory Controller HubHelps CPU communicate with rest of system
The ‘gatekeeper’ to the outside world from perspective of CPUTalks to memory, CPU, and I/O
I/O means busses and chips and boards which ultimately connect to things like hard drives, networks, keyboards, monitor, etc..
Maintains memorytells it when to refresh, etc.. (dynamic memory has to be refreshed or it forgets the data it holds)Detects errors in what is stored in memory (ECC) and periodically corrects these errors
KC uses Intel E7501 MCH
http://www.intel.com/design/chipsets/e7501/index.htm?iid=ipp_srvr_proc_xeon+e7501&
Actual Die: Passive (no fan) Heatsink mounts on top.
Package decoupling capacitors. Try to hold voltage rails (vcc) steady. Closer to the die the better.
12/4/2002Introduction
9
ICH
ICH – Input/Output Controller HubI/O device that deals with critical but slow speed devices
Video controller (basic 2-d video on servers, not meant for 3-d graphics)
Talks to Keyboard, mouse via ‘Super IO’ chip BIOS (contains software which computer runs when it’s
first powered on. BIOS gets computer up and running and then passes control to the OS)
IDE Hard drive (slow hard drive that OS boots off of. The big capacity hard drives do not connect to ICH)
Helps with initial bootup of systemTalks to mch over a slow version of the Hub Interface busKC uses Intel 82801CA ICH
Still uses wire-bond technology. Doesn’t require heatsink.
12/4/2002Introduction
10
Memory Modules Memory – High speed storage place for code and data used
by CPUMuch faster than hard drive storage, but not as massive in quantity
2-16 Gigabytes vs. 100’s of GigabytesSlower than CPU cache (small amount of memory internal to CPU)
Memory die’s are inside wire-bond packages called ‘DRAMs’Dynamic Random Access Memory
Many DRAMs solder to a memory board called the ‘DIMM’Dual In-line Memory ModuleAnywhere from 9 to 36 DRAMs on one memory board2 types of memory boards: Unbuffered and Registered. Unbuffered is used in desktop. KC (and most servers) only uses Registered DIMMs.
Memory board plugs into KC board via a connector.KC can support up to 8 DIMMs
Many different DIMM manufacturersMicron, Samsung, Elpida,Infineon, …
DRAM
2 DIMMs
http://www.micron.com/products/modules/ddrsdram/index.html
12/4/2002Introduction
11
PCI-bridge P64H2 – PCIX 64-bit to Hub Interface 2 bridge
Gateway to High Speed I/O devices such as hard drive arrays adapters or ethernet adaptersTalks to MCH on one side
Via high speed variey of ‘Hub Interface’ BusTalks to 2 separate pci-busses on other side
Several connectors connect to each pci busUser can slide a card into each of these connectorsTypically Hard Disk Controller Cards
Each Provides several SCSI or SATA connection to hard drive Can end up with quite a few hard drives and large storage
capacityOr Network Adapater Cards
Provide several 100Mb/s ethernet connections, which go to a switch or router
Servers tend to read data from the hard drive into memory, process the data with the CPUs, and then send it out to the requesting computer via the network (e.g., web server)
‘P64H2’ is the code name for the Intel 82870P2
P64H2 die, may require passive heatsink
12/4/2002Introduction
12
Identifying parts on the board CPUs
there are two of them and they have really huge fansinks.NOTE: If the fansink doesn’t run, and the system is on, the heatsink will get VERY hot (burn skin) and system will eventually shut itself off.
Memory memory consists of 8 wide connectors all in a row.
At least 2 DIMMs must be plugged in (‘populated’) for system to boot up. There are rules regarding what kind of dimms to use and which order to plug them into the connectors.
MCHIt’s the chip between the cpus and the memoryProbably has a passive heatsink (a small heatsink w/ no fan)
ICH & P64H2’sHard to find, but it looks like the chip pictured in the above slideThere is one ICH and 2 P64H2’s (second one not always populated, i.e. sometimes it’s missing)
PCI SlotsLarger connectors on opposite side of board from DIMM connectors.One slot is from ICH and is slow speed legacy PCI bus (33MHz, 32-bit)The rest are high speed (66-133MHz, 64-bit – i.e. 64 data bits wide)
12/4/2002
Interconnects on KC
‘the wires that connect the chips together’
12/4/2002Introduction
14
Front Side Bus (FSB) Function: CPUs talk to each other and to the MCH. This is
how the CPUs get data and code from memory, and communicate with rest of the system.
Technology: Source synchronous533MBit/second/wire (266MHz signals)
This means each data wire on this bus can send 533 million bits of data in one second
64 data bits wideSo the whole bus can transfer 64*533 million bits of data per second
Topology: Multi-drop bus. CPU1 has to listen to when CPU0 acceses memory because CPU1 may have that memory data in it’s cache
‘Cache Coherency’ Each cpu has a small superfast memory inside it called the ‘cpu cache’. The cache holds recently accessed memory data. If one cpu holds the data for a memory address that another cpu needs, it has to let it know not to get that data from main memory.
Termination: Parallel termination at both ends of bus (MCH and CPU0)
12/4/2002Introduction
15
FSB Topology
VTT = 1.5V
50 W
INSIDE DIE
Package Trace Board Trace Board Trace
VTT = 1.5V
50 W
INSIDE DIE
Package Trace
INSIDE DIENote: no 50 ohm
termination
Package Trace
MCHCPU #0
‘Middle Agent’
CPU #1‘End Agent’
12/4/2002Introduction
16
FSB Signal Groups Data Signal Group
HD[63:0]# – 64 bits of dataDINV[3:0]# - dynamic inversionHDSTBP[3:0]# - P strobesHDSTBN[3:0]# - N strobes
Example, HDSTBP[0]# & HDSTBN[0]# are used to strobe in HD[0-16]# & DINV[0]#
E.g., two strobe signals are used to clock in a set of data bits, more on this later
Address Signal GroupSwitches at ½ the rate of the FSB Data SignalsHAB[35:3]# - Address Bits. E.g., what memory address is being requestedHADSTB[1:0]# - Strobes for address signal group
HADSTB[0]# strobes A[16:3]#, HADSTB[1]# strobes the rest Common Clock Signals
There a whole lot of common clock signalsHCLKINP, HCLKINN – Common clock for all signals in the ‘Host Clock Domain’, i.e. all common clock signals, and outer loop timings for strobes (also called BCLK)
12/4/2002Introduction
17FSB Timings – 4X Source Synchronous
Timings not published for 533 MHz FSB, 400 MHz timing diagrams shown
12/4/2002Introduction
18FSB Timings – 2X Source Synchronous
Timings not published for 533 MHz FSB, 400 MHz timing diagrams shown
12/4/2002Introduction
19DDR (Double Data Rate) Memory Bus Function: MCH reads and writes data from memory over
this high speed bus. Signaling Technology: Source Synchronous
266Mbit/sec/wire (133Mhz signals)A DDR bus is 64 data bits wideKC’s MCH uses 2 DDR busses in parallel, so it likes having a 128-bit wide bus
Bandwidth is 128*266 Million bits per second
Topology:Multi-Drop Bus. Each of the two separate instances of this bus on this system go to four (4) DIMM slots.
Each DIMM can have up to 2GBytes of memory, for a max total of 16GB.
Topology and TimingsThe subject of the next class
12/4/2002Introduction
20Memory speed and size within a system
CPUL1
L2
L3
CPUL1
L2
L3
MCH MAIN MEMORYMAIN MEMORYMAIN MEMORYMAIN MEMORY
PCI BRIDGE(P64H2)
Adapter Card(s)
Fas
ter
Ret
rieva
l to
CP
U
Greater D
ensity
When we study DDR signaling and timings, it will be clear the overhead and latency that main memory exhibits, and thus the need for local L1-L3 CPU cache.
Hard Drives
12/4/2002Introduction
21
HI (Hub Interface) Bus Function: Intel proprietary bus to provide flexible high
speed data transfer from MCH to various downstream components
Signaling Technology: Source SynchronousHI1: From MCH to ICH: 266Mbit/sec/wire (133Mhz signals)
8 data bits wide, bandwidth = 8*266 Million bits per second
HI2: From MCH to P64H2’s: 533Mbit/sec/wire (266Mhz signals)
16 data bits wide, bandwidth = 16*533 Million bits per second
Topology: Point to Point. Termination: There are parallel terminations at both ends of the bus
12/4/2002Introduction
22
Hub Interface Topology
80 W
INSIDE DIE
Package Trace Board Trace
INSIDE DIE
Package Trace
50 W
80 W
50 W
VCC=1.2VCC=1.2
12/4/2002Introduction
23
Hub Interface Signal Groups
Data BitsHI[15:0] – Data SignalsPSTRB_[1:0], PSTRB_[1:0]# – Strobe Signals
PSTRB_[0] & PSTRB_[0]# strobe in HI[0:7]PSTRB_[1] & PSTRB_[1]# strobe in HI[15:8]
OtherHI[18:16] – Command Signals
Common Clock signals
GCLKIN66MHz common clock for HI[18:16] and outer loops of the PSTRB’s
Timings not published for Hub Interface
12/4/2002Introduction
24
PCI and PCI-X Function: Industry standard bus for plug-in peripherals
In servers, quite often plug-in hard drive adapters or network adapters are connected to the pci bus
Technology: Common ClockPCI-X: From P64H2 – 133Mbit/sec/wire, clocks are 133MHz signals, data is 66MHz
64 data bits wide, bandwidth = 64*133 Million bits per secondPCI: From ICH – 33Mbit/sec/sire, clocks are 33MHz, data is 16MHz
32 data bits wide, bandwidth = 32*33 Million bits per secondThis speed and width are common in current desktop pc pci slots
Topology: Multi-Drop Bus. The bus consists of devices soldered down on the board and slots for plug-in devices. They all dangle off the PCI bus.
Termination: Series terminated inside the transmitting agent. This signaling technolgy depends on a doubling of the waveform at the end of the bus (due to open-circuit) in order to function.
12/4/2002Introduction
25
PCIX-133 2-slot Topology
Package Trace
Board Trace
INSIDE DIE
Board Trace
INSIDE DIE INSIDE DIE
Plug-in Board Trace
Plug-in Board Trace
Package Trace
Package Tracex
xx
12/4/2002Introduction
26
PCI-X Signal Groups
Data and Address use same signals:AD[63:0]Common Clocked
Control SignalsDEVSEL, TRDY, … a bunch moreCommon Clocked
Asynchronous SignalsREQ, GNT – Arbitration signalsInterrupt Signals
ClocksPCLK[6:0] – goes to every device that hangs off the pci bus including the bridge itself
12/4/2002Introduction
27PCI-X Data to Common Clock Timings
ns
ns
12/4/2002Introduction
28
PCI-X Common Clock
12/4/2002Introduction
29
KC Clock Distribution
CK 408 Chip
P64H2 #1
P64H2 #2
CPU #2
MCH
14.31818MHz
66MHz
66MHz
CPU #1133MHz
133MHz133MHz
Differential Clock
Differential ClockDifferential Clock
66MHz
PCI to Gbit Ethernet
PCI BUS B
PCI BUS A
133 MHz
100 MHz
Dual Channel
SCSI
PCI BUS A
PCI BUS A
PCI “Riser Card”
Miscellaneous Other ClocksLPC (low pin count) Legacy BusVideo ClockICH PCI SlotOther Clocks used by ICH...
DDR Bus A Slots
DDR Bus B Slots
12/4/2002Introduction
30
Magic Decoder Ring Names for Signals given so far are from the device sheets KC Board uses it’s own names
Board’s have to name signals clearly and uniquely across the whole boardDevices only have to name then uniquely across the device
Decoder Ring:FSB: HD[63:0]# FSB_HD63_N
Rule is append FSB_ in front and replace ‘#’ with ‘_N’ in the device signal name
HI: HI[15:0] P64H2_1_HI[15:0]Rule is append P64H2_1_ before the device signal nameNote that the ‘1’ in P64H2_1_ may be a ‘1’ or a ‘2’ because KC has two P64H2 devices
PCI-X: AD[63:0] P64H2_1_PB_AD[63:0]Rule is append P64H2_1_PB to beginning of device signal nameNote that the ‘1’ in P64H2_1_PD.. Can be a ‘2’ or ‘3’ … depending on which pci bus the signal belongs too
DDR: Given Later