Post on 24-Dec-2015
1© Copyright 2008 EMC Corporation. All rights reserved.
Processor-Controlled TestUpdates and DFT Requirements
Jeffrey A MooreEMC Corp.
2© Copyright 2010 EMC Corporation. All rights reserved.
A Typical Storage Processor (SP) Core
CPU
XDP
DIMMs
PCT Pod
DO AS YOU’RE TOLD
Storage Processor Core
ICH
IOH
LAN
LAN
SIO
PCIe
NVRAMFLASH
POST
BIOS
COMM2
COMM1
LAN 1
LAN 0
PCT Software controls CPU through PCT POD & Adapter
CPU registers in turn initialize and operate busses at normal speeds
Tests progress outward from CPU one chip at a time until all chips are tested
No need for DRAM, or BIOS, or Flash memory to be working or loaded with data
Libraries of chip models make init and operations easy to program
3© Copyright 2010 EMC Corporation. All rights reserved.
Replace Custom Functional Test (CFT) w/ Basic Functional Test (BFT)
Test Stage Replacement
AOI
AXI
ICT ESS(FRU Point)
CFT CIT FC
AOI
AXI
ICT ESS(FRU Point)
BFT CIT FC
• Test Suite to Contain:
CMD Power Tests
Boundary Scan (BScan)
PCT Functional Test
Code Load,
Boot-up Custom Diag, POST, and Resume Write
4© Copyright 2010 EMC Corporation. All rights reserved.
Basic Functional Test (BFT) Improvements
BFT now combines 6 test steps into one– All steps using same test station, so no added tester costs or test stage– CMD Power test Checks voltages and margin status– BSCAN test checks shorts/opens and some memory tests– PCT test checks functionality of entire board– PCT / Embedded SAS Tests guarantee operation of 25 drive slots– Code load programs the CMD, MCU, BIOS, POST ROMs– Boot-up test allows board to boot and run POST to verify readiness for next
stage
CMD BSCAN PCT SAS LOOPCODE LOAD
BOOT-UP
5© Copyright 2010 EMC Corporation. All rights reserved.
Advantages of Combining all tests
CMD / BScan / PCT / Code Load merger into One Tester– Improved coverage significantly– Provides better diagnosis software at the CM for functional test– CMD provides quick debug of power problems– PCT reduces debug repair costs & eases lack of ability to use scope probes – PCT Diagnoses “No-Boot” problems to boost CFT, ESS, or CIT yield– SAS controller and SAS/Drive loop back bus tests provide quicker debug
and better coverage
6© Copyright 2010 EMC Corporation. All rights reserved.
BFT CMD Test Capabilities
CMD Power Sequencer Hardware (Control & Monitoring Device)– All CPU & I/O boards have a CMD chip (DSP w/many A/D and PCM ports)– CMD sequences, monitors & adjusts all DC-DC power circuits– If a power-up fault occurs, sequencer stores values of supplies before
shutting down– Communicates by TWI bus to test card’s USB-to-TWI pod
CMD Sequencer Test Software Features– Check CMD status– Change margins (normal/high/low) – User configuration data (Set margin limits)– Restore p-good limits to factory settings– Display Power down log (Useful for boards that won’t power up)– Reads the POL and Vin voltages and compare them against the PTR limit
7© Copyright 2010 EMC Corporation. All rights reserved.
Current SAS Testing at BFT
SAS Testing Carried out at BFT
– Check for access & training to the SAS Controller (read Device/Vendor ID)– Individually test 4 SAS Controller PHYs that connect to the SAS Expander – Test 4 SAS Controller PHYs connected to SAS Front Panel Expansion Port – Test access to the SAS Controller EEPROM and Flash– PHYs that connect the SAS Expander to the SAS Controller are tested (4
Direct and 4 via the SAS Cable.) Command sent to each Drive PHY to test device presence and speed The SAS Address of the Attached Device is read and the Device is registered. Buffer test is carried out on the Drive Buffer
8© Copyright 2010 EMC Corporation. All rights reserved.
Cost Savings Realized at BFT SAS Test
Removed SAS Controller Flash Prom.– Saved $2 per board - $1 for Flash part & $1 for Pre-programming it.– This eliminated ability to run SAS tests in BFT– Developed a method to store Flash data in memory and then have SAS
Controller download it from there over PCIe bus.– Now SAS test is running OK again.
Embedded Test Techniques Save Tester Cost– Removed expensive 6GB SAS drives (Qty. 25)– Replaced drives with loop-back midplane– Developed use of chip vendors’ internal HS serial loop back tests– Protocol agnostic – vendor supports loopback test– Requires some DFT in design and vendor supplied firmware– Reduced tester cost by $7500 and saved headaches from having to debug
drives through the fixture as they fail.– Able to show significant changes in the PMC DFE 0 and Gain values when a cap was
removed
9© Copyright 2010 EMC Corporation. All rights reserved.
SP (Storage Processor) w/ Custom 25-Drive Midplane
Low Speed GPIO’s
Midplane CPLD
I2C Slave interface
GPIO Expander
Loopbacks/CTRL
5V SB12V
SAS Expander
PCI-E
4x PCI-E (CMI Path)
Etch Loopback
Etch Loopback
Midplane CTRL
6GSAS
Controller
25x SAS 4x SAS
2x SAS
Ethernet to I2C
I2C TCP/IP
28-pinJTAG Conn
CMI
SAS Drives
Discrete Drive
Signals
Fault
Activity
Insert
Drive IO’s
10© Copyright 2010 EMC Corporation. All rights reserved.
Typical BFT Hardware
GbE or SFP Loopbacks*
SP(UUT)
A - sideHemlock-S30-pin TestConnector
TriggerTest
Board
J1J7
PCT 304x Pod
DPANehalem
ITP
J16
DPARibbon Cable
uMasterCable
GPIO Ribbon Cable
PC USB Port
SAS SXP Exp
GbE C – Management LAN
GbE D – Service LAN
RS232 Console
RS232 SPS
USB
Chassis W Midplane
010
1
USB to I2C
Aar
dvar
k U
SB
Cab
le
RS232 Loopbacks
Ethernet LAN cables
Test Box
SP(Captive)
B - side
USB Flash/Cable Assy
I/O Card
I/O Card
SAS Cable Loop0
0
I/O: 2 GbE LpbcksI/O: 4 SFP Lpbcks
SAS SPC Exp
0PC:ASSET
PCI-200EJCard
CMD / BScan/ PCT Scripts/ DIO Control
Port A
Port B
LOOP
BACK
MP
CMIPath
J3DB9
DB
9 F/M
I2C Arbitration
Logic
J1 DB9
Ribbon cable
11© Copyright 2010 EMC Corporation. All rights reserved.
Tests controlled and results displayed via custom “Block Diagram” GUI – GUI displays each chip tested and lights each one Green or Red as it is tested– Techs can use script language to flip bits, exercise buses, read back data, etc.– GUI written by Cork, Ireland Test Engineers using PCT vendor’s standard API
Test Time– Total BFT test time of UUT with 2 I/O cards is 13 minutes/unit– Code Load + Boot + POST is about 47 minutes – ouch!
Coverage– Was 51% coverage on robotic fault insertion vs 35-40% on CFT– Now have 87% coverage for BFT Tests– 90% diagnostic accuracy vs ~70% on CFT– In actual run of boards, of 11 faults found by PCT, 10 were predicted correctly to a
faulty device/device group
Yields– Prototype BFT yield was 83.5% (Boards had no ICT tests yet)– Subsequent CFT test had 92.8% yield– PCT test was improved to catch more faults, so final CFT yield is now 97.6%
EMC BFT Test Features
12© Copyright 2010 EMC Corporation. All rights reserved.
DFT Needed for PCT Test
XDP connector too expensive and weak
Route signals through front panel or midplane connector
TCK, TMS, TDI, TRST, TDO routed to Processor
Main board 3.3V reset & 1.1V Reset from Processor
2 Handshake signals from Processor (PREQ & PRDY)
Most are 1.1V levels so pay attention to signal quality
Need to keep round-trip-time short from test card – cable – CPU
EMC BFT runs at 10 MHz TCK
PCT can run up to 20MHz
Need to use mux to switch CPU signals between XDP & External Conn
Test card uses mux to switch BSCAN / PCT/ MCU signals to JTAG bus
13© Copyright 2010 EMC Corporation. All rights reserved.
Conclusions / Future Needs
Conclusions– PCT is now integral part of BFT test suite– BFT is now integral part of our test process– Achieved replacement of CFT stage on latest products– PCT Improves Diagnosability of BFT tests– Use of Embedded SAS tests reduced cost of tester– Decreases Time To Market for new products– Enables use of less experienced techs at CMs– Enables development of more embedded tests and instruments
Future Needs– Looking for ways to reduce Code Load time– Need to develop multiple test sites on one tester like CFT– Investigating use of many PCT pods over Ethernet to one tester