1© Copyright 2008 EMC Corporation. All rights reserved.

Processor-Controlled TestUpdates and DFT Requirements

Jeffrey A MooreEMC Corp.

2© Copyright 2010 EMC Corporation. All rights reserved.

A Typical Storage Processor (SP) Core

CPU

XDP

DIMMs

PCT Pod

DO AS YOU’RE TOLD

Storage Processor Core

ICH

IOH

LAN

LAN

SIO

PCIe

NVRAMFLASH

POST

BIOS

COMM2

COMM1

LAN 1

LAN 0

PCT Software controls CPU through PCT POD & Adapter

CPU registers in turn initialize and operate busses at normal speeds

Tests progress outward from CPU one chip at a time until all chips are tested

No need for DRAM, or BIOS, or Flash memory to be working or loaded with data

Libraries of chip models make init and operations easy to program

3© Copyright 2010 EMC Corporation. All rights reserved.

Replace Custom Functional Test (CFT) w/ Basic Functional Test (BFT)

Test Stage Replacement

AOI

AXI

ICT ESS(FRU Point)

CFT CIT FC

AOI

AXI

ICT ESS(FRU Point)

BFT CIT FC

• Test Suite to Contain:

CMD Power Tests

Boundary Scan (BScan)

PCT Functional Test

Code Load,

Boot-up Custom Diag, POST, and Resume Write

4© Copyright 2010 EMC Corporation. All rights reserved.

Basic Functional Test (BFT) Improvements

BFT now combines 6 test steps into one– All steps using same test station, so no added tester costs or test stage– CMD Power test Checks voltages and margin status– BSCAN test checks shorts/opens and some memory tests– PCT test checks functionality of entire board– PCT / Embedded SAS Tests guarantee operation of 25 drive slots– Code load programs the CMD, MCU, BIOS, POST ROMs– Boot-up test allows board to boot and run POST to verify readiness for next

stage

CMD BSCAN PCT SAS LOOPCODE LOAD

BOOT-UP

5© Copyright 2010 EMC Corporation. All rights reserved.

Advantages of Combining all tests

CMD / BScan / PCT / Code Load merger into One Tester– Improved coverage significantly– Provides better diagnosis software at the CM for functional test– CMD provides quick debug of power problems– PCT reduces debug repair costs & eases lack of ability to use scope probes – PCT Diagnoses “No-Boot” problems to boost CFT, ESS, or CIT yield– SAS controller and SAS/Drive loop back bus tests provide quicker debug

and better coverage

6© Copyright 2010 EMC Corporation. All rights reserved.

BFT CMD Test Capabilities

CMD Power Sequencer Hardware (Control & Monitoring Device)– All CPU & I/O boards have a CMD chip (DSP w/many A/D and PCM ports)– CMD sequences, monitors & adjusts all DC-DC power circuits– If a power-up fault occurs, sequencer stores values of supplies before

shutting down– Communicates by TWI bus to test card’s USB-to-TWI pod

CMD Sequencer Test Software Features– Check CMD status– Change margins (normal/high/low) – User configuration data (Set margin limits)– Restore p-good limits to factory settings– Display Power down log (Useful for boards that won’t power up)– Reads the POL and Vin voltages and compare them against the PTR limit

7© Copyright 2010 EMC Corporation. All rights reserved.

Current SAS Testing at BFT

SAS Testing Carried out at BFT

– Check for access & training to the SAS Controller (read Device/Vendor ID)– Individually test 4 SAS Controller PHYs that connect to the SAS Expander – Test 4 SAS Controller PHYs connected to SAS Front Panel Expansion Port – Test access to the SAS Controller EEPROM and Flash– PHYs that connect the SAS Expander to the SAS Controller are tested (4

Direct and 4 via the SAS Cable.) Command sent to each Drive PHY to test device presence and speed The SAS Address of the Attached Device is read and the Device is registered. Buffer test is carried out on the Drive Buffer

8© Copyright 2010 EMC Corporation. All rights reserved.

Cost Savings Realized at BFT SAS Test

Removed SAS Controller Flash Prom.– Saved $2 per board - $1 for Flash part & $1 for Pre-programming it.– This eliminated ability to run SAS tests in BFT– Developed a method to store Flash data in memory and then have SAS

Controller download it from there over PCIe bus.– Now SAS test is running OK again.

Embedded Test Techniques Save Tester Cost– Removed expensive 6GB SAS drives (Qty. 25)– Replaced drives with loop-back midplane– Developed use of chip vendors’ internal HS serial loop back tests– Protocol agnostic – vendor supports loopback test– Requires some DFT in design and vendor supplied firmware– Reduced tester cost by $7500 and saved headaches from having to debug

drives through the fixture as they fail.– Able to show significant changes in the PMC DFE 0 and Gain values when a cap was

removed

9© Copyright 2010 EMC Corporation. All rights reserved.

SP (Storage Processor) w/ Custom 25-Drive Midplane

Low Speed GPIO’s

Midplane CPLD

I2C Slave interface

GPIO Expander

Loopbacks/CTRL

5V SB12V

SAS Expander

PCI-E

4x PCI-E (CMI Path)

Etch Loopback

Etch Loopback

Midplane CTRL

6GSAS

Controller

25x SAS 4x SAS

2x SAS

Ethernet to I2C

I2C TCP/IP

28-pinJTAG Conn

CMI

SAS Drives

Discrete Drive

Signals

Fault

Activity

Insert

Drive IO’s

10© Copyright 2010 EMC Corporation. All rights reserved.

Typical BFT Hardware

GbE or SFP Loopbacks*

SP(UUT)

A - sideHemlock-S30-pin TestConnector

TriggerTest

Board

J1J7

PCT 304x Pod

DPANehalem

ITP

J16

DPARibbon Cable

uMasterCable

GPIO Ribbon Cable

PC USB Port

SAS SXP Exp

GbE C – Management LAN

GbE D – Service LAN

RS232 Console

RS232 SPS

USB

Chassis W Midplane

010

1

USB to I2C

Aar

dvar

k U

SB

Cab

le

RS232 Loopbacks

Ethernet LAN cables

Test Box

SP(Captive)

B - side

USB Flash/Cable Assy

I/O Card

I/O Card

SAS Cable Loop0

0

I/O: 2 GbE LpbcksI/O: 4 SFP Lpbcks

SAS SPC Exp

0PC:ASSET

PCI-200EJCard

CMD / BScan/ PCT Scripts/ DIO Control

Port A

Port B

LOOP

BACK

MP

CMIPath

J3DB9

DB

9 F/M

I2C Arbitration

Logic

J1 DB9

Ribbon cable

11© Copyright 2010 EMC Corporation. All rights reserved.

Tests controlled and results displayed via custom “Block Diagram” GUI – GUI displays each chip tested and lights each one Green or Red as it is tested– Techs can use script language to flip bits, exercise buses, read back data, etc.– GUI written by Cork, Ireland Test Engineers using PCT vendor’s standard API

Test Time– Total BFT test time of UUT with 2 I/O cards is 13 minutes/unit– Code Load + Boot + POST is about 47 minutes – ouch!

Coverage– Was 51% coverage on robotic fault insertion vs 35-40% on CFT– Now have 87% coverage for BFT Tests– 90% diagnostic accuracy vs ~70% on CFT– In actual run of boards, of 11 faults found by PCT, 10 were predicted correctly to a

faulty device/device group

Yields– Prototype BFT yield was 83.5% (Boards had no ICT tests yet)– Subsequent CFT test had 92.8% yield– PCT test was improved to catch more faults, so final CFT yield is now 97.6%

EMC BFT Test Features

12© Copyright 2010 EMC Corporation. All rights reserved.

DFT Needed for PCT Test

XDP connector too expensive and weak

Route signals through front panel or midplane connector

TCK, TMS, TDI, TRST, TDO routed to Processor

Main board 3.3V reset & 1.1V Reset from Processor

2 Handshake signals from Processor (PREQ & PRDY)

Most are 1.1V levels so pay attention to signal quality

Need to keep round-trip-time short from test card – cable – CPU

EMC BFT runs at 10 MHz TCK

PCT can run up to 20MHz

Need to use mux to switch CPU signals between XDP & External Conn

Test card uses mux to switch BSCAN / PCT/ MCU signals to JTAG bus

13© Copyright 2010 EMC Corporation. All rights reserved.

Conclusions / Future Needs

Conclusions– PCT is now integral part of BFT test suite– BFT is now integral part of our test process– Achieved replacement of CFT stage on latest products– PCT Improves Diagnosability of BFT tests– Use of Embedded SAS tests reduced cost of tester– Decreases Time To Market for new products– Enables use of less experienced techs at CMs– Enables development of more embedded tests and instruments

Future Needs– Looking for ways to reduce Code Load time– Need to develop multiple test sites on one tester like CFT– Investigating use of many PCT pods over Ethernet to one tester