CONTROL SYSTEM FRONT-END HARDWARE UPGRADE

SLAC

CONTROL SYSTEMFRONT-END

HARDWARE UPGRADEEric J. SiskindMarch 7, 2002

Stanford Linear Accelerator Center

E. Siskind, March 7, 2002 2

SLAC

Project Goals

• Replace Multibus-I micro hardware with commercial-off-the-shelf personal computers.

• Replace special purpose accelerator networks (SLCnet, KISnet, PNET) with a single commercial-off-the-shelf TCP/IP network such as switched gigabit Ethernet.

• Develop personal computer CAMAC interface to replace MBCD/MBCD-II.


SLAC

Project Impetus

• Multibus-I difficult to upgrade, maintain, and support.

• Not able to take advantage of Moore’s Law increases in CPU speed, memory size, and network bandwidth for ~10 years.

• Two orders of magnitude increase in that time:– CPU clock frequency from ~20 MHz to ~2 GHz;

– Memory size from ~10 megabytes to ~1 gigabyte;

– Network bandwidth from 1-10 to 100-1000 megabits/second.


SLAC Front-End Real-Time TCP/IP Networking

• Network must deliver 120/360Hz real-time micro-micro traffic in preference to slower Alpha-micro (or generic front-end client to micro) packets. There is currently no expertise in meeting this challenge on the SLAC site.

• Single network is preferable for reliability and cost of purchase/maintenance, but use of a separate real-time network at an early stage is possible.


SLAC

New CAMAC Features

• CAMAC interface differs from MBCD/MBCD-II in four major ways:– Two board design to support remote I/O;– Real-time preemptive hardware with recovery;– Maximizes CAMAC utilization when user

software environment in PC is multi-threaded;– Large (i. e. comparable to existing micro) user

programming environment within interface.


SLAC

Hardware Architecture

• Hardware consists of two circuit boards connected by up to 10 km of single mode fiber optics:– PCIL (“PCI Link”) board plugs into PCI option

slot in PC;– FECC (“Front-End CAMAC Controller”) is

double-width CAMAC module.

• PC moves from micro’s current location to MCC computer room, leaving behind only FECC.


SLAC

PCIL Features

• Basically an intelligent special-purpose Network Interface Card.

• Processor with DMA PCI block transfer master and DMA fiber optic link transmitter/receiver.

• Makes no assumptions about nature of PC’s real-time operating system.

• Semi-static assembly language PROM code with downloaded updates.

• First generation hardware doubles as Alpha SLCnet interface.


SLAC

PCIL Code Functions

• Supports PC FECC message passing.• Supports FECC remote read/write access to PC

memory with optional PC interrupt.• Performs function-specific scatter/gather DMA

to/from PC memory for all classes of operations, avoiding unnecessary buffer copying by PC CPU.

• Forwards trigger pattern from PCI registers to FECC (only in 1st generation; pattern forwarding in hardware in 2nd generation).


SLAC

FECC Features

• Logical successor to MBCD/MBCD-II.• Accesses host memory via point-to-point fiber

optic link instead of parallel I/O bus.• Gets power from CAMAC backplane and timing

triggers from PDU via auxiliary backplane.• 1st generation supports MBCD-II’s four strings of

SLAC serial CAMAC crate controllers (5 MHz, bit serial, half duplex).

• 2nd generation adds support for four strings of IEEE standard crate controllers (5 MHz, byte serial, full duplex) and Bitbus.


SLAC

Rapid Micro Replacement

• All PCs in MCC computer room have only three connections to outside world:– Front-end TCP/IP network interface;

– MPG IDOM connection to hard reset button (replaces PNET reset);

– Fiber optic connection to FECC.

• Only FECC connection is specific to a particular function; i. e. only attachment to PR02’s FECC converts an arbitrary PC on front-end network into PR02.


SLAC

Micro Replacement II

• MCC computer room patch panel has fiber optic links to all PCs’ PCILs and all regions’ FECCs.

• Switching to a spare PC requires changing TCP/IP address, updating IDOM bit number, and moving fiber optic patch cord.

• First two accomplished via logical physical micro association table.


SLAC

Real-Time Constraints• All accelerator operations based on 360 Hz timing;

2.78 milliseconds/pulse.• Many CAMAC operations must be performed on

each pulse, or on a particular pulse (e.g. F(19) PPYY broadcasts, BPM data acquisition).

• Real-time access to CAMAC for such time-critical usages is currently maintained by limiting all MBCD requests for groups (“packages”) of CAMAC cycles from non-time-critical micro jobs to one millisecond. Only one such micro job may have a single package outstanding in the MBCD at any given time.


SLAC PCIL-FECC Real-Time CAMAC Access Scheme

• All time-critical operations either employ hardware that is distinct from that used for non-time-critical operations, or else can interrupt non-time-critical operations on shared hardware.

• All software executing in PCIL or FECC runs under a real-time executive with interrupt-driven prioritized preemptive scheduling.


SLAC

Real-Time Latency

• Shared PCI hardware released in three PCI clock ticks (90/45 ns on 33/66 MHz bus).

• Shared 2nd generation fiber optic link accessed on next 4.9 microsecond cell (plus second cell if pattern forwarding required).

• Shared CAMAC hardware released in one CAMAC cycle (~10/~4 microseconds on SLAC/IEEE standard cable) unless locked.

• Real-time executive switches processes in 3-5 microseconds (plus similar interrupt latency if context switch already in progress).


SLAC CAMAC Interrupt Recovery & Control

• If a CAMAC block transfer is interrupted, hardware can repeat the previous write operation (address pointer load?) with updated write data before resuming the transfer.

• Hardware can prevent an interrupt between CAMAC cycles transferring two 16-bit halves of a single 32-bit word.

• Hardware can prevent an interrupt of a particular block transfer entirely (needed because of a current PIOP feature).


SLAC

MBCD-II Parallelism

• Multibus-I backplane bandwidth is ~1 megabyte/second (2 microseconds/16-bit word).

• SLAC serial CAMAC bandwidth is ~¼ of that (~8 microseconds/16-bit word in block transfer mode).

• Solution to limited bandwidth problem was to operate 4 CAMAC cables in parallel.


SLAC MBCD-II Performance Limits

• MBCD-II firmware only looks at one package at a time, so a package must use multiple crates on different cables in order to achieve parallelism.

• Non-time-critical tasks often require multiple packages to read one module, so no parallelism is achieved (e.g. analog status can’t read an entire SAM from collider arcs in a single package).

• Time-critical operations with small numbers of words/crate don’t realize full bandwidth because of limited Multibus-I package fetch bandwidth.


SLAC Package Fetch Improvements

• FECC CAMAC hardware has much higher ratio of local memory/CAMAC bandwidth to avoid package fetch bottlenecks.

• Very large PCI DMA bandwidth between PC memory and PCIL local memory, and link DMA bandwidth between PCIL local memory and FECC local memory:– Straw between FECC memory and CAMAC;

– Fire hose between PCIL memory and FECC memory;

– River between PC memory and PCIL memory.


SLAC

System Performance

• Many PC jobs now can have simultaneously outstanding long CAMAC packages.

• All packages are sent to FECC firmware ASAP, and parsed into their individual (block transfer) operations in a single crate (“packets”).

• Packets are immediately queued to hardware for the cable accessing the target crate.

• Cable operation begins as soon as any outstanding package has a packet requiring access to a crate on that cable.

• Parallelism now achieved over multiple packages.


SLAC

MBCD Intelligence

• Set of four 2911 4-bit slices to make 16-bit microsequencer (AMD 2900 series of bit-slice microprocessors).

• Code from a 32 x 16 bit PROM via a single 82S105 FPLS.

• 5 MHz FPLS clock; 2911 clock generated by FPLS at lower frequency as required.


SLAC

MBCD-II Intelligence

• Analog Devices ADSP-2101 12.5 MHz 16-bit DSP.

• 2k x 24 bit on-chip program memory; 1k x 16 bit on-chip data memory; 8k x 8 bit external program/data memory (unused).

• Actual application program uses no more than a few hundred instructions and very little data memory.


SLAC

PCIL/FECC Intelligence

• Analog Devices ADSP-21060 “SHARC” 40 MHz 32-bit DSP on each board.

• 40k x 48 bit on-chip program memory; 64k x 32 bit on-chip data memory.

• 512k/2048k x 48 bit external program/data memory (256k/512k x 48 in 2nd generation PCIL).

• External DMA I/O data memory:– 512k/2048k x 32 bit in 1st generation PCIL;– 256k/512k x 64 bit in 2nd generation PCIL;– 1024k/2048 x 32 bit in 2nd generation FECC.


SLAC

User FECC Programming• Real-time executive plus MBCD/Bitbus emulation

uses <10% of on-chip program memory and most of on-chip data memory.

• Remaining minimum of 7+ megabytes of memory on 40 MHz 32-bit processor similar to existing 386/486 CPU boards.

• Move existing 360 Hz interrupt driven code from micro to CAMAC interface to minimize CAMAC access delay.

• C run-time environment with dynamic heap allocation; full environment and stack preserved over context switch.


SLAC

1st Generation Hardware• PCIL uses 32 bit/33 MHz PCI slot.• Separate fiber optic links for real-time/non-real-

time PCIL FECC communication.• Each link runs at 12.5 megabytes/second half

duplex; back compatible with 10 megabyte/second BIPI link (SLCnet backbone).

• PCIL peeks and pokes at FECC memory.• PCI master and link are FPGA-based DMA

peripherals of PCIL SHARC.• CAMAC is programmed I/O peripheral of FECC

SHARC, but link uses SHARC DMA channels.


SLAC

1st Generation Status

• Hardware based on SHARC plus FPGA-based peripherals (PCIL-3; FECC-2).

• PCIL debugged in 1999; FECC in 2000.• Five PCIL/FECC board sets, plus 3 PCILs for

Alpha SLCnet interface.• Now intended primarily for development:

– Link protocol requires 3-4 round trips per message; bandwidth & latency degrade with distance.

– Link TAXI chips no longer available.


SLAC

2nd Generation Hardware

• PCIL2 uses 64 bit/66 MHz PCI slot.• Single specialized fiber optic link for PCIL2

FECC2 communication (NLC SBIR).• 125 megabyte/second full duplex link, but

9/19 used for hardware overhead.• All peripherals FPGA-based DMA except

for Bitbus (programmed I/O).• Hardware based on SHARC plus one large

FPGA per board.


SLAC

2nd Generation Link

• All traffic divided into 608 byte cells.• 320 bytes of user data (real-time, non-real-time,

pattern, no-op/zero); 32 bytes of hardware header; 256 bytes of error correction code.

• Cell requires 4.864 microseconds; can withstand 128 consecutive bytes or > 1 microsecond of lost data in that time.

• Klystron modulator pulse is > 5 microseconds wide with < 1 microsecond rise and fall times; can survive loss of all data on both rising and trailing edges of a single modulator pulse.


SLAC

2nd Generation Link II

• If uncorrectable errors, hardware timeout and retransmission of all lost cells; 100 microsecond (20 cell lengths) typical timeout length.

• Hardware flow control for 16 real-time receive buffers and 16 non-real-time receive buffers.

• No start-of-message sequences to be corrupted by noise after initial turn-on.

• Hardware header supports remote manipulation of initialization register; local and remote hardware error counters.


SLAC

Link Header Format

• All fields 16 bits unless noted.• Command :

– 4 bits - destination buffer number;– 1 bit - first cell in buffer;– 1 bit - last cell in buffer;– 2 bits - cell type (Pattern/R-T/N-R-T/NOP);– 1 bit - remote initialization register write enable;– 7 bits - unused.

• Offset in buffer to end of cell, i. e. cumulative buffer length including this cell (64 byte chunks).


SLAC

Link Header Format II

• Remote initialization register write data (32 bits).• 16 non-real-time receive buffer available bits

(backpressure flow control).• 16 real-time receive buffer available bits

(backpressure flow control).• 16 non-real-time receive buffer filled handshake

bits.• 16 real-time receive buffer filled handshake bits.


SLAC

Link Header Format III

• Outgoing cell sequence number.• Next expected incoming cell sequence number.• Multiple cells acknowledged event count.• Received cell out-of-sequence error count.• Timeout error count.• 8B/10B decoder error count.• Reed-Solomon corrected error count.• Uncorrectable Reed-Solomon error count.• Outgoing and incoming versions of last 8 counters

readable from SHARC.


SLAC

2nd Generation Status

• PCIL2 link now debugged; PCI master not yet tested.

• Two PCIL2 boards built in 2001; three more in 2002.

• FECC2 design in progress; complete by mid-summer; debugged by end of CY02.

• Initial build of two FECC2 boards in 2002; also requires I/O concentrator panel (too many connectors for CAMAC front panel).


SLAC

3rd Generation Design• Replace SHARCs by PowerPC embedded in each

board’s single FPGA.• Time scale: FPGAs available in CY02; begin in

CY03(???).• Benefits:

– Commercial real-time executive with improved software development tools;

– Clearly the upcoming embedded hardware/software technology.

• Shortcomings:– Need more time;– Need more money.


SLAC SLAC CAMAC Bandwidth

• ~7.4 microseconds per 16-bit word in block transfer mode.

• Add round trip propagation delay to target crate controller for each word (up to 2.7 microseconds in SLC arcs).

• Add 1.6 microseconds for 24-bit data.• Subtract 3.2 microseconds for control operation

(F8=1).• Add 3.6/4.8 microseconds for new CNAF in

read/write operations.


SLAC IEEE Standard CAMAC Bandwidth

• 3.6 microseconds per 16/24 bit word for read/write with new CNAF on each cycle.

• Add round-trip propagation to furthest crate controller.

• Subtract 0.8 microseconds for control operation.• Includes parity on each byte and message check

byte.• 2-3 times faster than SLAC serial CAMAC

controllers.


SLAC Pipelined IEEE Standard CAMAC Mode

• 1.0 microsecond per additional 16/24 bit read/write cycle (full CAMAC crate bandwidth).

• No change to CNAF allowed.• No round-trip propagation delay degradation.• Order of magnitude improvement over SLAC

serial CAMAC controllers.• Only supported by Kinetic Systems model 3952

“enhanced” L-2 crate controller.


SLAC

1st Generation Bandwidth

• ~1 megabyte/second bit serial CAMAC bandwidth (four SLAC CAMAC cables combined) (“straw”).

• 12.5 megabytes/second for each link; only one of read/write active at any time on each link (“fire hose”).

• 133 megabytes/second PCI master in block transfer read/write (“river”).


SLAC

2nd Generation Bandwidth

• 8 megabytes/second enhanced byte serial CAMAC bandwidth (four IEEE CAMAC cables combined).

• ~1 megabyte/second bit serial CAMAC bandwidth (four SLAC CAMAC cables combined).

• <0.05 megabyte/second Bitbus bandwidth.• 125 megabytes/second link burst rate in each

direction; maximum average rate is 10/19 of this or 66 megabytes/second.

• 533 megabytes/second PCI master in block transfer read/write.


SLAC

Bitbus Master

• Existing master based on MCS-51 2 MHz 8 bit microcontroller with SDLC/HDLC serial link.

• 8044BEM is an MCS-51 with mask-programmed firmware (iDCX-51 RTE plus Bitbus application).

• FECC2 has programmed I/O hardware (one SHARC interrupt/32 bits) to send a packet to one slave and receive that slave’s response.

• Poll list, counters, etc. maintained by new SHARC code in FECC2.

• Command queuing no longer an issue.


SLAC

Software Group Projects

• Build, boot, and debug iRMX-III in off-the-shelf personal computers.

• Make real-time micro micro data communications streams coexist with slower back-end front-end traffic on switched TCP/IP network.

• Move 360 Hz interrupt processing from micro to FECC.


SLAC

Software Projects II

• Change control of 360 Hz interrupt processing from shared memory to network link model.

• Modify fast feedback actuator to specify real-time priority for CAMAC operations.

• Modify slower micro jobs to send larger packages to CAMAC interface to maximize parallelism.

• Modify klystron job to protect PIOP CAMAC operations from interrupts.


SLAC

Software Projects III

• Support logical to physical micro mapping.

• Support PC access to file server delivering current versions of PCIL and FECC executable images appropriate to different types of micros.


SLAC

Schedule

• SWE development effort is currently in progress (TEG/RCS).

• All optimizations not necessary at early stage.

• Trying to deploy 1-2 PC-micros in production system during accelerator shutdown beginning this July.

• Begin AIP with 2nd generation PCIL-FECC in FY03/04.


SLAC Simplified Hardware Configuration

• No PCPC fast feedback (KISnet equivalent) network traffic.

• Use 1st generation PCIL-FECC hardware:– No IEEE CAMAC or Bitbus;– Only need 32 bit/33 MHz PCI slot.

• MPG PC (PNET equivalent) pattern broadcast network traffic on separate real-time TCP/IP network.

• PC-based MPG (MP10/MP11) gets pattern from MP00/MP01 via reflective memory link.

CONTROL SYSTEM FRONT-END HARDWARE UPGRADE

Documents

Transcript of CONTROL SYSTEM FRONT-END HARDWARE UPGRADE