Post on 03-Jan-2016
description
July 7-8, 2000 Kits Workshop 1WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
A Smart Port Card
Tutorial-
The Exercises
John DeHartWashington University
jdd@arl.wustl.eduhttp://www.arl.wustl.edu/~jdd
July 7-8, 2000 Kits Workshop 2WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Question?
How many
of you
have never built a
NetBSD kernel?
July 7-8, 2000 Kits Workshop 3WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Outline
• Exercises– Download a kernel
– Build a kernel for the SPC
– Modify a file system
– Build a file system from scratch
– Configure an SPC for IP connectivity
– Install the SPC hardware in a Switch• Use completely separate set of switches from other exercises
• Do this as you have free time throughout SATURDAY
July 7-8, 2000 Kits Workshop 4WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
If time permits...
• Open Exercises:– things we have not done yet
– things we can discuss and/or start working on.
• Building a kernel - you add the SPC mods
• Download program using AAL5 frames
• Investigate Linux kernel for SPC
July 7-8, 2000 Kits Workshop 5WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Setup
• We need to divide into 7 Groups:
• Each group gets its own:– login (spc1, spc2, spc3, … spc7 passwords: Spc1Tut, Spc2Tut, …)
– switch with SPC already installed on Port 4
– PC with• APIC connected to Port 3
• /usr/SPC directory tree
APIC
PC
Fiber Pair
WUGS Switch
Port 0Port 1
Port 7
Port 6Port 3
Port 5SPC/Port 4
Port 2
2 Serial Cables
July 7-8, 2000 Kits Workshop 6WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Directory Tree for each Group:
/usr/SPC/NetBSD
NetBSD source directories
/usr/SPC /bin
WUGS and SPC utilities binaries
/usr/SPC /boot_loader
Source for SPC boot loader
/usr/SPC /configuration
GBNSC config files and Jammer scripts
/usr/SPC /download
Directory from which to download kernels
/usr/SPC /root_wdir
Directory for portion of kernel build done as root
/usr/SPC /spc_utilities
Source code for SPC utilities
July 7-8, 2000 Kits Workshop 7WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Directory Tree for each Group:
• WUGS Utilities:– GBNSC
– Jammer
– GBNSC config file
– Jammer script for configuring switch
• SPC Utilities:– resetSPC
– download_APIC
– download_ENI
– mknl: make symbol-only netbsd
– mkimg: make BSS image
• download directory– pre-built bootloader
• boot.out
– bit file
– Pre-built kernel• spcKernel/netbsd.bss.ORIG
• SPC kernel source tree
• non-SPC kernel source tree
• Scripts for building kernel
• SPC Filesystem
Exercise 1: Kernel Downloading
July 7-8, 2000 Kits Workshop 9WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Exercise 1 - Download and boot a Kernel
• Reset switch• Run GBNSC and Jammer to configure switch:
– as root:> cd /usr/SPC/configuration> sh config.apic.connections
– as regular user:> cd /usr/SPC/configuration> ../bin/GBNSC -r -a config.switch.APIC &> ../bin/Jammer 0.1 wooster 3551 SPC_Control.js
July 7-8, 2000 Kits Workshop 10WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Exercise 1 - continued
• Configure the PC (/etc/remote) so you can use tip: spc1:dv=/dev/tty01:dc:br#9600:pa=none
spc0:dv=/dev/tty00:dc:br#9600:pa=none
• Start the tip processes in two separate windows as root:> tip spc0
> tip spc1
July 7-8, 2000 Kits Workshop 11WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Exercise 1 - continued
• Download bootloader and kernel
> cd /usr/SPC/download
> ../bin/download_APIC -a 0 41 0 42
– download will take 4-5 minutes…• make sure you see this fairly quickly:
boot loader started, sleep(2)
Sending entire kernel...sendfile(filename=netbsd.bss, load_addr =
0x100000, start = 0x0, sendlen = 0x40000000)
– while you’re waiting look at the scripts and source code:• /usr/SPC/configuratoin/SPC_Control.js
– Look at the WUGS connections to get to the SPC
• /usr/SPC/spc_utilities/download_APIC/main.cc
July 7-8, 2000 Kits Workshop 12WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Exercise 1 - continued• Boot to multi-user mode
– When you see this: “Enter pathname of shell or RETURN for sh:”• hit <return>
– Then at ‘#’ prompt, type ‘exit’
• Log in as root (no password) on the serial console– Use Terminal Type: vt100– Notice which port is the console port. tty00 or tty01?– Check the size of the netbsd kernel in the file system:
> ls -l /netbsd> du -s /netbsd
– Check the size of the file system:> df -k /
– While looking at your watch, try this:> sleep 20• what is the effect of these SPCs running at 25 MHz? Why?
July 7-8, 2000 Kits Workshop 13WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Exercise 1 - continued
• Send a reset control cell> /usr/SPC/bin/resetSPC -a 0 41 0 42
– See that the serial console is now dead
– Why does resetSPC hang? Look at code...
– Kill it with a <ctrl-C>
• Download bootloader and kernel again• See boot messages in tip window…
Exercise 2: Kernel and Filesystem
Building
July 7-8, 2000 Kits Workshop 15WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Exercise 2A - Building a kernel
• Do these steps as regular user (i.e. not as root)
• Make sure the kern and pci source code is configured:> cd /usr/SPC/NetBSD/usr/src/sys/kern
> make init_sysent.c
> cd ../dev/pci
> make -f Makefile.pcidevs
• Kernel config file provided:– /usr/SPC/NetBSD/usr/src/sys/arch/i386/conf/SPC_24MB
> cd /usr/SPC/NetBSD/usr/src/sys/arch/i386/conf
> config SPC_24MB
> cd ../compile/SPC_24MB
> make depend
July 7-8, 2000 Kits Workshop 16WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Exercise 2A - Building a kernel (con’t)
• Three shell scripts are provided: mk.1, mk.2, mk.3
• Its easier for this part if you have two windows open– root window in directory /usr/SPC/root_wdir– regular user window in directory:
/usr/SPC/NetBSD/usr/src/sys/arch/i386/compile/SPC_24MB
– Stage 1: mk.1:• run as regular user
• performs make to build kernel
• may take several minutes:– take a look at mk.1, mk.2, mk.3 while you are waiting
– where does the kernel file end up when it is ready for download?
> cd /usr/SPC/NetBSD/usr/src/sys/arch/i386/compile/SPC_24MB/
> ../mk.1
July 7-8, 2000 Kits Workshop 17WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Exercise 2A - Building a kernel (con’t)– Stage 2: mk.2
• run as root
• inserts symbol-only copy of /netbsd into filesystem
> cd /usr/SPC/root_wdir
> ./mk.2
– Stage 3: mk.3• run as regular user
• copies filesystem into netbsd kernel file
• takes a couple of minutes...
> cd /usr/SPC/NetBSD/usr/src/sys/arch/i386/compile/SPC_24MB/
> ../mk.3
• Download bootloader and kernel again– don’t forget to reset SPC
– and if you use resetSPC, don’t forget to kill it with <ctrl-C>
July 7-8, 2000 Kits Workshop 18WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Exercise 2B - Modifying a kernel• Modify the kernel
– Change kernel so it boots to multi-user mode• look in sys/arch/i386/i386/md_root.c (search for SINGLE)
– Change timing of clock in system FPGA • look at: sys/arch/i386/isa/timerreg.h (#define TIMER_FREQ)
– add a kernel config file option to change TIMER_FREQ
• If needed, configure switch as in Exercise 1
• Rebuild your kernel– You will need to do a ‘> make clean ‘ in the compile/SPC_24MB
directory
• If needed, configure switch as in Exercise 1 • If needed, send reset control cell to SPC• Download bootloader and kernel• Log in via serial port
– Check the file system and verify that it is what you built
July 7-8, 2000 Kits Workshop 19WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Exercise 2C - Modify a file system
• Modify the SPC Filesystem (as root, be careful...)> cd /usr/SPC/root_wdir
> vnconfig -t spc24MB -v -c /dev/vnd0d SPC24MB.fs
> mount /dev/vnd0d /mnt
> cd /mnt/etc
– Put something in it that will tell you that it is yours• e.g. a file in /mnt/etc/<your_name>
– Change console to other TTY port• modify /mnt/etc/ttys (in SPC filesystem)
– also change default terminal type for non-console tty to vt100
• also need to change the kernel config file and re-run config
> umount /mnt
> vnconfig -u /dev/vnd0d
July 7-8, 2000 Kits Workshop 20WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Exercise 2C - Modify a file system (con’t)
• Rebuild your kernel• If needed, configure switch as in Exercise 1 • If needed, send reset control cell to SPC• Download bootloader and kernel• Log in via serial port
– Check the file system and verify that it is what you built
July 7-8, 2000 Kits Workshop 21WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Exercise 2D - Build a file system
• Build your own Filesystem from scratch– Edit /etc/disktab to create a new sized disk: 18MB.
spc18MB|spc18MBkernSPC Kernel FileSystem Disk:\
:ty=simulated:se#512:nt#16:rm#300:ns#27:nc#80:\
:pa#34560:oa#0:ba#4096:fa#512:ta=4.2BSD:\
:pb#34560:ob#0:\
:pc#34560:oc#0:
> cd /usr/SPC/root_wdir
– Use dd(1) to create a file > dd if=/dev/zero of=<filename> count=<numBlocks>
– Use vnconfig(1) to configure the file as a vnode disk> vnconfig -t <disktab entry> -v -c /dev/vnd0d <filename>
– Use disklabel (1) to give it a label > disklabel -rw /dev/vnd0d <disktab entry> <label>
> disklabel -r /dev/vnd0d
– Use newfs(1) to put a filesystem on the disk > newfs -m 0 -o space -i 5120 -c 54 -T <disktab entry> /dev/rvnd0a
July 7-8, 2000 Kits Workshop 22WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Exercise 2D - Build a file system (con’t)
– Mount it. > mount /dev/vnd0a /mnt
– Populate the disk:• Use mknod(1) to make devices in /dev directory
– see next page for details!
• Mount the original SPC filesystem we used before on /mnt1> vnconfig -t spc24MB -v -c /dev/vnd1d <original filesystem filename>
> mount /dev/vnd1d /mnt1
• if /mnt1 does not exists you may have to create it with ‘mkdir’
• Look through /mnt1/* and populate your filesystem from there
• Useful command, this does a tar(1) from directory to directory:> cd <dir1>; tar cf - . | (cd <dir2>; tar xfBp -)
– Unmount> umount /mnt
> vnconfig -u /dev/vnd0d
> umount /mnt1
> vnconfig -u /dev/vnd1d
July 7-8, 2000 Kits Workshop 23WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Exercise 2D - (con’t) : mknod(1) Devices> cd /mnt> mkdir dev> cd dev> mknod bpf0 c 23 0> mknod bpf1 c 23 1> mknod bpf2 c 23 2> mknod bpf3 c 23 3> mknod console c 0 0> mknod drum c 4 0> mknod io c 2 14> mknod klog c 7 0> mknod kmem c 2 1> mknod lkm c 28 0> mknod mem c 2 0> mknod null c 2 2> mknod ptyp0 c 6 0> mknod ptyp1 c 6 1> mknod ptyp2 c 6 2> mknod ptyp3 c 6 3
> mknod rmd0a c 24 0> mknod rmd0b c 24 1> mknod rmd0c c 24 2> mknod stderr c 22 2> mknod stdin c 22 0> mknod stdout c 22 1> mknod tty c 1 0> mknod tty00 c 8 0> mknod tty01 c 8 1> mknod ttyp0 c 5 0> mknod ttyp1 c 5 1> mknod ttyp2 c 5 2> mknod ttyp3 c 5 3> mknod zero c 2 12> mknod md0a b 17 0> mknod md0b b 17 1> mknod md0c b 17 2
> chmod 600 console> chmod 640 drum> chmod 640 io> chmod 600 klog> chmod 640 kmem> chmod 640 mem> chmod 666 null> chmod 640 md0a> chmod 640 md0b> chmod 640 md0c> chmod 640 rmd0a> chmod 640 rmd0b> chmod 640 rmd0c> chmod 666 stderr> chmod 666 stdin> chmod 666 stdout> chmod 666 tty> chmod 644 tty00> chmod 644 tty01> chmod 666 zero
July 7-8, 2000 Kits Workshop 24WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Exercise 2D - (con’t) : mknod(1) Devices> cd /mnt> mkdir dev> cd dev> mknod bpf0 c 23 0> mknod bpf1 c 23 1> mknod bpf2 c 23 2> mknod bpf3 c 23 3> mknod console c 0 0> mknod drum c 4 0> mknod io c 2 14> mknod klog c 7 0> mknod kmem c 2 1> mknod lkm c 28 0> mknod mem c 2 0> mknod null c 2 2> mknod ptyp0 c 6 0> mknod ptyp1 c 6 1> mknod ptyp2 c 6 2> mknod ptyp3 c 6 3
> mknod rmd0a c 24 0> mknod rmd0b c 24 1> mknod rmd0c c 24 2> mknod stderr c 22 2> mknod stdin c 22 0> mknod stdout c 22 1> mknod tty c 1 0> mknod tty00 c 8 0> mknod tty01 c 8 1> mknod ttyp0 c 5 0> mknod ttyp1 c 5 1> mknod ttyp2 c 5 2> mknod ttyp3 c 5 3> mknod zero c 2 12> mknod md0a b 17 0> mknod md0b b 17 1> mknod md0c b 17 2
> chmod 600 console> chmod 640 drum> chmod 640 io> chmod 600 klog> chmod 640 kmem> chmod 640 mem> chmod 666 null> chmod 640 md0a> chmod 640 md0b> chmod 640 md0c> chmod 640 rmd0a> chmod 640 rmd0b> chmod 640 rmd0c> chmod 666 stderr> chmod 666 stdin> chmod 666 stdout> chmod 666 tty> chmod 644 tty00> chmod 644 tty01> chmod 666 zero
OK, OK. There is a script for this./usr/SPC/bin/mknods
July 7-8, 2000 Kits Workshop 25WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Exercise 2D - Build a file system (con’t)
• Rebuild your kernel– Change the kernel config file
– Look through the make scripts to change filesystem
– Re-config and re-make the kernel
• If needed, configure switch as in Exercise 1 • If needed, send reset control cell to SPC• Download bootloader and kernel• Log in via serial port
– Check the file system and verify that it is what you built
– Keep it running while you go on to next exercise ...
Exercise 3:IP Configuration
July 7-8, 2000 Kits Workshop 27WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Exercise 3A - IP Between SPC and PC
• Save any changes you make for the SPC in your filesystem
• Configure an IP address for the SPC’s APIC> ifconfig
• Configure an IP address for the PC’s APIC
• Configure an APIC connection on each> atm_ifconfig
• Add routes on PC and SPC:> route add ...
• Configure the necessary WUGS connection(s)
July 7-8, 2000 Kits Workshop 28WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Exercise 3A - IP Address Assignments
Team PC APIC IPAddress
SPC APIC IPAddress
1 192.168.10.10 192.168.10.12 192.168.20.10 192.168.20.13 192.168.30.10 192.168.30.14 192.168.40.10 192.168.40.15 192.168.50.10 192.168.50.16 192.168.60.10 192.168.60.17 192.168.70.10 192.168.70.1
> cat filename | telnet postscript.cs 9100
July 7-8, 2000 Kits Workshop 29WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Exercise 3A - IP Between SPC and PC
• Try to ping– Iterate until you can ping from PC to SPC– Tools for debugging the IP connection:
• netstat• APIC readCounts utility
– /usr/local/bin/readCounts on both PC and SPC
• WUGS cell counters (Jammer)
• Turn on telnetd– edit /etc/inetd.conf on SPC– kill -HUP <inetd pid>
– iterate until you can telnet from PC to SPC– Be patient
• on some machines telnet seems to take a while to connect.• Not sure why yet...
July 7-8, 2000 Kits Workshop 30WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Exercise 3A - IP Between SPC and PC
• Make sure you have saved all your changes• Reset switch• Rebuild kernel with your newly modified
filesystem and re-download. Does it still work...
July 7-8, 2000 Kits Workshop 31WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Exercise 3B - IP Between Neighbor SPCs
• Use SPCs as routers:
PC
192.168.10.10
192.168.10.1
192.168.10Domain
PC
192.168.20.10
192.168.20.1
192.168.20Domain
. . .
SPC SPC
July 7-8, 2000 Kits Workshop 32WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Exercise 3B - IP Between Neighbor SPCs
• Save any changes you make for the SPC in your filesystem
• Configure an IP address alias for the SPC’s APIC to put it in the neighboring SPC’s domain> ifconfig …alias
• Remove the network route that the above caused:> route delete -net 192.168.20
• Configure an APIC connection on each SPC> atm_ifconfig
• Add routes on PC and SPC:> route add ...
• Configure the necessary WUGS connection(s)
July 7-8, 2000 Kits Workshop 33WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Exercise 3B - IP Between Neighbor SPCs
• Try to ping
– Iterate until you can ping from PC to neighbor PC
– Tools for debugging the IP connection:• netstat
• APIC readCounts utility (/usr/local/bin/readCounts on SPC)
• WUGS cell counters
• Make sure you have saved all your changes• Reset switch• Rebuild kernel with your newly modified
filesystem and re-download. Does it still work...
Exercise 4:Hardware Installation
July 7-8, 2000 Kits Workshop 35WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Exercise 4 - Installing an SPC in a Switch
• Power Off• Open Switch• Removing Glink Line Card from Port 4 or 5• Install SPC• Install Serial Cables, feed in through front panel• Re-Install Glink Line Card on top of SPC• Close Switch• Connect serial cables to PC• Power On• Download and Boot Kernel
Tom will providea Slide or two
Open Exercises
July 7-8, 2000 Kits Workshop 37WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Open Exercises: SPC Mods for NetBSD 1.4.1
• Do this first if you are interested in SPC/Linux• Compare kernel config files:
– /usr/SPC/NetBSD/usr/src/sys/arch/i386/conf/SPC_24MB
– /usr/SPC/NetBSD/usr/src/sys/arch/i386/conf/APIC_1_4_1
– Note the major differences
– Find the network devices in both. Anything interesting?
• Compare source files:– /usr/SPC/NetBSD/usr.ORIG
– /usr/SPC/NetBSD/usr
– Directories where there are differences:sys/arch/i386/i386 sys/...
sys/arch/i386/isa
sys/dev/ic
sys/dev/pci
July 7-8, 2000 Kits Workshop 38WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Open Exercises: AAL5 Download Program
• currently we download via APIC control cells – 16 bytes of data per cell
– send one cell, wait for response cell before sending next one
– slow: 4-5 minutes for a 24MB kernel
• Preferred method would be AAL5– Use APIC Control cells to configure AAL5 descriptors
– Send Kernel in AAL5 frames to SPC
– Read it back to verify?
July 7-8, 2000 Kits Workshop 39WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Open Exercises: Linux Kernel for the SPC
• ONLY if you are already familiar with NetBSD/SPC mods
• ONLY if you are already VERY familiar with Linux
• Kernel Changes:– Memory Disk
– Serial port console
– APIC Configuration by APIC driver
• System FPGA impact:– Timer configuration
– RTC requirements
– Serial port configuration
– Static IRQ assignment
• Download and Boot– Boot loader
– Other things that Linux expects from the BIOS or bootloader?