1 Setting up a Testbed with the Netbed/Emulab Software.
-
Upload
kendall-sherick -
Category
Documents
-
view
225 -
download
1
Transcript of 1 Setting up a Testbed with the Netbed/Emulab Software.
![Page 1: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/1.jpg)
1
Setting up a Testbed with theNetbed/Emulab Software
![Page 2: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/2.jpg)
2
Detailed documentation of all this is in the source tree:
doc/setup*.txt
![Page 3: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/3.jpg)
3
Steps
• Get network infrastructure set up
• Install/configure FreeBSD on boss and ops– Linux should work, but never been done. Glitches will
occur, most likely in standard services, not Emulab itself. Rutgers Winlab about to tackle it.
• Run installation scripts on boss and ops
• Put information about the hardware into the database
• Bring up nodes
![Page 4: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/4.jpg)
4
![Page 5: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/5.jpg)
5
Control Network VLANs
• External – connection to campus network
• Node control – nodes’ control interfaces
• Private – boss and other ‘secure’ nodes
• Public – ops and other nodes with user accounts
• Control hardware – SNMP devices that must be protected from users
![Page 6: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/6.jpg)
6
Switch (Cisco) Setup
• Enable SNMP read-write access• Control network
– ‘set port host’– Router: Set ‘UDP helper’ address
• Experimental Network– Disable noisy automatic protocols– CDP, spantree, bpdu-guard
• Connect control and experimental networks– Port in the ‘Control Hardware’ VLAN on control– Some VLAN with switch IP interface on exper.
![Page 7: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/7.jpg)
7
Installing FreeBSD
• FreeBSD 4.9 recommended right now (11/03)
• Use standard FreeBSD installer
• Pick 'Developer' distribution set
• Install the ports collection
• Pick where you will put important filesystems
• Configure hostname, IP address, etc.
• Install your favorite editors, etc.
![Page 8: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/8.jpg)
8
Important Filesystems on ops
• /usr/testbed/
• /users/
• /proj/
• /groups/
• /share/
![Page 9: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/9.jpg)
9
Important Filesystems on boss
• /usr/
• /usr/testbed/
• /var/
![Page 10: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/10.jpg)
10
Configuring Testbed Software
• Done with GNU autoconf
• Most configuration options stored in 'defs' file– '—with-TBDEFS='
• 'defs-example' provides a template
• Use an object tree outside your source tree
• Uses GNU make (gmake in FreeBSD)
• Install targets– boss-install, ops-install– post-install, to be run as root, on boss
![Page 11: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/11.jpg)
11
Defs File Example# The name of this installation
THISHOMEBASE=Example.Emulab.Net
# Domain this testbed resides in
OURDOMAIN=example.emulab.net
# Host name of our web server (or host:port)
WWWHOST=www.example.emulab.net
# Fully-quailified hostname of the boss node
BOSSNODE=boss.example.emulab.net
# Fully-quailified hostname of the ops (also called users) node
USERNODE=ops.example.emulab.net
# Fully-quailified hostname of the fileserver (will probably # be the same as the ops node)
FSNODE=fs.example.emulab.net
![Page 12: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/12.jpg)
12
Mailing Lists
• testbed-ops– User support– The most serious error messages
• testbed-approval– Requests to start new projects
• testbed-logs– Experiment creation/teardown
• testbed-audit– Audit user and project creation and changes
![Page 13: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/13.jpg)
13
Mailing Lists (contd.)
• testbed-www– Web reports
• testbed-stated– Reports from state daemon
• testbed-users-archive– Archive for automatically-generated testbed-
users list – should be direct to a file
• testbed-active-users-archive– Archive for testbed-active-users lists
![Page 14: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/14.jpg)
14
Setting Up ops
• Run install/ops-install from your object tree
• Install the ops part of the testbed software– 'gmake ops-install'
• Special kernel if using Cyclades Z-series on FreeBSD
• Mailing lists on ops• Put boss in ops' /etc/hosts file• Reboot ops
![Page 15: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/15.jpg)
15
Setting Up boss
• Use same defs file you did with ops– Need a duplicate obj tree for boss
• Start install/boss-install• Install ports• Finish install/boss-install• Add {ops,users,fs} to /etc/hosts• Install the boss part of the software
– Two targets, boss-install and post-install
• ssh from boss to ops as root
![Page 16: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/16.jpg)
16
Other Things To Do on boss
• setuid bit on /usr/bin/suidperl
• Get boss set up as the nameserver for your domain
• SSL certificates for the web
• 1000Hz kernel
• Bootloaders and MFSes from Utah
• Disk images for nodes come from Utah, for now
![Page 17: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/17.jpg)
17
Bootstrapping Users and Projects
• Create first user with 'firstuser' script
• Create holding experiments via the web– In the 'emulab-ops' project– 'hwdown' – for nodes that have failed– 'reloadpending' – for nodes awaiting disk loads– 'reloading' – for nodes having their disks loaded
![Page 18: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/18.jpg)
18
What Goes in the Database
• Switches (nodes)– Note: Switch node_id must be resolvable
– Types (node_types)– Stacks (switch_stacks, switch_stack_types)– Interconnects (interfaces, interface_types, wires)
• Power controllers (nodes, node_types)
• Images and OSids (images, os_info)
![Page 19: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/19.jpg)
19
What Goes in The Database - Nodes
• Node itself (nodes)
• Virtual nodes (nodes)
• Node type (node_types)
• Interfaces (interfaces, interface_types)
• Connection to switches (wires)
• Serial line (tiplines)
• Power control outlet (outlets)
![Page 20: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/20.jpg)
20
Bringing up Nodes - Overview
• Basic idea: Let nodes report in automatically• Nodes boot with a temporary IP address• Unknown nodes boot a 'newnode' FreeBSD
MFS• Nodes report their hardware to boss• boss guesses node names and addresses• Script finds interface ports on switches• Administrator adds nodes to the testbed via a
Web interface (at any time)
![Page 21: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/21.jpg)
21
Bringing up Nodes – Details• Enter node and interface types
• BIOS setup– PXE in boot order– Power-loss behavior– Serial console redirection; password
• Dynamic IP range in dhcpd.conf– Start with dhcpd.conf.template– Use a range of IP addresses not to be used by
nodes
![Page 22: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/22.jpg)
22
Bringing up Nodes (contd.)
• Start with one node, to tweak name and IP– Others will be inferred from the first
• Re-numbering interfaces– Convert from FreeBSD to Linux order
• Searching switches for nodes’ interfaces– Searches all switches in the database– Switch ports must be enabled and in a VLAN
• Create nodes• Free nodes from ‘hwdown’ experiment• Scripts don't do serial and power yet
![Page 23: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/23.jpg)
23
Creating a Disk Imagefor Emulab Test PCs
![Page 24: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/24.jpg)
24
Step by step
• Install a stock {Linux,BSD} of your choice– Make one image with both OS’s (if both popular)
• Get the serial console working
• Setup root access– root password (for console login)– boss root pubkey to ~root/.ssh/authorized_keys
• Ensure perl, rpm and sudo installed
• Create or copy over node host keys
• “make client-install” from build tree
![Page 25: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/25.jpg)
25
Saving the Image
• If Emulab up and running:– Use image creation web form
• Otherwise:– Shutdown to single-user (FS’s still mounted)– Run /usr/local/etc/emulab/prepare– Umount FS’s and RO mount /– Reboot from CD or MFS– Use imagezip to create the image
![Page 26: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/26.jpg)
26
Details: Programs added
• Daemons: idle detection, HW health monitor, watchdog
• Agents: traffic generation, link control, program execution
• Synchronization server
• TMCC
• Emulab startup/shutdown hooks
![Page 27: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/27.jpg)
27
Details: Kernel
• Stock kernels mostly work• For FreeBSD
– Adding IPoD a simple patch
– Shaping nodes require additional features– BSD virtual nodes require massive changes
• For Linux– Adding IPoD a simple patch
![Page 28: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/28.jpg)
28
All Linux, All the Time
• Some environments just do Linux
• Doing FreeBSD also asking too much?
• Converting servers: boss, ops– Get Emulab software to build– Get RPM equivalents to ports– Easy as that!
![Page 29: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/29.jpg)
29
The BSD-free environment
• MFS: for creating and installing images– Should be PXE boot environments out there
• Delay nodes– Need Linux to act as a bridge– Is “tc” up to the task?
• Multiplexed node hosts– UML (probably not, too slow)– vservers (used in planetlab, which we support)– Xen (looks good)
![Page 30: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/30.jpg)
30
Emulab Idle Detection and Policy
![Page 31: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/31.jpg)
31
Idle Detection
• Why do it?– Ease contention for resources
– Encourage better resource use practices
• How activity is defined– CPU load (tunable threshold – currently 0.8)
– TTY activity (mod time on /dev entries)
– Network traffic• Control network vs. Experimental network: different levels
– External activity• E.g.: user-instantiated reboot.
![Page 32: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/32.jpg)
32
Goals and Challenges
• Don’t affect experiment’s performance!
• No false positives!– Could ruin an experimenter’s day— or week—
since swapout saves no node state (currently)
• Few false negatives
• Non-goal: detect intentional circumvention
![Page 33: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/33.jpg)
33
Idle Detection - slothd
• Small daemon runs on each test node• Tracks last observed metrics & compares vs.
current values: crossed activity thresholds?– Runs every 5 minutes when node is active– Runs every 5 seconds when inactive (aggressive mode).
• Instant detection of idle->active transition
• Reports via UDP to central collector– Every five minutes, or send immediately if transitions
from idle->active– Collector stores last activity timestamps in the DB
• One for each type of activity.
![Page 34: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/34.jpg)
34
Idle Detection – policy enforcement
• Script runs every 5 minutes on server.– May send email to user warning of inactivity.
– May swap out experiment (idle, or max duration).
• Policy is dictated by tunable site-vars.– Frequency of email warnings.
– When to swap experiment out.
• Honors per-experiment override settings.– E.g.: expt not swappable; max duration; idle threshold.
– User has control over these both at experiment creation, and after swapin by editing expt. metadata.
![Page 35: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/35.jpg)
35
Idle Detection cont.
• Last seen activity viewable on web page.– Per experiment.– Drill down at node level (CPU, TTY, Network).
• Issues– Easy to circumvent intentionally– Sometimes defeated unintentionally
• E.g.: User leaves traffic generator running
– Hard (impossible?) to determine what activity is meaningful (be context sensitive)
![Page 36: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/36.jpg)
36
Emulab’s Administrative Structure
![Page 37: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/37.jpg)
37
Executive Summary
• Emulab has a two level administrative structure: “Projects”, and “Groups”– Plus “Elab Admins” on top, and “Users” on the bottom
• Permits a variety of common organizational structures– A single user project with no groups
– A class project with multiple, isolated groups managed by TAs
• Administrative control is delegated to “leaders” at each level
![Page 38: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/38.jpg)
38
A “Project”
• Central administrative entity• Started by a faculty member or senior student
– Submitted through web interface– User account gets created for experiment leader
• Approval of project users delegated to leader– Saves on administrative overhead– Project leader responsible for users' behaviour– Leader may grant leader rights to one or more users
• Essential for lazy faculty
• Project gets its own disk space/tree• Users may join multiple projects
![Page 39: 1 Setting up a Testbed with the Netbed/Emulab Software.](https://reader036.fdocuments.net/reader036/viewer/2022081602/551c0c08550346ad4f8b5178/html5/thumbnails/39.jpg)
39
A “Group”
• Projects may have multiple groups
• Project groups created by the project leader– Designates group leaders for each– Delegates group membership approval to them
• Groups are independent– Files/experiments are protected from each other
• Groups can share– Share the common project file hierarchy