Blink - Making The World More Accessible, One Blink At A TimeTelly StroumbisThe Boeing Company

BIO – Telly StroumbisName: Telly StroumbisKey Technical Field: Software Engineering/ArchitectureE-mail: telly.stroumbis@boeing.com

Telly Stroumbis is a Computer Systems Architect and Technical Fellow for Boeing. He designs and leads Java and Java Enterprise Edition development efforts and provides technical support to numerous Boeing efforts. Telly was an early adopter and has been developing with JavaTM for the last 12 years. For the past 1.5 years, Telly has been working with Volunteers for Medical Engineering in the development of the Blink System. The Blink software is designed to aid those with MS, ALS and other diseases causing weak or paralyzed muscles. The software is designed to help severely paralyzed patients communicate and control their environments.

2

Agenda> Introducing Blink

Background Overview Capabilities

> The Tech Of Blink Plug-in Architecture Text To Speech Swing Hacks Device Control

> Questions3

Introducing Blink> Background> Overview> Capabilities

4

VME Concept> VME - Volunteers for Medical Engineering> Non-Profit Organization, Incorporated in 1982> Based In Baltimore, Maryland> Goal is to increase the independence of

individuals with disabilities and the elderly through: Custom design of new products Research and development Modifications to existing devices

5

VME Past Projects

6

*Images courtesy of Volunteers for Medical Engineering

VME Brown Bag"Almost 20 years ago, Volunteers for Medical Engineering (VME) developed a system to help a disabled man who could only blink to communicate with his family. They designed and built a mechanism and software to create what is now called the Blink System."

"The Blink System operates using a cursor that moves at a selected speed through rows, scanning to the bottom row, then back to the top, and then repeating the process. When the user blinks one eye the cursor stops scanning downward and starts stepping one box at a time across the row. The second blink stops the cursor and the contents of the box are spoken."

"Over time, however, the program controlling the Blink Switch became obsolete. "

7

Modernize Blink> Design for portability> Design for supportability> Design for configurability> Support a variety of input devices> Support speech output> Provide X10, INSTEON, & IR control capabilities> Provide Email & SMS capabilities

8

Portability> Develop using Java> Java code compiles to byte code> Byte code runs in a Java Virtual Machine (JVM)> JVM’s exist for all major platforms> Supports write once run anywhere> Tested On Windows, Linux, Mac OS

9

Supportability> Java is a modern language> Java is strongly typed> Java runs on everything from smart cards & cell

phones to big iron such as eBay and Amazon> JavaOnesm - Java Developers Conference, largest

technical conference (15K – 23K developers)> Java is Object Oriented> Develop using a plug-in architecture (features can

be added without modifying the code base)

10

Configurability> Plug-in based architecture> Capabilities defined in an XML file> New capabilities easily added> User Interface is fully customizable> Customizations saved in an XML file

11

Blink Design

12

Voice• Text To Speech• Volume• Voice

X10• Device On• Device Off• Dimmer

Email• To• Subject• Message

Command• Type• Display Text• Command• Data• Status

Input Devices> Support a wide variety of input devices> System operation using only a mouse> Scanner mode using only a left mouse click> Additional switches and sensors emulate a mouse

device. Examples include:Blink Switch, Sip Puff Switch, Twitch Switch, Bite Switch, Tongue Switch, Head Mouse, etc.

13

*Images courtesy of Origin Instruments Corporation

Text and Speech Output> Supports the capability to display words, phrases,

and sentences on the user display> Supports predictive words and phrases> Supports Text To Speech (TTS)> FreeTTS – An Open Source Java speech

synthesizer

14

Email> Supports communications through email> Supports SMS messaging using an SMS gateway

15

X10 & INSTEON Control> Supports control of X10 & INSTEON devices> Plug-in talks to an X10/INSTEON Gateway> EZSrve – Control X10 & INSTEON devices over a

network - Price $209.99http://www.smarthome.com/31279.html

16

IR Control> Supports control of IR devices> Plug-in talks to an IR Gateway> Global Caché – Control IR devices over a network

- Price $159.99http://www.smarthome.com/8114.html

17

Welcome To Blink

18

P rog ra m m a ble K e ys – Y o u c a n

c ha ng e the func t io n, la be l, a nd

a ppe a ra nc e o f the s e ke y s

P ull-do w n Me nus

A re a w he re y o ur te xt a ppe a rs

R e a l-t im e

C lo c k

S ug g e s tio ns – P ro vide s

s ug g e s te d w o rds fo r yo ur

c o m po s e d te xt

T a bs –

Y o u c a n a dd, de le te , a nd

re o rde r ta b s

Quic k A c tio n B a r -

F o r Ofte n Us e d

F unc tio ns

The Tech Of Blink> Switch Input Hack> Plug-in Architecture> Predictive Text> Text To Speech> Swing Hacks> Device Control

Note: In Code Samples, Error Handling Removed For Brevity

19

Switch Input On The Cheap

20

PLUG-IN ARCHITECTUREThe Tech Of Blink

21

Plug-in Architecture> All commands handled by plug-ins> Plug-ins bundled Into a jar file> “devices.xml” file configures 1 or more plug-ins> File located at root of jar file> Device entries include:

Commands – Actions to perform Properties – Configuration options Java Class to load

22

Sample Device Entry<device name="JavaSpeech"

class="vme.blink.device.SpeechDevice”> <commands> <command name="SAY" value="SAY"/> </commands> <properties> <property name="VOLUME" value="1.0"/> <property name="PITCH" value="100.0"/> <property name="RATE" value="150.0"/> <property name="VOICE" value="kevin16"/> </properties></device>

23

Finding Plug-ins> public URL getResource(String name)

Finds "the" resource with the given name Searches The Java CLASSPATH Returns The First Match

> public Enumeration<URL> getResources(String name) Finds "all the" resources with the given name

24

Create Hashmap Of Plug-insClassLoader loader = this.getClass().getClassLoader();Enumeration<URL> urls = loader.getResources("devices.xml"); while(urls.hasMoreElements()) {

URL url = urls.nextElement(); InputStream input = url.openStream();Document doc = BlinkImport.getDocument(input); ArrayList<DeviceEntry> deviceList = BlinkImport.getDevices(doc); for(int i=0; i < deviceList.size(); i++) {

DeviceEntry entry = deviceList.get(i);devices.put(entry.getDeviceName(), entry);

}}

25

Device Has A Lifecyclepublic interface Device { public void init(String deviceName, Properties commands, Properties properties); public void start(); public void stop(); public void destroy(); public CommandStatus process(Command command);

}

26

Processing A Command> Find the device> Invoke process() on the command> Plug-in Class gets loaded on demand

Device device = getDevice(command.getDeviceName()); CommandStatus status = device.process(command);

27

Dynamic Class InstantiationDevice device = deviceCache.get(deviceName); if(device == null) {

DeviceEntry entry = config.getDeviceEntry(deviceName); device = (Device)Class.forName(entry.getDeviceClass()).newInstance();

device.init(deviceName, entry.getCommands(), preferences.getDevicePreferences(deviceName));

device.start();deviceCache.put(deviceName, device);

} return device;

28

PREDICTIVE TEXTThe Tech Of Blink

29

Java DB> Embeddable database> 100% Java> Mature product> Long history

1996 – Cloudscape Inc 1999 – Informix Software, Inc 2001 – IBM 2004 – Apache Derby Java DB Is SUN's supported distribution of Derby

30

Simple Database Startupprivate static final String DB_URL = "jdbc:derby:BlinkDB";private Connection con; public SuggestionDAO (File blinkHome) {

// we want to specify the database locationSystem.setProperty("derby.system.home", blinkHome.getPath());

File dbDir = new File(blinkHome, "BlinkDB");

Class.forName("org.apache.derby.jdbc.EmbeddedDriver");

// Get a connection to the database this.con = DriverManager.getConnection(DB_URL);}

31

Simple Design> Words Table – Contains Words Used & Freq> Phrase Table – Contains Phrases Used & Freq

> Pre-Load With Small List Of Common Words Large Lists Tend To Be Bad Results In Suggestions Of Uncommon Words

> Suggestions Ordered By Frequency> Frequency Updated When Command Issued

32

Word Suggestions> Simple JDBC Query

String sql = "select name from WORDS where name like ? order by freq desc, name asc";

PreparedStatement stmt = con.prepareStatement(sql);

stmt.setString(1, word + "%");

ResultSet rset = stmt.executeQuery();

33

Updating Word Frequency> When Command Issued, Update Word/Phrase

Frequencies

> If Word Exists, Update, Otherwise Insert

String selectSql = "select freq from words where name = ?";

String updateSql = "update WORDS set freq=? where name = ?";String insertSql = "insert into WORDS VALUES (?, 1)";

34

Shutting Down> Always Shutdown A Database When Exiting

Speeds Up Database Startup Database Doesn't Have To Recover

public void shutdown() { try{ con.close(); DriverManager.getConnection(DB_URL + ";shutdown=true"); }catch(Exception e) { // successful shutdown always throws an exception }}

35

TEXT TO SPEECHThe Tech Of Blink

36

Java Speech API> There Is An Optional Java Speech API> FreeTTS Is An Open Source Implementation> http://freetts.sourceforge.net

> Speech Device Lifecycle We Create & Initialize on init() Resume It on start() Pause It On stop() Deallocate It On destroy()

37

Get The Speech SynthesizerFreeTTSEngineCentral central = new FreeTTSEngineCentral();

SynthesizerModeDesc desc = new SynthesizerModeDesc( null, // engine name "general", // mode name Locale.US, // locale null, // running null); // voice

EngineList list = central.createEngineList(desc);

if (list.size() > 0) { EngineCreate creator = (EngineCreate) list.get(0); synthesizer = (Synthesizer) creator.createEngine();

}

38

Prepare It To Speak

synthesizer.allocate();synthesizer.resume();

39

Find Voice Specified In Properties> Find The First Voice That Matches

desc = (SynthesizerModeDesc) synthesizer.getEngineModeDesc();

Voice[] voices = desc.getVoices();Voice voice = null; for (int i = 0; i < voices.length; i++) {

if (voices[i].getName().equals(voiceName)) {voice = voices[i];break;

}

40

Initialize It With Our Properties> The Importance Of Configurability

An Elderly Care Giver Had Trouble Understanding The Speech The Ability To Slow The Speaking Rate Resolved This

synthesizer.getSynthesizerProperties().setVoice(voice);synthesizer.getSynthesizerProperties().setPitch(pitch);synthesizer.getSynthesizerProperties().setSpeakingRate(rate);synthesizer.getSynthesizerProperties().setVolume(vol);

41

Now We Are Ready To Speak> There Is A Java Speech Markup Language> We Simply Use Plain Text> The Second Argument Is For An Optional Listener

synthesizer.speakPlainText(command.getData(), null);

42

SWING HACKSThe Tech Of Blink

43

Command Tabs

> Why Create A JTable For Each Tab?

44

> Looks Like A JTable In A JTabbedPane> Looks Can Be Deceiving

tabbedPane = new JTabbedPane(); table = new JTable();tabbedPane.addTab("title", table);

Command Tabs Hack> Create Custom Component> JTable Is Under The TabbedPane> TabbedPane's Component

Simple BlinkTableComponent Has Dimension(0, 0); Contains BlinkTableModel

> Drag & Drop Feature Based On DnDTabbedPane by TERAI Atsuhiro

45

BlinkTabbedPanepublic BlinkTabbedPane() { pane = new DnDTabbedPane(); pane.setTabLayoutPolicy(JTabbedPane.SCROLL_TAB_LAYOUT); pane.addChangeListener(this); table = new JTable(); table.setRowHeight(50); table.setShowGrid(false); table.setBorder(BorderFactory.createLineBorder(new Color(0, 0, 0))); setLayout(new BorderLayout()); add(pane, BorderLayout.NORTH); add(table, BorderLayout.CENTER);}

46

When State Changes, Update JTablepublic void stateChanged(ChangeEvent e) { // Get current tab int sel = pane.getSelectedIndex(); if(sel < 0) return; BlinkTableComponent c = getTableComponentAt(sel); if(c == null) return;

BlinkTableModel model = c.getModel();

table.setModel(model); // in case we have a different number of rows, adjust cell size resized();}

47

When Resized, Update Row Height table.addComponentListener(new ComponentAdapter(){

public void componentResized(ComponentEvent e) {resized();

} }); public void resized() { int tableHeight = table.getHeight(); int rows = table.getRowCount(); if(rows <= 0) return; int rowHeight = tableHeight / rows; if(rowHeight > 1) { table.setRowHeight(rowHeight); } }

48

JDK 1.6 Supports For Tab Componentspublic class BlinkTabComponent extends JComponent {

public BlinkTabComponent(String title, BlinkTabbedPane pane) {setLayout(new FlowLayout(FlowLayout.CENTER, 2, 1));

add(tabTitle);add(tabField);add(acceptButton);add(editButton);add(exportButton);add(deleteButton);

}

BlinkTabComponent tab = new BlinkTabComponent(title, this);pane.setTabComponentAt(index, tab);

49

The Look Of Blink> Ideas Taken From:

"TS-3548 - Extreme GUI Makeover 2007" Chris Campbell, Shannon Hickey, Romain Guy

> Create A Custom Cell Renderer> Set A Custom Border> Override paintComponent()> Gradient Goes From Top Left To Bottom Right> Cycles Once, Darker, Lighter, Darker

50

Rounded Borderprivate class RoundedBorder extends AbstractBorder {

private final Color FILL = new Color(135, 135, 135);private final Stroke STROKE = new BasicStroke(2f);

public void paintBorder(Component c, Graphics g,

int x, int y, int width, int height) { Graphics2D g2 = (Graphics2D)g.create(); g2.setRenderingHint(

RenderingHints.KEY_ANTIALIASING,RenderingHints.VALUE_ANTIALIAS_ON);

g2.setColor(FILL); g2.setStroke(STROKE); g2.drawRoundRect(4, 4, width-8, height-8, 10, 10);

g2.dispose();}

}

51

Paint Cell With Gradientpublic void paintComponent(Graphics graphic) {

Graphics2D g2 = (Graphics2D)graphic;g2.setRenderingHint(…);int width = getWidth();int height = getHeight();

Color startColor = getBackground();Color[] colors = ColorRange.getLighter(startColor, 0.15f);startColor = colors[0]; Color endColor = colors[1];

Paint bg = new GradientPaint(0f, 0f, startColor, getWidth()/2, getHeight()/2, endColor, true);

g2.setColor(Color.WHITE);g2.fillRoundRect(4, 4, width-8, height-8, 10, 10);g2.setPaint(bg);

g2.fillRoundRect(7, 7, width-11, height-11, 10, 10);}

52

Rounded Border

Fill White

Fill Gradient

DEVICE CONTROLThe Tech Of Blink

53

Device Control Challenges> Avoid OS specific device drivers> Don't want to tether user> Solution: Network Gateways

Gateway for INSTEON/X10 Gateway for IR Future: Maybe OpenRemote.Org

One device for both Complexity follows SEP principal

54

EZSrve Gateway> Standalone INSTEON & X10 Controller> Web Based GUI On Port 80> XML Interface On port 8002

We Can Control INSTEON & X10 Devices By Sending XML Messages On port 8002

<Command> Command Name<Parameter 1> Value </Parameter 1><Parameter 2> Value </Parameter 2><Parameter ….> Value </Parameter ….>

</Command>

55

INSTEON Commands> Each Module Has A Unique Address> Flags Allow Us To Specify How Many Hops> Some Commands Use 2 Parameters

ON Command has brightness level

<command>SndIns<parameter1>to id high byte</parameter1><parameter2> to id middle byte </parameter2><parameter3> to id low byte </parameter3><parameter4>flags</parameter4><parameter5>Command1</parameter5><parameter6>Command2</parameter6>

</command>56

Formatting An INSTEON Commandprivate static String INS_CMD = "<command>SndIns" + "<parameter1>0x{0}</parameter1>" + "<parameter2>0x{1}</parameter2>" + "<parameter3>0x{2}</parameter3>" + "<parameter4>0x{3}</parameter4>" + "<parameter5>0x{4}</parameter5>" + "<parameter6>0x{5}</parameter6></command>";

String[] insAddr = addr.split("\\."); String flag = "0F";String cmd2 = "00";if(command == INSCommand.ON) cmd2 = "FF";

String cmd = MessageFormat.format(INS_CMD, insAddr[0], insAddr[1],insAddr[2], flag, command.getValue(), cmd2);

57

X10 Commands> Module has a 2 part Selectable Address

House code: A-P Unit code: 1-16 4 most sig bits house code, 4 least sig bits unit code

> Action often requires two X10 Commands Select the unit (has flag: 0x00) Issue the command (has flag: 0x80)

> Command Always Issued To House Code<command>SndX10

<Parameter1> rawX10</Parameter1><Parameter2> X10Flag </Parameter2>

</command>58

Translation Tables Determine What's SentProperties x10HouseCodes = new Properties() { { setProperty("A", "6"); setProperty("B", "E"); setProperty("C", "2"); setProperty("D", "A"); setProperty("E", "1"); setProperty("F", "9"); setProperty("G", "5"); setProperty("H", "D"); setProperty("I", "7"); setProperty("J", "F"); setProperty("K", "3"); setProperty("L", "B"); setProperty("M", "0"); setProperty("N", "8"); setProperty("O", "4"); setProperty("P", "C"); } };

Properties x10UnitCodes = new Properties() { { setProperty("1", "6"); setProperty("2", "E"); setProperty("3", "2"); setProperty("4", "A"); setProperty("5", "1"); setProperty("6", "9"); setProperty("7", "5"); setProperty("8", "D"); setProperty("9", "7"); setProperty("10", "F"); setProperty("11", "3"); setProperty("12", "B"); setProperty("13", "0"); setProperty("14", "8"); setProperty("15", "4"); setProperty("16", "C"); } };

59

Formatting An X10 Commandprivate static String X10_SELECT = "<command>SndX10" + "<Parameter1>0x{0}{1}</Parameter1>" + "<Parameter2>0x00</Parameter2></command>"; private static String X10_CMD = "<command>SndX10" + "<Parameter1>0x{0}{1}</Parameter1>" + "<Parameter2>0x80</Parameter2></command>";

String house = x10HouseCodes.getProperty(houseCode);String unit = x10UnitCodes.getProperty(unitCode);

String sel = MessageFormat.format(X10_SELECT, house, unit);String cmd = MessageFormat.format(X10_CMD, house, command.getValue());

60

Sending Messages> Open A Socket> Write The Command> Read The Results> We Need To Add A Delay Between Commands

I'm using 800ms

socket = new Socket(address, port);out = socket.getOutputStream();in = socket.getInputStream();

out.write(xml.getBytes());out.flush();

61

IR Primer> IR light source

Alternates on/off Rate: Carrier frequency

> Encodings determine Mixture of on/off states Cycles on/off Duration on/off

> Multiple encodings Pulse Width (80% est.) Bi-Phase (RC-5)

62

1 2 3 4 5 6 7 8

40,000 Hz4 Cycles On, 4 Cycles Off1/40,000 Hz = 0.025 ms0.1 ms On, 0.1 ms Off

Pulse Width Encoding> Consists of on/off Pairs> Data encoded by varying on/off durations> Data often consists of

Lead-in data IR command data Lead-out data

63

Encoding Formats> Pronto Hex Format

Popular format for the Phillips Pronto Remotes http://www.remotecentral.com/

> LIRC Format Linux Infrared Remote Control http://www.lirc.org/

64

Pronto Hex Format> Series of hex values> First 4 words represent preamble> Followed by on/off durations in cycles> Sample preamble = 0000 0067 000 0015

0000 – Learned Code 0067 – 40,000 Hz freq (relative to pronto freq) 0000 – Length one-time burst (none exists) 0015 – Length of repeat burst (21 Pairs/42 Words)

Note: Frequency = 1000000/(N * .241246)Reference: http://www.hifi-remote.com/infrared/

65

LIRC Format> Set of instructions for generating data

Default freq 38,000 Hz, Timings in microsecondsbegin remote

 name  DVD  bits            7  flags SPACE_ENC|CONST_LENGTH  header     2400   600  one          1200   600  zero          600   600  ptrail        600  post_data_bits  12  post_data      0x5C9  gap          44749  min_repeat      3 begin codes

power_on                 0x3A

66

Converting LIRC To ProntoFreq = 38,000Cycle = 1,000,000 * (1/38,000) = 26 usec

600 usec off = 600 / 26 = 23 cycles or 0x17

header:     2400   600 = 005c 0017zero: 600 600 = 0017 0017one: 1200 600 = 002e 0017

power_on: 0x3A = 7 bits: 0111010 0 1 1 1 0 …0017 0017 002e 0017 002e 0017 002e 0017 0017 0017 ...

Add post data & gap67

These Sequences Look DifferentSample Used Freq = 40,000 Hz – LIRC Used Freq 38,000 Hz (10% tolerance OK)

Pronto Hex Sample:0000 0067 0000 0015 0060 0018 0018 0018 0030 0018 0030 0018 0030 0018 0018 0018 0030 0018 0018 0018 0018 0018 0030 0018 0018 0018 0030 0018 0030 0018 0030 0018 0018 0018 0018 0018 0030 0018 0018 0018 0018 0018 0030 0018 0018 03f6

LIRC Conversion To Pronto Hex:0000 006d 0000 0015 005c 0017 0017 0017 002e 0017 002e 0017 002e 0017 0017 0017 002e 0017 0017 0017 0017 0017 002e 0017 0017 0017 002e 0017 002e 0017 002e 0017 0017 0017 0017 0017 002e 0017 0017 0017 0017 0017 002e 0017 0017 06b9

68

Adjusting LIRC For Freq 40,000 HzLIRC specifies longer trailing gap – do nothing before next command

Pronto Hex Sample:0000 0067 0000 0015 0060 0018 0018 0018 0030 0018 0030 0018 0030 0018 0018 0018 0030 0018 0018 0018 0018 0018 0030 0018 0018 0018 0030 0018 0030 0018 0030 0018 0018 0018 0018 0018 0030 0018 0018 0018 0018 0018 0030 0018 0018 03f6

LIRC Conversion To Pronto Hex:0000 0067 0000 0015 0060 0018 0018 0018 0030 0018 0030 0018 0030 0018 0018 0018 0030 0018 0018 0018 0018 0018 0030 0018 0018 0018 0030 0018 0030 0018 0030 0018 0018 0018 0018 0018 0030 0018 0018 0018 0018 0018 0030 0018 0018 06fe

69

Global Caché IR Gateway> Data sent on port 4998> Command sent in ASCII text> Command terminated by a carriage return> On/Off values are in cycles of carrier frequency

How many cycles on How many cycles off Like Pronto Hex, but values in decimal and comma

separated

sendir,<connectoraddress>,<ID>,<frequency>,<count>,<offset>,<on1>, <off1>,<on2>,<off2>,….,onN,offN

String cmd = "sendir,2:1,1," + code.getGCCode() + "\r";

70

Status> Software is free> Distributed under GPLv3 license> Currently in alpha testing

71

72

Questions?

73

Telly Stroumbistelly.stroumbis@boeing.comtelly@blinksystem.comhttp://blinksystem.com