Slide 1

Yet More Symbol Systems; Relevant User Issues Seminar on Speech and Language Processing for Augmentative and Alternative Communication Spring 2010 Mahsa A.Yarmohammadi Slide 2 Papers for Today Patel, R., Schooley, K. & Radhakrishnan, R. (2006). Comparison of semantic versus syntactic message formulation: A pilot study. Assistive Technology Outcomes and Benefits, Volume 3, 96-110. Ottem, E., 2001. Use of pictographic-articulatory symbols to promote alphabetic reading in a language-impaired boy: Case study. Augmentative and Alternative Communication 17 (March), 52-60. Venkatagiri, H. S., 2002. Clinical implications of an augmentative and alternative communication taxonomy. Augmentative and Alternative Communication 18 (March), 45-57 Slide 3 Main topics of papers Patel et al. (2006) Compare message formulation using a semantic frame-based approach versus the conventional syntactic approach Ottem (2001) Use Blissymbols and a new type of pictographic- articulatory symbol to teach reading to a 9-year- old boy with specific language impairment Venkatagiri (2002) Offer a taxonomy of certain AAC system components (signs) as a framework for clinical decisions related to AAC Slide 4 Patel et al. (2006) The use of picture symbols for communication has gained increasing appeal Picture symbols paired with voice output serve as a powerful means for expressing ones ideas Most commercially available VOCAs use a linear(syntactic) approach to message construction I + want + soda At least basic knowledge of syntax is required No research support the notion that POS ordering of vocabulary icons assists in language learning Slide 5 Approach The use of semantic schema -associated with the verb- may facilitate message construction by making the rules of grammar more transparent probably reducing keystrokes easing the cognitive burden improving access to vocabulary Compare message formulation using a semantic frame-based approach (iconCHAT) versus the conventional syntactic approach (Default) in terms of: accuracy, speed, complexity, user preference Slide 6 Method Eight developing children between 7 and 10 without speech and language impairments The child was shown 18 pictures and 4 three- step sequences The child used the iconCHAT and Default prototypes to describe what was happening in the scene he/she was shown Both prototypes had the same vocabulary size, overall display size, icon size, graphic symbol set, and the same type of synthetic voice output A survey was done to assess ease of use and the satisfaction with both prototypes Slide 7 iconCHAT Slide 8 Default Slide 9 Stimuli Slide 10 Dependent measures Construction time per linguistic unit Time per button click Button clicks per linguistic unit Percentage accuracy semantic accurateness of the subject, verb, object Complexity the use of modifiers, one point for each modifier Use of quick reference panel one point for using the panel to select an agent or pronoun, for iconCHAT only Reuse of prior message constructions one point for each reuse, for iconCHAT only Slide 11 Results Some participants were faster and required fewer keystrokes using iconCHAT, others performed better using Default Slide 12 Results All participants were 100% accurate in terms of grammatical completeness using both prototypes Slide 13 Results Sentences structured in both prototypes were either in the SVO, SVMO, or SVMMO forms Slide 14 Results The heterogeneity of participant strategies for message formulation is evident in the results Slide 15 Results No statistically significant differences between the prototypes in terms of time per linguistic unit Slide 16 Results Although there was a statistically significant difference in time per button click between the two prototypes, the difference was rather small Slide 17 Results 5 sec/click in iconCHAT, 4 sec/click in Default Slide 18 Results A statistically significant difference in the number of button clicks per linguistic unit, iconCHAT requires fewer clicks Slide 19 Results Participants varied in the complexity of utterances they generated Slide 20 Results All participants used the quick reference panel to a large extent Slide 21 Results Only one child reused a prior message construction Slide 22 Survey results Various results across participants 2 participants thought that iconCHAT was easier to use, 4 thought that Default was easier to use, 2 rated both equally 5 participants said that they used the semantic schema as opposed to the serially construction 6 participants mentioned relying on the semantic schema rather than the serial ordering 4 participants mentioned that they liked how iconCHAT showed what needed to be filled in Slide 23 Survey results Some positive feedbacks regarding Default: faster to learn liked the details (modifiers) along the bottom liked having all the categories on one page Very little negative feedback for either prototype 2 comments on the need for more vocabulary One participant noted that having to select the verb first was odd in the beginning because it was not what Im used to but that it was cool though, I guess because it helped me. Slide 24 Conclusion Semantic-based approaches to message formulation are at least as effective in facilitating accurate and complete utterances for typically developing children. The 2-dimensional semantic schema and the serial ordering of icons may be beneficial to children who use AAC given that their constructions is limited to single word and has atypical syntax illustrates the relationships between icons provides feedback regarding the output of the speech recognizer Slide 25 Conclusion Semantic-based formulation may be especially useful for those children who struggle with grasping grammatical constructs semantic frame facilitate complete sentence production through visual cues Slide 26 Future work Future studies need to include children who use AAC The stimuli should be expanded to include a broader range of sentence complexity To assess the usefulness of the reuse feature, multiple stimuli with the same predicate are necessary Include a modifier panel along the bottom of the iconCHAT interface Slide 27 Ottem (2001) More than 80% of children with SLI experience problems with reading (Tallal et al.,1996) Reading problems of children with SLI are probably due to a deficit in phonological processing (Stanovich, 1998;Vellutino, 1979) To read using a grapheme-to-phoneme conversion (alphabetic) system, phoneme awareness is required Some studies support the view that articulatory awareness plays an important role in learning to read (Heilman et al.,1996;Montgomery,1981;Alexander et al.,1991) awareness of the movement of the position of ones own articulators is important in both phonological awareness and the ability to grapheme-to-phoneme conversion The treatment of dyslexia should include speech therapy to teach individuals to be aware of their articulatory gestures Slide 28 Ottem (2001) Graphic Representational Symbols (GRSs) may facilitate specific components of word awareness, but their overall impact on beginning reading may be minimal They dont contain information about how the basic units of the orthographic system are pronounced by the articulators Can a GRS that contains visual information about articularoty gestures facilitate orthographic reading in a child with SLI? Slide 29 Pictographic-articulatory system PAS was designed to portray the salient articulatory features of letter sounds PAS was used to teach letter sounds for alphabetic reading to a child with SLI Four phases of reading (Ehri, 1991): logographic phase: letters serve as visual cues transitional phase: letters provide alphabetic cues to the sounds of spoken words alphabetic phase: recode spellings phonologically into pronunciations orthographic phase: ability to read words without phonologically recoding constituent letters Slide 30 Pictographic-articulatory system Slide 31 Slide 32 Case study Erik is a child with SLI problems with the structural aspect of language very short and ungrammatical utterances at the age of 8 years poor word knowledge, unable to read any sight words limited knowledge of the letters of alphabet, unable to analyze or synthesize spoken or written words Use of Blissymbols and the PAS with Erik over a 2-year period is studied Slide 33 Assessments Erik was assessed using some standard tests Wechsler Intelligence Scale for Children-Revised Wechsler Preschool and Primary Scale of Intelligence Reynell Developmental Language Scales Illinois Test of Psycholinguistic Abilities Verbal Expression Manual Expression Sound Blending Erik was tested three times prior to the initiation of Blissymbols (8 years,2 months) prior to the introduction of the PAS (9 years,6 months) after termination of the use of the PAS (10 years, 3 months) Slide 34 First assessment Initial scores on all of the verbal tests were quite low Slide 35 Communication and Language Intervention: Blissymbols Erik was taught Blissymbols in order to improve his knowledge of the meaning of the words increase his vocabulary improve his verbal communication skills Teaching one 30-minutes session per day, for about 7 months He acquired more than 200 Blissymbols (150 nouns, 50 verbs, pronouns, adjectives, prepositions) He could read sentences composed of up to 7 Blissymbols Slide 36 Second assessment Both his verbal knowledge and his phonological awareness were facilitated Slide 37 Second assessment Eriks ability to communicate was improved, he spoke more often, spoke longer sentences and more variable words Slide 38 Reading Intervention: the PAS PAS was developed to proceed from Blissymbols to orthographic reading Blissymbols in combination with letters didnt facilitate Eriks ability to acquire beginning reading skills Teaching the PAS three hours per day in order to achieve four primary goals. Slide 39 Reading Intervention: the PAS Promoting knowledge about the articulators teaching the articulators and their functions using models, pictures, mirror work, and play What happens in your mouth when you make the sound ? Promoting knowledge of the PAS symbols and their relationship to orthographic letters link two symbols to form a word using the pictogram cards read the words using PAS symbols Linking sounds and PAS symbols to words read two- and three-symbol words, then rearrange the cards into new words Matching PAS symbols to letters Slide 40 The letters of the alphabet were paired with the PAS cards previously introduced A computer program for symbol-sound matching was used to practice the link between the sounds and the letters practice 30 min per day for 8 months Slide 41 Matching PAS symbols to letters Early in the process Erik understood the meanings of the constituent parts of the PAS symbols He understood that each pictogram corresponded to a letter sound After 7 months Erik knew 17 PAS symbols and their letters He was able to read simple stories comprised of either PAS symbols or letters He reached the alphabetic phase of reading Slide 42 Matching PAS symbols to letters After 8 months Erik refused to use the PAS any longer He could read without the aid of the symbols All of the remaining letters of the alphabet were taught orthographically only Over the next year Erik gradually reached the orthographic phase of reading He could fluently read quite complex instructions He still had difficulty interpreting the meaning of abstract words, because of his delayed conceptual development Slide 43 Third assessment WISC-R and ITPA verbal expression results are the same as the second assessment (or slightly declined in scaled scores) Slide 44 Third assessment The PAS didnt appear to contribute to Eriks conceptual or general cognitive development Slide 45 Third assessment His comprehension of spoken language and phonological awareness had improved Slide 46 Discussion Use of Blissymbols for 7 months didnt facilitate Eriks acquisition of reading, however it increased his interest in communication and improved his word knowledge promoted his understanding that words are basic units of language, so increasing his print awareness prepared him for the logographic phase of reading Use of the PAS facilitated Eriks reading acquisition by directing his attention to salient parts of the sound system Slide 47 Discussion Eriks experience was sequential and hence additive, so a different outcome is possible if PAS use had proceed that of Blyssimbols Additional AAC research that examines the impact of different types of GRSs on reading performance is encouraged Slide 48 Venkatagiri (2002) There are a huge number of AAC symbol/icon sets and symbol/icon retrieval techniques This paper offers a taxonomy of AAC system components Builds on and extends previous contributions to AAC taxonomy by Lloyd (1985), Lloyd and Fuller (1986), Lloyd and Blischak (1992), and Fuller, Lloyd, and Schlosser (1992) Slide 49 Current AAC taxonomies A widely used taxonomy: aided / unaided useful for distinguishing between the physical characteristics of AAC communication fails for differences in the complexity and scope A secondary-level distinction is static / dynamic confusion with respect to the use of these terms more movement and change for dynamic communication, or durability of message elements in time the term symbol includes all types of message elements involved in AAC communication, but fails to accommodate emotional states, comfort states, physical & psychological needs, cognitive states AAC traditionally has a speech-centered definition, however, it includes expressive communication in all of its forms Slide 50 AAC taxonomy Reasons for classifying AAC system components formulate a comprehensive and coherent theory of AAC provide guidelines for clinical decision making useful insight to the steps necessary to develop appropriate AAC systems At minimum, an AAC system consists of a set of signs and a retrieval method Slide 51 Classification of AAC signs Slide 52 Signs Communication takes place through signs Sign refers to all actions, events, and objects that have a communicative value objects, tokens, marks, images, movements, gestures, sound patterns, graphic patterns There are three main types of signs: indexes icons symbols Slide 53 Indexes When the relationship between a sign and what it represents is one of cause and effect and/or physical proximity, then the sign is an index reflexive vocalizations cry, moan, grunt, sigh facial expressions happiness, sadness, anger, surprise body postures and movements Communication through indexical signs is unintentional Communication partner is responsible for recognizing the communicative value of indexes Slide 54 Indexes It is useful to prepare a detailed list of indexical signs displayed by an AAC user Slide 55 Indexes Visceral indexes Emotional states fear, anger, happiness Comfort states fatigue, pain, feeling cold Visceral indexes are not amenable to shaping through learning Skeletal indexes Physical and Psychological needs hunger, companionship, toileting Cognitive states alertness, excitement, anticipation Skeletal indexes may be modified through behavior modification and social learning procedures Slide 56 Icons When the sign bears a physical resemblance to what it signifies, it is called an icon color, shape, size, texture, smell, sound, Examples: some road signs (a drawing of a falling rock), pictographic writing used by many African tribes, pantomime, imitations of sounds of animals The association of icons with their referents requires some learning Slide 57 Icons Transparent icons when a likeness of an object is used to represent that object a picture of fire to represent fire Translucent icons icons that represent a property of the referent a picture of fire to signify hot rather than fire itself Translucent icons involve greater learning demands than transparent ones Translucent icons must present salient physical or functional features of the referent, otherwise they are regarded as symbols Icons are well suited to represent everyday objects and events, not abstract concepts (courage), invisible objects (air), imaginary entities and events (miracle) Slide 58 Icons Transparent or translucent icons are either Gestural point to the nose to represent nose (transparent) a hand placed under the nose to signify smell (translucent) Auditory meow for the sound made by cats (transparent) meow for cat, noise of snoring for sleep (translucent) Material Pictographs a painting, photograph, etc, of object or event(transparent) or for representing a salient physical property of object (translucent) Miniatures a toy bed to represent bed (transparent) a small pillow to represent bed (translucent) Slide 59 Symbols When the relationship between a sign and what it represents is arbitrary and conventional, it is called a symbol The relationship between a symbol and what it signifies is entirely learned Examples: traffic signs such as two-way traffic, religious symbols, political symbols (flags) Symbols are divided into verbal and nonverbal categories verbal: symbols used within the framework of natural language nonverbal: arbitrary and conventional signs that are not part of a natural language Slide 60 Primary symbols Primary verbal symbols: speech (vocal) gestural language: sign languages (e.g., American Sign Language and signed English) Seamless and symbiotic relationship to cognition Cognition allows for producing meaningful spoken and gestural communication, at the same time, speech and gestural languages is used to organize ones cognitive domain Primary nonverbal symbols: speech (vocal): grunts, cries, sighs, and other vocal sounds when used volitionally to represent arbitrary meanings gestural language: idiosyncratic and commonly understood gestures Slide 61 Secondary symbols Secondary verbal symbols: symbols used for writing Graphic: traditional alphabetic or other type of graphic writing system Embedded-graphic: orthographic symbols in another modality Auditory Morse code Tactile-visual: three-dimensional cutouts of letters Visual-gestural finger spelling Secondary nonverbal symbols: artificial symbol sets Tactile: abstract 3-dimensional graphic symbols recognized through tactile channels Auditory: arbitrary tones and noises used for communication, have not been used in AAC so far Visual: abstract graphic symbols with assigned meaning many Blissymbols, color-coded geometric shapes, Lexigrams Slide 62 Flowchart of decision making The flowchart can be used to select indexes, icons, and symbols for an AAC system AAC evaluation and system development should occur simultaneously rather than consecutively It shows the order of integrating the different types of signs into AAC system the rate of integration depends on the users capabilities, motivations, and communication needs Slide 63 Indexes 1 2 Primary nonverbal symbols (speech) 3 Primary nonverbal symbols (gestural) 4 Primary nonverbal symbols (speech) 5 Gestural symbols 6 Secondary verbal symbols 7 Material icons Slide 64 Flowchart of decision making Indexes, volitional vocalizations, speech, gestures and gestural languages are preferred to material icons communication However there is a lack of research on indexical communication it deals with vital aspects of well-being care givers are primarily responsible for indexical communication it is highly productive Disadvantages of material icons problems of portability availability adaptability scope Slide 65 Conclusion Variables when selecting symbol/icon sets scope: what meaning may be expressed clarity: the degree of precision of meanings rate: the amount of information transferred/time learnability: the amount of time and effort to learn the skills cognitive and physical ease of message preparation availability: when communication can take place ease of comprehension for the partner Selection is consistent with the physical and cognitive capabilities of the AAC user Importance of these factors varies depending the needs and abilities of users and contexts Slide 66 Conclusion Indexes (e.g., yawning and squirming) and primary nonverbal symbols (e.g., grunts, idiosyncratic gestures) should be fully exploited in an AAC system high rate, high availability, cognitive simplicity, ease of physical effort Gestural and auditory icons (e.g., commonly understood gestures) should be important components of an AAC system highly learnable, early acquisition by developing children Primary verbal symbols (i.e., speech and gestural languages) can be helpful in AAC systems high rate on scope, clarity, speed, comprehensibility, availability, however they are more cognitively complex and require fine motor coordination Material icons (e.g., line drawings and miniatures) should be integrated if the above signs fail to completely meet the clients needs Secondary nonverbal symbols are problematic at best in AAC systems poor in scope, rate, clarity, comprehensibility, availability, learnability