Executive Summary · Web viewJoshua Crow - Sam Turley - Zachary Cordes - Ziwei Feng Contents...

Decision Making in Weather EmergenciesFinal Report

HCI 445 – Autumn 2018Joshua Crow - Sam Turley - Zachary Cordes - Ziwei Feng

ContentsExecutive Summary.....................................................................................................................................4

Introduction.................................................................................................................................................4

Methods......................................................................................................................................................4

Observations............................................................................................................................................4

Participants..........................................................................................................................................4

Data Collection Methods.....................................................................................................................4

Analysis Methods................................................................................................................................4

Interviews................................................................................................................................................4

Participants..........................................................................................................................................4



Surveys....................................................................................................................................................4

Participants..........................................................................................................................................4



Findings.......................................................................................................................................................4

Observations............................................................................................................................................4

Interviews................................................................................................................................................4

Pre-Storm: Source Perception.............................................................................................................4

Post-Storm: Personal Impacts..............................................................................................................4

Surveys....................................................................................................................................................4

Evacuation Likelihood..........................................................................................................................4

Source Trust and Influence..................................................................................................................4

Personas..................................................................................................................................................4

Features Matrix.......................................................................................................................................4

User Scenarios.........................................................................................................................................4

Discussion....................................................................................................................................................5

Limitations and Future Work.......................................................................................................................5

Bibliography................................................................................................................................................5

Appendix.....................................................................................................................................................5

Executive SummaryTropical cyclones, known in the United States as hurricanes, pose a risk to property and people’s

lives. Despite the risks and the advance warnings of when landfall is imminent, many people do not follow mandatory evacuation orders. In this project, we aimed to understand how technology can be used to improve informed decision-making by individuals preparing for the imminent threat of a hurricane. We first broke down the decision-making process into four steps to frame our research: (1) Awareness, (2) Seeking Information, (3), Comparing Information, and (4) Decision Making.

We first conducted scenario-based observations, focusing on steps 2 and 3 with six participants who live in areas at risk for tropical cyclone landfall. Two information categories were used by our participants: social sources, such as posts from Facebook friends, and authoritative, such as stories published by local news stations and from local government sources. We found that a majority of participants looked first for information from social sources, and most participants told us that they were more likely to make decisions based on social sources when compared to authoritative sources. Our findings indicated that technological-based interventions may increase informed decision-making by increasing the weight of the authority sources in the decision-making process or by increasing the likelihood that social sources will share accurate information.

We then conducted interviews to more deeply examine where residents of hurricane-prone areas place their trust and whether recent hurricane experiences affected their perceptions of storms. We found that participants said they were most likely to trust their own experiences and the experiences of their neighbors in deciding whether to evacuate before a hurricane was going to hit, and they indicated they weren’t likely to trust local news unless the information was coming from government officials or meteorologists. Most of the participants stated that they had experienced property damage as a result of the most recent hurricane to hit them, but only one participant said they felt they would definitely evacuate for a future storm of similar severity. Our findings at this stage indicated people were most likely to trust anecdotal experiences.

Lastly, we conducted a survey to collect qualitative data on pre-storm decision-making processes. We asked survey respondents in or from hurricane-prone location about their experiences deciding whether to evacuate for recent hurricanes, what sources they trusted in making that decision, and whether they believed their prior experiences would shape their decision making in the future.

Our findings provided statistical evidence that people were most likely to trust their own prior experiences over other information sources and use that experience in their decision-making processes. It also provided statistical evidence of higher levels of trust in meteorologists and government officials when compared to other information sources. However, we saw significant levels of refusal to follow evacuation orders, indicating that technological-based interventions could find a way to accurately and authoritatively convey the risks associated with staying behind.

IntroductionMany major cities face an annual risk of tropical cyclones, also known as hurricanes, cyclones, or

typhoons depending on the area the storm forms in (NOAA, 2013). The National Hurricane Center reports that “close to seven hurricanes every four years strike the United States,” for an average of 1.75 hurricanes making landfall each year (Landsea, 2018). But unlike other natural disasters such as earthquakes and tornadoes, the recurring threat of hurricanes comes with the advantage of advance warning. Meteorologists can provide hourly updates to a storm’s projected path as soon as it forms. More accurate predictions of the storm’s path and potential landfall can be made up to five days in

advance. This allows government officials to make decisions on whether to prepare shelters and issue evacuation orders (NOAA, 2018).

However, when evacuation orders have been issued, many people have not heeded them. Less than half of New Jersey residents in affected areas followed evacuation orders when Hurricane Sandy hit in 2012 (Monmouth University, 2013). Only about 35 percent of residents under evacuation orders in South Carolina chose to leave before Hurricane Matthew hit in 2016 (Merchant & Cope, 2016). The refusal comes despite warnings from experts and officials that the choosing to stay means risking death (Knapp, 2018) and despite repeated warnings made through sources such as television, government and news websites, and social media.

Prior to beginning our research, we analyzed some competitors in the field. We found that in addition to broadcasts made by local and national news networks, there is a wide range of information sources providing weather emergency advice and information online. For example, Ready.gov, by US Department of Homeland Security, has a section devoted to hurricane evacuation preparedness, and it is one of the top websites to appear in search engine results for hurricane preparedness. This site provides a large variety of information but does not consider the reasons for evacuation; it uses language such as, “Evacuate if told to do so” but ignores the fact that many people choose to not heed such orders. This site appears to assume that decision making surrounding a weather emergency has been made and provides guidance for preparing accordingly.

The National Weather Service (https://www.weather.gov/) also provides an enormous amount of information. Current weather advisories are shown at the top of the page where the pertinent information in obvious display . While there is plenty of information about imminent storms, advice on action is absent. Visitors to the site may feel expected to interpret weather information and make appropriate actions. This can be problematic considering that people often chose not to take precautions because they do not fully understand the risk at hand (Joslyn & Jared, 2012).

In this project, we first examined the types of information people access when a hurricane is imminent and the ways they access it through observations. Our observations showed a difference in the types of sources people get their hurricane information from and how trustworthy they consider those sources to be, so we then used interviews to more deeply examine what factors people consider when deciding whether to evacuate and what weight they give those factors.

Finally, we used common points identified in our interviews to create a survey to collect quantitative data on the potential relationship between source trust, hurricane experiences, and evacuation decision making.

Our goal was to determine how different information types shape evacuation decisions, and how technology-based interventions could be used to improve informed decision-making.

In the following sections, we discuss our methods, our findings, and the implication of those findings on the execution of our technology-based solution to improve decision-making.

MethodsIn the following sections, we describe our observation, interview, and survey methods.

ObservationsThe following three sections describe our participants, our data collection methods, and our data analysis methods for our observations.

ParticipantsTable 1. Participant information

Number Pseudonym Sex

Age Location Occupation

1 Jane F 40 Kill Devil Hills, North Carolina, United States

Communications manager

2 Susan F 59 Jacksonville, Florida, United States Social worker

3 Sean M 31 Jacksonville, Florida, United States Package handler

4 Sally F 72 Jacksonville, Florida, United States Payroll clerk

5 Zoe F 23 Shenzhen, Guangdong Province, China/Chicago, Illinois, United States*

Student

6 Dale M 45 Columbia, South Carolina, United States

HR manager

*This participant studies at DePaul University in Chicago but is from Shenzhen and resides there when not in school.

Data Collection MethodsWe conducted in-person observations; sessions lasted between 15 and 30 minutes. Four

observations took place in the participants’ homes, one via videoconferencing (Skype), and one in the DePaul University library. Five participants used their own computers, tablets, and/or smartphones. One participant began by accessing weather displays on their smart tv (Apple TV).

After obtaining informed consent, we asked questions about the participant’s personal experience with hurricanes. We then asked participants to demonstrate how they would gather information if a major hurricane was approaching their city.

Following the observation, we asked participants how they had prepared for hurricanes in the past, whether they had been told to evacuate before, and why they made their decision on whether to follow the evacuation orders. (see Appendix B: Observation Protocol). Each observer took notes, either by hand or on a computer, during their sessions.

Analysis MethodsWe used the AEIOU framework to organize our notes into Activities, Environments, Interactions,

Objects, and Users. We added our organized notes to Stormboard, an online affinity diagramming tool to virtually collaborate and identify common themes (see Appendix 7.3: Affinity Diagram).

Interviews The following three sections describe our participants, our data collection methods, and our

data analysis methods for our interviews.

ParticipantsWe recruited four of the five participants through friends and family members; the fifth

participant agreed to participate at a festival local to one interviewer. Four out of the five of the

participants did not take part in the observations that we previously conducted; Zoe took part in an observation and an interview (see Table 2: Interview Participant Information).

Table 2. Interview Participant information

Number Pseudonym Sex

Age Location Hurricane Discussed

Occupation

1 Jesse F 37 Nags Head, North Carolina, United States

Hurricane Florence(Sept. 2018)

Retail

2 Sandra F 51 Jacksonville, Florida, United States

Hurricane Irma(Sept. 2017)

Hospital manager

3 Sage F 22 Jacksonville, Florida, United States


Barista

4 Courtney F 28 Columbia, South Carolina, United States


Teacher

5 Zoe F 23 Shenzhen, Guangdong Province, China/Chicago, Illinois, United States*

Typhoon Shanzhu(Sept. 2018)

Student

*This participant studies at DePaul University in Chicago but is from Shenzhen and resides there when not in school.

Data Collection MethodsSandra and Sage were interviewed in their homes. Jesse was interviewed at a festival local to

one interviewer. Courtney was interviewed by phone. Zoe was interviewed on the campus of DePaul University.

Each of the four members of our team conducted interviews separately and in-person. All participants read and signed the Informed Consent Form. We asked participants warm-up questions about where they live, the most recent severe storm to hit their community, and whether they stayed in their home for the duration of the storm.

We then asked participants questions about the storm’s affects on them, the steps they took to prepare for the storm, the types of information they utilized in their decision-making process, and how they felt about the reliability of different information sources (see the Interview Protocol in the appendix).

Interviews took between 20 and 40 minutes. We audio recorded and transcribed each interview.

Analysis MethodsWe added each transcript to a shared document then performed an open inductive coding.

From the results, we developed common salient themes. We also identified the following potential spectrums describing attitudes, opinions and reported past behaviors:

Emotional discomfort Trust vs. mistrust of information:

o Regarding Social Mediao Regarding Neighbors

Past experiences to future actionso Preparation/Regrets

Financial costs of actions/inaction Likelihood to evacuate

We then placed each interviewee on the spectrums to develop our initial segmentation for personas.

SurveysThe following three sections describe our participants, our data collection methods, and our

data analysis methods for our interviews.

Participants

Eighty-two participants resided in North Carolina at the time of the survey. Out of the remaining participants, eight resided in Florida, five resided in Illinois, and one each lived in California, Colorado, Georgia, Maryland, and Virginia.

Eighty-one of the participants identified as female. Eighteen participants identified as male, and one chose not to say.

Data Collection MethodsWe hosted the survey on Google Forms. The survey was open for responses from November 7, 2018

to November 11, 2018. The survey consisted of three sections:

Hurricane Experiences - Participants were asked about how many hurricanes they recall being affected by as an adult, how many of those hurricanes they remained in their home for, and how they believed their prior experiences would affect their future hurricane plans.

Source Trust - Participants were asked how they to rate their trustworthiness of various types of authoritative and social sources and rate the level of influence each source had on their pre-storm decision making.

Demographics - Participants were asked their age, gender, and location.

Analysis MethodsWe formed two hypotheses before conducting our survey:

There is a relationship between surviving previous hurricanes and deciding whether to follow evacuation orders for an imminent storm.

There is a relationship between a potential hurricane victim's level of trust in an information source and its influence on their pre-storm decision-making.

We performed a Mann-Whitney U statistical test for the first hypothesis and a double Kruskal Wallis statistical test for the second hypothesis. We performed the test in the programming language R using R Studio.

FindingsObservations

To provide a framework for this project, we broke down the decision-making process into four steps:

Awareness, when the individual first finds out that a hurricane is threatening their location Seeking Information, when the individual begins to search for details about the storm Comparing Information, when the individual decides -- consciously and subconsciously -- which

information they will use to make their decisions when the information sources are in conflict Decision Making, when the individual commits to a choice on their storm preparations

Our observation focused primarily on Step 2. Step 1 was implied by the scenario provided to the participants; while steps 3 and 4 were not directly observed, we did collect data about the latter steps through participant comments and our questions.

The flowchart in figure 2 illustrates Step 2; we found information seeking thematically broke down into two types of information resources: social sources and authoritative sources.

Figure 2. Flowchart of information seeking

We defined a social source as a source where information is learned from peers as opposed to experts, such as meteorologists, or officials, such as city government leaders. We defined meteorologists, city government leaders, and similar individuals in positions of authority or knowledge as authoritative sources. We found that social media played a unique role by providing nearly-equal levels of accessibility to both social and authoritative sources through means such as the Facebook NewsFeed. Social media sources involved information in the form of posts directly by Facebook friends, directly by authoritative sources “liked” by the individual, and by friends sharing information from authoritative sources.

In four out of six observations (Jane, Susan, Sean, and Zoe), the participant navigated to Facebook first when beginning to search for information. In three of these instances, the participant specifically said they were looking to see what friends and family were posting in regard to the storm; we categorized this action as turning to social sources first. Each of these four participants proceeded to use Facebook to also look for information from authoritative sources; they also all navigated to the

pages of local television news stations for information. Jane also looked at the page of a local station’s chief meteorologist. Only Sean looked for information from the city’s official Facebook page.

Sally and Dale began their information search apart from Facebook, though both utilized it later in their searches. Sally navigated to Facebook after first viewing Google News results for the hurricane, which she found by typing the name of the hurricane into the search engine on her desktop browser’s homepage. This is the same approach Susan used when she went to Google after checking Facebook. Sally and Susan both viewed news articles from local news stations. Conversely, Dale navigated to Facebook and utilized Gmail to send personal messages as a last resort after researching “authoritative” sources.

Sean and Zoe relied exclusively on their smartphones when searching for information. Jane and Susan used their smartphones first before turning to their computers. Sally used her desktop exclusively and does not own a mobile phone with internet access. Dale began his search on his Apple TV but used his smartphone later in the observation, which he said was to minimize distractions but did not elaborate on.

Jane’s search behavior was non-typical among our observations because she searched for significantly more expert information - and more specific information - than the other participants. She was the only participant who visited websites dedicated to weather, and she visited websites dedicated specifically to tropical weather. She visited nhc.noaa.gov, the official website of the National Hurricane Center in the United States, and spaghettimodels.com, a website devoted to compiling weather models from various meteorological sources. She visited these authoritative sources after initially going on Facebook. Additionally, of the four participants who been told to evacuate for hurricanes within the past year, she was the only one to follow the order.

While not related directly to the observation, all three participants from Jacksonville, Florida, were under mandatory evacuation orders before Hurricane Irma hit the area in September 2017. However, they all ignored the order and stayed in their homes. At the time of the storm, Susan lived in a mobile home while Sean and Sally both lived within a third of a mile of a local river, putting all of them in the highest-risk flood zone according to city maps. Susan’s home was not damaged by the storm, but Sean and Sally’s homes were. Sally and her family had to move to the second floor of her home due to encroaching flood waters during the hurricane, and Sean and his family had to flee their home mid-storm due to rising waters inside their home. Sean’s home had approximately $50,000 in damage, and his vehicle was swept away by the waters. Despite this, none of the three Jacksonville participants would commit to evacuating should they face another evacuation order. Susan and Sally could not give a specific reason for their indecisiveness, while Sean stated that he didn’t expect “anything like that to happen again.” The three participants gave some similar reasons for not following the evacuation orders:

They were uncertain about how soon after the storm they would be required to return to work. They were concerned about reports of gas shortages and crowded roads. They were unwilling to go to shelters because they viewed them as dirty and dangerous.

In addition, Sean and Sally cited information from neighbors as a reason for not evacuating. Both said that in digital and in-person conversations with neighbors, they were told that the hurricane threat was being “overblown by the media.” Both said that long-time residents of their neighborhoods told them they weren’t worried because they’d never seen a hurricane do serious damage to the area.

InterviewsWe utilized the framework created for the observation to create our interview questions and

analyze our findings, focusing on steps 2, 3, and 4. We organized our interview findings into (1) Pre-Storm: Source Perception and (2) Post-Storm: Personal Impacts.

Pre-Storm: Source PerceptionWe organized the pre-storm perception into three sub-categories: (1) Trust of local knowledge; (2) Trust of social media; and (3) Trust of local news.

Trust of local knowledgeWe saw strong levels of trust in local knowledge by our participants; four out of five participants

said that before the storm hit, they put the greatest amount of trust into their own previous storm experiences and the experiences of their neighbors. None of the participants evacuated for the storm that they were interviewed about. Jesse said her neighbors influenced her to stay because they had been through so many storms, and Sandra said, “I trusted the people who had lived in this neighborhood. They’ve been through dozens of hurricanes. Once the storm hit, they said they’d never seen anything like it before, either.” Several participants also indicated that their personal history of safely experiencing tropical storms and hurricanes without evacuating influenced them to stay despite warnings, because they had done so in the past without damage or injury.

Trust of social mediaParticipants reported mixed levels of trust in social media. Each participant reported high levels

of smartphone use during the storm, with several participants saying they had checked social media “constantly.” However, participants did not report social media posts as trustworthy or useful in their preparations. Jesse said that the social media posts she remembered seeing were either of friends making preparations she had already completed or entertainment. “It was mostly funny,” she said, “like pictures of people boarding up with duct tape and doing stupid stuff.” Sandra reported that most of her social media interactions before the storm were with out-of-state friends and family that she felt were being overly dramatic. “They were freaked out by all the news about the storm,” she said. “They acted like I was gonna die if I stayed in Jacksonville.” Zoe stated that she got frustrated by posts from friends because she felt they were adding unnecessarily to local panic. “Everyone just starts posting news,” she said. “Some start immediately acting like, ‘Oh my god, this is horrible!’”

Trust of local newsEach participant stated that they viewed their local news sources as the least trustworthy source

of information, unless the news was directly reporting from an official source such as the weather service or local government. Official sources were generally viewed by our participants as trustworthy but not personally applicable; e.g., several participants stated that they understood why an evacuation was ordered, they didn’t personally need to evacuate because they had supplies and had taken steps to protect their home. Local news reports were criticized by four out of five participants. Sage said that she believed “their job is to make things sound scary,” while Courtney said that she felt storm coverage by her local news was repetitive and overly dramatic.

Post-Storm: Personal ImpactsWe organized the post-storm impact into two sub-categories: (1) Property Damage and

Emotional Toll and (2) Preparedness and Perception.

Property Damage and Emotional TollFour out of five participants (Jesse, Sandra, Sage, and Courtney) reported significant property

damage as a result of the storm they were interviewed about; none evacuated before the storm, although Sage had to escape from her home when floodwaters came inside during the storm. All four participants had their car damaged or destroyed. Sandra, Sage, and Courtney said their homes were damaged and some of their belongings were destroyed. Sandra said, “The months of trying to recover are just a fog. I can tell you that I lost all of my living room furniture, and the flooring had to be replaced. The baseboards, too. And parts of the walls had to be fixed. But none of that compares to all of the memories I lost. I cried every day I cleaned up. And cleaning up took months.”

Sandra was not the only participant to describe the storm and post-storm cleanup as an emotionally difficult experience. Sage said that seeing the floodwaters rise and having to escape with her family was the most terrifying few hours of her life, saying “It was f**king scary. We’re used to the street flooding, but when the water came in the door...I mean, how else do you react? I’ll be damned if I’m just gonna go upstairs. So we got out.”

Preparedness and perceptionOut of the four participants who experienced property damage as a result of the storms,

Courtney and Sage said that they have modified their storm plans for future storms. Zoe and Jesse felt that the preparation plan they had used during the previous storm was enough for future storms. Only Sandra stated that she felt like she would definitely evacuate for a future storm of similar strength. Sage said that while she would want to evacuate and would try to convince others to do so, she doubted that her family would. If they refused to evacuate, she said, she would stay with them again. “I couldn’t live with myself if something happened to them when I wasn’t there,” she said.Several participants explicitly stated that they would not have lower levels of trust in neighbors and personal experiences in preparation for future storms. Each participant said that they had spent a lot of -- or all of -- their life living in an area prone to hurricanes. “It’s just normal to me,” Zoe said.

SurveysWe tested two hypotheses: 1) there is a relationship between surviving previous hurricanes and

deciding whether to follow evacuation orders for an imminent storm, and 2) there is a relationship between a potential hurricane victim's level of trust in an information source and its influence on their pre-storm decision-making.

Evacuation likelihoodOut of the survey participants, 38 said they had not evacuated for their most recent hurricane

despite being under a mandatory evacuation order, as shown in Figure 3. Ten participants said they did not evacuate but were not under a mandatory evacuation order, and 43 said they did evacuate.

Figure 3. Evacuation decision for most recent hurricane

To test whether there is a relationship between surviving previous hurricanes and deciding whether to follow evacuation orders for an imminent storm, we conducted a Mann-Whitney U test. There was significant difference of the medians between the groups at an alpha level .05 (U (108) = 46, Z = 1.8, p = .03862). (The mean ranks of those who with experience and those without were 26.1 and 16.25, respectively). Those who had experienced a hurricane previously were reported that they are more likely to stay for the next one.

Source trust and influenceSocial media was the most common source for first learning about an imminent hurricane for nearly half of the participants, while government sources was the second-most common, as shown in Figure 4.

Figure 4. Source of first information about imminent hurricane

We investigated whether there were differences in reported in a potential hurricane victim’s level of trust in an information source and its influence on their pre-storm decision-making (those who chose to stay (42) and those who chose to leave (56)). The significance for the groups (stayed versus left) was tested individually across seven different information sources and their perceived trustworthiness, using a Kruskal Wallis. The differences were statistically significant in both tests.

For those who chose to stay, H(2) =32.9, p <.05, with a mean rank of 183.9 for social media advice, 186.8 for friends social media posts, 192.1 for local news, 192.6 for local government, 190.0 for the meteorologist, 199.5 for neighbors, and 209.8 for personal experiences.

For those who chose to leave, H(2) = 48.0, p <.05, with a mean rank of 150.5 for social media advice, 153.8 for friends social media posts, 170.6 for neighbors, 171.6 for local news, 202.4 for local government, 215.8 for the meteorologists, and 261.1 for personal experience.

Participants rated their own experience as most trustworthy source of information. Local meteorologists rated the second most trustworthy, with local government being rated slightly less, but still significantly trustworthy. Neighbors were on average trusted to some degree as was the News Media. Articles on social media, as well as advice of friends on social media, was not trusted.

Personas We created personas based on two types of resident: a new resident who is unfamiliar with life

in a hurricane-prone area and unsure of how to make pre-storm decisions, and a long-time resident who is resistant to evacuation based on previous experiences.

Features Matrix

F# Feature Priority Impact Feasibility User Type

F1 Testimony and damage videos from storm victims who didn’t follow evacuation orders

Medium

Medium High Evacuation skeptics

F2 Historical comparison of prior hurricanes to upcoming hurricane for community

High High Medium Evacuation skeptics

F3 Flooding/wind damage simulation based on user location

High High Medium Evacuation skeptics

F4 Financial toll calculator Medium

Medium Medium Evacuation skeptics

F5 Simulation of storm’s Low Medium Medium Evacuation skeptics/indecisive

current strength/damage capabilities

individuals

F6 Meteorological warning accuracy rates

Medium

Medium Medium Evacuation skeptics

F7 Fact checker for trending weather stories

Low Medium Medium Social media-reliant individuals

F8 Current and projected traffic conditions

Low Medium High Indecisive individuals

F9 Current fuel in stock at local gas stations/trends in stations running out

Medium

Medium Low Social media-reliant individuals/indecisive individuals

F9 Countdown to time when evacuation is no longer safe

Medium

Medium High Indecisive individuals

F10

Accurate preparation checklists for staying and for leaving

Medium

Low High Social media-reliant individuals/indecisive individuals/evacuation skeptics

F11

Assistant in categorizing items for insurance

Medium

Low Medium Indecisive individuals/evacuation skeptics

F11

Rapid push notifications of government warnings

Medium

Medium High Social media-reliant individuals/indecisive individuals

User ScenariosOur research findings hold potential for multiple applications and digital services. We envisioned

an app whose features primarily target individuals who are prone to ignoring mandatory evacuation orders with the acknowledgement that these features may be utilized by loved ones of individuals who are hesitant to evacuate. We believe these features will also be useful in helping indecisive individuals decide whether to leave as well as helping inexperienced residents understand the severity of the storm rather than relying on social media and neighbors’ anecdotal experiences.

Ella the New ResidentElla is used to tornadoes -- not hurricanes, having grown up in Oklahoma. She recently moved to

Miami and is anxious about the inevitable hurricane threat she’ll face there. Many of her colleagues and friends are Florida natives who brush off the threat, making jokes about “hurricane parties” and telling her she’ll get used to it soon enough.

When local meteorologists start showing a hurricane on track for south Florida, Ella takes matters into her own hands and downloads the StormPower app. She goes through the checklists to make sure she has the right supplies in the right places whether she decides to stay or go, gladly tucking away the birth certificate she’d have otherwise forgotten to keep safe. A friend invites Ella and her

roommate to stay at her house for the storm, but the hurricane is strengthening and projected to make landfall in three days, and Ella wants to head north to stay with her aunt in Orlando. Her roommate says she’s not going anywhere.

Ella opens the app to see that traffic is expected to be at its lowest at 10 that night and that the gas station closest to her is the only one that still has gas. She fuels up, heads home for a quick nap, and then leaves at 10 p.m., avoiding major traffic delays. She weathers the storm safely at her aunt’s home and returns to her apartment two days later to find her roommate is fine but shaken up. “I couldn’t sleep at all,” her roommate says. “I was too scared.”

Ian the Stubborn SkepticIan remembers being 12 years old and sitting in his bathtub with a flashlight when Hurricane

Allen hit his grandmother’s Louisiana home in 1980. Ian wasn’t scared then, and he isn’t scared now. A married man with a son of his own, Ian doesn’t think he’s flippant about his family’s safety; he just thinks that they can weather any storm by staying smart. He knows how to board up his windows, stockpile water, and has a rifle ready to protect his property and his loved ones. He doesn’t think there’s going to be a difference between the storms he’s been through and the storms still to come.

His wife, Anna, was pleased when they moved into a single-family home after years in a townhouse. It’s a sturdy home built in the 1940s, surrounded by trees that are even older and neighbors who have been there for decades. When a new storm starts brewing in the Atlantic, she keeps an eye on the forecasts and makes sure Ian does the same. Ian heads out for plywood and water jugs as soon as the forecast shows the storm heading their way. “It’ll ease up just like the others,” he says, but the Category 4 storm shows no signs of weakening.

Ian brings a neighbor in his 70s over to talk to Anna about her growing worries. “I’m still here, the house is still here, the trees are still here. We’re sturdier than any storm,” he tells them. But Anna downloads StormPower and sees that a storm of this strength has never hit their neighborhood. She pulls up a simulation showing the severity of damage that a Category 4 storm can do to a low-lying home like theirs. After watching it, Ian relents -- to a degree -- and says he’ll look into taking them out of state. But when Anna shows him a countdown to the time that meteorologists say it’ll be too dangerous to leave, Ian puts their bags and supplies in the truck and they head north to stay in a Tennessee hotel.

Four days after the storm hits, they get word that their neighborhood is accessible. They pull up to find a lot of their house standing, but a giant oak tree has crashed into their living room. Ian breathes a sigh of relief that his wife’s pressure may have saved their lives.

DiscussionIn our research, we examined what people do to inform their decisions when hurricanes are

imminent with the purpose of improving their decisions based on expert recommendations; specifically, we wanted to determine how technology can ensure people trust and follow mandatory evacuation orders. To frame our research, we created a four-step path to decision making: awareness, seeking information, comparing information, and decision making. Our observations found two distinct sources of information gathered: social sources and authoritative sources. Our interviews and surveys also revealed high reliance on personal experience to inform decision making.

Our observations and interviews indicated that social sources were often at odds with authoritative sources. Half of the observation participants and a majority of interview participants indicated that in past hurricanes, they had followed the advice of social sources over authoritative sources. Our survey, by contrast, showed lower levels of reported trust in social media. Our survey was in agreement with prior research in finding that residents who have ignored prior evacuation orders say they are likely to continue to do so.

We therefore believe future technology should consider methods to increase the value that people give to authoritative sources and decrease the value given to social sources and personal experiences. In addition, an early warning technology could mitigate the fears associated with evacuation expressed in our observations by providing: (a) the status and safety of local shelters: (b) current traffic condition and gas availability; and (c) best practices for evacuation if they choose to leave the region; e.g., a checklist of supplies and procedures. We believe technology could also be used to help visualize how storms could cause damage in a way that would allow individuals deciding to ignore orders based on personal experiences to see greater threats in oncoming storms.

In the observations, interviews, and surveys, participants said they still sought out information from authoritative sources, indicating they wanted to know about warnings from meteorologists and city leaders. However, previous research and our own research show a high likelihood of refusing to follow evacuation orders; this indicates a mistrust of authorities when it comes to hurricane readiness. This finding is supported by our surveys, which showed that participants who refused to follow evacuation orders still had relatively high levels of trust of the local government and meteorologists.

Our interviews shed more light on trust of neighbors and participants’ own experiences; while they said that listened to official information and indicated that they found it to be trustworthy, they didn’t consider it to be personally applicable, indicating that they believed they were adequately prepared. This is supported by survey data that shows participants rate their own experiences as the most trustworthy source of information. We believe that designs should also consider the emotional impacts of information delivery: our interviews indicate that people may be prone to dismissing severe warnings as fear-mongering and that they may try to seek reassurance from any available source.

Limitations and Future WorkWe were limited by a small sample sizes in our observations (n=6) and interviews (n=6), as such,

our findings may not generalize. We had a significantly larger sample size (n=100) for our survey, but the respondents were disproportionately female (81%). There were shortcomings of scenario-based observation because people may behave differently in an actual storm event. Additional shortcomings of the survey were a) the lack of reliability on respondents to accurately predict their actions ahead of future hurricanes, although it did allow for insight into their current perceptions of future storm risks, and b) the disproportionate number of respondents from North Carolina compared to other U.S. states. Future studies should draw from a participant pool made up more evenly of potential respondents in hurricane-prone areas (much of the Eastern seaboard and Gulf Coast states).

There are multiple areas for future study; one area to investigate is why people appear more likely to weigh personal experience so highly even when warned by officials not to and how to best counter that personal bias. Another potential area of research is into the complexities of trust surrounding official sources of information. While participants trusted official sources more than news media, many who were under evacuation orders chose to ignore the official advice.

Another research area is to observe social media usage in relation to “pre-storm” and “during storm” by tracking participant preference to see whether social media is still referred to first or if an alternate medium becomes more important as the storm advances. While offsite monitoring may aid this scenario, conducting a first-hand study in an anthropological manner may be more relevant.

Our survey demographic information showed another area for further research: the role of gender in decision making. While we were not able to access demographic information for the Facebook groups we got many of our participants from, men and women were seen interacting with other posts in the groups while women were overwhelmingly the respondents to the survey. This may simply be a result of a smaller sample size or a gender difference in the likelihood of participating in a survey;

however, it is worth examining whether women play greater roles in information gathering or decision making in hurricane preparations.

Finally, conducting explorative studies testing the viability of new media or nontraditional sources such as VR, online gaming interactions, smart speakers and smart watches may yield additional results or give additional insights for design. A VR demonstration shown on the Weather Channel as Hurricane Florence approached in September 2018 gained significant attention for its augmented portrayal of the residential impact of storm surge (Hill, 2018), indicating such visual tools could be a valuable and attention-gaining method of warning.

Bibliography

Hill, M. (2018, September 13). Weather Channel forecaster 'parts the waters' to showcase

flooding dangers. Retrieved from https://www.newscaststudio.com/2018/09/13/weather-

channel-flooding/

Joslyn, S. L., & LeClerc, J. E. (2012). Uncertainty forecasts improve weather-related

decisions and attenuate the effects of forecast error. Journal of Experimental

Psychology: Applied, 18(1), 126-140. doi:10.1037/a0025185

Knapp, A. (2018, September 11). SC Gov. McMaster’s evacuation for Florence

‘mandatory,’ but hard to enforce. The Post and Courier. Retrieved

from https://www.postandcourier.com/news/sc-gov-mcmaster-s-evacuation-for-florence-

mandatory-but-hard/article_10363150-b5c5-11e8-ae5c-0b9d9f00da15.html.

Landsea, C. (2018, August 1). How many direct hits by hurricanes of various categories.

Retrieved from http://www.aoml.noaa.gov/hrd/tcfaq/E19.html.

Merchant, B., & Cope, C. (2016, October 6). Haley says 175,000 in SC have evacuated:

‘That’s not enough’. The State. Retrieved from https://www.thestate.com/news/politics-

government/article106358012.html.

Monmouth University. (2013, February). Superstorm Sandy Survey: Impact on New Jersey

Coastal Residents. Retrieved from

https://www.monmouth.edu/polling-institute/documents/monmouthpoll_njsandycoast_05

0713.pdf/.

NOAA. (2018, May 22). Hurricane forecasting. Retrieved from

https://www.noaa.gov/explainers/hurricane-forecasting.

NOAA. (2013, June 28). What is the difference between a hurricane and a typhoon?

Retrieved from https://oceanservice.noaa.gov/facts/cyclone.html.

Appendix

Observation Informed Consent Form

ADULT CONSENT TO PARTICIPATE IN RESEARCH

Decision Making in Emergency Weather Events

Principal Investigators: Joshua Crow, Sam Turley, Zachary Cordes, Ziwei Feng - Graduate Students

Institution: DePaul University, Chicago, Illinois, USA

Department (School, College): Computing and Digital MediaFaculty Advisor: Cynthia Putnam, Ph.D. - CDM, DePaul University

What is the purpose of this research?We are asking you to be in a research study because we are trying to learn more about how people make decisions during weather emergencies. This study is being conducted by Joshua Crow, Sam Turley, Zachary Cordes, and Ziwei Feng, graduate students at DePaul University, as a part of a class research assignment required to obtain their Master’s degrees.

We hope to include about four people in the research.

Why are you being asked to be in the research?You are invited to participate in this study because you live in an area that is affected by strong weather events, like hurricanes. You are above the age of 18.

What is involved in being in the research study?If you agree to be in this study you will be asked to participate in a scenario based task. This will include a short interview, a period of observation, then followed by a short wrap up interview. Interviews will be recorded and if you so choose to use a computer, a video of the screen interaction will be recorded. We will be recording notes during the experience.

How much time will this take?This study will take about 30 minutes of your time. The time will be spent primarily in observation of the scenario based task.

Are there any risks involved in participating in this study?Being in this study does not involve any risks other than what you would encounter in daily life. You may feel uncomfortable or embarrassed about answering certain questions. You do not have to answer any question you do not want to. Are there any benefits to participating in this study?You will not personally benefit from being in this study. Your participation may lead to advancements in severe weather safety and preparedness.

Can you decide not to participate? Your participation is voluntary, which means you can choose not to participate. There will be no negative consequences, penalties, or loss of benefits if you decide not to participate or change your mind later and withdraw from the research after you begin participating.

Who will see my study information and how will the confidentiality of the information collected for the research be protected?The research records will be kept and stored securely. Your information will be combined with information from other people taking part in the study. When we write about the study or publish a paper to share the research with other researchers, we will write about the combined information we have gathered. We will not include your name or any information that will directly identify you. We will make every effort to prevent anyone who is not on the research team from knowing that you gave us information, or what that information is. However, some people might review or copy our records that may identify you in order to make sure we are following the required rules, laws, and regulations. For example, the DePaul University Institutional Review Board and/or the Data and Safety Monitoring Board may review your information. If they look at our records, they will keep your information confidential.

The audio recordings will be kept until accurate written notes have been made, then they will be destroyed.

Who should be contacted for more information about the research?Before you decide whether to accept this invitation to take part in the study, please ask any questions that might come to mind now. Later, if you have questions, suggestions, concerns, or complaints about the study or you want to get additional information or provide input about this research, you can contact the researchers:

Joshua [email protected](703) 919-7844

Sam [email protected](304) 208-5859

mailto:[email protected]


Zachary [email protected] (309) 669-7862

Ziwei [email protected](812) 369-1833

You can also contact the faculty advisor:Cynthia [email protected]

You may also contact DePaul’s Office of Research Services if:

Your questions, concerns, or complaints are not being answered by the research team. You cannot reach the research team. You want to talk to someone besides the research team.

You will be given a copy of this information to keep for your records.

Statement of Consent from the Subject:

I have read the above information. I have had all my questions and concerns answered. By signing below, I indicate my consent to be in the research.

Signature:_______________________________________________

Printed name: ____________________________________________

Date: _________________




Observation Protocol

Materials:

NotebooksLaptop or tablet computerSmartphonesAudio recorded

Step 1: Introduction

My name is ___________. Thank you for agreeing to be a part of this study. I am working on a project that aims to understand how people make decisions surrounding severe weather events.

Explain purpose:

The data collected today will only be used for this class assignment; only our research team will know your identity. This observation today will help us understand how people behave during weather emergencies and how technology might help inform those decisions.

Explain what will happen:

There are no right or wrong answers or behaviors. I simply wish to watch what you do and listen to what you say. I will take notes, record the session and possibly record your screen. I may ask questions before, during and after the observation period. This observation, and respective task, should last 20 to 30 minutes.

Informed consent:

This activity is voluntary. You have the right to not demonstrate any activity oranswer any question, and to stop the inquiry at any time or for any reason. Youractions and responses will be confidential and used only in connection with thisclass assignment. Only your first name will be used to identify you. If you wish,you can use a pseudonym rather than your real name.

[Give participant the informed consent form]

Please read over this informed consent form. Please feel free to ask me questions about this form and sign it if you consent to take part in this research.

Step 2: Warm up and rapport building

Do you live in an area that is affected by hurricanes/typhoons? (Phrasing depends on common usage in participant’s area)

Have you experienced a hurricane/typhoon first-hand?

Step 3: Observation

Now I would like to get started with our activity. I am interested in observing how you would go about finding information about a major hurricane that may impact where you are living, and what kind of information you would look for. Feel free to use any method of accomplishing this task. I have my laptop here for you to use if you would like. You can also use your phone or even your own computer if that is what you would typically use to find information.

[Observe]

Step 4: Wrap-up

What was this experience like for you? Could I have done something better or differently? Clarify questions noted during observation. Demographics? Have you evacuated before/why? How do you

prepare for a hurricane?

INTERVIEW PROTOCOL

Research Question:How can technology be used to increase informed decision-making by individuals preparing for an imminent weather emergency? Screener Questions:

1. Are you at 18 years or older?2. Is the area where you live (or have lived) at risk for hurricanes (typhoons,

cyclones)?

(If no to either or both questions, thank participant for indicating interest and inform them that they will not need to proceed with the interview.) IntroductionHello, my name is ______. Thank you for agreeing to be a part of this study. I am working with a team of DePaul University graduate students on a project that aims to understand how people make decisions ahead of severe weather events.

[Give participant the informed consent form.]

Do you mind if I [video or audio] record our interview?

Once we get started, let me know if you don’t understand a question so we can clarify. Keep in mind that there are no right or wrong answers. If there are any questions you don’t have an answer for or don’t feel comfortable answering, or you feel like it doesn’t apply to you, just tell me, and we’ll move on.

Do you have any questions before we start?

Let’s begin!

Warm-up Question[Here we want to build rapport and refresh their memory of severe storms they have encountered. We want to find out where they live or lived, what storms they dealt with, whether they have been under evacuation orders, how they decided to stay or leave, and how they felt about that decision.]

Please tell me about where you live. [Or lived, if applicable. Prompt for proximity to coast, rural/non-rural, type of housing, flooding potential. Let them talk - open ended.]

What was the most recent hurricane or tropical storm that significantly affected you? [Prompt for storm name.]

Do you remember the official intensity of that storm? [Tropical storm or hurricane? If hurricane, category 1, 2, 3, 4, or 5?]

Did you stay in your home for the duration of the storm? If not, where did you go?

Were you under evacuation orders? Was it a voluntary or mandatory evacuation? If so, how did you hear about the order? How long did it take for you to hear about the

order? What factors did you consider when deciding

whether to follow the order? How long did it take you to make a decision?

Do you regret that decision? Why or why not? [If participant stayed] How did you feel while the storm was

hitting you? [If participant evacuated] How did you determine when to

return home? Did the storm do any damage to your house or car?

If so, what kind? Can you tell me what the financial cost of the

damages were? Do you remember being hit by any other hurricanes or tropical

storms? [If yes, repeat previous questions, starting with intensity.]

Deep Focus - Past Storms and Technology[Here we want to find out how participants collected and evaluated information to inform their decisions. What pieces of technology or platforms did they use? How did they feel about these sources of information? Social vs. authoritative. Do they trust information sources? What information solidified their decisions? Why? How do they verify anecdotal information?]

Could you describe how you first heard about [name of most recent storm provided by participant]?

Before it hit, did you talk to your neighbors about the storm? Did talking to your neighbors make you feel more or less safe? Did they make you feel more or less prepared?

Before it hit, did you interact with social media posts about the storm on sites like Facebook? [If yes, ask the participant to tell you more about the types of posts. Who made them? What kind of comments were being made?]

Did these social media interactions make you feel more or less safe? Did they make you feel more or less prepared?

Before it hit, did you follow local news reports about the storm? If so:

Did you watch local news on TV? Did you visit local news websites? Did you read physical copies of local newspapers? Did you check local news websites? Did you visit local news pages on social media?

Did the local news reports make you feel more or less safe? Did they make you feel more or less prepared?

What information did you rely on to make your decision about staying or evacuating? It can be information that we haven’t talked about.

What information did you trust the most? Why? What information did you trust the least? Why? What kind of information was the most helpful? Why? What kind of information was the least helpful? Why?

Before the storm hit, were you using a smartphone to get information? If so, how often? What kind of information?

Before the storm hit, were you using a computer to get information?

If so, how often? What kind of information? Looking back on the storm, do you regret the decision you made about

staying or leaving? Would you make a different decision in the future? Why or why not?

Do you have an emergency plan for severe storms? If so, what are they?

Did you have this plan before [name of storm]? Did you follow it?

If not, why not?

ClosingThose are all of the questions I have for us today. Is there anything you’d like to add?

Do you have any questions for me?

Thank you for your participation, your insight and contribution will benefit others who face similar extreme-weather events as we continue to develop our platform.Survey Protocol

Research Questions What informs people’s decisions when hurricanes are imminent? How can technology-based solutions ensure people understand, trust,

and follow mandatory evacuation orders?

IntroductionWe are asking you to participate in this survey because we want to understand people’s decision making processes during extreme weather events such as hurricanes. This study is being conducted by Joshua Crow, Sam Turley, Zachary Cordes and Ziwei Feng of DePaul University. Our faculty advisor, Cynthia Putnam is supervising our research.

To participate in this study you must meet the following requirements: You must be 18 years of age or older. You must live (or have lived) in an area affected by hurricanes or

typhoons.

This survey will take approximately 15 to 20 minutes of your time. All responses are anonymous and we will not collect any identifiable personal

information. Your participation in this survey is voluntary; you may discontinue the survey at any time by closing the browser window.

If you have any further questions please contact:Joshua [email protected]

Sam [email protected]

Zachary [email protected] Ziwei [email protected]

You may also contact our faculty advisor Cynthia Putnam at [email protected] if:

Your questions, concerns, or complaints are not being answered by the research team.

You cannot reach the research team You want to talk to someone besides the research team

By clicking next, you are agreeing to participate and that you are 18 years or older and live (or have lived) in an area affected by hurricanes or typhoons.

Survey Questions

Question Instructions

Answer Options

Reasons

Approximately how far from the coastline, or flood prone body of water, is/was your residence the last time you experienced a hurricane?

0-5 miles6-10 miles11-20 miles20-50 miles50+ miles

To identify respondents storm impact potential.






How recently have you experienced a hurricane?

In the last 3 monthsIn the last 6 monthsIn the last yearIn the last 2 yearsIn the last 3 years or longer

To identify how recently respondents experienced hurricanes and current answers are.

Have you ever been under evacuation orders?

YesNo

To identify if respondents had to ever consider evacuation.

If yes, where did you first hear about the evacuation orders?

Choose one Friends/familySocial mediaGovernment sourcesLocal meteorologistNewsI do not remember

Approximately how many hurricanes, as an adult, have you lived through?

Open-ended To determine the number of hurricanes a respondent has experienced.

Among the hurricanes that you have experienced as an adult, when evacuation orders have been issued, how many times have you evacuated?

Open-ended To determine the percentage of hurricanes a respondent has or has not followed evacuation orders for.

For participants who chose not to evacuate

Think back to before the last hurricane hit and you were deciding what actions to take. Please indicate your level of

agreement or disagreement with with the following statements regarding making decisions before the hurricane hit.

“I trusted articles about the hurricane shared by friends on social media.”

Choose one (matrix)

Strongly DisagreeDisagreeNeutralAgreeStrongly AgreeNo Opinion

To determine trust in articles posted on social media as a source.

“I trusted advice posted by friends on social media.”


To determine trust in personal advice posted on social media as a source - differentiates types of sources inside the social media source.

“I trusted my neighbor’s advice.”


To determine trust in neighbors as a source.

“I trusted my own experience.”


To determine trust in personal, past experience.

“I trusted local government advice.”

Strongly DisagreeDisagreeNeutralAgreeStrongly Agree

To determine trust in authoritative sources.

No Opinion

“I trusted news media’s advice.”


To determine trust in new media sources.

“I trusted the local media meteorologist's advice.”


To differentiate sources within news media and authority.

Think back to before the last hurricane hit and you were deciding what actions to take. Please indicate your level of agreement or disagreement with the following statements. “My decision to not follow evacuation orders was influenced by the following information source:”

Articles shared on social media

Choose one (matrix)


To determine the effect articles shared on social media had on decision-making.

Friends' posts on social media


To determine the effect friends posts on social media had on decision-making.

No Opinion

Neighbor's advice Strongly DisagreeDisagreeNeutralAgreeStrongly AgreeNo Opinion

To determine the effect neighbor’s had on decision-making.

My own experience Strongly DisagreeDisagreeNeutralAgreeStrongly AgreeNo Opinion

To determine the effect the respondents person experience had on decision-making.

Local government advice Strongly DisagreeDisagreeNeutralAgreeStrongly AgreeNo Opinion

To determine the effect local government advice had on decision-making.

News media's advice Strongly DisagreeDisagreeNeutralAgreeStrongly AgreeNo Opinion

To determine the effect new media had on decision-making.

Local media meteorologist's advice


To determine the effect the local media meteorologist’s advice had on decision-making.

Were there any other sources of information that contributed to your

Open-ended

decision not to evacuate?

For participants who chose to evacuate

Think back to before the last hurricane hit and you were deciding what actions to take. Please indicate your level of agreement or disagreement with with the following statements regarding making decisions before the hurricane hit.

I trusted articles about the hurricane shared by friends on social media.

Choose one (matrix)


To determine trust in articles posted on social media as a source.

I trusted advice posted by friends on social media.


To determine trust in personal advice posted on social media as a source - differentiates types of sources inside the social media source.

I trusted my neighbor’s advice.


To determine trust in neighbors as a source.

I trusted my own experience.


To determine trust in personal, past experience.

I trusted local government advice.


To determine trust in authoritative sources.

I trusted news media’s advice.


To determine trust in new media sources.

I trusted the local media meteorologist's advice.


To differentiate sources within news media and authority.

Think back to before the last hurricane hit and you were deciding what actions to take. Please indicate your level of agreement or disagreement with the following statements. My decision to follow evacuation orders was influenced by the following information source:

Articles shared on social media

Choose one (matrix)


To determine the effect articles shared on social media had on decision-making.

Friends' posts on social media

Strongly DisagreeDisagreeNeutral

To determine the effect friends posts on social media had on decision-making.

AgreeStrongly Agree

Neighbor's advice Strongly DisagreeDisagreeNeutralAgreeStrongly Agree

To determine the effect neighbor’s had on decision-making.

My own experience Strongly DisagreeDisagreeNeutralAgreeStrongly Agree

To determine the effect the respondents person experience had on decision-making.

Local government advice Strongly DisagreeDisagreeNeutralAgreeStrongly Agree

To determine the effect local government advice had on decision-making.

News media's advice Strongly DisagreeDisagreeNeutralAgreeStrongly Agree

To determine the effect new media had on decision-making.

Local media meteorologist's advice


To determine the effect the local media meteorologist’s advice had on decision-making.

Were there any other sources of information that contributed to your decision to evacuate?

Open-ended

To what level do you agree or disagree: "My experience during the last hurricane that orders were issued for made me more likely to stay during future hurricanes."

Strongly DisagreeDisagreeNeutralAgreeStrongly AgreeNot applicable

To identify what value respondents place on past experience informing future disciplines.

What is your age? Drop-down menu

To determine the effect, if any, age has on response.

What gender do you identify as?

MaleFemaleOtherPrefer not to say

To determine the effect, if any, gender has on response.

If in the United States, what state do you live in?

Drop-down menu

To determine the effect, if any, geographic location has on response.

Do you have anything you wish to share/add?

Open-ended Allows for flexible response/ability to share opinion

Affinity Diagram - Observation

https://stormboard.com/invite/664204/snake383

https://stormboard.com/invite/664204/snake383

Affinity Diagram – Interviews

Link to Survey Results

https://docs.google.com/spreadsheets/d/13WgL2Q607Yucki1mUrBSfNx6xdCuK3z6plGaLfAP2nE/edit?usp=sharing



Executive Summary · Web viewJoshua Crow - Sam Turley - Zachary Cordes - Ziwei Feng Contents...

Documents

Transcript of Executive Summary · Web viewJoshua Crow - Sam Turley - Zachary Cordes - Ziwei Feng Contents...