Augmented Reality in Transportation ConstructionFHWA Contract DTFH6117C00027: LEVERAGING AUGMENTED REALITY FOR HIGHWAY CONSTRUCTION

Hoda Azari, Nondestructive Evaluation Research Program Manager

IHEEP 2018 – Lincoln, Nebraska. September 25, 2018

September • 2018

Augmented Reality Research Project

FHWA Contract DTFH6117C00027

Thorough review of state-of-the

art augmented reality (AR)

technologies and applications for

Highway Construction

Draft Report will be completed at

the end of 2018

Augmented Reality (xR) Terminologies

Reality Virtual RealityAugmented

RealityAugmented

Virtuality

The Reality – Virtuality Continuum

Mixed Reality (xR)

Augmented Reality (xR) Terminologies

Reality Virtual RealityAugmented

RealityAugmented

Virtuality

Mixed Reality (xR)

Daqri Smart Glasses (AR)

Google Daydream (VR)

Augmented Reality (xR) Terminologies

Reality Virtual RealityAugmented

RealityAugmented

Virtuality

Mixed Reality (xR)

Typical AR Devices are closer to the ‘Reality’ end of the spectrum:

• Primarily ‘See-through’ of reality• Some VR overlay

Augmented Reality Technologies

Components of AR System:

Sensory input

Display device

Haptic input (navigation/interaction)

Tracking Systems

Computing Device

Media Representation

Data Input / Storage

Augmented Reality Technologies

Display devices fit into two categories

Hand-held devices – tablets, smart phones

Head-mounted displays (HMD) / Glasses

Augmented Reality Technologies

Tracking Methods – Position and Orientation

GNSS / Wifi networks / V2X/I2X (radio systems for CAV)

Inertial sensors – track local movements of the device

Optical sensors – track 2D video or 3D imagery of the real-world

Sensor fusion – systems that combine all of the tracking methods

Marker-based position initialization

Augmented Reality Technologies

Hand-held devices and commercial AR applications

Google Translate

PokemonGo

Augmented Reality Technologies

Hand-held devices and AR applications Video see-through: real-world captured by camera on device

GNSS positioning / Inertial tracking for orientation and movement

Some devices use optical or 3D sensor tracking

Initial registration typically achieved with markers / targets in scene

Hand-helds on construction sites not necessarily new – AR could be add-on

to existing uses of mobile devices

Augmented Reality Technologies

Head-mounted displays Optical see-through: real-world seen directly through lenses

GNSS positioning / Inertial tracking for orientation and movement

Some devices use optical or 3D sensor tracking

Initial registration typically achieved with markers / targets in scene

AR Information Display Options

Categories of information display that AR will support:

Display what is not yet constructed• Compare design alternatives in context

• Check relationships between existing/future elements

• Site logistics, equipment movements

• Preview complex installation procedures

• Illustrate construction methods and sequencing

• Opportunities for training on-site

• Check traffic management/temporary structures

Display what was intended to be constructed• Site inspection and validation

• Code/standards/compliance checking

• Checking quantities and work progress

• Inspection training opportunities

• Check traffic management/temporary structures

AR Information Display Options

Categories of information display that AR will support:

Display what is hidden from view• Buried utilities or structural components

• Elements obstructed from the current view

Display abstract information aligned with real-world context• Alignment information, easements, site boundaries, ROW boundaries

• Environmental boundaries, such as flood levels or sea-level rise data

• Sensitive areas such as archeological and historic sites

• Meta-data tagged to associated real-world objects

• Potential work-zone hazards

Augmented Reality Challenges

Challenges with current systems

Field of View (FOV) • Handheld devices limited to FOV displayed on screen held at arms length• Current HMD’s typically limit virtual display to 60 degrees FOV (see next slide)

Field of View (FOV)

Right eye 135⁰Left eye 135⁰

Binocular Field of View 120⁰

Typical AR Display FOV60⁰

Augmented Reality Challenges

Challenges with current systems

Occlusion

• Current systems can only display overlaid virtual information in the foreground

• Virtual elements that are ‘behind’ real-world elements are not masked out

• Limits the immersive quality of the AR representation

Calculating occlusion requires that the AR application have detailed 3D information of both the real-world scene and the virtual model

Augmented Reality Challenges

Challenges with current systems

Ruggedness / Durability• Current devices not designed for daily outdoor use, safety issues

Performance / Portability• Display processing must be performed on-board the device

• Some devices tethered to ‘wearable’ computing device – adds weight

• Most applications require storage for ‘pre-loading’ of 3D assets onto the device

• WiFi connectivity / streaming could allow assets to be loaded on demand – no examples of this yet (solvable in the near future with 5G?)

Safety• Hand-held devices – use one or usually both hands and block view

• HMD’s – can limit peripheral vision, and possibly audio

Augmented Reality Challenges

Challenges with current hardware

Tracking technologies• GNSS requires visibility of satellite network• Most systems require marker-based or manual registration• Optical / 3D sensors require ‘visible’ detail in environment• Bright / outdoor scenes are difficult for optical tracking

These last two requirements could be key challenges with construction, especially in an outdoor work environment without details to track

Display brightness• Current display technology in HMD’s is limited in ability to display over optical

see-through devices in a bright outdoor environment

Matching virtual display to real-world brightness problematic, especially in outdoor environment

Augmented Reality Software

AR Ready ‘Cloud-based’ Platforms

Autodesk• BIM360 Platform – Not currently supporting Geo-referencing• Forge– development platform

Bentley• Bentley Connect

Trimble• Trimble Connect

AR Ready Developer Platforms• Apple ARkit, Google ARCore, Windows Mixed-Reality, WebAR

Game Platforms - Unity most common in AR

Augmented Reality Workflows

AR in 3D Model-based Workflows• AR should become a critical tool within existing BIM model workflows

• Piggyback on BIM data management and 3D modeling efforts

• Some current BIM platforms already supporting AR tools

AR in Workflows for other data types

• Leveraging other types of information (non-3D) will require new tools and workflows

• Abstract data could be included in 3D model if geolocated correctly

ARTC Workshop

Augmented Reality in Transportation Construction (ARTC)

Held May 9th 2018, at Turner Fairbanks Highway Research Facility

~40 participants

• FHWA / State DOT’s

• University / Research

• Contractors / Consultants

• AR Vendors / R&D

Goals:

• Educate participants in AR

• Identify new applications for AR

• Prioritize future applications of AR

ARTC Workshop

Conducted in Three Sessions

1: Overview of AR Technologies

• Current display technologies

• Characteristics of AR systems

• Challenges with AR tools for construction

2: Vendor technology presentations and hands-on demonstrations

• Participants were able to see and experience the AR tools

3: Brainstorming and participant polling of application opportunities

• Discussion and identification of potential AR applications

• Documentation of challenges for AR applications in construction

• Electronic ranking of top AR application concepts

ARTC Workshop

Key themes emerged in the brainstorming session on AR

application areas in construction

1: Visualization of design information

• Simplified 3D model components

• Abstract 2D information such as alignments or ROW

• Accessible through existing 3D workflows

2: AR to support construction inspection activities

• Comparing as-built condition with as-designed virtual information

• Access to context specific inspection criteria and specifications

3: AR as a training support tool

• In particular for construction inspection activities

ARTC Workshop

Key challenges identified in the second brainstorming session

on roadblocks to AR implementation

1: Challenges with the technology• AR systems and hardware still new

• 3D workflows required to implement AR

2: Challenges within STA’s in implementing AR• Cultural shift- changes in workflows, acceptance of new and untried

technologies

• Potential costs for new computing systems needed

• Labor costs for more advanced 3D workflows

ARTC Workshop

Informal Vote of Participants on Applications - Top 10 Ideas Identified:

Site planning and Logistics

AR support of automated code/standard compliance of proposed designs

AR to support road safety audits/hazard recognition/hazards to the traveling public

Provide visualization to the property owner to better understand a project’s impact and show

options to the property owner as well.

Visually tag variances in schedule and quality issues so that both sides are viewing the same issue

Streamline/Assist with NEPA and upholding other project commitments

AR to support training and certification for construction inspection

Verify the work is being installed properly by having the right information and right format for the

inspector

AR to facilitate faster approval of pay estimates

Automated layout of systems

ARTC WorkshopWhat Type of Agency or Field Do you Work In?

1. Federal Agency (23%)

2. State Agency (23%)

3. Technology Developer (35%)

4. Engineering and Construction Industry Firms (11%)

5. University or Research (8%)

( 26 RESPONDENTS )

Prioritization

Impact – the impact of the proposed augmented reality

application in improving project performance

1 – little to no impact

9 – great impact

Feasibility – level of effort required to implement the proposed

augmented reality application considering both technical and

human resource requirements

1 – significant effort

9 – little to no effort

ARTC Workshop Polling What would be the IMPACT of implementing “Concept X”

1. Extremely Low

2.

3. Low

4.

5. Neutral

6.

7. High

8.

9. Extremely High

ARTC Workshop PollingWhat would be the Feasibility of implementing “Concept X”

1. Extremely Difficult

2.

3. Difficult

4.

5. Neutral

6.

7. Easy

8.

9. Extremely Easy

ARTC Workshop PollingResults of Participant Polling on Applications - Top 5 by ‘Impact’ Rating:

Inspector’s Toolkit: use of augmented reality to verify that work is being installed properly by providing the

right information and right format for the inspector in the field.

Impact: 7.54, Feasibility: 3.70

NextGen Training and Certification: use of augmented reality supported training and certification for

construction inspection.

Impact: 7.24, Feasibility: 4.73

Augmented Reality support of Right-of-Way Acquisition: provide project visualization to property owners

to better understand a project’s impact and to show design options to the property owner as well.

Impact: 7.12, Feasibility: 6.08

Automated Inspection: use of augmented reality to support automated code/standard compliance checking

of installed items through machine learning.

Impact: 7.08, Feasibility: 3.44

Visual Variances: visually tag variances in the field with regards to schedule and quality issues to ensure all

parties are viewing the same issue.

Impact: 6.92, Feasibility: 4.54

ARTC Workshop PollingL

OW

IM

PA

CT

–H

IGH

FE

AS

IBIL

ITY

HIG

H I

MP

AC

T –

HIG

H F

EA

SIB

ILIT

Y

LO

W IM

PA

CT

–L

OW

FE

AS

IBIL

ITY

HIG

H IM

PA

CT

–L

OW

FE

AS

IBIL

ITY

1.5 1 0.5 -0.5 -1 -1.5

Site Planning and Logistics

Automated Code/Standard

Compliance

Hazard Recognition

ROW Visualization

Visually Tag

Variances NEPA/Commitments

Assist

Training/Certification

Provide Right Info

at Right Time

Pay Estimates

Automated Layout

Impact vs. Feasibility of Proposed AR Applications (Overall)2

1.5

1

0.5

0

-0.5

-1

-1.5

-2

Results of Participant Polling on Applications – Plot of Impact vs. Feasibility:

ARTC Workshop PollingResults of Participant Polling on Applications – Plot of Impact vs. Feasibility:

Site Planning and Logistics

Automated Code/Standard

ComplianceHazard Recognition

ROW Visualization

Visually Tag Variances

NEPA/Commitments

Assist

Training/Certification

Provide Right Info

at Right Time

Pay Estimates

Automated Layout

Impact vs. Feasibility of Proposed AR Applications (State & Federal)1.2

1

0.8

0.6

0.4

0.2

0

-0.2

-0.4

-0.6

-0.8

-1

-1.2

1.5 1 0.5 -0.5 -1 -1.5

LO

W I

MP

AC

T –

HIG

H F

EA

SIB

ILIT

YH

IGH

IM

PA

CT

–H

IGH

FE

AS

IBIL

ITY

LO

W IM

PA

CT

–L

OW

FE

AS

IBIL

ITY

HIG

H IM

PA

CT

–L

OW

FE

AS

IBIL

ITY

ARTC Workshop PollingResults of Participant Polling on Applications – Plot of Impact vs. Feasibility:

LO

W I

MP

AC

T –

HIG

H F

EA

SIB

ILIT

YH

IGH

IM

PA

CT

–H

IGH

FE

AS

IBIL

ITY

LO

W IM

PA

CT

–L

OW

FE

AS

IBIL

ITY

HIG

H IM

PA

CT

–L

OW

FE

AS

IBIL

ITY

Site Planning and Logistics

Automated Code/Standard

ComplianceHazard Recognition

ROW Visualization

Visually Tag Variances

NEPA/Commitments Assist

Training/Certification

Provide Right Info

at Right Time

Pay Estimates

Automated Layout

Impact vs. Feasibility of Proposed AR Applications (Engineering & Construction)3

2.5

2

1.5

1

0.5

0

-0.5

-1

-1.5

-2

-2.5

-3

1.5 1 0.5 -0.5 -1 -1.5

Site Planning and Logistics

Automated Code/Standard

Compliance

Hazard Recognition

ROW Visualization

Visually Tag Variances NEPA/Commitments

Assist

Training/Certification

Provide Right

Info at Right

Time

Pay Estimates

Automated Layout

Impact vs. Feasibility of Proposed AR Applications (Research & Development)

ARTC Workshop Polling

2

1.5

1

0.5

0

-0.5

-1

-1.5

-2

2.0 1.5 1 0.5 -0.5 -1 -1.5 -2.0

Results of Participant Polling on Applications – Plot of Impact vs. Feasibility:L

OW

IM

PA

CT

–H

IGH

FE

AS

IBIL

ITY

HIG

H I

MP

AC

T –

HIG

H F

EA

SIB

ILIT

Y

LO

W IM

PA

CT

–L

OW

FE

AS

IBIL

ITY

HIG

H IM

PA

CT

–L

OW

FE

AS

IBIL

ITY

Thank YouHODA AZARI NONDESTRUCTIVE EVALUATION RESEARCH PROGRAM DIRECTOR

TURNER FAIRBANKS HIGHWAY RESEARCH FACILITY, MCLEAN, VAhoda.azari@dot.gov

September • 2018

VENDOR DEMONSTRATION SESSIONS

• Autodesk

• Trimble

• Banc3

• Hololens

BRAINSTORMING SESSION 1

1. Based on what you have seen today, how could state transportation agencies immediately use augmented reality?

1. For construction inspection

2. Human Resource Development (training)

3. Improved project management and QA

4. Other areas?

2. If there was opportunity to further develop augmented reality, how would you like to use the technology on future projects?

1. For construction inspection

2. Human Resource Development (training)

3. Improved project management and QA

4. Other areas?

BRAINSTORMING SESSION 2

3. What are the greatest challenges in state

transportation agencies’ use of augmented

technology?

• Technology barriers?

• Human resource barriers?

• Process barriers?

BRAINSTORMING SESSION 3

4. What would it take for state transportation

agencies to successfully implement AR?

On their projects?

Within their agency?

POLLING OF POTENTIAL STRATEGIES

Capture Data from Attendees

Identify Key Strategies for AR Implementations