04/19/23 13:16 1
Can Real-Time Systems be built with
Off the Shelf Components?
Krithi Ramamritham
Real-Time Systems LaboratoryUniversity of Massachusetts, Amherst& Indian Institute of Technology, Bombay
04/19/23 13:16 2
Talk Outline Using off-the-shelf components for
Real-Time applications? Future application characteristics Problems and challenges
Characteristics and limitations of OTS OSs
OTS component based solutions for distributed Real-Time applications User-level scheduling of communicating RT tasks
Conclusions, recommendations, future
04/19/23 13:16 3
EWS
Network
Field Devices
Field Network
PLC
Field Devices
Field Network
PLC
Operator Commands
Video/audio
Device monitoring
Background
Server
M.Services
LynxOS
M. Services
VxWorks
M. Services M. Services M. ServicesM. Services
OperatorStations
M. Services NT
Industrial Control Environment
04/19/23 13:16 4
Future Application Characteristics
Interactions and communication
Human/device to human/device
Co-existence of multiple types of media Small control and signal data Periodic updates Bursty file/page/image access and transfer Continuous media
Computer and communication technology advances
04/19/23 13:16 5
What Are the New Challenges?
Communication centered Operating System is no-longer standalone
Support for real-time and non-real-time co-existence
Integrated, end-to-end solutions required Application-level control of system resources
Need to support new applications and technologies
04/19/23 13:16 6
System Research to Meet the Challenges
Network Architecture
Network
Interface
Design
Resource
ManagerScheduling
Algorithms
Scheduler
Middleware
Services
Traffic Management
Rate Control
04/19/23 13:16 7
Real-Time Spectrum
Hard
Soft
04/19/23 13:16 8
Real-Time OS Spectrum
Hard
Soft
Real-Time Operating System
General-PurposeOperatingSystem
VxWorks, Lynx, QNX, ...Intime, HyperKernel, RTLinux
(Windows NT, Linux)
(Windows NT, Linux)
04/19/23 13:16 9
Using General Purpose Operating
Systems
GPOS offer some capabilities useful for real-time system builders
RT applications can obtain leverage from existing development tools and applications
Some GPOSs accepted as de-facto standards for industrial applications
04/19/23 13:16 10
Windows NT -- for RT applications?
Scheduling and priorities Preemptive, priority-based scheduling
non-degradable priorities priority adjustment
No priority inheritance No priority tracking Limited number of priorities No explicit support for guaranteeing timing constraints
04/19/23 13:16 11
Windows NT -- for RT applications? (contd.)
Quick recognition of external events Priority inversion due to Deferred Procedure Calls (DPC)
I/O management Timers granularity and accuracy
High resolution counter with resolution of 0.8 sec. Periodic and one shot timers with resolution of 1 msec.
Rich set of synchronization objects and communication mechanisms. Object queues are FIFO
04/19/23 13:16 12
Talk Outline Using off-the-shelf components for
Real-Time applications? Future application characteristics Problems and challenges
Characteristics and limitations of OTS OSs
OTS component based solutions for distributed Real-Time applications User-level scheduling of communicating RT tasks
Conclusions, recommendations, future
04/19/23 13:16 13
Goals - I
Evaluate the real-time capabilities of NT.
Identify areas where NT is (not) suitable for real-time applications.
Determine to what extent the unpredictable parts of NT can be “masked”.
Offer recommendations to designers using NT.
04/19/23 13:16 14
Priority Model
Real-timeclass
2625242322
16 Idle
Above NormalNormalBelow NormalLowest
Highest31 Time-critical
Dynamicclasses
15 Time-critical
14131211
15
High class
1 Idle
987
11
Normal class10
5432
6
Idle class
ThreadLevel
04/19/23 13:16 15
Thread Priority = Process class + level
Real-timeclass
2625242322
16 Idle
Above NormalNormalBelow NormalLowest
Highest31 Time-critical
Dynamicclasses
15 Time-critical
14131211
15
High class
1 Idle
987
11
Normal class10
5432
6
Idle class
ThreadLevel
04/19/23 13:16 16
Scheduling
Threads scheduled by executive.
Priority based preemptive scheduling.
Interrupts
Deferred Procedure Calls (DPC)
System anduser-level threads
04/19/23 13:16 17
Servicing an interrupt
Power Failure...
APCNormal Exec.
Dispatch/DPC
Device X...
InterruptDispatch Table
1
Device interrupts
InterruptServiceRoutine
DPCRoutineDevice Driver
DPC
DPC DPC
DPC FIFO Queue
DPC DPC
DPC FIFO Queue
user-level threads
Interrupts
DPC
04/19/23 13:16 18
Servicing an interrupt
Power Failure...
APCNormal Exec.
Dispatch/DPC
Device X...
InterruptDispatch Table
1
Device interrupts
2
Transfercontrol to ISR
InterruptServiceRoutine
DPCRoutineDevice Driver
DPC
DPC DPC
DPC FIFO Queue
DPC DPC
DPC FIFO Queue
user-level threads
Interrupts
DPC
04/19/23 13:16 19
Servicing an interrupt
Power Failure...
APCNormal Exec.
Dispatch/DPC
Device X...
InterruptDispatch Table
1
Device interrupts
2
Transfercontrol to ISR
InterruptServiceRoutine
DPCRoutineDevice Driver
DPC
DPC DPC
DPC FIFO Queue
DPC DPC
DPC FIFO Queue
user-level threads
Interrupts
DPC
04/19/23 13:16 20
Servicing an interrupt
Power Failure...
APCNormal Exec.
Dispatch/DPC
Device X...
InterruptDispatch Table
1
Device interrupts
2
Transfercontrol to ISR
InterruptServiceRoutine
DPCRoutineDevice Driver
DPC
DPC DPC
DPC FIFO Queue
DPC DPC
DPC FIFO Queue
user-level threads
Interrupts
DPC
3
Stop int.. andqueue DPC
04/19/23 13:16 21
Servicing an interrupt
Power Failure...
APCNormal Exec.
Dispatch/DPC
Device X...
InterruptDispatch Table
1
Device interrupts
2
Transfercontrol to ISR
InterruptServiceRoutine
DPCRoutineDevice Driver
DPC
DPC DPC
DPC FIFO Queue
DPC DPC
DPC FIFO Queue
user-level threads
Interrupts
DPC
3
Stop int.. andqueue DPC
04/19/23 13:16 22
Servicing an interrupt
Power Failure...
APCNormal Exec.
Dispatch/DPC
Device X...
InterruptDispatch Table
1
Device interrupts
2
Transfercontrol to ISR
InterruptServiceRoutine
DPCRoutineDevice Driver
3
Stop int.. andqueue DPC
DPC
DPC DPC
DPC FIFO Queue
DPC DPC
DPC FIFO Queue
4 Task level drops and DPC can execute
user-level threads
Interrupts
DPC
04/19/23 13:16 23
Servicing an interrupt
Power Failure...
APCNormal Exec.
Dispatch/DPC
Device X...
InterruptDispatch Table
1
Device interrupts
2
Transfercontrol to ISR
InterruptServiceRoutine
DPCRoutineDevice Driver
3
Stop int.. andqueue DPC
DPC
DPC DPC
DPC FIFO Queue
DPC DPC
DPC FIFO Queue
4 Task level drops and DPC can execute
5
Transfercontrol to driver’s
DPC
user-level threads
Interrupts
DPC
04/19/23 13:16 24
Servicing an interrupt
Power Failure...
APCNormal Exec.
Dispatch/DPC
Device X...
InterruptDispatch Table
1
Device interrupts
2
Transfercontrol to ISR
InterruptServiceRoutine
DPCRoutineDevice Driver
3
Stop int.. andqueue DPC
DPC
DPC DPC
DPC FIFO Queue
DPC DPC
DPC FIFO Queue
4 Task level drops and DPC can execute
5
Transfercontrol to driver’s
DPC
6
Execution of DPCroutine
user-level threads
Interrupts
DPC
04/19/23 13:16 25
I/O Handling
I/O request is sent to device driver. Device completes operation and interrupts. Complete I/O request.
Buffered I/O Direct I/O
APC
Device
System
User’s space
Device
System
User’s space
(Keyboard, mouse) (disk, network)
04/19/23 13:16 26
Prototype Software Architecture
Operator StationAcquisition &Control Equipment
Heartbeat Timer Heartbeat Timer
Command
Ack.
RealVideo
Receiver
Producer
ConsumerBuffer
Highest Priority
Normal Priority
Real-Time Class
04/19/23 13:16 27
Performance Metrics
Round Trip Time (RTT) as seen by the operator input.
Rate of execution of sensor data processing entities.
Quality of the video output.
04/19/23 13:16 28
Workload
Number of sensor data streams (2-20).
Period of new sensor values (10-1000 ms).
Period of control messages (10-100 ms).
Amount of work done in processing data.
One Video and audio.
04/19/23 13:16 29
Operator Command Performance
Real-Time Priority Class
0
10
20
30
40
50
60
0 200 400 600 800 1000
Sample no
RTT
(ms)
High Priority Class
0
10
20
30
40
50
60
0 200 400 600 800 1000
Sample no.
RTT
(ms)
Normal Priority Class
0
10
20
30
40
50
60
0 200 400 600 800 1000
Sample no.
RTT
(ms)
• Two 1KB Sensor data streams • 1 second update rate• 30 ms period for control messages• No work
04/19/23 13:16 30
Operator command Performance
• NT Scheduler - same RT priority • 16 Sensor data streams (1KB) • Update rate (100, 200, 500, 1000 ms)• 90 ms period for control messages• work
Real-Time Priority Class
01020304050607080
0 200 400 600 800 1000
Sample #
RT
T (
mse
c)
04/19/23 13:16 31
Operator command Performance(user-level scheduling)
Real-time Priority Class
0
20
40
60
80
0 500 1000 1500 2000
Sample #
RTT
(ms)
• Rate Monotonic with limited levels• 16 Sensor data streams (1Kb)• Update rate (100, 200, 500, 1000 ms)• 50 ms period for control messages• work
Real-time Priority Class
0
20
40
60
80
0 200 400 600 800 1000
Sample #
RT
T (
ms)
• Time-cognizant dispatcher + plan• 8 Sensor data streams (1Kb)• Update rate (100, 200, 500, 1000 ms)• 100 ms period for control messages• work
04/19/23 13:16 32
Design principles and
recommendations:
Do not depend on NT scheduler to accomplish timing behavior in interactive applications.
Utilize user-level scheduling to achieve higher predictability.
If possible, characterize duration of I/O activity and its frequency.
Lock pages in memory for real-time threads. Manage and control utilization of systems
resources.
04/19/23 13:16 33
Use a General Purpose OS for RT?
It is possible to improve the predictability of real-time tasks. It is not possible to “mask” all sources of unpredictability
within NT “as is”. Designer needs to be aware of the effects DPC queue on any
user thread. I/O handling.
Prototype application demonstrates the uses of the recommendations.
K. Ramamritham, C. Shen, O. González, S. Sen, S.B. Shirgurkar. Using Windows NT for Real-Time Applications. In Proc. of the 4th IEEE Real-Time Technology and Applications Symposium, Denver, CO, June 1998.
04/19/23 13:16 34
Talk Outline Using off-the-shelf components for
Real-Time applications? Future application characteristics Problems and challenges
Characteristics and limitations of OTS OSs
OTS component based solutions for distributed Real-Time applications User-level scheduling of communicating RT tasks
Conclusions, recommendations, future
04/19/23 13:16 35
Challenges in SupportingCommunicating Real-Time Tasks
Network ArchitectureNetwork
Interface
Design
Resource
Manager
Scheduling
AlgorithmsScheduler
Middleware
Services
Traffic Management
Rate Control
04/19/23 13:16 36
Communications Middleware
Transporting multiple media types, control, data, visual,
image, alarms, video and audio
Integrating control with visual monitoring
Enabling plug-and-play
QoS management
04/19/23 13:16 37
Writer DPA_1
DPA_nReader_i
Network
Node 1 Node 2
ReMA
ReMA
ReMA = Reflective Memory Area
Real-Time Channel-based Reflective Memory (RT-CRM)
DPA = Data Push Agent
DRA = Data Receive Agent
DRA_n
04/19/23 13:16 38
RT-CRM Operation Modes
Synchronous/Asynchronous
Writer DPA_1
DPA_nReader_i
Network
Node 1 Node 2
ReMA
ReMA
Blocking / Non-blocking
04/19/23 13:16 39
Network
Node 1 Node 2
Using MidART services
Write Cmd
DPA
Process Cmd
DPA
Result
• Synchronous mode
04/19/23 13:16 40
End-to-End QoS Provisions
Writer DPA_1
DPA_nReader_i
Network
Node 1 Node 2
ReMA
ReMA
Memory-to-Memory
Application-to-Application
04/19/23 13:16 41
GPOS Problems
No priority inheritance Among user processes/threads
No Priority tracking From user to system/network threads
Limited number of priorities
No explicit support for guaranteeing timing constraints
04/19/23 13:16 42
Problem alleviated:
No priority inheritance
and limited priorities
No priority tracking
Hidden protocol stack
Priority inversion
Server-based User Level Scheduling
Solution:
Modified Dual Priority
Scheduling
04/19/23 13:16 43
Dual Priority Scheduling (York)
Middle Band
Low Band
High BandThree
Priority
Bands Non-real-time tasks
Real-time tasks before promotion time
Real-time tasks after promotion time
04/19/23 13:16 44
Dual Priority in MidART
Middle Band
Low Band
High Band
Time Critical Band
Four
Priority
BandsNon-real-time tasks
Real-time tasks before promotion time
Critical real-time tasks
Real-time tasks after promotion time
04/19/23 13:16 45
Communication Server
ReMA
DPA
ReMA
DPA
ReMA
DPA
High
Promoted
Middle
Low
QueuesRateControllers
DataPusher
To network
@ promotion time
Request = actual messages Queues
limited priorities
Calculation of promotion time
04/19/23 13:16 46
Admission control
To calculate the schedulability T_i:
Promotion Timei = Deadlinei - worst case response timei
worst case response timei = worst case delay (wi) + release jitteri
wi m+1 = Ci + So + [ w
i m + Jj C
j ] j hp(i) Tj
Computation time Blocking time Interference of high priority tasks
04/19/23 13:16 47
Rate Control with Uniform Message Size
Bounding the priority inversion time due to non-preemptive message transmission.
The “optimal” message size is a platform dependent parameter.
Receiving node capability is the key - the known live-lock problem.
Match value between the system timer granularity and the data push latency. (E.g., in our case, 8KB)
04/19/23 13:16 48
Sending Time for Data Pusher
0
2
4
6
8
10
12
14
16
0.5 1 2 4 8 9 10 12 16 24 32 48
Size of ReMA (KB)
Tim
e (
ms
ec
)
1K
8K
Single
04/19/23 13:16 49
Communication Server
ReMA
DPA
ReMA
DPA
ReMA
DPA
High
Promoted
Middle
Low
QueuesRateControllers
DataPusher
To network
@ promotion time
STOP
Uniform message size to allowtransmission with fix preemption points
04/19/23 13:16 50
Communication Server
ReMA
DPA
ReMA
DPA
ReMA
DPA
High
Promoted
Middle
Low
QueuesRateControllers
DataPusher
To network
Uniform message size to allowtransmission with fix preemption points
04/19/23 13:16 51
Problem alleviated:
No priority inheritance
and limited priorities
No priority tracking
Hidden protocol stack
Priority inversion
Server-based User Level Scheduling
Solution:
Modified Dual Priority
Scheduling
Real-time data pusher
Rate control
Uniform message size
Integration of real-time and non-real-time applications on the same computing and communication platform
04/19/23 13:16 52
Industrial Control Environment
EWS
Network
Field Devices
Field Network
PLC
Field Devices
Field Network
PLC
OperatorStations
Periodic Monitoring (64KB @ 10 msec)
Command and control (1KB @ 20 msec)
Server
04/19/23 13:16 53
Data sending period without User level scheduling
0
5
10
15
20
25
30
0 100 200 300 400 500 600 700 800 900
# Sample
Tim
e (
mse
c)
Command and ControlPeriodic Monitoring
04/19/23 13:16 54
Data sending period with User level scheduling
0
5
10
15
20
25
30
0 100 200 300 400 500 600 700 800 900 1000
# Samples
Tim
e (
ms
ec
)
Command and control
Periodic monitoring
04/19/23 13:16 55
Summary: User Level Communication
Communication Scheduler
End-to-End Task Model Local scheduling of precedence constrained distributed
tasks
Integration of real-time and non-real-time tasks
C. Shen, O. González, K. Ramamritham, and I. Mizunuma. User Level Scheduling of Communicating Real-Time Tasks. In Proc. Of fifth IEEE Real-Time Technology and Applications Symposium, Vancouver, Canada, June 1999.
04/19/23 13:16 56
Conclusion
Characterization and understanding of limitations imposed by GPOS.
Development and evaluation of an efficient user-level scheduling scheme for communicating real-time tasks Support coexistence of real-time and non-real-time
using general purpose operating systems and networks
Ongoing work for QoS management support at the middleware level
04/19/23 13:16 57
Incorporation of Quality of Service
QoS management is needed to support multiple real-time
applications on single platform Operating system
Application
Dynamic QoS management. Extend QoS guarantees provided by network to end point
applications
Predictable performance for real-time and multimedia applications
according to user QoS parameters.
04/19/23 13:16 58
System Research to Meet the Challenge
Network ArchitectureNetwork
Interface
Design
Resource
Manager
Scheduling
AlgorithmsScheduler
Middleware
Services
Traffic Management
Rate Control
Qos
Management
04/19/23 13:16 59
Research Questions
What type of support is needed to optimize resource use?
How to implement efficient QoS adaptation at the middleware
level?
The real problem
Integrated end-to-end Quality of Service management
From the mouse to the wire (User to Network)
From the wire to the glass (Network to Display)
04/19/23 13:16 60
Problems
Specification of user QoS requirements
Efficient adaptation mechanisms
Mode change
Integrated scheduling of multiple resources
Resource reservation
Account for interrupt in a GPOS
Support for management of trade-offs between different types of
requirements
dependability vs. performance
Top Related