Post on 21-Jan-2018
© 2016 Cisco and/or its affiliates. All rights reserved. 1
CiscoConnect
Model-Driven TelemetryViktor OsipchukTechnical Marketing Engineer
12 October 2017
© 2016 Cisco and/or its affiliates. All rights reserved. 2
Be sure to make right measurements before implementing automation @scale
What Automation Without Visibility Looks Like
© 2016 Cisco and/or its affiliates. All rights reserved. 3
Source: Google @ Bay Area OpenDaylight Meetup 06/16
© 2016 Cisco and/or its affiliates. All rights reserved. 4
Telemetry as the basis for networks @scale
https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45687.pdf
5© 2016 Cisco and/or its affiliates. All rights reserved.
Current challenges
© 2016 Cisco and/or its affiliates. All rights reserved. 6
syslog
SNMP
CLI
Too Slow
Incomplete
Network-Specific
Hard to Operationalize
Why Network Visibility Is Hard
© 2016 Cisco and/or its affiliates. All rights reserved. 7
What Happens When You Push SNMP Too Hard
• 10 second poll / push• 3 pollers / telemetry receivers• 30 minute measurement intervals
• 288 100Gig E Interfaces (Line Rate)• SNMP: IF-MIB (query by row)
8© 2016 Cisco and/or its affiliates. All rights reserved.
Telemetry fundamentals
© 2016 Cisco and/or its affiliates. All rights reserved. 9
Three Enablers for Telemetry
Push Not Pull
Analytics-Ready Data
Data-Model Driven
© 2016 Cisco and/or its affiliates. All rights reserved. 10
Push Beats SNMP Pull for Speed & Scale
• 10 second poll / push• 3 pollers / telemetry receivers• 30 minute measurement intervals
• 288 100Gig E Interfaces (Line Rate)• SNMP: IF-MIB (query by row)
© 2016 Cisco and/or its affiliates. All rights reserved. 11
“OSI model” of Telemetry
Data store
Data model
Producer
Exporter
Collector
Native (raw) data inside a router’s database
Raw data mapped to a model (YANG native, OpenConfig, etc)
Sending requested data in model format to the “Exporter” at defined intervals
Encoding and delivering data to the collector(s) destination(s)
Information collection for processing (e.g., data monitoring, automation, analytics)
“Data”“Layer”Te
lem
etry
end
-to-e
nd
© 2016 Cisco and/or its affiliates. All rights reserved. 12
YANG Is A Modeling Language
module ietf-interfaces {import ietf-yang-types {
prefix yang; }container interfaces {
list interface { key "name"; leaf name {
type string;}leaf enabled {
type boolean; default "true";
}…
Edited for
Brevity
Self-contained top-level hierarchy of nodes
Import or define data types
Leaf nodes for simple data
Lists for sequence of entries
Containers group related nodes
Other YANG Features
• RO or RW • Optional nodes• Choice• Augment
• When• Arbitrary XML• RPC• etc
© 2016 Cisco and/or its affiliates. All rights reserved. 13
Models are Available in Github
You Should Do This
• Telemetry only cares about operational (*-oper.yang) models.
• 143 oper YANG models published for XR 6.1.1• 151 oper YANG are for XR 6.1.2• 177 oper YANG are for XR 6.2.1• 180 oper YANG for XR 6.2.2• 198 oper YANG for XR 6.3.1https://github.com/YangModels/yang/tree/master/vendor/cisco/xr
© 2016 Cisco and/or its affiliates. All rights reserved. 14
“OSI model” of Telemetry
Data store
Data model
Producer
Exporter
Collector
Native (raw) data inside a router’s database
Raw data mapped to a model (YANG native, OpenConfig, etc)
Sending requested data in model format to the “Exporter” at defined intervals
Encoding and delivering data to the collector(s) destination(s)
Information collection for processing (e.d., data monitoring, automation, analytics)
“Data”“Layer”Te
lem
etry
end
-to-e
nd
© 2016 Cisco and/or its affiliates. All rights reserved. 15
Finding the Data You Want To Stream
$ pyang -f tree Cisco-IOS-XR-infra-statsd-oper.yang--tree-path infra-statistics/interfaces/interface/latest/generic-counters
telemetry model-driven
sensor-group SGROUP1
sensor-path Cisco-IOS-XR-infra-statsd-oper:infra-
statistics/interfaces/interface/latest/generic-counters
© 2016 Cisco and/or its affiliates. All rights reserved. 16
What Will Be Pushed With That Config
{"Timestamp": 1480547974706,"Keys": {
"interface-name": "MgmtEth0/RP0/CPU0/0"},"Content": {
"applique": 0,"availability-flag": 0,"broadcast-packets-received": 25035,"broadcast-packets-sent": 0,"bytes-received": 165321050,"bytes-sent": 233917498,"carrier-transitions": 3,"crc-errors": 0,"framing-errors-received": 0,"giant-packets-received": 0,"input-aborts": 0,"input-drops": 62,"input-errors": 0,"input-ignored-packets": 0,"input-overruns": 0,"input-queue-drops": 0,
"last-data-time": 1480547974,"last-discontinuity-time": 1479244159,"multicast-packets-received": 457,"multicast-packets-sent": 0,"output-buffer-failures": 0,"output-buffers-swapped-out": 0,"output-drops": 0,"output-errors": 104,"output-queue-drops": 0,"output-underruns": 0,"packets-received": 373156,"packets-sent": 311583,"parity-packets-received": 0,"resets": 0,"runt-packets-received": 0,"seconds-since-last-clear-counters": 0,"seconds-since-packet-received": 0,"seconds-since-packet-sent": 0,"throttled-packets-received": 0,"unknown-protocol-packets-received": 0
}
Repeated for all interfaces
© 2016 Cisco and/or its affiliates. All rights reserved. 17
“OSI model” of Telemetry
Data store
Data model
Producer
Exporter
Collector
Native (raw) data inside a router’s database
Raw data mapped to a model (YANG native, OpenConfig, etc)
Sending requested data in model format to the “Exporter” at defined intervals
Encoding and delivering data to the collector(s) destination(s)
Information collection for processing (e.d., data monitoring, automation, analytics)
“Data”“Layer”Te
lem
etry
end
-to-e
nd
© 2016 Cisco and/or its affiliates. All rights reserved. 18
Configuring Destination
telemetry model-driven
destination-group DGROUP
address family ipv4 192.168.1.1 port 2104
---- and/or ----
address family ipv6 2001:db8::1 port 2104
encoding self-describing-gpb
protocol tcp
Specify where you want to send your data
Specify how you want your data to look like
Specify how you want your data to be delivered
© 2016 Cisco and/or its affiliates. All rights reserved. 19
Basic Concept: Encoding
Encoding (or “serialization”) translates data (objects, state) into a format that can be transmitted across the network. When the receiver decodes (“de-serializes”) the data, it has an semantically identical copy of the original data.
DATA
DATA
“Decode”
“Encode”
IOS XR platforms Encodings
• Compact GPB• Key-Value GPB• JSON (6.3.1)
© 2016 Cisco and/or its affiliates. All rights reserved. 20
GPB Encoding
Design Goals
• Simplicity• Performance• Forward/Backward
Compatibility
Non-Goals
• Human-Readable• Self-Describing• Text-based
Google Protocol Buffers (GPB)
Call them “protobufs”
for short
“Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler.”
© 2016 Cisco and/or its affiliates. All rights reserved. 21
Telemetry Has Two GPB Encoding Options
data_gpb {row {
timestamp: 1485794640469keys: "\n\026GigabitEthernet0/0/0/0"content:
"\220\003\010\230\003\001\240\003\002\250\003\000\260\003\000\270\003\000\300\003\000\310\003\000\320\003\300\204=\330\003\000\340\003\000\350\003\000\360\003\377\001"}
2X fasterOperationally more complex (but not relative to SNMP!)
data_gpbkv {timestamp: 1485793813389fields {name: "keys"fields { name: "interface-name" string_value:
"GigabitEthernet0/0/0/0" }}fields {name: "content"fields { name: "input-data-rate" uint64_value: 8 }fields { name: "input-packet-rate" uint64_value: 1 }
<<< 9 lines are skipped >>> fields { name: "input-load" uint32_value: 0 }fields { name: "reliability" uint32_value: 255 }
}}...
3X largerNative models: still need heuristics for key names
GPB – “compact” GPB – “self-describing”
© 2016 Cisco and/or its affiliates. All rights reserved. 22
Dial-Out
• TCP & gRPC (from 6.1.1)• UDP (from 6.2.1)
Dial-In
• gRPC only (from 6.1.1)
Transport Options
Collector
Data
SYNSYN-ACK
ACK Collector
Data
SYNSYN-ACK
ACK
© 2016 Cisco and/or its affiliates. All rights reserved. 23
gRPC: Like REST But Different
Runs over HTTP/2Optimize for page load timeServer push, header compression, multiplexing, TLSRFC 7540 (May 2015)Preserves most HTTP1.1 syntax
Defines Services (“RPCs”) Encodes Using Google Protocol Buffers (“GPB” or “protobufs”)
Services and MessagesAuto-generate code in many languages
http://www.grpc.io/docs/#hello-grpc
© 2016 Cisco and/or its affiliates. All rights reserved. 24
“OSI model” of Telemetry
Data store
Data model
Producer
Exporter
Collector
Native (raw) data inside a router’s database
Raw data mapped to a model (YANG native, OpenConfig, etc)
Sending requested data in model format to the “Exporter” at defined intervals
Encoding and delivering data to the collector(s) destination(s)
Information collection for processing (e.d., data monitoring, automation, analytics)
“Data”“Layer”Te
lem
etry
end
-to-e
nd
© 2016 Cisco and/or its affiliates. All rights reserved. 25
A Telemetry Subscription
telemetry model-driven
subscription Sub1
sensor-group-id SGROUP1 sample-interval 30000
destination-id DGROUP1
*Omit Destination Group For gRPC Dial-In
26© 2016 Cisco and/or its affiliates. All rights reserved.
Telemetry on Cisco products
© 2016 Cisco and/or its affiliates. All rights reserved. 27
Cisco XR Telemetry overview
Classic XR ASR9k Evolved XR ASR9k NCS5500 NCS6kMDT support 6.1.1 6.1.1 6.1.1 6.1.3
Data modelsYANG
(native, OC, IETF) Link for models
YANG (native, OC, IETF)
Link for models
YANG (native, OC, IETF)
Link for models
YANG (native, OC, IETF)
Link for models
Transport(Control
protocols)
TCP (dial-out), UDP (dial-out)*
gRPC (dial-in, dial-out), TCP (dial-out), UDP (dial-out)*
gRPC (dial-in, dial-out), TCP (dial-out), UDP
(dial-out)*
TCP (dial-out), UDP (dial-out)*
Encoding GPB / GPB-KV / JSON** GPB / GPB-KV / JSON** GPB /
GPB-KV / JSON**GPB /
GPB-KV / JSON**
Collectors Pipeline*** Pipeline*** Pipeline*** Pipeline***
* UDP support from 6.2.1** JSON support from 6.3.1*** Open-sourced and ready to use: https://github.com/cisco/bigmuddy-network-telemetry-pipeline
© 2016 Cisco and/or its affiliates. All rights reserved. 28
NX OS IOS-XE
MDT support 7.0(3)I6(1) 16.6.1*
Data models Data Management Engine, NX-API, YANG (native, OC, IETF)
YANG (native**, IETF) Link for models
Transport(Control protocols)
gRPC* (dial-out), UDP** (dial-out), HTTP***(dial-out)
Netconf (for YANG), GNMI(16.8.1), gRPC (16.9.1)
Formats GPB/JSON XML, GPB (16.9.1)
Collectors Pipeline TBD
Min sample interval 5 sec 1 sec
Max # of dial-out destinations 5 TBD
Cisco NXOS/XE Telemetry high-level overview
* gRPC supports GPB only * supported on Catalyst 3650/3850/9300/9500, ASR1000, ISR4000** UDP from 7.0(3)I7(1), supports both, GPB and JSON ** Native models are different from YANG models in XR*** HTTP suppors JSON only
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/programmability/guide/b_Cisco_Nexus_9000_Series_NX-OS_Programmability_Guide_7x/b_Cisco_Nexus_9000_Series_NX-OS_Programmability_Guide_7x_chapter_011000.html
https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/prog/configuration/166/b_166_programmability_cg/model_driven_telemetry.html
© 2016 Cisco and/or its affiliates. All rights reserved. 29
Event-Driven Telemetry overview
Model-Driven Telemetry
100 interfaces UP / 0 interfaces DOWN
100 interfaces UP / 0 interfaces DOWN
100 interfaces UP / 0 interfaces DOWN
99 interfaces UP / 1 interfaces DOWN
99 interfaces UP / 1 interfaces DOWN
99 interfaces UP / 1 interfaces DOWN
Event-Driven Telemetry
Router X
100 interfaces UP / 0 interfaces DOWN
99 interfaces UP / 1 interfaces DOWN
Router X
Tim
e
Tim
e
© 2016 Cisco and/or its affiliates. All rights reserved. 30
Event-Driven Telemetry configurationA configuration example:
telemetry model-drivensensor-group interface
sensor-path Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface!subscription interface
sensor-group-id interface sample-interval 0 ## ”0” means EDT!
Please, go ahead
And tell us what you want to have!
IOS XR 6.3.1 includes support for:
1. Interface changes• Sensor path: Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface• Sensor path: Cisco-IOS-XR-ipv6-ma-oper:ipv6-network/nodes/node/interface-data/vrfs/vrf/global-briefs/global-brief (*)
2. RIB changes• Sensor path: Cisco-IOS-XR-ip-rib-ipv4-oper:rib/vrfs/vrf/afs/af/safs/saf/ip-rib-route-table-names/ip-rib-route-table-name/routes/route• Sensor path: Cisco-IOS-XR-ip-rib-ipv6-oper:ipv6-rib/vrfs/vrf/afs/af/safs/saf/ip-rib-route-table-names/ip-rib-route-table-
name/routes/route3. Syslog updates
• Sensor path: Cisco-IOS-XR-infra-syslog-oper:syslog/messages/message
More to come soon !
© 2016 Cisco and/or its affiliates. All rights reserved. 31
Some Commonly Used Models and SubtreesData Model
Interface Oper State sensor-path Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interfaceInterface Data Rate sensor-path Cisco-IOS-XR-infra-statsd-oper:infra-statistics/interfaces/interface/latest/data-rateInterfaces Stats sensor-path Cisco-IOS-XR-infra-statsd-oper:infra-statistics/interfaces/interface/latest/generic-countersOptics Ports Info sensor-path Cisco-IOS-XR-controller-optics-oper:optics-oper/optics-ports/optics-port/optics-InfoUptime Info sensor-path Cisco-IOS-XR-shellutil-oper:system-time/uptimeCPU State sensor-path Cisco-IOS-XR-wdsysmon-fd-oper:system-monitoring/cpu-utilizationMemory Info sensor-path Cisco-IOS-XR-nto-misc-oper:memory-summary/nodes/node/summaryProcesses Memory sensor-path Cisco-IOS-XR-procmem-oper:processes-memory/nodesLLDP Info sensor-path Cisco-IOS-XR-ethernet-lldp-oper:lldp/nodes/node/neighbors/summaries/summary
IPv4 RIB Infosensor-path Cisco-IOS-XR-ip-rib-ipv4-oper:rib/vrfs/vrf/afs/af/safs/saf/ip-rib-route-table-names/ip-rib-route-table-name/routes/route
IPv6 RIB Infosensor-path Cisco-IOS-XR-ip-rib-ipv6-oper:ipv6-rib/vrfs/vrf/afs/af/safs/saf/ip-rib-route-table-names/ip-rib-route-table-name/routes/route
BGP IPv4 Routes Infosensor-path Cisco-IOS-XR-ip-rib-ipv4-oper:rib/vrfs/vrf/afs/af/safs/saf/ip-rib-route-table-names/ip-rib-route-table-name/protocol/bgp/as/information
BGP IPv6 Routes Infosensor-path Cisco-IOS-XR-ip-rib-ipv6-oper:ipv6-rib/vrfs/vrf/afs/af/safs/saf/ip-rib-route-table-names/ip-rib-route-table-name/protocol/bgp/as/information
BGP ipv4 Neighbor Info sensor-path Cisco-IOS-XR-ipv4-bgp-oper:bgp/instances/instance/instance-active/default-vrf/neighbors/neighborMPLS-TE Tunnels Summary Info sensor-path Cisco-IOS-XR-mpls-te-oper:mpls-te/tunnels/summaryRSVP Interface Info sensor-path Cisco-IOS-XR-ip-rsvp-oper:rsvp/interface-briefs/interface-briefNCS5500 NPU Stats sensor-path Cisco-IOS-XR-fretta-bcm-dpa-npu-stats-oper:dpa/stats/nodes/nodeNCS5500 NPU Resources Info sensor-path Cisco-IOS-XR-fretta-bcm-dpa-hw-resources-oper:dpa/stats/nodes/node/hw-resources-datas/hw-resources-data
© 2016 Cisco and/or its affiliates. All rights reserved. 32
Your infrastructure should be readyData NCS5516
numbersgRPC/KVGPBbytes*
gRPC/KVGPBBW (mbps)
gRPC/GPBbytes*
gRPC/GPBBW (mbps)
UDP/JSONbytes*
UDP/JSONBW (mbps)
Interface Oper State* 2160 5105157 0.6806876 1089581 0.145277467 5774492 0.769932267Interface Data Rate 2160 795737 0.3182948 188765 0.075506 1003445 0.401378
Interfaces Stats 2160 2144454 0.8577816 377912 0.1511648 2520016 1.0080064Optics Ports Info 576 5369650 0.477302222 740693 0.065839378 5438332 0.483407289
Uptime Info 1 176 0.0002816 143 0.0002288 373 0.0005968CPU State 18 419895 0.335916 66315 0.053052 380407 0.3043256Memory Info 18 5045 0.004036 1448 0.0011584 6984 0.0055872Processes Memory 538 691910 0.061503111 222950 0.019817778 937089 0.0832968LLDP Info 574 231640 0.185312 87911 0.0703288 300228 0.2401824IPv4 RIB Info* 650,022 1065910551 71.0607034 246565671 16.4377114 1279423829 85.29492193IPv6 RIB Info* 14876 27311283 3.6415044 6427000 0.856933333 32784283 4.371237733BGP IPv4 Routes Info 650,000 593 0.000158333 303 0.0080466667 790 0.0002188BGP IPv6 Routes Info 12800 590 0.000157333 300 0.00008 784 0.000209067BGP ipv4 Neighbor Info 2 48784 0.0390272 6841 0.0054728 48904 0.0391232MPLS-TE Tunnels Summary Info 1003 5755 0.004604 704 0.0005632 5865 0.004692RSVP Interface Info 5 1919 0.0015352 532 0.0004256 2465 0.001972NCS5500 NPU Stats 96 119816982 1.59755976 19158017 0.255440227 129493262 1.726576827NCS5500 NPU Resources Info 96 256502 0.2052016 47352 0.0378816 266352 0.2130816
TOTAL Bandwidth ~ 80 mbps ~ 18 mbps ~ 95 mbps
Bytes are per a single collection* Go with EDT starting from IOS XR 6.3.x
33© 2016 Cisco and/or its affiliates. All rights reserved.
Collectors and analytics platforms
© 2016 Cisco and/or its affiliates. All rights reserved. 34
Kafka
Different Collection Models
Logstash
ElasticSearch
KibanaPanda
BYO
Custom Open Source, Customizable
Proprietaryor OS-based
Commercial Stack
Prometheus /InfluxDB
Grafana
Applications
Storage
Collection
© 2016 Cisco and/or its affiliates. All rights reserved. 35
Pipeline: An Open Source Collector
Pipeline
Kapacitor
Output to file, TSBD, Kafka…Ingest, transform, filterSelf-monitoring, horizontally scalable
© 2016 Cisco and/or its affiliates. All rights reserved. 36
TSDB basic comparisonData InfluxDB Prometheus Elasticsearch
Website https://influxdata.com/ https://prometheus.io/ https://www.elastic.co/products/elasticsearchCategory Real-time Analytics Monitoring System Real-time SearchSupported Measurements metrics, events metrics metrics, events
Sharding and Replication Manual Manual (supports federation) AutomaticHigh Availability (HA) Double writing 2 servers Double writing 2 servers ClusteringUnderlying Technology Golang Golang JavaOperational Complexity Low (medium with HA) Low MediumStorage Backend Custom Custom DocumentSupported Data Types int64, float64, bool, and string float64 string, int32, int64, float32, float64, bool, nullBytes per point after compression 2.2 1.3 22Metric Precision nano second milli second milli secondRecording type events fixed interval eventsWrite Performance - Single Node 470k metrics / sec (custom HW) 800k metrics / sec 30k metrics / secWrite Performance - 5 Node Cluster - - 120k metrics / sec (calculated)Query Performance (1 host, 12hr by 1m) 3.78 ms (min), 8.17 (avg) tbd 13.23 ms (min), 28.6 (avg)Query Performance Medium to Fast Moderate ModerateQuery Language InfluxQL (SQL like) PromQL Query DSLData Model metric names, fields, labels metric names, labels metric names, labelsQuery Language Functionality 4/5 5/5 3/5Query Language Usability 5/5 4/5 3/5Community Size large large largeLicense MIT Apache 2 Apache 2Latest Version v1.3.5 v2.0.0-beta.2 v5Maturity Stable Stable Stable
Full table: https://tinyurl.com/jsd4esyGood to read: https://tinyurl.com/ybaw4ww6
DB ranking: https://tinyurl.com/ya8rrrjpInfluxDB vs Elasticsearch: https://tinyurl.com/y7yxjf6v
InfluxDB vs OpenTSDB: https://tinyurl.com/y8ofbjyyInfluxDB vs Cassandra: https://tinyurl.com/y83vv9ys
© 2016 Cisco and/or its affiliates. All rights reserved. 37
TSDB: systems requirements
Pipeline
Grafana uses TSDB for the work, server requirements are small (250MB of RAM, 1CPU – the answer from the developer: https://tinyurl.com/ybdhexqu)
Prometheus doesn’t give exact requirements, but in many discussions shows 64GB of RAM and and 32 CPU: https://tinyurl.com/yd7l38ed)
Pipeline is a lightweight program, written in Go (with concurrency). You can also start several instances [on a single server] if needed.
InfluxDB requirements:https://tinyurl.com/yd9f3ntj
Server CPU,cores
RAM, GB IOPS
Low load [5k writes, 100k series, 5 queries per second] 2-4 2-4 500
Moderate load [250k writes, 1m series, 25 queries per second] 4-6 8-32 500-1000
High load [>250k writes, >1m series, >25 queries per second] 8+ 32+ 1000+
© 2016 Cisco and/or its affiliates. All rights reserved. 38
Real life deployments
https://promcon.io/2017-munich/slides/monitoring-cloudflares-planet-scale-edge-network-with-prometheus.pdf
© 2016 Cisco and/or its affiliates. All rights reserved. 39
MDT in real time
Demo
© 2016 Cisco and/or its affiliates. All rights reserved. 40
ASR 9006
Demo topology
BE 14 BE 13
BE 26BE 56
BE 15BE 46
BE 36 BE 12
BE 16
NCS 5502
NCS 5011 NCS 5501
NCS 5501 NCS 5502
Pipeline
Kapacitor
100GE link10GE link
Telemetry stream
© 2016 Cisco and/or its affiliates. All rights reserved. 41
Key Takeaways
• Speed & Scale Require Visibility• It’s Not Hard to Beat SNMP• Data Models Are Your Friends• A Big Data Platform Is In Your Future
© 2016 Cisco and/or its affiliates. All rights reserved. 42
ResourcesTutorials, Blogs, VoDs
https://xrdocs.github.io/telemetry/http://blogs.cisco.com/sp/the-limits-of-snmphttp://blogs.cisco.com/sp/boring-is-the-new-awesomehttp://blogs.cisco.com/sp/why-you-should-care-about-model-driven-telemetryhttps://youtu.be/tIN8BjHwpNs (NANOG 67: 10 Lessons from Telemetry)
YANGhttps://github.com/YangModels/yang/tree/master/vendor/cisco (Cisco YANG models)http://blogs.cisco.com/getyourbuildon/yang-opensource-tools-for-data-modeling-driven-management(YANG open source tools)https://developer.cisco.com/site/ydk/ (YDK intro)
Telemetry Tools :https://github.com/cisco/bigmuddy-network-telemetry-pipelinehttps://github.com/cisco/bigmuddy-network-telemetry-stacks
Demos and Labhttps://dcloud-cms.cisco.com/demo/mdt-ios-xr-611-v1 (dCloud)https://youtu.be/F_S9-ctNFe0 (demo on NCS 5508)
Thank you.