Tracking and Monitoring APIs at Scale
-
Upload
cosmin-stanciu -
Category
Data & Analytics
-
view
41 -
download
4
Transcript of Tracking and Monitoring APIs at Scale
![Page 1: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/1.jpg)
Tracking and Monitoring APIs at ScaleBuilding a big-data pipeline with Nginx
Cosmin StanciuLead Software Engineer, Adobe I/O
![Page 2: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/2.jpg)
![Page 3: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/3.jpg)
#nginx #nginxconf
Adobe API’s
Adobe Document Cloud
Adobe Creative Cloud
Adobe Marketing Cloud
![Page 4: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/4.jpg)
API Gateway
![Page 5: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/5.jpg)
Adobe API Gateway
Validation Caching Throttling Logging
![Page 6: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/6.jpg)
• apigateway• api-gateway-request-validation• api-gateway-cachemanager• api-gateway-aws• api-gateway-async-logger
Github
![Page 7: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/7.jpg)
metadata
request
response
600M/day
API Gateway
Kinesis / Kafka
![Page 8: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/8.jpg)
#nginx #nginxconf
Data
Analytics
Debugging
Business model
Monitoring
![Page 9: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/9.jpg)
![Page 10: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/10.jpg)
![Page 11: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/11.jpg)
#nginx #nginxconf
– Carl Sagan
If you wish to make an apple pie from scratch, you must first invent the universe.
![Page 12: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/12.jpg)
architecture
![Page 13: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/13.jpg)
architectureBatch
ServiceSpeed
HDFS SQLStreaming
{API}
Agg. Index
Speed IndexConsumers
Kinesis
S3
![Page 14: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/14.jpg)
- Streaming layer
Kinesis
BatchService
Speed
HDFS SQLStreaming
{API}
Agg. Index
Speed IndexConsumers
S3
![Page 15: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/15.jpg)
Kinesis Logger Config1. local logger_module = “api-gateway.logger.BufferedAsyncLogger"
3. local logger_opts = {4. flush_length = 500,5. flush_interval = 5,6. flush_concurrency = 16,7. flush_throughput = 10000, 8. sharedDict = “stats_kinesis”9. }
PutRecords - ever request up to 500 records
5s - interval in seconds to flush regardless if the buffer is full or
notMax parallel
threads used for sending logs
max logs / SECOND that can be sent to
the Kinesis backend
dict for caching the logs
![Page 16: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/16.jpg)
Kinesis Logger Config1. backend_opts = {2. aws_region = ngx.var.aws_region or "us-east-1",3. kinesis_stream_name = "api-gateway-stream",4. aws_credentials = {5. provider = "api-gateway.aws.AWSSTSCredentials",6. role_ARN = "arn:aws:iam::123456789012:user/admin,7. role_session_name = "kinesis-logger-session",8. shared_cache_dict = “aws_credentials”9. }10. }
Security Token Service
AsumeRole returns a set of temporary
security credentials
![Page 17: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/17.jpg)
Cluster
![Page 18: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/18.jpg)
Cluster deployment
VPCPublicPrivate
Stateless Body
Stateful BodyNucleus Membrane
Mesos Agents HDFS
NameNodeZookeeper
Mesos master
Stateless Workloads
Stateful Workloads
- HDFS DataNodes
API Gateway- Auto discover
- Load balancing- Security
![Page 19: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/19.jpg)
SpeedSpeed IndexConsumers
Kinesis
BatchService
HDFS SQLStreaming
{API}
Agg. Index
S3
Real-time data
Kibana UIDebugging
Monitoring
- Speed layer
![Page 20: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/20.jpg)
- Batch layer Batch
HDFS SQLStreamingAgg. Index
ServiceSpeed
{API}
Speed IndexConsumers
Kinesis
S3
Docker / Marathon Batch size
Checkpointing Kinesis / Spark
Store in Parquet format
Temporary storage Parquet files
S3 sync
Elasticsearch Index
Docker / ChronosDaily / Hourly aggregation
Run job hourly or daily
![Page 21: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/21.jpg)
#nginx #nginxconf
OLAP Data Cube
count
time
consumer
serviceThe Elasticsearch aggregated index can be represented as a Data Cube
The cube is actually a hypercube with more than 3 dimensions
Users can apply filters, roll-ups or drill-downs
![Page 22: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/22.jpg)
Service{API}
Agg. Index
Speed Index
S3 BatchSpeed
HDFS SQLStreaming
Consumers
Kinesis
- Service layer
![Page 23: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/23.jpg)
Testing
![Page 24: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/24.jpg)
Performance testing
![Page 25: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/25.jpg)
Functional testing
HDFS SQLStreaming
Kafka
Testing app
![Page 26: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/26.jpg)
Canary testing
{API}
Agg. Index Speed Index
Canary traffic
![Page 27: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/27.jpg)
Results
![Page 28: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/28.jpg)
![Page 29: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/29.jpg)
Results
![Page 30: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/30.jpg)
Monitoring
14 15 16 17 18 19 200
1750
3500
5250
7000
8750
Realtime Forcast
![Page 31: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/31.jpg)
Demo
![Page 32: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/32.jpg)
Functional testing
HDFS SQLStreaming
Kafka
Testing app
![Page 33: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/33.jpg)
Kafka Logger Config1. local logger_module = “api-gateway.logger.BufferedAsyncLogger"
3. local logger_opts = {4. flush_length = 500,5. flush_interval = 1,6. flush_concurrency = 16,7. flush_throughput = 10000, 8. sharedDict = “stats_kafka”9. }
![Page 34: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/34.jpg)
![Page 35: Tracking and Monitoring APIs at Scale](https://reader036.fdocuments.net/reader036/viewer/2022062412/587080f11a28ab57368b65ed/html5/thumbnails/35.jpg)
#nginx #nginxconf
Thank Youfacebook.com/selfxp
linkedin.com/in/cosminstanciu
@selfxp80