Logging service design
32
Logging Service Design For Web Application
-
Upload
wittawas-wisarnkanchana -
Category
Data & Analytics
-
view
204 -
download
1
Transcript of Logging service design
Logging Service DesignFor Web Application
Contents● Logging
● What is Logging?● Web application architecture● Problem● Centralizing● Middleware● Application log● Microservice● Events to log● Technical pitfalls
○ Disk space○ Transport protocols○ Choosing Log Storage
● Processing○ Lambda architecture
What is Logging?When application writing activities or events to somewhere. generally writing in file system.
Web Application Architecture
web server app server database
Web Application Architecture
web server app server database
Log Log Log
Logging Purposes
web server
Log
Monitoring on server status access/error/redirect.Commonly in the form of STDOUT/STDERR
Logging Purposes
app server
Log
Monitor events within application
Logging Purposes
Database
Log
Searching for underlying problems that happened in database level
E.g. Duplicate transactions
Web Application Architecture(cluster)
web server app server database
web server
web server app server
app server
database
database
Web Application Architecture(cluster)
web server app server database
web server
web server app server
app server
database
database
LogLogLog LogLogLog LogLogLog
Problem?Logging seperately in each instances cost extra work searching for problems and lack flexiblity to log other meaningful events.
Centralizing
web server
web server
web server
LogMiddleware
MiddlewareSet of tools that manage underlying work for software
E.g. a middleware to manage HTTP request, redirect assets path.
In this case, we use middleware to interrupt and send log to somewhere else
Centralizing example: NGINX
nginx
nginx
nginx
redisOpenrestyredis2-nginx
Centralizing example: NGINX(2)
nginx
nginx
nginx
ElasticsearchLogstashOpenresty
Application logLog all events in application
Using similar strategy to web server log.By modifying appication middleware and send log to somewhere else
Application log example: NodeJS(Express)
express
express
express
redisCustom
ExpressJSMidddleware
Other middleware to use?Rack in Ruby server
net/http in Go server
Etc. (existed in all popular programming langauge and framework)
Application log(custom)Log other meaningful events
Using different strategy.By modifying appication itself and insert log statement when events actually occur.
imagine console.log(‘other’); in browser
Events to logUser Sign-in
User Register
User select something
User click on something
Referer to that page
Typical everything you could, you wouldn’t know what problem you’ll want to solve in the future.
Application log(custom): ProblemInserting log statement directly in the application will used up computing, I/O resources and obstruct actual flows, may cause discomfort for users.
Microservice: How can it solve the problem?Only for 1 purpose, log processing.
Instead of compute directly, send keys and values to microservice and let it handle in another resources
Why Microservice?Application log need some processing to format log statement
Cause the obstruction
Can be distributed
Decoupled computing resources
Example: User sign-out with log
User sign-out
Sign-outcompleted
● Destroy User session from database● Create log statement object saying User
signing-out at certain time● Format log statement, save only user_id,
format datetime● Save log statement on storage● Redirect to sign-out completed
Example: User sign-out with log on microservice
User sign-out
Sign-outcompleted
● Destroy User session from database● Send log params to service● Redirect to sign-out completed
● Create log statement object saying User signing-out at certain time
● Format log statement, save only user_id, format datetime
● Save log statement on storage
Web Application Architecture (now)
web server app server database
middleware
Log storage (elasticsearch, redis, etc)
middleware middleware
service
Technical pitfall: Transport protocolUDP
TCP
HTTP
HTTPS/SSL
Technical pitfall: Transport protocolWhen sending log parameters to service
Use UDP with non-critical and have massive amount E.g. User clicked at navbar
Use TCP/HTTP connection in crucial informationE.g User go to checkout page
Use HTTPS/SSL when connecting to logging service directly from clients
No need for HTTPS/SSL in between infrastructure
Technical pitfall: Disk spacePrepare counter-measures for disk space management
When running out of disk space
When storage instances are down
When migrate old log statements to new storage
Warning: It used up a lot when in production.
Technical pitfall: Choosing Log StorageFile system is a no-no, consider using tool that can scale easily.
Redis
Elasticsearch
Cassandra
MongoDB
etc.
Where to go from here
Log Storage
Storm
Spark
ServingDatabase
Query from app
Q&A
