1. 1. Level 3 Support Team: Nurul Zaman Derek SUPPORT TOOLS & PROCESSES AT ICONNECT360
2. 2. Session Agenda Introduction Incident Management Problem Management Demonstration Conclusion Q & A
3. 3. INTRODUCTION
4. 4. 6 Applications
5. 5. iconnect360 Ezypay Secure Site
6. 6. iconnectHQ iconnect360 Mobile Sales App
7. 7. Online Signup Forms Recovery Payment Forms
8. 8. 8 Countries Australia New Zealand Singapore Malaysia Philippines Hong Kong Taiwan China
9. 9. Release Cycle Monthly Release Deliver Patches Every 2 Weeks Hot Patches
10. 10. How do we do that? Supporting a growing list of products From 1 country to 8 countries within 12 months Processes Exponential Growth in Support Demands
11. 11. Tools for Support Zendesk - Ticketing system for L1, L2, L3, Help Center Bugzilla - Defect Management Tool used by Support, Dev, UX & QA Kibana - Logging tools for Developers & Support staff WordPress - Manage Knowledgebase for sharing - Release Notes
12. 12. Be Inspired! Supporting a Live Plane Cat Herders
13. 13. INCIDENT MANAGEMENT
14. 14. Incident Management Resolving issue within the shortest possible time. Root cause is not the main concern. Get the service back online as soon as possible. Could be permanent or temporary. For example, providing a workaround.
15. 15. Incident Management Situation (Code Blue) - When a system wide fault occurs, it affects all or most of the customers. Resolution of the code blue will become the utmost priority for the support & technical teams. Proper communication & follow-up with multiple clients is required.
16. 16. Incident Management Two key measurement metrics Time To Respond Within the same working day Time for Resolution P1 Defect 2 Working Days Resolution could be a workaround or permanent solution Resolution could be advice or mini training For P2 and P3 defects with patches, customers would be notified with proper follow-up
17. 17. PROBLEM MANAGEMENT
18. 18. Problem Management Proper tracking of issues and defects are important The entire technical team could be involved: Support (Level 2 and Level 3) Developers Business Analysts QA Team Infrastructure Release / DevOps
19. 19. Problem Management Known Error Record (KER) - While the problem is being resolved, a workaround or temp solution would be used to circumvent the issues in production There are 2 types of Problem Management scenarios: Re-Active Problem Management Pro-Active Problem Management
20. 20. Problem Management (Re-active) These are issues that arise from incidents and are directly reported by the customers. Re-Active problems are given a higher priority to be resolved.
21. 21. Problem Management (Pro-active) Issues or problems do not always come from the end-users or customers. Problems could be identified internally Continuous improvement initiatives Communication to the end users
22. 22. Problem Management (Defect Prioritisation) Priority Definition Resolution P1 1. These are defects that affect multiple users. 2. A crucial feature (or the entire system) is not usable and there is no suitable workaround. 1. Need to be analysed and resolved as soon as possible. 2. Typically fixed & deployed to production within 2 working days. P2 1. These are defects that affect only some users. 2. The affected features are not used often or there is an easy workaround available. 1. Will be fixed as part of the monthly release cycle. P3 1. These are minor defects (e.g. UI, wordings). 1. There is no fixed time for resolution for these. 2. Will normally be attended to after P1 and P2 defects are fixed. 3. Increase/decrease priority accordingly.
23. 23. Problem Management (Root Cause Analysis)
24. 24. Problem Management (Root Cause Analysis) Finding the root cause and coming up with a viable solution. Require monitoring of the production system. Collecting and interpreting logs.
25. 25. Problem Management (Root Cause Analysis) Successful RCA: The problem can be eliminated completely. Lowering the risks of re-occurrence. Sometimes, we might have to contact vendors. For examples, Microsoft & Telerik to provide solutions.
26. 26. Problem Management (Root Cause Analysis) Define What is the Problem Determine the Scope and Goal Analyse Analyse the causes Why does it happen Prevent Develop appropriate solution Implement solution
27. 27. Problem Management (5 Whys) WHY ? WHY ? Why ? Why ? Why ? Problem Revelation RespectTrust Learn From the Past Look positively towards the Future Being Defensive Blame Game Disrespectful & Pessimistic
28. 28. Problem Management
29. 29. Problem Management
30. 30. Problem Manager Role Keeps track of the problems in the system and facilitates the resolution. Organises meetings and work with the teams Hosts Root Cause Analysis sessions. Reports to management and stakeholders.
31. 31. Problem Manager Role Aids in finding systematic issues, technical issues and process issues in the product and its supporting structure. Works closely with the Incident Manager on analysing the incident trends.
32. 32. Software Design Strategy for Better Support Proper error logging mechanism. Send out critical alerts when errors are detected. Easy to understand error messages with detailed information (e.g. Error Codes) for a speedier response and troubleshooting.
33. 33. Application Error Message
34. 34. Kibana Logging
35. 35. Out Support Structure Development / Engineering / vendor PM, CM 1. Final point for technical resolution 1. Perform RCA activities Level 3 IM, PM, CM 1. Able to conduct more in-depth technical investigation 1. Perform RCA activities Level 2 - IM 1. Able to handle more technical tasks 1. Database investigation 2. Hardware support & remote support Level 1 (Customer Service / Helpdesk) - IM 1. Initial point of contact 2. Provide solutions to simple and known issues 1. Perform straight forward tasks 2. Give advice and suggestions
36. 36. DEMO Zendesk Ticketing Creating a Defect on Bugzilla
37. 37. After 12 Months People 3 Level 2s 4 Level 3s Proper Prioritisation P1, P2, P3 Suitable Tools & Processes Incident Management Problem Management Zendesk, Bugzilla + + = RESULTS
38. 38. 69 64 103 130 28 13 71 62 0 20 40 60 80 100 120 140 Q3 2014 Q4 2014 Q1 2015 Q2 2015 360 HQ 257 234 646 758 110 131 113 137 0 200 400 600 800 Q3 2014 Q4 2014 Q1 2015 Q2 2015 360 HQ Bugzilla Q by Q Stats Chart Zendesk Q by Q Stats Chart RESULTS
39. 39. Conclusion Keep users happy by providing great support. Come up with a good support process: Incident Management Problem Management Proper Communication Encourage Level 3 Support Team to develop in-house checking tools to improve support.
40. 40. Q&A
41. 41. THANK YOU