20111006 synergie informatique-label-qualityof-experience_en
-
Upload
synergie-informatique -
Category
Technology
-
view
397 -
download
1
description
Transcript of 20111006 synergie informatique-label-qualityof-experience_en
Quality of Experience
Performance Benchmark on Messaging solutions Microsoft Office 365, IBM LotusLive and Google Apps
Webinar October 6 th 2011
Alexis Blandin, Cloud & Social Business Leader | Synergie Informatique
Arnaud Bécart, International Presales Engineer| ip-label
David Wais, Business Development Leader| ip-label
Synergie Informatique
Cloud Services Provider, France
Social Business Specialist
Expert in messaging and collaboration solutions integration
IBM, Microsoft and Google partner
Around the world, ip-label.newtest solutions measure the quality of applications from the user’s point of view: business applications as well as web, voice and even TV over IP platforms. Over 1500 direct and indirect customers rely on ip-label.newtest to validate the end-to-end reliability of their websites and critical applications worldwide
ip-label.newtest
Cloud goes Mainstream
Innovation = Public versus Enterprise Social Network Mobility, highly connected, convergence Increasing Data SaaS, a serious consideration
“ Cloud Computing Market Will Top $241 Billion in 2020 ” – Forrester Research “10% of Enterprise Email Will Be Based on a Cloud/SaaS Model in 2014” – Gartner
Messaging in the line of sight
Cloud Usage Scenarios
1. Prototyping
2. Test & Development
3. Messaging
4. Collaboration
5. Web Applications
6. Backup, Archiving
CIO Expected Advantages Commodities delegation Industrialisation Fast deployment Savings, ROI Self-service, flexibility / PAYG Innovation (« beta »)
Standardization, integration New challenges (mobility, …)
The CIO Evolution
Guarantees, disponibility (SLA)
Warranty and « Quality » improvement
Security, externalisation, reversibility
Sales relationships (B2B)
Contracting and legal management, (Patriot Act, CNIL)
Going from CAPEX to OPEX (subscription)
Operational and organisational restrains
Adapted Needs
Governance, « Cloud » Best Practices
Adapted procedures flowchart
More straight to the point (support, specifics, infrastructures)
Dependant to the service/ network
User experience
Metrology Tools
Editor’s Focus / Collaborative Suite
Identity, Strategy
1911 Services / Softwares
Social Business
1975 Windows / Office
« All In »
1998 Search
Pure Player
Product LotusLive Office 365 Google Apps For Business
Acquisition By modules
Yes Progressive (options) No
€/U/Y (public prices)
34 to 113 € 43 to 306 € 40 €
Advantages • Social and Collaborative
Network • External Opening
• Pack Office Integrated • Sharepoint Framework
• Web Experience • Google ecosystem
Weaknesses • Some missing features
(online office suite, blogs/wikis, call audio/video)
• SharePoint Out-of-the-Box (ergonomy, features)
• Document-Centric
• No web conference • Very basic integrated social and
collaborative tools
E-Mail Systems Focus
Product • LotusLive iNotes/Notes • Exchange Online • GMail
Features
• Classic codes
• Native Lotus Notes included
• Classic codes • Native Outlook integrated
• Codes different than a classic messaging suite
• GMail offline (Google Chrome)
Technical • Two offers: basic(web mail) +
advanced (Domino) • Hybrid Mode
• Hybride Mode • Full Web
Analysis, trend
• Lack of visibility, can look strong enough just for Domino clients
• Domino/LotusLive future vision available
• Leader On-Premise • Late Coming-Out Cloud
• Difficult move from BPOS to Office 365 : which
consequeces ?
• In position of conquest • Big efforts made in some
points (sécurity / administration)
Space 25 Go 25 Go 25 Go
SLA, Credit according to the available
%
99,9 % 2 à 10% monthly cost
99,9 % 25 à 100% monthly cost
99,9 % 3 to 15 days
“ Inbox = Tetris getting zero lines doesn’t win it just makes the next move easy “
Methodology Data Solution Pilot for all web applications , rich web or client (
Office 365, Outlook…) run like a user: mouse click, filling application forms, text control, click on image etc.
Fast launch on one of the 70 measuring points in the world (5 operators are tested in France, Orange, Neuf-SFR, Iliad-Free, Colt, Cogent)
Mode « Saas » on demand, from set up tests to operational robot and servers maintenance
Access to all measures via an extranet, reports sent via mail and warnings on incidents
Access to back up for all set up , expertise and training requests
Scenari recording tools for clients wanted to be autonomous
Solution Architecture
Tests Description 3 platforms tested
3 types of tests
– Access to the portal from 5 spots in France • https://portal.microsoftonline.com • https://apps.lotuslive.com • https://www.google.com/a/synergie-informatique.fr/ • Tests origins: France Telecom, Neuf-SFR, Iliad-Free, Colt, Cogent • Tests frequency: 15 minutes • Tests duration: 21/07/11 au 30/09/11 (71 jours)
– Scenario SENDING from 1 spot in France • Sending a mail from the Cloud platform to a external messging address • Delivery mail authentication in less than 60 seconds • Tests frequency: 30 minutes • Tests duration: 1/09/11 au 30/09/11 (30 jours)
– Scenario RECEIVING from 1 spot inFrance • Sending a mail from an external messaging address rto the Cloud platform • Receiving mail authentication in less than 60 seconds • Tests frequency: 30 minutes • Tests duration: 1/09/11 au 30/09/11 (30 jours)
Tests Description Scenario SENDING
– Connexion to the portal
– Authentication
– Access to the e-mail
– New online message to [email protected]
– Send the message and wait for POP from ip-label (mail.ip-label.com) – waiting time max 60 seconds
Scénario RECEIVING – Connexion to the portal
– Authentication
– Access to the e-mail
– Send a mail via SMTP from a server from ip-label (mail.ip-label.com)
– Wait for mail receipt on the tested platform – waiting time max 60 seconds
Each sent mail is automatically deleted after each test
Each portal account uses a distinctive external messaging account
Tests Description When the portal we are trying to access is considered unavailable?
– Non loading of the portal in less than 60 seconds
– Network incidents detected ( DNS faulty resolution , failed connexion) and server(loading slowness)
When a scenario is considered unavailable ?
– Non complete loading of a page in less than 30 seconds linked to network or server incidents
– Keywords not found
• Username to validate the loading of the portal
• Username/Password, connexion button, then button ‘new mail to validate the authentication step
• SEND : button new mail and send
• RECEIVING : refresh button to verify the mail had been received
– Non transmission of the e-mail ( external SMTP online messaging or online messaging SMTP) in less than 60 seconds
Portal Access Availibility and average access time to each portal
Results from 21/07/11 to 30/09/11 – Around 34000 tests per portal
– Google has the best portal loading time
– Lotus Live is quite fast (1,5 seconds) but the page is longer to start (600ms), with longer DNS resolution, connexion and server time
– Office 365 is the longest to load it is also the heaviest (1 Mo, versus 17 ko for google and 32 ko for Lotus Live)
– Lotus Live shows the best availibility with 7 failures, Google (8 échecs) and Office 365 (11 failures)
1,097 seconds 99.977 %
3,552 seconds 99.968 %
1,541 seconds 99.980 %
0 500 1000 1500 2000 2500 3000 3500 4000
Office 365
lotus live
google appsRésolution DNS
Temps de connexion
Temps de réponse du serveur
Chargement du contenu
Portal Access Evolution over a 3 months period
3474 3750
3432
1286 1357
1982
1119 1113 1061
0
500
1000
1500
2000
2500
3000
3500
4000
Juillet Aout Septembre
Office 365
lotus live
google apps
99
,98
99
,83
10
0
99
,98
99
,98
99
,98
99
,98
99
,96
99
,99
99,7
99,75
99,8
99,85
99,9
99,95
100
Juillet Aout Septembre
Access to portal performance Access to portal availibility
Conclusion
- For the LotusLive portal , the answering time network and server has increased since 31/08/11 ( for an identical content)
- Main errors encountered
- Google : 5 DNS resolution errors and 3 time outs
Lotus Live : 4 interrupted connections and 2 unreachable servers
Office 365 : 4 DNS resolution errors, 5 unreachable servers and 2 timeouts
- A maintenance intervention on the Office 365 portal on 9/9/11 from 5am to 8;15am was not taken into account
Authentication Authentication availibility and online messaging access
Results from 01/09/11 to 30/09/11 –About 2800 tests per portal
– Google is the strongest plaftform in terms of authentication 99.86 % in 24/24
– Office 365 99.76% in 24/24 on authentication (mainly linked to some timeouts and DNS errors); those incidents usually occur at night and from 8h to 23h the availibility is the same as Google
– LotusLive has the most errors at the authentication (99.17%); even if those errors occur mainly at night time, (99.56% availibility between 8h and 23h), one incident last September 18th between 8h and 10h penalise their position on this scale
– Google is the fastest platform to authentify yourself and access to messaging : 11,5 seconds, LotusLive 15 seconds ( in one page load in opposition to the others)
– Office 365 is the slowest to reach online Outlook : 19 seconds
98,52
99,56
99,17
99,81
99,89
99,86
99,54
99,89
99,76
97,50 98,00 98,50 99,00 99,50 100,00
DisponibilitéHNO
23h-8h
DisponibilitéHeures Ouvrées
8h-23h
Disponibilité24h/24
Office 365
google apps
lotus Live
19054
15238
11438
0 5000 10000 15000 20000
Performance Authentication and online messaging access
Authentication Authentication Incident for Lotus Live on 18/09
l 18
Timeout : incomplete page after login (88k out of 212ko expected)
Page error after login
Authentication Authentification errors examples
Authentification timeout on Office 365
Authentification timeout on Google Apps
Authentication Examples : Authentication errors on Google Apps portal
Erreur du serveur (HTTP 500) sur l’objet Dashboard qui devrait rediriger sur le portail
Impact utilisateur : le champ login absent de la page et l’utilisateur ne peut pas s’authentifier
Erreur
Cas normal
Sending/Receiving mail Performance
Average length of time to send and receive e-mails
– On average, to send a mail from a Cloud platform is faster than receiving a mail from an external platform
– Google is the platform which transmits mails the fastest, as well as sending to an external account (12,2 seconds on average) and receiving from an external account(7.5 seconds)
– Office 365 is faster than Lotus Live for receiving mail, but slower than LotusLive to send mails
15095
18895
10049
23213
7558
12271
0 5000 10000 15000 20000 25000
Temps d'envoi
Temps de réception
google apps
lotus live
Office 365
Sending/Receiving mail Stability
Average availibility for sending and receiving mails
– Lotus Live is the most available platform in terms of sending mails, if we consider as unavailable a length of time superior to 60 seconds ,to send a mail CloudExterne or ExterneCloud
– Office 365 (~75%) and Google Apps (~85%) : This means even with a fast sending and receiving mail process (<15 secondes), engorgements regularly slow down the e-mail sending or receiving process
– Sending and receiving mail process is more unstable for Google and Office 365
74,81
71,66
97,62
99,18
85,17
86,17
0 10 20 30 40 50 60 70 80 90 100
Disponibilité d'envoi (<60s)
Disponibilité de réception (<60s)
google apps
lotus live
Office 365
Google Apps Gmail can be considered as a reference in terms of online messaging access speed and you are guaranteed fast sending and receiving mail
IBM Lotus Live is a stable platform in terms of sending and receiving mails but can be a disappointment for users because of access unavailibility and average access time (15 sec)
Microsoft Office 365 has a good messaging availibility (99.89% during working hours) but some progress need to be made on page loading time ( particularly on authentication and access to Outlook) and on sending stability
1 3 2 Google Apps Lotus Live
Office 365
Portal performance 3 2 1
Portal availibility 2 3 1
Authentication availability and online messaging access 3 2 1
Authentifcation availability (working hours) 3 1 3
Receiving Time 3 1 2
Sending Time 3 2 1
Receiving Stability 2 3 1
Sending Stability 2 3 1
Total 21 17 11
Scores
Conclusions Results Details
Gmail Google
– Best Portal Access Performance
– Best authentication performance and availibity
– Best sending time to an external e-mail address and receiving from an external e-mail address
– Relative unstability in transfert time frames
• 15 mails out of 100 are sent in more than 60 seconds
• 14 mails out of 100 are received in more than 60 seconds
IBM Lotus Live
– Look out on the highest portal loading time since the beginning of September
– Best portal access availibility
– Best transfert stability
1 mails out of 100 are sent in more than 60 seconds
3 mails out of 100 are received in more than 60 seconds
– Authentication performance and availibility is not good
Microsoft Office 365
– Very good authentication disponibility during working hours.
– Portal access performance and access to messaging to be improved (19 seconds)
– Stabilité des durées de transmission des mails à améliorer
• 25 mails out of 100 are sent in more than 60 seconds
• 28 mails out of 100 are sent in more than 60 seconds
Conclusions Each editor commits himself to 99.9% availibility
– « Microsoft provides a financially-backed 99.9% uptime guarantee »
• http://www.microsoft.com/fr-fr/office365/what-is-office365.aspx
– « LotusLive Notes delivers the capabilities your business users need to increase efficiency while reducing IT complexity and associated bottom line costs - all backed by a dedicated IBM team with 24x7 support and a 99.9% up-time service level objective »
• https://www.lotuslive.com/styles/tours/transcripts/LLNotes_datasheet.pdf
– « Google Apps includes a 99.9% uptime SLA and synchronous replication »
• http://www.google.com/apps/intl/en/business/messaging.html
As a reminder data for September 2011…
According to you, what is the quality really
perceived by the online messaging users ?
Q/A
Were mails send with attachments ?
– No , only text content; the add on of attachments of different sizes could be a criteria to do more tests
Which internet navigator was used for the tests?
– Internet Explorer 8
Can those tests be run on a client (Lotus Notes, outlook) ?
Yesthe robots from ip-label can simulate any user running windows
Can this benchmark be testing with files /document sharing?
– Sharing files tools can also be tested by the robots ip-label
What about other navigators , especially if Microsoft Exchange is losing out on fonctionnality ?
– The robots can test Explorer or Firefox; the multi navigators cans be given through other solutions like Cloud Observer for instance
Can we audit audio / video flow?
– Ip-label can measure those flows using different measuring solutions on a user standpoint (streaming, voice quality analysis, video tests /audio conference, etc.)
Thank You
Synergie-informatique.fr
@SynergieInfo