Lies, damned lies, and statistics
Apr 9th, 2010 by sharps
First some insight into how my twisted mind works. I rarely believe any bar chart, pie-chart, percentage I see presented unless I can access the raw data myself and draw my own conclusions. I’m not a statistician by trade or education but I’ve spent a lot of time running surveys and analyzing large data sets; so I have the benefit of some experience.
Replay Solutions just published a survey about Java Platform usage. The questioning, subsequent analysis and presentation of the results was poor IMO. But they did one thing right – they provided the raw data. Thanks for that.
There’s a posting on TSS entitled “Why is JBoss at the bottom of this list ???”. In typical TSS style – few people actually bothered to read the survey results or question them and a long and rather pointless thread ensues. This post is an expansion of my comment at the end of the TSS thread.
The original report has this chart :
Update – Sunday 4/11/10
My initial (very quick) analysis was wrong. My formula for searching for different categories had a basic reg-ex flaw so I was over-counting JBoss by fair bit. I’ve fixed that mistake (spreadsheet here) and also removed duplicates (responses with both “JB + TC” and “Tomcat” or “JBoss” – I’d already admitted to this minor double counting (in the comments) – the original author’s analysis still includes this error.
So the ranking is now the same as the original author’s but the %’s are different. My apologies to IBM for originally stealing their #2 spot 
By the way – Red Hat fully supports both Tomcat and JBoss AS – so #1 and #3 rankings and being able to satisfy 87% of the market isn’t such a bad result for us.
[As percentages of respondents - that's : Tomcat = 59%, Websphere = 39%, JBoss = 28%, Weblogic = 23%, Other = 19%]
These rankings (WAS above JBoss) are more representative of larger organizations (where both WAS and WLS have traditionally been stronger) – the latest Eclipse survey shows a similar break-out. Unlike the Eclipse survey – this survey doesn’t have any information on the respondents and they seem to be largely self-selected.
My original points still stand – poor questioning, poor analysis and presentation are common in these kinds of surveys. Always ask for the source data !
Great post Rich!
Hi Rich, agree – I am uneasy with any survey data without understanding the methodology and actual questions asked.
I will look at the data in detail later – but I think you’ve made a mistake by counting “Multi Answers” back into the individual server choices. ***Doing this double counts the results.***
For eg: If a respondent selected “JBoss, Weblogic” for Question JS1 “Which App Servers will you use in 2010?” the data set puts a “1″ under JBoss, a “1″ under “WebLogic” and “1″ under “Multi Answers”.
So, the simplest thing to do is ignore “Multiple Answers” when looking at the individual server usage.
Which should get you back to data as reported by Replay Solutions (except with the “Multiple Answers” row removed).
Savio
IBM WebSphere (but a data geek at times)
You’re correct about “Multi Answers” they are included in the server counts (except “JB+TC” and “Jonas…”) – including it on the bar chart was bad form and threw me.
My numbers simply count the occurences of the “server” – this gives the correct count for everything but the “Jonas…” category – but those numbers are small enough to not make a difference. The only other error is that if “JB+TC” and “Tomcat” were specified I double count – but that only accounts for a small error.
Again – this is a flaw in the questions and presentation – that’s my main point.
My %’s are based on the total “server” responses so add up to 100% and I ignore “no answer”.
- Rich
Great analysis Rich. It’s a great followup to the material.
This is always the problem we have with these types of surveys.
Besides some of problems in the survey itself (like not listing GlassFish!), the survey says nothing about the population being sampled, and, w/o that, the value of the survey is limited. It says something, sure, but, what does it say?…
Surveys! Even the EDC’s surveys had significant methodological problems, but this one… BTW, did EDC stop doing AppServer surveys?
Looked at the raw data briefly. Honestly, given the flaws in the data and that we not even know the sample, it does not seem worth spending much time on this but….
* I don’t think you can reduce multiple-choice answers to flat market shares.
* The 18% “Other” and 4% Blank is a big chuck of the data. I looked and most of the “Others” are by themselves, not “WLS, WAS, Other”, but just “Other”.
* There is no data on deployment vs development. Does using Jetty in Maven count the same as using JBoss in production?
* Given that GlassFish is the largest major player omitted, I could argue a big chunk of that 18% is GF
. Maybe even some of the +4%. Or not. Who knows.
The survey also has weird sets of questions/answers. The selection for “Key JavaEE Services/Frameworks” is really strange.
Anyhow, unless I hear something more interesting about the survey, I’m going to ignore it…
Ok, thanks for the clarification – I guess I don’t really know what you mean by ‘reassigned “Multiple Answers”:
> and re-assigned “JBoss + Tomcat” and “Multiple Answers”.
In any case – I looked at the data in two ways.
Scenario 1: Follow the Replay Analysis Methodology & allow a user to vote for multiple servers (i.e. Total results > Total valid respondents for the question)
- For instance, if respondent A selected WebSphere and respondent B selected JBoss and WebSphere, the total results would be 2 for WebSphere and 1 for JBoss, for a sum of 3 across the total 2 respondents.
- In keeping with the point above (and Replay Solutions’ analysis methodology) I added all of “TC+JB” results (161 by my count) to the “Tomcat” result and also add 161 to the “JBoss” results.
- Total count = 2036 split as follows:
35% Tomcat & JB+TC
17% JBoss & JB+TC
12% WebLogic
20% WebSphere
5% Jetty
1% Jonas, Jrun, Orion
1% Resin
10% Other
100% Total
Not sure why the results don’t exactly match yours, but I could very easily have made an error (I did the spreadsheet work quickly while on a call!)
Scenario 2: Each valid respondent’s response is spread across the servers s/he selects (i.e. Total results == Total valid respondents for the question)
- For instance, if respondent A selected WebSphere and respondent B selected JBoss and WebSphere, the total results would be 1.5 for WebSphere and 0.5 for JBoss across the total 2 respondents.
- Total count = 1054 split as follows:
34% Tomcat & TC portion of “JB+TC”
12% JBoss & JB portion of “JB+TC”
11% WebLogic
24% WebSphere
4% Jetty
1% Jonas, Jrun, Orion
1% Resin
14% Other
100% Total
I am in no way saying that Scenario 2 is a valid representation of “actual usage” or anything like that. But interesting none the less…
I’ve uploaded the file here if you’re interested: http://drop.io/6sotiyj# (tried to get it up on gDocs but it kept crashing)
Savio
IBM WebSphere
@Savio – scenario 1 is pretty close to what I got – the diffs. could be the double counting on my side.
@Eduardo – I agree there are much bigger questions about the source of the data than I’ve posed. The data is tangible so an easier target for me.
My own analysis isn’t perfect but my point still stands – the Questioning, analysis and presentation of the data is poor.
The debate over how to properly count these responses just supports the notion that you can make the data say whatever you want, but I believe the proper counts are:
Tomcat – 625
WebSphere – 410
JBoss – 287
WebLogic – 239
Other – 197
(I’ve ignored the others as their counts are low)
Like Savio suggests, I’ve ignored “multiple answers” and I’ve also avoided double counting for JBoss and Tomcat by making sure that when JBoss or Tomcat is specified in the same response as JBoss+Tomcat I only count that response once. My results are more consistent with Savio’s so I suspect Rich’s minor double counting may not be so minor.
See http://ktschmidt.blogspot.com/2010/04/further-analysis-of-java-platform.html for some charts from my analysis.
> my point still stands – the Questioning, analysis and presentation of the data is poor.
I agree 100% w/ you.
@Savio, @Kevin :
a) you’re both wrong; as is the report’s author – but so was I. I’ve now corrected my post.
b) thanks for the peer-review !
c) we’ve already spent way more time on this than the orginal author !
- Rich