MG3 and CF Framework Performance Benchmarks
Whenever I work on the core of a Model-Glue version, I use a suite of sample applications and JMeter tests to make sure I'm not wrecking anything in a performance or resource context. I measure against both prior versions of MG and other frameworks, making sure nothing anomalous is sneaking into MG by watching for spikes in the numbers. I'm not trying to "compete" as much as I'm just looking for comparable benchmarks for Model-Glue, and performance profiles of the other frameworks are my best bet.
For running these tests, I maintain a set of applications (Model-Glue, Fusebox, Mach-ii, and now ColdBox) that are the simplest possible request - they don't even show any output. For Mach-II and Model-Glue, no listener functions are run. For ColdBox, it's just the minimum required handlers and a simple "main" handler that adds one view.
Basically, it's a test of how well each can spin up its equivalent of a request context, execute it, render one empty view, and shut down.
This morning, I slaved my PC to running these tests, which ramp up from 25 to 100 simultaneous users, each making 100 back-to-back requests. The following chart shows the results:
Note: don't use these numbers for choosing a framework. Application code is much more likely to be a bottleneck than the framework!
If anyone else would like to try these out (or if any framework authors would like to make sure I've made the sample apps as simple as possible!), I've attached the Eclipse project I use for all of this to this entry. Maybe it'd be a good RIAForge project?
Keep up the great work.
Cheers
Nice graphics! The only point I would make is that the coldbox application you used is not basic at all, you are doing file logging, ses interceptions, handler recreations, value settings and layout renderings. Hardly optimized for a comparison load test. I can send you a comparable app based on the other ones if you want, so you can do more testing. Let me know? I am a big fan of tests!!
So does the current build of MG3 ;) All you have to do is listen for onSessionStart, onSessionEnd, onApplicationStart, etc.
@Rob and Luis,
I'd love to get it optimized and rerun. I used to have an older version that seemed to clip along a little better (maybe new features are added now?)...I remembered it being neck and neck w/ MG and M2, but the number were the numbers...
I do _not_ want to get into a "who's framework is the fastest" - I want us all to have a baseline to test against because it gives all of the framework authors a good safety net.
If you could send me an updated version that did only what was necessary to invoke ColdBox's event handler mechanics with a simple event that doesn't do anything (populated w/ url, form, etc.) values that renders an empty view, that'd be splendid.
Having dug into ColdBox a bit to see how it invokes its initial event, I'm betting it'll be faster than MG or M2.
Going further, I'm beginning to think I should set up a second set of templates that do an equivalent post / redirect / empty view + empty layout, as it may give a more real world scenario. That's a good deal of work to maintain, though...