Apr 18 2008

MG3 and CF Framework Performance Benchmarks

Posted by Joe Rinehart at 10:27 AM
9 comments - Categories: Model-Glue | Frameworks

Whenever I work on the core of a Model-Glue version, I use a suite of sample applications and JMeter tests to make sure I'm not wrecking anything in a performance or resource context. I measure against both prior versions of MG and other frameworks, making sure nothing anomalous is sneaking into MG by watching for spikes in the numbers. I'm not trying to "compete" as much as I'm just looking for comparable benchmarks for Model-Glue, and performance profiles of the other frameworks are my best bet.

For running these tests, I maintain a set of applications (Model-Glue, Fusebox, Mach-ii, and now ColdBox) that are the simplest possible request - they don't even show any output. For Mach-II and Model-Glue, no listener functions are run. For ColdBox, it's just the minimum required handlers and a simple "main" handler that adds one view.

Basically, it's a test of how well each can spin up its equivalent of a request context, execute it, render one empty view, and shut down.

This morning, I slaved my PC to running these tests, which ramp up from 25 to 100 simultaneous users, each making 100 back-to-back requests. The following chart shows the results:

Coming back soon...getting an optimized application shell for ColdBox from its team.

Note: don't use these numbers for choosing a framework. Application code is much more likely to be a bottleneck than the framework!

If anyone else would like to try these out (or if any framework authors would like to make sure I've made the sample apps as simple as possible!), I've attached the Eclipse project I use for all of this to this entry. Maybe it'd be a good RIAForge project?

Comments

Neil

Neil wrote on 04/18/08 4:03 PM

@Joe good post, would like to mention couple of things, ColdBox is using all features of Application.cfc and also loading full blown SES engine as well.
Rob Gonda

Rob Gonda wrote on 04/18/08 5:49 PM

Joe, just to be fair, I wanted to point out that your instance of ColdBox had SES rewriting and logging enabled, and caching disabled. I know you threw this pretty quickly and didn't really spend time optimizing every framework or changing some of their settings. It would be nice to see some rules to compare them evenly. If you'd like, I can contribute with some ideas or modifying your base code to optimize ColdBox like you do for MG3.

Keep up the great work.

Cheers
Luis Majano

Luis Majano wrote on 04/18/08 8:41 PM

Hi Joe,
Nice graphics! The only point I would make is that the coldbox application you used is not basic at all, you are doing file logging, ses interceptions, handler recreations, value settings and layout renderings. Hardly optimized for a comparison load test. I can send you a comparable app based on the other ones if you want, so you can do more testing. Let me know? I am a big fan of tests!!
Joe Rinehart

Joe Rinehart wrote on 04/18/08 9:04 PM

@Neil,

So does the current build of MG3 ;) All you have to do is listen for onSessionStart, onSessionEnd, onApplicationStart, etc.

@Rob and Luis,

I'd love to get it optimized and rerun. I used to have an older version that seemed to clip along a little better (maybe new features are added now?)...I remembered it being neck and neck w/ MG and M2, but the number were the numbers...

I do _not_ want to get into a "who's framework is the fastest" - I want us all to have a baseline to test against because it gives all of the framework authors a good safety net.

If you could send me an updated version that did only what was necessary to invoke ColdBox's event handler mechanics with a simple event that doesn't do anything (populated w/ url, form, etc.) values that renders an empty view, that'd be splendid.

Having dug into ColdBox a bit to see how it invokes its initial event, I'm betting it'll be faster than MG or M2.

Going further, I'm beginning to think I should set up a second set of templates that do an equivalent post / redirect / empty view + empty layout, as it may give a more real world scenario. That's a good deal of work to maintain, though...
Ian

Ian wrote on 04/19/08 5:09 AM

I notice that the Fusebox app is being used in development-circuit-load mode. I would expect it to perform similar to "No Framework" once it is switched to production mode.
Luis Majano

Luis Majano wrote on 04/22/08 6:34 PM

HI Joe,

I updated the template you posted and left only the bare bones as you described. You can find it at http://www.coldboxframework.com/downloads/loadtesttemplate.zip

Please go over it and let me know if there is something missing and also, if you would like me to create another template with more stuff like you mention, I would be delighted. Thanks for your time, effort and the nice graphics!
Brad B

Brad B wrote on 04/23/08 2:35 AM

I like coldbox a lot. Can you post the optimized version? I mean, what would be the point of a JMeter test on a framework version thats n/a?
Sean Corfield

Sean Corfield wrote on 04/23/08 4:00 AM

I hope you'll update this with ColdBox soon - just because the "default" ColdBox behavior isn't optimized doesn't seem like a good reason to remove the entire comparison...
Oyun

Oyun wrote on 07/11/08 8:02 PM

harika bir olay kutlarim. Thank you
http://www.aylak.com

Write your comment



(it will not be displayed)







Categories

Monthly Archives

Search Archives

Favorite Links