although I think Google Website Optimizer would be a good choice, I assume you could build your own system via:
- A @Before method which generates a random value for the session (a test seed let's say) or a value based on a parameter you decide (revenue, etc)
- Some logic around the value of the seed in the business layer
- Tags for the UI part, in which the tag shows some text or another (or image, or...) depending on the value of the seed
- A table in the database to store information linked to a session (to trace the movements of the user as he comes into your site)
- Some reporting over the data in the table (I can only recommend Jasper or Birt, haven't work with others)
But it will be quite a lot of work.
Honestly I truly believe using existing systems (Google Website) is a better idea. Think that all the time you spend setting your environment for your custom A/B testing is time you are not developing your product. And although A/B is good to increase conversion, you will need many iterations over it to find the best choice, so the most time you spend on other things, the more money you are potentially loosing.
In fact, from your complains on Google's tool:
- I have to say that the main point of A/B testing is the presentation layer, as that's what the user sees and what drives their actions. They don't care if your implementation of a merge sort is nicer, they care that your logo is cute.
- Whatever you do, you will clutter your templates. That or you will have to create many "clones" of a page, one for each version you want to show. Either way, it's not "nice" code.
- You need to runt he tests for a while to have a valid number of samples before being able to decide something. That means you can't change the options every day (unless your site has thousands of users daily) or you won't have statistically significant data
So from your 4 points, I believe that 3 are debatable. I would advise to first use some existing tool to increase the performance. Once you have the best conversion possible then you can think on doing some extra custom work based on other parameters.