Most AB testing you hear about is referring to client-side tests powered by injecting JS in the browser. Testing in a Java app requires a different approach.
You can use a free, open-source tool such as Planout. This serves as a basic traffic splitter and uses a deterministic hashing algorithm so that you get consistent variations as long as you keep using the same user ID.
If you need more robust functionality, you may need to go for a commercial package. For in-app testing (mobile, web, backend), these usually take 1 of 2 forms:
- API approach: your app can send a request to the vendor-managed server asking for a variation for the current user. The downside here is the performance hit of waiting for a response every time you need the user's variation.
- SDK: vendor-provided SDK you implement in your Java code base that uses a deterministic hash to determine the variation for the user. The SDK will need retrieve some kind of data file with experiment status, traffic allocation, etc from a server prior to running the experiment (and on some update frequency), but then lets you determine variations in memory (no blocking network calls). Tracking calls can be sent async.
A commercial package will give you a few benefits over a free solution, like a user-friendly web portal to manage experiments, a statistical calculator, QA tools, feature flags, user permissions, ongoing updates and support, etc.
Disclaimer: I work for Optimizely who develop an SDK-driven platform for AB testing & feature flags.