Since you're working with genetic programming, you spend lot of operations doing one (or many) of the following :
- Computing next generation (crossover, mutation)
- Fitness function computing
- Garbage collecting if your population is very large
To determine the hot spots of your code, use JVisualVM, the sampling pane will be enough for your use case.
From my experience, I would say there are two main probable hotspots:
- Slow crossover and memory exhaustion: this is related to uncompact individual representation. You can gain significant performance by reducing individual's memory footprint. Try to put it in a primitive array whenever possible.
- Slow fitness : This is more difficult, you may want to reduce the number of program evaluation, or go for more significant examples.
If you're working with a fixed-size population, you may consider working off-heap, but this may significantly complexify your program.
I don't know if you are using a framework like JGAP, but this latter is deadly slow, too many encapsulations and large call stacks. It is very good for proof of concepts, but when things get intensive, you have to do it yourself.
EDIT: reading the comments, I see that your code is not parallelized, you must absolutely start with that. This is trivial for GP, just run the fitness in parallel, then the next generation computation in parallel.