In general, don't do optimization unless you know exactly that you have a performance problem and why it occurs.
The decision whether to use the builder pattern or not should be based upon a single aspect: Does it improve the code quality (i.e. maintainability, readability, ...)?
If you later see that it causes a performance problem (which is unlikely), you could still find a solution, maybe even with keeping the builder pattern.
There are several reasons why it is unlikely that it causes problems:
- In general, GC is really fast for objects that aren't referenced at all after usage, which is typically the case for builder instances.
- If objects are only used in a single stackframe (i.e. they are not passed around to other methods or as return values), allocation and GC is even quicker (search for "escape analysis" if you want to know details). In many cases, builders are used in this way.
- JIT compilation is also negligible, because it's a once-per-runtime task and the builder code is trivial, as it mainly consists of simple setters and instance variables.