There is no correct way. But we may break it down to this:
Do you know the amount of JavaFX elements to display before your application has started? If yes, you can build your GUI with SceneBuilder or directly in XML writing the FXML File. This is true for most GUI screen setups where the overall layout (MenuBar, ToolBar, content, sidebar, ...) is well known and does not change over time.
Now, custom components do not know how to display itself. There are many options to correspond for and most custom components are build up dynamically or can change drastically as the user changes these options. You can not write a single FXML file but instead would have to write many single files, each would build a small part of the custom component. This is not really handy to manage, so most developers choose to do it without FXML, directly in code.
And on top of that: There are many custom components which use a special layout which is not built in into JavaFX, to achieve that you have to write your own containers and wire them up with custom components to get the perfect result.
In the end, custom components are built (or better: should be built) to be used inside FXML / SceneBuilder.