The JLayeredPane works by defualt with no Layout manager, which means that you are using absolute positioning and no resizing. You could add a resize listener and adjust positions and size of inner components from code, as you see fit.
If you don't want to do this from code, you will need a layout manager, nothing fancy, just something to fill the container as it resizes. But here's the thing... if you add a layout manager, it will layout the components as if they are all in one layer, but most layout managers don't overlap their children so they are useless. The only one you could use is the OverlayLayout - it can also resize children. But using an OverlayLayout with JLayeredPane is overkill. You can just use OverlayLayout with a JPanel. So, yes, JLayeredPane is kind of useless. I recommend using a JPanel with an OverlayLayout instead.
Here is how to set things up so that you can have great control over almost any overlapping UI scenario out there: Using a JPanel with an OverlayLayout, have a separate transparent JPanel for each layer. In this way you can combine various LayoutManagers on different layers, by setting a diferent layout manager for each pane, including absolute positioning if necessary. Then add your visible components inside the panels representing the layers. Don't add them directly to the OverlayLayout panel. Just make sure that all of the JPanels you are using as layers have setAlignmentX and Y to center (0.5f) so that they fill the entire OverlayLayout panel as it resizes.