Removing components not updating JPanel/JFrame

Question

I had a larger program where i encounterd a weird bug, i managed to isolate it into this small code line but from here i can't say what i'm doing wrong.

The problem is that for some reason the JFrame/JPanel forgets to update it's graphics when adding and removing a panel from it. it works if i don't set a preferred size for the "pbase" panel and or fixes itself if i resize the window while it's removed.

what am i missing?

public class ADdREMOVEJPANELS extends JFrame implements KeyListener{
public static void main(String[] args) {
    new ADdREMOVEJPANELS();
}
JPanel p = new JPanel();
JPanel pbase = new JPanel();
public ADdREMOVEJPANELS(){
    this.setDefaultCloseOperation(EXIT_ON_CLOSE);
    this.addKeyListener(this);

    p.setBackground(Color.red);
    p.setPreferredSize(new Dimension(200,200));

    pbase.setBackground(Color.blue);
    pbase.setLayout(new BorderLayout());
    pbase.setPreferredSize(new Dimension(200,200));
    pbase.add(p);

    this.setBackground(Color.green);
    this.add(pbase);


    this.pack();
    this.setVisible(true);
}
@Override
public void keyPressed(KeyEvent e) {

    if(e.getKeyCode() == KeyEvent.VK_UP){
        System.out.println("works");
        pbase.remove(p);
        pbase.setLayout(new BorderLayout());
        this.pack();

    }else if(e.getKeyCode() == KeyEvent.VK_DOWN){
        System.out.println("worksdouble");
        pbase.add(p);
        this.pack();
    }
}
}