when hovering on FAB, I wanna not to rebuild CustomPaint widget

Question

I thought that when I am hovering on FAB, only the FAB would change and everything else would stay the same, but when I just hovering on, the CustomPaint widget is being repainted. I'm wondering why this is happening, as I thought CustomPaint was only repainting when I pressed FAB.

My code is as follows

class HomePage extends HookWidget {
  const HomePage({super.key});

  @override
  Widget build(BuildContext context) {
    final toggle = useState(false);

    debugPrint('HomePage rebuild');

    return Scaffold(
      body: Center(
        child: Container(
          height: 250,
          width: 250,
          child: CustomPaint(
            painter: PanelBackend(Colors.red, toggle),
            foregroundPainter:
                PanelFrontend(Colors.blue.withOpacity(1), toggle),
            child: Container(
              height: 100.0,
              width: 100.0,
              color: Colors.transparent,
              child: const Text('Middle'),
            ),
          ),
        ),
      ),
      floatingActionButton: FloatingActionButton(
        onPressed: () => toggle.value = !toggle.value,
        child: const Icon(Icons.add),
      ),
    );
  }
}

class PanelBackend extends CustomPainter {
  final Color color;
  final ValueNotifier<bool> notifier;

  PanelBackend(this.color, this.notifier) : super(repaint: notifier);

  @override
  void paint(Canvas canvas, Size size) {
    final paint = Paint()..color = color;

    final stamp = DateTime.now().millisecondsSinceEpoch;
    debugPrint('$stamp : Backend painted, size: ${size.toString()}');

    canvas.drawRect(
        Rect.fromPoints(Offset.zero, Offset(size.width, size.height)), paint);
  }

  @override
  bool shouldRepaint(covariant PanelBackend oldDelegate) {
    return notifier.value != oldDelegate.notifier.value;
  }
}

class PanelFrontend extends CustomPainter {
  final Color color;

  final ValueNotifier<bool> notifier;

  PanelFrontend(this.color, this.notifier) : super(repaint: notifier);

  @override
  void paint(Canvas canvas, Size size) {
    final paint = Paint()..color = color;

    final stamp = DateTime.now().millisecondsSinceEpoch;
    debugPrint('$stamp : Front painted, size: ${size.toString()}');

    canvas.drawRect(
        Rect.fromPoints(Offset.zero, Offset(size.width, size.height)), paint);
  }

  @override
  bool shouldRepaint(covariant PanelFrontend oldDelegate) {
    return notifier.value != oldDelegate.notifier.value;
  }
}

I tried using ElevatedButton instead of FAB, but with the same result, i.e. when hovering on, the CustomPaint has rebuild.

However, I tried using GestureDetector instead of FAB or ElevatedButton, then It was just as I thought it would be.