Flutter: Hide or unhide field based on a button without rebuilding the widget

Question

I have a parent widget which has 2 child widgets for input fields. So, I want one field to be hidden in the beginning and by pressing a button to hide or unhide it. I managed to achieve that by wrapping the child widget (input field) in a visibility widget and then adding the button in the parent widget with a function onPressed() to setState() of the variables. However, this approach rebuild the parent widget since the setState happens in the parent level and my problem is that I don't want to lose the values inside the input fields when I press the button. Therefore, I want just to hide or unhide the child widget.

Some parts of code are: Two input fields inside the LocationSearchDialog Widget

Table(columnWidths: const <int, TableColumnWidth>{
                          0: FlexColumnWidth(85),
                          1: FlexColumnWidth(15),
                        }, children: [
                          TableRow(
                            children: [
                              Padding(
                                  padding:
                                      const EdgeInsets.fromLTRB(15, 0, 15, 10),
                                  child: Visibility(
                                      visible: widget.showFromField,
                                      child: LocationSearchDialog(
                                        labelText: 'From',
                                        addMarker: addStartPointMarker,
                                        currentLocation: widget.currentLocation,
                                      ))),
                              Text('')
                              //this widget was added just to align the other 2 widgets in the table
                            ],
                          ),
                          TableRow(
                            children: [
                              Padding(
                                  padding:
                                      const EdgeInsets.fromLTRB(15, 0, 15, 0),
                                  child: LocationSearchDialog(
                                    labelText: 'To',
                                    addMarker: addDestinationPointMarker,
                                    currentLocation: widget.currentLocation,
                                  )),
                              Padding(
                                  padding:
                                      const EdgeInsets.fromLTRB(15, 0, 15, 0),
                                  child: ShowStartField(updateFromFieldVisibility: updateFromFieldVisibility))
                            ],
                          ),
                        ])

The button that needs to update the visibility of the field

Transform.rotate(
    angle: 180 * widget.rotateButton / 180,
    child: IconButton(
    icon: Icon(Icons.arrow_forward_ios),
    iconSize: 15,
    color: Colors.blue,
    tooltip: 'Add pick up point',
    onPressed: () {
    updateFromFieldVisibility;
    },
    ));
  

Function to update the visibility

  void updateFromFieldVisibility() {
    setState(() {
      widget.showFromField = !widget.showFromField;
      widget.rotateButton = widget.rotateButton == 1.5 ? 0 : 1.5;
    });
    print(widget.showFromField.toString() + ' ' + widget.rotateButton.toString());

  }

All of these widget and functionality exist in one main file which is the stateful parent widget.

Thank you in advance guys!