NestedScrollView No Scroll when List is Short with Float SliverAppBar

Question

I am having a ListView, that is required to use pull to refresh function, and a floating header is needed on the design. Therefore I have tried to use the NestedScrollView to fulfill the task. However, I find that even if the list is short, as the SliverAppBar is floating, the body is always a bit longer to make the list able to scroll to hide the app bar which is not what expected on the our design flow.

The design flow considered is to

1. pin the appbar (not able to scroll) when the body is shorter than the available space
2. able to float app bar when list body is longer than the available space

I have tried to listen the scroll notification, but it seems not work as what i want to obtain. Please give me some suggestions if there is any way can solve this problem. Thank you!

The following is the code so far i have tried:

return Scaffold(
      body: Builder(
        builder: (context, ) {
          return SafeArea(
            child:  NotificationListener<ScrollNotification>(
              onNotification: (info){
                debugPrint('scroll = ${info.metrics.pixels}');
                bool isFloat = info.metrics.pixels>57.0;
                if(_isFloat!=isFloat){
                  _isFloat= isFloat;
                  WidgetsBinding.instance?.addPostFrameCallback((timeStamp) {
                    setState((){});
                  });
                }
                return true;
              },
              child: NestedScrollView(
                floatHeaderSlivers: false,
                headerSliverBuilder: (context, isInnerScroll) {
                  return [
                    SliverAppBar(
                      title: Text('Demo'),
                      floating: _isFloat,
                      pinned: !_isFloat,
                    ),
                  ];
                },
                body: SmartRefresher(
                  controller: _refreshController,
                  onRefresh: () {
                    Future.delayed(Duration(milliseconds: 1000), () {
                      _refreshController.refreshCompleted();
                    });
                  },
                  child: ListView.builder(
                    itemCount: 5,
                    itemBuilder: (context, index) {
                      return ListTile(
                        title: Text('$index'),
                      );
                    },
                  ),
                ),
              ),
            ),
          );
        }
      ),
    );