Preventing SwiftUI ScrollView View Refresh When Appending New Items in LazyVGrid

Question

Issue: SwiftUI scrollview with LazyVgrid refresh entire view on appending new items

I am trying to implement a gallery view where i fetch images remotely, the api i use support pagination and hence i fetch only n number of images at a time, i want the behavior to be when user scroll to the bottom of the scrollview i make an api call and update the image list, the issue i am seeing is whenever i make an api call and update the imagelist, entire view gets refreshed and scroll position resets and i am on top of the scrollview, at the same time new set of images anyway gets appended to the list, can anyone suggest a solution/pointers on what i am doing wrong here and how can i eliminate the view refresh and just append new items inside the scrollview, thanks

@ObservedObject var imageList: ImageList
@State private var previousCount = 0

    var gridItemLayout = [GridItem(.flexible(minimum: 40), spacing: FilesGridView.gridSpacing),
                                  GridItem(.flexible(minimum: 40), spacing: FilesGridView.gridSpacing),
                                  GridItem(.flexible(minimum: 40), spacing: FilesGridView.gridSpacing),
                                  GridItem(.flexible(minimum: 40), spacing: FilesGridView.gridSpacing)]

var body: some View {
    ScrollViewReader { scrollViewProxy in
        ScrollView(.vertical) {
            LazyVGrid(columns: gridItemLayout, spacing: 10) {
                ForEach(imageList.images) { item in
                    ImageView(url: item.thumbnail, size: CGSize(width: 80, height: 80))
                        .id(item.id)
                        .cornerRadius(5)
                        .task {
                            if hasReachedEnd(image: item) {
                                loadMore?()
                            }
                        }
                }
            }
            .onChange(of: imageList.images.count) { count in
                if count > previousCount {
                    scrollViewProxy.scrollTo(imageList.images.last?.id)
                }
                previousCount = count
            }
        }
        .padding(.horizontal)
    }
}