How to make react-native-reanimated-carousel work with ImageZoom (react-native-image-pan-zoom)?

Question

I want to have a carousel that each item is a zoomable image, the carousel should be all over the screen so I use Portal for that. To support zoom I use ImageZoom component from react-native-image-pan-zoom, and the carousel is from react-native-reanimated-carousel, in the following way:

      <Portal>
        <Carousel
          loop
          windowSize={1}
          width={SCREEN_WIDTH}
          height={SCREEN_HEIGHT}
          data={images}
          style={{
            flex: 1,
            backgroundColor: "black",
          }}
          defaultIndex={imageIndexToDisplay}
          onSnapToItem={handleImageChange}
          renderItem={({ item: { url, width, height } }) => (
            <ImageZoom
              cropWidth={SCREEN_WIDTH}
              cropHeight={SCREEN_HEIGHT}
              imageWidth={width}
              imageHeight={height}
              enableSwipeDown
              onSwipeDown={handleClose}
              onClick={handlePress}
              useNativeDriver
            >
              <Image
                imageSource={url}
                defaultImage={defaultImage}
                imageStyle={{ height, width }}
              />
            </ImageZoom>
          )}
        />
      </Portal>

What happens is that the carousel barely let me scroll left or right since it seems like the ImageZoom responds first to the scrolls. I tried to set onStartShouldSetPanResponder={() => false} on the ImageView which solves the Carousel scrolling but doesn't let me use the ImageZoom to zoom since it appears like the Carousel now responds first to gestures.

I would like to know if there is any way to make them both work together.

Thanks ahead!