References to ExoPlayerImplInternal causes too much use of RAM

Question

I have two instances of ExoPlayer in each Fragment in a ViewPager. After swiping a lot of times, the app reaches ~120MB of RAM, and starts running really slow. I never get an OutOfMemoryException though.

Taking a look at the hprof file, I see that there 37 instances of ExoPlayerImplInternal, even though I make sure to release it properly every time the Fragment gets its view destroyed in onDestroy(), onDetach() and onDestroyView().

This is how I initialize the player:

public void initPlayer(Context context) {
    initProxy(context);

    if (player == null) {
        DefaultBandwidthMeter bandwidthMeter = new DefaultBandwidthMeter();
        TrackSelection.Factory videoTrackSelectionFactory =
                new AdaptiveTrackSelection.Factory(bandwidthMeter);
        TrackSelector trackSelector =
                new DefaultTrackSelector(videoTrackSelectionFactory);

        // 2. Create the player
        player = ExoPlayerFactory.newSimpleInstance(context, trackSelector);

        final LoopingMediaSource loopingSource = new LoopingMediaSource(mediaSource(videoUrl));

        player.prepare(loopingSource);
        player.setVideoScalingMode(C.VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING);
    }
}

In the Fragment this is the release method, called in onDestroy(), onDetach() and onDestroyView():

public void release() {
    if (player != null) {
        player.removeListener(playerEventListener);
        player.stop();
        player.release();
        player = null;
    }
    setPlayerPlaying(false);
    if (simpleExoPlayerView != null) {
        simpleExoPlayerView.setPlayer(null);
        simpleExoPlayerView = null;
    }
    playerEventListener = null;

    removeExoPlayerLayout();

    if (uiProgressHandler != null) {
        uiProgressHandler.removeCallbacks(uiProgressRunner);
        uiProgressHandler.removeCallbacks(null);
        uiProgressHandler = null;
    }
    uiProgressRunner = null;
    handler = null;
    listener = null;

    released = true;
}

A snapshot of the hprof file, when the app reaches ~120MB:

I'm kind of lost of where I could be keeping references to all those instances, being that in the ViewPager the only initialized players are the visible one and the two adjacent Fragments.