Swipe to refresh in listfragment when there are no items

Question

I did a Swipe to refresh in my list fragment and it works good when i have items in my list but when i have no items it's working too but you can only see the loading refresh button a few seconds.

Here is my layout :

<?xml version="1.0" encoding="utf-8"?>
<android.support.v4.widget.SwipeRefreshLayout
    xmlns:android="http://schemas.android.com/apk/res/android"
    android:id="@+id/swipe"
    android:layout_width="match_parent"
    android:layout_height="match_parent"

    >

    <RelativeLayout
        xmlns:fab="http://schemas.android.com/apk/res-auto"
        android:layout_width="match_parent"
        android:layout_height="match_parent">

        <ListView
            android:id="@id/android:list"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:layout_alignParentBottom="true"
            android:layout_alignParentLeft="true"
            android:layout_alignParentStart="true"
            android:layout_alignParentTop="true"
            android:longClickable="true"/>

        <TextView
            android:id="@android:id/empty"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:layout_centerHorizontal="true"
            android:layout_centerVertical="true"
            android:text=""
            android:textSize="30sp"/>

        <com.getbase.floatingactionbutton.AddFloatingActionButton
            android:id="@+id/semi_transparent"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:layout_alignParentBottom="true"
            android:layout_alignParentEnd="true"
            android:layout_alignParentRight="true"
            fab:fab_colorNormal="@color/primaryColor"
            fab:fab_colorPressed="@color/AccentColor"
            fab:fab_plusIconColor="@color/primaryTextColor"/>
        <TextView
            android:id="@+id/imageView5"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:layout_centerHorizontal="true"
            android:layout_centerVertical="true"
            android:width="200dp"
            android:height="200dp"
            android:text="Vous n&apos;avez aucun coffy commencez par en ajouter un en cliquant sur le bouton +"
            android:textSize="22dp"
            android:visibility="gone"/>

        <WebView
            android:id="@+id/webView3"
            android:layout_width="10dp"
            android:layout_height="10dp"
            android:layout_above="@+id/semi_transparent"
            android:layout_centerHorizontal="true"
            android:visibility="invisible"/>
    </RelativeLayout>
</android.support.v4.widget.SwipeRefreshLayout>

I did the pull to refresh like this (this is working also when there a no items but you can't see the icon when there are no items) :

final SwipeRefreshLayout swipeView = (SwipeRefreshLayout) rootview.findViewById(R.id.swipe);

        swipeView.setColorScheme(R.color.primaryColor, R.color.AccentColor, R.color.primaryColorDark, R.color.primaryTextColor);
        swipeView.setOnRefreshListener(new SwipeRefreshLayout.OnRefreshListener() {
            @Override
            public void onRefresh() {
                swipeView.setRefreshing(true);
                Log.d("Swipe", "Refreshing Number");
                (new Handler()).postDelayed(new Runnable() {
                    @Override
                    public void run() { }