Google-Fit API Client does not connect without Google-Fit App installed

Question

I know the documentation (https://developers.google.com/fit/overview) states:

Note: The Google Fit app uses the Google Fit platform, which is included in Google Play services. Your fitness or wellness app can use the Google Fit platform without requiring users to install the Google Fit app.

but I am having problem even connecting my client when I uninstall the Google-Fit App from my device. It works perfectly while Google-Fit in installed on my device but now I uninstalled to check if it's still working but now I don't get any callbacks. Neither onConnected now onConnectionSuspended is called. Only onConnectionFailed if I decline to give access in the Request-Sheet that pops up. It's just not doing anything. Also in Logcat I can't find anything useful...

This is the code I am using to connect to the client.

Could anybody tell me what I am missing here?

    GoogleApiClient.Builder builder = new GoogleApiClient.Builder(mContext)
            .addApi(Fitness.HISTORY_API)
            .addScope(new Scope(FITNESS_ACTIVITY_READ))
            .addConnectionCallbacks(new GoogleApiClient.ConnectionCallbacks() {
                @Override
                public void onConnected(Bundle bundle) {
                    // Do some magic
                }

                @Override
                public void onConnectionSuspended(int i) {
                    // Do some magic
                }
            })
            .enableAutoManage(mFragmentActivity, new GoogleApiClient.OnConnectionFailedListener() {
                @Override
                public void onConnectionFailed(@NonNull ConnectionResult connectionResult) {
                    // Do some magic

                    // This here is called if I decline to the Login-Sheet
                }
            });

    mClient = builder.build();