AlertDialog not showing in Fragment

Question

I've got a really strange problem where on some KitKat devices, my simple yes/no AlertDialog will appear behind the current fragment and not in the foreground. The reason I say the dialog appears behind the current fragment is because the dialog appears in the foreground only after I rotate the device. The app has a MainActivity that switches between different fragments that take up most of the screen.

MainActivity.java

@Override
public void onBackPressed() {
    Builder builder = new AlertDialog.Builder(this);
    builder.setMessage(getString(R.string.exit_confirm_summary))
            .setTitle(getString(R.string.exit_confirm_title))
            .setCancelable(true)
            .setPositiveButton(getString(R.string.ok),
                    new OnClickListener() {
                        @Override
                        public void onClick(DialogInterface dialog, int which) {
                            //...
                        }
                    })
            .setNegativeButton(getString(R.string.cancel),
                    new OnClickListener() {
                        @Override
                        public void onClick(DialogInterface dialog, int which) {
                            //...
                        }
                    });
    AlertDialog dialog = builder.create();
    dialog.show();
}

After doing some research I found that it is best to use DialogFragment when using Fragments in your app, so I changed my code to this:

MainActivity.java

@Override
public void onBackPressed() {
    AlertDialogFragment adf = new AlertDialogFragment();
    adf.setRetainInstance(true);
    adf.show(getFragmentManager(), "dialog");
}

AlertDialogFragment.java

public class AlertDialogFragment extends DialogFragment {
public AlertDialogFragment() {}

@Override
public Dialog onCreateDialog(Bundle savedInstanceState) {
    return new AlertDialog.Builder(getActivity())
            .setCancelable(false)
            .setTitle("Alert DialogFragment")
            .setMessage("AlertDialogFragment Test")
            .setPositiveButton("OK", new DialogInterface.OnClickListener() {
                public void onClick(DialogInterface dialog, int which) {
                    // ...
                }
            })
            .setNegativeButton("Cancel", new DialogInterface.OnClickListener() {
                public void onClick(DialogInterface dialog, int which) {
                    // ...
                }
            }).create();
    }
}

However, the effect is still the same. The dialog should appear when I press the back button, but is only visible after I press the back button and then rotate the device. It also becomes visible after I go home and come back into the app. I've noticed it only happens on a few devices but I'd like to get rid of this problem for good.

Note: this behavior happens for all dialogs in the app, not just this one.

Anyone have any ideas what is going on?