Nanohttpd-Webserver is not stoppable after Activity-Resume

Question

I have an activity with included webserver-class (simplified):

public class MyActivity extends AppCompatActivity {

    SimpleWebServer webServer = new SimpleWebServer(8088);

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_my);
    }

    @Override protected void onResume() {
        super.onResume();
        // Starting webserver
        try {
            webServer.start();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    public void changeActivityButtonClicked(View view){
        //change activity
    }

    public void disconnectButtonClicked(View view){
        // Stopping webserver
        webServer.stop();
    }

    //Webserver-Class
    private class SimpleWebServer extends NanoHTTPD {
        public static final String TEXT_404 = "404 File Not Found";

        public SimpleWebServer(int port) {
            super(port);
        }

        @Override
        public Response serve(IHTTPSession session) {
            return new Response("You didn't use the API");
        }
    }
}

This code works as expected when I don't leave the activity. But when I exit this activity (via the changeActivityButtonClicked()-method or whatever) and re-enter it, I can't use the webServer.stop() command anymore. The server runs forever.

I guess I'm loosing focus of the actual webServer-object when changing the activity, but I have no idea how to reaccess it.

Do you have any hints?