You cannot locally detach a global listener, you have to detach it globally, ie via
EventManager::getInstance()->off(/* ... */);
However you may want to consider whether passing options to the finder may be a better solution, so that your controllers do not have to know about listeners and the like, but just execute the find calls as needed, like
$Table->find('all', ['doThisAndThat' => false]);
and your listener could then act accordingly.
Quote from the docs
[...] Any options that are not in this list will be passed to beforeFind listeners where they can be used to modify the query object.
Cookbook > Database Access & ORM > Retrieveing Data & Result Sets > Using Finders to Load Data