The documentation isn't clear on this point, and the developercommunity post is incorrect:
https://developercommunity.visualstudio.com/content/problem/175267/release-environment-artifact-triggers-not-availabl.html
The comment below from @MarinaLiu-MSF confirms Environment Artifact Filters feature are not available for onsite TFS 2017 nor for the first release of onsite TFS 2018. To date, this functionality is only currently available in the online VSTS.
I found a post on the VisualStudio User Voice site that mentions it being available in the next major on-prem version: https://visualstudio.uservoice.com/forums/330519-visual-studio-team-services/suggestions/13413054-trigger-release-definition-only-for-specific-branc
ADMIN
VSTS Team (Product group, Microsoft Visual Studio) responded · July 11, 2017
We are happy to let you know that your suggestion is now available on Team Services.
It will become available to the on-prem version in the next major version of TFS.
You can read more about it at https://www.visualstudio.com/en-us/docs/build/concepts/definitions/release/triggers#release-triggers