There are 2 options that I can think of:
1) The better option is to ensure apps always have that variable you reference for quota populated in the first place. This eliminates the need for 2 variables, and quota policy can be used as you referenced. Furthermore, an admin could override or assign a different quota if necessary. The Dev Connect can be configured to ensure an app custom attribute is assigned upon app creation or registration.
2) Alternatively, you can check from 1 source(like an app custom variable), and then if it does not have value, you can use another source (like the API product quota setting).
Unfortunately, I do not believe this can be done all within the Quota policy. Rather, you could use a service callout policy to set 1 quota variable based on what is available.
That... or you could use 2 different quota policies where either would get triggered based on their conditions. Their conditions would reference the variables you mentioned to check whether they exist (or not).
<Step>
<Condition>(app.quota_var is null)</Condition>
<Name>QuotaPolicyUsingApiProductQuotaReference</Name>
</Step>
<Step>
<!-- if the app custom variable is there, you must mean to use it -->
<Condition>(app.quota_var != null)</Condition>
<Name>QuotaPolicyUsingAppQuotaReference</Name>
</Step>