To begin with a quote:
Mathematics is the art of giving the same name to different things.
Ferdinand Verhulst
Indeed, according to Wikipedia's page on Monotonic functions, the use of "anti" (before "monotone" or "monotonic") for a function in the realm of order theory is different than its use in calculus and analysis.
In order theory, "a monotone function is also called isotone, or order-preserving. The dual notion is often called antitone, anti-monotone, or order-reversing". It only means that the order of the images of the function is inverted.
But generally speaking, we deal with calculus. There, your first definition is the right one : a function "is called monotonic if and only if it is either entirely non-increasing, or entirely non-decreasing."And if a function increases and decreases, it would be simply called non-monotonic.
In data mining, what would be a monotonic function would be the support function of an itemset (its frequency in the transaction database). But when "frequent" (i.e sup(X) > supmin) is our criteria :
"if a set is frequent, then all of its subset are frequent too", and also "if a set is infrequent then all of its superset are also infrequent."
The combination of both means the anti-monotonicity in this context.