I'm going to throw my two cents in here with a third option that gets it based on finding the 1st of each day, and then adding multiples of 7 to get to the final result.
Function
CREATE FUNCTION `ISNTHDAYINMONTH`(checkDate DATE, weekNumber INTEGER, dayOfWeek INTEGER, monthOfYear INTEGER) RETURNS INTEGER
NO SQL
DETERMINISTIC
BEGIN
DECLARE firstOfMonth DATE;
SET firstOfMonth = DATE_SUB(checkDate, INTERVAL DAYOFMONTH(checkDate) - 1 DAY); #Find the first day of the current month
IF DAYOFWEEK(checkDate) = dayOfWeek AND MONTH(checkDate) = monthOfYear #Make sure at least this matches
AND DAYOFMONTH(checkDate) = ((1 + (7 + dayOfWeek - DAYOFWEEK(firstOfMonth)) % 7) + 7 * (weekNumber - 1)) THEN #When the date matches the nth dayOfWeek day of the month
RETURN 1;
ELSE
RETURN 0; #Nope
END IF;
END
Use
SELECT ISNTHDAYINMONTH(datecol, weekNumber, dayOfWeek, monthOfYear);
Where weekNumber is 1 through 5, dayOfWeek is 1 through 7 (1 = Sun), and monthOfYear is 1 through 12.
Example
To check if datecol is the second Tuesday of April:
SELECT ISNTHDAYINMONTH(datecol, 2, 3, 4);
Calculations
Let's break it down. This line gets the first day of the month for the date that is passed in.
SET firstOfMonth = DATE_SUB(checkDate, INTERVAL DAYOFMONTH(checkDate) - 1 DAY); #Find the first day of the current month
This check ensures that the date has the correct day of the week, and correct month (just in case the user passed in a date that isn't even the right day of the week).
DAYOFWEEK(checkDate) = dayOfWeek AND MONTH(checkDate) = monthOfYear
Now for the big one:
DAYOFMONTH(checkDate) = ((1 + (7 + dayOfWeek - DAYOFWEEK(firstOfMonth)) % 7) + 7 * (weekNumber - 1))
To get the amount we need to add to 1 to get the date, we calculate dayOfWeek that we are looking at, minus the day of the week that the first day of the month falls on to get the offset mod 7 ((dayOfWeek - DayOfWeek(first)) % 7).
Note that because the first of the month could land on a day before or on the day we are looking at, we add an additional 7 and then mod by seven. This is needed because MySQL's Mod function does not properly compute mod. I.e. -1 % 7 should be 6, but MySQL returns -1. Adding 7 and taking the modulus ensures the result will always be correct.
To summarize:
NumberToAddToOneToGetDateOfFirstWeekDay = (7 + DayOfWeek - DayOfWeek(firstDayOfMonth)) %
Then we add one and how every many multiples of 7 are needed to get to the correct week.