This will depend on your containment relationship; whether it is multivalued (its "upperBound" is set to "-1" i.e. it can hold as many Y as you want) or monovalued (its "upperBound" is "1" or unchanged, it can only hold a single Y).
If multivalued, the reference will never be "null" (or "oclIsUndefined"). When it does not hold a single Y, it will be an empty list and you thus need to check for the size :
[if (not X.relationname.isEmpty())]
Otherwise, for monovalued references, you can check for null
(what you have done in your answer seems to indicate that it is the case for you here) :
[if (not X.relationname.oclIsUndefined())]
On the contrary, what you have done in your answer is a little different :
[if((X.relationname.attributename->size()).oclIsUndefined() <> true)]
This will actually retrieve the Y associated with your X and access its attribute values. This will not be null
if there is no "Y" : it will be "invalid" i.e. it will fail. Of course, "oclInvalid" (the "failure" object) is different from "true" so your "<>" works... even though it is clunky (you'd usually use the "not" operation instead of testing against a boolean value).