According given example it is not needed to order by maximum value of two, but order by date that is modified date when available and otherwise createdDate. That can be done as follows.
Right tool for that is COALESCE expression. It returns first non-null value. When all arguments are null, then null is returned.
CriteriaBuilder cb = em.getCriteriaBuilder();
CriteriaQuery<Object[]> cq = cb.createQuery(Object[].class);
Root<YourEntity> root = cq.from(YourEntity.class);
cq.multiselect(root, cb.coalesce(
root.get("modifiedDate"),
root.get("createdDate")));
cq.orderBy(cb.asc(cb.coalesce(
root.get("modifiedDate"),
root.get("createdDate"))));
List<Object[]> result = em.createQuery(cq).getResultList();
JPQL is bit more readable:
SELECT y, (y.modifiedDate, y.createdDate)
FROM YourEntity y
ORDER BY coalesce(y.modifiedDate, y.createdDate)