You can do this by composing predicates. If you have a predicate that can compare variables, and a predicate that can compare coefficients, then you can easily create a new predicate that checks with one, returning a definite answer if the first predicate provides a definite answer, or deferring to the second predicate, in the case that it doesn't. This will be reusable for other applications, too:
(defun and-then (original-predicate next-predicate)
"Returns a new predicate constructed from ORIGINAL-PREDICATE and
NEXT-PREDICATE. The new predicate compares two elements, x and y, by
checking first with ORIGINAL-PREDICATE. If x is less than y under
ORIGINAL-PREDICATE, then the new predicate returns true. If y is less
than x under ORIGINAL-PREDICATE, then the new predicate returns false.
Otherwise, the new predicate compares x and y using NEXT-PREDICATE."
(lambda (x y)
(cond
((funcall original-predicate x y) t)
((funcall original-predicate y x) nil)
(t (funcall next-predicate x y)))))
Then it's easy enough to make a call to (and-then 'variable< 'coefficient<). First, some accessors and predicates:
(defun term-coefficient (term)
(first term))
(defun coefficient< (term1 term2)
(< (term-coefficient term1)
(term-coefficient term2)))
(defun term-variable (term)
(second term))
(defun variable< (term1 term2)
(string< (term-variable term1)
(term-variable term2)))
Now the test:
(defparameter *sample*
'((1 x)(2 y)(1 x)(2 x)(3 y)(2 y)))
CL-USER> (sort (copy-list *sample*) 'coefficient<)
((1 X) (1 X) (2 Y) (2 X) (2 Y) (3 Y))
CL-USER> (sort (copy-list *sample*) 'variable<)
((1 X) (1 X) (2 X) (2 Y) (3 Y) (2 Y))
CL-USER> (sort (copy-list *sample*) (and-then 'variable< 'coefficient<))
((1 X) (1 X) (2 X) (2 Y) (2 Y) (3 Y))
You could define a compare-by function to create some of these predicate functions, which could make their definitions a bit simpler, or possibly removable altogether.
(defun compare-by (predicate key)
"Returns a function that uses PREDICATE to compare values extracted
by KEY from the objects to compare."
(lambda (x y)
(funcall predicate
(funcall key x)
(funcall key y))))
You could simply the predicate definitions:
(defun coefficient< (term1 term2)
(funcall (compare-by '< 'term-coefficient) term1 term2))
(defun variable< (term1 term2)
(funcall (compare-by 'string< 'term-variable) term1 term2))
or get rid of them altogether:
(defun varsort (p1)
(sort (copy-list p1)
(and-then (compare-by '< 'term-coefficient)
(compare-by 'string< 'term-variable))))