Yes, they work... somewhat. They certainly affect behaviour, but it depends on the cat.
A "real-world" experiment showed some deterrence:
The efficacy of an ultrasonic cat deterrent, Sarah Helen Nelson, Andrew David Evans, and Richard Brian Bradbury, Applied Animal Behaviour Science
Volume 96, Issues 1-2, January 2006, Pages 83-91
Ultrasound deterrents for a variety of mammals, including cats, are widely available in the commercial market, but few have been independently tested for efficacy. This study tested the efficacy of an ultrasonic cat deterrent ‘Catwatch©’, using 63 and 96 volunteer observers in two long-running (18 and 33 weeks) blind experiments. Results indicated that the device did have a moderate deterrent effect, reducing the probability of a cat intrusion into a garden by approximately 32% in the first experiment, but not in the second. The average duration of intrusions was reduced by approximately 38 and 22% in the two experiments, respectively. The magnitude of the deterrent effect appeared to increase with time, since the device was deployed. It is likely that the size of the deterrent effect could be increased by positioning the device(s) more carefully with regard to entry points to the garden that are regularly used by cats.
There has also been a study of cats in more "lab-like" conditions, that also showed differences in behaviour - but perhaps not the ones desired (emphasis mine):
Evaluation of the welfare implications and efficacy of an ultrasonic ‘deterrent’ for cats, D. S. Mills, S. L. Bailey, R. E. Thurstans, Veterinary Record 2000;147:678-680 doi:10.1136/vr.147.24.678
The effect of the ultrasonic output of a commercial cat ‘deterrent’ was assessed by measuring the behaviour responses of 10 cats in a standard test arena. The cats were introduced to the arena approximately nine metres outside the stated range of efficacy of the product and small food piles were placed at one metre intervals towards the device. When the cats were released from the basket, their behaviour and location were recorded continuously. The behaviour of the cats with the device on and off was compared by using a general linear model and chi-squared analysis. Differences between individual cats were a significant factor in explaining the variance associated with the amount of ‘relaxed behaviours’ (P<0.001), and the time spent within the range of the device (P=0.006). The only significant behaviour changes recorded when the device was on, were an increased likelihood of ear flicking (P<0.001), less time spent actively exploring (P=0.043), and an increase in the amount of time spent in the reported ultrasonic range of the device (P=0.003).