Shouldn't rgl.postscript() work for a headless server, i.e. when options(rgl.useNULL = TRUE)? I know that rgl.snapshot() won't work.
library(rgl)
options(rgl.useNULL = TRUE)
open3d()
x <- sort(rnorm(1000))
y <- rnorm(1000)
z <- rnorm(1000) + atan2(x, y)
plot3d(x, y, z, col = rainbow(1000))
rgl.postscript("test.pdf",fmt="pdf")
This gives me "In rgl.postscript("test.pdf", fmt = "pdf") : Postscript conversion failed".