Update
OK, I've tracked down the error! I'm using fonts from http://www.typography.com/ and if I remove the link to the fonts from the <head> (or even put it in the body instead) the site is fetched correctly every time!
Summary: If you're using webfonts which are loaded from a remote domain (with some kind of license approval process taking place as well) then the spiderable package will break!
The original question:
So I got this simple site built using meteor.js. It's on Digital Ocean, deployed using meteor up (with phantomjs enabled) and it's using the spiderable package.
Here's the site, it's a simple portfolio.
Now when I for exampel do curl http://portfolio.new-doc.com/?_escaped_fragment_= it will first return an empty body (classic meteor-without-spiderable-behaviour), but if I do the same curl within a few seconds it returns the correct result. (The same is true if I curl localhost:3000 on my machine).
So first the spiderable package does not do it's thing, and then it does. It kind of feels like on the first curl it returns the empty site (but loads up all the publiscations/subscriptions on the server) and on the second curl it uses the now loaded subscriptions and returns the correct result.
This is also true for Google webmaster tools. My first fetch as google bot returns an empty body, and the second one (if made quickly after the first one) returns the correct page.
The site only has one publish and one subscription. The publish either returns one or more pages from a subscription or runs this.stop(). The subscription is set up in a waitOn function in the app's only iron-router route. No complicated stuff here.
Since the curl command returns the correct result sometimes I don't think the error is in the publish/subs?
I've gotten the spiderable package to work in the past, but I've also spent a lot of time battling it!
Quite frustrating.
Any ideas? Thanks!
