Going to try answer this myself, as it seems a pure CSS solution is not possible. I've found a small workaround here that works for me. It's not ideal, but may work for others too..
I just render the number behind the plural class name and wrap this around the s. Then I use a class for plural1 that hides the s.
This works for me because I use templates to render my html and it's easy to slot the numbers in.
.plural1 {
display: none;
}
<div>-1 book<span class="plural-1">s</span>
</div>
<div>0 book<span class="plural0">s</span>
</div>
<div>1 book<span class="plural1">s</span>
</div>
<div>2 book<span class="plural2">s</span>
</div>
Pros: This can work for other endings and other languages too. Negative numbers are supported.
Cons: Adds unused classes to your elements (plural2, plural3, etc.) that the browser will need to read and ignore. Not really an issue for a small number of object though (eg. 100).