There is a document "iBookstore Asset Guide", that is only available in iTunes Connect when logged in with a publisher account. It's not directly about the .ibooks-format, the iBooks Author creates, but about iBooks-specifics of publications for the iBookstore.
This document provides delivery information for all accepted media and files for the iBookstore. It describes the basics of EPUB structure and design, as well as guidelines specific to each type of EPUB and Multi-Touch books.
I can't give you a link, as sharing seems to be not allowed:
All information, documentation, and examples must not be shared external to Apple
Someone might provide a download of some kind for some reason at some place ;o)
The ibooks-format is not "open", is generated by iBooks Author and not really manually modifiable. To see, what iBooks Author is capable of, simply use it or its documentation. A simple answer is: iBooks Author is good for simple, quickly and easily created fixed-layout iBooks. If you want any special functionality, like JavaScript-interaction, animation etc, stick to the epub-format. iBooks allows the use of some epub3-features while the epub-files must be formatted epub2-style (content.opf, toc.ncx, html-files). The iBookstore Asset Guide and the provided samples are a good entrypoint for that.