A version string can refer to a collection of independently versioned API's/packages, a single API, an API and implementation, etc. Basically you have to be cognizant of exactly what it is you are applying a version number to, and what your customers expect. Semantic versioning is all about telling your customers about the risks you think they will take, if they accept an update from you.
From your description, I think you have three things that need to be versioned:
- The file format.
- The library.
- the CLI utility application.
Whether you develop them in one, two or three repos is up to you. It's not uncommon for a file format specification to be developed and versioned independently of any implementations that depend on it. Whether your library and CLI utility belong together or in separate repo's is a matter of your development process logistics and customer expectations. Do you have consumers of the library that don't require your utility?
Whether they reside in one repo, two or three, you should package and version them separately. This implies independent feeds and packaging scripts.