Building an NPAPI host is non-trivial; if you really want to do it yourself from scratch your best bet would be to look at the source for Gecko, WebKit, and/or Chromium, and re-use whatever you can from there (assuming your app's source license makes that possible).
Other than that, Mozilla's NPAPI documentation is the best source, along with the list of accepted NPAPI extensions for more recent changes (since you mentioned the Mac, you'll need to read and understand at least the Cocoa, Core Graphics, Core Animation, and negotiation proposals). Just keep in mind that a lot of how NPAPI works is de-facto standard based on what other browsers do, so just the documentation won't be enough.
You mentioned in a comment that you don't want to use WebKit because of a "security issue"—if you know of a security issue in WebKit, why not fix it instead of building a really complicated system from scratch? It's very, very unlikely that if you make a from-scratch implementation of an NPAPI host you will end up making fewer security mistakes than exist in a mature implementation that's been tested, debugged, and improved over the course of a number of years.