So after much playing around I think I have achieved a little bit of success.
Two problems that I had to work around were:
- Buttons detect both NSAppearance and Material and change their design in relation to there superviews settings. You can easily change the appearance of a NSView but as far as I'm aware you're only able to change the material of a NSVisualEffectView. I wanted to achieve the same look as a FaceTime window, where the rows are a NSVisualEffectView in Dark when not selected and Light when they are. This is quite possible by embedding a NSVisualEffectView within your cell, but when adding buttons and having a load of rows in your table, you start getting graphical errors as you fling the table from one end to the other.
- My thought was that I needed to cut down on VisualEffectViews and as such have one embedded behind my TableView and have made all rows/clips/ views transparent. My problem here is that despite trying to make each view allow vibrancy I'm still unable to route the vibrancy down to the cells in my table. This also doesn't solve the problem of getting an individual row with a different vibrancy effect.
So luckily I wanted to use custom buttons anyway and as such didn't need to worry about the material aspect, but was caught up making my row highlight with a Light NSVisualEffectView when selected. I'd worked out that having hundred of effect views was probably not best, so figured maybe I needed to draw one between my row and my cell when selected... but turns out I managed to find an either simpler technique, by drawing a transparent white fill on my row view:
In my already subclassed TableRowView class:
override func drawSelectionInRect(dirtyRect: NSRect) {
let color = NSColor(white: 1, alpha: 0.6)
color.setFill()
NSRectFill(dirtyRect);
}