NSLayoutManager hides new line characters no matter what I do

Question

I'm trying to show invisible characters like the new line character in my NSTextView subclass. The usual approach like overriding drawGlyph method of NSLayoutManager is a bad idea because it's too slow and not work properly with multi-paged layout.

What I'm trying to do is to override the setGlyph method of the NSLayoutManager so it would replace invisible "\n" glyph with "¶" glyph and " " with "∙".

And it works on the " " space glyphs but has no effect on the new line characters.

public override func setGlyphs(_ glyphs: UnsafePointer<CGGlyph>, properties props: UnsafePointer<NSGlyphProperty>, characterIndexes charIndexes: UnsafePointer<Int>, font aFont: Font, forGlyphRange glyphRange: NSRange) {
    var substring = (self.currentTextStorage.string as NSString).substring(with: glyphRange)

    // replace invisible characters with visible
    if PreferencesManager.shared.shouldShowInvisibles == true {
        substring = substring.replacingOccurrences(of: " ", with: "\u{00B7}")
        substring = substring.replacingOccurrences(of: "\n", with: "u{00B6}")
    }

    // create a CFString
    let stringRef = substring as CFString
    let count = CFStringGetLength(stringRef)

    // convert processed string to the C-pointer
    let cfRange = CFRangeMake(0, count)
    let fontRef = CTFontCreateWithName(aFont.fontName as CFString?, aFont.pointSize, nil)
    let characters = UnsafeMutablePointer<UniChar>.allocate(capacity: MemoryLayout<UniChar>.size * count)
    CFStringGetCharacters(stringRef, cfRange, characters)

    // get glyphs for the pointer of characters
    let glyphsRef = UnsafeMutablePointer<CGGlyph>.allocate(capacity: MemoryLayout<CGGlyph>.size * count)
    CTFontGetGlyphsForCharacters(fontRef, characters, glyphsRef, count)

    // set those glyphs
    super.setGlyphs(glyphsRef, properties:props, characterIndexes: charIndexes, font: aFont, forGlyphRange: glyphRange)
}

Then I came up with an idea: it looks like NSTypesetter marks new line char ranges like those it shouldn't process at all. So I subclassed NSTypesetter and did override a method:

override func setNotShownAttribute(_ flag: Bool, forGlyphRange glyphRange: NSRange) {
    let theFlag = PreferencesManager.shared.shouldShowInvisibles == true ? false : true
    super.setNotShownAttribute(theFlag, forGlyphRange: glyphRange)
}

But it's not working. NSLayoutManager still won't generate a glyph for the new line character, no matter what glyph I create.

What am I doing wrong?

score 5 · Answer 1 · answered Feb 20 '19 at 04:58

This is probably way past the point of usefulness for this question in particular, but I arrived here via Google.

As least as of testing on macOS 10.14, -[NSLayoutManager setNotShownAttribute:forGlyphAtIndex:] is able to change its value after the glyph for that index is generated. The key is that it will unconditionally setNotShownAttribute:YES for the newline glyph at the end of laying out the line. You get -[NSLayoutManagerDelegate layoutManager:shouldUseAction:forControlCharacterAtIndex:] after it finishes and you can reset it there:

- (NSControlCharacterAction)layoutManager:(NSLayoutManager *)layoutManager shouldUseAction:(NSControlCharacterAction)action forControlCharacterAtIndex:(NSUInteger)characterIndex {
  if (layoutManager.showsInvisibleCharacters && (action & NSControlCharacterActionLineBreak)) {
    [layoutManager setNotShownAttribute:NO forGlyphAtIndex:[layoutManager glyphIndexForCharacterAtIndex:characterIndex]];
  }

  return action;
}

func layoutManager(_ layoutManager: NSLayoutManager, shouldUse action: NSLayoutManager.ControlCharacterAction, forControlCharacterAt characterIndex: Int) -> NSLayoutManager.ControlCharacterAction {
  if layoutManager.showsInvisibleCharacters, action.contains(.lineBreak) {
    let glyphIndex = layoutManager.glyphIndexForCharacter(at: characterIndex)
    layoutManager.setNotShownAttribute(false, forGlyphAt: glyphIndex)
  }

  return action
}

You'll notice I use showsInvisibleCharacters in the sample, and it actually works for the built-in method even though it has no mapping for that character, yielding the "I don't know" glyph:

With -[NSLayoutManagerDelegate layoutManager:shouldGenerateGlyphs:properties:characterIndexes:font:forGlyphRange:, you can get it working perfect:

Great timing! I'm trying to rewrite my old implementation for better performance. I implemented your shouldUseAction... delegate method. I setShows... control and invisible characters. Depending on which font I use, I get an image like your first one. I'm struggling to find a good resource to learn more about the layoutManager:shouldGenerateGlyphs:... method to use the glyphs shown in your second image. Any pointers? Thanks!!! — Fletcher T. Penney, Feb 22 '19 at 05:07
After hours of "brute forcing" -- my working solution consists of the typesetter setNotShown..., layoutManager:shouldGenerateGlyphs... (to swap new glyphs for space and newline (tab doesn't work here?), and using drawBackgroundForGlyphRange to provide visible tabs (inspired by https://stackoverflow.com/questions/36666502/uitextview-draw-invisible-whitespace-characters). It seems strange to use separate approaches for tab as opposed to space/newline. Am I doing this right, or is there a better way? (Either way, this gives me faster and more accurate results than my old method!) — Fletcher T. Penney, Feb 22 '19 at 08:22

Vitalii Vashchenko · Accepted Answer · 2016-10-04T14:41:05.797

As I figured out, the default implementation of NSTypesetter's setNotShownAttribute: of the class doesn't change already generated glyphs in its glyph storage. So, call of super doesn't produce any effect. I just have to replace glyphs manually before calling super.

So, the most efficient implementation of showing invisible characters (you will see the difference while zooming the view) is this:

Limitations of this approach: if your app has to have multiple fonts in text view, then this approach might not be such a good idea, because the font of those displayed invisible characters will be different as well. And that's not what you might want to achieve.

Subclass NSLayoutManager and override setGlyphs to show space chars:

public override func setGlyphs(_ glyphs: UnsafePointer<CGGlyph>, properties props: UnsafePointer<NSGlyphProperty>, characterIndexes charIndexes: UnsafePointer<Int>, font aFont: Font, forGlyphRange glyphRange: NSRange) {
    var substring = (self.currentTextStorage.string as NSString).substring(with: glyphRange)

    // replace invisible characters with visible
    if PreferencesManager.shared.shouldShowInvisibles == true {
        substring = substring.replacingOccurrences(of: " ", with: "\u{00B7}")
    }

    // create a CFString
    let stringRef = substring as CFString
    let count = CFStringGetLength(stringRef)

    // convert processed string to the C-pointer
    let cfRange = CFRangeMake(0, count)
    let fontRef = CTFontCreateWithName(aFont.fontName as CFString?, aFont.pointSize, nil)
    let characters = UnsafeMutablePointer<UniChar>.allocate(capacity: MemoryLayout<UniChar>.size * count)
    CFStringGetCharacters(stringRef, cfRange, characters)

    // get glyphs for the pointer of characters
    let glyphsRef = UnsafeMutablePointer<CGGlyph>.allocate(capacity: MemoryLayout<CGGlyph>.size * count)
    CTFontGetGlyphsForCharacters(fontRef, characters, glyphsRef, count)

    // set those glyphs
    super.setGlyphs(glyphsRef, properties:props, characterIndexes: charIndexes, font: aFont, forGlyphRange: glyphRange)
}

Subclass NSATSTypesetter and assign it to your NSLayoutManager subclas. The subclass will display the new line characters and make sure that every invisible character will be drawn with a different color:

class CustomTypesetter: NSATSTypesetter {

    override func setNotShownAttribute(_ flag: Bool, forGlyphRange glyphRange: NSRange) {
        var theFlag = flag

        if PreferencesManager.shared.shouldShowInvisibles == true   {
            theFlag = false

            // add new line glyphs into the glyph storage
            var newLineGlyph = yourFont.glyph(withName: "paragraph")
            self.substituteGlyphs(in: glyphRange, withGlyphs: &newLineGlyph)

            // draw new line char with different color
            self.layoutManager?.addTemporaryAttribute(NSForegroundColorAttributeName, value: NSColor.invisibleTextColor, forCharacterRange: glyphRange)
        }

        super.setNotShownAttribute(theFlag, forGlyphRange: glyphRange)
    }

    /// Currently hadn't found any faster way to draw space glyphs with different color
    override func setParagraphGlyphRange(_ paragraphRange: NSRange, separatorGlyphRange paragraphSeparatorRange: NSRange) {
        super.setParagraphGlyphRange(paragraphRange, separatorGlyphRange: paragraphSeparatorRange)

        guard PreferencesManager.shared.shouldShowInvisibles == true else { return }

        if let substring = (self.layoutManager?.textStorage?.string as NSString?)?.substring(with: paragraphRange) {
            let expression = try? NSRegularExpression.init(pattern: "\\s", options: NSRegularExpression.Options.useUnicodeWordBoundaries)
            let sunstringRange = NSRange(location: 0, length: substring.characters.count)

            if let matches = expression?.matches(in: substring, options: NSRegularExpression.MatchingOptions.withoutAnchoringBounds, range: sunstringRange) {
                for match in matches {
                    let globalSubRange = NSRange(location: paragraphRange.location + match.range.location, length: 1)
                    self.layoutManager?.addTemporaryAttribute(NSForegroundColorAttributeName, value: Color.invisibleText, forCharacterRange: globalSubRange)
                }
            }
        }
    }
}

To show/hide invisible characters just call:

let storageRange = NSRange(location: 0, length: currentTextStorage.length)
layoutManager.invalidateGlyphs(forCharacterRange: storageRange, changeInLength: 0, actualCharacterRange: nil)
layoutManager.ensureGlyphs(forGlyphRange: storageRange)

Seems this code is in Swift3, how can we convert it in Swift 2.3? I noticed that "MemoryLayout" is not available in Swift 2.3. — Shiva Kumar, Oct 06 '16 at 06:31

NSLayoutManager hides new line characters no matter what I do

2 Answers2

Linked