Unicode string with diacritics split by chars

Question

I have this Unicode string: Ааа́Ббб́Ввв́Г㥴Дд

And I want to it split by chars. Right now if I try to loop truth all chars I get something like this:
A a a ' Б ...

Is there a way to properly split this string to chars: А а а́ ?

Answer 1

To do this properly, what you want is the algorithm for working out the grapheme cluster boundaries, as defined in UAX 29. Unfortunately this requires knowledge of which characters are members of which classes, from the Unicode Character Database, and JavaScript doesn't make that information available(*). So you'd have to include a copy of the UCD with your script, which would make it pretty bulky.

An alternative if you only need to worry about the basic accents used by Latin or Cyrillic would be to take only the Combining Diacritical Marks block (U+0300-U+036F). This would fail for other languages and symbols, but might be enough for what you want to do.

function findGraphemesNotVeryWell(s) {
    var re= /.[\u0300-\u036F]*/g;
    var match, matches= [];
    while (match= re.exec(s))
        matches.push(match[0]);
    return matches;
}

findGraphemesNotVeryWell('Ааа́Ббб́Ввв́Г㥴Дд');
["А", "а", "а́", "Б", "б", "б́", "В", "в", "в́", "Г", "г", "Ґ", "ґ", "Д", "д"]

(*: there might be a way to extract the information by letting the browser render the string, and measuring the positions of selections in it... but it would surely be very messy and difficult to get working cross-browser.)

Answer 2

A little update on this.

As ES6 came by, there are new string methods and ways of dealing with strings. There are solutions for two problems present in this.

1) Emoji and surrogate pairs

Emoji and other Unicode characters that fall above the Basic Multilingual Plane (BMP) (Unicode "code points" in the range 0x0000 - 0xFFFF) can be worked out as the strings in ES6 adhere to the iterator protocol, so you can do like this:

let textWithEmoji = '\ud83d\udc0e\ud83d\udc71\u2764'; //horse, happy face and heart
[...textWithEmoji].length //3
for (char of textWithEmoji) { console.log(char) } //will log 3 chars

2) Diacritics

A harder problem to solve, as you start to work with "grapheme clusters" (a character and it's diacritics). In ES6 there is a method that simplify working with this, but it's still hard to work. The String.prototype.normalize method eases the work, but as Mathias Bynens puts:

(A) code points with multiple combining marks applied to them always result in a single visual glyph, but may not have a normalized form, in which case normalization doesn’t help.

More insight can be found here:

https://ponyfoo.com/articles/es6-strings-and-unicode-in-depth https://mathiasbynens.be/notes/javascript-unicode

Answer 3

This package might help you: https://www.npmjs.com/package/runes

const runes = require('runes')

const example = 'Emoji '
example.split('') // ["E", "m", "o", "j", "i", " ", "�", "�"] 
runes(example)    // ["E", "m", "o", "j", "i", " ", ""] 

Answer 4

If you're writing an application that needs to consume chunks of data from a Node.js stream, then you can probably just pipe through utf8-stream to prevent this:

https://github.com/substack/utf8-stream

Answer 5

Using the Unicode properties Grapheme_Base

"Ааа́Ббб́Ввв́Г㥴Дд".match(/\p{Grapheme_Base}/gu)
> ['А', 'а', 'а', 'Б', 'б', 'б', 'В', 'в', 'в', 'Г', 'г', 'Ґ', 'ґ', 'Д', 'д']

and Grapheme_Extend

"Ааа́Ббб́Ввв́Г㥴Дд".match(/\p{Grapheme_Extend}/gu)
> ['́', '́', '́']

combining these into

"Ааа́Ббб́Ввв́Г㥴Дд".match(/\p{Grapheme_Base}\p{Grapheme_Extend}|\p{Grapheme_Base}/gu)
> ['А', 'а', 'а́', 'Б', 'б', 'б́', 'В', 'в', 'в́', 'Г', 'г', 'Ґ', 'ґ', 'Д', 'д']

Answer 6

The problem of your string are surogate pairs ("a" "́) which get combined to signle character only when displayed by browser. For your case, it is enough if you attach \u0301 to the previous character but this is by no means a general solution.

var a="Ааа́Ббб́Ввв́Г㥴Дд",
    i =0,
    chars=[];

while(a.charAt(i)) {
  if (a.charAt(i+1) == "\u0301") {
    chars.push(a.charAt(i++)+a.charAt(i++));
  } else {
    chars.push(a.charAt(i++));}}

To clarify the issue, go and read Mathias Bynens's blog post.