[label] Full-width hyphen in Japanese ー is accidently disappeared

[Kanahiro]

maplibre-gl-js version: latest

browser: Chrome

Steps to Trigger Behavior

1. write style to show label, without "text-font" or with "text-font" not including ー
2. show labels including full-width hyphen in Japanese ー
3. ー is not shown.

Link to Demonstration

Following two are using same style.

GL JS v4

https://jsbin.com/wosopasika/edit?html,output

GL JS v3

https://jsbin.com/qiduxehete/edit?html,output

• You can see A-BーC in v3 but in v4 A-BC.
• It seems that there might be some regression from v3 to v4.

Expected Behavior

v3 is correct.

Actual Behavior

ー is missing. It occurs also on latest version.

[HarelM]

Thanks for taking the time to report this issue!
Any chance you could take a look into this issue?

[Kanahiro]

Yes, I'm researching this now.👍

[Kanahiro]

When text-font isn't defined OR defined font doesn't exist in glyphs, getGlyphs returns object as result that result[stack][id] become null.
This seems to occur this problem.

maplibre-gl-js/src/render/glyph_manager.ts

Line 44 in bedcb86

async getGlyphs(glyphs: {[stack: string]: Array<number>}): Promise<GetGlyphsResponse> {

[HarelM]

That can be a result of my refactoring of the callback hell, including the glyphs code, which was a nightmare to read and understand...

[Kanahiro]

The problem is here:

maplibre-gl-js/src/render/glyph_manager.ts

Line 129 in bedcb86

/\p{Ideo}|\p{sc=Hang}|\p{sc=Hira}|\p{sc=Kana}/u.test(String.fromCodePoint(id));

This regexp judges whether a character is CJK or not.

Japansese full-width dash ー is not included sc=Hira or sc=Kana. It is included in sc=Common.
sc=Common includes very broad characters: some Japanese symbols 「 ☆ and some half-width symbols like whitespace.

sc=Common includes not only Japanese symbols but also some symbols used globally.
Simply adding sc=Common in the regexp break current testcase: code

IMHO, there are two ways:

1. hardcode all characters not included in sc=Hira or sc=Kana
2. revert changes in the regexp to v3 codes

[HarelM]

I think this was changed by @1ec5. @1ec5 can you take a look at this issue please?

[Kanahiro]

IMO, we can use unicodeBlockLookup

https://github.com/maplibre/maplibre-gl-js/blob/main/src/util/is_char_in_unicode_block.ts

sc=Kana in Regexp may not be enough unicode classfying like above.

[Kanahiro]

I made draft: #5421

IMO:

• It is better to use unicodeBlockLookup.
• It may be possible to broaden blocks to be judged as CJK (e.g. // symbol in pull request)

I'm not sure how much is a cost to broaden it but it looks useful.