module TermBuf::Unicode

Overview

Character property lookup for the terminal buffer: display width, grapheme cluster break class, and the two auxiliary properties UAX #29 needs.

Every code point falls into exactly one range of Tables::RANGE_START, so a lookup is a binary search with no miss case. Code points below FAST_LIMIT skip the search entirely via a direct-indexed table built at startup, which covers Latin, Greek, Cyrillic, Hebrew, Arabic, the combining diacriticals, and most of the Indic scripts.

Defined in:

termbuf/unicode/grapheme.cr
termbuf/unicode/overrides.cr
termbuf/unicode/policy.cr
termbuf/unicode/tables.cr
termbuf/unicode/utf8.cr
termbuf/unicode/width.cr

Constant Summary

EMOJI_PRESENTATION = '️'

Variation selector 16, forcing emoji presentation and so a width of two.

FAST_LIMIT = 4096

Code points below this are resolved by direct index rather than search.

TEXT_PRESENTATION = '︎'

Variation selector 15, forcing text presentation and so a width of one.

ZERO_WIDTH_JOINER = '\u200D'

Joins emoji into one cluster, and on most terminals into one glyph.

Class Method Summary

• .ambiguous?(char : Char) : Bool

  Whether char is East Asian Ambiguous, and so rendered at a width the terminal's configuration decides.

• .ambiguous_width : Int32

  Cells an East Asian Ambiguous character takes.

• .char_width(char : Char, ambiguous : Int32 = ambiguous_width) : Int32

  Number of terminal cells char occupies on its own, ignoring any grapheme cluster it may belong to.

• .conjunct_class(char : Char) : Tables::Incb

  Indic conjunct break class of char, used by rule GB9c.

• .each_grapheme(string : String, policy : WidthPolicy = Unicode.policy, & : Grapheme -> ) : Nil

  Yields each extended grapheme cluster of string in order, without allocating: a Grapheme locates its cluster by byte offset.

• .grapheme_class(char : Char) : Tables::Gcb

  Grapheme cluster break class of char (UAX #29).

• .graphemes(string : String) : Array(String)

  All extended grapheme clusters of string, as separate strings.

• .pictographic?(char : Char) : Bool

  Whether char has the Extended_Pictographic property, used by rule GB11.

• .policy : WidthPolicy

  How clusters are measured when no policy is given.

• .policy=(policy : WidthPolicy)

  How clusters are measured when no policy is given.

• .properties(codepoint : Int32) : UInt16

  Packed property word for codepoint.

• .string_width(string : String, policy : WidthPolicy = Unicode.policy) : Int32

  Total number of terminal cells string occupies, under policy.

• .utf8_continuation?(byte : UInt8) : Bool

  Whether byte continues a character rather than beginning one.

• .utf8_length(lead : UInt8) : Int32

  How many bytes the character starting with lead takes, or zero if lead is not a lead byte at all.

• .utf8_prefix(bytes : Bytes) : Int32

  How much of bytes forms whole characters.

Class Method Detail