Some characters break grid tables

[lggruspe]

Explain the problem.

pandoc example.md -f markdown+grid_tables -s

The first table works just fine. The second table breaks even though the only difference is that some entries have additional diacritics (U+031E COMBINING DOWN TACK BELOW)

+-----------+-----------+-----------+-----------+-------+
| vowels    | front                 | central   | back  |
|           +-----------+-----------+           |       |
|           | unrounded | rounded   |           |       |
+===========+===========+===========+===========+=======+
| close     | i         | y         |           | u     |
+-----------+-----------+-----------+-----------+-------+
| mid       | e         | ø         |           | o     |
+-----------+-----------+-----------+-----------+-------+
| open      |           |           | ä         |       |
+-----------+-----------+-----------+-----------+-------+

+-----------+-----------+-----------+-----------+-------+
| vowels    | front                 | central   | back  |
|           +-----------+-----------+           |       |
|           | unrounded | rounded   |           |       |
+===========+===========+===========+===========+=======+
| close     | i         | y         |           | u     |
+-----------+-----------+-----------+-----------+-------+
| mid       | e̞         | ø̞         |           | o̞     |
+-----------+-----------+-----------+-----------+-------+
| open      |           |           | ä         |       |
+-----------+-----------+-----------+-----------+-------+

Pandoc version?
pandoc 3.1.8, fedora 38

[jgm]

jgm/doclayout has code to calculate widths for East Asian characters (which can be 0, 1, or 2 width). But this character is not in the lookup table doclayout uses -- presumably because it's a western accent combining character. We should modify doclayout so that it knows about all combining characters, not just those in the East Asian set.

Moving this issue to doclayout, where it will need to be addressed.

[jgm]

Sorry: there is in fact code to deal with these combining characters, so I'm not yet sure what is going on here.

[jgm]

OK, transferring back to pandoc.
I think the problem lies there. Note if you take off the fancy header, the second table works fine:

+-----------+-----------+-----------+-----------+-------+
| close     | i         | y         |           | u     |
+-----------+-----------+-----------+-----------+-------+
| mid       | e̞         | ø̞         |           | o̞     |
+-----------+-----------+-----------+-----------+-------+
| open      |           |           | ä         |       |
+-----------+-----------+-----------+-----------+-------+

So, my guess is that somewhere in the code that was added to deal with row/colspans, someone used T.length instead of realLength.

[jgm]

A more minimal test case:

+-----------+-------+
| a         | back  |
+===========+=======+
| ø̞         | o̞     |
+-----------+-------+

This is parsed as having rowspan=2. Replacing the characters on the bottom row with regular ascii makes the problem go away.

[jgm]

The problem seems to lie in the gridtables package, which parses this table as:

ArrayTable {arrayTableCells = array ((1,1),(2,2)) [((1,1),ContentCell 2 1 [" a         ","==========="," \248\798         "]),((1,2),ContentCell 2 1 [" back  ","======="," o\798     "]),((2,1),ContinuationCell (1,1)),((2,2),ContinuationCell (1,2))], arrayTableHead = Nothing, arrayTableFoot = Nothing, arrayTableColSpecs = array (1,2) [(1,(AlignDefault,11)),(2,(AlignDefault,7))]

@tarleb can you have a look at this?

[jgm]

Submitted issue quarto-dev/gridtables#10