RST input: "Simple Tables" problems (with leading spaces and tabs)

[jschleus]

Converting the RST input file https://www.kernel.org/doc/Documentation/process/license-rules.rst
to HTML with pandoc (current version 3.8) results in a severe formatting problem of a so-called "simple table" at the end of the file (as can be seen e.g. at https://fossies.org/linux/kernel/v6.17/linux-6.17-rc6.tar.gz/linux-6.17-rc6/Documentation/process/license-rules.rst). The expected behavior (converted by sphinx) can be seen e.g. at https://www.kernel.org/doc/html/v6.17-rc6/process/license-rules.html (also at the end under "MODULE_LICENSE").

Since I am not familiar with RST and pandoc, I could not determine whether the error is in the input file or whether it is a pandoc rendering error. At least for me, the problem seems to be related to the whitespace at the beginning of the table content and the combination of spaces and tabs. Unfortunately, I have not yet found a specification for how whitespace should be handled in this context.

By the way, the "simple table" example within the pytablewriter’s documentation
also did not format correctly with pandoc.

For simplification I created four simpler test cases (T=tab, expanding here to 8 spaces) and checked it with "sphinx" and "pandoc" (by the way, "rst2html" from the "docutils" package gave the same results as "sphinx"):

Example 1 (starting point):

The width of the first column is deliberately chosen as 9 (1 for a character + 8 for an expanded tab)
to avoid further problems.

Source:

========= ======
1         abcdef
2         abcdef
========= ======

1a) "sphinx" HTML output rendered by "firefox" (only a basic representation):

1         abcdef
2         abcdef

Okay, result as expected.

1b) "pandoc" HTML output rendered by "firefox" (only a basic representation):

1 abcdef
2 abcdef

Hmm, result principally okay, but the table width for column 1 is ignored.

Example 2 (before the "2" is a single space):

Source:

========= ======
1         abcdef
 2        abcdef
========= ======

2a) "sphinx" HTML output rendered by "firefox" (only a basic representation):

1         abcdef
2         abcdef

Okay, result identical to example 1.

2b) "pandoc" HTML output rendered by "firefox" (only a basic representation):

1              abcdef

       2       abcdef

Hmm, an added empty line and formatting problems.

Example 3 (before the "2" is a single tab):

Source:

========= ======
1         abcdef
T2 abcdef
========= ======

3a) "sphinx" HTML output rendered by "firefox" (only a basic representation):

1         abcdef
2         abcdef

Okay, result identical to example 1
(it appears that "sphinx" expands existing tabs to 8 characters each before converting).

3b) "pandoc" rendered HTML (principally):

1              abcdef

    2 abcd     ef

Hmm, an added empty line and more formatting problems.

Example 4 (more like the problematic part in the original mentioned Linux license file; after the "1" is a single tab; column 1 of row 2 empty resp. "without" a character -> column spanning?):

Source:

========= =============
1T  abcdef
T  abcdef
========= =============

4a) "sphinx" HTML output rendered by "firefox" (only a basic representation):

1         abcdef abcdef

Okay (?), the "spanned" text from row 2 is added to column 2 of row 1.

4b) "pandoc" rendered HTML (principally):

1 abcd        ef

     abcd     ef

Hmm, not the expected single line with a spanned column 2, but an added empty line and more formatting problems.

Sorry for the somewhat incomplete report.

[jgm]

If you're assuming an 8-space tab stop, try with --tab-stop=8. By default pandoc uses 4.

[jschleus]

Great, I didn't know about this option, but using it results now in correct pandoc rendering of the mentioned Linux license-rules.rst file (with row spanning).

But still two comments:

First, how could one determine the correct or optimal value for the --tab-stop option?

For a single file, you could rely on the first line of equal signs that determine the column structure, or simply use trial and error. But on a server like fossies.org (of which I am the administrator), where hundreds of pandoc conversions are performed daily while user are browsing FOSS project documentation, this is impossible.

Based on the experience here, and since e.g. the vim editor also uses tabstop=8 by default, I will probably use now generally for pandoc the value --tab-stop=8 instead of the default 4 (which hopefully won't cause problems elsewhere).

By the way, due to the strong dependence on column position, using tabs seems to me to be a very bad idea, at least for RST tables.

And secondly: Aside from the fact that the fundamental problem that triggered this issue has been resolved, the somewhat exotic problem (leading space) in my second example file remains, which demonstrates a general difference in the output between sphinx and pandoc:

Sphinx seems to simply ignore leading spaces in a column header and outputs the text left-aligned. Pandoc seems to interpret them this way, centering the text in the cell in question and inserting a blank line both above and below the relevant line. Bug or feature?

And finally, I would like to express my sincere thanks to the pandoc team, as the pandoc tool usually does its difficult job really well or even perfectly, and lives up to its name.

[jgm]

Markdown always had a tab = 4 spaces conversion, so that's the default in pandoc.
I don't know how you'd determine what the right setting is for a random document. For markdown it should always be 4; maybe for RST it should always be 8?

As for the other issue you mention, I guess it is your Example 2. We use a common parser for markdown and RST simple tables, and it may be that RST simple tables don't have the column alignment feature we have in our markdown simple tables. [Confirmed. I see that we already handle that in the RST writer, left-aligning everything. We should also change the reader so that everything is default aligned.]

[jgm]

Actually, it's not about alignment - the RST reader produces AlignDefault for every column.

The issue is that the RST reader is parsing the indented cell as a BlockQuote.

<table>
<tbody>
<tr>
<td>1</td>
<td>abcdef</td>
</tr>
<tr>
<td><blockquote>
<p>2</p>
</blockquote></td>
<td>abcdef</td>
</tr>
</tbody>
</table>

[jschleus]

The different behavior can also be observed well in the "simple table" example mentioned above within the pytablewriter’s documentation. All rows have there at least one column entry beginning with a space.

By the way, I implemented now on fossies.org your suggestion. Thanks.

[jgm]

The commit I made above fixes the leading space problem.

But now I'm worried about a discrepancy I'm seeing between the released pandoc and the latest dev build. It only appears with

.. table:: example_table

    ===  =====  ====  =====  ========  ========================
    int  float  str   bool     mix               time
    ===  =====  ====  =====  ========  ========================
      0   0.10  hoge  True          0  2017-01-01 03:04:05+0900
      2  -2.23  foo   False            2017-12-23 12:34:51+0900
      3   0.00  bar   True   Infinity  2017-03-03 22:44:55+0900
    -10  -9.90        False       NaN  2017-01-01 00:00:00+0900
    ===  =====  ====  =====  ========  ========================

With the dev build this just results in...nothing. With the release we get a table.
The problem is NOT the latest commit, though. I have found by bisecting that the culprit is commit 47e1cc4 . @TuongNM can you have a look?

[jgm]

Here's a simpler demonstration of the bug, without .. table:

 % pandoc -f rst
===  =====  ====  =====  ========  ========================
int  float  str   bool     mix               time
===  =====  ====  =====  ========  ========================
  0   0.10  hoge  True          0  2017-01-01 03:04:05+0900
  2  -2.23  foo   False            2017-12-23 12:34:51+0900
  3   0.00  bar   True   Infinity  2017-03-03 22:44:55+0900
-10  -9.90        False       NaN  2017-01-01 00:00:00+0900
===  =====  ====  =====  ========  ========================
^D
<p>=== ===== ==== ===== ======== ======================== int float str
bool mix time === ===== ==== ===== ======== ======================== 0
0.10 hoge True 0 2017-01-01 03:04:05+0900 2 -2.23 foo False 2017-12-23
12:34:51+0900 3 0.00 bar True Infinity 2017-03-03 22:44:55+0900 -10
-9.90 False NaN 2017-01-01 00:00:00+0900 === ===== ==== ===== ========
========================</p>

[jgm]

I created a new issue, #11150 to track this bug.