Issues with Blocks extensions and `md_in_html`

[lmartellotto]

Description

Hello!

Help, I'm lost 😅 I don't know if it's a bug or if I just don't write correctly my code.

Context: in a .md file in a mkdocs project where pymdown-extensions is used.
It seems impossible to put HTML content in a Block Tab which is itself in a HTML div. See below the steps.

I need some help 🥺
Thanks!

Minimal Reproduction

1. Block Tabs containing HTML tags are OK:

/// tab | STRONG
<strong>Hello i'm in a tab!</strong>
///

/// tab | P
<p>Hello i'm a p in a tab!</p>
///

Result:

<div class="tabbed-set tabbed-alternate" data-tabs="1:2" style="--md-indicator-x: 0px; --md-indicator-width: 95px;">
  <input checked="checked" id="__tabbed_1_1" name="__tabbed_1" type="radio">
  <input id="__tabbed_1_2" name="__tabbed_1" type="radio">
  <div class="tabbed-labels tabbed-labels--linked"><label for="__tabbed_1_1"><a href="#__tabbed_1_1" tabindex="-1">STRONG</a></label><label for="__tabbed_1_2"><a href="#__tabbed_1_2" tabindex="-1">P</a></label></div>
  <div class="tabbed-content">
     <div class="tabbed-block">
        <p><strong>Hello i'm in a tab!</strong></p>
     </div>
     <div class="tabbed-block">
        <p>Hello i'm a p in a tab!</p>
     </div>
  </div>
  <div class="tabbed-control tabbed-control--prev" hidden=""><button class="tabbed-button" tabindex="-1" aria-hidden="true"></button></div><div class="tabbed-control tabbed-control--next" hidden=""><button class="tabbed-button" tabindex="-1" aria-hidden="true"></button></div>
</div>

2. HTML div containing HTML tags are OK:

<div class="my-div" markdown>
<strong>Hello i'm in a simple div!</strong>
<p>Hello i'm a 'p' in a simple div!</p>
</div>

Result:

 <div class="my-div">
 <p><strong>Hello i'm in a simple div!</strong></p>
 <p>Hello i'm a 'p' in a simple div!</p>
 </div>

3. HTML div containing Block Tabs containing HTML tags are NOT OK:

<div class="my-div" markdown>

/// tab | :desktop_computer: STRONG
<strong>Hello i'm in a tab which in in a div!</strong>
///

/// tab | :desktop_computer: P
<p>Hello i'm a 'p' in a tab which in in a div!</p>
///

</div>

Result:

 <div class="my-div">
   <div class="tabbed-set tabbed-alternate" data-tabs="1:2" style="--md-indicator-x: 0px; --md-indicator-width: 95px;">
     <input checked="checked" id="__tabbed_1_1" name="__tabbed_1" type="radio">
     <input id="__tabbed_1_2" name="__tabbed_1" type="radio">
     <div class="tabbed-labels tabbed-labels--linked"><label for="__tabbed_1_1"><a href="#__tabbed_1_1" tabindex="-1">STRONG</a></label><label for="__tabbed_1_2"><a href="#__tabbed_1_2" tabindex="-1">P</a></label></div>
     <div class="tabbed-content">
        <div class="tabbed-block">
           <p><strong>Hello i'm in a tab!</strong></p>
        </div>
        <div class="tabbed-block">
        </div>
     </div>
     <div class="tabbed-control tabbed-control--prev" hidden=""><button class="tabbed-button" tabindex="-1" aria-hidden="true"></button></div><div class="tabbed-control tabbed-control--next" hidden=""><button class="tabbed-button" tabindex="-1" aria-hidden="true"></button></div>
   </div>
<p>Hello i'm a p in a tab!</p>
<p>///</p>
</div>

The <p> tag is outside the Block Tab, and I don't understand why. I've tried to add some space around <div> tags too.

Version(s) & System Info

• Operating System: MacOs & Linux
• Python Version: 3.8
• Package Version:
  • pymdown-extensions 10.14
  • mkdocs 1.6.1

[facelessuser]

@lmartellotto You are doing nothing wrong, the problem is Python Markdown and how it is handling things. Let me illustrate.

Here I have a simple example. Where I have nested HTML. I place /// 1 before the HTML and /// 2 after the HTML.

<div class="my-div" markdown>

/// 1

<p>
Hello i'm a 'p' in a tab which in in a div!
</p>

/// 2

</div>
"""

I added some print statements in the Block processor to print the lines as soon as it sees them. I get the following which shows the end is processed before the start.

test /// 2
test /// 1

When I remove the outer <div class="my-div" markdown></div>, I get the start first as expected, the placeholder for the processed HTML and then the end line.

test /// 1
test wzxhzdk:0
test /// 2

So the problem is when Python Markdown enters its special Markdown processing, and then encounters another block HTML element (inline seem to be fine), it can start processing Markdown out of order, which makes no sense to me. This breaks expectations.

So the Tab block in this case processes the end out of order, then later processes the beginning, but now can't find the end because Python Markdown handled it out of order. It causes everything to break down.

I would argue this needs to be fixed upstream in md_in_html, but, you could workaround it in a couple ways.

1. Enable pymdownx.blocks.html and then apply HTML that processes blocks like this. Notice nested fences use a different number of ///.

   //// html | div.my-div
   
   /// tab | Tab
   <p>Hello i'm a 'p' in a tab which in in a div!</p>
   ///
   
   ////
   

2. Or avoid using block HTML nested in the tab when wrapped by a md_in_html tag that is processing Markdown which causes things to break. That can be done by letting Markdown create the <p>.

   <div class="my-div" markdown>
   
   /// tab | Tab
   Hello i'm a 'p' in a tab which in in a div!
   ///
   
   </div>
   

So again, there is definitely a bug, but it exists due to how md_in_html is messing up the order of processing.

[facelessuser]

A Python Markdown issue has been created: #1502. There is no guarantee that it will be fixed as I am not sure of the limitations in the code yet, but I am part of that project and will try to take a look when I have time to see how hard a fix would be.

[lmartellotto]

@facelessuser Thanks a lot for your time!

Your solutions could be great, but unfortunately, my HTML code is pretty more complex than a simple <p> (I simplified for the issue). In reality, my HTML looks like:

<div class="mf-generated" markdown>
Hello I'm in a div which can contain markdown!
</div>

Yes, I hear you: "keep it simple..." 😅 I generate markdown pages (HTML+markdown content) and I include them with mkdocs-include-markdown-plugin in Blocks Tabs. Yeaah!

I tried to target the problem, and here i'm:

//// html | div.my-div

/// tab | TEST
<div class="mf-generated" markdown>
Hello I'm in a div which can contain markdown!
</div>
///

////

This code doesn't work because of markdown attribute in div. If I remove it, it works. I tried all values of markdown option for html block (https://facelessuser.github.io/pymdown-extensions/extensions/blocks/plugins/html/#usage).

Fun fact: I switch from the old pymdownx.details lib to your new blocks lib (which is much more convenient!) and this code worked:

??? tip "Procédure"

    /// tab | TEST
    <div class="mf-generated" markdown>
    Hello I'm in a div which can contain markdown!
    </div>
    ///

I tried with the new Details Block, and it doesn't work:

//// details | Some summary

/// tab | TEST
<div class="mf-generated" markdown>
Hello I'm in a div which can contain markdown!
</div>
///

////

Wrapping blocks with the new lib doesn't seem to like HTML which contain markdown... But, is is a bug or is it logic? 😅

[facelessuser]

I don't know what you mean, this works just fine.

//// html | div.my-div

/// tab | TEST
<div class="mf-generated" markdown>
Hello I'm in a div which can contain **markdown**!
</div>
///

////

<div class="my-div">
<div class="tabbed-set" data-tabs="1:1"><input checked="checked" id="__tabbed_1_1" name="__tabbed_1" type="radio" /><label for="__tabbed_1_1">TEST</label><div class="tabbed-content"></div>
</div>
<div class="mf-generated">
<p>Hello I'm in a div which can contain <strong>markdown</strong>!</p>
</div>
</div>

Please be clear with your examples, if you are oversimplifying the examples, and they don't reproduce, I can't offer any guidance.

[lmartellotto]

It doesn't work because my <div class="mf-generated" markdown> is not contain in the <div class="tabbed-content"></div> :/

In your result, the <div class="mf-generated"> is after the <div class="tabbed-content"></div>.

The correct expected result is:

<div class="my-div">
  <div class="tabbed-set" data-tabs="1:1">...
    <div class="tabbed-content">
      <div class="tabbed-block">
        <div class="mf-generated" markdown="" style="border: medium; padding: 0px;">
          Hello I'm in a div which can contain markdown!
        </div>
      </div>
    </div>
  </div>
</div>

[facelessuser]

Okay, let me look a bit closer. I see what you are showing now.

[facelessuser]

There's something very wrong with md_in_html. It seems to be altering normal flow of Markdown. It may just be another upstream issue. In this case, the content is in order, but the reporting is weird and somehow not flowing through the blocks as intended. I'll be digging in, but it may take me a few days to figure out how md_in_html is messing with the process and what can possibly be done.

[facelessuser]

Okay, I better understand what Python Markdown is doing. md_in_html will create the block element with no parent inside the tree, essentially an orphan, and use that as the parent for processing Markdown under it. Then, it inserts the element after the Block processor has done its work. So it is inserted in the wrong location.

So, this is a problem for our extension as it gets a parent with no ancestry. We can't immediately determine where it is supposed to go. With a little code change, we can guess okay. But even if we do, md_in_html will still insert the created element in the wrong location. Notice the empty div and notice that we are not wrapped in it. We captured the content, but not the parent element.

<div>
<div class="tabbed-set tabbed-alternate" data-tabs="1:1"><input checked="checked" id="__tabbed_1_1" name="__tabbed_1" type="radio" /><div class="tabbed-labels"><label for="__tabbed_1_1">what</label></div>
<div class="tabbed-content">
<div class="tabbed-block">
<p>Hello I'm in a div which can contain <strong>markdown</strong>!</p>
</div>
</div>
</div>
<div></div>
</div>

So right now, we are fighting with md_in_html to try and do the right thing.

Earlier you stated that the old details works fine, but it doesn't, it just breaks in a different way. If you look at the output, it will wrap your <div markdown> with a <p> tag. This is because it is completely confused and ignores the markdown. It is basically treating the content as inline and wrapping it with a paragraph, which it should not for a <div>.

This was one of the reasons (not the only reason) why I wanted to do the new block format because md_in_html was broken inside admonitions, details, tabs, etc. It leads to various quirks. Unfortunately, md_in_html still causes chaos in a different way with the new blocks. I can see what it is doing, but I don't have an immediate solution yet.

It should be noted that the author of md_in_html has stated numerous times that only HTML at the root level of a document is supported. While this is technically at root level (indentation-wise), the HTML is nested under a block construct in Markdown.

Anyway, I need to see if there is some clever trick I can do or not. It may be that something could be done in md_in_html, but that would be an upstream change.

[facelessuser]

There's a possibility that if we bump handling before md_in_html we might avoid some of the issues. This would then mean we need to handle indentation which was previously handled for us, but it just might help us avoid some of the issues.

[facelessuser]

In short, there appears to be two issues.

1. In md_in_html elements lines can be processed in reverse which can break logic.

2. md_in_html is processed before Blocks in general and can cause insertion of content to be done in the wrong location.

Initial tests show that moving the Blocks processor before md_in_html can solve 2) as it will allow the Blocks processor to properly place the content before md_in_html performs any shenanigans.

Unfortunately, this will not solve 1). This would need to be addressed in md_in_html.

---

Currently, md_in_html follows the rules specified as mentioned in this comment Python-Markdown/markdown#1427 (comment).

The reason for this behavior is based on the original Markdown rules, which state in part (emphasis added):

The only restrictions are that block-level HTML elements — e.g. <div>, <table>, <pre>, <p>, etc. — must be separated from surrounding content by blank lines, and the start and end tags of the block should not be indented with tabs or spaces.

The md-in-html extension relies on the standard rules for identifying what is considered a "block-level" element. If you want to diverge from the standard rules, then you need to look at third-party extensions.

Basically, it is designed with this in mind, meaning that when md_in_html was developed, it didn't really consider it could be nested under other constructs as fenced Markdown blocks did not exist at the time like Blocks implements.

In the short term, until a fix is fully explored and implemented, I think success with nesting md_in_html blocks under other constructs, whether old style !!! "Details" or new style /// details will have issues one way or the other, some may be more severe depending. The most surefire way to avoid such issues is to use something like the following.

/// HTML div.myclass
//// tab | Tab
///// html | div.my-other-class
Content
/////
////
///

Maybe not ideal, but workable if absolutely needed.

I'll be looking into a way to fix 2). And when I get time, I'll look into fixing 1) in Python Markdown directly (if possible).