footnote XML output contains <<unknown>> tags

[zkamvar]

(edit: not sure how the reprex lost its formatting, but I fixed it)

The new footnotes feature might be useful for us at {tinkr}, but I'm not sure how to parse them as each footnote contains identical tags.

from: ropensci/tinkr#92 (comment)

txt <- c("a statement[^1][^2]\n", "[^1]: this is true", "[^2]: this is false")
commonmark::markdown_xml(txt, footnotes = TRUE) |> writeLines()
#> <?xml version="1.0" encoding="UTF-8"?>
#> <!DOCTYPE document SYSTEM "CommonMark.dtd">
#> <document xmlns="http://commonmark.org/xml/1.0">
#>   <paragraph>
#>     <text xml:space="preserve">a statement</text>
#>     <<unknown> />
#>     <<unknown> />
#>   </paragraph>
#>   <<unknown>>
#>     <paragraph>
#>       <text xml:space="preserve">this is true</text>
#>     </paragraph>
#>   </<unknown>>
#>   <<unknown>>
#>     <paragraph>
#>       <text xml:space="preserve">this is false</text>
#>     </paragraph>
#>   </<unknown>>
#> </document>

^{Created on 2023-03-22 with reprex v2.0.2}

[jeroen]

Ew that's weird. Posted it upstream here: github/cmark-gfm#316

[zkamvar]

I've added a potential fix for this in github/cmark-gfm#362, but I'm not sure how active the maintainers are there.

[yihui]

I feel there is no hope that they would ever merge the fix upstream. @jeroen Do you think it will be easy enough to apply @zkamvar's patch to the commonmark R package here? Of course, I mean in a maintainable way (e.g., git apply after pulling cmark-gfm), not just as a one-time manual job.

[jeroen]

Yes we can do this. It looks like the upstream project is stale, sadly.

[jeroen]

Can you confirm your issue has been fixed? Could you maybe help me by adding a unit test to this package to ensure we do not regress in the future?

[zkamvar]

Thank you! I'll submit a PR with the tests by the end of the week 🤞🏽

[zkamvar]

And I have confirmed this works:

txt <- c("a statement[^1][^2]\n", "[^1]: this is true", "[^2]: this is false")
commonmark::markdown_xml(txt, footnotes = TRUE) |> writeLines()
#> <?xml version="1.0" encoding="UTF-8"?>
#> <!DOCTYPE document SYSTEM "CommonMark.dtd">
#> <document xmlns="http://commonmark.org/xml/1.0">
#>   <paragraph>
#>     <text xml:space="preserve">a statement</text>
#>     <fnref id="fnref-1" destination="fn-1" />
#>     <fnref id="fnref-2" destination="fn-2" />
#>   </paragraph>
#>   <fn id="fn-1">
#>     <paragraph>
#>       <text xml:space="preserve">this is true</text>
#>     </paragraph>
#>   </fn>
#>   <fn id="fn-2">
#>     <paragraph>
#>       <text xml:space="preserve">this is false</text>
#>     </paragraph>
#>   </fn>
#> </document>

^{Created on 2024-10-08 with reprex v2.1.1}

[yihui]

I think I still need another patch (github/cmark-gfm#314), otherwise there will be no way for me to identify footnotes in LaTeX output or post-process it to turn footnotes into \footnote{}. I'll take a closer look and open a new issue. Thanks!

P.S. I've also verified that the XML issue has been fixed.