Pass VFile Data Back

[yordis]

Initial checklist

• I read the support docs
• I read the contributing guide
• I agree to follow the code of conduct
• I searched issues and couldn’t find anything (or linked relevant results below)

Subject

Pass the VFile data back to mdx-bundler

Problem

First, my apologies if I sound a little bit all over the place, has been one night and way too many tools and things that I don't fully comprehend.

I am using mdx-bundler, with the following rehype plugin @stefanprobst/rehype-extract-toc here: https://github.com/straw-hat-team/adr/blob/9171dd779c13ff86caf951ac33c01d4cf9fc3d57/website/src/pages/adrs/%5Bslug%5D.tsx#L30

@stefanprobst/rehype-extract-toc does the following https://github.com/stefanprobst/rehype-extract-toc/blob/6136d102bba79a2457a3749e193e913a4e3b6ee2/src/index.js#L13

vfile.data.toc = createTree(headings) || []

Such code adds toc to the VFile data.

I traced the code, and until here:

xdm/lib/integration/esbuild.js

Line 135 in 051b49f

value = file.value

there vfile.data.toc is there.

But then we don't do anything with, here:

xdm/lib/integration/esbuild.js

Line 202 in 051b49f

return {contents: value, errors, warnings, resolveDir: p.cwd()}

So the data is lost, therefore I didn't get the vfile.data in my compiled output in mdx-bundler here https://github.com/straw-hat-team/adr/blob/9171dd779c13ff86caf951ac33c01d4cf9fc3d57/website/src/helpers/mdx.ts#L99

Solution

Pass somehow vfile.data back, I think we need to update the contents to be a JSON with two keys at least, so we can pass that information as well.

Alternatives

🤷🏻 don't understand much, I am not sure

[wooorm]

Hey!

vfile is an object that maintains metadata about a file, for example, the path to where it exists on disk.
esbuild’s contents is the string value of a file.
So I think the proposed solution—put metadata into the file’s contents—isn’t the right way to go about it. Because well, we don’t put the path to the file into the file either.

And the file contains mangled JavaScript — you probably don’t want that. You want an array or so?

What I think is a good solution, would be to somehow add a new export to the file.

export const tableOfContents = { … }

export default function MDXContent() { … }

Things like these are possible through plugins, in userland.

Here is one similar example: https://github.com/remcohaszing/rehype-mdx-title/blob/main/src/index.ts. It adds an export for the title.
Another one is this: https://github.com/remcohaszing/remark-mdx-frontmatter/blob/main/src/index.ts. It turns YAML frontmatter into an export.

I think it would be quite useful for other people to have such a plugin!

[yordis]

Let me see if I can make it to work ... honestly, I have no clue what I am doing.

Should I make it to work over JS-DevTools/rehype-toc#8 since that is what most people seem to be using?

[yordis]

Doing what I can, stefanprobst/rehype-extract-toc#2

But I am just copy-pasting code, and try-and-error until works, any mentoring is welcome ❤️

[yordis]

I am very lost now, I am using rehype-mdx-title but I don't see the metadata anywhere 🤔

https://github.com/straw-hat-team/adr/blob/71e79025567d0389ffbb1e73a05b7ff1b04b9c23/website/src/pages/adrs/%5Bslug%5D.tsx#L29

[yordis]

Also, how can we extract the information without having to inject it back to some output? The intention is not to expose it but collect it thou (but I digress I take anything at this point)

[yordis]

(Sorry I am spamming here, learning in the public)

I think I am using the wrong tooling and/or there is some ecosystem context that I am missing.

From one side, definitely handle exporting things as part of the file, on the other hand, I want to extract the ToC.

And lastly, mdx-bundler abstraction makes it harder ... just lost in way too many tools 😭

Why would I keep adding plugins to mdx-bundler that extract things or whatever when it seems that doesn't work.

So, do I suppose to include only plugins that will help xdm to generate the component directly and that should be it?

[rehypeMdxTitle, { name: 'IMAHERE' }]

Not working, for example

[ChristianMurphy]

I think I am using the wrong tooling and/or there is some ecosystem context that I am missing.

I'd highly recommend reading https://unifiedjs.com/learn/ if you haven't already, it introduces the ecosystem and some of the common patterns.
As well as the MDX 2 docs https://v2.mdxjs.com/ which introduce many MDX specific concepts.

More broadly your question increasingly sounds like an XY problem.
It would be good to take a step back, and explain what your goal is with TOC and how you want to use it.

Reading between the lines of your comments, you want the TOC external to the component being generated? Something like a sitemap? If so could you expand on how you want to use this information external to the component?

---

I am very lost now, I am using rehype-mdx-title but I don't see the metadata anywhere

MDX can share values through exports, https://github.com/remcohaszing/rehype-mdx-title#usage
These export can be used during runtime.
For example if used from a loader

import MDXContentCompoent, { title } from './example.mdx'

[yordis]

@ChristianMurphy definitely reading as much as I can right now, honestly speaking, I read many things before thou, but way too many things to figure out.

import MDXContentCompoent, { title } from './example.mdx'

That totally makes sense, I think mdx-bundler abstraction is what is making it harder for me to follow since I lost the context of having the AST produced directly and I am just watching compiled JS code.

I am so sorry folks, I am trying to learn.

Reading between the lines of your comments, do you want the TOC external to the component being generated? Something like a sitemap? If so could you expand on how you want to use this information external to the component?

Right, I need the TOC externally from the MDXContentComponent because I need to render it in a particular place outside MDXContentComponent, an example page: https://google.aip.dev/122

So I need to extract the frontmatter, and the TOC.

---

If you don't mind helping me thou, should I drop mdx-bundler? At some point I switched because of some blog posts. But then they don't give me back things like rehype-mdx-title as something I can work with.

btw, I do see the rehype-mdx-title information being injected, but still, it is a string, so I will have to eval or do something of that nature in order to use it externally.

Sorry folks, really appreciate your patience and support.

[ChristianMurphy]

Right, I need the TOC externally from the MDXContentComponent because I need to render it in a particular place outside MDXContentComponent

Would importing the data, and feeding it to your table of contents component, similar to:

import MDXContentCompoent, { title, tableOfContents } from './example.mdx'

<TableOfContentsComponent tableOfContents={tableOfContents} />

work?

---

should I drop mdx-bundler?

Again, understanding the why is important, this is another XY question.
It is likely possible either with or without mdx-bundler, what's easier depends on your goal and your stack.
For example:

At some point I switched because of some blog posts.

What did you switch from? Why did you switch? Is there a feature from mdx-bundler that you want or need for your solution?

---

so I will have to eval or do something of that nature in order to use it externally.

Right, It's a JavaScript file as a string, it needs to be evaluated for it to run.

[yordis]

Yeah ... let me go back to .mdx files directly because I definitely understand now, just that the string in between from mdx-bundler screwed me over.

Honestly, I think I talked myself from mdx to mdx-bundler due to "performance" (meh, I don't even have an issue), and that I think I didn't know about import {compile} from '@mdx-js/mdx' ... I think ... one of those that you hear esbuild, fast, and the sell-out of simpler LOL (that one is on me thou, no pointing singers other than myself)

Let me reset, and focus on the goal, hopefully, I can make it work now since I start to understand a bit better, definitely that v2 website is helping for sure

[yordis]

Things like, https://github.com/kentcdodds/mdx-bundler#:~:text=mdx%2Dbundler%20is%20an%20MDX,built%20version%20of%20your%20files.

"Why does this use XDM instead of @mdx-js?"
It has more features, fewer bugs, and no runtime!

Which @wooorm you are part the mdx crew, so maybe you have some feedback on the topic? maybe that is not true anymore with mdx-js v2?

[wooorm]

@yordis I understand you mean well, but there’s a couple of things going wrong here:

• You’ve been working on this for 2 days now, it’s still unclear what you actually want to make. please respect your own time (this may help: https://rubberduckdebugging.com)
• You’re jotting down notes as you work here, asking lots of questions that you later realise the answers to, or realise that you don’t need an answer to. I now come in to this thread and like, I can take 2 hours to write down answers to all your questions, which you don’t need anymore? That’s a really big ask. please respect our time.
• You are asking a bunch of questions that you can find by reading before doing. There’s a pinned issue about xdm/mdx here: Question: What is the roadmap for XDM and how does it relate to MDX 2? #94.
• In the first post I already gave you an answer: make a plugin based on two examples. If you don’t know how to do that, read about plugins, try stuff out, get an understanding, look at examples, read docs. And if you’re having troubles, then finally ask specific questions.

[yordis]

You’re jotting down notes as you work here, asking lots of questions that you later realise the answers to, or realise that you don’t need an answer to. I now come in to this thread and like, I can take 2 hours to write down answers to all your questions, which you don’t need anymore? That’s a really big ask. please respect our time.
You are asking a bunch of questions that you can find by reading before doing. There’s a pinned issue about xdm/mdx here: Question: What is the roadmap for XDM and how does it relate to MDX 2? #94.

@wooorm sorry, something got lost, as I mentioned before, trying to learn in public to let a trace of my failures rather than keeping them for myself for the next person that needs things.

I am not demanding anything from anyone as far as I can tell, unless I made a mistake, my apologies if so, but you are entitled to your time, and whatever you decide to do with it it is up to you.

You are free to ignore the thread and unsubscribe, otherwise, if you wish me to stop sharing thins here I will since it is your space.

For what is worth, thank you so much for your patience and support. I will figure it out.

[wooorm]

I think the quite vocal learning in public here in issues caught me off guard. I have no problem with you learning in public, but I don’t think an issue here on GitHub is the right place. Perhaps a blog post / digital garden might be the right place?

I see issues as a place to talk about actionable things: if there’s no need to change something in the respective project, the conversation is over/moves somewhere else.

This has in part to do with how GitHub and related things is set up: I get notifications any time, anyone comments, on any of the hundreds of projects I maintain, and notifications/issues are something to be solved/closed, otherwise projects are treated as unmaintained.
This also has to do with how I treat issues. But it’s possible that you’ll find other open source maintainers treating them the same.

[yordis]

All good, no need for explanation, it is totally reasonable and understandable where you are coming from, I made a mistake. Will not happen again.

Cheers,

[max-pod]

Hey @yordis, I've got a similar use-case and trying to follow your tracks.
I saw that you did some work /w https://github.com/stefanprobst/rehype-extract-toc -- is there anywhere else I should look?