Looking proc_macro2 source as guidence for implementing ParseSlice on custom token type.

[devdave]

I am trying to implement tracing for my token system but I can't figure out how to implement ParseSlice to support the $() operator.

My code is roughly like ->

enum TType {
    Start,
    Name,
    Operator,
    etc...
    Stop,
}

struct Token {
     r#type: TType,
    Position: start,  // Holds Line and column
    Position: end,   //ditto
    text: String,  //The actual raw token content
}

TType has a few more elements/values but that is the gist of it. In peg-macros at https://github.com/kevinmehall/rust-peg/blob/master/peg-macros/tokens.rs what is the structure/definition of TokenStream and Span which are imported from proc_macro2 which doesn't seem to be in the repo?

For reference the current parser rule set is here -> https://github.com/devdave/rython4/blob/d33e78316497d73f7f48a460491104463a509f27/src/parser/grammar.rs I think I am missing a rule to consume multiple NL's ("\n") but I am not sure.

Aside from my question, peg-rust is a great contribution to the Rust & open source ecosystem.

[kevinmehall]

peg_macros/tokens.rs is a little complicated because it's translating between the tree-structured tokens that rustc uses and a flat array of tokens that rust-peg expects. TokenStream is the Rust compiler's token implementation used in procedural macros. proc-macro2 is the top of several abstraction layers over the implementation in the Rust compiler internals, but it comes with a fallback implementation that might be easier to understand if you're curious how it works. However you don't need any of that to use rust-peg unless you're trying to use it to implement a procedural Rust macro.

It looks like you have a Vec<Rc<Token>>, so the better example to look at would be the implementation for [T]. You probably just want to take a slice of your Vec and return &'input [Rc<Token>].

[devdave]

Thank you Kevin for the pointer in the right direction. I got tracing to work and hopefully after I do some refactoring I will be able to track down where my mistake is in my rule set/grammar that's causing the parse error.