Serializing Structs into Network Data Packets

[hgupta12]

Presently, Scalpel handles the deserialization of network packets across multiple protocols. However, to offer sculpting functionality, it's imperative to extend Scalpel's capabilities to support the serialization of Rust structs into network packets.

[gabhijit]

It would be great to have a functionality that can parse the inputs in the scapy format for example something like

 pkt = Ether() / IP(dst="github.com") / ICMP()

Initially, it's fine if the DNS resolution is not supported. Also, may be we can even start with a json format.

There might be additional changes required in the Packet structure to account for padding bits/bytes.

the Layer trait should also include a function to_bytes that returns a Vec<u8> for a given layer.

Some experimental code would be good to start.

[hgupta12]

Yes sir. I'll start looking into how to implement to_bytes for each layer.

[gabhijit]

I think most of it is kinda straight forward. A bit harder part is - things like checksum. As long as checksum is within the same layer it's not a problem (we always have it locally available with that particular structure). If checksum is based on some inner layer, this becomes a bit tricky, but we'll see that in a bit.

Also, it's probably better to have Vec<Vec<u8>> and later flatten it into the wire format, where the inner Vec<u8> is for a given layer.

Will be slightly tricky, but once we get it right, it will be very useful functionality.

May be you can start a draft PR and mention this discussion and let's see the mileage.

[aadilshabier]

I was thinking of a design similar to this, what do you think?

let eth_packet = Ethernet{...};
let ipv4_packet = IPv4{...};
let builder: PacketBuilder = PacketBuilder::new()
                   .with(eth_packet)
                   .with(ipv4_packet)
                   .with_bytes(&[0x12, 0x32, ...]);

let result: Vec<Vec<u8>> = builder.build().unwrap();

[gabhijit]

My opinion about this is - neutral. The builder idea is a good one, but then it would be probably a good idea to have a builder for every layer. Which kind of also makes sense since we already have a #[derive(Default)].

I was thinking something along the lines of the following from scapy.

let pkt_data = "ETH()/IP()"

let pkt = Packet::parse(pkt_data)?;

let to_wire = pkt.to_bytes();

let round_trip = Packet::from_bytes(to_wire)?;
assert!(round_trip == pkt);

The Packet::parse could probably call parse for each 'Layer'. Something like Ethernet::parse looks like -

impl Ethernet { 
    fn parse(input: &str) -> Result<Self> { 
      let mut eth = Ethernet::default();
      // ... split by , and then split by = and update the default 
      // This is where we *may* use builder as above. 
      Ok(eth)
    }
}   

This would mean - we will start 'parsing' the scapy packet strings. Let me know what you all think.
Later, we'll see if we can implement a REPL a-la scapy.

[hgupta12]

How do we go about the parsing of scapy packet strings? We should probably scope to one or two protocols as POC. Which ones do you think would be a good start?

[gabhijit]

As I see, there are two parts two solving this.

1. Implementation of to_bytes for each of the Layer. One of you can start with that for Ethernet, IPv4 and TCP with some payload.

2. The other person can then start with the parse function as discussed above. We should start with something very basic first, that is let's not try to support all the fancy features that are supported by scapy like name resolution. Also, if possible we should be able to handle the correct ETHERTYPE ,IP protocols etc. from the input (ie. ETH()/IP() and ETH()/IP6() should use proper ETHERTYPE and similarly for IP protocols.

[hgupta12]

Can I take up the implementation of parse function? Are there any crates we can use for the lexer and parser? Something similar to flex and yacc in C.

[gabhijit]

There are crates like lex and yacc (I forgot the name), but we can write a simple parser. Probably a good idea to take a look at how scapy implements it.

I believe nom might be useful here.

You can write everything by hand - First 'split' the string using "/" and get each of the layers. May be you can have a "global-map" with keys as "Ethernet", "IP" etc. and values as some layer creator function. The parse function may need to peek into next layer, to determine the demux type (like say ETHERTYPE).

Let's do some experimentation (even a couple of you can give it a shot or work-together) and then we'll see how to converge.

[aadilshabier]

I think the parse function can internally use the builder.

fn parse(input: &str) -> Result<Vec<Vec<u8>>> {
    let tokens = ...;
    let builder: PacketBuilder = PacketBuilder::new();
    for token in tokens {
        builder = builder.stack(to_layer(token));
    }
    return builder.build();
}

PacketBuilder.build() calls Layer::to_bytes() for each layer. To implement checksum, packet length easier, me and Harshit felt it might make sense to build from the topmost layer first.

[gabhijit]

@aadilshabier : Yes it makes sense to experiment with a couple of approaches and see which works.

I was looking at the scapy. What scapy is essentially doing is - it's using the Python's call syntax and overridden __rdiv__ method. Thus Ether() / Ip() - would call something like Packet.__div__(Ether, IP) (Not strictly - but I hope you get the point.)

I believe that's a bit too much to emulate. Let's look at builder and a parser approach (or may be combined) and see how it goes.

It's been a while we are discussing 'ideas' probably, let's look at some implementation and reason about it? Likely we may have to iterate over those a couple of times, before we get it right.

What do you all say?

[aadilshabier]

Yes, I've started working on a crude implementation, and will make a draft PR in a while.

[aadilshabier]

Draft PR related to the discussion: #65

[gabhijit]

This is merged into master! I believe this is a good start for this functionality. Thanks @aadilshabier for this.

The final implementation may deviate from this once we start ironing out issues, but this is certainly a very good start.

[gabhijit]

#70 : Makes sculpting an Opt-In Feature. Till this is usable enough let's keep this opt-in. Makes Sense?

[aadilshabier]

Hey @gabhijit, I was looking at the checksum problem for TCP/UDP once again, and I wanted to to come up with a generalized solution. For that I wanted to understand whether there exists any similar relations to lower level protocols for calculating fields.
In the TCP/UDP case, the relation is just from layer n to n+1, are there other less popular protocols (maybe something used for virtulization) where there is an n to n+2 or even n+3 relation?