PCAP File Rewriting and Traffic Replaying with Network Expect

Two common tasks that we all have to eventually deal with are PCAP file rewriting and traffic replaying. PCAP file rewriting consists of opening a PCAP file, reading each packet in the file, perform some conversion, and then write the converted packets to another file. Traffic replaying is similar to PCAP file rewriting but sends converted/processed packets to the network instead of to another file.

Network Expect can help with some of these PCAP file rewriting and replaying situations, and this page contains some examples and comments about how Network Expect can help with these tasks.

Traffic Anonymization

Sometimes we have a PCAP file that contains traffic that we would like to show to somebody else but we can't share as it is because it would disclose details about the topology of the network where the packet capture took place. What we have to do in these cases is to re-write the original PCAP file so the original IP addresses are replaced with some other dummy IP addresses, like RFC 1918 addresses. The following script just does that:

# Provides a new unique IPv4 address per unique IPv4 address
proc get_v4_addr {v4_addr} {
    global v4_dict base_ip

    if {[info exists v4_dict($v4_addr)] == 0} {
        set v4_dict($v4_addr) [ipaddr next base_ip]
    }

    return $v4_dict($v4_addr)
}

if {$argc != 2} {
    puts "Usage: $argv0 <input file> <output file>"
    exit 1
}

set base_ip [ipaddr new 192.168.1.1++]

# -fullspeed: read from PCAP file at full speed; do not preserve original
# packet arrival times.
# -r ...: input PCAP file
# -w ...: output PCAP file
# "ip": PCAP filter so we only read IP packets
spawn_network -fullspeed -r [lindex $argv 0] -w [lindex $argv 1] ip

expect_network {1} {
    send_network -fixups "
        ether()/
        ip(cksum = 0 /* don't waste time calculating checksums;
                        they will be fixed up post-build */,
           src=[get_v4_addr ${ip.src}],
           dst=[get_v4_addr ${ip.dst}],
           proto=${ip.proto} /* need to force IP proto */)/
        data('[barray slice _(packet) $pdu(3,offset)l$pdu(3,tot_len)]')"
    nexp_continue   ;# Keep reading input file
} eof {
    # It's necessary to have an "eof" case to be able to exit the
    # expect_network statement
}

close_network nexp0     ;# close output file

puts "IPs were anonymized:\n"

foreach oldip [array names v4_dict] {
    puts "$oldip is now $v4_dict($oldip)"
}

Here's how this works: the function get_v4_addr takes as argument one IP address, and returns a new IP address that is always the same for the same IP address passed as argument to the function. It uses a base or starting IP address, which is set to 192.168.1.1 and that increments with each new IP address passed to get_v4_addr.

The spawn_network statement opens both the existing PCAP file (which we want to anonymize) and the new PCAP file (which will have the anonymized packets.)

The main loop is performed by the network_expect command. What makes the statement a loop is the nexp_continue statement in the first instruction body (which corresponds to the expect case '1', i.e. always true.)

For each packet read by the expect_network command we execute a send_network command to write the packet to the output file. It is in this send_network command where the anonymization takes place - note that we create a new IP PDU (with "ip(...)") and for the source and destination IP addresses we call the get_v4_addr function passing it the IP address in the original packet. Note that the send_network command doesn't worry about the payload of the IP packet - it just appends the original payload to the new IP PDU with "data(...)". Since changing IP addresses will render the original layer 4 checksums invalid we must use the "-fixups" flag of the send_network command so layer 4 checksums are recalculated and fixed.

Local Versus Remote Addresses

The astute reader will notice that the above script has what could be considered a flaw, depending on the needs and the situation: since all addresses being anonymized are converted to addresses that start at the base address 192.168.1.1, all anonymized addresses will end very likely in the same IP subnetwork. If the traffic being anonymized traverses a layer 3 device, and the objective of the exercise is to anonymize to show someone else how that layer 3 device is behaving, then the anonymization process is removing important information that allows to determine who is local and who is remote.

This problem can easily be fixed by making a minor change to the get_v4_addr function so it uses the NetExpect Tcl function islocal:

proc get_v4_addr {addr} {
    global v4_dict local remote localnet

    if {[info exists v4_dict($addr)] == 0} {
        if {[islocal $localnet $addr] != 0} {
            set v4_dict($addr) [ipaddr next local]
        } else {
            set v4_dict($addr) [ipaddr next remote]
        }
    }

    return $v4_dict($addr)
}

islocal receives as a parameter a network address (a.b.c.d/nn or a.b.c.d/nn.nn.nn.nn) and an IP address. It will return 1 if the IP address is local to the network address or 0 if it is remote.

The new get_v4_addr uses three new global variables, local, remote, and localnet, defined as follows:

set localnet 10.83.4.175/255.255.255.240
set local [ipaddr new 192.168.1.1++]
set remote [ipaddr new 172.16.0.1++]

localnet defines what the local IP network is. local and remote are the address pools that will be used as the new IP addresses after the anonymization process.

Converting an IPv6 Capture to an IPv4 Capture

Let's say we have a PCAP file with some TCP over IPv6 traffic and we want to convert that traffic to TCP over IPv4 traffic, possibly for traffic replaying later. The following script will do that. Each IPv6 address is replaced by a 192.168.1.x IPv4 address. The script is very similar to the anonymizing script above:

Note: this script is based on a Scapy (Python) script written by Shiva Persaud <shivapd at cisco dot com> from Cisco to solve a real-world problem. Special thanks to Shiva for sharing his script with me so I could make sure Network Expect is able to tackle this problem.

# Provides a unique IPv4 address per unique IPv6 address
proc get_v4_addr {v6_addr} {
    global v4_dict base_ip

    if {[info exists v4_dict($v6_addr)] == 0} {
        set v4_dict($v6_addr) [ipaddr next base_ip]
    }

    return $v4_dict($v6_addr)
}

if {$argc != 2} {
    puts "Usage: $argv0 <input file> <output file>"
    exit 1
}

set npackets 0

set base_ip [ipaddr new 192.168.1.1++]

# -fullspeed: read from PCAP file at full speed; do not preserve original
# packet arrival times.
# -R tcp: Wireshark display filter to read only packets carrying TCP
# -r ...: input PCAP file
# -w ...: output PCAP file
spawn_network -fullspeed -r [lindex $argv 0] -w [lindex $argv 1] -R tcp

expect_network {1} {
    send_network -fixups "
        ether()/
        ip(cksum = 0 /* don't waste time calculating checksums;
                        they will be fixed up post-build */,
           src=[get_v4_addr ${ipv6.src}],
           dst=[get_v4_addr ${ipv6.dst}],
           proto=6 /* need to force IP proto */)/
        data('[barray slice _(packet) $pdu(4,offset)l$pdu(4,tot_len)]')"
    incr npackets
    nexp_continue   ;# Keep reading input file
} eof {
    puts "$npackets packets processed."
    close_network nexp0 ;# close output file
}

Converting an IPv4 Capture to an IPv6 Capture

Someone asked me how to do the opposite of the above, i.e. convert an IPv4 packet capture to an IPv6 packet capture. The process is similar. However, because we currently lack a way to generate IPv6 objects that can auto-increment, we need to manually link IPv4 and IPv6 addresses. In other words, since we cannot do something like this today:

set base_ip6 [ip6addr new fe80::21f:e2ff:fe1a:7380++]
...
set v4to6_dict($v4_addr) [ip6addr next base_ip6]

we have to do this:

array set v4to6_dict {
    192.168.1.10    fe80::21f:e2ff:fe1a:7380
    172.18.104.134  fe80::21f:e2ff:fe1a:7381
}

(Adding IPv6 object functionality should not be hard, and should come at some point.)

Here's a sample script to do a conversion of TCP over IPv4 traffic in a PCAP file to TCP over IPv6 traffic:

if {$argc != 2} {
    puts "Usage: $argv0 <input file> <output file>"
    exit 1
}

set npackets 0

array set v4to6_dict {
    192.168.1.10    fe80::21f:e2ff:fe1a:7380
    172.18.104.134  fe80::21f:e2ff:fe1a:7381
}

# -fullspeed: read from PCAP file at full speed; do not preserve original
# packet arrival times.
# -R tcp: Wireshark display filter to read only packets carrying TCP
# -r ...: input PCAP file
# -w ...: output PCAP file
spawn_network -r [lindex $argv 0] -w [lindex $argv 1] -R tcp

expect_network {1} {
    if {![info exists v4to6_dict(${ip.src})]} {
        puts "No IPv6 address for ${ip.src}"
        nexp_continue
    }

    if {![info exists v4to6_dict(${ip.dst})]} {
        puts "No IPv6 address for ${ip.dst}"
        nexp_continue
    }

    send_network -fixups "
    ether()/
    ip6(
       src=$v4to6_dict(${ip.src}),
       dst=$v4to6_dict(${ip.dst}),
       next-header=6 /* need to force IP proto */)/
    data(data = '[barray slice _(packet) $pdu(3,offset)l$pdu(3,tot_len)]')"
    incr npackets
    nexp_continue   ;# Keep reading input file
} eof {
    puts "$npackets packets processed."
    close_network nexp0    ;# close output file
}

RewriteAndReplay (last edited 2011-04-29 19:11:57 by EloyParis)