An Interactive Session with Network Expect

While NetworkExpect is usually used via scripts, it can also be used interactively by issuing commands from a prompt. What follows is an annotated interactive session with NetworkExpect that has good educational value. This is similar in spirit to (and inspired by) the quick demo of an interactive session with Scapy, the powerful interactive packet manipulation program written in Python by Philippe Biondi and other contributors.

NetworkExpect's executable is called nexp. Because of the low level network operations that are normally done with NetworkExpect, nexp usually needs to be run as root. As soon as nexp is run one gets the "netexpect> " prompt. NetworkExpect (and Tcl) commands can be entered there. The following fragment shows how to enter the NetworkExpect command interpreter as well as the results from running the command "nexp_version", which displays version information:

shell# nexp
netexpect> nexp_version 
Network Expect version 0.17pre1
libpcap version 1.0.0
Wireshark Packet Analyzer version 1.0.7

The "send_network" command sends "stuff" to ..., well, to the network. "stuff" is a Protocol Data Unit (PDU), and is defined as part of the command's arguments. The following example shows an ICMP echo request being sent to google.com:

netexpect> send_network ip(dst = google.com)/icmp-echo()/data(data = '1234567890')
!
1

PDU definitions are intended to be simple, intuitive, and powerful. Fields in a PDU can take different values, and when sending a PDU with fields that have multiple values, the "send_network" command will send as many packets as necessary to cover all permutations. For example, the following command:

netexpect> send_network ip(dst = '192.168.1.1/30', id = 1++)/tcp(seq = 'random', dst=<80, 443>, syn)
!!!!!!!!
8

will send eight packets: two packets with destination TCP ports 80 and 443 to each of the four IP addresses in the 192.168.1.1/30 network (including network and broadcast addresses). The IP ID will start at 1 and will be incremented by one in each packet, and the TCP sequence number will be a random number in each packet.

PDU "objects" (note: these are not objects as in the Object-Oriented Programming paradigm) allow to store a PDU definition. PDU objects are manipulated using the NetworkExpect command "pdu". The following interaction shows how a PDU object is created and later built:

netexpect> set mypdu [pdu new ip(dst = google.com)/icmp-echo()/data(data = '1234567890')]
ip(dst = google.com)/icmp-echo()/data(data = '1234567890')
netexpect> set mypacket [pdu build mypdu]
10.116.188.57 -> 74.125.45.100 ICMP Echo (ping) request

Once built, a PDU object becomes a packet object (again, not an "object" as in the Object-Oriented Programming paradigm). Packet objects are manipulated using the NetworkExpect command "packet", and as can be seen in the output from the last command above, the string representation of a packet object is a one line summary of the dissected packet, just as in the output from "tshark" in non-verbose mode.

The following interaction shows some commands run on the packet object that was created by building a PDU object:

netexpect> packet show mypacket
---[ frame ]---
  frame.time: "Oct  2, 2009 16:14:26.415904000"
  frame.time_delta: 0.2147483647
  frame.time_delta_displayed: 0.2147483647
  frame.time_relative: 0.2147483647
  frame.number: 0
  frame.pkt_len: 38
  frame.len: 38
  frame.cap_len: 38
  frame.marked: 0
  frame.protocols: raw:ip:icmp:data
---[ raw ]---
  Text label: No link information available
---[ ip ]---
  ip.version: 4
  ip.hdr_len: 20
  ip.dsfield: 0
    ip.dsfield.dscp: 0
    ip.dsfield.ect: 0
    ip.dsfield.ce: 0
  ip.len: 38
  ip.id: 256
  ip.flags: 0
    ip.flags.rb: 0
    ip.flags.df: 0
    ip.flags.mf: 0
  ip.frag_offset: 0
  ip.ttl: 64
  ip.proto: 1
  ip.checksum: 15177
    ip.checksum_good: 1
    ip.checksum_bad: 0
  ip.src: 10.116.188.57
  ip.addr: 10.116.188.57
  ip.src_host: 10.116.188.57
  ip.host: 10.116.188.57
  ip.dst: 74.125.45.100
  ip.addr: 74.125.45.100
  ip.dst_host: 74.125.45.100
  ip.host: 74.125.45.100
---[ icmp ]---
  icmp.type: 8
  icmp.code: 0
  icmp.checksum: 40075
  icmp.ident: 20847
  icmp.seq: 0
---[ data ]---
    data.data: 31:32:33:34:35:36:37:38:39:30

The previous output shows an important detail: while the PDU was defined with just "ip(dst = google.com)/icmp-echo()/data(data = '1234567890')", we can see in the dissection of the packet that some fields have sensible defaults. For instance, the IP version is 4, the IP header length is 20, the source IP address is the address of the interface used to reach google.com, etc.

A more verbose way of displaying a packet object (this corresponds to the output from "tshark -V"):

netexpect> packet show -verbose mypacket
Frame 0 (38 bytes on wire, 38 bytes captured)
  Arrival Time: Oct  2, 2009 16:14:26.415904000
  [Time delta from previous captured frame: 0.2147483647 seconds]
  [Time delta from previous displayed frame: 0.2147483647 seconds]
  [Time since reference or first frame: 0.2147483647 seconds]
  Frame Number: 0
  Packet Length: 38 bytes
  Frame Length: 38 bytes
  Capture Length: 38 bytes
  [Frame is marked: False]
  [Protocols in frame: raw:ip:icmp:data]
Raw packet data
  No link information available
Internet Protocol, Src: 10.116.188.57 (10.116.188.57), Dst: 74.125.45.100 (74.125.45.100)
  Version: 4
  Header length: 20 bytes
  Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
    0000 00.. = Differentiated Services Codepoint: Default (0x00)
    .... ..0. = ECN-Capable Transport (ECT): 0
    .... ...0 = ECN-CE: 0
  Total Length: 38
  Identification: 0x0100 (256)
  Flags: 0x00
    0... = Reserved bit: Not set
    .0.. = Don't fragment: Not set
    ..0. = More fragments: Not set
  Fragment offset: 0
  Time to live: 64
  Protocol: ICMP (0x01)
  Header checksum: 0x3b49 [correct]
    [Good: True]
    [Bad : False]
  Source: 10.116.188.57 (10.116.188.57)
  Source or Destination Address: 10.116.188.57 (10.116.188.57)
  [Source Host: 10.116.188.57]
  [Source or Destination Host: 10.116.188.57]
  Destination: 74.125.45.100 (74.125.45.100)
  Source or Destination Address: 74.125.45.100 (74.125.45.100)
  [Destination Host: 74.125.45.100]
  [Source or Destination Host: 74.125.45.100]
Internet Control Message Protocol
  Type: 8 (Echo (ping) request)
  Code: 0 ()
  Checksum: 0x9c8b [correct]
  Identifier: 0x516f
  Sequence number: 0 (0x0000)
  Data (10 bytes)
    Data: 31323334353637383930

We can dump the packet's contents in hexadecimal:

netexpect> packet dump mypacket
Packet length: 38
Packet data:
00000000  4500 0026 0100 0000 4001 3b49 0a74 bc39 E..&....@.;I.t.9
00000010  4a7d 2d64 0800 9c8b 516f 0000 3132 3334 J}-d....Qo..1234
00000020  3536 3738 3930                          567890

We can also obtain a string representation of the packet:

netexpect> packet data mypacket
E\x00\x00&\x01\x00\x00\x00@\x01;I\nt\xbc9J}-d\x08\x00\x9c\x8bQo\x00\x001234567890

The built-in array "iface" contains information about the system's network interfaces:

netexpect> parray iface
iface(eth0,flags)       = UP BROADCAST MULTICAST
iface(eth0,hw_addr)     = 00:1f:e2:1a:73:80
iface(eth0,mtu)         = 1500
iface(eth0,type)        = Ethernet
iface(lo,flags)         = UP LOOPBACK
iface(lo,ip)            = 127.0.0.1
iface(lo,mtu)           = 16436
iface(lo,netmask)       = 255.0.0.0
iface(lo,type)          = Loopback
iface(vboxnet0,flags)   = BROADCAST MULTICAST
iface(vboxnet0,hw_addr) = 0a:00:27:00:00:00
iface(vboxnet0,mtu)     = 1500
iface(vboxnet0,type)    = Ethernet
iface(wlan0,flags)      = UP BROADCAST MULTICAST
iface(wlan0,hw_addr)    = 00:21:5c:54:17:77
iface(wlan0,ip)         = 10.116.188.57
iface(wlan0,mtu)        = 1500
iface(wlan0,netmask)    = 255.255.255.240
iface(wlan0,type)       = Ethernet
iface(wmaster0,flags)   = UP BROADCAST MULTICAST
iface(wmaster0,hw_addr) = 00:21:5c:54:17:77
iface(wmaster0,mtu)     = 1500
iface(wmaster0,type)    = Ethernet

Let's now do some more interesting things... this creates a network "listener" that will receive ICMP messages that do not come from ourselves:

netexpect> spawn_network -i wlan0 icmp and not src $iface(wlan0,ip)

The "spawn_network -info" command will display information about existing network "listeners" and "speakers". Note in the following output the listener we just created (called "nexp0"), as well as the default speakers ("ip", "ip6", "lo", "eth0", "wmaster0", "wlan0", "hex", "stdout", and "null"):

netexpect> spawn_network -info
Listeners:
  Number of listeners: 1
  Listener #0:
    Name: nexp0
    Type: live capture
    Capture (pcap) filter: icmp and not src 10.116.188.57
    Display (libwireshark) filter: none
    Packet capture descriptor at: 0x9382d00
    File descriptor: 17
    Reading packets from: interface wlan0

Speakers:
  Number of speakers: 9
  Speaker #0:
    Name: ip
    Type: IPv4 packets (kernel-routed)
  Speaker #1:
    Name: ip6
    Type: IPv6 packets (kernel-routed)
  Speaker #2:
    Name: lo
    Type: raw Ethernet frames
    Interface: lo
  Speaker #3:
    Name: eth0
    Type: raw Ethernet frames
    Interface: eth0
  Speaker #4:
    Name: wmaster0
    Type: raw Ethernet frames
    Interface: wmaster0
  Speaker #5:
    Name: wlan0
    Type: raw Ethernet frames
    Interface: wlan0
  Speaker #6:
    Name: hex
    Type: hexdump to standard output
  Speaker #7:
    Name: stdout
    Type: standard output
  Speaker #8:
    Name: null
    Type: black hole

Now let's use the listener we just created. First, we send a packet:

netexpect> send_network ip(dst = google.com)/icmp-echo()/data(data = 'netexpect')
!
1

Next, we run the "expect_network" command to read data from the network:

netexpect> expect_network {1} {puts "Received an echo reply"}
Received an echo reply

The "expect_network" command will read a packet from a listener, will evaluate a condition after a packet has been read, and then will execute some code if the condition evaluates to "true". In the above example, the condition is "1" (always true), and the code that executes if the condition is true is the "puts" statement. The "expect_network" command always reads data from a network listener. The "-i <listener ID>" switch ("-i" for "input") of the "expect_network" command explicitely tells the "expect_network" command what listener to read from. If there is no "-i" switch to the "expect_network" command, like in the example above, then an implicit network listener is used -- the ID of implicit network listener is stored in the Tcl variable listener_id, and this variable always contains the last listener that was created via the "spawn_network" command.

(As a side note, the "send_network" command sends packets via a network "speaker". The "send_network" command has a "-o <speaker ID>" switch that allows to explicitely select the network speaker to send packets to. As in the case of the "expect_network" command and the lack of a "-i <listener ID>" switch, if a "send_network" command is executed without a "-o" switch to explicitely select a network speaker, an implicit network speaker is used. The name of this implicit speaker is stored in the Tcl variable speaker_id. The default network speaker is the "ip" speaker if netexpect runs with root privileges and allows to inject IP packets that are routed by the kernel, or the "hex" speaker if netexpect runs without root privileges and just produces a hexadecimal dump of the packet sent. However, creating a new speaker with the "spawn_network" command will set the speaker_id variable with the name of the new speaker.)

After executing the "expect_network" command, the packet that was read is dissected, and the dissection results are available to the NetworkExpect script via Tcl variables. The special Tcl variable "pdu" (a Tcl array) has important information about the packet that was dissected:

netexpect> parray pdu  ;# Let's see the contents of the "pdu" Tcl array
pdu(0,hdr_len)     = 60
pdu(0,offset)      = 0
pdu(0,proto)       = frame
pdu(1,hdr_len)     = 14
pdu(1,offset)      = 0
pdu(1,payload_len) = 46
pdu(1,proto)       = eth
pdu(1,tot_len)     = 60
pdu(2,hdr_len)     = 20
pdu(2,offset)      = 14
pdu(2,payload_len) = 26
pdu(2,proto)       = ip
pdu(2,tot_len)     = 46
pdu(3,hdr_len)     = 26
pdu(3,offset)      = 34
pdu(3,payload_len) = 0
pdu(3,proto)       = icmp
pdu(3,tot_len)     = 26
pdu(list)          = frame eth ip icmp

So in this case the dissected packet has the following layers (or "protocols", in Wireshark speak): "frame", "eth", "ip", and "icmp". To see all the variables that were created we can use the Tcl "info vars" command.

The variables for the "frame" protocol are:

netexpect> info vars frame*
frame.number frame.cap_len frame.marked frame.time_relative frame.time frame.time_delta_displayed frame.len frame.time_delta frame.protocols

The variables for the "eth" protocol are:

netexpect> info vars eth*
eth.lg eth.src eth.addr eth.ig eth.dst eth.type

The variables for the "ip" protocol are:

netexpect> info vars ip*
ip.src ip.dsfield.dscp ip.frag_offset ip.flags.mf ip.dsfield.ce ip.checksum ip.id ip.dsfield.ect ip.version ip.dst ip.checksum_good ip.proto ip.flags ip.len ip.hdr_len ip.checksum_bad ip.ttl ip.flags.rb ip.dsfield ip.flags.df

For the "icmp" protocol:

netexpect> info vars icmp*
icmp.checksum icmp.seq icmp.type icmp.code icmp.ident

And finally, the "data" protocol:

netexpect> info vars data*
data.data

The list of all Tcl variables that were created during the behind-the-scenes packet dissection performed by the "expect_network" command can also be obtained via the command "show dissection vars", which returns a Tcl list of all such variables:

netexpect> show dissection vars 
icmp.checksum frame.time_relative frame.number frame.protocols ip.hdr_len icmp.ident ip.proto eth.lg frame.time ip.dsfield ip.len ip.id ip.dsfield.ect pdu ip.flags.mf frame.marked frame.time_delta ip.version ip.dst eth.src eth.type frame.time_delta_displayed icmp.code eth.addr frame.len eth.ig ip.dsfield.dscp ip.dsfield.ce ip.checksum frame.cap_len icmp.type ip.checksum_bad ip.flags.df _ ip.flags.rb ip.frag_offset ip.ttl icmp.seq data.data ip.flags ip.checksum_good eth.dst ip.src

As you can see, the names of the variables are the same names used in Wireshark's display filters. This is not a coincidence - NetworkExpect uses libwireshark, from the Wireshark project, for all packet dissection tasks.

We can do anything we want with the variables since these are regular Tcl variables:

netexpect> set ip.src
74.125.45.100
netexpect> set ip.dst
10.116.188.57
netexpect> puts "${eth.src} -> ${eth.dst}: ${ip.src} -> ${ip.dst}"
00:22:90:24:2f:50 -> 00:21:5c:54:17:77: 74.125.45.100 -> 10.116.188.57
netexpect> set data.data
netexpect

All Tcl variables that were created during the behind-the-scenes packet dissection performed by the "expect_network" command can be removed with the "clear dissection vars" command:

netexpect> clear dissection vars 
netexpect> show dissection vars 
netexpect> # no variables shown
netexpect> set ip.src
can't read "ip.src": no such variable
while evaluating {set ip.src}
netexpect> # the variables are really gone

NetworkExpect has a special and powerful command called "send_expect" that allows to build network tools in a few lines. This command does something very similar to what the "send-and-receive" family of functions in Scapy does (in fact, all credit for NetworkExpect's "send_expect" goes to Philippe Biondi, author of Scapy, and the Scapy developers).

In a nutshell, the "send_expect" command injects some stimuli and then waits for responses to the injected stimuli. Once responses are received it then matches injected stimuli with received answers.

The following example shows the use of the "send_expect" command to send ICMP echo requests to the /29 network where google.com is located and see who replies back with an ICMP echo reply:

netexpect> send_expect ip(dst = 'google.com/29')/icmp-echo(id = 'random')/data(data = 'blah')
Begin stimulus injection (pass 1/1)... done (sent 8 packets)
!!!!!!!!
Sent 8 packets; received 8 answers; 0 unanswered

The "send_expect" commands creates three Tcl lists after it has finished injecting stimuli and matching received responses with injected stimuli: one list of packets sent, another list of the corresponding responses, and another list of packets sent but that received no responses. Here's how these three Tcl lists look like at a higher lever:

netexpect> set _(sent)
{10.116.188.57 -> 74.125.67.96 ICMP Echo (ping) request} {10.116.188.57 -> 74.125.67.97 ICMP Echo (ping) request} {10.116.188.57 -> 74.125.67.98 ICMP Echo (ping) request} {10.116.188.57 -> 74.125.67.99 ICMP Echo (ping) request} {10.116.188.57 -> 74.125.67.100 ICMP Echo (ping) request} {10.116.188.57 -> 74.125.67.101 ICMP Echo (ping) request} {10.116.188.57 -> 74.125.67.102 ICMP Echo (ping) request} {10.116.188.57 -> 74.125.67.103 ICMP Echo (ping) request}
netexpect> set _(received)
{74.125.67.96 -> 10.116.188.57 ICMP Echo (ping) reply} {74.125.67.97 -> 10.116.188.57 ICMP Echo (ping) reply} {74.125.67.98 -> 10.116.188.57 ICMP Echo (ping) reply} {74.125.67.99 -> 10.116.188.57 ICMP Echo (ping) reply} {74.125.67.100 -> 10.116.188.57 ICMP Echo (ping) reply} {74.125.67.101 -> 10.116.188.57 ICMP Echo (ping) reply} {74.125.67.102 -> 10.116.188.57 ICMP Echo (ping) reply} {74.125.67.103 -> 10.116.188.57 ICMP Echo (ping) reply}
netexpect> set _(unanswered)
netexpect> # empty list since all sent packets received a response

After the "send_expect" finishes execution one normally would want to do something with the results. The following example shows precisely how to do something with the results from a "send_expect" command, in this case, machines in google.com/29 that responded to our previous ICMP echo request:

netexpect> foreach r $_(received) {
>packet dissect r
>puts "ICMP echo reply from ${ip.src}"
>}
ICMP echo reply from 74.125.67.96
ICMP echo reply from 74.125.67.97
ICMP echo reply from 74.125.67.98
ICMP echo reply from 74.125.67.99
ICMP echo reply from 74.125.67.100
ICMP echo reply from 74.125.67.101
ICMP echo reply from 74.125.67.102
ICMP echo reply from 74.125.67.103

Since each list of received packets contains packet objects, we can just do the following:

netexpect> foreach r $_(received) { puts "$r" }
74.125.67.96 -> 10.116.188.57 ICMP Echo (ping) reply
74.125.67.97 -> 10.116.188.57 ICMP Echo (ping) reply
74.125.67.98 -> 10.116.188.57 ICMP Echo (ping) reply
74.125.67.99 -> 10.116.188.57 ICMP Echo (ping) reply
74.125.67.100 -> 10.116.188.57 ICMP Echo (ping) reply
74.125.67.101 -> 10.116.188.57 ICMP Echo (ping) reply
74.125.67.102 -> 10.116.188.57 ICMP Echo (ping) reply
74.125.67.103 -> 10.116.188.57 ICMP Echo (ping) reply

This works because, as previously mentioned, the string representation of a packet object is a one line summary of the dissected packet, just as in the output from "tshark" in non-verbose mode.

To exit out of the NetworkExpect command interpreter just send the end-of-file character (usually Control-D) or execute the "exit" command:

netexpect> exit
shell#

InteractiveSession (last edited 2009-10-03 01:26:46 by EloyParis)