We’re going to look at network statistics to form a timeline over the event so far.
Learning Objectives #
- Gain knowledge of the network traffic data format.
- Understand the differences between full packet captures and network flows.
- Learn how to process network flow data.
- Discover the SiLK tool suite.
- Gain hands-on experience in network flow analysis with SiLK.
Network Traffic Data #
Network communication and traffic are everywhere! It’s a constant flow of data, including personal interactions and business transactions. The table below highlights the importance and key benifits of each of these aspects:
| Description | |
|---|---|
| Network Management | Monitoring performance, bandwith bottlenecks, resource allocation |
| Troubleshooting | Issues (latency and connectivity issues), performance baselines |
| Incident Response | Scope, root cause analysis, assessment of the compliance aspects of incidents |
| Threat Hunting | Proactive analysis for signs of suspicious and malicious patterns, IoCs etc. |
Packet captures (PCAP) format is the first thing to come to mind. It provides all possible data from packets (also known as deep packet inspection). However, it requires storage, processing and analysis capacities to provide comprehensive insight. It’s very useful for detailed analysis but it’s not practical for fast analysis.
Netfork flow data is a lightweight alternative to PCAPs. It’s commonly used in NetFlow format (developed by Cisco) that focuses on the metadata part of the traffic. It provides a “summary” of the traffic.
Closer Look at PCAPs and Flows #
| Feature | PCAP | Network Flow |
|---|---|---|
| Model | Packet capture | Protocol flow records |
| Depth of Information | Detailed granular data. Contains the packet details and payload. |
Summary data Doesn’t contain the packet details and payload |
| Main Purpose | Deep packet analytics | Summary of the traffic flow |
| Pros | Provides high visibility of packet details | Provides a high-level summary of the big picture Encryption is not an obstacle (the flows don’t use the packet payload) |
| Cons | Hard to process and requires time and resources to store and analyse Encryption is an obstacle |
Summary only; no payload |
| Available Fields | Layer headers and payload data | Packet metadata |
The table above gives some highlights over the differences between PCAP and network flow.
Detailed comparison #
Key Data Files of PCAP Format
- Link layer information
- Timestamp
- Packet length
- MAC addresses:
- Source MAC
- Destination MAC
- IP and Port information:
- Source IP Address
- Destination IP Address
- Source Port
- Destination Port
- TCP/UDP Information
- Application layer protocol details
- Packet data and payload
Key Data Fields of Network Flow Format
- IP and Port information:
- Source IP Address
- Destination IP Address
- Source Port
- Destination Port
- IP Protocol
- Volume details in byte and packet metrics
- TCP flags
- Time details:
- Start time
- Duration
- End time
- Sensor info
- Application layer protocol information
How to Collect and Process Network Data #
Network data collection and processing typically involves using network monitoring and analysis tools (Wireshark, tshark, tcpdump). There’s also product and system-based solutions that help collect network data in flow format. The tools and methods used depends on the size and complexity of the network and the objectives.
In todays task, we focus on using SiLK to analyze the network flow.
SiLK #
SiLK stands for System for Internet Level Knowledge tool suite and was developed by the CERT Situational Awareness group and Carnegie Mellon University’s Software Engineering Institute. It contains various tools and binaries that allow users to collect, parse, filter and analyze network traffic data. It can process direct flows, PCAP files and binary flow data.
SiLK has two parts: the Packaging System and the Analysis Suite.
- Packaging System: Supports the collection of network flow types (IPFIX, NetFlow v9, NetFlow v5) and stores them in binary files.
- Analysis Suite: Contains the tools needed to carry out various operations (list, sort, count).
rwcut #
silk_config -vShows the version of SiLK installed.rwfileinfo <FILENAME>Overviewing the file info.rwcut <FILENAME>Print out all the records without any filtering.rwcut <FILENAME> --num-recs=5Limits the output to only show the first five records.rwcut <FILENAME> --tail-rec=5Limits the output to only show the last five records.rwcut <FILENAME> --fields=protocol,sIP,sPort,dIP,dPortShows the protocol type, source and destination IPs and ports.
One thing to notice is the protocol column. It shows the numeric values in binary form, instead of TCP or UDP.
- ICMP = 1
- IPv4 = 4
- TCP = 6
- UDP = 17
Column filtering:
- Source IP:
sIP - Destination IP:
dIP - Source Port:
sPort - Destination Port:
dPort - Duration:
duration - Start time:
sTime - End time:
eTime
rwfilter #
rwfilter helps implement conditional and logical filters to extract records.
rwfilter <FILENAME> --proto=17 --pass=stdout | rwcut --num-recs=5Filters all UDP records, passes it torwcutand displays the first five records.
--pass=stdout must be set to process the output with pipe and rwcut.
Filtering field options:
- Protocols:
--proto, possible values are 0-255. - Any port:
--aport - Source port:
--sport - Destination port:
--dport - Any IP address:
--any-address - Source address:
--saddress - Destination address:
--daddress - Number of the packets:
--packets - Number of the bytes:
--bytes
rwstats #
rwstats helps for a quicker and more automated overview of the events.
It needs to have --fields parameters to run.
rwstats <FILENAME> --fields=dPort --values=records,packets,bytes,sIP-Distinct,dIP-Distinct --count=10--countlimits the number of records printed to the terminal--values=records,packets,bytesshows the measurement in flows, packets and bytes--values=sIP-Distinct,dIP-Distinctshows the number of unique IP addresses that used the filtered field.
Bring it together #
List the top talkers on the network:
rwstats <FILENAME> --fields=sIP --values=bytes --count=10 --top
List the top communication pairs:
rwstats <FILENAME> --fields=sIP,dIP --values=records,bytes,packets --count=10
Filter out all records that use Port 53:
rwfilter <FILENAME> --aport=53 --pass=stdout | rwstats --fields=sIP,dIP --values=records,bytes,packets --count=10
View the frequency of the requets:
rwfilter <FILENAME> --saddress=IP-address --dport=53 --pass=stdout | rwcut --fields=sIP,dIP,sTime | head -10
There are many more commands used in the room, but this is the most meaningful and in combination with other filters already described above, this room is now over… :)