----    0-TH SEC:    ----
<--------------------------|  |                 |  |<-------------------CUBIC
                           |  |      500ms      |  |
CUBIC--------------------->|  |      ^^^^^5ms   |  |------------------------>
                           |  |  10ms|   |      |  |
                           |  |---120Mbps,20ms--|  |
                           |  |                 |  |
                           |  |    15-TH SEC:   |  |
VEGAS---120Mbps,2000pkts-->|  |                 |  |--120Mbps,5ms,3000pkts-->
                           |  |                 |  |
VEGAS---120Mbps,2000pkts-->|  |                 |  |--120Mbps,5ms,3000pkts-->
                           ----     30 SECS     ----
                    

CoCo-Beholder is a human-friendly virtual network emulator providing the popular dumbbell topology of any size. Each link of the topology may have individual rate, delay, and queue size. The central link may also have a variable delay with optional jitter. Flows of different schemes may run together in the topology for a specified runtime of seconds. For each flow, its direction and starting second of the runtime can be chosen.

Each flow has a host in the left half and a host in the right half of the topology and the hosts exchange a scheme's traffic with one host being the sender and one being the receiver. There is the left router that interconnects all the hosts in the left half and the right router that interconnects all the hosts in the right half of the topology. All the flows share the common central link between the two routers.

Testing
Analysis
Plots and Statistics Generation
Installation
Troubleshooting a Scheme
Adding a New Scheme
Python Support
Third-Party Libraries
Authorship
Contact

This command specifies the path to the collection containing the schemes to test and runs the testing for 30 seconds, with the central link having 120 Mbps rate and the variable delay (the base delay 20 ms, delta 500 ms, step 10 ms, jitter 5 ms):

./run.py -p ~/pantheon 20ms 0.5s 10ms 5ms -t 30 -r 120 -s 12345

If this is the first run of the script, the default layout file layout.yml, shown below, is generated and used. The resulting testing setup is present in the drawing of the dumbbell topology at the top of this page. The layout file can be edited to get much more complex testing setups with more flows belonging to various schemes and having diverse network settings.

# Delays/rates are optional: if lacking or null, they are set to 0us/0.0
# and for netem, to set delay/rate to zero is same as to leave it unset.
# Sizes of queues are optional: if lacking or null, they are set to 1000.
- direction: <-
  flows: 1
  left-delay: null
  left-queues: null
  left-rate: null
  right-delay: null
  right-queues: null
  right-rate: null
  scheme: cubic
  start: 0
- direction: ->
  flows: 2
  left-delay: 0us
  left-queues: 2000
  left-rate: 120.0
  right-delay: 5ms
  right-queues: 3000
  right-rate: 120
  scheme: vegas
  start: 15
- direction: ->
  flows: 1
  scheme: cubic
  start: 0

The rate, delay, and queue size are always installed at both the interfaces at the ends of each link in the topology using tc qdisc NetEm link emulator. In particular, this means that the RTT of a link is twice the (one-way) delay. Only the central link may have two different queue sizes of the interfaces at its ends – see -q1, -q2, -q arguments in the help message of the script. By default, both the queues are of 1000 packets.

The variable delay at the central link is defined by four positional arguments: the base delay, delta, step, and jitter, where the jitter can be skipped. Each delta time, the delay is increased or decreased by step depending on a pseudorandom generator, whose seed can be specified with -s argument or is assigned the current UNIX time. To have a constant delay at the central link, choose the delta >= the runtime -t.

Into a chosen output directory, metadata.json file is written containing all the parameters of the test, including the generator seed. The file may be fed to CoCo-Beholder in the future to fully reproduce the test. Also, during the testing, PCAP dump files are recorded at all the hosts of the dumbbell topology into the output directory using tcpdump. So for the example in the drawing, eight PCAP dump files were recorded.

Note #1: The maximum delay for every link – side and central (jitter not counted) – can be specified with -m option. To have a square-wave delay at the central link, set the maximum delay to the sum of the base delay and step.

Note #2: The testing script calls setup_after_reboot on Pantheon wrapper of each scheme in the layout file before launching any flows of the schemes, so the user does not have to manually set schemes up after every reboot.

Analysis script only accepts two arguments: the input folder and output folder – dumps and graphs/data by default.

First, the script simply copies the metadata file from the input folder into the output folder. Then, the script processes a pair of PCAP dumps <flow's starting #>-<scheme>-<sender/receiver>.pcap of each flow and extracts information on the flow's packets into the flow's individual json data log file. E.g., for two ~12 GB dumps, one ~300 MB data log file is produced.

The partial output of the analysis script for the example in the drawing:

./analyze.py

cubic scheme, flow 1:

sender   dump: 100.0% in 5.18s

Total: 222667 pkts/224631734 bytes, from sender: 144906 pkts/219351284 bytes

receiver dump: 100.0% in 5.37s

Total: 222245 pkts/223376026 bytes, from sender: 144052 pkts/218058328 bytes

♥ Union of data from sender recorded on both sides: 144906 pkts/219351284 bytes
♦ Subset of ♥ which was not recorded at sender    : 0 pkts/0 bytes
♣ Subset of ♥ which was not recorded at receiver  : 854 pkts/1292956 bytes
♠ Loss (ratio of ♣ bytes to ♥ bytes)              : 0.589%

Saving the data of the flow to the file...

==========================================

# and so on for the three more flows...

For forthcoming plots and statistics generation, the PCAP dumps are not needed anymore. The analysis of the PCAP dumps is performed only once, and then the plotting script may be run as many times as needed over the data log files to produce various plots quickly.

Plotting script plot.py reads data log files of the flows and generates plots and statistics into the output folder (graphs by default). Different types of plots and statistics can be generated:

  -f, --per-flow        Graphs and stats are generated per flow, i.e. each
                        graph has a separate curve per flow
                        
  -t, --total           Total graphs and stats are generated for all flows
                        altogether, i.e. each graph has only one curve
                        
  -s "FIELD1 FIELD2...", --per-subset "FIELD1 FIELD2..."
                        Graphs and stats are generated per subset, i.e. each
                        graph has one curve per subset. Flows are in one
                        subset if they have the same values of the chosen
                        layout field(s). E.g. for -s "scheme direction", each
                        graph will have one curve per subset of flows having
                        both the same scheme name and direction. Currently
                        allowed layout fields: ['scheme', 'direction'].

For a selected type, three line plots and one scatter plot are generated:

• average throughput
• average one-way delay
• average Jain's index
• per-packet one-way delay

E.g., above, one can see the per-flow per-packet one-way delay plot for the setup in the drawing.

Average plots are averaged per an aggregation time interval: any positive float number supplied with -i argument or 0.5 second by default.

Average Jain's index plot always contains one curve that is computed over the curves present in the corresponding average throughput plot.

Arguments -c and -j allow changing colors of the curves in plots flexibly.

For a selected type, the statistics file is generated. It does not depend on the chosen aggregation interval and contains:

• each curve's overall average throughput over the curve's whole duration (Mbps)
• each curve's overall average one-way delay over all the curve's packets (ms)
• each curve's overall median, average, and 95th percentile per-packet one-way delays over all the curve's packets (ms)
• overall Jain's index value for all the curves together, computed over the average throughput statistics of the curves

The overall average one-way delay and overall average per-packet one-way delay of a curve are the same values computed in different ways.

The installation process is as follows:

• Install Pantheon collection of congestion control schemes and, if needed, add more schemes to the collection locally

• Install CoCo-Beholder emulator using its installation script

• Done. Test the schemes in the collection using CoCo-Beholder.

Installing CoCo-Beholder itself is trivial, and its installation script install.sh is super easy and short. However, installing Pantheon collection of schemes often causes lots of problems. Besides, some Linux distributions just have bugs. Thus, please, see the detailed instructions for installation on Ubuntu 16.04, Ubuntu 18.04, and Debian 10.

git clone https://github.com/StanfordSNR/pantheon.git && cd pantheon
git submodule update --init --recursive

rm -r src/wrappers/patches

sudo apt-get install autoconf                              # for verus
sudo apt-get install nodejs-legacy                         # for webrtc
sudo apt-get install python-pip && sudo pip install pyyaml # for setup.py

src/experiments/setup.py --install-deps (--all | --schemes "<cc1> <cc2> ...")
src/experiments/setup.py --setup (--all | --schemes "<cc1> <cc2> ...")

cd coco-beholder && sudo ./install.sh

git clone https://github.com/StanfordSNR/pantheon.git && cd pantheon
git submodule update --init --recursive

rm -r src/wrappers/patches

sudo apt-get install autoconf                              # for verus
sudo apt-get install python-pip && sudo pip install pyyaml # for setup.py

src/experiments/setup.py --install-deps (--all | --schemes "<cc1> <cc2> ...")
src/experiments/setup.py --setup (--all | --schemes "<cc1> <cc2> ...")

cd coco-beholder && sudo ./install.sh

git clone https://github.com/StanfordSNR/pantheon.git && cd pantheon
git submodule update --init --recursive

rm -r src/wrappers/patches

sed -i 's/chromium-browser/chromium/g' src/wrappers/webrtc.py # for webrtc
sudo apt-get install python-pip && sudo pip install pyyaml    # for setup.py

src/experiments/setup.py --install-deps (--all | --schemes "<cc1> <cc2> ...")

sudo apt-get update
sudo apt-get remove libalglib-dev             # remove  3.14 version
sudo apt-get install -t stretch libalglib-dev # install 3.10 version

sudo apt-get install autoconf                             # for verus
myregex='s/milliseconds(\(.\ \))/milliseconds(int(\1))/g'    
sed -i $myregex third_party/verus/src/verus_client.cpp
sed -i $myregex third_party/verus/src/verus_server.cpp 

sudo apt-get install pkg-config                           # for sprout

sudo apt-get install libtinfo5                            # for quic
# During the setup, do not be afraid of CERTIFICATE_VERIFY_FAILED errors by quic

src/experiments/setup.py --setup (--all | --schemes "<cc1> <cc2> ...")

cd coco-beholder && sudo ./install.sh

# matplotlib will give Python backports.functools_lru_cache error. To solve:
sudo pip install arrow==0.12.0

cd pantheon
src/wrappers/vegas.py setup_after_reboot # setup the scheme once after reboot
src/wrappers/vegas.py run_first          # who is server: sender or receiver
# receiver
src/wrappers/vegas.py receiver 54321         # start server in one shell
src/wrappers/vegas.py sender 127.0.0.1 54321 # start client in another shell
sudo pkill -9 -f vegas                       # kill all the started processes

cd coco-beholder
myregex='s/(\(.\ \)).pid/(\1, stdout=None, stderr=None).pid/g'
sed -i "$myregex" variable_delay/src/test/test.py

find /lib/modules/`(uname -r)`/kernel -type f -name *cdg*
# /lib/modules/4.19.0-6-amd64/kernel/net/ipv4/tcp_cdg.ko

  cdg:
    name: TCP CDG
    color: red
    marker: 'x'

cp pantheon/src/wrappers/vegas.py pantheon/src/wrappers/cdg.py
sed -i 's/vegas/cdg/g' pantheon/src/wrappers/cdg.py

CoCo-Beholder is ensured to work with Python 2.7, 3.5, 3.6, and 3.7. CoCo-Beholder's installation script install.sh installs Python 2 library dependencies by default. For Python 3, please, comment out the corresponding lines in the installation script.

CoCo-Beholder utilizes Mininet library: its API that enables to create a virtual host as a UNIX shell in a separate network namespace, to create a veth pair link between a pair of virtual hosts, and to launch processes at a virtual host. CoCo-Beholder does not use Controller, Switch, Topology, TCLink or other higher-level entities of Mininet. To prevent any future compatibility issues and to make the installation of CoCo-Beholder easier, the needed parts of Mininet 2.3.0d5 are included in CoCo-Beholder repository as a third-party library according to Mininet license.

Evgeniya Khasina implemented the program in the scope of the Master's thesis "The CoCo-Beholder: Enabling Comprehensive Evaluation of Congestion Control Algorithms" at Saarland University in 2019 under the supervision of Professor Anja Feldmann and Dr. Corinna Coupette at Internet Architecture Department of Max Planck Institute for Informatics.

For any questions on the program, you may write to me, Evgeniya Khasina, to my E-mail address.