4.3.5. Execution of vswitchperf testcases by Yardstick

4.3.5.1. General

Yardstick is a generic framework for a test execution, which is used for validation of installation of OPNFV platform. In the future, Yardstick will support two options of vswitchperf testcase execution:

  • plugin mode, which will execute native vswitchperf testcases; Tests will be executed natively by vsperf, and test results will be processed and reported by yardstick.
  • traffic generator mode, which will run vswitchperf in trafficgen mode only; Yardstick framework will be used to launch VNFs and to configure flows to ensure, that traffic is properly routed. This mode will allow to test OVS performance in real world scenarios.

In Colorado release only the traffic generator mode is supported.

4.3.5.2. Yardstick Installation

In order to run Yardstick testcases, you will need to prepare your test environment. Please follow the installation instructions to install the yardstick.

Please note, that yardstick uses OpenStack for execution of testcases. OpenStack must be installed with Heat and Neutron services. Otherwise vswitchperf testcases cannot be executed.

4.3.5.3. VM image with vswitchperf

A special VM image is required for execution of vswitchperf specific testcases by yardstick. It is possible to use a sample VM image available at OPNFV artifactory or to build customized image.

4.3.5.3.1. Sample VM image with vswitchperf

Sample VM image is available at vswitchperf section of OPNFV artifactory for free download:

$ wget http://artifacts.opnfv.org/vswitchperf/vnf/vsperf-yardstick-image.qcow2

This image can be used for execution of sample testcases with dummy traffic generator.

NOTE: Traffic generators might require an installation of client software. This software is not included in the sample image and must be installed by user.

NOTE: This image will be updated only in case, that new features related to yardstick integration will be added to the vswitchperf.

4.3.5.3.2. Preparation of custom VM image

In general, any Linux distribution supported by vswitchperf can be used as a base image for vswitchperf. One of the possibilities is to modify vloop-vnf image, which can be downloaded from http://artifacts.opnfv.org/vswitchperf.html/ (see vloop-vnf).

Please follow the Installing vswitchperf to install vswitchperf inside vloop-vnf image. As vswitchperf will be run in trafficgen mode, it is possible to skip installation and compilation of OVS, QEMU and DPDK to keep image size smaller.

In case, that selected traffic generator requires installation of additional client software, please follow appropriate documentation. For example in case of IXIA, you would need to install IxOS and IxNetowrk TCL API.

4.3.5.3.3. VM image usage

Image with vswitchperf must be uploaded into the glance service and vswitchperf specific flavor configured, e.g.:

$ glance --os-username admin --os-image-api-version 1 image-create --name \
  vsperf --is-public true --disk-format qcow2 --container-format bare --file \
  vsperf-yardstick-image.qcow2

$ nova --os-username admin flavor-create vsperf-flavor 100 2048 25 1

4.3.5.4. Testcase execution

After installation, yardstick is available as python package within yardstick specific virtual environment. It means, that yardstick environment must be enabled before the test execution, e.g.:

source ~/yardstick_venv/bin/activate

Next step is configuration of OpenStack environment, e.g. in case of devstack:

source /opt/openstack/devstack/openrc
export EXTERNAL_NETWORK=public

Vswitchperf testcases executable by yardstick are located at vswitchperf repository inside yardstick/tests directory. Example of their download and execution follows:

git clone https://gerrit.opnfv.org/gerrit/vswitchperf
cd vswitchperf

yardstick -d task start yardstick/tests/rfc2544_throughput_dummy.yaml

NOTE: Optional argument -d shows debug output.

4.3.5.5. Testcase customization

Yardstick testcases are described by YAML files. vswitchperf specific testcases are part of the vswitchperf repository and their yaml files can be found at yardstick/tests directory. For detailed description of yaml file structure, please see yardstick documentation and testcase samples. Only vswitchperf specific parts will be discussed here.

Example of yaml file:

...
scenarios:
-
  type: Vsperf
  options:
    testname: 'p2p_rfc2544_throughput'
    trafficgen_port1: 'eth1'
    trafficgen_port2: 'eth3'
    external_bridge: 'br-ex'
    test_params: 'TRAFFICGEN_DURATION=30;TRAFFIC={'traffic_type':'rfc2544_throughput}'
    conf_file: '~/vsperf-yardstick.conf'

  host: vsperf.demo

  runner:
    type: Sequence
    scenario_option_name: frame_size
    sequence:
    - 64
    - 128
    - 512
    - 1024
    - 1518
  sla:
    metrics: 'throughput_rx_fps'
    throughput_rx_fps: 500000
    action: monitor

context:
...

4.3.5.5.1. Section option

Section option defines details of vswitchperf test scenario. Lot of options are identical to the vswitchperf parameters passed through --test-params argument. Following options are supported:

  • frame_size - a packet size for which test should be executed; Multiple packet sizes can be tested by modification of Sequence runner section inside YAML definition. Default: ‘64’
  • conf_file - sets path to the vswitchperf configuration file, which will be uploaded to VM; Default: ‘~/vsperf-yardstick.conf’
  • setup_script - sets path to the setup script, which will be executed during setup and teardown phases
  • trafficgen_port1 - specifies device name of 1st interface connected to the trafficgen
  • trafficgen_port2 - specifies device name of 2nd interface connected to the trafficgen
  • external_bridge - specifies name of external bridge configured in OVS; Default: ‘br-ex’
  • test_params - specifies a string with a list of vsperf configuration parameters, which will be passed to the --test-params CLI argument; Parameters should be stated in the form of param=value and separated by a semicolon. Configuration of traffic generator is driven by TRAFFIC dictionary, which can be also updated by values defined by test_params. Please check VSPERF documentation for details about available configuration parameters and their data types. In case that both test_params and conf_file are specified, then values from test_params will override values defined in the configuration file.

In case that trafficgen_port1 and/or trafficgen_port2 are defined, then these interfaces will be inserted into the external_bridge of OVS. It is expected, that OVS runs at the same node, where the testcase is executed. In case of more complex OpenStack installation or a need of additional OVS configuration, setup_script can be used.

NOTE It is essential to specify a configuration for selected traffic generator. In case, that standalone testcase is created, then traffic generator can be selected and configured directly in YAML file by test_params. On the other hand, if multiple testcases should be executed with the same traffic generator settings, then a customized configuration file should be prepared and its name passed by conf_file option.

4.3.5.5.2. Section runner

Yardstick supports several runner types. In case of vswitchperf specific TCs, Sequence runner type can be used to execute the testcase for given list of frame sizes.

4.3.5.5.3. Section sla

In case that sla section is not defined, then testcase will be always considered as successful. On the other hand, it is possible to define a set of test metrics and their minimal values to evaluate test success. Any numeric value, reported by vswitchperf inside CSV result file, can be used. Multiple metrics can be defined as a coma separated list of items. Minimal value must be set separately for each metric.

e.g.:

sla:
    metrics: 'throughput_rx_fps,throughput_rx_mbps'
    throughput_rx_fps: 500000
    throughput_rx_mbps: 1000

In case that any of defined metrics will be lower than defined value, then testcase will be marked as failed. Based on action policy, yardstick will either stop test execution (value assert) or it will run next test (value monitor).

NOTE The throughput SLA (or any other SLA) cannot be set to a meaningful value without knowledge of the server and networking environment, possibly including prior testing in that environment to establish a baseline SLA level under well-understood circumstances.