DPDK documentation - Read the Docs [PDF]

DPDK documentation Release 16.07.0

July 29, 2016

Contents

1 Getting Started Guide for Linux

1

2 Getting Started Guide for FreeBSD

26

3 Xen Guide

37

4 Programmer’s Guide

45

5 Network Interface Controller Drivers

224

6 Crypto Device Drivers

310

7 Sample Applications User Guide

322

8 Testpmd Application User Guide

547

9 FAQ

586

10 How To User Guides

591

11 Release Notes

611

12 Contributor’s Guidelines

666

i

CHAPTER 1

Getting Started Guide for Linux

1.1 Introduction This document contains instructions for installing and configuring the # Identify NIC devices for DPDK apps to use and load nic_uio driver: hw.nic_uio.bdfs="2:0:0,2:0:1" nic_uio_load="YES"

2.2.2 Compiling and Running the Example Applications When the DPDK has been installed from the ports collection it installs its example applications in /usr/local/share/dpdk/examples - also accessible via symlink as /usr/local/share/examples/dpdk. These examples can be compiled and run as described in Compiling and Running Sample Applications. In this case, the required environmental variables should be set as below: • RTE_SDK=/usr/local/share/dpdk • RTE_TARGET=x86_64-native-bsdapp-clang

2.2. Installing DPDK from the Ports Collection

27

DPDK documentation, Release 16.07.0

Note: To install a copy of the DPDK compiled using gcc, please download the official DPDK package from http://dpdk.org/ and install manually using the instructions given in the next chapter, Compiling the DPDK Target from Source An example application can therefore be copied to a user’s home directory and compiled and run as below: export RTE_SDK=/usr/local/share/dpdk export RTE_TARGET=x86_64-native-bsdapp-clang cp -r /usr/local/share/dpdk/examples/helloworld . cd helloworld/ gmake CC main.o LD helloworld INSTALL-APP helloworld INSTALL-MAP helloworld.map sudo ./build/helloworld -c F -n 2 EAL: EAL: EAL: EAL: EAL: EAL: EAL: EAL: EAL: EAL: EAL: EAL:

Contigmem driver has 2 buffers, each of size 1GB Sysctl reports 8 cpus Detected lcore 0 Detected lcore 1 Detected lcore 2 Detected lcore 3 Support maximum 64 logical core(s) by configuration. Detected 4 lcore(s) Setting up physically contiguous memory... Mapped memory segment 1 @ 0x802400000: len 1073741824 Mapped memory segment 2 @ 0x842400000: len 1073741824 WARNING: clock_gettime cannot use CLOCK_MONOTONIC_RAW and HPET is not available - clock timings may be less accurate. EAL: TSC frequency is ~3569023 KHz EAL: PCI scan found 24 devices EAL: Master core 0 is ready (tid=0x802006400) EAL: Core 1 is ready (tid=0x802006800) EAL: Core 3 is ready (tid=0x802007000) EAL: Core 2 is ready (tid=0x802006c00) EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x80074a000 EAL: PCI memory mapped at 0x8007ca000 EAL: PCI device 0000:01:00.1 on NUMA socket 0 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x8007ce000 EAL: PCI memory mapped at 0x80084e000 EAL: PCI device 0000:02:00.0 on NUMA socket 0 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x800852000 EAL: PCI memory mapped at 0x8008d2000 EAL: PCI device 0000:02:00.1 on NUMA socket 0 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x801b3f000 EAL: PCI memory mapped at 0x8008d6000 hello from core 1 hello from core 2 hello from core 3

2.2. Installing DPDK from the Ports Collection

28

DPDK documentation, Release 16.07.0

hello from core 0

Note: To run a DPDK process as a non-root user, adjust the permissions on the /dev/contigmem and /dev/uio device nodes as described in section Running DPDK Applications Without Root Privileges

Note: For an explanation of the command-line parameters that can be passed to an DPDK application, see section Running a Sample Application.

2.3 Compiling the DPDK Target from Source 2.3.1 System Requirements The DPDK and its applications require the GNU make system (gmake) to build on FreeBSD. Optionally, gcc may also be used in place of clang to build the DPDK, in which case it too must be installed prior to compiling the DPDK. The installation of these tools is covered in this section. Compiling the DPDK requires the FreeBSD kernel sources, which should be included during the installation of FreeBSD on the development platform. The DPDK also requires the use of FreeBSD ports to compile and function. To use the FreeBSD ports system, it is required to update and extract the FreeBSD ports tree by issuing the following commands: portsnap fetch portsnap extract

If the environment requires proxies for external communication, these can be set using: setenv http_proxy : setenv ftp_proxy :

The FreeBSD ports below need to be installed prior to building the DPDK. In general these can be installed using the following set of commands: cd /usr/ports/ make config-recursive make install make clean

Each port location can be found using: whereis

The ports required and their locations are as follows: • dialog4ports: /usr/ports/ports-mgmt/dialog4ports • GNU make(gmake): /usr/ports/devel/gmake • coreutils: /usr/ports/sysutils/coreutils

2.3. Compiling the DPDK Target from Source

29

DPDK documentation, Release 16.07.0

For compiling and using the DPDK with gcc, the compiler must be installed from the ports collection: • gcc: version 4.8 is recommended /usr/ports/lang/gcc48. Ensure that CPU_OPTS is selected (default is OFF). When running the make config-recursive command, a dialog may be presented to the user. For the installation of the DPDK, the default options were used. Note: To avoid multiple dialogs being presented to the user during make install, it is advisable before running the make install command to re-run the make config-recursive command until no more dialogs are seen.

2.3.2 Install the DPDK and Browse Sources First, uncompress the archive and move to the DPDK source directory: unzip DPDK-.zip cd DPDK- ls app/ config/ examples/ lib/ LICENSE.GPL LICENSE.LGPL Makefile mk/ scripts/ tools/

The DPDK is composed of several directories: • lib: Source code of DPDK libraries • app: Source code of DPDK applications (automatic tests) • examples: Source code of DPDK applications • config, tools, scripts, mk: Framework-related makefiles, scripts and configuration

2.3.3 Installation of the DPDK Target Environments The format of a DPDK target is: ARCH-MACHINE-EXECENV-TOOLCHAIN

Where: • ARCH is: x86_64 • MACHINE is: native • EXECENV is: bsdapp • TOOLCHAIN is: gcc | clang The configuration files for the DPDK targets can be found in the DPDK/config directory in the form of: defconfig_ARCH-MACHINE-EXECENV-TOOLCHAIN

Note: Configuration files are provided with the RTE_MACHINE optimization level set. Within the configuration files, the RTE_MACHINE configuration value is set to native, which means that

2.3. Compiling the DPDK Target from Source

30

DPDK documentation, Release 16.07.0

the compiled software is tuned for the platform on which it is built. For more information on this setting, and its possible values, see the DPDK Programmers Guide. To make the target, use gmake install T=. For example to compile for FreeBSD use: gmake install T=x86_64-native-bsdapp-clang

Note: If the compiler binary to be used does not correspond to that given in the TOOLCHAIN part of the target, the compiler command may need to be explicitly specified. For example, if compiling for gcc, where the gcc binary is called gcc4.8, the command would need to be gmake install T= CC=gcc4.8.

2.3.4 Browsing the Installed DPDK Environment Target Once a target is created, it contains all the libraries and header files for the DPDK environment that are required to build customer applications. In addition, the test and testpmd applications are built under the build/app directory, which may be used for testing. A kmod directory is also present that contains the kernel modules to install: ls x86_64-native-bsdapp-gcc app build include kmod lib Makefile

2.3.5 Loading the DPDK contigmem Module To run a DPDK application, physically contiguous memory is required. In the absence of nontransparent superpages, the included sources for the contigmem kernel module provides the ability to present contiguous blocks of memory for the DPDK to use. The contigmem module must be loaded into the running kernel before any DPDK is run. The module is found in the kmod sub-directory of the DPDK target directory. The amount of physically contiguous memory along with the number of physically contiguous blocks to be reserved by the module can be set at runtime prior to module loading using: kenv hw.contigmem.num_buffers=n kenv hw.contigmem.buffer_size=m

The kernel environment variables can also be specified during boot by placing the following in /boot/loader.conf: hw.contigmem.num_buffers=n hw.contigmem.buffer_size=m

The variables can be inspected using the following command: sysctl -a hw.contigmem

Where n is the number of blocks and m is the size in bytes of each area of contiguous memory. A default of two buffers of size 1073741824 bytes (1 Gigabyte) each is set during module load if they are not specified in the environment. The module can then be loaded using kldload (assuming that the current directory is the DPDK target directory): kldload ./kmod/contigmem.ko

2.3. Compiling the DPDK Target from Source

31

DPDK documentation, Release 16.07.0

It is advisable to include the loading of the contigmem module during the boot process to avoid issues with potential memory fragmentation during later system up time. This can be achieved by copying the module to the /boot/kernel/ directory and placing the following into /boot/loader.conf: contigmem_load="YES"

Note: The contigmem_load directive should be placed after any definitions of hw.contigmem.num_buffers and hw.contigmem.buffer_size if the default values are not to be used. An error such as: kldload: can't load ./x86_64-native-bsdapp-gcc/kmod/contigmem.ko: Exec format error

is generally attributed to not having enough contiguous memory available and can be verified via dmesg or /var/log/messages: kernel: contigmalloc failed for buffer

To avoid this error, reduce the number of buffers or the buffer size.

2.3.6 Loading the DPDK nic_uio Module After loading the contigmem module, the nic_uio module must also be loaded into the running kernel prior to running any DPDK application. This module must be loaded using the kldload command as shown below (assuming that the current directory is the DPDK target directory). kldload ./kmod/nic_uio.ko

Note: If the ports to be used are currently bound to a existing kernel driver then the hw.nic_uio.bdfs sysctl value will need to be set before loading the module. Setting this value is described in the next section below. Currently loaded modules can be seen by using the kldstat command and a module can be removed from the running kernel by using kldunload . To load the module during boot, copy the nic_uio module to /boot/kernel and place the following into /boot/loader.conf: nic_uio_load="YES"

Note: nic_uio_load="YES" must appear after the contigmem_load directive, if it exists. By default, the nic_uio module will take ownership of network ports if they are recognized DPDK devices and are not owned by another module. However, since the FreeBSD kernel includes support, either built-in, or via a separate driver module, for most network card devices, it is likely that the ports to be used are already bound to a driver other than nic_uio. The following sub-section describe how to query and modify the device ownership of the ports to be used by DPDK applications.

2.3. Compiling the DPDK Target from Source

32

DPDK documentation, Release 16.07.0

Binding Network Ports to the nic_uio Module Device ownership can be viewed using the pciconf -l command. The example below shows four Intel® 82599 network ports under if_ixgbe module ownership. pciconf -l ix0@pci0:1:0:0: ix1@pci0:1:0:1: ix2@pci0:2:0:0: ix3@pci0:2:0:1:

class=0x020000 class=0x020000 class=0x020000 class=0x020000

card=0x00038086 card=0x00038086 card=0x00038086 card=0x00038086

chip=0x10fb8086 chip=0x10fb8086 chip=0x10fb8086 chip=0x10fb8086

rev=0x01 rev=0x01 rev=0x01 rev=0x01

hdr=0x00 hdr=0x00 hdr=0x00 hdr=0x00

The first column constitutes three components: 1. Device name: ixN 2. Unit name: pci0 3. Selector (Bus:Device:Function): 1:0:0 Where no driver is associated with a device, the device name will be none. By default, the FreeBSD kernel will include built-in drivers for the most common devices; a kernel rebuild would normally be required to either remove the drivers or configure them as loadable modules. To avoid building a custom kernel, the nic_uio module can detach a network port from its current device driver. This is achieved by setting the hw.nic_uio.bdfs kernel environment variable prior to loading nic_uio, as follows: hw.nic_uio.bdfs="b:d:f,b:d:f,..."

Where a comma separated list of selectors is set, the list must not contain any whitespace. For example to re-bind ix2@pci0:2:0:0 and ix3@pci0:2:0:1 to the nic_uio module upon loading, use the following command: kenv hw.nic_uio.bdfs="2:0:0,2:0:1"

The variable can also be specified during boot by placing the following into /boot/loader.conf, before the previously-described nic_uio_load line - as shown: hw.nic_uio.bdfs="2:0:0,2:0:1" nic_uio_load="YES"

Binding Network Ports Back to their Original Kernel Driver If the original driver for a network port has been compiled into the kernel, it is necessary to reboot FreeBSD to restore the original device binding. Before doing so, update or remove the hw.nic_uio.bdfs in /boot/loader.conf. If rebinding to a driver that is a loadable module, the network port binding can be reset without rebooting. To do so, unload both the target kernel module and the nic_uio module, modify or clear the hw.nic_uio.bdfs kernel environment (kenv) value, and reload the two drivers first the original kernel driver, and then the nic_uio driver. Note: the latter does not need to be reloaded unless there are ports that are still to be bound to it. Example commands to perform these steps are shown below: kldunload nic_uio kldunload # To clear the value completely:

2.3. Compiling the DPDK Target from Source

33

DPDK documentation, Release 16.07.0

kenv -u hw.nic_uio.bdfs # To update the list of ports to bind: kenv hw.nic_uio.bdfs="b:d:f,b:d:f,..." kldload kldload nic_uio

# optional

2.4 Compiling and Running Sample Applications The chapter describes how to compile and run applications in a DPDK environment. It also provides a pointer to where sample applications are stored.

2.4.1 Compiling a Sample Application Once a DPDK target environment directory has been created (such as x86_64-native-bsdapp-clang), it contains all libraries and header files required to build an application. When compiling an application in the FreeBSD environment on the DPDK, the following variables must be exported: • RTE_SDK - Points to the DPDK installation directory. • RTE_TARGET - Points to the DPDK target environment directory. For FreeBSD, this is the x86_64-native-bsdapp-clang or x86_64-native-bsdapp-gcc directory. The following is an example of creating the helloworld application, which runs in the DPDK FreeBSD environment. While the example demonstrates compiling using gcc version 4.8, compiling with clang will be similar, except that the CC= parameter can probably be omitted. The helloworld example may be found in the ${RTE_SDK}/examples directory. The directory contains the main.c file. This file, when combined with the libraries in the DPDK target environment, calls the various functions to initialize the DPDK environment, then launches an entry point (dispatch application) for each core to be utilized. By default, the binary is generated in the build directory. setenv RTE_SDK /home/user/DPDK cd $(RTE_SDK) cd examples/helloworld/ setenv RTE_SDK $HOME/DPDK setenv RTE_TARGET x86_64-native-bsdapp-gcc gmake CC=gcc48 CC main.o LD helloworld INSTALL-APP helloworld INSTALL-MAP helloworld.map ls build/app helloworld helloworld.map

Note: In the above example, helloworld was in the directory structure of the DPDK. However, it could have been located outside the directory structure to keep the DPDK structure

2.4. Compiling and Running Sample Applications

34

DPDK documentation, Release 16.07.0

intact. In the following case, the helloworld application is copied to a new directory as a new starting point. setenv RTE_SDK /home/user/DPDK cp -r $(RTE_SDK)/examples/helloworld my_rte_app cd my_rte_app/ setenv RTE_TARGET x86_64-native-bsdapp-gcc gmake CC=gcc48 CC main.o LD helloworld INSTALL-APP helloworld INSTALL-MAP helloworld.map

2.4.2 Running a Sample Application 1. The contigmem and nic_uio modules must be set up prior to running an application. 2. Any ports to be used by the application must be already bound to the nic_uio module, as described in section Binding Network Ports to the nic_uio Module, prior to running the application. The application is linked with the DPDK target environment’s Environment Abstraction Layer (EAL) library, which provides some options that are generic to every DPDK application. The following is the list of options that can be given to the EAL: ./rte-app -c COREMASK [-n NUM] [-b ] \ [-r NUM] [-v] [--proc-type ]

Note: EAL has a common interface between all operating systems and is based on the Linux notation for PCI devices. For example, a FreeBSD device selector of pci0:2:0:1 is referred to as 02:00.1 in EAL. The EAL options for FreeBSD are as follows: • -c COREMASK: A hexadecimal bit mask of the cores to run on. Note that core numbering can change between platforms and should be determined beforehand. • -n NUM: Number of memory channels per processor socket. • -b : Blacklisting of ports; prevent EAL from using specified PCI device (multiple -b options are allowed). • --use-device: Use the specified Ethernet device(s) only. Use comma-separate [domain:]bus:devid.func values. Cannot be used with -b option. • -r NUM: Number of memory ranks. • -v: Display version information on startup. • --proc-type: The type of process instance. Other options, specific to Linux and are not supported under FreeBSD are as follows: • socket-mem: Memory to allocate from hugepages on specific sockets. • --huge-dir: The directory where hugetlbfs is mounted.

2.4. Compiling and Running Sample Applications

35

DPDK documentation, Release 16.07.0

• --file-prefix: The prefix text used for hugepage filenames. • -m MB: Memory to allocate from hugepages, regardless of processor socket. It is recommended that --socket-mem be used instead of this option. The -c option is mandatory; the others are optional. Copy the DPDK application binary to your target, then run the application as follows (assuming the platform has four memory channels, and that cores 0-3 are present and are to be used for running the application): ./helloworld -c f -n 4

Note: The --proc-type and --file-prefix EAL options are used for running multiple DPDK processes. See the “Multi-process Sample Application” chapter in the DPDK Sample Applications User Guide and the DPDK Programmers Guide for more details.

2.4.3 Running DPDK Applications Without Root Privileges Although applications using the DPDK use network ports and other hardware resources directly, with a number of small permission adjustments, it is possible to run these applications as a user other than “root”. To do so, the ownership, or permissions, on the following file system objects should be adjusted to ensure that the user account being used to run the DPDK application has access to them: • The userspace-io device files in /dev, for example, /dev/uio0, /dev/uio1, and so on • The userspace contiguous memory device: /dev/contigmem Note: Please refer to the DPDK Release Notes for supported applications.

2.4. Compiling and Running Sample Applications

36

CHAPTER 3

Xen Guide

3.1 DPDK Xen Based Packet-Switching Solution 3.1.1 Introduction DPDK provides a para-virtualization packet switching solution, based on the Xen hypervisor’s Grant Table, Note 1, which provides simple and fast packet switching capability between guest domains and host domain based on MAC address or VLAN tag. This solution is comprised of two components; a Poll Mode Driver (PMD) as the front end in the guest domain and a switching back end in the host domain. XenStore is used to exchange configure information between the PMD front end and switching back end, including grant reference IDs for shared Virtio RX/TX rings, MAC address, device state, and so on. XenStore is an information storage space shared between domains, see further information on XenStore below. The front end PMD can be found in the DPDK directory lib/ librte_pmd_xenvirt and back end example in examples/vhost_xen. The PMD front end and switching back end use shared Virtio RX/TX rings as para- virtualized interface. The Virtio ring is created by the front end, and Grant table references for the ring are passed to host. The switching back end maps those grant table references and creates shared rings in a mapped address space. The following diagram describes the functionality of the DPDK Xen Packet- Switching Solution. Note 1 The Xen hypervisor uses a mechanism called a Grant Table to share memory between domains (http://wiki.xen.org/wiki/Grant Table). A diagram of the design is shown below, where “gva” is the Guest Virtual Address, which is the 0_mempool_va="0x7fcbc6881000" 0_tx_vring_gref="3049" 0_rx_vring_gref="3053" 0_ether_addr="4e:0b:d0:4e:aa:f1" 0_vring_flag="3054" ...

Multiple mempools and multiple Virtios may exist in the guest domain, the first number is the index, starting from zero. The idx#_mempool_va stores the guest virtual address for mempool idx#.

3.1. DPDK Xen Based Packet-Switching Solution

39

DPDK documentation, Release 16.07.0

The idx#_ether_adder stores the MAC address of the guest Virtio device. For idx#_rx_ring_gref, idx#_tx_ring_gref, and idx#_mempool_gref, the value is a list of Grant references. Take idx#_mempool_gref node for example, the host maps those Grant references to a continuous virtual address space. The real Grant reference information is stored in this virtual address space, where (gref, pfn) pairs follow each other with -1 as the terminator.

Fig. 3.3: Mapping Grant references to a continuous virtual address space After all gref# IDs are retrieved, the host maps them to a continuous virtual address space. With the guest mempool virtual address, the host establishes 1:1 address mapping. With multiple guest mempools, the host establishes multiple address translation regions. Switching Back End The switching back end monitors changes in XenStore. When the back end detects that a new Virtio device has been created in a guest domain, it will: 1. Retrieve Grant and configuration information from XenStore. 2. Map and create a Virtio ring. 3. Map mempools in the host and establish address translation between the guest address and host address. 4. Select a free VMDQ pool, set its affinity with the Virtio device, and set the MAC/ VLAN filter. Packet Reception When packets arrive from an external network, the MAC?VLAN filter classifies packets into queues in one VMDQ pool. As each pool is bonded to a Virtio device in some guest domain, the switching back end will: 1. Fetch an available entry from the Virtio RX ring. 2. Get gva, and translate it to hva. 3. Copy the contents of the packet to the memory buffer pointed to by gva. The DPDK application in the guest domain, based on the PMD front end, is polling the shared Virtio RX ring for available packets and receives them on arrival. Packet Transmission When a Virtio device in one guest domain is to transmit a packet, it puts the virtual address of the packet’s eth_xenvirt0,mac=00:00:00:00:00:11" --vdev="eth_xenvirt1;mac=00:00:00:00:00:22"

Usage Examples: Injecting a Packet Stream Using a Packet Generator Loopback Mode

Run TestPMD in a guest VM: ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 --vdev="eth_xenvirt0,mac=00:00:00:00:00:11" testpmd> set fwd mac testpmd> start

Example output of the vhost_switch would be:

Run TestPMD in guest VM2: ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 --vdev="eth_xenvirt0,mac=00:00:00:00:00:22"

Configure a packet stream in the packet generator, and set the destination MAC address to 00:00:00:00:00:11 and VLAN to 1000. The packets received in Virtio in guest VM1 will be forwarded to Virtio in guest VM2 and then sent out through hardware with destination MAC address 00:00:00:00:00:33. The packet flow is: packet generator->Virtio in guest VM1->switching backend->Virtio in guest VM2->switching backend->wire

3.1. DPDK Xen Based Packet-Switching Solution

44

CHAPTER 4

Programmer’s Guide

4.1 Introduction This document provides software architecture information, development environment information and optimization guidelines. For programming examples and for instructions on compiling and running each sample application, see the DPDK Sample Applications User Guide for details. For general information on compiling and running applications, see the DPDK Getting Started Guide.

4.1.1 Documentation Roadmap The following is a list of DPDK documents in the suggested reading order: • Release Notes (this document): Provides release-specific information, including supported features, limitations, fixed issues, known issues and so on. Also, provides the answers to frequently asked questions in FAQ format. • Getting Started Guide : Describes how to install and configure the DPDK software; designed to get users up and running quickly with the software. • FreeBSD* Getting Started Guide : A document describing the use of the DPDK with FreeBSD* has been added in DPDK Release 1.6.0. Refer to this guide for installation and configuration instructions to get started using the DPDK with FreeBSD*. • Programmer’s Guide (this document): Describes: – The software architecture and how to use it (through examples), specifically in a Linux* application (linuxapp) environment – The content of the DPDK, the build system (including the commands that can be used in the root DPDK Makefile to build the development kit and an application) and guidelines for porting an application – Optimizations used in the software and those that should be considered for new development A glossary of terms is also provided. • API Reference : Provides detailed information about DPDK functions,

This command runs the kni sample application with two physical ports. Each port pins two forwarding cores (ingress/egress) in user space. 4. Assign a raw socket to vhost-net during qemu-kvm startup. The DPDK does not provide a script to do this since it is easy for the user to customize. The following shows the key steps to launch qemu-kvm with kni-vhost: #!/bin/bash echo 1 > /sys/class/net/vEth0/sock_en fd=`cat /sys/class/net/vEth0/sock_fd`

4.20. Kernel NIC Interface

124

DPDK documentation, Release 16.07.0

qemu-kvm \ -name vm1 -cpu host -m 2048 -smp 1 -hda /opt/vm-fc16.img \ -netdev tap,fd=$fd,id=hostnet1,vhost=on \ -device virti-net-pci,netdev=hostnet1,id=net1,bus=pci.0,addr=0x4

It is simple to enable raw socket using sysfs sock_en and get raw socket fd using sock_fd under the KNI device node. Then, using the qemu-kvm command with the -netdev option to assign such raw socket fd as vhost’s backend. Note: The key word tap must exist as qemu-kvm now only supports vhost with a tap backend, so here we cheat qemu-kvm by an existing fd.

Compatibility Configure Option There is a CONFIG_RTE_KNI_VHOST_VNET_HDR_EN configuration option in DPDK configuration file. By default, it set to n, which means do not turn on the virtio net header, which is used to support additional features (such as, csum offload, vlan offload, generic-segmentation and so on), since the kni-vhost does not yet support those features. Even if the option is turned on, kni-vhost will ignore the information that the header contains. When working with legacy virtio on the guest, it is better to turn off unsupported offload features using ethtool -K. Otherwise, there may be problems such as an incorrect L4 checksum error.

4.21 Thread Safety of DPDK Functions The DPDK is comprised of several libraries. Some of the functions in these libraries can be safely called from multiple threads simultaneously, while others cannot. This section allows the developer to take these issues into account when building their own application. The run-time environment of the DPDK is typically a single thread per logical core. In some cases, it is not only multi-threaded, but multi-process. Typically, it is best to avoid sharing {'name' : '', ...}\"";

These strings can then be searched for by external tools to determine the hardware support of a given library or application.

4.29. Development Kit Build System

204

DPDK documentation, Release 16.07.0

Useful Variables Provided by the Build System • RTE_SDK: The absolute path to the DPDK sources. When compiling the development kit, this variable is automatically set by the framework. It has to be defined by the user as an environment variable if compiling an external application. • RTE_SRCDIR: The path to the root of the sources. When compiling the development kit, RTE_SRCDIR = RTE_SDK. When compiling an external application, the variable points to the root of external application sources. • RTE_OUTPUT: The path to which output files are written. Typically, it is $(RTE_SRCDIR)/build, but it can be overridden by the O= option in the make command line. • RTE_TARGET: A string identifying the target for which we are building. The format is arch-machine-execenv-toolchain. When compiling the SDK, the target is deduced by the build system from the configuration (.config). When building an external application, it must be specified by the user in the Makefile or as an environment variable. • RTE_SDK_BIN: References $(RTE_SDK)/$(RTE_TARGET). • RTE_ARCH: Defines the architecture (i686, x86_64). It is the same value as CONFIG_RTE_ARCH but without the double-quotes around the string. • RTE_MACHINE: Defines the machine. It is the same value as CONFIG_RTE_MACHINE but without the double-quotes around the string. • RTE_TOOLCHAIN: Defines the toolchain (gcc , icc). It is the same value as CONFIG_RTE_TOOLCHAIN but without the double-quotes around the string. • RTE_EXEC_ENV: Defines the executive environment (linuxapp). It is the same value as CONFIG_RTE_EXEC_ENV but without the double-quotes around the string. • RTE_KERNELDIR: This variable contains the absolute path to the kernel sources that will be used to compile the kernel modules. The kernel headers must be the same as the ones that will be used on the target machine (the machine that will run the application). By default, the variable is set to /lib/modules/$(shell uname -r)/build, which is correct when the target machine is also the build machine. • RTE_DEVEL_BUILD: Stricter options (stop on warning). It defaults to y in a git tree. Variables that Can be Set/Overridden in a Makefile Only • VPATH: The path list that the build system will search for sources. RTE_SRCDIR will be included in VPATH.

By default,

• CFLAGS: Flags to use for C compilation. The user should use += to append

The above lines load the contigmem kernel module during boot process and allocate 2 x 1G blocks of contiguous memory to be used for DPDK later on. This is to avoid issues with potential memory fragmentation during later system up time, which may result in failure of allocating the contiguous memory required for the contigmem kernel module. 4. Restart the system and ensure the contigmem module is loaded successfully: reboot kldstat | grep "contigmem"

Example output: 2

1 0xffffffff817f1000 3118

contigmem.ko

5. Repeat step 1 to ensure that you are in the DPDK source directory. 6. Load the cxgbe kernel module: kldload if_cxgbe

7. Get the PCI bus addresses of the interfaces bound to t5nex driver: pciconf -l | grep "t5nex"

Example output: 5.4. CXGBE Poll Mode Driver

235

DPDK documentation, Release 16.07.0

t5nex0@pci0:2:0:4: class=0x020000 card=0x00001425 chip=0x54011425 rev=0x00

In the above example, the t5nex0 is bound to 2:0:4 bus address. Note: Both the interfaces of a Chelsio T5 2-port adapter are bound to the same PCI bus address. 8. Unload the kernel module: kldunload if_cxgbe

9. Set the PCI bus addresses to hw.nic_uio.bdfs kernel environment parameter: kenv hw.nic_uio.bdfs="2:0:4"

This automatically binds 2:0:4 to nic_uio kernel driver when it is loaded in the next step. Note: Currently, CXGBE PMD only supports the binding of PF4 for Chelsio T5 NICs. 10. Load nic_uio kernel driver: kldload ./x86_64-native-bsdapp-clang/kmod/nic_uio.ko

11. Start testpmd with basic parameters: ./x86_64-native-bsdapp-clang/app/testpmd -c 0xf -n 4 -w 0000:02:00.4 -- -i

Example output: [...] EAL: PCI device 0000:02:00.4 on NUMA socket 0 EAL: probe driver: 1425:5401 rte_cxgbe_pmd EAL: PCI memory mapped at 0x8007ec000 EAL: PCI memory mapped at 0x842800000 EAL: PCI memory mapped at 0x80086c000 PMD: rte_cxgbe_pmd: fw: 1.13.32.0, TP: 0.1.4.8 PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter Interactive-mode selected Configuring Port 0 (socket 0) Port 0: 00:07:43:2D:EA:C0 Configuring Port 1 (socket 0) Port 1: 00:07:43:2D:EA:C8 Checking link statuses... PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted Port 0 Link Up - speed 10000 Mbps - full-duplex Port 1 Link Up - speed 10000 Mbps - full-duplex Done testpmd>

Note: Flow control pause TX/RX is disabled by default and can be enabled via testpmd. Refer section Enable/Disable Flow Control for more details.

5.4. CXGBE Poll Mode Driver

236

DPDK documentation, Release 16.07.0

5.4.8 Sample Application Notes Enable/Disable Flow Control Flow control pause TX/RX is disabled by default and can be enabled via testpmd as follows: testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 0 testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 1

To disable again, run: testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 0 testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 1

Jumbo Mode There are two ways to enable sending and receiving of jumbo frames via testpmd. One method involves using the mtu command, which changes the mtu of an individual port without having to stop the selected port. Another method involves stopping all the ports first and then running max-pkt-len command to configure the mtu of all the ports with a single command. • To configure each port individually, run the mtu command as follows: testpmd> port config mtu 0 9000 testpmd> port config mtu 1 9000

• To configure all the ports at once, stop all the ports first and run the max-pkt-len command as follows: testpmd> port stop all testpmd> port config all max-pkt-len 9000

5.5 Driver for VM Emulated Devices The DPDK EM poll mode driver supports the following emulated devices: • qemu-kvm emulated Intel® 82540EM Gigabit Ethernet Controller (qemu e1000 device) • VMware* emulated Intel® 82545EM Gigabit Ethernet Controller • VMware emulated Intel® 8274L Gigabit Ethernet Controller.

5.5.1 Validated Hypervisors The validated hypervisors are: • KVM (Kernel Virtual Machine) with Qemu, version 0.14.0 • KVM (Kernel Virtual Machine) with Qemu, version 0.15.1 • VMware ESXi 5.0, Update 1

5.5. Driver for VM Emulated Devices

237

DPDK documentation, Release 16.07.0

5.5.2 Recommended Guest Operating System in Virtual Machine The recommended guest operating system in a virtualized environment is: • Fedora* 18 (64-bit) For supported kernel versions, refer to the DPDK Release Notes.

5.5.3 Setting Up a KVM Virtual Machine The following describes a target environment: • Host Operating System: Fedora 14 • Hypervisor: KVM (Kernel Virtual Machine) with Qemu version, 0.14.0 • Guest Operating System: Fedora 14 • Linux Kernel Version: Refer to the DPDK Getting Started Guide • Target Applications: testpmd The setup procedure is as follows: 1. Download qemu-kvm-0.14.0 from http://sourceforge.net/projects/kvm/files/qemu-kvm/ and install it in the Host OS using the following steps: When using a recent kernel (2.6.25+) with kvm modules included: tar xzf qemu-kvm-release.tar.gz cd qemu-kvm-release ./configure --prefix=/usr/local/kvm make sudo make install sudo /sbin/modprobe kvm-intel

When using an older kernel or a kernel from a distribution without the kvm modules, you must download (from the same link), compile and install the modules yourself: tar xjf kvm-kmod-release.tar.bz2 cd kvm-kmod-release ./configure make sudo make install sudo /sbin/modprobe kvm-intel

Note that qemu-kvm installs in the /usr/local/bin directory. For more details about KVM configuration and usage, please refer to: http://www.linuxkvm.org/page/HOWTO1. 2. Create a Virtual Machine and install Fedora 14 on the Virtual Machine. This is referred to as the Guest Operating System (Guest OS). 3. Start the Virtual Machine with at least one emulated e1000 device. Note: The Qemu provides several choices for the emulated network device backend. Most commonly used is a TAP networking backend that uses a TAP networking device in the host. For more information about Qemu supported networking backends and different options for configuring networking at Qemu, please refer to: — http://www.linux-kvm.org/page/Networking

5.5. Driver for VM Emulated Devices

238

DPDK documentation, Release 16.07.0

— http://wiki.qemu.org/Documentation/Networking — http://qemu.weilnetz.de/qemu-doc.html For example, to start a VM with two emulated e1000 devices, issue the following command: /usr/local/kvm/bin/qemu-system-x86_64 -cpu host -smp 4 -hda qemu1.raw -m 1024 -net nic,model=e1000,vlan=1,macaddr=DE:AD:1E:00:00:01 -net tap,vlan=1,ifname=tapvm01,script=no,downscript=no -net nic,model=e1000,vlan=2,macaddr=DE:AD:1E:00:00:02 -net tap,vlan=2,ifname=tapvm02,script=no,downscript=no

where: — -m = memory to assign — -smp = number of smp cores — -hda = virtual disk image This command starts a new virtual machine with two emulated 82540EM devices, backed up with two TAP networking host interfaces, tapvm01 and tapvm02. # ip tuntap show tapvm01: tap tapvm02: tap

4. Configure your TAP networking interfaces using ip/ifconfig tools. 5. Log in to the guest OS and check that the expected emulated devices exist:

# lspci -d 8086:100e 00:04.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03 00:05.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03

6. Install the DPDK and run testpmd.

5.5.4 Known Limitations of Emulated Devices The following are known limitations: 1. The Qemu e1000 RX path does not support multiple descriptors/buffers per packet. Therefore, rte_mbuf should be big enough to hold the whole packet. For example, to allow testpmd to receive jumbo frames, use the following: testpmd [options] – –mbuf-size= 2. Qemu e1000 does not validate the checksum of incoming packets. 3. Qemu e1000 only supports one interrupt source, so link and Rx interrupt should be exclusive. 4. Qemu e1000 does not support interrupt auto-clear, application should disable interrupt immediately when woken up.

5.5. Driver for VM Emulated Devices

239

DPDK documentation, Release 16.07.0

5.6 ENA Poll Mode Driver The ENA PMD is a DPDK poll-mode driver for the Amazon Elastic Network Adapter (ENA) family.

5.6.1 Overview The ENA driver exposes a lightweight management interface with a minimal set of memory mapped registers and an extendable command set through an Admin Queue. The driver supports a wide range of ENA adapters, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and it negotiates and supports an extendable feature set. ENA adapters allow high speed and low overhead Ethernet traffic processing by providing a dedicated Tx/Rx queue pair per CPU core. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds.

5.6.2 Management Interface ENA management interface is exposed by means of: • Device Registers • Admin Queue (AQ) and Admin Completion Queue (ACQ) ENA device memory-mapped PCIe space for registers (MMIO registers) are accessed only during driver initialization and are not involved in further normal device operation. AQ is used for submitting management commands, and the results/responses are reported asynchronously through ACQ. ENA introduces a very small set of management commands with room for vendor-specific extensions. Most of the management operations are framed in a generic Get/Set feature command. The following admin queue commands are supported: • Create I/O submission queue • Create I/O completion queue • Destroy I/O submission queue • Destroy I/O completion queue • Get feature • Set feature • Get statistics

5.6. ENA Poll Mode Driver

240

DPDK documentation, Release 16.07.0

Refer to ena_admin_defs.h for the list of supported Get/Set Feature properties.

5.6.3

This command generates one network device vEth0 for physical port. If specify more physical ports, the generated network device will be vEth1, vEth2, and so on. For each physical port, kni creates two user threads. One thread loops to fetch packets from the physical NIC port into the kni receive queue. The other user thread loops to send packets in the kni transmit queue. For each physical port, kni also creates a kernel thread that retrieves packets from the kni receive queue, place them onto kni’s raw socket’s queue and wake up the vhost kernel thread to exchange packets with the virtio virt queue. For more details about kni, please refer to Kernel NIC Interface. 3. Enable the kni raw socket functionality for the specified physical NIC port, get the generated file descriptor and set it in the qemu command line parameter. Always remember to set ioeventfd_on and vhost_on. Example: echo 1 > /sys/class/net/vEth0/sock_en fd=`cat /sys/class/net/vEth0/sock_fd` exec qemu-system-x86_64 -enable-kvm -cpu host \ -m 2048 -smp 4 -name dpdk-test1-vm1 \ -drive file=/

6.3 AES-NI GCM Crypto Poll Mode Driver The AES-NI GCM PMD (librte_pmd_aesni_gcm) provides poll mode crypto driver support for utilizing Intel multi buffer library (see AES-NI Multi-buffer PMD documentation to learn more about it, including installation). The AES-NI GCM PMD has current only been tested on Fedora 21 64-bit with gcc.

6.3.1 Features AESNI GCM PMD has support for: Cipher algorithms: • RTE_CRYPTO_CIPHER_AES_GCM Authentication algorithms: • RTE_CRYPTO_AUTH_AES_GCM

6.3.2 Initialization In order to enable this virtual crypto PMD, user must: • Export the environmental variable AESNI_MULTI_BUFFER_LIB_PATH with the path where the library was extracted. • Build the multi buffer library (go to Installation section in AES-NI MB PMD documentation). • Set CONFIG_RTE_LIBRTE_PMD_AESNI_GCM=y in config/common_base. To use the PMD in an application, user must: • Call rte_eal_vdev_init(“cryptodev_aesni_gcm_pmd”) within the application. • Use –vdev=”cryptodev_aesni_gcm_pmd” in the EAL options, rte_eal_vdev_init() internally.

which will call

The following parameters (all optional) can be provided in the previous two calls: • socket_id: Specify the socket where the memory for the device is going to be allocated (by default, socket_id will be the socket where the core that is creating the PMD is running on). • max_nb_queue_pairs: Specify the maximum number of queue pairs in the device (8 by default). • max_nb_sessions: Specify the maximum number of sessions that can be created (2048 by default). Example: ./l2fwd-crypto -c 40 -n 4 --vdev="cryptodev_aesni_gcm_pmd,socket_id=1,max_nb_sessions=128"

6.3. AES-NI GCM Crypto Poll Mode Driver

313

DPDK documentation, Release 16.07.0

6.3.3 Limitations • Chained mbufs are not supported. • Hash only is not supported. • Cipher only is not supported. • Only in-place is currently supported (destination address is the same as source address). • Only supports session-oriented API implementation (session-less APIs are not supported). • Not performance tuned.

6.4 KASUMI Crypto Poll Mode Driver The KASUMI PMD (librte_pmd_kasumi) provides poll mode crypto driver support for utilizing Intel Libsso library, which implements F8 and F9 functions for KASUMI UEA1 cipher and UIA1 hash algorithms.

6.4.1 Features KASUMI PMD has support for: Cipher algorithm: • RTE_CRYPTO_SYM_CIPHER_KASUMI_F8 Authentication algorithm: • RTE_CRYPTO_SYM_AUTH_KASUMI_F9

6.4.2 Limitations • Chained mbufs are not supported. • KASUMI(F9) supported only if hash offset field is byte-aligned. • In-place bit-level operations for KASUMI(F8) are not supported (if length and/or offset of

6.5 Null Crypto Poll Mode Driver The Null Crypto PMD (librte_pmd_null_crypto) provides a crypto poll mode driver which provides a minimal implementation for a software crypto device. As a null device it does not modify the

6.6 SNOW 3G Crypto Poll Mode Driver The SNOW 3G PMD (librte_pmd_snow3g) provides poll mode crypto driver support for utilizing Intel Libsso library, which implements F8 and F9 functions for SNOW 3G UEA2 cipher and UIA2 hash algorithms.

6.6.1 Features SNOW 3G PMD has support for: Cipher algorithm: • RTE_CRYPTO_SYM_CIPHER_SNOW3G_UEA2 6.6. SNOW 3G Crypto Poll Mode Driver

316

DPDK documentation, Release 16.07.0

Authentication algorithm: • RTE_CRYPTO_SYM_AUTH_SNOW3G_UIA2

6.6.2 Limitations • Chained mbufs are not supported. • Snow3g(UIA2) supported only if hash offset field is byte-aligned. • In-place bit-level operations for Snow3g(UEA2) are not supported (if length and/or offset of

6.6. SNOW 3G Crypto Poll Mode Driver

317

DPDK documentation, Release 16.07.0

6.7 Quick Assist Crypto Poll Mode Driver The QAT PMD provides poll mode crypto driver support for Intel QuickAssist Technology DH895xxC hardware accelerator.

6.7.1 Features The QAT PMD has support for: Cipher algorithms: • RTE_CRYPTO_SYM_CIPHER_AES128_CBC • RTE_CRYPTO_SYM_CIPHER_AES192_CBC • RTE_CRYPTO_SYM_CIPHER_AES256_CBC • RTE_CRYPTO_SYM_CIPHER_AES128_CTR • RTE_CRYPTO_SYM_CIPHER_AES192_CTR • RTE_CRYPTO_SYM_CIPHER_AES256_CTR • RTE_CRYPTO_SYM_CIPHER_SNOW3G_UEA2 • RTE_CRYPTO_CIPHER_AES_GCM Hash algorithms: • RTE_CRYPTO_AUTH_SHA1_HMAC • RTE_CRYPTO_AUTH_SHA256_HMAC • RTE_CRYPTO_AUTH_SHA512_HMAC • RTE_CRYPTO_AUTH_AES_XCBC_MAC • RTE_CRYPTO_AUTH_SNOW3G_UIA2

6.7.2 Limitations • Chained mbufs are not supported. • Hash only is not supported except Snow3G UIA2. • Cipher only is not supported except Snow3G UEA2. • Only supports the session-oriented API implementation (session-less APIs are not supported). • Not performance tuned. • Snow3g(UEA2) supported only if cipher length, cipher offset fields are byte-aligned. • Snow3g(UIA2) supported only if hash length, hash offset fields are byte-aligned. • No BSD support as BSD QAT kernel driver not available.

6.7. Quick Assist Crypto Poll Mode Driver

318

DPDK documentation, Release 16.07.0

6.7.3 Installation To use the DPDK QAT PMD an SRIOV-enabled QAT kernel driver is required. The VF devices exposed by this driver will be used by QAT PMD. If you are running on kernel 4.4 or greater, see instructions for Installation using kernel.org driver below. If you are on a kernel earlier than 4.4, see Installation using 01.org QAT driver .

6.7.4 Installation using 01.org QAT driver Download the latest QuickAssist Technology Driver from 01.org Consult the Getting Started Guide at the same URL for further information. The steps below assume you are: • Building on a platform with one DH895xCC device. • Using package qatmux.l.2.3.0-34.tgz. • On Fedora21 kernel 3.17.4-301.fc21.x86_64. In the BIOS ensure that SRIOV is enabled and VT-d is disabled. Uninstall any existing QAT driver, for example by running: • ./installer.sh uninstall in the directory where originally installed. • or rmmod qat_dh895xcc; rmmod intel_qat. Build and install the SRIOV-enabled QAT driver: mkdir /QAT cd /QAT # copy qatmux.l.2.3.0-34.tgz to this location tar zxof qatmux.l.2.3.0-34.tgz export ICP_WITHOUT_IOMMU=1 ./installer.sh install QAT1.6 host

You can use cat /proc/icp_dh895xcc_dev0/version to confirm the driver is correctly installed. You can use lspci -d:443 to confirm the bdf of the 32 VF devices are available per DH895xCC device. To complete the installation - follow instructions in Binding the available VFs to the DPDK UIO driver . Note: If using a later kernel and the build fails with an error relating to strict_stroul not being available apply the following patch:

/QAT/QAT1.6/quickassist/utilities/downloader/Target_CoreLibs/uclo/include/linux/uclo_platform.h + #if LINUX_VERSION_CODE >= KERNEL_VERSION(3,18,5) + #define STR_TO_64(str, base, num, endPtr) {endPtr=NULL; if (kstrtoul((str), (base), (num))) p + #else #if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,38) #define STR_TO_64(str, base, num, endPtr) {endPtr=NULL; if (strict_strtoull((str), (base), (num #else #if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,25) #define STR_TO_64(str, base, num, endPtr) {endPtr=NULL; strict_strtoll((str), (base), (num));} #else #define STR_TO_64(str, base, num, endPtr) \ do { \ if (str[0] == '-') \

6.7. Quick Assist Crypto Poll Mode Driver

319

DPDK documentation, Release 16.07.0

{

\ *(num) = -(simple_strtoull((str+1), &(endPtr), (base))); \ }else { \ \ *(num) = simple_strtoull((str), &(endPtr), (base)); } \ } while(0) + #endif #endif #endif

If the build fails due to missing header files you may need to do following: • sudo yum install zlib-devel • sudo yum install openssl-devel If the build or install fails due to mismatching kernel sources you may need to do the following: • sudo yum install kernel-headers-‘uname -r‘ • sudo yum install kernel-src-‘uname -r‘ • sudo yum install kernel-devel-‘uname -r‘

6.7.5 Installation using kernel.org driver Assuming you are running on at least a 4.4 kernel, you can use the stock kernel.org QAT driver to start the QAT hardware. The steps below assume you are: • Running DPDK on a platform with one DH895xCC device. • On a kernel at least version 4.4. In BIOS ensure that SRIOV is enabled and VT-d is disabled. Ensure the QAT driver is loaded on your system, by executing: lsmod | grep qat

You should see the following output: qat_dh895xcc intel_qat

5626 82336

0 1 qat_dh895xcc

Next, you need to expose the VFs using the sysfs file system. First find the bdf of the DH895xCC device: lspci -d : 435

You should see output similar to: 03:00.0 Co-processor: Intel Corporation Coleto Creek PCIe Endpoint

Using the sysfs, enable the VFs: echo 32 > /sys/bus/pci/drivers/dh895xcc/0000\:03\:00.0/sriov_numvfs

If you get an error, it’s likely you’re using a QAT kernel driver earlier than kernel 4.4. To verify that the VFs are available for use - use lspci -d:443 to confirm the bdf of the 32 VF devices are available per DH895xCC device.

6.7. Quick Assist Crypto Poll Mode Driver

320

DPDK documentation, Release 16.07.0

To complete the installation - follow instructions in Binding the available VFs to the DPDK UIO driver . Note: If the QAT kernel modules are not loaded and you see an error like Failed to load MMP firmware qat_895xcc_mmp.bin this may be as a result of not using a distribution, but just updating the kernel directly. Download firmware from the kernel firmware repo at: http://git.kernel.org/cgit/linux/kernel/git/firmware/linuxfirmware.git/tree/ Copy qat binaries to /lib/firmware: * cp qat_895xcc.bin /lib/firmware * cp qat_895xcc_mmp.bin /lib/firmware cd to your linux source root directory and start the qat kernel modules: * insmod ./drivers/crypto/qat/qat_common/intel_qat.ko * insmod ./drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.ko Note:The following warning in /var/log/messages can be ignored: IOMMU should be enabled for SR-IOV to work correctly

6.7.6 Binding the available VFs to the DPDK UIO driver The unbind command below assumes bdfs of 03:01.00-03:04.07, if yours are different adjust the unbind command below: cd $RTE_SDK modprobe uio insmod ./build/kmod/igb_uio.ko for device in $(seq 1 4); do \ for fn in $(seq 0 7); do \ echo -n 0000:03:0${device}.${fn} > \ /sys/bus/pci/devices/0000\:03\:0${device}.${fn}/driver/unbind; \ done; \ done echo "8086 0443" > /sys/bus/pci/drivers/igb_uio/new_id

You can use lspci -vvd:443 to confirm that all devices are now in use by igb_uio kernel driver.

6.7. Quick Assist Crypto Poll Mode Driver

321

CHAPTER 7

Sample Applications User Guide

7.1 Introduction This document describes the sample applications that are included in the (0,4,6,8),(1,5,7,9)"

7.11.5 KNI Operations Once the KNI application is started, one can use different Linux* commands to manage the net interfaces. If more than one KNI devices configured for a physical port, only the first KNI device will be paired to the physical device. Operations on other KNI devices will not affect the physical port handled in user space application. Assigning an IP address: #ifconfig vEth0_0 192.168.0.1

Displaying the NIC registers: #ethtool -d vEth0_0

Dumping the network traffic: #tcpdump -i vEth0_0

When the DPDK userspace application is closed, all the KNI devices are deleted from Linux*.

7.11.6 Explanation The following sections provide some explanation of code. Initialization Setup of mbuf pool, driver and queues is similar to the setup done in the L2 Forwarding Sample Application (in Real and Virtualized Environments).. In addition, one or more kernel NIC interfaces are allocated for each of the configured ports according to the command line parameters. The code for allocating the kernel NIC interfaces for a specific port is as follows: static int kni_alloc(uint8_t port_id) {

7.11. Kernel NIC Interface Sample Application

355

DPDK documentation, Release 16.07.0

uint8_t i; struct rte_kni *kni; struct rte_kni_conf conf; struct kni_port_params **params = kni_port_params_array; if (port_id >= RTE_MAX_ETHPORTS || !params[port_id]) return -1; params[port_id]->nb_kni = params[port_id]->nb_lcore_k ? params[port_id]->nb_lcore_k : 1; for (i = 0; i < params[port_id]->nb_kni; i++) { /* Clear conf at first */ memset(&conf, 0, sizeof(conf)); if (params[port_id]->nb_lcore_k) { snprintf(conf.name, RTE_KNI_NAMESIZE, "vEth%u_%u", port_id, i); conf.core_id = params[port_id]->lcore_k[i]; conf.force_bind = 1; } else snprintf(conf.name, RTE_KNI_NAMESIZE, "vEth%u", port_id); conf.group_id = (uint16_t)port_id; conf.mbuf_size = MAX_PACKET_SZ; /* * * * */

The first KNI device associated to a port is the master, for multiple kernel thread environment.

if (i == 0) { struct rte_kni_ops ops; struct rte_eth_dev_info dev_info;

memset(&dev_info, 0, sizeof(dev_info)); rte_eth_dev_info_get(port_id, &dev_inf conf.addr = dev_info.pci_dev->addr; conf.id = dev_info.pci_dev->id; memset(&ops, 0, sizeof(ops)); ops.port_id = port_id; ops.change_mtu = kni_change_mtu; ops.config_network_if = kni_config_network_interface; kni = rte_kni_alloc(pktmbuf_pool, &conf, &ops); } else kni = rte_kni_alloc(pktmbuf_pool, &conf, NULL); if (!kni) rte_exit(EXIT_FAILURE, "Fail to create kni for " "port: %d\n", port_id); params[port_id]->kni[i] = kni; } return 0; }

The other step in the initialization process that is unique to this sample application is the association of each port with lcores for RX, TX and kernel threads. • One lcore to read from the port and write to the associated one or more KNI devices

7.11. Kernel NIC Interface Sample Application

356

DPDK documentation, Release 16.07.0

• Another lcore to read from one or more KNI devices and write to the port • Other lcores for pinning the kernel threads on one by one This is done by using the‘kni_port_params_array[]‘ array, which is indexed by the port ID. The code is as follows: static int parse_config(const char *arg) { const char *p, *p0 = arg; char s[256], *end; unsigned size; enum fieldnames { FLD_PORT = 0, FLD_LCORE_RX, FLD_LCORE_TX, _NUM_FLD = KNI_MAX_KTHREAD + 3, }; int i, j, nb_token; char *str_fld[_NUM_FLD]; unsigned long int_fld[_NUM_FLD]; uint8_t port_id, nb_kni_port_params = 0; memset(&kni_port_params_array, 0, sizeof(kni_port_params_array)); while (((p = strchr(p0, '(')) != NULL) && nb_kni_port_params < RTE_MAX_ETHPORTS) { p++; if ((p0 = strchr(p, ')')) == NULL) goto fail; size = p0 - p; if (size >= sizeof(s)) { printf("Invalid config parameters\n"); goto fail; } snprintf(s, sizeof(s), "%.*s", size, p); nb_token = rte_strsplit(s, sizeof(s), str_fld, _NUM_FLD, ','); if (nb_token = RTE_MAX_ETHPORTS) { printf("Port ID %u could not exceed the maximum %u\n", port_id, RTE_MAX_ETHPORTS); goto fail; } if (kni_port_params_array[port_id]) {

7.11. Kernel NIC Interface Sample Application

357

DPDK documentation, Release 16.07.0

printf("Port %u has been configured\n", port_id); goto fail; }

kni_port_params_array[port_id] = (struct kni_port_params*)rte_zmalloc("KNI_port_params" kni_port_params_array[port_id]->port_id = port_id; kni_port_params_array[port_id]->lcore_rx = (uint8_t)int_fld[i++]; kni_port_params_array[port_id]->lcore_tx = (uint8_t)int_fld[i++];

if (kni_port_params_array[port_id]->lcore_rx >= RTE_MAX_LCORE || kni_port_params_array[ printf("lcore_rx %u or lcore_tx %u ID could not " "exceed the maximum %u\n", kni_port_params_array[port_id]->lcore_rx, kni_port_params_array[port_id]->l goto fail; } for (j = 0; i < nb_token && j < KNI_MAX_KTHREAD; i++, j++) kni_port_params_array[port_id]->lcore_k[j] = (uint8_t)int_fld[i]; kni_port_params_array[port_id]->nb_lcore_k = j; } print_config(); return 0; fail: for (i = 0; i < RTE_MAX_ETHPORTS; i++) { if (kni_port_params_array[i]) { rte_free(kni_port_params_array[i]); kni_port_params_array[i] = NULL; } } return -1; }

Packet Forwarding After the initialization steps are completed, the main_loop() function is run on each lcore. This function first checks the lcore_id against the user provided lcore_rx and lcore_tx to see if this lcore is reading from or writing to kernel NIC interfaces. For the case that reads from a NIC port and writes to the kernel NIC interfaces, the packet reception is the same as in L2 Forwarding sample application (see Receive, Process and Transmit Packets). The packet transmission is done by sending mbufs into the kernel NIC interfaces by rte_kni_tx_burst(). The KNI library automatically frees the mbufs after the kernel successfully copied the mbufs. /** * */

Interface to burst rx and enqueue mbufs into rx_q

static void kni_ingress(struct kni_port_params *p) { uint8_t i, nb_kni, port_id; unsigned nb_rx, num; struct rte_mbuf *pkts_burst[PKT_BURST_SZ];

7.11. Kernel NIC Interface Sample Application

358

DPDK documentation, Release 16.07.0

if (p == NULL) return; nb_kni = p->nb_kni; port_id = p->port_id; for (i = 0; i < nb_kni; i++) { /* Burst rx from eth */ nb_rx = rte_eth_rx_burst(port_id, 0, pkts_burst, PKT_BURST_SZ); if (unlikely(nb_rx > PKT_BURST_SZ)) { RTE_LOG(ERR, APP, "Error receiving from eth\n"); return; } /* Burst tx to kni */ num = rte_kni_tx_burst(p->kni[i], pkts_burst, nb_rx); kni_stats[port_id].rx_packets += num; rte_kni_handle_request(p->kni[i]); if (unlikely(num < nb_rx)) { /* Free mbufs not tx to kni interface */ kni_burst_free_mbufs(&pkts_burst[num], nb_rx - num); kni_stats[port_id].rx_dropped += nb_rx - num; } } }

For the other case that reads from kernel NIC interfaces and writes to a physical NIC port, packets are retrieved by reading mbufs from kernel NIC interfaces by rte_kni_rx_burst(). The packet transmission is the same as in the L2 Forwarding sample application (see Receive, Process and Transmit Packets). /** * */

Interface to dequeue mbufs from tx_q and burst tx

static void kni_egress(struct kni_port_params *p) { uint8_t i, nb_kni, port_id; unsigned nb_tx, num; struct rte_mbuf *pkts_burst[PKT_BURST_SZ]; if (p == NULL) return; nb_kni = p->nb_kni; port_id = p->port_id; for (i = 0; i < nb_kni; i++) { /* Burst rx from kni */ num = rte_kni_rx_burst(p->kni[i], pkts_burst, PKT_BURST_SZ); if (unlikely(num > PKT_BURST_SZ)) { RTE_LOG(ERR, APP, "Error receiving from KNI\n"); return; } /* Burst tx to eth */ nb_tx = rte_eth_tx_burst(port_id, 0, pkts_burst, (uint16_t)num); kni_stats[port_id].tx_packets += nb_tx;

7.11. Kernel NIC Interface Sample Application

359

DPDK documentation, Release 16.07.0

if (unlikely(nb_tx < num)) { /* Free mbufs not tx to NIC */ kni_burst_free_mbufs(&pkts_burst[nb_tx], num - nb_tx); kni_stats[port_id].tx_dropped += num - nb_tx; } } }

Callbacks for Kernel Requests To execute specific PMD operations in user space requested by some Linux* commands, callbacks must be implemented and filled in the struct rte_kni_ops structure. Currently, setting a new MTU and configuring the network interface (up/ down) are supported. static struct rte_kni_ops kni_ops = { .change_mtu = kni_change_mtu, .config_network_if = kni_config_network_interface, }; /* Callback for request of changing MTU */ static int kni_change_mtu(uint8_t port_id, unsigned new_mtu) { int ret; struct rte_eth_conf conf; if (port_id >= rte_eth_dev_count()) { RTE_LOG(ERR, APP, "Invalid port id %d\n", port_id); return -EINVAL; } RTE_LOG(INFO, APP, "Change MTU of port %d to %u\n", port_id, new_mtu); /* Stop specific port */ rte_eth_dev_stop(port_id); memcpy(&conf, &port_conf, sizeof(conf)); /* Set new MTU */ if (new_mtu > ETHER_MAX_LEN) conf.rxmode.jumbo_frame = 1; else conf.rxmode.jumbo_frame = 0; /* mtu + length of header + length of FCS = max pkt length */ conf.rxmode.max_rx_pkt_len = new_mtu + KNI_ENET_HEADER_SIZE + KNI_ENET_FCS_SIZE; ret = rte_eth_dev_configure(port_id, 1, 1, &conf); if (ret < 0) { RTE_LOG(ERR, APP, "Fail to reconfigure port %d\n", port_id); return ret; } /* Restart specific port */ ret = rte_eth_dev_start(port_id);

7.11. Kernel NIC Interface Sample Application

360

DPDK documentation, Release 16.07.0

if (ret < 0) { RTE_LOG(ERR, APP, "Fail to restart port %d\n", port_id); return ret; } return 0; } /* Callback for request of configuring network interface up/down */ static int kni_config_network_interface(uint8_t port_id, uint8_t if_up) { int ret = 0; if (port_id >= rte_eth_dev_count() || port_id >= RTE_MAX_ETHPORTS) { RTE_LOG(ERR, APP, "Invalid port id %d\n", port_id); return -EINVAL; } RTE_LOG(INFO, APP, "Configure network interface of %d %s\n", port_id, if_up ? "up" : "down"); if (if_up != 0) { /* Configure network interface up */ rte_eth_dev_stop(port_id); ret = rte_eth_dev_start(port_id); } else /* Configure network interface down */ rte_eth_dev_stop(port_id); if (ret < 0) RTE_LOG(ERR, APP, "Failed to start port %d\n", port_id); return ret; }

7.12 Keep Alive Sample Application The Keep Alive application is a simple example of a heartbeat/watchdog for packet processing cores. It demonstrates how to detect ‘failed’ DPDK cores and notify a fault management entity of this failure. Its purpose is to ensure the failure of the core does not result in a fault that is not detectable by a management entity.

7.12.1 Overview The application demonstrates how to protect against ‘silent outages’ on packet processing cores. A Keep Alive Monitor Agent Core (master) monitors the state of packet processing cores (worker cores) by dispatching pings at a regular time interval (default is 5ms) and monitoring the state of the cores. Cores states are: Alive, MIA, Dead or Buried. MIA indicates a missed ping, and Dead indicates two missed pings within the specified time interval. When a core is Dead, a callback function is invoked to restart the packet processing core; A real life application might use this callback function to notify a higher level fault management entity of the core failure in order to take the appropriate corrective action. Note: Only the worker cores are monitored. A local (on the host) mechanism or agent to supervise the Keep Alive Monitor Agent Core DPDK core is required to detect its failure.

7.12. Keep Alive Sample Application

361

DPDK documentation, Release 16.07.0

Note: This application is based on the L2 Forwarding Sample Application (in Real and Virtualized Environments). As such, the initialization and run-time paths are very similar to those of the L2 forwarding application.

7.12.2 Compiling the Application To compile the application: 1. Go to the sample application directory: export RTE_SDK=/path/to/rte_sdk cd ${RTE_SDK}/examples/keep_alive

2. Set the target (a default target is used if not specified). For example: export RTE_TARGET=x86_64-native-linuxapp-gcc

See the DPDK Getting Started Guide for possible RTE_TARGET values. 3. Build the application: make

7.12.3 Running the Application The application has a number of command line options: ./build/l2fwd-keepalive [EAL options] \ -- -p PORTMASK [-q NQ] [-K PERIOD] [-T PERIOD]

where, • p PORTMASK: A hexadecimal bitmask of the ports to configure • q NQ: A number of queues (=ports) per lcore (default is 1) • K PERIOD: Heartbeat check period in ms(5ms default; 86400 max) • T PERIOD: statistics will be refreshed each PERIOD seconds (0 to disable, 10 default, 86400 maximum). To run the application in linuxapp environment with 4 lcores, 16 ports 8 RX queues per lcore and a ping interval of 10ms, issue the command: ./build/l2fwd-keepalive -c f -n 4 -- -q 8 -p ffff -K 10

Refer to the DPDK Getting Started Guide for general information on running applications and the Environment Abstraction Layer (EAL) options.

7.12.4 Explanation The following sections provide some explanation of the The Keep-Alive/’Liveliness’ conceptual scheme. As mentioned in the overview section, the initialization and run-time paths are very similar to those of the L2 Forwarding Sample Application (in Real and Virtualized Environments). The Keep-Alive/’Liveliness’ conceptual scheme: • A Keep- Alive Agent Runs every N Milliseconds. • DPDK Cores respond to the keep-alive agent. 7.12. Keep Alive Sample Application

362

DPDK documentation, Release 16.07.0

• If keep-alive agent detects time-outs, it notifies the fault management entity through a callback function. The following sections provide some explanation of the code aspects that are specific to the Keep Alive sample application. The keepalive functionality is initialized with a struct rte_keepalive and the callback function to invoke in the case of a timeout. rte_global_keepalive_info = rte_keepalive_create(&dead_core, NULL); if (rte_global_keepalive_info == NULL) rte_exit(EXIT_FAILURE, "keepalive_create() failed");

The function that issues the pings keepalive_dispatch_pings() is configured to run every check_period milliseconds. if (rte_timer_reset(&hb_timer, (check_period * rte_get_timer_hz()) / 1000, PERIODICAL, rte_lcore_id(), &rte_keepalive_dispatch_pings, rte_global_keepalive_info ) != 0 ) rte_exit(EXIT_FAILURE, "Keepalive setup failure.\n");

The rest of the initialization and run-time path follows the same paths as the the L2 forwarding application. The only addition to the main processing loop is the mark alive functionality and the example random failures. rte_keepalive_mark_alive(&rte_global_keepalive_info); cur_tsc = rte_rdtsc(); /* Die randomly within 7 secs for demo purposes.. */ if (cur_tsc - tsc_initial > tsc_lifetime) break;

The rte_keepalive_mark_alive function simply sets the core state to alive. static inline void rte_keepalive_mark_alive(struct rte_keepalive *keepcfg) { keepcfg->state_flags[rte_lcore_id()] = ALIVE; }

7.13 L2 Forwarding with Crypto Sample Application The L2 Forwarding with Crypto (l2fwd-crypto) sample application is a simple example of packet processing using the

or to enable CAT and CDP on cpus 1,3: ./build/l2fwd-cat -c 2 -n 4 -- --l3ca="(0x00C00,0x00300)@(1,3)"

If CDP is not supported it will fail with following error message: PQOS: CDP requested but not supported. PQOS: Requested CAT configuration is not valid! PQOS: Shutting down PQoS library... EAL: Error - exiting with code: 1 Cause: PQOS: L3CA init failed!

The option to enable CAT is: • --l3ca=’[,