First shot at running VPP inside LXD container
Today I've tried to build and run VPP entirely within the LXD container,
without vagrant or VMs.
My host is running the latest and greatest 16.04:
ayourtch@xenial:~$ cat /etc/issue
Ubuntu Xenial Xerus (development branch) \n \l
ayourtch@xenial:~$
After installing the LXD and configuring it to use the ZFS backend (not describing here),
I've created a container for ubuntu 14.04, and started it.
ayourtch@xenial:~$ lxc launch ubuntu:14.04 vpp
After that I got into the container:
ayourtch@xenial:~$ lxc exec vpp -- su -l ubuntu
ubuntu@vpp:~$
Next we'll need to install git, to be able to check out the sources of VPP and the
build-essential package, which has gcc, make, etc:
ubuntu@vpp:~$ sudo apt-get update && sudo apt-get -y install git build-essential
The next step is to get the VPP sources. https://wiki.fd.io/view/VPP has a lot of info,
but in the nutshell, what we need is this:
ubuntu@vpp:~$ git clone https://gerrit.fd.io/r/vpp
Now we can cd into the vpp subdirectory, and type "make":
ubuntu@vpp:~/vpp$ ls
build-data build-root dpdk g2 gmod Makefile perftool sample-plugin svm test vlib vlib-api vnet vpp vppapigen vpp-api-test vppinfra vpp-japi
ubuntu@vpp:~/vpp$ make
Make Targets:
bootstrap - prepare tree for build
install-dep - install software dependencies
wipe - wipe all products of debug build
wipe-release - wipe all products of release build
build - build debug binaries
build-release - build release binaries
rebuild - wipe and build debug binares
rebuild-release - wipe and build release binares
run - run debug binary
run-release - run release binary
debug - run debug binary with debugger
debug-release - run release binary with debugger
build-vat - build vpp-api-test tool
run-vat - run vpp-api-test tool
pkg-deb - build DEB packages
pkg-rpm - build RPM packages
ctags - (re)generate ctags database
cscope - (re)generate cscope database
Make Arguments:
V=[0|1] - set build verbosity level
STARTUP_CONF= - startup configuration file
(e.g. /etc/vpp/startup.conf)
STARTUP_DIR= - startup drectory (e.g. /etc/vpp)
It also sets STARTUP_CONF if
startup.conf file is present
GDB= - gdb binary to use for debugging
PLATFORM= - target platform. default is vpp
Current Argumernt Values:
V = 0
STARTUP_CONF =
STARTUP_DIR = /home/ubuntu/vpp
GDB = gdb
PLATFORM = vpp
DPDK_VERSION =
ubuntu@vpp:~/vpp$
The next step is to install the necessary dependencies. This is done like this:
ubuntu@vpp:~/vpp$ make install-dep
After the necessary packages are installed, we can prepare the tree for the build:
ubuntu@vpp:~/vpp$ make bootstrap
Finally, let's start the build:
ubuntu@vpp:~/vpp$ make build
The build process most probably will end with errors like this:
ln -nsf `/home/ubuntu/vpp/build-root/build-vpp_debug-native/dpdk/dpdk-2.2.0/scripts/relpath.sh /home/ubuntu/vpp/build-root/build-vpp_debug-native/dpdk/dpdk-2.2.0/lib/librte_eal/linuxapp/eal/include/exec-env/rte_dom0_common.h /home/ubuntu/vpp/build-root/install-vpp_debug-native/dpdk/include/exec-env` /home/ubuntu/vpp/build-root/install-vpp_debug-native/dpdk/include/exec-env
ar crDs librte_eal.a eal.o eal_hugepage_info.o eal_memory.o eal_thread.o eal_log.o eal_pci.o eal_pci_uio.o eal_pci_vfio.o eal_pci_vfio_mp_sync.o eal_debug.o eal_lcore.o eal_timer.o eal_interrupts.o eal_alarm.o eal_common_lcore.o eal_common_timer.o eal_common_memzone.o eal_common_log.o eal_common_launch.o eal_common_pci.o eal_common_pci_uio.o eal_common_memory.o eal_common_tailqs.o eal_common_errno.o eal_common_cpuflags.o eal_common_string_fns.o eal_common_hexdump.o eal_common_devargs.o eal_common_dev.o eal_common_options.o eal_common_thread.o rte_malloc.o malloc_elem.o malloc_heap.o rte_keepalive.o
INSTALL-LIB librte_eal.a
cp -f librte_eal.a /home/ubuntu/vpp/build-root/install-vpp_debug-native/dpdk/lib
make[8]: *** [linuxapp] Error 2
make[7]: *** [librte_eal] Error 2
make[6]: *** [lib] Error 2
make[5]: *** [all] Error 2
make[4]: *** [pre_install] Error 2
make[3]: *** [install] Error 2
make[3]: Leaving directory `/home/ubuntu/vpp/build-root/build-vpp_debug-native/dpdk/dpdk-2.2.0'
make[2]: *** [/home/ubuntu/vpp/build-root/build-vpp_debug-native/dpdk/.build.ok] Error 2
make[2]: Leaving directory `/home/ubuntu/vpp/dpdk'
make[1]: *** [dpdk-install] Error 2
make[1]: Leaving directory `/home/ubuntu/vpp/build-root'
make: *** [build] Error 2
ubuntu@vpp:~/vpp$
If you scroll a bit up, you will see what is going wrong:
== Build lib/librte_eal/linuxapp
== Build lib/librte_eal/linuxapp/igb_uio
== Build lib/librte_eal/linuxapp/eal
make: *** /lib/modules/4.4.0-11-generic/build: No such file or directory. Stop.
make[10]: *** [igb_uio.ko] Error 2
make[9]: *** [igb_uio] Error 2
make[9]: *** Waiting for unfinished jobs....
Unfortunately, there is no exact headers:
ubuntu@vpp:~/vpp$ sudo apt-cache search linux-headers-4.4.0
linux-headers-4.4.0-13 - Header files related to Linux kernel version 4.4.0
linux-headers-4.4.0-13-generic - Linux kernel headers for version 4.4.0 on 64 bit x86 SMP
linux-headers-4.4.0-13-lowlatency - Linux kernel headers for version 4.4.0 on 64 bit x86 SMP
linux-headers-4.4.0-14 - Header files related to Linux kernel version 4.4.0
linux-headers-4.4.0-14-generic - Linux kernel headers for version 4.4.0 on 64 bit x86 SMP
linux-headers-4.4.0-14-lowlatency - Linux kernel headers for version 4.4.0 on 64 bit x86 SMP
ubuntu@vpp:~/vpp$
Therefore we will install the 4.4.0-14 version and symlink the 4.4.0-11 directory to it (it's a bit of a dirty trick but we do not intend to use the kernel modules anyway, at least for now!)
ubuntu@vpp:~/vpp$ sudo apt-get install linux-headers-4.4.0-14-generic
ubuntu@vpp:~/vpp$ sudo ln -s /lib/modules/4.4.0-14-generic /lib/modules/4.4.0-11-generic
Now we can retry "make build":
ubuntu@vpp:~/vpp$ make build
....
/usr/bin/install -c -m 644 /home/ubuntu/vpp/build-data/../vpp/api/vpe_all_api_h.h /home/ubuntu/vpp/build-data/../vpp/api/vpe_msg_enum.h api/vpe.api.h '/home/ubuntu/vpp/build-root/install-vpp_debug-native/vpp/include/api'
CC app/version.o
CCLD vpp
/bin/mkdir -p '/home/ubuntu/vpp/build-root/install-vpp_debug-native/vpp/bin'
/usr/bin/install -c vpp '/home/ubuntu/vpp/build-root/install-vpp_debug-native/vpp/bin'
make[4]: Leaving directory `/home/ubuntu/vpp/build-root/build-vpp_debug-native/vpp'
make[3]: Leaving directory `/home/ubuntu/vpp/build-root/build-vpp_debug-native/vpp'
make[2]: Leaving directory `/home/ubuntu/vpp/build-root/build-vpp_debug-native/vpp'
make[1]: Leaving directory `/home/ubuntu/vpp/build-root'
ubuntu@vpp:~/vpp$
Now we can try to run it:
ubuntu@vpp:~/vpp$ make run
WARNING: STARTUP_CONF not defined or file doesn't exist.
Running with minimal startup config: unix { interactive } dpdk { no-pci socket-mem 1024 }
vlib_plugin_early_init:201: plugin path /usr/lib/vpp_plugins
dpdk_config: mount failed 1
ubuntu@vpp:~/vpp$
The problem with the "mount failed" is that hugetlbfs can not be mounted... First thing we need
to do is to check that the hugepages are enabled:
ubuntu@vpp:~/vpp$ sudo sysctl vm.nr_hugepages
vm.nr_hugepages = 0
ubuntu@vpp:~/vpp$
Looks like they aren't so we need to go to the host and enable them:
ayourtch@xenial:~$ echo "vm.nr_hugepages=1024" | sudo tee -a /etc/sysctl.d/20-hugepages.conf
vm.nr_hugepages=1024
ayourtch@xenial:~$
ayourtch@xenial:~$ sudo sysctl --system
However, without the hugepage support exposed from host to guest, the VPP won't start.
Therefore we need to create this profile back on the host and apply to the container:
lxc profile create hugepages
lxc profile edit hugepages <<__END__
name: hugepages
config:
raw.lxc: |
lxc.mount.entry = hugetlbfs dev/hugepages hugetlbfs rw,relatime,create=dir 0 0
security.privileged: "true"
__END__
After that we apply the hugepages profile and restart the vpp container:
ayourtch@xenial:~$ lxc profile apply vpp default,hugepages
Profile default,hugepages applied to vpp
ayourtch@xenial:~$
ayourtch@xenial:~$ lxc stop vpp
ayourtch@xenial:~$ lxc start vpp
ayourtch@xenial:~$ lxc exec vpp -- su -l ubuntu
ubuntu@vpp:~$
Now we can start the VPP:
ubuntu@vpp:~$ cd vpp
ubuntu@vpp:~/vpp$ make run
WARNING: STARTUP_CONF not defined or file doesn't exist.
Running with minimal startup config: unix { interactive } dpdk { no-pci socket-mem 1024 }
vlib_plugin_early_init:201: plugin path /usr/lib/vpp_plugins
ls: cannot access /sys/class/net/*/device: No such file or directory
Usage: egrep [OPTION]... PATTERN [FILE]...
Try 'egrep --help' for more information.
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 0 on socket 0
EAL: Detected lcore 5 as core 1 on socket 0
EAL: Detected lcore 6 as core 2 on socket 0
EAL: Detected lcore 7 as core 3 on socket 0
EAL: Support maximum 256 logical core(s) by configuration.
EAL: Detected 8 lcore(s)
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x2ae661400000 (size = 0x200000)
EAL: Ask a virtual area of 0x1c00000 bytes
EAL: Virtual area found at 0x2ae661600000 (size = 0x1c00000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x2ae663200000 (size = 0x200000)
EAL: Ask a virtual area of 0x1e800000 bytes
EAL: Virtual area found at 0x2ae663400000 (size = 0x1e800000)
EAL: Ask a virtual area of 0xa200000 bytes
EAL: Virtual area found at 0x2ae681c00000 (size = 0xa200000)
EAL: Ask a virtual area of 0x55400000 bytes
EAL: Virtual area found at 0x2ae68be00000 (size = 0x55400000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x2ae6e1200000 (size = 0x200000)
EAL: Requesting 512 pages of size 2MB from socket 0
EAL: TSC frequency is ~2294788 KHz
EAL: Master lcore 0 is ready (tid=6fa52c0;cpuset=[0])
PMD: rte_i40e_pmd_init(): >>
PMD: rte_i40evf_pmd_init(): >>
PMD: rte_igbvf_pmd_init(): >>
PMD: rte_ixgbe_pmd_init(): >>
PMD: rte_ixgbevf_pmd_init(): >>
PMD: rte_virtio_pmd_init(): IOPL call failed - cannot use virtio PMD
PMD: rte_vmxnet3_pmd_init(): >>
DPDK physical memory layout:
Segment 0: phys:0x20200000, len:2097152, virt:0x2ae661400000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
Segment 1: phys:0x33c00000, len:29360128, virt:0x2ae661600000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
Segment 2: phys:0x40200000, len:2097152, virt:0x2ae663200000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
Segment 3: phys:0x5cc00000, len:511705088, virt:0x2ae663400000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
Segment 4: phys:0x7fa00000, len:169869312, virt:0x2ae681c00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
Segment 5: phys:0x3fd000000, len:358612992, virt:0x2ae68be00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
dpdk_lib_init:272: DPDK drivers found no ports...
dpdk_lib_init:276: DPDK drivers found 0 ports...
DBGvpp# show interface
Name Idx State Counter Count
local0 0 down
pg/stream-0 1 down
pg/stream-1 2 down
pg/stream-2 3 down
pg/stream-3 4 down
DBGvpp#
Index of /blog/2016-04-01-First-shot-at-running-VPP-inside-LXD-container/