First shot at running VPP inside LXD container

<h1>First shot at running VPP inside LXD container </h1>
Today I've tried to build and run VPP entirely within the LXD container,
without vagrant or VMs.

My host is running the latest and greatest 16.04:
<pre>
ayourtch@xenial:~$ cat /etc/issue
Ubuntu Xenial Xerus (development branch) \n \l

ayourtch@xenial:~$ 
</pre>

After installing the LXD and configuring it to use the ZFS backend (not describing here),
I've created a container for ubuntu 14.04, and started it.

<pre>
ayourtch@xenial:~$ lxc launch ubuntu:14.04 vpp
</pre>

After that I got into the container:
<pre>
ayourtch@xenial:~$ lxc exec vpp -- su -l ubuntu
ubuntu@vpp:~$ 
</pre>

Next we'll need to install git, to be able to check out the sources of VPP and the
build-essential package, which has gcc, make, etc:

<pre>
ubuntu@vpp:~$ sudo apt-get update && sudo apt-get -y install git build-essential
</pre>

The next step is to get the VPP sources. https://wiki.fd.io/view/VPP has a lot of info,
but in the nutshell, what we need is this:

<pre>
ubuntu@vpp:~$ git clone https://gerrit.fd.io/r/vpp
</pre>

Now we can cd into the <b>vpp</b> subdirectory, and type "make":

<pre>
ubuntu@vpp:~/vpp$ ls
build-data  build-root  dpdk  g2  gmod  Makefile  perftool  sample-plugin  svm  test  vlib  vlib-api  vnet  vpp  vppapigen  vpp-api-test  vppinfra  vpp-japi
ubuntu@vpp:~/vpp$ make
Make Targets:
 bootstrap           - prepare tree for build
 install-dep         - install software dependencies
 wipe                - wipe all products of debug build 
 wipe-release        - wipe all products of release build 
 build               - build debug binaries
 build-release       - build release binaries
 rebuild             - wipe and build debug binares
 rebuild-release     - wipe and build release binares
 run                 - run debug binary
 run-release         - run release binary
 debug               - run debug binary with debugger
 debug-release       - run release binary with debugger
 build-vat           - build vpp-api-test tool
 run-vat             - run vpp-api-test tool
 pkg-deb             - build DEB packages
 pkg-rpm             - build RPM packages
 ctags               - (re)generate ctags database
 cscope              - (re)generate cscope database

Make Arguments:
 V=[0|1]             - set build verbosity level
 STARTUP_CONF=<path> - startup configuration file
                       (e.g. /etc/vpp/startup.conf)
 STARTUP_DIR=<path>  - startup drectory (e.g. /etc/vpp)
                       It also sets STARTUP_CONF if
                       startup.conf file is present
 GDB=<path>          - gdb binary to use for debugging
 PLATFORM=<name>     - target platform. default is vpp

Current Argumernt Values:
 V            = 0
 STARTUP_CONF = 
 STARTUP_DIR  = /home/ubuntu/vpp
 GDB          = gdb
 PLATFORM     = vpp
 DPDK_VERSION = 
ubuntu@vpp:~/vpp$ 
</pre>

The next step is to install the necessary dependencies. This is done like this:

<pre>
ubuntu@vpp:~/vpp$ make install-dep
</pre>

After the necessary packages are installed, we can prepare the tree for the build:
<pre>
ubuntu@vpp:~/vpp$ make bootstrap
</pre>

Finally, let's start the build:

<pre>
ubuntu@vpp:~/vpp$ make build
</pre>

The build process most probably will end with errors like this:

<pre>
ln -nsf `/home/ubuntu/vpp/build-root/build-vpp_debug-native/dpdk/dpdk-2.2.0/scripts/relpath.sh /home/ubuntu/vpp/build-root/build-vpp_debug-native/dpdk/dpdk-2.2.0/lib/librte_eal/linuxapp/eal/include/exec-env/rte_dom0_common.h /home/ubuntu/vpp/build-root/install-vpp_debug-native/dpdk/include/exec-env` /home/ubuntu/vpp/build-root/install-vpp_debug-native/dpdk/include/exec-env
ar crDs librte_eal.a eal.o eal_hugepage_info.o eal_memory.o eal_thread.o eal_log.o eal_pci.o eal_pci_uio.o eal_pci_vfio.o eal_pci_vfio_mp_sync.o eal_debug.o eal_lcore.o eal_timer.o eal_interrupts.o eal_alarm.o eal_common_lcore.o eal_common_timer.o eal_common_memzone.o eal_common_log.o eal_common_launch.o eal_common_pci.o eal_common_pci_uio.o eal_common_memory.o eal_common_tailqs.o eal_common_errno.o eal_common_cpuflags.o eal_common_string_fns.o eal_common_hexdump.o eal_common_devargs.o eal_common_dev.o eal_common_options.o eal_common_thread.o rte_malloc.o malloc_elem.o malloc_heap.o rte_keepalive.o 
  INSTALL-LIB librte_eal.a
cp -f librte_eal.a /home/ubuntu/vpp/build-root/install-vpp_debug-native/dpdk/lib
make[8]: *** [linuxapp] Error 2
make[7]: *** [librte_eal] Error 2
make[6]: *** [lib] Error 2
make[5]: *** [all] Error 2
make[4]: *** [pre_install] Error 2
make[3]: *** [install] Error 2
make[3]: Leaving directory `/home/ubuntu/vpp/build-root/build-vpp_debug-native/dpdk/dpdk-2.2.0'
make[2]: *** [/home/ubuntu/vpp/build-root/build-vpp_debug-native/dpdk/.build.ok] Error 2
make[2]: Leaving directory `/home/ubuntu/vpp/dpdk'
make[1]: *** [dpdk-install] Error 2
make[1]: Leaving directory `/home/ubuntu/vpp/build-root'
make: *** [build] Error 2
ubuntu@vpp:~/vpp$ 

</pre>

If you scroll a bit up, you will see what is going wrong:

<pre>
== Build lib/librte_eal/linuxapp
== Build lib/librte_eal/linuxapp/igb_uio
== Build lib/librte_eal/linuxapp/eal
make: *** /lib/modules/4.4.0-11-generic/build: No such file or directory.  Stop.
make[10]: *** [igb_uio.ko] Error 2
make[9]: *** [igb_uio] Error 2
make[9]: *** Waiting for unfinished jobs....
</pre>

Unfortunately, there is no exact headers:

<pre>
ubuntu@vpp:~/vpp$ sudo apt-cache search linux-headers-4.4.0
linux-headers-4.4.0-13 - Header files related to Linux kernel version 4.4.0
linux-headers-4.4.0-13-generic - Linux kernel headers for version 4.4.0 on 64 bit x86 SMP
linux-headers-4.4.0-13-lowlatency - Linux kernel headers for version 4.4.0 on 64 bit x86 SMP
linux-headers-4.4.0-14 - Header files related to Linux kernel version 4.4.0
linux-headers-4.4.0-14-generic - Linux kernel headers for version 4.4.0 on 64 bit x86 SMP
linux-headers-4.4.0-14-lowlatency - Linux kernel headers for version 4.4.0 on 64 bit x86 SMP
ubuntu@vpp:~/vpp$
</pre>

Therefore we will install the 4.4.0-14 version and symlink the 4.4.0-11 directory to it (it's a bit of a dirty trick but we do not intend to use the kernel modules anyway, at least for now!)

<pre>
ubuntu@vpp:~/vpp$ sudo apt-get install linux-headers-4.4.0-14-generic
</pre>

<pre>
ubuntu@vpp:~/vpp$ sudo ln -s /lib/modules/4.4.0-14-generic /lib/modules/4.4.0-11-generic
</pre>

Now we can retry "make build":
<pre>
ubuntu@vpp:~/vpp$ make build
....
 /usr/bin/install -c -m 644  /home/ubuntu/vpp/build-data/../vpp/api/vpe_all_api_h.h /home/ubuntu/vpp/build-data/../vpp/api/vpe_msg_enum.h api/vpe.api.h '/home/ubuntu/vpp/build-root/install-vpp_debug-native/vpp/include/api'
  CC       app/version.o
  CCLD     vpp
 /bin/mkdir -p '/home/ubuntu/vpp/build-root/install-vpp_debug-native/vpp/bin'
  /usr/bin/install -c vpp '/home/ubuntu/vpp/build-root/install-vpp_debug-native/vpp/bin'
make[4]: Leaving directory `/home/ubuntu/vpp/build-root/build-vpp_debug-native/vpp'
make[3]: Leaving directory `/home/ubuntu/vpp/build-root/build-vpp_debug-native/vpp'
make[2]: Leaving directory `/home/ubuntu/vpp/build-root/build-vpp_debug-native/vpp'
make[1]: Leaving directory `/home/ubuntu/vpp/build-root'
ubuntu@vpp:~/vpp$ 

</pre>

Now we can try to run it:

<pre>
ubuntu@vpp:~/vpp$ make run
WARNING: STARTUP_CONF not defined or file doesn't exist.
         Running with minimal startup config: unix { interactive } dpdk { no-pci socket-mem 1024 }

vlib_plugin_early_init:201: plugin path /usr/lib/vpp_plugins
dpdk_config: mount failed 1
ubuntu@vpp:~/vpp$ 
</pre>

The problem with the "mount failed" is that hugetlbfs can not be mounted... First thing we need
to do is to check that the hugepages are enabled:
<pre>
ubuntu@vpp:~/vpp$ sudo sysctl vm.nr_hugepages
vm.nr_hugepages = 0
ubuntu@vpp:~/vpp$ 
</pre>

Looks like they aren't so we need to go to the host and enable them:

<pre>
ayourtch@xenial:~$ echo "vm.nr_hugepages=1024" | sudo tee -a /etc/sysctl.d/20-hugepages.conf
vm.nr_hugepages=1024
ayourtch@xenial:~$
ayourtch@xenial:~$ sudo sysctl --system
</pre>

However, without the hugepage support exposed from host to guest, the VPP won't start.
Therefore we need to create this profile back on the host and apply to the container:

<pre>
lxc profile create hugepages
lxc profile edit hugepages <<__END__
name: hugepages
config:
  raw.lxc: |
    lxc.mount.entry = hugetlbfs dev/hugepages hugetlbfs rw,relatime,create=dir 0 0
  security.privileged: "true"
__END__
</pre>

After that we apply the hugepages profile and restart the vpp container:
<pre>
ayourtch@xenial:~$ lxc profile apply vpp default,hugepages
Profile default,hugepages applied to vpp
ayourtch@xenial:~$ 
ayourtch@xenial:~$ lxc stop vpp
ayourtch@xenial:~$ lxc start vpp
ayourtch@xenial:~$ lxc exec vpp -- su -l ubuntu
ubuntu@vpp:~$ 
</pre>

Now we can start the VPP:

<pre>
ubuntu@vpp:~$ cd vpp
ubuntu@vpp:~/vpp$ make run
WARNING: STARTUP_CONF not defined or file doesn't exist.
         Running with minimal startup config: unix { interactive } dpdk { no-pci socket-mem 1024 }

vlib_plugin_early_init:201: plugin path /usr/lib/vpp_plugins
ls: cannot access /sys/class/net/*/device: No such file or directory
Usage: egrep [OPTION]... PATTERN [FILE]...
Try 'egrep --help' for more information.
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 0 on socket 0
EAL: Detected lcore 5 as core 1 on socket 0
EAL: Detected lcore 6 as core 2 on socket 0
EAL: Detected lcore 7 as core 3 on socket 0
EAL: Support maximum 256 logical core(s) by configuration.
EAL: Detected 8 lcore(s)
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x2ae661400000 (size = 0x200000)
EAL: Ask a virtual area of 0x1c00000 bytes
EAL: Virtual area found at 0x2ae661600000 (size = 0x1c00000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x2ae663200000 (size = 0x200000)
EAL: Ask a virtual area of 0x1e800000 bytes
EAL: Virtual area found at 0x2ae663400000 (size = 0x1e800000)
EAL: Ask a virtual area of 0xa200000 bytes
EAL: Virtual area found at 0x2ae681c00000 (size = 0xa200000)
EAL: Ask a virtual area of 0x55400000 bytes
EAL: Virtual area found at 0x2ae68be00000 (size = 0x55400000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x2ae6e1200000 (size = 0x200000)
EAL: Requesting 512 pages of size 2MB from socket 0
EAL: TSC frequency is ~2294788 KHz
EAL: Master lcore 0 is ready (tid=6fa52c0;cpuset=[0])
PMD: rte_i40e_pmd_init():  >>
PMD: rte_i40evf_pmd_init():  >>
PMD: rte_igbvf_pmd_init():  >>
PMD: rte_ixgbe_pmd_init():  >>
PMD: rte_ixgbevf_pmd_init():  >>
PMD: rte_virtio_pmd_init(): IOPL call failed - cannot use virtio PMD
PMD: rte_vmxnet3_pmd_init():  >>
DPDK physical memory layout:
Segment 0: phys:0x20200000, len:2097152, virt:0x2ae661400000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
Segment 1: phys:0x33c00000, len:29360128, virt:0x2ae661600000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
Segment 2: phys:0x40200000, len:2097152, virt:0x2ae663200000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
Segment 3: phys:0x5cc00000, len:511705088, virt:0x2ae663400000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
Segment 4: phys:0x7fa00000, len:169869312, virt:0x2ae681c00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
Segment 5: phys:0x3fd000000, len:358612992, virt:0x2ae68be00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
dpdk_lib_init:272: DPDK drivers found no ports...
dpdk_lib_init:276: DPDK drivers found 0 ports...
DBGvpp# show interface
              Name               Idx       State          Counter          Count     
local0                            0        down      
pg/stream-0                       1        down      
pg/stream-1                       2        down      
pg/stream-2                       3        down      
pg/stream-3                       4        down      
DBGvpp# 

</pre>