Standalone virtiofs usage

This document describes how to set up virtiofs for standalone QEMU testing without Kata Containers or libvirt. In general it's easier to debug basic issues in this environment than inside Kata Containers.

Components

The following components are required:

A Linux 5.4 or later guest kernel with virtiofs support
QEMU 4.2.0 or later for built-in virtiofs support
virtiofsd

The instructions assume that you already have available a Linux guest image to run under QEMU and a Linux host on which you can build and run the components.

The guest kernel

An appropriately configured Linux 5.4 or later kernel can be used for virtiofs. Many distributions enable virtiofs so it is not necessary to build from source. However, if you want to compile a custom kernel or modify the virtiofs.ko guest driver you need to build from source:

      git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

Configure, build and install this kernel inside your guest VM, ensuring that the following config options are selected:

      CONFIG_VIRTIO_FS
      CONFIG_FUSE_DAX   # optional

Build and install the kernel in the guest, on most distros this can be most easily achieved with the commandline:

      make -j 8 && make -j 8 modules && make -j 8 modules_install && make -j 8 install

Boot the guest and ensure it boots normally.

Note: An alternative is to build the kernel on the host and pass the kernel on the QEMU command line; although this can take some work to get initrd's to work right.

Building QEMU

QEMU 4.2.0 or later has built-in virtiofs support. Normally it is not necessary to build QEMU from source in order to use virtiofs. If you wish to develop virtiofs then you may need to build QEMU from source in order to modify the vhost-user-fs device setup code although most of the host-side virtiofs code lives in virtiofsd instead of QEMU.

On the host, download the QEMU tree by:

      git clone https://gitlab.com/qemu-project/qemu.git

Inside the checkout create a build directory, and from inside that build directory:

        ../configure --prefix=$PWD --target-list=x86_64-softmmu
        make -j 8

Building virtiofsd

Usually your distribution's virtiofsd package is sufficient and you do not need to compile virtiofsd from source. If you wish to develop virtiofsd or test the latest code then you can build it as follows:

        git clone https://gitlab.com/virtio-fs/virtiofsd.git
        cd virtiofsd
        cargo build --release

See virtiofsd's readme file for information on build dependencies and build options.

Running with virtiofs

A shared directory for testing is needed, this can initially be empty, but it's useful if it contains a file that you can check from inside the guest; we assume that $TESTDIR points to it.

First start virtiofsd daemon:

Launch virtiofsd (its location may vary depending on whether you built it from source or which distribution you are using):

        /usr/libexec/virtiofsd --socket-path=/tmp/vhostqemu -o source=$TESTDIR -o cache=always

The socket path will also need to be passed to QEMU below.
Now start QEMU, for virtiofs we need to add parameters

to create the communications socket

      -chardev socket,id=char0,path=/tmp/vhostqemu

instantiate the device
```
      -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs
```
The tag name is arbitrary and must match the tag given in the guests mount command.

force use of memory sharable with virtiofsd.

      -m 4G -object memory-backend-file,id=mem,size=4G,mem-path=/dev/shm,share=on -numa node,memdev=mem

Add all these options to your standard QEMU command line; note the '-m' option and values are replacements for the existing option to set the memory size.
A typical QEMU command line might be:

   qemu-system-x86_64 -M pc -cpu host --enable-kvm -smp 2 \
     -m 4G -object memory-backend-file,id=mem,size=4G,mem-path=/dev/shm,share=on -numa node,memdev=mem \
     -chardev socket,id=char0,path=/tmp/vhostqemu -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs \
     -chardev stdio,mux=on,id=mon -mon chardev=mon,mode=readline -device virtio-serial-pci -device virtconsole,chardev=mon -vga none -display none \
     -drive if=virtio,file=rootfsimage.qcow2

That assumes that 'rootfsimage.qcow2' is the VM built with the modified kernel. Log into the guest as root, and issue the mount command:

   mount -t virtiofs myfs /mnt

Note that Linux 4.19-based virtiofs kernels required a different mount syntax mount -t virtio_fs none /mnt -o tag=myfs,rootmode=040000,user_id=0,group_id=0 instead.

The contents of the /mnt directory in the guest should now reflect the $TESTDIR on the host.

Enabling DAX

DAX mapping allows the guest to directly access the file contents from the hosts caches and thus avoids duplication between the guest and host. This feature is not complete and only available when building from source.
A mapping area ('cache') is shared between virtiofsd and QEMU; this size must be specified on the command lines for QEMU, the command line for virtiofsd is unchanged.

The device section of the qemu command line changes to:

     -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs,cache-size=2G

Inside the guest the mount command becomes:

   mount -t virtiofs myfs /mnt -o dax

Note that Linux 4.19-based virtiofs kernels required a different mount syntax mount -t virtio_fs none /mnt -o tag=myfs,rootmode=040000,user_id=0,group_id=0,dax instead.

Note that the size of the 'cache' used doesn't increase the host RAM used directly, since it's just a mapping area for files.