Standalone virtio-fs usage

This document describes how to setup the virtio-fs components for standalone testing with QEMU and without Kata Containers. In general it's easier to debug basic issues in this environment than inside Kata Containers.


The following components need building:
  1. A guest kernel with virtio-fs support
  2. A QEMU with virtio-fs support
  3. The example virtio-fs daemon (virtiofsd)
  4. (optionally) The 'ireg' cache daemon
The instructions assume that you already have available a Linux guest image to run under QEMU and a Linux host on which you can build and run the components.

The guest kernel

Download the virtio-fs kernel tree by:
      git clone
Configure, build and install this kernel inside your guest VM, ensuring that the following config options are selected:
Build and install the kernel in the guest, on most distros this can be most easily achieved with the commandline:
      make -j 8 && make -j 8 modules && make -j 8 modules_install && make -j 8 install
Boot the guest and ensure it boots normally.

Note: An alternative is to build the kernel on the host and pass the kernel on the QEMU command line; although this can take some work to get initrd's to work right.

Building QEMU

On the host, download the virtio-fs QEMU tree by:
      git clone
Inside the checkout create a build directory, and from inside that build directory:
        ../configure --prefix=$PWD --target-list=x86_64-softmmu
        make -j 8
now also build the virtiofsd included in the qemu source:
        make -j 8 virtiofsd

Running with virtio-fs

A shared directory for testing is needed, this can initially be empty, but it's useful if it contains a file that you can check from inside the guest; we assume that $TESTDIR points to it.

First start virtiofsd daemon:

In the qemu build directory, run:
        ./virtiofsd -o vhost_user_socket=/tmp/vhostqemu -o source=$TESTDIR -o cache=always
The socket path will also be passed to the QEMU.
Now start QEMU, for virtio-fs we need to add parameters Add all these options to your standard QEMU command line; note the '-m' option and values are replacements for the existing option to set the memory size.
A typical QEMU command line (run from the qemu build directory) might be:
   ./x86_64-softmmu/qemu-system-x86_64 -M pc -cpu host --enable-kvm -smp 2 \
     -m 4G -object memory-backend-file,id=mem,size=4G,mem-path=/dev/shm,share=on -numa node,memdev=mem \
     -chardev socket,id=char0,path=/tmp/vhostqemu -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs \
     -chardev stdio,mux=on,id=mon -mon chardev=mon,mode=readline -device virtio-serial-pci -device virtconsole,chardev=mon -vga none -display none \
     -drive if=virtio,file=rootfsimage.qcow2
That assumes that 'rootfsimage.qcow2' is the VM built with the modified kernel. Log into the guest as root, and issue the mount command:
   mount -t virtiofs myfs /mnt

Note that Linux 4.19-based virtio-fs kernels required a different mount syntax mount -t virtio_fs none /mnt -o tag=myfs,rootmode=040000,user_id=0,group_id=0 instead.

The contents of the /mnt directory in the guest should now reflect the $TESTDIR on the host.

Enabling DAX

DAX mapping allows the guest to directly access the file contents from the hosts caches and thus avoids duplication between the guest and host.
A mapping area ('cache') is shared between virtiofsd and QEMU; this size must be specified on the command lines for QEMU, the command line for virtiofsd is unchanged.

The device section of the qemu command line changes to:

     -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs,cache-size=2G
Inside the guest the mount command becomes:
   mount -t virtiofs myfs /mnt

Note that Linux 4.19-based virtio-fs kernels required a different mount syntax mount -t virtio_fs none /mnt -o tag=myfs,rootmode=040000,user_id=0,group_id=0,dax instead.

Note that the size of the 'cache' used doesn't increase the host RAM used directly, since it's just a mapping area for files.

Building ireg

On the host, download the virtio-fs ireg tree by:
      git clone
and build with:

Enabling ireg [more experimental]

'ireg' is an external daemon that provides a shared cache of meta-data updates and allows guest kernels to check for changes to files quickly.
Start ireg as root:
      ./ireg &
Start virtiofsd, passing it the flag to set the caching mode to shared:
      ./virtiofsd -o virtio_socket=/tmp/vhostqemu / -o source=/home/dgilbert/virtio-fs/fs -o shared
Start qemu chanding the device option to point to the shared meta-data table:
      -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs,cache-size=2G,versiontable=/dev/shm/fuse_shared_versions
The mount options are unchanged.