Standalone virtiofs usage

This document describes how to setup the virtiofs components for standalone testing with QEMU and without Kata Containers. In general it's easier to debug basic issues in this environment than inside Kata Containers.

Components

The following components need building:

A guest kernel with virtiofs support
A QEMU with virtiofs support
The example virtiofs daemon (virtiofsd)
(optionally) The 'ireg' cache daemon

The instructions assume that you already have available a Linux guest image to run under QEMU and a Linux host on which you can build and run the components.

The guest kernel

An appropriately configured Linux 5.4 or later can be used for virtiofs, however if you want access to development features, download the virtiofs kernel tree by:

      git clone https://gitlab.com/virtio-fs/linux.git
      git checkout virtio-fs-dev

Configure, build and install this kernel inside your guest VM, ensuring that the following config options are selected:

      CONFIG_VIRTIO
      CONFIG_VIRTIO_FS
      CONFIG_DAX
      CONFIG_FS_DAX
      CONFIG_DAX_DRIVER
      CONFIG_ZONE_DEVICE

Build and install the kernel in the guest, on most distros this can be most easily achieved with the commandline:

      make -j 8 && make -j 8 modules && make -j 8 modules_install && make -j 8 install

Boot the guest and ensure it boots normally.

Note: An alternative is to build the kernel on the host and pass the kernel on the QEMU command line; although this can take some work to get initrd's to work right.

Building QEMU

On the host, download the virtiofs QEMU tree by:

      git clone https://gitlab.com/virtio-fs/qemu.git

Inside the checkout create a build directory, and from inside that build directory:

        ../configure --prefix=$PWD --target-list=x86_64-softmmu
        make -j 8

now also build the virtiofsd included in the qemu source:

        make -j 8 virtiofsd

Running with virtiofs

A shared directory for testing is needed, this can initially be empty, but it's useful if it contains a file that you can check from inside the guest; we assume that $TESTDIR points to it.

First start virtiofsd daemon:

In the qemu build directory, run:

        ./virtiofsd --socket-path=/tmp/vhostqemu -o source=$TESTDIR -o cache=always

The socket path will also be passed to the QEMU.
Now start QEMU, for virtiofs we need to add parameters

to create the communications socket

      -chardev socket,id=char0,path=/tmp/vhostqemu

instantiate the device
```
      -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs
```
The tag name is arbitrary and must match the tag given in the guests mount command.

force use of memory sharable with virtiofsd.

      -m 4G -object memory-backend-file,id=mem,size=4G,mem-path=/dev/shm,share=on -numa node,memdev=mem

Add all these options to your standard QEMU command line; note the '-m' option and values are replacements for the existing option to set the memory size.
A typical QEMU command line (run from the qemu build directory) might be:

   ./x86_64-softmmu/qemu-system-x86_64 -M pc -cpu host --enable-kvm -smp 2 \
     -m 4G -object memory-backend-file,id=mem,size=4G,mem-path=/dev/shm,share=on -numa node,memdev=mem \
     -chardev socket,id=char0,path=/tmp/vhostqemu -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs \
     -chardev stdio,mux=on,id=mon -mon chardev=mon,mode=readline -device virtio-serial-pci -device virtconsole,chardev=mon -vga none -display none \
     -drive if=virtio,file=rootfsimage.qcow2

That assumes that 'rootfsimage.qcow2' is the VM built with the modified kernel. Log into the guest as root, and issue the mount command:

   mount -t virtiofs myfs /mnt

Note that Linux 4.19-based virtiofs kernels required a different mount syntax mount -t virtio_fs none /mnt -o tag=myfs,rootmode=040000,user_id=0,group_id=0 instead.

The contents of the /mnt directory in the guest should now reflect the $TESTDIR on the host.

Enabling DAX

DAX mapping allows the guest to directly access the file contents from the hosts caches and thus avoids duplication between the guest and host.
A mapping area ('cache') is shared between virtiofsd and QEMU; this size must be specified on the command lines for QEMU, the command line for virtiofsd is unchanged.

The device section of the qemu command line changes to:

     -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs,cache-size=2G

Inside the guest the mount command becomes:

   mount -t virtiofs myfs /mnt -o dax

Note that Linux 4.19-based virtiofs kernels required a different mount syntax mount -t virtio_fs none /mnt -o tag=myfs,rootmode=040000,user_id=0,group_id=0,dax instead.

Note that the size of the 'cache' used doesn't increase the host RAM used directly, since it's just a mapping area for files.

Building ireg

On the host, download the virtiofs ireg tree by:

      git clone https://gitlab.com/virtio-fs/ireg.git

and build with:

      make

Enabling ireg [more experimental]

'ireg' is an external daemon that provides a shared cache of meta-data updates and allows guest kernels to check for changes to files quickly.
Start ireg as root:

      ./ireg &

Start virtiofsd, passing it the flag to set the caching mode to shared:

      ./virtiofsd -o virtio_socket=/tmp/vhostqemu / -o source=/home/dgilbert/virtio-fs/fs -o shared

Start qemu chanding the device option to point to the shared meta-data table:

      -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs,cache-size=2G,versiontable=/dev/shm/fuse_shared_versions

The mount options are unchanged.