1

We would like to run ZFS with QAT offloading for compressions and checksums.

The distribution is Centos 8.2 with the stock kernel:

[root@dellqat ~]# uname -a
Linux dellqat 4.18.0-193.19.1.el8_2.x86_64 #1 SMP Mon Sep 14 14:37:00 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

for the QAT we have:

qat1.7.l.4.11.0-00001 (latest from intel)

./configure --enable-icp-trace --enable-icp-debug --enable-icp-log-syslog --enable-kapi
make
make install

for ZFS we have:

[root@dellqat zfs_latest]# git status
On branch zfs-2.0-release
Your branch is up to date with 'origin/zfs-2.0-release'.

nothing to commit, working tree clean

export ICP_ROOT=/opt/A3C/qat1.7.l.4.11.0-00001
./configure --with-qat=/opt/A3C/qat1.7.l.4.11.0-00001
make
make install
ldconfig

The problem is the module loading order: ZFS does not use correctly the QAT because the zfs module is loaded before the QAT. Not only that, but the QAT modules are loaded in an init script qat_service where use of a program adf_ctl is done to init the QAT services. This qat_service is scheduled as the last service to run!

The qat_service basically launch a program adf_ctl to start the qat engines, hence it is not only a problem of module loading order: before modprobing for zfs adf_ctl must be run.

As I see zfs comes with a dracut module zfs-dracut for loading the modules in initramfs for root partitions in zfs.

Has anyone experience to share on the use of Intel QAT with ZFS ?

Ideally we would like to write a dracut module that goes before the zfs dracut module for performing init of QAT in initramfs before loading ZFS.

fstrati70
  • 11
  • 1

1 Answers1

0

I sort of got it working with a custom initramfs with dracut...

You must exclude the dracut modules for zfs: 02zfsexpandknowledge and 90zfs.

You then add a custom module for qat:

[root@dellqat ~]# cd /usr/lib/dracut/modules.d/
[root@dellqat modules.d]# mkdir 89qat

inside 89qat you put two files:

[root@dellqat modules.d]# cd 89qat/
[root@dellqat 89qat]# cat module-setup.sh 
#!/usr/bin/env bash

check() {
    return 0
}

depends() {
    return 0
}

installkernel() {
    instmods qat_dh895xcc
    instmods qat_api
    instmods usdm_drv
    instmods intel_qat
    instmods uio
}

install() {
    inst_hook pre-mount 10 "${moddir}/qat_start.sh"
    inst /etc/dh895xcc_dev0.conf /etc/dh895xcc_dev0.conf
    inst /usr/local/bin/adf_ctl /sbin/adf_ctl
    inst /usr/bin/sleep /sbin/sleep
}

and

[root@dellqat 89qat]# cat qat_start.sh
#!/bin/sh

modprobe -r qat_api;
/sbin/sleep 1;
modprobe -r qat_dh895xcc;
/sbin/sleep 1;
modprobe qat_dh895xcc;
/sbin/sleep 1;
modprobe qat_api;
/sbin/sleep 1;
/sbin/adf_ctl restart 1>/dev/null 2>&1;
/sbin/sleep 1;

both files shall be chmod 755.

you then rebuild the image with dracut: dracut foobar.img $(uname -r) --force -v

which create the "foobar.img" initramfs image for the current kernel.

I don't know why but the modules don't load in the correct order if you keep also the zfs-dracut modules: 02zfsexpandknowledge and 90zfs.

Maybe someone with more experience with dracut may help?

fstrati70
  • 11
  • 1