Timeouts

As storage for the applications I had chosen the same pool as in which the TrueNAS Jails were running. Just a simple set of HDD’s, running in a RAIDZ2. For most applications not a problem, it just works. However almost every time when I update an application the system complains of timeouts, and the application stays in “deploying” for ever. So the server or my Home Lab is running not as I like. Annoying to say the least.

Moving Pools

For transcoding purposes I have setup a couple of 250GB SATA disk Crucial BX500 240GB 3D NAND SATA 2.5-inch SSD in a stripe configuration.

Moving to another pool is really easy, just a matter of selecting a different pool and TrueNAS handles it all. Smooth sailing.

Choosing the faster pool patly solved the problem, some applications do update now, others not. An even faster storage medium is needed.

Adding M.2 NVMe SSD’s to the Server

At the moment M.2 NVMe SSD are the fasted SSD one can get. Albeit more costly. And there is the problem to install them in the Server. I read somewhere online of the possibility of adding a PCIe card that would make it possible to add storage directly to the server.

  • Would this be possible?
  • What would it take?
  • How much is such a adapter card?
  • How much are the SSD’s going to be?

Some questions that need to be answered.

Finding the right adapter card

After some searching online, the right card is found: ASUS Hyper M.2 x16 Gen 4 Card

However the motherboard of the server needs to support bifurcation. This feature enables you to divide the PCIe x16 slot into a slot that separates the 16 lanes into 4 x x4 lanes. A nice simple feature, does the SuperMicro H11SSL-i motherboard support this? Yes, it does. Pretty simple, boot into Bios, make the change on the slot and ready to add the card.

And the card is not that expensive either. Less then € 60.

There are enough PCIe lanes available on the EPYC CPU, and having the x16 slot available also helps.

NVMe SSD

Thanks to the Amazon Spring sale, the opportunity was there. The Crucial P3 500GB PCIe M.2 2280 SSD seems like a nice fit, and for a nice price. The 500GB storage is enough for this purpose.

Installing the card

The card is huge, it almost did not fit. TrueNAS recognized the SSDs directly and setting up another storage pool is quickly done. I already had another SSD already installed in the server using the one NVMe M.2 slot. Another pool is created, with using the 5 SSDs running in a RAIDZ2. This enables me with roughly 1150 GB storage. Plenty for storing and running the Scale Applications and then some.

Additional purpose

Next to storing the applications on a medium that has enough speed performance, I also could run the VMs on this volume. Moving these is slightly more complicated.

Creating Snapshot and send snapshot

Thanks to this post it is doable.

  • Shutdown the VM
  • Create dataset on destination pool. (In my case on the just created pool using the NVMe SSDs.)
  • Create the snapshot using the web interface
  • and using the terminal send the snapshot to the other pool

All in all not to bad.

Performance

How fast is the new setup? That is simple using the command fio, this tool helps in measuring the I/O and simulating reading and writing files to a storage.

For me the command turned out to be:

fio --bs=128k --direct=1 --directory=<MountPoint/TargetDirectory> --gtod_reduce=1 --ioengine=posixaio --iodepth=32 --group_reporting --name=randrw --numjobs=12 --ramp_time=10 --runtime=60 --rw=randrw --size=256M --time_based

Test Results

Running the test on the NVMe SSD pool:

Run status group 0 (all jobs):
   READ: bw=1437MiB/s (1507MB/s), 1437MiB/s-1437MiB/s (1507MB/s-1507MB/s), io=84.3GiB (90.5GB), run=60060-60060msec
  WRITE: bw=1436MiB/s (1506MB/s), 1436MiB/s-1436MiB/s (1506MB/s-1506MB/s), io=84.2GiB (90.5GB), run=60060-60060msec

And when running the test on the HDD pool:

Run status group 0 (all jobs):
   READ: bw=426MiB/s (446MB/s), 426MiB/s-426MiB/s (446MB/s-446MB/s), io=25.0GiB (26.9GB), run=60201-60201msec
  WRITE: bw=425MiB/s (446MB/s), 425MiB/s-425MiB/s (446MB/s-446MB/s), io=25.0GiB (26.8GB), run=60201-60201msec

That is a pretty clear result, a lot faster.

Conclusion

Updating the applications has improved, no more timeouts when I update any application and the performance of VM’s has also improved.

All in all I am very content with the end result, a nice side project with a nice performance improvement.