Sunday, 15 November 2026

Linux Kernel Tuning for High-Concurrency NVMe Storage: I/O Schedulers, Queue Depth, and Sector Geometry

 

 


NVMe storage can handle thousands of I/O operations concurrently, but installing an NVMe SSD or selecting an NVMe-backed VPS doesn't automatically guarantee the best possible disk performance.

The Linux kernel plays an important role in how storage requests move from applications to physical or virtual disks. Its block layer manages I/O queues, request scheduling, completion handling, and communication with storage drivers.

For busy database servers, high-traffic websites, and virtualized environments, these settings can affect latency and throughput. A poorly chosen I/O scheduler or an overloaded request queue can introduce unnecessary waiting, even when the underlying SSD has spare capacity.

Linux NVMe performance tuning starts with understanding the multi-queue block layer (blk-mq), testing the available I/O schedulers, examining queue behavior, and checking filesystem alignment. The goal is to improve measurable workload performance without compromising stability or data integrity.

1. How the Linux Block Layer Handles NVMe Storage

The Linux block layer sits between applications and block-storage devices.

When MySQL writes transaction logs, PostgreSQL reads database pages, or a web server accesses files, the filesystem submits I/O requests through the kernel's block infrastructure.

Modern Linux kernels use the multi-queue block I/O mechanism, commonly called blk-mq.

Instead of relying on a single software request queue for all storage activity, blk-mq allows requests to be distributed across multiple software and hardware dispatch queues.

This architecture is particularly useful for NVMe devices, which support multiple command queues and can process concurrent requests efficiently.

A simplified request path looks like this:

Application → Filesystem → Linux blk-mq → Storage Driver → NVMe Device

On a VPS, the request may also pass through virtual block devices, a hypervisor, or networked storage infrastructure.

That distinction matters. Tuning a virtual disk inside a VPS does not necessarily change how the hosting provider's physical NVMe drives operate.

2. Understanding Linux I/O Schedulers

An I/O scheduler determines how the Linux block layer handles pending storage requests before dispatching them to the device.

Traditional schedulers were developed when mechanical hard drives were common. Rearranging requests could reduce physical seek movements and improve disk efficiency.

NVMe SSDs operate differently.

They have no mechanical seek heads and are designed to process many storage commands concurrently. As a result, complex request reordering may provide fewer benefits on some NVMe workloads.

Modern Linux systems commonly expose schedulers such as none, mq-deadline, bfq, and sometimes kyber, depending on the kernel and distribution.

The none Scheduler

The none scheduler avoids the additional request-reordering logic used by more elaborate schedulers.

With blk-mq, requests can move toward the device's dispatch queues without a separate elevator algorithm organizing them.

This can reduce scheduling overhead, especially for fast storage with substantial internal parallelism.

For some NVMe workloads, none is a sensible starting point because the SSD controller already performs its own command management.

But none is not automatically the fastest scheduler for every NVMe VPS.

If several applications compete for storage, or if the underlying virtual disk has unusual latency behavior, another scheduler may perform better.

The mq-deadline Scheduler

mq-deadline is designed for the multi-queue block layer.

It organizes requests with attention to read and write deadlines, helping prevent some operations from waiting excessively behind others.

This can be useful when a server handles mixed workloads.

For example, a database may issue latency-sensitive reads while backup processes generate sustained write activity.

Depending on the workload, mq-deadline may provide more predictable response times than none.

The bfq Scheduler

BFQ stands for Budget Fair Queueing.

It focuses on distributing I/O service among competing processes and can be useful in environments where fairness and responsiveness matter.

However, its scheduling overhead may make it less attractive for some high-throughput NVMe workloads.

BFQ should be tested rather than selected purely because it offers more scheduling features.

What About CFQ?

CFQ, or Completely Fair Queueing, was associated with older Linux block-layer implementations.

It is not a standard selectable scheduler on current mainstream blk-mq systems.

Modern administrators should work with the schedulers actually available on their running kernel instead of following older CFQ tuning guides.

3. Check the Active I/O Scheduler on Linux

Before changing any kernel setting, identify the storage device and inspect its current scheduler.

Run:

lsblk -o NAME,SIZE,TYPE,ROTA,FSTYPE,MOUNTPOINTS

This shows the block devices visible to the operating system.

A physical NVMe device may appear as nvme0n1, but a virtualized server could expose storage as vda, sda, or another device name.

Next, check the scheduler:

cat /sys/block/nvme0n1/queue/scheduler

Replace nvme0n1 with your actual device name.

An example result could look like:

[none] mq-deadline

The scheduler inside square brackets is currently active.

In this example, none is selected and mq-deadline is also available.

If the server already uses none, there is no reason to change it merely to follow a tuning checklist.

4. Change the NVMe I/O Scheduler Temporarily

Linux allows administrators to switch between supported I/O schedulers while the system is running.

To select none, run:

echo none | sudo tee /sys/block/nvme0n1/queue/scheduler

Verify the change:

cat /sys/block/nvme0n1/queue/scheduler

To test mq-deadline instead:

echo mq-deadline | sudo tee /sys/block/nvme0n1/queue/scheduler

These commands require sufficient permissions and the selected scheduler must be available for the device.

The change usually lasts until the device is reinitialized or the system restarts.

On a managed VPS, the provider may restrict changes to block-device settings.

Avoid forcing modifications when the kernel or virtual storage device does not expose the required control.

Which Scheduler Should You Choose?

For a high-concurrency NVMe workload, begin by recording the current scheduler and measuring its performance.

Then compare none and mq-deadline using the same workload.

Look at IOPS, CPU utilization, average latency, and p95/p99 response times.

If none reduces overhead without increasing tail latency, it may be a suitable choice.

If mq-deadline produces more consistent response times under mixed reads and writes, retaining it may be better.

The appropriate scheduler depends on the application rather than the NVMe label alone.

5. Make NVMe Scheduler Changes Persistent with Udev

Temporary scheduler changes may disappear after a restart.

On Linux systems using systemd-udevd, administrators can define udev rules to apply settings when matching devices are detected.

This approach can be used on Ubuntu, Debian, AlmaLinux, and other distributions with compatible udev support.

Create a rule file:

sudo nano /etc/udev/rules.d/60-nvme-scheduler.rules

Add:

ACTION=="add", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", KERNEL=="nvme*n*", ATTR{queue/scheduler}="none"

This rule targets whole NVMe namespace block devices and attempts to select the none scheduler.

It does not target partition devices because of the DEVTYPE condition.

Save the file and reload the rules:

sudo udevadm control --reload-rules

Reloading the rules does not automatically guarantee that existing devices have been reconfigured.

For an existing device, apply the scheduler temporarily using the command in the previous section. Then verify persistence during a planned restart or device reinitialization.

After restarting, check:

cat /sys/block/nvme0n1/queue/scheduler

If the selected scheduler remains none, the rule has taken effect.

Some systems use distribution-specific udev rules or other storage management services that can override these settings. Inspect existing configuration before adding competing rules.

Reverting the Change

To remove the persistent rule:

sudo rm /etc/udev/rules.d/60-nvme-scheduler.rules

sudo udevadm control --reload-rules

Restore the previous scheduler manually if required, and verify the configuration after the next restart.

6. Understanding NVMe Queue Depth and Request Limits

Queue depth represents the number of I/O operations that can be outstanding while waiting for completion.

NVMe devices are designed to handle many commands concurrently, but the queue capacity available to an application depends on the hardware, driver, filesystem, and virtualization environment.

Higher queue depth can improve throughput when a workload has enough independent operations to keep the storage device busy.

It can also increase latency when requests accumulate faster than the device can complete them.

Inspect Block-Layer Request Settings

Check:

cat /sys/block/nvme0n1/queue/nr_requests

The nr_requests parameter controls request allocation limits in the Linux block layer.

It is not the same as the NVMe hardware queue depth.

Increasing this value does not automatically increase the number of commands supported by the physical SSD.

If testing identifies an actual request-allocation bottleneck, an administrator might experiment with a different value:

echo 256 | sudo tee /sys/block/nvme0n1/queue/nr_requests

This is an example test setting, not a universally recommended configuration.

Some devices and schedulers expose different constraints, and the kernel may reject unsupported values.

For most servers, leave the default unchanged until monitoring or benchmarks demonstrate a reason to modify it.

Why Increasing Queue Depth Can Hurt Performance

Suppose a VPS runs a transactional database.

Increasing the number of concurrent I/O requests may initially improve throughput.

Once the storage system reaches its practical capacity, additional requests begin waiting longer.

The reported IOPS may stop increasing while p99 latency becomes worse.

For applications that require consistent transaction response times, this can be more damaging than a slightly lower maximum throughput.

The objective is to identify a useful balance between concurrency and latency.

7. Linux Sysctl Settings and File Descriptor Limits

Linux provides many kernel parameters through the sysctl interface.

One frequently discussed setting is fs.file-max, which controls the system-wide maximum number of allocated file handles.

Check its current value:

sysctl fs.file-max

You can also inspect file-handle usage:

cat /proc/sys/fs/file-nr

Busy web servers and database systems may maintain many open files, sockets, and database-related resources.

If monitoring reveals system-wide file-handle exhaustion, raising the limit may be appropriate.

For example, create a configuration file:

sudo nano /etc/sysctl.d/90-file-limits.conf

Add:

fs.file-max = 2097152

Apply the setting:

sudo sysctl --system

This is not an NVMe performance optimization. Increasing fs.file-max will not directly improve random IOPS, reduce disk latency, or accelerate fsync() operations.

It addresses a separate resource limit, and the example value should not be applied without evidence of file-handle pressure.

Applications may also be restricted by process-specific limits, including the systemd LimitNOFILE setting.

Always check the relevant application limits before changing kernel-wide parameters.

8. Sector Geometry and 4K Alignment

Storage performance depends partly on how data is organized and aligned.

Many modern storage devices use 4 KiB physical sectors, although the logical sector size presented to the operating system may differ.

A drive might expose 512-byte logical sectors while using 4 KiB physical sectors internally.

This arrangement is often called 512e, meaning 512-byte emulation.

When writes are poorly aligned with the underlying storage geometry, extra read-modify-write activity may occur.

That can increase storage overhead.

Inspect Logical and Physical Sector Sizes

Run:

lsblk -o NAME,PHY-SEC,LOG-SEC,MIN-IO,OPT-IO

This reports the physical and logical sector sizes and certain I/O alignment characteristics exposed by the device.

You can also inspect the values through sysfs:

cat /sys/block/nvme0n1/queue/logical_block_size

cat /sys/block/nvme0n1/queue/physical_block_size

These values describe the block device visible to the operating system.

On a VPS, they may reflect virtual device characteristics rather than the physical geometry of the provider's SSD.

For most modern Linux installations, default partitioning tools already handle common alignment requirements correctly.

Avoid repartitioning a production server simply because you discover different logical and physical sector sizes.

9. Understanding ashift=12 in ZFS

The ashift setting is relevant to systems using OpenZFS.

It determines the minimum sector alignment and allocation unit for a ZFS virtual device.

The relationship is:

Sector size = 2^ashift

For example:

  • ashift=9 represents 512 bytes.
  • ashift=12 represents 4096 bytes.
  • ashift=13 represents 8192 bytes.

When the underlying storage has 4 KiB physical sectors, ashift=12 can help avoid misaligned writes.

OpenZFS normally attempts to detect an appropriate value automatically, but explicit configuration can be useful when storage devices report misleading sector sizes.

For a new ZFS pool, an example configuration is:

sudo zpool create -o ashift=12 tank mirror /dev/disk/by-id/DEVICE_A /dev/disk/by-id/DEVICE_B

Do not execute this command on existing production disks. Creating a pool is a destructive storage operation for the selected devices.

Replace the placeholder devices only when creating a new pool using verified, dedicated storage disks.

The ashift value is effectively fixed when a ZFS vdev is created. It is not an ordinary setting that can be safely adjusted later to optimize an existing pool.

It also does not apply to ext4 or XFS filesystems.

For Linux VPS users without direct control over the provider's ZFS storage pools, this tuning may be unavailable or unnecessary.

10. Configure TRIM and Discard for NVMe SSDs

Solid-state storage can benefit from discard operations that inform the underlying device when blocks are no longer needed.

This mechanism is commonly associated with TRIM.

Discard support allows compatible storage systems to identify blocks that the filesystem no longer uses.

On Linux, administrators can use periodic fstrim operations instead of issuing continuous discard requests for every file deletion.

Check Whether Discard Is Supported

Run:

lsblk -D

Review the discard-related columns for the relevant device.

You can also inspect:

cat /sys/block/nvme0n1/queue/discard_max_bytes

A zero value indicates that discard requests are not supported through that particular queue interface.

On virtual servers, discard support depends on the hypervisor and storage backend.

Enable Periodic TRIM

Ubuntu, Debian, and many other systemd-based distributions provide an fstrim.timer unit.

Check its status:

systemctl status fstrim.timer

If your filesystem and storage infrastructure support discard, enable the timer:

sudo systemctl enable --now fstrim.timer

Check the schedule:

systemctl list-timers fstrim.timer

Periodic TRIM can be preferable to continuous discard for some workloads, but administrators should verify the storage provider's recommendations.

Do not assume that discard requests reach a physical NVMe SSD simply because the guest filesystem accepts them.

11. Monitor NVMe Storage Performance

Kernel tuning should begin with measurements.

Changing schedulers, request limits, or filesystem settings without a performance baseline makes it difficult to determine whether the change actually helped.

Install monitoring tools on Ubuntu or Debian:

sudo apt update

sudo apt install -y sysstat fio

On AlmaLinux or Rocky Linux:

sudo dnf install -y sysstat fio

Check Disk Latency with iostat

Run:

iostat -xz 1

Important measurements include:

r_await: Average time required for read requests to complete, including queue time.

w_await: Average time required for write requests to complete, including queue time.

aqu-sz: Average number of I/O requests outstanding during the measurement period.

%util: Percentage of elapsed time during which the device had I/O activity.

Do not interpret 100% utilization as definitive evidence of saturation on a modern parallel NVMe device.

The relationship between utilization and capacity is more complicated for hardware that can execute many operations concurrently.

Compare throughput, request latency, queue occupancy, and application responsiveness together.

12. Benchmark none vs mq-deadline with fio

Fio can generate controlled storage workloads to compare scheduler behavior.

For example, a random-read test could use:

fio --name=nvme-read-test \

  --filename=./nvme-testfile \

  --size=2G \

  --rw=randread \

  --bs=4k \

  --direct=1 \

  --ioengine=libaio \

  --iodepth=32 \

  --runtime=60 \

  --time_based \

  --group_reporting

Use a dedicated test directory on an authorized, non-production storage volume.

For meaningful random-read measurements, ensure the test file has first been fully populated with data. Otherwise, sparse-file behavior may distort the results.

The working set, test duration, and queue depth should be adjusted to suit the available storage capacity and application requirements.

A useful comparison procedure is:

  1. Record the current scheduler.
  2. Prepare the benchmark file.
  3. Run the workload several times and record results.
  4. Switch to the alternative scheduler.
  5. Repeat the same test using identical settings.
  6. Compare IOPS, average latency, p95/p99 latency, and CPU utilization.
  7. Restore the original scheduler if the new configuration provides no meaningful benefit.

For a detailed explanation of SSD architecture, IOPS measurements, and storage performance comparisons, see the NVMe vs SSD VPS benchmarks and storage architecture guide.

How to Interpret the Results

Suppose the none scheduler produces slightly higher throughput but noticeably worse p99 latency during mixed I/O.

A database handling interactive transactions may perform better with the more consistent configuration.

For a batch-processing system, maximum throughput might be more important.

There is no universal scheduler choice that guarantees the best outcome for every workload.

Use production-like measurements rather than selecting settings solely from online optimization guides.

13. Common NVMe Kernel Tuning Mistakes

Assuming none Is Always the Fastest Scheduler

The none scheduler reduces request scheduling complexity, but it does not guarantee better application latency under every workload.

Increasing nr_requests Without Measurements

Request allocation limits are not the same as the NVMe controller's physical queue capacity.

A larger value may increase outstanding work without improving the device's ability to complete it.

Applying Legacy CFQ Tuning Instructions

Older Linux guides sometimes recommend CFQ options that no longer apply to modern blk-mq installations.

Use the scheduler interfaces supported by the running kernel.

Changing ZFS ashift on an Existing Pool

ZFS sector geometry is established when vdevs are created. Treat ashift as a storage design decision, not a routine runtime optimization.

Ignoring VPS Virtualization

A virtual disk may not expose the physical NVMe controller or its native queue structure.

The hosting platform can introduce limits or scheduling behavior outside the guest operating system.

Making Multiple Changes at Once

Changing the scheduler, queue limits, readahead, and other parameters simultaneously makes the results difficult to interpret.

Change one relevant setting at a time and preserve a clear rollback procedure.

 

14. Recommended NVMe Tuning Checklist

 

Before changing Linux storage settings, confirm that:

  • The block device and active I/O scheduler have been identified.
  • The VPS permits changes to the relevant device settings.
  • Baseline throughput and latency measurements have been recorded.
  • Both none and mq-deadline have been considered where available.
  • Request allocation limits are only modified when necessary.
  • Physical and logical sector geometry have been inspected.
  • ZFS-specific settings are not applied to unrelated filesystems.
  • TRIM support has been checked before enabling periodic discard.
  • Persistent udev rules have been reviewed and verified.
  • A rollback procedure exists for each configuration change.

 

Conclusion

 

Modern Linux kernels already include a multi-queue block architecture designed to work efficiently with fast storage.

For high-concurrency NVMe workloads, the none scheduler may reduce unnecessary scheduling overhead, while mq-deadline can be useful when mixed workloads require more consistent latency.

Queue depth, sector alignment, TRIM configuration, and filesystem behavior also affect the storage path, but none of these settings should be changed without understanding the underlying system.

For cloud VPS administrators, the most effective approach is to measure the current configuration, change one setting at a time, and evaluate how the real application responds.

 

No comments:

Post a Comment

Your Post is Publishing

Featured Post

  The demand to have strong, stable and high-performance server solutions has never been higher in the ever-changing digital space. Enterpri...