Let's face it, using synchronization primitives such as RCU can be frustrating. And it is only natural to wish to get back, somehow, at the source of such frustration. In short, it is quite understandable to want to torture RCU. (And other synchronization primitives as well, but you have to start somewhere!) Another benefit of torturing RCU is that doing so sometimes uncovers bugs in other parts of the kernel. You see, RCU is not always willing to suffer alone.
One long-standing RCU-torture approach is to use modprobe
to install and remove the rcutorture
module, as described in the torture-test documentation
. However, this approach requires considerable manual work to check for errors.
On the other hand, this approach avoids any concern about the underlying architecture or virtualization technology. This means that use of modprobe
is the method of choice if you wish to torture RCU on (say) SPARC or when running on Hyper-V (this last according to people actually doing this). This method is also necessary when you want to torture RCU on a very specific kernel configuration or when you need to torture RCU on bare metal.
But for those of us running mainline kernels on x86 systems supporting virtualization, the approach described in the remainder of this document will usually be more convenient.
Running rcutorture in a Guest OS
If you have an x86 system (or, with luck, an ARMv8 or PowerPC system) set up to run qemu
and KVM, you can instead use the rcutorture scripting
, which automates running rcutorture
over a full set of configurations, as well as automating analysis of the build products and console output. Running this can be as simple as:
As of v5.8-rc1, this will build and run each of nineteen combinations of Kconfig options, with each run taking 30 minutes for a total of 8.5 hours, not including the time required to build the kernel, boot the guest OS, and analyze the test results. Given that a number of the scenarios use only a single CPU, this approach can be quite wasteful, especially on the well-endowed systems of the year 2020.
This waste can be avoided by using the --cpus
argument, for example, for the 12-hardware-thread laptop on which I am typing this, you could do the following:
tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 12
This command would run up to 12 CPUs worth of rcutorture
scenarios concurrently, so that the nineteen combinations would be run in eight batches. Because TREE03
each want 16 CPUs, rcutorture
will complain in its run summary as follows:
--- Mon Jun 15 10:23:02 PDT 2020 Test summary:
Results directory: /home/git/linux/tools/testing/selftests/rcutorture/res/2020.06.15-10.23.02
tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 12 --duration 5 --trust-make
RUDE01 ------- 2102 GPs (7.00667/s) [tasks-rude: g0 f0x0 ]
SRCU-N ------- 42229 GPs (140.763/s) [srcu: g549860 f0x0 ]
SRCU-P ------- 11887 GPs (39.6233/s) [srcud: g110444 f0x0 ]
SRCU-t ------- 59641 GPs (198.803/s) [srcu: g1 f0x0 ]
SRCU-u ------- 59209 GPs (197.363/s) [srcud: g1 f0x0 ]
TASKS01 ------- 1029 GPs (3.43/s) [tasks: g0 f0x0 ]
TASKS02 ------- 1043 GPs (3.47667/s) [tasks: g0 f0x0 ]
TASKS03 ------- 1019 GPs (3.39667/s) [tasks: g0 f0x0 ]
TINY01 ------- 43373 GPs (144.577/s) [rcu: g0 f0x0 ] n_max_cbs: 34463
TINY02 ------- 46519 GPs (155.063/s) [rcu: g0 f0x0 ] n_max_cbs: 2197
TRACE01 ------- 756 GPs (2.52/s) [tasks-tracing: g0 f0x0 ]
TRACE02 ------- 559 GPs (1.86333/s) [tasks-tracing: g0 f0x0 ]
TREE01 ------- 8930 GPs (29.7667/s) [rcu: g64765 f0x0 ]
TREE02 ------- 17514 GPs (58.38/s) [rcu: g138645 f0x0 ] n_max_cbs: 18010
TREE03 ------- 15920 GPs (53.0667/s) [rcu: g159973 f0x0 ] n_max_cbs: 1025308
CPU count limited from 16 to 12
TREE04 ------- 10821 GPs (36.07/s) [rcu: g70293 f0x0 ] n_max_cbs: 81293
TREE05 ------- 16942 GPs (56.4733/s) [rcu: g123745 f0x0 ] n_max_cbs: 99796
TREE07 ------- 8248 GPs (27.4933/s) [rcu: g52933 f0x0 ] n_max_cbs: 183589
CPU count limited from 16 to 12
TREE09 ------- 39903 GPs (133.01/s) [rcu: g717745 f0x0 ] n_max_cbs: 83002
However, other than these two complaints, this is what the summary of an uneventful rcutorture run looks like.Whatever is the meaning of all those numbers in the summary???
The console output for each run and much else besides may be found in the /home/git/linux/tools/testing/selftests/rcutorture/res/2020.06.15-10.23.02
directory called out above.
The more CPUs you have, the fewer batches are required:
If you specify more CPUs than your system actually has, kvm.sh
will ignore your fantasies in favor of your system's reality.
Specifying Specific Scenarios
Sometimes it is useful to take one's ire out on a specific type of RCU, for example, SRCU. You can use the --configs
argument to select specific scenarios:
tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 12 \
--configs "SRCU-N SRCU-P SRCU-t SRCU-u"
This runs in two batches, but the second batch uses only two CPUs, which is again wasteful. Given that SRCU-P
requires eight CPUs, SRCU-N
four CPUs, and SRCU-t
one each, it would cost nothing to run two instances of each of these scenarios other than SRCU-N
tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 12 \
--configs "SRCU-N 2*SRCU-P 2*SRCU-t 2*SRCU-u"
This same notation can be used to run multiple copies of the entire list of scenarios. For example (again, in v5.7), a system with 384 CPUs can use --configs 4*CFLIST
to run four copies of of the full set of scenarios as follows:
tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 384 --configs "4*CFLIST"
Mixing and matching is permissible, for example:
tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 384 --configs "3*CFLIST 12*TREE02"
script that is to run on a wide variety of systems can benefit from --allcpus
(expected to appear in v5.9), which acts like --cpus N
, where N
is the number of CPUs on the current system:
tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --configs "3*CFLIST 12*TREE02"
Build time can dominate when running a large number of short-duration runs, for example, when chasing down a low-probability non-deterministic boot-time failure. Use of --trust-make
can be very helpful in this case:
tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 384 --duration 2 \
--configs "1000*TINY01" --trust-make
will play it safe by forcing your source tree to a known state between each build. In addition to --trust-make
, there are a number of tools such as ccache
that can also greatly reduce build times.
Locating Test Failures
Although the ability to automatically run many tens of scenarios can be very convenient, it can also cause significant eyestrain staring through a long “summary” checking for test failures. Therefore, if there are failures, this is noted at the end of the summary, for example, as shown in the following abbreviated output from a --configs "28*TREE03"
TREE03.8 ------- 1195094 GPs (55.3284/s) [rcu: g11475633 f0x0 ] n_max_cbs: 1449125
TREE03.9 ------- 1202936 GPs (55.6915/s) [rcu: g11572377 f0x0 ] n_max_cbs: 1514561
3 runs with runtime errors.
Of course, picking the three errors out of the 28 runs can also cause eyestrain, so there is yet another useful little script:
This will run your editor on the make
output for each build error and on the console output for each runtime failure, greatly reducing eyestrain. Users of vi
can also edit a summary of the runtime errors from each failing run as follows:
Enlisting Torture Assistance
produces a failure-free run, that is a failure on the part of rcutorture
. After all, there are bugs in there somewhere, and rcutorture
failed to find them!
One approach is to increase the duration, for example, to 12 hours (also known as 720 minutes):
tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 12 --duration 720
Another approach is to enlist the help of other in-kernel torture features, for example, lockdep
. The --kconfig
parameter to kvm.sh
can be used to this end:
tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 12 --configs "TREE03" \
--kconfig "CONFIG_DEBUG_LOCK_ALLOC=y CONFIG_PROVE_LOCKING=y"
The aid of the kernel address sanitizer (KASAN) can be enlisted using the --kasan
tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 12 --kasan
The kernel concurrency sanitizer (KCSAN) can also be brought to bear, but proper use of KCSAN requires some thought (see part 1
and part 2
of the LWN “Concurrency bugs should fear the big bad data-race detector” article) and also version 11 or later of Clang/LLVM (and a patch for GCC has been accepted). Once you have all of that in place, the --kcsan
argument invokes KCSAN and also generates a summary as described in part 1
of the aforementioned LWN article. Note again that only very recent compiler versions (such as Clang-11) support KCSAN, so a --kmake "CC=clang-11"
or similar argument might also be necessary.
Sometimes enlisting debugging aid is the best approach, but other times greater selectivity is the best way forward.
Sometimes simply building a kernel is torture enough, especially when building with unusual Kconfig options (see the discussion of --kconfig
above). In this case, specifying the --buildonly
argument will build the kernels, but refrain from running them. This approach can also be useful for running multiple copies of the resulting binaries on multiple systems: You can use the --buildonly
to build the kernels and qemu-cmd
scripts, and then run these files on the other systems, given suitable adjustments to the qemu-cmd
Other times it is useful to torture some specific portion of RCU. For example, one wishing to vent their ire solely on expedited grace periods could add --bootargs "rcutorture.gp_exp=1"
to the kvm.sh
command line. This argument causes rcutorture
to run a stress test using only expedited RCU grace periods, which can be helpful when attempting to work out whether a too-short RCU grace period is due to a bug in the normal or the expedited grace-period code. Similarly, the callback-flooding aspects of rcutorture
stress testing can be disabled using --bootargs "rcutorture.fwd_progress=0"
. It is possible to specify both in one run using --bootargs "rcutorture.gp_exp=1 rcutorture.fwd_progress=0"
Enlisting Debugging Assistance
Still other times, it is helpful to enable event tracing. For example, if the rcu_barrier()
event traces are of interest, use --bootargs "trace_event=rcu:rcu_barrier"
. The trace buffer will be dumped automatically upon specific rcutorture failures. If the failure mode is instead a non-rcutorture-specific oops, use this: --bootargs "trace_event=rcu:rcu_barrier ftrace_dump_on_oops"
. If it is also necessary to dump the trace buffers on warnings, a (heavy handed) way to achieve this is to use --bootargs "trace_event=rcu:rcu_barrier ftrace_dump_on_oops panic_on_warn"
If you have many tens of rcutorture
instances that all decide to flush their trace buffers at about the same time, the combined flushing operations can take considerable time, especially if the underlying system features rotating rust. If only the most recent activity is of interest, specifying a small trace buffer can help: --bootargs "trace_event=rcu:rcu_barrier ftrace_dump_on_oops panic_on_warn trace_buf_size=3k"
If only the oopsing/warning CPU's traces are relevant, the orig_cpu
modifier can be helpful: --bootargs "trace_event=rcu:rcu_barrier ftrace_dump_on_oops=orig_cpu panic_on_warn trace_buf_size=3k"
More information on tracing can be found in Documentation/trace
, and more on kernel boot parameters in general may be found in kernel-parameters.txt
. Given the multi-thousand-line heft of this latter, there is clearly great scope for tweaking your torturing of RCU!
Why Stop at Torturing RCU?
After all, locking can sometimes be almost as annoying as RCU. And it is possible to torture locking:
tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --torture lock
stress test does not get as much love and attention as does rcutorture
, but it is at least a start.
There are also a couple of RCU performance tests and an upcoming smp_call_function*()
stress test that use this same torture-test infrastructure. Please note that the details of the summary output varies from test to test.
In short, you can do some serious torturing of RCU, and much else besides! So show them no mercy!!! :-)