An earlier post
discussed ways of making rcutorture
more aggressive, but even with these techniques, rcutorture
's level of aggression is limited by build time on the one hand and the confines of a single system on the other. This post describes some recent ways around those limitations.
Play It Again, Sam!
A full rcutorture
run will do about 20 kernel builds, which can take some tens of minutes or, on slower systems, well over an hour. This can be extremely annoying when you simply want to re-run the last test in order to obtain better failure statistics or to get more test time on a recent bug fix.
The traditional rcutorture
way of avoiding rebuilds is to optionally edit the qemu-cmd
files for each scenario to be re-run, then manually invoke sh
on each resulting file. The editing step allows you to avoid overwriting the previous run's console output, but may be omitted if you don't care about that console output or if you have already saved it off somewhere. This works, but is painstaking and error-prone.
This is where the new kvm-again.sh
script comes in. Its first argument is the path to the directory for the old run, for one example on my laptop, tools/testing/selftests/rcutorture/res/2021.03.31-10.52.56
. This can be a relative pathname as in this example, but use of absolute pathnames can make your life easier when reviewing output from prior kvm-again.sh
runs. By default, the new run will have the same duration as the old run, but the --duration
argument may be used to specify the new run's duration. Also by default, kvm-again.sh
will generate the new run's directory based on the current date and time (suffixed with -again
), but the --rundir
argument may be used to specify some other location. Finally, and again by default, hard links are used to “copy” the needed data from the old run directory (such as the Linux kernel), but the --link
argument can be used to specify soft
links or explicit copy
operations. The full set of scenarios generates some 20 kernels, each of which is somewhat larger than they would have been in the past. You may therefore need to exercise some caution when using --link copy
, especially if you are doing repeated kvm-again.sh
file in the new run directory gives the pathname of the old run directory. Although you can give a run directory produced by a prior kvm-again.sh
invocation to a later kvm-again.sh
invocation, best practice is to continue specifying the original run directory. If nothing else, following this best practice avoids ever-growing qemu-cmd
Of course, the shorter the runs, the greater an advantage kvm-again.sh
provides. In the extreme case, it can be amazingly helpful when testing for rare boot-time failures.
Strength in Numbers
It seems likely that there are quite a few more people with access to eight 16-CPU systems than there are people with access to a single 128-CPU system. You can of course run kvm.sh
on each of eight 16-CPU systems, but working out which scenarios to run on each of those systems can be time-consuming and error-prone. And this is why the new kvm-remote.sh
Build or Buy?
This script can be invoked in two different modes. In both cases, the first argument is a quoted list of system names, as in names that the ssh
command understands. Specifying localhost
or any of its synonyms might
work, but is an option for the brave at this point. Should this prove useful, it will be take care of in a later version of this script.
The first form builds all needed kernels on the system on which the kvm-remote.sh
script is run. In this case, the second and subsequent arguments can be anything accepted by the kvm.sh
In the second form, the second and subsequent arguments must be suitable for the kvm-again.sh
script, that is, the second argument must specify the path to an old run directory and the third and subsequent arguments can be --duration
, and </tt>--link</tt>.
In both forms, once the kernels are available, a tarball of all scenarios is downloaded to all of the systems. Each such download is run sequentially, which means that downloading can take significant time, especially if low-bandwidth network links are involved. Once all systems have had the tarball downloaded and expanded, batches of scenarios are parceled out among the systems specified by the first argument. If there are more batches than there are systems, once a system completes its current batch, it will be given another batch.
Once all batches have completed, the results from each system are uploaded back to the system running the kvm-remote.sh
script, where the usual end-of-run error-checking and analysis is carried out.
This script assumes that all systems have the same number of CPUs. Addressing this limitations is future work. In the meantime, one workaround is to do multiple --buildonly
runs of kvm.sh
, one for each type of system. Then multiple runs of the second form of the kvm-remote.sh
script can safely be run concurrently on the same build system. Because all the pre-built kernels for each type of system are safely collected up in the corresponding old-run directory, the multiple invocations of kvm-remote.sh
will not interfere with each other.
script uses ssh
to do all downloading, control, and uploading operations. This might seem to be a poor choice in this age of Kubernetes and friends, but the fact remains that ssh
is widely available, easy to configure, and reasonably robust. In contrast, there is a wide variety of Kubernetes-like systems, and they can be configured in a wide variety of ways. It would be impossible to choose just one of these systems, and it would be quite difficult to accommodate all of the configurations, versions, and variants of even one of them.
However, please note that kvm-remote.sh
assumes that all of the systems have been set up properly. This means that low-level virtualization support must be in place, and it also means that running an ssh
command to any of the specified systems must complete without the need for any human interaction. For example, if ssh foo date
does not open a connection to system foo
, run the date
command, and print the result without any need to type any sort password or passphrase, then system foo
is not yet set up properly.
does not take any actions that might be necessary to reserve system foo
for your exclusive use, nor does it do anything to release this system upon completion of the test. Thus, these system-configuration, reservation, and release operations are jobs for which you may wish to enlist the help of Kubernetes or of similar frameworks. For example, I use (admittedly crude) scripts that interact with Facebook's internal environment to reserve and configure the desired number and type of systems, invoke kvm-remote.sh
once everything is set up, and then release those systems.
What Might The Future Hold?
Although the kvm-remote.sh
approach of using ssh
works reasonably well on a few tens of systems, if someone wanted to run rcutorture
on thousands of systems, something else would likely be required. On the other hand, there are not that many sites where one would reasonably devote anywhere near that many systems to rcutorture
. There might be downloading improvements at some point, most likely in the form of allowing a script to be provided to allow kvm-remote.sh
to use some site-specific optimized multi-system download utility. Both kvm-again.sh
might someday need a way to specify that only a subset of a prior run's scenarios be re-run, for example, to chase down a bug that occurred in only a few of those scenarios.
And as mentioned earlier, perhaps a future version of kvm-remote.sh
will gracefully handle remote systems with varying numbers of CPUs or running actual tests on the system running the kvm-remote.sh
But if things go as they usually do, a fair fraction of the future changes will come as complete surprises.