jcs.org
qemu with hax to log dma reads & writes
jcs.org/2018/11/12/vfio
docs: Add docs/devel/testing.rst
To make our efforts on QEMU testing easier to consume by contributors,
let's add a document. For example, Patchew reports build errors on
patches that should be relatively easy to reproduce with a few steps, and
it is much nicer if there is such a documentation that it can refer to.
This focuses on how to run existing tests and how to write new test
cases, without going into the frameworks themselves.
The VM based testing section is moved from tests/vm/README which now
is a single line pointing to the new doc.
Signed-off-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20180201022046.9425-1-famz@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
+487
-89
+486
docs/devel/testing.rst
+486
docs/devel/testing.rst
···
1
+
===============
2
+
Testing in QEMU
3
+
===============
4
+
5
+
This document describes the testing infrastructure in QEMU.
6
+
7
+
Testing with "make check"
8
+
=========================
9
+
10
+
The "make check" testing family includes most of the C based tests in QEMU. For
11
+
a quick help, run ``make check-help`` from the source tree.
12
+
13
+
The usual way to run these tests is:
14
+
15
+
.. code::
16
+
17
+
make check
18
+
19
+
which includes QAPI schema tests, unit tests, and QTests. Different sub-types
20
+
of "make check" tests will be explained below.
21
+
22
+
Before running tests, it is best to build QEMU programs first. Some tests
23
+
expect the executables to exist and will fail with obscure messages if they
24
+
cannot find them.
25
+
26
+
Unit tests
27
+
----------
28
+
29
+
Unit tests, which can be invoked with ``make check-unit``, are simple C tests
30
+
that typically link to individual QEMU object files and exercise them by
31
+
calling exported functions.
32
+
33
+
If you are writing new code in QEMU, consider adding a unit test, especially
34
+
for utility modules that are relatively stateless or have few dependencies. To
35
+
add a new unit test:
36
+
37
+
1. Create a new source file. For example, ``tests/foo-test.c``.
38
+
39
+
2. Write the test. Normally you would include the header file which exports
40
+
the module API, then verify the interface behaves as expected from your
41
+
test. The test code should be organized with the glib testing framework.
42
+
Copying and modifying an existing test is usually a good idea.
43
+
44
+
3. Add the test to ``tests/Makefile.include``. First, name the unit test
45
+
program and add it to ``$(check-unit-y)``; then add a rule to build the
46
+
executable. Optionally, you can add a magical variable to support ``gcov``.
47
+
For example:
48
+
49
+
.. code::
50
+
51
+
check-unit-y += tests/foo-test$(EXESUF)
52
+
tests/foo-test$(EXESUF): tests/foo-test.o $(test-util-obj-y)
53
+
...
54
+
gcov-files-foo-test-y = util/foo.c
55
+
56
+
Since unit tests don't require environment variables, the simplest way to debug
57
+
a unit test failure is often directly invoking it or even running it under
58
+
``gdb``. However there can still be differences in behavior between ``make``
59
+
invocations and your manual run, due to ``$MALLOC_PERTURB_`` environment
60
+
variable (which affects memory reclamation and catches invalid pointers better)
61
+
and gtester options. If necessary, you can run
62
+
63
+
.. code::
64
+
make check-unit V=1
65
+
66
+
and copy the actual command line which executes the unit test, then run
67
+
it from the command line.
68
+
69
+
QTest
70
+
-----
71
+
72
+
QTest is a device emulation testing framework. It can be very useful to test
73
+
device models; it could also control certain aspects of QEMU (such as virtual
74
+
clock stepping), with a special purpose "qtest" protocol. Refer to the
75
+
documentation in ``qtest.c`` for more details of the protocol.
76
+
77
+
QTest cases can be executed with
78
+
79
+
.. code::
80
+
81
+
make check-qtest
82
+
83
+
The QTest library is implemented by ``tests/libqtest.c`` and the API is defined
84
+
in ``tests/libqtest.h``.
85
+
86
+
Consider adding a new QTest case when you are introducing a new virtual
87
+
hardware, or extending one if you are adding functionalities to an existing
88
+
virtual device.
89
+
90
+
On top of libqtest, a higher level library, ``libqos``, was created to
91
+
encapsulate common tasks of device drivers, such as memory management and
92
+
communicating with system buses or devices. Many virtual device tests use
93
+
libqos instead of directly calling into libqtest.
94
+
95
+
Steps to add a new QTest case are:
96
+
97
+
1. Create a new source file for the test. (More than one file can be added as
98
+
necessary.) For example, ``tests/test-foo-device.c``.
99
+
100
+
2. Write the test code with the glib and libqtest/libqos API. See also existing
101
+
tests and the library headers for reference.
102
+
103
+
3. Register the new test in ``tests/Makefile.include``. Add the test executable
104
+
name to an appropriate ``check-qtest-*-y`` variable. For example:
105
+
106
+
``check-qtest-generic-y = tests/test-foo-device$(EXESUF)``
107
+
108
+
4. Add object dependencies of the executable in the Makefile, including the
109
+
test source file(s) and other interesting objects. For example:
110
+
111
+
``tests/test-foo-device$(EXESUF): tests/test-foo-device.o $(libqos-obj-y)``
112
+
113
+
Debugging a QTest failure is slightly harder than the unit test because the
114
+
tests look up QEMU program names in the environment variables, such as
115
+
``QTEST_QEMU_BINARY`` and ``QTEST_QEMU_IMG``, and also because it is not easy
116
+
to attach gdb to the QEMU process spawned from the test. But manual invoking
117
+
and using gdb on the test is still simple to do: find out the actual command
118
+
from the output of
119
+
120
+
.. code::
121
+
make check-qtest V=1
122
+
123
+
which you can run manually.
124
+
125
+
QAPI schema tests
126
+
-----------------
127
+
128
+
The QAPI schema tests validate the QAPI parser used by QMP, by feeding
129
+
predefined input to the parser and comparing the result with the reference
130
+
output.
131
+
132
+
The input/output data is managed under the ``tests/qapi-schema`` directory.
133
+
Each test case includes four files that have a common base name:
134
+
135
+
* ``${casename}.json`` - the file contains the JSON input for feeding the
136
+
parser
137
+
* ``${casename}.out`` - the file contains the expected stdout from the parser
138
+
* ``${casename}.err`` - the file contains the expected stderr from the parser
139
+
* ``${casename}.exit`` - the expected error code
140
+
141
+
Consider adding a new QAPI schema test when you are making a change on the QAPI
142
+
parser (either fixing a bug or extending/modifying the syntax). To do this:
143
+
144
+
1. Add four files for the new case as explained above. For example:
145
+
146
+
``$EDITOR tests/qapi-schema/foo.{json,out,err,exit}``.
147
+
148
+
2. Add the new test in ``tests/Makefile.include``. For example:
149
+
150
+
``qapi-schema += foo.json``
151
+
152
+
check-block
153
+
-----------
154
+
155
+
``make check-block`` is a legacy command to invoke block layer iotests and is
156
+
rarely used. See "QEMU iotests" section below for more information.
157
+
158
+
GCC gcov support
159
+
----------------
160
+
161
+
``gcov`` is a GCC tool to analyze the testing coverage by instrumenting the
162
+
tested code. To use it, configure QEMU with ``--enable-gcov`` option and build.
163
+
Then run ``make check`` as usual. There will be additional ``gcov`` output as
164
+
the testing goes on, showing the test coverage percentage numbers per analyzed
165
+
source file. More detailed reports can be obtained by running ``gcov`` command
166
+
on the output files under ``$build_dir/tests/``, please read the ``gcov``
167
+
documentation for more information.
168
+
169
+
QEMU iotests
170
+
============
171
+
172
+
QEMU iotests, under the directory ``tests/qemu-iotests``, is the testing
173
+
framework widely used to test block layer related features. It is higher level
174
+
than "make check" tests and 99% of the code is written in bash or Python
175
+
scripts. The testing success criteria is golden output comparison, and the
176
+
test files are named with numbers.
177
+
178
+
To run iotests, make sure QEMU is built successfully, then switch to the
179
+
``tests/qemu-iotests`` directory under the build directory, and run ``./check``
180
+
with desired arguments from there.
181
+
182
+
By default, "raw" format and "file" protocol is used; all tests will be
183
+
executed, except the unsupported ones. You can override the format and protocol
184
+
with arguments:
185
+
186
+
.. code::
187
+
188
+
# test with qcow2 format
189
+
./check -qcow2
190
+
# or test a different protocol
191
+
./check -nbd
192
+
193
+
It's also possible to list test numbers explicitly:
194
+
195
+
.. code::
196
+
197
+
# run selected cases with qcow2 format
198
+
./check -qcow2 001 030 153
199
+
200
+
Cache mode can be selected with the "-c" option, which may help reveal bugs
201
+
that are specific to certain cache mode.
202
+
203
+
More options are supported by the ``./check`` script, run ``./check -h`` for
204
+
help.
205
+
206
+
Writing a new test case
207
+
-----------------------
208
+
209
+
Consider writing a tests case when you are making any changes to the block
210
+
layer. An iotest case is usually the choice for that. There are already many
211
+
test cases, so it is possible that extending one of them may achieve the goal
212
+
and save the boilerplate to create one. (Unfortunately, there isn't a 100%
213
+
reliable way to find a related one out of hundreds of tests. One approach is
214
+
using ``git grep``.)
215
+
216
+
Usually an iotest case consists of two files. One is an executable that
217
+
produces output to stdout and stderr, the other is the expected reference
218
+
output. They are given the same number in file names. E.g. Test script ``055``
219
+
and reference output ``055.out``.
220
+
221
+
In rare cases, when outputs differ between cache mode ``none`` and others, a
222
+
``.out.nocache`` file is added. In other cases, when outputs differ between
223
+
image formats, more than one ``.out`` files are created ending with the
224
+
respective format names, e.g. ``178.out.qcow2`` and ``178.out.raw``.
225
+
226
+
There isn't a hard rule about how to write a test script, but a new test is
227
+
usually a (copy and) modification of an existing case. There are a few
228
+
commonly used ways to create a test:
229
+
230
+
* A Bash script. It will make use of several environmental variables related
231
+
to the testing procedure, and could source a group of ``common.*`` libraries
232
+
for some common helper routines.
233
+
234
+
* A Python unittest script. Import ``iotests`` and create a subclass of
235
+
``iotests.QMPTestCase``, then call ``iotests.main`` method. The downside of
236
+
this approach is that the output is too scarce, and the script is considered
237
+
harder to debug.
238
+
239
+
* A simple Python script without using unittest module. This could also import
240
+
``iotests`` for launching QEMU and utilities etc, but it doesn't inherit
241
+
from ``iotests.QMPTestCase`` therefore doesn't use the Python unittest
242
+
execution. This is a combination of 1 and 2.
243
+
244
+
Pick the language per your preference since both Bash and Python have
245
+
comparable library support for invoking and interacting with QEMU programs. If
246
+
you opt for Python, it is strongly recommended to write Python 3 compatible
247
+
code.
248
+
249
+
Docker based tests
250
+
==================
251
+
252
+
Introduction
253
+
------------
254
+
255
+
The Docker testing framework in QEMU utilizes public Docker images to build and
256
+
test QEMU in predefined and widely accessible Linux environments. This makes
257
+
it possible to expand the test coverage across distros, toolchain flavors and
258
+
library versions.
259
+
260
+
Prerequisites
261
+
-------------
262
+
263
+
Install "docker" with the system package manager and start the Docker service
264
+
on your development machine, then make sure you have the privilege to run
265
+
Docker commands. Typically it means setting up passwordless ``sudo docker``
266
+
command or login as root. For example:
267
+
268
+
.. code::
269
+
270
+
$ sudo yum install docker
271
+
$ # or `apt-get install docker` for Ubuntu, etc.
272
+
$ sudo systemctl start docker
273
+
$ sudo docker ps
274
+
275
+
The last command should print an empty table, to verify the system is ready.
276
+
277
+
An alternative method to set up permissions is by adding the current user to
278
+
"docker" group and making the docker daemon socket file (by default
279
+
``/var/run/docker.sock``) accessible to the group:
280
+
281
+
.. code::
282
+
283
+
$ sudo groupadd docker
284
+
$ sudo usermod $USER -G docker
285
+
$ sudo chown :docker /var/run/docker.sock
286
+
287
+
Note that any one of above configurations makes it possible for the user to
288
+
exploit the whole host with Docker bind mounting or other privileged
289
+
operations. So only do it on development machines.
290
+
291
+
Quickstart
292
+
----------
293
+
294
+
From source tree, type ``make docker`` to see the help. Testing can be started
295
+
without configuring or building QEMU (``configure`` and ``make`` are done in
296
+
the container, with parameters defined by the make target):
297
+
298
+
.. code::
299
+
300
+
make docker-test-build@min-glib
301
+
302
+
This will create a container instance using the ``min-glib`` image (the image
303
+
is downloaded and initialized automatically), in which the ``test-build`` job
304
+
is executed.
305
+
306
+
Images
307
+
------
308
+
309
+
Along with many other images, the ``min-glib`` image is defined in a Dockerfile
310
+
in ``tests/docker/dockefiles/``, called ``min-glib.docker``. ``make docker``
311
+
command will list all the available images.
312
+
313
+
To add a new image, simply create a new ``.docker`` file under the
314
+
``tests/docker/dockerfiles/`` directory.
315
+
316
+
A ``.pre`` script can be added beside the ``.docker`` file, which will be
317
+
executed before building the image under the build context directory. This is
318
+
mainly used to do necessary host side setup. One such setup is ``binfmt_misc``,
319
+
for example, to make qemu-user powered cross build containers work.
320
+
321
+
Tests
322
+
-----
323
+
324
+
Different tests are added to cover various configurations to build and test
325
+
QEMU. Docker tests are the executables under ``tests/docker`` named
326
+
``test-*``. They are typically shell scripts and are built on top of a shell
327
+
library, ``tests/docker/common.rc``, which provides helpers to find the QEMU
328
+
source and build it.
329
+
330
+
The full list of tests is printed in the ``make docker`` help.
331
+
332
+
Tools
333
+
-----
334
+
335
+
There are executables that are created to run in a specific Docker environment.
336
+
This makes it easy to write scripts that have heavy or special dependencies,
337
+
but are still very easy to use.
338
+
339
+
Currently the only tool is ``travis``, which mimics the Travis-CI tests in a
340
+
container. It runs in the ``travis`` image:
341
+
342
+
.. code::
343
+
344
+
make docker-travis@travis
345
+
346
+
Debugging a Docker test failure
347
+
-------------------------------
348
+
349
+
When CI tasks, maintainers or yourself report a Docker test failure, follow the
350
+
below steps to debug it:
351
+
352
+
1. Locally reproduce the failure with the reported command line. E.g. run
353
+
``make docker-test-mingw@fedora J=8``.
354
+
2. Add "V=1" to the command line, try again, to see the verbose output.
355
+
3. Further add "DEBUG=1" to the command line. This will pause in a shell prompt
356
+
in the container right before testing starts. You could either manually
357
+
build QEMU and run tests from there, or press Ctrl-D to let the Docker
358
+
testing continue.
359
+
4. If you press Ctrl-D, the same building and testing procedure will begin, and
360
+
will hopefully run into the error again. After that, you will be dropped to
361
+
the prompt for debug.
362
+
363
+
Options
364
+
-------
365
+
366
+
Various options can be used to affect how Docker tests are done. The full
367
+
list is in the ``make docker`` help text. The frequently used ones are:
368
+
369
+
* ``V=1``: the same as in top level ``make``. It will be propagated to the
370
+
container and enable verbose output.
371
+
* ``J=$N``: the number of parallel tasks in make commands in the container,
372
+
similar to the ``-j $N`` option in top level ``make``. (The ``-j`` option in
373
+
top level ``make`` will not be propagated into the container.)
374
+
* ``DEBUG=1``: enables debug. See the previous "Debugging a Docker test
375
+
failure" section.
376
+
377
+
VM testing
378
+
==========
379
+
380
+
This test suite contains scripts that bootstrap various guest images that have
381
+
necessary packages to build QEMU. The basic usage is documented in ``Makefile``
382
+
help which is displayed with ``make vm-test``.
383
+
384
+
Quickstart
385
+
----------
386
+
387
+
Run ``make vm-test`` to list available make targets. Invoke a specific make
388
+
command to run build test in an image. For example, ``make vm-build-freebsd``
389
+
will build the source tree in the FreeBSD image. The command can be executed
390
+
from either the source tree or the build dir; if the former, ``./configure`` is
391
+
not needed. The command will then generate the test image in ``./tests/vm/``
392
+
under the working directory.
393
+
394
+
Note: images created by the scripts accept a well-known RSA key pair for SSH
395
+
access, so they SHOULD NOT be exposed to external interfaces if you are
396
+
concerned about attackers taking control of the guest and potentially
397
+
exploiting a QEMU security bug to compromise the host.
398
+
399
+
QEMU binary
400
+
-----------
401
+
402
+
By default, qemu-system-x86_64 is searched in $PATH to run the guest. If there
403
+
isn't one, or if it is older than 2.10, the test won't work. In this case,
404
+
provide the QEMU binary in env var: ``QEMU=/path/to/qemu-2.10+``.
405
+
406
+
Make jobs
407
+
---------
408
+
409
+
The ``-j$X`` option in the make command line is not propagated into the VM,
410
+
specify ``J=$X`` to control the make jobs in the guest.
411
+
412
+
Debugging
413
+
---------
414
+
415
+
Add ``DEBUG=1`` and/or ``V=1`` to the make command to allow interactive
416
+
debugging and verbose output. If this is not enough, see the next section.
417
+
418
+
Manual invocation
419
+
-----------------
420
+
421
+
Each guest script is an executable script with the same command line options.
422
+
For example to work with the netbsd guest, use ``$QEMU_SRC/tests/vm/netbsd``:
423
+
424
+
.. code::
425
+
426
+
$ cd $QEMU_SRC/tests/vm
427
+
428
+
# To bootstrap the image
429
+
$ ./netbsd --build-image --image /var/tmp/netbsd.img
430
+
<...>
431
+
432
+
# To run an arbitrary command in guest (the output will not be echoed unless
433
+
# --debug is added)
434
+
$ ./netbsd --debug --image /var/tmp/netbsd.img uname -a
435
+
436
+
# To build QEMU in guest
437
+
$ ./netbsd --debug --image /var/tmp/netbsd.img --build-qemu $QEMU_SRC
438
+
439
+
# To get to an interactive shell
440
+
$ ./netbsd --interactive --image /var/tmp/netbsd.img sh
441
+
442
+
Adding new guests
443
+
-----------------
444
+
445
+
Please look at existing guest scripts for how to add new guests.
446
+
447
+
Most importantly, create a subclass of BaseVM and implement ``build_image()``
448
+
method and define ``BUILD_SCRIPT``, then finally call ``basevm.main()`` from
449
+
the script's ``main()``.
450
+
451
+
* Usually in ``build_image()``, a template image is downloaded from a
452
+
predefined URL. ``BaseVM._download_with_cache()`` takes care of the cache and
453
+
the checksum, so consider using it.
454
+
455
+
* Once the image is downloaded, users, SSH server and QEMU build deps should
456
+
be set up:
457
+
458
+
- Root password set to ``BaseVM.ROOT_PASS``
459
+
- User ``BaseVM.GUEST_USER`` is created, and password set to
460
+
``BaseVM.GUEST_PASS``
461
+
- SSH service is enabled and started on boot,
462
+
``$QEMU_SRC/tests/keys/id_rsa.pub`` is added to ssh's ``authorized_keys``
463
+
file of both root and the normal user
464
+
- DHCP client service is enabled and started on boot, so that it can
465
+
automatically configure the virtio-net-pci NIC and communicate with QEMU
466
+
user net (10.0.2.2)
467
+
- Necessary packages are installed to untar the source tarball and build
468
+
QEMU
469
+
470
+
* Write a proper ``BUILD_SCRIPT`` template, which should be a shell script that
471
+
untars a raw virtio-blk block device, which is the tarball data blob of the
472
+
QEMU source tree, then configure/build it. Running "make check" is also
473
+
recommended.
474
+
475
+
Image fuzzer testing
476
+
====================
477
+
478
+
An image fuzzer was added to exercise format drivers. Currently only qcow2 is
479
+
supported. To start the fuzzer, run
480
+
481
+
.. code::
482
+
483
+
tests/image-fuzzer/runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test qcow2
484
+
485
+
Alternatively, some command different from "qemu-img info" can be tested, by
486
+
changing the ``-c`` option.
+1
-89
tests/vm/README
+1
-89
tests/vm/README
···
1
-
=== VM test suite to run build in guests ===
2
-
3
-
== Intro ==
4
-
5
-
This test suite contains scripts that bootstrap various guest images that have
6
-
necessary packages to build QEMU. The basic usage is documented in Makefile
7
-
help which is displayed with "make vm-test".
8
-
9
-
== Quick start ==
10
-
11
-
Run "make vm-test" to list available make targets. Invoke a specific make
12
-
command to run build test in an image. For example, "make vm-build-freebsd"
13
-
will build the source tree in the FreeBSD image. The command can be executed
14
-
from either the source tree or the build dir; if the former, ./configure is not
15
-
needed. The command will then generate the test image in ./tests/vm/ under the
16
-
working directory.
17
-
18
-
Note: images created by the scripts accept a well-known RSA key pair for SSH
19
-
access, so they SHOULD NOT be exposed to external interfaces if you are
20
-
concerned about attackers taking control of the guest and potentially
21
-
exploiting a QEMU security bug to compromise the host.
22
-
23
-
== QEMU binary ==
24
-
25
-
By default, qemu-system-x86_64 is searched in $PATH to run the guest. If there
26
-
isn't one, or if it is older than 2.10, the test won't work. In this case,
27
-
provide the QEMU binary in env var: QEMU=/path/to/qemu-2.10+.
28
-
29
-
== Make jobs ==
30
-
31
-
The "-j$X" option in the make command line is not propagated into the VM,
32
-
specify "J=$X" to control the make jobs in the guest.
33
-
34
-
== Debugging ==
35
-
36
-
Add "DEBUG=1" and/or "V=1" to the make command to allow interactive debugging
37
-
and verbose output. If this is not enough, see the next section.
38
-
39
-
== Manual invocation ==
40
-
41
-
Each guest script is an executable script with the same command line options.
42
-
For example to work with the netbsd guest, use $QEMU_SRC/tests/vm/netbsd:
43
-
44
-
$ cd $QEMU_SRC/tests/vm
45
-
46
-
# To bootstrap the image
47
-
$ ./netbsd --build-image --image /var/tmp/netbsd.img
48
-
<...>
49
-
50
-
# To run an arbitrary command in guest (the output will not be echoed unless
51
-
# --debug is added)
52
-
$ ./netbsd --debug --image /var/tmp/netbsd.img uname -a
53
-
54
-
# To build QEMU in guest
55
-
$ ./netbsd --debug --image /var/tmp/netbsd.img --build-qemu $QEMU_SRC
56
-
57
-
# To get to an interactive shell
58
-
$ ./netbsd --interactive --image /var/tmp/netbsd.img sh
59
-
60
-
== Adding new guests ==
61
-
62
-
Please look at existing guest scripts for how to add new guests.
63
-
64
-
Most importantly, create a subclass of BaseVM and implement build_image()
65
-
method and define BUILD_SCRIPT, then finally call basevm.main() from the
66
-
script's main().
67
-
68
-
- Usually in build_image(), a template image is downloaded from a predefined
69
-
URL. BaseVM._download_with_cache() takes care of the cache and the
70
-
checksum, so consider using it.
71
-
72
-
- Once the image is downloaded, users, SSH server and QEMU build deps should
73
-
be set up:
74
-
75
-
* Root password set to BaseVM.ROOT_PASS
76
-
* User BaseVM.GUEST_USER is created, and password set to BaseVM.GUEST_PASS
77
-
* SSH service is enabled and started on boot,
78
-
$QEMU_SRC/tests/keys/id_rsa.pub is added to ssh's "authorized_keys" file
79
-
of both root and the normal user
80
-
* DHCP client service is enabled and started on boot, so that it can
81
-
automatically configure the virtio-net-pci NIC and communicate with QEMU
82
-
user net (10.0.2.2)
83
-
* Necessary packages are installed to untar the source tarball and build
84
-
QEMU
85
-
86
-
- Write a proper BUILD_SCRIPT template, which should be a shell script that
87
-
untars a raw virtio-blk block device, which is the tarball data blob of the
88
-
QEMU source tree, then configure/build it. Running "make check" is also
89
-
recommended.
1
+
See docs/devel/testing.rst for help.