qemu with hax to log dma reads & writes jcs.org/2018/11/12/vfio

qapi: force a UTF-8 locale for running Python

Python2 did not validate locale correctness when reading input data, so
would happily read UTF-8 data in non-UTF-8 locales. Python3 is strict so
if you try to read UTF-8 data in the C locale, it will raise an error
for any UTF-8 bytes that aren't representable in 7-bit ascii encoding.
e.g.

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 54: ordinal not in range(128)
Traceback (most recent call last):
File "/tmp/qemu-test/src/scripts/qapi-commands.py", line 317, in <module>
schema = QAPISchema(input_file)
File "/tmp/qemu-test/src/scripts/qapi.py", line 1468, in __init__
parser = QAPISchemaParser(open(fname, 'r'))
File "/tmp/qemu-test/src/scripts/qapi.py", line 301, in __init__
previously_included)
File "/tmp/qemu-test/src/scripts/qapi.py", line 348, in _include
exprs_include = QAPISchemaParser(fobj, previously_included, info)
File "/tmp/qemu-test/src/scripts/qapi.py", line 271, in __init__
self.src = fp.read()
File "/usr/lib64/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]

More background on this can be seen in

https://www.python.org/dev/peps/pep-0538/

Many distros support a new C.UTF-8 locale that is like the C locale,
but with UTF-8 instead of 7-bit ASCII. That is not entirely portable
though. This patch thus sets the LANG to "C", but overrides LC_CTYPE
to be en_US.UTF-8 locale. This gets us pretty close to C.UTF-8, but
in a way that should be portable to everywhere QEMU builds.

This patch only forces UTF-8 for QAPI scripts, since that is the one
showing the immediate error under Python3 with C locale, but potentially
we ought to force this for all python scripts used in the build process.

Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Message-Id: <20180116134217.8725-9-berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>

authored by

Daniel P. Berrange and committed by
Eduardo Habkost
d4e5ec87 f7a5376d

+12 -10
+12 -10
Makefile
··· 20 20 all: 21 21 include config-host.mak 22 22 23 + PYTHON_UTF8 = LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8 $(PYTHON) 24 + 23 25 git-submodule-update: 24 26 25 27 .PHONY: git-submodule-update ··· 487 489 488 490 qga/qapi-generated/qga-qapi-types.c qga/qapi-generated/qga-qapi-types.h :\ 489 491 $(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-types.py $(qapi-py) 490 - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py \ 492 + $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-types.py \ 491 493 $(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \ 492 494 "GEN","$@") 493 495 qga/qapi-generated/qga-qapi-visit.c qga/qapi-generated/qga-qapi-visit.h :\ 494 496 $(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-visit.py $(qapi-py) 495 - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py \ 497 + $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-visit.py \ 496 498 $(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \ 497 499 "GEN","$@") 498 500 qga/qapi-generated/qga-qmp-commands.h qga/qapi-generated/qga-qmp-marshal.c :\ 499 501 $(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-commands.py $(qapi-py) 500 - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py \ 502 + $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-commands.py \ 501 503 $(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \ 502 504 "GEN","$@") 503 505 ··· 518 520 519 521 qapi-types.c qapi-types.h :\ 520 522 $(qapi-modules) $(SRC_PATH)/scripts/qapi-types.py $(qapi-py) 521 - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py \ 523 + $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-types.py \ 522 524 $(gen-out-type) -o "." -b $<, \ 523 525 "GEN","$@") 524 526 qapi-visit.c qapi-visit.h :\ 525 527 $(qapi-modules) $(SRC_PATH)/scripts/qapi-visit.py $(qapi-py) 526 - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py \ 528 + $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-visit.py \ 527 529 $(gen-out-type) -o "." -b $<, \ 528 530 "GEN","$@") 529 531 qapi-event.c qapi-event.h :\ 530 532 $(qapi-modules) $(SRC_PATH)/scripts/qapi-event.py $(qapi-py) 531 - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-event.py \ 533 + $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-event.py \ 532 534 $(gen-out-type) -o "." $<, \ 533 535 "GEN","$@") 534 536 qmp-commands.h qmp-marshal.c :\ 535 537 $(qapi-modules) $(SRC_PATH)/scripts/qapi-commands.py $(qapi-py) 536 - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py \ 538 + $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-commands.py \ 537 539 $(gen-out-type) -o "." $<, \ 538 540 "GEN","$@") 539 541 qmp-introspect.h qmp-introspect.c :\ 540 542 $(qapi-modules) $(SRC_PATH)/scripts/qapi-introspect.py $(qapi-py) 541 - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-introspect.py \ 543 + $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-introspect.py \ 542 544 $(gen-out-type) -o "." $<, \ 543 545 "GEN","$@") 544 546 ··· 811 813 docs/interop/qemu-qmp-qapi.texi docs/interop/qemu-ga-qapi.texi: $(SRC_PATH)/scripts/qapi2texi.py $(qapi-py) 812 814 813 815 docs/interop/qemu-qmp-qapi.texi: $(qapi-modules) 814 - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi2texi.py $< > $@,"GEN","$@") 816 + $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi2texi.py $< > $@,"GEN","$@") 815 817 816 818 docs/interop/qemu-ga-qapi.texi: $(SRC_PATH)/qga/qapi-schema.json 817 - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi2texi.py $< > $@,"GEN","$@") 819 + $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi2texi.py $< > $@,"GEN","$@") 818 820 819 821 qemu.1: qemu-doc.texi qemu-options.texi qemu-monitor.texi qemu-monitor-info.texi 820 822 qemu.1: qemu-option-trace.texi