docker: Improved image checksum · jcs.org/qemu@c1958e9

qemu with hax to log dma reads & writes jcs.org/2018/11/12/vfio

docker: Improved image checksum

When a base image locally defined by QEMU, such as in the debian images,
is updated, the dockerfile checksum mechanism in docker.py still skips
updating the derived image, because it only looks at the literal content
of the dockerfile, without considering changes to the base image.

For example we have a recent fix e58c1f9b35e81 that fixed
debian-win64-cross by updating its base image, debian8-mxe, but due to
above "feature" of docker.py the image in question is automatically NOT
rebuilt unless you add NOCACHE=1. It is noticed on Shippable:

https://app.shippable.com/github/qemu/qemu/runs/541/2/console

because after the fix is merged, the error still occurs, and the log
shows the container image is, as explained above, not updated.

This is because at the time docker.py was written, there wasn't any
dependencies between QEMU's docker images.

Now improve this to preprocess any "FROM qemu:*" directives in the
dockerfiles while doing checksum, and inline the base image's dockerfile
content, recursively. This ensures any changes on the depended _QEMU_
images are taken into account.

This means for external images that we expect to retrieve from docker
registries, we still do it as before. It is not perfect, because
registry images can get updated too. Technically we could substitute the
image name with its hex ID as obtained with $(docker images $IMAGE
--format="{{.Id}}"), but --format is not supported by RHEL 7, so leave
it for now.

Reported-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20171103131229.4737-1-famz@redhat.com>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Fam Zheng <famz@redhat.com>

Fam Zheng 8 years ago c1958e9d b0fbe46a

+23 -1

1 changed file

expand all

tests

docker

docker.py

+23 -1

tests/docker/docker.py

··· 105 105 so_path = os.path.dirname(l) 106 106 _copy_with_mkdir(l , dest_dir, so_path) 107 107 108 + def _read_qemu_dockerfile(img_name): 109 + df = os.path.join(os.path.dirname(__file__), "dockerfiles", 110 + img_name + ".docker") 111 + return open(df, "r").read() 112 + 113 + def _dockerfile_preprocess(df): 114 + out = "" 115 + for l in df.splitlines(): 116 + if len(l.strip()) == 0 or l.startswith("#"): 117 + continue 118 + from_pref = "FROM qemu:" 119 + if l.startswith(from_pref): 120 + # TODO: Alternatively we could replace this line with "FROM $ID" 121 + # where $ID is the image's hex id obtained with 122 + # $ docker images $IMAGE --format="{{.Id}}" 123 + # but unfortunately that's not supported by RHEL 7. 124 + inlining = _read_qemu_dockerfile(l[len(from_pref):]) 125 + out += _dockerfile_preprocess(inlining) 126 + continue 127 + out += l + "\n" 128 + return out 129 + 108 130 class Docker(object): 109 131 """ Running Docker commands """ 110 132 def __init__(self): ··· 196 218 checksum = self.get_image_dockerfile_checksum(tag) 197 219 except Exception: 198 220 return False 199 - return checksum == _text_checksum(dockerfile) 221 + return checksum == _text_checksum(_dockerfile_preprocess(dockerfile)) 200 222 201 223 def run(self, cmd, keep, quiet): 202 224 label = uuid.uuid1().hex