bitmaps.md: Convert to rST; move it into 'interop' dir

qemu with hax to log dma reads & writes jcs.org/2018/11/12/vfio

This is part of the on-going effort to convert QEMU upstream
documentation syntax to reStructuredText (rST).

The conversion to rST was done using:

$ pandoc -f markdown -t rst bitmaps.md -o bitmaps.rst

Then, make a couple of small syntactical adjustments. While at it,
reword a statement to avoid ambiguity. Addressing the feedback from
this thread:

https://lists.nongnu.org/archive/html/qemu-devel/2017-06/msg05428.html

Signed-off-by: Kashyap Chamarthy <kchamart@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20170717105205.32639-2-kchamart@redhat.com
Signed-off-by: Jeff Cody <jcody@redhat.com>

authored by

Kashyap Chamarthy and committed by

Jeff Cody 8 years ago 7746cf8a ca4e667d

+555 -505

2 changed files

expand all

docs

devel

bitmaps.md

interop

bitmaps.rst

-505

docs/devel/bitmaps.md

··· 1 -  8 - 9 - # Dirty Bitmaps and Incremental Backup 10 - 11 - * Dirty Bitmaps are objects that track which data needs to be backed up for the 12 - next incremental backup. 13 - 14 - * Dirty bitmaps can be created at any time and attached to any node 15 - (not just complete drives.) 16 - 17 - ## Dirty Bitmap Names 18 - 19 - * A dirty bitmap's name is unique to the node, but bitmaps attached to different 20 - nodes can share the same name. 21 - 22 - * Dirty bitmaps created for internal use by QEMU may be anonymous and have no 23 - name, but any user-created bitmaps may not be. There can be any number of 24 - anonymous bitmaps per node. 25 - 26 - * The name of a user-created bitmap must not be empty (""). 27 - 28 - ## Bitmap Modes 29 - 30 - * A Bitmap can be "frozen," which means that it is currently in-use by a backup 31 - operation and cannot be deleted, renamed, written to, reset, 32 - etc. 33 - 34 - * The normal operating mode for a bitmap is "active." 35 - 36 - ## Basic QMP Usage 37 - 38 - ### Supported Commands ### 39 - 40 - * block-dirty-bitmap-add 41 - * block-dirty-bitmap-remove 42 - * block-dirty-bitmap-clear 43 - 44 - ### Creation 45 - 46 - * To create a new bitmap, enabled, on the drive with id=drive0: 47 - 48 - ```json 49 - { "execute": "block-dirty-bitmap-add", 50 - "arguments": { 51 - "node": "drive0", 52 - "name": "bitmap0" 53 - } 54 - } 55 - ``` 56 - 57 - * This bitmap will have a default granularity that matches the cluster size of 58 - its associated drive, if available, clamped to between [4KiB, 64KiB]. 59 - The current default for qcow2 is 64KiB. 60 - 61 - * To create a new bitmap that tracks changes in 32KiB segments: 62 - 63 - ```json 64 - { "execute": "block-dirty-bitmap-add", 65 - "arguments": { 66 - "node": "drive0", 67 - "name": "bitmap0", 68 - "granularity": 32768 69 - } 70 - } 71 - ``` 72 - 73 - ### Deletion 74 - 75 - * Bitmaps that are frozen cannot be deleted. 76 - 77 - * Deleting the bitmap does not impact any other bitmaps attached to the same 78 - node, nor does it affect any backups already created from this node. 79 - 80 - * Because bitmaps are only unique to the node to which they are attached, 81 - you must specify the node/drive name here, too. 82 - 83 - ```json 84 - { "execute": "block-dirty-bitmap-remove", 85 - "arguments": { 86 - "node": "drive0", 87 - "name": "bitmap0" 88 - } 89 - } 90 - ``` 91 - 92 - ### Resetting 93 - 94 - * Resetting a bitmap will clear all information it holds. 95 - 96 - * An incremental backup created from an empty bitmap will copy no data, 97 - as if nothing has changed. 98 - 99 - ```json 100 - { "execute": "block-dirty-bitmap-clear", 101 - "arguments": { 102 - "node": "drive0", 103 - "name": "bitmap0" 104 - } 105 - } 106 - ``` 107 - 108 - ## Transactions 109 - 110 - ### Justification 111 - 112 - Bitmaps can be safely modified when the VM is paused or halted by using 113 - the basic QMP commands. For instance, you might perform the following actions: 114 - 115 - 1. Boot the VM in a paused state. 116 - 2. Create a full drive backup of drive0. 117 - 3. Create a new bitmap attached to drive0. 118 - 4. Resume execution of the VM. 119 - 5. Incremental backups are ready to be created. 120 - 121 - At this point, the bitmap and drive backup would be correctly in sync, 122 - and incremental backups made from this point forward would be correctly aligned 123 - to the full drive backup. 124 - 125 - This is not particularly useful if we decide we want to start incremental 126 - backups after the VM has been running for a while, for which we will need to 127 - perform actions such as the following: 128 - 129 - 1. Boot the VM and begin execution. 130 - 2. Using a single transaction, perform the following operations: 131 - * Create bitmap0. 132 - * Create a full drive backup of drive0. 133 - 3. Incremental backups are now ready to be created. 134 - 135 - ### Supported Bitmap Transactions 136 - 137 - * block-dirty-bitmap-add 138 - * block-dirty-bitmap-clear 139 - 140 - The usages are identical to their respective QMP commands, but see below 141 - for examples. 142 - 143 - ### Example: New Incremental Backup 144 - 145 - As outlined in the justification, perhaps we want to create a new incremental 146 - backup chain attached to a drive. 147 - 148 - ```json 149 - { "execute": "transaction", 150 - "arguments": { 151 - "actions": [ 152 - {"type": "block-dirty-bitmap-add", 153 - "data": {"node": "drive0", "name": "bitmap0"} }, 154 - {"type": "drive-backup", 155 - "data": {"device": "drive0", "target": "/path/to/full_backup.img", 156 - "sync": "full", "format": "qcow2"} } 157 - ] 158 - } 159 - } 160 - ``` 161 - 162 - ### Example: New Incremental Backup Anchor Point 163 - 164 - Maybe we just want to create a new full backup with an existing bitmap and 165 - want to reset the bitmap to track the new chain. 166 - 167 - ```json 168 - { "execute": "transaction", 169 - "arguments": { 170 - "actions": [ 171 - {"type": "block-dirty-bitmap-clear", 172 - "data": {"node": "drive0", "name": "bitmap0"} }, 173 - {"type": "drive-backup", 174 - "data": {"device": "drive0", "target": "/path/to/new_full_backup.img", 175 - "sync": "full", "format": "qcow2"} } 176 - ] 177 - } 178 - } 179 - ``` 180 - 181 - ## Incremental Backups 182 - 183 - The star of the show. 184 - 185 - **Nota Bene!** Only incremental backups of entire drives are supported for now. 186 - So despite the fact that you can attach a bitmap to any arbitrary node, they are 187 - only currently useful when attached to the root node. This is because 188 - drive-backup only supports drives/devices instead of arbitrary nodes. 189 - 190 - ### Example: First Incremental Backup 191 - 192 - 1. Create a full backup and sync it to the dirty bitmap, as in the transactional 193 - examples above; or with the VM offline, manually create a full copy and then 194 - create a new bitmap before the VM begins execution. 195 - 196 - * Let's assume the full backup is named 'full_backup.img'. 197 - * Let's assume the bitmap you created is 'bitmap0' attached to 'drive0'. 198 - 199 - 2. Create a destination image for the incremental backup that utilizes the 200 - full backup as a backing image. 201 - 202 - * Let's assume it is named 'incremental.0.img'. 203 - 204 - ```sh 205 - # qemu-img create -f qcow2 incremental.0.img -b full_backup.img -F qcow2 206 - ``` 207 - 208 - 3. Issue the incremental backup command: 209 - 210 - ```json 211 - { "execute": "drive-backup", 212 - "arguments": { 213 - "device": "drive0", 214 - "bitmap": "bitmap0", 215 - "target": "incremental.0.img", 216 - "format": "qcow2", 217 - "sync": "incremental", 218 - "mode": "existing" 219 - } 220 - } 221 - ``` 222 - 223 - ### Example: Second Incremental Backup 224 - 225 - 1. Create a new destination image for the incremental backup that points to the 226 - previous one, e.g.: 'incremental.1.img' 227 - 228 - ```sh 229 - # qemu-img create -f qcow2 incremental.1.img -b incremental.0.img -F qcow2 230 - ``` 231 - 232 - 2. Issue a new incremental backup command. The only difference here is that we 233 - have changed the target image below. 234 - 235 - ```json 236 - { "execute": "drive-backup", 237 - "arguments": { 238 - "device": "drive0", 239 - "bitmap": "bitmap0", 240 - "target": "incremental.1.img", 241 - "format": "qcow2", 242 - "sync": "incremental", 243 - "mode": "existing" 244 - } 245 - } 246 - ``` 247 - 248 - ## Errors 249 - 250 - * In the event of an error that occurs after a backup job is successfully 251 - launched, either by a direct QMP command or a QMP transaction, the user 252 - will receive a BLOCK_JOB_COMPLETE event with a failure message, accompanied 253 - by a BLOCK_JOB_ERROR event. 254 - 255 - * In the case of an event being cancelled, the user will receive a 256 - BLOCK_JOB_CANCELLED event instead of a pair of COMPLETE and ERROR events. 257 - 258 - * In either case, the incremental backup data contained within the bitmap is 259 - safely rolled back, and the data within the bitmap is not lost. The image 260 - file created for the failed attempt can be safely deleted. 261 - 262 - * Once the underlying problem is fixed (e.g. more storage space is freed up), 263 - you can simply retry the incremental backup command with the same bitmap. 264 - 265 - ### Example 266 - 267 - 1. Create a target image: 268 - 269 - ```sh 270 - # qemu-img create -f qcow2 incremental.0.img -b full_backup.img -F qcow2 271 - ``` 272 - 273 - 2. Attempt to create an incremental backup via QMP: 274 - 275 - ```json 276 - { "execute": "drive-backup", 277 - "arguments": { 278 - "device": "drive0", 279 - "bitmap": "bitmap0", 280 - "target": "incremental.0.img", 281 - "format": "qcow2", 282 - "sync": "incremental", 283 - "mode": "existing" 284 - } 285 - } 286 - ``` 287 - 288 - 3. Receive an event notifying us of failure: 289 - 290 - ```json 291 - { "timestamp": { "seconds": 1424709442, "microseconds": 844524 }, 292 - "data": { "speed": 0, "offset": 0, "len": 67108864, 293 - "error": "No space left on device", 294 - "device": "drive1", "type": "backup" }, 295 - "event": "BLOCK_JOB_COMPLETED" } 296 - ``` 297 - 298 - 4. Delete the failed incremental, and re-create the image. 299 - 300 - ```sh 301 - # rm incremental.0.img 302 - # qemu-img create -f qcow2 incremental.0.img -b full_backup.img -F qcow2 303 - ``` 304 - 305 - 5. Retry the command after fixing the underlying problem, 306 - such as freeing up space on the backup volume: 307 - 308 - ```json 309 - { "execute": "drive-backup", 310 - "arguments": { 311 - "device": "drive0", 312 - "bitmap": "bitmap0", 313 - "target": "incremental.0.img", 314 - "format": "qcow2", 315 - "sync": "incremental", 316 - "mode": "existing" 317 - } 318 - } 319 - ``` 320 - 321 - 6. Receive confirmation that the job completed successfully: 322 - 323 - ```json 324 - { "timestamp": { "seconds": 1424709668, "microseconds": 526525 }, 325 - "data": { "device": "drive1", "type": "backup", 326 - "speed": 0, "len": 67108864, "offset": 67108864}, 327 - "event": "BLOCK_JOB_COMPLETED" } 328 - ``` 329 - 330 - ### Partial Transactional Failures 331 - 332 - * Sometimes, a transaction will succeed in launching and return success, 333 - but then later the backup jobs themselves may fail. It is possible that 334 - a management application may have to deal with a partial backup failure 335 - after a successful transaction. 336 - 337 - * If multiple backup jobs are specified in a single transaction, when one of 338 - them fails, it will not interact with the other backup jobs in any way. 339 - 340 - * The job(s) that succeeded will clear the dirty bitmap associated with the 341 - operation, but the job(s) that failed will not. It is not "safe" to delete 342 - any incremental backups that were created successfully in this scenario, 343 - even though others failed. 344 - 345 - #### Example 346 - 347 - * QMP example highlighting two backup jobs: 348 - 349 - ```json 350 - { "execute": "transaction", 351 - "arguments": { 352 - "actions": [ 353 - { "type": "drive-backup", 354 - "data": { "device": "drive0", "bitmap": "bitmap0", 355 - "format": "qcow2", "mode": "existing", 356 - "sync": "incremental", "target": "d0-incr-1.qcow2" } }, 357 - { "type": "drive-backup", 358 - "data": { "device": "drive1", "bitmap": "bitmap1", 359 - "format": "qcow2", "mode": "existing", 360 - "sync": "incremental", "target": "d1-incr-1.qcow2" } }, 361 - ] 362 - } 363 - } 364 - ``` 365 - 366 - * QMP example response, highlighting one success and one failure: 367 - * Acknowledgement that the Transaction was accepted and jobs were launched: 368 - ```json 369 - { "return": {} } 370 - ``` 371 - 372 - * Later, QEMU sends notice that the first job was completed: 373 - ```json 374 - { "timestamp": { "seconds": 1447192343, "microseconds": 615698 }, 375 - "data": { "device": "drive0", "type": "backup", 376 - "speed": 0, "len": 67108864, "offset": 67108864 }, 377 - "event": "BLOCK_JOB_COMPLETED" 378 - } 379 - ``` 380 - 381 - * Later yet, QEMU sends notice that the second job has failed: 382 - ```json 383 - { "timestamp": { "seconds": 1447192399, "microseconds": 683015 }, 384 - "data": { "device": "drive1", "action": "report", 385 - "operation": "read" }, 386 - "event": "BLOCK_JOB_ERROR" } 387 - ``` 388 - 389 - ```json 390 - { "timestamp": { "seconds": 1447192399, "microseconds": 685853 }, 391 - "data": { "speed": 0, "offset": 0, "len": 67108864, 392 - "error": "Input/output error", 393 - "device": "drive1", "type": "backup" }, 394 - "event": "BLOCK_JOB_COMPLETED" } 395 - 396 - * In the above example, "d0-incr-1.qcow2" is valid and must be kept, 397 - but "d1-incr-1.qcow2" is invalid and should be deleted. If a VM-wide 398 - incremental backup of all drives at a point-in-time is to be made, 399 - new backups for both drives will need to be made, taking into account 400 - that a new incremental backup for drive0 needs to be based on top of 401 - "d0-incr-1.qcow2." 402 - 403 - ### Grouped Completion Mode 404 - 405 - * While jobs launched by transactions normally complete or fail on their own, 406 - it is possible to instruct them to complete or fail together as a group. 407 - 408 - * QMP transactions take an optional properties structure that can affect 409 - the semantics of the transaction. 410 - 411 - * The "completion-mode" transaction property can be either "individual" 412 - which is the default, legacy behavior described above, or "grouped," 413 - a new behavior detailed below. 414 - 415 - * Delayed Completion: In grouped completion mode, no jobs will report 416 - success until all jobs are ready to report success. 417 - 418 - * Grouped failure: If any job fails in grouped completion mode, all remaining 419 - jobs will be cancelled. Any incremental backups will restore their dirty 420 - bitmap objects as if no backup command was ever issued. 421 - 422 - * Regardless of if QEMU reports a particular incremental backup job as 423 - CANCELLED or as an ERROR, the in-memory bitmap will be restored. 424 - 425 - #### Example 426 - 427 - * Here's the same example scenario from above with the new property: 428 - 429 - ```json 430 - { "execute": "transaction", 431 - "arguments": { 432 - "actions": [ 433 - { "type": "drive-backup", 434 - "data": { "device": "drive0", "bitmap": "bitmap0", 435 - "format": "qcow2", "mode": "existing", 436 - "sync": "incremental", "target": "d0-incr-1.qcow2" } }, 437 - { "type": "drive-backup", 438 - "data": { "device": "drive1", "bitmap": "bitmap1", 439 - "format": "qcow2", "mode": "existing", 440 - "sync": "incremental", "target": "d1-incr-1.qcow2" } }, 441 - ], 442 - "properties": { 443 - "completion-mode": "grouped" 444 - } 445 - } 446 - } 447 - ``` 448 - 449 - * QMP example response, highlighting a failure for drive2: 450 - * Acknowledgement that the Transaction was accepted and jobs were launched: 451 - ```json 452 - { "return": {} } 453 - ``` 454 - 455 - * Later, QEMU sends notice that the second job has errored out, 456 - but that the first job was also cancelled: 457 - ```json 458 - { "timestamp": { "seconds": 1447193702, "microseconds": 632377 }, 459 - "data": { "device": "drive1", "action": "report", 460 - "operation": "read" }, 461 - "event": "BLOCK_JOB_ERROR" } 462 - ``` 463 - 464 - ```json 465 - { "timestamp": { "seconds": 1447193702, "microseconds": 640074 }, 466 - "data": { "speed": 0, "offset": 0, "len": 67108864, 467 - "error": "Input/output error", 468 - "device": "drive1", "type": "backup" }, 469 - "event": "BLOCK_JOB_COMPLETED" } 470 - ``` 471 - 472 - ```json 473 - { "timestamp": { "seconds": 1447193702, "microseconds": 640163 }, 474 - "data": { "device": "drive0", "type": "backup", "speed": 0, 475 - "len": 67108864, "offset": 16777216 }, 476 - "event": "BLOCK_JOB_CANCELLED" } 477 - ``` 478 - 479 -

+555

docs/interop/bitmaps.rst

··· 1 + .. 2 + Copyright 2015 John Snow <jsnow@redhat.com> and Red Hat, Inc. 3 + All rights reserved. 4 + 5 + This file is licensed via The FreeBSD Documentation License, the full 6 + text of which is included at the end of this document. 7 + 8 + ==================================== 9 + Dirty Bitmaps and Incremental Backup 10 + ==================================== 11 + 12 + - Dirty Bitmaps are objects that track which data needs to be backed up 13 + for the next incremental backup. 14 + 15 + - Dirty bitmaps can be created at any time and attached to any node 16 + (not just complete drives). 17 + 18 + .. contents:: 19 + 20 + Dirty Bitmap Names 21 + ------------------ 22 + 23 + - A dirty bitmap's name is unique to the node, but bitmaps attached to 24 + different nodes can share the same name. 25 + 26 + - Dirty bitmaps created for internal use by QEMU may be anonymous and 27 + have no name, but any user-created bitmaps must have a name. There 28 + can be any number of anonymous bitmaps per node. 29 + 30 + - The name of a user-created bitmap must not be empty (""). 31 + 32 + Bitmap Modes 33 + ------------ 34 + 35 + - A bitmap can be "frozen," which means that it is currently in-use by 36 + a backup operation and cannot be deleted, renamed, written to, reset, 37 + etc. 38 + 39 + - The normal operating mode for a bitmap is "active." 40 + 41 + Basic QMP Usage 42 + --------------- 43 + 44 + Supported Commands 45 + ~~~~~~~~~~~~~~~~~~ 46 + 47 + - ``block-dirty-bitmap-add`` 48 + - ``block-dirty-bitmap-remove`` 49 + - ``block-dirty-bitmap-clear`` 50 + 51 + Creation 52 + ~~~~~~~~ 53 + 54 + - To create a new bitmap, enabled, on the drive with id=drive0: 55 + 56 + .. code:: json 57 + 58 + { "execute": "block-dirty-bitmap-add", 59 + "arguments": { 60 + "node": "drive0", 61 + "name": "bitmap0" 62 + } 63 + } 64 + 65 + - This bitmap will have a default granularity that matches the cluster 66 + size of its associated drive, if available, clamped to between [4KiB, 67 + 64KiB]. The current default for qcow2 is 64KiB. 68 + 69 + - To create a new bitmap that tracks changes in 32KiB segments: 70 + 71 + .. code:: json 72 + 73 + { "execute": "block-dirty-bitmap-add", 74 + "arguments": { 75 + "node": "drive0", 76 + "name": "bitmap0", 77 + "granularity": 32768 78 + } 79 + } 80 + 81 + Deletion 82 + ~~~~~~~~ 83 + 84 + - Bitmaps that are frozen cannot be deleted. 85 + 86 + - Deleting the bitmap does not impact any other bitmaps attached to the 87 + same node, nor does it affect any backups already created from this 88 + node. 89 + 90 + - Because bitmaps are only unique to the node to which they are 91 + attached, you must specify the node/drive name here, too. 92 + 93 + .. code:: json 94 + 95 + { "execute": "block-dirty-bitmap-remove", 96 + "arguments": { 97 + "node": "drive0", 98 + "name": "bitmap0" 99 + } 100 + } 101 + 102 + Resetting 103 + ~~~~~~~~~ 104 + 105 + - Resetting a bitmap will clear all information it holds. 106 + 107 + - An incremental backup created from an empty bitmap will copy no data, 108 + as if nothing has changed. 109 + 110 + .. code:: json 111 + 112 + { "execute": "block-dirty-bitmap-clear", 113 + "arguments": { 114 + "node": "drive0", 115 + "name": "bitmap0" 116 + } 117 + } 118 + 119 + Transactions 120 + ------------ 121 + 122 + Justification 123 + ~~~~~~~~~~~~~ 124 + 125 + Bitmaps can be safely modified when the VM is paused or halted by using 126 + the basic QMP commands. For instance, you might perform the following 127 + actions: 128 + 129 + 1. Boot the VM in a paused state. 130 + 2. Create a full drive backup of drive0. 131 + 3. Create a new bitmap attached to drive0. 132 + 4. Resume execution of the VM. 133 + 5. Incremental backups are ready to be created. 134 + 135 + At this point, the bitmap and drive backup would be correctly in sync, 136 + and incremental backups made from this point forward would be correctly 137 + aligned to the full drive backup. 138 + 139 + This is not particularly useful if we decide we want to start 140 + incremental backups after the VM has been running for a while, for which 141 + we will need to perform actions such as the following: 142 + 143 + 1. Boot the VM and begin execution. 144 + 2. Using a single transaction, perform the following operations: 145 + 146 + - Create ``bitmap0``. 147 + - Create a full drive backup of ``drive0``. 148 + 149 + 3. Incremental backups are now ready to be created. 150 + 151 + Supported Bitmap Transactions 152 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 153 + 154 + - ``block-dirty-bitmap-add`` 155 + - ``block-dirty-bitmap-clear`` 156 + 157 + The usages are identical to their respective QMP commands, but see below 158 + for examples. 159 + 160 + Example: New Incremental Backup 161 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 162 + 163 + As outlined in the justification, perhaps we want to create a new 164 + incremental backup chain attached to a drive. 165 + 166 + .. code:: json 167 + 168 + { "execute": "transaction", 169 + "arguments": { 170 + "actions": [ 171 + {"type": "block-dirty-bitmap-add", 172 + "data": {"node": "drive0", "name": "bitmap0"} }, 173 + {"type": "drive-backup", 174 + "data": {"device": "drive0", "target": "/path/to/full_backup.img", 175 + "sync": "full", "format": "qcow2"} } 176 + ] 177 + } 178 + } 179 + 180 + Example: New Incremental Backup Anchor Point 181 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 182 + 183 + Maybe we just want to create a new full backup with an existing bitmap 184 + and want to reset the bitmap to track the new chain. 185 + 186 + .. code:: json 187 + 188 + { "execute": "transaction", 189 + "arguments": { 190 + "actions": [ 191 + {"type": "block-dirty-bitmap-clear", 192 + "data": {"node": "drive0", "name": "bitmap0"} }, 193 + {"type": "drive-backup", 194 + "data": {"device": "drive0", "target": "/path/to/new_full_backup.img", 195 + "sync": "full", "format": "qcow2"} } 196 + ] 197 + } 198 + } 199 + 200 + Incremental Backups 201 + ------------------- 202 + 203 + The star of the show. 204 + 205 + **Nota Bene!** Only incremental backups of entire drives are supported 206 + for now. So despite the fact that you can attach a bitmap to any 207 + arbitrary node, they are only currently useful when attached to the root 208 + node. This is because drive-backup only supports drives/devices instead 209 + of arbitrary nodes. 210 + 211 + Example: First Incremental Backup 212 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 213 + 214 + 1. Create a full backup and sync it to the dirty bitmap, as in the 215 + transactional examples above; or with the VM offline, manually create 216 + a full copy and then create a new bitmap before the VM begins 217 + execution. 218 + 219 + - Let's assume the full backup is named ``full_backup.img``. 220 + - Let's assume the bitmap you created is ``bitmap0`` attached to 221 + ``drive0``. 222 + 223 + 2. Create a destination image for the incremental backup that utilizes 224 + the full backup as a backing image. 225 + 226 + - Let's assume the new incremental image is named 227 + ``incremental.0.img``. 228 + 229 + .. code:: bash 230 + 231 + $ qemu-img create -f qcow2 incremental.0.img -b full_backup.img -F qcow2 232 + 233 + 3. Issue the incremental backup command: 234 + 235 + .. code:: json 236 + 237 + { "execute": "drive-backup", 238 + "arguments": { 239 + "device": "drive0", 240 + "bitmap": "bitmap0", 241 + "target": "incremental.0.img", 242 + "format": "qcow2", 243 + "sync": "incremental", 244 + "mode": "existing" 245 + } 246 + } 247 + 248 + Example: Second Incremental Backup 249 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 250 + 251 + 1. Create a new destination image for the incremental backup that points 252 + to the previous one, e.g.: ``incremental.1.img`` 253 + 254 + .. code:: bash 255 + 256 + $ qemu-img create -f qcow2 incremental.1.img -b incremental.0.img -F qcow2 257 + 258 + 2. Issue a new incremental backup command. The only difference here is 259 + that we have changed the target image below. 260 + 261 + .. code:: json 262 + 263 + { "execute": "drive-backup", 264 + "arguments": { 265 + "device": "drive0", 266 + "bitmap": "bitmap0", 267 + "target": "incremental.1.img", 268 + "format": "qcow2", 269 + "sync": "incremental", 270 + "mode": "existing" 271 + } 272 + } 273 + 274 + Errors 275 + ------ 276 + 277 + - In the event of an error that occurs after a backup job is 278 + successfully launched, either by a direct QMP command or a QMP 279 + transaction, the user will receive a ``BLOCK_JOB_COMPLETE`` event with 280 + a failure message, accompanied by a ``BLOCK_JOB_ERROR`` event. 281 + 282 + - In the case of an event being cancelled, the user will receive a 283 + ``BLOCK_JOB_CANCELLED`` event instead of a pair of COMPLETE and ERROR 284 + events. 285 + 286 + - In either case, the incremental backup data contained within the 287 + bitmap is safely rolled back, and the data within the bitmap is not 288 + lost. The image file created for the failed attempt can be safely 289 + deleted. 290 + 291 + - Once the underlying problem is fixed (e.g. more storage space is 292 + freed up), you can simply retry the incremental backup command with 293 + the same bitmap. 294 + 295 + Example 296 + ~~~~~~~ 297 + 298 + 1. Create a target image: 299 + 300 + .. code:: bash 301 + 302 + $ qemu-img create -f qcow2 incremental.0.img -b full_backup.img -F qcow2 303 + 304 + 2. Attempt to create an incremental backup via QMP: 305 + 306 + .. code:: json 307 + 308 + { "execute": "drive-backup", 309 + "arguments": { 310 + "device": "drive0", 311 + "bitmap": "bitmap0", 312 + "target": "incremental.0.img", 313 + "format": "qcow2", 314 + "sync": "incremental", 315 + "mode": "existing" 316 + } 317 + } 318 + 319 + 3. Receive an event notifying us of failure: 320 + 321 + .. code:: json 322 + 323 + { "timestamp": { "seconds": 1424709442, "microseconds": 844524 }, 324 + "data": { "speed": 0, "offset": 0, "len": 67108864, 325 + "error": "No space left on device", 326 + "device": "drive1", "type": "backup" }, 327 + "event": "BLOCK_JOB_COMPLETED" } 328 + 329 + 4. Delete the failed incremental, and re-create the image. 330 + 331 + .. code:: bash 332 + 333 + $ rm incremental.0.img 334 + $ qemu-img create -f qcow2 incremental.0.img -b full_backup.img -F qcow2 335 + 336 + 5. Retry the command after fixing the underlying problem, such as 337 + freeing up space on the backup volume: 338 + 339 + .. code:: json 340 + 341 + { "execute": "drive-backup", 342 + "arguments": { 343 + "device": "drive0", 344 + "bitmap": "bitmap0", 345 + "target": "incremental.0.img", 346 + "format": "qcow2", 347 + "sync": "incremental", 348 + "mode": "existing" 349 + } 350 + } 351 + 352 + 6. Receive confirmation that the job completed successfully: 353 + 354 + .. code:: json 355 + 356 + { "timestamp": { "seconds": 1424709668, "microseconds": 526525 }, 357 + "data": { "device": "drive1", "type": "backup", 358 + "speed": 0, "len": 67108864, "offset": 67108864}, 359 + "event": "BLOCK_JOB_COMPLETED" } 360 + 361 + Partial Transactional Failures 362 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 363 + 364 + - Sometimes, a transaction will succeed in launching and return 365 + success, but then later the backup jobs themselves may fail. It is 366 + possible that a management application may have to deal with a 367 + partial backup failure after a successful transaction. 368 + 369 + - If multiple backup jobs are specified in a single transaction, when 370 + one of them fails, it will not interact with the other backup jobs in 371 + any way. 372 + 373 + - The job(s) that succeeded will clear the dirty bitmap associated with 374 + the operation, but the job(s) that failed will not. It is not "safe" 375 + to delete any incremental backups that were created successfully in 376 + this scenario, even though others failed. 377 + 378 + Example 379 + ^^^^^^^ 380 + 381 + - QMP example highlighting two backup jobs: 382 + 383 + .. code:: json 384 + 385 + { "execute": "transaction", 386 + "arguments": { 387 + "actions": [ 388 + { "type": "drive-backup", 389 + "data": { "device": "drive0", "bitmap": "bitmap0", 390 + "format": "qcow2", "mode": "existing", 391 + "sync": "incremental", "target": "d0-incr-1.qcow2" } }, 392 + { "type": "drive-backup", 393 + "data": { "device": "drive1", "bitmap": "bitmap1", 394 + "format": "qcow2", "mode": "existing", 395 + "sync": "incremental", "target": "d1-incr-1.qcow2" } }, 396 + ] 397 + } 398 + } 399 + 400 + - QMP example response, highlighting one success and one failure: 401 + 402 + - Acknowledgement that the Transaction was accepted and jobs were 403 + launched: 404 + 405 + .. code:: json 406 + 407 + { "return": {} } 408 + 409 + - Later, QEMU sends notice that the first job was completed: 410 + 411 + .. code:: json 412 + 413 + { "timestamp": { "seconds": 1447192343, "microseconds": 615698 }, 414 + "data": { "device": "drive0", "type": "backup", 415 + "speed": 0, "len": 67108864, "offset": 67108864 }, 416 + "event": "BLOCK_JOB_COMPLETED" 417 + } 418 + 419 + - Later yet, QEMU sends notice that the second job has failed: 420 + 421 + .. code:: json 422 + 423 + { "timestamp": { "seconds": 1447192399, "microseconds": 683015 }, 424 + "data": { "device": "drive1", "action": "report", 425 + "operation": "read" }, 426 + "event": "BLOCK_JOB_ERROR" } 427 + 428 + .. code:: json 429 + 430 + { "timestamp": { "seconds": 1447192399, "microseconds": 431 + 685853 }, "data": { "speed": 0, "offset": 0, "len": 67108864, 432 + "error": "Input/output error", "device": "drive1", "type": 433 + "backup" }, "event": "BLOCK_JOB_COMPLETED" } 434 + 435 + - In the above example, ``d0-incr-1.qcow2`` is valid and must be kept, 436 + but ``d1-incr-1.qcow2`` is invalid and should be deleted. If a VM-wide 437 + incremental backup of all drives at a point-in-time is to be made, 438 + new backups for both drives will need to be made, taking into account 439 + that a new incremental backup for drive0 needs to be based on top of 440 + ``d0-incr-1.qcow2``. 441 + 442 + Grouped Completion Mode 443 + ~~~~~~~~~~~~~~~~~~~~~~~ 444 + 445 + - While jobs launched by transactions normally complete or fail on 446 + their own, it is possible to instruct them to complete or fail 447 + together as a group. 448 + 449 + - QMP transactions take an optional properties structure that can 450 + affect the semantics of the transaction. 451 + 452 + - The "completion-mode" transaction property can be either "individual" 453 + which is the default, legacy behavior described above, or "grouped," 454 + a new behavior detailed below. 455 + 456 + - Delayed Completion: In grouped completion mode, no jobs will report 457 + success until all jobs are ready to report success. 458 + 459 + - Grouped failure: If any job fails in grouped completion mode, all 460 + remaining jobs will be cancelled. Any incremental backups will 461 + restore their dirty bitmap objects as if no backup command was ever 462 + issued. 463 + 464 + - Regardless of if QEMU reports a particular incremental backup job 465 + as CANCELLED or as an ERROR, the in-memory bitmap will be 466 + restored. 467 + 468 + Example 469 + ^^^^^^^ 470 + 471 + - Here's the same example scenario from above with the new property: 472 + 473 + .. code:: json 474 + 475 + { "execute": "transaction", 476 + "arguments": { 477 + "actions": [ 478 + { "type": "drive-backup", 479 + "data": { "device": "drive0", "bitmap": "bitmap0", 480 + "format": "qcow2", "mode": "existing", 481 + "sync": "incremental", "target": "d0-incr-1.qcow2" } }, 482 + { "type": "drive-backup", 483 + "data": { "device": "drive1", "bitmap": "bitmap1", 484 + "format": "qcow2", "mode": "existing", 485 + "sync": "incremental", "target": "d1-incr-1.qcow2" } }, 486 + ], 487 + "properties": { 488 + "completion-mode": "grouped" 489 + } 490 + } 491 + } 492 + 493 + - QMP example response, highlighting a failure for ``drive2``: 494 + 495 + - Acknowledgement that the Transaction was accepted and jobs were 496 + launched: 497 + 498 + .. code:: json 499 + 500 + { "return": {} } 501 + 502 + - Later, QEMU sends notice that the second job has errored out, but 503 + that the first job was also cancelled: 504 + 505 + .. code:: json 506 + 507 + { "timestamp": { "seconds": 1447193702, "microseconds": 632377 }, 508 + "data": { "device": "drive1", "action": "report", 509 + "operation": "read" }, 510 + "event": "BLOCK_JOB_ERROR" } 511 + 512 + .. code:: json 513 + 514 + { "timestamp": { "seconds": 1447193702, "microseconds": 640074 }, 515 + "data": { "speed": 0, "offset": 0, "len": 67108864, 516 + "error": "Input/output error", 517 + "device": "drive1", "type": "backup" }, 518 + "event": "BLOCK_JOB_COMPLETED" } 519 + 520 + .. code:: json 521 + 522 + { "timestamp": { "seconds": 1447193702, "microseconds": 640163 }, 523 + "data": { "device": "drive0", "type": "backup", "speed": 0, 524 + "len": 67108864, "offset": 16777216 }, 525 + "event": "BLOCK_JOB_CANCELLED" } 526 + 527 + .. raw:: html 528 + 529 +