qemu with hax to log dma reads & writes jcs.org/2018/11/12/vfio

qapi: Implement boxed types for commands/events

Turn on the ability to pass command and event arguments in
a single boxed parameter, which must name a non-empty type
(although the type can be a struct with all optional members).
For structs, it makes it possible to pass a single qapi type
instead of a breakout of all struct members (useful if the
arguments are already in a struct or if the number of members
is large); for other complex types, it is now possible to use
a union or alternate as the data for a command or event.

The empty type may be technically feasible if needed down the
road, but it's easier to forbid it now and relax things to allow
it later, than it is to allow it now and have to special case
how the generated 'q_empty' type is handled (see commit 7ce106a9
for reasons why nothing is generated for the empty type). An
alternate type is never considered empty, but now that a boxed
type can be either an object or an alternate, we have to provide
a trivial QAPISchemaAlternateType.is_empty(). The new call to
arg_type.is_empty() during QAPISchemaCommand.check() requires
that we first check the type in question; but there is no chance
of introducing a cycle since objects do not refer back to commands.

We still have a split in syntax checking between ad-hoc parsing
up front (merely validates that 'boxed' has a sane value) and
during .check() methods (if 'boxed' is set, then 'data' must name
a non-empty user-defined type).

Generated code is unchanged, as long as no client uses the
new feature.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <1468468228-27827-10-git-send-email-eblake@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Test files renamed to *-boxed-*]
Signed-off-by: Markus Armbruster <armbru@redhat.com>

authored by

Eric Blake and committed by
Markus Armbruster
c818408e 48825ca4

+129 -20
+25 -2
docs/qapi-code-gen.txt
··· 410 410 === Commands === 411 411 412 412 Usage: { 'command': STRING, '*data': COMPLEX-TYPE-NAME-OR-DICT, 413 - '*returns': TYPE-NAME, 413 + '*returns': TYPE-NAME, '*boxed': true, 414 414 '*gen': false, '*success-response': false } 415 415 416 416 Commands are defined by using a dictionary containing several members, ··· 461 461 => { "execute": "my-second-command" } 462 462 <= { "return": [ { "value": "one" }, { } ] } 463 463 464 + The generator emits a prototype for the user's function implementing 465 + the command. Normally, 'data' is a dictionary for an anonymous type, 466 + or names a struct type (possibly empty, but not a union), and its 467 + members are passed as separate arguments to this function. If the 468 + command definition includes a key 'boxed' with the boolean value true, 469 + then 'data' is instead the name of any non-empty complex type 470 + (struct, union, or alternate), and a pointer to that QAPI type is 471 + passed as a single argument. 472 + 473 + The generator also emits a marshalling function that extracts 474 + arguments for the user's function out of an input QDict, calls the 475 + user's function, and if it succeeded, builds an output QObject from 476 + its return value. 477 + 464 478 In rare cases, QAPI cannot express a type-safe representation of a 465 479 corresponding Client JSON Protocol command. You then have to suppress 466 480 generation of a marshalling function by including a key 'gen' with ··· 484 498 485 499 === Events === 486 500 487 - Usage: { 'event': STRING, '*data': COMPLEX-TYPE-NAME-OR-DICT } 501 + Usage: { 'event': STRING, '*data': COMPLEX-TYPE-NAME-OR-DICT, 502 + '*boxed': true } 488 503 489 504 Events are defined with the keyword 'event'. It is not allowed to 490 505 name an event 'MAX', since the generator also produces a C enumeration ··· 504 519 { "event": "EVENT_C", 505 520 "data": { "b": "test string" }, 506 521 "timestamp": { "seconds": 1267020223, "microseconds": 435656 } } 522 + 523 + The generator emits a function to send the event. Normally, 'data' is 524 + a dictionary for an anonymous type, or names a struct type (possibly 525 + empty, but not a union), and its members are passed as separate 526 + arguments to this function. If the event definition includes a key 527 + 'boxed' with the boolean value true, then 'data' is instead the name of 528 + any non-empty complex type (struct, union, or alternate), and a 529 + pointer to that QAPI type is passed as a single argument. 507 530 508 531 509 532 == Client JSON Protocol introspection ==
+2 -1
scripts/qapi-commands.py
··· 30 30 31 31 argstr = '' 32 32 if boxed: 33 - assert False # not implemented 33 + assert arg_type and not arg_type.is_empty() 34 + argstr = '&arg, ' 34 35 elif arg_type: 35 36 assert not arg_type.variants 36 37 for memb in arg_type.members:
+4 -1
scripts/qapi-event.py
··· 79 79 QObject *obj; 80 80 Visitor *v; 81 81 ''') 82 - ret += gen_param_var(arg_type) 82 + if not boxed: 83 + ret += gen_param_var(arg_type) 84 + else: 85 + assert not boxed 83 86 84 87 ret += mcgen(''' 85 88
+50 -13
scripts/qapi.py
··· 522 522 523 523 def check_command(expr, expr_info): 524 524 name = expr['command'] 525 + boxed = expr.get('boxed', False) 525 526 527 + args_meta = ['struct'] 528 + if boxed: 529 + args_meta += ['union', 'alternate'] 526 530 check_type(expr_info, "'data' for command '%s'" % name, 527 - expr.get('data'), allow_dict=True, allow_optional=True, 528 - allow_metas=['struct']) 531 + expr.get('data'), allow_dict=not boxed, allow_optional=True, 532 + allow_metas=args_meta) 529 533 returns_meta = ['union', 'struct'] 530 534 if name in returns_whitelist: 531 535 returns_meta += ['built-in', 'alternate', 'enum'] ··· 537 541 def check_event(expr, expr_info): 538 542 global events 539 543 name = expr['event'] 544 + boxed = expr.get('boxed', False) 540 545 546 + meta = ['struct'] 547 + if boxed: 548 + meta += ['union', 'alternate'] 541 549 events.append(name) 542 550 check_type(expr_info, "'data' for event '%s'" % name, 543 - expr.get('data'), allow_dict=True, allow_optional=True, 544 - allow_metas=['struct']) 551 + expr.get('data'), allow_dict=not boxed, allow_optional=True, 552 + allow_metas=meta) 545 553 546 554 547 555 def check_union(expr, expr_info): ··· 694 702 raise QAPIExprError(info, 695 703 "'%s' of %s '%s' should only use false value" 696 704 % (key, meta, name)) 705 + if key == 'boxed' and value is not True: 706 + raise QAPIExprError(info, 707 + "'%s' of %s '%s' should only use true value" 708 + % (key, meta, name)) 697 709 for key in required: 698 710 if key not in expr: 699 711 raise QAPIExprError(info, ··· 725 737 add_struct(expr, info) 726 738 elif 'command' in expr: 727 739 check_keys(expr_elem, 'command', [], 728 - ['data', 'returns', 'gen', 'success-response']) 740 + ['data', 'returns', 'gen', 'success-response', 'boxed']) 729 741 add_name(expr['command'], info, 'command') 730 742 elif 'event' in expr: 731 - check_keys(expr_elem, 'event', [], ['data']) 743 + check_keys(expr_elem, 'event', [], ['data', 'boxed']) 732 744 add_name(expr['event'], info, 'event') 733 745 else: 734 746 raise QAPIExprError(expr_elem['info'], ··· 1163 1175 def visit(self, visitor): 1164 1176 visitor.visit_alternate_type(self.name, self.info, self.variants) 1165 1177 1178 + def is_empty(self): 1179 + return False 1180 + 1166 1181 1167 1182 class QAPISchemaCommand(QAPISchemaEntity): 1168 1183 def __init__(self, name, info, arg_type, ret_type, gen, success_response, ··· 1181 1196 def check(self, schema): 1182 1197 if self._arg_type_name: 1183 1198 self.arg_type = schema.lookup_type(self._arg_type_name) 1184 - assert isinstance(self.arg_type, QAPISchemaObjectType) 1185 - assert not self.arg_type.variants # not implemented 1186 - assert not self.boxed # not implemented 1199 + assert (isinstance(self.arg_type, QAPISchemaObjectType) or 1200 + isinstance(self.arg_type, QAPISchemaAlternateType)) 1201 + self.arg_type.check(schema) 1202 + if self.boxed: 1203 + if self.arg_type.is_empty(): 1204 + raise QAPIExprError(self.info, 1205 + "Cannot use 'boxed' with empty type") 1206 + else: 1207 + assert not isinstance(self.arg_type, QAPISchemaAlternateType) 1208 + assert not self.arg_type.variants 1209 + elif self.boxed: 1210 + raise QAPIExprError(self.info, 1211 + "Use of 'boxed' requires 'data'") 1187 1212 if self._ret_type_name: 1188 1213 self.ret_type = schema.lookup_type(self._ret_type_name) 1189 1214 assert isinstance(self.ret_type, QAPISchemaType) ··· 1205 1230 def check(self, schema): 1206 1231 if self._arg_type_name: 1207 1232 self.arg_type = schema.lookup_type(self._arg_type_name) 1208 - assert isinstance(self.arg_type, QAPISchemaObjectType) 1209 - assert not self.arg_type.variants # not implemented 1210 - assert not self.boxed # not implemented 1233 + assert (isinstance(self.arg_type, QAPISchemaObjectType) or 1234 + isinstance(self.arg_type, QAPISchemaAlternateType)) 1235 + self.arg_type.check(schema) 1236 + if self.boxed: 1237 + if self.arg_type.is_empty(): 1238 + raise QAPIExprError(self.info, 1239 + "Cannot use 'boxed' with empty type") 1240 + else: 1241 + assert not isinstance(self.arg_type, QAPISchemaAlternateType) 1242 + assert not self.arg_type.variants 1243 + elif self.boxed: 1244 + raise QAPIExprError(self.info, 1245 + "Use of 'boxed' requires 'data'") 1211 1246 1212 1247 def visit(self, visitor): 1213 1248 visitor.visit_event(self.name, self.info, self.arg_type, self.boxed) ··· 1648 1683 1649 1684 def gen_params(arg_type, boxed, extra): 1650 1685 if not arg_type: 1686 + assert not boxed 1651 1687 return extra 1652 1688 ret = '' 1653 1689 sep = '' 1654 1690 if boxed: 1655 - assert False # not implemented 1691 + ret += '%s arg' % arg_type.c_param_type() 1692 + sep = ', ' 1656 1693 else: 1657 1694 assert not arg_type.variants 1658 1695 for memb in arg_type.members:
+5
tests/Makefile.include
··· 284 284 qapi-schema += args-any.json 285 285 qapi-schema += args-array-empty.json 286 286 qapi-schema += args-array-unknown.json 287 + qapi-schema += args-bad-boxed.json 288 + qapi-schema += args-boxed-anon.json 289 + qapi-schema += args-boxed-empty.json 290 + qapi-schema += args-boxed-string.json 287 291 qapi-schema += args-int.json 288 292 qapi-schema += args-invalid.json 289 293 qapi-schema += args-member-array-bad.json ··· 317 321 qapi-schema += escape-outside-string.json 318 322 qapi-schema += escape-too-big.json 319 323 qapi-schema += escape-too-short.json 324 + qapi-schema += event-boxed-empty.json 320 325 qapi-schema += event-case.json 321 326 qapi-schema += event-nest-struct.json 322 327 qapi-schema += flat-union-array-branch.json
+1
tests/qapi-schema/args-bad-boxed.err
··· 1 + tests/qapi-schema/args-bad-boxed.json:2: 'boxed' of command 'foo' should only use true value
+1
tests/qapi-schema/args-bad-boxed.exit
··· 1 + 1
+2
tests/qapi-schema/args-bad-boxed.json
··· 1 + # 'boxed' should only appear with value true 2 + { 'command': 'foo', 'boxed': false }
tests/qapi-schema/args-bad-boxed.out

This is a binary file and will not be displayed.

+1
tests/qapi-schema/args-boxed-anon.err
··· 1 + tests/qapi-schema/args-boxed-anon.json:2: 'data' for command 'foo' should be a type name
+1
tests/qapi-schema/args-boxed-anon.exit
··· 1 + 1
+2
tests/qapi-schema/args-boxed-anon.json
··· 1 + # 'boxed' can only be used with named types 2 + { 'command': 'foo', 'boxed': true, 'data': { 'string': 'str' } }
tests/qapi-schema/args-boxed-anon.out

This is a binary file and will not be displayed.

+1
tests/qapi-schema/args-boxed-empty.err
··· 1 + tests/qapi-schema/args-boxed-empty.json:3: Cannot use 'boxed' with empty type
+1
tests/qapi-schema/args-boxed-empty.exit
··· 1 + 1
+3
tests/qapi-schema/args-boxed-empty.json
··· 1 + # 'boxed' requires a non-empty type 2 + { 'struct': 'Empty', 'data': {} } 3 + { 'command': 'foo', 'boxed': true, 'data': 'Empty' }
tests/qapi-schema/args-boxed-empty.out

This is a binary file and will not be displayed.

+1
tests/qapi-schema/args-boxed-string.err
··· 1 + tests/qapi-schema/args-boxed-string.json:2: 'data' for command 'foo' cannot use built-in type 'str'
+1
tests/qapi-schema/args-boxed-string.exit
··· 1 + 1
+2
tests/qapi-schema/args-boxed-string.json
··· 1 + # 'boxed' requires a complex (not built-in) type 2 + { 'command': 'foo', 'boxed': true, 'data': 'str' }
tests/qapi-schema/args-boxed-string.out

This is a binary file and will not be displayed.

+1 -1
tests/qapi-schema/args-union.err
··· 1 - tests/qapi-schema/args-union.json:4: 'data' for command 'oops' cannot use union type 'Uni' 1 + tests/qapi-schema/args-union.json:3: 'data' for command 'oops' cannot use union type 'Uni'
+1 -2
tests/qapi-schema/args-union.json
··· 1 - # we do not allow union arguments 2 - # TODO should we support this? 1 + # use of union arguments requires 'boxed':true 3 2 { 'union': 'Uni', 'data': { 'case1': 'int', 'case2': 'str' } } 4 3 { 'command': 'oops', 'data': 'Uni' }
+1
tests/qapi-schema/event-boxed-empty.err
··· 1 + tests/qapi-schema/event-boxed-empty.json:2: Use of 'boxed' requires 'data'
+1
tests/qapi-schema/event-boxed-empty.exit
··· 1 + 1
+2
tests/qapi-schema/event-boxed-empty.json
··· 1 + # 'boxed' requires a non-empty type 2 + { 'event': 'FOO', 'boxed': true }
tests/qapi-schema/event-boxed-empty.out

This is a binary file and will not be displayed.

+4
tests/qapi-schema/qapi-schema-test.json
··· 127 127 { 'command': 'guest-get-time', 'data': {'a': 'int', '*b': 'int' }, 128 128 'returns': 'int' } 129 129 { 'command': 'guest-sync', 'data': { 'arg': 'any' }, 'returns': 'any' } 130 + { 'command': 'boxed-struct', 'boxed': true, 'data': 'UserDefZero' } 131 + { 'command': 'boxed-union', 'data': 'UserDefNativeListUnion', 'boxed': true } 130 132 131 133 # For testing integer range flattening in opts-visitor. The following schema 132 134 # corresponds to the option format: ··· 154 156 'data': { '*a': 'int', '*b': 'UserDefOne', 'c': 'str' } } 155 157 { 'event': 'EVENT_D', 156 158 'data': { 'a' : 'EventStructOne', 'b' : 'str', '*c': 'str', '*enum3': 'EnumOne' } } 159 + { 'event': 'EVENT_E', 'boxed': true, 'data': 'UserDefZero' } 160 + { 'event': 'EVENT_F', 'boxed': true, 'data': 'UserDefAlternate' } 157 161 158 162 # test that we correctly compile downstream extensions, as well as munge 159 163 # ticklish names
+8
tests/qapi-schema/qapi-schema-test.out
··· 30 30 boxed=False 31 31 event EVENT_D q_obj_EVENT_D-arg 32 32 boxed=False 33 + event EVENT_E UserDefZero 34 + boxed=True 35 + event EVENT_F UserDefAlternate 36 + boxed=True 33 37 object Empty1 34 38 object Empty2 35 39 base Empty1 ··· 153 157 case __org.qemu_x-value: __org.qemu_x-Struct2 154 158 command __org.qemu_x-command q_obj___org.qemu_x-command-arg -> __org.qemu_x-Union1 155 159 gen=True success_response=True boxed=False 160 + command boxed-struct UserDefZero -> None 161 + gen=True success_response=True boxed=True 162 + command boxed-union UserDefNativeListUnion -> None 163 + gen=True success_response=True boxed=True 156 164 command guest-get-time q_obj_guest-get-time-arg -> int 157 165 gen=True success_response=True boxed=False 158 166 command guest-sync q_obj_guest-sync-arg -> any
+8
tests/test-qmp-commands.c
··· 59 59 return arg; 60 60 } 61 61 62 + void qmp_boxed_struct(UserDefZero *arg, Error **errp) 63 + { 64 + } 65 + 66 + void qmp_boxed_union(UserDefNativeListUnion *arg, Error **errp) 67 + { 68 + } 69 + 62 70 __org_qemu_x_Union1 *qmp___org_qemu_x_command(__org_qemu_x_EnumList *a, 63 71 __org_qemu_x_StructList *b, 64 72 __org_qemu_x_Union2 *c,