qemu with hax to log dma reads & writes jcs.org/2018/11/12/vfio

qapi: Make doc comments optional where we don't need them

Since we added the documentation generator in commit 3313b61, doc
comments are mandatory. That's a very good idea for a schema that
needs to be documented, but has proven to be annoying for testing.

Make doc comments optional again, but add a new directive

{ 'pragma': { 'doc-required': true } }

to let a QAPI schema require them.

Add test cases for the new pragma directive. While there, plug a
minor hole in includ directive test coverage.

Require documentation in the schemas we actually want documented:
qapi-schema.json and qga/qapi-schema.json.

We could probably make qapi2texi.py cope with incomplete
documentation, but for now, simply make it refuse to run unless the
schema has 'doc-required': true.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <1489582656-31133-3-git-send-email-armbru@redhat.com>
[qapi-code-gen.txt wording tweaked]
Reviewed-by: Eric Blake <eblake@redhat.com>

+92 -14
+30 -13
docs/qapi-code-gen.txt
··· 117 117 118 118 ==== Expression documentation ==== 119 119 120 - Each expression that isn't an include directive must be preceded by a 120 + Each expression that isn't an include directive may be preceded by a 121 121 documentation block. Such blocks are called expression documentation 122 122 blocks. 123 + 124 + When documentation is required (see pragma 'doc-required'), expression 125 + documentation blocks are mandatory. 123 126 124 127 The documentation block consists of a first line naming the 125 128 expression, an optional overview, a description of each argument (for ··· 204 207 not used by any commands or events in the Client JSON Protocol, for 205 208 the side effect of generated C code used internally. 206 209 207 - There are seven top-level expressions recognized by the parser: 208 - 'include', 'command', 'struct', 'enum', 'union', 'alternate', and 209 - 'event'. There are several groups of types: simple types (a number of 210 - built-in types, such as 'int' and 'str'; as well as enumerations), 211 - complex types (structs and two flavors of unions), and alternate types 212 - (a choice between other types). The 'command' and 'event' expressions 213 - can refer to existing types by name, or list an anonymous type as a 214 - dictionary. Listing a type name inside an array refers to a 215 - single-dimension array of that type; multi-dimension arrays are not 216 - directly supported (although an array of a complex struct that 217 - contains an array member is possible). 210 + There are eight top-level expressions recognized by the parser: 211 + 'include', 'pragma', 'command', 'struct', 'enum', 'union', 212 + 'alternate', and 'event'. There are several groups of types: simple 213 + types (a number of built-in types, such as 'int' and 'str'; as well as 214 + enumerations), complex types (structs and two flavors of unions), and 215 + alternate types (a choice between other types). The 'command' and 216 + 'event' expressions can refer to existing types by name, or list an 217 + anonymous type as a dictionary. Listing a type name inside an array 218 + refers to a single-dimension array of that type; multi-dimension 219 + arrays are not directly supported (although an array of a complex 220 + struct that contains an array member is possible). 218 221 219 222 All names must begin with a letter, and contain only ASCII letters, 220 223 digits, hyphen, and underscore. There are two exceptions: enum values ··· 282 285 QType QType JSON string matching enum QType values 283 286 284 287 285 - === Includes === 288 + === Include directives === 286 289 287 290 Usage: { 'include': STRING } 288 291 ··· 300 303 from making a forward reference to a type that is only introduced by 301 304 an outer file. The parser may be made stricter in the future to 302 305 prevent incomplete include files. 306 + 307 + 308 + === Pragma directives === 309 + 310 + Usage: { 'pragma': DICT } 311 + 312 + The pragma directive lets you control optional generator behavior. 313 + The dictionary's entries are pragma names and values. 314 + 315 + Pragma's scope is currently the complete schema. Setting the same 316 + pragma to different values in parts of the schema doesn't work. 317 + 318 + Pragma 'doc-required' takes a boolean value. If true, documentation 319 + is required. Default is false. 303 320 304 321 305 322 === Struct types ===
+2
qapi-schema.json
··· 49 49 # 50 50 ## 51 51 52 + { 'pragma': { 'doc-required': true } } 53 + 52 54 # QAPI common definitions 53 55 { 'include': 'qapi/common.json' } 54 56
+2
qga/qapi-schema.json
··· 11 11 # 12 12 ## 13 13 14 + { 'pragma': { 'doc-required': true } } 15 + 14 16 ## 15 17 # @guest-sync-delimited: 16 18 #
+23 -1
scripts/qapi.py
··· 37 37 'QType': 'QTYPE_QSTRING', 38 38 } 39 39 40 + # Are documentation comments required? 41 + doc_required = False 42 + 40 43 # Whitelist of commands allowed to return a non-dictionary 41 44 returns_whitelist = [ 42 45 # From QMP: ··· 277 280 "Value of 'include' must be a string") 278 281 self._include(include, info, os.path.dirname(abs_fname), 279 282 previously_included) 283 + elif "pragma" in expr: 284 + if len(expr) != 1: 285 + raise QAPISemError(info, "Invalid 'pragma' directive") 286 + pragma = expr['pragma'] 287 + if not isinstance(pragma, dict): 288 + raise QAPISemError( 289 + info, "Value of 'pragma' must be a dictionary") 290 + for name, value in pragma.iteritems(): 291 + self._pragma(name, value, info) 280 292 else: 281 293 expr_elem = {'expr': expr, 282 294 'info': info} ··· 307 319 exprs_include = QAPISchemaParser(fobj, previously_included, info) 308 320 self.exprs.extend(exprs_include.exprs) 309 321 self.docs.extend(exprs_include.docs) 322 + 323 + def _pragma(self, name, value, info): 324 + global doc_required 325 + if name == 'doc-required': 326 + if not isinstance(value, bool): 327 + raise QAPISemError(info, 328 + "Pragma 'doc-required' must be boolean") 329 + doc_required = value 330 + else: 331 + raise QAPISemError(info, "Unknown pragma '%s'" % name) 310 332 311 333 def accept(self, skip_comment=True): 312 334 while True: ··· 863 885 expr = expr_elem['expr'] 864 886 info = expr_elem['info'] 865 887 866 - if 'doc' not in expr_elem: 888 + if 'doc' not in expr_elem and doc_required: 867 889 raise QAPISemError(info, 868 890 "Expression missing documentation comment") 869 891
+3
scripts/qapi2texi.py
··· 254 254 sys.exit(1) 255 255 256 256 schema = qapi.QAPISchema(argv[1]) 257 + if not qapi.doc_required: 258 + print >>sys.stderr, ("%s: need pragma 'doc-required' " 259 + "to generate documentation" % argv[0]) 257 260 print texi(schema.docs) 258 261 259 262
+5
tests/Makefile.include
··· 381 381 qapi-schema += doc-invalid-return.json 382 382 qapi-schema += doc-invalid-section.json 383 383 qapi-schema += doc-invalid-start.json 384 + qapi-schema += doc-missing.json 384 385 qapi-schema += doc-missing-colon.json 385 386 qapi-schema += doc-missing-expr.json 386 387 qapi-schema += doc-missing-space.json ··· 422 423 qapi-schema += ident-with-escape.json 423 424 qapi-schema += include-before-err.json 424 425 qapi-schema += include-cycle.json 426 + qapi-schema += include-extra-junk.json 425 427 qapi-schema += include-format-err.json 426 428 qapi-schema += include-nested-err.json 427 429 qapi-schema += include-no-file.json ··· 439 441 qapi-schema += missing-type.json 440 442 qapi-schema += nested-struct-data.json 441 443 qapi-schema += non-objects.json 444 + qapi-schema += pragma-doc-required-crap.json 445 + qapi-schema += pragma-extra-junk.json 446 + qapi-schema += pragma-non-dict.json 442 447 qapi-schema += qapi-schema-test.json 443 448 qapi-schema += quoted-structural-chars.json 444 449 qapi-schema += redefined-builtin.json
+1
tests/qapi-schema/doc-missing.err
··· 1 + tests/qapi-schema/doc-missing.json:5: Expression missing documentation comment
+1
tests/qapi-schema/doc-missing.exit
··· 1 + 1
+5
tests/qapi-schema/doc-missing.json
··· 1 + # Expression documentation required 2 + 3 + { 'pragma': { 'doc-required': true } } 4 + 5 + { 'command': 'undocumented' }
tests/qapi-schema/doc-missing.out

This is a binary file and will not be displayed.

+1
tests/qapi-schema/include-extra-junk.err
··· 1 + tests/qapi-schema/include-extra-junk.json:3: Invalid 'include' directive
+1
tests/qapi-schema/include-extra-junk.exit
··· 1 + 1
+3
tests/qapi-schema/include-extra-junk.json
··· 1 + # 'include' must be the sole member 2 + 3 + { 'include': 'comments.json', 'junk': true }
tests/qapi-schema/include-extra-junk.out

This is a binary file and will not be displayed.

+1
tests/qapi-schema/pragma-doc-required-crap.err
··· 1 + tests/qapi-schema/pragma-doc-required-crap.json:3: Pragma 'doc-required' must be boolean
+1
tests/qapi-schema/pragma-doc-required-crap.exit
··· 1 + 1
+3
tests/qapi-schema/pragma-doc-required-crap.json
··· 1 + # 'doc-required' must be bool 2 + 3 + { 'pragma': { 'doc-required': {} } }
tests/qapi-schema/pragma-doc-required-crap.out

This is a binary file and will not be displayed.

+1
tests/qapi-schema/pragma-extra-junk.err
··· 1 + tests/qapi-schema/pragma-extra-junk.json:3: Invalid 'pragma' directive
+1
tests/qapi-schema/pragma-extra-junk.exit
··· 1 + 1
+3
tests/qapi-schema/pragma-extra-junk.json
··· 1 + # 'pragma' must be the sole member 2 + 3 + { 'pragma': { 'doc-required': true }, 'junk': true }
tests/qapi-schema/pragma-extra-junk.out

This is a binary file and will not be displayed.

+1
tests/qapi-schema/pragma-non-dict.err
··· 1 + tests/qapi-schema/pragma-non-dict.json:3: Value of 'pragma' must be a dictionary
+1
tests/qapi-schema/pragma-non-dict.exit
··· 1 + 1
+3
tests/qapi-schema/pragma-non-dict.json
··· 1 + # Value of 'pragma' must be a dictionary 2 + 3 + { 'pragma': [] }
tests/qapi-schema/pragma-non-dict.out

This is a binary file and will not be displayed.