jon.recoil.org
this repo has no description
1{0 How to Drive [odoc]}
2
3[odoc] is a CLI tool to create API and documentation for OCaml
4projects. However, it operates at a rather low level, taking individual files
5through several distinct phases until the HTML output is generated.
6
7For this reason, just like for building any multifiles OCaml project, [odoc]
8needs to be driven by a higher level tool. The driver will take care of calling
9the [odoc] command with the right arguments throughout the different
10phases. The {{!/odoc-driver/page-index}odoc-driver} package contains a "reference driver", that is kept up-to-date
11with the latest development of [odoc],
12
13Several drivers for [odoc] exist, such as:
14{{:https://dune.readthedocs.io/en/stable/documentation.html}dune} and
15{{:https://erratique.ch/software/odig/doc/}odig}.
16
17This document explains how to drive [odoc], as of version 3. It is not needed to
18know any of this to {e use} [odoc], it is targeted at driver authors, tools that
19interact with [odoc], or any curious passerby. This includes several subjects:
20
21- A big picture view of the doc generation model,
22- A unified explanation for the various command line flags,
23- An explanation of the [odoc] pipeline,
24- A convention for building docs for [opam]-installed packages.
25
26In addition to the documentation, the reference driver is a good tool to
27understand how to build [odoc] projects. It can be useful to look at the
28implementation code, but it can also help to simply look at all invocations of
29[odoc] during a run of the driver.
30
31{1:units Trees of documentation}
32
33In its third major version, [odoc] has been improved so that the same
34documentation can work on multiple scenarios, from local switches to big
35monorepos, or the {{:https://ocaml.org/packages}ocaml.org} hub of documentation for all packages, without
36anything breaking, especially references.
37
38The idea is that we have named groups of documentation, that we'll call {e trees}
39here. We have two kinds of trees: page trees, and modules trees. Inside the
40trees, the hierarchy is managed by [odoc]. The driver is free to "root" them
41however they like in the overall hierarchy. So [odoc] is responsible for the
42hierarchy below the trees root, and the driver is responsible for the one
43outside of the trees. In order to reference another tree, a documentation author
44can use the name of the tree in the reference.
45
46Different situations will give different meanings to the trees. In the case of
47[opam] packages, though, there is a natural meaning to give to those trees
48(you'll find more details in the {{!section-conv}convention for opam-installed packages}). Any
49opam package will have an associated "documentation tree", named with the name
50of the package. Any of its libraries will have an associated "module tree",
51named with the name of the library. Another package can thus refer to the doc
52using the package name, or to any of its library using the library name, no
53matter where the package is located in the hierarchy.
54
55{1 The doc generation pipeline}
56
57Just like when compiling OCaml modules, generating docs for these modules need
58to be run in a specific order, as some information for generating docs for a
59file might reside in another one. However, [odoc] actually allows a particular
60file to reference a module that depends on it, seemingly creating a circular
61dependency.
62
63This circular dependency problem is one of the reasons we have several phases in
64[odoc]. Let's review them:
65
66- The [compile] phase, which is used to create the [.odoc] artifacts from
67 [.cm{i;t;ti}] and [.mld] files. This is where [odoc] does similar work to
68 that of the OCaml compiler, computing expansions for each module types. The dependencies
69 between are the same as the ones for the [.cm{i;t;ti}] input.
70
71- The [link] phase transforms the [.odoc] artifacts to [.odocl]. The main result
72 of this phase is to resolve odoc references, which also has an effect on
73 canonical modules.
74
75- The [indexing] phase generates [.odoc-index] files from sets of [.odocl]
76 files. These index files will be used both for generating a global sidebar,
77 and for generating a search index.
78
79- The [generation] phase takes the [.odocl] and [.odoc-index] files and turns
80 them into either HTML, man pages or Latex files.
81
82{2 The compile phase}
83
84The compile phase takes as input a set of [.cm{i;t;ti}] as well as [.mld] files,
85and builds a directory hierarchy of [.odoc] files.
86
87There are distinct commands for this phase: [odoc compile] for interfaces and
88pages, [odoc compile-impl] for implementations, and [odoc compile-asset] for
89assets.
90
91{3 Compiling interfaces}
92
93Let's have a look at a generic invocation of [odoc] during the compile phase:
94
95{@shell[
96 $ odoc compile --output-dir <od> --parent-id <pid> -I <dir1> -I <dir2> <input-file>.<ext>
97]}
98
99- [<input-file>.<ext>] is the input file, either a [.cm{i;t;ti}] file or an [.mld]
100 file. Prefer [.cmti] files over the other formats!
101
102- [--output-dir <od>] allows to specify the directory that will contain all the
103 [.odoc] files. This directory has to be fully managed by [odoc] and should not
104 be modified by another tool! The output file depends on the [--parent-id]
105 option.
106
107- [--parent-id <pid>] allows to place the output [.odoc] file in the
108 documentation hierarchy. This consists in a [/] separated sequence of non
109 empty strings (used as directory name). This "path" determines where the
110 [.odoc] file will be located below the [<od>] output dir. The name of the
111 output file is [<input-file>.odoc] for modules, and [page-<input-file>.odoc]
112 for pages. Documentation artifacts that will be in the same {{!units}unit of
113 documentation} need to hare a common root in their parent id.
114
115- [-I <dir>] corresponds to the search path for other [.odoc] files. Multiple
116 directory can be added to the search path, so every required [.odoc] file is
117 in the search path. The required [.odoc] files are the one generated from a
118 [.cm{i;t;ti}] file listed when calling [odoc compile-deps] on the input
119 file.
120
121A concrete example for such command would be:
122
123{@shell[
124 $ odoc compile
125 ~/.opam/5.2.0/lib/ppxlib/ppxlib__Extension.cmti
126 --output-dir _odoc/
127 -I _odoc/ocaml-base-compiler/compiler-libs.common
128 -I _odoc/ocaml-base-compiler/stdlib
129 -I _odoc/ocaml-compiler-libs/ocaml-compiler-libs.common
130 -I _odoc/ppxlib/ppxlib
131 -I _odoc/ppxlib/ppxlib.ast
132 -I _odoc/ppxlib/ppxlib.astlib
133 -I _odoc/ppxlib/ppxlib.stdppx
134 -I _odoc/ppxlib/ppxlib.traverse_builtins
135 -I _odoc/sexplib0/sexplib0
136 --parent-id ppxlib/ppxlib
137]}
138
139{3 Compiling implementations}
140
141A [compile-impl] command is pretty similar:
142
143{@shell[
144 $ odoc compile-impl --output-dir <od> --source-id <sid> --parent-id <pid> -I <dir1> -I <dir2> <input-file>.<ext>
145]}
146
147- [<input-file>.cmt] is the input file, it has to be a [.cmt] file.
148
149- [--output-dir <od>] has the same meaning as for [odoc compile].
150
151- [--parent-id <pid>] also has the same meaning as for [odoc compile]. However,
152 the name of the output file is [impl-<input-file>.odoc]. Implementations need
153 to be available through the [-I] search path, so it is very likely that one
154 wants the implementation and interface [.odoc] files to share the same parent
155 id.
156
157- [-I <dir>] also corresponds to the search path for other [.odoc] files.
158
159- [source-id <sid>] is a new argument specific to [compile-impl]. This corresponds to the location of the rendering of the source, which is required to generate links to it.
160
161A concrete example for such command would be:
162
163{@shell[
164 $ odoc compile-impl
165 ~/.opam/5.2.0/lib/ppxlib/ppxlib__Spellcheck.cmt
166 --output-dir _odoc/
167 -I _odoc/ocaml-base-compiler/compiler-libs.common
168 -I _odoc/ocaml-base-compiler/stdlib
169 -I _odoc/ocaml-compiler-libs/ocaml-compiler-libs.common
170 -I _odoc/ppxlib/ppxlib
171 -I _odoc/ppxlib/ppxlib.ast
172 -I _odoc/ppxlib/ppxlib.astlib
173 -I _odoc/ppxlib/ppxlib.stdppx
174 -I _odoc/sexplib0/sexplib0
175 --enable-missing-root-warning
176 --parent-id ppxlib/ppxlib
177 --source-id ppxlib/src/ppxlib/spellcheck.ml
178]}
179
180{3 Compiling assets}
181
182Assets are given during the generation phase. But we still need to create an
183[.odoc] file, for [odoc]'s resolution mechanism.
184
185{@shell[
186 $ odoc compile-asset --output-dir <od> --parent-id <pid> --name <assetname>
187]}
188
189- [--output-dir] and [--parent-id] are identical to the [compile] and
190 [compile-impl] commands,
191
192- [--name <assetname>] gives the asset name.
193
194- The output file name is computed from the previous values as
195 [<output-dir>/<parent-id>/asset-<assetname>.odoc].
196
197{2 The link phase}
198
199The link phase requires the directory of the [compile] phase to generate its set
200of [.odocl] files. This phase resolves references and canonicals.
201
202A generic link command is:
203
204{@shell[
205 $ odoc link
206 -I <dir1> -I <dir2>
207 -P <pname1>:<pdir1> -P <pname2>:<pdir2>
208 -L <lname1>:<ldir1> -L <lname2>:<ldir2>
209 <path/to/file.odoc>
210]}
211
212- [<path/to/file.odoc] is the input [.odoc] file. The result of this command is
213 [path/to/file.odocl]. This path was determined by [--output-dir] and
214 [--parent-id] from the link phase, and it is important for the indexing phase
215 that it stays in the same location.
216
217- [-P <name>:<dir>] are used to list the "page trees", used to resolve
218 references such as [{!/ocamlfind/index}].
219
220- [-L <name>:<dir>] are used to list the "module trees", used to resolve
221 references such as [{!/findlib.dynload/Fl_dynload}]. This also adds [<dir>] to
222 the search path.
223
224- [-I <dir>] adds [<dir>] to the search path. The search path is used to resolve
225 references that do not use the "named tree" mechanism, such as [{!Module}] and
226 [{!page-pagename}].
227
228{2 The indexing phase}
229
230The indexing phase refers to the "crunching" of information split in several
231[.odocl] files. Currently, there are two use-cases for this phase:
232
233- Generating a search index. This requires all information from linked
234 interfaces and pages, but also form linked implementations in order to sort
235 results (by number of occurrences).
236
237- Generating a global sidebar.
238
239{3 Counting occurrences}
240
241This step counts the number of occurrences of each value/type/... in the
242implementation, and stores them in a table. A generic invocation is:
243
244{@shell[
245 $ odoc count-occurrences <dir1> <dir2> -o <path/to/name.odoc-occurrences>
246]}
247
248An example of such command:
249
250{@shell[
251 $ odoc count-occurrences _odoc/ -o _odoc/occurrences-all.odoc-occurrences
252]}
253
254{3 Indexing entries}
255
256The [odoc compile-index] produces an [.odoc-index] file, from [.odocl] files,
257other [.odoc-index] files, and possibly some [.odoc-occurrences] files.
258
259To create an index for the page and documentation units, we use the [-P] and
260[-L] arguments.
261
262{@shell[
263 $ odoc compile-index
264 -o path/to/<indexname>.odoc-index
265 -P <pname1>:<ppath1>
266 -P <pname2>:<ppath2>
267 -L <lname1>:<lpath1>
268 -L <lname2>:<lpath2>
269 --occurrences <path/to/name.odoc-occurrences>
270]}
271
272An example of such command:
273
274{@shell[
275 $ odoc compile-index
276 -o _odoc/ppxlib/index.odoc-index
277 -P ppxlib:_odoc/ppxlib
278 -L ppxlib:_odoc/ppxlib/ppxlib
279 -L ppxlib.ast:_odoc/ppxlib/ppxlib.ast
280 -L ppxlib.astlib:_odoc/ppxlib/ppxlib.astlib
281 -L ppxlib.metaquot:_odoc/ppxlib/ppxlib.metaquot
282 -L ppxlib.metaquot_lifters:_odoc/ppxlib/ppxlib.metaquot_lifters
283 -L ppxlib.print_diff:_odoc/ppxlib/ppxlib.print_diff
284 -L ppxlib.runner:_odoc/ppxlib/ppxlib.runner
285 -L ppxlib.runner_as_ppx:_odoc/ppxlib/ppxlib.runner_as_ppx
286 -L ppxlib.stdppx:_odoc/ppxlib/ppxlib.stdppx
287 -L ppxlib.traverse:_odoc/ppxlib/ppxlib.traverse
288 -L ppxlib.traverse_builtins:_odoc/ppxlib/ppxlib.traverse_builtins
289 --occurrences _odoc/occurrences-all.odoc-occurrences
290]}
291
292{2 The generation phase}
293
294The generation phase is the phase that takes all information computed in
295previous files, and actually generates the documentation. It can take the form
296of HTML, Latex and manpages, although currently HTML is the [odoc] backend that
297supports the most functionalities (such as images, videos, ...).
298
299In this manual, we describe the HTML generation usecase. Usually,
300generating for other backend boils down to replacing [html-generate] by
301[latex-generate] or [man-generate], refer to the manpage to see the diverging
302options.
303
304Given an [.odocl] file, [odoc] might generate a single [.html] file, or a
305complete directory of [.html] files. The [--output-dir] option specifies the
306root for generating those outputs.
307
308{3 A JavaScript file for search requests}
309
310[odoc] provides a way to plugin a JavaScript file, containing the code to answer
311user's queries. In order to never block the UI, this file will be loaded in a
312web worker to perform searches:
313
314- The search query will be sent as a plain string to the web worker, using the
315 standard mechanism of message passing.
316
317- The web worker has to send back the result as a message to the main thread,
318 containing the results. The format for the result message is a string that can
319 be parsed as a list of JSON objects. Each object contain two keys-value pairs:
320 a key ["url"] which contains a relative URL from [--ouput-dir]; and a key
321 ["html"] which contain the html showing the search entry.
322
323- The JavaScript file must be manually put in the [--output-dir] values, the driver can
324 decide where.
325
326{3 Interfaces and pages}
327
328A generic [html-generate] command for interfaces has the following form:
329
330{@shell[
331 $ odoc html-generate
332 --output-dir <odir>
333 --index <path/to/file.odoc-index>
334 --search-uri <relative/to/output-dir/file.js>
335 --search-uri <relative/to/output-dir/file2.js>
336 <path/to/file.odocl>
337]}
338
339- [--output-dir <odir>] is used to specify the root output for the generated
340 [.html].
341
342- [--index <path/to/file.odoc-index>] is given to [odoc] for sidebar generation.
343
344- [--search-uri <relative/to/output-dir/file.js>] tells [odoc] which file(s) to
345 load in a web worker.
346
347The output directory or file can be computed from this command's [--output-dir],
348the initial [--parent-id] given when creating the [.odoc] file, as well as the
349unit name. In the case of a module, the output is a directory named with the
350name of the module. In the case of a page, the output is a file with the name of
351the page and the [.html] extension.
352
353An example of such command is:
354
355{@shell[
356 $ odoc html-generate
357 _odoc/ppxlib/page-index.odocl
358 --index _odoc/ppxlib/index.odoc-index
359 --search-uri ppxlib/sherlodoc_db.js
360 --search-uri sherlodoc.js
361 -o _html/
362]}
363
364{3 Source code}
365
366{@shell[
367 $ odoc html-generate-source --output-dir <odir> --impl <path/to/impl-file.odocl> <path/to/source/file.ml>
368]}
369
370- [--output-dir <odir>] has been covered already
371
372- [--impl <path/to/impl-file.odocl>] allows to give the implementation file.
373
374- [<path/to/source/file.ml>] is the source file.
375
376The output file can be computed from this command's [--output-dir], and the
377initial [--source-id] and [--name] given when creating the [impl-*.odoc] file.
378
379An example of such command is:
380
381{@shell[
382 $ odoc html-generate-source
383 --impl _odoc/ppxlib/ppxlib/impl-ppxlib__Reconcile.odocl
384 /home/panglesd/.opam/5.2.0/lib/ppxlib/reconcile.ml
385 -o _html/
386]}
387
388{3 Generating docs for assets}
389
390This is the phase where we pass the actual asset. We pass it as a positional
391argument, and give the asset unit using [--asset-unit].
392
393{@shell[
394 $ odoc html-generate-asset --output-dir <odir> --asset-unit <path/to/asset-file.odocl> <path/to/asset/file.ext>
395]}
396
397{1:conv Convention for installed packages}
398
399In order to build the documentation for installed packages, the driver needs to
400give a meaning to the various concepts above. In particular, it needs to
401define the pages and libraries trees, know where to find the pages and assets,
402what id to give them, when linking it needs to know to which trees the artifact
403may be linking...
404
405So that the different drivers and installed packages play well together, we
406define here a convention for building installed packages. If both the package
407and the driver follow it, building the docs should go well!
408
409{2 The [-P] and [-L] trees, and their root ids}
410
411Each package defines a set of trees, each of them having a root id. These roots
412will be used in [--parent-id] and in [-P] and [-L].
413
414The driver can decide any set of mutually disjoint set of roots, without posing
415problem to the reference resolution. For instance, both [-P
416pkg:<output_dir>/pkg] and [-P pkg:<output_dir>/pkg/version] are
417acceptable versions. However, we define here "canonical" roots:
418
419Each installed package [<p>] defines a single page root id: [<p>].
420
421For each package [<p>], each library [<l>] defines a library root id:
422[<p>/<l>].
423
424For instance, a package [foo] with two libraries: [foo] and [foo.bar] will
425define three trees:
426
427- A documentation tree named [foo], with root id [foo]. When referred
428 from other trees, a [-P foo:<odoc_dir>/foo] argument needs to be added
429 at the link phase.
430
431- A module tree named [foo], with root id [foo/foo]. When referred from
432 other trees, a [-L foo:<odoc_dir>/foo/foo] argument needs to be added
433 at the link phase.
434
435- A module tree named [foo.bar], with root id [foo/foo.bar]. When referred from
436 other trees, a [-L foo.bar:<odoc_dir>/foo/foo.bar] argument needs to be
437 added at the link phase.
438
439{2 Link-time dependencies}
440
441Installed OPAM packages need to specify which trees they may be referencing
442during the link phase, so that the proper [-P] and [-L] arguments are
443added. (Note that these dependencies can be circular, as they
444happen during the link phase and only require the artifact from the compile
445phase.)
446
447An installed package [<p>] specifies its tree dependencies in a file at
448[<opam root>/doc/<p>/odoc-config.sexp]. This file contains s-expressions.
449
450Stanzas of the form [(packages p1 p2 ...)] specifies that page trees [p1],
451[p2], ..., should be added using the [-P] argument: with the canonical roots, it
452would be [-P p1:<output_dir>/p1 -P p2:<output_dir>/p2 -P ...].
453
454Stanzas of the form [(libraries l1 l2 ...)] specifies that module trees [l1],
455[l2], ..., should be added using the [-L] argument: with the canonical roots, it
456would be [-L l1:<output_dir>/p1/l1 -L l2<output_dir>/p2/l2 -L ...],
457where [p1] is the package [l1] is in, etc.
458
459{2 The units}
460
461The module units of a package [p] are all files installed by [p] that can be
462found in [<opam root>/lib/p/] or a subdirectory.
463
464The page units are those files that can be found in [<opam
465root>/doc/odoc-pages/] or a subdirectory, and that have an [.mld] extension.
466
467The asset units are those files that can be found in [<opam
468root>/doc/odoc-pages/] or a subdirectory, but that do not have an [.mld]
469extension. Additionally, they are all files found in [<opam
470root>/doc/odoc-assets/].
471
472{2 The [--parent-id] arguments}
473
474Interface and implementation units have as parent id the root of the library
475tree they belong to: with "canonical" roots, [<pkgname>/<libname>].
476
477Page units that are found in [<opam
478root>/doc/<pkgname>/odoc-pages/<relpath>/<name>.mld] have the parent id from
479their page tree, followed by [<relpath>]. So, with canonical roots,
480[<pkgname>/<relpath>].
481
482Asset units that are found in [<opam
483root>/doc/<pkgname>/odoc-pages/<relpath>/<name>.<ext>] have the parent id from
484their page tree, followed by [<relpath>]. With canonical roots,
485[<pkgname>/<relpath>].
486
487Asset units that are found in [<opam root>/doc/<pkgname>/odoc-assets/<filename>]
488have the parent id from their page tree, followed by [_asset/<filename>]
489[<p>/_assets/<filename>].
490
491{2 The [--source-id] arguments}
492
493The driver could choose the source id without breaking references. However,
494following the canonical roots convention, implementation units must have as
495source id: [<pkgname>/src/<libraryname>/<filename>.ml].
496
497{2 Ordering the generated pages}
498
499The canonical hierarchy introduces directories (one per library) that may not be
500ordered by the author, either by omitting it in the [@children_order] tag or by
501not specifying any [@children_order] tag. In this case, [odoc] needs to come up
502with a reasonable default order, which may not be easy without some help from
503the driver.
504
505In auto-generated pages (either [index.mld] in a directory, or [page.mld]),
506[odoc] supports the [@order_category <category_name>] tag, to help sorting the
507pages, if it is not sorted by the parent's [@children_order]. The resulting
508order is:
509
510- First, pages in the order given in their parent's [@children_order],
511- Then, pages ordered lexicographically by their [@order_category]. An undefined
512 category comes before a defined one.
513- Inside a category, pages are ordered lexicographically by their first title,
514- Two pages with the same name will be ordered using their file name!
515
516Note that [@order_category] is not suitable for author use, as it may change in
517the future. Use this tag only in the driver's autogenerated pages!