freshlybakedca.ke / git
Git fork
fork atom

doc: add large-object-promisors.adoc to the docs build

Commit 5040f9f164 ("doc: add technical design doc for large object
promisors", 2025-02-18) added the large object promisors document
as a technical document (with a '.txt' extension). The merge commit
2c6fd30198 ("Merge branch 'cc/lop-remote'", 2025-03-05) seems to
have renamed the file with an '.adoc' extension.

Despite the '.adoc' extension, this document was not being formatted
by asciidoc(tor) as part of the docs build. In order to do so, add
the document to the make and meson build files.

Having added the document to the build, asciidoc and asciidoctor find
(slightly different) problems with the syntax of the input document.

The first set of warnings (only issued by asciidoc) relate to some
'section title out of sequence: expected level 3, got level 4'. This
document uses 'setext' style of section headers, using a series of
underline characters, where the character used denotes the level of
the title. From document title to level 5 (see [1]), these characters
are =, -, ~, ^, +. This does not seem to fit the error message, which
implies that those characters denote levels 0 -> 4. Replacing the headings
underlined with '+' by the '^' character eliminates these warnings.

The second set of warnings (only issued by asciidoctor) relate to some
headings which seem to use both arabic and roman numerals as part of
a single 'list' sequence. This elicited either 'unterminated listing
block' or (for example) 'list item index: expected I, got II' warnings.
In order not to mix arabic and roman numerals, remove the numeral from
the '0) Non goals' heading. Similarly, the remaining roman numeral
entries had the ')' removed and turned into regular headings with I, II,
III ... at the beginning.

[1] https://asciidoctor.org/docs/asciidoc-recommended-practices/

Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

authored by

Ramsay Jones and committed by
Junio C Hamano 1c1fc86d b770ed95

+34 -32
unified split
+1
Documentation/Makefile
··· 122 122 TECH_DOCS += technical/commit-graph 123 123 TECH_DOCS += technical/directory-rename-detection 124 124 TECH_DOCS += technical/hash-function-transition 125 + TECH_DOCS += technical/large-object-promisors 125 126 TECH_DOCS += technical/long-running-process-protocol 126 127 TECH_DOCS += technical/multi-pack-index 127 128 TECH_DOCS += technical/packfile-uri
+32 -32
Documentation/technical/large-object-promisors.adoc
··· 34 34 35 35 https://lore.kernel.org/git/xmqqbkdometi.fsf@gitster.g/ 36 36 37 - 0) Non goals 38 - ------------ 37 + Non goals 38 + --------- 39 39 40 40 - We will not discuss those client side improvements here, as they 41 41 would require changes in different parts of Git than this effort. ··· 90 90 even more to host content with larger blobs or more large blobs 91 91 than currently. 92 92 93 - I) Issues with the current situation 94 - ------------------------------------ 93 + I Issues with the current situation 94 + ----------------------------------- 95 95 96 96 - Some statistics made on GitLab repos have shown that more than 75% 97 97 of the disk space is used by blobs that are larger than 1MB and ··· 138 138 complaining that these tools require significant effort to set up, 139 139 learn and use correctly. 140 140 141 - II) Main features of the "Large Object Promisors" solution 142 - ---------------------------------------------------------- 141 + II Main features of the "Large Object Promisors" solution 142 + --------------------------------------------------------- 143 143 144 144 The main features below should give a rough overview of how the 145 145 solution may work. Details about needed elements can be found in ··· 166 166 other objects. 167 167 168 168 Note 1 169 - ++++++ 169 + ^^^^^^ 170 170 171 171 To clarify, a LOP is a normal promisor remote, except that: 172 172 ··· 178 178 itself. 179 179 180 180 Note 2 181 - ++++++ 181 + ^^^^^^ 182 182 183 183 Git already makes it possible for a main remote to also be a promisor 184 184 remote storing both regular objects and large blobs for a client that ··· 186 186 to avoid that. 187 187 188 188 Rationale 189 - +++++++++ 189 + ^^^^^^^^^ 190 190 191 191 LOPs aim to be good at handling large blobs while main remotes are 192 192 already good at handling other objects. 193 193 194 194 Implementation 195 - ++++++++++++++ 195 + ^^^^^^^^^^^^^^ 196 196 197 197 Git already has support for multiple promisor remotes, see 198 198 link:partial-clone.html#using-many-promisor-remotes[the partial clone documentation]. ··· 213 213 underlying object storage appear like a remote to Git. 214 214 215 215 Note 216 - ++++ 216 + ^^^^ 217 217 218 218 A LOP can be a promisor remote accessed using a remote helper by 219 219 both some clients and the main remote. 220 220 221 221 Rationale 222 - +++++++++ 222 + ^^^^^^^^^ 223 223 224 224 This looks like the simplest way to create LOPs that can cheaply 225 225 handle many large blobs. 226 226 227 227 Implementation 228 - ++++++++++++++ 228 + ^^^^^^^^^^^^^^ 229 229 230 230 Remote helpers are quite easy to write as shell scripts, but it might 231 231 be more efficient and maintainable to write them using other languages ··· 247 247 storage for large files handled by Git LFS. 248 248 249 249 Rationale 250 - +++++++++ 250 + ^^^^^^^^^ 251 251 252 252 This would simplify the server side if it wants to both use a LOP and 253 253 act as a Git LFS server. ··· 259 259 LOP all its blobs with a size over a configurable threshold. 260 260 261 261 Rationale 262 - +++++++++ 262 + ^^^^^^^^^ 263 263 264 264 This makes it easy to set things up and to clean things up. For 265 265 example, an admin could use this to manually convert a repo not using ··· 268 268 to regularly make sure the large blobs are moved to the LOP. 269 269 270 270 Implementation 271 - ++++++++++++++ 271 + ^^^^^^^^^^^^^^ 272 272 273 273 Using something based on `git repack --filter=...` to separate the 274 274 blobs we want to offload from the other Git objects could be a good ··· 284 284 perhaps pushed, into it. 285 285 286 286 Rationale 287 - +++++++++ 287 + ^^^^^^^^^ 288 288 289 289 A main remote containing many oversize blobs would defeat the purpose 290 290 of LOPs. 291 291 292 292 Implementation 293 - ++++++++++++++ 293 + ^^^^^^^^^^^^^^ 294 294 295 295 The way to offload to a LOP discussed in 4) above can be used to 296 296 regularly offload oversize blobs. About preventing oversize blobs from ··· 326 326 fetch those blobs from the LOP to be able to serve the client. 327 327 328 328 Note 329 - ++++ 329 + ^^^^ 330 330 331 331 For fetches instead of clones, a protocol negotiation might not always 332 332 happen, see the "What about fetches?" FAQ entry below for details. 333 333 334 334 Rationale 335 - +++++++++ 335 + ^^^^^^^^^ 336 336 337 337 Security, configurability and efficiency of setting things up. 338 338 339 339 Implementation 340 - ++++++++++++++ 340 + ^^^^^^^^^^^^^^ 341 341 342 342 A "promisor-remote" protocol v2 capability looks like a good way to 343 343 implement this. The way the client and server use this capability ··· 356 356 but might not need anymore, to the LOP. 357 357 358 358 Note 359 - ++++ 359 + ^^^^ 360 360 361 361 It might depend on the context if it should be OK or not for clients 362 362 to offload large blobs they have created, instead of fetched, directly ··· 367 367 implementing this feature. 368 368 369 369 Rationale 370 - +++++++++ 370 + ^^^^^^^^^ 371 371 372 372 On the client, the easiest way to deal with unneeded large blobs is to 373 373 offload them. 374 374 375 375 Implementation 376 - ++++++++++++++ 376 + ^^^^^^^^^^^^^^ 377 377 378 378 This is very similar to what 4) above is about, except on the client 379 379 side instead of the server side. So a good solution to 4) could likely ··· 385 385 a LOP, it is likely, and can easily be confirmed, that the LOP still 386 386 has them, so that they can just be removed from the client. 387 387 388 - III) Benefits of using LOPs 389 - --------------------------- 388 + III Benefits of using LOPs 389 + -------------------------- 390 390 391 391 Many benefits are related to the issues discussed in "I) Issues with 392 392 the current situation" above: ··· 406 406 407 407 - Reduced storage needs on the client side. 408 408 409 - IV) FAQ 410 - ------- 409 + IV FAQ 410 + ------ 411 411 412 412 What about using multiple LOPs on the server and client side? 413 413 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ··· 533 533 on a promisor remote. 534 534 535 535 Regular fetch 536 - +++++++++++++ 536 + ^^^^^^^^^^^^^ 537 537 538 538 In a regular fetch, the client will contact the main remote and a 539 539 protocol negotiation will happen between them. It's a good thing that ··· 551 551 using, or not using, the same LOP(s) as last time. 552 552 553 553 "Backfill" or "lazy" fetch 554 - ++++++++++++++++++++++++++ 554 + ^^^^^^^^^^^^^^^^^^^^^^^^^^ 555 555 556 556 When there is a backfill fetch, the client doesn't necessarily contact 557 557 the main remote first. It will try to fetch from its promisor remotes ··· 576 576 token when performing a protocol negotiation with the main remote (see 577 577 section II.6 above). 578 578 579 - V) Future improvements 580 - ---------------------- 579 + V Future improvements 580 + --------------------- 581 581 582 582 It is expected that at the beginning using LOPs will be mostly worth 583 583 it either in a corporate context where the Git version that clients
+1
Documentation/technical/meson.build
··· 13 13 'commit-graph.adoc', 14 14 'directory-rename-detection.adoc', 15 15 'hash-function-transition.adoc', 16 + 'large-object-promisors.adoc', 16 17 'long-running-process-protocol.adoc', 17 18 'multi-pack-index.adoc', 18 19 'packfile-uri.adoc',