freshlybakedca.ke
Git fork
Merge branch 'tb/cruft-packs'
A mechanism to pack unreachable objects into a "cruft pack",
instead of ejecting them into loose form to be reclaimed later, has
been introduced.
* tb/cruft-packs:
sha1-file.c: don't freshen cruft packs
builtin/gc.c: conditionally avoid pruning objects via loose
builtin/repack.c: add cruft packs to MIDX during geometric repack
builtin/repack.c: use named flags for existing_packs
builtin/repack.c: allow configuring cruft pack generation
builtin/repack.c: support generating a cruft pack
builtin/pack-objects.c: --cruft with expiration
reachable: report precise timestamps from objects in cruft packs
reachable: add options to add_unseen_recent_objects_to_traversal
builtin/pack-objects.c: --cruft without expiration
builtin/pack-objects.c: return from create_object_entry()
t/helper: add 'pack-mtimes' test-tool
pack-mtimes: support writing pack .mtimes files
chunk-format.h: extract oid_version()
pack-write: pass 'struct packing_data' to 'stage_tmp_packfiles'
pack-mtimes: support reading .mtimes files
Documentation/technical: add cruft-packs.txt
+1
Documentation/Makefile
+1
Documentation/Makefile
+14
-7
Documentation/config/gc.txt
+14
-7
Documentation/config/gc.txt
···
81
81
to enable it within all non-bare repos or it can be set to a
82
82
boolean value. The default is `true`.
83
83
84
+
gc.cruftPacks::
85
+
Store unreachable objects in a cruft pack (see
86
+
linkgit:git-repack[1]) instead of as loose objects. The default
87
+
is `false`.
88
+
84
89
gc.pruneExpire::
85
-
When 'git gc' is run, it will call 'prune --expire 2.weeks.ago'.
86
-
Override the grace period with this config variable. The value
87
-
"now" may be used to disable this grace period and always prune
88
-
unreachable objects immediately, or "never" may be used to
89
-
suppress pruning. This feature helps prevent corruption when
90
-
'git gc' runs concurrently with another process writing to the
91
-
repository; see the "NOTES" section of linkgit:git-gc[1].
90
+
When 'git gc' is run, it will call 'prune --expire 2.weeks.ago'
91
+
(and 'repack --cruft --cruft-expiration 2.weeks.ago' if using
92
+
cruft packs via `gc.cruftPacks` or `--cruft`). Override the
93
+
grace period with this config variable. The value "now" may be
94
+
used to disable this grace period and always prune unreachable
95
+
objects immediately, or "never" may be used to suppress pruning.
96
+
This feature helps prevent corruption when 'git gc' runs
97
+
concurrently with another process writing to the repository; see
98
+
the "NOTES" section of linkgit:git-gc[1].
92
99
93
100
gc.worktreePruneExpire::
94
101
When 'git gc' is run, it calls
+9
Documentation/config/repack.txt
+9
Documentation/config/repack.txt
···
30
30
If set to false, linkgit:git-repack[1] will not run
31
31
linkgit:git-update-server-info[1]. Defaults to true. Can be overridden
32
32
when true by the `-n` option of linkgit:git-repack[1].
33
+
34
+
repack.cruftWindow::
35
+
repack.cruftWindowMemory::
36
+
repack.cruftDepth::
37
+
repack.cruftThreads::
38
+
Parameters used by linkgit:git-pack-objects[1] when generating
39
+
a cruft pack and the respective parameters are not given over
40
+
the command line. See similarly named `pack.*` configuration
41
+
variables for defaults and meaning.
+5
Documentation/git-gc.txt
+5
Documentation/git-gc.txt
···
54
54
be performed as well.
55
55
56
56
57
+
--cruft::
58
+
When expiring unreachable objects, pack them separately into a
59
+
cruft pack instead of storing the loose objects as loose
60
+
objects.
61
+
57
62
--prune=<date>::
58
63
Prune loose objects older than date (default is 2 weeks ago,
59
64
overridable by the config variable `gc.pruneExpire`).
+30
Documentation/git-pack-objects.txt
+30
Documentation/git-pack-objects.txt
···
13
13
[--no-reuse-delta] [--delta-base-offset] [--non-empty]
14
14
[--local] [--incremental] [--window=<n>] [--depth=<n>]
15
15
[--revs [--unpacked | --all]] [--keep-pack=<pack-name>]
16
+
[--cruft] [--cruft-expiration=<time>]
16
17
[--stdout [--filter=<filter-spec>] | <base-name>]
17
18
[--shallow] [--keep-true-parents] [--[no-]sparse] < <object-list>
18
19
···
94
95
+
95
96
Incompatible with `--revs`, or options that imply `--revs` (such as
96
97
`--all`), with the exception of `--unpacked`, which is compatible.
98
+
99
+
--cruft::
100
+
Packs unreachable objects into a separate "cruft" pack, denoted
101
+
by the existence of a `.mtimes` file. Typically used by `git
102
+
repack --cruft`. Callers provide a list of pack names and
103
+
indicate which packs will remain in the repository, along with
104
+
which packs will be deleted (indicated by the `-` prefix). The
105
+
contents of the cruft pack are all objects not contained in the
106
+
surviving packs which have not exceeded the grace period (see
107
+
`--cruft-expiration` below), or which have exceeded the grace
108
+
period, but are reachable from an other object which hasn't.
109
+
+
110
+
When the input lists a pack containing all reachable objects (and lists
111
+
all other packs as pending deletion), the corresponding cruft pack will
112
+
contain all unreachable objects (with mtime newer than the
113
+
`--cruft-expiration`) along with any unreachable objects whose mtime is
114
+
older than the `--cruft-expiration`, but are reachable from an
115
+
unreachable object whose mtime is newer than the `--cruft-expiration`).
116
+
+
117
+
Incompatible with `--unpack-unreachable`, `--keep-unreachable`,
118
+
`--pack-loose-unreachable`, `--stdin-packs`, as well as any other
119
+
options which imply `--revs`. Also incompatible with `--max-pack-size`;
120
+
when this option is set, the maximum pack size is not inferred from
121
+
`pack.packSizeLimit`.
122
+
123
+
--cruft-expiration=<approxidate>::
124
+
If specified, objects are eliminated from the cruft pack if they
125
+
have an mtime older than `<approxidate>`. If unspecified (and
126
+
given `--cruft`), then no objects are eliminated.
97
127
98
128
--window=<n>::
99
129
--depth=<n>::
+11
Documentation/git-repack.txt
+11
Documentation/git-repack.txt
···
63
63
Also run 'git prune-packed' to remove redundant
64
64
loose object files.
65
65
66
+
--cruft::
67
+
Same as `-a`, unless `-d` is used. Then any unreachable objects
68
+
are packed into a separate cruft pack. Unreachable objects can
69
+
be pruned using the normal expiry rules with the next `git gc`
70
+
invocation (see linkgit:git-gc[1]). Incompatible with `-k`.
71
+
72
+
--cruft-expiration=<approxidate>::
73
+
Expire unreachable objects older than `<approxidate>`
74
+
immediately instead of waiting for the next `git gc` invocation.
75
+
Only useful with `--cruft -d`.
76
+
66
77
-l::
67
78
Pass the `--local` option to 'git pack-objects'. See
68
79
linkgit:git-pack-objects[1].
+123
Documentation/technical/cruft-packs.txt
+123
Documentation/technical/cruft-packs.txt
···
1
+
= Cruft packs
2
+
3
+
The cruft packs feature offer an alternative to Git's traditional mechanism of
4
+
removing unreachable objects. This document provides an overview of Git's
5
+
pruning mechanism, and how a cruft pack can be used instead to accomplish the
6
+
same.
7
+
8
+
== Background
9
+
10
+
To remove unreachable objects from your repository, Git offers `git repack -Ad`
11
+
(see linkgit:git-repack[1]). Quoting from the documentation:
12
+
13
+
[quote]
14
+
[...] unreachable objects in a previous pack become loose, unpacked objects,
15
+
instead of being left in the old pack. [...] loose unreachable objects will be
16
+
pruned according to normal expiry rules with the next 'git gc' invocation.
17
+
18
+
Unreachable objects aren't removed immediately, since doing so could race with
19
+
an incoming push which may reference an object which is about to be deleted.
20
+
Instead, those unreachable objects are stored as loose objects and stay that way
21
+
until they are older than the expiration window, at which point they are removed
22
+
by linkgit:git-prune[1].
23
+
24
+
Git must store these unreachable objects loose in order to keep track of their
25
+
per-object mtimes. If these unreachable objects were written into one big pack,
26
+
then either freshening that pack (because an object contained within it was
27
+
re-written) or creating a new pack of unreachable objects would cause the pack's
28
+
mtime to get updated, and the objects within it would never leave the expiration
29
+
window. Instead, objects are stored loose in order to keep track of the
30
+
individual object mtimes and avoid a situation where all cruft objects are
31
+
freshened at once.
32
+
33
+
This can lead to undesirable situations when a repository contains many
34
+
unreachable objects which have not yet left the grace period. Having large
35
+
directories in the shards of `.git/objects` can lead to decreased performance in
36
+
the repository. But given enough unreachable objects, this can lead to inode
37
+
starvation and degrade the performance of the whole system. Since we
38
+
can never pack those objects, these repositories often take up a large amount of
39
+
disk space, since we can only zlib compress them, but not store them in delta
40
+
chains.
41
+
42
+
== Cruft packs
43
+
44
+
A cruft pack eliminates the need for storing unreachable objects in a loose
45
+
state by including the per-object mtimes in a separate file alongside a single
46
+
pack containing all loose objects.
47
+
48
+
A cruft pack is written by `git repack --cruft` when generating a new pack.
49
+
linkgit:git-pack-objects[1]'s `--cruft` option. Note that `git repack --cruft`
50
+
is a classic all-into-one repack, meaning that everything in the resulting pack is
51
+
reachable, and everything else is unreachable. Once written, the `--cruft`
52
+
option instructs `git repack` to generate another pack containing only objects
53
+
not packed in the previous step (which equates to packing all unreachable
54
+
objects together). This progresses as follows:
55
+
56
+
1. Enumerate every object, marking any object which is (a) not contained in a
57
+
kept-pack, and (b) whose mtime is within the grace period as a traversal
58
+
tip.
59
+
60
+
2. Perform a reachability traversal based on the tips gathered in the previous
61
+
step, adding every object along the way to the pack.
62
+
63
+
3. Write the pack out, along with a `.mtimes` file that records the per-object
64
+
timestamps.
65
+
66
+
This mode is invoked internally by linkgit:git-repack[1] when instructed to
67
+
write a cruft pack. Crucially, the set of in-core kept packs is exactly the set
68
+
of packs which will not be deleted by the repack; in other words, they contain
69
+
all of the repository's reachable objects.
70
+
71
+
When a repository already has a cruft pack, `git repack --cruft` typically only
72
+
adds objects to it. An exception to this is when `git repack` is given the
73
+
`--cruft-expiration` option, which allows the generated cruft pack to omit
74
+
expired objects instead of waiting for linkgit:git-gc[1] to expire those objects
75
+
later on.
76
+
77
+
It is linkgit:git-gc[1] that is typically responsible for removing expired
78
+
unreachable objects.
79
+
80
+
== Caution for mixed-version environments
81
+
82
+
Repositories that have cruft packs in them will continue to work with any older
83
+
version of Git. Note, however, that previous versions of Git which do not
84
+
understand the `.mtimes` file will use the cruft pack's mtime as the mtime for
85
+
all of the objects in it. In other words, do not expect older (pre-cruft pack)
86
+
versions of Git to interpret or even read the contents of the `.mtimes` file.
87
+
88
+
Note that having mixed versions of Git GC-ing the same repository can lead to
89
+
unreachable objects never being completely pruned. This can happen under the
90
+
following circumstances:
91
+
92
+
- An older version of Git running GC explodes the contents of an existing
93
+
cruft pack loose, using the cruft pack's mtime.
94
+
- A newer version running GC collects those loose objects into a cruft pack,
95
+
where the .mtime file reflects the loose object's actual mtimes, but the
96
+
cruft pack mtime is "now".
97
+
98
+
Repeating this process will lead to unreachable objects not getting pruned as a
99
+
result of repeatedly resetting the objects' mtimes to the present time.
100
+
101
+
If you are GC-ing repositories in a mixed version environment, consider omitting
102
+
the `--cruft` option when using linkgit:git-repack[1] and linkgit:git-gc[1], and
103
+
leaving the `gc.cruftPacks` configuration unset until all writers understand
104
+
cruft packs.
105
+
106
+
== Alternatives
107
+
108
+
Notable alternatives to this design include:
109
+
110
+
- The location of the per-object mtime data, and
111
+
- Storing unreachable objects in multiple cruft packs.
112
+
113
+
On the location of mtime data, a new auxiliary file tied to the pack was chosen
114
+
to avoid complicating the `.idx` format. If the `.idx` format were ever to gain
115
+
support for optional chunks of data, it may make sense to consolidate the
116
+
`.mtimes` format into the `.idx` itself.
117
+
118
+
Storing unreachable objects among multiple cruft packs (e.g., creating a new
119
+
cruft pack during each repacking operation including only unreachable objects
120
+
which aren't already stored in an earlier cruft pack) is significantly more
121
+
complicated to construct, and so aren't pursued here. The obvious drawback to
122
+
the current implementation is that the entire cruft pack must be re-written from
123
+
scratch.
+19
Documentation/technical/pack-format.txt
+19
Documentation/technical/pack-format.txt
···
294
294
295
295
All 4-byte numbers are in network order.
296
296
297
+
== pack-*.mtimes files have the format:
298
+
299
+
All 4-byte numbers are in network byte order.
300
+
301
+
- A 4-byte magic number '0x4d544d45' ('MTME').
302
+
303
+
- A 4-byte version identifier (= 1).
304
+
305
+
- A 4-byte hash function identifier (= 1 for SHA-1, 2 for SHA-256).
306
+
307
+
- A table of 4-byte unsigned integers. The ith value is the
308
+
modification time (mtime) of the ith object in the corresponding
309
+
pack by lexicographic (index) order. The mtimes count standard
310
+
epoch seconds.
311
+
312
+
- A trailer, containing a checksum of the corresponding packfile,
313
+
and a checksum of all of the above (each having length according
314
+
to the specified hash function).
315
+
297
316
== multi-pack-index (MIDX) files have the following format:
298
317
299
318
The multi-pack-index files refer to multiple pack-files and loose objects.
+2
Makefile
+2
Makefile
···
740
740
TEST_BUILTINS_OBJS += test-oidmap.o
741
741
TEST_BUILTINS_OBJS += test-oidtree.o
742
742
TEST_BUILTINS_OBJS += test-online-cpus.o
743
+
TEST_BUILTINS_OBJS += test-pack-mtimes.o
743
744
TEST_BUILTINS_OBJS += test-parse-options.o
744
745
TEST_BUILTINS_OBJS += test-parse-pathspec-file.o
745
746
TEST_BUILTINS_OBJS += test-partial-clone.o
···
996
997
LIB_OBJS += pack-bitmap-write.o
997
998
LIB_OBJS += pack-bitmap.o
998
999
LIB_OBJS += pack-check.o
1000
+
LIB_OBJS += pack-mtimes.o
999
1001
LIB_OBJS += pack-objects.o
1000
1002
LIB_OBJS += pack-revindex.o
1001
1003
LIB_OBJS += pack-write.o
+9
-1
builtin/gc.c
+9
-1
builtin/gc.c
···
42
42
43
43
static int pack_refs = 1;
44
44
static int prune_reflogs = 1;
45
+
static int cruft_packs = 0;
45
46
static int aggressive_depth = 50;
46
47
static int aggressive_window = 250;
47
48
static int gc_auto_threshold = 6700;
···
152
153
git_config_get_int("gc.auto", &gc_auto_threshold);
153
154
git_config_get_int("gc.autopacklimit", &gc_auto_pack_limit);
154
155
git_config_get_bool("gc.autodetach", &detach_auto);
156
+
git_config_get_bool("gc.cruftpacks", &cruft_packs);
155
157
git_config_get_expiry("gc.pruneexpire", &prune_expire);
156
158
git_config_get_expiry("gc.worktreepruneexpire", &prune_worktrees_expire);
157
159
git_config_get_expiry("gc.logexpiry", &gc_log_expire);
···
331
333
{
332
334
if (prune_expire && !strcmp(prune_expire, "now"))
333
335
strvec_push(&repack, "-a");
334
-
else {
336
+
else if (cruft_packs) {
337
+
strvec_push(&repack, "--cruft");
338
+
if (prune_expire)
339
+
strvec_pushf(&repack, "--cruft-expiration=%s", prune_expire);
340
+
} else {
335
341
strvec_push(&repack, "-A");
336
342
if (prune_expire)
337
343
strvec_pushf(&repack, "--unpack-unreachable=%s", prune_expire);
···
551
557
{ OPTION_STRING, 0, "prune", &prune_expire, N_("date"),
552
558
N_("prune unreferenced objects"),
553
559
PARSE_OPT_OPTARG, NULL, (intptr_t)prune_expire },
560
+
OPT_BOOL(0, "cruft", &cruft_packs, N_("pack unreferenced objects separately")),
554
561
OPT_BOOL(0, "aggressive", &aggressive, N_("be more thorough (increased runtime)")),
555
562
OPT_BOOL_F(0, "auto", &auto_gc, N_("enable auto-gc mode"),
556
563
PARSE_OPT_NOCOMPLETE),
···
670
677
die(FAILED_RUN, repack.v[0]);
671
678
672
679
if (prune_expire) {
680
+
/* run `git prune` even if using cruft packs */
673
681
strvec_push(&prune, prune_expire);
674
682
if (quiet)
675
683
strvec_push(&prune, "--no-progress");
+291
-13
builtin/pack-objects.c
+291
-13
builtin/pack-objects.c
···
36
36
#include "trace2.h"
37
37
#include "shallow.h"
38
38
#include "promisor-remote.h"
39
+
#include "pack-mtimes.h"
39
40
40
41
/*
41
42
* Objects we are going to pack are collected in the `to_pack` structure.
···
194
195
static int keep_unreachable, unpack_unreachable, include_tag;
195
196
static timestamp_t unpack_unreachable_expiration;
196
197
static int pack_loose_unreachable;
198
+
static int cruft;
199
+
static timestamp_t cruft_expiration;
197
200
static int local;
198
201
static int have_non_local_packs;
199
202
static int incremental;
···
1260
1263
&to_pack, written_list, nr_written);
1261
1264
}
1262
1265
1266
+
if (cruft)
1267
+
pack_idx_opts.flags |= WRITE_MTIMES;
1268
+
1263
1269
stage_tmp_packfiles(&tmpname, pack_tmp_name,
1264
1270
written_list, nr_written,
1265
-
&pack_idx_opts, hash, &idx_tmp_name);
1271
+
&to_pack, &pack_idx_opts, hash,
1272
+
&idx_tmp_name);
1266
1273
1267
1274
if (write_bitmap_index) {
1268
1275
size_t tmpname_len = tmpname.len;
···
1521
1528
return 1;
1522
1529
}
1523
1530
1524
-
static void create_object_entry(const struct object_id *oid,
1525
-
enum object_type type,
1526
-
uint32_t hash,
1527
-
int exclude,
1528
-
int no_try_delta,
1529
-
struct packed_git *found_pack,
1530
-
off_t found_offset)
1531
+
static struct object_entry *create_object_entry(const struct object_id *oid,
1532
+
enum object_type type,
1533
+
uint32_t hash,
1534
+
int exclude,
1535
+
int no_try_delta,
1536
+
struct packed_git *found_pack,
1537
+
off_t found_offset)
1531
1538
{
1532
1539
struct object_entry *entry;
1533
1540
···
1544
1551
}
1545
1552
1546
1553
entry->no_try_delta = no_try_delta;
1554
+
1555
+
return entry;
1547
1556
}
1548
1557
1549
1558
static const char no_closure_warning[] = N_(
···
3403
3412
string_list_clear(&exclude_packs, 0);
3404
3413
}
3405
3414
3415
+
static void add_cruft_object_entry(const struct object_id *oid, enum object_type type,
3416
+
struct packed_git *pack, off_t offset,
3417
+
const char *name, uint32_t mtime)
3418
+
{
3419
+
struct object_entry *entry;
3420
+
3421
+
display_progress(progress_state, ++nr_seen);
3422
+
3423
+
entry = packlist_find(&to_pack, oid);
3424
+
if (entry) {
3425
+
if (name) {
3426
+
entry->hash = pack_name_hash(name);
3427
+
entry->no_try_delta = no_try_delta(name);
3428
+
}
3429
+
} else {
3430
+
if (!want_object_in_pack(oid, 0, &pack, &offset))
3431
+
return;
3432
+
if (!pack && type == OBJ_BLOB && !has_loose_object(oid)) {
3433
+
/*
3434
+
* If a traversed tree has a missing blob then we want
3435
+
* to avoid adding that missing object to our pack.
3436
+
*
3437
+
* This only applies to missing blobs, not trees,
3438
+
* because the traversal needs to parse sub-trees but
3439
+
* not blobs.
3440
+
*
3441
+
* Note we only perform this check when we couldn't
3442
+
* already find the object in a pack, so we're really
3443
+
* limited to "ensure non-tip blobs which don't exist in
3444
+
* packs do exist via loose objects". Confused?
3445
+
*/
3446
+
return;
3447
+
}
3448
+
3449
+
entry = create_object_entry(oid, type, pack_name_hash(name),
3450
+
0, name && no_try_delta(name),
3451
+
pack, offset);
3452
+
}
3453
+
3454
+
if (mtime > oe_cruft_mtime(&to_pack, entry))
3455
+
oe_set_cruft_mtime(&to_pack, entry, mtime);
3456
+
return;
3457
+
}
3458
+
3459
+
static void show_cruft_object(struct object *obj, const char *name, void *data)
3460
+
{
3461
+
/*
3462
+
* if we did not record it earlier, it's at least as old as our
3463
+
* expiration value. Rather than find it exactly, just use that
3464
+
* value. This may bump it forward from its real mtime, but it
3465
+
* will still be "too old" next time we run with the same
3466
+
* expiration.
3467
+
*
3468
+
* if obj does appear in the packing list, this call is a noop (or may
3469
+
* set the namehash).
3470
+
*/
3471
+
add_cruft_object_entry(&obj->oid, obj->type, NULL, 0, name, cruft_expiration);
3472
+
}
3473
+
3474
+
static void show_cruft_commit(struct commit *commit, void *data)
3475
+
{
3476
+
show_cruft_object((struct object*)commit, NULL, data);
3477
+
}
3478
+
3479
+
static int cruft_include_check_obj(struct object *obj, void *data)
3480
+
{
3481
+
return !has_object_kept_pack(&obj->oid, IN_CORE_KEEP_PACKS);
3482
+
}
3483
+
3484
+
static int cruft_include_check(struct commit *commit, void *data)
3485
+
{
3486
+
return cruft_include_check_obj((struct object*)commit, data);
3487
+
}
3488
+
3489
+
static void set_cruft_mtime(const struct object *object,
3490
+
struct packed_git *pack,
3491
+
off_t offset, time_t mtime)
3492
+
{
3493
+
add_cruft_object_entry(&object->oid, object->type, pack, offset, NULL,
3494
+
mtime);
3495
+
}
3496
+
3497
+
static void mark_pack_kept_in_core(struct string_list *packs, unsigned keep)
3498
+
{
3499
+
struct string_list_item *item = NULL;
3500
+
for_each_string_list_item(item, packs) {
3501
+
struct packed_git *p = item->util;
3502
+
if (!p)
3503
+
die(_("could not find pack '%s'"), item->string);
3504
+
p->pack_keep_in_core = keep;
3505
+
}
3506
+
}
3507
+
3508
+
static void add_unreachable_loose_objects(void);
3509
+
static void add_objects_in_unpacked_packs(void);
3510
+
3511
+
static void enumerate_cruft_objects(void)
3512
+
{
3513
+
if (progress)
3514
+
progress_state = start_progress(_("Enumerating cruft objects"), 0);
3515
+
3516
+
add_objects_in_unpacked_packs();
3517
+
add_unreachable_loose_objects();
3518
+
3519
+
stop_progress(&progress_state);
3520
+
}
3521
+
3522
+
static void enumerate_and_traverse_cruft_objects(struct string_list *fresh_packs)
3523
+
{
3524
+
struct packed_git *p;
3525
+
struct rev_info revs;
3526
+
int ret;
3527
+
3528
+
repo_init_revisions(the_repository, &revs, NULL);
3529
+
3530
+
revs.tag_objects = 1;
3531
+
revs.tree_objects = 1;
3532
+
revs.blob_objects = 1;
3533
+
3534
+
revs.include_check = cruft_include_check;
3535
+
revs.include_check_obj = cruft_include_check_obj;
3536
+
3537
+
revs.ignore_missing_links = 1;
3538
+
3539
+
if (progress)
3540
+
progress_state = start_progress(_("Enumerating cruft objects"), 0);
3541
+
ret = add_unseen_recent_objects_to_traversal(&revs, cruft_expiration,
3542
+
set_cruft_mtime, 1);
3543
+
stop_progress(&progress_state);
3544
+
3545
+
if (ret)
3546
+
die(_("unable to add cruft objects"));
3547
+
3548
+
/*
3549
+
* Re-mark only the fresh packs as kept so that objects in
3550
+
* unknown packs do not halt the reachability traversal early.
3551
+
*/
3552
+
for (p = get_all_packs(the_repository); p; p = p->next)
3553
+
p->pack_keep_in_core = 0;
3554
+
mark_pack_kept_in_core(fresh_packs, 1);
3555
+
3556
+
if (prepare_revision_walk(&revs))
3557
+
die(_("revision walk setup failed"));
3558
+
if (progress)
3559
+
progress_state = start_progress(_("Traversing cruft objects"), 0);
3560
+
nr_seen = 0;
3561
+
traverse_commit_list(&revs, show_cruft_commit, show_cruft_object, NULL);
3562
+
3563
+
stop_progress(&progress_state);
3564
+
}
3565
+
3566
+
static void read_cruft_objects(void)
3567
+
{
3568
+
struct strbuf buf = STRBUF_INIT;
3569
+
struct string_list discard_packs = STRING_LIST_INIT_DUP;
3570
+
struct string_list fresh_packs = STRING_LIST_INIT_DUP;
3571
+
struct packed_git *p;
3572
+
3573
+
ignore_packed_keep_in_core = 1;
3574
+
3575
+
while (strbuf_getline(&buf, stdin) != EOF) {
3576
+
if (!buf.len)
3577
+
continue;
3578
+
3579
+
if (*buf.buf == '-')
3580
+
string_list_append(&discard_packs, buf.buf + 1);
3581
+
else
3582
+
string_list_append(&fresh_packs, buf.buf);
3583
+
strbuf_reset(&buf);
3584
+
}
3585
+
3586
+
string_list_sort(&discard_packs);
3587
+
string_list_sort(&fresh_packs);
3588
+
3589
+
for (p = get_all_packs(the_repository); p; p = p->next) {
3590
+
const char *pack_name = pack_basename(p);
3591
+
struct string_list_item *item;
3592
+
3593
+
item = string_list_lookup(&fresh_packs, pack_name);
3594
+
if (!item)
3595
+
item = string_list_lookup(&discard_packs, pack_name);
3596
+
3597
+
if (item) {
3598
+
item->util = p;
3599
+
} else {
3600
+
/*
3601
+
* This pack wasn't mentioned in either the "fresh" or
3602
+
* "discard" list, so the caller didn't know about it.
3603
+
*
3604
+
* Mark it as kept so that its objects are ignored by
3605
+
* add_unseen_recent_objects_to_traversal(). We'll
3606
+
* unmark it before starting the traversal so it doesn't
3607
+
* halt the traversal early.
3608
+
*/
3609
+
p->pack_keep_in_core = 1;
3610
+
}
3611
+
}
3612
+
3613
+
mark_pack_kept_in_core(&fresh_packs, 1);
3614
+
mark_pack_kept_in_core(&discard_packs, 0);
3615
+
3616
+
if (cruft_expiration)
3617
+
enumerate_and_traverse_cruft_objects(&fresh_packs);
3618
+
else
3619
+
enumerate_cruft_objects();
3620
+
3621
+
strbuf_release(&buf);
3622
+
string_list_clear(&discard_packs, 0);
3623
+
string_list_clear(&fresh_packs, 0);
3624
+
}
3625
+
3406
3626
static void read_object_list_from_stdin(void)
3407
3627
{
3408
3628
char line[GIT_MAX_HEXSZ + 1 + PATH_MAX + 2];
···
3535
3755
uint32_t pos,
3536
3756
void *_data)
3537
3757
{
3538
-
add_object_entry(oid, OBJ_NONE, "", 0);
3758
+
if (cruft) {
3759
+
off_t offset;
3760
+
time_t mtime;
3761
+
3762
+
if (pack->is_cruft) {
3763
+
if (load_pack_mtimes(pack) < 0)
3764
+
die(_("could not load cruft pack .mtimes"));
3765
+
mtime = nth_packed_mtime(pack, pos);
3766
+
} else {
3767
+
mtime = pack->mtime;
3768
+
}
3769
+
offset = nth_packed_object_offset(pack, pos);
3770
+
3771
+
add_cruft_object_entry(oid, OBJ_NONE, pack, offset,
3772
+
NULL, mtime);
3773
+
} else {
3774
+
add_object_entry(oid, OBJ_NONE, "", 0);
3775
+
}
3539
3776
return 0;
3540
3777
}
3541
3778
···
3559
3796
return 0;
3560
3797
}
3561
3798
3562
-
add_object_entry(oid, type, "", 0);
3799
+
if (cruft) {
3800
+
struct stat st;
3801
+
if (stat(path, &st) < 0) {
3802
+
if (errno == ENOENT)
3803
+
return 0;
3804
+
return error_errno("unable to stat %s", oid_to_hex(oid));
3805
+
}
3806
+
3807
+
add_cruft_object_entry(oid, type, NULL, 0, NULL,
3808
+
st.st_mtime);
3809
+
} else {
3810
+
add_object_entry(oid, type, "", 0);
3811
+
}
3563
3812
return 0;
3564
3813
}
3565
3814
···
3799
4048
if (unpack_unreachable_expiration) {
3800
4049
revs->ignore_missing_links = 1;
3801
4050
if (add_unseen_recent_objects_to_traversal(revs,
3802
-
unpack_unreachable_expiration))
4051
+
unpack_unreachable_expiration, NULL, 0))
3803
4052
die(_("unable to add recent objects"));
3804
4053
if (prepare_revision_walk(revs))
3805
4054
die(_("revision walk setup failed"));
···
3872
4121
unpack_unreachable = 1;
3873
4122
if (arg)
3874
4123
unpack_unreachable_expiration = approxidate(arg);
4124
+
}
4125
+
return 0;
4126
+
}
4127
+
4128
+
static int option_parse_cruft_expiration(const struct option *opt,
4129
+
const char *arg, int unset)
4130
+
{
4131
+
if (unset) {
4132
+
cruft = 0;
4133
+
cruft_expiration = 0;
4134
+
} else {
4135
+
cruft = 1;
4136
+
if (arg)
4137
+
cruft_expiration = approxidate(arg);
3875
4138
}
3876
4139
return 0;
3877
4140
}
···
3965
4228
OPT_CALLBACK_F(0, "unpack-unreachable", NULL, N_("time"),
3966
4229
N_("unpack unreachable objects newer than <time>"),
3967
4230
PARSE_OPT_OPTARG, option_parse_unpack_unreachable),
4231
+
OPT_BOOL(0, "cruft", &cruft, N_("create a cruft pack")),
4232
+
OPT_CALLBACK_F(0, "cruft-expiration", NULL, N_("time"),
4233
+
N_("expire cruft objects older than <time>"),
4234
+
PARSE_OPT_OPTARG, option_parse_cruft_expiration),
3968
4235
OPT_BOOL(0, "sparse", &sparse,
3969
4236
N_("use the sparse reachability algorithm")),
3970
4237
OPT_BOOL(0, "thin", &thin,
···
4091
4358
4092
4359
if (!HAVE_THREADS && delta_search_threads != 1)
4093
4360
warning(_("no threads support, ignoring --threads"));
4094
-
if (!pack_to_stdout && !pack_size_limit)
4361
+
if (!pack_to_stdout && !pack_size_limit && !cruft)
4095
4362
pack_size_limit = pack_size_limit_cfg;
4096
4363
if (pack_to_stdout && pack_size_limit)
4097
4364
die(_("--max-pack-size cannot be used to build a pack for transfer"));
···
4117
4384
4118
4385
if (stdin_packs && use_internal_rev_list)
4119
4386
die(_("cannot use internal rev list with --stdin-packs"));
4387
+
4388
+
if (cruft) {
4389
+
if (use_internal_rev_list)
4390
+
die(_("cannot use internal rev list with --cruft"));
4391
+
if (stdin_packs)
4392
+
die(_("cannot use --stdin-packs with --cruft"));
4393
+
if (pack_size_limit)
4394
+
die(_("cannot use --max-pack-size with --cruft"));
4395
+
}
4120
4396
4121
4397
/*
4122
4398
* "soft" reasons not to use bitmaps - for on-disk repack by default we want
···
4174
4450
the_repository);
4175
4451
prepare_packing_data(the_repository, &to_pack);
4176
4452
4177
-
if (progress)
4453
+
if (progress && !cruft)
4178
4454
progress_state = start_progress(_("Enumerating objects"), 0);
4179
4455
if (stdin_packs) {
4180
4456
/* avoids adding objects in excluded packs */
···
4182
4458
read_packs_list_from_stdin();
4183
4459
if (rev_list_unpacked)
4184
4460
add_unreachable_loose_objects();
4461
+
} else if (cruft) {
4462
+
read_cruft_objects();
4185
4463
} else if (!use_internal_rev_list) {
4186
4464
read_object_list_from_stdin();
4187
4465
} else if (pfd.have_revs) {
+161
-24
builtin/repack.c
+161
-24
builtin/repack.c
···
18
18
#include "pack-bitmap.h"
19
19
#include "refs.h"
20
20
21
+
#define ALL_INTO_ONE 1
22
+
#define LOOSEN_UNREACHABLE 2
23
+
#define PACK_CRUFT 4
24
+
25
+
#define DELETE_PACK 1
26
+
#define CRUFT_PACK 2
27
+
28
+
static int pack_everything;
21
29
static int delta_base_offset = 1;
22
30
static int pack_kept_objects = -1;
23
31
static int write_bitmaps = -1;
24
32
static int use_delta_islands;
25
33
static int run_update_server_info = 1;
26
34
static char *packdir, *packtmp_name, *packtmp;
35
+
static char *cruft_expiration;
27
36
28
37
static const char *const git_repack_usage[] = {
29
38
N_("git repack [<options>]"),
···
35
44
"--no-write-bitmap-index or disable the pack.writebitmaps configuration."
36
45
);
37
46
47
+
struct pack_objects_args {
48
+
const char *window;
49
+
const char *window_memory;
50
+
const char *depth;
51
+
const char *threads;
52
+
const char *max_pack_size;
53
+
int no_reuse_delta;
54
+
int no_reuse_object;
55
+
int quiet;
56
+
int local;
57
+
};
38
58
39
59
static int repack_config(const char *var, const char *value, void *cb)
40
60
{
61
+
struct pack_objects_args *cruft_po_args = cb;
41
62
if (!strcmp(var, "repack.usedeltabaseoffset")) {
42
63
delta_base_offset = git_config_bool(var, value);
43
64
return 0;
···
59
80
run_update_server_info = git_config_bool(var, value);
60
81
return 0;
61
82
}
83
+
if (!strcmp(var, "repack.cruftwindow"))
84
+
return git_config_string(&cruft_po_args->window, var, value);
85
+
if (!strcmp(var, "repack.cruftwindowmemory"))
86
+
return git_config_string(&cruft_po_args->window_memory, var, value);
87
+
if (!strcmp(var, "repack.cruftdepth"))
88
+
return git_config_string(&cruft_po_args->depth, var, value);
89
+
if (!strcmp(var, "repack.cruftthreads"))
90
+
return git_config_string(&cruft_po_args->threads, var, value);
62
91
return git_default_config(var, value, cb);
63
92
}
64
93
···
131
160
fname = xmemdupz(e->d_name, len);
132
161
133
162
if ((extra_keep->nr > 0 && i < extra_keep->nr) ||
134
-
(file_exists(mkpath("%s/%s.keep", packdir, fname))))
163
+
(file_exists(mkpath("%s/%s.keep", packdir, fname)))) {
135
164
string_list_append_nodup(fname_kept_list, fname);
136
-
else
137
-
string_list_append_nodup(fname_nonkept_list, fname);
165
+
} else {
166
+
struct string_list_item *item;
167
+
item = string_list_append_nodup(fname_nonkept_list,
168
+
fname);
169
+
if (file_exists(mkpath("%s/%s.mtimes", packdir, fname)))
170
+
item->util = (void*)(uintptr_t)CRUFT_PACK;
171
+
}
138
172
}
139
173
closedir(dir);
140
174
···
153
187
strbuf_release(&buf);
154
188
}
155
189
156
-
struct pack_objects_args {
157
-
const char *window;
158
-
const char *window_memory;
159
-
const char *depth;
160
-
const char *threads;
161
-
const char *max_pack_size;
162
-
int no_reuse_delta;
163
-
int no_reuse_object;
164
-
int quiet;
165
-
int local;
166
-
};
167
-
168
190
static void prepare_pack_objects(struct child_process *cmd,
169
191
const struct pack_objects_args *args)
170
192
{
···
219
241
} exts[] = {
220
242
{".pack"},
221
243
{".rev", 1},
244
+
{".mtimes", 1},
222
245
{".bitmap", 1},
223
246
{".promisor", 1},
224
247
{".idx"},
···
306
329
die(_("could not finish pack-objects to repack promisor objects"));
307
330
}
308
331
309
-
#define ALL_INTO_ONE 1
310
-
#define LOOSEN_UNREACHABLE 2
311
-
312
332
struct pack_geometry {
313
333
struct packed_git **pack;
314
334
uint32_t pack_nr, pack_alloc;
···
366
386
if (string_list_has_string(existing_kept_packs, buf.buf))
367
387
continue;
368
388
}
389
+
if (p->is_cruft)
390
+
continue;
369
391
370
392
ALLOC_GROW(geometry->pack,
371
393
geometry->pack_nr + 1,
···
572
594
573
595
string_list_insert(include, strbuf_detach(&buf, NULL));
574
596
}
597
+
598
+
for_each_string_list_item(item, existing_nonkept_packs) {
599
+
if (!((uintptr_t)item->util & CRUFT_PACK)) {
600
+
/*
601
+
* no need to check DELETE_PACK, since we're not
602
+
* doing an ALL_INTO_ONE repack
603
+
*/
604
+
continue;
605
+
}
606
+
string_list_insert(include, xstrfmt("%s.idx", item->string));
607
+
}
575
608
} else {
576
609
for_each_string_list_item(item, existing_nonkept_packs) {
577
-
if (item->util)
610
+
if ((uintptr_t)item->util & DELETE_PACK)
578
611
continue;
579
612
string_list_insert(include, xstrfmt("%s.idx", item->string));
580
613
}
···
628
661
return finish_command(&cmd);
629
662
}
630
663
664
+
static int write_cruft_pack(const struct pack_objects_args *args,
665
+
const char *pack_prefix,
666
+
struct string_list *names,
667
+
struct string_list *existing_packs,
668
+
struct string_list *existing_kept_packs)
669
+
{
670
+
struct child_process cmd = CHILD_PROCESS_INIT;
671
+
struct strbuf line = STRBUF_INIT;
672
+
struct string_list_item *item;
673
+
FILE *in, *out;
674
+
int ret;
675
+
676
+
prepare_pack_objects(&cmd, args);
677
+
678
+
strvec_push(&cmd.args, "--cruft");
679
+
if (cruft_expiration)
680
+
strvec_pushf(&cmd.args, "--cruft-expiration=%s",
681
+
cruft_expiration);
682
+
683
+
strvec_push(&cmd.args, "--honor-pack-keep");
684
+
strvec_push(&cmd.args, "--non-empty");
685
+
strvec_push(&cmd.args, "--max-pack-size=0");
686
+
687
+
cmd.in = -1;
688
+
689
+
ret = start_command(&cmd);
690
+
if (ret)
691
+
return ret;
692
+
693
+
/*
694
+
* names has a confusing double use: it both provides the list
695
+
* of just-written new packs, and accepts the name of the cruft
696
+
* pack we are writing.
697
+
*
698
+
* By the time it is read here, it contains only the pack(s)
699
+
* that were just written, which is exactly the set of packs we
700
+
* want to consider kept.
701
+
*/
702
+
in = xfdopen(cmd.in, "w");
703
+
for_each_string_list_item(item, names)
704
+
fprintf(in, "%s-%s.pack\n", pack_prefix, item->string);
705
+
for_each_string_list_item(item, existing_packs)
706
+
fprintf(in, "-%s.pack\n", item->string);
707
+
for_each_string_list_item(item, existing_kept_packs)
708
+
fprintf(in, "%s.pack\n", item->string);
709
+
fclose(in);
710
+
711
+
out = xfdopen(cmd.out, "r");
712
+
while (strbuf_getline_lf(&line, out) != EOF) {
713
+
if (line.len != the_hash_algo->hexsz)
714
+
die(_("repack: Expecting full hex object ID lines only "
715
+
"from pack-objects."));
716
+
string_list_append(names, line.buf);
717
+
}
718
+
fclose(out);
719
+
720
+
strbuf_release(&line);
721
+
722
+
return finish_command(&cmd);
723
+
}
724
+
631
725
int cmd_repack(int argc, const char **argv, const char *prefix)
632
726
{
633
727
struct child_process cmd = CHILD_PROCESS_INIT;
···
644
738
int show_progress;
645
739
646
740
/* variables to be filled by option parsing */
647
-
int pack_everything = 0;
648
741
int delete_redundant = 0;
649
742
const char *unpack_unreachable = NULL;
650
743
int keep_unreachable = 0;
651
744
struct string_list keep_pack_list = STRING_LIST_INIT_NODUP;
652
745
struct pack_objects_args po_args = {NULL};
746
+
struct pack_objects_args cruft_po_args = {NULL};
653
747
int geometric_factor = 0;
654
748
int write_midx = 0;
655
749
···
659
753
OPT_BIT('A', NULL, &pack_everything,
660
754
N_("same as -a, and turn unreachable objects loose"),
661
755
LOOSEN_UNREACHABLE | ALL_INTO_ONE),
756
+
OPT_BIT(0, "cruft", &pack_everything,
757
+
N_("same as -a, pack unreachable cruft objects separately"),
758
+
PACK_CRUFT),
759
+
OPT_STRING(0, "cruft-expiration", &cruft_expiration, N_("approxidate"),
760
+
N_("with -C, expire objects older than this")),
662
761
OPT_BOOL('d', NULL, &delete_redundant,
663
762
N_("remove redundant packs, and run git-prune-packed")),
664
763
OPT_BOOL('f', NULL, &po_args.no_reuse_delta,
···
699
798
OPT_END()
700
799
};
701
800
702
-
git_config(repack_config, NULL);
801
+
git_config(repack_config, &cruft_po_args);
703
802
704
803
argc = parse_options(argc, argv, prefix, builtin_repack_options,
705
804
git_repack_usage, 0);
···
710
809
if (keep_unreachable &&
711
810
(unpack_unreachable || (pack_everything & LOOSEN_UNREACHABLE)))
712
811
die(_("options '%s' and '%s' cannot be used together"), "--keep-unreachable", "-A");
812
+
813
+
if (pack_everything & PACK_CRUFT) {
814
+
pack_everything |= ALL_INTO_ONE;
815
+
816
+
if (unpack_unreachable || (pack_everything & LOOSEN_UNREACHABLE))
817
+
die(_("options '%s' and '%s' cannot be used together"), "--cruft", "-A");
818
+
if (keep_unreachable)
819
+
die(_("options '%s' and '%s' cannot be used together"), "--cruft", "-k");
820
+
}
713
821
714
822
if (write_bitmaps < 0) {
715
823
if (!write_midx &&
···
794
902
if (pack_everything & ALL_INTO_ONE) {
795
903
repack_promisor_objects(&po_args, &names);
796
904
797
-
if (existing_nonkept_packs.nr && delete_redundant) {
905
+
if (existing_nonkept_packs.nr && delete_redundant &&
906
+
!(pack_everything & PACK_CRUFT)) {
798
907
for_each_string_list_item(item, &names) {
799
908
strvec_pushf(&cmd.args, "--keep-pack=%s-%s.pack",
800
909
packtmp_name, item->string);
···
856
965
if (!names.nr && !po_args.quiet)
857
966
printf_ln(_("Nothing new to pack."));
858
967
968
+
if (pack_everything & PACK_CRUFT) {
969
+
const char *pack_prefix;
970
+
if (!skip_prefix(packtmp, packdir, &pack_prefix))
971
+
die(_("pack prefix %s does not begin with objdir %s"),
972
+
packtmp, packdir);
973
+
if (*pack_prefix == '/')
974
+
pack_prefix++;
975
+
976
+
if (!cruft_po_args.window)
977
+
cruft_po_args.window = po_args.window;
978
+
if (!cruft_po_args.window_memory)
979
+
cruft_po_args.window_memory = po_args.window_memory;
980
+
if (!cruft_po_args.depth)
981
+
cruft_po_args.depth = po_args.depth;
982
+
if (!cruft_po_args.threads)
983
+
cruft_po_args.threads = po_args.threads;
984
+
985
+
cruft_po_args.local = po_args.local;
986
+
cruft_po_args.quiet = po_args.quiet;
987
+
988
+
ret = write_cruft_pack(&cruft_po_args, pack_prefix, &names,
989
+
&existing_nonkept_packs,
990
+
&existing_kept_packs);
991
+
if (ret)
992
+
return ret;
993
+
}
994
+
859
995
string_list_sort(&names);
860
996
861
997
for_each_string_list_item(item, &names) {
···
910
1046
* was given) and that we will actually delete this pack
911
1047
* (if `-d` was given).
912
1048
*/
913
-
item->util = (void*)(intptr_t)!string_list_has_string(&names, sha1);
1049
+
if (!string_list_has_string(&names, sha1))
1050
+
item->util = (void*)(uintptr_t)((size_t)item->util | DELETE_PACK);
914
1051
}
915
1052
}
916
1053
···
934
1071
if (delete_redundant) {
935
1072
int opts = 0;
936
1073
for_each_string_list_item(item, &existing_nonkept_packs) {
937
-
if (!item->util)
1074
+
if (!((uintptr_t)item->util & DELETE_PACK))
938
1075
continue;
939
1076
remove_redundant_pack(packdir, item->string);
940
1077
}
+1
-1
bulk-checkin.c
+1
-1
bulk-checkin.c
+12
chunk-format.c
+12
chunk-format.c
···
181
181
182
182
return CHUNK_NOT_FOUND;
183
183
}
184
+
185
+
uint8_t oid_version(const struct git_hash_algo *algop)
186
+
{
187
+
switch (hash_algo_by_ptr(algop)) {
188
+
case GIT_HASH_SHA1:
189
+
return 1;
190
+
case GIT_HASH_SHA256:
191
+
return 2;
192
+
default:
193
+
die(_("invalid hash version"));
194
+
}
195
+
}
+3
chunk-format.h
+3
chunk-format.h
+3
-15
commit-graph.c
+3
-15
commit-graph.c
···
193
193
return xstrfmt("%s/info/commit-graphs/commit-graph-chain", odb->path);
194
194
}
195
195
196
-
static uint8_t oid_version(void)
197
-
{
198
-
switch (hash_algo_by_ptr(the_hash_algo)) {
199
-
case GIT_HASH_SHA1:
200
-
return 1;
201
-
case GIT_HASH_SHA256:
202
-
return 2;
203
-
default:
204
-
die(_("invalid hash version"));
205
-
}
206
-
}
207
-
208
196
static struct commit_graph *alloc_commit_graph(void)
209
197
{
210
198
struct commit_graph *g = xcalloc(1, sizeof(*g));
···
365
353
}
366
354
367
355
hash_version = *(unsigned char*)(data + 5);
368
-
if (hash_version != oid_version()) {
356
+
if (hash_version != oid_version(the_hash_algo)) {
369
357
error(_("commit-graph hash version %X does not match version %X"),
370
-
hash_version, oid_version());
358
+
hash_version, oid_version(the_hash_algo));
371
359
return NULL;
372
360
}
373
361
···
1924
1912
hashwrite_be32(f, GRAPH_SIGNATURE);
1925
1913
1926
1914
hashwrite_u8(f, GRAPH_VERSION);
1927
-
hashwrite_u8(f, oid_version());
1915
+
hashwrite_u8(f, oid_version(the_hash_algo));
1928
1916
hashwrite_u8(f, get_num_chunks(cf));
1929
1917
hashwrite_u8(f, ctx->num_commit_graphs_after - 1);
1930
1918
+3
-15
midx.c
+3
-15
midx.c
···
41
41
42
42
#define PACK_EXPIRED UINT_MAX
43
43
44
-
static uint8_t oid_version(void)
45
-
{
46
-
switch (hash_algo_by_ptr(the_hash_algo)) {
47
-
case GIT_HASH_SHA1:
48
-
return 1;
49
-
case GIT_HASH_SHA256:
50
-
return 2;
51
-
default:
52
-
die(_("invalid hash version"));
53
-
}
54
-
}
55
-
56
44
const unsigned char *get_midx_checksum(struct multi_pack_index *m)
57
45
{
58
46
return m->data + m->data_len - the_hash_algo->rawsz;
···
134
122
m->version);
135
123
136
124
hash_version = m->data[MIDX_BYTE_HASH_VERSION];
137
-
if (hash_version != oid_version()) {
125
+
if (hash_version != oid_version(the_hash_algo)) {
138
126
error(_("multi-pack-index hash version %u does not match version %u"),
139
-
hash_version, oid_version());
127
+
hash_version, oid_version(the_hash_algo));
140
128
goto cleanup_fail;
141
129
}
142
130
m->hash_len = the_hash_algo->rawsz;
···
420
408
{
421
409
hashwrite_be32(f, MIDX_SIGNATURE);
422
410
hashwrite_u8(f, MIDX_VERSION);
423
-
hashwrite_u8(f, oid_version());
411
+
hashwrite_u8(f, oid_version(the_hash_algo));
424
412
hashwrite_u8(f, num_chunks);
425
413
hashwrite_u8(f, 0); /* unused */
426
414
hashwrite_be32(f, num_packs);
+3
-1
object-file.c
+3
-1
object-file.c
···
997
997
return check_and_freshen_nonlocal(oid, 0);
998
998
}
999
999
1000
-
static int has_loose_object(const struct object_id *oid)
1000
+
int has_loose_object(const struct object_id *oid)
1001
1001
{
1002
1002
return check_and_freshen(oid, 0);
1003
1003
}
···
2039
2039
{
2040
2040
struct pack_entry e;
2041
2041
if (!find_pack_entry(the_repository, oid, &e))
2042
+
return 0;
2043
+
if (e.p->is_cruft)
2042
2044
return 0;
2043
2045
if (e.p->freshened)
2044
2046
return 1;
+11
-1
object-store.h
+11
-1
object-store.h
···
115
115
freshened:1,
116
116
do_not_close:1,
117
117
pack_promisor:1,
118
-
multi_pack_index:1;
118
+
multi_pack_index:1,
119
+
is_cruft:1;
119
120
unsigned char hash[GIT_MAX_RAWSZ];
120
121
struct revindex_entry *revindex;
121
122
const uint32_t *revindex_data;
122
123
const uint32_t *revindex_map;
123
124
size_t revindex_size;
125
+
/*
126
+
* mtimes_map points at the beginning of the memory mapped region of
127
+
* this pack's corresponding .mtimes file, and mtimes_size is the size
128
+
* of that .mtimes file
129
+
*/
130
+
const uint32_t *mtimes_map;
131
+
size_t mtimes_size;
124
132
/* something like ".git/objects/pack/xxxxx.pack" */
125
133
char pack_name[FLEX_ARRAY]; /* more */
126
134
};
···
326
334
* references.
327
335
*/
328
336
int has_loose_object_nonlocal(const struct object_id *);
337
+
338
+
int has_loose_object(const struct object_id *);
329
339
330
340
/**
331
341
* format_object_header() is a thin wrapper around s xsnprintf() that
+129
pack-mtimes.c
+129
pack-mtimes.c
···
1
+
#include "git-compat-util.h"
2
+
#include "pack-mtimes.h"
3
+
#include "object-store.h"
4
+
#include "packfile.h"
5
+
6
+
static char *pack_mtimes_filename(struct packed_git *p)
7
+
{
8
+
size_t len;
9
+
if (!strip_suffix(p->pack_name, ".pack", &len))
10
+
BUG("pack_name does not end in .pack");
11
+
return xstrfmt("%.*s.mtimes", (int)len, p->pack_name);
12
+
}
13
+
14
+
#define MTIMES_HEADER_SIZE (12)
15
+
16
+
struct mtimes_header {
17
+
uint32_t signature;
18
+
uint32_t version;
19
+
uint32_t hash_id;
20
+
};
21
+
22
+
static int load_pack_mtimes_file(char *mtimes_file,
23
+
uint32_t num_objects,
24
+
const uint32_t **data_p, size_t *len_p)
25
+
{
26
+
int fd, ret = 0;
27
+
struct stat st;
28
+
uint32_t *data = NULL;
29
+
size_t mtimes_size, expected_size;
30
+
struct mtimes_header header;
31
+
32
+
fd = git_open(mtimes_file);
33
+
34
+
if (fd < 0) {
35
+
ret = -1;
36
+
goto cleanup;
37
+
}
38
+
if (fstat(fd, &st)) {
39
+
ret = error_errno(_("failed to read %s"), mtimes_file);
40
+
goto cleanup;
41
+
}
42
+
43
+
mtimes_size = xsize_t(st.st_size);
44
+
45
+
if (mtimes_size < MTIMES_HEADER_SIZE) {
46
+
ret = error(_("mtimes file %s is too small"), mtimes_file);
47
+
goto cleanup;
48
+
}
49
+
50
+
data = xmmap(NULL, mtimes_size, PROT_READ, MAP_PRIVATE, fd, 0);
51
+
52
+
header.signature = ntohl(data[0]);
53
+
header.version = ntohl(data[1]);
54
+
header.hash_id = ntohl(data[2]);
55
+
56
+
if (header.signature != MTIMES_SIGNATURE) {
57
+
ret = error(_("mtimes file %s has unknown signature"), mtimes_file);
58
+
goto cleanup;
59
+
}
60
+
61
+
if (header.version != 1) {
62
+
ret = error(_("mtimes file %s has unsupported version %"PRIu32),
63
+
mtimes_file, header.version);
64
+
goto cleanup;
65
+
}
66
+
67
+
if (!(header.hash_id == 1 || header.hash_id == 2)) {
68
+
ret = error(_("mtimes file %s has unsupported hash id %"PRIu32),
69
+
mtimes_file, header.hash_id);
70
+
goto cleanup;
71
+
}
72
+
73
+
74
+
expected_size = MTIMES_HEADER_SIZE;
75
+
expected_size = st_add(expected_size, st_mult(sizeof(uint32_t), num_objects));
76
+
expected_size = st_add(expected_size, 2 * (header.hash_id == 1 ? GIT_SHA1_RAWSZ : GIT_SHA256_RAWSZ));
77
+
78
+
if (mtimes_size != expected_size) {
79
+
ret = error(_("mtimes file %s is corrupt"), mtimes_file);
80
+
goto cleanup;
81
+
}
82
+
83
+
cleanup:
84
+
if (ret) {
85
+
if (data)
86
+
munmap(data, mtimes_size);
87
+
} else {
88
+
*len_p = mtimes_size;
89
+
*data_p = data;
90
+
}
91
+
92
+
close(fd);
93
+
return ret;
94
+
}
95
+
96
+
int load_pack_mtimes(struct packed_git *p)
97
+
{
98
+
char *mtimes_name = NULL;
99
+
int ret = 0;
100
+
101
+
if (!p->is_cruft)
102
+
return ret; /* not a cruft pack */
103
+
if (p->mtimes_map)
104
+
return ret; /* already loaded */
105
+
106
+
ret = open_pack_index(p);
107
+
if (ret < 0)
108
+
goto cleanup;
109
+
110
+
mtimes_name = pack_mtimes_filename(p);
111
+
ret = load_pack_mtimes_file(mtimes_name,
112
+
p->num_objects,
113
+
&p->mtimes_map,
114
+
&p->mtimes_size);
115
+
cleanup:
116
+
free(mtimes_name);
117
+
return ret;
118
+
}
119
+
120
+
uint32_t nth_packed_mtime(struct packed_git *p, uint32_t pos)
121
+
{
122
+
if (!p->mtimes_map)
123
+
BUG("pack .mtimes file not loaded for %s", p->pack_name);
124
+
if (p->num_objects <= pos)
125
+
BUG("pack .mtimes out-of-bounds (%"PRIu32" vs %"PRIu32")",
126
+
pos, p->num_objects);
127
+
128
+
return get_be32(p->mtimes_map + pos + 3);
129
+
}
+26
pack-mtimes.h
+26
pack-mtimes.h
···
1
+
#ifndef PACK_MTIMES_H
2
+
#define PACK_MTIMES_H
3
+
4
+
#include "git-compat-util.h"
5
+
6
+
#define MTIMES_SIGNATURE 0x4d544d45 /* "MTME" */
7
+
#define MTIMES_VERSION 1
8
+
9
+
struct packed_git;
10
+
11
+
/*
12
+
* Loads the .mtimes file corresponding to "p", if any, returning zero
13
+
* on success.
14
+
*/
15
+
int load_pack_mtimes(struct packed_git *p);
16
+
17
+
/* Returns the mtime associated with the object at position "pos" (in
18
+
* lexicographic/index order) in pack "p".
19
+
*
20
+
* Note that it is a BUG() to call this function if either (a) "p" does
21
+
* not have a corresponding .mtimes file, or (b) it does, but it hasn't
22
+
* been loaded
23
+
*/
24
+
uint32_t nth_packed_mtime(struct packed_git *p, uint32_t pos);
25
+
26
+
#endif
+6
pack-objects.c
+6
pack-objects.c
···
170
170
171
171
if (pdata->layer)
172
172
REALLOC_ARRAY(pdata->layer, pdata->nr_alloc);
173
+
174
+
if (pdata->cruft_mtime)
175
+
REALLOC_ARRAY(pdata->cruft_mtime, pdata->nr_alloc);
173
176
}
174
177
175
178
new_entry = pdata->objects + pdata->nr_objects++;
···
197
200
198
201
if (pdata->layer)
199
202
pdata->layer[pdata->nr_objects - 1] = 0;
203
+
204
+
if (pdata->cruft_mtime)
205
+
pdata->cruft_mtime[pdata->nr_objects - 1] = 0;
200
206
201
207
return new_entry;
202
208
}
+25
pack-objects.h
+25
pack-objects.h
···
168
168
/* delta islands */
169
169
unsigned int *tree_depth;
170
170
unsigned char *layer;
171
+
172
+
/*
173
+
* Used when writing cruft packs.
174
+
*
175
+
* Object mtimes are stored in pack order when writing, but
176
+
* written out in lexicographic (index) order.
177
+
*/
178
+
uint32_t *cruft_mtime;
171
179
};
172
180
173
181
void prepare_packing_data(struct repository *r, struct packing_data *pdata);
···
287
295
if (!pack->layer)
288
296
CALLOC_ARRAY(pack->layer, pack->nr_alloc);
289
297
pack->layer[e - pack->objects] = layer;
298
+
}
299
+
300
+
static inline uint32_t oe_cruft_mtime(struct packing_data *pack,
301
+
struct object_entry *e)
302
+
{
303
+
if (!pack->cruft_mtime)
304
+
return 0;
305
+
return pack->cruft_mtime[e - pack->objects];
306
+
}
307
+
308
+
static inline void oe_set_cruft_mtime(struct packing_data *pack,
309
+
struct object_entry *e,
310
+
uint32_t mtime)
311
+
{
312
+
if (!pack->cruft_mtime)
313
+
CALLOC_ARRAY(pack->cruft_mtime, pack->nr_alloc);
314
+
pack->cruft_mtime[e - pack->objects] = mtime;
290
315
}
291
316
292
317
#endif
+80
-13
pack-write.c
+80
-13
pack-write.c
···
2
2
#include "pack.h"
3
3
#include "csum-file.h"
4
4
#include "remote.h"
5
+
#include "chunk-format.h"
6
+
#include "pack-mtimes.h"
7
+
#include "oidmap.h"
8
+
#include "chunk-format.h"
9
+
#include "pack-objects.h"
5
10
6
11
void reset_pack_idx_option(struct pack_idx_option *opts)
7
12
{
···
181
186
182
187
static void write_rev_header(struct hashfile *f)
183
188
{
184
-
uint32_t oid_version;
185
-
switch (hash_algo_by_ptr(the_hash_algo)) {
186
-
case GIT_HASH_SHA1:
187
-
oid_version = 1;
188
-
break;
189
-
case GIT_HASH_SHA256:
190
-
oid_version = 2;
191
-
break;
192
-
default:
193
-
die("write_rev_header: unknown hash version");
194
-
}
195
-
196
189
hashwrite_be32(f, RIDX_SIGNATURE);
197
190
hashwrite_be32(f, RIDX_VERSION);
198
-
hashwrite_be32(f, oid_version);
191
+
hashwrite_be32(f, oid_version(the_hash_algo));
199
192
}
200
193
201
194
static void write_rev_index_positions(struct hashfile *f,
···
288
281
return rev_name;
289
282
}
290
283
284
+
static void write_mtimes_header(struct hashfile *f)
285
+
{
286
+
hashwrite_be32(f, MTIMES_SIGNATURE);
287
+
hashwrite_be32(f, MTIMES_VERSION);
288
+
hashwrite_be32(f, oid_version(the_hash_algo));
289
+
}
290
+
291
+
/*
292
+
* Writes the object mtimes of "objects" for use in a .mtimes file.
293
+
* Note that objects must be in lexicographic (index) order, which is
294
+
* the expected ordering of these values in the .mtimes file.
295
+
*/
296
+
static void write_mtimes_objects(struct hashfile *f,
297
+
struct packing_data *to_pack,
298
+
struct pack_idx_entry **objects,
299
+
uint32_t nr_objects)
300
+
{
301
+
uint32_t i;
302
+
for (i = 0; i < nr_objects; i++) {
303
+
struct object_entry *e = (struct object_entry*)objects[i];
304
+
hashwrite_be32(f, oe_cruft_mtime(to_pack, e));
305
+
}
306
+
}
307
+
308
+
static void write_mtimes_trailer(struct hashfile *f, const unsigned char *hash)
309
+
{
310
+
hashwrite(f, hash, the_hash_algo->rawsz);
311
+
}
312
+
313
+
static const char *write_mtimes_file(const char *mtimes_name,
314
+
struct packing_data *to_pack,
315
+
struct pack_idx_entry **objects,
316
+
uint32_t nr_objects,
317
+
const unsigned char *hash)
318
+
{
319
+
struct hashfile *f;
320
+
int fd;
321
+
322
+
if (!to_pack)
323
+
BUG("cannot call write_mtimes_file with NULL packing_data");
324
+
325
+
if (!mtimes_name) {
326
+
struct strbuf tmp_file = STRBUF_INIT;
327
+
fd = odb_mkstemp(&tmp_file, "pack/tmp_mtimes_XXXXXX");
328
+
mtimes_name = strbuf_detach(&tmp_file, NULL);
329
+
} else {
330
+
unlink(mtimes_name);
331
+
fd = xopen(mtimes_name, O_CREAT|O_EXCL|O_WRONLY, 0600);
332
+
}
333
+
f = hashfd(fd, mtimes_name);
334
+
335
+
write_mtimes_header(f);
336
+
write_mtimes_objects(f, to_pack, objects, nr_objects);
337
+
write_mtimes_trailer(f, hash);
338
+
339
+
if (adjust_shared_perm(mtimes_name) < 0)
340
+
die(_("failed to make %s readable"), mtimes_name);
341
+
342
+
finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
343
+
CSUM_HASH_IN_STREAM | CSUM_CLOSE | CSUM_FSYNC);
344
+
345
+
return mtimes_name;
346
+
}
347
+
291
348
off_t write_pack_header(struct hashfile *f, uint32_t nr_entries)
292
349
{
293
350
struct pack_header hdr;
···
484
541
const char *pack_tmp_name,
485
542
struct pack_idx_entry **written_list,
486
543
uint32_t nr_written,
544
+
struct packing_data *to_pack,
487
545
struct pack_idx_option *pack_idx_opts,
488
546
unsigned char hash[],
489
547
char **idx_tmp_name)
490
548
{
491
549
const char *rev_tmp_name = NULL;
550
+
const char *mtimes_tmp_name = NULL;
492
551
493
552
if (adjust_shared_perm(pack_tmp_name))
494
553
die_errno("unable to make temporary pack file readable");
···
501
560
rev_tmp_name = write_rev_file(NULL, written_list, nr_written, hash,
502
561
pack_idx_opts->flags);
503
562
563
+
if (pack_idx_opts->flags & WRITE_MTIMES) {
564
+
mtimes_tmp_name = write_mtimes_file(NULL, to_pack, written_list,
565
+
nr_written,
566
+
hash);
567
+
}
568
+
504
569
rename_tmp_packfile(name_buffer, pack_tmp_name, "pack");
505
570
if (rev_tmp_name)
506
571
rename_tmp_packfile(name_buffer, rev_tmp_name, "rev");
572
+
if (mtimes_tmp_name)
573
+
rename_tmp_packfile(name_buffer, mtimes_tmp_name, "mtimes");
507
574
}
508
575
509
576
void write_promisor_file(const char *promisor_name, struct ref **sought, int nr_sought)
+4
pack.h
+4
pack.h
···
44
44
#define WRITE_IDX_STRICT 02
45
45
#define WRITE_REV 04
46
46
#define WRITE_REV_VERIFY 010
47
+
#define WRITE_MTIMES 020
47
48
48
49
uint32_t version;
49
50
uint32_t off32_limit;
···
109
110
#define PH_ERROR_PROTOCOL (-3)
110
111
int read_pack_header(int fd, struct pack_header *);
111
112
113
+
struct packing_data;
114
+
112
115
struct hashfile *create_tmp_packfile(char **pack_tmp_name);
113
116
void stage_tmp_packfiles(struct strbuf *name_buffer,
114
117
const char *pack_tmp_name,
115
118
struct pack_idx_entry **written_list,
116
119
uint32_t nr_written,
120
+
struct packing_data *to_pack,
117
121
struct pack_idx_option *pack_idx_opts,
118
122
unsigned char hash[],
119
123
char **idx_tmp_name);
+17
-2
packfile.c
+17
-2
packfile.c
···
334
334
p->revindex_data = NULL;
335
335
}
336
336
337
+
static void close_pack_mtimes(struct packed_git *p)
338
+
{
339
+
if (!p->mtimes_map)
340
+
return;
341
+
342
+
munmap((void *)p->mtimes_map, p->mtimes_size);
343
+
p->mtimes_map = NULL;
344
+
}
345
+
337
346
void close_pack(struct packed_git *p)
338
347
{
339
348
close_pack_windows(p);
340
349
close_pack_fd(p);
341
350
close_pack_index(p);
342
351
close_pack_revindex(p);
352
+
close_pack_mtimes(p);
343
353
oidset_clear(&p->bad_objects);
344
354
}
345
355
···
363
373
364
374
void unlink_pack_path(const char *pack_name, int force_delete)
365
375
{
366
-
static const char *exts[] = {".pack", ".idx", ".rev", ".keep", ".bitmap", ".promisor"};
376
+
static const char *exts[] = {".pack", ".idx", ".rev", ".keep", ".bitmap", ".promisor", ".mtimes"};
367
377
int i;
368
378
struct strbuf buf = STRBUF_INIT;
369
379
size_t plen;
···
718
728
if (!access(p->pack_name, F_OK))
719
729
p->pack_promisor = 1;
720
730
731
+
xsnprintf(p->pack_name + path_len, alloc - path_len, ".mtimes");
732
+
if (!access(p->pack_name, F_OK))
733
+
p->is_cruft = 1;
734
+
721
735
xsnprintf(p->pack_name + path_len, alloc - path_len, ".pack");
722
736
if (stat(p->pack_name, &st) || !S_ISREG(st.st_mode)) {
723
737
free(p);
···
869
883
ends_with(file_name, ".pack") ||
870
884
ends_with(file_name, ".bitmap") ||
871
885
ends_with(file_name, ".keep") ||
872
-
ends_with(file_name, ".promisor"))
886
+
ends_with(file_name, ".promisor") ||
887
+
ends_with(file_name, ".mtimes"))
873
888
string_list_append(data->garbage, full_name);
874
889
else
875
890
report_garbage(PACKDIR_FILE_GARBAGE, full_name);
+50
-8
reachable.c
+50
-8
reachable.c
···
13
13
#include "worktree.h"
14
14
#include "object-store.h"
15
15
#include "pack-bitmap.h"
16
+
#include "pack-mtimes.h"
16
17
17
18
struct connectivity_progress {
18
19
struct progress *progress;
···
60
61
struct recent_data {
61
62
struct rev_info *revs;
62
63
timestamp_t timestamp;
64
+
report_recent_object_fn *cb;
65
+
int ignore_in_core_kept_packs;
63
66
};
64
67
65
68
static void add_recent_object(const struct object_id *oid,
69
+
struct packed_git *pack,
70
+
off_t offset,
66
71
timestamp_t mtime,
67
72
struct recent_data *data)
68
73
{
···
103
108
die("unable to lookup %s", oid_to_hex(oid));
104
109
105
110
add_pending_object(data->revs, obj, "");
111
+
if (data->cb)
112
+
data->cb(obj, pack, offset, mtime);
113
+
}
114
+
115
+
static int want_recent_object(struct recent_data *data,
116
+
const struct object_id *oid)
117
+
{
118
+
if (data->ignore_in_core_kept_packs &&
119
+
has_object_kept_pack(oid, IN_CORE_KEEP_PACKS))
120
+
return 0;
121
+
return 1;
106
122
}
107
123
108
124
static int add_recent_loose(const struct object_id *oid,
109
125
const char *path, void *data)
110
126
{
111
127
struct stat st;
112
-
struct object *obj = lookup_object(the_repository, oid);
128
+
struct object *obj;
129
+
130
+
if (!want_recent_object(data, oid))
131
+
return 0;
132
+
133
+
obj = lookup_object(the_repository, oid);
113
134
114
135
if (obj && obj->flags & SEEN)
115
136
return 0;
···
126
147
return error_errno("unable to stat %s", oid_to_hex(oid));
127
148
}
128
149
129
-
add_recent_object(oid, st.st_mtime, data);
150
+
add_recent_object(oid, NULL, 0, st.st_mtime, data);
130
151
return 0;
131
152
}
132
153
···
134
155
struct packed_git *p, uint32_t pos,
135
156
void *data)
136
157
{
137
-
struct object *obj = lookup_object(the_repository, oid);
158
+
struct object *obj;
159
+
timestamp_t mtime = p->mtime;
160
+
161
+
if (!want_recent_object(data, oid))
162
+
return 0;
163
+
164
+
obj = lookup_object(the_repository, oid);
138
165
139
166
if (obj && obj->flags & SEEN)
140
167
return 0;
141
-
add_recent_object(oid, p->mtime, data);
168
+
if (p->is_cruft) {
169
+
if (load_pack_mtimes(p) < 0)
170
+
die(_("could not load cruft pack .mtimes"));
171
+
mtime = nth_packed_mtime(p, pos);
172
+
}
173
+
add_recent_object(oid, p, nth_packed_object_offset(p, pos), mtime, data);
142
174
return 0;
143
175
}
144
176
145
177
int add_unseen_recent_objects_to_traversal(struct rev_info *revs,
146
-
timestamp_t timestamp)
178
+
timestamp_t timestamp,
179
+
report_recent_object_fn *cb,
180
+
int ignore_in_core_kept_packs)
147
181
{
148
182
struct recent_data data;
183
+
enum for_each_object_flags flags;
149
184
int r;
150
185
151
186
data.revs = revs;
152
187
data.timestamp = timestamp;
188
+
data.cb = cb;
189
+
data.ignore_in_core_kept_packs = ignore_in_core_kept_packs;
153
190
154
191
r = for_each_loose_object(add_recent_loose, &data,
155
192
FOR_EACH_OBJECT_LOCAL_ONLY);
156
193
if (r)
157
194
return r;
158
-
return for_each_packed_object(add_recent_packed, &data,
159
-
FOR_EACH_OBJECT_LOCAL_ONLY);
195
+
196
+
flags = FOR_EACH_OBJECT_LOCAL_ONLY | FOR_EACH_OBJECT_PACK_ORDER;
197
+
if (ignore_in_core_kept_packs)
198
+
flags |= FOR_EACH_OBJECT_SKIP_IN_CORE_KEPT_PACKS;
199
+
200
+
return for_each_packed_object(add_recent_packed, &data, flags);
160
201
}
161
202
162
203
static int mark_object_seen(const struct object_id *oid,
···
217
258
218
259
if (mark_recent) {
219
260
revs->ignore_missing_links = 1;
220
-
if (add_unseen_recent_objects_to_traversal(revs, mark_recent))
261
+
if (add_unseen_recent_objects_to_traversal(revs, mark_recent,
262
+
NULL, 0))
221
263
die("unable to mark recent objects");
222
264
if (prepare_revision_walk(revs))
223
265
die("revision walk setup failed");
+8
-1
reachable.h
+8
-1
reachable.h
···
3
3
4
4
struct progress;
5
5
struct rev_info;
6
+
struct object;
7
+
struct packed_git;
8
+
9
+
typedef void report_recent_object_fn(const struct object *, struct packed_git *,
10
+
off_t, time_t);
6
11
7
12
int add_unseen_recent_objects_to_traversal(struct rev_info *revs,
8
-
timestamp_t timestamp);
13
+
timestamp_t timestamp,
14
+
report_recent_object_fn cb,
15
+
int ignore_in_core_kept_packs);
9
16
void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
10
17
timestamp_t mark_recent, struct progress *);
11
18
+56
t/helper/test-pack-mtimes.c
+56
t/helper/test-pack-mtimes.c
···
1
+
#include "git-compat-util.h"
2
+
#include "test-tool.h"
3
+
#include "strbuf.h"
4
+
#include "object-store.h"
5
+
#include "packfile.h"
6
+
#include "pack-mtimes.h"
7
+
8
+
static void dump_mtimes(struct packed_git *p)
9
+
{
10
+
uint32_t i;
11
+
if (load_pack_mtimes(p) < 0)
12
+
die("could not load pack .mtimes");
13
+
14
+
for (i = 0; i < p->num_objects; i++) {
15
+
struct object_id oid;
16
+
if (nth_packed_object_id(&oid, p, i) < 0)
17
+
die("could not load object id at position %"PRIu32, i);
18
+
19
+
printf("%s %"PRIu32"\n",
20
+
oid_to_hex(&oid), nth_packed_mtime(p, i));
21
+
}
22
+
}
23
+
24
+
static const char *pack_mtimes_usage = "\n"
25
+
" test-tool pack-mtimes <pack-name.mtimes>";
26
+
27
+
int cmd__pack_mtimes(int argc, const char **argv)
28
+
{
29
+
struct strbuf buf = STRBUF_INIT;
30
+
struct packed_git *p;
31
+
32
+
setup_git_directory();
33
+
34
+
if (argc != 2)
35
+
usage(pack_mtimes_usage);
36
+
37
+
for (p = get_all_packs(the_repository); p; p = p->next) {
38
+
strbuf_addstr(&buf, basename(p->pack_name));
39
+
strbuf_strip_suffix(&buf, ".pack");
40
+
strbuf_addstr(&buf, ".mtimes");
41
+
42
+
if (!strcmp(buf.buf, argv[1]))
43
+
break;
44
+
45
+
strbuf_reset(&buf);
46
+
}
47
+
48
+
strbuf_release(&buf);
49
+
50
+
if (!p)
51
+
die("could not find pack '%s'", argv[1]);
52
+
53
+
dump_mtimes(p);
54
+
55
+
return 0;
56
+
}
+1
t/helper/test-tool.c
+1
t/helper/test-tool.c
···
48
48
{ "oidmap", cmd__oidmap },
49
49
{ "oidtree", cmd__oidtree },
50
50
{ "online-cpus", cmd__online_cpus },
51
+
{ "pack-mtimes", cmd__pack_mtimes },
51
52
{ "parse-options", cmd__parse_options },
52
53
{ "parse-pathspec-file", cmd__parse_pathspec_file },
53
54
{ "partial-clone", cmd__partial_clone },
+1
t/helper/test-tool.h
+1
t/helper/test-tool.h
···
38
38
int cmd__oidmap(int argc, const char **argv);
39
39
int cmd__oidtree(int argc, const char **argv);
40
40
int cmd__online_cpus(int argc, const char **argv);
41
+
int cmd__pack_mtimes(int argc, const char **argv);
41
42
int cmd__parse_options(int argc, const char **argv);
42
43
int cmd__parse_pathspec_file(int argc, const char** argv);
43
44
int cmd__partial_clone(int argc, const char **argv);
+739
t/t5329-pack-objects-cruft.sh
+739
t/t5329-pack-objects-cruft.sh
···
1
+
#!/bin/sh
2
+
3
+
test_description='cruft pack related pack-objects tests'
4
+
. ./test-lib.sh
5
+
6
+
objdir=.git/objects
7
+
packdir=$objdir/pack
8
+
9
+
basic_cruft_pack_tests () {
10
+
expire="$1"
11
+
12
+
test_expect_success "unreachable loose objects are packed (expire $expire)" '
13
+
git init repo &&
14
+
test_when_finished "rm -fr repo" &&
15
+
(
16
+
cd repo &&
17
+
18
+
test_commit base &&
19
+
git repack -Ad &&
20
+
test_commit loose &&
21
+
22
+
test-tool chmtime +2000 "$objdir/$(test_oid_to_path \
23
+
$(git rev-parse loose:loose.t))" &&
24
+
test-tool chmtime +1000 "$objdir/$(test_oid_to_path \
25
+
$(git rev-parse loose^{tree}))" &&
26
+
27
+
(
28
+
git rev-list --objects --no-object-names base..loose |
29
+
while read oid
30
+
do
31
+
path="$objdir/$(test_oid_to_path "$oid")" &&
32
+
printf "%s %d\n" "$oid" "$(test-tool chmtime --get "$path")"
33
+
done |
34
+
sort -k1
35
+
) >expect &&
36
+
37
+
keep="$(basename "$(ls $packdir/pack-*.pack)")" &&
38
+
cruft="$(echo $keep | git pack-objects --cruft \
39
+
--cruft-expiration="$expire" $packdir/pack)" &&
40
+
test-tool pack-mtimes "pack-$cruft.mtimes" >actual &&
41
+
42
+
test_cmp expect actual
43
+
)
44
+
'
45
+
46
+
test_expect_success "unreachable packed objects are packed (expire $expire)" '
47
+
git init repo &&
48
+
test_when_finished "rm -fr repo" &&
49
+
(
50
+
cd repo &&
51
+
52
+
test_commit packed &&
53
+
git repack -Ad &&
54
+
test_commit other &&
55
+
56
+
git rev-list --objects --no-object-names packed.. >objects &&
57
+
keep="$(basename "$(ls $packdir/pack-*.pack)")" &&
58
+
other="$(git pack-objects --delta-base-offset \
59
+
$packdir/pack <objects)" &&
60
+
git prune-packed &&
61
+
62
+
test-tool chmtime --get -100 "$packdir/pack-$other.pack" >expect &&
63
+
64
+
cruft="$(git pack-objects --cruft --cruft-expiration="$expire" $packdir/pack <<-EOF
65
+
$keep
66
+
-pack-$other.pack
67
+
EOF
68
+
)" &&
69
+
test-tool pack-mtimes "pack-$cruft.mtimes" >actual.raw &&
70
+
71
+
cut -d" " -f2 <actual.raw | sort -u >actual &&
72
+
73
+
test_cmp expect actual
74
+
)
75
+
'
76
+
77
+
test_expect_success "unreachable cruft objects are repacked (expire $expire)" '
78
+
git init repo &&
79
+
test_when_finished "rm -fr repo" &&
80
+
(
81
+
cd repo &&
82
+
83
+
test_commit packed &&
84
+
git repack -Ad &&
85
+
test_commit other &&
86
+
87
+
git rev-list --objects --no-object-names packed.. >objects &&
88
+
keep="$(basename "$(ls $packdir/pack-*.pack)")" &&
89
+
90
+
cruft_a="$(echo $keep | git pack-objects --cruft --cruft-expiration="$expire" $packdir/pack)" &&
91
+
git prune-packed &&
92
+
cruft_b="$(git pack-objects --cruft --cruft-expiration="$expire" $packdir/pack <<-EOF
93
+
$keep
94
+
-pack-$cruft_a.pack
95
+
EOF
96
+
)" &&
97
+
98
+
test-tool pack-mtimes "pack-$cruft_a.mtimes" >expect.raw &&
99
+
test-tool pack-mtimes "pack-$cruft_b.mtimes" >actual.raw &&
100
+
101
+
sort <expect.raw >expect &&
102
+
sort <actual.raw >actual &&
103
+
104
+
test_cmp expect actual
105
+
)
106
+
'
107
+
108
+
test_expect_success "multiple cruft packs (expire $expire)" '
109
+
git init repo &&
110
+
test_when_finished "rm -fr repo" &&
111
+
(
112
+
cd repo &&
113
+
114
+
test_commit reachable &&
115
+
git repack -Ad &&
116
+
keep="$(basename "$(ls $packdir/pack-*.pack)")" &&
117
+
118
+
test_commit cruft &&
119
+
loose="$objdir/$(test_oid_to_path $(git rev-parse cruft))" &&
120
+
121
+
# generate three copies of the cruft object in different
122
+
# cruft packs, each with a unique mtime:
123
+
# - one expired (1000 seconds ago)
124
+
# - two non-expired (one 1000 seconds in the future,
125
+
# one 1500 seconds in the future)
126
+
test-tool chmtime =-1000 "$loose" &&
127
+
git pack-objects --cruft $packdir/pack-A <<-EOF &&
128
+
$keep
129
+
EOF
130
+
test-tool chmtime =+1000 "$loose" &&
131
+
git pack-objects --cruft $packdir/pack-B <<-EOF &&
132
+
$keep
133
+
-$(basename $(ls $packdir/pack-A-*.pack))
134
+
EOF
135
+
test-tool chmtime =+1500 "$loose" &&
136
+
git pack-objects --cruft $packdir/pack-C <<-EOF &&
137
+
$keep
138
+
-$(basename $(ls $packdir/pack-A-*.pack))
139
+
-$(basename $(ls $packdir/pack-B-*.pack))
140
+
EOF
141
+
142
+
# ensure the resulting cruft pack takes the most recent
143
+
# mtime among all copies
144
+
cruft="$(git pack-objects --cruft \
145
+
--cruft-expiration="$expire" \
146
+
$packdir/pack <<-EOF
147
+
$keep
148
+
-$(basename $(ls $packdir/pack-A-*.pack))
149
+
-$(basename $(ls $packdir/pack-B-*.pack))
150
+
-$(basename $(ls $packdir/pack-C-*.pack))
151
+
EOF
152
+
)" &&
153
+
154
+
test-tool pack-mtimes "$(basename $(ls $packdir/pack-C-*.mtimes))" >expect.raw &&
155
+
test-tool pack-mtimes "pack-$cruft.mtimes" >actual.raw &&
156
+
157
+
sort expect.raw >expect &&
158
+
sort actual.raw >actual &&
159
+
test_cmp expect actual
160
+
)
161
+
'
162
+
163
+
test_expect_success "cruft packs tolerate missing trees (expire $expire)" '
164
+
git init repo &&
165
+
test_when_finished "rm -fr repo" &&
166
+
(
167
+
cd repo &&
168
+
169
+
test_commit reachable &&
170
+
test_commit cruft &&
171
+
172
+
tree="$(git rev-parse cruft^{tree})" &&
173
+
174
+
git reset --hard reachable &&
175
+
git tag -d cruft &&
176
+
git reflog expire --all --expire=all &&
177
+
178
+
# remove the unreachable tree, but leave the commit
179
+
# which has it as its root tree intact
180
+
rm -fr "$objdir/$(test_oid_to_path "$tree")" &&
181
+
182
+
git repack -Ad &&
183
+
basename $(ls $packdir/pack-*.pack) >in &&
184
+
git pack-objects --cruft --cruft-expiration="$expire" \
185
+
$packdir/pack <in
186
+
)
187
+
'
188
+
189
+
test_expect_success "cruft packs tolerate missing blobs (expire $expire)" '
190
+
git init repo &&
191
+
test_when_finished "rm -fr repo" &&
192
+
(
193
+
cd repo &&
194
+
195
+
test_commit reachable &&
196
+
test_commit cruft &&
197
+
198
+
blob="$(git rev-parse cruft:cruft.t)" &&
199
+
200
+
git reset --hard reachable &&
201
+
git tag -d cruft &&
202
+
git reflog expire --all --expire=all &&
203
+
204
+
# remove the unreachable blob, but leave the commit (and
205
+
# the root tree of that commit) intact
206
+
rm -fr "$objdir/$(test_oid_to_path "$blob")" &&
207
+
208
+
git repack -Ad &&
209
+
basename $(ls $packdir/pack-*.pack) >in &&
210
+
git pack-objects --cruft --cruft-expiration="$expire" \
211
+
$packdir/pack <in
212
+
)
213
+
'
214
+
}
215
+
216
+
basic_cruft_pack_tests never
217
+
basic_cruft_pack_tests 2.weeks.ago
218
+
219
+
test_expect_success 'cruft tags rescue tagged objects' '
220
+
git init repo &&
221
+
test_when_finished "rm -fr repo" &&
222
+
(
223
+
cd repo &&
224
+
225
+
test_commit packed &&
226
+
git repack -Ad &&
227
+
228
+
test_commit tagged &&
229
+
git tag -a annotated -m tag &&
230
+
231
+
git rev-list --objects --no-object-names packed.. >objects &&
232
+
while read oid
233
+
do
234
+
test-tool chmtime -1000 \
235
+
"$objdir/$(test_oid_to_path $oid)"
236
+
done <objects &&
237
+
238
+
test-tool chmtime -500 \
239
+
"$objdir/$(test_oid_to_path $(git rev-parse annotated))" &&
240
+
241
+
keep="$(basename "$(ls $packdir/pack-*.pack)")" &&
242
+
cruft="$(echo $keep | git pack-objects --cruft \
243
+
--cruft-expiration=750.seconds.ago \
244
+
$packdir/pack)" &&
245
+
test-tool pack-mtimes "pack-$cruft.mtimes" >actual.raw &&
246
+
cut -f1 -d" " <actual.raw | sort >actual &&
247
+
248
+
(
249
+
cat objects &&
250
+
git rev-parse annotated
251
+
) >expect.raw &&
252
+
sort <expect.raw >expect &&
253
+
254
+
test_cmp expect actual &&
255
+
cat actual
256
+
)
257
+
'
258
+
259
+
test_expect_success 'cruft commits rescue parents, trees' '
260
+
git init repo &&
261
+
test_when_finished "rm -fr repo" &&
262
+
(
263
+
cd repo &&
264
+
265
+
test_commit packed &&
266
+
git repack -Ad &&
267
+
268
+
test_commit old &&
269
+
test_commit new &&
270
+
271
+
git rev-list --objects --no-object-names packed..new >objects &&
272
+
while read object
273
+
do
274
+
test-tool chmtime -1000 \
275
+
"$objdir/$(test_oid_to_path $object)"
276
+
done <objects &&
277
+
test-tool chmtime +500 "$objdir/$(test_oid_to_path \
278
+
$(git rev-parse HEAD))" &&
279
+
280
+
keep="$(basename "$(ls $packdir/pack-*.pack)")" &&
281
+
cruft="$(echo $keep | git pack-objects --cruft \
282
+
--cruft-expiration=750.seconds.ago \
283
+
$packdir/pack)" &&
284
+
test-tool pack-mtimes "pack-$cruft.mtimes" >actual.raw &&
285
+
286
+
cut -d" " -f1 <actual.raw | sort >actual &&
287
+
sort <objects >expect &&
288
+
289
+
test_cmp expect actual
290
+
)
291
+
'
292
+
293
+
test_expect_success 'cruft trees rescue sub-trees, blobs' '
294
+
git init repo &&
295
+
test_when_finished "rm -fr repo" &&
296
+
(
297
+
cd repo &&
298
+
299
+
test_commit packed &&
300
+
git repack -Ad &&
301
+
302
+
mkdir -p dir/sub &&
303
+
echo foo >foo &&
304
+
echo bar >dir/bar &&
305
+
echo baz >dir/sub/baz &&
306
+
307
+
test_tick &&
308
+
git add . &&
309
+
git commit -m "pruned" &&
310
+
311
+
test-tool chmtime -1000 "$objdir/$(test_oid_to_path $(git rev-parse HEAD))" &&
312
+
test-tool chmtime -1000 "$objdir/$(test_oid_to_path $(git rev-parse HEAD^{tree}))" &&
313
+
test-tool chmtime -1000 "$objdir/$(test_oid_to_path $(git rev-parse HEAD:foo))" &&
314
+
test-tool chmtime -500 "$objdir/$(test_oid_to_path $(git rev-parse HEAD:dir))" &&
315
+
test-tool chmtime -1000 "$objdir/$(test_oid_to_path $(git rev-parse HEAD:dir/bar))" &&
316
+
test-tool chmtime -1000 "$objdir/$(test_oid_to_path $(git rev-parse HEAD:dir/sub))" &&
317
+
test-tool chmtime -1000 "$objdir/$(test_oid_to_path $(git rev-parse HEAD:dir/sub/baz))" &&
318
+
319
+
keep="$(basename "$(ls $packdir/pack-*.pack)")" &&
320
+
cruft="$(echo $keep | git pack-objects --cruft \
321
+
--cruft-expiration=750.seconds.ago \
322
+
$packdir/pack)" &&
323
+
test-tool pack-mtimes "pack-$cruft.mtimes" >actual.raw &&
324
+
cut -f1 -d" " <actual.raw | sort >actual &&
325
+
326
+
git rev-parse HEAD:dir HEAD:dir/bar HEAD:dir/sub HEAD:dir/sub/baz >expect.raw &&
327
+
sort <expect.raw >expect &&
328
+
329
+
test_cmp expect actual
330
+
)
331
+
'
332
+
333
+
test_expect_success 'expired objects are pruned' '
334
+
git init repo &&
335
+
test_when_finished "rm -fr repo" &&
336
+
(
337
+
cd repo &&
338
+
339
+
test_commit packed &&
340
+
git repack -Ad &&
341
+
342
+
test_commit pruned &&
343
+
344
+
git rev-list --objects --no-object-names packed..pruned >objects &&
345
+
while read object
346
+
do
347
+
test-tool chmtime -1000 \
348
+
"$objdir/$(test_oid_to_path $object)"
349
+
done <objects &&
350
+
351
+
keep="$(basename "$(ls $packdir/pack-*.pack)")" &&
352
+
cruft="$(echo $keep | git pack-objects --cruft \
353
+
--cruft-expiration=750.seconds.ago \
354
+
$packdir/pack)" &&
355
+
356
+
test-tool pack-mtimes "pack-$cruft.mtimes" >actual &&
357
+
test_must_be_empty actual
358
+
)
359
+
'
360
+
361
+
test_expect_success 'repack --cruft generates a cruft pack' '
362
+
git init repo &&
363
+
test_when_finished "rm -fr repo" &&
364
+
(
365
+
cd repo &&
366
+
367
+
test_commit reachable &&
368
+
git branch -M main &&
369
+
git checkout --orphan other &&
370
+
test_commit unreachable &&
371
+
372
+
git checkout main &&
373
+
git branch -D other &&
374
+
git tag -d unreachable &&
375
+
# objects are not cruft if they are contained in the reflogs
376
+
git reflog expire --all --expire=all &&
377
+
378
+
git rev-list --objects --all --no-object-names >reachable.raw &&
379
+
git cat-file --batch-all-objects --batch-check="%(objectname)" >objects &&
380
+
sort <reachable.raw >reachable &&
381
+
comm -13 reachable objects >unreachable &&
382
+
383
+
git repack --cruft -d &&
384
+
385
+
cruft=$(basename $(ls $packdir/pack-*.mtimes) .mtimes) &&
386
+
pack=$(basename $(ls $packdir/pack-*.pack | grep -v $cruft) .pack) &&
387
+
388
+
git show-index <$packdir/$pack.idx >actual.raw &&
389
+
cut -f2 -d" " actual.raw | sort >actual &&
390
+
test_cmp reachable actual &&
391
+
392
+
git show-index <$packdir/$cruft.idx >actual.raw &&
393
+
cut -f2 -d" " actual.raw | sort >actual &&
394
+
test_cmp unreachable actual
395
+
)
396
+
'
397
+
398
+
test_expect_success 'loose objects mtimes upsert others' '
399
+
git init repo &&
400
+
test_when_finished "rm -fr repo" &&
401
+
(
402
+
cd repo &&
403
+
404
+
test_commit reachable &&
405
+
git repack -Ad &&
406
+
git branch -M main &&
407
+
408
+
git checkout --orphan other &&
409
+
test_commit cruft &&
410
+
# incremental repack, leaving existing objects loose (so
411
+
# they can be "freshened")
412
+
git repack &&
413
+
414
+
tip="$(git rev-parse cruft)" &&
415
+
path="$objdir/$(test_oid_to_path "$tip")" &&
416
+
test-tool chmtime --get +1000 "$path" >expect &&
417
+
418
+
git checkout main &&
419
+
git branch -D other &&
420
+
git tag -d cruft &&
421
+
git reflog expire --all --expire=all &&
422
+
423
+
git repack --cruft -d &&
424
+
425
+
mtimes="$(basename $(ls $packdir/pack-*.mtimes))" &&
426
+
test-tool pack-mtimes "$mtimes" >actual.raw &&
427
+
grep "$tip" actual.raw | cut -d" " -f2 >actual &&
428
+
test_cmp expect actual
429
+
)
430
+
'
431
+
432
+
test_expect_success 'expiring cruft objects with git gc' '
433
+
git init repo &&
434
+
test_when_finished "rm -fr repo" &&
435
+
(
436
+
cd repo &&
437
+
438
+
test_commit reachable &&
439
+
git branch -M main &&
440
+
git checkout --orphan other &&
441
+
test_commit unreachable &&
442
+
443
+
git checkout main &&
444
+
git branch -D other &&
445
+
git tag -d unreachable &&
446
+
# objects are not cruft if they are contained in the reflogs
447
+
git reflog expire --all --expire=all &&
448
+
449
+
git rev-list --objects --all --no-object-names >reachable.raw &&
450
+
git cat-file --batch-all-objects --batch-check="%(objectname)" >objects &&
451
+
sort <reachable.raw >reachable &&
452
+
comm -13 reachable objects >unreachable &&
453
+
454
+
git repack --cruft -d &&
455
+
456
+
mtimes=$(ls .git/objects/pack/pack-*.mtimes) &&
457
+
test_path_is_file $mtimes &&
458
+
459
+
git gc --cruft --prune=now &&
460
+
461
+
git cat-file --batch-all-objects --batch-check="%(objectname)" >objects &&
462
+
463
+
comm -23 unreachable objects >removed &&
464
+
test_cmp unreachable removed &&
465
+
test_path_is_missing $mtimes
466
+
)
467
+
'
468
+
469
+
test_expect_success 'cruft packs are not included in geometric repack' '
470
+
git init repo &&
471
+
test_when_finished "rm -fr repo" &&
472
+
(
473
+
cd repo &&
474
+
475
+
test_commit reachable &&
476
+
git repack -Ad &&
477
+
git branch -M main &&
478
+
479
+
git checkout --orphan other &&
480
+
test_commit cruft &&
481
+
git repack -d &&
482
+
483
+
git checkout main &&
484
+
git branch -D other &&
485
+
git tag -d cruft &&
486
+
git reflog expire --all --expire=all &&
487
+
488
+
git repack --cruft &&
489
+
490
+
find $packdir -type f | sort >before &&
491
+
git repack --geometric=2 -d &&
492
+
find $packdir -type f | sort >after &&
493
+
494
+
test_cmp before after
495
+
)
496
+
'
497
+
498
+
test_expect_success 'repack --geometric collects once-cruft objects' '
499
+
git init repo &&
500
+
test_when_finished "rm -fr repo" &&
501
+
(
502
+
cd repo &&
503
+
504
+
test_commit reachable &&
505
+
git repack -Ad &&
506
+
git branch -M main &&
507
+
508
+
git checkout --orphan other &&
509
+
git rm -rf . &&
510
+
test_commit --no-tag cruft &&
511
+
cruft="$(git rev-parse HEAD)" &&
512
+
513
+
git checkout main &&
514
+
git branch -D other &&
515
+
git reflog expire --all --expire=all &&
516
+
517
+
# Pack the objects created in the previous step into a cruft
518
+
# pack. Intentionally leave loose copies of those objects
519
+
# around so we can pick them up in a subsequent --geometric
520
+
# reapack.
521
+
git repack --cruft &&
522
+
523
+
# Now make those objects reachable, and ensure that they are
524
+
# packed into the new pack created via a --geometric repack.
525
+
git update-ref refs/heads/other $cruft &&
526
+
527
+
# Without this object, the set of unpacked objects is exactly
528
+
# the set of objects already in the cruft pack. Tweak that set
529
+
# to ensure we do not overwrite the cruft pack entirely.
530
+
test_commit reachable2 &&
531
+
532
+
find $packdir -name "pack-*.idx" | sort >before &&
533
+
git repack --geometric=2 -d &&
534
+
find $packdir -name "pack-*.idx" | sort >after &&
535
+
536
+
{
537
+
git rev-list --objects --no-object-names $cruft &&
538
+
git rev-list --objects --no-object-names reachable..reachable2
539
+
} >want.raw &&
540
+
sort want.raw >want &&
541
+
542
+
pack=$(comm -13 before after) &&
543
+
git show-index <$pack >objects.raw &&
544
+
545
+
cut -d" " -f2 objects.raw | sort >got &&
546
+
547
+
test_cmp want got
548
+
)
549
+
'
550
+
551
+
test_expect_success 'cruft repack with no reachable objects' '
552
+
git init repo &&
553
+
test_when_finished "rm -fr repo" &&
554
+
(
555
+
cd repo &&
556
+
557
+
test_commit base &&
558
+
git repack -ad &&
559
+
560
+
base="$(git rev-parse base)" &&
561
+
562
+
git for-each-ref --format="delete %(refname)" >in &&
563
+
git update-ref --stdin <in &&
564
+
git reflog expire --all --expire=all &&
565
+
rm -fr .git/index &&
566
+
567
+
git repack --cruft -d &&
568
+
569
+
git cat-file -t $base
570
+
)
571
+
'
572
+
573
+
test_expect_success 'cruft repack ignores --max-pack-size' '
574
+
git init max-pack-size &&
575
+
(
576
+
cd max-pack-size &&
577
+
test_commit base &&
578
+
# two cruft objects which exceed the maximum pack size
579
+
test-tool genrandom foo 1048576 | git hash-object --stdin -w &&
580
+
test-tool genrandom bar 1048576 | git hash-object --stdin -w &&
581
+
git repack --cruft --max-pack-size=1M &&
582
+
find $packdir -name "*.mtimes" >cruft &&
583
+
test_line_count = 1 cruft &&
584
+
test-tool pack-mtimes "$(basename "$(cat cruft)")" >objects &&
585
+
test_line_count = 2 objects
586
+
)
587
+
'
588
+
589
+
test_expect_success 'cruft repack ignores pack.packSizeLimit' '
590
+
(
591
+
cd max-pack-size &&
592
+
# repack everything back together to remove the existing cruft
593
+
# pack (but to keep its objects)
594
+
git repack -adk &&
595
+
git -c pack.packSizeLimit=1M repack --cruft &&
596
+
# ensure the same post condition is met when --max-pack-size
597
+
# would otherwise be inferred from the configuration
598
+
find $packdir -name "*.mtimes" >cruft &&
599
+
test_line_count = 1 cruft &&
600
+
test-tool pack-mtimes "$(basename "$(cat cruft)")" >objects &&
601
+
test_line_count = 2 objects
602
+
)
603
+
'
604
+
605
+
test_expect_success 'cruft repack respects repack.cruftWindow' '
606
+
git init repo &&
607
+
test_when_finished "rm -fr repo" &&
608
+
(
609
+
cd repo &&
610
+
611
+
test_commit base &&
612
+
613
+
GIT_TRACE2_EVENT=$(pwd)/event.trace \
614
+
git -c pack.window=1 -c repack.cruftWindow=2 repack \
615
+
--cruft --window=3 &&
616
+
617
+
grep "pack-objects.*--window=2.*--cruft" event.trace
618
+
)
619
+
'
620
+
621
+
test_expect_success 'cruft repack respects --window by default' '
622
+
git init repo &&
623
+
test_when_finished "rm -fr repo" &&
624
+
(
625
+
cd repo &&
626
+
627
+
test_commit base &&
628
+
629
+
GIT_TRACE2_EVENT=$(pwd)/event.trace \
630
+
git -c pack.window=2 repack --cruft --window=3 &&
631
+
632
+
grep "pack-objects.*--window=3.*--cruft" event.trace
633
+
)
634
+
'
635
+
636
+
test_expect_success 'cruft repack respects --quiet' '
637
+
git init repo &&
638
+
test_when_finished "rm -fr repo" &&
639
+
(
640
+
cd repo &&
641
+
642
+
test_commit base &&
643
+
GIT_PROGRESS_DELAY=0 git repack --cruft --quiet 2>err &&
644
+
test_must_be_empty err
645
+
)
646
+
'
647
+
648
+
test_expect_success 'cruft --local drops unreachable objects' '
649
+
git init alternate &&
650
+
git init repo &&
651
+
test_when_finished "rm -fr alternate repo" &&
652
+
653
+
test_commit -C alternate base &&
654
+
# Pack all objects in alterate so that the cruft repack in "repo" sees
655
+
# the object it dropped due to `--local` as packed. Otherwise this
656
+
# object would not appear packed anywhere (since it is not packed in
657
+
# alternate and likewise not part of the cruft pack in the other repo
658
+
# because of `--local`).
659
+
git -C alternate repack -ad &&
660
+
661
+
(
662
+
cd repo &&
663
+
664
+
object="$(git -C ../alternate rev-parse HEAD:base.t)" &&
665
+
git -C ../alternate cat-file -p $object >contents &&
666
+
667
+
# Write some reachable objects and two unreachable ones: one
668
+
# that the alternate has and another that is unique.
669
+
test_commit other &&
670
+
git hash-object -w -t blob contents &&
671
+
cruft="$(echo cruft | git hash-object -w -t blob --stdin)" &&
672
+
673
+
( cd ../alternate/.git/objects && pwd ) \
674
+
>.git/objects/info/alternates &&
675
+
676
+
test_path_is_file $objdir/$(test_oid_to_path $cruft) &&
677
+
test_path_is_file $objdir/$(test_oid_to_path $object) &&
678
+
679
+
git repack -d --cruft --local &&
680
+
681
+
test-tool pack-mtimes "$(basename $(ls $packdir/pack-*.mtimes))" \
682
+
>objects &&
683
+
! grep $object objects &&
684
+
grep $cruft objects
685
+
)
686
+
'
687
+
688
+
test_expect_success 'MIDX bitmaps tolerate reachable cruft objects' '
689
+
git init repo &&
690
+
test_when_finished "rm -fr repo" &&
691
+
(
692
+
cd repo &&
693
+
694
+
test_commit reachable &&
695
+
test_commit cruft &&
696
+
unreachable="$(git rev-parse cruft)" &&
697
+
698
+
git reset --hard $unreachable^ &&
699
+
git tag -d cruft &&
700
+
git reflog expire --all --expire=all &&
701
+
702
+
git repack --cruft -d &&
703
+
704
+
# resurrect the unreachable object via a new commit. the
705
+
# new commit will get selected for a bitmap, but be
706
+
# missing one of its parents from the selected packs.
707
+
git reset --hard $unreachable &&
708
+
test_commit resurrect &&
709
+
710
+
git repack --write-midx --write-bitmap-index --geometric=2 -d
711
+
)
712
+
'
713
+
714
+
test_expect_success 'cruft objects are freshend via loose' '
715
+
git init repo &&
716
+
test_when_finished "rm -fr repo" &&
717
+
(
718
+
cd repo &&
719
+
720
+
echo "cruft" >contents &&
721
+
blob="$(git hash-object -w -t blob contents)" &&
722
+
loose="$objdir/$(test_oid_to_path $blob)" &&
723
+
724
+
test_commit base &&
725
+
726
+
git repack --cruft -d &&
727
+
728
+
test_path_is_missing "$loose" &&
729
+
test-tool pack-mtimes "$(basename "$(ls $packdir/pack-*.mtimes)")" >cruft &&
730
+
grep "$blob" cruft &&
731
+
732
+
# write the same object again
733
+
git hash-object -w -t blob contents &&
734
+
735
+
test_path_is_file "$loose"
736
+
)
737
+
'
738
+
739
+
test_done