freshlybakedca.ke
Git fork
1Use of index and Racy Git problem
2=================================
3
4Background
5----------
6
7The index is one of the most important data structures in Git.
8It represents a virtual working tree state by recording list of
9paths and their object names and serves as a staging area to
10write out the next tree object to be committed. The state is
11"virtual" in the sense that it does not necessarily have to, and
12often does not, match the files in the working tree.
13
14There are cases where Git needs to examine the differences between the
15virtual working tree state in the index and the files in the
16working tree. The most obvious case is when the user asks `git
17diff` (or its low level implementation, `git diff-files`) or
18`git-ls-files --modified`. In addition, Git internally checks
19if the files in the working tree are different from what are
20recorded in the index to avoid stomping on local changes in them
21during patch application, switching branches, and merging.
22
23In order to speed up this comparison between the files in the
24working tree and the index entries, the index entries record the
25information obtained from the filesystem via `lstat(2)` system
26call when they were last updated. When checking if they differ,
27Git first runs `lstat(2)` on the files and compares the result
28with this information (this is what was originally done by the
29`ce_match_stat()` function, but the current code does it in
30`ce_match_stat_basic()` function). If some of these "cached
31stat information" fields do not match, Git can tell that the
32files are modified without even looking at their contents.
33
34Note: not all members in `struct stat` obtained via `lstat(2)`
35are used for this comparison. For example, `st_atime` obviously
36is not useful. Currently, Git compares the file type (regular
37files vs symbolic links) and executable bits (only for regular
38files) from `st_mode` member, `st_mtime` and `st_ctime`
39timestamps, `st_uid`, `st_gid`, `st_ino`, and `st_size` members.
40With a `USE_STDEV` compile-time option, `st_dev` is also
41compared, but this is not enabled by default because this member
42is not stable on network filesystems. With `USE_NSEC`
43compile-time option, `st_mtim.tv_nsec` and `st_ctim.tv_nsec`
44members are also compared. On Linux, this is not enabled by default
45because in-core timestamps can have finer granularity than
46on-disk timestamps, resulting in meaningless changes when an
47inode is evicted from the inode cache. See commit 8ce13b0
48of git://git.kernel.org/pub/scm/linux/kernel/git/tglx/history.git
49([PATCH] Sync in core time granularity with filesystems,
502005-01-04). This patch is included in kernel 2.6.11 and newer, but
51only fixes the issue for file systems with exactly 1 ns or 1 s
52resolution. Other file systems are still broken in current Linux
53kernels (e.g. CEPH, CIFS, NTFS, UDF), see
54https://lore.kernel.org/lkml/5577240D.7020309@gmail.com/
55
56Racy Git
57--------
58
59There is one slight problem with the optimization based on the
60cached stat information. Consider this sequence:
61
62 : modify 'foo'
63 $ git update-index 'foo'
64 : modify 'foo' again, in-place, without changing its size
65
66The first `update-index` computes the object name of the
67contents of file `foo` and updates the index entry for `foo`
68along with the `struct stat` information. If the modification
69that follows it happens very fast so that the file's `st_mtime`
70timestamp does not change, after this sequence, the cached stat
71information the index entry records still exactly match what you
72would see in the filesystem, even though the file `foo` is now
73different.
74This way, Git can incorrectly think files in the working tree
75are unmodified even though they actually are. This is called
76the "racy Git" problem (discovered by Pasky), and the entries
77that appear clean when they may not be because of this problem
78are called "racily clean".
79
80To avoid this problem, Git does two things:
81
82. When the cached stat information says the file has not been
83 modified, and the `st_mtime` is the same as (or newer than)
84 the timestamp of the index file itself (which is the time `git
85 update-index foo` finished running in the above example), it
86 also compares the contents with the object registered in the
87 index entry to make sure they match.
88
89. When the index file is updated that contains racily clean
90 entries, cached `st_size` information is truncated to zero
91 before writing a new version of the index file.
92
93Because the index file itself is written after collecting all
94the stat information from updated paths, `st_mtime` timestamp of
95it is usually the same as or newer than any of the paths the
96index contains. And no matter how quick the modification that
97follows `git update-index foo` finishes, the resulting
98`st_mtime` timestamp on `foo` cannot get a value earlier
99than the index file. Therefore, index entries that can be
100racily clean are limited to the ones that have the same
101timestamp as the index file itself.
102
103The callers that want to check if an index entry matches the
104corresponding file in the working tree continue to call
105`ce_match_stat()`, but with this change, `ce_match_stat()` uses
106`ce_modified_check_fs()` to see if racily clean ones are
107actually clean after comparing the cached stat information using
108`ce_match_stat_basic()`.
109
110The problem the latter solves is this sequence:
111
112 $ git update-index 'foo'
113 : modify 'foo' in-place without changing its size
114 : wait for enough time
115 $ git update-index 'bar'
116
117Without the latter, the timestamp of the index file gets a newer
118value, and falsely clean entry `foo` would not be caught by the
119timestamp comparison check done with the former logic anymore.
120The latter makes sure that the cached stat information for `foo`
121would never match with the file in the working tree, so later
122checks by `ce_match_stat_basic()` would report that the index entry
123does not match the file and Git does not have to fall back on more
124expensive `ce_modified_check_fs()`.
125
126
127Runtime penalty
128---------------
129
130The runtime penalty of falling back to `ce_modified_check_fs()`
131from `ce_match_stat()` can be very expensive when there are many
132racily clean entries. An obvious way to artificially create
133this situation is to give the same timestamp to all the files in
134the working tree in a large project, run `git update-index` on
135them, and give the same timestamp to the index file:
136
137 $ date >.datestamp
138 $ git ls-files | xargs touch -r .datestamp
139 $ git ls-files | git update-index --stdin
140 $ touch -r .datestamp .git/index
141
142This will make all index entries racily clean. The linux project, for
143example, there are over 20,000 files in the working tree. On my
144Athlon 64 X2 3800+, after the above:
145
146 $ /usr/bin/time git diff-files
147 1.68user 0.54system 0:02.22elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
148 0inputs+0outputs (0major+67111minor)pagefaults 0swaps
149 $ git update-index MAINTAINERS
150 $ /usr/bin/time git diff-files
151 0.02user 0.12system 0:00.14elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
152 0inputs+0outputs (0major+935minor)pagefaults 0swaps
153
154Running `git update-index` in the middle checked the racily
155clean entries, and left the cached `st_mtime` for all the paths
156intact because they were actually clean (so this step took about
157the same amount of time as the first `git diff-files`). After
158that, they are not racily clean anymore but are truly clean, so
159the second invocation of `git diff-files` fully took advantage
160of the cached stat information.
161
162
163Avoiding runtime penalty
164------------------------
165
166In order to avoid the above runtime penalty, post 1.4.2 Git used
167to have a code that made sure the index file
168got a timestamp newer than the youngest files in the index when
169there were many young files with the same timestamp as the
170resulting index file otherwise would have by waiting
171before finishing writing the index file out.
172
173I suspected that in practice the situation where many paths in the
174index are all racily clean was quite rare. The only code paths
175that can record recent timestamp for large number of paths are:
176
177. Initial `git add .` of a large project.
178
179. `git checkout` of a large project from an empty index into an
180 unpopulated working tree.
181
182Note: switching branches with `git checkout` keeps the cached
183stat information of existing working tree files that are the
184same between the current branch and the new branch, which are
185all older than the resulting index file, and they will not
186become racily clean. Only the files that are actually checked
187out can become racily clean.
188
189In a large project where raciness avoidance cost really matters,
190however, the initial computation of all object names in the
191index takes more than one second, and the index file is written
192out after all that happens. Therefore the timestamp of the
193index file will be more than one second later than the
194youngest file in the working tree. This means that in these
195cases there actually will not be any racily clean entry in
196the resulting index.
197
198Based on this discussion, the current code does not use the
199"workaround" to avoid the runtime penalty that does not exist in
200practice anymore. This was done with commit 0fc82cff on Aug 15,
2012006.