freshlybakedca.ke / git
Git fork
fork atom

trace2: discard new traces if target directory has too many files

trace2 can write files into a target directory. With heavy usage, this
directory can fill up with files, causing difficulty for
trace-processing systems.

This patch adds a config option (trace2.maxFiles) to set a maximum
number of files that trace2 will write to a target directory. The
following behavior is enabled when the maxFiles is set to a positive
integer:
When trace2 would write a file to a target directory, first check
whether or not the traces should be discarded. Traces should be
discarded if:
* there is a sentinel file declaring that there are too many files
* OR, the number of files exceeds trace2.maxFiles.
In the latter case, we create a sentinel file named git-trace2-discard
to speed up future checks.

The assumption is that a separate trace-processing system is dealing
with the generated traces; once it processes and removes the sentinel
file, it should be safe to generate new trace files again.

The default value for trace2.maxFiles is zero, which disables the file
count check.

The config can also be overridden with a new environment variable:
GIT_TRACE2_MAX_FILES.

Signed-off-by: Josh Steadmon <steadmon@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

authored by

Josh Steadmon and committed by
Junio C Hamano 83e57b04 22541013

+112
unified split
+6
Documentation/config/trace2.txt
··· 54 54 By default, these errors are suppressed and tracing is 55 55 silently disabled. May be overridden by the 56 56 `GIT_TRACE2_DST_DEBUG` environment variable. 57 + 58 + trace2.maxFiles:: 59 + Integer. When writing trace files to a target directory, do not 60 + write additional traces if we would exceed this many files. Instead, 61 + write a sentinel file that will block further tracing to this 62 + directory. Defaults to 0, which disables this check.
+17
t/t0212-trace2-event.sh
··· 265 265 test_cmp expect actual 266 266 ' 267 267 268 + test_expect_success 'discard traces when there are too many files' ' 269 + mkdir trace_target_dir && 270 + test_when_finished "rm -r trace_target_dir" && 271 + ( 272 + GIT_TRACE2_MAX_FILES=5 && 273 + export GIT_TRACE2_MAX_FILES && 274 + cd trace_target_dir && 275 + test_seq $GIT_TRACE2_MAX_FILES >../expected_filenames.txt && 276 + xargs touch <../expected_filenames.txt && 277 + cd .. && 278 + GIT_TRACE2_EVENT="$(pwd)/trace_target_dir" test-tool trace2 001return 0 279 + ) && 280 + echo git-trace2-discard >>expected_filenames.txt && 281 + ls trace_target_dir >ls_output.txt && 282 + test_cmp expected_filenames.txt ls_output.txt 283 + ' 284 + 268 285 test_done
+84
trace2/tr2_dst.c
··· 8 8 */ 9 9 #define MAX_AUTO_ATTEMPTS 10 10 10 11 + /* 12 + * Sentinel file used to detect when we should discard new traces to avoid 13 + * writing too many trace files to a directory. 14 + */ 15 + #define DISCARD_SENTINEL_NAME "git-trace2-discard" 16 + 17 + /* 18 + * When set to zero, disables directory file count checks. Otherwise, controls 19 + * how many files we can write to a directory before entering discard mode. 20 + * This can be overridden via the TR2_SYSENV_MAX_FILES setting. 21 + */ 22 + static int tr2env_max_files = 0; 23 + 11 24 static int tr2_dst_want_warning(void) 12 25 { 13 26 static int tr2env_dst_debug = -1; ··· 32 45 dst->need_close = 0; 33 46 } 34 47 48 + /* 49 + * Check to make sure we're not overloading the target directory with too many 50 + * files. First get the threshold (if present) from the config or envvar. If 51 + * it's zero or unset, disable this check. Next check for the presence of a 52 + * sentinel file, then check file count. If we are overloaded, create the 53 + * sentinel file if it doesn't already exist. 54 + * 55 + * We expect that some trace processing system is gradually collecting files 56 + * from the target directory; after it removes the sentinel file we'll start 57 + * writing traces again. 58 + */ 59 + static int tr2_dst_too_many_files(const char *tgt_prefix) 60 + { 61 + int file_count = 0, max_files = 0, ret = 0; 62 + const char *max_files_var; 63 + DIR *dirp; 64 + struct strbuf path = STRBUF_INIT, sentinel_path = STRBUF_INIT; 65 + struct stat statbuf; 66 + 67 + /* Get the config or envvar and decide if we should continue this check */ 68 + max_files_var = tr2_sysenv_get(TR2_SYSENV_MAX_FILES); 69 + if (max_files_var && *max_files_var && ((max_files = atoi(max_files_var)) >= 0)) 70 + tr2env_max_files = max_files; 71 + 72 + if (!tr2env_max_files) { 73 + ret = 0; 74 + goto cleanup; 75 + } 76 + 77 + strbuf_addstr(&path, tgt_prefix); 78 + if (!is_dir_sep(path.buf[path.len - 1])) { 79 + strbuf_addch(&path, '/'); 80 + } 81 + 82 + /* check sentinel */ 83 + strbuf_addbuf(&sentinel_path, &path); 84 + strbuf_addstr(&sentinel_path, DISCARD_SENTINEL_NAME); 85 + if (!stat(sentinel_path.buf, &statbuf)) { 86 + ret = 1; 87 + goto cleanup; 88 + } 89 + 90 + /* check file count */ 91 + dirp = opendir(path.buf); 92 + while (file_count < tr2env_max_files && dirp && readdir(dirp)) 93 + file_count++; 94 + if (dirp) 95 + closedir(dirp); 96 + 97 + if (file_count >= tr2env_max_files) { 98 + creat(sentinel_path.buf, 0666); 99 + ret = 1; 100 + goto cleanup; 101 + } 102 + 103 + cleanup: 104 + strbuf_release(&path); 105 + strbuf_release(&sentinel_path); 106 + return ret; 107 + } 108 + 35 109 static int tr2_dst_try_auto_path(struct tr2_dst *dst, const char *tgt_prefix) 36 110 { 37 111 int fd; ··· 49 123 strbuf_addch(&path, '/'); 50 124 strbuf_addstr(&path, sid); 51 125 base_path_len = path.len; 126 + 127 + if (tr2_dst_too_many_files(tgt_prefix)) { 128 + strbuf_release(&path); 129 + if (tr2_dst_want_warning()) 130 + warning("trace2: not opening %s trace file due to too " 131 + "many files in target directory %s", 132 + tr2_sysenv_display_name(dst->sysenv_var), 133 + tgt_prefix); 134 + return 0; 135 + } 52 136 53 137 for (attempt_count = 0; attempt_count < MAX_AUTO_ATTEMPTS; attempt_count++) { 54 138 if (attempt_count > 0) {
+3
trace2/tr2_sysenv.c
··· 49 49 "trace2.perftarget" }, 50 50 [TR2_SYSENV_PERF_BRIEF] = { "GIT_TRACE2_PERF_BRIEF", 51 51 "trace2.perfbrief" }, 52 + 53 + [TR2_SYSENV_MAX_FILES] = { "GIT_TRACE2_MAX_FILES", 54 + "trace2.maxfiles" }, 52 55 }; 53 56 /* clang-format on */ 54 57
+2
trace2/tr2_sysenv.h
··· 24 24 TR2_SYSENV_PERF, 25 25 TR2_SYSENV_PERF_BRIEF, 26 26 27 + TR2_SYSENV_MAX_FILES, 28 + 27 29 TR2_SYSENV_MUST_BE_LAST 28 30 }; 29 31