commit f1dbec24f9a9553e22659f6c592c91e96221f6b9
from: Stefan Sperling <stsp@stsp.name>
date: Tue Nov 23 19:11:53 2021 UTC

fix a size_t multiplication overflow in diff_meyrs.c

Found on an OpenBSD armv7 machine running Got regression tests:
test_status_shows_no_mods_after_complete_merge Segmentation fault (core dumped)

The problematic multiplication is kd_len * kd_len in diff_algo_myers()
with kd_len set to 65537.

(gdb) p (int)(65537 * 65537)
$64 = 131073
(gdb) p (int)(65537 + 65537)
$65 = 131074
(gdb) p (unsigned int)(size_t)(-1)
$68 = 4294967295
(gdb) p (4294967295 / kd_len) 
$71 = 65535

Detect such overflow and run the fallback diff algorithm instead.

commit - 346d4986568ffbcfdc53cce45dd8a468535a68f2
commit + f1dbec24f9a9553e22659f6c592c91e96221f6b9
blob - 09e07bf366400bc6bbc7e18eba214c253d078cd7
blob + c886d1a285863fa2148b2b295b733d08fe7c9040
--- lib/diff_myers.c
+++ lib/diff_myers.c
@@ -1098,6 +1098,7 @@ diff_algo_myers(const struct diff_algo_config *algo_co
 	size_t kd_state_size = kd_buf_size * sizeof(int);
 	debug("state size: %zu\n", kd_state_size);
 	if (kd_buf_size < kd_len /* overflow? */
+	    || (SIZE_MAX / kd_len ) < kd_len
 	    || kd_state_size > algo_config->permitted_state_size) {
 		debug("state size %zu > permitted_state_size %zu, use fallback_algo\n",
 		      kd_state_size, algo_config->permitted_state_size);