ERROR in find_local_syms, couldn't find matching XXX local symbols in vmlinux symbol table

[ziyangc97]

hi, I met the following error when making kpatch.

My environment infor

kpatch version: 0.9.7
kernel version: 6.4.0
architecture: x86_64
linux: openEuler
gcc: gcc-12

/root/.kpatch/build.log:

  NM      System.map
  SORTTAB vmlinux
create-diff-object: ERROR: time.o: kpatch_bundle_symbols: 242: symbol __do_sys_adjtimex at offset 16 within section .text.__do_sys_adjtimex, expected 0: Success
ERROR: version-timestamp.o: 1 unsupported section change(s)
create-diff-object: unreconcilable difference: Success

The following is my kernel patch:

--- kernel/time/time.c  2023-09-06 11:19:31.000000000 +0800
+++ kernel/time/time.c.new      2023-09-11 19:06:21.674000162 +0800
@@ -170,6 +170,9 @@ int do_sys_settimeofday64(const struct t
 {
        static int firsttime = 1;
        int error = 0;
+
+       static int testdata=1;
+       printk("Test testdata=%d\n", testdata);

        if (tv && !timespec64_valid_settod(tv))
                return -EINVAL;

Addition info produced by debug_info flag:

+ [[ x86_64 = \p\p\c\6\4\l\e ]]
+ /usr/libexec/kpatch/create-diff-object orig/init/version-timestamp.o patched/init/version-timestamp.o vmlinux /root/.kpatch/tmp/vmlinux.symtab /usr/src/linux-6.4.0-6.0.0.14.oe2309.x86_64/Module.symvers klp_testklp output/init/version-timestamp.o
+ logger 1
+ local to_stdout=1
+ [[ 1 -ge 2 ]]
+ [[ 1 -eq 1 ]]
+ tee -a /root/.kpatch/build.log
ERROR: version-timestamp.o: 1 unsupported section change(s)
create-diff-object: unreconcilable difference: Success
version-timestamp.o: changed section .data not selected for inclusion
+ check_pipe_status create-diff-object
+ rc=2
+ [[ 2 = 139 ]]
+ [[ 2 -eq 0 ]]
+ [[ 2 -eq 3 ]]
+ ERROR=2
+ [[ 2 -eq 0 ]]
+ [[ 2 -ne 0 ]]
+ die '2 error(s) encountered'
+ [[ -z 2 error(s) encountered ]]
+ msg='2 error(s) encountered'
+ [[ -e /root/.kpatch/build.log ]]
+ warn '2 error(s) encountered. Check /root/.kpatch/build.log for more details.'
+ echo 'ERROR: 2 error(s) encountered. Check /root/.kpatch/build.log for more details.'
ERROR: 2 error(s) encountered. Check /root/.kpatch/build.log for more details.
+ exit 1

Thanks in advance for any suggestion or help! Is there any patch that may fix this error already?

[jpoimboe]

Please try the latest kpatch code

[ziyangc97]

Hi, thanks for advice.

I upgrade kpatch to latest version but still fail to make kpatch.
It seems that this is caused by new kernel 6.4.0 ?

My environment:

kpatch version: latest version 0.9.9 downloaded from github
kernel version: 6.4.0
architecture: x86_64
linux: openEuler

[root@openEuler patch_workspace]# ./make_hotpatch -i testklp -d .v1 -j 32
kernel version:6.4.0-6.0.0.14.oe2309.x86_64
grep: warning: stray \ before -
grep: warning: stray \ before -
grep: warning: stray \ before -
detect change files:/usr/src/linux-6.4.0-6.0.0.14.oe2309.x86_64/kernel/time/ntp.c.v1
make patch /tmp/klp_testklp/testklp.patch
Using source directory at /usr/src/linux-6.4.0-6.0.0.14.oe2309.x86_64
Testing patch file(s)
Reading special section data
Building original source
Building patched source
Extracting new and modified ELF sections
create-diff-object: ERROR: ntp.o: kpatch_bundle_symbols: 251: czy symbol ntp_update_frequency at offset 16 within section .text.ntp_update_frequency, expected 0
ERROR: 1 error(s) encountered. Check /root/.kpatch/build.log for more details.
error: invoke kpatch-build shell script to build patch failed

[jpoimboe]

Might be a problem with the handling of CONFIG_PREFIX_SYMBOLS. This should have been handled by 3e54c63 but maybe there's a gap somewhere. Please post the output of readelf -Ws /usr/src/linux-6.4.0-6.0.0.14.oe2309.x86_64/kernel/time/ntp.o.

[ziyangc97]

[root@openEuler ~]# readelf -Ws /usr/src/linux-6.4.0-6.0.0.14.oe2309.x86_64/kernel/time/ntp.o

Symbol table '.symtab' contains 88 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND
     1: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS ntp.c
     2: 0000000000000000     0 SECTION LOCAL  DEFAULT    4 .text.ntp_update_frequency
     3: 0000000000000010   108 FUNC    LOCAL  DEFAULT    4 ntp_update_frequency

[ziyangc97]

I compare the ELF data of ntp.o produced by kernel-5.10, the st_value is differenet

readelf -Ws /usr/src/kernels/linux-5.10.0-xxxxx/kernel/time/ntp.o

Symbol table '.symtab' contains 105 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND
     1: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS ntp.c
     2: 0000000000000000     0 SECTION LOCAL  DEFAULT    1 .text
     3: 0000000000000000     0 SECTION LOCAL  DEFAULT    2 .data
     4: 0000000000000000     0 SECTION LOCAL  DEFAULT    3 .bss
     5: 0000000000000000     0 SECTION LOCAL  DEFAULT    4 .text.ntp_update_frequency
     7: 0000000000000000   136 FUNC    LOCAL  DEFAULT    4 ntp_update_frequency

[jpoimboe]

Yes, the 16 byte offset is expected, and should be dealt with by 3e54c63, are you sure you have that commit?

Can you also post the output of readelf -Ws /usr/src/linux-6.4.0-6.0.0.14.oe2309.x86_64/kernel/time/ntp.o |grep ntp_update_frequency (lower case "s" this time)?

[ziyangc97]

1. I am sure my current kpatch have commit 3e54c6 by adding debug log.
2. There is no _pfx symbol of ntp_update_frequency

[root@openEuler patch_workspace]# readelf -Ws /usr/src/linux-6.4.0-6.0.0.14.oe2309.x86_64/kernel/time/ntp.o | grep ntp_update_frequency
     2: 0000000000000000     0 SECTION LOCAL  DEFAULT    4 .text.ntp_update_frequency
     3: 0000000000000010   108 FUNC    LOCAL  DEFAULT    4 ntp_update_frequency

3. I belive the problem occurs in sym->pfx code:

	list_for_each_entry(sym, &kelf->symbols, list) {
		if (is_bundleable(sym)) {
			if (sym->pfx)
				expected_offset = 16;
			else if (is_gcc6_localentry_bundled_sym(kelf, sym))

In my case, when iterating ntp_update_frequency symbol, sym->pfx returns false and results mismatch between st_value and expected_offset

[ziyangc97]

Maybe this error is releated to this PR :#1350

[ziyangc97]

I comment out CONFIG_X86_KERNEL_IBT and the error is gone, a new one come up:

Maybe this error is found or fix before?

[he7850]

I comment out CONFIG_X86_KERNEL_IBT and the error is gone, a new one come up: Maybe this error is found or fix before?

this error comes from making this patch:

diff --git a/fs/proc/version.c b/fs/proc/version.c.new
index 02e3c3c..47445f9 100644
--- a/fs/proc/version.c
+++ b/fs/proc/version.c.new
@@ -13,6 +13,7 @@ static int version_proc_show(struct seq_file *m, void *v)
 		utsname()->sysname,
 		utsname()->release,
 		utsname()->version);
+	seq_printf(m, "hotpatched\n");
 	return 0;
 }
 

/root/.kpatch/tmp/vmlinux.symtab:

rreadelf -sW /root/.kpatch/tmp/orig/fs/proc/version.o:

readelf -sW /usr/src/linux-6.4.0-1.0.1.4.oe2309.x86_64/vmlinux:

[he7850]

so find_local_syms error happens because vmlinux.symtab does not contains __pfx_ symbols but version.o contains them, and local symbols cannot match.

maybe maybe_discarded_sym() should add __pfx_ pattern?

[he7850]

so find_local_syms error happens because vmlinux.symtab does not contains "_pfx" symbols but version.o contains them, and local symbols cannot match.

maybe maybe_discarded_sym() should add "_pfx" pattern?

seems that this patch can solve the find_local_syms error:

diff --git a/kpatch-build/lookup.c b/kpatch-build/lookup.c
index bd2b732..2008cce 100644
--- a/kpatch-build/lookup.c
+++ b/kpatch-build/lookup.c
@@ -83,6 +83,7 @@ static bool maybe_discarded_sym(const char *name)
            !strncmp(name, "__brk_reservation_fn_", 21) ||
            !strncmp(name, "__func_stack_frame_non_standard_", 32) ||
            strstr(name, "__addressable_") ||
+           strstr(name, "__pfx_") ||
            strstr(name, "__UNIQUE_ID_") ||
            !strncmp(name, ".L.str", 6) ||
            is_ubsan_sec(name))