Support for weak or lazy symbols

[ghaerr]

Hello @jmalak,

I have a need to mark a symbol such that, when linking, other libraries will not normally be searched to find and pull in the referenced symbol unless the symbol has already been defined in object files already committed to being linked. In GCC, this is known as weak symbols, and can be specified using the GCC __attribute__((__weak__)) extension.

Can the OpenWatcom linker support weak symbols? I can't seem to find any direct documentation on it, although by looking at the OW source, I see that OBJ EXTRN comment records are apparently used (for import) to specify weak and lazy symbol attributes. (See

open-watcom-v2/bld/cg/intel/h/x86objd.h

Line 87 in 8eb58ce

#define WEAK_EXTRN_COMMENT CMT_NP( WKEXT )

as well as IS_SYM_LINK_WEAK in wl/c/objpass1.c, and WEAK in many more places).

There is also some documentation identified on google but which I cannot find the full manual for, saying "Open Watcom Linker User's Guide ... The "EXPORT" directive is used to tell the Open Watcom Linker which symbols are available for import ... A lazy external reference is one which has two" but when I folllow the link I can't find the referenced text. So possibly there is a way to use the EXPORT or IMPORT directives to specify a weak symbol, rather than C source.

My requirement to use weak symbols is for optionally including various routines only when needed in the ELKS C Library, e.g. for including another function printf might call to perform float-to-string display only when floats are used, etc.

Can you point me into the right direction for how to take advantage of this possible internal capability in OpenWatcom?

Thank you!

[jmalak]

It is not supported by OW compilers for user code.
It is used internally by code generator for common block (fortran) or for C++ code I think. May be other reason.
The linker is fully capable to process it.
But use of it can be danger and it is not required for current code.
We can do it as extension but it need review.
Probably we could extend #pragma aux for it.

[ghaerr]

Below is code generator calling compilers front-end for such info (it is OMF output).

    case IMPORT_IS_WEAK:
        OutShort( WEAK_EXTRN_COMMENT, cmt );
        OutIdx( idx, cmt );
        OutIdx( def_idx, cmt );
        break;

Ah yes, great! Given that the WKEXT comment record contains both idx, the weak EXTDEF index and def_idx, the default EXTDEF symbol to resolve to when resolved as weak, how are you thinking we might force an unresolved weak symbol to have an address of 0? It would seem we might need an internal symbol that is guaranteed to be linked to address 0? Or would a new pragma aux just take two symbols (weak and default resolution) and leave it up to the programmer to work out whether &sym == &default_resolution_sym or not? That would likely require two extern definitions in C source file, with the default resolution being strong and always linked. Correct?

[ghaerr]

how are you thinking we might force an unresolved weak symbol to have an address of 0?

Looking at the wlink source, it seems that when a weak symbol is never defined, the linker doesn't bother setting the weak symbol value to the default symbol in the OMF record. Instead, it just leaves it as is, which would be zero:

void ResolveUndefined( void )
    ...
        for( sym = HeadSym; sym != NULL; sym = sym->link ) {
            if( ( (sym->info & SYM_DEFINED) == 0 && !IS_SYM_WEAK_REF( sym )
                || (FmtData.type & MK_NOVELL) && IS_SYM_IMPORTED( sym )
                  && (sym->info & (SYM_REFERENCED | SYM_LOCAL_REF)) )
              && (sym->info & SYM_IS_ALTDEF) == 0 ) {
                LibFind( sym->name.u.ptr, (sym->info & SYM_CHECKED) != 0 );
            }
            sym->info |= SYM_CHECKED;
        }

[jmalak]

Below is code generator calling compilers front-end for such info (it is OMF output).

    case IMPORT_IS_WEAK:
        OutShort( WEAK_EXTRN_COMMENT, cmt );
        OutIdx( idx, cmt );
        OutIdx( def_idx, cmt );
        break;

Ah yes, great! Given that the WKEXT comment record contains both idx, the weak EXTDEF index and def_idx, the default EXTDEF symbol to resolve to when resolved as weak, how are you thinking we might force an unresolved weak symbol to have an address of 0? It would seem we might need an internal symbol that is guaranteed to be linked to address 0? Or would a new pragma aux just take two symbols (weak and default resolution) and leave it up to the programmer to work out whether &sym == &default_resolution_sym or not? That would likely require two extern definitions in C source file, with the default resolution being strong and always linked. Correct?

The default symbol is not used because such feature doesn't exist in ELF or COFF and what is used must be portable over platforms.
That OMF enable it doesn't means that it must be used. Many OMF features is not used for reason portability to other formats. Code generator is designed to use features which is common for all output formats. OMF generator is little specific and is not included in OWL.
Anyway wlink define how it will be used, compiler only output info about symbols necessary for linker. Compilation unit can have only single definition for such symbol.

[jmalak]

Below is code generator calling compilers front-end for such info (it is OMF output).

    case IMPORT_IS_WEAK:
        OutShort( WEAK_EXTRN_COMMENT, cmt );
        OutIdx( idx, cmt );
        OutIdx( def_idx, cmt );
        break;

Ah yes, great! Given that the WKEXT comment record contains both idx, the weak EXTDEF index and def_idx, the default EXTDEF symbol to resolve to when resolved as weak, how are you thinking we might force an unresolved weak symbol to have an address of 0? It would seem we might need an internal symbol that is guaranteed to be linked to address 0? Or would a new pragma aux just take two symbols (weak and default resolution) and leave it up to the programmer to work out whether &sym == &default_resolution_sym or not? That would likely require two extern definitions in C source file, with the default resolution being strong and always linked. Correct?

you must look on this problem in wider context over all OW supported platforms don't concentrate on single platform. it is not necessary feature and is not available on all platforms therefore is not used at all.

[ghaerr]

you must look on this problem in wider context over all OW supported platforms

The default symbol is not used because such feature doesn't exist in ELF or COFF and what is used must be portable over platforms.

Yes, I see now. Agreed. And this makes things quite a bit simpler, as in the OMF case, idx and def_idx can be the same (weak) symbol index.

[jmalak]

OW support only alias for regular symbol. It is not same , but it could help you in some situation.
#pragma alias(id1/"name1", id2/"name2")
That you can reference existing code with some symbol by new name, then both symbols point to single instance of code.

[ghaerr]

Another option would be post-processing of OBJ file and adding WKEXT record. However, that is lots of work unless there's an existing OMF/OBJ manipulation library

I just found a very small OBJ post-processing utility: fcenable (bld/wl/fcenable/c). This utility normally enables the far call optimization performed by the linker by inserting another OMF record in OBJ files produced by other tools. It is a very small and simple program.

In our case, it would seem to be quite simple to enhance it to take a symbol name and add a WKEXT record in the same way, thus defining the passed symbol as weak, run as a compiler post-processing tool: fcenable -w weaksym vfprintf.obj, for instance. The WKEXT record could take idx and def_idx to be the same symbol, since it appears def_idx isn't used.

Of course, this would be a bit of a kluge but might be quite a bit simpler than introducing a compiler change, and could be done first for testing. What do you think?

[jmalak]

I have a question why you need GCC __attribute__((__weak__)) ? I didn't need something like this with OW.
If I understand it then gcc is not able determine correct version of some symbol if it is defined multiple times.
The OW linker use explicit order of object files and libraries used for linking and also have priority for libraries used during linking.
I think you don't need something as this extension because OW linker use first occurence of symbol with regard to order of used/serched modules. If you need replace some symbol in some library simply create module with this symbol and add it to linker list of modules and linker pickup this module and ignore this symbol in any other module or library. Simply what is first on command line has highest priority. Also modules has priority before libraries. you can define library for linker in source code with appropriate priority too.

[ghaerr]

Don't go this way, much simpler is add some pragma for front-end and it will generate necessary info

It is simple attribute for symbol nothing strange.

Ok. Yes, all we need is simple weak attribute tagged to extern symbol, and everything should work.

If I understand it then gcc is not able determine correct version of some symbol if it is defined multiple times.

No, the problem is not determine correct version of a symbol, and library ordering can't solve it. The problem being solved is that of optionally including a routine (in the vfprintf case, dtostr double-to-string conversion) only when an application needs it. When an application does not need it, the resulting code size is smaller.

In the current ELKS C library using OW, the resulting binary application sizes are large and always include floating point conversion dtostr routine, even when not needed. In ELKS application, a macro __STDIO_PRINT_FLOATS is used within to signal that dtostr should be included into vfprintf.

In order to keep things simple, I have only told you half the story. The other half is that an application, in this case basic, will contain a statement, something like __STDIO_PRINT_FLOATS which is a macro which ends up defining a strong reference to the same (dtostr in this case) symbol. That ultimately causes the double-to-string conversion routine to pulled into vfprintf, so that %g used in basic will work.

For more details, see libc/stdio.h:

void dtostr(double val, int style, int preci, char *buf);
/* use this macro to link in libc %e,%f,%g printf/sprintf support into user program */
#define __STDIO_PRINT_FLOATS    __LINK_SYMBOL(dtostr)

and then sys/linksym.h:

#ifdef __GNUC__
#define __LINK_SYMBOL(sym) \
                          asm(".pushsection .discard; " \
                              ".long " #sym "; " \
                              ".popsection")
#endif

#ifdef __WATCOMC__
extern void __LINK_SYMBOL(void (*sym)());
#pragma aux __LINK_SYMBOL = __parm [ __ax];
#endif

This code is all quite tricky, but I have included it for your understanding of the problem. This mechanism is used in other places in the ELKS C library. I would like to port it using weak symbols rather than using a new method for the entire C library just because porting to OW.

All said, can you point me to source code in OW C compiler that deals with pragma aux processing? I am not familiar with what routines compiler must call in order to tag an extern definition as weak.

[ghaerr]

Also there are others that are interested in weak symbol support in OW, see https://stackoverflow.com/questions/76284513/does-openwatcom-support-an-equivalent-to-elf-weak-symbols.

[jmalak]

The problem with floating point stuff is simply solvable. OW compilers generate reference to special symbol _fltused_ into module code if any floating point variable is used in user code that linker add appropriate floating point code to executable as soon as you used any floating point. It uses default math library name instruction in module code for linker to automaticaly load floating point math library for linking that proper version of code is loaded. It doesn't need such jigery-pokery and code is transparent and you don't need such constructs in header files.

[ghaerr]

The problem with floating point stuff is simply solvable.

I mention floating point only as an example. Look at other parts of vfprintf.c:

         case 'k':              /* Pticks */
          usproc:
            l = lval? va_arg(ap, unsigned long) : (unsigned long)va_arg(ap, unsigned int);
#ifndef __WATCOMC__
            if (*fmt == 'k') {
                if (_weaken(ptostr)) {
                    (_weaken(ptostr))(l, ptmp);
                    preci = -1;
                    goto printit;
                }
                /* if precision timing not linked in, display as unsigned */
            }
#endif

Our precision timing routines have to be removed because of lack of OW support. We can go ahead and ifdef out code when Open Watcom is used, but I would prefer to have support. This isn't jiggery-pokery but instead macros that support weak symbols on various platforms in a way that keeps binary executable sizes quite small for various reasons. You must think of the bigger picture and not just OW when looking at ELKS C library.

OW compilers generate reference to special symbol fltused into module code if any floating point variable is used in user code

Of course. However, in your PR ghaerr/elks#1944 and committed in ghaerr/elks#1948, the -zls compiler option is now used which removes generating special symbols, at your suggestion. I mention all this to you before, and have been planning on using weak symbols when turning off OWC special symbols, knowing full well how it works. What I didn't know is that weak symbols aren't supported on OW.

But that's not the issue here - I'm trying to answer your question of why weak symbol support is needed. I will try modifying fcenable.c to emit the WKEXT OMF record for further testing.

[ghaerr]

I will try modifying fcenable.c to emit the WKEXT OMF record for further testing.

Although having a #pragma aux sym __weak would be great, in my case, more jigery-pokery might have to be done, just as the klugery of running a post-processor also is. That is, instead of having a self-contained macro like _weaken(sym) that automatically defines the symbol as weak and returns its value, the macro would have to contain a #pragma in itself, which I don't think is possible. Thus, the programmer would have to use the pragma outside and along with the _weaken macro in order to support both OW and GCC.

[ghaerr]

More information about weak symbol support in current Open Watcom linker and C compiler:

I went ahead and modified fcenable's (far call optimizer) OBJ manipulation routines to emit a WKREF weak symbol reference for a specified symbol, which now works. This allowed further testing to see how wlink handles weak symbols, both when resolved and left unresolved. For small code models, the result works as expected: the symbol (in the tested case, an external function's address) is indeed set to 0.

For large code models, there are some potential problems. Since the C compiler didn't know about the weak symbol, the code generation was identical, but I'm pretty sure the code generated would have to be the same. Before calling a weak function symbol, it must be tested to determine if the function is in fact linked in or not.

In large model, the code generated for testing the address of a weak function symbol is:

extern int ptostr(); // <-- marked as weak with OBJ file post-processing
    if (ptostr) {
    ....

0302  B8 00 00                          mov             ax,seg ptostr_
0305  85 C0                             test            ax,ax
0307  75 07                             jne             L$49
0309  B8 00 00                          mov             ax,offset ptostr_
030C  85 C0                             test            ax,ax
030E  74 36                             je              L$52
0310                            L$49:

What happens in the linker is that the value of seg ptostr_ is left as the current code segment, not 0. The offset is zero. Thus in large model, the test for a weak symbol being non-zero as set by the linker always succeeds and the function would be called as SEG:0, and the segment is never zero. This usually causes a program crash.

This can be kind of fixed by just testing the low 16 bits of the function address; but in large model it is possible that if the weak symbol happens to be linked into the first (zero) offset of a new segment, its strong reference offset would also be zero. With the ELKS C library, the first code segment offset zero won't be used, so this would work for programs < 64k code.

I am guessing that even with pragma aux weak the C compiler would have to generate a segment reference, as the compiler doesn't know whether the reference will ultimately be resolved weak or strongly by the linker. This means that even with pragma aux weak, the linker probably needs modification to set the segment value of an unresolved weak symbol to zero in order for things to function properly. [EDIT: it is possible the linker is already doing this and there is a non-cancelled FIXUP relocation entry that is causing the segment value to be set from non-zero. I will have to check more carefully to see if that is the reason for this problem.]

Another possible fix would be to somehow tell the linker to not place any functions at address zero in any code segment. I haven't found a way to specify or guarantee that with existing commands/options.

[jmalak]

No, it need to use also code generator support to implement it by some #pragma for weak symbols.
It is not only about generate object record but also about segments as you noted (for segmented 16-bit architecture if symbol is far, near symbol is trivial).
Code generator must generate some different segment (probably shared) that linker can do resulting decision and can assign final segment because linker must decide, code generator doesn't know anything about it.
OW code generator has such code for generating C++ code where it is commonly used.
That implementation need simple modification in C front end and combine it with usage appropriate code in code generator used for segment support for weak symbols.
I will need to look in the appropriate code generator code in details to implement weak symbols in C compiler, some new attribute will be need probably.
Add new pragma to front-end is simple, but modify code generator appropriate code need more investigation for me to not broke anything. I am planning some work in code generator for various problems that I will add weak symbol implementation to this activities.

I looked on IBM compilers #pragma weak that it could use same syntax for OW, but need some investigation if 16-bit segmented architecture need something specific or not. Probably we will need extend it to support native and C symbols due to symbol names mangling.