What is the purpose of libcmd.a (nothing related with libmd)

[phidebian]

Hi All,

I am working (investigating) about doing some loadable builtins of my own, as I used to do with dtksh :-)

I discovered that OOTB build don't bring builtin loading capability.

TC$ builtin -s -f ./libbksh.so sum
builtin: adding built-ins not supported
Usage: builtin [-dls] [-f lib] [pathname ...]
 Help: builtin [ --help | --man ] 2>&1

Then I discovered that libcmd/ provide a set of loadable builtins, a good start to ramp up my knowledge and try to build my own hello world first one, but looking at the RELEASE file I can read

This file is of historical interest only. For recent changes in both ksh 93u+m
and the accompanying libraries, see the file NEWS in the top-level directory.

Then I discovered that despite this scary message the OOTB build do build libcmd.a that nobody use, may be could be used with some SHOPT.sh flags setting, but I am not that far.

I still discovered that I can bring loadable builtin capability by editing SHOPT.sh

SHOPT DYNAMIC=0<bs>1                         # dynamic loading for builtins

This build complete, with some QA failure (4 errors), yet now I can do builtins loading. I made some quick and dirty hack in the BE to get libcmd.so instead of libcmd.a and try to load a builtin from there and it seems to work.

TC$ builtin -s -f ./libcmd.so wc

TC$ type wc
wc is a shell builtin

TC$ echo -n 123 | wc -c
       3

So my questions.

• Does the libcmd.a is still used to build a non DYNAMIC ksh with all the libcmd/ pre installed as additional builtins ? (sum, wc, etc...) if so how?

• Does a SHOPT DYNAMIC=1 build supposed to work ? i.e should I spend time to fix the few regression in the QA, or is it to be abandoned?

• What is the concept of PLUGINS I can see some mention of it, but from my ancient dtksh knowledge I don't know what this is, is that the same as dynamic builtin load or is it something else.

Thanx in advance.

[phidebian]

I ramping up my knowledge (slowly) of loadable builtin, plugin concept. I got the impression that plugin libdll/dllplug.c is something built around dll load to implement a kind of auto loading for builtins, may be something to avoid DGK function autoloading doing the builtin load.

Anyway trying to understand how the prebuild autoloadable builtins work the ones found in libcmd and then trying to get a workable QA suite for SHOPT_DYNAMIC 1 build I discover this that sounds shocking to me, but may be it works as expected.

On a regular ksh build (static build)

TC$ builtin                                                                   
.
/opt/ast/bin/basename
/opt/ast/bin/cat
/opt/ast/bin/cp
/opt/ast/bin/cut
/opt/ast/bin/dirname
/opt/ast/bin/getconf
/opt/ast/bin/ln
/opt/ast/bin/mktemp
/opt/ast/bin/mv
:
[
alarm
alias
autoload
bg
break
builtin
cd
command
compound
continue
....

TC$ type cat
cat is a tracked alias for /usr/bin/cat

TC$ type /opt/ast/bin/cat     
/opt/ast/bin/cat is a shell builtin
TC$ /opt/ast/bin/cat   --help
Usage: cat [ options ] [file ...]
 Help: cat [ --help | --man ] 2>&1
OPTIONS
  -b, --number-nonblank

TC$ ls /opt/ast/bin
ls: cannot access '/opt/ast/bin': No such file or directory

To me it is shocking to have builtins (like cat) located in a non existing directory /opt/ast/bin.

Is that expected, I don't see any ref of /opt/ast/bin on my man ksh.

I planned a patch to not use such a dir if not existing to solve a a test failure in path.sh test when compiled SHOPT_DYNAMIC 1, but before doing so I want to know if this is the way to approach this.

Thanx in advance.

[phidebian]

Ha, I knew I red something about sh shared libraries but could not find it, it is in fact the on going work at
#357

Well so far I use my own build env, and I simply turn SHOPT_DYNAMIC 1 and I got loadable builtins.

I got some regression in QA mostly path.sh, and is related to /opt/ast/bin/* that I discovered is call a virtual something (file, dir). I am not yet in the versioning problematics.

[phidebian]

What is the official way to build SHOPT_DYNAMIC 1

• Is it by editing ./src/cmd/ksh93/SHOPT.sh and set SHOPT DYNAMIC=1
• Is it by giving bin/package make some CCFLAGS like
  ./bin/package make SHOPT_DYNAMIC=1 CCFLAGS='-m64 -O0 -g3'

Thanx in advance.

[JohnoKing]

The official way to build with dynamic library support is incomplete (cf. #357), though the correct way is by setting SHOPT DYNAMIC to 1 in src/cmd/ksh93/SHOPT.sh (don't pass the flags to bin/package manually). Additionally, to enable all of the path-bound builtins (i.e., those tied to the virtual /opt/ast/bin dir), you can set SHOPT ALL_LIBCMD to 1 in the same file (this doesn't depend on SHOPT_DYNAMIC).

For more information on the /opt/ast/bin builtins, these issues and pull requests are relevant:
#349
#138
#275

[phidebian]

Thanx @JohnoKing for the pointers I will review the various # ref.
Meanwhile I had figured the purpose of SHOPT.sh::ALL_LIBCMD purpose, I see how additional builtin comes in.

For what I see so far, the SHOPT_DYNAMIC=1 seems good, almost all works except a little glitch in ./src/cmd/ksh93/sh/path.c I proposed patch #673 with that patch I can proceed.

I got the impression that what is called PLUGIN here is something to avoid DGK initial setting for builtin autoloading that was based on ksh93 function autoloading, the autoloaded function doing the job of finding and loading the lib.sl well lib.so nowaday. The autoloaded function could do a lot of things, and I used this (while back) to implement a package manager (ala perl, python, name it) to centralise function and in some case C code (builtin calling C code)

I think the PLUGIN adds a lot of complexity in the code along with the potential bugs that goes with, I like the keep it simple principle along with if not broken don't fix it. So in my project I will stay away from plugin doing only explicit builtin -f.

I find strange that SHOPT_DYNAMIC=1 is not the default build, it was a major plus at the time (dtksh era) over bash or any other shell, it took decade for bash to start think of it, and they started some implementations, not sure they have function autoloading.