Remove usage of grep, sed, find from functions.sh

[dcermak]

Problem description

We are currently building very minimal container images and try to exclude as much as possible from them, for instance sed and grep. Unfortunately, that results in kiwi's config.sh failing as it relies on sed and grep to exist.

It would be nice if kiwi could work without sed and grep being present in the build root.

OS and Software information

• KIWI version: 9.23.20
• Operating system: (irrelevant)
• OBS version: build.opensuse.org

[dcermak]

I'm wondering whether we could simply leverage python for anything that is not part bash, as kiwi depends on python anyway?

[schaefi]

The entire functions.sh needs a refactor. Please note most of the stuff is only there for backward compatibility with custom scripts that might use this methods.

Ideally I would like to get rid of almost everything functions.sh provided. We added these methods to support users with common functions for simple setup tasks. In the end however, it has turned out that this causes a lot of maintenance effort as distributions changes all the time and there is actually no common way to setup a system for feature X. Looking back I think it was a mistake to offer it. There are many opportunities to roll out system configurations and way better tools like ansible and friends to do it. Integrating them in a kiwi image is pretty easy and moves the responsibility for the system setup to the right place.

As it's hard to simply remove published stuff the code is there and gets worse while the distros changes. So from my perspective it would be really good if we can find a strategy to refactor/reduce the code from functions.sh into a state that is useful, stable and reliable.

Thoughts ?

[dcermak]

Ideally I would like to get rid of almost everything functions.sh provided. We added these methods to support users with common functions for simple setup tasks. In the end however, it has turned out that this causes a lot of maintenance effort as distributions changes all the time and there is actually no common way to setup a system for feature X. Looking back I think it was a mistake to offer it.

I'm afraid this is going to be very hard as many config.sh scripts in the buildservice just get copied over from the previous release and no one even knows what these functions do and whether they are necessary (hint: in 99% of the cases, they aren't!). So if you eventually really remove certain functions, you'll get a lot of angry people even if you add huge THIS WILL BE REMOVED!!! warnings (because who looks at build logs if they don't fail your build?). Maybe a less brutal way would be to make deprecated functions error out unless the user does a EXPORT DONT_ERROR_OUT_ON_DEPRECATED_FUNCTIONS_I_KNOW_THIS_WILL_BITE_ME_LATER=1?

There are many opportunities to roll out system configurations and way better tools like ansible and friends to do it. Integrating them in a kiwi image is pretty easy and moves the responsibility for the system setup to the right place.

It is doable, but not super straight forward. I think what would make people really switch to Ansible or Salt, if you could include your playbook/state files in the image description directly.

As it's hard to simply remove published stuff the code is there and gets worse while the distros changes. So from my perspective it would be really good if we can find a strategy to refactor/reduce the code from functions.sh into a state that is useful, stable and reliable.

My recommendation would be to start deprecating everything that does not provide meaningful functionality or is just there for legacy reasons and gate the rest using a test suite like I started introducing in #1827.

[schaefi]

My recommendation would be to start deprecating everything that does not provide meaningful functionality or is just there for >legacy reasons and gate the rest using a test suite like I started introducing in #1827.

yes I fully agree and I also like to the idea to gate the rest code via a test suite. In addition to your test suite draft I have created my idea of refactoring the code. As expected this will cause conflicts but I'm sure they can be solved relatively easy

[schaefi]

Fixed with #1861

[dcermak]

Fixed with #1861

Not really, that was just one tiny bit:

✦ ❯ rg 'sed|grep|find' kiwi/config/functions.sh|wc
     34     238    1815

[schaefi]

ok but if we keep this open as a generic issue to fix step by step I suggest that we add more structure, define a task list for the methods that should be fixed and allow to share work across people. It might also be a good idea to create a function test first then adapt. Here is the list to start with imho

• baseUpdateSysConfig (sed)
• baseStripModules (sed, find)
• baseQuoteFile (sed, grep)
• baseFixupKernelModuleDependencies (sed, find)
• baseStripTranslations (grep, find)
• baseStripFirmware (grep, find)
• baseStripAndKeep (grep)
• baseStripLocales (find)
• baseStripKernelModules (find)
• baseStripTools (find)

[dcermak]

It might also be a good idea to create a function test first then adapt.

That should be a must imho.

[SamirPS]

Hello where is this issue, what need to be done now ? @schaefi @dcermak

[schaefi]

@SamirPS Hi This is about the code you can find in https://github.com/OSInside/kiwi/blob/master/kiwi/config/functions.sh

The idea here is to reduce the external requirements for tools called as part of this code. For example sed, find, grep ... if possible. The driver for the change is that with a reduced set of external tools you don't need them anymore in the environment and so you can build with less dependencies, smaller images

The tough part here is to make changes without introducing regressions. Because of that @dcermak created an integration testing framework for the shell code which allows us to run the methods see their results, then refactor and expect the results to stay unchanged.

If you would like to participate here it would be much appreciated as I don't expect us to have time for this one. I'm sure @dcermak will help you to get started especially on the testing area

Thanks much

[SamirPS]

Can you show me an example on how replace grep ?

[SamirPS]

@schaefi or @dcermak

[dcermak]

@SamirPS that really depends on the exact usage of grep, but usually you can switch to reading the file line by line and then regex matching each line with bash or sh. See e.g.: https://github.com/OSInside/kiwi/pull/1861/files

Feel free to drop by in the #kiwi chatroom and we can have a chat there.

[SamirPS]

I will try to solve this issue

[SamirPS]

Hello @dcermak when are you available to talk about this issue ?

[dcermak]

@SamirPS You mentioned in #kiwi that you would like to replace find in:

kiwi/kiwi/config/functions.sh

Line 203 in 29b1fe0

find /usr/lib/locale -mindepth 1 -maxdepth 1 -type d 2>/dev/null |\

This shouldn't be too difficult, find -mindepth 1 -maxdepth 1 -type d means that you list all subdirectories of a folder. That can be replicated in pure bash roughly like this:

for f in $(ls -a .); do
  if [ -d $f ]; then
    # do something with $f
  fi
done

There is a catch with the above snippet though: it includes . and .., you'd have to filter these out and of course, write some tests for this function first ;-)

[SamirPS]

Sorry but i didn't have time to finish all, if someone want to take the lead, i let him