Add support for AppBundle, FlatImage... and other formats!

[ivan-hc]

I'm the author of the AppBundle format and tooling, integration is fairly easy to do. You can use the following flags if you already handle integration for AppImages:
--pbundle_link, this command is provided by the runtime and lets you run any command but from within the AppBundle. Its very powerful. Let's you do this, for example:
--pbundle_link sh -c "cp \$SELF_TEMPDIR/.DirIcon ~/.local/share/icons/firefox.png"
You also have the following commands:
--pbundle_desktop | base64 -d
--pbundle_pngIcon | base64 -d
--pbundle_appstream | base64 -d
--pbundle_list

You can check the exact build-date of an AppBundle via --pbundle_help.

I'm working on AppBundleHUB, which should provide metadata for the AppBundles it generates.
If you've looked at any of the recipes, you'll realice they're mostly one-liners like the following:
pelfCreator -m "xplshn" -n "org.chromium.Chromium" -p "chromium" -e "chromium.desktop", when they include a -e that ends in .desktop, that means that the produced AppBundle will have an icon & desktop file. (if you don't like this automatic method of building/doing things, you can use the pelf-dwfs tool to create AppBundles of any AppDir, or the bs2appbundle tool to convert conty-based AppImages to AppBundle and it also supports normal AppImages.)

Anyhow. I'm open to suggestions :)

Originally posted by @xplshn in #1079 (comment)

[ivan-hc]

@xplshn @Azathothas and @Samueru-sama , with participation request of @ruanformigoni , welcome everyone.

Please read carefully what I am about to tell you.

Now that the 9.1 release supports Toolpacks, it is good to explain how they are installed with "AM". This is the function that is used:

	if echo "$arg" | grep -q "appimage$"; then
		curl -Ls "$AMREPO"/templates/AM-SAMPLE-AppImage > "$CACHEDIR/toolpack/$arg.toolpack" || exit 1
	else
		curl -Ls "$AMREPO"/templates/AM-SAMPLE-Archive > "$CACHEDIR/toolpack/$arg.toolpack" || exit 1
	fi
	sed -i "s#APP=SAMPLE#APP=$arg#g; s#FUNCTION#$toolpack_version#g; s#wget \"\$version\"#wget \$($toolpak_url)#g" "$CACHEDIR/toolpack/$arg.toolpack"
	sed -e '/\[ -e/ s/^#*/#/' -i "$CACHEDIR/toolpack/$arg.toolpack" 2>/dev/null

as you can see, an if/fi statement is used to check which of the two patterns should be used to create the on-the-fly installation script.

Well, the scripts (authored by @Samueru-sama) if you look closely, are pretty much identical:

• AppImages https://github.com/ivan-hc/AM/blob/main/templates/AM-SAMPLE-AppImage
• Archives https://github.com/ivan-hc/AM/blob/main/templates/AM-SAMPLE-Archive

what changes is the final part. The AppImage script ends with this:

# LAUNCHER & ICON
./"$APP" --appimage-extract *.desktop 1>/dev/null && mv ./squashfs-root/*.desktop ./"$APP".desktop
./"$APP" --appimage-extract .DirIcon 1>/dev/null && mv ./squashfs-root/.DirIcon ./DirIcon
COUNT=0
while [ "$COUNT" -lt 10 ]; do # Tries to get the actual icon/desktop if it is a symlink to another symlink
	if [ -L ./"$APP".desktop ]; then
		LINKPATH="$(readlink ./"$APP".desktop | sed 's|^\./||' 2>/dev/null)"
		./"$APP" --appimage-extract "$LINKPATH" 1>/dev/null && mv ./squashfs-root/"$LINKPATH" ./"$APP".desktop
	fi
	if [ -L ./DirIcon ]; then
		LINKPATH="$(readlink ./DirIcon | sed 's|^\./||' 2>/dev/null)"
		./"$APP" --appimage-extract "$LINKPATH" 1>/dev/null && mv ./squashfs-root/"$LINKPATH" ./DirIcon
	fi
	[ ! -L ./"$APP".desktop ] && [ ! -L ./DirIcon ] && break
	COUNT=$((COUNT + 1))
done
sed -i "s#Exec=[^ ]*#Exec=$APP#g; s#Icon=.*#Icon=/opt/$APP/icons/$APP#g" ./"$APP".desktop
mv ./"$APP".desktop /usr/local/share/applications/"$APP"-AM.desktop && mv ./DirIcon ./icons/"$APP" 1>/dev/null
rm -R -f ./squashfs-root

...the other script has nothing, as it is meant to be completed via the -t option asking the user to add a URL to an icon and create a .desktop file.

Why am I telling you this?

If you want, the Archives script can also be used for AppImages, with the only difference that that piece should be added at the end.

If the script is not unique it is because of the creation of an AM-updater file for updates that contains additional steps, but that's another story.

What interests us is that it is possible to use a HEREDOC to add at the end of the file the necessary functions to extract/create icons and launchers. And all this "on the fly"! Just like it happens with the new Toolpacks implementation, in the new "AM" 9.1.

That said, keep in mind that little piece I posted here, and made by @Samueru-sama, in the AppImage scripts, and try to adapt it to AppBundle and FlatImage.

You will have a much better integration of apps installed via Toolpacks or other repositories that you will want to implement in the future.

[Samueru-sama]

AppBundle could just implement --appimage-extract and make it much easier for us.

[xplshn]

AppBundle could just implement --appimage-extract and make it much easier for us.

I don't want to keep compatibility with bugs and end up writting Xorg but in shell. If such a compatibility option were to be added, it'd have to use squashfs-root, but that'd be misleading because we use DWARFS, we don't use SQUASHFS.

I'll invest some time, hopefully tomorrow, to test AM and try and integrate my format to it.

[Samueru-sama]

I don't want to keep compatibility with bugs and end up writting Xorg but in shell. If such a compatibility option were to be added, it'd have to use squashfs-root, but that'd be misleading because we use DWARFS, we don't use SQUASHFS.

You can just make a squashfs-root symlink, which is what the uruntime does already.

This is linux where /usr no longer means user and where the X in XDG no longer means X11.

[Azathothas]

It's best if we stop attributing other formats to be similar to Appimages.
They aren't. Otherwise they would just be Appimage variant like NixAppImage is.

Each format is its own thing, and deserves the extra code that would be needed to be written to integrate that format.
Be it running --pbundle_iconPng etc for AppBundles, or having to create a desktop.json and fetching icon remotely for flatimage,
These need to be handled by their own respective integrator function.

There's going to be more and more of these formats in the future, and not all of them will be easy as Appimage or hard (unless @ruanformigoni adds a way to embed the desktop.json and icon files directly within the flatimage at creation time) as flatimage.
There's also going to be self extracting portable archives, for which you would have to download/create the desktop/icon files manually.

The current way of extracting DirIcon, desktop will have to be changed for any script that aims to install/integrate based on the specific format.
So there's no general solution that fits all, unless ofc AM wants to rely on pkgforge, which does offer downloadable desktop/icon files for all packages that are added, but I don't think that's the preferred choice.

[ivan-hc]

@Azathothas In truth, I know practically nothing about AppBundle and FlatImage, but I find them very interesting.

If it is possible to extract in any way an icon and a .desktop file from the bundle (or script, I have seen the content of an AppBundle) then I can dedicate a function in AppImage style that however uses the commands for AppBundle. Nothing simpler. In fact... if it were necessary to download such files from external sources, it would be good to add additional columns to the file you provided, in which there are the URLs to the icon and the .desktop file.

We can do whatever you want, just tell me how to do these things, and the Archive script will be compiled on the fly.

[xplshn]

I forked and made a simple modification to the AppImage template, its however, not yet tested. I've yet to fire up a container and change the rest of the code, in theory, integrating AppBundles is much easier than AppImages, since a flag is specifically provided to extract the icon, the desktop file, and etc, and you can even do all of these at once if you use --pbundle_link sh -c "cp \$SELF_TEMPDIR/.DirIcon ./DirIcon ; \$SELF_TEMPDIR/*.desktop .", wilder things can be done, but just those flags do everything we need.

Also, for actual integration, there's a program named pelfd that doubles as a daemon, but can also be used as a simple CLI command to integrate a specific AppBundle OR AppImage (yes, it supports both)

Anyways, I'll spin up a container, set up AM (since I don't have most of the deps of AM installed, and don't plan to use apk to install stuff like bash on my system), and then test if the template works.

[ivan-hc]

hey folks, the template provided by @xplshn works! #1103

[ivan-hc]

merged!

simplescreenrecorder-2024-11-12_02.21.42.mkv.mp4

[ivan-hc]

@ruanformigoni as you can see in my previous comment, I was able to handle AppBundles thanks to the template that @xplshn has provided here #1102 and in my test the app is 100% integrated

Basically, the differences between

• https://github.com/ivan-hc/AM/blob/main/templates/AM-SAMPLE-AppImage

...and...

• https://github.com/ivan-hc/AM/blob/main/templates/AM-SAMPLE-AppBundle

is this, for AppImages the end of the script is this

# LAUNCHER & ICON
./"$APP" --appimage-extract *.desktop 1>/dev/null && mv ./squashfs-root/*.desktop ./"$APP".desktop
./"$APP" --appimage-extract .DirIcon 1>/dev/null && mv ./squashfs-root/.DirIcon ./DirIcon
COUNT=0
while [ "$COUNT" -lt 10 ]; do # Tries to get the actual icon/desktop if it is a symlink to another symlink
	if [ -L ./"$APP".desktop ]; then
		LINKPATH="$(readlink ./"$APP".desktop | sed 's|^\./||' 2>/dev/null)"
		./"$APP" --appimage-extract "$LINKPATH" 1>/dev/null && mv ./squashfs-root/"$LINKPATH" ./"$APP".desktop
	fi
	if [ -L ./DirIcon ]; then
		LINKPATH="$(readlink ./DirIcon | sed 's|^\./||' 2>/dev/null)"
		./"$APP" --appimage-extract "$LINKPATH" 1>/dev/null && mv ./squashfs-root/"$LINKPATH" ./DirIcon
	fi
	[ ! -L ./"$APP".desktop ] && [ ! -L ./DirIcon ] && break
	COUNT=$((COUNT + 1))
done
sed -i "s#Exec=[^ ]*#Exec=$APP#g; s#Icon=.*#Icon=/opt/$APP/icons/$APP#g" ./"$APP".desktop
mv ./"$APP".desktop /usr/local/share/applications/"$APP"-AM.desktop && mv ./DirIcon ./icons/"$APP" 1>/dev/null
rm -R -f ./squashfs-root

for AppBundles instead is this

# LAUNCHER & ICON
./"$APP" --pbundle_desktop | base64 -d > ./"$APP".desktop
./"$APP" --pbundle_pngIcon | base64 -d > ./DirIcon
sed -i "s#Exec=[^ ]*#Exec=$APP#g; s#Icon=.*#Icon=/opt/$APP/icons/$APP#g" ./"$APP".desktop
mv ./"$APP".desktop /usr/local/share/applications/"$APP"-AM.desktop && mv ./DirIcon ./icons/"$APP" 1>/dev/null

plus two extension references that change from ./tmp/*mage to ./tmp/*pp*undle... but the scripts are exactly the same.

In your case the tw extension references should change from ./tmp/*mage to ./tmp/*lat*mage (if the extension is "FlatImage"), that is the easier thing.

All we really need is the final step, to have the icon and the launcher.

Would you like to try?

[ruanformigoni]

Hello! Yes, I'll give it a shot.

[ivan-hc]

New release is out https://github.com/ivan-hc/AM/releases/tag/9.1.1

...I hope to improve support for these new package formats in the next release.

[sandboiii]

Hey, I was looking into Flatpak format, as many application developers prefer it over providing AppImages. There are some Flatpak applications I'd like to install with am (I don't like the Flatpak manager).

There's a discussion about the possibility of running Flatpak without installation. Someone has developed a Rust program that might achieve this.

Also, that discussion turned into another about repacking Flatpak into AppImage format (which could then be used in the conventional way). And another person has created a PoC script that does this conversion.

My technical knowledge isn't sufficient to fully grasp how these programs function. If you could somehow integrate Flatpak into am, it would be an incredible feature to have.

[ivan-hc]

hi @sandboiii , "AM" is "limited" to the installation of portable programs and is intended for portable programs.

But as you saw, through installation scripts and in the same way I approached AppBundle here, creating them on the fly, I could handle any other packaging format. It is enough to create a dedicated template, with creation of dedicated scripts for updating and removing into it.

However, I don't want to install or touch a single Flatpak on my system. I had them and used them time ego, even with pleasure to tell the truth... but they adopt an installation scheme that I had little opportunity to study, and that was enough for me to avoid them. It is difficult to completely remove them once installed. Years later I still had directories scattered throughout the system because of that model.

Here, I wrote "AM" specifically to avoid that model.

But if you were to think of creating a template like the ones already existing, I could consider an approach like the one in AppBundle. It's not a problem for me.

[Samueru-sama]

There's a flatpak/flatpak#408 about the possibility of running Flatpak without installation.

I find it funny that probono commented that "Flatpak is optimized for other aspects, e.g., download size efficiency." when this is no longer the case, multiple flatpaks use different runtimes and the user is likely to end up with different runtimes and even different versions of the same runtime, and that explodes the storage usage of flatpak compared to appimage.

Also, that discussion turned into AppImage/AppImageKit#297 about repacking Flatpak into AppImage format (which could then be used in the conventional way).

There is already NixAppImage which is very similar, and I think you can find every single app under the sun in nix.

[xplshn]

Nix is too fat, I made an alternative that generates AppDirs of any Alpine Linux package, but using it for arch is also trivial

[ivan-hc]

Just added NixAppImage support f12b5f4

[ivan-hc]

Since there is just chromium.nixappimage in the list @Azathothas provided, I have tested this on it.

simplescreenrecorder-2024-11-22_21.52.00.mkv.mp4

[Azathothas]

Since there is just chromium.nixappimage in the list @Azathothas provided, I have tested this on it.

No demand for NixAppImages, there's only some rarely: #1151
But the size scares them away.
I have fixed most of the issues, but things like OpenGL needs to be fixed upstream directly, which is out of my control.
So, probably won't be an increase in the number of NixAppImage anytime soon

[ivan-hc]

This new release is an example of how AM would keep have new apps out od its own database, in case I'll can't keep maintain it

https://github.com/ivan-hc/AM/releases/tag/9.1.3

the reason I did this is that AppBundle are now only on a dedicated database. Toolpacks may have portable binaries that may already exist in AM. This is the difference.

[ivan-hc]

@Azathothas I contacted you on Discord, when you have time I left some messages

EDIT: it is urgent

[Azathothas]

Hi, I won't be able to view discord messages as I am no longer there.
If you aren't able to dm me on telegram, please write it here

[ivan-hc]

I noticed that your list of toolpacks has a lt of duplicates with different descriptions, can you solve this?

I also noticed that -a is affected, it gets the first reference, and all first references in duplicates have tha same description

[Azathothas]

As stated in #1079 (comment)
These are package family

If you want me to remove them, I will have to exclude the entire baseutil dir
If that's what you want, sure

[ivan-hc]

Maybe the missing ones deserve a separated list I guess... but in the meantime its better you exclude them. Thanks.

[Azathothas]

pkgforge/pkgcache@8485365 now excludes the entire baseutils dir
https://bin.pkgforge.dev/aarch64/AM.txt
https://bin.pkgforge.dev/x86_64/AM.txt

[ivan-hc]

thank you @Azathothas

It's strange to see so few programs compared to before, but since they are divided into families, they should have deserved a separate extension to be distinguished from each other (I would have removed the extension during installation). But for now it's better that we solved it like this. Thanks again.

[Azathothas]

The way AM is designed, I don't see how pkg_family could be supported easily.
Xplshn has to create his own metadata because he couldn't figure out either.

This is how it's done in rust: https://github.com/pkgforge/soar/blob/main/src/registry/fetcher.rs#L46
It's splitter based on URL, if you can translate that to bash, you can support family, no unnecessary metadata modification needed

The correct count of total pkg is:

[ivan-hc]

I normally rely on extensions for this, as I did with AppBundle, I also have packages like brave and brave-appimage, which are installed as brave, or firefox and firefox-appimage or avidemux and debian-testing-avidemux which are installed as firefox and avidemux respectively. Extensions are truncated, and the -a option detects which package was used to install the app

I started writing this program more than 3 years ago. If I had known that one day I would implement third-party databases, I would have structured it differently.

[ivan-hc]

That said, it seems that families of programs cand have a dedicated list. This would help me in adding different extensions, depending on the list provided (below the completion list, note how extension changes when switching from "AM" to AppBundles to Toolpacks)

[ivan-hc]

@Azathothas can you show me the URLs of (for example) all awk you have for each family?

I just got an idea, but I'm notsure if it will work. I need to do some tests.

[Azathothas]

Hi, please for the last time, join telegram: https://t.me/official_loonix
The texts here aren't realtime, and I won't be able to respond properly.

GitHub Issues shouldn't be used as chat, and this is going offtopic now.
If you aren't willing to use telegram, I am not willing to respond here either.
Hope you understand.

[ivan-hc]

OK, never mind. Thanks anyway.