Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Define a co-maintainership with Qualcomm, and migrate the layer to a Qualcomm hosted Github org #680

Open
ndechesne opened this issue Nov 22, 2024 · 17 comments
Open

Define a co-maintainership with Qualcomm, and migrate the layer to a Qualcomm hosted Github org #680

ndechesne opened this issue Nov 22, 2024 · 17 comments

Comments

@ndechesne
Copy link
Contributor

hey,

as discussed with a few people privately, we intend to migrate meta-qcom into a new Github org (most likely github.com/quic-yocto) in the near future. Qualcomm has now started to develop a BSP based on Yocto development branch, using linux-yocto kernel (e.g. meta-qcom-hwe layer), and there is no strong reason to maintain 2 competing Yocto BSP. We want to merge meta-qcom and meta-qcom-hwe 'master' branches and have all development done jointly on meta-qcom.

We have a rough goal for this migration to be done between Dec 24 and Feb 25.

There are many technical details to iron out, so let's use this issue to coordinate.

Here is a summary of what we've discussed so far

  1. The migration should be implemented as a Github ownership transfer, with redirection, to minimize the chance of breaking users
  2. meta-qcom is a BSP layer, and any policy/distro configs should be done outside of meta-qcom. Qualcomm maintains meta-qcom-distro for that purpose
  3. We want to migrate the current master branch (and future yocto stable branch will follow from that), but we are not doing any retro-active changes. meta-qcom-hwe will continue to support QC Linux 1.x releases with the kirkstone branch, and all other branches on meta-qcom won't change.
  4. CI: we will move away from current Linaro tuxmake based CI for master branch and use existing Meta-qcom-hwe (main) CI infrastructure.
  5. CI: Keep Linaro CI for old branches (through tuxmake)
  6. Machines. Meta-qcom has ‘generic’ machines only, and no plans to add board specific machines. Linaro will continue to maintain/support the generic machines in the merged layer. There are discussions/efforts to support genericarm64 and genericarm machines from meta-yocto-bsp, but for that this is in addition to meta-qcom generic machines. Qualcomm is interested in the generic machines from meta-yocto-bsp, in addition to the specific machine conf file currently in meta-qcom-hwe (which will migrate to meta-qcom)
  7. Meta-qcom: find . -name *.bb | wc -l  108, let’s review them and remove unnecessary recipes (If any)
  8. Understand what initramfs-firmware-image.bb and firmware-woa are, and how we want to support them after the merge.
  9. Meta-qcom has support for meta-linux-mainline kernel recipes. is anyone using that? do we want to maintain them?
  10. Meta-qcom maintains Mesa_git recipe. Is that still needed? For which purpose?
  11. What are the Binary blobs which are needed in meta-qcom, and how do we handle their redistribution.
  12. meta-qcom will be open for contributions, Qualcomm and Linaro will co-maintain and thus review external contributions.
  13. meta-qcom-hwe supports 2 flavor of BSP (Base and Custom), and these 2 profiles will continue to exist.
  14. Future Qualcomm Linux releases will be based off meta-qcom only (e.g. 2.x, 3.x..), and will be maintain by Qualcomm for productization/commercialization.

We will shortly open a branch (somewhere still TBD) on github.com/quic-yocto, to start working on this
@lumag
Copy link
Collaborator

lumag commented Nov 22, 2024 •
edited
Loading

Here is a list of all the recipes and
appends in the meta-qcom layer together with their descriptions and
some notes.

./recipes-bsp/hexagon-dsp-binaries/hexagon-dsp-binaries_20241017.bb

fastrpc_shell and libraries for the DSP, redistributable, covered by
LICENSE.qcom.txt

./recipes-bsp/firmware/firmware-ath6kl_git.bb

additional ath6k firmware, not suitable for linux-firmware, redistributable

./recipes-bsp/firmware/firmware-qcom-dragonboard410c-bootloader-sdcard_17.09.bb
./recipes-bsp/firmware/firmware-qcom-dragonboard845c_20190529180356-v4.bb
./recipes-bsp/firmware/firmware-qcom-rb3gen2_00039.2.bb

recipes to package redistributable bootloader and firmware binaries for
supported devices.

Note: DB845c archive and recipe also install firmware for the Renesas xHCI
PCI controller. It has been provided to us by Thundercomm in the same ZIP
archive and claimed to be covered by LICENSE.qcom.txt, but further
investigation could not prove this claim. It is being used by the layer on
the premises of being abandonware by the Renesas side, but we don't have a
good enough licence to get it into the linux-firmware. Maybe Qualcomm can
help with that by providing a bigger push on Renesas, especially granted
that RB3gen2 also uses the same chip (which actually requires the
firmware).

./recipes-bsp/firmware/firmware-qcom-dragonboard-apq8074.bb
./recipes-bsp/firmware/firmware-qcom-ifc6410.bb
./recipes-bsp/firmware/firmware-qcom-ifc6560.bb
./recipes-bsp/firmware/firmware-qcom-sm8150-hdk.bb
./recipes-bsp/firmware/firmware-qcom-sm8350-hdk.bb
./recipes-bsp/firmware/firmware-qcom-sm8450-hdk.bb
./recipes-bsp/firmware/firmware-qcom-sm8550-hdk.bb
./recipes-bsp/firmware/firmware-qcom-sm8650-hdk.bb

recipes to package non-redistributable binary firmware for supported
devices (HDKs, devboards). Requires the user to explicitly specify the
NON-HLOS.bin / adreno.tar.gz / dspso.bin (pending) in order to generate
non-empty packages.

./recipes-bsp/firmware-nexus/firmware-qcom-nexus4.bb
./recipes-bsp/firmware-nexus/firmware-qcom-nexus5.bb
./recipes-bsp/firmware-nexus/firmware-qcom-nexus5x.bb
./recipes-bsp/firmware-nexus/firmware-qcom-nexus6.bb
./recipes-bsp/firmware-nexus/firmware-qcom-nexus6p.bb
./recipes-bsp/firmware-nexus/firmware-qcom-nexus7-2013.bb
./recipes-bsp/firmware-nexus/firmware-qcom-pixel.bb
./recipes-bsp/firmware-nexus/firmware-qcom-pixel2.bb
./recipes-bsp/firmware-nexus/firmware-qcom-pixel3.bb
./recipes-bsp/firmware-nexus/firmware-qcom-pixel3a.bb
./recipes-bsp/firmware-nexus/firmware-qcom-pixel4.bb
./recipes-bsp/firmware-nexus/firmware-qcom-pixel4a-5g.bb
./recipes-bsp/firmware-nexus/firmware-qcom-pixel4a.bb
./recipes-bsp/firmware-nexus/firmware-qcom-pixel5.bb
./recipes-bsp/firmware-nexus/firmware-qcom-pixel5a-5g.bb

 recipes to package non-redistributable binary firmware for supported
 Google phones and tablets. Downloads firmware automatically from the
 Google's website.

./recipes-bsp/packagegroups/packagegroup-firmware-dragonboard-apq8074.bb
./recipes-bsp/packagegroups/packagegroup-firmware-dragonboard410c.bb
./recipes-bsp/packagegroups/packagegroup-firmware-dragonboard820c.bb
./recipes-bsp/packagegroups/packagegroup-firmware-dragonboard845c.bb
./recipes-bsp/packagegroups/packagegroup-firmware-ifc6410.bb
./recipes-bsp/packagegroups/packagegroup-firmware-ifc6560.bb
./recipes-bsp/packagegroups/packagegroup-firmware-lenovo-x13s.bb
./recipes-bsp/packagegroups/packagegroup-firmware-rb1.bb
./recipes-bsp/packagegroups/packagegroup-firmware-rb2.bb
./recipes-bsp/packagegroups/packagegroup-firmware-rb3gen2.bb
./recipes-bsp/packagegroups/packagegroup-firmware-rb5.bb
./recipes-bsp/packagegroups/packagegroup-firmware-sm8150-hdk.bb
./recipes-bsp/packagegroups/packagegroup-firmware-sm8350-hdk.bb
./recipes-bsp/packagegroups/packagegroup-firmware-sm8450-hdk.bb
./recipes-bsp/packagegroups/packagegroup-firmware-sm8550-hdk.bb
./recipes-bsp/packagegroups/packagegroup-firmware-sm8650-hdk.bb

packagegroups, selecting necessary firmware and DSP binary packages for
corresponding devices

./recipes-bsp/images/initramfs-firmware-dragonboard820c-image.bb
./recipes-bsp/images/initramfs-firmware-pixel3-image.bb
./recipes-bsp/images/initramfs-firmware-db8074-image.bb
./recipes-bsp/images/initramfs-firmware-sm8550-hdk-image.bb
./recipes-bsp/images/initramfs-firmware-nexus-image.bb
./recipes-bsp/images/initramfs-firmware-rb12-image.bb
./recipes-bsp/images/initramfs-firmware-sm8150-hdk-image.bb
./recipes-bsp/images/initramfs-firmware-ifc6560-image.bb
./recipes-bsp/images/initramfs-firmware-sm8650-hdk-image.bb
./recipes-bsp/images/initramfs-firmware-sm8350-hdk-image.bb
./recipes-bsp/images/initramfs-firmware-sm8450-hdk-image.bb

initramfs images containing firmware to start up remoteprocs and/or Adreno
on corresponding devices

./recipes-bsp/images/initramfs-firmware-image.bb

All-In image pulling firmware for supported RBx and Dragonboard devices.

./recipes-bsp/images/initramfs-firmware-mega-image.bb

All-In firmware image, pulling all packages for all devices, mostly used
for signature validation and (possible) conflict detection

./dynamic-layers/openembedded-layer/recipes-bsp/firmware-woa/firmware-qcom-ecs-liva-qc710_200.0.10.0.bb
./dynamic-layers/openembedded-layer/recipes-bsp/firmware-woa/firmware-qcom-lenovo-miix-630_200.0.6.0.bb
./dynamic-layers/openembedded-layer/recipes-bsp/firmware-woa/firmware-qcom-lenovo-yoga-c630_200.0.19.0.bb
./dynamic-layers/openembedded-layer/recipes-bsp/firmware-woa/firmware-qcom-sc8180x_200.0.111.0.bb
./dynamic-layers/openembedded-layer/recipes-bsp/firmware-woa/firmware-qcom-x1e80100_200.0.32.0.bb

Recipes packaging firmware for several WoA devices by downloading the
binaries community's GitHub repo. Licence status is pretty unclear, as WSUS
archives contain little to no licence information. Downloading from GH repo
can be replaced with accessing WSUS directly, the tool has been implemented
by the UUPDownload project, but it is written in .NET, so it has a huge set
of external dependencies. Ideally the tool should be reimplemented in
Python or in C, so that we can use it to create WoA firmware packages on
laptops themselves (for "normal" distros).

./dynamic-layers/openembedded-layer/recipes-bsp/packagegroups/packagegroup-firmware-ecs-liva-qc710.bb
./dynamic-layers/openembedded-layer/recipes-bsp/packagegroups/packagegroup-firmware-lenovo-miix-630.bb
./dynamic-layers/openembedded-layer/recipes-bsp/packagegroups/packagegroup-firmware-lenovo-yoga-c630.bb
./dynamic-layers/openembedded-layer/recipes-bsp/packagegroups/packagegroup-firmware-sc8180x.bb
./dynamic-layers/openembedded-layer/recipes-bsp/packagegroups/packagegroup-firmware-x1e80100-crd.bb

Packagegroups pulling corresponding firmware images (both from
linux-firmware and from WoA firmware packages).

./dynamic-layers/openembedded-layer/recipes-bsp/images/initramfs-firmware-ecs-liva-qc710-image.bb
./dynamic-layers/openembedded-layer/recipes-bsp/images/initramfs-firmware-lenovo-miix-630-image.bb
./dynamic-layers/openembedded-layer/recipes-bsp/images/initramfs-firmware-lenovo-yoga-c630-image.bb
./dynamic-layers/openembedded-layer/recipes-bsp/images/initramfs-firmware-sc8180x-image.bb
./dynamic-layers/openembedded-layer/recipes-bsp/images/initramfs-firmware-x1e80100-crd-image.bb
./dynamic-layers/openembedded-layer/recipes-bsp/images/initramfs-firmware-mega-image.bbappend

Initramfs images containing these firmware files.

./recipes-kernel/linux-firmware/linux-firmware_%.bbappend

Layer-specific customization to handle conflicts with other

firmware packagages

./recipes-support/initrdscripts/initramfs-module-copy-modules_1.0.bb
./recipes-kernel/packagegroups/packagegroup-qcom-boot.bb
./recipes-kernel/images/initramfs-qcom-image.bb
./recipes-kernel/images/initramfs-rootfs-image.bb

Various pieces of the "jump to the rootfs" lightweight initramfs image,
used by Linaro and Qualcomm developers.

./recipes-support/fastrpc/fastrpc_git.bb
./recipes-support/pd-mapper/pd-mapper_git.bb
./recipes-support/qbootctl/qbootctl_0.1.2.bb
./recipes-support/qrtr/qrtr_git.bb
./recipes-support/rmtfs/rmtfs_git.bb
./recipes-support/tqftpserv/tqftpserv_git.bb

Target Qualcomm-specific tools

./recipes-devtools/debugcc/debugcc_git.bb
./recipes-test/mybw/mybw_git.bb
./recipes-test/bootrr/bootrr_git.bb
./recipes-test/diag/diag_git.bb

Target Qualcomm-specific debugging tools

./recipes-devtools/pil-squasher/pil-squasher_git.bb
./recipes-devtools/qc-image-unpacker/qc-image-unpacker_git.bb
./recipes-devtools/qca-swiss-army-knife/qca-swiss-army-knife_git.bb
./recipes-devtools/qdl/qdl_git.bb
./recipes-devtools/qmic/qmic_git.bb

Native Qualcomm-specific tools

./recipes-kernel/images/esp-qcom-image.bb
./recipes-kernel/images/linux-qcom-uki.bb
./recipes-vendor-in/python/python3-pefile_2024.8.26.bb

UKI support, to be replaced / ported to uki.bbclass

./recipes-devtools/skales/skales_git.bb

Lightweight mkbootimg implementation

./recipes-test/images/initramfs-kerneltest-full-image.bb
./recipes-test/images/initramfs-kerneltest-image.bb
./recipes-test/images/initramfs-test-full-image.bb
./recipes-test/images/initramfs-test-image.bb
./recipes-test/images/initramfs-tiny-image.bb

Several initramfs images useful for testing and development

./recipes-kernel/linux/linux-linaro-qcomlt_6.6.bb

Old 6.6 QcomLT tree, to be removed once qcom-armv7a-modem is ported to
linux-yocto

./recipes-devtools/qtestsign/qtestsign_git.bb
./recipes-bsp/u-boot/u-boot-qcom_git.bb

U-Boot support

./recipes-support/rust-android-sparse/rust-android-sparse_0.6.0.bb

img2simg / simg2img implementation (for handling Google pixel firmware and
for implementing sparse image generation)

./dynamic-layers/openembedded-layer/recipes-devtools/android-tools/android-tools-adbd-cmdline.bb

Layer-specific ADBD customization, used for initramfs-test images

./dynamic-layers/meta-linux-mainline/recipes-kernel/linux/linux-stable_%.bbappend
./dynamic-layers/meta-linux-mainline/recipes-kernel/linux/linux-mainline.bbappend

mainline linux support, used for testing at some point, might be dropped now

./dynamic-layers/openembedded-layer/recipes-graphics/images/qcom-x11-image.bb
./recipes-graphics/mesa/mesa.bb
./recipes-test/lava-test-shell/lava-test-shell.bb

Obsolete, can be dropped ?

./recipes-graphics/xorg-xserver/xserver-xorg_%.bbappend
./recipes-graphics/xorg-xserver/xserver-xf86-config_0.1.bbappend

Check if these can be dropped. Might be handled in the upstream

./recipes-graphics/mesa/mesa.bbappend

Platform-specific customization to enable freedreno driver

./recipes-kernel/linux/linux-yocto_6.6.bbappend
./recipes-kernel/linux/linux-yocto_%.bbappend

Platform-specific linux-yocto implementation

./dynamic-layers/openembedded-layer/recipes-devtools/android-tools/android-tools-conf-configfs_%.bbappend

Platform-specific ADBD customization

./dynamic-layers/openembedded-layer/recipes-navigation/gpsd/gpsd_%.bbappend

Qualcomm-specific GPSd driver, works only on db410c

@ricardosalveti
Copy link
Contributor

  • What are the Binary blobs which are needed in meta-qcom, and how do we handle their redistribution.

The non-redistributable binaries would need to be removed (including renesas xhci), should we create a new repository under Linaro for them (e.g. meta-qcom-linaro)?

@lumag
Copy link
Collaborator

lumag commented Nov 22, 2024

I think meta-qcom-linaro makes sense. At the same time ideally Qualcomm should try utilizing their power to get a sane licence for that file because, if I remember correctly, it is also required to get that host to work on RB3gen2 (old RB3 had it flashed in the ROM, I don't know if RB3gen2 has the ROM or not).

@ndechesne
Copy link
Contributor Author

@lumag can you please confirm if the Renesas firmware you have (for the original RB3) is the same as here: https://www.renesas.com/en/products/interface/usb-switches-hubs/upd720201-usb-30-host-controller#documents? That's the one which is recommended to use for RB3 Gen2 (see https://docs.qualcomm.com/bundle/publicresource/topics/80-70015-254/how_to.html#update-usb-and-ethernet-controller-firmware). It is not redistributed by QCOM, but there are instructions for users to download it and flash it.

@ndechesne
Copy link
Contributor Author

At least based on RB3 and RB3 Gen2 schematics, it's indeed the same Renesas bridge being used on both (uPD720201K8-701-BAC-A)

@lumag
Copy link
Collaborator

lumag commented Nov 25, 2024

Yes, it's the same.

My point is that if we distribute rootfs images, they (ideally) should just work without downloading any binaries. Likewise if a user builds rootfs, it should be possible to provide the rom / zip file at the build time. I'm quite insisting here because if a company develops an end-user product using RB3gen2, they should be able to support user products (like providing rootfs, providing updates, etc), without users having to download anything for their device from the 3rd party vendor (Renesas).

@ricardosalveti
Copy link
Contributor

From our investigation there is no ROM in rb3gen2 and we are bound to Renesas terms. It is indeed a pain to manage.

@lumag
Copy link
Collaborator

lumag commented Nov 26, 2024

Regarding meta-qcom-linaro. I think instead I will spawn linux-msm/meta-qcom-extras and start moving questionable recipes to that repo. @davidinux @ndechesne @ricardosalveti WDYT?

@ndechesne
Copy link
Contributor Author

Regarding meta-qcom-linaro. I think instead I will spawn linux-msm/meta-qcom-extras and start moving questionable recipes to that repo. @davidinux @ndechesne @ricardosalveti WDYT?

This is a good idea. I was not sure meta-qcom-linaro was great.. I like your idea much better.

@ricardosalveti
Copy link
Contributor

Yeah, better under linux-msm.

@davidinux
Copy link
Contributor

davidinux commented Nov 27, 2024 via email

@lumag
Copy link
Collaborator

lumag commented Nov 27, 2024

I started the split by preparing to remove Nexus/Pixel and WoA recipes.
Are we okay with leaving package-the-binaries recipes for HDKs and db8074 or should I move them to the external repo too?

@lumag
Copy link
Collaborator

lumag commented Nov 27, 2024

See #681 , #682 .

@ndechesne
Copy link
Contributor Author

thanks. Both PRs above look pretty nice cleanups. thanks.
Let's keep the HDK and original dragonboard in meta-qcom for now. they are Qualcomm reference designs, so it makes sense. If that becomes a problem in the future we can stage 'legacy' hardware in -extras.

@davidinux
Copy link
Contributor

davidinux commented Nov 27, 2024 via email

@lumag
Copy link
Collaborator

lumag commented Nov 27, 2024

I still have hopes that one day we might have HDK binaries under the 'redistributable' licence.
DB8074 and IFC board -- probably no hopes on that side.

@ndechesne ndechesne mentioned this issue Nov 28, 2024
Open
@ndechesne
Copy link
Contributor Author

I've given this a first try, see quic-yocto/meta-qcom-hwe#92

My intent is to prepare a branch that is the result of the merge of the 2 unrelated git histories into a single branch. This branch will become meta-qcom 'master' eventually (when it's reviewed and functional).

I've tried to explain clearly the process I followed in the PR and the commits I made

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@lumag @ndechesne @davidinux @ricardosalveti