Move airflow sources to airflow-core package

[potiuk]

This is continuation of the separation of the Airflow codebase into
separate distributions. This one splits airflow into two of them:

• apache-airflow - becomes an empty, meta no-code distribution that
  only has dependencies to apache-airflow-core and task-sdk
  distributions and it has preinstalled provider distributions
  added in standard "wheel" distribution. All "extras" lead
  either to "apache-airflow-core" extras or to providers - the
  dependencies and optional dependencies are calculated differently
  depending on "editable" or "standard" mode - in editable mode,
  just provider dependencies are installed for preinstalled providers
  in standard mode - those preinstalled providers are dependencies.

• the apache-airflow-core distribution contains all airflow core
  sources (previously in apache-airflow) and it has no provider
  extras. Thanks to that apache-airflow distribution does not
  have any dynamically calculated dependencies.

• the apache-airflow-core distribution hs "hatch_build_airflow_core.py"
  build hooks that add custom build target and implement custom
  cleanup in order to implement compiling assets as part of the build.

• During the move, the following changes were applied for consistency:

  • packages when used in context of distribution packages have been
    renamed to "distributions" - including all documentations and
    commands in breeze to void confusion with import packages
    (see
    https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/)

  • all tests in airflow-core follow now the same convention
    where tests are in unit, system and integration package.
    no extra package has been as second level, because all the
    provider tests have "" there, so we just have to avoid
    naming airflow unit."" with the same name as provider.

  • all tooling in CI/DEV have been updated to follow the new
    structure. We should always build to packages now when we
    are building them using breeze.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[jscheffl]

Wow! Impressive metric! More files moved than code changed :-D

[potiuk]

Wow! Impressive metric! More files moved than code changed :-D

Indeed :)

[potiuk]

Wow! Impressive metric! More files moved than code changed :-D

This changed now :) .. During making the PR green I used the opportunity since I was changing anyway the package preparation steps - that I finally will fix the long standing "what is a package" problem.

In the current incarnation of the change I made a consistent change to name what we produce "distributions" not "packages" - because "package" is really ambiguous.

See https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/

So we will be:

• preparing distribution (in a give distribution format - wheel, or sdist)
• releasing distribution
• verifying distribution
• working on distribution

We will have provider distributions rather than provider packages. This is mostly changing breeze DISTRIBUTION in our code and documentation links etc.

I think this is the right time of doing it especially that we are going to have those DISTRIBUTIONS:

• apache-airflow
• apache-airlfow-core
• apache-airlfow-task-sdk
• apache-airflow-providers-* -> 90+ of those

And most of those will have Python airflow package inside.

[potiuk]

OK. That one has a big chance of being green

[gopidesupavan]

Wooho its green :)

[potiuk]

Last rebase I hope.

[potiuk]

Damn... have to rebase AGAIN :)

[potiuk]

YOLO! Merging after rebase without waiting.

[zlonghofer]

@potiuk the Apache Airflow website has a GitHub "Suggest a change on this page" widget that still links to the old location and returns a 404.

This is true for all of the pages I've tried and probably reflects a templating update to the main documentation website is required to resolve the broken links. Is it appropriate to open an issue on the Airflow repo to implement this change, or does a different entity manage the links on the main documentation site?

[amoghrajesh]

@zlonghofer thanks for reporting, i created an issue for this: #48178

This change will have to go in airflow-site repository, not here.