feat(build_wheels): 1st step of rewritting the wheel builds - assembl…

…ed the list of packages

.github/workflows/build-wheels-windows-dispatch_new.yml

@@ -0,0 +1,33 @@
+name: windows-dispatch
+
+on:
+  pull_request:
+  push:
+  workflow_dispatch:
+
+jobs:
+  build-python-wheels:
+    name: Build Python Wheels for windows-latest
+    runs-on: windows-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ['3.11.4']
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v2
+
+      - name: Set up Python
+        uses: actions/setup-python@v3
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Get Python version
+        run: python --version
+
+      - name: Install build dependencis
+        run: pip install -r .\build_requirements.txt
+
+      - name: Build wheels for IDF
+        run: python .\build_wheels.py

README.md

@@ -1,6 +1,6 @@
 # ESP idf-python-wheels
 
-The goal of this project is to automate build and upload process of required Python Wheels by IDF tools using GitHub Actions. We are able to build wheels for multiple OSes and architectures with multiple versions of Python.  
+The goal of this project is to automate build and upload process of required Python Wheels by IDF tools using GitHub Actions. We are able to build wheels for multiple OSes and architectures with multiple versions of Python.
 
 Supported architectures:
 * ubuntu-latest - x64
@@ -10,9 +10,9 @@ Supported architectures:
 * linux-armv7-self-hosted - arm32
 * linux-aarch64-self-hosted - arm64
 
-Each architecture has it's own workflow in .github/workflows except aarch64 Linux - not ready as GitHub runner yet. 
+Each architecture has it's own workflow in .github/workflows.
 
-For each architecture the user can select Python version to built wheels. On self-hosted runners can handle multiple versions of Python with pyenv. 
+For each architecture the user can select Python version to built wheels. On self-hosted runners can handle multiple versions of Python with pyenv.
 
 The build contains all wheels required by branches:
 * release/v4.3
@@ -21,7 +21,7 @@ The build contains all wheels required by branches:
 
 ## Configuration
 - user can set `IDF_branch` input parameter to add wheels for another branch.
-- currently, it is possible to run a workflow for the whole requirenments.txt
+- currently, it is possible to run a workflow for the whole requirements.txt
 - to add new architecture, set up GitHub runner, and create new GitHub Action
 - workflows need to be started manually
 
@@ -39,3 +39,15 @@ The build contains all wheels required by branches:
 .\Build-Wheels.ps1 -Branch "master" -Arch "-arm64" -CompileWheels @("greenlet", "gevent<2.0,>=1.2.2", "cryptography", "windows-curses", "python-pkcs11") -Python python3.9
 .\Test-Wheels.ps1 -Branch "master"
 ```
+
+## Requirements lists
+These lists are .txt files for requirements which should be added or excluded from the main requirements list which is automatically assembled.
+
+### include_list.txt
+File for additional Python packages to the main requirements list.
+
+### exclude_list.txt
+File for excluded Python packages in the main requirements list.
+
+### build_requirements.txt
+File for the requirements needed for build process and the build script.

build_requirements.txt

@@ -0,0 +1,4 @@
+# This is a list of Python packages needed for build process and script. This file is used with pip.
+# ----- build script -----
+requests
+packaging

build_wheels.py

@@ -0,0 +1,113 @@
+import json
+
+import requests
+from packaging.requirements import InvalidRequirement
+from packaging.requirements import Requirement
+
+idf_constraints_url = 'https://dl.espressif.com/dl/esp-idf/espidf.constraints.'
+
+# --- Global variables for branches and constraints files to be downloaded requirements for --- #
+# IDF branches are automatically fetched from github, every release branch higher or equal than 5.0 and master is taken
+# For constraints file the version is resolved from IDF release branch
+# The constraints file for master branch is considered to be the latest release version
+# incremented by 1 in minor field (e.g when last release is v5.2 the constrains file for master is v5.3)
+# if there is no such a version, the last release branch constraints file will be used
+# This can be overwritten by setting the variable 'AUTO_VERSION_CONSTRAINS_FILE' to False
+# and append into the list (in if statement bellow) desired version for master branch
+
+AUTO_VERSION_CONSTRAINS_FILE = True
+
+# ESP-IDF branches list
+IDF_BRANCHES_URL = 'https://api.github.com/repos/espressif/esp-idf/branches'
+r = requests.get(IDF_BRANCHES_URL)
+if r.status_code == 200:
+    idf_repo_branches = [branch['name'] for branch in r.json()]
+else:
+    print('Failed to fetch ESP-IDF branches.', '\n', r.text)
+    exit()
+
+idf_branches = []
+for branch in idf_repo_branches:
+    if branch.startswith('release'):
+        if float(branch.split('v')[1]) >= 5.0:
+            idf_branches.append(branch)
+idf_branches.append('master')
+
+# Constraints files versions list
+idf_constraints = []
+for branch in idf_branches:
+    try:
+        splitted_branch = branch.split('release/')
+        idf_constraints.append(splitted_branch[1])
+    except IndexError:
+        if AUTO_VERSION_CONSTRAINS_FILE:
+            idf_last_release = idf_constraints[-1].split('v')[1].split('.')
+            idf_master_minor = str(int(idf_last_release[1])+1)
+            if requests.get(f'{idf_constraints_url}v{idf_last_release[0]}.{idf_master_minor}.txt').status_code == 200:
+                idf_constraints.append(f'v{idf_last_release[0]}.{idf_master_minor}')
+            else:
+                idf_constraints.append(idf_constraints[-1])
+        else:
+            idf_constraints.append('6.0')
+
+print(f'ESP-IDF branches to be downloaded requirements for:\n{idf_branches}\n')
+print(f'ESP-IDF constrains files versions to be downloaded requirements for:\n{idf_constraints}\n')
+
+idf_resources_url = 'https://raw.githubusercontent.com/espressif/esp-idf/'
+requirements_txt = []
+requirements_set = set()
+
+# --- Download all requirements from all the branches requirements and constraints files --- #
+for i, branch in enumerate(idf_branches):
+    idf_requirements_json_url = f'{idf_resources_url}{branch}/tools/requirements.json'
+    constraint_file_url = f'https://dl.espressif.com/dl/esp-idf/espidf.constraints.{idf_constraints[i]}.txt'
+
+    r = requests.get(idf_requirements_json_url)
+    if r.status_code == 200:
+        idf_requirements_json = json.loads(r.content)
+    else:
+        print(f'\nFailed to download requirements JSON for branch {branch}')
+        continue
+
+    # Download requirements files from all requirements groups specified in JSON
+    print(f'---------- ESP-IDF BRANCH {branch} ----------')
+    for feature in idf_requirements_json['features']:
+        r = requests.get(f"{idf_resources_url}{branch}/{feature['requirement_path']}")
+        if r.status_code == 200:
+            requirements_txt += r.text.splitlines()
+            print(f"Added ESP-IDF {feature['name']} requirements")
+        else:
+            print(f"Failed to download feature (requirement group) '{feature['name']}'")
+
+    # Download constraints file for specific branch
+    r = requests.get(constraint_file_url)
+    if r.status_code == 200:
+        requirements_txt += r.text.splitlines()
+        print(f'Added ESP-IDF constraints file {idf_constraints[i]} for branch {branch}')
+    else:
+        print(f'Failed to download ESP-IDF constraints file {idf_constraints[i]} for branch {branch}')
+
+    # Add requirements into set to prevent duplicates
+    for line in map(str.strip, requirements_txt):
+        if line != '\n' or line != '\r\n' or line != '\r':
+            # check if in the line or the line itself is not a comment
+            if '#' in line:
+                splitted_line = line.split('#')
+                if splitted_line[0] != '':
+                    line = splitted_line[0]
+                else:
+                    continue
+            try:
+                requirements_set.add(Requirement(line))
+            except InvalidRequirement:
+                # TODO not a classic requirement
+                requirements_set.add(line)
+
+
+##############################################################
+# TODO setuptools requirement should be first
+
+# Print requirements
+print('\n---------- REQUIREMENTS ----------')
+for req in requirements_set:
+    print(req)

include_list.txt

@@ -0,0 +1,15 @@
+# This is a list of Python additional packages needed. This file is used with pip.
+
+# ----- IDF v4.3 and v4.4 -----
+# cffi 1.15 supports Python 3.6+ only
+cffi<1.15;python_version<'3.6'
+
+# esptool requirements (see components/esptool_py/esptool/setup.py)
+reedsolo>=1.5.3,<=1.5.4
+# bitstring 4 dropped support for Python 3.6 and older
+bitstring>=3.1.6,<4
+ecdsa>=0.16.0
+
+# espcoredump requirements
+# This is the last version supports both 2.7 and 3.4
+construct==2.10.54