Add library to read/write firmware update HEADER files #74

jonathanhaigh-arm · 2019-01-28T10:31:24Z

When mbl-cloud-client wants to know what version of firmware is running
on a device it runs a script "arm_update_active_details.sh" that passes
mbl-cloud-client a "HEADER" file that contains information about the
last firmware update.

Usually, this HEADER file is passed to the device along with a firmware
update from the Pelion cloud, but there are two cases where that doesn't
happen:

When an update is sent to the device by the mbl-cli tool rather than
the Pelion cloud.
When the first version of software is initially written directly to
the deivce's flash.

We therefore need a mechanism to create these HEADER files independently
of the Pelion Cloud and mbl-cloud-client.

This commit adds:

The mbl-firmware-update-header Python library for creating and reading
HEADER files.
Tests for the mbl-firmware-update-header library.
An update to the mbl-firmware-update-manager library (which is used to
install updates that do not arrive via the Pelion cloud) to create
a HEADER file for incoming updates.

Synchronization
Must be merged before: PelionIoT/meta-mbl#367

Jenkins Link
http://jenkins.mbed-linux.arm.com/blue/organizations/jenkins/jh-test/detail/jh-test/34/pipeline (successful)
http://jenkins.mbed-linux.arm.com/blue/organizations/jenkins/jh-test/detail/jh-test/36/pipeline (successful; after addressing review comments).
http://jenkins.mbed-linux.arm.com/blue/organizations/jenkins/jh-test/detail/jh-test/37/pipeline (aborted; after addressing further review comments and fixing sanity check errors).
http://jenkins.mbed-linux.arm.com/blue/organizations/jenkins/jh-test/detail/jh-test/38/pipeline (successful; after more changes).
http://jenkins.mbed-linux.arm.com/blue/organizations/jenkins/jh-test/detail/jh-test/40/pipeline (in progress; after module rename).

Testing done

Unit tests for the new library have been added and pass.
I have manually checked that the versions (timestamps) in HEADER files created by mbl-firmware-update-manager are reasonable.
I have checked that the firmware versions in HEADER files generated by mbl-firmware-update-manager show up in the Pelion Device Management Web UI as the devices' "manifest timestamp" attributes and the "PkgVersion" resources (/10252/0/6).
I have manually checked that the firmware hashes in HEADER files created by mbl-firmware-update-manager match the firmware hashes in HEADER files created by installing the same payloads via the Pelion cloud.
I have manually checked that the firmware hashes in HEADER files generated by mbl-firmware-update manager show up correctly as the "firmware checksum" attributes of devices.

When mbl-cloud-client wants to know what version of firmware is running on a device it runs a script "arm_update_active_details.sh" that passes mbl-cloud-client a "HEADER" file that contains information about the last firmware update. Usually, this HEADER file is passed to the device along with a firmware update from the Pelion cloud, but there are two cases where that doesn't happen: * When an update is sent to the device by the mbl-cli tool rather than the Pelion cloud. * When the first version of software is initially written directly to the deivce's flash. We therefore need a mechanism to create these HEADER files independently of the Pelion Cloud and mbl-cloud-client. This commit adds: * The mbl-firmware-update-header Python library for creating and reading HEADER files. * Tests for the mbl-firmware-update-header library. * An update to the mbl-firmware-update-manager library (which is used to install updates that do not arrive via the Pelion cloud) to create a HEADER file for incoming updates.

...e-management/mbl-firmware-update-header/mbl/firmware_update_header/firmware_update_header.py

firmware-management/mbl-firmware-update-header/mbl/firmware_update_header/__init__.py

amirb1974 · 2019-01-28T15:18:30Z

...e-management/mbl-firmware-update-header/mbl/firmware_update_header/firmware_update_header.py

+
+        if len(data) < self.__header_and_crc_size:
+            raise FormatError(
+                "Data is too short (expecting at least {}B)".format(


Suggest to add len(data) in the message

Good point. Will do that.

Added in latest code.

amirb1974 · 2019-01-28T15:19:58Z

...e-management/mbl-firmware-update-header/mbl/firmware_update_header/firmware_update_header.py

+        if header_magic != self.__HEADER_MAGIC:
+            raise FormatError(
+                "Invalid header magic value 0x{:x} (should be 0x{:x})".format(
+                    header_magic, self.__HEADER_MAGIC


I assume this header magic confidential, right?

LOL meant "NOT confidential"

amirb1974 · 2019-01-28T15:20:57Z

...e-management/mbl-firmware-update-header/mbl/firmware_update_header/firmware_update_header.py

+
+        if header_version != self.HEADER_FORMAT_VERSION:
+            raise FormatError(
+                "Unsupported header version {}".format(header_version)


I suggest to add self.HEADER_FORMAT_VERSION to the message

Added in latest code.

amirb1974 · 2019-01-28T15:28:45Z

...anagement/mbl-firmware-update-manager/mbl/firmware_update_manager/firmware_update_manager.py

+        # Create a "HEADER" file for the update - this is a blob that contains
+        # information about the update
+        with open(HEADER_FILE, "wb") as header_file:
+            header_file.write(_create_header_data(firmware_update_file_path))


I think your new script throws exceptions. Are you handling them here in some way?

Yes, this is sloppy, should be fixed in the latest code.

jonathanhaigh-arm · 2019-01-28T18:05:52Z

Adding DO NOT MERGE label - I need to re-test after the changes for review comments.

...e-management/mbl-firmware-update-header/mbl/firmware_update_header/firmware_update_header.py

jonathanhaigh-arm · 2019-01-29T13:39:00Z

Regarding the remaining sanity check failures: they are spurious; see PelionIoT/mbl-tools#126

* Confusingly, the code was previously sometimes treating firmware hashes as 512-bit values and sometimes treating them as 256-bit values. It sort-of worked by chance, but let's be explicit about what's happenning: the firmware_hash field in HEADER blobs is 512 bits long, but we only use the first 256 bits of it to store sha256 hashes. * Add a missing "self" parameter in _create_header_data.

jonathanhaigh-arm · 2019-01-30T12:10:48Z

Even after PelionIoT/mbl-tools#126 was merged there are still sanity check failures... They're in unrelated code and are being dealt with by PelionIoT/mbl-tools#129.

I think I've addressed all review comments and I've re-tested, so, reviewers, please re-review.

jonathanhaigh-arm · 2019-01-30T14:16:13Z

Reviewers, sorry for the extra hassle but I want to make another change before this is merged.

In another PR I'm going to create a recipe in meta-mbl to use the Python module added in this PR to actually create a firmware update HEADER file... It seems sensible to call this new recipe "mbl-firmware-update-header", so I want to rename the Python module to be called "mbl.firmware_update_header.util".

jonathanhaigh-arm · 2019-01-30T15:04:21Z

I've (partially) retested after the module rename and added a new Jenkins link (build in progress).

...anagement/mbl-firmware-update-header-util/tests/host/test_mbl_firmware_update_header_util.py

...l-firmware-update-header-util/mbl/firmware_update_header_util/firmware_update_header_util.py

Co-Authored-By: jonathanhaigh-arm <[email protected]>

* Remove optional "0" in some slice syntax. * Remove unnecessary line break.

…ed/mbl-core into jh-update-header-IOTMBL-1099

rwalton-arm · 2019-02-05T13:34:55Z

Thanks for the changes.

jonathanhaigh-arm added the SYNCHRONIZE MERGES label Jan 28, 2019

jonathanhaigh-arm mentioned this pull request Jan 28, 2019

Get mbl-firmware-update-manager to generate valid HEADER files PelionIoT/meta-mbl#367

Merged

jonathanhaigh-arm requested review from amirb1974, gillevko, diego-sueiro and rwalton-arm January 28, 2019 11:08

rwalton-arm suggested changes Jan 28, 2019

View reviewed changes

amirb1974 reviewed Jan 28, 2019

View reviewed changes

TO BE SQUASHED: Address review comments

e8d5d3f

jonathanhaigh-arm added the DO NOT MERGE label Jan 28, 2019

rwalton-arm suggested changes Jan 29, 2019

View reviewed changes

...e-management/mbl-firmware-update-header/mbl/firmware_update_header/firmware_update_header.py Outdated Show resolved Hide resolved

jonathanhaigh-arm added 2 commits January 29, 2019 11:10

TO BE SQUASHED: Add some more error checking

9cca820

SQUASH: fix sanity-check errors; put private function at end of file.

1aa6baf

jonathanhaigh-arm added 2 commits January 29, 2019 14:54

SQUASH: Whitespace changes to keep the checkers happy

3e8a153

jonathanhaigh-arm removed the DO NOT MERGE label Jan 29, 2019

SQUASH: empty commit to force re-run of sanity checks

ad3e63f

SQUASH: empty commit to force re-run of sanity checks

0f231aa

jonathanhaigh-arm added the DO NOT MERGE label Jan 30, 2019

SQUASH: rename to mbl-firmware-update-header-util

c2f7589

jonathanhaigh-arm removed the DO NOT MERGE label Jan 30, 2019

rwalton-arm suggested changes Feb 1, 2019

View reviewed changes

rwalton-arm and others added 2 commits February 1, 2019 13:39

SQUASH: Address review comment - use default "bytes" constructor

ddd344d

Co-Authored-By: jonathanhaigh-arm <[email protected]>

SQUASH: Address review comments

6c94ca9

* Remove optional "0" in some slice syntax. * Remove unnecessary line break.

Merge branch 'jh-update-header-IOTMBL-1099' of ssh://github.com/ARMmb…

fd89bca

…ed/mbl-core into jh-update-header-IOTMBL-1099

rwalton-arm approved these changes Feb 5, 2019

View reviewed changes

Merge branch 'master' into jh-update-header-IOTMBL-1099

5c524db

jonathanhaigh-arm merged commit 7553159 into master Feb 6, 2019

jonathanhaigh-arm deleted the jh-update-header-IOTMBL-1099 branch February 6, 2019 11:05

jonathanhaigh-arm mentioned this pull request Feb 12, 2019

MBL version is not reported correctly in Pelion Cloud #46

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add library to read/write firmware update HEADER files #74

Add library to read/write firmware update HEADER files #74

jonathanhaigh-arm commented Jan 28, 2019 •

edited

Loading

amirb1974 Jan 28, 2019

jonathanhaigh-arm Jan 28, 2019

jonathanhaigh-arm Jan 29, 2019

amirb1974 Jan 28, 2019

jonathanhaigh-arm Jan 28, 2019

amirb1974 Jan 28, 2019

amirb1974 Jan 28, 2019

jonathanhaigh-arm Jan 28, 2019

jonathanhaigh-arm Jan 29, 2019

amirb1974 Jan 28, 2019

jonathanhaigh-arm Jan 29, 2019

jonathanhaigh-arm commented Jan 28, 2019

jonathanhaigh-arm commented Jan 29, 2019

jonathanhaigh-arm commented Jan 30, 2019

jonathanhaigh-arm commented Jan 30, 2019

jonathanhaigh-arm commented Jan 30, 2019

rwalton-arm commented Feb 5, 2019

Add library to read/write firmware update HEADER files #74

Add library to read/write firmware update HEADER files #74

Conversation

jonathanhaigh-arm commented Jan 28, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonathanhaigh-arm commented Jan 28, 2019

jonathanhaigh-arm commented Jan 29, 2019

jonathanhaigh-arm commented Jan 30, 2019

jonathanhaigh-arm commented Jan 30, 2019

jonathanhaigh-arm commented Jan 30, 2019

rwalton-arm commented Feb 5, 2019

jonathanhaigh-arm commented Jan 28, 2019 •

edited

Loading