Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement OTA Firmware upgrade using CC3000 #3

Open
satishgn opened this issue May 17, 2013 · 15 comments
Open

Implement OTA Firmware upgrade using CC3000 #3

satishgn opened this issue May 17, 2013 · 15 comments
Assignees

Comments

@satishgn
Copy link
Contributor

No description provided.

@satishgn
Copy link
Contributor Author

Before implementing the OTA/WLAN based DFU, we have the following things in place:
1)Integration of working SPI Flash driver with DFU codebase.
2)Integration of working CC3000/WLAN driver with DFU codebase.
3)Application Backup functionality from Internal Flash to External Flash.
4)Application Restore functionality from External flash to Internal Flash.

Note:
Application "marvin" address in Internal Flash has been fixed at 0x08007000
and External Flash Backup/Restore address has been fixed at 0x00007000

Here is the next plan and TODO items:

When the marvin application is running, it will wait for command/message from the API server for new Firmware availability.

As soon as the "Upgrade_Firmware" message is received, the core will reset itself into DFU mode.

Now there are few ways how we can proceed with OTA firmware upgrade in DFU codebase:

1)Implement a basic TFTP server on core. Use a TFTP client on PC/Server side to initiate the firmware file transfer for writing on to the Internal Flash memory. This should be very easy to implement.
2)Implement a basic HTTP server on core. Upload Firmware file using HTML forms. To send raw binary data, the HTML POST method is used instead of the GET.
3)Implement TCP socket communication for firmware upgrade. Here we need a server side code which sends blocks of firmware upgrade code along with CRC checksum and core perform a CRC check before writing onto the Flash.

Zac, If you wish to take #3 approach, I need your help in getting the server side code implemented.
Let me know if you have other plan on OTA-DFU approach.

@zsup
Copy link

zsup commented May 23, 2013

Good suggestions for OTA firmware update. I think the biggest concern is that whatever solution we choose in the DFU codebase must be relatively foolproof, since it will be difficult to update or change once the Cores are in peoples' hands. Which of the three do you think would be least likely to requires changes or updates further down the road?

Zach Supalla
312.953.3413

On Thursday, May 23, 2013 at 11:42 AM, Satish Nair wrote:

Before implementing the OTA/WLAN based DFU, we have the following things in place:
1)Integration of working SPI Flash driver with DFU codebase.
2)Integration of working CC3000/WLAN driver with DFU codebase.
3)Application Backup functionality from Internal Flash to External Flash.
4)Application Restore functionality from External flash to Internal Flash.
Note:
Application "marvin" address in Internal Flash has been fixed at 0x08007000
and External Flash Backup/Restore address has been fixed at 0x00007000
Here is the next plan and TODO items:
When the marvin application is running, it will wait for command/message from the API server for new Firmware availability.
As soon as the "Upgrade_Firmware" message is received, the core will reset itself into DFU mode.
Now there are few ways how we can proceed with OTA firmware upgrade in DFU codebase:
1)Implement a basic TFTP server on core. Use a TFTP client on PC/Server side to initiate the firmware file transfer for writing on to the Internal Flash memory. This should be very easy to implement.
2)Implement a basic HTTP server on core. Upload Firmware file using HTML forms. To send raw binary data, the HTML POST method is used instead of the GET.
3)Implement TCP socket communication for firmware upgrade. Here we need a server side code which sends blocks of firmware upgrade code along with CRC checksum and core perform a CRC check before writing onto the Flash.
Zac, If you wish to take #3 (#3) approach, I need your help in getting the server side code implemented.
Let me know if you have other plan on OTA-DFU approach.


Reply to this email directly or view it on GitHub (#3 (comment)).

@towynlin
Copy link
Contributor

We should use approach 3, sending the data with appropriate checksums over the existing socket. Understood: I and/or someone else we hire will work with you to write the server-side code.

@satishgn Some things to keep in mind:

  1. We will soon switch to protocol buffers for the communication between the device and server.
  2. Will need to move to C++ soon.
  3. We need to move toward test-driven development. Can you request expense reimbursement through oDesk? If so (assuming you don't already have a copy), please buy a $25 eBook of Test Driven Development for Embedded C here: http://pragprog.com/book/jgade/test-driven-development-for-embedded-c and Spark will reimburse you for it. Read it as you have time, and start thinking about testing our existing code and writing tests firsts for new code.

@satishgn
Copy link
Contributor Author

Sure, will try to get my hands on that eBook.

On Fri, May 24, 2013 at 2:27 AM, Zachary Crockett
[email protected]:

We should use approach 3, sending the data with appropriate checksums over
the existing socket. Understood: I and/or someone else we hire will work
with you to write the server-side code.

@satishgn https://github.com/satishgn Some things to keep in mind:

  1. We will soon switch to protocol buffers for the communication
    between the device and server.
  2. Will need to move to C++ soon.
  3. We need to move toward test-driven development. Can you request
    expense reimbursement through oDesk? If so (assuming you don't already have
    a copy), please buy a $25 eBook of Test Driven Development for Embedded C
    here:
    http://pragprog.com/book/jgade/test-driven-development-for-embedded-cand Spark will reimburse you for it. Read it as you have time, and start
    thinking about testing our existing code and writing tests firsts for new
    code.


Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-18371167
.

@satishgn
Copy link
Contributor Author

I have a suggestion on the partition of our Internal Flash structure.
Currently we have:
1)DFU code (USB:complete + OTA:under development)at the start of Flash at address 0x08000000 (28KB reserved)
2)Marvin code at address 0x08007000 (100KB available)

My suggestion is since our USB-DFU is relatively foolproof and reliable, we can keep it separate from OTA-DFU.
We can have the following:
1)DFU (USB) code at 0x08000000 (19KB reserved) and locked(non-writable)
2)System/User Flags at 0x08004C00 (1KB reserved). This is the alternative for the Backup registers/External Flash (as per Andy's feedback)
3)OTA-DFU code at 0x08005000 (28KB reserved)
4)Marvin code at 0x0800A000 (80KB available)

While shipping out core, we can flash the USB-DFU code and use it to flash 3 and 4 using dfu-util tool.
To enter USB-DFU mode, the button needs to be pressed at reset.
To enter OTA-DFU mode, the Server API will send a request to marvin application to reset itself to OTA upgrade.

Let me know your viewpoints on the above.

@andyw-lala
Copy link

I am not seeing the big picture. To run ota-dfu is it required for the
stm32 to read external flash, program internal flash with the ota-dfu code,
then write the ota-dfu payload to external flash ? Or does it execute
ota-dfu from ram ?

Don't forget that you will want 2 copies of anything downloadable (perhaps
you consider this done with a running copy in stm32 flash and an external
copy.

We must avoid the situation where a failed download results in losing a
previously working firmware.
On 24 May 2013 00:16, "Satish Nair" [email protected] wrote:

I have a suggestion on the partition of our Internal Flash structure.
Currently we have:
1)DFU code (USB:complete + OTA:under development)at the start of Flash at
address 0x08000000 (28KB reserved)
2)Marvin code at address 0x08007000 (100KB available)

My suggestion is since our USB-DFU is relatively foolproof and reliable,
we can keep it separate from OTA-DFU.
We can have the following:
1)DFU (USB) code at 0x08000000 (19KB reserved) and locked(non-writable)
2)System/User Flags at 0x08004C00 (1KB reserved). This is the alternative
for the Backup registers/External Flash (as per Andy's feedback)
3)OTA-DFU code at 0x08005000 (28KB reserved)
4)Marvin code at 0x0800A000 (80KB available)

While shipping out core, we can flash the USB-DFU code and use it to flash
3 and 4 using dfu-util tool.
To enter USB-DFU mode, the button needs to be pressed at reset.
To enter OTA-DFU mode, the Server API will send a request to marvin
application to reset itself to OTA upgrade.

Let me know your viewpoints on the above.


Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-18386999
.

@satishgn
Copy link
Contributor Author

All the executable code(usb-dfu, ota-dfu & marvin) will be organized in Internal flash.
The External Flash will just contain the backup copies of working marvin code.

With a fresh core in hand,
spark will first program the usb-dfu application using a JTAG programmer. The dfu pages will be write-protected.
Then using dfu-util, ota-dfu is programmed followed by marvin via USB.

Flags in Backup registers will be used for switching between ota-dfu and marvin after System reset. This is a good use of Backup Registers.

When in ota-dfu mode, the first step is to create a backup copy of marvin(application code) in External flash.
Then the marvin code in Internal flash will be erased and programmed with the latest firmware.
In the event that the crc checksum of the latest firmware fails or other failed download conditions,
the backup marvin copy in External flash will be Restored/programmed back to the Internal flash.

Also since the usb-dfu is failproof and reliable, the core user can anytime program both the Internal and External Flash via USB.

@andyw-lala
Copy link

OK I think I understand. I would recommend that the production
test/programming fixture blows a combo image into stm32 flash and then also
loads a small programming into ram that is responsible for formatting
external flash and copying sections from stm32 flash out to external. The
combo image can contain the per unit personalisation data too.
On 24 May 2013 03:26, "Satish Nair" [email protected] wrote:

All the executable code(usb-dfu, ota-dfu & marvin) will be organized in
Internal flash.
The External Flash will just contain the backup copies of working marvin
code.

With a fresh core in hand,
spark will first program the usb-dfu application using a JTAG programmer.
The dfu pages will be write-protected.
Then using dfu-util, ota-dfu is programmed followed by marvin via USB.

Flags in Backup registers will be used for switching between ota-dfu and
marvin after System reset. This is a good use of Backup Registers.

When in ota-dfu mode, the first step is to create a backup copy of
marvin(application code) in External flash.
Then the marvin code in Internal flash will be erased and programmed with
the latest firmware.
In the event that the crc checksum of the latest firmware fails or other
failed download conditions,
the backup marvin copy in External flash will be Restored/programmed back
to the Internal flash.

Also since the usb-dfu is failproof and reliable, the core user can
anytime program both the Internal and External Flash via USB.


Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-18392469
.

@satishgn
Copy link
Contributor Author

Sounds good. During production, all the binaries can be combined together
into a single file and programmed at one go.

On Fri, May 24, 2013 at 3:01 PM, andyw-lala [email protected]:

OK I think I understand. I would recommend that the production
test/programming fixture blows a combo image into stm32 flash and then
also
loads a small programming into ram that is responsible for formatting
external flash and copying sections from stm32 flash out to external. The
combo image can contain the per unit personalisation data too.
On 24 May 2013 03:26, "Satish Nair" [email protected] wrote:

All the executable code(usb-dfu, ota-dfu & marvin) will be organized in
Internal flash.
The External Flash will just contain the backup copies of working marvin
code.

With a fresh core in hand,
spark will first program the usb-dfu application using a JTAG
programmer.
The dfu pages will be write-protected.
Then using dfu-util, ota-dfu is programmed followed by marvin via USB.

Flags in Backup registers will be used for switching between ota-dfu and
marvin after System reset. This is a good use of Backup Registers.

When in ota-dfu mode, the first step is to create a backup copy of
marvin(application code) in External flash.
Then the marvin code in Internal flash will be erased and programmed
with
the latest firmware.
In the event that the crc checksum of the latest firmware fails or other
failed download conditions,
the backup marvin copy in External flash will be Restored/programmed
back
to the Internal flash.

Also since the usb-dfu is failproof and reliable, the core user can
anytime program both the Internal and External Flash via USB.


Reply to this email directly or view it on GitHub<
https://github.com/sprk/dfu/issues/3#issuecomment-18392469>
.


Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-18395171
.

@andyw-lala
Copy link

But since JTAG cannot write external flash, plan for a program to init
that, based on the internal flash contents. Best way to do that without
further flash hacking is to load a ram-based binary via JTAG.
Disclaimer: I've done this for msp430 but not cortex yet, but I understand
it should be possible.
On 24 May 2013 04:38, "Satish Nair" [email protected] wrote:

Sounds good. During production, all the binaries can be combined together
into a single file and programmed at one go.

On Fri, May 24, 2013 at 3:01 PM, andyw-lala [email protected]:

OK I think I understand. I would recommend that the production
test/programming fixture blows a combo image into stm32 flash and then
also
loads a small programming into ram that is responsible for formatting
external flash and copying sections from stm32 flash out to external.
The
combo image can contain the per unit personalisation data too.
On 24 May 2013 03:26, "Satish Nair" [email protected] wrote:

All the executable code(usb-dfu, ota-dfu & marvin) will be organized
in
Internal flash.
The External Flash will just contain the backup copies of working
marvin
code.

With a fresh core in hand,
spark will first program the usb-dfu application using a JTAG
programmer.
The dfu pages will be write-protected.
Then using dfu-util, ota-dfu is programmed followed by marvin via USB.

Flags in Backup registers will be used for switching between ota-dfu
and
marvin after System reset. This is a good use of Backup Registers.

When in ota-dfu mode, the first step is to create a backup copy of
marvin(application code) in External flash.
Then the marvin code in Internal flash will be erased and programmed
with
the latest firmware.
In the event that the crc checksum of the latest firmware fails or
other
failed download conditions,
the backup marvin copy in External flash will be Restored/programmed
back
to the Internal flash.

Also since the usb-dfu is failproof and reliable, the core user can
anytime program both the Internal and External Flash via USB.


Reply to this email directly or view it on GitHub<
https://github.com/sprk/dfu/issues/3#issuecomment-18392469>
.


Reply to this email directly or view it on GitHub<
https://github.com/sprk/dfu/issues/3#issuecomment-18395171>
.


Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-18395464
.

@satishgn
Copy link
Contributor Author

I don't see a need to program External Flash during production.
The External Flash will be erased and programmed with the back copy of the
marvin code when the OTA-DFU code is executed. The External Flash can also
be programmed over USB by the dfu-util tool when the USB-DFU code is
executed.

On Fri, May 24, 2013 at 3:55 PM, andyw-lala [email protected]:

But since JTAG cannot write external flash, plan for a program to init
that, based on the internal flash contents. Best way to do that without
further flash hacking is to load a ram-based binary via JTAG.
Disclaimer: I've done this for msp430 but not cortex yet, but I understand
it should be possible.
On 24 May 2013 04:38, "Satish Nair" [email protected] wrote:

Sounds good. During production, all the binaries can be combined
together
into a single file and programmed at one go.

On Fri, May 24, 2013 at 3:01 PM, andyw-lala [email protected]:

OK I think I understand. I would recommend that the production
test/programming fixture blows a combo image into stm32 flash and then
also
loads a small programming into ram that is responsible for formatting
external flash and copying sections from stm32 flash out to external.
The
combo image can contain the per unit personalisation data too.
On 24 May 2013 03:26, "Satish Nair" [email protected] wrote:

All the executable code(usb-dfu, ota-dfu & marvin) will be organized
in
Internal flash.
The External Flash will just contain the backup copies of working
marvin
code.

With a fresh core in hand,
spark will first program the usb-dfu application using a JTAG
programmer.
The dfu pages will be write-protected.
Then using dfu-util, ota-dfu is programmed followed by marvin via
USB.

Flags in Backup registers will be used for switching between ota-dfu
and
marvin after System reset. This is a good use of Backup Registers.

When in ota-dfu mode, the first step is to create a backup copy of
marvin(application code) in External flash.
Then the marvin code in Internal flash will be erased and programmed
with
the latest firmware.
In the event that the crc checksum of the latest firmware fails or
other
failed download conditions,
the backup marvin copy in External flash will be Restored/programmed
back
to the Internal flash.

Also since the usb-dfu is failproof and reliable, the core user can
anytime program both the Internal and External Flash via USB.


Reply to this email directly or view it on GitHub<
https://github.com/sprk/dfu/issues/3#issuecomment-18392469>
.


Reply to this email directly or view it on GitHub<
https://github.com/sprk/dfu/issues/3#issuecomment-18395171>
.


Reply to this email directly or view it on GitHub<
https://github.com/sprk/dfu/issues/3#issuecomment-18395464>
.


Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-18397302
.

@andyw-lala
Copy link

As long as all the possible failure modes have been carefully thought
through perhaps you do not. OTA requires very careful and methodical
planning.
On 24 May 2013 05:33, "Satish Nair" [email protected] wrote:

I don't see a need to program External Flash during production.
The External Flash will be erased and programmed with the back copy of the
marvin code when the OTA-DFU code is executed. The External Flash can also
be programmed over USB by the dfu-util tool when the USB-DFU code is
executed.

On Fri, May 24, 2013 at 3:55 PM, andyw-lala [email protected]:

But since JTAG cannot write external flash, plan for a program to init
that, based on the internal flash contents. Best way to do that without
further flash hacking is to load a ram-based binary via JTAG.
Disclaimer: I've done this for msp430 but not cortex yet, but I
understand
it should be possible.
On 24 May 2013 04:38, "Satish Nair" [email protected] wrote:

Sounds good. During production, all the binaries can be combined
together
into a single file and programmed at one go.

On Fri, May 24, 2013 at 3:01 PM, andyw-lala [email protected]:

OK I think I understand. I would recommend that the production
test/programming fixture blows a combo image into stm32 flash and
then
also
loads a small programming into ram that is responsible for
formatting
external flash and copying sections from stm32 flash out to
external.
The
combo image can contain the per unit personalisation data too.
On 24 May 2013 03:26, "Satish Nair" [email protected]
wrote:

All the executable code(usb-dfu, ota-dfu & marvin) will be
organized
in
Internal flash.
The External Flash will just contain the backup copies of working
marvin
code.

With a fresh core in hand,
spark will first program the usb-dfu application using a JTAG
programmer.
The dfu pages will be write-protected.
Then using dfu-util, ota-dfu is programmed followed by marvin via
USB.

Flags in Backup registers will be used for switching between
ota-dfu
and
marvin after System reset. This is a good use of Backup Registers.

When in ota-dfu mode, the first step is to create a backup copy of
marvin(application code) in External flash.
Then the marvin code in Internal flash will be erased and
programmed
with
the latest firmware.
In the event that the crc checksum of the latest firmware fails or
other
failed download conditions,
the backup marvin copy in External flash will be
Restored/programmed
back
to the Internal flash.

Also since the usb-dfu is failproof and reliable, the core user
can
anytime program both the Internal and External Flash via USB.


Reply to this email directly or view it on GitHub<
https://github.com/sprk/dfu/issues/3#issuecomment-18392469>
.


Reply to this email directly or view it on GitHub<
https://github.com/sprk/dfu/issues/3#issuecomment-18395171>
.


Reply to this email directly or view it on GitHub<
https://github.com/sprk/dfu/issues/3#issuecomment-18395464>
.


Reply to this email directly or view it on GitHub<
https://github.com/sprk/dfu/issues/3#issuecomment-18397302>
.


Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-18397605
.

@zsup
Copy link

zsup commented May 24, 2013

I agree wholeheartedly with the initial suggestion to keep USB-DFU and OTA-DFU separate, since USB-DFU is robust and that means that, no matter what happens, anything can be fixed with an over-USB firmware upgrade. In a consumer product this would be a terrible failsafe, but with the hobbyist market that we're pursuing first, this is fine.

In the long run, I think our goal should be to ensure that OTA-DFU is as robust as USB-DFU by the time we're positioning the Core for inclusion in consumer products. So perhaps just as important as making it failsafe now is designing a system where, when there is a failure, we are aware of it so that we can squash bugs as we go.

Part of the design of OTA-DFU, therefore, should be that as often as possible, when there is a checksum failure (or any other kind of failure), we send a signal back to our servers with an error code.

Next steps, then, are as follows:

  1. Satish, I think you should go ahead and implement the system as you have spec'ed it out below, adding in a mechanism to report back errors.
  2. In parallel, to Andy's point, we should think through all possible failure modes, document them, give them error codes, and start tracking them.
  3. If we encounter failure modes from which the system cannot recover (either through working through them on our own or by encountering them with our beta testers or eventually with the larger group of backers), we should make changes and improvements as we go to OTA-DFU.

Thoughts?

Zach Supalla
312.953.3413

On Friday, May 24, 2013 at 5:45 AM, andyw-lala wrote:

As long as all the possible failure modes have been carefully thought
through perhaps you do not. OTA requires very careful and methodical
planning.
On 24 May 2013 05:33, "Satish Nair" <[email protected] (mailto:[email protected])> wrote:

I don't see a need to program External Flash during production.
The External Flash will be erased and programmed with the back copy of the
marvin code when the OTA-DFU code is executed. The External Flash can also
be programmed over USB by the dfu-util tool when the USB-DFU code is
executed.

On Fri, May 24, 2013 at 3:55 PM, andyw-lala <[email protected] (mailto:[email protected])>wrote:

But since JTAG cannot write external flash, plan for a program to init
that, based on the internal flash contents. Best way to do that without
further flash hacking is to load a ram-based binary via JTAG.
Disclaimer: I've done this for msp430 but not cortex yet, but I
understand
it should be possible.
On 24 May 2013 04:38, "Satish Nair" <[email protected] (mailto:[email protected])> wrote:

Sounds good. During production, all the binaries can be combined
together
into a single file and programmed at one go.

On Fri, May 24, 2013 at 3:01 PM, andyw-lala <[email protected] (mailto:[email protected])>wrote:

OK I think I understand. I would recommend that the production
test/programming fixture blows a combo image into stm32 flash and
then
also
loads a small programming into ram that is responsible for
formatting
external flash and copying sections from stm32 flash out to
external.
The
combo image can contain the per unit personalisation data too.
On 24 May 2013 03:26, "Satish Nair" <[email protected] (mailto:[email protected])>
wrote:

All the executable code(usb-dfu, ota-dfu & marvin) will be
organized
in
Internal flash.
The External Flash will just contain the backup copies of working
marvin
code.

With a fresh core in hand,
spark will first program the usb-dfu application using a JTAG
programmer.
The dfu pages will be write-protected.
Then using dfu-util, ota-dfu is programmed followed by marvin via
USB.

Flags in Backup registers will be used for switching between
ota-dfu
and
marvin after System reset. This is a good use of Backup Registers.

When in ota-dfu mode, the first step is to create a backup copy of
marvin(application code) in External flash.
Then the marvin code in Internal flash will be erased and
programmed
with
the latest firmware.
In the event that the crc checksum of the latest firmware fails or
other
failed download conditions,
the backup marvin copy in External flash will be
Restored/programmed
back
to the Internal flash.

Also since the usb-dfu is failproof and reliable, the core user
can
anytime program both the Internal and External Flash via USB.


Reply to this email directly or view it on GitHub<
https://github.com/sprk/dfu/issues/3#issuecomment-18392469>
.


Reply to this email directly or view it on GitHub<
https://github.com/sprk/dfu/issues/3#issuecomment-18395171>
.


Reply to this email directly or view it on GitHub<
https://github.com/sprk/dfu/issues/3#issuecomment-18395464>
.


Reply to this email directly or view it on GitHub<
https://github.com/sprk/dfu/issues/3#issuecomment-18397302>
.


Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-18397605
.


Reply to this email directly or view it on GitHub (#3 (comment)).

@satishgn
Copy link
Contributor Author

Agree with you Zach.

Zac, can you please create a new project "ota-dfu" on Github and also rename existing "dfu" project to "usb-dfu" if possible ?

Thanks,
Satish.

@towynlin
Copy link
Contributor

Agreed. Done. You should all have access to the new ota-dfu repository. You'll have to update your existing dfu project to point to the new github URL with this command:

git remote set-url origin [email protected]:sprk/usb-dfu.git

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants