Release 2018.07 - RC1

[cladmi]

This issue lists the status of all tests for the Release Candidate 1 of the 2018.07 release.

Specs tested:

• 01-ci @cladmi (Related PR: shell/sc_can: fix uninitialized warning [backport 2018.07] RIOT#9644 pkg/libfixmath: replace __FILE__ by RIOT_FILE_NOPATH RIOT#9643 tests/*: memory blacklisting for new Docker image (edit cladmi: and current image) RIOT#9507)
  • Task #01 Issues but PR provided. I will re-run the failed tests now that they are backported.
  • Task #02
  • Task #03
  • Task #04 Increase test timeout unittests: increase test timeout RIOT#9590
• 02-tests @cladmi (Related PR: pkg/pkg.mk: allow overwriting GITAMFLAGS from environment var RIOT#9637)
  • Task #01
  • Task #02
  • Task #03
• 03-single-hop-ipv6-icmp @PeterKietzmann
  • Task #01
  • Task #02
  • Task #03
• 04-single-hop-6lowpan-icmp @PeterKietzmann
  • Task #01
  • Task #02
  • Task #03
  • Task #04
  • (Task #05 ) @PeterKietzmann
  • (Task #06) @PeterKietzmann
  • (Task #07)
  • (Task #08)
• 05-single-hop-route (Related Issue: 05-single-hop-route needs special compile flags #63) @jia200x
  • Task #01 (0% packet loss)
  • Task #02
  • Task #03
  • Task #04
• 06-single-hop-udp (Related Issues: Task 6: Questionable test parameters #53) @jia200x
  • Task #01 (0.2% packet loss)
  • Task #02 (0.2% packet loss)
• 07-multi-hop (Related PR: 07-multi-hop: Fix issues found when doing Release 2018.04 #66 Issue: Task 7: arbitrary error margin? #54)
  • Task #01
  • Task #02
  • Task #03
  • Task #04
• 08-interop
  • Task #01 @PeterKietzmann
  • Task #02
  • Task #03
  • Task #04
  • Task #05
  • Task #06
  • Task #07
  • Task #08
  • Task #09 @PeterKietzmann
  • Task #10

[cladmi]

I needed to fix some timeouts to run task02, task04. But other than that they work. I will provide PRs here and in RIOT for these.

[cladmi]

PR to update the scripts in 01-CI: #67

[kYc0o]

Contiki test done and successful.

[cladmi]

For 01-Ci, It took me a long time to actually make it run on my computer entirely (see the PR #67 for updating the test scripts)

I arrived with issue that were already mentioned in the PR to update the docker container to bionic RIOT-OS/RIOT#9507 There are PRs for fixing the issues already open.

[miri64]

You are still on RC1 for testing? With all the fixes merged, shouldn't you proceed to RC2 by now?

[cladmi]

I was only running the automated tests for the moment and fixes are not merged for them. I will do a report for Task 2.

[cladmi]

For Task 02 I used the script I wrote for running compilation and tests for one board: #68 The results of the tests are stored in https://github.com/cladmi/RIOT_2018_07_tests_results

So I have a summary for more boards than the task requires.
Here is the summary:

Compilation failed

Compilation was done for

arduino-zero
b-l072z-lrwan1
iotlab-a8-m3
iotlab-m3
native
pba-d-01-kw2x
samr21-xpro
stm32f4discovery
wsn430-v1_3b
wsn430-v1_4

• ccn-lite-relay failed on computer with an old version of cmake examples: unable to build ccn-lite for cortex-m RIOT#9064 which is known
  • https://github.com/cladmi/RIOT_2018_07_tests_results/blob/master/cladmi-iotlab/iotlab-m3/examples/ccn-lite-relay/compilation.failed
  • @aabadie it is currently not possible to build ccn-lite-relay on IoT-LAB frontends
• tests/pkg_fatfs_vfs and tests/pkg_u8g2 for wsn430 platforms it is just not compiling for a long time and should be put as not enough memory tests/*: memory blacklisting for new Docker image (edit cladmi: and current image) RIOT#9507
  • https://github.com/cladmi/RIOT_2018_07_tests_results/blob/master/cladmi-iotlab/wsn430-v1_3b/tests/pkg_fatfs_vfs/compilation.failed
  • https://github.com/cladmi/RIOT_2018_07_tests_results/tree/master/cladmi-iotlab/wsn430-v1_3b/tests/pkg_u8g2
• tests/driver_at fails with arm-none-eabi-gcc 8.1.0 which means all other applications look ok with it (at least for stm32f4discovery There is a -Werror=cast-function-type warning which would need a fix someday
  • https://github.com/cladmi/RIOT_2018_07_tests_results/blob/master/jcarrano-linuxdesktop/stm32f4discovery/tests/driver_at/compilation.failed

Tests failed comes in the next one.

[cladmi]

Test failed

Tests where run on the same boards except wsn430-* as flashing was broken after two or three flash.

arduino-zero
b-l072z-lrwan1
iotlab-a8-m3
iotlab-m3
native
pba-d-01-kw2x
samr21-xpro
stm32f4discovery

• tests/cbor only succeeded on native, which is known to be broken on other.
• driver_ds1307 test fails on iotlab-m3, iotlab-a8-m3, stm32f4discovery
  • https://github.com/cladmi/RIOT_2018_07_tests_results/blob/master/cladmi-iotlab/iotlab-m3/tests/driver_ds1307/test.failed (same for iotlab-a8-m3)
  • https://github.com/cladmi/RIOT_2018_07_tests_results/blob/master/jcarrano-linuxdesktop/stm32f4discovery/tests/driver_ds1307/test.failed
• driver_groove_ledbar which fails for stm43f4discovery and b-l072z-lrwan1. It crashes where-as it does not on iotlab but just runs through. Maybe it should be more robust.
  • https://github.com/cladmi/RIOT_2018_07_tests_results/blob/master/jcarrano-linuxdesktop/stm32f4discovery/tests/driver_grove_ledbar/test.failed
  • https://github.com/cladmi/RIOT_2018_07_tests_results/blob/master/jia200x-linuxlaptop/b-l072z-lrwan1/tests/driver_grove_ledbar/test.failed
• driver_hd4478 has a hardfault on arduino-zero but it should require a special setup, so maybe just need to handle errors and say a "test skip" or something
  • https://github.com/cladmi/RIOT_2018_07_tests_results/blob/master/cladmi-iotlab/arduino-zero/tests/driver_hd44780/test.failed
• driver_my9221 failed assertion on all boards as nothing is connected
• gnrc_ipv6_ext fails on arduino-zero and stm43f4discovery
  • Hardfault https://github.com/cladmi/RIOT_2018_07_tests_results/blob/master/cladmi-iotlab/arduino-zero/tests/gnrc_ipv6_ext/test.failed
  • TImeout for pkt->users: 0 https://github.com/cladmi/RIOT_2018_07_tests_results/blob/master/jcarrano-linuxdesktop/stm32f4discovery/tests/gnrc_ipv6_ext/test.failed
• gnrc_netif fails everywhere, it is a test script issue that was known but not yet fixed tests: add more hardware in the loop tests RIOT#9007
• gnrc_sixlowpan fails on everything:
  • child.expect_exact("tag: 0x1") failed on iotlab-m3, samr21-xpro but the test also sometime worked when run on IoT-LAB and not on my computer https://github.com/cladmi/RIOT_2018_07_tests_results/blob/master/cladmi-linuxlaptop/iotlab-m3/tests/gnrc_sixlowpan/test.failed
  • hardfault on b-l072z-lrwan1 https://github.com/cladmi/RIOT_2018_07_tests_results/blob/master/jia200x-linuxlaptop/b-l072z-lrwan1/tests/gnrc_sixlowpan/test.failed
  • Assertion failed on stm32f4discovery and arduino-zero https://github.com/cladmi/RIOT_2018_07_tests_results/blob/master/jcarrano-linuxdesktop/stm32f4discovery/tests/gnrc_sixlowpan/test.failed https://github.com/cladmi/RIOT_2018_07_tests_results/blob/master/cladmi-iotlab/arduino-zero/tests/gnrc_sixlowpan/test.failed
  • Segmentation fault in native https://github.com/cladmi/RIOT_2018_07_tests_results/blob/master/cladmi-linuxlaptop/native/tests/gnrc_sixlowpan/test.failed
• libfixmath test is sometime failing, must be a timeout not big enough issue
  • b-l072z-lrwan1
  • iotlab-m3
  • pba-d-01-kw2x
  • samr21-xpro
• libfixmath_unittests also failing with a timeout on b-l072z-lrwan1 also expect timing issue here
• pkg_fatfs_vfs it fails on all hardware as it supposed to have a setup step using an sdcard
• ps_schedstatistics fails with hardfault in cascade for pba-d-01-kw2x
  • https://github.com/cladmi/RIOT_2018_07_tests_results/blob/master/dpetry-ubuntu16_04_laptop/pba-d-01-kw2x/tests/ps_schedstatistics/test.failed
• pthread_condition_variable timeouts for b-l072z-lrwan1 maybe just a timeout issue
  • https://github.com/cladmi/RIOT_2018_07_tests_results/blob/master/jia200x-linuxlaptop/b-l072z-lrwan1/tests/pthread_condition_variable/test.failed
• trickle test failed with failures found in the test firmware
  • iotlab-a8-m3: https://github.com/cladmi/RIOT_2018_07_tests_results/blob/master/cladmi-iotlab/iotlab-a8-m3/tests/trickle/test.failed
  • stm32f4discovery: https://github.com/cladmi/RIOT_2018_07_tests_results/blob/master/jcarrano-linuxdesktop/stm32f4discovery/tests/trickle/test.failed
• warn_conflict failed for smt43f4discovery looks related to GCC 8
  • https://github.com/cladmi/RIOT_2018_07_tests_results/blob/master/jcarrano-linuxdesktop/stm32f4discovery/tests/warn_conflict/test.failed
• xtimer_periodic_wakeup failed for pba-d-01-kw2x.
  • https://github.com/cladmi/RIOT_2018_07_tests_results/blob/master/dpetry-ubuntu16_04_laptop/pba-d-01-kw2x/tests/xtimer_periodic_wakeup/test.failed

[miri64]

child.expect_exact("tag: 0x1") failed on iotlab-m3, samr21-xpro but the test also sometime worked when run on IoT-LAB and not on my computer https://github.com/cladmi/RIOT_2018_07_tests_results/blob/master/cladmi-linuxlaptop/iotlab-m3/tests/gnrc_sixlowpan/test.failed

due to RIOT-OS/RIOT#9585 this also happens in master for native.

[miri64]

gnrc_netif fails everywhere, it is a test script issue that was known but not yet fixed RIOT-OS/RIOT#9007

Comments made in RIOT-OS/RIOT#9007 still uphold

[miri64]

gnrc_ipv6_ext fails on arduino-zero and stm43f4discovery

Might also be related that to the fact, that the default interface instead of a dummy interface is used. Both these boards don't have a netdev on board.

[jia200x]

06-single-hop-udp OK. In average, less than 0.2% packet loss.

[PeterKietzmann]

04-single-hop-6lowpan-icmp Task #06 passed with 7% loss. I assume this relates to #7304

[cladmi]

@cgundogan may you can take a look at trickle errors ? #65 (comment)

[miri64]

Mhh... Rerunning gnrc_netif actually reveals a segfault that wasn't there before. I will investigate.

[cladmi]

@miri64 I added a PR to enable it in CI to try having it for the release RIOT-OS/RIOT#9669

[miri64]

Provided several fixes (I linked them in RIOT-OS/RIOT#9669).

[cladmi]

Closing in favor of #69