Test automation improvements - tracking issue

[danpetry]

Summary

This issue gives an indication of the automation improvements that can be made in the testing.

As of writing, merging of the automation scripts is largely blocked pending a community decision on test framework and interfaces to be used globally. This would be a good area to apply effort in order to progress the automation of the testing here. (Synchronizing an unmerged PR with merged test specifications isn't an easily manageable process.)

Note: Currently the "Automated?" and "ease of automation" statuses are very WIP. I'll be updating them as I get more familiar with the testing (by doing the testing). In the meantime, if anyone sees any corrections or updates they can make to them, please go ahead and make them.

Test 01-ci

Task no	Description	Automated?	Ease of automation
1	Compile all applications for all boards	Yes	N/A
2	Run all unit tests in native	Yes	N/A
3	Run all unit tests separately in native	Yes	N/A
4	Run all unit tests on an IoT-Lab M3 node	Yes	N/A

Test 02-tests

Task no	Description	Automated?	Ease of automation
1	Compile and run tests on native	Yes	N/A
2	Compile and run tests on IoT-Lab M3	Yes	N/A
3	Compile and run tests on SAMR21-xpro	Yes	N/A

Test 03-single-hop-ipv6-icmp

Task no	Description	Automated?	Ease of automation
1	ICMPv6 multicast echo on native	In PR 79	N/A
2	ICMPv6 link local echo on native - rapid	In PR 79	N/A
3	ICMPv6 link local echo on native - extended	In PR 79	N/A
4	ICMPv6 stress test on native	No	Easy
5	ICMPv6 neighbour cache stress test on native	No	Easy

Test 04-single-hop-6lowpan-icmp

Task no	Description	Automated?	Ease of automation
1	ICMPv6 link local echo on IoT-Lab M3	In PR 79	N/A
2	ICMPv6 multicast echo between SAMR21-xpro and IoT-Lab M3	In PR 79	N/A
3	ICMPv6 echo with large payload	In PR 79	N/A
4	ICMPv6 echo between SAMR21-xpro and IoT-Lab M3 - 15 minutes	In PR 79	N/A
5	ICMPv6 multicast echo between SAMR21-xpro and Zolertia Remote	No	Easy
6	ICMPv6 link-local echo between SAMR21-xpro and Zolertia Remote	No	Easy
7	ICMPv6 multicast echo between SAMR21-xpro and Arduino Zero + XBee	In PR 79	N/A
8	ICMPv6 echo between SAMR21-xpro and Arduino Zero + XBee	In PR 79	N/A
9	ICMPv6 stress test between several IoT-Lab M3 nodes	No	Easy

Test 05-single-hop-route

Task no	Description	Automated?	Ease of automation
1	ICMPv6 echo unicast addresess on native (default route)	In PR 79	N/A
2	ICMPv6 echo unicast addresess on iotlab-m3 (default route)	In PR 79	N/A
3	ICMPv6 echo unicast addresess on native (specific route)	In PR 79	N/A
4	ICMPv6 echo unicast addresess on iotlab-m3 (static route)	In PR 79	N/A

Test 06-single-hop-udp

Task no	Description	Automated?	Ease of automation
1	UDP on iotlab-m3	In PR 79	N/A
2	UDP on iotlab-m3 (UDP port compression)	In PR 79	N/A
3	UDP on native (non-existent neighbor)	No	Easy
4	UDP on iotlab-m3 (non-existent neighbor)	No	Easy

Test 07-multi-hop

Task no	Description	Automated?	Ease of automation
1	ICMPv6 echo on iotlab-m3 with three hops (static route)	No	Easy
2	UDP on iotlab-m3 with three hops (static route)	No	Easy
3	ICMPv6 echo on iotlab-m3 with three hops (RPL route)	No	Easy
4	UDP on iotlab-m3 with three hops (RPL route)	No	Easy

Test 08-interop

Task no	Description	Automated?	Ease of automation
1	ICMPv6 echo between native and Linux	No	Easy
2	ICMPv6 echo between iotlab-m3 and Linux with 6LowPAN	No	Hard
3	ICMPv6 echo between iotlab-m3 and Contiki	No	Hard
4	ICMPv6 echo between iotlab-m3 and Internet host through Linux with 6LowPAN	No	Hard
5	ICMPv6 echo between iotlab-m3 and Internet host through RIOT border router	No	Hard
6	UDP between iotlab-m3 and Internet host through RIOT border router	No	Hard
7	UDP between iotlab-m3 and Internet host through RIOT border router (200b payload)	No	Hard
8	UDP between GNRC and lwIP on iotlab-m3	No	Hard
9	UDP between GNRC and emb6 on iotlab-m3	No	Hard
10	UDP between lwIP and emb6 on iotlab-m3	No	Hard

Test 09-CoAP

Task no	Description	Automated?	Ease of automation
1	CORD Endpoint	No	Easy
2	Confirmable retries	No	Easy
3	Block1	No	Easy
4	Block2	No	Easy
5	Observe registration and notification	No	Easy

Test 10-icmpv6-error

Task no	Description	Automated?	Ease of automation
1	Destination unreachable - no route to destination	No	Easy
2	Destination unreachable - Beyond scope of source address	No	Easy
3	Destination unreachable - address unreachable (target node address)	No	Easy
4	Destination unreachable - address unreachable (neighbor cache miss)	No	Easy
5	Destination unreachable - port unreachable	No	Easy
6	Destination unreachable - port unreachable (large payload)	No	Easy
7	Packet too big	No	Easy
8	Time exceeded - hop limit exceeded in transit	No	Easy
9	Parameter problem - erroneous header field encountered	No	Easy
10	IPv6-in-IPv6 encapsulation	No	Easy

Test 11-lorawan

Task no	Description	Automated?	Ease of automation
1	LoRaWAN example	No	Hard
2	OTAA join procedure	No	Hard
3	ABP join procedure	No	Hard
4	LoRaWAN device parameters persistence	No	Hard

Test 99-compile-and-test-one-board

Task no	Description	Automated?	Ease of automation
1	Run test 02 on additional hardware	Yes	N/A