fd90f5e0ce
Adding vendor porperty to reports, handling that in upload script
2015-11-24 17:39:39 -06:00
d6ff1189e0
Fixes issue where library is compiled multiple times during testing
2015-11-16 13:24:46 -06:00
33558d3a03
Merge pull request #1425 from bridadan/lib-dep-build-order-fix
...
Fix for issue with libraries being built out of order
2015-11-16 08:40:17 +00:00
0657050098
Fixes issue with libraries being built out of order
2015-11-13 11:48:44 -06:00
750b0ed41a
Using description for build reports
2015-11-12 11:01:08 -06:00
7f28e18afe
Allowing builds to report under different package name
2015-11-12 11:01:07 -06:00
736cae108e
Consolidating reporting in build_release and test_api
2015-11-12 11:01:03 -06:00
6ee94dee1e
Merged build_release report format into test junit format
2015-11-12 11:01:02 -06:00
3951ea2273
Handling failures correctly and adding timeout status
2015-11-05 11:22:46 -06:00
ecf04be150
Merge pull request #1352 from bridadan/fix-consolidate-waterfall-result
...
Fixes reporting of failure when one test passes when using --consolidate-waterfall
2015-09-25 15:35:03 +02:00
6eebc9926a
Fixes reporting of failure when one test passes when using --consolidate-waterfall
2015-09-24 12:57:28 -05:00
1d946c2160
Adding checks with mbedls throughout testing process
2015-09-23 11:21:11 -05:00
cefba29822
Added verbose mode to mbed library build when using singletest.py -v (verbose) switch
2015-08-12 09:43:43 +01:00
b5191580fd
Removed mbed-ls dependency from test framework
2015-07-22 16:40:42 -05:00
af7630d525
Merge pull request #1139 from PrzemekWirkus/devel_interoperability
...
Tools: Interoperability test suite: Added simple interface chip tests / checks
I'm merging this pull request, we can add more interoperability tests in the future. For now we can use this is CI.
I'm expecting some issues with this but this is not essential functionality and we can always improve it in the future.
2015-07-06 10:29:25 +01:00
abd4cbcf00
Changing return status of singletest.py if a build failure occurs
2015-06-19 11:53:38 -05:00
0d2effd024
Add option to consolidate waterfall loops into one test case in test report. Useful for CI.
2015-06-08 14:51:13 -05:00
f7fad72e44
Fixed runtime error when test fails due to build failure and outputting test report file
2015-06-08 12:45:30 -05:00
70a21e9656
Refactored interoperability tests, added separate classes of tests and CLI
2015-05-27 15:40:22 +01:00
cc6be1a16e
Added support for multiple targets of the same type (hardware platfrom)
2015-05-03 02:32:04 +01:00
9a720be3d4
Small improvements to reporting build failures
2015-05-02 23:08:00 +01:00
373e8c40c0
Fixed test build failure, now BUILD_FAILURE is one of possible results.
...
Added build report after build completion.
Fixed result from execute() capture when result is NoneType value
2015-04-30 19:39:30 +01:00
2f4d668382
Renamed Jenkins specific options to general "Build" options
2015-04-17 09:39:26 -05:00
5eb4894e71
Added skipped mbed library build reporting and removed unnecessary print
2015-04-14 13:39:29 -05:00
a574e1f27e
Broke apart some template stuff, added tests-build report
2015-04-13 17:07:37 -05:00
cb4e13c1e3
Reorganized test_api.py to build all required libraries (RTOS, Ethernet, etc) for tests up front, before any tests are built
2015-04-13 17:07:21 -05:00
cf7f03254a
Added to option --auto handler for -f <mcu> filter switch. Now using -f switch
...
will filter target platforms to test. Use comma to pass more MCU names to
filter.
```
$ singletest.py --auto -j 8 -O --config
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected NUCLEO_L053R8, port: COM35, mounted: E:
MBEDLS: Detected KL25Z, port: COM89, mounted: F:
MUTs configuration in auto-detected:
+-------+-------------+---------------+------+-------+
| index | peripherals | mcu | disk | port |
+-------+-------------+---------------+------+-------+
| 1 | | NUCLEO_L053R8 | E: | COM35 |
| 2 | | KL25Z | F: | COM89 |
+-------+-------------+---------------+------+-------+
Test specification in auto-detected:
+---------------+-----+------+
| mcu | ARM | uARM |
+---------------+-----+------+
| KL25Z | Yes | - |
| NUCLEO_L053R8 | - | Yes |
+---------------+-----+------+
```
Building original configuration (no filter):
```
$ singletest.py --auto -j 8 -O
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected NUCLEO_L053R8, port: COM35, mounted: E:
MBEDLS: Detected KL25Z, port: COM89, mounted: F:
Building library CMSIS (KL25Z, ARM)
Building library MBED (KL25Z, ARM)
Building project DETECT (KL25Z, ARM)
.
.
.
Building library CMSIS (NUCLEO_L053R8, uARM)
Building library MBED (NUCLEO_L053R8, uARM)
Building project DETECT (NUCLEO_L053R8, uARM)
.
.
.
Completed in 3.68 sec
```
```
$ singletest.py --auto -j 8 -O -f KL25Z --config
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected NUCLEO_L053R8, port: COM35, mounted: E:
MBEDLS: Detected KL25Z, port: COM89, mounted: F:
MUTs configuration in auto-detected:
+-------+-------------+-------+------+-------+
| index | peripherals | mcu | disk | port |
+-------+-------------+-------+------+-------+
| 2 | | KL25Z | F: | COM89 |
+-------+-------------+-------+------+-------+
Test specification in auto-detected:
+-------+-----+
| mcu | ARM |
+-------+-----+
| KL25Z | Yes |
+-------+-----+
```
Building original configuration (with applied filter):
```
$ singletest.py --auto -j 8 -O -f KL25Z
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected NUCLEO_L053R8, port: COM35, mounted: E:
MBEDLS: Detected KL25Z, port: COM89, mounted: F:
Building library CMSIS (KL25Z, ARM)
Building library MBED (KL25Z, ARM)
Building project DETECT (KL25Z, ARM)
.
.
.
Completed in 1.33 sec
```
2015-03-13 15:30:44 +00:00
4c45ed1cb1
Parallel functionality, marge conflicts resolved (v2)
...
Description
===========
Added thread model for function SingleTest::execute where for each target we
have separate thread.
Added new experimental feature. Currently we are only building in parallel
mbed SDK and libraries with dependencies.
Each MUT is flashed separately, but with this new feature we are able to
flash multiple boards at the same time and get results in parallel.
Test execution time is reduced by n where n is number of MUTs (MUT: mbed under
test).
using --parallel means that:
* separate thread will be spawned to build mbed SDK and dependencies (libraries)
for each unique target with all declared toolchains.
E.g:
Thread(n==1): LPC1768: ARM, uARM, GCC_ARM
Thread(n==2): K64F: ARM, GCC_ARM
Thread(n==3): NUCLEO_X: uARM
.
.
.
Thread(4): for toolchain in NUCLEO_Z[toolchains]
-> build mbed SDK
-> build libs
-> build project
-> run MUT testrunner
return test results for NUCLEO_Z[toolchain]
Done:
1. Added option --parallel.
2. Decoupled execute function so it can be run in parallel with other execute
functions.
3. Thread join via Queue, not Thread::Join() to avoid deadlocks or waits for
particular thread to finish.
4. All builds are in parallel, but because each target and library for each
toolchain have different directory we do not worry about building in
parallel and compiler collisions.
Missing:
1. No sync for 'print' (TODO).
2. No sync on test result structures - not needed because we only append to
them (?).
Experimental --parallel bugfix: self.test_suite_properties_ext dict was not
populated with target name.
2015-03-12 09:55:11 +00:00
50ef6207f0
Added new switch -p to force peripheral in autodetection mode (--auto)
2015-03-04 09:48:39 +00:00
dfa9599ded
Added mbed assertion detection
...
Test suite will detect mbed assertion print and end test execution with MBED_ASSERT error
2015-02-24 13:46:45 +00:00
0fe5f5f674
Increased initial timeout for each host test so we can be sure binary will be
...
flashed completely and we can proceed with target reset.
Before fix:
====
Test summary:
+---------+--------+-----------+---------+----------------------------+--------------------+---------------+-------+
| Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) | Loops |
+---------+--------+-----------+---------+----------------------------+--------------------+---------------+-------+
| OK | K64F | ARM | NET_1 | TCP client hello world | 3.37 | 20 | 1/1 |
| OK | K64F | ARM | NET_13 | TCP client echo loop | 2.06 | 20 | 1/1 |
| OK | K64F | ARM | NET_2 | NIST Internet Time Service | 2.41 | 20 | 1/1 |
| TIMEOUT | K64F | ARM | NET_3 | TCP echo server | 10.21 | 10 | 0/1 |
| TIMEOUT | K64F | ARM | NET_4 | TCP echo client | 10.21 | 10 | 0/1 |
| TIMEOUT | K64F | ARM | NET_5 | UDP echo server | 10.21 | 10 | 0/1 |
| TIMEOUT | K64F | ARM | NET_6 | UDP echo client | 10.21 | 10 | 0/1 |
| OK | K64F | ARM | NET_7 | HTTP client hello world | 6.46 | 15 | 1/1 |
| TIMEOUT | K64F | ARM | NET_8 | NTP client | 10.21 | 10 | 0/1 |
+---------+--------+-----------+---------+----------------------------+--------------------+---------------+-------+
Result: 4 OK / 5 TIMEOUT
Completed in 116.32 sec
After fix:
====
Test summary:
+--------+--------+-----------+---------+----------------------------+--------------------+---------------+-------+
| Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) | Loops |
+--------+--------+-----------+---------+----------------------------+--------------------+---------------+-------+
| OK | K64F | ARM | NET_1 | TCP client hello world | 3.32 | 20 | 1/1 |
| OK | K64F | ARM | NET_13 | TCP client echo loop | 2.06 | 20 | 1/1 |
| OK | K64F | ARM | NET_2 | NIST Internet Time Service | 2.4 | 20 | 1/1 |
| OK | K64F | ARM | NET_3 | TCP echo server | 1.54 | 20 | 1/1 |
| OK | K64F | ARM | NET_4 | TCP echo client | 1.54 | 20 | 1/1 |
| OK | K64F | ARM | NET_5 | UDP echo server | 1.47 | 20 | 1/1 |
| OK | K64F | ARM | NET_6 | UDP echo client | 1.6 | 20 | 1/1 |
| OK | K64F | ARM | NET_7 | HTTP client hello world | 6.46 | 15 | 1/1 |
| OK | K64F | ARM | NET_8 | NTP client | 2.4 | 15 | 1/1 |
+--------+--------+-----------+---------+----------------------------+--------------------+---------------+-------+
Result: 9 OK
Completed in 125.06 sec
2015-02-17 11:22:45 +00:00
f2f0f51d16
Added colorama to toolchain notification function and basic test suite prompts
2015-02-10 23:38:01 +00:00
8910d95ae6
Fixed bug in --tc switch
2015-02-10 21:29:11 +00:00
f38a53ba21
Added new switch --auto to force autodetection with mbed_lstools
...
Added switch --tc to force certain toolchains.
Switch --auto and -tc works well with switch --config so do not hesitate to
check your configuration before running your test suite.
2015-02-10 21:19:01 +00:00
35c034c911
Added timeout detection from aut-detection MUT printout
...
Timeout is captured by application supervising host_test and duration is modiffied
Added functionality preventing MUT printouts to reset device's timeout value or script execution timer counter
2015-02-10 09:15:46 +00:00
bcfbc3575d
Added -f (filter) to configuration switch --config
...
We can now use --config -f <mcu> to filter only targets we are interested in
2015-01-30 08:38:15 +00:00
b877fe32f9
Added test case time measurement instead of whole test time measurement for 'Elapsed Time (sec)' report in singletest.py console reports
2015-01-28 15:33:11 +00:00
f7f79866d1
Unlocked --db switch. Added check when parsing DB connection string
2014-11-21 13:57:23 +00:00
eadd2013c2
Merge pull request #719 from PrzemekWirkus/host-test-improvements-part-2
...
Tools: Host test plugin improvements, part 2: Copy plugins
2014-11-19 00:24:28 -08:00
858196ffff
When users are using 'build only flag' (switch -O) and test do not have specified peripherals (muts.json file is just empty dictionary) we can allow test building by default. This supposed to be Yotta related improvement.
2014-11-12 18:02:07 +00:00
7646c51d7e
Host test plugins: minor updates
2014-10-30 14:26:52 +00:00
93a87748a3
Host test plugins: bug-fix: serial_timeout was used as function name and class member. Caused runtime script problem
...
Host test plugins: Increased test suite timeout on each test execution twice - for now, so tests do not timeout. TODO: exclude coopy and reset procedures from testcase timeout measurement
2014-10-29 17:21:55 +00:00
c4397b75b7
Host test plugins: moved program cycle sleep time to host_test - this delay is used to give time device to flash copied file
2014-10-29 14:28:02 +00:00
4d8dfed27f
Host test copy functionality migration: copy functionality partially moved to host_test (tested with basic set of tests, not with e.g. NET tests.
...
Host test copy functionality: TODO: modify rest of host tests to use self.mbed.reset() and self.mbed.copy_image() according to curent test scipt API
2014-10-27 16:38:19 +00:00
16dbfa3f72
Host test copy functionality migration: removed all extra port and disk functionality. It will be added when it is really needed
2014-10-27 15:51:26 +00:00
ae4be96c86
Host test copy functionality migration: added required parameters for host_test.py command line: path to image, copy method
2014-10-27 15:46:15 +00:00
97c220d56e
Host test plugins: rearranged singletest.py script parameters list so now for option -c and -r (copy method and reset method respectively) we will see plugin related options also - changed dynamicaly with plugin loads
2014-10-27 12:42:48 +00:00
2f03b85bfc
Host test plugins: Added generic error printing to host test plugin base class
2014-10-23 17:14:18 +01:00
17f42519f0
Host test plugins: Added MPS2 and Firfox copy methods (both not stable and require further development)
...
Host test plugins: Added 'stable' attribute to each plugin, by default all plugins are not stable
Moved all MPS2 related functions to MPS2 plugins in host test directory
2014-10-23 16:56:05 +01:00
6a1bdb588e
Host test plugins: added copy pligins, and removed Selenium and eACommander dependency from Test API
2014-10-23 16:30:17 +01:00
Brian Daniels
Martin Kojtal
Przemek Wirkus
Brian Daniels