abd4cbcf00
Changing return status of singletest.py if a build failure occurs
2015-06-19 11:53:38 -05:00
0d2effd024
Add option to consolidate waterfall loops into one test case in test report. Useful for CI.
2015-06-08 14:51:13 -05:00
f7fad72e44
Fixed runtime error when test fails due to build failure and outputting test report file
2015-06-08 12:45:30 -05:00
cc6be1a16e
Added support for multiple targets of the same type (hardware platfrom)
2015-05-03 02:32:04 +01:00
9a720be3d4
Small improvements to reporting build failures
2015-05-02 23:08:00 +01:00
373e8c40c0
Fixed test build failure, now BUILD_FAILURE is one of possible results.
...
Added build report after build completion.
Fixed result from execute() capture when result is NoneType value
2015-04-30 19:39:30 +01:00
2f4d668382
Renamed Jenkins specific options to general "Build" options
2015-04-17 09:39:26 -05:00
5eb4894e71
Added skipped mbed library build reporting and removed unnecessary print
2015-04-14 13:39:29 -05:00
a574e1f27e
Broke apart some template stuff, added tests-build report
2015-04-13 17:07:37 -05:00
cb4e13c1e3
Reorganized test_api.py to build all required libraries (RTOS, Ethernet, etc) for tests up front, before any tests are built
2015-04-13 17:07:21 -05:00
cf7f03254a
Added to option --auto handler for -f <mcu> filter switch. Now using -f switch
...
will filter target platforms to test. Use comma to pass more MCU names to
filter.
```
$ singletest.py --auto -j 8 -O --config
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected NUCLEO_L053R8, port: COM35, mounted: E:
MBEDLS: Detected KL25Z, port: COM89, mounted: F:
MUTs configuration in auto-detected:
+-------+-------------+---------------+------+-------+
| index | peripherals | mcu | disk | port |
+-------+-------------+---------------+------+-------+
| 1 | | NUCLEO_L053R8 | E: | COM35 |
| 2 | | KL25Z | F: | COM89 |
+-------+-------------+---------------+------+-------+
Test specification in auto-detected:
+---------------+-----+------+
| mcu | ARM | uARM |
+---------------+-----+------+
| KL25Z | Yes | - |
| NUCLEO_L053R8 | - | Yes |
+---------------+-----+------+
```
Building original configuration (no filter):
```
$ singletest.py --auto -j 8 -O
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected NUCLEO_L053R8, port: COM35, mounted: E:
MBEDLS: Detected KL25Z, port: COM89, mounted: F:
Building library CMSIS (KL25Z, ARM)
Building library MBED (KL25Z, ARM)
Building project DETECT (KL25Z, ARM)
.
.
.
Building library CMSIS (NUCLEO_L053R8, uARM)
Building library MBED (NUCLEO_L053R8, uARM)
Building project DETECT (NUCLEO_L053R8, uARM)
.
.
.
Completed in 3.68 sec
```
```
$ singletest.py --auto -j 8 -O -f KL25Z --config
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected NUCLEO_L053R8, port: COM35, mounted: E:
MBEDLS: Detected KL25Z, port: COM89, mounted: F:
MUTs configuration in auto-detected:
+-------+-------------+-------+------+-------+
| index | peripherals | mcu | disk | port |
+-------+-------------+-------+------+-------+
| 2 | | KL25Z | F: | COM89 |
+-------+-------------+-------+------+-------+
Test specification in auto-detected:
+-------+-----+
| mcu | ARM |
+-------+-----+
| KL25Z | Yes |
+-------+-----+
```
Building original configuration (with applied filter):
```
$ singletest.py --auto -j 8 -O -f KL25Z
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected NUCLEO_L053R8, port: COM35, mounted: E:
MBEDLS: Detected KL25Z, port: COM89, mounted: F:
Building library CMSIS (KL25Z, ARM)
Building library MBED (KL25Z, ARM)
Building project DETECT (KL25Z, ARM)
.
.
.
Completed in 1.33 sec
```
2015-03-13 15:30:44 +00:00
4c45ed1cb1
Parallel functionality, marge conflicts resolved (v2)
...
Description
===========
Added thread model for function SingleTest::execute where for each target we
have separate thread.
Added new experimental feature. Currently we are only building in parallel
mbed SDK and libraries with dependencies.
Each MUT is flashed separately, but with this new feature we are able to
flash multiple boards at the same time and get results in parallel.
Test execution time is reduced by n where n is number of MUTs (MUT: mbed under
test).
using --parallel means that:
* separate thread will be spawned to build mbed SDK and dependencies (libraries)
for each unique target with all declared toolchains.
E.g:
Thread(n==1): LPC1768: ARM, uARM, GCC_ARM
Thread(n==2): K64F: ARM, GCC_ARM
Thread(n==3): NUCLEO_X: uARM
.
.
.
Thread(4): for toolchain in NUCLEO_Z[toolchains]
-> build mbed SDK
-> build libs
-> build project
-> run MUT testrunner
return test results for NUCLEO_Z[toolchain]
Done:
1. Added option --parallel.
2. Decoupled execute function so it can be run in parallel with other execute
functions.
3. Thread join via Queue, not Thread::Join() to avoid deadlocks or waits for
particular thread to finish.
4. All builds are in parallel, but because each target and library for each
toolchain have different directory we do not worry about building in
parallel and compiler collisions.
Missing:
1. No sync for 'print' (TODO).
2. No sync on test result structures - not needed because we only append to
them (?).
Experimental --parallel bugfix: self.test_suite_properties_ext dict was not
populated with target name.
2015-03-12 09:55:11 +00:00
50ef6207f0
Added new switch -p to force peripheral in autodetection mode (--auto)
2015-03-04 09:48:39 +00:00
dfa9599ded
Added mbed assertion detection
...
Test suite will detect mbed assertion print and end test execution with MBED_ASSERT error
2015-02-24 13:46:45 +00:00
0fe5f5f674
Increased initial timeout for each host test so we can be sure binary will be
...
flashed completely and we can proceed with target reset.
Before fix:
====
Test summary:
+---------+--------+-----------+---------+----------------------------+--------------------+---------------+-------+
| Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) | Loops |
+---------+--------+-----------+---------+----------------------------+--------------------+---------------+-------+
| OK | K64F | ARM | NET_1 | TCP client hello world | 3.37 | 20 | 1/1 |
| OK | K64F | ARM | NET_13 | TCP client echo loop | 2.06 | 20 | 1/1 |
| OK | K64F | ARM | NET_2 | NIST Internet Time Service | 2.41 | 20 | 1/1 |
| TIMEOUT | K64F | ARM | NET_3 | TCP echo server | 10.21 | 10 | 0/1 |
| TIMEOUT | K64F | ARM | NET_4 | TCP echo client | 10.21 | 10 | 0/1 |
| TIMEOUT | K64F | ARM | NET_5 | UDP echo server | 10.21 | 10 | 0/1 |
| TIMEOUT | K64F | ARM | NET_6 | UDP echo client | 10.21 | 10 | 0/1 |
| OK | K64F | ARM | NET_7 | HTTP client hello world | 6.46 | 15 | 1/1 |
| TIMEOUT | K64F | ARM | NET_8 | NTP client | 10.21 | 10 | 0/1 |
+---------+--------+-----------+---------+----------------------------+--------------------+---------------+-------+
Result: 4 OK / 5 TIMEOUT
Completed in 116.32 sec
After fix:
====
Test summary:
+--------+--------+-----------+---------+----------------------------+--------------------+---------------+-------+
| Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) | Loops |
+--------+--------+-----------+---------+----------------------------+--------------------+---------------+-------+
| OK | K64F | ARM | NET_1 | TCP client hello world | 3.32 | 20 | 1/1 |
| OK | K64F | ARM | NET_13 | TCP client echo loop | 2.06 | 20 | 1/1 |
| OK | K64F | ARM | NET_2 | NIST Internet Time Service | 2.4 | 20 | 1/1 |
| OK | K64F | ARM | NET_3 | TCP echo server | 1.54 | 20 | 1/1 |
| OK | K64F | ARM | NET_4 | TCP echo client | 1.54 | 20 | 1/1 |
| OK | K64F | ARM | NET_5 | UDP echo server | 1.47 | 20 | 1/1 |
| OK | K64F | ARM | NET_6 | UDP echo client | 1.6 | 20 | 1/1 |
| OK | K64F | ARM | NET_7 | HTTP client hello world | 6.46 | 15 | 1/1 |
| OK | K64F | ARM | NET_8 | NTP client | 2.4 | 15 | 1/1 |
+--------+--------+-----------+---------+----------------------------+--------------------+---------------+-------+
Result: 9 OK
Completed in 125.06 sec
2015-02-17 11:22:45 +00:00
f2f0f51d16
Added colorama to toolchain notification function and basic test suite prompts
2015-02-10 23:38:01 +00:00
8910d95ae6
Fixed bug in --tc switch
2015-02-10 21:29:11 +00:00
f38a53ba21
Added new switch --auto to force autodetection with mbed_lstools
...
Added switch --tc to force certain toolchains.
Switch --auto and -tc works well with switch --config so do not hesitate to
check your configuration before running your test suite.
2015-02-10 21:19:01 +00:00
35c034c911
Added timeout detection from aut-detection MUT printout
...
Timeout is captured by application supervising host_test and duration is modiffied
Added functionality preventing MUT printouts to reset device's timeout value or script execution timer counter
2015-02-10 09:15:46 +00:00
bcfbc3575d
Added -f (filter) to configuration switch --config
...
We can now use --config -f <mcu> to filter only targets we are interested in
2015-01-30 08:38:15 +00:00
b877fe32f9
Added test case time measurement instead of whole test time measurement for 'Elapsed Time (sec)' report in singletest.py console reports
2015-01-28 15:33:11 +00:00
f7f79866d1
Unlocked --db switch. Added check when parsing DB connection string
2014-11-21 13:57:23 +00:00
eadd2013c2
Merge pull request #719 from PrzemekWirkus/host-test-improvements-part-2
...
Tools: Host test plugin improvements, part 2: Copy plugins
2014-11-19 00:24:28 -08:00
858196ffff
When users are using 'build only flag' (switch -O) and test do not have specified peripherals (muts.json file is just empty dictionary) we can allow test building by default. This supposed to be Yotta related improvement.
2014-11-12 18:02:07 +00:00
7646c51d7e
Host test plugins: minor updates
2014-10-30 14:26:52 +00:00
93a87748a3
Host test plugins: bug-fix: serial_timeout was used as function name and class member. Caused runtime script problem
...
Host test plugins: Increased test suite timeout on each test execution twice - for now, so tests do not timeout. TODO: exclude coopy and reset procedures from testcase timeout measurement
2014-10-29 17:21:55 +00:00
c4397b75b7
Host test plugins: moved program cycle sleep time to host_test - this delay is used to give time device to flash copied file
2014-10-29 14:28:02 +00:00
4d8dfed27f
Host test copy functionality migration: copy functionality partially moved to host_test (tested with basic set of tests, not with e.g. NET tests.
...
Host test copy functionality: TODO: modify rest of host tests to use self.mbed.reset() and self.mbed.copy_image() according to curent test scipt API
2014-10-27 16:38:19 +00:00
16dbfa3f72
Host test copy functionality migration: removed all extra port and disk functionality. It will be added when it is really needed
2014-10-27 15:51:26 +00:00
ae4be96c86
Host test copy functionality migration: added required parameters for host_test.py command line: path to image, copy method
2014-10-27 15:46:15 +00:00
97c220d56e
Host test plugins: rearranged singletest.py script parameters list so now for option -c and -r (copy method and reset method respectively) we will see plugin related options also - changed dynamicaly with plugin loads
2014-10-27 12:42:48 +00:00
2f03b85bfc
Host test plugins: Added generic error printing to host test plugin base class
2014-10-23 17:14:18 +01:00
17f42519f0
Host test plugins: Added MPS2 and Firfox copy methods (both not stable and require further development)
...
Host test plugins: Added 'stable' attribute to each plugin, by default all plugins are not stable
Moved all MPS2 related functions to MPS2 plugins in host test directory
2014-10-23 16:56:05 +01:00
6a1bdb588e
Host test plugins: added copy pligins, and removed Selenium and eACommander dependency from Test API
2014-10-23 16:30:17 +01:00
b3c342bf7c
Unlocked option -f (--filter) to filter result by first column
2014-10-20 10:51:02 +01:00
c0ed6ba791
Disabled --db switch - reasoning: DB schema is still nto in test suite and it's not 100% ready so before I introduce MySQL DB schema officialy let's make this option dorment
2014-10-20 10:34:13 +01:00
12a9051aba
Removed few minor pyFlakes warnings
2014-10-15 16:33:30 +01:00
50966fdf89
Refactoring of report exporter to support HTML and JUNIT: Fixed issues with non ASCII characters read from serial port in ProcessObserver. Also changed reporting for errors and failures in TestSuite JUnit class
2014-10-15 16:05:57 +01:00
2274651892
Refactoring of report exporter to support HTML and JUNIT. Added unicode encoding for strings coming from serial console
2014-10-15 13:38:20 +01:00
091cccaf6e
Refactoring of report exporter to support HTML and JUNIT: Simple variable scope bug-fix
2014-10-15 11:36:47 +01:00
7ce6287474
Refactoring of report exporter to support HTML and JUNIT. Added properties for each unique test suite. Each test suite is sorted out as different toolchain
2014-10-15 11:33:41 +01:00
567f4a170f
Refactoring of report exporter to support HTML and JUNIT. Modified host tests to support serial and copy issues in more generic way. Copy in test suite and serial in host_test control flow.
2014-10-15 11:33:37 +01:00
51223ef6d1
Refactoring of report exporter to support HTML and JUNIT. TODO: add support for TEXT output.
...
Added JUtnit exporter
2014-10-15 11:33:30 +01:00
9fe239dcad
Added HTML reporting feature, switch --report-html FILENAME to singletest.py so now users can export test results to external HTML file. This feature can be used to display HTML reports on CI dashboards
2014-09-25 17:42:08 +01:00
95519c1334
Minor modifications to test api scripts
2014-09-25 17:42:04 +01:00
2ab8ad9b47
Feature: Device detection via serial - part 4
...
Interrupt In test pinology for NUCLEO changed
2014-09-24 14:58:14 +01:00
6690db1444
Feature: Device detection via serial - part 2
2014-09-24 14:58:07 +01:00
2e58f7922c
Feature: Device detection via serial - part 1
2014-09-24 14:58:03 +01:00
b031920624
Added clean build switch --clean to singletest script
2014-09-18 11:17:57 +01:00
7d9048fccc
Simple refactoring around MUT copy functionality - 'copy_method' avail in MUTs definition
2014-09-10 14:09:25 +01:00
Brian Daniels
Brian Daniels
Przemek Wirkus
Martin Kojtal