Tools: Interoperability test suite: Added simple interface chip tests / checks
I'm merging this pull request, we can add more interoperability tests in the future. For now we can use this is CI.
I'm expecting some issues with this but this is not essential functionality and we can always improve it in the future.
Description
===========
Added thread model for function SingleTest::execute where for each target we
have separate thread.
Added new experimental feature. Currently we are only building in parallel
mbed SDK and libraries with dependencies.
Each MUT is flashed separately, but with this new feature we are able to
flash multiple boards at the same time and get results in parallel.
Test execution time is reduced by n where n is number of MUTs (MUT: mbed under
test).
using --parallel means that:
* separate thread will be spawned to build mbed SDK and dependencies (libraries)
for each unique target with all declared toolchains.
E.g:
Thread(n==1): LPC1768: ARM, uARM, GCC_ARM
Thread(n==2): K64F: ARM, GCC_ARM
Thread(n==3): NUCLEO_X: uARM
.
.
.
Thread(4): for toolchain in NUCLEO_Z[toolchains]
-> build mbed SDK
-> build libs
-> build project
-> run MUT testrunner
return test results for NUCLEO_Z[toolchain]
Done:
1. Added option --parallel.
2. Decoupled execute function so it can be run in parallel with other execute
functions.
3. Thread join via Queue, not Thread::Join() to avoid deadlocks or waits for
particular thread to finish.
4. All builds are in parallel, but because each target and library for each
toolchain have different directory we do not worry about building in
parallel and compiler collisions.
Missing:
1. No sync for 'print' (TODO).
2. No sync on test result structures - not needed because we only append to
them (?).
Experimental --parallel bugfix: self.test_suite_properties_ext dict was not
populated with target name.
Adde missing check for optional auto_detect parameter in CLI options.
Test procedure:
* Check with mbed-ls installed:
$ singletest.py --auto -j 8
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected K64F, port: COM61, mounted: E:
Building library CMSIS (K64F, ARM)
Copy: startup_MK64F12.o
Copy: sys.o
Copy: cmsis_nvic.o
Copy: system_MK64F12.o
* Uninstall mbed-ls iools:
$ pip uninstall mbed-ls
Uninstalling mbed-ls:
c:\python27\lib\site-packages\mbed_ls-0.1.4-py2.7.egg
c:\python27\scripts\mbedls-script.py
c:\python27\scripts\mbedls.exe
c:\python27\scripts\mbedls.exe.manifest
Proceed (y/n)? y
Successfully uninstalled mbed-ls
$ mbedls
'mbedls' is not recognized as an internal or external command,
operable program or batch file.
$ python
Python 2.7.8 (default, Jun 30 2014, 16:03:49) [MSC v.1500 32 bit (Intel)] win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import mbed_lstools
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named mbed_lstools
* Check singletest.py work flow without mbed-ls:
$ singletest.py -i test_spec.json -M muts_all.json
Building library CMSIS (K64F, ARM)
Building library MBED (K64F, ARM)
Building project DETECT (K64F, ARM)
TargetTest::K64F::ARM::DTCT_1::Simple detect test [OK] in 0.50 of 10 sec
Building project DEV_NULL (K64F, ARM)
TargetTest::K64F::ARM::EXAMPLE_1::/dev/null [OK] in 3.49 of 20 sec
Building project HELLO (K64F, ARM)
TargetTest::K64F::ARM::MBED_10::Hello World [OK] in 0.38 of 5 sec
Building project TICKER (K64F, ARM)
TargetTest::K64F::ARM::MBED_11::Ticker Int [OK] in 11.35 of 15 sec
Added switch --tc to force certain toolchains.
Switch --auto and -tc works well with switch --config so do not hesitate to
check your configuration before running your test suite.
Added separate file to store test suite DB access interface
Added new functionality to database interface: database name, connection status, hostname + uname functionality
Added option --db to singletest.py so users can define database connection url for database access
Added database configuration check with switch --config. Now users can combine switch --db and --config to check database conenctivity
sg-
Mihail Stoyanov
Brian Daniels
Przemek Wirkus
rnestler
Brian Daniels