ARMmbed
/
mbed-os
mirror of https://github.com/ARMmbed/mbed-os.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Merge pull request #11710 from jamesbeyond/exp_github 

Update examples testing scripts
pull/11756/head
Martin Kojtal 2019-10-28 09:50:36 +01:00 committed by GitHub
parent 87a5e47b42 a211cf16a9
commit 83c4a80082
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
4 changed files with 523 additions and 332 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

167
tools/test/examples/README.md Normal file
View File

@ -0,0 +1,167 @@
# Examples testing script
The scripts in this folder are used for testing `mbed-os` official examples. It contains the following files:
- `examples.py` - the main script that serves as the command-line interface.
- `examples.json` - the default configuration file for the script, which contains the information to the examples. If required, you can pass a customized configuration file as an argument to the script.
- `examples_lib.py` - the library file, which contains the main function of the testing scripts.
## The scripts
`examples.py` provides command-line interface and subcommands that makes easier to test examples. Included subcommands are:
* **import** - imports each of the repos and its dependencies (.lib files) associated with the specific examples name from the .json configuration file. If there is already a clone of the repo, it will first be removed to ensure a clean, up-to-date cloning.
* **clone** - clones each of the repos associated with the specific examples name from the .json configuration file. If there is already a clone of the repo, it will first be removed to ensure a clean, up-to-date cloning.
* **deploy** - if the example directory exists as provided by the .json configuration file, pulls in the examples dependencies by using `mbed-cli deploy`.
* **update** - for each example repo identified in the config .json object, updates the version of `mbed-os` to that specified by the supplied GitHub tag. This function assumes that each example repo has already been cloned.
* **compile** - compiles combinations of example programs, targets and compile chains.
* **export** - exports and builds combinations of example programs, targets and IDEs.
* **list** - displays examples in a configuration file in a table.
* **symlink** - creates a symbolic link to a given `mbed-os` PATH.
For more detailed options, please use `-h` or `--help`.
## The configuration file
Here is the section of default configuration file:
```json
{
"examples": [
{
"name": "mbed-os-example-blinky",
"github": "https://github.com/ARMmbed/mbed-os-example-blinky",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : [],
"toolchains" : [],
"exporters": [],
"compile" : true,
"export": true,
"test" : true,
"baud_rate": 9600,
"compare_log": ["mbed-os-example-blinky/tests/blinky.log"],
"auto-update" : true
},
...
{
"name": "mbed-os-example-tls",
"github": "https://github.com/ARMmbed/mbed-os-example-tls",
"sub-repo-example": true,
"subs": [
"benchmark",
"tls-client",
"hashing",
"authcrypt"
],
"features" : [],
"targets" : ["K66F", "NUCLEO_F429ZI"],
"toolchains" : ["GCC_ARM", "ARM"],
"exporters": [],
"compile" : false,
"export": false,
"test" : false,
"baud_rate": 9600,
"compare_log": [
"mbed-os-example-tls/tests/benchmark.log",
"mbed-os-example-tls/tests/tls-client.log",
"mbed-os-example-tls/tests/hashing.log",
"mbed-os-example-tls/tests/authcrypt.log"
],
"auto-update" : true
}
]
}
```
### Fields
* **name** - name of the example. It should be the base name of the example repository address and will throw if it doesn't match.
* **github** - example GitHub repository address.
* **sub-repo-example** - specifies if the example repository has a subfolder for each example.
* **subs** - array of subexample names.
* **features** - the features that must be in the features array of a target in `targets.json`.
* **baud_rate** - example default baud rate.
* **compare_log** - array of log compared to command-line output during testing. If the example has many subexamples, the order of log should match the order of subexamples.
* **targets** - list of `mbed-os` development boards that run this example.
* **targets** - list of targeted development boards.
* **toolchains** - toolchain to use for compiling.
* **exporters** - allowed exporters.
* **compile** - enables compiling.
* **export** - enables exporting.
* **test** - enables testing.
### Values
`[ ]` means all possible alternatives.
## Typical use
In the Mbed OS CI, we follow the below steps to compile and test Mbed OS examples.
1. Clone `mbed-os` repository to the current folder:
```
git clone https://github.com/ARMmbed/mbed-os.git
```
1. Clone the examples repo to the current folder. Users can pass an `-e` option to the script to filter out the rest of the examples, so the scripts only run on one particular example:
```
python mbed-os/tools/test/examples/examples.py clone
```
1. Create a symbolic link to `mbed-os` for every example. This step lets all the examples share a single `mbed-os` folder, rather than checking out the `mbed-os` folder many times. We highly recommend you pass an absolute path as the argument:
```
python mbed-os/tools/test/examples/examples.py symlink $PWD/mbed-os
```
1. Deploy other dependency libraries:
```
python mbed-os/tools/test/examples/examples.py deploy
```
1. Compile the test for the examples on a specific target:
```
python mbed-os/tools/test/examples/examples.py compile -m <target>
```
After the compile test finished, the scripts print the result table:
```
Passed example compilation:
+---------------------------------+--------+-----------+----------+--------------+
| EXAMPLE NAME | TARGET | TOOLCHAIN | TEST GEN | BUILD RESULT |
+---------------------------------+--------+-----------+----------+--------------+
| mbed-os-example-kvstore | K64F | GCC_ARM | TEST_ON | PASSED |
| mbed-os-example-tls-socket | K64F | GCC_ARM | TEST_ON | PASSED |
| mbed-os-example-blockdevice | K64F | GCC_ARM | TEST_ON | PASSED |
| mbed-os-example-wifi | K64F | GCC_ARM | TEST_OFF | PASSED |
| mbed-os-example-error-handling | K64F | GCC_ARM | TEST_ON | PASSED |
| mbed-os-example-sd-driver | K64F | GCC_ARM | TEST_ON | PASSED |
| mbed-os-example-crash-reporting | K64F | GCC_ARM | TEST_ON | PASSED |
| mbed-os-example-filesystem | K64F | GCC_ARM | TEST_ON | PASSED |
| mbed-os-example-blinky | K64F | GCC_ARM | TEST_ON | PASSED |
| mbed-os-example-bootloader | K64F | GCC_ARM | TEST_OFF | PASSED |
| mbed-os-example-cpu-stats | K64F | GCC_ARM | TEST_ON | PASSED |
| mbed-os-example-sys-info | K64F | GCC_ARM | TEST_ON | PASSED |
| mbed-os-example-attestation | K64F | GCC_ARM | TEST_ON | PASSED |
+---------------------------------+--------+-----------+----------+--------------+
Number of failures = 0
```
After the compilation stage, a `test_spec.json` file is generated. Later, Greentea tests will consume this file. They test the compiled example on hardware platform.

177
tools/test/examples/examples.json
View File

@ -3,10 +3,8 @@
{
"name": "mbed-os-example-blinky",
"github": "https://github.com/ARMmbed/mbed-os-example-blinky",
"mbed": [
"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-blinky"
],
"test-repo-source": "github",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : [],
"toolchains" : [],
@ -21,34 +19,33 @@
{
"name": "mbed-os-example-blinky-baremetal",
"github": "https://github.com/ARMmbed/mbed-os-example-blinky-baremetal",
"mbed": [
"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-blinky-baremetal"
],
"test-repo-source": "github",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : ["K66F", "NUCLEO_F429ZI", "ARCH_PRO", "LPC1768"],
"toolchains" : [],
"exporters": ["iar", "make_armc5", "make_armc6", "make_gcc_arm", "make_iar"],
"compile" : true,
"export": true,
"test" : false,
"auto-update" : true
},
{
"name": "mbed-os-example-tls",
"github": "https://github.com/ARMmbed/mbed-os-example-tls",
"mbed": [
"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-tls-benchmark",
"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-tls-tls-client",
"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-tls-hashing",
"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-tls-authcrypt"
"sub-repo-example": true,
"subs": [
"benchmark",
"tls-client",
"hashing",
"authcrypt"
],
"test-repo-source": "mbed",
"features" : [],
"targets" : ["K66F", "NUCLEO_F429ZI"],
"toolchains" : ["GCC_ARM", "ARM"],
"exporters": [],
"compile" : true,
"export": true,
"compile" : false,
"export": false,
"test" : false,
"baud_rate": 9600,
"compare_log": [
@ -62,31 +59,31 @@
{
"name": "mbed-os-example-ble",
"github":"https://github.com/ARMmbed/mbed-os-example-ble",
"mbed": [
"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-ble-Beacon",
"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-ble-HeartRate",
"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-ble-Thermometer",
"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-ble-LEDBlinker",
"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-ble-LED",
"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-ble-GAPButton",
"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-ble-Button",
"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-ble-BatteryLevel"
"sub-repo-example": true,
"subs": [
"BLE_Beacon",
"BLE_HeartRate",
"BLE_Thermometer",
"BLE_LEDBlinker",
"BLE_LED",
"BLE_GAPButton",
"BLE_Button",
"BLE_BatteryLevel"
],
"test-repo-source": "mbed",
"features" : ["BLE"],
"targets" : ["NRF51_DK", "NRF52_DK", "K66F", "NUCLEO_F401RE"],
"toolchains" : [],
"exporters": [],
"compile" : true,
"compile" : false,
"export": false,
"test" : false,
"auto-update" : true
},
{
"name": "mbed-os-example-sockets",
"github":"https://github.com/ARMmbed/mbed-os-example-sockets",
"mbed": [
],
"test-repo-source": "github",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : ["K66F", "NUCLEO_F429ZI", "NUMAKER_PFM_NUC472", "FVP_MPS2_M3"],
"toolchains" : [],
@ -101,9 +98,8 @@
{
"name": "mbed-os-example-tls-socket",
"github":"https://github.com/ARMmbed/mbed-os-example-tls-socket",
"mbed": [
],
"test-repo-source": "github",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : ["K64F", "DISCO_F746NG"],
"toolchains" : [],
@ -118,61 +114,64 @@
{
"name": "mbed-os-example-wifi",
"github":"https://github.com/ARMmbed/mbed-os-example-wifi",
"mbed": [
],
"test-repo-source": "github",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : [],
"toolchains" : [],
"exporters": [],
"compile" : true,
"export": true,
"test" : false,
"auto-update" : true
},
{
"name": "nanostack-border-router",
"github":"https://github.com/ARMmbed/nanostack-border-router",
"mbed": [],
"test-repo-source": "github",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : ["K66F", "NUCLEO_F429ZI"],
"toolchains" : [],
"exporters": [],
"compile" : true,
"export": true,
"test" : false,
"auto-update" : true
},
{
"name": "mbed-os-example-cellular",
"github":"https://github.com/ARMmbed/mbed-os-example-cellular",
"mbed": [],
"test-repo-source": "github",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : ["MTS_DRAGONFLY_F411RE"],
"toolchains" : [],
"exporters": [],
"compile" : true,
"export": true,
"test" : false,
"auto-update" : true
},
{
"name": "mbed-os-example-lorawan",
"github":"https://github.com/ARMmbed/mbed-os-example-lorawan",
"mbed": [],
"test-repo-source": "github",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : ["DISCO_L072CZ_LRWAN1", "MTB_MTS_XDOT", "MTS_MDOT_F411RE"],
"toolchains" : [],
"exporters": [],
"compile" : true,
"export": false,
"test" : false,
"auto-update" : true
},
{
"name": "mbed-os-example-nvstore",
"github":"https://github.com/ARMmbed/mbed-os-example-nvstore",
"mbed": [],
"test-repo-source": "github",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : ["K66F", "FVP_MPS2_M3"],
"toolchains" : [],
@ -187,8 +186,8 @@
{
"name": "mbed-os-example-devicekey",
"github":"https://github.com/ARMmbed/mbed-os-example-devicekey",
"mbed": [],
"test-repo-source": "github",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : ["K66F"],
"toolchains" : [],
@ -203,8 +202,8 @@
{
"name": "mbed-os-example-thread-statistics",
"github":"https://github.com/ARMmbed/mbed-os-example-thread-statistics",
"mbed": [],
"test-repo-source": "github",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : ["K66F", "FVP_MPS2_M3"],
"toolchains" : [],
@ -219,8 +218,8 @@
{
"name": "mbed-os-example-sys-info",
"github":"https://github.com/ARMmbed/mbed-os-example-sys-info",
"mbed": [],
"test-repo-source": "github",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : ["K64F", "FVP_MPS2_M3"],
"toolchains" : [],
@ -235,8 +234,8 @@
{
"name": "mbed-os-example-cpu-usage",
"github":"https://github.com/ARMmbed/mbed-os-example-cpu-usage",
"mbed": [],
"test-repo-source": "github",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : ["K66F", "FVP_MPS2_M3"],
"toolchains" : [],
@ -251,8 +250,8 @@
{
"name": "mbed-os-example-cpu-stats",
"github":"https://github.com/ARMmbed/mbed-os-example-cpu-stats",
"mbed": [],
"test-repo-source": "github",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : ["K64F", "FVP_MPS2_M3"],
"toolchains" : [],
@ -267,8 +266,8 @@
{
"name": "mbed-os-example-socket-stats",
"github":"https://github.com/ARMmbed/mbed-os-example-socket-stats",
"mbed": [],
"test-repo-source": "github",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : ["K66F"],
"toolchains" : [],
@ -283,8 +282,8 @@
{
"name": "mbed-os-example-error-handling",
"github":"https://github.com/ARMmbed/mbed-os-example-error-handling",
"mbed": [],
"test-repo-source": "github",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : ["K64F", "FVP_MPS2_M3"],
"toolchains" : [],
@ -299,10 +298,8 @@
{
"name": "mbed-os-example-filesystem",
"github":"https://github.com/ARMmbed/mbed-os-example-filesystem",
"mbed": [
"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-filesystem"
],
"test-repo-source": "github",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : ["K64F","K82F"],
"toolchains" : [],
@ -317,10 +314,8 @@
{
"name": "mbed-os-example-mesh-minimal",
"github":"https://github.com/ARMmbed/mbed-os-example-mesh-minimal",
"mbed": [
"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-mesh-minimal"
],
"test-repo-source": "github",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : ["DISCO_F469NI", "DISCO_F746NG", "K66F",
"NUCLEO_F429ZI", "NUCLEO_F439ZI", "NUCLEO_F746ZG",
@ -330,35 +325,35 @@
"exporters": [],
"compile" : true,
"export": true,
"test" : false,
"auto-update" : true
},
{
"name": "mbed-os-example-bootloader",
"github":"https://github.com/ARMmbed/mbed-os-example-bootloader",
"mbed": [
"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-bootloader"
],
"test-repo-source": "github",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : ["K64F", "NUCLEO_F429ZI", "UBLOX_EVK_ODIN_W2"],
"toolchains" : [],
"exporters": [],
"compile" : true,
"export": true,
"test" : false,
"auto-update" : true
},
{
"name": "mbed-os-example-mbed-crypto",
"github":"https://github.com/ARMmbed/mbed-os-example-mbed-crypto",
"mbed": [],
"test-repo-source": "github",
"sub-repo-example": true,
"subs": ["getting-started"],
"features" : [],
"targets" : ["K64F"],
"toolchains" : [],
"exporters": [],
"compile" : true,
"export": true,
"test" : true,
"compile" : false,
"export": false,
"test" : false,
"baud_rate": 9600,
"compare_log": ["mbed-os-example-mbed-crypto/tests/getting-started.log"],
"auto-update" : true
@ -366,23 +361,25 @@
{
"name": "mbed-os-example-nfc",
"github": "https://github.com/ARMmbed/mbed-os-example-nfc",
"mbed": [
"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-nfc-SmartPoster"
"sub-repo-example": true,
"subs": [
"NFC_EEPROM",
"NFC_SmartPoster"
],
"test-repo-source": "mbed",
"features" : [],
"targets" : ["NUCLEO_F401RE", "DISCO_L475VG_IOT01A"],
"toolchains" : [],
"exporters": [],
"compile" : true,
"export": true,
"compile" : false,
"export": false,
"test" : false,
"auto-update" : true
},
{
"name": "mbed-os-example-blockdevice",
"github":"https://github.com/armmbed/mbed-os-example-blockdevice",
"mbed": [],
"test-repo-source": "github",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : ["K64F"],
"toolchains" : [],
@ -397,8 +394,8 @@
{
"name": "mbed-os-example-kvstore",
"github":"https://github.com/ARMmbed/mbed-os-example-kvstore",
"mbed": [],
"test-repo-source": "github",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : ["K64F", "FVP_MPS2_M3"],
"toolchains" : [],
@ -413,8 +410,8 @@
{
"name": "mbed-os-example-crash-reporting",
"github":"https://github.com/ARMmbed/mbed-os-example-crash-reporting",
"mbed": [],
"test-repo-source": "github",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : ["K64F", "DISCO_L475VG_IOT01A"],
"toolchains" : [],
@ -429,8 +426,8 @@
{
"name": "mbed-os-example-sd-driver",
"github":"https://github.com/ARMmbed/mbed-os-example-sd-driver",
"mbed": [],
"test-repo-source": "github",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : ["K64F"],
"toolchains" : [],
@ -445,8 +442,8 @@
{
"name": "mbed-os-example-attestation",
"github": "https://github.com/ARMmbed/mbed-os-example-attestation",
"mbed": [],
"test-repo-source": "github",
"sub-repo-example": false,
"subs": [],
"features" : [],
"targets" : ["CY8CKIT_062_WIFI_BT_PSA",
"K64F",

87
tools/test/examples/examples.py
View File

@ -1,3 +1,5 @@
#!/usr/bin/env python
"""
Copyright (c) 2017-2019 ARM Limited. All rights reserved.
@ -19,6 +21,7 @@ limitations
from argparse import ArgumentParser
import os
from os.path import dirname, abspath, basename
from prettytable import PrettyTable
import os.path
import sys
import subprocess
@ -35,9 +38,8 @@ from tools.build_api import get_mbed_official_release
import examples_lib as lib
from examples_lib import SUPPORTED_TOOLCHAINS, SUPPORTED_IDES
def main():
"""Entry point"""
def parse_args():
"""Parse the arguments passed to the script."""
official_targets = get_mbed_official_release("5")
official_target_names = [x[0] for x in official_targets]
@ -49,16 +51,21 @@ def main():
type=argparse_many(lambda x: x),
default=[])
subparsers = parser.add_subparsers()
import_cmd = subparsers.add_parser("import")
import_cmd = subparsers.add_parser("import", help="import of examples in config file" )
import_cmd.set_defaults(fn=do_import)
clone_cmd = subparsers.add_parser("clone")
clone_cmd = subparsers.add_parser("clone", help="clone examples in config file" )
clone_cmd.set_defaults(fn=do_clone)
deploy_cmd = subparsers.add_parser("deploy")
list_cmd = subparsers.add_parser("list", help="list examples in config file in a table")
list_cmd.set_defaults(fn=do_list)
symlink_cmd = subparsers.add_parser("symlink", help="create symbolic link to given mbed-os PATH")
symlink_cmd.add_argument("PATH", help=" path of mbed-os to be symlinked")
symlink_cmd.set_defaults(fn=do_symlink)
deploy_cmd = subparsers.add_parser("deploy", help="mbed deploy for examples in config file" )
deploy_cmd.set_defaults(fn=do_deploy)
version_cmd = subparsers.add_parser("tag")
version_cmd.add_argument("tag")
version_cmd.set_defaults(fn=do_versionning)
compile_cmd = subparsers.add_parser("compile")
version_cmd = subparsers.add_parser("update", help="update mbed-os to sepcific tags")
version_cmd.add_argument("TAG", help=" tag of mbed-os")
version_cmd.set_defaults(fn=do_update)
compile_cmd = subparsers.add_parser("compile", help="compile of examples" )
compile_cmd.set_defaults(fn=do_compile),
compile_cmd.add_argument(
"toolchains", nargs="*", default=SUPPORTED_TOOLCHAINS,
@ -73,10 +80,12 @@ def main():
official_target_names, "MCU")),
default=official_target_names)
compile_cmd.add_argument("--profile",
help=("build profile file"),
metavar="profile")
compile_cmd.add_argument(
"--profiles",
nargs='+',
metavar="profile",
help="build profile(s)")
compile_cmd.add_argument("-j", "--jobs",
dest='jobs',
metavar="NUMBER",
@ -90,8 +99,8 @@ def main():
default=False,
help="Verbose diagnostic output")
export_cmd = subparsers.add_parser("export")
export_cmd.set_defaults(fn=do_export),
export_cmd = subparsers.add_parser("export", help="export of examples")
export_cmd.set_defaults(fn=do_export)
export_cmd.add_argument(
"ide", nargs="*", default=SUPPORTED_IDES,
type=argparse_force_uppercase_type(SUPPORTED_IDES,
@ -104,25 +113,32 @@ def main():
argparse_force_uppercase_type(
official_target_names, "MCU")),
default=official_target_names)
args = parser.parse_args()
return parser.parse_args()
def main():
"""Entry point"""
args = parse_args()
config = json.load(open(os.path.join(os.path.dirname(__file__),
args.config)))
all_examples = []
for example in config['examples']:
all_examples = all_examples + [basename(x['repo']) for x in lib.get_repo_list(example)]
examples = [x for x in all_examples if x in args.example] if args.example else all_examples
return args.fn(args, config, examples)
name = basename(example['github'])
if name != example['name']:
exit("ERROR : repo basename '%s' and example name '%s' not match " % (name, example['name']))
all_examples.append(name)
exp_filter = [x for x in all_examples if x in args.example] if args.example else all_examples
return args.fn(args, config, exp_filter)
def do_export(args, config, examples):
"""Do export and build step"""
results = {}
results = lib.export_repos(config, args.ide, args.mcu, examples)
lib.print_summary(results, export=True)
failures = lib.get_num_failures(results, export=True)
print("Number of failures = %d" % failures)
failures = lib.get_export_summary(results)
return failures
@ -143,16 +159,27 @@ def do_deploy(_, config, examples):
def do_compile(args, config, examples):
"""Do the compile step"""
results = lib.compile_repos(config, args.toolchains, args.mcu, args.profile, args.verbose, examples, args.jobs)
lib.print_summary(results)
failures = lib.get_num_failures(results)
print("Number of failures = %d" % failures)
results = lib.compile_repos(config, args.toolchains, args.mcu, args.profiles, args.verbose, examples, args.jobs)
failures = lib.get_build_summary(results)
return failures
def do_versionning(args, config, examples):
def do_update(args, config, examples):
""" Test update the mbed-os to the version specified by the tag """
return lib.update_mbedos_version(config, args.tag, examples)
return lib.update_mbedos_version(config, args.TAG, examples)
def do_list(_, config, examples):
"""List the examples in the config file"""
exp_table = PrettyTable()
exp_table.hrules = 1
exp_table.field_names = ["Name", "Subs", "Feature", "Targets", "Compile", "Test"]
for example in config["examples"]:
exp_table.add_row([example['name'], '\n'.join(example['subs']),'\n'.join(example['features']),'\n'.join(example['targets']),example['compile'],example['test']])
print(exp_table)
return 0
def do_symlink(args, config, examples):
"""Create Symbolic link for given mbed-os PATH"""
return lib.symlink_mbedos(config, args.PATH, examples)
if __name__ == "__main__":
sys.exit(main())

424
tools/test/examples/examples_lib.py
View File

@ -1,3 +1,5 @@
#!/usr/bin/env python
"""
Copyright (c) 2017-2019 ARM Limited. All rights reserved.
@ -19,9 +21,14 @@ import os
from os.path import dirname, abspath, basename, join, normpath
import os.path
import sys
import copy
import stat
import subprocess
from shutil import rmtree
import json
import logging
logging.basicConfig(level=logging.DEBUG, format='[EXAMPLES]> %(levelname)-8s %(message)s')
""" Import and bulid a bunch of example programs
@ -30,8 +37,9 @@ import json
"""
ROOT = abspath(dirname(dirname(dirname(dirname(__file__)))))
sys.path.insert(0, ROOT)
MBED_OS_ROOT = abspath(dirname(dirname(dirname(dirname(__file__)))))
CWD = os.getcwd()
sys.path.insert(0, MBED_OS_ROOT)
from tools.build_api import get_mbed_official_release
from tools.targets import TARGET_MAP
@ -39,65 +47,74 @@ from tools.export import EXPORTERS
from tools.project import EXPORTER_ALIASES
from tools.toolchains import TOOLCHAINS
from tools.utils import write_json_to_file
from prettytable import PrettyTable
SUPPORTED_TOOLCHAINS = list(TOOLCHAINS - set(u'uARM'))
SUPPORTED_IDES = [exp for exp in list(EXPORTERS) + list(EXPORTER_ALIASES)
if exp != "cmsis" and exp != "zip"]
def print_list(lst):
"""Prints to screen the contents of a list
def get_build_summary(results):
"""Prints to screen the complication results of example programs.
Args:
lst - a list of any type, to be displayed
results - results of the compilation stage. which is the output of compile_repos()
Returns: Numbers of failed results
"""
if lst:
for thing in lst:
print("# %s" % thing)
pass_table = PrettyTable()
pass_table.field_names = ["EXAMPLE NAME", "TARGET", "TOOLCHAIN", "TEST GEN", "BUILD RESULT"]
pass_table.align["EXAMPLE NAME"] = "l"
fail_table = copy.deepcopy(pass_table)
failure_counter = 0
for exp, status in list(results.items()):
for summary in status[2]:
pass_table.add_row([summary["name"], summary["target"], summary["toolchain"], summary["test"], "PASSED"])
for summary in status[3]:
fail_table.add_row([summary["name"], summary["target"], summary["toolchain"], summary["test"], "FAILED"])
failure_counter+=1
print("\n\nPassed Example Compilation:")
print(pass_table)
if (failure_counter > 0):
print("\n\nFailed Example Compilation:")
print(fail_table)
print("Number of failures = %d" % failure_counter)
return failure_counter
def print_category(results, index, message):
summary = [example for key, summ in list(results.items())
for example in summ[index]]
if all(len(s) == 0 for s in summary):
return
print("#")
print("#" * 80)
print("# %s" % message)
print("#" * 80)
split_summ = [s.rsplit(" ", 1) for s in summary]
print_list(summary)
def print_summary(results, export=False):
"""Prints to screen the results of compiling/exporting combinations of example programs,
targets and compile toolchains/IDEs.
def get_export_summary(results):
"""Prints to screen the exporting results of example programs.
Args:
results - results of the compilation stage. See compile_repos() and export_repos()
for details of the format.
results - results of the compilation stage. which is the output of and export_repos()
Returns: Numbers of failed results
"""
print("#"*80)
print("# Examples compilation summary")
print("#"*80)
print_category(results, 2, "Passed example combinations")
second_result = "Failed example combinations" if not export else \
"Failed export example combinations"
print_category(results, 3, second_result)
if export:
print_category(results, 4, "Failed build combinations")
print_category(results, 5, "Skipped build combinations")
print("#")
print("#"*80)
pass_table = PrettyTable()
pass_table.field_names = ["EXAMPLE NAME", "TARGET", "IDE", "EXPORT RESULT", "BUILD RESULT"]
pass_table.align["EXAMPLE NAME"] = "l"
fail_table = copy.deepcopy(pass_table)
failure_counter = 0
for exp, status in list(results.items()):
for summary in status[2]:
pass_table.add_row([summary["name"], summary["target"], summary["ide"], "PASSED", "PASSED"])
for summary in status[3]:
fail_table.add_row([summary["name"], summary["target"], summary["ide"], "FAILED", ""])
failure_counter+=1
for summary in status[4]:
fail_table.add_row([summary["name"], summary["target"], summary["ide"], "PASSED", "FAILED"])
failure_counter+=1
for summary in status[5]:
pass_table.add_row([summary["name"], summary["target"], summary["ide"], "PASSED", "SKIPPED"])
print("\n\nPassed Example Exporting:")
print(pass_table)
if (failure_counter > 0):
print("\n\nFailed Example Exporting:")
print(fail_table)
print("Number of failures = %d" % failure_counter)
return failure_counter
def valid_choices(allowed_choices, all_choices):
if len(allowed_choices) > 0:
@ -144,34 +161,17 @@ def target_cross_ide(allowed_targets, allowed_ides, features=[], toolchains=[]):
yield target, ide
def get_repo_list(example):
""" Returns a list of all the repos and their types associated with the
specific example in the json config file.
If the key 'test-repo-source' is set to 'mbed', then it will return the
mbed section as a list. Otherwise, it will return the single github repo.
NOTE: This does not currently deal with multiple examples underneath a github
sourced exampe repo.
Args:
example - Example for which the repo list is requested
"""
repos = []
if example['test-repo-source'] == 'mbed':
for repo in example['mbed']:
repos.append({
'repo': repo,
'type': 'hg'
})
def get_sub_examples_list(example):
""" Get the names of sub examples. if no sub examples, return the name of main example"""
sub_examples = []
if example['sub-repo-example']:
for sub in example['subs']:
sub_examples.append("%s/%s" % (example["name"], sub))
else:
repos.append({
'repo': example['github'],
'type': 'git'
})
return repos
sub_examples.append(example["name"])
return sub_examples
def source_repos(config, examples):
def source_repos(config, exp_filter):
""" Imports each of the repos and its dependencies (.lib files) associated
with the specific examples name from the json config file. Note if
there is already a clone of the repo then it will first be removed to
@ -182,22 +182,20 @@ def source_repos(config, examples):
"""
print("\nImporting example repos....\n")
for example in config['examples']:
for repo_info in get_repo_list(example):
name = basename(repo_info['repo'])
if name in examples:
if os.path.exists(name):
print("'%s' example directory already exists. Deleting..." % name)
rmtree(name)
cmd = "mbed-cli import %s" %repo_info['repo']
result = subprocess.call(cmd, shell=True)
if result:
return result
name = example['name']
if name in exp_filter:
if os.path.exists(name):
logging.warning("'%s' example directory already exists. Deleting..." % name)
rmtree(name)
cmd = "mbed-cli import %s" % example['github']
logging.info("Executing command '%s'..." % cmd)
result = subprocess.call(cmd, shell=True)
if result:
return result
return 0
def clone_repos(config, examples , retry = 3):
def clone_repos(config, exp_filter , retry = 3):
""" Clones each of the repos associated with the specific examples name from the
json config file. Note if there is already a clone of the repo then it will first
be removed to ensure a clean, up to date cloning.
@ -207,22 +205,23 @@ def clone_repos(config, examples , retry = 3):
"""
print("\nCloning example repos....\n")
for example in config['examples']:
for repo_info in get_repo_list(example):
name = basename(repo_info['repo'])
if name in examples:
if os.path.exists(name):
print("'%s' example directory already exists. Deleting..." % name)
rmtree(name)
cmd = "%s clone %s" %(repo_info['type'], repo_info['repo'])
for i in range(0, retry):
if not subprocess.call(cmd, shell=True):
break
else:
print("ERROR : unable to clone the repo {}".format(name))
return 1
name = example['name']
if name in exp_filter:
if os.path.exists(name):
logging.warning("'%s' example directory already exists. Deleting..." % name)
rmtree(name, onerror=remove_readonly)
cmd = "git clone %s" % example['github']
for i in range(0, retry):
logging.info("Executing command '%s'..." % cmd)
if not subprocess.call(cmd, shell=True):
break
else:
logging.error("unable to clone the repo '%s'" % name)
return 1
return 0
def deploy_repos(config, examples):
def deploy_repos(config, exp_filter):
""" If the example directory exists as provided by the json config file,
pull in the examples dependencies by using `mbed-cli deploy`.
Args:
@ -231,39 +230,24 @@ def deploy_repos(config, examples):
"""
print("\nDeploying example repos....\n")
for example in config['examples']:
for repo_info in get_repo_list(example):
name = basename(repo_info['repo'].strip('/'))
if name in examples:
if os.path.exists(name):
os.chdir(name)
result = subprocess.call("mbed-cli deploy", shell=True)
os.chdir("..")
if result:
print("mbed-cli deploy command failed for '%s'" % name)
return result
else:
print("'%s' example directory doesn't exist. Skipping..." % name)
return 1
name = example['name']
if name in exp_filter:
if os.path.exists(name):
os.chdir(name)
logging.info("In folder '%s'" % name)
cmd = "mbed-cli deploy"
logging.info("Executing command '%s'..." % cmd)
result = subprocess.call(cmd, shell=True)
os.chdir(CWD)
if result:
logging.error("mbed-cli deploy command failed for '%s'" % name)
return result
else:
logging.info("'%s' example directory doesn't exist. Skipping..." % name)
return 1
return 0
def get_num_failures(results, export=False):
""" Returns the number of failed compilations from the results summary
Args:
results - results summary of the compilation stage. See compile_repos() for
details of the format.
num_failures
"""
num_failures = 0
for key, val in list(results.items()):
num_failures = num_failures + len(val[3])
if export:
num_failures += len(val[4])
return num_failures
def export_repos(config, ides, targets, examples):
def export_repos(config, ides, targets, exp_filter):
"""Exports and builds combinations of example programs, targets and IDEs.
The results are returned in a [key: value] dictionary format:
@ -288,12 +272,10 @@ def export_repos(config, ides, targets, examples):
ides - List of IDES to export to
"""
results = {}
valid_examples = set(examples)
print("\nExporting example repos....\n")
for example in config['examples']:
example_names = [basename(x['repo']) for x in get_repo_list(example)]
common_examples = valid_examples.intersection(set(example_names))
if not common_examples:
if example['name'] not in exp_filter:
continue
export_failures = []
build_failures = []
@ -302,41 +284,39 @@ def export_repos(config, ides, targets, examples):
exported = True
pass_status = True
if example['export']:
for repo_info in get_repo_list(example):
example_project_name = basename(repo_info['repo'])
os.chdir(example_project_name)
for name in get_sub_examples_list(example):
os.chdir(name)
logging.info("In folder '%s'" % name)
# Check that the target, IDE, and features combinations are valid and return a
# list of valid combinations to work through
for target, ide in target_cross_ide(valid_choices(example['targets'], targets),
valid_choices(example['exporters'], ides),
example['features'], example['toolchains']):
example_name = "{} {} {}".format(example_project_name, target,
ide)
def status(message):
print(message + " %s" % example_name)
sys.stdout.flush()
status("Exporting")
proc = subprocess.Popen(["mbed-cli", "export", "-i", ide,
"-m", target])
example_summary = {"name" : name, "target" : target, "ide" : ide }
summary_string = "%s %s %s" % (name, target, ide)
logging.info("Exporting %s" % summary_string)
cmd = ["mbed-cli", "export", "-i", ide, "-m", target]
logging.info("Executing command '%s'..." % " ".join(cmd))
proc = subprocess.Popen(cmd)
proc.wait()
if proc.returncode:
export_failures.append(example_name)
status("FAILURE exporting")
export_failures.append(example_summary)
logging.error("FAILURE exporting %s" % summary_string)
else:
status("SUCCESS exporting")
status("Building")
logging.info("SUCCESS exporting %s" % summary_string)
logging.info("Building %s" % summary_string)
try:
if EXPORTERS[ide].build(example_project_name, cleanup=False):
status("FAILURE building")
build_failures.append(example_name)
if EXPORTERS[ide].build(name, cleanup=False):
logging.error("FAILURE building %s" % summary_string)
build_failures.append(example_summary)
else:
status("SUCCESS building")
successes.append(example_name)
logging.info("SUCCESS building %s" % summary_string)
successes.append(example_summary)
except TypeError:
successes.append(example_name)
build_skips.append(example_name)
os.chdir("..")
successes.append(example_summary)
build_skips.append(example_summary)
os.chdir(CWD)
if len(build_failures+export_failures) > 0:
pass_status= False
@ -349,7 +329,7 @@ def export_repos(config, ides, targets, examples):
return results
def compile_repos(config, toolchains, targets, profile, verbose, examples, jobs=0):
def compile_repos(config, toolchains, targets, profiles, verbose, exp_filter, jobs=0):
"""Compiles combinations of example programs, targets and compile chains.
The results are returned in a [key: value] dictionary format:
@ -370,68 +350,62 @@ def compile_repos(config, toolchains, targets, profile, verbose, examples, jobs=
targets - list of target names
profile - build profile path or name if in default place
verbose - enabling verbose
examples - List of examples to be build
exp_filter - List of exp_filter to be build
jobs - Number of compile jobs
"""
results = {}
test_json = {"builds":{}}
valid_examples = set(examples)
base_path = os.getcwd()
print("\nCompiling example repos....\n")
for example in config['examples']:
example_names = [basename(x['repo']) for x in get_repo_list(example)]
common_examples = valid_examples.intersection(set(example_names))
if not common_examples:
if example['name'] not in exp_filter:
continue
failures = []
successes = []
compiled = True
pass_status = True
if 'test' in example and example['test'] and 'baud_rate' in example and 'compare_log'in example:
test_example = True
else:
test_example = False
if example['test']:
if not ('baud_rate' in example and 'compare_log'in example):
logging.warning("'baud_rate' or 'compare_log' keys are missing from config json file")
example['test'] = False
if example['compile']:
for repo_info in get_repo_list(example):
name = basename(repo_info['repo'])
for name in get_sub_examples_list(example):
os.chdir(name)
logging.info("In folder '%s'" % name)
# Check that the target, toolchain and features combinations are valid and return a
# list of valid combinations to work through
for target, toolchain in target_cross_toolchain(valid_choices(example['targets'], targets),
valid_choices(example['toolchains'], toolchains),
example['features']):
example_summary = {"name" : name, "target" : target, "toolchain" : toolchain, "test": "UNSET"}
summary_string = "%s %s %s" % (name, target, toolchain)
logging.info("Compiling %s" % summary_string)
build_command = ["mbed-cli", "compile", "-t", toolchain, "-m", target, "-j", str(jobs)] + (['-vv'] if verbose else [])
if profile:
build_command.append("--profile")
build_command.append(profile)
print("Compiling [%s] for [%s] with toolchain [%s]\n\n> %s" % (name, target, toolchain, " ".join(build_command)))
if profiles:
for profile in profiles:
build_command.extend(["--profile", profile])
logging.info("Executing command '%s'..." % " ".join(build_command))
proc = subprocess.Popen(build_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
std_out, std_err = proc.communicate()
std_out = std_out.decode('utf-8')
std_out = std_out.decode()
std_err = std_err.decode()
print ("\n#### STDOUT ####\n%s\n#### STDERR ####\n%s\n#### End of STDOUT/STDERR ####\n" % (std_out,std_err))
if test_example:
log = example['compare_log'].pop(0)
# example['compare_log'] is a list of log file/files, which matches each examples/sub-examples from same repo.
# pop the log file out of list regardless the compilation for each example pass of fail
image = fetch_output_image(std_out)
if image:
image_info = [{"binary_type": "bootable","path": normpath(join(name,image)),"compare_log":log}]
else:
print ("Warning: could not find built image for example %s" % name)
example_summary = "{} {} {}".format(name, target, toolchain)
if proc.returncode:
failures.append(example_summary)
else:
if test_example:
test_group = "{}-{}-{}".format(target, toolchain, example['baud_rate'])
if example['test']:
log = example['compare_log'].pop(0)
# example['compare_log'] is a list of log file/files, which matches each examples/sub-examples from same repo.
# pop the log file out of list regardless the compilation for each example pass of fail
image = fetch_output_image(std_out)
if image:
image_info = [{"binary_type": "bootable","path": normpath(join(name,image)),"compare_log":log}]
test_group = "{}-{}-{}".format(target, toolchain, example['baud_rate'])
if not test_group in test_json['builds']:
test_json['builds'][test_group] = {
"platform":target ,
@ -440,16 +414,17 @@ def compile_repos(config, toolchains, targets, profile, verbose, examples, jobs=
"baud_rate": int(example['baud_rate']),
"tests":{} }
test_json['builds'][test_group]['tests'][name]={"binaries":image_info}
test_status = "TEST_ON"
example_summary["test"] = "TEST_ON"
else:
test_status = "NO_IMAGE"
logging.warning("could not find built image for example %s" % name)
example_summary["test"] = "NO_IMAGE"
else:
print("Warning: Test for %s will not be generated." % name)
print("One or more of 'test', 'baud_rate', and 'compare_log' keys are missing from the example config json file\n")
test_status = "TEST_OFF"
successes.append(example_summary + " " + test_status)
logging.warning("Test for %s will not be generated." % name)
example_summary["test"] = "TEST_OFF"
successes.append(example_summary)
os.chdir("..")
os.chdir(CWD)
# If there are any compilation failures for the example 'set' then the overall status is fail.
if len(failures) > 0:
@ -463,7 +438,7 @@ def compile_repos(config, toolchains, targets, profile, verbose, examples, jobs=
return results
def update_mbedos_version(config, tag, examples):
def update_mbedos_version(config, tag, exp_filter):
""" For each example repo identified in the config json object, update the version of
mbed-os to that specified by the supplied GitHub tag. This function assumes that each
example repo has already been cloned.
@ -473,29 +448,54 @@ def update_mbedos_version(config, tag, examples):
tag - GitHub tag corresponding to a version of mbed-os to upgrade to.
"""
print("Updating mbed-os in examples to version %s\n" % tag)
print("\nUpdating mbed-os in examples to version '%s'\n" % tag)
for example in config['examples']:
if example['name'] not in examples:
if example['name'] not in exp_filter:
continue
for repo_info in get_repo_list(example):
update_dir = basename(repo_info['repo']) + "/mbed-os"
print("\nChanging dir to %s\n" % update_dir)
for name in get_sub_examples_list(example):
update_dir = name + "/mbed-os"
os.chdir(update_dir)
logging.info("In folder '%s'" % name)
cmd = "mbed-cli update %s --clean" %tag
logging.info("Executing command '%s'..." % cmd)
result = subprocess.call(cmd, shell=True)
os.chdir("../..")
os.chdir(CWD)
if result:
return result
return 0
def symlink_mbedos(config, path, exp_filter):
""" Create a symbolic link in each example folder to given path
If a mbed-os.lib can be found in the folder, it will be removed
"""
print("\nCreating mbed-os Symbolic link to '%s'\n" % path)
for example in config['examples']:
if example['name'] not in exp_filter:
continue
for name in get_sub_examples_list(example):
os.chdir(name)
logging.info("In folder '%s'" % name)
if os.path.exists("mbed-os.lib"):
logging.info("Removing 'mbed-os.lib' in '%s'" % name)
os.remove("mbed-os.lib")
else:
logging.warning("No 'mbed-os.lib' found in '%s'" % name)
if os.path.exists("mbed-os"):
logging.warning("'mbed-os' already existed in '%s'" % name)
else:
logging.info("Creating Symbolic link '%s'->'mbed-os'" % path)
os.symlink(path, "mbed-os")
os.chdir(CWD)
return 0
def fetch_output_image(output):
"""Find the build image from the last 5 lines of a given log"""
"""Find the build image from the last 30 lines of a given log"""
lines = output.splitlines()
last_index = -6 if len(lines)>4 else (-1 - len(lines))
last_index = -31 if len(lines)>29 else (-1 - len(lines))
for index in range(-1,last_index,-1):
if lines[index].startswith("Image:"):
image = lines[index][7:]
if os.path.isfile(image):
return image
return False
return False