mbed-os/tools/test/examples

Examples testing script

The scripts in this folder are used for testing mbed-os official examples. It contains the following files:

• examples.py - the main script that serves as the command-line interface.
• examples.json - the default configuration file for the script, which contains the information to the examples. If required, you can pass a customized configuration file as an argument to the script.
• examples_lib.py - the library file, which contains the main function of the testing scripts.

The scripts

examples.py provides command-line interface and subcommands that makes easier to test examples. Included subcommands are:

• import - imports each of the repos and its dependencies (.lib files) associated with the specific examples name from the .json configuration file. If there is already a clone of the repo, it will first be removed to ensure a clean, up-to-date cloning.

• clone - clones each of the repos associated with the specific examples name from the .json configuration file. If there is already a clone of the repo, it will first be removed to ensure a clean, up-to-date cloning.

• deploy - if the example directory exists as provided by the .json configuration file, pulls in the examples dependencies by using mbed-cli deploy.

• update - for each example repo identified in the config .json object, updates the version of mbed-os to that specified by the supplied GitHub tag. This function assumes that each example repo has already been cloned.

• compile - compiles combinations of example programs, targets and compile chains.

• export - exports and builds combinations of example programs, targets and IDEs.

• list - displays examples in a configuration file in a table.

• symlink - creates a symbolic link to a given mbed-os PATH.

For more detailed options, please use -h or --help.

The configuration file

Here is the section of default configuration file:

{
 "examples": [
   {
     "name": "mbed-os-example-blinky",
     "github": "https://github.com/ARMmbed/mbed-os-example-blinky",
     "sub-repo-example": false,
     "subs": [],
     "features" : [],
     "targets" : [],
     "toolchains" : [],
     "exporters": [],
     "compile" : true,
     "export": true,
     "test" : true,
     "baud_rate": 9600,
     "compare_log": ["mbed-os-example-blinky/tests/blinky.log"],
     "auto-update" : true
   },
     
   ...
   
   {
     "name": "mbed-os-example-tls",
     "github": "https://github.com/ARMmbed/mbed-os-example-tls",
     "sub-repo-example": true,
     "subs": [
         "benchmark",
         "tls-client",
         "hashing",
         "authcrypt"
     ],
     "features" : [],
     "targets" : ["K66F", "NUCLEO_F429ZI"],
     "toolchains" : ["GCC_ARM", "ARM"],
     "exporters": [],
     "compile" : false,
     "export": false,
     "test" : false,
     "baud_rate": 9600,
     "compare_log": [
         "mbed-os-example-tls/tests/benchmark.log",
         "mbed-os-example-tls/tests/tls-client.log",
         "mbed-os-example-tls/tests/hashing.log",
         "mbed-os-example-tls/tests/authcrypt.log"
     ],
     "auto-update" : true
   }
 ]
}

Fields

• name - name of the example. It should be the base name of the example repository address and will throw if it doesn't match.
• github - example GitHub repository address.
• sub-repo-example - specifies if the example repository has a subfolder for each example.
• subs - array of subexample names.
• features - the features that must be in the features array of a target in targets.json.
• baud_rate - example default baud rate.
• compare_log - array of log compared to command-line output during testing. If the example has many subexamples, the order of log should match the order of subexamples.
• targets - list of mbed-os development boards that run this example.
• targets - list of targeted development boards.
• toolchains - toolchain to use for compiling.
• exporters - allowed exporters.
• compile - enables compiling.
• export - enables exporting.
• test - enables testing.

Values

[ ] means all possible alternatives.

Typical use

In the Mbed OS CI, we follow the below steps to compile and test Mbed OS examples.

1. Clone mbed-os repository to the current folder:

   git clone https://github.com/ARMmbed/mbed-os.git
   

2. Clone the examples repo to the current folder. Users can pass an -e option to the script to filter out the rest of the examples, so the scripts only run on one particular example:

   python mbed-os/tools/test/examples/examples.py clone
   

3. Create a symbolic link to mbed-os for every example. This step lets all the examples share a single mbed-os folder, rather than checking out the mbed-os folder many times. We highly recommend you pass an absolute path as the argument:

   python mbed-os/tools/test/examples/examples.py symlink $PWD/mbed-os
   

4. Deploy other dependency libraries:

   python mbed-os/tools/test/examples/examples.py deploy
   

5. Compile the test for the examples on a specific target:

   python mbed-os/tools/test/examples/examples.py compile -m <target>
   

After the compile test finished, the scripts print the result table:

Passed example compilation:
+---------------------------------+--------+-----------+----------+--------------+
| EXAMPLE NAME                    | TARGET | TOOLCHAIN | TEST GEN | BUILD RESULT |
+---------------------------------+--------+-----------+----------+--------------+
| mbed-os-example-kvstore         |  K64F  |  GCC_ARM  | TEST_ON  |    PASSED    |
| mbed-os-example-tls-socket      |  K64F  |  GCC_ARM  | TEST_ON  |    PASSED    |
| mbed-os-example-blockdevice     |  K64F  |  GCC_ARM  | TEST_ON  |    PASSED    |
| mbed-os-example-wifi            |  K64F  |  GCC_ARM  | TEST_OFF |    PASSED    |
| mbed-os-example-error-handling  |  K64F  |  GCC_ARM  | TEST_ON  |    PASSED    |
| mbed-os-example-sd-driver       |  K64F  |  GCC_ARM  | TEST_ON  |    PASSED    |
| mbed-os-example-crash-reporting |  K64F  |  GCC_ARM  | TEST_ON  |    PASSED    |
| mbed-os-example-filesystem      |  K64F  |  GCC_ARM  | TEST_ON  |    PASSED    |
| mbed-os-example-blinky          |  K64F  |  GCC_ARM  | TEST_ON  |    PASSED    |
| mbed-os-example-bootloader      |  K64F  |  GCC_ARM  | TEST_OFF |    PASSED    |
| mbed-os-example-cpu-stats       |  K64F  |  GCC_ARM  | TEST_ON  |    PASSED    |
| mbed-os-example-sys-info        |  K64F  |  GCC_ARM  | TEST_ON  |    PASSED    |
| mbed-os-example-attestation     |  K64F  |  GCC_ARM  | TEST_ON  |    PASSED    |
+---------------------------------+--------+-----------+----------+--------------+
Number of failures = 0

After the compilation stage, a test_spec.json file is generated. Later, Greentea tests will consume this file. They test the compiled example on hardware platform.