ARMmbed
/
mbed-os
mirror of https://github.com/ARMmbed/mbed-os.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Publish tests documentation for DNS, Emac, NetworkInterface and Wifi 

Documentation moved from internal Confluence pages.
pull/9630/head
Michal Paszta 2019-02-06 13:17:43 +02:00
parent 57d8915a89
commit ea3d70e0b1
4 changed files with 1153 additions and 55 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

326
TESTS/netsocket/README.md
View File

@ -18,14 +18,15 @@ The target for this plan is to test:
- [UDPSocket](https://github.com/ARMmbed/mbed-os/blob/master/features/netsocket/UDPSocket.h).
- [TCPSocket](https://github.com/ARMmbed/mbed-os/blob/master/features/netsocket/TCPSocket.h).
- [TLSSocket](https://github.com/ARMmbed/mbed-os/blob/master/features/netsocket/TLSSocket.h).
- [DNS](https://github.com/ARMmbed/mbed-os/blob/master/features/netsocket/DNS.h).
Reference documentation: https://os.mbed.com/docs/latest/reference/network-socket.html
See [Network Socket Documentation](https://os.mbed.com/docs/mbed-os/latest/apis/network-socket.html) for reference.
Tools to use
----------------
- Mbed OS.
- Standard Mbed OS development tools as described in https://os.mbed.com/docs/latest/tools/index.html.
- Standard Mbed OS development tools as described in [Arm Mbed tools overview](https://os.mbed.com/docs/latest/tools/index.html).
- Test server.
These test cases themselves do not require any special tooling, other than
@ -227,6 +228,20 @@ pass the test if the driver implements the feature in question.
| 60 | TLSSOCKET_SIMULTANEOUS_TEST | SHOULD |
| 61 | TLSSOCKET_ECHOTEST_BURST | SHOULD |
| 62 | TLSSOCKET_ECHOTEST_BURST_NONBLOCK | SHOULD |
| 63 | ASYNCHRONOUS_DNS | MUST |
| 64 | ASYNCHRONOUS_DNS_CACHE | MUST |
| 65 | ASYNCHRONOUS_DNS_CANCEL | MUST |
| 66 | ASYNCHRONOUS_DNS_EXTERNAL_EVENT_QUEUE | MUST |
| 67 | ASYNCHRONOUS_DNS_INVALID_HOST | MUST |
| 68 | ASYNCHRONOUS_DNS_NON_ASYNC_AND_ASYNC | MUST |
| 69 | ASYNCHRONOUS_DNS_SIMULTANEOUS_CACHE | MUST |
| 70 | ASYNCHRONOUS_DNS_SIMULTANEOUS_REPEAT | MUST |
| 71 | ASYNCHRONOUS_DNS_SIMULTANEOUS | MUST |
| 72 | ASYNCHRONOUS_DNS_TIMEOUTS | MUST |
| 73 | SYNCHRONOUS_DNS | MUST |
| 74 | SYNCHRONOUS_DNS_CACHE | MUST |
| 75 | SYNCHRONOUS_DNS_INVALID_HOST | MUST |
| 76 | SYNCHRONOUS_DNS_MULTIPLE | MUST |
@ -1654,6 +1669,311 @@ through a dedicated socket
Echo server returns data to both threads and received data matches to
send data. The additional thread isn't stopped prematurely.
Test cases for DNS class
---------------------------
### ASYNCHRONOUS_DNS
**Description:**
Verify basic functionality of asynchronous DNS. Call `NetworkInterface::gethostbyname_async()` with a valid host name and verify result.
**Preconditions:**
1. Network interface is initialised.
2. Network connection is up.
**Test steps:**
1. Call `gethostbyname_async()` with a valid host name and a callback.
2. Verify that callback is called with correct parameters.
**Expected result:**
Callback is called with NSAPI_ERROR_OK and IP address.
### ASYNCHRONOUS_DNS_SIMULTANEOUS
**Description:**
Verify that simultaneous asynchronous DNS queries work correctly. Call `NetworkInterface::gethostbyname_async()` in a row 6 times with a different host names. Wait for all requests to complete and verify result. Cache shall not contain host names used in asynchronous request.
**Preconditions:**
1. Network interface is initialised.
2. Network connection is up.
**Test steps:**
1. Call `gethostbyname_async()` in a row 6 times with a different host names. Host names shall not be found from cache.
2. Verify that last `gethostbyname_async()` operation is rejected since there is room only for 5 simultaneous operations.
3. Verify that callback is called with correct parameters 5 times.
**Expected result:**
Sixth `gethostbyname_async()` is rejected. Callback is called with NSAPI_ERROR_OK and IP address 5 times.
### ASYNCHRONOUS_DNS_SIMULTANEOUS_CACHE
**Description:**
Verify that the caching of DNS results works correctly with simultaneous asynchronous DNS queries. Call `NetworkInterface::gethostbyname_async()` in a row 6 times with a different host names. Wait for all requests to complete and verify result. Cache shall contain at least one host name used in asynchronous request. This can be achieved e.g. by running test "Asynchronous DNS simultaneous" before this test and using same host names in this run.
**Preconditions:**
1. Network interface is initialised.
2. Network connection is up.
**Test steps:**
1. Call `gethostbyname_async()` in a row 6 times with a different host names. At least one host name shall be found from cache.
2. Verify that callback is called with correct parameters 6 times.
**Expected result:**
Callback is called with NSAPI_ERROR_OK and IP address 6 times.
### ASYNCHRONOUS_DNS_CACHE
**Description:**
Verify that the caching of DNS results works correctly. Call `NetworkInterface::gethostbyname_async()` 5 times with the same host name and verify result after each request. For first request, cache shall not contain the host name. Verify that first request completes slower than the requests made after it (where the response is found from cache).
**Preconditions:**
1. Network interface is initialised.
2. Network connection is up.
**Test steps:**
1. Call `gethostbyname_async()` with a host name. For first request, host name shall not be found from cache.
2. Verify that callback is called with correct parameters.
3. Repeat the sequence 4 times using the same host name.
4. For each request, calculate how long time it takes for DNS query to complete.
**Expected result:**
Callback is called with NSAPI_ERROR_OK and IP address 5 times. First request shall complete slower than the requests made after it (where the response is found from cache).
### ASYNCHRONOUS_DNS_NON_ASYNC_AND_ASYNC
**Description:**
Verify that non-asynchronous i.e. blocking DNS queries and asynchronous i.e. non-blocking queries work at the same time. Call `NetworkInterface::gethostbyname_async()`. Right after that make 6 non-asynchronous `NetworkInterface::gethostbyname()` calls with different host names.
**Preconditions:**
1. Network interface is initialised.
2. Network connection is up.
**Test steps:**
1. Call `gethostbyname_async()` with a host name. Host name shall not be found from cache.
2. Call `gethostbyname()` 6 times with a different host names (none of the names shall be same as in step 1).
3. Verify that each `gethostbyname()` returns success.
4. Verify that the asynchronous callback is called with correct parameters.
**Expected result:**
All operations shall return NSAPI_ERROR_OK and IP address.
### ASYNCHRONOUS_DNS_CANCEL
**Description:**
Verify that asynchronous DNS query cancel works correctly. Call `NetworkInterface::gethostbyname_async()` in a row 6 times with a different host names. Cache shall contain 3 host names used in requests. This can be achieved e.g. by running test "Asynchronous DNS non-asynchronous and asynchronous" before this test and using same host names in this run. For each request that was given an unique id, call cancel. Verify that callback is not called for cancelled requests.
**Preconditions:**
1. Network interface is initialised.
2. Network connection is up.
**Test steps:**
1. Call `gethostbyname_async()` in a row 6 times with a different host names. Cache shall contain in maximum 3 host names used in requests.
2. Call `gethostbyname_async_cancel()` for each request that was given an unique id.
3. Verify that for cancelled requests, callback is not called.
4. Verify that for other request, callback is called.
**Expected result:**
Callback shall be called only for requests that were not cancelled.
### ASYNCHRONOUS_DNS_EXTERNAL_EVENT_QUEUE
**Description:**
Verify that providing an external event queue works correctly. Define a thread and an event queue running on it. Define a DNS call in callback function that uses the event queue (call_in_callback_cb_t). Enable external event queue. Call `NetworkInterface::gethostbyname_async()` in a row 6 times with a different host names.
**Preconditions:**
1. Network interface is initialised.
2. Network connection is up.
**Test steps:**
1. Define a thread and an event queue running on it.
2. Define a DNS call in callback function that uses the event queue (call_in_callback_cb_t).
3. Start thread and event queue.
4. Set DNS callback function using the `nsapi_dns_call_in_set()` call.
5. Call `gethostbyname_async()` in a row 6 times with a different host names. Host names shall not be found from cache.
6. Verify that last `gethostbyname_async()` operation is rejected since there is room only for 5 simultaneous operations.
7. Verify that callback is called with correct parameters 5 times.
8. Restore default DNS callback function using the `nsapi_dns_call_in_set()` call.
**Expected result:**
Sixth `gethostbyname_async()` is rejected. Callback is called with NSAPI_ERROR_OK and IP address 5 times.
### ASYNCHRONOUS_DNS_INVALID_HOST
**Description:**
Verify that DNS failure error is provided for invalid hosts. Call `NetworkInterface::gethostbyname_async()` in a row 6 times with a different host names. First, third and fifth host name shall be invalid.
**Preconditions:**
1. Network interface is initialised.
2. Network connection is up.
**Test steps:**
1. Call `gethostbyname_async()` in a row 6 times with a different host names. Host names shall not be found from cache. First, third and fifth host name shall be invalid.
2. Verify that last `gethostbyname_async()` operation is rejected since there is room only for 5 simultaneous operations.
3. Verify that callback is called with correct parameters 5 times.
**Expected result:**
Sixth `gethostbyname_async()` is rejected. Callback is called with NSAPI_ERROR_DNS_FAILURE for first, third and fifth host name. Callback is called with NSAPI_ERROR_OK and IP address for second and fourth host name.
### ASYNCHRONOUS_DNS_TIMEOUTS
**Description:**
Test DNS timeouts using an external event queue that is modified to timeout the events faster that standard event queue. In this test event queue shall not delay events, instead it handles those immediately. Call `NetworkInterface::gethostbyname_async()` in a row 6 times with a different host names. All or some of the request shall timeout and timeout return value is returned.
**Preconditions:**
1. Network interface is initialised.
2. Network connection is up.
**Test steps:**
1. Define a thread and an event queue running on it.
2. Define a DNS call in callback function that uses the event queue (call_in_callback_cb_t). Callback function shall not delay callbacks; instead it shall handle those immediately.
3. Start thread and event queue.
4. Set DNS callback function using the `nsapi_dns_call_in_set()` call.
5. Call `gethostbyname_async()` in a row 6 times with a different host names. Host names shall not be found from cache.
6. Verify that last `gethostbyname_async()` operation is rejected since there is room only for 5 simultaneous operations.
7. Verify that callback is called with correct parameters 5 times.
**Expected result:**
Sixth `gethostbyname_async()` is rejected. At least for one operation, callback is called with NSAPI_ERROR_TIMEOUT value.
### ASYNCHRONOUS_DNS_SIMULTANEOUS_REPEAT
**Description:**
Verify that simultaneous asynchronous DNS queries work correctly when repeated in sequence. Call `NetworkInterface::gethostbyname_async()` in a row 5 times with a different host names. Wait for all requests to complete and verify result. Repeat the procedure 100 times.
**Preconditions:**
1. Network interface is initialised.
2. Network connection is up.
**Test steps:**
1. Call `gethostbyname_async()` in a row 5 times with a different host names.
2. Verify that callback is called with correct parameters 5 times for first operation.
3. Repeat 100 times steps 1 to 2.
**Expected result:**
Callback is called with NSAPI_ERROR_OK and IP address 5 times for every repeat.
### SYNCHRONOUS_DNS
**Description:**
Verify basic functionality of synchronous DNS. Call `NetworkInterface::gethostbyname()` with a valid host name and verify result.
**Preconditions:**
1. Network interface is initialised.
2. Network connection is up.
**Test steps:**
1. Call `gethostbyname()` with a valid host name.
2. Verify that address was resolved and return value was valid.
**Expected result:**
Return value is NSAPI_ERROR_OK and IP address is obtained from the function call.
### SYNCHRONOUS_DNS_MULTIPLE
**Description:**
Verify basic functionality of synchronous DNS. Call `NetworkInterface::gethostbyname()` with a list of 6 host names and verify result.
**Preconditions:**
1. Network interface is initialised.
2. Network connection is up.
**Test steps:**
1. Call `gethostbyname()` with a list of 6 host names
2. Verify that each of the addresses was resolved and return value was valid.
**Expected result:**
Return value is NSAPI_ERROR_OK and IP addresses are obtained from the function call.
### SYNCHRONOUS_DNS_CACHE
**Description:**
Verify that the caching of DNS results works correctly. Call `NetworkInterface::gethostbyname()` 5 times with the same host name and verify result after each request. For first request, cache shall not contain the host name. Verify that first request completes slower than the requests made after it (where the response is found from cache).
**Preconditions:**
1. Network interface is initialised.
2. Network connection is up.
**Test steps:**
1. Call `gethostbyname()` with a host name. For the first request, host name shall not be found from cache.
2. Verify that address was resolved and return value was valid.
3. Repeat the sequence 4 times using the same host name.
4. For each request, calculate how long time it takes for DNS query to complete.
**Expected result:**
Return value is NSAPI_ERROR_OK and IP address is obtained from the function call 5 times. First request shall complete slower than the requests made after it (where the response is found from cache).
### SYNCHRONOUS_DNS_INVALID_HOST
**Description:**
Verify that DNS failure error is provided for invalid hosts. Call `NetworkInterface::gethostbyname()` in a row 6 times with a different host names. First, third and fifth host name shall be invalid.
**Preconditions:**
1. Network interface is initialised.
2. Network connection is up.
**Test steps:**
1. Call `gethostbyname()` in a row 6 times with a different host names. Host names shall not be found from cache. First, third and fifth host name shall be invalid.
2. Verify that return value was valid and for valid hostnames the address was resolved 6 times.
**Expected result:**
Return value is NSAPI_ERROR_DNS_FAILURE for first, third and fifth host name. Return value is NSAPI_ERROR_OK and IP address is obtained for second and fourth host name.
Subset for driver test
----------------------
@ -1677,4 +1997,4 @@ Subset for driver test
### For socket layer driver (AT-driven, external IP stack):
All Socket, UDPSocket, TCPSocket and TLSSocket testcases.
All Socket, UDPSocket, TCPSocket and TLSSocket testcases.

210
TESTS/network/emac/README.md
View File

@ -37,6 +37,26 @@ For Wi-Fi set the `json` configuration to:
For Wi-Fi you also need to configure Wi-Fi SSID and security options to the configuration file.
Test case priorities
--------------------
Please refer to the following table for priorities of test cases. Priorities
are labeled as MUST and SHOULD. MUST means this is a requirement and
therefore mandatory to pass the test. SHOULD means it is recommended to
pass the test if the driver implements the feature in question.
| | Test case | Priority |
|-----|-----------------------------------------|----------|
| 1 | EMAC initialize | MUST |
| 2 | EMAC broadcast | MUST |
| 3 | EMAC unicast | MUST |
| 4 | EMAC unicast frame length | MUST |
| 5 | EMAC unicast burst | MUST |
| 6 | EMAC unicast long | MUST |
| 7 | EMAC multicast filter | MUST |
| 8 | EMAC memory | MUST |
## Example commands
### CTP echo server
@ -107,90 +127,176 @@ To verify whether the echo server is receiving CTP Ethernet frames, enable `echo
### EMAC initialize
The test case initializes the EMAC driver and the test network stack.
**Description:**
The EMAC test environment uses the test network stack as the default stack. To enable the stack, set the `nsapi.default-stack` option in the `json` file of the test environment application to value `TEST`.
Test case initializes and connects the EMAC driver and the test network stack.
The test network stack is a bare minimum implementation and has the functionality needed to set up the network interface. The test network stack is defined in the `emac_TestNetworkStack.h` and `emac_TestNetworkStack.cpp` files. The stack uses the test memory manager for the EMAC. The test memory manager is defined in the `emac_TestMemoryManager.h` and `emac_TestMemoryManager.cpp` files. Message buffers sent to the EMAC in `link_out()` are allocated from the buffer pool of the test memory manager. The test memory manager pool allocation unit (buffer size) is 610 bytes.
**Preconditions:**
The initialization test constructs the network interface and connects to it. The test network stack and the EMAC are bound to the network interface using `get_default_instance()` calls to the stack and to the EMAC.
1. Device is ready to be connected (Ethernet cable is plugged or WIFI base station is available).
After the construction, the network interface is connected. A connect call triggers a set up call to the test network stack. The set up call triggers a call to `emac_if_init()` function in the EMAC initialization test case.
**Test steps:**
The `emac_if_init()` function of the test case configures and powers up the EMAC.
1. Constructs the network interface
The configuration steps are:
2. Connects the network interface
* Setting the test memory manager for the EMAC.
* Setting the EMAC link input and state callbacks to call the test environment input and state callback handlers.
* Reading and setting the Ethernet MAC address.
**Expected result:**
Network interface is connected.
### EMAC broadcast
1. Sends three CTP broadcast messages (100 bytes each)
2. Waits for three seconds
3. Sends three CTP broadcast messages (60 bytes each).
4. Listens for the CTP echo server responses.
5. Stores the addresses of the echo servers if replies are received.
**Description:**
The test case passes if there are no responses from the echo server, but further test cases are skipped.
Test case tests basic broadcast functionality and resolves CTP echo server MAC address.
**Preconditions:**
1. Network interface is connected.
**Test steps:**
1. Sends three CTP broadcast messages (100 bytes each)
2. Waits for three seconds
3. Sends three CTP broadcast messages (60 bytes each).
4. Listens for the CTP echo server responses.
5. Stores the addresses of the echo servers if replies are received.
**Expected result:**
Echo server replies to broadcast messages. For each sent broadcast message reply is waited for three seconds. In case reply is not received sending is repeated six times before failing the test.
### EMAC unicast
1. Sends three CTP unicast messages (100 bytes each) to the CTP echo server.
2. Verifies that all are replied.
**Description:**
Test case tests basic unicast functionality and that the CTP echo server replies to unicast messages.
**Preconditions:**
1. Network interface is connected.
**Test steps:**
1. Sends three CTP unicast messages (100 bytes each) to the CTP echo server.
2. Verifies that all are replied.
**Expected result:**
Echo server replies to unicast messages. For each sent unicast message reply is waited for three seconds. In case reply is not received sending is repeated five times before failing the test.
### EMAC unicast frame length
1. Sends CTP unicast messages with Ethernet message length from 100 bytes to the maximum defined by the MTU of the EMAC with 50 bytes increments.
2. Verifies that all are replied.
**Description:**
Test case tests Ethernet frame lengths.
**Preconditions:**
1. Network interface is connected.
**Test steps:**
1. Sends CTP unicast messages with Ethernet message length from 100 bytes to the maximum defined by the MTU of the EMAC with 50 bytes increments.
2. Verifies that all are replied.
**Expected result:**
Echo server replies to messages that are sent to it. For each sent unicast message reply is waited for 500ms. In case reply is not received sending is repeated five times before failing the test.
### EMAC unicast burst
1. Sends CTP unicast messages with Ethernet message length from 100 bytes to the maximum defined by the MTU of the EMAC with 50 bytes increments.
2. Repeats the sending 10 times.
3. Verifies that all are replied.
**Description:**
Test case tests Ethernet echoing at full speed.
**Preconditions:**
1. Network interface is connected.
**Test steps:**
1. Sends CTP unicast messages with Ethernet message length from 100 bytes to the maximum defined by the MTU of the EMAC with 50 bytes increments.
2. Repeats the sending 10 times.
3. Verifies that all are replied.
**Expected result:**
Echo server replies to messages that are sent to it. For each sent unicast message reply is waited for five seconds. In case reply is not received sending is repeated five times before failing the test.
### EMAC unicast long
1. Sends CTP unicast messages with random Ethernet message length.
2. Repeats the sending 50000 times.
3. Verifies that all are replied.
**Description:**
Test case tests Ethernet echoing at full speed for an extended time.
**Preconditions:**
1. Network interface is connected.
**Test steps:**
1. Sends CTP unicast messages with random Ethernet message length.
2. Repeats the sending 50000 times.
3. Verifies that all are replied.
**Expected result:**
Echo server replies to messages that are sent to it. For each sent unicast message reply is waited for 350ms. In case reply is not received sending is repeated five times before failing the test.
### EMAC multicast filter
Tests multicast filtering. Multicast filtering is an optional feature for the EMAC. The test does not fail if filtering is not implemented.
The multicast testing requests the CTP echo server to forward the CTP messages to a specified multicast address as the destination address.
**Description:**
Test steps:
Test case tests multicast filtering. Multicast filtering is an optional feature for the EMAC. The test does not fail if filtering is not implemented.
1. Using unicast, verify that the echo server responses are received.
2. Set the IPv6 multicast filter address and the echo server reply (forward) address to different values. Check if the echo response is filtered.
3. Set the IPv6 multicast filter address and the echo server reply address to same value. Check that the response is not filtered.
4. Set the IPv4 multicast filter address and the echo server reply address to different values. Check if the response is filtered.
5. Set the IPv4 multicast filter address and the echo server reply address to same value. Check that the response is not filtered.
6. Enable the receiving of all multicasts. Check that the response is not filtered.
**Preconditions:**
1. Network interface is connected.
**Test steps:**
1. Using unicast, verify that the echo server responses are received.
2. Set the IPv6 multicast filter address and the echo server reply (forward) address to different values. Check if the echo response is filtered.
3. Set the IPv6 multicast filter address and the echo server reply address to same value. Check that the response is not filtered.
4. Set the IPv4 multicast filter address and the echo server reply address to different values. Check if the response is filtered.
5. Set the IPv4 multicast filter address and the echo server reply address to same value. Check that the response is not filtered.
6. Enable the receiving of all multicasts. Check that the response is not filtered.
**Expected result:**
Echo server replies to are received as expected by the test step. Supporting of the filtering will affect what messages are received.
### EMAC memory
Tests memory manager out-of-memory situations. The test case configures the test memory manager to reject memory buffer allocations made by the EMAC. Memory buffer allocations are divided into output and input memory allocations:
**Description:**
* The output memory allocations are the ones made by the EMAC in the `link_out()` function called by the network stack (test case).
* The input memory allocations are other memory allocations made by the EMAC.
Test case tests memory manager out-of-memory situations. The test case configures the test memory manager to reject memory buffer allocations made by the EMAC. Memory buffer allocations are divided into output and input memory allocations:
- The output memory allocations are the ones made by the EMAC in the \`link\_out()\` function called by the network stack (test case).
- The input memory allocations are other memory allocations made by the EMAC.
Depending on the EMAC implementation, it may or may not allocate memory manager buffers in the link output function. If the memory manager buffers are not allocated, disabling the link output memory allocations in the test does not affect the functionality.
In each test step, the test case sends CTP unicast messages with Ethernet message length from 100 bytes to the maximum defined by the MTU of the EMAC with 50 bytes increments. Memory buffers sent to the EMAC in the `link_out()` function are forced to be non-aligned in this test case.
In each test step, the test case sends CTP unicast messages with Ethernet message length from 100 bytes to the maximum defined by the MTU of the EMAC with 50 bytes increments. Memory buffers sent to the EMAC in the \`link\_out()\` function are forced to be non-aligned in this test case.
Test steps:
**Preconditions:**
1. Memory buffer allocations are allowed. Verify that echo server responses are received.
2. Disable input memory buffer allocations. The echo server responses should not be received.
3. Allow memory buffer allocations. Verify that the echo server responses are received.
4. Disable output memory buffer allocations. The echo server responses may or may not be received depending on the EMAC link out implementation.
5. Allow memory buffer allocations. Verify that the echo server responses are received.
6. Disable input and output memory buffer allocations. The echo server responses should not be received.
7. Allow memory buffer allocations. Verify that the echo server responses are received.
8. Allocate memory buffers that are sent to the EMAC in link out from the heap (contiguous memory). Verify that the echo server responses are received.
1. Network interface is connected.
**Test steps:**
1. Memory buffer allocations are allowed. Verify that echo server responses are received.
2. Disable input memory buffer allocations. The echo server responses should not be received.
3. Allow memory buffer allocations. Verify that the echo server responses are received.
4. Disable output memory buffer allocations. The echo server responses may or may not be received depending on the EMAC link out implementation.
5. Allow memory buffer allocations. Verify that the echo server responses are received.
6. Disable input and output memory buffer allocations. The echo server responses should not be received.
7. Allow memory buffer allocations. Verify that the echo server responses are received.
8. Allocate memory buffers that are sent to the EMAC in link out from the heap (contiguous memory). Verify that the echo server responses are received.
**Expected result:**
Echo server replies to messages that are sent to it based on whether the driver can allocate memory for frame or not. For each sent unicast message reply is waited for 500ms. In case reply is not received sending is repeated three times. If test expects that memory should be available and sending fails three times, test is failed.

166
TESTS/network/interface/README.md Normal file
View File

@ -0,0 +1,166 @@
Network Interface test plan
========================
Target API
----------
The target for this plan is to test:
- [NetworkInterface.h](https://github.com/ARMmbed/mbed-os/blob/master/features/netsocket/NetworkInterface.h)
Tools to be used
----------------
- Mbed OS.
- Standard Mbed OS development tools as described in [Arm Mbed tools overview](https://os.mbed.com/docs/latest/tools/index.html).
Test environment
----------------
- [Greentea](https://os.mbed.com/docs/mbed-os/latest/tools/greentea-testing-applications.html)
Test case priorities
--------------------
Please refer to the following table for priorities of test cases. Priorities
are labeled as MUST and SHOULD. MUST means this is a requirement and
therefore mandatory to pass the test. SHOULD means it is recommended to
pass the test if the driver implements the feature in question.
| | Test case | Priority |
|-----|-----------------------------------------|----------|
| 1 | NETWORKINTERFACE_CONN_DISC_REPEAT | MUST |
| 2 | NETWORKINTERFACE_STATUS | MUST |
| 3 | NETWORKINTERFACE_STATUS_NONBLOCK | MUST |
| 4 | NETWORKINTERFACE_STATUS_GET | MUST |
Building test binaries
----------------------
For testing the board and driver, test against the Mbed OS
master branch to get the most recent, up-to-date test cases and drivers.
To create a build environment:
```.sh
mbed new network_interface_test
cd network_interface_test
cd mbed-os
git checkout master
cd ..
```
Now build test binaries:
```.sh
mbed test --compile -t <toolchain> -m <target> -n mbed-os-tests-network-interface
```
Running tests
-------------
When device is connected to network, or in case of wireless device near
the access point.
```.sh
mbed test -n mbed-os-tests-network-interface
```
Test cases for NetworkInterface class
-------------------------------------
### NETWORKINTERFACE_CONN_DISC_REPEAT
**Description:**
Test `NetworkInterface::connect()` and `NetworkInterface::disconnect()`
**Preconditions:**
1. Network interface is initialized.
**Test steps:**
1. Connect interface
2. Disconnect interface
3. Repeat connect and disconnect steps 1 to 2 four times
**Expected result:**
`Connect()` and `disconnect()` calls return NSAPI_ERROR_OK.
### NETWORKINTERFACE_STATUS
**Description:**
Test `NetworkInterface::attach()` and status indications.
**Preconditions:**
1. Network interface is initialized
2. Status callback is attached to network interface
3. Network interface is on blocking mode
**Test steps:**
1. Connect interface
2. Check that interface indicates status NSAPI_STATUS_CONNECTING
3. Optional: check that interface indicates status NSAPI_STATUS_LOCAL_UP (whether this is indicated depends on network interface type)
4. Check that interface indicates status NSAPI_STATUS_GLOBAL_UP
5. Disconnect interface
6. Check that interface indicates status NSAPI_STATUS_DISCONNECTED
7. Repeat connect and disconnect steps 1 to 6 four times
**Expected result:**
Callback statuses are correct. `Connect()` and `disconnect()` calls return NSAPI_ERROR_OK. Note: network interface may indicate same status several times a row depending on network interface type. This will not fail the test.
### NETWORKINTERFACE_STATUS_NONBLOCK
**Description:**
Test `NetworkInterface::attach()` and status indications on non-blocking mode.
**Preconditions:**
1. Network interface is initialized
2. Status callback is attached to network interface
3. Network interface is on non-blocking mode
**Test steps:**
1. Connect interface
2. Check that interface indicates status NSAPI_STATUS_CONNECTING
3. Optional: check that interface indicates status NSAPI_STATUS_LOCAL_UP (whether this is indicated depends on network interface type)
4. Check that interface indicates status NSAPI_STATUS_GLOBAL_UP
5. Disconnect interface
6. Check that interface indicates status NSAPI_STATUS_DISCONNECTED
7. Repeat connect and disconnect steps 1 to 6 four times
**Expected result:**
Callback statuses are correct. `Connect()` and `disconnect()` calls return NSAPI_ERROR_OK. Note: network interface may indicate same status several times a row depending on network interface type. This will not fail the test.
### NETWORKINTERFACE_STATUS_GET
**Description:**
Test `NetworkInterface::get_connection_status()`.
**Preconditions:**
1. Network interface is initialized
2. Network interface is on blocking mode
**Test steps:**
1. Check that `get_connection_status()` returns status NSAPI_STATUS_DISCONNECTED
2. Connect interface
3. Poll the `get_connection_status()` until it returns status NSAPI_STATUS_GLOBAL_UP
4. Disconnect interface
5. Check that `get_connection_status()` returns status NSAPI_STATUS_DISCONNECTED
6. Repeat connect and disconnect steps 2 to 5 four times
**Expected result:**
`Connect()` and `disconnect()` calls return NSAPI_ERROR_OK. Right status is returned by `get_connection_status()`.

506
TESTS/network/wifi/README.md Normal file
View File

@ -0,0 +1,506 @@
Wifi test plan
========================
This is a test plan for Mbed OS Wifi API.
Target API
----------
Target for this plan is to test [WiFiInterface](https://github.com/ARMmbed/mbed-os/blob/master/features/netsocket/WiFiInterface.h) class.
Tools to be used
----------------
- Mbed OS
- Secure WiFi access point
- Unsecure WiFi access point
This test plan is written in tool agnostic way and do no specify how test cases should be written, build or run.
Test environment
----------------
General test environment consist of DUTs, base stations, network connection and test server.
![Wi-Fi](../../netsocket/wifi_environment.png)
- All DUTS should be in RAAS
- Two Wifi APs required
- non-secured one. SSID later referred as `<ssid:unsecure>`
- secured one: SSID later referred as `<ssid:secure>`, password referred as `<pw:secure>`
- IPv4 and IPv6 must be routable between Wifi network IP address space and echo server.
- Firewall must not block test traffic to and from test server **echo.mbedcloudtesting.com**
- Echo Protocol, [RFC 862](https://tools.ietf.org/html/rfc862) is enabled in both TCP and UDP. Port 7.
- Discard Protocol, [RFC 863](https://tools.ietf.org/html/rfc863) is enabled in both TCP and UDP. Port 9
- Character generator protocol, [RFC 864](https://tools.ietf.org/html/rfc864) is enabled in both TCP and UDP. Port 19
- Daytime protocol, [RFC 867](https://tools.ietf.org/html/rfc867) in both TCP and UDP. Port 13.
- Time protocol, [RFC 868](https://tools.ietf.org/html/rfc868) in both TCP and UDP. Port 37.
- Channels to be used must be different for both APs. For secure on channel number is later referred as `<ch:secure>` and for unsecure on `<ch:unsecure>`
- MAC addresses of Wifi APs must be known. Later referred as `<mac:secure>` and `<mac:unsecure>`
**NOTE:** Echo server is referred here as it is a requirement for running Socket API tests. It is not directly used by test cases defined in this document.
Test case priorities
--------------------
Please refer to following table for priorities of test cases. Priorities are labelled as MUST and SHOULD. MUST means this is a requirement and therefore mandatory to pass the test. SHOULD means it is recommended to pass the test if the driver implements the feature in question.
| | Test case | | Priority |
|-----|-----------------------------------------|----------------------------|----------|
| 1 | WIFI_CONSTRUCTOR | | MUST |
| 2 | WIFI_CONNECT_NOCREDENTIALS | | MUST |
| 3 | WIFI_SET_CREDENTIAL | | MUST |
| 4 | WIFI_SET_CHANNEL | | SHOULD |
| 5 | WIFI_GET_RSSI | | SHOULD |
| 6 | WIFI_CONNECT_PARAMS_NULL | | MUST |
| 7 | WIFI_CONNECT_PARAMS_VALID_UNSECURE | | MUST |
| 8 | WIFI_CONNECT_PARAMS_VALID_SECURE | With security type: | |
| | | NSAPI_SECURITY_WEP | SHOULD |
| | | NSAPI_SECURITY_WPA | SHOULD |
| | | NSAPI_SECURITY_WPA2 | SHOULD |
| | | NSAPI_SECURITY_WPA_WPA2 | MUST |
| 9 | WIFI_CONNECT_PARAMS_CHANNEL | | SHOULD |
| 10 | WIFI_CONNECT_PARAMS_CHANNEL_FAIL | | SHOULD |
| 11 | WIFI_CONNECT | | MUST |
| 12 | WIFI_CONNECT_SECURE | With security type: | |
| | | NSAPI_SECURITY_WEP | SHOULD |
| | | NSAPI_SECURITY_WPA | SHOULD |
| | | NSAPI_SECURITY_WPA2 | SHOULD |
| | | NSAPI_SECURITY_WPA_WPA2 | MUST |
| 13 | WIFI_CONNECT_SECURE_FAIL | | MUST |
| 14 | WIFI_CONNECT_DISCONNECT_REPEAT | | MUST |
| 15 | WIFI_SCAN_NULL | | SHOULD |
| 16 | WIFI_SCAN | | SHOULD |
Building test binaries
----------------------
For testing the board and driver, test against the Mbed OS
master branch to get the most recent, up-to-date test cases and drivers.
To create a build environment:
```.sh
mbed new wifi_test
cd wifi_test
cd mbed-os
git checkout master
cd ..
```
Now build test binaries:
```.sh
mbed test --compile -t <toolchain> -m <target> -n mbed-os-tests-network-wifi
```
Running tests
-------------
Bear in mind that RAAS user and password have to be set in the shell to access it.
```.sh
mbedgt -g <target>:raas_client:ruka.mbedcloudtesting.com:8000 -n mbed-os-tests-network-wifi -V -v
```
Test cases for WifiInterface class
-------------------------------------
### WIFI_CONSTRUCTOR
**Description:**
 Test that constructor of the driver works.
**Preconditions:**
1. None
**Test steps:**
1. Construct the driver by calling the constructor.
**Expected result:**
Constructor returns.
### WIFI_CONNECT_NOCREDENTIALS
**Description:**
 Test `WiFiInterface::connect()` without parameters. Don't set parameters with `set_credentials()`
This must be a first test to run after constructing the driver. No credentials should be given for the driver before this test.
**Preconditions:**
1. Test enviroment is set up as specified in "Test Environment" chapter.
**Test steps:**
1. Initialise the driver
2. Call `WiFiInterface::connect()`
3. `disconnect()`
**Expected result:**
`connect()` call returns NSAPI_ERROR_PARAMETER or NSAPI_ERROR_NO_SSID
### WIFI_SET_CREDENTIAL
**Description:**
This test case is to test whether the driver accepts valid credentials and reject ones that are not valid.
**Preconditions:**
1. Driver class is initialised.
**Test steps:**
1. Call `set_credentials()` with various parameters described in "Expected results"
**Expected result:**
| Parameters | Password | Security | Expected return code |
|------------|---------------------------|----------------------------------|-----------------------------------------------------|
| ssid=NULL | NULL | NSAPI_SECURITY_NONE | NSAPI_ERROR_PARAMETER |
| ssid="OK" | NULL | NSAPI_SECURITY_WPA_WPA2 | NSAPI_ERROR_PARAMETER |
| ssid="OK" | NULL | NSAPI_SECURITY_WEP | NSAPI_ERROR_PARAMETER because password is missing. |
| ssid="OK" | NULL | NSAPI_SECURITY_NONE | NSAPI_ERROR_OK |
| ssid="OK" | [any 64 character string] | NSAPI_SECURITY_WPA2 | NSAPI_ERROR_PARAMETER because password is too long |
| ssid="OK" | [any 63 character string] | NSAPI_SECURITY_WPA2 | NSAPI_ERROR_OK |
| ssid="OK" | "" | NSAPI_SECURITY_WPA_WPA2 | NSAPI_ERROR_PARAMETER |
| ssid="OK" | "" | NSAPI_SECURITY_WEP | NSAPI_ERROR_PARAMETER because password is missing. |
| ssid="OK" | "" | NSAPI_SECURITY_NONE | NSAPI_ERROR_OK |
| ssid="OK" | "12345678" | NSAPI_SECURITY_WPA_WPA2 | NSAPI_ERROR_OK |
| ssid="OK" | "12345678" | NSAPI_SECURITY_WPA2 | NSAPI_ERROR_OK |
| ssid="OK" | "12345678" | NSAPI_SECURITY_WPA | NSAPI_ERROR_OK |
| ssid="OK" | "12345678" | NSAPI_SECURITY_WEP | NSAPI_ERROR_OK or NSAPI_ERROR_UNSUPPORTED |
| ssid="" | "" | NSAPI_SECURITY_NONE | NSAPI_ERROR_PARAMETER |
### WIFI_SET_CHANNEL
**Description:**
Test validity of `WiFiInterface::set_channel()`.
When proper channel number is feed, it should return NSAPI_ERROR_OK. API documentation is unclear what to report on error case, but I'm assuming NSAPI_ERROR_PARAMETER.
If the driver does not support setting channels for scan, it should report NSAPI_ERROR_UNSUPPORTED for all of the channels.
**Preconditions:**
1. Driver class is initialised
**Test steps:**
1. Call `set_channel()` with various parameters described in "Expected results"
**Expected result:**
For 2.4 Ghz chips
| channel | Expected return code |
|---------|-----------------------------------------------------------------------------------|
| 0 | NSAPI_ERROR_OK (0 = any channel) |
| 1 | NSAPI_ERROR_OK |
| 13 | NSAPI_ERROR_OK if supporting European frequencies |
| | NSAPI_ERROR_PARAMETER if configured for North America |
| 15 | NSAPI_ERROR_PARAMETER |
| | because not a valid 2.4 Ghz channel |
For 5 Ghz chips
| channel | Expected return code |
|---------|-----------------------------------------------------------------------------------|
| 3 | NSAPI_ERROR_PARAMETER |
| | because not a valid 5 Ghz channel number |
| 36 | NSAPI_ERROR_OK |
| 169 | NSAPI_ERROR_PARAMETERs |
| | should not be allowed in any country |
Drivers not supporting
| channel | Expected return code |
|---------|-----------------------------------------------------------------------------------|
| <any> | NSAPI_ERROR_UNSUPPORTED should be returned for ALL channel numbers > 0. |
### WIFI_GET_RSSI
**Description:**
 Test `WiFiInterface::get_rssi()` API. When connected, it should return valid RSSI value. When unconnected it should return 0.
API is a bit unclear whether 0 can also mean that driver does not support reporting of RSSI.
**Preconditions:**
1. Test enviroment is set up as specified in "Test Environment" chapter.
**Test steps:**
1. Initialise the driver.
2. Call `get_rssi()`, store the result.
3. Call `connect()` with valid SSID parameters. Wait for connection
4. Call `get_rssi()`
**Expected result:**
First call should return zero, second call should return negative number between -10 and -100.
### WIFI_CONNECT_PARAMS_NULL
**Description:**
Test `WiFiInterface::connect(ssid, pass, security, channel)` with NULL parameters
**Preconditions:**
1. Test enviroment is set up as specified in "Test Environment" chapter.
**Test steps:**
1. Initialise the driver
2. Call `WiFiInterface::connect(NULL, NULL)`
3. Call `WiFiInterface::connect("", "")`
**Expected result:**
Both `connect()` calls return NSAPI_ERROR_PARAMETER
### WIFI_CONNECT_PARAMS_VALID_UNSECURE
**Description:**
 Test `WiFiInterface::connect(ssid, pass, security)` with valid parameters for unsecure network
**Preconditions:**
1. Test enviroment is set up as specified in "Test Environment" chapter.
**Test steps:**
1. Initialise the driver
2. Call `WiFiInterface::connect( <ssid:unsecure>, NULL)`
3. `disconnect()`
4. Call `WiFiInterface::connect( <ssid:unsecure>, "")`
5. `disconnect()`
**Expected result:**
`connect()` calls return NSAPI_ERROR_OK
### WIFI_CONNECT_PARAMS_VALID_SECURE
**Description:**
 Test `WiFiInterface::connect(ssid, pass, security)` with valid parameters for secure network
**Preconditions:**
1. Test enviroment is set up as specified in "Test Environment" chapter.
**Test steps:**
1. Initialise the driver
2. Call `WiFiInterface::connect( <ssid:secure>, <pw:secure>, NSAPI_SECURITY_WPA2)`
3. `disconnect()`
**Expected result:**
`connect()` call returns NSAPI_ERROR_OK
 
### WIFI_CONNECT_PARAMS_CHANNEL
**Description:**
 Test `WiFiInterface::connect(ssid, pass, security, channel)` with valid parameters for secure network using channel specified.
This test only applies to devices which support selecting the channel (passes WIFI-SET-CHANNEL).
**Preconditions:**
1. Test enviroment is set up as specified in "Test Environment" chapter.
**Test steps:**
1. Initialise the driver
2. Call `WiFiInterface::connect( <ssid:secure>, <pw:secure>, NSAPI_SECURITY_WPA2, <ch:secure>)`
3. `disconnect()`
**Expected result:**
`connect()` call returns NSAPI_ERROR_OK
 
### WIFI_CONNECT_PARAMS_CHANNEL_FAIL
**Description:**
 Test `WiFiInterface::connect(ssid, pass, security, channel)` with valid parameters for secure network using channel specified. Channel specified must be wrong so that connect call should fail.
This test only applies to devices which support selecting the channel (passes WIFI-SET-CHANNEL).
**Preconditions:**
1. Test enviroment is set up as specified in "Test Environment" chapter.
**Test steps:**
1. Initialise the driver
2. Call `WiFiInterface::connect( <ssid:secure>, <pw:secure>, NSAPI_SECURITY_WPA2, <ch:unsecure>)`
3. `disconnect()`
**Expected result:**
connect() call returns NSAPI_ERROR_CONNECTION_TIMEOUT or NSAPI_ERROR_NO_CONNECTION
### WIFI_CONNECT
**Description:**
 Test `WiFiInterface::connect()` without parameters. Use `set_credentials()` for setting parameters. Checks that driver does not try to retrieve SSID from location it's not controlling.
**Preconditions:**
1. Test enviroment is set up as specified in "Test Environment" chapter.
**Test steps:**
1. Initialise the driver
2. `Call WiFiInterface::set_credentials( <ssid:unsecure>, NULL)`
3. `Call WiFiInterface::connect()`
4. `disconnect()`
5. `Call WiFiInterface::set_credentials( <ssid:unsecure>, "")`
6. `Call WiFiInterface::connect()`
7. `disconnect()`
8. trash the memory storing SSID
9. `Call WiFiInterface::set_credentials( <ssid:unsecure>, "")`
10. `Call WiFiInterface::connect()`
11. `disconnect()`
**Expected result:**
`connect()` calls return NSAPI_ERROR_OK
### WIFI_CONNECT_SECURE
**Description:**
 Test `WiFiInterface::connect()` without parameters. Use secure settings for `set_credentials`.
**Preconditions:**
1. Test enviroment is set up as specified in "Test Environment" chapter.
**Test steps:**
1. Initialise the driver
2. Call `WiFiInterface::set_credentials( <ssid:secure>, <pw:secure>, NSAPI_SECURITY_WPA2)`
3. Call `WiFiInterface::connect()`
4. `disconnect()`
**Expected result:**
`connect()` call returns NSAPI_ERROR_OK
### WIFI_CONNECT_SECURE_FAIL
**Description:**
 Test `WiFiInterface::connect()` failing with wrong password.
**Preconditions:**
1. Test enviroment is set up as specified in "Test Environment" chapter.
**Test steps:**
1. Initialise the driver
2. Call `WiFiInterface::set_credentials( <ssid:secure>, <any 8 character string, not pw:secure>, NSAPI_SECURITY_WPA2)`
3. Call `WiFiInterface::connect()`
4. disconnect()
**Expected result:**
`connect()` call returns NSAPI_ERROR_AUTH_FAILUR, NSAPI_ERROR_CONNECTION_TIMEOUT or NSAPI_ERROR_NO_CONNECTION
### WIFI_CONNECT_DISCONNECT_REPEAT
**Description:**
 Test `WiFiInterface::connect()` - `disconnect()` repeatition works.
**Preconditions:**
1. Test enviroment is set up as specified in "Test Environment" chapter.
**Test steps:**
1. Initialise the driver
2. Call `WiFiInterface::set_credentials( <ssid:unsecure>, NULL)`
3. Repeat 10 times
1. Call `WiFiInterface::connect()`
2. disconnect()
**Expected result:**
Each `connect()` call returns NSAPI_ERROR_OK
Each `disconnect()` returns NSAPI_ERROR_OK
### WIFI_SCAN_NULL
**Description:**
 Call `WiFiInterface::scan()` with null parameters to get number of networks available.
**Preconditions:**
1. Test enviroment is set up as specified in "Test Environment" chapter.
**Test steps:**
1. Initialise the driver
2. Call `WiFiInterface::scan(NULL, 0)`;
**Expected result:**
`scan()` returns positive number that is higher or equivalent of two. (>=2).
### WIFI_SCAN
**Description:**
 Call `WiFiInterface::scan()` with valid accesspoint list allocated
**Preconditions:**
1. Test enviroment is set up as specified in "Test Environment" chapter.
2. Test environment must contain less than 10 WiFi SSID within the listening range of DUT, otherwise adjust the value used in step 2.
**Test steps:**
1. Initialise the driver
2. Allocate array of 10 WiFiAccessPoint objects.
3. Call `WiFiInterface::scan(<array>, 10)`;
**Expected result:**
`scan()` returns positive number that is higher or equivalent of two. (>=2).
Scan elements from array indexes between zero and the number returned.
- Call `get_ssid()` for each element. `<ssid:secure>` and `<ssid:unsecure>` must be found.
- Call `get_rssid()` for each element. Value must be between -10 and -100.
For those known networks, test that following return values are found:
| get_ssid() | get_bssid() | get_security() | get_channel() |
|-------------------------|------------------------|-----------------------|-----------------------|
| `<ssid:unsecure>` | `<mac:unsecure>` | NSAPI_SECURITY_NONE | `<ch:unsecure>` |
| `<ssid:secure>` | `<mac:secure>` | NSAPI_SECURITY_WPA2 | `<ch:secure>` |