There is no problem with this test during the morph, but some issue has been noticed while testing new Jenkins CI in Oulu on NRF52_DK.
I was able to reproduce the issue locally. The difference between morph and local run is that CPU statistics are enabled on morph. This makes the difference and test passes.
The sleep test case perform sleep for 100 us, 200 us, ... ,1000 us in loop (us ticker wakes the board) and verifies if sleep time matches the assumption.
I got the following results:
sleep wake-up after
100 us ~100 us ok
200 us ~200 us ok
300 us ~300 us ok
400 us ~400 us ok
500 us ~14 us (??)
When requested sleep time is equal to 500 us some unexpected interrupt occurs which wakeup the board and force the test to fail.
Register state just after exit from sleep:
Control and State Register: 0x00400000 (ISRPENDING - Interrupt pending flag is set).
NVIC Interrupt Set-pending Register[0]: 0x00000004 (UARTE0_UART0_IRQn) or 0x00000200 (TIMER1_IRQn - timer used by us ticker).
UART interrupt is generated because of green-tea transmission. We know that it is performed while test is executed since we need to wait before going into deep-sleep since otherwise the transmission will be broken. So to take care of UART interrupt we need to wait before sleep test in the same way like it is done in deep-sleep test.
Test case was assuming that secure and unsecure SSID were on different
channels.
This is not a requirement and it should be OK to run on same channel.
Fixed the testcase by using +1 on channel number to get a wrong channel.
2.4GHz and 5GHz channels might be using the same SSID. Wifi scan
might also fail occasionally to find secure- and unsecure channels
on same scan so lets not assume that we'll find both.
Add a test to ensure that devices have at least 2K free ram
and 2K free heap. This test should be the first test that fails
due to running out of ram or heap.
Change the base time value to more realistic:
START_TIME = 1537789823 # GMT: Monday, 24 September 2018 11:50:23
This fix has been proposed by STM in order to enhance test efficiency.
Current test version did not detect problem with RTC reset on F1 family boards since the base time was too small.
- code refactoring and preparation for enabling DPI/QPI tests
- reduced multiple test count to 4
- use common flash config header for all MX25RXX35F chips
- fix sector erase max time on N25Q128A
Remove the double free RTOS tests since this was never defined
behavior of CMSIS-RTOS. This allows testing to pass.
The RTX commit which caused this test to start failing is:
c3b123ef4256f65537e2597af475fc20ec9a383e
RTX5: updated MemoryPoolFree (removed count check) [SDCMSIS-801]
Note - Double freeing an element from a memory pool was never safe.
The error return value when double freeing was misleading since memory
corruption may still be occurring in that case. For more information
on this see SDCMSIS-801.
Increase timeout for EMAC tests. Old timeout (600 seconds) was not
enough for slower devices to complete the tests.
Check that connection was successful when running test case
emac_test_initialize.
Main idea of introduced changes is to ease adding support for new flash chips
Major changes:
- move implementation of all memory chip specific functions to memory config file (no weak functions)
- add support for 1-2-2 write
To handle timer rollovers the test tests-mbed_hal-common_tickers_freq
calls intf->set_interrupt(0). For this to work correctly the ticker
implementation must fire an interrupt on every rollover event though
intf->set_interrupt(0) was called only once. Whether an interrupt will
fire only once or multiple times is undefined behavior which
cannot be relied upon.
To avoid this undefined behavior this patch continually schedules an
interrupt and performs overflow detection on every read. This also
removes the possibility of race conditions due to overflowCounter
incrementing at the wrong time.
The thread stack size was restricted due to some boards have really limited RAM sizes,
and out of heap memory on multiple threads tests.
The side effect was on the debug profile build, the tests will get stack overflow.
We need the build the test with debug profile in order to do the code coverage analysis.
So increased the thread stack size on FastModel targets.
FastModels targets are simulator running on the x86 hosts.
As the nature of non-RealTime x86 OS and FastModels, timing accuracy is not guaranteed
So skipping the time drifting tests on FastModel targets
When the define LPTICKER_DELAY_TICKS is set deep sleep can be randomly
disallowed when using the low power ticker. This is because a Timer
object, which locks deep sleep, is used to protect from back-to-back
writes to lp tickers which can't support that. This causes tests which
assert that deep sleep is allowed to intermittently fail.
To fix this intermittent failure this patch adds the function
sleep_manager_can_deep_sleep_test_check() which checks if deep sleep
is allowed over a duration. It updates all the tests to use
sleep_manager_can_deep_sleep_test_check() rather
than sleep_manager_can_deep_sleep() so the tests work even if deep
sleep is spuriously blocked.
I'm submitting a test plan for Mbed OS sockets, so that we have
reference that describes wanted behaviour, required test environment,
and priorities of test cases.
This servers as a both documentation and specification. It needs to live
with source code, as opposed to Handbook, because it will be extend, modified
and fixed for faster cycle than what should be done for Handbook.
Part of this document (How to run -section) will be submitted to Handbook
with links to this.
Test is giving problems in some networks. Could be that DNS server
is regarding repeated DNS enquiries on the test to be caused by
malfunctioning device.
This is solution to issue #7724.
Ticker free tests have been removed since ticker free function has been implemented only for CI boards.
Implementation for the remaining platforms will be done on feature branch feature-hal-ticker-free.
RTX waits, such as sem.wait(N), delays for between N-1 and N cycles.
As lp_ticker is not so accurate as us_ticker,
sometines, wait is expiring just before the lp_timeout.
In `ticker speed` test case execution time of ticker API functions is measured using Timer object. Test replaces original ticker handler for testing purposes, so test should not relay on higher level ticker based features(like Timer).
Use low level ticker API for time measuring.
DEVICE_EMAC is defined only for boards with default Ethernet emac.
It is not defined for example for Realtek RTL8195AM board that
does not have Ethernet. Removed the check for device emac from emac
greentea tests.
Made to prevent timeout if a single test case fails. The goal is that
each test case might wait only half of the remaining time reserved for
running TCP test cases.
In mbed-os-tests-mbed_hal-common_tickers/Microsecond ticker overflow test, some targets
would fail to catch specified ticker value near overflow in time and so fail. This commit
alleviates the issue by checking ticker value range rather than one exact ticker value near
overflow.
Serial buffer must be flushed before entering deep sleep mode. In the test this is done by the additional delay which is implemented on the busy loop which decrements given value down to 0 (`void wait_cycles(volatile unsigned int cycles)`). This solution is not appropriate since it is very target specific and the cycles value has been already increased few times. Additionally very big number of loop cycles which is suitable for fast targets may take much longer on slower boards and results in test timeout.
It has been verified that 20ms is sufficient delay for the green-tea transmission. In this test we cannot simply use `wait_ms(20)` since this potentially may put board to sleep and wake up using lp ticker. The test re-initialzies the lp ticker(disables ticker interrupt) and this operation may break the schedule and time tracing by the upper layer. But we can use us ticker which is not affected by this test. The solution is to add a delay routine based on busy loop and us ticker only. This way are able to wait exactly 20 ms.
The test sometimes fails on NRF51_DK (test case: "Test: LowPowerTimer - time measurement 1 ms.") in morph tests.
The test verifies if LowPowerTimer class correctly counts elapsed time. Sometimes we got measured ~1600 us for delay 1000 us (delta 550 us).
The delay is performed using `wait_us()` function which for delays greater than or equal to 1 ms (our case) calls `Thread::wait((uint32_t)ms);`. This causes rescheduling and potentially can put board into sleep (deep sleep mode is disabled by `wait_us()`). For our test purposes we don't need rescheduling/sleep since this actions takes extra time and have influence on the time measurement accuracy.
The solution is to implement function for delay which is based on busy loop and uses us ticker. It has been verified that this solves the problem. With this fix when measurement of 1 ms is repeated 1000 times we got usually measured time equal to ~1080 us, and sometimes ~1300us (checked that this is caused by systick interrupt handling). Since this is test for drivers layer and the results are acceptable I decided to not disabling systick in the test).
As we do not include rtx_lib header file anymore, these symbols are not available.
Use core util for if ISR is active. And for OS tick, pull in os_tick header file.
Mark mbed-os-tests-mbed_hal-common_tickers/time_cpu_cycles not in-lined.
Otherwise, time_cpu_cycles may be in-lined somewhere and takes different
CPU cycles on invocation.
Increase timeout for netsocket TCP and UDP tests. Old timeout
(240 seconds) was not enough for slower devices to complete the
tests.
Accept error codes NSAPI_ERROR_CONNECTION_TIMEOUT and
NSAPI_ERROR_NO_CONNECTION when running test wifi_connect_secure_fail
The new configuration make Error history tracking switched off by default and enabled by using the config flag MBED_CONF_PLATFORM_ERROR_HIST_ENABLED.
Config flag MBED_CONF_PLATFORM_ERROR_ALL_THREADS_INFO enables printing info of all threads. This will be turned off by default.
Increases tolerance value for sleep_usticker_test to cover extra time needed
for cpu stats computation (for more details see MBED_CPU_STATS_ENABLED).
Prevent scheduling interrupt during ticker initialization (in lp_ticker_init)
while test execution.
Make multiticker test more reliable when scheduling very early interrupts.
When very early interrupt is scheduled (e.g now + 2[ticks]) then it's likely
that it won't be fired in some circumstances and as a result there is overdue
event in queue. That overdue event can be mistakenly scheduled again by
`Ticker::detach` (detach calls schedule if head was removed). That's why we
should check interrupts counter immediately after wait period and before detach loop
Updated EMAC memory manager to use libservice nsdynmemlib
for EMAC memory buffers. Located the nsdynmemlib buffer heap
to DMA safe memory bank on LPCxx boards. Optimized placement of
static variables on EMAC test environment for LPCxx boards to
maximize available memory.
In case when base tick count is different than next tick count check first if the difference is equal to 1 tick (this is what we are looking for) if not then decrease the tick count.
Repeat counting process few times before incrementing the number of cycles in case when base tick count is equal to next tick count.
RTC reset test was failing when board has been just powered and RTC reset test was executed for the first time (issue has been detected on CI). In such case RTC initialization takes more time than in futher rtc_init calls. This has impact on green-tea communication. Commands send by host immediately after init command (write, read) were not handled on the device side and no response to the host was provided.
To fix this problem test communication flow has been modified. Device sends ack to the host after RTC init is done. Host sends further RTC commands when ack from the device has been received.
Edit:
There are still some communication issues on the CI. Add ack from the device after each executed command. Increase test timeout and wait after reset.
Since some platforms use RTC for low power timer RTC may be already enabled.
Because of that we will only verify if rtc_isenabled() returns 1 in case when init is done and RTC time is set.
Test inserts event into the TimerEvent object which is scheduled 50 ms in the future. Then test thread hangs on the semaphore for 51 ms and after that time we expect that event handler has been executed. This test fails sometimes on NRF51_DK, NRF52_DK since different clocks are used for event handling and delay. TimerEvent class uses fast us ticker and semaphores are based on lp ticker (NRF51_DK) or System Tick Timer (NRF52_DK).
We assume that ticker measurement error is 10% and our 1 [ms] extra corresponds to 2% error. I suggest to increase delay to 2 [ms] (4%). This should be enough for all boards at this moment.
It is possible that the difference between base and next tick count on some platforms is greater than 1, in this case we need to repeat counting with the reduced number of cycles (for slower boards).
If number of cycles has been reduced than base tick count needs to be redefined. This operation is missing and is added by this patch.
It has been noticed on NRF51_DK board that occurrence of system tick have impact on test execution.
NRF51_DK is a slow board with fast us ticker and handling of system tick interrupt takes about 250 us ticker ticks which have huge influence on interrupt tests where interrupt is set to <current count> + 100, 200, 500 ticks.
This test fails sometimes while testing 1 ms timeout on NRF51_DK board.
This board is very slow and lp ticker is inaccurate.
I noticed that measured results are very close to the tolerance value. For 1ms delay we measure in failure case 1560 us and tolerance is set to 550.
Since measurement method is also significantly flawed (extra time for setting up ticker and execution of a ticker irq handler) I recommend to increase tolerance by least +100 us for now.
Since now us ticker is disabled in deep-sleep mode and tests-mbed_drivers-timerevent verifies TimerEvent class which uses us ticker then deep-sleep mode needs to be disabled during the test execution.
On some boards without SYSTICK support lp ticker is used to trace system ticks.
Event queue uses system ticker to count delay when callback is to be fired.
New ticker implementation for NRF51_DK provides fast and accurate counter for us ticker which shows inaccuracy of the system ticker(which is based on slow and inaccurate lp ticker). This is why measured error grows linearly and percentage form needs to be used.
Currently we use constant tolerance value equal to: 5 [ms].
This patch modifies tolerance to the following form: 5% of measured time [ms].
In this test the lowest delay is equal to 100 [ms], so 5% corresponds to 5 [ms] tolerance - original value. This means that min tolerance is 5 [ms] so this change should not have negative impact on other targets.
For NR51_DK US_TICKER_OV_LIMIT needs to be increased since if test is run few times in row sometimes fails. This is because NR51_DK is a slow board (16 MHz) with fast and short us ticker counter 1 MHz/16 bits.
Since this change concerns new ticker driver for NRF51_DK it goas on ticker feature branch.
On this branch us ticker is based on 1 MHz higher precision counter (on master is based on 32kHz RTC). As a result of using higher precision counter measurement error increased:
:485::FAIL: Expected 1.060000 Was 1.070478
For delay 1060 ms we measured ~1070 ms so we have about ~1% measurement error which is not bed for this board.
To make it work with 1MHz us ticker increase test tolerance. Define tolerance as follows:
tolerance = 500us + 0.02 * delay.
Since this change concerns new ticker driver for NRF51_DK it goas on ticker feature branch.
On master NRF51_DK Ticker driver uses 32kHz RTC for lp and us ticker. Test uses us ticker to measure lp ticker interrupt scheduling (LowPowerTicker.attach), but since the same hardware is used for both tickers the measurement error is constant. On this branch us ticker uses 1 MHz higher precision clock and the results are different - measurement error grows linearly and shows inaccuracy of the RTC.
Test implements constant delta for measured time equal to 2000 us.
Change delta so it depends on lp ticker tested timeout value:
500us for measurement inaccuracy + 5% tolerance for LowPowerTicker
Update the ticker common test to clean up after each case by restoring
the ticker IRQ handler. This allows tickers to function normally after
each case has completed.
- count_ticks: fix counter overflow handling,
- count_ticks: use reg_cycles variable in while loop,
- increment test: reduce number of cycles for slow cores if measure process needs to be repeated (difference is greater than 1).
On slow targets with fast high frequency tickers like NRF51_DK (16 Mhz CPU/1MHz ticker) time window for test case execution without overflow needs to be increased. Add parameter to `overflow_protect` function to be able to set different time window without overflow for us ticker and lp ticker.
Increment test proves that ticker counter is incremented by one.
This is done indirectly for high frequency counters where it is impossible to read 'value' and 'value + 1' in two successive ticker reads.
This check is done indirectly by counting ticker ticks elapsed during execution of N cycles of empty while loop. Unfortunately on slow boards with fast tickers like NRF51_DK(16 MHz CPU/1MHz hf ticker) it is possible that for the same N cycles measured number of elapsed ticks in two successive calls is greater than 1. This patch provides fix for such case - measure operation is repeated with the same number of cycles.
Since target specific ticker drivers are not ready also features which uses ticker in upper layers may not work correctly and tests for these features.
We need to disable also failing higher level ticker related tests (by adding check if DEVICE_USTICKER symbol is available and raise #error [NOT_SUPPORTED] if not).
Note:
tests-mbed_drivers-rtc is new and uses us ticker to perform a delay.
count_ticks() function counts ticker ticks elapsed during execution of N cycles of empty while loop.
In current version N value (cycles) is given as volatile paramater in order to disable possible compiler optimalisation.
There was a problem with measured time on different compilers and additionally results on ARM compiler were unexpected (the difference beetween measured elapsed ticks for the same number of cycles was to large). This might be caused by the memory access in order to store updated variable in memory. To fix this issue given numer of cycles has been stored into register and register is decremented (no memory access).
With this fix count_ticks(NUM_OF_CYCLES, 1) call returns 2500 +/-1 for us ticker ticks using each compiler (K64F).
On some platforms (e.g. K64F) different counters are used for time measurement and interrupt generation.
Because of that we should relax interrupt test case and give additional time before checking if interrupt handler has been executed.
Since according to the ticker requirements min acceptable counter size is 12 bits (low power timer) for which max count is 4095, then all test cases must be executed in this time window.
HAL ticker layer handles overflow and it is not handled in the target ticker drivers.
The intention was to use ticker_overflow_delta equal to 0 for low power ticker tests and ticker_overflow_delta equal to 50 for high frequency ticker tests. Current implementation is invalid since for devices which provide LOW_POWER_TIMER feature delta is equal to 0 and for devices without this feature delta is equal 50.
Since target specific ticker drivers are not ready also features which uses ticker in upper layers may not work correctly and tests for these features.
We need to disable also failing higher level ticker related tests (by adding check if DEVICE_USTICKER symbol is available and raise #error [NOT_SUPPORTED] if not).
Cruz Monrreal
Veijo Pesonen
Przemyslaw Stekiel
Martin Kojtal
Amir Cohen
Seppo Takalo
Russ Butler
ccli8
Yossi Levy
David Saada
Anna Bridge
Ron Eldor
Jarno Lamsa
Maciej Bocianski
paul-szczepanek-arm
Donatien Garnier
Qinghao Shi
Kevin Bracey
Amanda Butler
Deepika
Steven
Russ Butler
Mika Leppänen
Juha Ylinen
Ben Cooke
Nir Sonnenschein
jeromecoutant
kegilbert
Jimmy Brisson
Senthil Ramakrishnan
Cruz Monrreal
Wilfried Chauveau
Marc Emmers
Kimmo Vaisanen
Bartek Szatkowski