Commit Graph

127 Commits (23d1d35ea9486561b181ce5fe901bbe909f2625e)

Author SHA1 Message Date
Kris Gesling d4abc37861 sleep prior to tests to assist in bug diagnosis 2020-07-17 15:53:59 +09:30
Kris Gesling 247c6d30a3 add test - specific skill should not reply 2020-06-18 22:33:22 +09:30
Åke Forslund e08f630bea Minor header fixes in voight_kampff
- Copyright year
- typo
2020-05-18 14:48:19 +02:00
Åke Forslund 46c8a6b51f Add Voight Kampff support setting configurations
Adds the "Given the user's {config} is {value}" step implementation

This will patch the configuration with a section from a dictionary that
can either be a global (shipped in
mycroft/res/{lang}/configurations.json) or shipped with the test
definition. The file should be named the same as the feature file but
instead of ".feature" the extension should be ".config.json".

mycroft/res/text/en-us/configurations.json contains a couple of
pre-defined configurations that can be applied
- units (metric/imperial)
- location (Stockholm)

After each scenario any applied patch will be cleared
2020-05-18 14:48:14 +02:00
Bart Ribbers 4d01269f35
Explicitely run the messagebus test with Python3
/usr/bin/python might not always point to Python 3, and it might not
even exist, depending on the distribution. To make sure we always run
with Python 3, explicitely call /usr/bin/python3

Also update the service start command and remove the outdated message
method tests
2020-05-06 14:28:27 +02:00
Kris Gesling 1a96f04020 Add single Voight Kampff module interface
Takes in arguments for both test_setup.py and behave test runner. Parses
any args for test_setup and passes any remaining arguments to behave.

This moves argparsing out of the test_setup main() allowing the helper commands
to pass in pre-parsed arguments rather than adding logic inside main to
differentiate between a list and a preparsed arument object
2020-03-25 07:40:01 +01:00
Åke Forslund 9fe158d841 Prepare for multi-lang support 2020-03-24 10:42:07 +01:00
Åke Forslund 00c83c5139 Add repo and branch selection for test_setup.py
--url can be added to specify the repo url
--branch can be added to specify a specific branch
2020-03-24 10:42:07 +01:00
Åke Forslund c33f9ee875 Restructure test_setup.py according to review 2020-03-24 10:42:07 +01:00
Åke Forslund f8c6107ea5 Update readme with some new steps 2020-03-24 10:42:07 +01:00
Åke Forslund 8e7e19ecc2 Add help text to test_setup.py parameters 2020-03-24 10:42:07 +01:00
Åke Forslund daeac964e0 Rename tested-skills argument to test-skills 2020-03-24 10:42:07 +01:00
Åke Forslund 5d6cb83899 Make all fields in config optional 2020-03-24 10:42:07 +01:00
Åke Forslund e0dec90d17 Allow test_setup to update skills 2020-03-24 10:42:07 +01:00
Åke Forslund 2a7b0ccef9 Update voight kampff Readme with some extra info 2020-03-24 10:42:07 +01:00
Åke Forslund a6d1e91942 Add auto-retry
This switches behave to the current dev release including the autoretry
system.
2020-03-24 10:42:07 +01:00
Åke Forslund 0ad91ce321 Add an event that can be waited on instead of sleep 2020-03-24 10:42:07 +01:00
Åke Forslund 0f860162e3 Update run_test_suite.sh
- Pulseaudio is launched if on CI
- remove hard-coding of start-mycroft.sh
2020-03-24 10:42:02 +01:00
Åke Forslund 353239cbc2 Move more common functions to voight kampff tools 2020-03-23 11:41:13 +01:00
Åke Forslund 0d5520be4a Reduce sleep time while waiting for response 2020-03-23 11:41:13 +01:00
Åke Forslund 2055b50238 Add before_feature logging current test 2020-03-23 11:41:13 +01:00
Åke Forslund fa88562a8c Fix timeout of wait for message
The counter didn't increment as intended
2020-03-23 11:41:13 +01:00
Åke Forslund 9a5f1f6f7e Add shared tools 2020-03-23 11:41:13 +01:00
Åke Forslund 0ea1af126d Reduce safety timeouts 2020-03-23 11:41:13 +01:00
Åke Forslund d451e1c1e0 Add more info on captured response (skill and dialog) 2020-03-23 11:41:13 +01:00
Åke Forslund b79fe04eb8 Extend timeout after scenario 2020-03-23 11:41:13 +01:00
Åke Forslund 51f07a5d96 Add assert message if no dialog is recorded 2020-03-23 11:41:13 +01:00
Åke Forslund c159377acb Consume matched messages. 2020-03-23 11:41:13 +01:00
Åke Forslund b43785a403 Recreate messed up assert messages 2020-03-23 11:41:13 +01:00
Åke Forslund 1470151c2b Add alternatives for user replies 2020-03-23 11:41:13 +01:00
Åke Forslund 0c873b5065 Update skill list for default skills test 2020-03-23 11:41:13 +01:00
Åke Forslund 75f92a66dd Add handler to capture messages of specific type 2020-03-23 11:41:13 +01:00
Åke Forslund 12f2e63cc3 Share only identity and allure in separate volume
- Sharing only the identity file removes the need for clearing the skill
sandbox dir and padatious cache
- Make things a bit cleaner with separate Allure volume
2020-03-23 11:41:13 +01:00
Åke Forslund 2592c9b1c2 Add clearer feedback on failures
Will now print all responses received from Mycroft
2020-03-23 11:41:13 +01:00
Åke Forslund 15148fadf5 Rework bus hooks
Override the bus clients on_message method and collect all messages in a
list. This will make it harder to miss a message during a test
2020-03-23 11:41:13 +01:00
Åke Forslund 38d43f2bb7 Move Jenkins specific commands into Jenkinsfile
Allure commandline and chown command are now run through the Jenkinsfile
instead of through the run_test_suite.sh
2020-03-23 11:41:13 +01:00
Chris Veilleux b436e5575a Add allure test reports
Behave will generate test results in the allure format, this will be
picked up by Jenkins and sent of to a standalone webserver.
2020-03-23 11:41:13 +01:00
Åke Forslund fc9c462a37 Add possibility to send a user response as Then step 2020-03-23 11:41:13 +01:00
Åke Forslund 4a1dbd7e04 Add dockerfile for running voight_kampff test 2020-03-23 11:41:13 +01:00
Åke Forslund b4507b7866 Add Voight Kampff test
The Voight kampff test is an integration test collecting and running
behave test of skills.
2020-03-23 11:41:09 +01:00
Åke Forslund 8e12aee1dc Remove refs to Settings class from skill tester
Skill tester used a "mocked" skill test class overriding some methods.
This is no longer necessary since a normal dict will work just as well.
2020-02-27 08:07:12 +01:00
maxbachmann 7a48297132 cleanup python2 unicode 2019-11-26 23:09:32 +01:00
Åke Forslund af79c7c2d2 Make skill's logging visible
The skill logger was still set to the temporary logger, this resets it
to the intended logger name and handler
2019-10-10 09:19:59 +02:00
Åke Forslund 48e93f3504 Fix TestSettings
Updates the settings to work with the new ones
2019-09-25 13:01:45 +02:00
Åke Forslund 3b2c7267e7 Rename Message type arg to msg_type
Name change to remove collision with the builin type.
2019-09-02 13:30:55 +02:00
Åke Forslund edc6192124 Reduce complexity of EvaluationRule
Creates rules.py containing logic for the small rule snippets for each
entry in the json
2019-09-02 10:03:45 +02:00
Åke Forslund f35ccae83b Fix broken startup log capture 2019-09-02 10:03:45 +02:00
Åke Forslund 69231c5ed1 Make execute_test less complex
Split the long function into several smaller ones just retaining the
main logic.
2019-09-02 10:03:45 +02:00
Åke Forslund 98c1a74ce8 Add dummy wait_for_response() method to test emitter 2019-09-02 10:03:45 +02:00
Chris Veilleux 3bd3dd1bed Refactor skill manager (#2237)
Split skill_manager into three separate classes, SkillManager, SkillUpdater and SkillLoader splitting the responsibility into logical units

* Split the SkillManager.__init__ code to determine the download times into a new method
* Make docstrings consistent and PEP257 compliant.  Also fixed a couple of spelling errors
* fixed two issues introduced in the previous refactoring
* removed unnecessary assignment of an instance attribute to a local variable
* updated the unit test to mock out code that reaches outside of core, like MSM and the configuration manager.
* add several unittests and refactored load_priority method.
* add a test for the _get_last_modified_date function.
* add "quick" argument to docstring
* removed unused import
* new class containing the logic to periodically update/install skills and send skill manifests to the backend.
* import MsmException from where it is defined, not from the skill manager.
* add some logging to the skill updater
* remove code now in SkillUpdater from SkillManager
* added imports to __init__.py to define the API into the message bus package
* new base class for unit tests and module for reusable mocks
* new skill loader class that will replace the _load_or_reload_skill() method in the SkillManager class.
* moved skill loading logic from core.py into skill_loader.py, resulting in some refactoring of skill loader and skill manager.  change unit tests to match.
* added back some spacing that was inadvertently removed.
* change skill tester to use new SkillLoader class.
* Separate reload required check from performing reload to make logic easier
to follow
* Track skills that failed to load to handle infinite loop at first load
if skill fails to load
* Allow reloading skills that has failed to load
* Simplify first load of skills
- create activate, deactivate and unload methods for skill_loader
objects
- add sanity checks before activating and deactivating skills
- Update activation/deactivation test cases
2019-08-20 12:02:39 +02:00