Commit Graph

83 Commits (c6b1e3dc92c2e6e41bd5e3f7ca465ecdfb43a9b7)

Author SHA1 Message Date
Åke Forslund 4a7cf45ada Add possibility to add a custom test runner
The custom test runner can be used for mocking third-party applications
or services.

To use create a __init__.py in the SKILL_DIR/test

The base of the file should look something like:

```python
from test.integrationtests.skills.skill_tester import SkillTest

def skill_runner(skill, example, emitter, loader, m1, m2):
    return SkillTest(skill, example, emitter).run(loader)
```

Then the skill_runner can be decorated by `mock.patch` to mock out resources.
2018-05-14 14:27:37 -05:00
Åke Forslund cab556450e Add expected_data option
Allows the intent test to check expected data content.

Example:

{
  "utterance": "set a weekend alarm at  9 am",
  "expected_data": {
      "ampm": "am",
      "time": "9",
      "daytype": "weekend"
  },
  "expected_response": "Okay. Setting a .* alarm"
}
2018-05-14 14:27:37 -05:00
Åke Forslund 9c4ac178b9 Give better feedback when skill isn't loaded
If a skill required for a test case isn't loaded an exception will be
raised.
2018-05-14 14:27:37 -05:00
Åke Forslund 3917726bb5 Use queue to get skill completion
This allows the skill test to exit directly after skill handler
completes.
2018-05-14 14:27:37 -05:00
Åke Forslund e7ead7ca6e Reduce timeout when skill has been handled 2018-05-14 14:27:37 -05:00
Matthew D. Scholefield 79bf94e48e Speed up intent tests by responding to converse requests 2018-05-14 14:27:37 -05:00
Åke Forslund 02565efd59 Use pytest to discover test cases 2018-05-14 14:27:37 -05:00
Matthew D. Scholefield 154ccd59d7 Expose single threaded training option from Padatious Service
This is necessary to prevent a deadlock when training during intent tests using pytest
2018-05-14 14:27:37 -05:00
Åke Forslund 921bc5f23d Add support for "expected dialog"
Loads dialogs and checks if any skills responds with one of the possible
dialogs.
2018-05-14 14:27:37 -05:00
Åke Forslund d10ba29f70 Fix pep8 issues 2018-05-14 14:27:37 -05:00
Åke Forslund 3fcb40c2bb Add new rule 'exists'
Rule is satisfied if the entry exists in the data field not taking into
account it's value at all:

Example:
{
    "utterance": "set an alarm on july 4th 2016 at 3pm",
    "evaluation_timeout": 10,
    "assert": "[['and', ['exists', 'ampm'], ['exists', 'time'] ]]"
}
2018-05-14 14:27:37 -05:00
Åke Forslund 90e719c29e Add padatious and fallbacks 2018-05-14 14:27:37 -05:00
Åke Forslund 3a9242c1c7 Allow utterance without intent_type and intents 2018-05-14 14:27:37 -05:00
Åke Forslund a206807606 Add simple script for running a single skill
Based on the skill_developers_testrunner.py, takes the path to a skill
as argument and runs any tests for that skill.
2018-05-14 14:27:37 -05:00
CarstenAgerskov 8ab30500d7 Cleaner and better format for examples in message_tester 2018-05-14 14:27:37 -05:00
CarstenAgerskov 1cbedaea3c Added test_all_skills. More doc. 2018-05-14 14:27:37 -05:00
CarstenAgerskov 6c226ea4d9 Added message tester utility. Reduced line length to <80 chars in general 2018-05-14 14:27:37 -05:00
CarstenAgerskov a16c2a0ecc Added support for context, mycroft response and more. 2018-05-14 14:27:37 -05:00
CarstenAgerskov 7e9a05f4e0 First cut at skill tester
Minimum viable solution for running existing testcases (json files).
Currently considered additions:
(16, 3) # TODO: Make template for testing one skill only, for the skill
developer to use
(138, 11) # TODO: Pass something to intent, that tells that this is a
test run. The skill intent can then avoid side effects
(144, 11) # TODO: add optional timeout parameter to test_case
(155, 11) # TODO: Check that all intents are checked (what about
context)
(169, 3) # TODO: Add command line utility to test an event against a
test_case, allow for debugging tests
(173, 11) # TODO: Add support for expected response, and others
2018-05-14 14:27:37 -05:00
Matthew D. Scholefield fa11ac7dbe Rename MycroftSkill.shutdown to MycroftSkill._shutdown
This removes the need for skills to call Super(MySkill, self).shutdown()
2018-04-11 21:29:14 -05:00
Matthew D. Scholefield 5c8cf053a0 Remove code that waits for websocket to open
This is unnecessary now that WebsocketClient does this automatically
2018-03-28 19:19:56 -05:00
penrods 8f2e5d9498 Change to Apache 2.0 license from GPLv3.0
This commit officially switches the mycroft-core repository from
GPLv3.0 licensing to Apache 2.0.  All dependencies on GPL'ed code
have been removed and we have contacted all previous contributors
with still-existing code in the repository to agree to this change.

Going forward, all contributors will sign a Contributor License
Agreement (CLA) by visiting https://mycroft.ai/cla, then they will
be included in the Mycroft Project's overall Contributor list,
found at: https://github.com/MycroftAI/contributors.  This cleanly
protects the project, the contributor and all who use the technology
to build upon.

Futher discussion can be found at this blog post:
https://mycroft.ai/blog/right-license/

This commit also removes all __author__="" from the code.  These
lines are painful to maintain and the etiquette surrounding their
maintainence is unclear.  Do you remove a name from the list if the
last line of code the wrote gets replaced?  Etc.  Now all
contributors are publicly acknowledged in the aforementioned repo,
and actual authorship is maintained by Github in a much more
effective and elegant way!

Finally, a few references to "Mycroft AI" were changed to the correct
legal entity name "Mycroft AI Inc."

==== Fixed Issues ====
#403 Update License.md and file headers to Apache 2.0
#400 Update LICENSE.md

====  Documentation Notes ====
Deprecated the ScheduledSkill and ScheduledCRUDSkill classes.
These capabilities have been superceded by the more flexible MycroftSkill
class methods schedule_event(), schedule_repeating_event(), update_event(),
and cancel_event().
2017-10-04 01:28:44 -05:00
Matthew D. Scholefield 5e392f34aa Optimize imports
Remove unused imports and group local vs external alphabetically
2017-09-18 16:07:50 -05:00
Åke Forslund 12bf78cd5f Move funcs only used in skill_tester from core.py
====  Tech Notes ====
Some functions have been orphaned in core.py and are only used in the
tests. To clean up these have been moved to where they're used.
2017-08-21 15:59:48 +02:00
Åke Forslund 4e7defa89f Fix pep8 issues. 2017-08-17 02:05:02 -05:00
Åke Forslund 1c5da3e25d Move messagebus test to integration tests 2017-08-17 02:05:02 -05:00
harika 173bee9f95 Fixed pep8 format 2017-08-17 02:05:02 -05:00
harika 4c3dd3aa7c Updating skill_tester to automate skills testing(regression) 2017-08-17 02:05:02 -05:00
harika 632df2a498 Update tests 2017-08-17 02:05:02 -05:00
harika e2811ccde2 added some stuffs --arron says it's awesome! 2017-08-17 02:05:02 -05:00
Åke Forslund c8dd1a9b46 Make discover test directory settable
The skill directory can be specified when calling the discovery_test

`PYTHONPATH=./ python test/integrationtests/skills/discovery_test.py
 SKILL_DIR`
2017-08-17 02:05:02 -05:00
Åke Forslund f0a7a71e01 Fix shutdown error.
None type can be in skill list
2017-08-17 02:05:02 -05:00
Åke Forslund 39d0451143 Separate tests into unittests and integrationtests 2017-08-17 02:05:02 -05:00