Several visual changes to the logs (no functional difference)
* Added 'with' to close test_case_file, quieting warning
* Highlight Mycroft's utterannces in output and moved print to fix order of printing
* Better highlight sending a response
* Highlight the test utterance
While working on the Alarm skill I discovered several issues with the
event scheduler. This PR cleans up those findings and resolves several
other potential issues:
1) To avoid thread synchronization issues, the EventScheduler had several
queues which independently held objects to be added/deleted/updated. However, the order of the events was undefined and got mixed since they were all batched together. So, for instance, if skill code performed:
self.add_event("foo", self.handle_foo)
if SomeReason:
self.cancel_event("foo")
The actual order of queue handling would perform Remove first, then Add which resulted in "foo" not being found for delete, but then added and left as an active event.
Now the EventScheduler protects the list using a Lock and the queues have been removed. Modifications to the list happen immediately after obtaining the lock and are not batched up.
2) One-time events were triggered while the event was still in the EventScheduler list. Now the entry is removed before invoking the handler.
3) Within the MycroftSkill.add_event(name, handler) is a local 'wrapper' method that actually makes the callback. The MycroftSkill.remove_event(name) method attempted to find entries in the events list and the associated handler entries in the self.emitter to remove. However, the emitter actually held the wrapper(handler), not the handler itself. So the emitter handlers were left behind.
This was a quiet bug until the next time you scheduled an event of the same name. When that second event finally triggered, it would fire off both the new and the old handler -- which snowballed in the 'skill-alarm:Beep' case, doubling and redoubling with every beep.
Now this cancels all the emitter listeners by name. There is a very slim chance that someone has registered a listener with the same name, but since it is namespaced to "skill-name:Event" I think this is pretty safe.
Not technically related, but a failure that has been lurking for
some time and is a French unit test that doesn't work depending
on the time of day when the test is run.
This is reimplementation of #1649 which became divergent.
## Description
Adds a Ogg123Service and a play_ogg exactly like Mpg123Service and play_mp3
## How to test
I have a skill for a podcast which does not have an mp3 feed:
https://github.com/joshuacox/skill-GNUworldOrder
## Contributor license agreement signed?
signed by @joshuacox
Response formatting for German language ordinals depending on cases/prepositions for dates
"am 1. März" -> "am ersten März" (on the first of March)
"der 1. März" -> "der erste März" (the first of March)
"1. März" -> "erster März" (first of March)
Response formatting for mathematical results
"10 ^ 2" -> "10 hoch 2" (ten to the power of two)
Can be tested via the corresponding test_format_de or by using wolfram alpha skill:
"Was ist die Fläche von Canada"
"Wann ist George Washington geboren"
* Improve the standalone skill tester
Using the skill tester was difficult -- the skill author had to copy a file
locally, and figuring out what was wrong with a test wasn't obvious. This
change improves it in several ways:
* The tester can now be run as a module, allowing any skill to be tested
by entering the skill folder and running:
```
python -m test.integrationtests.skills.runner
```
Optionally you can pass along a path to the skill.
* The runner will display help with a '--help' parameter, pointing to
documentation on creating the tests.
* Information on where the tests are expected is printed during execution
* The *.intent.json was reduced to simply *.json since this is under an
```test/intent/``` folder already. (This is backwards compatible for
existing intent tests.)
* The failing rule is now displayed at the bottom of the run report, making
it easier to figure out where issues exist during test creation
* Headers and terminal colors are used in the output, making it easier to
visually parse the output from the execution of tests.
==== Documentation Notes ====
Update the skill documentation to reflect using
python -m test.integrationtests.skills.runner
instead of copying the skill_developers_testrunner.py.
NOTE: This does have to be performed within a developer venv in order to
access the test suite. Consider moving this into a mycroft-core-dev package
in the future for Mark 1 / Picroft users.
* Quieting warning from Codacy
* Replace single_test with runner
Adds support for "test_env" to the runner script
* Update the discover_test to match the runner
- catch intent tests from *.json files
- add failure msg to assert
* Turn off color using MST_NO_COLOR env variable
Many cases that were missed in the unittests for extract_datetime()
from the original source. Restored those tests and made code
adjustments to support them all.
Also adding the mycroft.util.time module. This supports:
* mycroft.util.time.default_timezone()
Returns the user-configured timezone based on location
* mycroft.util.time.now_utc()
Returns the time in UTC
* mycroft.util.time.now_local()
Returns the time in the user's timezone
* mycroft.util.time.to_utc()
Converts to UTC
* mycroft.util.time.to_local()
Converts to user's timezone
NOTE: Several skills should be updated to use these now.
==== Fixed Issues ====
Several issues for skills regarding parsing of "today"
==== Documentation Notes ====
Note the new module: mycroft.util.time
==== Localization Notes ====
Localized versions of extract_datetime() likely need to be
updated, as most were based on the original English implementation
* Restore extractdatetime to return False
- Restore extractdatetime to return False if no time was found
- Add tests to make sure this is true
- Add a extract_datetime function to keep coherency with the rest of the functions. (the old extractdatetime still exists for compatibility)
- Update documentation to match
* Minor corrections to docstrings.
* Update format for skill listing
Now send the skills with id and active status
* Add commands to activate/deactivate skills
* Add "unload all except one" functionallity
* Update after rebasing
- fix identifying skills
* Unload skills if they're removed from disk
* Rename _shutdown to default_shutdown
The method is not intended to be non-public, and this should shut up
codacy bot.
* Handle keep command without argument
* Add new commands to help
- Split help into multiple pages as needed
* Support :activate all
This also removes the ability to pass the skills dir as the first argument. Instead this was moved to an environment variable called SKILLS_DIR to prevent conflict with pytest arguments
Forcing all content to string limits the amount of tests that can be run with this. For example a list of strings will fail. This keeps the original type from the json
Adds the possibility to add a responses list in the test case. The test
will respond with each of the entries in the list to get_response
requests.
Ex:
{
[...]
"responses": ["yes", "Miami"]
}
expected_data would fail in combination with Intent tests, this
differentiates it from the other rules
Add message type as "__type__" in the data for processing using
expected_data.
The custom test runner can be used for mocking third-party applications
or services.
To use create a __init__.py in the SKILL_DIR/test
The base of the file should look something like:
```python
from test.integrationtests.skills.skill_tester import SkillTest
def skill_runner(skill, example, emitter, loader, m1, m2):
return SkillTest(skill, example, emitter).run(loader)
```
Then the skill_runner can be decorated by `mock.patch` to mock out resources.
Allows the intent test to check expected data content.
Example:
{
"utterance": "set a weekend alarm at 9 am",
"expected_data": {
"ampm": "am",
"time": "9",
"daytype": "weekend"
},
"expected_response": "Okay. Setting a .* alarm"
}
Rule is satisfied if the entry exists in the data field not taking into
account it's value at all:
Example:
{
"utterance": "set an alarm on july 4th 2016 at 3pm",
"evaluation_timeout": 10,
"assert": "[['and', ['exists', 'ampm'], ['exists', 'time'] ]]"
}
Minimum viable solution for running existing testcases (json files).
Currently considered additions:
(16, 3) # TODO: Make template for testing one skill only, for the skill
developer to use
(138, 11) # TODO: Pass something to intent, that tells that this is a
test run. The skill intent can then avoid side effects
(144, 11) # TODO: add optional timeout parameter to test_case
(155, 11) # TODO: Check that all intents are checked (what about
context)
(169, 3) # TODO: Add command line utility to test an event against a
test_case, allow for debugging tests
(173, 11) # TODO: Add support for expected response, and others
This is necessary because in Python 3, hash(x) changes every single start of the application. Using the skill folder makes it consistent. In addition, the skill folder makes it easier to debug parts of the application in comparison to using something like an md5sum
Lots of minor fixes including, sorting dicts, making ints of strings,
MagicMock file spec and some other things
A couple of issues in the mycroft-core code base were identified and
fixed. Most notably the incorrect version check for python three when
adding basestring.
Update .travis.yml
- Engines now specify if they support ssml rather than the configuration
- The text client strips out ssml tags
- Engines can modify tags via the `self.modify_tag` method
* Fix error message for enable_intent
The error was printed for each intent name mismatch instead of after all intents had been checked.
* Make sure intents aren't munged multiple times
Previously intents could be munged multiple times (This happened when enabling a disabled intent), resulting in an invalid name.
* Add test case for disable/enable intent
* Improve unmunging of messages
This make sure that only skill id's in the beginning of messages are removed and should speed up the process slightly
* Fix munging for register_vocab and register_regex
* Add testcases for register vocab and regex
Add support for:
* mycroft.util.format.nice_time()
* mycroft.util.format.prounce_number()
* implemented unittests for above
Also renamed the helper method convert_number() to
_convert_to_mixed_fraction()
- add_event() now accepts the parameter once, registring the event as a one shot event.
- remove_event for non-existing events is handled
- added a test for this
Move the language specific functions and constants into separate files.
This will avoid many unnecessary conflicts due to involuntary encoding
changes.
When a token has been generated for the provided developer credentials
id the method will return it as json, if it doesn't exist HTTPError will
be raised (404 not found)
Previously the coverage counted the entire mycroft core including things
such as the docs and the tests themself. The inclusion of the actual
tests in the coverage metric scewed the metric quite a bit and the
inclusion of irrelevant files made the information presented by
coveralls cluttered.
As noted on the Chromium Dev How-to [1] and on the
SpeechRecognition library docs [2], the Google TTS API
is really not the right API to call. While it's simple,
and allows you to authenticate via a simple token, it is
also limited to a max of *50* requests per day.
That's not a lot. I don't think many people will find that
useful, outside of quick testing (if they can even get an
API key - I couldn't figure out how to generate one that
worked correctly).
This commit introduces a new STT backend, google_cloud, so that
the Google STT backend can be deprecated eventually. To do so, we
needed to:
- Install the Google API Client Library
- Create a new STT class which knows how to turn a provided Google
JSON credentials file into a string (the SpeechRecognition library
expects you to pass in the JSON string, not a file path, nor an object).
Any person wishing to use this will need to:
- Enable the Cloud Speech API on the Google Cloud Platform console
- Create a new Service Account Key, and download the credentials to
a secure location
- Configure that location in mycroft.conf, like so:
"stt": {
"google_cloud": {
"credential" {
"json": { contents of downloaded credentials }
}
}
}
It's worth noting that the Cloud Speech API has a free quota of
60 minutes per month, which would probably stretch further than 50
individual requests. So for hobbyists, it should be nothing but
a net benefit.
[1] http://www.chromium.org/developers/how-tos/api-keys
[2] https://github.com/Uberi/speech_recognition/blob/master/reference/library-reference.rst#recognizer_instancerecognize_googleaudio_data-key--none-language--en-us-show_all--false
PR 1049 introduced several cosmetic PEP8 errors that were easily fixed.
Additionally there are unittests that include non-ASCII characters which are
failing. As Pt-PT support is a work-in-progress, I just commented them out
with TODOs next to them.
==== Tech Notes ====
- removed old main.py
- replace reference to ConfigurationManager in api tests
- reset configuration after use in configuration test
- Pep-8 issue
added capability to auto upload changes from settingsmeta.json to home.mycroft.ai
==== Documentation Notes ====
If a developer make changes to the settingsmeta.json, then this will be auto uploaded to home.mycroft.ai
==== Protocol Notes ====
hash and uuid are now stored as variables in files located in ~/.mycroft/skills/{skill-name}
This commit officially switches the mycroft-core repository from
GPLv3.0 licensing to Apache 2.0. All dependencies on GPL'ed code
have been removed and we have contacted all previous contributors
with still-existing code in the repository to agree to this change.
Going forward, all contributors will sign a Contributor License
Agreement (CLA) by visiting https://mycroft.ai/cla, then they will
be included in the Mycroft Project's overall Contributor list,
found at: https://github.com/MycroftAI/contributors. This cleanly
protects the project, the contributor and all who use the technology
to build upon.
Futher discussion can be found at this blog post:
https://mycroft.ai/blog/right-license/
This commit also removes all __author__="" from the code. These
lines are painful to maintain and the etiquette surrounding their
maintainence is unclear. Do you remove a name from the list if the
last line of code the wrote gets replaced? Etc. Now all
contributors are publicly acknowledged in the aforementioned repo,
and actual authorship is maintained by Github in a much more
effective and elegant way!
Finally, a few references to "Mycroft AI" were changed to the correct
legal entity name "Mycroft AI Inc."
==== Fixed Issues ====
#403 Update License.md and file headers to Apache 2.0
#400 Update LICENSE.md
==== Documentation Notes ====
Deprecated the ScheduledSkill and ScheduledCRUDSkill classes.
These capabilities have been superceded by the more flexible MycroftSkill
class methods schedule_event(), schedule_repeating_event(), update_event(),
and cancel_event().
==== Tech Notes ====
Some functions have been orphaned in core.py and are only used in the
tests. To clean up these have been moved to where they're used.
==== Fixed Issues ====
NONE - replace with associated issue numbers, e.g. #123, #304
==== Tech Notes ====
NONE - explain new algorithms in detail, tool changes, etc.
==== Documentation Notes ====
NONE - description of a new feature or notes on behavior changes
==== Localization Notes ====
NONE - point to new strings, language specific functions, etc.
==== Environment Notes ====
NONE - new package requirements, new files being written to disk, etc.
==== Protocol Notes ====
NONE - message types added or changed, new signals, APIs, etc.
==== Fixed Issues ====
NONE - replace with associated issue numbers, e.g. #123, #304
==== Tech Notes ====
NONE - explain new algorithms in detail, tool changes, etc.
==== Documentation Notes ====
NONE - description of a new feature or notes on behavior changes
==== Localization Notes ====
NONE - point to new strings, language specific functions, etc.
==== Environment Notes ====
NONE - new package requirements, new files being written to disk, etc.
==== Protocol Notes ====
NONE - message types added or changed, new signals, APIs, etc.
This is just a basic test of sending between websocket clients. It makes sure the message gets to the other side. It doesn't check for correct ness. More testing sure could be done.
Signed-off-by: Art McGee <amcgee7@jaguarlandrover.com>
Adds the ExtractDateTime parse function from Christopher. When imported from mycroft/util/parse.py, it'll take a sentence like "What's the weather like 5 weeks from next Wednesday?" and will extract a python datetime object for that date.
* Added requirements.txt change for importing dateutil
This addresses this in several ways:
* Created mechanism to load 'commented' JSON (using '//' or '#' comments on a single line)
* Embedded comments into the mycroft.conf, indicating use, legal values, and where they get overridden
* Create ConfigurationManager.instance() static method to replace ConfigurationManager.get(). This produces more readable code like:
ConfigurationManager.instance().get("value") instead of ConfigurationManager.get().get("value")
* Made _ConfigurationListener 'private'
* docstring'ed things
* intent_failure message now carries along the utterance's lang code
* normalizing query for Wolfram Alpha
* added normalization of "whats" to "what is". This is technically incorrect ("whats" means more than one instance of "what", as in "the whats and whys of open source"), but that is a rare phrase. Unfortunately, several STT engines incorrectly output things like "whats 8 + 4", which is grammatically incorrect. So we'll handle the common and potentially screw up the uncommon.
* more parsing test cases, including a few corrections
The utterance is now placed on the bus along with its language code. If not specified, it uses "en-us".
Added a new mycroft.util.parse module. It contains the normalize() function. Normalization currently does two things:
* Expands contractions ("they're" -> "they are", etc)
* Optionally removes articles ("a", "an", "the"). Removing is the default.
* Textual numbers become digits, up to 20. E.g. "What is the weather in four days" becomes "What is weather in 4 days".
NOTE: This is potentially a breaking change! Remove "the", "a" and "an" from your .voc files!
Skill changes:
* I cleaned up the .voc files for the default Skills.
* Split the date_time keyword into an extra entity. Now a "QueryKeyword.voc" exists, with "what|tell" instead of combing that into "what is time" in the TimeKeyword.voc.
* Volume skill now accepts 1-11, e.g. "turn volume to 11"
Add cleanup procedure to skills:
MycroftSkill cleanup does nothing, ScheduledSkill will cancel running timers to make sure it's not blocking termination.
Tests updated to use this allowing unit test to exit after completion.
* Added new listener config options
* Fixed audio unit tests
This adds a cleareraudio file for the wakeup test and changesto the new LocalRecognizer constructor
* Added .dict files to .gitignore
This is because they are now auto-generated on startup rather than stored permanently
* Fixed audio accuracy test for new LocalRecognizer constructor
* Added support for spaces in wake word config
In the phonemes a new word is indicated by a period character. The separating of the words actually changes the way pocketsphinx interprets the sound of it and in this case improves it
* Fixed unit test
* Added new compiled regex for what's
This is for verbs that don't have a space between the question word and verb
* Added support for 'whats' in wolfram
* Added tests for the EnglishQuestionParser
* Issues 192 - WA skill will now search again with top alternative
* Issues 192 - Add tests for new method
* Issues 192 - Fix pep8
* Issues 192 - Update test syntax
* Issues 192 - Change WA skill to use dialog
* Issues 192 - Address feedback
* Issues 158 - Update ScheduledSkill time formatting
* Issues 158 - Add unit tests for new method changes
* Issues 158 - Change to not use date-based implementation to fix tests
* Issues 158 - Address feedback
Steve Penrod
Åke Forslund
Matthew D. Scholefield
Michael Nguyen
Steve Penrod
Josh Cox
f-e-l-i-x
JarbasAI
Carsten Agerskov
jarbasal
Matthew D. Scholefield
Kévin C
Augusto Monteiro
Cakeh
Ale
gaurav
Andrew Hayworth
Art McGee
harika
Connor Penrod
Augusto Monteiro
kfezer
ProsperousHeart
Ashwin Venkatesan
SoloVeniaASaludar
Jonathan D'Orleans
Ethan Ward
Jonathan D'Orleans
Ryan Sipes