- Tests for TTS base class
- Test for TTS playback queue
- Test for TTS Factory create
- Tests for Mimic2 TTS
- Tests for Mimic TTS
- Tests for Google TTS
- Fix: isFractional_es() parsed fractions incorrectly
- Update: earlier commit msg suggested another fix:
- Month parsing not fixed
- Several failing tests (skipped) document problem
- TODO and an issue also created
- Substantially improve parse_es.py test coverage
- TODO or comment several found bugs
- Many lines remain uncovered, incl possible bugs
- Fix bug causing extractnumber_es to return a sum instead of a list
- Add Spanish parser to extract_numbers and extract_number
==== Fixed Issues ====
Closes#2310
==== Tech Notes ====
Further obscures #2056: short_scale and ordinals parameters added to the
Spanish parsers, but they don't do anything. Present for compat only.
There is a TODO for this.
==== Localization Notes ====
It's all Spanish stuff!
This will require a native Spanish speaker to analyze the relationship
between extractnumber and isFractional, and determine why certain
fractions do not parse correctly. There is a TODO for this.
- Remove unreachable conditions from parse_en:1122-1133
- input string passed through clean_string() on line 763
- articles stripped from input before line 1122
- removed conditions relied on presence of "the" in input
- Improved test coverage on parse_en.py by approx 65 statements
- Directly test certain helper functions which are difficult to
invoke indirectly
- Add tests for certain missed conditions
- Approx. 20 uncovered statements remain
Add one more optional parameter: bool clock, always produces
digital clock-like output. "0h 3m" becomes "0:03:00".
Has no effect on resolutions YEARS or DAYS, and MINUTES won't print hrs.
- Companion enum:
mycroft.util.format.TimeResolution
offers YEARS, DAYS, HOURS, MINUTES, SECONDS, or MILLISECONDS
- Will only return ms if MILLISECONDS is chosen. Default: SECONDS
- Update tests
Split skill_manager into three separate classes, SkillManager, SkillUpdater and SkillLoader splitting the responsibility into logical units
* Split the SkillManager.__init__ code to determine the download times into a new method
* Make docstrings consistent and PEP257 compliant. Also fixed a couple of spelling errors
* fixed two issues introduced in the previous refactoring
* removed unnecessary assignment of an instance attribute to a local variable
* updated the unit test to mock out code that reaches outside of core, like MSM and the configuration manager.
* add several unittests and refactored load_priority method.
* add a test for the _get_last_modified_date function.
* add "quick" argument to docstring
* removed unused import
* new class containing the logic to periodically update/install skills and send skill manifests to the backend.
* import MsmException from where it is defined, not from the skill manager.
* add some logging to the skill updater
* remove code now in SkillUpdater from SkillManager
* added imports to __init__.py to define the API into the message bus package
* new base class for unit tests and module for reusable mocks
* new skill loader class that will replace the _load_or_reload_skill() method in the SkillManager class.
* moved skill loading logic from core.py into skill_loader.py, resulting in some refactoring of skill loader and skill manager. change unit tests to match.
* added back some spacing that was inadvertently removed.
* change skill tester to use new SkillLoader class.
* Separate reload required check from performing reload to make logic easier
to follow
* Track skills that failed to load to handle infinite loop at first load
if skill fails to load
* Allow reloading skills that has failed to load
* Simplify first load of skills
- create activate, deactivate and unload methods for skill_loader
objects
- add sanity checks before activating and deactivating skills
- Update activation/deactivation test cases
Refactors code so no request is sent to backend if the local identity
file is empty / has no uuid. Since if this part is missing core already
knows that it's not paired correctly.
Message bus config loading is now shared by service and client.
messagebus.client.ws file is still available in case skills are using it. It is a backport that inherits from the new MessageBusClient class. Adds depreciation warning.
The initial framework for live streaming Speech to Text. STT derived
classes now can report that they support live streaming via their
"can_stream" property. If True, the following member functions will be
called on the object:
stream_start() - Called when a stream is starting. Should setup any
persistent state required to process speech to text.
stream_data(chunk) - Called when a chunk of audio data has been
captured.
stream_stop() - Called when a stream should be stopped. Should tear down
any state setup with stream_start(). Note that this may be called at
anytime (even if stream_start() has not been called).
Note that the execute() API is still used to get the text utterance from
the class.
A typical sequence of calls would be:
stream_start()
stream_data(...)
stream_data(...)
stream_data(...)
stream_data(...)
...
execute(...)
stream_stop()
* Refactor skill test input utterances
Slight cleanup to make adding more different cases easier.
* Add test for common QA
"question" can now be used instead of "utterance" to test a common qa skill
Revert "Fix a couple of minor issues intruduced by skill_gid (#2079)"
This reverts commit e046377ce1.
Revert "Merge pull request #2075 from forslund/bugfix/msm_wrapper-license"
This reverts commit 18cfbce0ca, reversing
changes made to 82fa314ce9.
Revert "Feature/skillsmeta gid (#2074)"
This reverts commit 82fa314ce9.
* Add global id basics to settings meta
- All skills will upload a blank settingsmeta
- a skill_gid will be appended to all settingsmeta upload-data
- Added basic function for generating skill_gid
* Use new skill_gid field.
Populate skill_gid directly from metadata
* Separate travis tmp-dirs
- Update travis script to use tempdir for each python version
- Update test script to handle nonstandard tempdirs
- Generate msm folder using tempdir when running create_msm test
* Add title field with pretty name
* Collect and expand "title" as needed
For title use market-place title or name in settings meta or skillname
* Switch skill_manager create_msm test to 19.02
* Remove leading / trailing Skill in display name
Also rename title displayname to match new mycroft-skills-data
* Lock msm_create and mock the name info test_settings
Much of the code used "en-us" as the default value when not specified.
This limited the internationalization potential. Changing the default
to None and adds the ability to define the default lang code from other
locations in code. E.g.
```python
from mycroft.util.lang import set_default_lang
set_default_lang("en-us")
print("English date: "+nice_date(dt))
set_default_lang("de-de")
print("German date: "+nice_date(dt))
```
This allows easier localization of Skills by having the framework set the default without any changes necessary by the Skill writers.
Other minor changes:
* Changed the default return value of get_gender*() to None instead of False
Merge consecutive .*'s into a single .*
The process for "{{modifier}} {{precip}} is expected on {{day}}" will first
replace the {{}} by .* as previously:
".* .* is expected on .*"
then a second pass is made replacing any consecutive .* resulting in
".* is expected on .*"
This commit add the initial translations of core functions for format
numbers: nice_number_es, pronounce_number_es and nice_time_es.
==== Localization Notes ====
NONE - Castillian (Spain's spanish)
* Run emitter using threadpool
This moves the thread pool from the websocket client into the eventemitter allowing each registered function to run in a separate thread and not just each event.
This speeds up cases where there is a one to many call such as the common play framework and the upcomming common query framework.
* Add unit tests for threaded event emitter
* Add standard header
* Add standard header
* date and time - for tonight, weekdays
* Updating the previous commit test_extractdatetime_en
* Editing comments for extract_date_time_en
* Generalized as a marker
Expanded this from just handling "weekends" to any day or plural of the day, "weekend", "weekday" or "weekdays".
* Let skill tester expand dialogs
The skill tester need to expand the dialogs in the same way as the dialog renderer to be able to correctly assert the expected_dialog criteria.
* Minor docstring changes
* Attempt to create skill directory if not existing
* Handle missing priority skills
* Minor update of comments
* Handle skill load exception
Make sure an exception while trying to load/reload skill doesn't shutdown thread.
* Handle MsmException during SkillManager creation
If SkillManager can't be created due to an MsmException wait for network connection and retry.
* Update immediately if skill install file is missing
Missing skill install file indicates that this is a new venv and the requirements of the skills will need to be reinstalled.
* Add basic test for skill_manager
Basically only creating the skill_manager but it ensures that msm can be used on all supported python versions
- Create a read_vocab_file() function that normal vocab loading and voc_match both uses. This function handles blank lines and comments
- Use a simpler regex instead of word logic to match
- Add a couple of test cases for the method
* New formatters: nice_duration() and join_list()
Adding two new formatting functions:
* nice_duration(duration, lang="en-us", speech=True)
Accept seconds or duration and produce a nice sounding duration.
Example: nice_duration(61) == "one minute one second"
nice_duration(61, speech=False) == "1:01"
* join_list(items, connector, sep=None, lang="en-us")
Example: join_list(["a", "b", "c"], "and") == "a, b and c"
This includes a translation helper that uses text files in the
mycroft/res/text/LANG/ directory, such as "second.word".
This improves the utility of the _ReplaceableNumber class, and updates
most of the number parsing functions to take tokens rather than text.
This simplifies the interactions between many of the functions, as there
is no need to convert back and forth between text and tokens.
This also adds some tests. Note that there are a few regressions that
will be fixed in a subsequent commit.
* Add tests for DialogLoader
* Handle Path/PosixPath
LOG messages when files/directories were missing would fail when a PosixPath/Path object was sent to as argument. This uses format to get the correct string representation.
* Add test for dialog.get()
Issues fixed:
Lists, e.g. "some words one two three" would return (3, "one two three")
Negaitve words were not included in output, e.g. "negative five" would
return (-5, "five").
This is in support of issues-1959.
Methods implemented include:
extract_number_with_text
extract_numbers_with_text
convert_words_to_numbers
extract_duration
This is in support of issues-1959. This continues the work of
returning the relevant text that corresponds to a number
parsed from a string.
* Fix mimic2 negative numbers
Make the regex extracting numbers also match negative numbers when preparsing phrases sent to the mimic2 service
* Update pronounce_number to use "minus" for negatives
After discussion in the chat it was suggested to use "minus" for negatives as default.
When scientific notation is used the term "negative " is still used.
New tests for nice_number():
-test_specify_denominator
-test_no_speech
New tests for pronounce_number
-test_convert_hundreds
-test_convert_scientific_notation
-test_large_numbers
add support for decades, centuries, millemniums
add support for "within the hour", "in a second/minute",
add support for "a couple time_unit" and "a couple of time_unit"
* Add lt and gt to skill tester evaluation vocabulary
lt returns True if message item is LESS THAN the value in the config
gt returns True if message item is GREATER THAN the value in the config
* Add separate Exception for Skilltest errors
* Add support for common playback skill messages
CPS_query:
new test json possibilities
play_query: Emits a message that can be catched by CPS_match_query_phrase()
play_query_match: Structure with info of the expected match
"phrase": matched phrase
"confidence_threshold": The minimum confidence the phrase should result in
Example:
{
"play_query": "the news",
"play_query_match": {
"phrase": "the news",
"confidence_threshold": 0.8
}
}
"play_start": Emits message that can be catched by CPS_start using sub-fields.
"phrase": matched phrase
"callback_data": dict with info for the function
Example:
{
"play_start": {
"phrase": "the news",
"callback_data": {
}
},
"expected_data": {"__type__": "mycroft.audio.service.play"}
}
Add tests for dialogs with multiple lines, remove comment test as it didn't work
Also fix unittest for unknown templates
==== Fixed Issues ====
1829
==== Tech Notes ====
NONE - explain new algorithms in detail, tool changes, etc.
==== Documentation Notes ====
NONE - description of a new feature or notes on behavior changes
==== Localization Notes ====
NONE - point to new strings, language specific functions, etc.
==== Environment Notes ====
NONE - new package requirements, new files being written to disk, etc.
==== Protocol Notes ====
NONE - message types added or changed, new signals, APIs, etc.
This is the template for commits to mycroft-core and also an example of
a commit message. To use this template, change the first line (think of
it as a "headline" for the commit) and then edit this message with a
longer description of the change. To be nice, keep lines to 72
characters or less (The first two lines of this template are exactly
that length). Also use imperative writing, e.g. "Fix broken code" or
"Implement my new feature", not "Fixes the broken code" or
"Implemented my new feature".
Sections below can then be filled out and edited as appropriate.
Unused sections can optionally be removed.
Please fill this out carefully. This moment is when the code is
clearest in the mind of you, the foremost expert in this change.
Please pause to think of side effects and impacts.
==== Fixed Issues ====
NONE - replace with associated issue numbers, e.g. #123, #304
==== Tech Notes ====
NONE - explain new algorithms in detail, tool changes, etc.
==== Documentation Notes ====
NONE - description of a new feature or notes on behavior changes
==== Localization Notes ====
NONE - point to new strings, language specific functions, etc.
==== Environment Notes ====
NONE - new package requirements, new files being written to disk, etc.
==== Protocol Notes ====
NONE - message types added or changed, new signals, APIs, etc.
==== Fixed Issues ====
1829
==== Tech Notes ====
NONE - explain new algorithms in detail, tool changes, etc.
==== Documentation Notes ====
NONE - description of a new feature or notes on behavior changes
==== Localization Notes ====
NONE - point to new strings, language specific functions, etc.
==== Environment Notes ====
NONE - new package requirements, new files being written to disk, etc.
==== Protocol Notes ====
NONE - message types added or changed, new signals, APIs, etc.
Fix trillion being saved with wrong number (10e10 instead of 10e12)
==== Fixed Issues ====
1718
==== Tech Notes ====
NONE - explain new algorithms in detail, tool changes, etc.
==== Documentation Notes ====
NONE - description of a new feature or notes on behavior changes
==== Localization Notes ====
NONE - point to new strings, language specific functions, etc.
==== Environment Notes ====
NONE - new package requirements, new files being written to disk, etc.
==== Protocol Notes ====
NONE - message types added or changed, new signals, APIs, etc.
==== Fixed Issues ====
==== Tech Notes ====
NONE - explain new algorithms in detail, tool changes, etc.
==== Documentation Notes ====
NONE - description of a new feature or notes on behavior changes
==== Localization Notes ====
NONE - point to new strings, language specific functions, etc.
==== Environment Notes ====
NONE - new package requirements, new files being written to disk, etc.
==== Protocol Notes ====
NONE - message types added or changed, new signals, APIs, etc.
This makes a cross context call be treated as one level when calculating the probability. this makes previous contexes not be completely invalidated when a cross context call is sent.
* Fix ambiguous time handling
In certain cases the ambiguous time handling skipped a day forward. This updates the logic to handle a bit better.
* Adds a test for the ambiguous time
* Remove references to timeStr
timeStr was never set and the logic that used it would never activate.
* Remove rename of currentDate
* Add extract_datetime parameter default_time
If a time is not found in the input string the time will be set from the
datetime/time object passed in as the default_time argument. If None the
time will be Midnight as previously.
The skill log is redirected to a string during loading and if the skill
fails to load the loading logs are outputed when the first test for the
skill is executed.
Allowing multiple dialog choices can help in cases where a skill has a number of dialog files that each can be triggered independently by the same intent. For example, the weather skill inquiry "will it rain" can trigger either a response when there is an upcoming rain and another if there's no rain in the near future.
```
"expected_dialog": ["dialog1", "dialog2"]
```
and
```
"expected_response": ["text 1", "text 2"]
```
is now possible. This will pass the test if a line from either dialog1 or dialog2 is matched. (or "text 1" or "text 2" is matched for "expected_response")
The test setup function will be run after the skill is loaded but before the testing starts.
This provides a place to setup things that is standard for all test cases.
Several visual changes to the logs (no functional difference)
* Added 'with' to close test_case_file, quieting warning
* Highlight Mycroft's utterannces in output and moved print to fix order of printing
* Better highlight sending a response
* Highlight the test utterance
While working on the Alarm skill I discovered several issues with the
event scheduler. This PR cleans up those findings and resolves several
other potential issues:
1) To avoid thread synchronization issues, the EventScheduler had several
queues which independently held objects to be added/deleted/updated. However, the order of the events was undefined and got mixed since they were all batched together. So, for instance, if skill code performed:
self.add_event("foo", self.handle_foo)
if SomeReason:
self.cancel_event("foo")
The actual order of queue handling would perform Remove first, then Add which resulted in "foo" not being found for delete, but then added and left as an active event.
Now the EventScheduler protects the list using a Lock and the queues have been removed. Modifications to the list happen immediately after obtaining the lock and are not batched up.
2) One-time events were triggered while the event was still in the EventScheduler list. Now the entry is removed before invoking the handler.
3) Within the MycroftSkill.add_event(name, handler) is a local 'wrapper' method that actually makes the callback. The MycroftSkill.remove_event(name) method attempted to find entries in the events list and the associated handler entries in the self.emitter to remove. However, the emitter actually held the wrapper(handler), not the handler itself. So the emitter handlers were left behind.
This was a quiet bug until the next time you scheduled an event of the same name. When that second event finally triggered, it would fire off both the new and the old handler -- which snowballed in the 'skill-alarm:Beep' case, doubling and redoubling with every beep.
Now this cancels all the emitter listeners by name. There is a very slim chance that someone has registered a listener with the same name, but since it is namespaced to "skill-name:Event" I think this is pretty safe.
Not technically related, but a failure that has been lurking for
some time and is a French unit test that doesn't work depending
on the time of day when the test is run.
This is reimplementation of #1649 which became divergent.
## Description
Adds a Ogg123Service and a play_ogg exactly like Mpg123Service and play_mp3
## How to test
I have a skill for a podcast which does not have an mp3 feed:
https://github.com/joshuacox/skill-GNUworldOrder
## Contributor license agreement signed?
signed by @joshuacox
Response formatting for German language ordinals depending on cases/prepositions for dates
"am 1. März" -> "am ersten März" (on the first of March)
"der 1. März" -> "der erste März" (the first of March)
"1. März" -> "erster März" (first of March)
Response formatting for mathematical results
"10 ^ 2" -> "10 hoch 2" (ten to the power of two)
Can be tested via the corresponding test_format_de or by using wolfram alpha skill:
"Was ist die Fläche von Canada"
"Wann ist George Washington geboren"
* Improve the standalone skill tester
Using the skill tester was difficult -- the skill author had to copy a file
locally, and figuring out what was wrong with a test wasn't obvious. This
change improves it in several ways:
* The tester can now be run as a module, allowing any skill to be tested
by entering the skill folder and running:
```
python -m test.integrationtests.skills.runner
```
Optionally you can pass along a path to the skill.
* The runner will display help with a '--help' parameter, pointing to
documentation on creating the tests.
* Information on where the tests are expected is printed during execution
* The *.intent.json was reduced to simply *.json since this is under an
```test/intent/``` folder already. (This is backwards compatible for
existing intent tests.)
* The failing rule is now displayed at the bottom of the run report, making
it easier to figure out where issues exist during test creation
* Headers and terminal colors are used in the output, making it easier to
visually parse the output from the execution of tests.
==== Documentation Notes ====
Update the skill documentation to reflect using
python -m test.integrationtests.skills.runner
instead of copying the skill_developers_testrunner.py.
NOTE: This does have to be performed within a developer venv in order to
access the test suite. Consider moving this into a mycroft-core-dev package
in the future for Mark 1 / Picroft users.
* Quieting warning from Codacy
* Replace single_test with runner
Adds support for "test_env" to the runner script
* Update the discover_test to match the runner
- catch intent tests from *.json files
- add failure msg to assert
* Turn off color using MST_NO_COLOR env variable
Many cases that were missed in the unittests for extract_datetime()
from the original source. Restored those tests and made code
adjustments to support them all.
Also adding the mycroft.util.time module. This supports:
* mycroft.util.time.default_timezone()
Returns the user-configured timezone based on location
* mycroft.util.time.now_utc()
Returns the time in UTC
* mycroft.util.time.now_local()
Returns the time in the user's timezone
* mycroft.util.time.to_utc()
Converts to UTC
* mycroft.util.time.to_local()
Converts to user's timezone
NOTE: Several skills should be updated to use these now.
==== Fixed Issues ====
Several issues for skills regarding parsing of "today"
==== Documentation Notes ====
Note the new module: mycroft.util.time
==== Localization Notes ====
Localized versions of extract_datetime() likely need to be
updated, as most were based on the original English implementation
* Restore extractdatetime to return False
- Restore extractdatetime to return False if no time was found
- Add tests to make sure this is true
- Add a extract_datetime function to keep coherency with the rest of the functions. (the old extractdatetime still exists for compatibility)
- Update documentation to match
* Minor corrections to docstrings.
* Update format for skill listing
Now send the skills with id and active status
* Add commands to activate/deactivate skills
* Add "unload all except one" functionallity
* Update after rebasing
- fix identifying skills
* Unload skills if they're removed from disk
* Rename _shutdown to default_shutdown
The method is not intended to be non-public, and this should shut up
codacy bot.
* Handle keep command without argument
* Add new commands to help
- Split help into multiple pages as needed
* Support :activate all
This also removes the ability to pass the skills dir as the first argument. Instead this was moved to an environment variable called SKILLS_DIR to prevent conflict with pytest arguments
Forcing all content to string limits the amount of tests that can be run with this. For example a list of strings will fail. This keeps the original type from the json
Adds the possibility to add a responses list in the test case. The test
will respond with each of the entries in the list to get_response
requests.
Ex:
{
[...]
"responses": ["yes", "Miami"]
}
expected_data would fail in combination with Intent tests, this
differentiates it from the other rules
Add message type as "__type__" in the data for processing using
expected_data.
The custom test runner can be used for mocking third-party applications
or services.
To use create a __init__.py in the SKILL_DIR/test
The base of the file should look something like:
```python
from test.integrationtests.skills.skill_tester import SkillTest
def skill_runner(skill, example, emitter, loader, m1, m2):
return SkillTest(skill, example, emitter).run(loader)
```
Then the skill_runner can be decorated by `mock.patch` to mock out resources.
Allows the intent test to check expected data content.
Example:
{
"utterance": "set a weekend alarm at 9 am",
"expected_data": {
"ampm": "am",
"time": "9",
"daytype": "weekend"
},
"expected_response": "Okay. Setting a .* alarm"
}
Rule is satisfied if the entry exists in the data field not taking into
account it's value at all:
Example:
{
"utterance": "set an alarm on july 4th 2016 at 3pm",
"evaluation_timeout": 10,
"assert": "[['and', ['exists', 'ampm'], ['exists', 'time'] ]]"
}
Minimum viable solution for running existing testcases (json files).
Currently considered additions:
(16, 3) # TODO: Make template for testing one skill only, for the skill
developer to use
(138, 11) # TODO: Pass something to intent, that tells that this is a
test run. The skill intent can then avoid side effects
(144, 11) # TODO: add optional timeout parameter to test_case
(155, 11) # TODO: Check that all intents are checked (what about
context)
(169, 3) # TODO: Add command line utility to test an event against a
test_case, allow for debugging tests
(173, 11) # TODO: Add support for expected response, and others
This is necessary because in Python 3, hash(x) changes every single start of the application. Using the skill folder makes it consistent. In addition, the skill folder makes it easier to debug parts of the application in comparison to using something like an md5sum
Lots of minor fixes including, sorting dicts, making ints of strings,
MagicMock file spec and some other things
A couple of issues in the mycroft-core code base were identified and
fixed. Most notably the incorrect version check for python three when
adding basestring.
Update .travis.yml
- Engines now specify if they support ssml rather than the configuration
- The text client strips out ssml tags
- Engines can modify tags via the `self.modify_tag` method
* Fix error message for enable_intent
The error was printed for each intent name mismatch instead of after all intents had been checked.
* Make sure intents aren't munged multiple times
Previously intents could be munged multiple times (This happened when enabling a disabled intent), resulting in an invalid name.
* Add test case for disable/enable intent
* Improve unmunging of messages
This make sure that only skill id's in the beginning of messages are removed and should speed up the process slightly
* Fix munging for register_vocab and register_regex
* Add testcases for register vocab and regex
Add support for:
* mycroft.util.format.nice_time()
* mycroft.util.format.prounce_number()
* implemented unittests for above
Also renamed the helper method convert_number() to
_convert_to_mixed_fraction()
- add_event() now accepts the parameter once, registring the event as a one shot event.
- remove_event for non-existing events is handled
- added a test for this
Move the language specific functions and constants into separate files.
This will avoid many unnecessary conflicts due to involuntary encoding
changes.
When a token has been generated for the provided developer credentials
id the method will return it as json, if it doesn't exist HTTPError will
be raised (404 not found)
Åke Forslund
Chris Rogers
maxbachmann
Christian Clauss
Mike Woudenberg
Kris Gesling
David Wagner
ChanceNCounter
Chris Veilleux
mathmauney
c0r73x
Joshua Watt
Kathy Reid
jarbasal
Ruthvicp
Steve Penrod
Matthew D. Scholefield
Andreas Lorensen
Angel Docampo
Ale
Kris Gesling
Steve Penrod
G3RB3N
danielwine
JarbasAI
Niels Mündler
Michael Nguyen
Michael Nguyen
Josh Cox
f-e-l-i-x
Carsten Agerskov
Matthew D. Scholefield
Kévin C
Augusto Monteiro
Cakeh