The new IBM Watson TTS service now provides a unique url for your TTS
API resource. This adds a configuration option to override the default
url. As the url provided by IBM contains the api_path, we check for it
and if found remove it, as the RemoteTTS.__request method appends it
when making the call.
Remove the use of the separate error message and use the wait_for_reply method
to get the converse result. The error message is left to guarantee
compatibility.
Sets up a thread with a queue interface to handle the upload of timing
information to the backend.
Previously this was done in the emitter thread after a skill or
subsystem had delivered their content to avoid slowing down the system.
However massively parallell calls like the "mycroft.stop" message would
starve the threadpool casuing the upload time to limit the execution
speed of these.
This ensures that the original context isn't modified, for example when
converse is used the reply there would switch the source/destination for
the original message making the skill speak to the wrong destination.
Added a RemoteTTSException exception for non-timeout exceptions, Mimic2
will consider HTTP status code 200-299 as OK while other responses will
trigger raise RemoteTTSException.
The hash sing makes it difficult to deal with the json file. Even using tools like [jsmin](https://github.com/tikitu/jsmin/) does not help since hashes are not supported. To enhance interoperability and consistency, they can be removed (also because only two lines use them).
This removes the whole mycroft.util.lang submodule replacing it with a
minimum needed set of functions
extract_datetime: when a date was missing en-us would return the current date
to not mess up existing skills. This is no longer the case, if an
utterance doesn't contain a date the function will return None as
indicated by the docstring
- Move to test directory
- Make runable module
- Add some waiting to give Wakeword engine time to report back
- Disable beep on found wakeword
- Update gitignore directory
The functionallity of lingua franca has been verified so the language
specific test cases aren't needed anymore. The base test cases for
english for format.py and parse.py is left as long as those remain as
part of the utils module.
The functions are still there providing the docstrings but calling the
lingua_franca functions.
Note, there is a INTENTIONAL addition of an inconsistence. Making the
extract_datetime return todays date instead of None as is documented.
This needs to be removed as soon as the mycroft default skills have been
added.
The queue will store all settingsmeta upload calls and execute them
after pairing can be verified.
Starts settings download after upload has started. When settings meta upload
has started for all skills the skill download process is triggered.
This guarantees that the settingsmeta skill_gid has been populated
properly before skills try to figure out which part of the skills data
belongs to them.
Add docstrings to upload queue
The DEBUG, has been filtered by default. This will disregard the DEBUG
filter completely since the same effect can be achieved by setting the
debug level using ":log level debug".
When mycroft_skill.find_resource fails to load a resource for self.lang
fall back to lang 'en-us'
as most skills/resources are available in english by default.
==== Fixed Issues ====
At some point during refactoring the ability to hit Ctrl+C to exit the basic CLI was lost.
The signal.SIGINT handler is needed by the graphical GUI, but it
interfered with the use of sys.stdin.readline() in the simple CLI.
Rearrange the installation of the handler for only the GUI.
To test, run:
mycroft-cli-client --simple
Then hit Ctrl+C. Before this change nothing happens (you just see the "^C" printed out). After the change it exits as expected.
If the server doesn't support getting the http head for the feed a 501
may be generated the guess_mime function will in turn fail if the url
ends with "?arg=x" or similar. This does an extra parsing step trying to
remove such arguments.
The check for "this" performed in the case of "this quarter of an hour"
was copy/pasted in a large checkin. The change it made to
daySpecified made no difference since the or in the later code
looked for an unspecified absolute hour. I can come up with no
senseable phrase that matches the conditions and couldn't even
create a nonsensical phrase that made a difference.
Since all of the existing unit tests pass without this code,
I'm going to just remove it.
Verify that skill directory exists before writing to ensure that the
shutdown method doesn't throw exception if a skill is unloaded due to
being removed.
This resolves the Runtime Warning "'mycroft.messagebus.send' found in sys.modules after import of package 'mycroft.messagebus', but prior to execution of 'mycroft.messagebus.send'; this may result in unpredictable behaviour"
Account ID will now be refetched if invalid the first time.
This also adds some general exception handling to avoid client crashing due
to the servers invalid 204 response.
- Fix: isFractional_es() parsed fractions incorrectly
- Update: earlier commit msg suggested another fix:
- Month parsing not fixed
- Several failing tests (skipped) document problem
- TODO and an issue also created
- Substantially improve parse_es.py test coverage
- TODO or comment several found bugs
- Many lines remain uncovered, incl possible bugs
If rendering a chunk of a sentence takes too long time, the audio queue
may run out and trigger the listening.
This moves the listening trigger to after the last chunk.
- Fix bug causing extractnumber_es to return a sum instead of a list
- Add Spanish parser to extract_numbers and extract_number
==== Fixed Issues ====
Closes#2310
==== Tech Notes ====
Further obscures #2056: short_scale and ordinals parameters added to the
Spanish parsers, but they don't do anything. Present for compat only.
There is a TODO for this.
==== Localization Notes ====
It's all Spanish stuff!
This will require a native Spanish speaker to analyze the relationship
between extractnumber and isFractional, and determine why certain
fractions do not parse correctly. There is a TODO for this.
- Remove unreachable conditions from parse_en:1122-1133
- input string passed through clean_string() on line 763
- articles stripped from input before line 1122
- removed conditions relied on presence of "the" in input
- Improved test coverage on parse_en.py by approx 65 statements
- Directly test certain helper functions which are difficult to
invoke indirectly
- Add tests for certain missed conditions
- Approx. 20 uncovered statements remain
Add one more optional parameter: bool clock, always produces
digital clock-like output. "0h 3m" becomes "0:03:00".
Has no effect on resolutions YEARS or DAYS, and MINUTES won't print hrs.
- Companion enum:
mycroft.util.format.TimeResolution
offers YEARS, DAYS, HOURS, MINUTES, SECONDS, or MILLISECONDS
- Will only return ms if MILLISECONDS is chosen. Default: SECONDS
- Update tests
==== Fixed Issues ====
#2333
==== Tech Notes ====
extract_datetime("day after tomorrow") attempted to check the previous
word. This resulted in an index error. Fixed by introducing a nested
condition. Seems to have solved it.
A similar bug is present in other parts of this file, such as line 843.
I have also discovered a number of other oddities, but they may be
outside the scope of this issue. At least Mycroft can now extract from
"day after tomorrow."
==== Localization Notes ====
I haven't checked parsers for any other languages.
Assign hardcoded value from MustacheDialogRenderer.render() to variable.
Fix its value to properly accommodate short dialog files.
Remove superfluous check on lines 104-105 of mycroft.dialog.init
==== Tech Notes ====
The new dialog logic, which prevents Mycroft from repeating itself, uses
a list of recently-spoken lines from the skill's .dialog files.
The size of this list is capped, currently at 3. However, .dialog files
can be shorter than 3 entries. An offset is used to accommodate this.
In the previous commit, the offset was wrong. Now it's 2.
==== Documentation Notes ====
If presented with a .dialog file containing 4 or more entries,
Mycroft will avoid saying the same one for at least a few rounds. If the
dialog file contains only 3 entries, Mycroft will avoid saying the same
line twice in a row. If only 1-2 entries are present in a .dialog file,
Mycroft will just pick one and say it, without retaining a history.
Attempt to use a different line each time a .dialog file is invoked.
==== Tech Notes ====
MustacheDialogRenderer:
- Keep a running list of recently spoken lines (up to 3)
- If there's more than one line in a dialog file, don't repeat lines
- Pop lines from the list when full or when dialog file is running low
==== Documentation Notes ====
Consider encouraging skill authors to include multi-line .dialog files,
as this will help to give Mycroft a more natural feel
This makes sure to shutdown any Timer handling settingsmetadata upload
on skill shutdown to make sure no duplicates are running.
(Easily occurs when a skill gid doesn't match the one sent in skill
manifest)
This captures any unhandled expressions and will at least make sure
they're logged properly. (Exceptions in threads are mostly not printed
to stdout).
This sleeps slightly longer than normal (10 seconds) and then tries to resume
normal operation.
This handles capture portals as well.
The standard connection logic is now
Check outside ip is reachable and after that check that www.google.com
is resolvable and connectable.
TODO: create endpoint on backend with known response to perform check
upon.
Change logic that checked for an internet connection after an issue occurred during initial boot where the old connection check gave a false positive.
Reordered imports to be PEP8 compliant. Minor refactoring to remove issues identified in PyCharm
During skill shutdown remove_all_handlers() was called on all registered
events. This would unregister all global events, such as settings
updating, stop for all skills.
Now the remove only will remove the instance connected to the skills
method.
Pulseaudio allows corking / ducking for streams with prioritized roles
This sets the role for the mycroft speech to "phone" and all other to
"music" if the config tts->pulse_duck is set to true.
- Remove once-logic from the default handler wrapper, no need to do it there.
- Add docstring for handle_wrapper
- add the on_start, on_end handlers and move skill logic to respective
handlers.
- Create separate method for regisetering mycroft system event handlers
- Fix some string concatenations
- Group acknowledge together with speak and speak_dialog
This will recurse down to MycroftSkill class but not include it. for
most skills this will mean only the methods / variables added by the
skill creator
An exception in skill's initialize method would be reraised and cause
the main loading routine to halt. This does not reraise the exception,
instead the instance reference is removed.
Split skill_manager into three separate classes, SkillManager, SkillUpdater and SkillLoader splitting the responsibility into logical units
* Split the SkillManager.__init__ code to determine the download times into a new method
* Make docstrings consistent and PEP257 compliant. Also fixed a couple of spelling errors
* fixed two issues introduced in the previous refactoring
* removed unnecessary assignment of an instance attribute to a local variable
* updated the unit test to mock out code that reaches outside of core, like MSM and the configuration manager.
* add several unittests and refactored load_priority method.
* add a test for the _get_last_modified_date function.
* add "quick" argument to docstring
* removed unused import
* new class containing the logic to periodically update/install skills and send skill manifests to the backend.
* import MsmException from where it is defined, not from the skill manager.
* add some logging to the skill updater
* remove code now in SkillUpdater from SkillManager
* added imports to __init__.py to define the API into the message bus package
* new base class for unit tests and module for reusable mocks
* new skill loader class that will replace the _load_or_reload_skill() method in the SkillManager class.
* moved skill loading logic from core.py into skill_loader.py, resulting in some refactoring of skill loader and skill manager. change unit tests to match.
* added back some spacing that was inadvertently removed.
* change skill tester to use new SkillLoader class.
* Separate reload required check from performing reload to make logic easier
to follow
* Track skills that failed to load to handle infinite loop at first load
if skill fails to load
* Allow reloading skills that has failed to load
* Simplify first load of skills
- create activate, deactivate and unload methods for skill_loader
objects
- add sanity checks before activating and deactivating skills
- Update activation/deactivation test cases
In some instances, the backend returns number setting values as an integer rather than a string.
This has been seen in the wild but has been difficult to replicate. The circumstances under which it happens are still unclear. I was able to semi-consistently have a number returned as int from the [Severe Weather Skill](https://github.com/domcross/severe-weather-information-skill) when editing an unrelated setting. To test, remove the 3 character country code (`[A-Z]{3} - `) from the service options of this skill .
This seems to be the quickest fix for it, but worth investigating further from the backend.
The pairing trigger should only be triggered by the 401 status for the
Mycroft hosted STT backends. Doing it for other STT's is only confusing.
This creates the decorator 'requires_pairing' that can be applied to the STT modules where it's relevant.
This adds the decorator to the MycroftSTT and the MycroftDeepSpeechSTT
This small change adds a check that the intent has been registered
before removing it.
If an unregistered intent was removed padatious would silently throw an
exception due to a list operation error. But when run synchronously from
the skill tester this silent exception was actually loud causing skills
to fail to load.
Major refactoring of the skills startup sequence
- Restructure to a less nested structure
- Remove usage of globals by wrapping a lot of state variables into a class this allows for things like caching a negative pairing status throughout the startup process
Refactors code so no request is sent to backend if the local identity
file is empty / has no uuid. Since if this part is missing core already
knows that it's not paired correctly.
- MycroftSkill and related core functions to mycroft_skill.py
- FallbackSkill to fallback_skill.py
- Add important classes decorators and functions to __init__.py
- move SkillGUI class to the enclosure along with the other enclosure
interfaces
core.py retains the same members as previously by means of imports to
retain backwards compatibility. When most of the available skills starts
using this new structure properly
Message bus config loading is now shared by service and client.
messagebus.client.ws file is still available in case skills are using it. It is a backport that inherits from the new MessageBusClient class. Adds depreciation warning.
- Add module docstring
- Add comment in the init class method about setting up the log level
- Add mycroft.util.log to the readthedocs documentation
- Add comment in the config about the log_level
Checks if the skills haven't been updated for more than 2 weeks or if
the last_download is undefined. last_download may be undefined if
the .msm file is missing or if the list of installed requirements is
missing so skill requirements may be missing.
This also makes sure the skills are normally loaded before the an update
attemt is tried.
Upgrade msm to 0.7.8
Using the new max_threads parameters to MycroftSkillsManager.apply() the
skill loading is reduced to 2 threads if it's during runtime while a
fast update (20 threads) are used if not all default skills are
installed to speed up getting core into a 100% operational state.
A backwards compatibility piece of code creates a hotword from the
listener settings if the listener referes to a non-existing hotword.
This makes sure the generated config doesn't affect the loaded
configuration to avoid extra reloads of precise.
The reload check is now using a more reliable method and only a subset
of the configuration. Currently set to "listener", "hotwords", "opt_in"
and "stt".
pychromecast needed to be updated to fix pip package version conflict
when using streaming google STT.
Minor issues with startup of audio needed to be handled, mainly adding
explicit waits to ensure commands would be accepted
the old "__metaclass__" has been ignored since the switch to python 3
this restores the metaclass functionality by updating it to the new
class kwarg syntax
Pulling method originally implemented in the default Mycroft skill-alarm, but useful by many for
converting a different timezone to whatever the native timezone is on the host machine.
The MycroftSkill.stop() method does not need to be implemented by the majority of skills, no reason to 'require' its implementation with an abstractmethod.
The initial framework for live streaming Speech to Text. STT derived
classes now can report that they support live streaming via their
"can_stream" property. If True, the following member functions will be
called on the object:
stream_start() - Called when a stream is starting. Should setup any
persistent state required to process speech to text.
stream_data(chunk) - Called when a chunk of audio data has been
captured.
stream_stop() - Called when a stream should be stopped. Should tear down
any state setup with stream_start(). Note that this may be called at
anytime (even if stream_start() has not been called).
Note that the execute() API is still used to get the text utterance from
the class.
A typical sequence of calls would be:
stream_start()
stream_data(...)
stream_data(...)
stream_data(...)
stream_data(...)
...
execute(...)
stream_stop()
* Handle DelayRequests when uploading blank settingsmeta
* Move Api methods to DeviceApi
This includes delete, put and get skill settings/meta
* Cache skill settings for 30 seconds
The skill settings are cached to reduce number of requests to the backend and increase skill loading speed.
* Remove _request_other_settings method
The endpoint is not used anymore
* Fix issue when the settingsmeta fails to load
Remove an old reference to "BLANK_META" left behind in the exception handler for failing to load settingsmeta.
When opt in is set and local saving of the file no mtd would be generated and the cause a undefined variable exception. This generates metadata if it's missing
* Restored ability to store wakewords locally
The code that handles the local save of wake words configuration -
"record_wake_words" - was removed some time ago. This restores that
capability.
Add the following to mycroft.conf
{
"listener": {
"record_wake_words": true
}
}
Then restart Mycroft. It should begin saving .wav files under the
/tmp/mycroft_wake_words directory.
The filename, will be filename structure is:
<accountId>_<engine>_<model>_<name>_<sessionId>_<time>.wav
* Add documentation for 'record_wake_words' and improve the doc for
'save_utterances'.
* Catch the new system.gui.user.interaction
This will in turn trigger the gui.page_interaction similar to the page flip but page_number will be None
* Fix issue when adding pages
Seems like appending would fail at times
* Add lock around namespace modifications
* Improve argument description for override_idle
* Log error when creating Padatious
* Update msm to v0.7.6
This handles connection errors during the startup procedure and improves
the skill_gid when no network is available
Padatious is registered as a FallbackSkill and was pushing settings to
the backend.
This change allows Padatious to flag to the MycroftSkill class
that no settings should be created. This is done via the new optional
MycroftSkill parameter use_settings.
When the mimic2 TTS instance is created the phrases from mycroft-core and mycroft-wifi setup (if available) is generated and stored locally (defaults to /opt/mycroft/preloaded_cache but can be changed with the mimic2 config parameter "preloaded_cache"
On startup the cache will be copied into the cache directory to speed up default interactions.
Mycroft-core won't retry for 5 minutes on a 422, 500 or 501 error.
This adds a DelayRequests Exception which will cause the settings
fetching Timer thread to sleep for 5 minutes
* Set default log level to INFO
* Restore echo functionality
- Make sure that the mycroft.debug.log message isn't removed by the
whitelist
- whitelist now only checks the beginning of a message so a whitelisted
"mycroft.audio.service" will allow messages such as
"mycroft.audio.service.play"
* Make MycroftSkill.log track root logger
modify root logger when changing log level will now change the log level
of the skill logger.
* Changes for debugging
* Switched bus logging to use INFO instead of DEBUG so it always displays when turned on
* Add exception output
* Replaced multiple calls into a msg_type variable
Kris Gesling
Åke Forslund
dalgwen
devs-mycroft
ludwhe
Chris Veilleux
jarbasal
Francesco Belvedere
andlo
Ansgar Tuemmers
Sylvia van Os
ChanceNCounter
JarbasAI
Philipp Fischbeck
ChanceNCounter
domcross
Steve Penrod
maxbachmann
Christian Clauss
Mike Woudenberg
Daniel Morse
David Wagner
Kirill Kirilenko
Andrew Kish
c0r73x
AJ Jordan
MichaIng
Chris Rogers
Joshua Watt
Aditya Mehra
Tanel.Alumae
Ruthvicp
Matthew D. Scholefield
Kate Canaveral
Thomas Doczkal
Kris Gesling