pychromecast needed to be updated to fix pip package version conflict
when using streaming google STT.
Minor issues with startup of audio needed to be handled, mainly adding
explicit waits to ensure commands would be accepted
the old "__metaclass__" has been ignored since the switch to python 3
this restores the metaclass functionality by updating it to the new
class kwarg syntax
Pulling method originally implemented in the default Mycroft skill-alarm, but useful by many for
converting a different timezone to whatever the native timezone is on the host machine.
The MycroftSkill.stop() method does not need to be implemented by the majority of skills, no reason to 'require' its implementation with an abstractmethod.
The initial framework for live streaming Speech to Text. STT derived
classes now can report that they support live streaming via their
"can_stream" property. If True, the following member functions will be
called on the object:
stream_start() - Called when a stream is starting. Should setup any
persistent state required to process speech to text.
stream_data(chunk) - Called when a chunk of audio data has been
captured.
stream_stop() - Called when a stream should be stopped. Should tear down
any state setup with stream_start(). Note that this may be called at
anytime (even if stream_start() has not been called).
Note that the execute() API is still used to get the text utterance from
the class.
A typical sequence of calls would be:
stream_start()
stream_data(...)
stream_data(...)
stream_data(...)
stream_data(...)
...
execute(...)
stream_stop()
* Handle DelayRequests when uploading blank settingsmeta
* Move Api methods to DeviceApi
This includes delete, put and get skill settings/meta
* Cache skill settings for 30 seconds
The skill settings are cached to reduce number of requests to the backend and increase skill loading speed.
* Remove _request_other_settings method
The endpoint is not used anymore
* Fix issue when the settingsmeta fails to load
Remove an old reference to "BLANK_META" left behind in the exception handler for failing to load settingsmeta.
When opt in is set and local saving of the file no mtd would be generated and the cause a undefined variable exception. This generates metadata if it's missing
* Restored ability to store wakewords locally
The code that handles the local save of wake words configuration -
"record_wake_words" - was removed some time ago. This restores that
capability.
Add the following to mycroft.conf
{
"listener": {
"record_wake_words": true
}
}
Then restart Mycroft. It should begin saving .wav files under the
/tmp/mycroft_wake_words directory.
The filename, will be filename structure is:
<accountId>_<engine>_<model>_<name>_<sessionId>_<time>.wav
* Add documentation for 'record_wake_words' and improve the doc for
'save_utterances'.
* Catch the new system.gui.user.interaction
This will in turn trigger the gui.page_interaction similar to the page flip but page_number will be None
* Fix issue when adding pages
Seems like appending would fail at times
* Add lock around namespace modifications
* Improve argument description for override_idle
* Log error when creating Padatious
* Update msm to v0.7.6
This handles connection errors during the startup procedure and improves
the skill_gid when no network is available
Padatious is registered as a FallbackSkill and was pushing settings to
the backend.
This change allows Padatious to flag to the MycroftSkill class
that no settings should be created. This is done via the new optional
MycroftSkill parameter use_settings.
When the mimic2 TTS instance is created the phrases from mycroft-core and mycroft-wifi setup (if available) is generated and stored locally (defaults to /opt/mycroft/preloaded_cache but can be changed with the mimic2 config parameter "preloaded_cache"
On startup the cache will be copied into the cache directory to speed up default interactions.
Mycroft-core won't retry for 5 minutes on a 422, 500 or 501 error.
This adds a DelayRequests Exception which will cause the settings
fetching Timer thread to sleep for 5 minutes
* Set default log level to INFO
* Restore echo functionality
- Make sure that the mycroft.debug.log message isn't removed by the
whitelist
- whitelist now only checks the beginning of a message so a whitelisted
"mycroft.audio.service" will allow messages such as
"mycroft.audio.service.play"
* Make MycroftSkill.log track root logger
modify root logger when changing log level will now change the log level
of the skill logger.
* Changes for debugging
* Switched bus logging to use INFO instead of DEBUG so it always displays when turned on
* Add exception output
* Replaced multiple calls into a msg_type variable
The simple audio service can be paused. The audio is "paused" but placing the
audio playback mechanism (usually aplay or mpg123) into the background, kinda
like hitting Ctrl+Z. While in the background that process can no longer react
to the SIGTERM signal. So paused audio has to be brought to the foreground
using SIGCONT, and _then_ sent SIGTERM.
Also added a belt to go with these suspenders and perform a process.kill() if
the audio process doesn't go away after 1 second.
The message returned is a dict of all available audio backends as keys
containing 'supported_uris' listing supported uri types (file:// http://
etc.), 'remote' (bool) True if remote), 'default' (bool) true if the
default audio backend.
Revert "Fix a couple of minor issues intruduced by skill_gid (#2079)"
This reverts commit e046377ce1.
Revert "Merge pull request #2075 from forslund/bugfix/msm_wrapper-license"
This reverts commit 18cfbce0ca, reversing
changes made to 82fa314ce9.
Revert "Feature/skillsmeta gid (#2074)"
This reverts commit 82fa314ce9.
* Fix skill_gid for modified skills
Used the old '_' after device uuid
* Identify skill using gid
When receiving data from server the "identifier" entry is now removed and in it's stead the skill_gid should be used.
This also removes the "fetch other settings" functionallity since that is no longer relevant.
* Add global id basics to settings meta
- All skills will upload a blank settingsmeta
- a skill_gid will be appended to all settingsmeta upload-data
- Added basic function for generating skill_gid
* Use new skill_gid field.
Populate skill_gid directly from metadata
* Separate travis tmp-dirs
- Update travis script to use tempdir for each python version
- Update test script to handle nonstandard tempdirs
- Generate msm folder using tempdir when running create_msm test
* Add title field with pretty name
* Collect and expand "title" as needed
For title use market-place title or name in settings meta or skillname
* Switch skill_manager create_msm test to 19.02
* Remove leading / trailing Skill in display name
Also rename title displayname to match new mycroft-skills-data
* Lock msm_create and mock the name info test_settings
If no match was found for the non-normalized utterance it would jump to
the exception handler for StopIteration skipping the normalized step
altogether
This changes the next/StopIteration system for list comprehensions
Previously Padatious intent matches were performed on non-normalized text, meaning that things like "what's the weather" wouldn't match a Padatious intent but
"what is the weather" would.
The "utterance" in Adapt intent data will still be non-normalized even if the intent match occurred on a normalized utterance. Retaining the existing behavior.
The "intent_failure" data. In there, "utterance" is always the raw version, "norm_utt" is the normalized one.
Also added better debugging info for intent matching to the log.
Also addresses a rare issue with the old code where the Adapt context could
have been updated even if the Adapt intent wasn't actually invoked due to
a higher Padatious intent match.
* Add the SkillGUI method send_event()
Sends raw mycroft.events.triggered messages to the gui for the skills namespace.
Example usage:
self.gui.send_event('event_name' {'param1': 12, 'param2': 'abc'})
* Fix mimic 2 long sentences
Fixes bug in the second and third chunking pass incorrectly by
concatinating strings with lists resulting in chunks of single
characters.
* Handle mimic2 chunking correctly
- Move preprocessing from get_tts() to a method called from tts execute,
this allows all parts to be spoken and the caching to work correctly
- Remove duplicate of phonetic spelling in mimic2_tts
The loose (conf > 0.5) Padatious match was previously occurring as Fallback
priority 99. The AIML fallback at priority 90 would consume lots of
utterances, interferring with many skills. Now Padatious runs at priority
89.
Additionally, added documentation of the intent and fallback system, including
guidelines for priorities.
Much of the code used "en-us" as the default value when not specified.
This limited the internationalization potential. Changing the default
to None and adds the ability to define the default lang code from other
locations in code. E.g.
```python
from mycroft.util.lang import set_default_lang
set_default_lang("en-us")
print("English date: "+nice_date(dt))
set_default_lang("de-de")
print("German date: "+nice_date(dt))
```
This allows easier localization of Skills by having the framework set the default without any changes necessary by the Skill writers.
Other minor changes:
* Changed the default return value of get_gender*() to None instead of False
* Split ADJUST into ADJUST and SET
* Use enum names instead of values.
* Add properties to CommonIoTSkill
* property -> attribute to avoid conflict with builtin
* Add increase and decrease actions
* Fix copy/paste error
The restore_volume would previously unpause the service even if the user just asked for pausing playback.
This separates the paused flag from the pausing/resuming action so lowering/restoring volume doesn't affect the logical pausing state and can check it to determine if lowering/restoring is needed/wanted.
This adds a very simple mechanism for pausing and resuming audio streams
that are actively playing. The process is suspended and resumed when
requested.
This also uses that same method to implement lower_volume()/restore_volume() for simple "ducking" of audio.
Mimic2 voices has several issues:
* Numeric generation was all goofed up for decimal numbers. It was
independently doing number to speech for the values before and after
the period on a decimal number. E.g. 1.100 became "one.one hundred".
Removed local normalization altogether since it is handled
correctly on the Mimic2 server now.
* Sentence splitting was being applied in the middle of things like
numbers or abbreviations. E.g. "I.B.M."
* Chunker algorithms were making unnecessary copies of string arrays
While in there I also:
* Removed several unused imports
* Cleaned up docstrings
* Prefixed private helpers with _
* Added debug logging of mimic2 request text
Padatious was accepting fairly low confidence matches (0.5). This
would match a phrase such as "Where is the Empire State Building?" to the
intent "Where were you born?" (Which is a 0.54 match)
Now there are two passes mad on the Padatious fallback. The first is
looking for high confidence matches (> 0.8). Then other fallbacks -- such
as the CommonQuery fallbacks -- have an opportunity to consume the
utterance. If they don't consume it, Padatious is invoked again with
looser confidence before finally invoking the Unknown fallback.
This commit add the initial translations of core functions for format
numbers: nice_number_es, pronounce_number_es and nice_time_es.
==== Localization Notes ====
NONE - Castillian (Spain's spanish)
Now the eyes go to a khaki color and "< < < LOADING < < <" displays
during the skill startup. When Padatious finishes the first training
we now emit a "mycroft.ready" on the messagebus.
A change to the skill_mark_1 to look for "mycroft.ready" instead of
"mycroft.skills.initialized" provides a good visual change to show that
it is ready for use.
The deprecation warning was firing off at the load of every skill when
the decorators are being iterated through by internal code (which
includes the self.config with an @property). Add check for these
special cases to not show a warning.
The scoring for CommonPlay would favor skills that responded with overly long
matches. E.g. "Huey Lewis and the News" would be considered a better fit than
just "News" for the request "Play the News"
* Deprecate self.config in skills
Skills should contain their own settings, the self.config concept is being
deprecated.
Also removed the defaults set for several old MycroftAI skills. The 19.02
version of these Skills initializes the default values using:
```python
self.settings["key"] = default
```
* Update padatious config to work with the config property.
* Run emitter using threadpool
This moves the thread pool from the websocket client into the eventemitter allowing each registered function to run in a separate thread and not just each event.
This speeds up cases where there is a one to many call such as the common play framework and the upcomming common query framework.
* Add unit tests for threaded event emitter
* Add standard header
* Add standard header
* Add base common_iot_skill
* can_run takes IoTRequest
* Minor cleanup + documentation
* Fix pep8 issues
* Adjust scene and entity registration.
The controller skill is not guaranteed to be alive before
CommonIoTSkills, so we must call for values when it is alive, in
addition to accepting them at any time from other skills.
* Safer parsing
* Address PR feedback
* Add skill_id to register message
* Minor docstring edits
* date and time - for tonight, weekdays
* Updating the previous commit test_extractdatetime_en
* Editing comments for extract_date_time_en
* Generalized as a marker
Expanded this from just handling "weekends" to any day or plural of the day, "weekend", "weekday" or "weekdays".
* Add remove page methods
* Implement SkillGUI.clear()
The method will now remove the namespace entirely from the gui.
This adds the message gui.clear.namespace
* Remove debug prints from SkillGUI
* Correcting docstring
* More docstring changes
* Remove whitespace added by Github webUI
* Attempt to create skill directory if not existing
* Handle missing priority skills
* Minor update of comments
* Handle skill load exception
Make sure an exception while trying to load/reload skill doesn't shutdown thread.
* Handle MsmException during SkillManager creation
If SkillManager can't be created due to an MsmException wait for network connection and retry.
* Update immediately if skill install file is missing
Missing skill install file indicates that this is a new venv and the requirements of the skills will need to be reinstalled.
* Add basic test for skill_manager
Basically only creating the skill_manager but it ensures that msm can be used on all supported python versions
* Update to pyee 5.0.0
- Update requirement
- Make the SkillSettings class hashable
* Update adapt to 0.3.2
* Upgrade websocket-client
* Remove special threading for common query
This ensures no attributes are None and simplifies some checks within the function
Otherwise, if, for instance, there were no 'data' field in a message, when deserializing, the caller would have to check for None
This sends a new system.update.check message when the device is booted and not paired
It is not sent every boot because the command could upgrade across major versions which the user could not want.
- Create a read_vocab_file() function that normal vocab loading and voc_match both uses. This function handles blank lines and comments
- Use a simpler regex instead of word logic to match
- Add a couple of test cases for the method
Several changes to the logging using in all processes:
* Disable logging of bus messages by default (massively cleans up the logs)
* Add "mycroft.debug.log" bus message. It supports the data={"bus": True/False} as well as
data={"log": LEVEL}, where level is a Python logging level.
CLI now supports several commands:
* ```:log level XXX``` where XXX is the name of a standard Python level (e.g. 'debug')
* ```:log bus on``` or ```:log bus off```
* Removed the requirement for "log" in ```:clear log```. ```:clear``` works now.
The mic-level meter that appears in the Mycroft CLI could stop updating in
rare cases when a file I/O operation failed. It would also fail to show
a meter ever if the mic_level file hadn't been created before the CLI was
started.
Previously the voc_match() method had several problems:
* Blank lines in the .voc file would cause it to match all strings
* Comments weren't ignored
* Substrings matched so "book" would match "ok", for instance
Ambiguous times previously used a generally unexpected rule about
when to jump 12 hours when parsing times. Now it follows the rule:
If a time is spoken without am/pm indicator, assume the next time
that hasn't passed was intended.
For example:
"at 7 o'clock"
Would mean 7:00am if spoken at 6:59am, but would mean 7:00pm
if spoken at 7:01am.
* Remove the old single viseme message
Instead a single viseme message is sent
* Correct the spelling of viseme.
Ref: https://en.wikipedia.org/wiki/Viseme
* Remove debug print.
This was probably left in by mistake, removing to clean up the audio log somewhat.
In the case where a network call during the initialization of the
settings poll fails the first time, it would never be tried again.
Now it will retry initialization once a minute.
The settings code worked, but was noisy and generally messy about
a few exceptional but common situations:
* When the .mycroft/skills/<SkillName> folder didn't already exist
* When network timeouts and such occcurred
I also slipped in a couple trivial code cleanups for an unused variable
and a log message.
* New formatters: nice_duration() and join_list()
Adding two new formatting functions:
* nice_duration(duration, lang="en-us", speech=True)
Accept seconds or duration and produce a nice sounding duration.
Example: nice_duration(61) == "one minute one second"
nice_duration(61, speech=False) == "1:01"
* join_list(items, connector, sep=None, lang="en-us")
Example: join_list(["a", "b", "c"], "and") == "a, b and c"
This includes a translation helper that uses text files in the
mycroft/res/text/LANG/ directory, such as "second.word".
This extracts the logging logic that was being used in a few places to
indicate certain parcing functions are not supported in particular
languages, and adds the logging to extract_duration.
- Move "save_utterenaces" comments to correct block
- Correct information, records utterance not wakeword
- replace "record_utterances" with the new "save_utterances"
Change to match the documented and more intuitive name "save_utterances",
but add backwards compatibility code to support the original
"record_utterances".
Making things weak private, to limit surface area of support. As things
become increasingly stable/tested/useful, it may make sense to open them
up, but for now, keeping them private will limit risk.
This is in support of issue-1959.
After many changes, things had gotten a little disorganized, and the
docs were a little out of date. This brings them up to date.
This is in support of issues-1959.
This improves the utility of the _ReplaceableNumber class, and updates
most of the number parsing functions to take tokens rather than text.
This simplifies the interactions between many of the functions, as there
is no need to convert back and forth between text and tokens.
This also adds some tests. Note that there are a few regressions that
will be fixed in a subsequent commit.
This was calling convert_words_to_numbers and parsing out the resuling
numbers, which was a simple way of getting the numebrs in order, but it
choked on anything that didn't match the regex being used to parse
numbers, in particular numbers of the form '6e18'. The better solution
is to directly use extract_numbers_with_text (which now sorts by
start_index) and get the values from there directly.
This is in support of issues-1959.
"Five hours seven and a half minutes" was parsing as 5.5. This is
resolved. Multiple fractions/decimals still cause problems, e.g.
convert_words_to_numbers("seven and a half and nine and a half")
Out[5]: '7 and a 0.5 and 9 and a 0.5'
This is in support of issues-1959.
* Add tests for DialogLoader
* Handle Path/PosixPath
LOG messages when files/directories were missing would fail when a PosixPath/Path object was sent to as argument. This uses format to get the correct string representation.
* Add test for dialog.get()
Slight refactoring to accommodate for this in a nice way. Created
function connect_to_mycroft() handling fetching the config and
connecting to the mycroft messagebus since these are related and the
order is important to maintain for it to work.
* Refactor mimic2 to use the shared tts architecture
* Make sure the queue is cleared
- Add a convenience method grouping clear_queue and clear_visemes
- The start time is now set before the lock to allow multiple speech requests queued before the stop signal to also be cancelled
- Make sure the any pending TTS generation is cleared from the queue by calling tts.clear() when breaking from the chunking loop.
To simplify the process of adding an idle page to a skill the decorator "resting_screen_handler" was added. In a skill class the decorator can be applied to a method to register it to handle idle.
@resting_page_handler("My Idle Page")
def handler(self, message):
...
The decorator will Register the method with the Mark-2 skill and perform all communications needed to make it work smoothly.
* Add new api command to send visemes as single list. This allows more efficient use of the messagebus and gives implementors flexibility in how they handle the visualization.
* Switch mark1 to use viseme_list
The wrong method was registered, instead of the wrapped function call
the original method was registered. This led to not being able to
unregister fallbacks.
A small bug caused things like "two hundred twenty" to return only the
"hundred tenty" for the text. This has been fixed.
extract_numbers_with_text was updated to deal with the new return types
of the functions it depends on. Specifically, it accounts for the start
and end index values.
This is in support of issues-1959.
Previously it was assumed that the orgiginal text would be enough to
determine where in a string a number should go, however, in some
scenarios, that does not work, and results in the wrong values being
parsed.
A different, and smarter approach is being taken now, in which the
original string is initially split into a list of tuples of
(index, word) where index is the index of the word within the string.
All subsequent processing is done on these tuples, meaning we always
know exactly where the words were in the orginal string. This should
make text replacement perfect, as we can always sub out the exact,
correct words, based on their indicies.
extract_number_with_text_en now returns the number parsed, the text that
represents the number, the start index, and the end index.
Things are not yet working perfectly. Here is roughly the current state
of the world:
from mycroft.util.lang.parse_en import *
extract_number_with_text_en("this is some two hundred thousand twenty
two hours")
Out[3]: (200022, 'hundred thousand twenty two', 4, 7)
extract_number_with_text_en("this is some twenty two hours")
Out[4]: (22, 'twenty two', 3, 4)
extract_number_with_text_en("this is some twenty hours")
Out[5]: (20, 'twenty', 3, 3)
extract_number_with_text_en("this is some two and a half hours")
Out[6]: (2, 'two', 3, 3)
extract_number_with_text_en("this is some two point five hours")
Out[7]: (2, 'two', 3, 3)
The list of tuples is a bit of a hassle to deal with. In a future
commite the will be replaced with dictionaries, or even better, Token
objects, that contain the word and it's index. This would make the
code easier to reason about (removing lots things like words[0][1]
which has no meaning without deep understanding of the code).
This is in support of issues-1959.
Articles (a, an, the) that appeared immediately before the number were
included in the returned text. This fixes those issues.
The solution isn't super clean - it craetes a new function to wrap the
old one (unavoidable, since the articles can be needed for fractions),
and splits the returned string, the strips leading artivles.
Since strings are immutable, this is probably not super efficient. Might
be better to eventually use lists as much as possible, and only create a
string at the end (though the lists will come with their own problems,
so that could turn out to be a wash). In any case, this works for now.
Issues fixed:
Lists, e.g. "some words one two three" would return (3, "one two three")
Negaitve words were not included in output, e.g. "negative five" would
return (-5, "five").
This is in support of issues-1959.
Any articles (a, an, the) appearing before the number would be included
in the text, e.g. "set a timer for eight minutes" returned:
(8, 'a eight')
This fix clears the number words list if no number words have been
found. Note that articles can't simple be filtered, as they are
often a part of the numebr (e.g. three and a half).
This is in support of issues-1959.
Methods implemented include:
extract_number_with_text
extract_numbers_with_text
convert_words_to_numbers
extract_duration
This is in support of issues-1959. This continues the work of
returning the relevant text that corresponds to a number
parsed from a string.
This is in support of issues-1959. This begins the work of returning the
relevant text that corresponds to a number parsed from a string. Here's
a sample showing the basic functionality/state of the world (showing
some of the remaining cases to handle)
>>> from mycroft.util.lang.parse_en import *
>>> extractnumber_en_with_text("three hours twenty minutes")
(3, 'three')
>>> extractnumber_en_with_text("twenty minutes")
(20, 'twenty')
>>> extractnumber_en_with_text("twenty five minutes")
(25, 'twenty five')
>>> extractnumber_en_with_text("two hundred twenty five minutes")
(225, 'two hundred twenty five')
>>> extractnumber_en_with_text("three and a half minutes")
(3.5, 'three and a half minutes')
>>> extractnumber_en_with_text("three point five minutes")
(3.5, 'three point five minutes')
==== Tech Notes ====
Checks if the word being parsed is relevant to number parsing. If it is
not, and we have already found number words, we return what we have. If
it is, we add it to a list of words representing the current number
being parsed.
==== Documentation Notes ====
The old implementation of extractnumber_en seems to generally return the
last number in the text. This change will cause the first number to be
returned.
This is part of a refactor of extractnumber_en, with the ultimate
goal of making it easier to maintain and extend (should also
improve perf). This is in support of issues-1959.
This is part of a refactor of extractnumber_en, with the ultimate
goal of making it easier to maintain and extend (should also
improve perf). This is in support of issues-1959.
All tests (minus extract_duration, which has not yet been implemented)
are passing at this stage.
This is part of a refactor of extractnumber_en, with the ultimate
goal of making it easier to maintain and extend (should also
improve perf). This is in support of issues-1959.
All tests (minus extract_duration, which has not yet been implemented)
are passing at this stage.
Audiotest now prints the device and samplerate used as well as the commandline used to playback the audio for easier debugging if things doesn't work as intended.
This is part of a refactor of extractnumber_en, with the ultimate goal
of making it easier to maintain and extend (should also improve perf).
This is in support of issues-1959.
Begins a refactor of extractnumber_en, with the ultimate goal of making
it easier to maintain and extend (should also improve perf). This is
in support of issues-1959.
* Add communication from GUI to skills
- "set" events from Qt will set/update a variable in the skills .gui member
- It's possible to add general event handlers using self.gui.register_handler()
- Moved registration of skill_id to just after skill init
* Ensure that simultaneously writes doesn't occur
Wrap WebSocketHandler.write_message() with a lock in an attempt to handle "buffererror: existing exports of data: object cannot be re-sized."
* Add better logging to help debug disconnect issue
* Allow overriding the idle page
SkillGUI.show_page() and SkillGUI.show_pages() now takes an optional
override_idle parameter. This is used as a hint by the mark-2 skill
and if possible the idle screen will not be shown.
* Improve debugging using Logger
* Raise exception when sending a non-existing gui page
* Restore running state to new connections
When a GUI is connected data and running namespaces are synchronised and
shown.
This refactors the code quite a bit moving the GUI state from the GUIConnection
object to the Enclosure.
The GUIConnection object does the handles the sync in the on_connection_open()
method.
* Add gui.page_interaction message
Currently triggered on page change on the display.
* Handle message when gui changes sessionData
* Check if socket exists on gui before sending data
* Increase port on each failure and retry
- Adds RemoteAudioBackend class, which is used to differentiate between
Remotes and local audio backends
- When searching for audio backends the backends are storted to first
check local ones and secondly remote ones
If we are missing the ".mycroft_cli.conf" file we print a message to the
user informing them that we are ignoring the missing file along with a stack trace. The stack trace is really not necessary so this removes it.
Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
To avoid UnicodeDecodeError's when opening files tell Python that we
want to open the file as a utf8 encoded file.
Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
Ctrl+C was not being properly handled by the CLI, resulting in an unclean
shutdown and other unexpected consequences -- e.g. a
"./start-mycroft.sh debug" would kill the background processes on exit. The
KeyboardException was never being triggered.
Now the SIGINT is being captured and Ctrl+C behaves (as intended) just like
pressing Ctrl+X.
This moves the thread pool from the websocket client into the eventemitter allowing each registered function to run in a separate thread and not just each event.
This speeds up cases where there is a one to many call such as the common play framework and the upcomming common query framework.
The audioservice can now jump forward and backward in the audio stream/file
The functionality is accessed via the audioservice class's seek_forward(),
seek_backward() and seek() methods
* Try to improve stability
- Remove sync_active
- Update the way variables are sent to the gui
* Do not show full path for pages
The cli now shows the basename of the current page to make it easier to
determine if the correct page is displayed.
* Make elements in loaded list named tuples
This makes the intent of the code a bit cleaner.
Normally Mycroft will use the default PortAudio input device as the
microphone input for the user. However in some cases there is reason
to specify a different input.
The "device_index" under the "listener" section in mycroft.conf has
always allowed a user to select the microphone explicitly. But on
some systems the indices can change from reboot to reboot. So this
adds the "device_name" setting. If it exists (and the "device_index"
has not been specified explicitly), a regex match will be run against
the PortAudio device names.
When "device_name" is used, the voice.log will contain a listing of
the devices as they are tested. This can be used to debug if a
name isn't matching as expected.
EXAMPLES:
/etc/mycroft/mycroft.conf
```
{
"listener": {
"device_name": "aawsrc"
}
}
```
Would find a match against "aawsrc" or "aawsrcplug". To force a specific
match, you can use a regex such as "aawsrc$".
/etc/mycroft/mycroft.conf
```
{
"listener": {
"device_index": 2
}
}
```
The PortAudio device index specified will be used.
Names and indexes for PortAudio are difficult to guess. The simplest way to
view them is either enter a value for "device_name" and look at the names
which appear in the log when starting Mycroft, or to run a simple program
such as:
```python
import pyaudio
pa = pyaudio.PyAudio()
for i in range(pa.get_device_count()):
dev = pa.get_device_info_by_index(i)
print((i, dev['name'], dev['maxInputChannels']))
```
* Fix mimic2 negative numbers
Make the regex extracting numbers also match negative numbers when preparsing phrases sent to the mimic2 service
* Update pronounce_number to use "minus" for negatives
After discussion in the chat it was suggested to use "minus" for negatives as default.
When scientific notation is used the term "negative " is still used.
Several additions to the GUI protocol support
These changes allow switching between pages successfully with the current
mycroft-gui widget:
* Optimized commands to handle the active skill list
* MycroftSkill.gui.show_pages(list, idx) allows multiple-pages to be displayed
at a time starting with the given index visible.
* Merge SkillGUI.show_page with show_pages
This limits code duplication and makes things a bit more maintainable.
* Do not reload on changed .qmlc files
* Make EnclosureGeneric derive from Enclosure
* Update show function to match mycroft-gui-app
- adds internal representation of all loaded skills
- uses new commands to switch between pages and namespaces
* Add Extra debug output in enclosure
- Log if starting websocket fails
- Log the sending of page info in more detail
* Update GUI Debug client in CLI
- The CLI GUI now handles the new messages for switching pages
- Handle different data types better by using format instead of string concatenation
* Disable syncing code.
The sync code at startup outdated and needs to be reworked. Disabling it for now
to allow better interaction.
* Minor cleanups
- do not inherit from object
- use format instead of string concatenations
- remove duplicated self.loaded
- correct private member access
* Refactor GUIConnection.show()
Move the actions into separate methods for better overview of the logic
* Flipped "valid_file" to become "ignored_file"
The ability to "barge-in" has been lacking from Mycroft Core. The Mark 1
microphone was unable to support this due to physical limitations, but the
Mark II and other implementations with more advanced mic tech which can
hear over themselves are able to continuously listen.
To enable this while retaining backwards compatibility with simpler mic.py
systems, there are now two mycroft.conf values:
{
"listener": {
"mute_during_output" : true,
"duck_while_listening" : 0.3
}
}
The above values are defaults, and implementers will likely override them
using the /etc/mycroft/mycroft.conf file when appropriate.
The duck_while_listening setting is currently handled in Mycroft's
skill-volume. The mute_during_output is handled within mycroft-core itself.
- Add overridable stop() method to HotwordEngine Class
- Add stop implementation to precise shutting down the runner
- Call wakeword_recognizer.stop() before reloading the listener configuration
Several small changes based on the code review feedback:
* Drop '_' from classes like Enclosure_Mark1
* Adopt Python 3 style for class definitions and don't explicitly list '(object)'
* Slightly better documentation
* Moved MycroftSkill.show_html() to SkillGUI, resulting in code like self.gui.show_page('Weather.qml')
* Renamed SkillGUI.__dict to SkillGUI.__session_data. This better reflects the
how values are accessed in the QML.
Adds support for negotiating best answer for a questions
Currently three levels of confidence are defined
EXACT: If the query could be identified exactly and a response is returned.
Example: The cockail skill could find a cocktail in the query that exists in
it's database.
CATEGORY: A category of questions the skill handles could be identified.
Example: The wiki-data skill can identify that the question is regarding
A date of birth and finds an answer.
GENERAL: A general question and answer service could parse the question.
Example: The wolfram alpha skill got a match for "How tall is Abraham
Lincoln".
The QML files are typically not translated like other Mycroft resources,
they have internal translation tools. And at times there are other
resources that don't need to be translated all the time, for example
color strings like "AliceBlue" which might be used by non-English
speakers.
So now the translation mechanism looks for resource file X as follows:
1) Look for <res_dir>/<lang>/X
2) Look for <res_dir>/X
3) Look for anywhere under locale/<lang>/.../X
And now the show_page() method starts looking for resources under the skill/ui folder.
is kept in order, but old namespaces/skills are never culled.
* The active namespace list is synced on a GUIConnection level
* QML display requests now are sent as a list instead of a single entry, i.e.
with "gui_urls" instead of "gui_url". Currently a skill can only send a single
QML, however.
* Change CLI GUI client to handle "gui_urls" instead of "gui_url"
Make sure the older version of the install skill will still work. The SettingsManager.load/write_skills_data() will return the version 0 format of the skills_data and the write_skills_data() will convert to the version 1 format before doing the write.
- Lock msm when doing an update
- Add device enpoint for uploading skills.json
- Add configuration value for skill store updates
- Upload skills manifest after update
The GUI client built-in to the CLI now has these features:
* Activate/deactivate via Ctrl+G
* GUI 'window' shows active page title and all namespace variables
* Add fool-proof primitive draw(x,y, msg) that takes care of clipping, and padding
Further fleshing out of the GUI mechanisms
* Support for data and page from Mycroft -> GUIConnection
* Add a 'reconnecting' event for the messagebus
* Add MycroftSkill.show_url()
* Plumb MycroftSkill.gui into the messagebus
* Implement MycroftSkill.gui dictionary
CLI extensions for the GUI:
* Can now act as a simple GUIConnection
* Minor revamp of messagebus connection, provides kinder handling when
messagebus isn't found or ready.
* BUGFIX: An empty filter would filter ALL messages
* BUGFIX: Input wider than the screen would cause a crash
* BUGFIX: "filter" or "find" with no param would filer "filter" or find "find"
add support for decades, centuries, millemniums
add support for "within the hour", "in a second/minute",
add support for "a couple time_unit" and "a couple of time_unit"
The common play skill sends a stop message in the CPS_start(), if the play message is sent too close to the stop message it may be executed before the stop message causing playback to immediately stop.
To circumvent this a 1 second stop ignore time is added.
Still very much a work in progress.
For understand and testing, here is the sequence:
STEP 1: GUI announces itself
* Connect to the main Mycroft messagebus
* Send: "mycroft.gui.connected" with data { "gui_id": XXX } where XXX is a uniq ID (uuid)
STEP 2: Mycroft creates GUI socket
* Mycroft extracts the gui_id
* Mycroft prepares a socket and announces its availability on the Mycroft messagebus with:
self.bus.emit(Message("mycroft.gui.port",
{"port": self.GUIs[gui_id].port,
"gui_id": gui_id}))
STEP 3: GUI connects
In python, a very minimal test socket handler on the GUI side would look like this
from websocket import create_connection
port = 18181 (from the message above)
ws = create_connection("ws://0.0.0.0:"+port+"/gui")
ws.send("Hello, World")
print("Sent")
print("Receiving...")
result = ws.recv()
print("Received '%s'" % result)
ws.close()
Enclosures
* Create a mechanism to instantiate unique Enclosure classes, depending on the platform found in the SYSTEM mycroft.conf
* Implement a generic Enclosure, which support the new GUI protocol
* Implement a Mark 1 Enclosure (expects the serial connection to an Arduino)
* Implement the start of a Mark II enclosure
* Implement a generic enclosure (no screen)
* Implement the GUI announcement and protocol basics
MycroftSkill
* Implement the basis of the GUI-controlling interfaces. Namely:
- MycroftSkill.show_text()
- MycroftSkill.show_image()
- MycroftSkill.show_html()
- MycroftSkill.show_page()
- MycroftSkill.gui to set values for page displays.
Configuration
* Add "gui_websocket" to the mycroft.config.py
The registered intents are now stored in a list. When a detach_skill message is received the list is checked for matching intents and the intents are removed
The CommonPlaySkill base class provides an easy base class for any
skill wishing to use the "Common Play" framework. This allows multiple
skills to jointly handle requests such as "play Janet Joplin",
"play my Sled Zepplin playlist", "play NPS news" or "play Strump's
speech to the UN". Previously the "wildcard" intents needed to handle
this were basically impossible -- only one skill got a shot at handling
the request. Now several skills to search their service to see if they
have anything that can service the request. The service with which
reports the highest confidence gets invoked.
The CommonPlaySkill makes it easy to implement this. Simply derive a
skill from CommonPlaySkill (instead of MycroftSkill) and override
the two required methods CPS__match_query_phrase() and CPS__start().
The skill can then use self.CPS__play(url) to begin playback, or invoke
a unique player to interact with a custom service.
Fix trillion being saved with wrong number (10e10 instead of 10e12)
==== Fixed Issues ====
1718
==== Tech Notes ====
NONE - explain new algorithms in detail, tool changes, etc.
==== Documentation Notes ====
NONE - description of a new feature or notes on behavior changes
==== Localization Notes ====
NONE - point to new strings, language specific functions, etc.
==== Environment Notes ====
NONE - new package requirements, new files being written to disk, etc.
==== Protocol Notes ====
NONE - message types added or changed, new signals, APIs, etc.
==== Fixed Issues ====
==== Tech Notes ====
NONE - explain new algorithms in detail, tool changes, etc.
==== Documentation Notes ====
NONE - description of a new feature or notes on behavior changes
==== Localization Notes ====
NONE - point to new strings, language specific functions, etc.
==== Environment Notes ====
NONE - new package requirements, new files being written to disk, etc.
==== Protocol Notes ====
NONE - message types added or changed, new signals, APIs, etc.
This makes a cross context call be treated as one level when calculating the probability. this makes previous contexes not be completely invalidated when a cross context call is sent.
The delay while loading files have caused the Mark-1 Button to respond very slowly. This moves the sleep out of the loading section and is handled by the token refresh instead.
* Make the combo lock handle multiple users
On the Mark-1 the locking would fail since the first time the lock is
accessed the process is run under root (script checking the enclosure
version). This caused the locking to fail since the normal mycroft
processes got an access error (lock-file owned by root)
This change creates lock-files and sets the permissions to 0o777
if the file doesn't exist so it's accessible for all users
Also added an Apache license header
* Add protection for naive skill authors
It is fairly common for new skill authors to attempt actions in the __init__()
method which are not legal yet, as the Skill has not been fully connected to
the Mycroft system. This adds an @property protection layer for the two most common
issues:
* Accessing MycroftSkill.bus
* Accessing MycroftSkill.enclosure
Now those are properties instead of variables and provide appropriate warnings
when used before they exist.
Also enhancing the handling of error logs in the CLI to highlight problems such
as this:
* Color "- ERROR -" log messages in red
* Retaining leading characters from log messages, improving readability in formatted messages
The test for today/tomorrow/yesterday was only looking at the day,
not the month and year. So on July 14, 2018 it would claim that
the date June 14 2018 and July 14 2020 are all "today".
Now the full date is used in the comparison
* Fix Ctrl+C and Ctrl+X in CLI
There were several quirks in the shutdown of the CLI client:
* Consumed Ctrl+C wasn't being handled cleanly
* main() got called twice, requiring two Ctrl+Cs to exit
This eliminates a lot of the noise in the log files. Later I'll add features in the CLI to
assist watching the messagebus messages rather than writing them all to logs.
Also corrected some language and formatting in settings.py docstrings.
Description
Fix issues with identity-locking.
How to test
Remove identity file, pair device and check that mycroft can answer questions such as what is the weather and how tall is abraham lincon.
Adds the mycroft.util.combo_lock ComboLock class for interprocess/Thread
lock.
Loading updated to be more reliable:
- Flush and sync file
- wait 1.2 seconds before load
Split the logic from the locking so the lock can be avoided when calling
update from save or load from get.
The "phonetic_spellings.txt" mechanism converts odd words to strings that
represent what they should sound like when spoken. For example, "mycroftai"
to "Mycroft A.I.". This provides an easy mechanism to provide hints to
lots of Text to Speech engines.
This adds it to Mimic2, along with a spelling of "corgi".
* Add "wait" option to MycroftSkill.speak() and speak_dialog()
The new "wait" option will cause the speak function to block until all
of the given dialog has been spoken by Mycroft. This means:
self.speak("Hello world", wait=True)
is now equivalent to:
self.speak("Hello world")
wait_while_speaking()
* Add ability to schedule event in seconds
The MycroftSkill.schedule_event() method now accepts an integer in addition to
a datetime for the 'when' parameter. The integer represents the number of
seconds in the future to fire off the event. E.g.
```python
self.schedule_event(some_handler, 7)
```
Will invoke some_handler() seven seconds from now.
Also unified language used in event docstrings.
Switch to an OrderedDict() for translate_namedvalues(), maintaining the
sequence of values defined in the original "list.value" file. This is
useful in circumstances where there are multiple values, but the order
of listing indicates some sort of preference.
This is used in the Alarm skill to allow synonyms like "weekdays"/"weekday",
"Mondays/Monday", but the first value is used when building the status string.
For example "You have an alarm for 8am on Mondays". Generically, this lets
translators consistently provide preferred names for values by adjusting the
order.
The mycroft.util.time.now_utc() was returning a naive datetime object, potentially causing
issues in skills running on instances that aren't using UTC system-wide. This didn't impact
Picroft or Mark 1, but Github installs would experience issues with skills such as the Alarm.
Adapt doesn't populate the entry from the one_of correctly from context. To work around the issue intent structure is scanned for empty keys and tries to populate them from entities in __tags__
* Add repeat option to audioservice
The audioservice.play() method now accepts a repeat parameter. If this
parameter is True the playlist passed to the audio service will be
repeated.
* Add repeat support to vlc
* Add the repeat parameter to all services
Not functional but playback will work at least. Hacktoberfest?
Added functions for the spanish parser
* Added spanish function calls in parser.py for extractnumber_es
and extract_datetime_es
* Added spanish functions in util/paser_es.py for extractnumber_es
extract_datetime_es and added some missing numbers
Merged in changes from #1804
* Fix ambiguous time handling
In certain cases the ambiguous time handling skipped a day forward. This updates the logic to handle a bit better.
* Adds a test for the ambiguous time
* Remove references to timeStr
timeStr was never set and the logic that used it would never activate.
* Remove rename of currentDate
* Add extract_datetime parameter default_time
If a time is not found in the input string the time will be set from the
datetime/time object passed in as the default_time argument. If None the
time will be Midnight as previously.
* Add cleanup if the skill loading fails
If a skill throws an exception in initialize the registered handlers
need to be removed. To cleanup this the skill shutdown procedure is
run.
* Don't try converse method on non-loaded skills
Checks if a valid skill instance is present before calling the converse method
to filter out skills that wasn't loaded.
* Mark unloaded skills as inactive in CLI
* Use correct method to guess mime type
* Use the commands from mycroft.conf
- Remove hardcoded commands and use the ones from mycroft.util to handle
configurations.
- Improve error handling
The recent changes to support the 'locale' directory were incorrectly
joining the os.walk() path and filename, resulting in double-directories,
like "./vocab/en-us/./vocab/en-us/Something.voc"
This is a step towards abstracting the idea of an Enclosure which ties Mycroft to the hardware that is running Mycroft.
- Move the enclosure API to mycroft.enclosure.api (previously was mycroft.client.enclosure.api)
- Move display_manager out of enclosure client to mycroft.enclosure.display_manager
- Merge EnclosureWeather into EnclosureMouth
- Wrap display manager in a class
* Add MycroftSkill.find_resource() that locates a localization resource file
from either under the old vocab/regex/dialog folder, or under any folder in
the new 'locale' folder. The locale folder unifies the three different
folders and allows arbitrary subfolders underneath it.
* MycroftSkill.speak_dialog() will now speak the entry name if no dialog file
is found. Periods are replaced with spaces, so
```self.speak_dialog("this.is.a.test.")``` would return "this is a test" if
no file named this.is.a.test.dialog is found.
* Remove MycroftSkills.vocab_dir value
* Minor edits to several docstrings
The method first tries to use the skill vocab directory and then tries
the mycroft/res/text... directory.
After the vocab is loaded from file once it's stored in a cache
dictionary and further uses of the method won't hit the disk.
Example:
if self.voc_match(utt, "Yes):
...
Skills list shared the help screen id it was immediately overwritten by
the help.
- Give the skills page a separate page number
- Replace screen id numbers with simple identifiers
- Add simple handling of multiple pages
The initial implementation of web settings returned values as
strings all the time, even Boolean and numeric values. This
required unnecessarily complicated code, such as:
```python
if setting["show_time"] == "true":
# do whatever...
```
Now a value defined in metadata as a "checkbox" gets cast to a boolean,
and values defined in metadata as a "number" is typecast to int or
float, as appropriate. This allows cleaner code such as:
```python
if setting["show_time"]:
# do whatever...
```
NOTE: This can be a breaking change, verify you skill which uses 'checkbox'
webUI types handles this correctly for 18.08.
The API.find/find_setting/find_location() methods were replaced by
API.get/get_setting/get_location() some time ago. Permanently removing
the deprecated functions.
The list holding the names of scheduled events was being populated with decorated names,
but other code assumed it held the "friendly" name. Corrected this assumption.
Several locations also removed list entries while iterating on the same list, resulting
in entries being skipped and left un-deleted. This left behind phantom scheduled events
when skills were reloaded, goofing up many schedules after the reload.
Several minor and simple CLI fixes:
* The CLI was locking up with at Lock() deadlock when rebuilding the
filtered log.
* The simple_cli() wasn't registering handle_speak() on the websocket,
even though the handler was setup for it. The utterances are already
clear from the Input: line or from the logs being printed, so no need
to have a handle_utterance() registered in the simple_cli() case.
* Help wasn't refreshing the screen, so was invisible.
* An error could be thrown if Ctrl+Z was hit to suspend the 'fg' to
return to foreground before any key was pressed.
Numbers are now normalized to the text equivalent. This is to solve problems where mimic2 returns a speed up text generation when saying number values.
Allow a Padatious intent to override Adapt when it is VERY
certain that the utterance is directed at it. (95% confidence
or greater.) Right now that only occurs if the intent match
for the given phrase is perfect.
This solves this kind of issue:
* Adapt: Matching on "Set" and "Alarm"
* Padatious: Handling "is an alarm set"
* Fix logic error for when no Padatious intent
Previously, the location of the log files changed depending
on whether you were running on a Mark1/Picroft or under a
"Github" install. Now they are always under the same directory
at /var/log/mycroft-*.log
This also updates the CLI to pull from that location always.
Additionally:
* Removed the nonfunctional 'wifi' option from start-mycroft.sh
* Made the validation for dev_setup.sh exit instead of just show
a warning message in start-mycroft.sh
* Added code to allow dev_setup.sh to be run from different
directories successfully
The CLI would often have temporary screen corruption. This reworks
several things to correct that issue, although it looks like the
ultimate cause was drawing to the screen while in the middle of
waiting on a VT100 ESC keycode sequence.
* Rearchitected all screen drawing to run in a single thread. Now
call set_screen_dirty() instead of draw_screen()
* Added a lock on accessing the various Log buffers, preventing
changes to the buffer in the middle of a redraw
* Modified key reading logic when ESC character is received. Now
uses a 1 second timeout if no subsequent keycodes are sent
* Catch several special exceptions. Curses throws exceptions in some cases during get_wch(), such as
when you Ctrl+Z suspend the CLI the resume the process. Also moved Ctrl+C processing into this exception handler.
While working on the Alarm skill I discovered several issues with the
event scheduler. This PR cleans up those findings and resolves several
other potential issues:
1) To avoid thread synchronization issues, the EventScheduler had several
queues which independently held objects to be added/deleted/updated. However, the order of the events was undefined and got mixed since they were all batched together. So, for instance, if skill code performed:
self.add_event("foo", self.handle_foo)
if SomeReason:
self.cancel_event("foo")
The actual order of queue handling would perform Remove first, then Add which resulted in "foo" not being found for delete, but then added and left as an active event.
Now the EventScheduler protects the list using a Lock and the queues have been removed. Modifications to the list happen immediately after obtaining the lock and are not batched up.
2) One-time events were triggered while the event was still in the EventScheduler list. Now the entry is removed before invoking the handler.
3) Within the MycroftSkill.add_event(name, handler) is a local 'wrapper' method that actually makes the callback. The MycroftSkill.remove_event(name) method attempted to find entries in the events list and the associated handler entries in the self.emitter to remove. However, the emitter actually held the wrapper(handler), not the handler itself. So the emitter handlers were left behind.
This was a quiet bug until the next time you scheduled an event of the same name. When that second event finally triggered, it would fire off both the new and the old handler -- which snowballed in the 'skill-alarm:Beep' case, doubling and redoubling with every beep.
Now this cancels all the emitter listeners by name. There is a very slim chance that someone has registered a listener with the same name, but since it is namespaced to "skill-name:Event" I think this is pretty safe.
Not technically related, but a failure that has been lurking for
some time and is a French unit test that doesn't work depending
on the time of day when the test is run.
This is reimplementation of #1649 which became divergent.
## Description
Adds a Ogg123Service and a play_ogg exactly like Mpg123Service and play_mp3
## How to test
I have a skill for a podcast which does not have an mp3 feed:
https://github.com/joshuacox/skill-GNUworldOrder
## Contributor license agreement signed?
signed by @joshuacox
At startup, Padatious was blocking in the fallback while the
training occurred. As a result, any attempts to use Mycroft
during that period would queue up rather than giving the
user feedback. Now it immediately returns False, allowing
user notification to occur elsewhere.
Response formatting for German language ordinals depending on cases/prepositions for dates
"am 1. März" -> "am ersten März" (on the first of March)
"der 1. März" -> "der erste März" (the first of March)
"1. März" -> "erster März" (first of March)
Response formatting for mathematical results
"10 ^ 2" -> "10 hoch 2" (ten to the power of two)
Can be tested via the corresponding test_format_de or by using wolfram alpha skill:
"Was ist die Fläche von Canada"
"Wann ist George Washington geboren"
* Fix cases where the unmunge misses keywords
The unmunge would invariably miss keys due to the fact that the dict is modified while being iterated. This breaks out the keys into a list before iterating through them to ensure that all original keys are checked.
* Clean up the unmunge function slightly
This makes the dialog parameter dual-purpose. It can be used as a
literal spoken string, or a dialog resource. In the future the
announcement parameter can be retired.
* added asynchronous request to mimic2 backend to break up audio into chunks
* add chracter threshold
* refactored split_by_punctuation
* add punctuation function to sentence chunks
* fix spelling
* some more spelling
* added mimic2 in mycroftconf
* removed unused imports
The pronunciation of English date-time was awkward without the word "the"
for dates like "Monday, the 16th at eight a.m."
Also corrected the tests to use the newer lower case "a.m." instead of "AM"
Switched from "AM" to "a.m.", and "PM" to "p.m". These are pronounced better
in Mimic, and are also the more normally accepted ways of writing the
abbreviations for ante meridiem and post meridiem (at least according to the
AP and Chicago Style guides).
Many cases that were missed in the unittests for extract_datetime()
from the original source. Restored those tests and made code
adjustments to support them all.
Also adding the mycroft.util.time module. This supports:
* mycroft.util.time.default_timezone()
Returns the user-configured timezone based on location
* mycroft.util.time.now_utc()
Returns the time in UTC
* mycroft.util.time.now_local()
Returns the time in the user's timezone
* mycroft.util.time.to_utc()
Converts to UTC
* mycroft.util.time.to_local()
Converts to user's timezone
NOTE: Several skills should be updated to use these now.
==== Fixed Issues ====
Several issues for skills regarding parsing of "today"
==== Documentation Notes ====
Note the new module: mycroft.util.time
==== Localization Notes ====
Localized versions of extract_datetime() likely need to be
updated, as most were based on the original English implementation
New function allows simple yes/no queries to be asked, capturing common
affirmations (e.g. "yes", "yeah", "yep", "sure", ...) and rejections
("no", "nope", ...) that are consistent across all skills.
The method will return a normalized 'yes', 'no' or whatever else is spoken
for further parsing. None is also possible.
This also adds the MycroftSkill.is_match() method to assist in matching
translated synonyms within an utterance.
- Remove setup scripts for mycroft-skills-sdk since it's not used anymore
- Rename mycroft-base-setup.in/MANIFEST.in to setup.py and MANIFEST.in
- Remove skill-container, since it hasn't been used or kept up to date since 17.08 and the cli commands to remove and activate skills is easier to work with
==== Environment Notes ====
Small update of the packaging script is needed due to this change
* Restored VT100 function key support
* ESC now clears the entry line
* Spoken commands now become part of the history
* Enhanced help screen with auto word-wrap
* Bugfix: "Show meter" setting didn't persist, now it does
This fixes a stray lock.release() causing a silent
RuntimeError: release unlocked lock
after TTS execution but just before reporting the timing for the TTS system.
* Handle stop correctly in the audio service.
This allows for example the news skill playback to be stopped without
the listening being triggered.
* Handle case where service stops at end of playlist
- Fix issue when receiving multiple stop signals
- Stop method of services now returns True or False depending on if audio was playing
When running on a platform for which there was not DEFAULT.platform
file in the mycroft-skills repo, the downloading of skills would
crash. Now an informational message is shown and the DEFAULT
skills alone are loaded.
The Mark 1 button press can now be "consumed" when a skill handles
the Stop command. When this happens, the button press will not
trigger listening mode. An additional press would be needed to
trigger listening.
This introduces the "mycroft.stop.handled" messagebus message. It
carries a data field called "by" which identifies who handled it.
Currently the values are "TTS" for when speaking ends or the name
of a skill which implements Stop and returns True from the call.
Also fixed a potential bug when the flag to clear queued visemes
was left set after a button press.
* Restore extractdatetime to return False
- Restore extractdatetime to return False if no time was found
- Add tests to make sure this is true
- Add a extract_datetime function to keep coherency with the rest of the functions. (the old extractdatetime still exists for compatibility)
- Update documentation to match
* Minor corrections to docstrings.
* Update format for skill listing
Now send the skills with id and active status
* Add commands to activate/deactivate skills
* Add "unload all except one" functionallity
* Update after rebasing
- fix identifying skills
* Unload skills if they're removed from disk
* Rename _shutdown to default_shutdown
The method is not intended to be non-public, and this should shut up
codacy bot.
* Handle keep command without argument
* Add new commands to help
- Split help into multiple pages as needed
* Support :activate all
Also strips out old alternative ways to enable wake word upload
The logic for uploading wake words is now:
- Only if opt-in is enabled
- With an additional "disable" setting to selectively prevent it
The default socket timeout was changed when checking the connectivity through connecting to a remote server (8.8.8.8). This has now been updated to only change the timeout for the socket used for the connection.
To make reply work in the same manner as a newly created message data
shall be optional. This allows mistakes like L1141 of
mycroft/skills/core.py, where the data field isn't set. (And causes an
exception when the line is hit.)
The default method for the services call stop, in the chromecast case this isn't desirable since any playing application would be shutdown (even if mycroft didn't initiate it).
This overrides default behaviour and will merely disconnect from the device at shutdown.
Previously failed wake words would collect and each one would attempt an upload and fail. Now, this does the same process but failing on the first wake word that fails to upload
The scp upload shows the account id which isn't the best to put in logs
The saved audio seconds used to contain audio before so that it could be used to identify multiple recordings that didn't activate the device. However, we've since moved to tagging only the last 3 seconds
This is necessary because in Python 3, hash(x) changes every single start of the application. Using the skill folder makes it consistent. In addition, the skill folder makes it easier to debug parts of the application in comparison to using something like an md5sum
This keeps track of the skills whose dependencies have already been installed. While it won't automatically register newly installed skills, it will attempt to reinstall dependencies the next boot only one time so it shouldn't be a big issue.
Checking against basestring was necessary in python two since unicode and string were separate classes. In python3 basestring was removed since different string classes have been removed.
Lots of minor fixes including, sorting dicts, making ints of strings,
MagicMock file spec and some other things
A couple of issues in the mycroft-core code base were identified and
fixed. Most notably the incorrect version check for python three when
adding basestring.
Update .travis.yml
- Engines now specify if they support ssml rather than the configuration
- The text client strips out ssml tags
- Engines can modify tags via the `self.modify_tag` method
When loading voc and regex files, lines starting with "#" are now
ignored, so developers and translators can use them to document their
decisions.
==== Fixed Issues ====
This sends a ctrl+c signal to each process which will allow code to exit properly by handling KeyboardInterrupt
Other notable changes:
- create_daemon method used to clean up create daemon threads
- create_echo_function used to reduce code duplication with messagebus
echo functions
- wait_for_exit_signal used to wait for ctrl+c (SIGINT)
- reset_sigint_handler used to ensure SIGINT will raise KeyboardInterrupt
Allow intents to be enabled or disabled from outside of the skill via
the messagebus.
Adds mycroft.skill.enable_intent and mycroft.skill.disable_intent
This defaults skill's configs to an empty dict instead of None,
which simplifies getting specific values in the config for skills.
It removes the check for None config.
==== Fixed Issues ====
==== Documentation Notes ====
Skill writers no longer need to check for None configs.
The emitted event "open" will in many cases call ws.emit, and this will
lock if the connected_event isn't set. This makes sure that the
connected_event is set before emitting the open event.
If the sentence is normalized the utterance_remainder() method fails if
the intent keyword expects normalization.
Example the joking skill has an intent "tell me joke" which is the
normalized equivalent of "tell me a joke". In this case the intent wouldn't be removed from the remainder and instead of an empty string the utterance remainder would contain "tell me joke"
Modified the debugging message output after an utterance is complete, to
make it clear that metric data is only sent if opt_in is enabled.
This addresses #1494, which I filed in error, thinking that opt_in was
never checked (due to reading the log output and not knowing it was
checked elsewhere)
==== Fixed Issues ====
==== Localization Notes ====
Slight change to existing, unlocalized text
Previously if duplicates of a skill should be launched or a skill isn't properly
shutdown when reloaded multiple handlers could be registred. This commit disables multiples of repeating events.
An exception raised during the shutdown would previously cause the skill reload process to halt. This catches any exception and reports the error without halting the SkillManager
If super's shutdown was called before trying to cancel events an exception would be thrown since shutdown removed all registered events by setting self.events to None.
This replaces this with an empty list to allow skills to try to remove events/cancel events without incidents.
When a fresh image first updates the settings after pairing it is handled somewhat differently and this was missed in the original implementation of the callback handling.
This minor change includes this case as well.
If the scheduler was frozen for some time and the repeating event
scheduled time passes it would trigger the event constantly until the
next time is in the future again.
This will make the scheduler to disallow scheduling in the past and
instead schedule the next call one repeat period from now.
This can be replaced by ensuring the final except clause is 'except Exceptions'. This works because SystemExit and KeyboardInterrupt do not inherit from the base 'Exception' class
Several tweaks to CLI behavior
* The "show/hide meter" setting is now persisted
* Fix behavior when scrolled back and logs are rolling off
* Fix potential crash when no logs exist
* Add Ctrl+R to force a refresh (undocumented for now-- help page is getting long and we need an automated system to resize help for smaller screens)
The calulated limit ensures that at least one message will be displayed.
Using a hard-coded constant is ugly, but such are scattered throughout
the script. Untangling those constants has been left for future patch.
* Fix error message for enable_intent
The error was printed for each intent name mismatch instead of after all intents had been checked.
* Make sure intents aren't munged multiple times
Previously intents could be munged multiple times (This happened when enabling a disabled intent), resulting in an invalid name.
* Add test case for disable/enable intent
* Improve unmunging of messages
This make sure that only skill id's in the beginning of messages are removed and should speed up the process slightly
* Fix munging for register_vocab and register_regex
* Add testcases for register vocab and regex
When creating event handlers with add_event now by default there are no
messages about start and completion of the handler.
The handler info base name is now passed as an argument to the method
allowing for custom messages. skill intent handlers still report
skill.handler.start and skill.handler.complete but for example scheduled
events wont send these start/stop (but could instead trigger for example
skill.scheduled_event.start/complete)
* Update pyee to v5.0.0
The old version of pyee (1.0.1) could not remove events registered as "once" (instead of "on")
This fixes canceling scheduled events
* Restore MycroftSkill.remove_event() return value
The return statement in remove_event() was missing, probably lost while handling a conflict.
The data field of the message sent to the event handler may not always be a dictionary, (Example case Timer skill, which sets data to the timer name)
This validates the message type and the type of the data field before trying to unmunge.
* Add interface to Mark 1 faceplace capabilities
This adds API interfaces for two Mark 1 faceplace capabilities to the
mycroft.client.enclosure.EnclosureAPI()
EnclosureAPI.setpixel(index, r,g,b)
- Set individual eye pixels to any color. The indices go from 0-23 with
the 0-12 corresponding to the right eye and 11-23 to the left.
EnclosureAPI.fill(percentage)
- Fill the eyes to a percentage, for use in meters or countdowns. The
right eye is 0-50%, the left eye also fills if going up to 100%.
* Skip skill update on startup if recent
Skills aren't updated on startup if the last update was less than 12
hours ago.
- msm adds a .msm file in the skills directory with a timestamp and a
list of the skills installed by default
- the UPDATING screen message is moved to the SkillManger when direct
update is scheduled.
- On internet connection the skill update time is scheduled
- Skills (other than priority skills) are not loaded until
- The timeout for when direct update is necessary is settable in the
config.
internet connection has been verified (message on messagebus)
All methods relating to loading vocabulary, dialog and regular
expressions has grown quite large. To make the core functionallity of
the skills more readable these are moved to the new module skill_data.
Additional method documentation has been addedi as well.
Convert keyword names to unique names by prepending the keyword with a
letter string derived from the unique skill id.
This commit modifies required keywords, optional keywords, one_of
keywords and regex matches.
This also munges the context keyword when that is sent to match the
intent correctly
Many thanks to mikonse for troubleshooting and suggesting this fix.
==== Fixed Issues ====
==== Tech Notes ====
Disabling this exception should free up the audio producer thread to
continue deliver the available sound data.
* Add wait_for_response method to Websock client
The client handles the basic case when wanting to do a syncronous
request-response action.
The method sets up a handler waits for the response and handles timeout.
The expected format is that the reply message should have the same type as
the original message with ".response" appended.
An method in the Message class has been added to create a standard response for
a message.
* Let the :skills command use wait_for_response
* Minor docstring changes
Fixed typos and refined text
* Made MycroftSkill.remove_event() return a bool, preventing unnecessary/misleading message from being posted by MycroftSkill.cancel_scheduled_event()
* More doc and several minor renames around intent processing
* Several minor typo and doc corrections
An error occured when testing skill settings when checking if the settings should be sent when trying to upload a field without a value. To guard against this there is a check if the field has a value before checking.
SkillManager now handles the skillmanager.list message and will reply with the
mycroft.skills.list message including a list of the loaded skills.
==== Protocol Notes ====
Added messages:
- skillmanager.list: skill manager send list of skills on messagebus
- mycroft.skills.list message with skill list
The bug was due to identifier collisions in the backend as well as settingsmeta schemas with a label field with no name attribute. This fix will make sure there are no more identifier collisions with already existing broken settingsmeta schemas in the database.
Add support for:
* mycroft.util.format.nice_time()
* mycroft.util.format.prounce_number()
* implemented unittests for above
Also renamed the helper method convert_number() to
_convert_to_mixed_fraction()
* Add support for deepspeech_server
deepspeech_server is a server running deepspeech (obviously). It's quite
easy to install and run (package available in pip and then a config file
pointing to the model)
This pr adds the DeepSpeechServerTTS class, a STT interface allowing
mycroft to use one of these servers.
config needed:
"stt": {
"module": "deepspeech_server",
"deepspeech_server": {
"uri": "http://IP-ADDRESS:PORT/stt"
}
}
* Add deepspeech_server example to mycroft.conf
Withe the NTP checks in place, the sequence of visual and audio queues
was a little clunky. This refines it slightly for normal use and to
play better with the pairing process.
* Implement max-line to limit memory usage
The major point of the PR is to limit the memory usage of the CLI by
implementing a maximum log limit. It defaults to 5000.
Other changes:
* Add "--debug" option to support troubleshooting/debugging the CLI itself
* Add support for jumping to the top (Ctrl+T/Ctrl+PgUp) or bottom (Ctrl+B/Ctrl+PgDn) of the logs.
* Remove the "OLDEST" message from the log. It was really no longer necessary since the log navigation issues got straightened out, and it complicated the max log line logic.
Raspberry Pi's don't have a built-in clock, so at boot-up the clock just picks up from when they were last running. Normally this is corrected very quickly by NTP from an internet server, but if there is no network connection that cannot happen.
When an out-of-the-box Mark 1 or Picroft is being setup, the clock is set to whenever the image was created. Upon completing the Wifi setup step the NTP service can finally sync with the internet, so time suddenly "jumps" to weeks later -- usually. In either case (when the date jumps or when the date is erronously months old), there is potential for havoc.
These changes deal with that situation. Upon network connection, an NTP synchonization is forced. If it is detected that a major time jump happened
(more than 1 hour), then the user is notified that the clock change requires
a reboot and the system restarts.
Other changes:
* use the new "system." message namespace
* add pause before the system.reboot during a WIPE, allowing reset to totally complete
*
==== Tech Notes ====
When a new track starts playing the audio service will send a message
indicating which track has started.
==== Protocol Notes ====
"mycroft.audio.service.track_start" message added
- add_event() now accepts the parameter once, registring the event as a one shot event.
- remove_event for non-existing events is handled
- added a test for this
The speak method digs through the stack trying to find a Message object
and if found uses the context from that message when sending the data to
the speech subsystem.
==== Tech Notes ====
STT, intent handling, intent fallbacks, skill handlers are now timed and
tied together with a ident (consistent through the chain so the flow from
STT until completion of the skill handler can be follewed.
TTS execution time is also measured, right now this is not tied into the
ident due to the nature of the speech.
The report is always called "timing" and always contain the following
fields:
- id: Identifier grouping the metrics into interactions
- system: Which part (STT, intent service, skill handler, etc)
- start_time: timestamp for when the action started
- time: how long it took to execute the action
The different system adds their own specific information, for example
the intent_service adds the intent_type, i.e. which handler was matched.
==== Protocol Notes ====
mycroft.skills.loaded is sent togheter with skill id and skill name
whenever a skill is loaded. This is used in the intent_service to
convert from id to skill name when reporting
Basically at any place where handle_metric is called I was adding a try-except
block to catch possible network/http issues. Due to this fact I feel
it's best to add it here.
Move the language specific functions and constants into separate files.
This will avoid many unnecessary conflicts due to involuntary encoding
changes.
Åke Forslund
MichaIng
Chris Rogers
David Wagner
Steve Penrod
Joshua Watt
devs-mycroft
Aditya Mehra
Tanel.Alumae
Ruthvicp
Matthew D. Scholefield
Kate Canaveral
Thomas Doczkal
Kris Gesling
Kris Gesling
jarbasal
Ruthvicp
Steve Penrod
Andreas Lorensen
Angel Docampo
Luis Reinoso
Ale
Andreas Reinle
Alistair Francis
Alistair Francis
G3RB3N
ludwhe
Michael Nguyen
danielwine
JarbasAI
Jaime Muñoz Martín
Francis Tyers
Kathy Reid
mpolidori
Silvia O'Dwyer
Niels Mündler
Michael Nguyen
Angel Docampo
Dominik
Josh Cox
Julien Kassar
Dominik
Nolan Darilek
Josh Cox
f-e-l-i-x
Carsten Agerskov
nabice
Matthew D. Scholefield
reginaneon
Sergio Oller
Greg Cowan
InconsolableCellist
Zachary T Welch
Cakeh
Kévin C
Augusto Monteiro
Jony