* Add category registry
* Add entity registry support
* Update homeassistant/components/config/entity_registry.py
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Use ulid instead
* Add tests for adding same name in different scopes
* Handle keyerror on update
* Lookup tweak
* Omit categories from entity registry snapshots
* Use base registry
* Update snapshots
* Update snapshots
---------
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Add a helper to import modules in the event loop
Replaces the one used for triggers with a more generic helper
that can be reused and uses a future to avoid importing concurrently
* Add a helper to import modules in the event loop
Replaces the one used for triggers with a more generic helper
that can be reused and uses a future to avoid importing concurrently
* coverage
* make sure we do not retry
* coverage
Await discovery_function in the DiscoveryFlowHandler instead of wrapping
This function was always a coro so it can be awaited
directly instead of wrapping it in add_job
* Phase out periodic tasks
* false by default or some tests will block forever, will need to fix each one manually
* kwarg works
* kwarg works
* kwarg works
* fixes
* fix more tests
* fix more tests
* fix lifx
* opensky
* pvpc_hourly_pricing
* adjust more
* adjust more
* smarttub
* adjust more
* adjust more
* adjust more
* adjust more
* adjust
* no eager executor
* zha
* qnap_qsw
* fix more
* fix fix
* docs
* its a wrapper now
* add more coverage
* coverage
* cover all combos
* more fixes
* more fixes
* more fixes
* remaining issues are legit bugs in tests
* make tplink test more predictable
* more fixes
* feedreader
* grind out some more
* make test race safe
* one more
* Schedule periodic coordinator updates as background tasks.
Currently, the coordinator's periodic refreshes delay startup because they are not scheduled as background tasks. We will wait if the startup takes long enough for the first planned refresh. Another coordinator's scheduled refresh will be fired on busy systems, further delaying the startup. This chain of events results in the startup taking a long time and hitting the safety timeout because too many coordinators are refreshing.
This case can also happen with scheduled entity refreshes, but it's less common. A future PR will address that case.
* periodic_tasks
* periodic_tasks
* periodic_tasks
* merge
* merge
* merge
* merge
* merge
* fix test that call the sync api from async
* one more place
* cannot chain
* async_run_periodic_hass_job
* sun and pattern time changes from automations also block startup
* Revert "sun and pattern time changes from automations also block startup"
This reverts commit 6de2defa05.
* make sure polling is cancelled when config entry is unloaded
* Revert "Revert "sun and pattern time changes from automations also block startup""
This reverts commit e8f12aad55.
* remove DisabledError from homewizard test as it relies on a race
* fix race
* direct coverage
These were very likely to be cached so they were low on the
list to migrate, but since they are called in the event loop
its best to be sure we do no blocking I/O
discovered in https://github.com/home-assistant/core/pull/112295#discussion_r1513505710
We only checked if the last language had files to load
instead of all of them. The checks for each language
are the same because the only reason we would skip
a language is a missing/broken integration or the integration
is a single file. Both of these loop conditions are always
the same reguardless of the language so the check worked
Some of the data we had to search for was already available
in a dict or underlying data structure. Make it available
instead of having to build it every time.
There are more places these can be used, but I only did
the device registry cleanup for now
* Add 'issues' template function for listing active issues.
* Add issue template function test
* Add 'issue' template function for getting specific issue by domain and issue_id
* Remove comment
* Fix function description
* Remove reduntant function,
Fix tests
* remove pass_context
* remove issues filter
Co-authored-by: Erik Montnemery <erik@montnemery.com>
---------
Co-authored-by: Erik Montnemery <erik@montnemery.com>
* Avoid expensive inspect calls in config validators
inspect has a performance problem https://github.com/python/cpython/issues/92041
We now avoid calling inspect unless we are going to log
* remove unused
* reduce
* get_integration_logger
* Deprecate @bind_hass and log error if used inside custom component
* Log also when accessing `hass.components`
* Log warning only when `hass.components` is used
* Change version
* Process code review
* Reduce latancy to load storage by making the task eager
This changes the semantics a bit under the hood because it
can raise sooner which means we do not store the task
as _load_task if it raises right away. That means
concurrent calls that result in failure are likely to try
again now which will be a tiny performance hit for this
case.
* fix
* will now finish in time
* Avoid circular import in Storage.async_delay_save
We call Storage.async_delay_save for every entity being added or removed
from the registry. The late import took more time than everything else
in the function.
* Avoid reschedule churn in Storage.async_delay_save
When we are adding or removing entities we will call async_delay_save
quite often which has to add and remove a TimerHandle on the event loop
which can add up when there are a lot of registry items changing.
If the timer handle still has 80% of the time remaining on it
we will avoid resceduling and let it fire at the time the
original async_delay_save call was made. This ensures we
do not force the event loop to rebuild its heapq because
too many timer handlers were cancelled at once
* div0
* add coverage for 0 since we had none
* fix bad conflict
* tweaks
* tweaks
* tweaks
* tweaks
* tweaks
* tweaks
* more test fixes
* mqtt tests rely on event loop overhead
* Convert debouncer async_shutdown to be a normal function
nothing was being awaited here and the shutdown call was only used
in integrations marked internal and other internals. Its possible
that a custom component might have been using the method but it
seemed uncommon enough that it did not warrent marking as a breaking
change. The update coordinator is no longer awaiting anything in
async_shutdown either now but it seemed likely that this use
would get subclassed.
* fix
* Add async_schedule_call to the Debouncer
async_schedule_call allows the Debouncer to schedule a call
from a callback without having to create tasks to run
async_call
* Update homeassistant/helpers/debounce.py
* Make device registry cleanup all callback function
* fix typing, code supported callback functions, but typing did not
* fixes
* fixes
* fix
* we had no coverage for other job types
* we had no coverage for other job types
* Reduce overhead to load multiple languages in translations
Instead of loading in a task, we now group everything
to be loaded into a single executor job
* fixes
* fixes
* fixes
* fixes
* fixes
* update tests
* add missing coverage (was existing)
* Avoid scheduling registry loads as tasks in tests
Since we patch out async_load in Store, these will not yield
to the event loop so it makes sense to await them instead
of creating tasks
This reduced my local test run times ~2.5% on average
* mock out save as well so we do not schedule tasks to save empty data
* tweaks
* fix lingering files
* another one
* too much for one PR, reduce
* fix targets
* Add valve to HassTurnOn/Off
* Add set position for valves
* Add set position to covers
* Add HassTurnOn/Off for vacuums
* Add media player intents
* Split out vacuum intents
* Address comments
* Extra test
* Avoid creating tasks for checking integrations platforms
This is a followup to #110743 to avoid creating a task to check
if the integration platform exists. We created tasks because
we needed to await async_get_integrations but since its always
called from EVENT_COMPONENT_LOADED firing, we can use the
async_get_loaded_integration version which does not need
to be awaited. This eliminates one task for every loaded
component
* there is no more race risk
* reduce
* coro or callback
* reduce
* tweak
* race safe
* fix type
* fixes
* use built-in helper to make it smaller
* use built-in helper to make it smaller
* use built-in helper to make it smaller
* add coverage to ensure exceptions are logged
* improve readability a bit
* platforms
* Move late import of config flows in loader to load time
There does not seem to be any reason to import the
generated flows late. Import them at load time
with the rest of the generated files
* tests
* Add failing tests for package config validation error wrapping
* Wrap package schema validation errors in HomeAssistantError
* Fix yamllint errors
* Rework package merge validation
Ignore invalid package definitions instead of failing startup.
Output error messages with locations if possible when a package
definition has errors.
* Ruff format
* Fix linter errors
* Move package_definition_schema to module scope
* Move inner function to module level
* Merge exception handlers
Merge exception handlers for config schema validation and package merge
to avoid untested code branches
* Fix long lines and doc strings
* More minor changes to exception handler
---------
Co-authored-by: Erik Montnemery <erik@montnemery.com>
* Mask sensitive data in google_assistant logs
* Move common code to homeassistant/util/redact.py
* Move to helpers
* Add tests
* Tweak
* Redact additional logs
* Fix stale docstring
* Don't reveal the length of masked data
* Update test
* Improve tests of script trace
* Update tests after rebase
* Apply suggestions from code review
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Apply suggestions from code review
* Adjust
---------
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* add placeholder support to entity name translation
* add negativ tests
* make property also available via description
* fix doc string in translation_placeholders()
* fix detection of placeholder
* validate placeholders for localized strings
* add test
* Cache translation_placeholders property
* Make translation_placeholders uncondotionally return dict
* Fall back to unsubstituted name in case of mismatch
* Only replace failing translations with English
* Update snapshots
* Blow up on non stable releases
* Fix test
* Update entity.py
---------
Co-authored-by: Erik <erik@montnemery.com>
Co-authored-by: Joost Lekkerkerker <joostlek@outlook.com>
Reduce overhead connect dispatcher
- We tend to have 1000s (or 10000s) of connected dispatchers which
makes these prime targets to reduce overhead/memory
- Instead of creating new functions to wrap log exceptions each time
use partials which reuses the function body and only create new
arguments
Previous optimizations #103307#93602
* Keep capabilities up to date in the entity registry
* Warn if entities update their capabilities very often
* Fix updating of device class
* Stop tracking capability updates once flooding is logged
* Only sync registry if state changed
* Improve test
* Revert "Only sync registry if state changed"
This reverts commit 1c52571596c06444df234d4b088242b494b630f2.
* Avoid calculating device class twice
* Address review comments
* Revert using dataclass
* Fix unintended revert
* Add helper method
* Drop use of regex in helpers.extract_domain_configs
* Update test
* Revert test update
* Add domain_from_config_key helper
* Add validator
* Address review comment
* Update snapshots
* Inline domain_from_config_key in validator
* Improve OAuth error handling in configuration flows
* Update strings for all integrations that use oauth2 config flow
* Remove invalid_auth strings
* Revert change to release
* Revert close change in aiohttp mock
* Differentiate between warnings and errors in check_config helper
* Update tests
* Treat configuration errors in frontend and its dependencies as errors
* Improve test coverage
* Address review comments
* Improve test coverage
* Improve test coverage
* Address review comments
* Add comment
* Prevent accidentally reusing an entity object
* Fix group reload service
* Revert "Fix group reload service"
* Improve test
* Add tests aserting entity can't be reused
* add supports_response to platform entity services
* support multiple entities in entity_service_call
* support legacy response format for service calls
* revert changes to script/shell_command
* add back test for multiple responses for legacy service
* remove SupportsResponse.ONLY_LEGACY
* Apply suggestion
Co-authored-by: Allen Porter <allen.porter@gmail.com>
* test for entity_id remove None
* revert Apply suggestion
* return EntityServiceResponse from _handle_entity_call
* Use asyncio.gather
* EntityServiceResponse not Optional
* styling
---------
Co-authored-by: Allen Porter <allen.porter@gmail.com>
* Use loop time to set context
loop time is faster than utcnow, and since its only used internally it can
be switched without a breaking change
* fix mocking
* Allow specifying a custom log function for template render
* Bypass template cache when reporting errors + fix tests
* Send errors as events
* Fix logic for creating new TemplateEnvironment
* Add strict mode back
* Only send error events if report_errors is True
* Force test of websocket_api only
* Debug test
* Run pytest with higher verbosity
* Timeout after 1 minute, enable syslog output
* Adjust timeout
* Add debug logs
* Fix unsafe call to WebSocketHandler._send_message
* Remove debug code
* Improve test coverage
* Revert accidental change
* Include severity in error events
* Remove redundant information from error events
* Fix memory leak in dispatcher removal
When we removed the last job/callable from the dict for the
signal we did not remove the dict for the signal which meant
it leaked
* comment
* cleanup a bit more
* Set state of entity with invalid state to unknown
* Add test
* Apply suggestions from code review
Co-authored-by: Robert Resch <robert@resch.dev>
* Update test_entity.py
---------
Co-authored-by: Robert Resch <robert@resch.dev>
* Don't blow up if validators can't access the issue registry
* Make the check_config script open issue_registry read only
* Update tests/helpers/test_issue_registry.py
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
---------
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Improve handling of unrecoverable storage corruption
fixes#96574
If something in storage gets corrupted core can boot loop
or if its integration specific, the integration will fail to
start.
We now complainly loudly in the log, move away the corrupt data
and start fresh to allow startup to proceed so the user can
get to the UI and restore from backup without having to attach
a console (or otherwise login to the OS and manually modify files).
* test for corruption
* ensure OSError is still fatal
* one more case
* create an issue for corrupt storage
* fix key
* persist
* feedback
* feedback
* better to give the full path
* tweaks
* grammar
* add time
* feedback
* adjust
* try to get issue_domain from storage key
* coverage
* tweak wording some more
* Ensure async_get_system_info does not fail if supervisor is unavailable
fixes#96470
* fix i/o in the event loop
* fix tests
* handle some more failure cases
* more I/O here
* coverage
* coverage
* Update homeassistant/helpers/system_info.py
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
* remove supervisor detection fallback
* Update tests/helpers/test_system_info.py
---------
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
* Differentiate between device info types
* Update allowed fields
* Update homeassistant/helpers/entity_platform.py
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Split up message in 2 lines
* Use dict for device info types
* Extract device info function and test error checking
* Simplify parsing device info
* move checks around
* Simplify more
* Move error checking around
* Fix order
* fallback config entry title to domain
* Remove fallback for name to config entry domain
* Ensure mocked configuration URLs are strings
* one more test case
* Apply suggestions from code review
Co-authored-by: Erik Montnemery <erik@montnemery.com>
---------
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
Co-authored-by: Erik Montnemery <erik@montnemery.com>
* Mark scripts as response optional, make it always return a response if return_response is set
* Update test_init.py
* Revert "Update test_init.py"
This reverts commit 8e113e54db.
* Split + add test
* Avoid regex for negative zero check in sensor
We can avoid calling the regex for every sensor value
since most of the time values are not negative zero
* tweak
* tweak
* Apply suggestions from code review
* simpler
* cover
* safer and still fast
* safer and still fast
* prep for py3.11
* fix check
* add missing cover
* more coverage
* coverage
* coverage
* Update log message when referenced entity not found
* Update homeassistant/helpers/service.py
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Update test
---------
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Add service response data for listing calendar events
Add the capability of response data for for the entity component.
* Rename input arguments and add service description
* Improve list events to be more user friendly
Allow the end date to be determined based on a relative time duration. Make the start time optional and set to "now". Add additional test coverage. Update demo calendar to actually perform date range checks.
* Wrap docstrings properly.
* Increase test coverage
* Update to use new API calls
* Readability improvements
* Wrap docstrings
* Require at least one of end or duration
* Check for multiple entity matches earlier in the request
* Update documentation strings
* Improve service response data APIs
Make the API naming more consistent, and require registration that a
service supports response data so that we can better integrate with
the UI and avoid user confusion with better error messages.
* Improve test coverage
* Add an enum for registering response values
* Assign enum values
* Convert SupportsResponse to StrEnum
* Update service call test docstrings
* Add tiny missing full stop in comment
---------
Co-authored-by: Franck Nijhof <frenck@frenck.nl>
* Allow scripts service actions to save return values
* Simplify script service response data
* Rename result_variable to response_variable based on feedback
* Speed up entity service calls
- Avoid permissions check if the caller is an admin
- Use set intersection instead of linear search of entity platforms to find entities
* tweak
* fix light test to not use an admin user
* Update return signature of service calls
* Add timeout error handling in websocket api for service calls
* Update recorder tests to remove assertion on service call
* Remove timeout behavior and update callers that depend on it today
* Fix tests
* Add missing else
* await coro directly
* Fix more tests
* Update the intent task to use wait instead of timeout
* Remove script service call limits and limit constants
* Update tests that depend on service call limits
* Use wait instead of wait_for and add test
* Update homeassistant/helpers/intent.py
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
---------
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Remove unneeded checks for Entity.platform
* Update tests
* Prevent breaking integrations without an EntityComponent
* Warn when entity has no platform
* support live reload of intent_script
* add services.yaml
* update tesls for full code coverage
* Update based on feedback
* fix intent_script reload when no intent_script config
* Update homeassistant/helpers/intent.py
* update tests to handle no_existing better
---------
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
* Migrate restore_state helper to use registry loading pattern
As more entities have started using restore_state over time, it
has become a startup bottleneck as each entity being added is
creating a task to load restore state data that is already loaded
since it is a singleton
We now use the same pattern as the registry helpers
* fix refactoring error -- guess I am tired
* fixes
* fix tests
* fix more
* fix more
* fix zha tests
* fix zha tests
* comments
* fix error
* add missing coverage
* s/DATA_RESTORE_STATE_TASK/DATA_RESTORE_STATE/g
* Fix refactoring error with updating in sequence
see #93649
* coverage
* make sure entities are being updated in parallel
* make sure entities are being updated in sequence
* Switch some frequently called call_laters to call_at
call_at is a bit faster than call_later since call_later
is a wrapper around call_at.
We call at lot of these at startup so it helps a bit when
we are resource constrained
* update test
* update test
* Support marking an assist pipeline as preferred
* Adjust
* Revert unneeded change
* Send preferred pipeline id in pipeline list
* Don't use property functions for the preferred pipeline
* Fix memory churn in state templates
The LRU for state templates was limited to 512 states. As soon
as it was exaused, system performance would tank as each template
that iterated all states would have to create and GC any state
> 512
* does it scale?
* avoid copy on all
* comment
* preen
* cover
* cover
* comments
* comments
* comments
* preen
* preen
* Speed up entity filter when there are many glob matchers
Since we do no care about which glob matches we can
combine all the translated globs into a single regex
which reduces the overhead
* delete unused code
* preen
* Allow passing an optional name to async_track_time_interval
This is the same idea as passing a name to asyncio.create_task which
makes it easier to track down bugs
* more
* short
* still cannot find it
* add a few more
* test
* Add filters to climate and light service descriptions
* Allow specifying enums directly
* Update service descriptions
* Adjust test
* Cache entity features
* Lint
* Improve error handling, add list of known base components
* Don't allow specifying an entity feature as int
* Adds a loader to enable jinja imports.
* Switch to in-memory
* Move loading custom_jinja off of the event loop
* Raise TemplateNotFound if template doesn't exist
* Fix docstring
* Adds a service to reload custom jinja
* Remove IO from test setup
* Improve coverage and small refactor
* Incorporate feedback and use .jinja extension
* Check the loaded sources in test.
* Incorporate PR feedback.
* Update homeassistant/helpers/template.py
Co-authored-by: Erik Montnemery <erik@montnemery.com>
---------
Co-authored-by: Erik Montnemery <erik@montnemery.com>
* Migrate integration_platform helper to use async_get_integrations
We were fetching integrations inside the gather one
at a time. This is inefficent.
* cleanup
* cleanup
* add task name
* small tweaks
* gather only if we have tasks
When a template render raised an exception we would start listening for
all states until the template did not raise an exception anymore. This
was not needed since the entity that is causing the exception was
already in the tracker.
Re-rendering on all state changes can be extremely expensive and can
bring an instance into a sluggish or unresponsive state when updating
from a much older version that did not raise ValueError when a default
was missing.
Sid
Jan-Philipp Benecke
soonsouth
Franck Nijhof
J. Nick Koston
Joost Lekkerkerker
Paulus Schoutsen
Marc Mueller
Erik Montnemery
RoboMagus
Michael Hansen
chammp
Piotr Machowski
Bram Kragten
Martijn van der Pol
karwosts
Daniel
Lars R
Michael
Robert Resch
Ville Skyttä
Jan Bouwhuis
Paul Bottein
schelv
Aarni Koskela
Allen Porter
Kevin Stillhammer
Raman Gupta
G Johansson
Nicolas van de Walle
Meow
Michael Benz
Mike Degatano
epenet
David Poll
Petro31
ehendrix23
Penny Wood
Lucas Mindêllo de Andrade