* Turn on thread safety checks in async_dispatcher_send
We keep seeing issues where async_dispatcher_send is called from
a thread which means we call the callback function on the other
side in the thread as well which usually leads to a crash
* Turn on thread safety checks in async_dispatcher_send
We keep seeing issues where async_dispatcher_send is called from
a thread which means we call the callback function on the other
side in the thread as well which usually leads to a crash
* adjust
* Refactor entity_platform polling to avoid double time fetch
Replace async_track_time_interval with loop.call_later
to avoid the useless time fetch every time the listener
fired since we always throw it away
* fix test
* Add title feature to notify entity platform
* Add overload variants
* Remove overloads, update signatures
* Improve test coverage
* Apply suggestions from code review
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Do not use const
* fix typo
---------
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Add thread safety checks to async_create_task
Calling async_create_task from a thread almost always results in an
fast crash. Since most internals are using async_create_background_task
or other task APIs, and this is the one integrations seem to get wrong
the most, add a thread safety check here
* Add thread safety checks to async_create_task
Calling async_create_task from a thread almost always results in an
fast crash. Since most internals are using async_create_background_task
or other task APIs, and this is the one integrations seem to get wrong
the most, add a thread safety check here
* missed one
* Update homeassistant/core.py
* fix mocks
* one more internal
* more places where internal can be used
* more places where internal can be used
* more places where internal can be used
* internal one more place since this is high volume and was already eager_start
It turns out we have custom components that are writing to the area registry using the async APIs from threads. We now catch it at the point async_fire is called. Instead we should check sooner and use async_fire_internal so we catch the unsafe operation before it can corrupt the registry.
* Move thread safety check in entity_registry sooner
It turns out we have a lot of custom components that are writing
to the entity registry using the async APIs from threads. We now
catch it at the point async_fire is called. Instread we should check
sooner and use async_fire_internal so we catch the unsafe operation
before it can corrupt the registry.
* coverage
* Apply suggestions from code review
It turns out we have custom components that are writing to the device registry using the async APIs from threads. We now catch it at the point async_fire is called. Instead we should check sooner and use async_fire_internal so we catch the unsafe operation before it can corrupt the registry.
* Fix lying docstring for relative_time template function
* Update homeassistant/helpers/template.py
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
---------
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Make release channel a hardcoded enum rather than a free form string
* Update enum comparison to remove equality and us identity comparison
* Fix comparison condition to match the previous implementation
* Update tests to use Enum instead of string
* Keep track of top level components
Currently we have to do a set comp for icons, translations,
and integration platforms every time to split the top level
components from the platforms. Keep track of the top level
components in a seperate set so avoid having to do the setcomp
every time.
* remove impossible paths
* remove unused code
* preen
* preen
* fix
* coverage and fixes
* Update homeassistant/core.py
* Update homeassistant/core.py
* Update tests/test_core.py
* Remove unused state translations
There have been replaced with entity translations
https://github.com/home-assistant/developers.home-assistant/pull/1557https://github.com/home-assistant/core/pull/82701
* nothing does merging anymore
* useless dispatch
* remove
* remove platform code from hassfest
* preen
* Update homeassistant/helpers/translation.py
* ruff
* fix merge
* check is impossible now since we already know if translations exist or not
* keep the function for now
* remove unreachable code since we filter out `.` before now
* reduce
* reduce
* fix merge conflict (again)
* Fix race in TimestampDataUpdateCoordinator
The last_update_success_time value was being set after the listeners
were fired which could lead to a loop because the listener may
re-trigger an update because it thinks the data is stale
* coverage
* docstring
* Add notify entity component
* Device classes, restore state, icons
* Add icons file
* Add tests for kitchen_sink
* Remove notify from no_entity_platforms in hassfest icons, translation link
* ruff
* Remove `data` feature
* Only message support
* Complete initial device classes
* mypy pylint
* Remove device_class implementation
* format
* Follow up comments
* Remove _attr_supported_features
* Use setup_test_component_platform
* User helper at other places
* last comment
* Add entry unload test and non async test
* Avoid default mutable object in constructor
* Avoid checking for polling if an entity fails to add
* no need to do protected access
* no need to do protected access
* no need to do protected access
* no need to do protected access
* coverage
* fix test
* fix
* broken one must be first
* Bump aiodns to 3.2.0
changelog: https://github.com/saghul/aiodns/compare/v3.1.1...v3.2.0
* Switch to using the AsyncResolver with aiohttp
This avoids creating executor jobs to do DNS resolution
AsyncResolver was not usable before https://github.com/aio-libs/aiohttp/pull/8270
because it did not fallback to fallback to A records and only returned AAAA records
in most cases when IPv6 was available
This is a backport of https://github.com/aio-libs/aiohttp/pull/8270
* Switch to using the AsyncResolver with aiohttp
This avoids creating executor jobs to do DNS resolution
AsyncResolver was not usable before https://github.com/aio-libs/aiohttp/pull/8270
because it did not fallback to fallback to A records and only returned AAAA records
in most cases when IPv6 was available
This is a backport of https://github.com/aio-libs/aiohttp/pull/8270
* Switch to using the AsyncResolver with aiohttp
This avoids creating executor jobs to do DNS resolution
AsyncResolver was not usable before https://github.com/aio-libs/aiohttp/pull/8270
because it did not fallback to fallback to A records and only returned AAAA records
in most cases when IPv6 was available
This is a backport of https://github.com/aio-libs/aiohttp/pull/8270
* Switch to using the AsyncResolver with aiohttp
This avoids creating executor jobs to do DNS resolution
AsyncResolver was not usable before https://github.com/aio-libs/aiohttp/pull/8270
because it did not fallback to fallback to A records and only returned AAAA records
in most cases when IPv6 was available
This is a backport of https://github.com/aio-libs/aiohttp/pull/8270
* fixes
* fix mocking in next_dns
* fix unmocked calls in blink
* more mocking fixes
* more fixes
* more fixes
* Fix missing mocking in nextdns tests
extracted from #114539
* extract from context
* DNM: Make run_immediately the default for listeners
This is a test to see how much progress we have made twords this goal
https://github.com/home-assistant/core/pull/113727#issuecomment-2004587947
* fix shutdown
* Revert "fix shutdown"
This reverts commit a8969d7db9.
* set false since it break utility meter tests
* one more
* fix rfxtrx test
* test needs to be explict now
* fix matrix
* fail sooner
* Adjust async_load_platform comment
Its likely the deadlock here has been fixed for a long time, however
we should still do these in a task because it has to wait for base
components if they are not loaded yet.
* Adjust async_load_platform comment
Its likely the deadlock here has been fixed for a long time, however
we should still do these in a task because it has to wait for base
components if they are not loaded yet.
* Add missing platforms_exist guard to check_config
related issue #112811
When the exception hits, the config will end up being saved in the traceback
so the memory is never released.
This matches the check_config code to homeassistant.config to avoid having
the exception thrown.
* patch
* merge branch
This function was only ever used in homekit, and since there is
now an index of devices in the entity registry, homekit no longer
uses it.
I searched github code for all references to async_get_device_class_lookup
and the only think I could find using it were forks of core. It seems
unlikely that any custom components are affected by removing this
function
Co-authored-by: Sid <27780930+autinerd@users.noreply.github.com>
Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
Co-authored-by: J. Nick Koston <nick@koston.org>
* Small performance improvement to template expand
- Avoid fetching entity sources each loop
- Skip already found entities
- Avoid startswith in favor of equality check
* unneeded changes
* Use CONF_CORE to reference `homeassistant` domain
* Just use DOMAIN
* USE DOMAIN for `homeasistant` domain in config_schema.py
* Use DOMAIN_HA as constant for homeassistant domain
* Rename CONF_CORE to DOMAIN_HA
* Rename DOMAIN_HA to HA_DOMAIN
* Use relative import
* Use direct imports
* Refactor rate limit helper to track time in seconds
Currently we created datetime and timedelta objects to enforce the
rate limit. When the rate limit was being hit hard, this got expensive.
We now use floats everywhere instead as they are much cheaper which
is important when we are running up against a rate limit, which is
by definition a hot path
The rate limit helper is currently only used for templates and
we do not have any code in the code base that directly passes
in a rate limit so the impact to custom components is expected
to be negligible if any
* misesd two
* Cache parsing the url for the device registry
There are lots of hub integrations that use the same url for every
sub-device which results in a lot of url parsing at startup. The
logic can be simplified quite a bit here by only using yarl
for URLs
* fix onvif
* Refactor setup time tracking to exclude time waiting on other operations
We now exclude the import time and th time waiting on
base platforms to setup from the setup times
* tweak
* tweak
* tweak
* tweak
* adjust
* fixes
* fixes
* preen
* preen
* tweak
* tweak
* adjust
* tweak
* reduce
* do not count integrtion platforms against their parent integration
* handle legacy tts platforms
* stt as well
* one more wait
* use the same pattern in all the legacy
* fix tts and stt legacy
* fix
* fix
* reduce
* preen
* entity comp does not wait for platforms
* scene blocks as well
* fix test
* test fixes
* coverage
* coverage
* coverage
* fix test
* Update tests/test_setup.py
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Update tests/test_setup.py
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Update homeassistant/setup.py
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* strip
* strip WAIT_PLATFORM_INTEGRATION
* strip WAIT_PLATFORM_INTEGRATION
* strip WAIT_PLATFORM_INTEGRATION
* strip WAIT_PLATFORM_INTEGRATION
* remove complexity
* Apply suggestions from code review
* no longer works that way
* fixes
* fixes
* fixes
---------
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Add category registry
* Add entity registry support
* Update homeassistant/components/config/entity_registry.py
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Use ulid instead
* Add tests for adding same name in different scopes
* Handle keyerror on update
* Lookup tweak
* Omit categories from entity registry snapshots
* Use base registry
* Update snapshots
* Update snapshots
---------
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Reduce script overhead by avoiding creation of many tasks
* no eager stop
* reduce
* make sure wait being cancelled is handled
* make sure wait being cancelled is handled
* make sure wait being cancelled is handled
* preen
* preen
* result already raises cancelled error, remove redundant code
* no need to raise it into the future
* will never set an exception
* Simplify long action script implementation
* comment
* preen
* dry
* dry
* preen
* dry
* preen
* no need to access protected
* no need to access protected
* dry
* name
* dry
* dry
* dry
* dry
* reduce name changes
* drop one more task
* stale comment
* stale comment
We save the device and entity registry to disk quite often, and
the cost of serializing them to the storage can block the event
loop for >100ms. Add a cache to reduce the change the loop is blocked
at an inopportune time at run time. The first write after startup
will still be a little slow but we do have to serialize the
bulk of it at least once as there is no way to avoid this
```
2024-03-14 11:28:19.765 WARNING (MainThread) [homeassistant.helpers.storage] Writing data with data_func: core.device_registry
2024-03-14 11:28:20.020 WARNING (MainThread) [homeassistant.helpers.storage] Writing data with data_func: core.entity_registry
2024-03-14 11:28:20.178 WARNING (MainThread) [asyncio] Executing <TimerHandle cancelled when=2319925.760294916 Store._async_schedule_callback_delayed_write() created at /Users/bdraco/home-assistant/homeassistant/helpers/storage.py:328> took 0.159 seconds
```
Construct storage data in the executor
Constructing storage data can be expensive for large files and can block the event
loop. While ideally we optimize the construction of the data, there are some places
we cannot make it any faster. To avoid blocking the loop, the construction of
the data is now done in the executor by running the data_func in the executor.
2024-03-14 11:28:20.178 WARNING (MainThread) [asyncio] Executing <TimerHandle cancelled when=2319925.760294916 Store._async_schedule_callback_delayed_write() created at /Users/bdraco/home-assistant/homeassistant/helpers/storage.py:328> took 0.159 seconds
There is some risk that the data_func is not thread-safe and needs to be run in
the event loop, but I could not find any cases in our existing code where it
would be a problem
* Add a helper to import modules in the event loop
Replaces the one used for triggers with a more generic helper
that can be reused and uses a future to avoid importing concurrently
* Add a helper to import modules in the event loop
Replaces the one used for triggers with a more generic helper
that can be reused and uses a future to avoid importing concurrently
* coverage
* make sure we do not retry
* coverage
There were a few places we were missing the set to list
conversions in the registries. We do this before its
cached to avoid the JSON serializer having to fallback
to the default method every time since its expensive
to switch back from the native code into python context
for every set.
Await discovery_function in the DiscoveryFlowHandler instead of wrapping
This function was always a coro so it can be awaited
directly instead of wrapping it in add_job
* Run coroutines as eager tasks in async_run_hass_job
Note that this does not change async_add_hass_job
Do not merge this. For test run only
* Phase out periodic tasks
* false by default or some tests will block forever, will need to fix each one manually
* kwarg works
* kwarg works
* kwarg works
* fixes
* fix more tests
* fix more tests
* fix lifx
* opensky
* pvpc_hourly_pricing
* adjust more
* adjust more
* smarttub
* adjust more
* adjust more
* adjust more
* adjust more
* adjust
* no eager executor
* zha
* qnap_qsw
* fix more
* fix fix
* docs
* its a wrapper now
* add more coverage
* coverage
* cover all combos
* more fixes
* more fixes
* more fixes
* remaining issues are legit bugs in tests
* make tplink test more predictable
* more fixes
* feedreader
* grind out some more
* make test race safe
* limit first scope to triggers
* one more
* Start tasks eagerly in for async_at_start(ed)
A few of these can avoid being scheduled on the loop
during startup
* fix cloud
* Revert "fix cloud"
This reverts commit 5eb3ce695d.
* fix test to do what start does
* flip flag
* flip flag
* Fix here_travel_time creating many refresh requests at startup
- Each entity would try to refresh the coordinator which
created many tasks. Move the refresh to a single
async_at_started
- The tests fired the EVENT_HOMEASSISTANT_START event
but the code used async_at_started which only worked
because the tests did not set CoreState to not_running
* fix azure
* remove kw
* remove kw
* rip
* cover
* more rips
* more rips
* more rips
* Phase out periodic tasks
* false by default or some tests will block forever, will need to fix each one manually
* kwarg works
* kwarg works
* kwarg works
* fixes
* fix more tests
* fix more tests
* fix lifx
* opensky
* pvpc_hourly_pricing
* adjust more
* adjust more
* smarttub
* adjust more
* adjust more
* adjust more
* adjust more
* adjust
* no eager executor
* zha
* qnap_qsw
* fix more
* fix fix
* docs
* its a wrapper now
* add more coverage
* coverage
* cover all combos
* more fixes
* more fixes
* more fixes
* remaining issues are legit bugs in tests
* make tplink test more predictable
* more fixes
* feedreader
* grind out some more
* make test race safe
* one more
* Schedule periodic coordinator updates as background tasks.
Currently, the coordinator's periodic refreshes delay startup because they are not scheduled as background tasks. We will wait if the startup takes long enough for the first planned refresh. Another coordinator's scheduled refresh will be fired on busy systems, further delaying the startup. This chain of events results in the startup taking a long time and hitting the safety timeout because too many coordinators are refreshing.
This case can also happen with scheduled entity refreshes, but it's less common. A future PR will address that case.
* periodic_tasks
* periodic_tasks
* periodic_tasks
* merge
* merge
* merge
* merge
* merge
* fix test that call the sync api from async
* one more place
* cannot chain
* async_run_periodic_hass_job
* sun and pattern time changes from automations also block startup
* Revert "sun and pattern time changes from automations also block startup"
This reverts commit 6de2defa05.
* make sure polling is cancelled when config entry is unloaded
* Revert "Revert "sun and pattern time changes from automations also block startup""
This reverts commit e8f12aad55.
* remove DisabledError from homewizard test as it relies on a race
* fix race
* direct coverage
These were very likely to be cached so they were low on the
list to migrate, but since they are called in the event loop
its best to be sure we do no blocking I/O
discovered in https://github.com/home-assistant/core/pull/112295#discussion_r1513505710
We only checked if the last language had files to load
instead of all of them. The checks for each language
are the same because the only reason we would skip
a language is a missing/broken integration or the integration
is a single file. Both of these loop conditions are always
the same reguardless of the language so the check worked
Some of the data we had to search for was already available
in a dict or underlying data structure. Make it available
instead of having to build it every time.
There are more places these can be used, but I only did
the device registry cleanup for now
* Add 'issues' template function for listing active issues.
* Add issue template function test
* Add 'issue' template function for getting specific issue by domain and issue_id
* Remove comment
* Fix function description
* Remove reduntant function,
Fix tests
* remove pass_context
* remove issues filter
Co-authored-by: Erik Montnemery <erik@montnemery.com>
---------
Co-authored-by: Erik Montnemery <erik@montnemery.com>
* Avoid trying to import platforms that do not exist
* adjust
* fixes
* cleanup
* cleanup
* cleanup
* Apply suggestions from code review
* docs
* fixes
* fixes
* comment
* coverage
* coverage
* coverage
* Switch config to use async_get_component
This was another path where integrations that were marked to load in the executor
would be loaded in the loop
* Switch config to use async_get_component/async_get_platform
This was another path where integrations that were marked to load in the executor
would be loaded in the loop
* merge
* refactor
* refactor
* coverage
* preen
* preen
* Avoid expensive inspect calls in config validators
inspect has a performance problem https://github.com/python/cpython/issues/92041
We now avoid calling inspect unless we are going to log
* remove unused
* reduce
* get_integration_logger
* Move HassSetPosition to homeassistant domain
* Add test for unsupported domain with HassSetPosition
* Split service intent handler
* cleanup
---------
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
* Deprecate @bind_hass and log error if used inside custom component
* Log also when accessing `hass.components`
* Log warning only when `hass.components` is used
* Change version
* Process code review
* Reduce latancy to load storage by making the task eager
This changes the semantics a bit under the hood because it
can raise sooner which means we do not store the task
as _load_task if it raises right away. That means
concurrent calls that result in failure are likely to try
again now which will be a tiny performance hit for this
case.
* fix
* will now finish in time
* Use an eager task to setup entity platforms
Ideally we would have awaited this function instead, but we want
to shield it from cancellation so we wrap it in asyncio.shield
which schedules it as a task. Since we have integrations that
never suspend in async_setup_entry, we can avoid scheduling on
the evnet loop with an eager task for this case
* its an executor future
* its an executor future
* fix
* doc string lied
* Use an eager task in the update coordinator scheduled refresh
We have a lot of places that will not suspend because the refresh function
decides it does not need to update. Currently these have to be scheduled
on the event loop even though they are a noop.
Since _handle_refresh_interval is subclassed in some integrations, I created
a dunder wrapper function to avoid integraions subclassing it
* fix time fires outside of patch
Marc Mueller
Sid
J. Nick Koston
Jan Bouwhuis
G Johansson
Erik Montnemery
rlippmann
Robert Resch
Martijn van der Pol
Alberto Montes
Franck Nijhof
Michael Hansen
Jan-Philipp Benecke
Joost Lekkerkerker
Paul Chanvin
Paulus Schoutsen
RoboMagus