* Speed up async_get_loaded_integrations
Use a setcomp and difference to find the components to split
to avoid the loop. A setcomp is inlined in python3.12 so its
much faster
* Speed up async_get_loaded_integrations
Use a setcomp and difference to find the components to split
to avoid the loop. A setcomp is inlined in python3.12 so its
much faster
* simplify
* fix compat
* bootstrap
* fix tests
* Small speed up to setting up integration and config entries
When profiling tests, I noticed many calls to get_running_loop. In the places
where we are already in a coro, pass the existing loop so it does not have to
be looked up. I did not do this for places were we are not in a coro since there
is risk that an integration could be doing a non-thread-safe call and its better
that the code raises when trying to fetch the running loop vs the performance
improvement for these cases.
* fix merge
* missed some
* Add thread safety checks to async_create_task
Calling async_create_task from a thread almost always results in an
fast crash. Since most internals are using async_create_background_task
or other task APIs, and this is the one integrations seem to get wrong
the most, add a thread safety check here
* Add thread safety checks to async_create_task
Calling async_create_task from a thread almost always results in an
fast crash. Since most internals are using async_create_background_task
or other task APIs, and this is the one integrations seem to get wrong
the most, add a thread safety check here
* missed one
* Update homeassistant/core.py
* fix mocks
* one more internal
* more places where internal can be used
* more places where internal can be used
* more places where internal can be used
* internal one more place since this is high volume and was already eager_start
Co-authored-by: Sid <27780930+autinerd@users.noreply.github.com>
Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
Co-authored-by: J. Nick Koston <nick@koston.org>
* Move setup time logging into the context manager
We were fetching the time twice but since the context
manager already has the timing, move it there
* remove log setup assertions from integration test
* tweak logging to give us better data for tracking issues
* redundant
* adjust
* preen
* fixes
* adjust
* make api change internal so nobody uses it
* coverage
* fix test
* fix more tests
* coverage
* more tests assuming internal calls
* fix more
* adjust
* adjust
* fix axis tests
* fix broadlink -- it does not call async_forward_entry_setup
* missed some
* remove useless patch
* rename, detect it both ways
* clear
* debug
* try to fix
* handle phase finishing out while paused
* where its set does not need to know its late as that is an implemenation detail of setup
* where its set does not need to know its late as that is an implemenation detail of setup
* tweak
* simplify
* reduce complexity
* revert order change as it makes review harder
* revert naming changes as it makes review harder
* improve comment
* improve debug
* late dispatch test
* test the other way as well
* Update setup.py
* Update setup.py
* Update setup.py
* simplify
* reduce
* Refactor setup time tracking to exclude time waiting on other operations
We now exclude the import time and th time waiting on
base platforms to setup from the setup times
* tweak
* tweak
* tweak
* tweak
* adjust
* fixes
* fixes
* preen
* preen
* tweak
* tweak
* adjust
* tweak
* reduce
* do not count integrtion platforms against their parent integration
* handle legacy tts platforms
* stt as well
* one more wait
* use the same pattern in all the legacy
* fix tts and stt legacy
* fix
* fix
* reduce
* preen
* entity comp does not wait for platforms
* scene blocks as well
* fix test
* test fixes
* coverage
* coverage
* coverage
* fix test
* Update tests/test_setup.py
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Update tests/test_setup.py
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Update homeassistant/setup.py
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* strip
* strip WAIT_PLATFORM_INTEGRATION
* strip WAIT_PLATFORM_INTEGRATION
* strip WAIT_PLATFORM_INTEGRATION
* strip WAIT_PLATFORM_INTEGRATION
* remove complexity
* Apply suggestions from code review
* no longer works that way
* fixes
* fixes
* fixes
---------
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Add support for pre-imports at setup time
alternative solution to #111331
* refactor
* refactor
* refactor
* mark >1.0s integrations
* no point in executor if already loaded
* no point in executor if already loaded
* cleanup
* cleanup
* two more
* one more
* analytics loads a lot more integrations
* cloud
* debug
* psutil, hardwre
* try zha
* Update homeassistant/setup.py
* await
* comments
* coverage
* coverage
* coverage
* move logic to loader
* move logic to loader
* preserve comments
To wait for after dependencies we created a task to wait an asyncio.Event object,
instead of using an Event we can use an asyncio.Future instead and avoid the need
for a task wrapper
* Refactor integration startup time tracking to reduce overhead
- Use monotonic time for watching integration startup time as it avoids incorrect values if time moves backwards because of ntp during startup and reduces many time conversions since we want durations in seconds and not local time
- Use loop scheduling instead of a task
- Moves all the dispatcher logic into the new _WatchPendingSetups
* websocket as well
* tweaks
* simplify logic
* preserve logic
* preserve logic
* lint
* adjust
* Improve performance of _async_when_setup
Use an event filter to avoid creating a task until the
filter matches
* Improve performance of _async_when_setup
Use an event filter to avoid creating a task until the
filter matches
`2023-06-17 12:44:37.961 ERROR (MainThread) [homeassistant.setup] Setup failed for switchbot: Unable to import component: cannot import name DEFAULT_CIPHERS from urllib3.util.ssl_ (/Users/bdraco/home-assistant/venv/lib/python3.11/site-packages/urllib3/util/ssl_.py)` is not very helpful as it does not show which module tried to import.
adding a traceback makes it more obvious, and since ImportError is usually not something
the user can easily solve, it makes issue reports much more helpful
```
DEFAULT_CIPHERS from urllib3.util.ssl_ (/Users/bdraco/home-assistant/venv/lib/python3.11/site-packages/urllib3/util/ssl_.py)
Traceback (most recent call last):
File "/Users/bdraco/home-assistant/homeassistant/setup.py", line 213, in _async_setup_component
component = integration.get_component()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bdraco/home-assistant/homeassistant/loader.py", line 813, in get_component
ComponentProtocol, importlib.import_module(self.pkg_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/Users/bdraco/home-assistant/homeassistant/components/switchbot/__init__.py", line 5, in <module>
import switchbot
File "/Users/bdraco/home-assistant/venv/lib/python3.11/site-packages/switchbot/__init__.py", line 22, in <module>
from .devices.lock import SwitchbotLock
File "/Users/bdraco/home-assistant/venv/lib/python3.11/site-packages/switchbot/devices/lock.py", line 12, in <module>
import boto3
File "/Users/bdraco/home-assistant/venv/lib/python3.11/site-packages/boto3/__init__.py", line 16, in <module>
from boto3.session import Session
File "/Users/bdraco/home-assistant/venv/lib/python3.11/site-packages/boto3/session.py", line 17, in <module>
import botocore.session
File "/Users/bdraco/home-assistant/venv/lib/python3.11/site-packages/botocore/session.py", line 29, in <module>
import botocore.credentials
File "/Users/bdraco/home-assistant/venv/lib/python3.11/site-packages/botocore/credentials.py", line 34, in <module>
from botocore.config import Config
File "/Users/bdraco/home-assistant/venv/lib/python3.11/site-packages/botocore/config.py", line 16, in <module>
from botocore.endpoint import DEFAULT_TIMEOUT, MAX_POOL_CONNECTIONS
File "/Users/bdraco/home-assistant/venv/lib/python3.11/site-packages/botocore/endpoint.py", line 22, in <module>
from botocore.awsrequest import create_request_object
File "/Users/bdraco/home-assistant/venv/lib/python3.11/site-packages/botocore/awsrequest.py", line 24, in <module>
import botocore.utils
File "/Users/bdraco/home-assistant/venv/lib/python3.11/site-packages/botocore/utils.py", line 32, in <module>
import botocore.httpsession
File "/Users/bdraco/home-assistant/venv/lib/python3.11/site-packages/botocore/httpsession.py", line 10, in <module>
from urllib3.util.ssl_ import (
ImportError: cannot import name DEFAULT_CIPHERS from urllib3.util.ssl_ (/Users/bdraco/home-assistant/venv/lib/python3.11/site-packages/urllib3/util/ssl_.py)
```
Marc Mueller
J. Nick Koston
Sid
Joost Lekkerkerker
Erik Montnemery
Jan Bouwhuis