* Signficantly reduce executor contention during bootstrap
At startup we have a thundering herd wanting to use the executor
to load manifiest.json. Since we know which integrations we are
about to load in each resolver step, group the manifest loads
into single executor jobs by calling async_get_integrations on
the deps of the integrations after they are resolved.
In practice this reduced the number of executor jobs
by 80% during bootstrap
* merge
* naming
* tweak
* tweak
* not enough contention to be worth it there
* refactor to avoid waiting
* refactor to avoid waiting
* tweaks
* tweaks
* tweak
* background is fine
* comment
pkg_resources is abandoned and we need to move away
from using it https://github.com/pypa/pkg_resources
In the mean time we need to keep it working. This fixes
a new exception in py3.11 when a module is not installed
which allows proper fallback to pkg_resources.Requirement.parse
when needed
```
2023-02-25 15:46:21.101 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
File "/opt/homebrew/lib/python3.11/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request
resp = await request_handler(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/aiohttp/web_app.py", line 504, in _handle
resp = await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/aiohttp/web_middlewares.py", line 117, in impl
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bdraco/home-assistant/homeassistant/components/http/security_filter.py", line 60, in security_filter_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bdraco/home-assistant/homeassistant/components/http/forwarded.py", line 100, in forwarded_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bdraco/home-assistant/homeassistant/components/http/request_context.py", line 28, in request_context_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bdraco/home-assistant/homeassistant/components/http/ban.py", line 80, in ban_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bdraco/home-assistant/homeassistant/components/http/auth.py", line 235, in auth_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bdraco/home-assistant/homeassistant/components/http/view.py", line 146, in handle
result = await result
^^^^^^^^^^^^
File "/Users/bdraco/home-assistant/homeassistant/components/config/config_entries.py", line 148, in post
return await super().post(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bdraco/home-assistant/homeassistant/components/http/data_validator.py", line 72, in wrapper
result = await method(view, request, data, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bdraco/home-assistant/homeassistant/helpers/data_entry_flow.py", line 71, in post
result = await self._flow_mgr.async_init(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bdraco/home-assistant/homeassistant/config_entries.py", line 826, in async_init
flow, result = await task
^^^^^^^^^^
File "/Users/bdraco/home-assistant/homeassistant/config_entries.py", line 844, in _async_init
flow = await self.async_create_flow(handler, context=context, data=data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bdraco/home-assistant/homeassistant/config_entries.py", line 950, in async_create_flow
await async_process_deps_reqs(self.hass, self._hass_config, integration)
File "/Users/bdraco/home-assistant/homeassistant/setup.py", line 384, in async_process_deps_reqs
await requirements.async_get_integration_with_requirements(
File "/Users/bdraco/home-assistant/homeassistant/requirements.py", line 52, in async_get_integration_with_requirements
return await manager.async_get_integration_with_requirements(domain)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bdraco/home-assistant/homeassistant/requirements.py", line 171, in async_get_integration_with_requirements
await self._async_process_integration(integration, done)
File "/Users/bdraco/home-assistant/homeassistant/requirements.py", line 186, in _async_process_integration
await self.async_process_requirements(
File "/Users/bdraco/home-assistant/homeassistant/requirements.py", line 252, in async_process_requirements
await self._async_process_requirements(name, missing)
File "/Users/bdraco/home-assistant/homeassistant/requirements.py", line 284, in _async_process_requirements
installed, failures = await self.hass.async_add_executor_job(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.11/3.11.1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bdraco/home-assistant/homeassistant/requirements.py", line 113, in _install_requirements_if_missing
if pkg_util.is_installed(req) or _install_with_retry(req, kwargs):
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bdraco/home-assistant/homeassistant/util/package.py", line 40, in is_installed
pkg_resources.get_distribution(package)
File "/opt/homebrew/lib/python3.11/site-packages/pkg_resources/__init__.py", line 478, in get_distribution
dist = get_provider(dist)
^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/pkg_resources/__init__.py", line 354, in get_provider
return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
~~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range
``
If a package fails to install or partially installed importlib
version can return None. We now try pkg_resources first, then
try importlib, and handle the case where version unexpectedly
returns None
* Don't treat typing as an "in-between" module for import order
That was a < 3.5 era thing.
* Tighten scope of some pylint unused-import disables
To avoid isort moving a top level one around, undesirably broadening its
scope.
* Upgrade pylint to 1.8.1
* Fix no-else-return
* Fix bad-whitespace
* Fix too-many-nested-blocks
* Fix raising-format-tuple
See https://github.com/PyCQA/pylint/blob/master/doc/whatsnew/1.8.rst
* Fix len-as-condition
* Fix logging-not-lazy
Not sure about that TEMP_CELSIUS though, but internally it's probably just like if you concatenated any other (variable) string
* Fix stop-iteration-return
* Fix useless-super-delegation
* Fix trailing-comma-tuple
Both of these seem to simply be bugs:
* Nest: The value of self._humidity never seems to be used anywhere
* Dovado: The called API method seems to expect a "normal" number
* Fix redefined-argument-from-local
* Fix consider-using-enumerate
* Fix wrong-import-order
* Fix arguments-differ
* Fix missed no-else-return
* Fix no-member and related
* Fix signatures-differ
* Revert "Upgrade pylint to 1.8.1"
This reverts commit af78aa00f125a7d34add97b9d50c14db48412211.
* Fix arguments-differ
* except for device_tracker
* Cleanup
* Fix test using positional argument
* Fix line too long
I forgot to run flake8 - shame on me... 🙃
* Fix bad-option-value for 1.6.5
* Fix arguments-differ for device_tracker
* Upgrade pylint to 1.8.2
* 👕 Fix missed no-member
* Use pip install --user if venv not active
* Set PYTHONUSERBASE to deps directory, when installing with --user
option.
* Reset --prefix option to workaround incompatability when installing
with --user option. This requires pip version 8.0.0 or greater.
* Require pip version 8.0.3.
* Do not delete deps directory on home assistant upgrade.
* Fix local lib mount and check package exist.
* Update and add tests
* Fix upgrade from before version 0.46
* Extract function to get user site
* Add function(s) to package util to get user site.
* Use async subprocess for one of the functions to get user site.
* Add function to package util to check if virtual environment is
active.
* Add and update tests.
* Update version for last removal of deps dir
* Address comments
* Rewrite package util tests with pytest
* Rewrite all existing unittest class based tests for package util as
test functions, and capitalize pytest fixtures.
* Add test for installing with target inside venv.
J. Nick Koston
Franck Nijhof
Marc Mueller
henryptung
Erik Montnemery
Allen Porter
Ville Skyttä
Jeffery To
Christian Clauss
Bas Nijholt
Paulus Schoutsen
Pascal Vizeli
Pascal Vizeli
Paulus Schoutsen
Paulus Schoutsen
Andrey
Julius Mittenzwei
Otto Winter
Martin Hjelmare
Fabian Affolter
Pierre Ståhl
Andrzej
Fabian Heredia Montiel
Fabian Affolter
Ryan Kraus