459ea048ba
Fix old indices never being removed with PostgreSQL ( #89599 )
2023-03-12 14:07:05 -10:00
c41f91be89
Deduplicate entity_id in the states table ( #89557 )
2023-03-12 10:01:58 -10:00
50c31a5355
Move legacy database queries and models to prepare for schema v38 ( #89532 )
2023-03-11 11:26:30 -10:00
16b420d660
Fix get_significant_states_with_session query looking at legacy columns ( #89558 )
2023-03-11 10:37:00 -10:00
8bd43760b6
Deduplicate event_types in the events table ( #89465 )
...
* Deduplicate event_types in the events table
* Deduplicate event_types in the events table
* more fixes
* adjust
* adjust
* fix product
* fix tests
* adjust
* migrate
* migrate
* migrate
* more test fixes
* more test fixes
* fix
* migration test
* adjust
* speed up
* fix index
* fix more tests
* handle db failure
* preload
* tweak
* adjust
* fix stale docs strings, remove dead code
* refactor
* fix slow tests
* coverage
* self join to resolve query performance
* fix typo
* no need for quiet
* no need to drop index already dropped
* remove index that will never be used
* drop index sooner as we no longer use it
* Revert "remove index that will never be used"
This reverts commit 461aad2c52
2023-03-11 14:54:55 -05:00
52cea16f74
Remove unused code in RecorderRuns.entity_ids ( #89526 )
2023-03-11 12:46:12 +01:00
a0f725dfcb
Add type hints to tests ( #89497 )
2023-03-10 12:06:50 +01:00
170a13302c
Reduce overhead to store context ids in the database ( #88942 )
2023-03-08 14:51:45 -10:00
3c70dd9b42
Make sql subqueries threadsafe ( #89254 )
...
* Make sql subqueries threadsafe
fixes #89224
* fix join outside of lambda
* move statement generation into a seperate function to make it easier to test
* add cache key tests
* no need to mock hass
2023-03-06 20:44:11 -05:00
1bd9767d8c
Handle InnoDB deadlocks during migration ( #89073 )
...
* Handle slow InnoDB rollback when encountering duplicates during migration
fixes #89069
* adjust
* fix mock
* tests
* return on success
2023-03-03 22:00:13 -05:00
c2b770bcb9
Load pending state attributes and event data ids at startup ( #88444 )
...
* Load pending state attributes and event data ids at startup
Since we queue all events to be processed after startup
we can have a thundering herd of queries to prime the
LRUs of event data and state attributes ids. Since we
know we are about to process a chunk of events we can
fetch all the ids in two queries
* lru
* fix hang
* Fix recorder LRU being destroyed if event session is reopened
We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened
* Fix recorder LRU being destroyed if event session is reopened
We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened
* cleanup
2023-02-19 21:26:38 -05:00
3533e77ec9
Fix using MariaDB slow range select workaround with Postgresql ( #88459 )
...
* Mark PostgreSQL range select as fast
Currently we were using the slow range select workaround for
PostgreSQL that was original developed for MariaDB but
its actually slower on PostgreSQ
fixes #83253
* Mark PostgreSQL range select as fast
Currently we were using the slow range select workaround for
PostgreSQL that was original developed for MariaDB but
its actually slower on PostgreSQ
fixes #83253
2023-02-19 21:08:43 -05:00
a9731a7b26
Make the recorder LRU adjustment threadsafe ( #88443 )
2023-02-19 19:30:08 +01:00
9df00bd285
Adjust recorder LRU cache to align with the number of entities ( #88350 )
...
* Adjust size of recorder LRU based on number of entities
If there are a large number of entities the cache would
get thrashed as there were more state attributes being
recorded than the size of the cache. This meant we had
to go back to the database to do lookups frequently when
an instance has more than 2048 entities that change
frequently
* add a test
* do not actually record 4096 states
* patch target
* patch target
2023-02-19 00:02:36 -05:00
3ca9f3c0d5
Only build the recorder SupportedDialect enum once ( #88402 )
...
Every call to dialect_name was creating a new enum object
in the recorder
2023-02-18 20:21:14 -05:00
83e5bf7ae8
Use entity_sources to determine integration in recorder platforms ( #88382 )
2023-02-18 07:21:41 -06:00
289bab6f87
Make recorder platform attribute exclude integration aware ( #88357 )
2023-02-18 10:08:59 +01:00
a16e298599
Adjust json imports in tests ( #88248 )
2023-02-17 19:13:48 +01:00
185cd61cbd
Add type hints to integration tests (recorder) ( #88313 )
2023-02-17 18:50:09 +01:00
b9f29c27ab
Cleanup stale recorder code ( #88275 )
...
- Removes dead commit function
- Small cleanup to debug handling in execute
2023-02-16 23:12:11 -05:00
d3ed8de232
Update Ruff to v0.0.247 ( #88210 )
...
* Update Ruff to v0.0.247
* Fix new found violations
2023-02-15 16:46:03 -05:00
ed79265843
Enable Ruff PT006 ( #88165 )
...
* Enable Ruff PT006
* Adjust existing cases
* Fix tests
* Remove unneeded parentheses
2023-02-15 14:09:50 +01:00
efef915fc7
Add mariadb 10.3.32 to database tests ( #87741 )
...
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-13 16:28:39 -06:00
4ca0a24f87
Increase test scope of MariaDB + PostgreSQL tests ( #87019 )
...
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-13 15:58:13 -06:00
7b9293941e
The recorder/statistic(s)_during_period websocket API now require the statistic_id(s) field ( #87937 )
2023-02-12 14:51:10 -06:00
c35661947a
Optimize fetching statistics by avoiding timestamp conversions ( #87845 )
...
* Optimize fetching statistics by avoiding timestamp conversions
* remove accidential unrelated change
* fix test
* recreate so we handle timezone changes
2023-02-10 11:26:46 -05:00
cbaf4764e7
Ensure MySQL tests cleanup connections and raise an exception if they do not ( #87767 )
...
Co-authored-by: epenet <6771947+epenet@users.noreply.github.com>
Co-authored-by: Erik <erik@montnemery.com>
2023-02-09 16:27:53 -06:00
509de02044
Ensure recorder still shuts down if the final commit fails ( #87799 )
2023-02-09 15:12:40 -06:00
abf0c87e40
Migrate statistics to use timestamp columns ( #87321 )
2023-02-09 12:24:19 -06:00
2fae82a9ef
Really ensure recorder test fixture is setup before hass fixture ( #87796 )
2023-02-09 11:43:14 -06:00
db503c0daa
Always use UTC time for the MariaDB/MySQL session ( #87583 )
2023-02-09 11:39:45 -06:00
c05a7b29e6
Terminate stale MySQL connections at the end of test runs ( #87794 )
2023-02-09 11:13:13 -06:00
03b47f1931
Disable test_entity_ids on MariaDB ( #87766 )
...
Dropping the database after this test will fail on
MySQL and hang forever because it causes an InnoDB deadlock
```
| 2042 | root | localhost:52698 | NULL | Query | 41 | Waiting for table metadata lock | DROP DATABASE `homeassistant-test` | 0.000 |
```
2023-02-09 15:30:35 +01:00
f4a81a8b5b
Disable test_migration_in_progress on MySQL ( #87765 )
...
This test was recently enabled on MySQL in
https://github.com/home-assistant/core/pull/87753
Since it the migration is still in progress in the background
when the test ends, it causes deadlock with InnoDB when its dropped
out from under it
2023-02-09 15:27:27 +01:00
483b0cd017
Skip db locking test with postgresql/MySQL ( #87760 )
...
* Skip db locking test with postgresql/MySQL
https://github.com/home-assistant/core/pull/87756#issuecomment-1423828389
* switch to the guards instead
2023-02-09 13:58:32 +01:00
77cdde94f3
Turn back off test_shutdown_before_startup_finishes test on MySQL ( #87761 )
2023-02-09 03:12:10 -06:00
58618b2978
Improve recorder tests to use MariaDB / PostgreSQL ( #87756 )
2023-02-09 02:18:20 -06:00
b0f5bdd504
Improve migration tests to use MariaDB / PostgreSQL database ( #87753 )
2023-02-09 08:42:32 +01:00
ba85fdcd61
Add type hints to integration tests (q-s) ( #87706 )
2023-02-08 08:12:54 -08:00
94519de8dd
Upgrade SQLAlchemy to 2.0.2 ( #86436 )
...
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-08 08:17:32 -06:00
720f51657d
Add CI job which runs recorder tests on PostgreSQL ( #80614 )
...
Co-authored-by: Franck Nijhof <git@frenck.dev>
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-07 09:46:14 -06:00
59ca7780fa
Add typing to tests with single hass argument ( #87631 )
2023-02-07 15:01:16 +01:00
0a1d5c85ad
Optimize history.get_last_state_changes query ( #87554 )
...
fixes undefined
2023-02-07 07:50:44 -06:00
4142f0d15d
Add return type to tests without arguments ( #87613 )
...
* Add return type to tests without arguments
* Black
* Cancel fixture amends
2023-02-07 14:20:06 +01:00
aa00114c2f
Add return type to async tests without arguments ( #87612 )
2023-02-07 10:26:56 +01:00
342b406dc0
Add Self typing (1) [mypy 1.0] ( #87598 )
2023-02-06 22:29:47 -06:00
c7980bda82
Fix recorder run history during schema migration and startup ( #87492 )
...
Fix recorder run history during schema migration
RunHistory.get and RunHistory.current can be called before
RunHistory.start. We need to return a RecorderRuns object
with the recording_start time that will be used when start
it called to ensure history queries still work as expected.
fixes #87112
2023-02-05 22:12:41 -05:00
c11f3ffa17
Speed up purge time with newer MariaDB versions ( #87409 )
...
* Speed up purge time with newer MariaDB versions
* fix
* document
* document
* document
* rename
* self review
* Update homeassistant/components/recorder/util.py
* fixes
2023-02-05 17:58:34 -05:00
19e27f4f52
Ensure recorder api aligns with sensor converters ( #87257 )
2023-02-04 14:28:47 -06:00
389fc515a1
Prevent overly large attributes from being stored in the database ( #87105 )
2023-02-02 11:49:33 -06:00
5e81d28116
Update black to 23.1.0 ( #87188 )
2023-02-02 18:35:24 +01:00
58596d205c
Ensure sensor and recorder converters align ( #87138 )
2023-02-02 10:36:03 +01:00
1fd58b6cb7
Fix a few cartesian products in recorder tests ( #87106 )
...
Fixes
```
recorder/test_init.py:251: SAWarning: SELECT statement has a cartesian product between FROM element(s) "states" and FROM element "state_attributes". Apply join condition(s) between each element to resolve.
```
2023-02-02 07:47:09 +01:00
fb55933b0a
Reduce chance of queue overflow during schema migration ( #87090 )
...
Co-authored-by: Franck Nijhof <git@frenck.dev>
2023-02-01 18:34:40 +01:00
5284837c8f
Add a repair issue when using MariaDB is affected by MDEV-25020 ( #87040 )
...
closes https://github.com/home-assistant/core/issues/83787
2023-01-31 20:42:07 +01:00
53c5f02ca2
Remove some dead code from recorder ( #86697 )
2023-01-30 18:37:48 +01:00
c612a92cfb
Use python defaults for comparing State, LazyState, and Event objects ( #86856 )
...
* Speed up comparing State and Event objects
Use default python implementation for State and Event __hash__ and __eq__
The default implementation compared based on the id() of the object
which is effectively what we want here anyways. These overrides are
left over from the days when these used to be attrs objects
By avoiding implementing these ourselves all of the equality checks
can happen in native code
* tweak
* adjust tests
* write out some more
* fix test to not compare objects
* more test fixes
* more test fixes
* correct stats tests
* fix more tests
* fix more tests
* update sensor recorder tests
2023-01-29 13:31:43 -05:00
a79885ceaf
Enable Ruff SIM117 ( #86783 )
2023-01-27 11:52:49 +01:00
e4a78420b8
Enable Ruff PT013 ( #86757 )
...
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2023-01-27 09:09:46 +01:00
fea30c1ce9
Terminate strings at NUL when recording states and events ( #86687 )
2023-01-26 11:11:03 +01:00
52ea64d1d0
Fix repr for States and Events without a timestamp ( #86391 )
2023-01-22 08:11:42 -10:00
30bf0634fe
Add per-file-ignore to pylint ( #86289 )
2023-01-22 17:26:24 +01:00
233332c3a0
Fix fetching history include_start_time_state when timezone is not UTC ( #85983 )
2023-01-16 09:01:32 +01:00
72d3fa6d89
Replace the usage of unit constants by enumerations in Tests [o-r] ( #85936 )
2023-01-15 14:50:24 +01:00
b0d4b73874
Add unitless unit converter ( #85694 )
...
* Add unitless unit converter
* Adjust type hints
* Adjust tests
* Rename to UnitlessRatioConverter
2023-01-12 09:20:00 +01:00
d81febd3f4
Small speed up to frequently called datetime functions ( #85399 )
2023-01-08 09:42:29 -10:00
b8a1537b58
Improve performance of fetching and storing history and events with the database ( #84870 )
2023-01-02 13:26:08 -10:00
587381440f
Fix repairing datetime precision for PostgreSQL ( #83351 )
2022-12-05 22:33:06 -05:00
c576a68d33
Upgrade pytest-aiohttp ( #82475 )
...
* Upgrade pytest-aiohttp
* Make sure executors, tasks and timers are closed
Some test will trigger warnings on garbage collect, these warnings
spills over into next test.
Some test trigger tasks that raise errors on shutdown, these spill
over into next test.
This is to mimic older pytest-aiohttp and it's behaviour on test
cleanup.
Discussions on similar changes for pytest-aiohttp are here:
https://github.com/pytest-dev/pytest-asyncio/pull/309
* Replace loop with event_loop
* Make sure time is frozen for tests
* Make sure the ConditionType is not async
/home-assistant/homeassistant/helpers/template.py:2082: RuntimeWarning: coroutine 'AsyncMockMixin._execute_mock_call' was never awaited
def wrapper(*args, **kwargs):
Enable tracemalloc to get traceback where the object was allocated.
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
* Increase litejet press tests with a factor 10
The times are simulated anyway, and we can't stop the normal
event from occuring.
* Use async handlers for aiohttp
tests/components/motioneye/test_camera.py::test_get_still_image_from_camera
tests/components/motioneye/test_camera.py::test_get_still_image_from_camera
tests/components/motioneye/test_camera.py::test_get_stream_from_camera
tests/components/motioneye/test_camera.py::test_get_stream_from_camera
tests/components/motioneye/test_camera.py::test_camera_option_stream_url_template
tests/components/motioneye/test_camera.py::test_camera_option_stream_url_template
/Users/joakim/src/hass/home-assistant/venv/lib/python3.9/site-packages/aiohttp/web_urldispatcher.py:189: DeprecationWarning: Bare functions are deprecated, use async ones
warnings.warn(
* Switch to freezegun in modbus tests
The tests allowed clock to tick in between steps
* Make sure skybell object are fully mocked
Old tests would trigger attempts to post to could services:
```
DEBUG:aioskybell:HTTP post https://cloud.myskybell.com/api/v3/login/ Request with headers: {'content-type': 'application/json', 'accept': '*/*', 'x-skybell-app-id': 'd2b542c7-a7e4-4e1e-b77d-2b76911c7c46', 'x-skybell-client-id': '1f36a3c0-6dee-4997-a6db-4e1c67338e57'}
```
* Fix sorting that broke after rebase
2022-11-29 22:36:36 +01:00
f869ce9d06
Validate common statistics DB schema errors on start ( #79707 )
...
* Validate common statistics db schema errors on start
* Fix test
* Add tests
* Adjust tests
* Disable statistics schema validation in tests
* Update after rebase
2022-11-29 10:16:08 +01:00
a038314d8b
Add display unit to WS recorder/get_statistics_metadata ( #82870 )
2022-11-28 13:46:57 -05:00
2fe8e95309
Add helper to calculate statistic period start and end ( #82493 )
...
* Add helper to calculate statistic period start and end
* Don't parse values in resolve_period
* Add specific test for resolve_period
* Improve typing
* Move to recorder/util.py
* Extract period schema
2022-11-26 13:00:40 -05:00
b94e1e9ef8
Fire events when long term statistics is updated ( #82492 )
...
* Fire events when long term statistics is updated
* Allow the new events to be subscribed to by anyone
* Address review comments
* Finish renaming events
* Finish renaming events
* Fix do_adhoc_statistics
* Adjust tests
* Adjust tests
2022-11-24 22:01:36 +01:00
607a0e7697
Reduce size of get_statistics_during_period WS API response ( #82131 )
2022-11-16 17:36:30 +01:00
9b8f94363c
Fix statistic_during_period for data with holes ( #81847 )
2022-11-16 12:46:29 +01:00
68346599d2
Add WS API recorder/statistic_during_period ( #80663 )
2022-10-27 15:51:09 -04:00
8175dab7ab
Add week period to recorder statistics api ( #80784 )
...
* add week period to get statistics api
* add test
2022-10-25 20:07:28 +02:00
e84e5f134e
Use US_CUSTOMARY_SYSTEM in tests ( #80658 )
...
* Use US_CUSTOMARY_SYSTEM in tests
* Don't update test_unit_system
2022-10-20 15:42:23 +02:00
f4951a4f31
Add CI job which runs recorder tests on MariaDB ( #80586 )
...
Co-authored-by: Franck Nijhof <git@frenck.dev>
2022-10-19 13:04:28 +02:00
31a787558f
Ensure recorder test fixture is setup before hass fixture ( #80528 )
...
* Ensure recorder test fixture is setup before hass fixture
* Adjust more tests
2022-10-19 07:58:47 +02:00
2c206ad050
Fix flaky recorder test ( #80246 )
...
* Fix flaky recorder test
* Update tests/components/recorder/test_init.py
Co-authored-by: epenet <6771947+epenet@users.noreply.github.com>
Co-authored-by: epenet <6771947+epenet@users.noreply.github.com>
2022-10-14 09:10:38 +02:00
04cc2ae264
Correct initialization of new databases ( #80234 )
2022-10-13 13:01:27 +02:00
acb1477673
Avoid time traveling in recorder tests ( #80247 )
2022-10-13 11:51:27 +02:00
4462f2fc46
Fix recorder tests related to mysql ( #80238 )
2022-10-13 11:44:48 +02:00
466c4656ca
Refactor recorder migration ( #80175 )
...
* Refactor recorder migration
* Improve test coverage
2022-10-13 08:11:54 +02:00
4a1c40f09b
Revert "Refactor recorder migration"
...
This reverts commit 69e10e5982
2022-10-12 15:12:12 +02:00
69e10e5982
Refactor recorder migration
2022-10-12 15:00:28 +02:00
9aa6043255
Set character set to utf8mb4 when connecting to MySQL or MariaDB databases ( #79755 )
2022-10-11 14:01:46 +02:00
e8af007124
Disable echo for non SQLite databases ( #80032 )
...
* Disable echo for non SQLite databases
* Add test
2022-10-10 22:42:38 +03:00
a2e3978d53
Don't normalize units of long term statistics ( #79320 )
...
* Don't normalize units of long term statistics
* Update statistics.py
2022-10-03 21:42:44 +02:00
2de273500e
Remove state_unit_of_measurement from metadata DB table ( #79370 )
...
* Remove state_unit_of_measurement from metadata DB table
* Adjust test
2022-10-01 18:55:00 +02:00
bc2dffabc4
Improve naming of units used in statistics ( #79276 )
2022-09-30 08:38:44 +02:00
12e4d18038
Add volume to SensorDeviceClass ( #77960 )
...
* Add volume to SensorDeviceClass
* Adjust recorder
* Adjust tests
* Adjust sensor UNIT_CONVERTERS
* Adjust recorder
* Update strings.json
2022-09-27 19:37:52 +02:00
7c448416e1
Add speed to SensorDeviceClass ( #77953 )
...
* Add speed to SensorDeviceClass
* Adjust recorder
* Adjust tests
* Adjust sensor UNIT_CONVERTERS
* Add tests
* Add websocket tests
* Update strings.json
2022-09-27 17:19:34 +01:00
bfcc18e5b8
Add distance to SensorDeviceClass ( #77951 )
...
* Add distance to SensorDeviceClass
* Adjust recorder
* Adjust tests
* Adjust recorder
* Update __init__.py
* Update test_websocket_api.py
* Update test_websocket_api.py
* Update test_websocket_api.py
* Update strings.json
* Fix tests
* Adjust docstring
2022-09-27 15:34:00 +01:00
c52d0f7495
Support converting statistics to another unit ( #79117 )
2022-09-27 08:44:58 +02:00
df47fda2a0
Remove parametrization in recorder websocket api tests ( #78864 )
...
Remove parametrization in websocket api tests
2022-09-27 08:27:26 +02:00
08ebb9f31a
Test sum AND mean in recorder tests ( #78998 )
2022-09-23 09:55:17 -04:00
cd6697615f
Validate units when importing statistics ( #78891 )
2022-09-21 12:08:53 -04:00
dae00c70de
Allow selecting display unit when fetching statistics ( #78578 )
2022-09-20 17:43:57 -04:00
019d297ff0
Apply hass-relative-import to recorder tests ( #78734 )
2022-09-19 12:35:03 +02:00
dd20a7ea62
Display statistics in the source's unit ( #78031 )
2022-09-15 12:01:24 -04:00
0a1fd36e03
Catch up with statistics after DB migration is done ( #78469 )
...
* Catch up with statistics after DB migration is done
* Don't access the database from the event loop
* Fix deadlocking test
* Fix test
2022-09-15 07:40:56 +02:00
7937bfeedb
Deprecate history integration's statistics API ( #78056 )
2022-09-08 16:03:43 -04:00
008ac8d10d
Improve statistics metadata WS API ( #77209 )
2022-08-31 11:30:45 +02:00
f61edf0778
Fix updating of statistics metadata name ( #77207 )
...
* Fix updating of statistics metadata name
* Fix test
* Test renaming
2022-08-23 17:51:17 +02:00
fd6ffef52f
Support non-live database migration ( #72433 )
...
* Support non-live database migration
* Tweak startup order, add test
* Address review comments
* Fix typo
* Clarify comment about promoting dependencies
* Tweak
* Fix merge mistake
* Fix some tests
* Fix additional test
* Fix additional test
* Adjust tests
* Improve test coverage
2022-07-22 15:11:34 +02:00
606d544157
Use recorder get_instance function to improve typing ( #75567 )
2022-07-22 11:58:26 +02:00
1d7d2875e1
Add websocket command recorder/import_statistics ( #73937 )
...
* Expose ws_add_external_statistics in websocket API
* Refactor
* Add tests
* Improve test coverage
Co-authored-by: Thibault Cohen <titilambert@users.noreply.github.com>
Co-authored-by: Erik <erik@montnemery.com>
2022-07-21 12:36:49 +02:00
a8349a4866
Adjust entity filters to make includes stronger than excludes ( #74080 )
...
* Adjust entity filters to make includes stronger than excludes
Fixes #59080
* adjust test for stronger entity glob includes
* sync with docs
2022-06-28 11:42:51 -05:00
00a79635c1
Revert "Remove sqlalchemy lambda_stmt usage from history, logbook, and statistics ( #73191 )" ( #73917 )
2022-06-23 13:59:55 -05:00
7ae8bd5137
Remove sqlalchemy lambda_stmt usage from history, logbook, and statistics ( #73191 )
2022-06-07 16:15:50 -07:00
5f2b4001f3
Separate recorder database schema from other classes ( #72977 )
...
* Separate recorder database schema from other classes
* fix logbook imports
* migrate new tests
* few more
* last one
* fix merge
Co-authored-by: J. Nick Koston <nick@koston.org>
2022-06-07 14:41:43 +02:00
de2e9b6d77
Fix state_changes_during_period history query when no entities are passed ( #73139 )
2022-06-06 12:50:52 -07:00
c66b000d34
Reduce branching in generated lambda_stmts ( #73042 )
2022-06-05 21:13:31 -07:00
5b31414225
Fix misalignments between sql based filtering with the entityfilter based filtering ( #72936 )
2022-06-02 20:52:53 -07:00
f79e5e002b
Ensure recorder shuts down when its startup future is canceled out from under it ( #72866 )
2022-06-01 22:13:09 -07:00
c6e56c26b3
Fix logbook not setting up with an recorder filter that has empty fields ( #72869 )
2022-06-01 22:12:00 -07:00
d8b0376942
Fix purge of legacy database events that are not state changed ( #72815 )
2022-06-01 11:56:06 +02:00
587fd05603
Make logbook inherit the recorder filter ( #72728 )
2022-05-30 17:34:32 -07:00
040e120101
Fix recorder system health when the db_url is lacking a hostname ( #72612 )
2022-05-27 10:32:26 -07:00
23bd64b7a2
Prevent duplication of statistics metadata ( #71637 )
...
* Prevent duplication of statistics metadata
* Add models_schema_28.py
* Handle entity renaming as a recorder job
* Improve tests
2022-05-24 15:34:46 +02:00
4dbc1ed7a7
Add dialect (database engine) and version to recorder system health data ( #72339 )
2022-05-22 21:29:30 -07:00
9c3f949165
Add live streaming logbook websocket endpoint ( #72258 )
...
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2022-05-22 14:57:54 -05:00
1001f9e39a
Fix last state in history minimal respones when all the states at the end are skipped ( #72203 )
2022-05-19 20:58:32 -07:00
edd7a3427c
Remove support for databases without ROW_NUMBER ( #72092 )
2022-05-18 21:52:38 -05:00
037f6947d8
Fail recorder setup with unsupported dialect or version ( #70888 )
2022-05-18 16:52:46 +02:00
a4c1bcefb9
Tune sqlite based on configured settings ( #72016 )
2022-05-17 18:12:15 -05:00
a70e2a33dc
Fixing purging legacy rows and improve performance ( #71916 )
2022-05-15 21:25:07 -07:00
98809675ff
Convert history queries to use lambda_stmt ( #71870 )
...
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2022-05-15 10:47:29 -05:00
8c2743bb67
Avoid storing last_changed in the database if its the same as last_updated ( #71843 )
2022-05-14 12:06:31 -07:00
a8f1dda004
Use ciso8601 for parsing MySQLdb datetimes ( #71818 )
...
* Use ciso8601 for parsing MySQLDB datetimes
The default parser is this:
5340191feb/MySQLdb/times.py (L66)
2022-05-13 19:26:09 -05:00
7ab4960b1e
Restore v23 stats migration tests ( #71743 )
2022-05-12 21:14:02 -04:00
1d9fb4bca8
Fix process_datetime_to_timestamp and add test coverage ( #71755 )
2022-05-12 17:12:50 -07:00
e2cef55162
Add history/history_during_period websocket endpoint ( #71688 )
2022-05-11 17:52:22 -05:00
222baa53dd
Make database access in the eventloop raise an exception ( #71547 )
2022-05-09 15:22:08 -05:00
7c9c0e911a
Move do_adhoc_statistics to recorder test helpers ( #71544 )
2022-05-08 12:45:57 -07:00
a8aa0e1cca
Add Estimated Database Size to the recorder system health ( #71463 )
2022-05-07 21:02:54 -07:00
eb77f8db85
Complete strict typing for recorder ( #71274 )
...
* Complete strict typing for recorder
* update tests
* Update tests/components/recorder/test_migrate.py
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Update tests/components/recorder/test_migrate.py
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Remove the asserts
* remove ignore comments
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2022-05-04 12:22:50 -05:00
13ce0a7d6a
Fix history using pre v25 queries during v26 migration ( #71295 )
2022-05-04 08:56:50 -07:00
e30940ef2a
Move processing of recorder service call arguments into services.py ( #71260 )
2022-05-03 13:56:22 -07:00
e1be6dd34f
Move recorder services to services.py ( #71249 )
2022-05-03 10:38:44 -07:00
29bda196b5
Break apart recorder into tasks and core modules ( #71222 )
2022-05-02 21:53:56 -07:00
75026f9fed
Separate recorder logic for state_changed and non-state_changed events ( #71204 )
2022-05-02 17:22:53 -05:00
188040b8bb
Use lambda_stmt for recorder queries and migrate them to queries module ( #71219 )
2022-05-02 15:17:21 -07:00
5db014666c
Avoid recording state_changed events in the events table ( #71165 )
...
* squash
fix mypy
* Update homeassistant/components/recorder/models.py
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
* pass all columns
* fix commented out code
* reduce logbook query complexity
* merge
* comment
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2022-05-02 02:10:34 -05:00
c23866e5e5
De-duplicate event data into a new event_data table ( #71135 )
2022-05-01 21:01:17 -05:00
ff48720c6a
Remove db schema v23 stats migration tests ( #71137 )
2022-05-01 16:11:11 -07:00
66a21e0bc3
Add basic system health data to the recorder ( #71086 )
2022-04-30 11:10:20 -05:00
195811843b
Remove get_state and get_states history api calls ( #70830 )
2022-04-26 16:05:43 -07:00
6ce768465f
Improve purge performance for non-sqlite databases ( #70576 )
2022-04-26 13:11:57 -07:00
1c4a785fb3
Prevent autoflush from happening during attrs lookup ( #70768 )
2022-04-26 13:04:58 -07:00
f073f17040
Refactor tracking of the recorder run history ( #70456 )
...
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2022-04-26 09:59:43 -10:00
3016b5fbfc
Use recorder fixtures and helpers in tests ( #70773 )
2022-04-26 06:08:00 -10:00
89e3a48751
Use database executor in migration test ( #70774 )
2022-04-26 07:42:28 -07:00
d045e8678d
Use recorder test fixtures in tests ( #70652 )
2022-04-25 14:23:52 +02:00
28ebab9c5a
Simplify waiting for recorder in tests ( #70647 )
2022-04-25 12:04:47 +02:00
9f11063724
Fix state_changes_during_period bakery caching for limit and descending ( #70610 )
2022-04-24 21:47:21 -07:00
982e314de6
Use recorder_mock in tests ( #70363 )
...
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2022-04-22 19:29:44 -10:00
3737b58e85
Avoid fetching metadata multiple times during stat compile ( #70397 )
2022-04-22 00:25:42 -10:00
3142a106fc
Additional prep work for sqlalchemy 2.0 ( #70358 )
2022-04-21 12:39:18 -10:00
89807f0d2d
Reduce the number of queries needed to compile statistics ( #69731 )
...
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2022-04-15 09:13:29 -07:00
c85387290a
Increase recorder queue max backlog to 40k, improve message ( #70065 )
2022-04-14 11:45:07 -10:00
eb3458a3d2
Add MutexPool for recorder tests ( #69410 )
...
* Add MutexPool for recorder tests
* Fix get_schema_version
* Update test test_last_run_was_recently_clean
* Update test test_shutdown_before_startup_finishes
* Revert comments in test_write_lock_db
* Make the MutexPool lock a class variable
* Remove stale comment
* Move MutexPool
* Tweak debug prints
2022-04-12 07:41:46 -10:00
72fffde77a
Fix spelling of periodic in recorder ( #69658 )
...
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2022-04-07 23:37:02 -10:00
66f0a3816a
Reduce memory pressure during database migration ( #69628 )
2022-04-07 18:29:31 -10:00
949b0e1b65
Don't allow in-memory SQLite database ( #69616 )
2022-04-08 00:43:09 +02:00
4a7c978f69
Speedup recorder test test_write_lock_db ( #69578 )
2022-04-07 15:36:58 +02:00
5c7c09726a
Cleanup recorder history typing ( #69408 )
2022-04-07 00:09:05 -10:00
bde3646293
Always use a commit interval of 0 for the in memory db in tests ( #69330 )
2022-04-05 12:06:06 -10:00
ec131d685e
Auto repack the database on the second sunday of the month ( #69314 )
2022-04-04 21:39:12 -10:00
f5a13fc51b
Add ability to exclude attributes from being recorded by entity domain ( #68824 )
2022-03-29 20:13:08 -07:00
32b2d1e5c9
Add backup platform to recorder ( #68229 )
2022-03-25 23:17:11 -07:00
a566d3943c
Fix history queries while the database migration is in progress ( #68598 )
2022-03-24 09:49:13 -10:00
e911936a0d
Remove direct usage of concurrent.futures from recorder ( #68593 )
2022-03-24 09:48:49 -10:00
61cc8e32f3
Include has_mean + has_sum in statistics metadata WS response ( #68546 )
...
* Include has_mean + has_sum in statistics metadata WS response
* Don't include has_mean/has_sum in history/list_statistic_ids
* Adjust tests
* Do include has_mean/has_sum in history/list_statistic_ids
2022-03-24 10:12:01 +01:00
b5c5da96ac
Add WS API to adjust incorrect energy statistics ( #65147 )
...
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2022-03-22 15:18:30 -07:00
bcb8c7ec3c
Add API endpoint get_statistics_metadata ( #68471 )
...
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2022-03-21 21:14:47 -07:00
3150915cb7
Convert unindexed domain queries to entity_id queries ( #68404 )
2022-03-20 01:28:17 -10:00
816695cc96
Avoid selecting attributes in the history api when `no_attributes` is passed ( #68352 )
2022-03-19 23:47:22 -10:00
cf4033b1bc
Simplify time zone setting in tests ( #68330 )
...
* Simplify timezone setting in tests
* Fix typo
* Adjust caldav tests
* Adjust input_datetime tests
* Adjust time_date tests
* Adjust tod tests
* Adjust helper tests
* Adjust recorder tests
* Adjust risco tests
* Adjust aemet tests
* Adjust flux tests
* Adjust forecast_solar tests
* Revert unnecessary change in forecast_solar test
* Adjust climacell tests
* Adjust google tests
* Adjust sensor tests
* Adjust sonarr tests
* Adjust template tests
* Adjust zodiac tests
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2022-03-20 10:25:15 +01:00
7ee647cc78
Fix FOREIGN KEY constraint failed when removing state_attributes ( #68364 )
2022-03-19 14:21:28 -07:00
9215702388
Separate attrs into another table (reduces database size) ( #68224 )
2022-03-18 00:23:13 -10:00
bc862e97ed
Use a dedicated executor pool for database operations ( #68105 )
...
Co-authored-by: Erik Montnemery <erik@montnemery.com>
Co-authored-by: Franck Nijhof <git@frenck.dev>
2022-03-17 23:09:01 -10:00
95f20500ca
Commit any pending changes before running non-EventTasks in the recorder ( #68287 )
2022-03-17 21:33:22 -07:00
d360ac91ca
Disable recorder nightly jobs in tests ( #68188 )
2022-03-15 09:23:52 -07:00
b18096fc54
Remove unused columns from states/events tables ( #68078 )
2022-03-13 23:44:55 -07:00
910b1f1ec8
Speed up deletion of duplicated statistics ( #66014 )
2022-02-07 09:11:52 -08:00
41ab12cb88
Don't use shared session during recorder migration ( #65672 )
2022-02-04 09:55:11 -08:00
ac7662c82d
Remove limit of amount of duplicated statistics ( #65641 )
2022-02-04 14:49:45 +01:00
d057850971
Update typing - collections.abc (3) ( #63947 )
2022-01-12 08:04:17 +01:00
0a9927d18e
Avoid locking the database for non-SQLite backends ( #63847 )
...
* Avoid locking the database for non-SQLite backends
Currently we only have a lock implementation for SQLite. Just return
success for all other databases as they are not expected to store data
in the config directory and the caller can assume that a backup can
be safely taken.
This fixes `RuntimeError: generator didn't yield` errors when creating
a backup with the current Supervisor dev builds.
2022-01-11 16:17:56 +01:00
2f8e65a9b0
Store deleted duplicated statistics in .storage ( #62574 )
2021-12-22 14:27:56 +01:00
48b3d6e1c0
Save original + duplicate pairs when deleting duplicated statistics ( #62498 )
2021-12-21 15:24:36 +01:00
cab2a74b5f
Don't pollute config dir with deleted duplicated statistics ( #62489 )
2021-12-21 14:27:35 +01:00
474ef54477
Fix threading error in recorder tests ( #62187 )
2021-12-17 13:17:48 +01:00
bceeaec2f8
Remove duplicated statistics rows ( #61146 )
...
* Remove duplicated statistics
* Fix misleading docstring
* Pylint knows best
* Correct test
* Oops
* Prevent insertion of duplicated statistics
* Tweak
* pylint
* Add models_schema_23.py
* Tweak
2021-12-13 14:15:36 +01:00
dc5888ab4a
Correct recorder.statistics.get_last_statistics ( #61421 )
2021-12-10 10:09:29 -08:00
a13ae85982
Introduce only_supervisor for @websocket_api.ws_require_user() ( #61298 )
2021-12-08 16:49:35 -08:00
f30eb05870
Refactor recorder queue handling ( #61161 )
...
* Refactor recorder queue handling
* Address pylint's concerns
* Implement workaround for mypy bug
* Address review comments
2021-12-08 16:54:26 +01:00
f0006b92be
Allow to lock SQLite database during backup ( #60874 )
...
* Allow to set CONF_DB_URL
This is useful for test which need a custom DB path.
* Introduce write_lock_db helper to lock SQLite database
* Introduce Websocket API which allows to lock database during backup
* Fix isort
* Avoid mutable default arguments
* Address pylint issues
* Avoid holding executor thread
* Set unlock event in case timeout occures
This makes sure the database is left unlocked even in case of a race
condition.
* Add more unit tests
* Address new pylint errors
* Lower timeout to speedup tests
* Introduce queue overflow test
* Unlock database if necessary
This makes sure that the test runs through in case locking actually
succeeds (and the test fails).
* Make DB_LOCK_TIMEOUT a global
There is no good reason for this to be an argument. The recorder needs
to pick a sensible value.
* Add Websocket Timeout test
* Test lock_database() return
* Update homeassistant/components/recorder/__init__.py
Co-authored-by: Erik Montnemery <erik@montnemery.com>
* Fix format
Co-authored-by: J. Nick Koston <nick@koston.org>
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2021-12-07 13:16:24 +01:00
bcd9f3c05f
Correct end time for monthly statistics summary ( #59551 )
...
* Correct end time for monthly statistics summary
* Add tests
2021-11-12 15:01:40 +01:00
e5ee38d3b0
Adjust recorder tests according to #59087 ( #59143 )
2021-11-05 10:49:58 +01:00
470b01e4ce
Move async_migration_in_progress ( #59087 )
2021-11-05 10:40:56 +01:00
185f7beafc
Improve recorder migration tests ( #59075 )
2021-11-04 20:21:38 -07:00
4c5aca93df
Add recorder status WS API ( #58989 )
...
* Add recorder status WS API
* Rename recorder/status to recorder/info
* Silence pylint
* Improve tests
* Address review comments
* Tweak
* Try to fix tests
* Try to debug flaky tests
* Try to fix tests
* Revert changes to async_migration_in_progress
* Try to fix tests
* Remove debug prints
* Apply suggestions from code review
2021-11-04 16:46:45 +01:00
be4e9f91b6
Change minimum supported SQLite version to 3.31.0 ( #59073 )
2021-11-04 16:34:35 +01:00
dfa50a842a
Simplify recorder PgSQL version checks ( #58533 )
2021-10-27 16:05:40 +02:00
ac5e32d648
Corrections for external statistics ( #58469 )
2021-10-26 14:05:45 +02:00
e9ba5f3b4b
Warn when recorder connects to an unsupported database ( #58161 )
2021-10-26 13:41:59 +02:00
f594bc353b
Add support for external statistics ( #56607 )
...
* Support external statistics
* Update tests
* Lint
* Adjust code after rebase
* Separate external statistic_id with :, add name to metadata
* Adjust tests
* Simplify get_metadata_with_session
* Address review comments
* Allow updating external statistics
* Validate input
* Adjust tests after rebase
* Pylint
* Adjust tests
* Improve test coverage
2021-10-26 10:26:50 +02:00
b301ab25a3
Purge short term statistics ( #58028 )
...
* Purge short term statistics
* Less meep
* Add tests
2021-10-20 16:00:59 +02:00
8ef8838801
Correct detection of row_number support for MariaDB ( #57663 )
2021-10-14 11:19:39 -07:00
0139bfa749
Detect if mysql and sqlite support row_number ( #57475 )
2021-10-11 21:17:18 -07:00
a4d9019ffc
Refactor persistent notification to no longer route all data via a service ( #57157 )
...
* Convert persistent notification tests to async
* Create/dismiss persistent notifications in exposed functions, not service calls
* Fix notify persistent_notification
* Remove setting up persistent_notification
* Drop more setups
* Empty methods
* Undeprecate sync methods because too big task
* Fix setup clearing notifications
* Fix a bunch of tests
* Fix more tests
* Uno mas
* Test persistent notification events
* Clean up stale comment
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-10-07 12:58:00 +02:00
69875cbd11
Improve sensor statistics validation ( #56892 )
2021-10-04 09:47:44 -07:00
8567aa9e13
Evict purged states from recorder's old_state cache ( #56877 )
...
Co-authored-by: J. Nick Koston <nick@koston.org>
2021-10-04 08:21:40 -07:00
40ecf22bac
Remove automatic splitting of net meters from statistics ( #56772 )
2021-09-29 17:20:17 +02:00
00651a4055
Optimize _get_states_with_session ( #56734 )
...
* Optimize _get_states_with_session
* Move custom filters to derived table
* Remove useless derived table
* Filter old states after grouping
* Split query
* Add comments
* Simplify state update period criteria
* Only apply custom filters if we didn't get an include list of entities
Co-authored-by: J. Nick Koston <nick@koston.org>
2021-09-29 17:08:27 +02:00
8ef123259e
Add WS API for updating unit_of_measurement in statistics metadata ( #56184 )
...
* Add WS API for updating statistics metadata
* Update homeassistant/components/recorder/websocket_api.py
Co-authored-by: Bram Kragten <mail@bramkragten.nl>
* Update homeassistant/components/recorder/websocket_api.py
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
* Fix typo
Co-authored-by: Bram Kragten <mail@bramkragten.nl>
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2021-09-28 00:43:29 +02:00
5976f898da
Add WS API for removing statistics for a list of statistic_ids ( #55078 )
...
* Add WS API for removing statistics for a list of statistic_ids
* Refactor according to code review, enable foreign keys support for sqlite
* Adjust tests
* Move clear_statistics WS API to recorder
* Adjust tests after rebase
* Update docstring
* Update homeassistant/components/recorder/websocket_api.py
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
* Adjust tests after rebase
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2021-09-27 23:30:13 +02:00
64393b462d
Add migration for 5-minute statistics ( #56585 )
...
* Add migration for 5-minute statistics
* Tweaks
2021-09-24 09:19:22 +02:00
92253f5192
Minor refactoring of periodic statistics ( #56492 )
2021-09-22 13:31:33 -07:00
6f36419c6f
Improve statistics validation ( #56457 )
2021-09-20 17:54:25 +02:00
8c5efafdd8
Add 5-minute statistics for sensors ( #56006 )
...
* Add 5-minute statistics for sensors
* Address pylint issues
* Black
* Apply suggestion from code review
* Apply suggestions from code review
* Improve tests
2021-09-16 10:57:15 +02:00
d899d15a1e
Add statistics validation ( #56020 )
...
* Add statistics validation
* Remove redundant None-check
* Move validate_statistics WS API to recorder
* Apply suggestion from code review
2021-09-13 13:44:22 +02:00
d2a9f7904a
Include end time of statistics data points in API response ( #56063 )
...
* Include end time of statistics data points in API response
* Correct typing
* Update tests
2021-09-13 10:02:24 +02:00
80fd330479
Add sum_decrease and sum_increase statistics ( #55850 )
2021-09-08 23:35:53 -07:00
27764e9985
Fix handling of imperial units in long term statistics ( #55959 )
2021-09-08 08:08:48 -07:00
22e6ddf8df
Do not let one bad statistic spoil the bunch ( #55942 )
2021-09-08 07:55:40 -07:00
8877f37da0
Fix statistics for sensors setting last_reset ( #55136 )
...
* Re-add state_class total to sensor
* Make energy cost sensor enforce state_class total_increasing
* Drop state_class total
* Only report energy sensor issues once
2021-08-24 17:02:34 +02:00
32e297f4a0
Compile missing statistics ( #54690 )
2021-08-19 22:10:45 -07:00
e1926caeb9
Remove STATE_CLASS_TOTAL and last_reset from sensor ( #54755 )
...
* Remove STATE_CLASS_TOTAL
* Update mill sensor
* Update tests
* Kill last_reset
* Return ATTR_LAST_RESET to utility_meter
* Update energy cost sensor
* Restore last_reset for backwards compatibility
* Re-add and update deprecation warning
* Update tests
* Fix utility_meter
* Update EnergyCostSensor
* Tweak
* Fix rebase mistake
* Fix test
2021-08-18 10:03:27 +02:00
557cc792e9
Fix SQLAlchemy test warnings ( #54116 )
2021-08-08 20:33:47 -07:00
e541bcd54d
Update statistics meta data on entity_id change ( #52755 )
2021-07-14 13:23:11 +02:00
11fd9d9525
Avoid duplicated database queries when fetching statistics ( #52433 )
2021-07-02 15:40:54 +02:00
0476c7f9ee
Normalize temperature statistics to °C ( #52297 )
...
* Normalize temperature statistics to °C
* Fix tests
* Support temperature conversion to and from K, improve tests
* Fix test
* Add tests, pylint
2021-06-30 14:17:58 +02:00
42c4317628
Avoid drift in recorder purge cut-off ( #52135 )
2021-06-25 11:29:38 -10:00
805ef3f90b
Allow fetching multiple statistics ( #51996 )
2021-06-18 21:03:13 +02:00
39c94e8daa
Fix flaky statistics tests ( #51214 )
...
* Fix flaky statistics tests
* Tweak
2021-05-28 22:37:17 +02:00
b339d73109
Weight sensor average statistics by state durations ( #51150 )
...
* Weight sensor average statistics by state durations
* Fix test
2021-05-28 13:16:52 +02:00
aa9b99713c
Add purge_entities service call to recorder ( #48069 )
2021-05-22 10:30:05 -05:00
e16a8063a5
Compile statistics for energy sensors ( #50829 )
...
* Compile statistics for energy sensors
* Update tests
* Rename abs_value to state
* Tweak
* Recreate statistics table
* Pylint
* Try to fix test
* Fix statistics for multiple energy sensors
* Fix energy statistics when last_reset is not set
2021-05-20 13:05:15 +02:00
e7f7e61e88
Ensure a wal checkpoint is scheduled nightly ( #50746 )
2021-05-17 16:27:51 -05:00
72288710ca
Increase the sqlite cache size from ~2MiB to 8MiB ( #50747 )
2021-05-17 19:42:12 +02:00
89dd3292ba
Initial draft of statistics ( #49852 )
2021-05-16 10:23:37 -07:00
973f59e423
Refactor history component ( #50287 )
...
* Refactor history component
* Update tests
* Address review comments
* Correct deprecated functions
2021-05-11 09:21:57 +02:00
bf2d40adfe
Migrate from pytz to python-dateutil ( #49643 )
...
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2021-05-07 22:46:26 -07:00
1c8d9ca68b
Check exception causes for matching strings during recorder migration ( #49999 )
2021-05-02 20:57:42 -07:00
1bd9826684
Handle different entity_id formats ( #49969 )
2021-05-01 20:30:28 -07:00
J. Nick Koston
epenet
Erik Montnemery
Franck Nijhof
Marc Mueller
Michael
Joakim Plate
Thibault Cohen
Joakim Sørensen
Pawel
Stefan Agner
Paulus Schoutsen
PeteBa