Commit Graph

173 Commits (f9aa0a7cd887456857ae7ed9466f8ded9a7cbbf5)

Author SHA1 Message Date
J. Nick Koston f5a13fc51b
Add ability to exclude attributes from being recorded by entity domain (#68824) 2022-03-29 20:13:08 -07:00
Joakim Sørensen 32b2d1e5c9
Add backup platform to recorder (#68229) 2022-03-25 23:17:11 -07:00
J. Nick Koston a566d3943c
Fix history queries while the database migration is in progress (#68598) 2022-03-24 09:49:13 -10:00
J. Nick Koston e911936a0d
Remove direct usage of concurrent.futures from recorder (#68593) 2022-03-24 09:48:49 -10:00
Erik Montnemery 61cc8e32f3
Include has_mean + has_sum in statistics metadata WS response (#68546)
* Include has_mean + has_sum in statistics metadata WS response

* Don't include has_mean/has_sum in history/list_statistic_ids

* Adjust tests

* Do include has_mean/has_sum in history/list_statistic_ids
2022-03-24 10:12:01 +01:00
Erik Montnemery b5c5da96ac
Add WS API to adjust incorrect energy statistics (#65147)
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2022-03-22 15:18:30 -07:00
Pawel bcb8c7ec3c
Add API endpoint get_statistics_metadata (#68471)
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2022-03-21 21:14:47 -07:00
J. Nick Koston 3150915cb7
Convert unindexed domain queries to entity_id queries (#68404) 2022-03-20 01:28:17 -10:00
J. Nick Koston 816695cc96
Avoid selecting attributes in the history api when `no_attributes` is passed (#68352) 2022-03-19 23:47:22 -10:00
Erik Montnemery cf4033b1bc
Simplify time zone setting in tests (#68330)
* Simplify timezone setting in tests

* Fix typo

* Adjust caldav tests

* Adjust input_datetime tests

* Adjust time_date tests

* Adjust tod tests

* Adjust helper tests

* Adjust recorder tests

* Adjust risco tests

* Adjust aemet tests

* Adjust flux tests

* Adjust forecast_solar tests

* Revert unnecessary change in forecast_solar test

* Adjust climacell tests

* Adjust google tests

* Adjust sensor tests

* Adjust sonarr tests

* Adjust template tests

* Adjust zodiac tests

Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2022-03-20 10:25:15 +01:00
J. Nick Koston 7ee647cc78
Fix FOREIGN KEY constraint failed when removing state_attributes (#68364) 2022-03-19 14:21:28 -07:00
J. Nick Koston 9215702388
Separate attrs into another table (reduces database size) (#68224) 2022-03-18 00:23:13 -10:00
J. Nick Koston bc862e97ed
Use a dedicated executor pool for database operations (#68105)
Co-authored-by: Erik Montnemery <erik@montnemery.com>
Co-authored-by: Franck Nijhof <git@frenck.dev>
2022-03-17 23:09:01 -10:00
J. Nick Koston 95f20500ca
Commit any pending changes before running non-EventTasks in the recorder (#68287) 2022-03-17 21:33:22 -07:00
Erik Montnemery d360ac91ca
Disable recorder nightly jobs in tests (#68188) 2022-03-15 09:23:52 -07:00
J. Nick Koston b18096fc54
Remove unused columns from states/events tables (#68078) 2022-03-13 23:44:55 -07:00
Erik Montnemery 910b1f1ec8
Speed up deletion of duplicated statistics (#66014) 2022-02-07 09:11:52 -08:00
Erik Montnemery 41ab12cb88
Don't use shared session during recorder migration (#65672) 2022-02-04 09:55:11 -08:00
Erik Montnemery ac7662c82d
Remove limit of amount of duplicated statistics (#65641) 2022-02-04 14:49:45 +01:00
Marc Mueller d057850971
Update typing - collections.abc (3) (#63947) 2022-01-12 08:04:17 +01:00
Stefan Agner 0a9927d18e
Avoid locking the database for non-SQLite backends (#63847)
* Avoid locking the database for non-SQLite backends

Currently we only have a lock implementation for SQLite. Just return
success for all other databases as they are not expected to store data
in the config directory and the caller can assume that a backup can
be safely taken.

This fixes `RuntimeError: generator didn't yield` errors when creating
a backup with the current Supervisor dev builds.
2022-01-11 16:17:56 +01:00
Erik Montnemery 2f8e65a9b0
Store deleted duplicated statistics in .storage (#62574) 2021-12-22 14:27:56 +01:00
Erik Montnemery 48b3d6e1c0
Save original + duplicate pairs when deleting duplicated statistics (#62498) 2021-12-21 15:24:36 +01:00
Erik Montnemery cab2a74b5f
Don't pollute config dir with deleted duplicated statistics (#62489) 2021-12-21 14:27:35 +01:00
Erik Montnemery 474ef54477
Fix threading error in recorder tests (#62187) 2021-12-17 13:17:48 +01:00
Erik Montnemery bceeaec2f8
Remove duplicated statistics rows (#61146)
* Remove duplicated statistics

* Fix misleading docstring

* Pylint knows best

* Correct test

* Oops

* Prevent insertion of duplicated statistics

* Tweak

* pylint

* Add models_schema_23.py

* Tweak
2021-12-13 14:15:36 +01:00
Erik Montnemery dc5888ab4a
Correct recorder.statistics.get_last_statistics (#61421) 2021-12-10 10:09:29 -08:00
Stefan Agner a13ae85982
Introduce only_supervisor for @websocket_api.ws_require_user() (#61298) 2021-12-08 16:49:35 -08:00
Erik Montnemery f30eb05870
Refactor recorder queue handling (#61161)
* Refactor recorder queue handling

* Address pylint's concerns

* Implement workaround for mypy bug

* Address review comments
2021-12-08 16:54:26 +01:00
Stefan Agner f0006b92be
Allow to lock SQLite database during backup (#60874)
* Allow to set CONF_DB_URL

This is useful for test which need a custom DB path.

* Introduce write_lock_db helper to lock SQLite database

* Introduce Websocket API which allows to lock database during backup

* Fix isort

* Avoid mutable default arguments

* Address pylint issues

* Avoid holding executor thread

* Set unlock event in case timeout occures

This makes sure the database is left unlocked even in case of a race
condition.

* Add more unit tests

* Address new pylint errors

* Lower timeout to speedup tests

* Introduce queue overflow test

* Unlock database if necessary

This makes sure that the test runs through in case locking actually
succeeds (and the test fails).

* Make DB_LOCK_TIMEOUT a global

There is no good reason for this to be an argument. The recorder needs
to pick a sensible value.

* Add Websocket Timeout test

* Test lock_database() return

* Update homeassistant/components/recorder/__init__.py

Co-authored-by: Erik Montnemery <erik@montnemery.com>

* Fix format

Co-authored-by: J. Nick Koston <nick@koston.org>
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2021-12-07 13:16:24 +01:00
Erik Montnemery bcd9f3c05f
Correct end time for monthly statistics summary (#59551)
* Correct end time for monthly statistics summary

* Add tests
2021-11-12 15:01:40 +01:00
Erik Montnemery e5ee38d3b0
Adjust recorder tests according to #59087 (#59143) 2021-11-05 10:49:58 +01:00
Erik Montnemery 470b01e4ce
Move async_migration_in_progress (#59087) 2021-11-05 10:40:56 +01:00
Erik Montnemery 185f7beafc
Improve recorder migration tests (#59075) 2021-11-04 20:21:38 -07:00
Erik Montnemery 4c5aca93df
Add recorder status WS API (#58989)
* Add recorder status WS API

* Rename recorder/status to recorder/info

* Silence pylint

* Improve tests

* Address review comments

* Tweak

* Try to fix tests

* Try to debug flaky tests

* Try to fix tests

* Revert changes to async_migration_in_progress

* Try to fix tests

* Remove debug prints

* Apply suggestions from code review
2021-11-04 16:46:45 +01:00
Erik Montnemery be4e9f91b6
Change minimum supported SQLite version to 3.31.0 (#59073) 2021-11-04 16:34:35 +01:00
Erik Montnemery dfa50a842a
Simplify recorder PgSQL version checks (#58533) 2021-10-27 16:05:40 +02:00
Erik Montnemery ac5e32d648
Corrections for external statistics (#58469) 2021-10-26 14:05:45 +02:00
Erik Montnemery e9ba5f3b4b
Warn when recorder connects to an unsupported database (#58161) 2021-10-26 13:41:59 +02:00
Erik Montnemery f594bc353b
Add support for external statistics (#56607)
* Support external statistics

* Update tests

* Lint

* Adjust code after rebase

* Separate external statistic_id with :, add name to metadata

* Adjust tests

* Simplify get_metadata_with_session

* Address review comments

* Allow updating external statistics

* Validate input

* Adjust tests after rebase

* Pylint

* Adjust tests

* Improve test coverage
2021-10-26 10:26:50 +02:00
Erik Montnemery b301ab25a3
Purge short term statistics (#58028)
* Purge short term statistics

* Less meep

* Add tests
2021-10-20 16:00:59 +02:00
Erik Montnemery 8ef8838801
Correct detection of row_number support for MariaDB (#57663) 2021-10-14 11:19:39 -07:00
Erik Montnemery 0139bfa749
Detect if mysql and sqlite support row_number (#57475) 2021-10-11 21:17:18 -07:00
Paulus Schoutsen a4d9019ffc
Refactor persistent notification to no longer route all data via a service (#57157)
* Convert persistent notification tests to async

* Create/dismiss persistent notifications in exposed functions, not service calls

* Fix notify persistent_notification

* Remove setting up persistent_notification

* Drop more setups

* Empty methods

* Undeprecate sync methods because too big task

* Fix setup clearing notifications

* Fix a bunch of tests

* Fix more tests

* Uno mas

* Test persistent notification events

* Clean up stale comment

Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-10-07 12:58:00 +02:00
Erik Montnemery 69875cbd11
Improve sensor statistics validation (#56892) 2021-10-04 09:47:44 -07:00
Erik Montnemery 8567aa9e13
Evict purged states from recorder's old_state cache (#56877)
Co-authored-by: J. Nick Koston <nick@koston.org>
2021-10-04 08:21:40 -07:00
Erik Montnemery 40ecf22bac
Remove automatic splitting of net meters from statistics (#56772) 2021-09-29 17:20:17 +02:00
Erik Montnemery 00651a4055
Optimize _get_states_with_session (#56734)
* Optimize _get_states_with_session

* Move custom filters to derived table

* Remove useless derived table

* Filter old states after grouping

* Split query

* Add comments

* Simplify state update period criteria

* Only apply custom filters if we didn't get an include list of entities

Co-authored-by: J. Nick Koston <nick@koston.org>
2021-09-29 17:08:27 +02:00
Erik Montnemery 8ef123259e
Add WS API for updating unit_of_measurement in statistics metadata (#56184)
* Add WS API for updating statistics metadata

* Update homeassistant/components/recorder/websocket_api.py

Co-authored-by: Bram Kragten <mail@bramkragten.nl>

* Update homeassistant/components/recorder/websocket_api.py

Co-authored-by: Paulus Schoutsen <balloob@gmail.com>

* Fix typo

Co-authored-by: Bram Kragten <mail@bramkragten.nl>
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2021-09-28 00:43:29 +02:00
Erik Montnemery 5976f898da
Add WS API for removing statistics for a list of statistic_ids (#55078)
* Add WS API for removing statistics for a list of statistic_ids

* Refactor according to code review, enable foreign keys support for sqlite

* Adjust tests

* Move clear_statistics WS API to recorder

* Adjust tests after rebase

* Update docstring

* Update homeassistant/components/recorder/websocket_api.py

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>

* Adjust tests after rebase

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2021-09-27 23:30:13 +02:00