28ebab9c5a
Simplify waiting for recorder in tests ( #70647 )
2022-04-25 12:04:47 +02:00
9f11063724
Fix state_changes_during_period bakery caching for limit and descending ( #70610 )
2022-04-24 21:47:21 -07:00
982e314de6
Use recorder_mock in tests ( #70363 )
...
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2022-04-22 19:29:44 -10:00
3737b58e85
Avoid fetching metadata multiple times during stat compile ( #70397 )
2022-04-22 00:25:42 -10:00
3142a106fc
Additional prep work for sqlalchemy 2.0 ( #70358 )
2022-04-21 12:39:18 -10:00
89807f0d2d
Reduce the number of queries needed to compile statistics ( #69731 )
...
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2022-04-15 09:13:29 -07:00
c85387290a
Increase recorder queue max backlog to 40k, improve message ( #70065 )
2022-04-14 11:45:07 -10:00
eb3458a3d2
Add MutexPool for recorder tests ( #69410 )
...
* Add MutexPool for recorder tests
* Fix get_schema_version
* Update test test_last_run_was_recently_clean
* Update test test_shutdown_before_startup_finishes
* Revert comments in test_write_lock_db
* Make the MutexPool lock a class variable
* Remove stale comment
* Move MutexPool
* Tweak debug prints
2022-04-12 07:41:46 -10:00
72fffde77a
Fix spelling of periodic in recorder ( #69658 )
...
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2022-04-07 23:37:02 -10:00
66f0a3816a
Reduce memory pressure during database migration ( #69628 )
2022-04-07 18:29:31 -10:00
949b0e1b65
Don't allow in-memory SQLite database ( #69616 )
2022-04-08 00:43:09 +02:00
4a7c978f69
Speedup recorder test test_write_lock_db ( #69578 )
2022-04-07 15:36:58 +02:00
5c7c09726a
Cleanup recorder history typing ( #69408 )
2022-04-07 00:09:05 -10:00
bde3646293
Always use a commit interval of 0 for the in memory db in tests ( #69330 )
2022-04-05 12:06:06 -10:00
ec131d685e
Auto repack the database on the second sunday of the month ( #69314 )
2022-04-04 21:39:12 -10:00
f5a13fc51b
Add ability to exclude attributes from being recorded by entity domain ( #68824 )
2022-03-29 20:13:08 -07:00
32b2d1e5c9
Add backup platform to recorder ( #68229 )
2022-03-25 23:17:11 -07:00
a566d3943c
Fix history queries while the database migration is in progress ( #68598 )
2022-03-24 09:49:13 -10:00
e911936a0d
Remove direct usage of concurrent.futures from recorder ( #68593 )
2022-03-24 09:48:49 -10:00
61cc8e32f3
Include has_mean + has_sum in statistics metadata WS response ( #68546 )
...
* Include has_mean + has_sum in statistics metadata WS response
* Don't include has_mean/has_sum in history/list_statistic_ids
* Adjust tests
* Do include has_mean/has_sum in history/list_statistic_ids
2022-03-24 10:12:01 +01:00
b5c5da96ac
Add WS API to adjust incorrect energy statistics ( #65147 )
...
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2022-03-22 15:18:30 -07:00
bcb8c7ec3c
Add API endpoint get_statistics_metadata ( #68471 )
...
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2022-03-21 21:14:47 -07:00
3150915cb7
Convert unindexed domain queries to entity_id queries ( #68404 )
2022-03-20 01:28:17 -10:00
816695cc96
Avoid selecting attributes in the history api when `no_attributes` is passed ( #68352 )
2022-03-19 23:47:22 -10:00
cf4033b1bc
Simplify time zone setting in tests ( #68330 )
...
* Simplify timezone setting in tests
* Fix typo
* Adjust caldav tests
* Adjust input_datetime tests
* Adjust time_date tests
* Adjust tod tests
* Adjust helper tests
* Adjust recorder tests
* Adjust risco tests
* Adjust aemet tests
* Adjust flux tests
* Adjust forecast_solar tests
* Revert unnecessary change in forecast_solar test
* Adjust climacell tests
* Adjust google tests
* Adjust sensor tests
* Adjust sonarr tests
* Adjust template tests
* Adjust zodiac tests
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2022-03-20 10:25:15 +01:00
7ee647cc78
Fix FOREIGN KEY constraint failed when removing state_attributes ( #68364 )
2022-03-19 14:21:28 -07:00
9215702388
Separate attrs into another table (reduces database size) ( #68224 )
2022-03-18 00:23:13 -10:00
bc862e97ed
Use a dedicated executor pool for database operations ( #68105 )
...
Co-authored-by: Erik Montnemery <erik@montnemery.com>
Co-authored-by: Franck Nijhof <git@frenck.dev>
2022-03-17 23:09:01 -10:00
95f20500ca
Commit any pending changes before running non-EventTasks in the recorder ( #68287 )
2022-03-17 21:33:22 -07:00
d360ac91ca
Disable recorder nightly jobs in tests ( #68188 )
2022-03-15 09:23:52 -07:00
b18096fc54
Remove unused columns from states/events tables ( #68078 )
2022-03-13 23:44:55 -07:00
910b1f1ec8
Speed up deletion of duplicated statistics ( #66014 )
2022-02-07 09:11:52 -08:00
41ab12cb88
Don't use shared session during recorder migration ( #65672 )
2022-02-04 09:55:11 -08:00
ac7662c82d
Remove limit of amount of duplicated statistics ( #65641 )
2022-02-04 14:49:45 +01:00
d057850971
Update typing - collections.abc (3) ( #63947 )
2022-01-12 08:04:17 +01:00
0a9927d18e
Avoid locking the database for non-SQLite backends ( #63847 )
...
* Avoid locking the database for non-SQLite backends
Currently we only have a lock implementation for SQLite. Just return
success for all other databases as they are not expected to store data
in the config directory and the caller can assume that a backup can
be safely taken.
This fixes `RuntimeError: generator didn't yield` errors when creating
a backup with the current Supervisor dev builds.
2022-01-11 16:17:56 +01:00
2f8e65a9b0
Store deleted duplicated statistics in .storage ( #62574 )
2021-12-22 14:27:56 +01:00
48b3d6e1c0
Save original + duplicate pairs when deleting duplicated statistics ( #62498 )
2021-12-21 15:24:36 +01:00
cab2a74b5f
Don't pollute config dir with deleted duplicated statistics ( #62489 )
2021-12-21 14:27:35 +01:00
474ef54477
Fix threading error in recorder tests ( #62187 )
2021-12-17 13:17:48 +01:00
bceeaec2f8
Remove duplicated statistics rows ( #61146 )
...
* Remove duplicated statistics
* Fix misleading docstring
* Pylint knows best
* Correct test
* Oops
* Prevent insertion of duplicated statistics
* Tweak
* pylint
* Add models_schema_23.py
* Tweak
2021-12-13 14:15:36 +01:00
dc5888ab4a
Correct recorder.statistics.get_last_statistics ( #61421 )
2021-12-10 10:09:29 -08:00
a13ae85982
Introduce only_supervisor for @websocket_api.ws_require_user() ( #61298 )
2021-12-08 16:49:35 -08:00
f30eb05870
Refactor recorder queue handling ( #61161 )
...
* Refactor recorder queue handling
* Address pylint's concerns
* Implement workaround for mypy bug
* Address review comments
2021-12-08 16:54:26 +01:00
f0006b92be
Allow to lock SQLite database during backup ( #60874 )
...
* Allow to set CONF_DB_URL
This is useful for test which need a custom DB path.
* Introduce write_lock_db helper to lock SQLite database
* Introduce Websocket API which allows to lock database during backup
* Fix isort
* Avoid mutable default arguments
* Address pylint issues
* Avoid holding executor thread
* Set unlock event in case timeout occures
This makes sure the database is left unlocked even in case of a race
condition.
* Add more unit tests
* Address new pylint errors
* Lower timeout to speedup tests
* Introduce queue overflow test
* Unlock database if necessary
This makes sure that the test runs through in case locking actually
succeeds (and the test fails).
* Make DB_LOCK_TIMEOUT a global
There is no good reason for this to be an argument. The recorder needs
to pick a sensible value.
* Add Websocket Timeout test
* Test lock_database() return
* Update homeassistant/components/recorder/__init__.py
Co-authored-by: Erik Montnemery <erik@montnemery.com>
* Fix format
Co-authored-by: J. Nick Koston <nick@koston.org>
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2021-12-07 13:16:24 +01:00
bcd9f3c05f
Correct end time for monthly statistics summary ( #59551 )
...
* Correct end time for monthly statistics summary
* Add tests
2021-11-12 15:01:40 +01:00
e5ee38d3b0
Adjust recorder tests according to #59087 ( #59143 )
2021-11-05 10:49:58 +01:00
470b01e4ce
Move async_migration_in_progress ( #59087 )
2021-11-05 10:40:56 +01:00
185f7beafc
Improve recorder migration tests ( #59075 )
2021-11-04 20:21:38 -07:00
4c5aca93df
Add recorder status WS API ( #58989 )
...
* Add recorder status WS API
* Rename recorder/status to recorder/info
* Silence pylint
* Improve tests
* Address review comments
* Tweak
* Try to fix tests
* Try to debug flaky tests
* Try to fix tests
* Revert changes to async_migration_in_progress
* Try to fix tests
* Remove debug prints
* Apply suggestions from code review
2021-11-04 16:46:45 +01:00
be4e9f91b6
Change minimum supported SQLite version to 3.31.0 ( #59073 )
2021-11-04 16:34:35 +01:00
dfa50a842a
Simplify recorder PgSQL version checks ( #58533 )
2021-10-27 16:05:40 +02:00
ac5e32d648
Corrections for external statistics ( #58469 )
2021-10-26 14:05:45 +02:00
e9ba5f3b4b
Warn when recorder connects to an unsupported database ( #58161 )
2021-10-26 13:41:59 +02:00
f594bc353b
Add support for external statistics ( #56607 )
...
* Support external statistics
* Update tests
* Lint
* Adjust code after rebase
* Separate external statistic_id with :, add name to metadata
* Adjust tests
* Simplify get_metadata_with_session
* Address review comments
* Allow updating external statistics
* Validate input
* Adjust tests after rebase
* Pylint
* Adjust tests
* Improve test coverage
2021-10-26 10:26:50 +02:00
b301ab25a3
Purge short term statistics ( #58028 )
...
* Purge short term statistics
* Less meep
* Add tests
2021-10-20 16:00:59 +02:00
8ef8838801
Correct detection of row_number support for MariaDB ( #57663 )
2021-10-14 11:19:39 -07:00
0139bfa749
Detect if mysql and sqlite support row_number ( #57475 )
2021-10-11 21:17:18 -07:00
a4d9019ffc
Refactor persistent notification to no longer route all data via a service ( #57157 )
...
* Convert persistent notification tests to async
* Create/dismiss persistent notifications in exposed functions, not service calls
* Fix notify persistent_notification
* Remove setting up persistent_notification
* Drop more setups
* Empty methods
* Undeprecate sync methods because too big task
* Fix setup clearing notifications
* Fix a bunch of tests
* Fix more tests
* Uno mas
* Test persistent notification events
* Clean up stale comment
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-10-07 12:58:00 +02:00
69875cbd11
Improve sensor statistics validation ( #56892 )
2021-10-04 09:47:44 -07:00
8567aa9e13
Evict purged states from recorder's old_state cache ( #56877 )
...
Co-authored-by: J. Nick Koston <nick@koston.org>
2021-10-04 08:21:40 -07:00
40ecf22bac
Remove automatic splitting of net meters from statistics ( #56772 )
2021-09-29 17:20:17 +02:00
00651a4055
Optimize _get_states_with_session ( #56734 )
...
* Optimize _get_states_with_session
* Move custom filters to derived table
* Remove useless derived table
* Filter old states after grouping
* Split query
* Add comments
* Simplify state update period criteria
* Only apply custom filters if we didn't get an include list of entities
Co-authored-by: J. Nick Koston <nick@koston.org>
2021-09-29 17:08:27 +02:00
8ef123259e
Add WS API for updating unit_of_measurement in statistics metadata ( #56184 )
...
* Add WS API for updating statistics metadata
* Update homeassistant/components/recorder/websocket_api.py
Co-authored-by: Bram Kragten <mail@bramkragten.nl>
* Update homeassistant/components/recorder/websocket_api.py
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
* Fix typo
Co-authored-by: Bram Kragten <mail@bramkragten.nl>
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2021-09-28 00:43:29 +02:00
5976f898da
Add WS API for removing statistics for a list of statistic_ids ( #55078 )
...
* Add WS API for removing statistics for a list of statistic_ids
* Refactor according to code review, enable foreign keys support for sqlite
* Adjust tests
* Move clear_statistics WS API to recorder
* Adjust tests after rebase
* Update docstring
* Update homeassistant/components/recorder/websocket_api.py
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
* Adjust tests after rebase
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2021-09-27 23:30:13 +02:00
64393b462d
Add migration for 5-minute statistics ( #56585 )
...
* Add migration for 5-minute statistics
* Tweaks
2021-09-24 09:19:22 +02:00
92253f5192
Minor refactoring of periodic statistics ( #56492 )
2021-09-22 13:31:33 -07:00
6f36419c6f
Improve statistics validation ( #56457 )
2021-09-20 17:54:25 +02:00
8c5efafdd8
Add 5-minute statistics for sensors ( #56006 )
...
* Add 5-minute statistics for sensors
* Address pylint issues
* Black
* Apply suggestion from code review
* Apply suggestions from code review
* Improve tests
2021-09-16 10:57:15 +02:00
d899d15a1e
Add statistics validation ( #56020 )
...
* Add statistics validation
* Remove redundant None-check
* Move validate_statistics WS API to recorder
* Apply suggestion from code review
2021-09-13 13:44:22 +02:00
d2a9f7904a
Include end time of statistics data points in API response ( #56063 )
...
* Include end time of statistics data points in API response
* Correct typing
* Update tests
2021-09-13 10:02:24 +02:00
80fd330479
Add sum_decrease and sum_increase statistics ( #55850 )
2021-09-08 23:35:53 -07:00
27764e9985
Fix handling of imperial units in long term statistics ( #55959 )
2021-09-08 08:08:48 -07:00
22e6ddf8df
Do not let one bad statistic spoil the bunch ( #55942 )
2021-09-08 07:55:40 -07:00
8877f37da0
Fix statistics for sensors setting last_reset ( #55136 )
...
* Re-add state_class total to sensor
* Make energy cost sensor enforce state_class total_increasing
* Drop state_class total
* Only report energy sensor issues once
2021-08-24 17:02:34 +02:00
32e297f4a0
Compile missing statistics ( #54690 )
2021-08-19 22:10:45 -07:00
e1926caeb9
Remove STATE_CLASS_TOTAL and last_reset from sensor ( #54755 )
...
* Remove STATE_CLASS_TOTAL
* Update mill sensor
* Update tests
* Kill last_reset
* Return ATTR_LAST_RESET to utility_meter
* Update energy cost sensor
* Restore last_reset for backwards compatibility
* Re-add and update deprecation warning
* Update tests
* Fix utility_meter
* Update EnergyCostSensor
* Tweak
* Fix rebase mistake
* Fix test
2021-08-18 10:03:27 +02:00
557cc792e9
Fix SQLAlchemy test warnings ( #54116 )
2021-08-08 20:33:47 -07:00
e541bcd54d
Update statistics meta data on entity_id change ( #52755 )
2021-07-14 13:23:11 +02:00
11fd9d9525
Avoid duplicated database queries when fetching statistics ( #52433 )
2021-07-02 15:40:54 +02:00
0476c7f9ee
Normalize temperature statistics to °C ( #52297 )
...
* Normalize temperature statistics to °C
* Fix tests
* Support temperature conversion to and from K, improve tests
* Fix test
* Add tests, pylint
2021-06-30 14:17:58 +02:00
42c4317628
Avoid drift in recorder purge cut-off ( #52135 )
2021-06-25 11:29:38 -10:00
805ef3f90b
Allow fetching multiple statistics ( #51996 )
2021-06-18 21:03:13 +02:00
39c94e8daa
Fix flaky statistics tests ( #51214 )
...
* Fix flaky statistics tests
* Tweak
2021-05-28 22:37:17 +02:00
b339d73109
Weight sensor average statistics by state durations ( #51150 )
...
* Weight sensor average statistics by state durations
* Fix test
2021-05-28 13:16:52 +02:00
aa9b99713c
Add purge_entities service call to recorder ( #48069 )
2021-05-22 10:30:05 -05:00
e16a8063a5
Compile statistics for energy sensors ( #50829 )
...
* Compile statistics for energy sensors
* Update tests
* Rename abs_value to state
* Tweak
* Recreate statistics table
* Pylint
* Try to fix test
* Fix statistics for multiple energy sensors
* Fix energy statistics when last_reset is not set
2021-05-20 13:05:15 +02:00
e7f7e61e88
Ensure a wal checkpoint is scheduled nightly ( #50746 )
2021-05-17 16:27:51 -05:00
72288710ca
Increase the sqlite cache size from ~2MiB to 8MiB ( #50747 )
2021-05-17 19:42:12 +02:00
89dd3292ba
Initial draft of statistics ( #49852 )
2021-05-16 10:23:37 -07:00
973f59e423
Refactor history component ( #50287 )
...
* Refactor history component
* Update tests
* Address review comments
* Correct deprecated functions
2021-05-11 09:21:57 +02:00
bf2d40adfe
Migrate from pytz to python-dateutil ( #49643 )
...
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2021-05-07 22:46:26 -07:00
1c8d9ca68b
Check exception causes for matching strings during recorder migration ( #49999 )
2021-05-02 20:57:42 -07:00
1bd9826684
Handle different entity_id formats ( #49969 )
2021-05-01 20:30:28 -07:00
b27e9e376d
Use StaticPool for recorder and NullPool for all other threads with sqlite3 ( #49693 )
2021-04-26 19:20:31 -10:00
c351098f04
HomeAssistantType —> HomeAssistant for Integrations p* - s* ( #49558 )
2021-04-22 19:58:02 +02:00
c10836fcee
Upgrade to sqlalchemy 1.4.11 ( #49538 )
2021-04-21 20:29:36 -10:00
6d137d2316
Increase recorder test coverage ( #49362 )
...
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-04-19 08:22:38 -07:00
e24f5831a2
Force recorder shutdown at final write event ( #49145 )
...
* Force recorder shutdown at EVENT_HOMEASSISTANT_FINAL_WRITE
* remove unreachable
* remove unreachable
* simplify
* cancel in async
2021-04-19 11:24:17 +02:00
53853f035d
Prevent calling stop or restart services during db upgrade ( #49098 )
2021-04-12 17:18:38 -07:00
Erik Montnemery
J. Nick Koston
Joakim Sørensen
Pawel
Marc Mueller
Stefan Agner
Paulus Schoutsen
PeteBa
jan iversen