5db014666c
Avoid recording state_changed events in the events table ( #71165 )
...
* squash
fix mypy
* Update homeassistant/components/recorder/models.py
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
* pass all columns
* fix commented out code
* reduce logbook query complexity
* merge
* comment
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2022-05-02 02:10:34 -05:00
c23866e5e5
De-duplicate event data into a new event_data table ( #71135 )
2022-05-01 21:01:17 -05:00
9b10658d01
[ci skip] Translation update
2022-05-02 00:22:04 +00:00
2d720973ee
Fix incomplete recorder typing ( #71158 )
2022-05-01 19:53:47 +03:00
ba386b5841
[ci skip] Translation update
2022-05-01 00:28:25 +00:00
66a21e0bc3
Add basic system health data to the recorder ( #71086 )
2022-04-30 11:10:20 -05:00
b9c7a89b47
Prevent sqlalchemy Transparent SQL Compilation Caching from filling up during purge ( #71015 )
2022-04-28 23:48:28 -07:00
37384f7eb3
Bump sqlalchemy to 1.4.36 ( #71039 )
2022-04-28 21:32:46 -07:00
cf90e34776
Type recorder retry decorator ( #70993 )
2022-04-28 21:04:41 +02:00
27a4a9eed4
Adjust get_latest_short_term_statistics query to be postgresql compatible ( #70953 )
2022-04-27 21:19:36 -07:00
195811843b
Remove get_state and get_states history api calls ( #70830 )
2022-04-26 16:05:43 -07:00
6ce768465f
Improve purge performance for non-sqlite databases ( #70576 )
2022-04-26 13:11:57 -07:00
1c4a785fb3
Prevent autoflush from happening during attrs lookup ( #70768 )
2022-04-26 13:04:58 -07:00
f073f17040
Refactor tracking of the recorder run history ( #70456 )
...
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2022-04-26 09:59:43 -10:00
96d64bd6b7
Avoid selecting last_updated from the db when filtering on last_updated==last_changed ( #70459 )
2022-04-25 08:46:37 -10:00
9f11063724
Fix state_changes_during_period bakery caching for limit and descending ( #70610 )
2022-04-24 21:47:21 -07:00
022deb20fd
Fix history not including start time state ( #70447 )
2022-04-24 14:34:00 -07:00
521579fc6a
Fixed syntax error in ALTER TABLE statement ( #70304 ) ( #70336 )
2022-04-22 21:29:25 -10:00
3737b58e85
Avoid fetching metadata multiple times during stat compile ( #70397 )
2022-04-22 00:25:42 -10:00
9761a7310e
Adjust doc string for recorder.Recorder.async_periodic_statistics ( #70398 )
2022-04-21 22:04:42 -07:00
3142a106fc
Additional prep work for sqlalchemy 2.0 ( #70358 )
2022-04-21 12:39:18 -10:00
de9f39745b
Fix race in _process_recorder_platform ( #70339 )
...
* Fix race in _process_recorder_platform
* Update homeassistant/components/recorder/__init__.py
Co-authored-by: J. Nick Koston <nick@koston.org>
* Update tests
Co-authored-by: J. Nick Koston <nick@koston.org>
2022-04-21 09:03:05 +02:00
37d39ac6d3
Add restored to the set of attributes excluded from being recorded in the db ( #70169 )
...
* Add restored to the set of attributes excluded from being recorded in the database
- Noticed on restart that the table gets 100s of new rows because
of these
* Update homeassistant/components/recorder/const.py
2022-04-16 23:09:08 +02:00
89807f0d2d
Reduce the number of queries needed to compile statistics ( #69731 )
...
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2022-04-15 09:13:29 -07:00
c85387290a
Increase recorder queue max backlog to 40k, improve message ( #70065 )
2022-04-14 11:45:07 -10:00
e407960f68
Update pylint to 2.13.5 ( #69989 )
2022-04-13 18:36:05 +02:00
ba07663e7d
Fix adjusting statistics in ft³ ( #69913 )
...
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2022-04-12 14:08:38 -07:00
eb3458a3d2
Add MutexPool for recorder tests ( #69410 )
...
* Add MutexPool for recorder tests
* Fix get_schema_version
* Update test test_last_run_was_recently_clean
* Update test test_shutdown_before_startup_finishes
* Revert comments in test_write_lock_db
* Make the MutexPool lock a class variable
* Remove stale comment
* Move MutexPool
* Tweak debug prints
2022-04-12 07:41:46 -10:00
80ff1ecb2a
Fix adjusting 5-minute statistics ( #69921 )
2022-04-12 18:10:17 +02:00
fe6a4bfb1d
Remove EVENT_TIME_CHANGED and EVENT_TIMER_OUT_OF_SYNC ( #69643 )
...
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2022-04-09 09:05:54 -10:00
5076437413
Reduce indent in recorder event processing ( #69659 )
2022-04-07 23:37:20 -10:00
72fffde77a
Fix spelling of periodic in recorder ( #69658 )
...
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2022-04-07 23:37:02 -10:00
66f0a3816a
Reduce memory pressure during database migration ( #69628 )
2022-04-07 18:29:31 -10:00
949b0e1b65
Don't allow in-memory SQLite database ( #69616 )
2022-04-08 00:43:09 +02:00
23b84449e6
Bump sqlalchemy to 1.4.35 ( #69534 )
...
Changes: https://docs.sqlalchemy.org/en/14/changelog/changelog_14.html#change-1.4.35
2022-04-07 14:59:23 +03:00
0ad9da9dd0
Always use a commit interval of 0 for the in memory db in tests ( #69556 )
2022-04-07 13:04:08 +02:00
5c7c09726a
Cleanup recorder history typing ( #69408 )
2022-04-07 00:09:05 -10:00
def305cf46
Improve the use of bakeries in recorder ( #69418 )
2022-04-06 08:00:10 -10:00
8e090a8c04
Minor tweak of recorder typing ( #69415 )
2022-04-06 08:08:00 -07:00
ec131d685e
Auto repack the database on the second sunday of the month ( #69314 )
2022-04-04 21:39:12 -10:00
f9a47f0f9e
Bump sqlalchemy to 1.4.34 ( #69202 )
...
Changelog: https://github.com/sqlalchemy/sqlalchemy/releases/tag/rel_1_4_34
https://github.com/sqlalchemy/sqlalchemy/releases/tag/rel_1_4_33
2022-04-03 18:39:07 +03:00
522a9bb6b1
Exclude supported features and attribution from being recorded in the database ( #69165 )
2022-04-03 13:51:42 +02:00
8b04c676ac
Fix typing on recorder.history ( #68917 )
2022-03-30 09:50:21 -10:00
d75f577b88
Additional strict typing for recorder ( #68860 )
2022-03-30 09:20:44 -07:00
ecd43f391f
Prettify json (manifest.json) ( #68886 )
2022-03-30 10:56:37 +02:00
f5a13fc51b
Add ability to exclude attributes from being recorded by entity domain ( #68824 )
2022-03-29 20:13:08 -07:00
d7634d1cb1
Additional strict typing for additional recorder internals ( #68689 )
...
* Strict typing for additional recorder internals
* revert
* fix refactoring error
2022-03-29 09:45:25 +02:00
53110f8cb7
Update pylint to 2.13.2 ( #68704 )
2022-03-27 16:08:24 +02:00
32b2d1e5c9
Add backup platform to recorder ( #68229 )
2022-03-25 23:17:11 -07:00
911b159281
Cleanup after pylint update ( #68657 )
2022-03-26 00:34:12 +01:00
53245c6523
Update pylint to 2.13.0 ( #68656 )
2022-03-25 15:14:48 -07:00
225f7a989b
Add strict typing for recorder util ( #68681 )
2022-03-25 15:03:46 -07:00
c5c34bc0d7
Typing and code quality for recorder history ( #68647 )
2022-03-24 17:58:38 -07:00
a566d3943c
Fix history queries while the database migration is in progress ( #68598 )
2022-03-24 09:49:13 -10:00
e911936a0d
Remove direct usage of concurrent.futures from recorder ( #68593 )
2022-03-24 09:48:49 -10:00
3777fa52f0
Ensure recorder statistics process registry updates in the db executor ( #68633 )
2022-03-24 09:48:09 -10:00
cbf5b5ead5
Use recorder threadpool in WS recorder/get_statistics_metadata ( #68615 )
2022-03-24 14:15:09 +01:00
61cc8e32f3
Include has_mean + has_sum in statistics metadata WS response ( #68546 )
...
* Include has_mean + has_sum in statistics metadata WS response
* Don't include has_mean/has_sum in history/list_statistic_ids
* Adjust tests
* Do include has_mean/has_sum in history/list_statistic_ids
2022-03-24 10:12:01 +01:00
8c10963bc0
Small cleanups for recorder ( #68551 )
2022-03-23 12:12:37 -10:00
08d6a3d9d4
Cache newly written state attribute ids ( #68355 )
2022-03-22 21:21:56 -10:00
7deeb92485
Switch sqlalchemy execute to use .all() instead of list() on the iterator ( #68540 )
2022-03-22 19:24:21 -10:00
b5c5da96ac
Add WS API to adjust incorrect energy statistics ( #65147 )
...
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2022-03-22 15:18:30 -07:00
bcb8c7ec3c
Add API endpoint get_statistics_metadata ( #68471 )
...
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2022-03-21 21:14:47 -07:00
3c10ac308d
Fix migration to schema v25 with Postgresql ( #68426 )
2022-03-20 21:49:11 +01:00
6ffef8373b
Add shutdown guard to Recorder pool in case there is no connection ( #68407 )
2022-03-20 01:28:44 -10:00
3150915cb7
Convert unindexed domain queries to entity_id queries ( #68404 )
2022-03-20 01:28:17 -10:00
816695cc96
Avoid selecting attributes in the history api when `no_attributes` is passed ( #68352 )
2022-03-19 23:47:22 -10:00
0c0df07c52
Avoid hashing attributes when they are already in the cache ( #68395 )
2022-03-19 23:33:37 -07:00
7ee647cc78
Fix FOREIGN KEY constraint failed when removing state_attributes ( #68364 )
2022-03-19 14:21:28 -07:00
9864090e0b
Cache parsing attr in LazyState ( #68232 )
2022-03-18 07:44:37 -07:00
9215702388
Separate attrs into another table (reduces database size) ( #68224 )
2022-03-18 00:23:13 -10:00
bc862e97ed
Use a dedicated executor pool for database operations ( #68105 )
...
Co-authored-by: Erik Montnemery <erik@montnemery.com>
Co-authored-by: Franck Nijhof <git@frenck.dev>
2022-03-17 23:09:01 -10:00
95f20500ca
Commit any pending changes before running non-EventTasks in the recorder ( #68287 )
2022-03-17 21:33:22 -07:00
b18096fc54
Remove unused columns from states/events tables ( #68078 )
2022-03-13 23:44:55 -07:00
36acb09cae
Update sqlalchemy to 1.4.32 ( #68075 )
2022-03-12 22:52:47 -10:00
596644d715
Fix typo [recorder] ( #66879 )
2022-02-19 16:04:20 +01:00
cb736eaeaf
Add type ignore error codes [recorder] ( #66780 )
2022-02-18 10:37:38 +01:00
39ed628cca
Suppress unwanted error messages during recorder migration ( #66004 )
2022-02-07 15:46:40 -08:00
910b1f1ec8
Speed up deletion of duplicated statistics ( #66014 )
2022-02-07 09:11:52 -08:00
6d4df93bc7
Correct description of entity_globs ( #65805 )
2022-02-06 18:36:38 +01:00
1f8e8926fe
Only remove duplicated statistics on error ( #65653 )
2022-02-04 11:31:12 -08:00
020953e943
Improve recorder migration for PostgreSQL when columns already exist ( #65680 )
2022-02-04 10:55:28 -08:00
41ab12cb88
Don't use shared session during recorder migration ( #65672 )
2022-02-04 09:55:11 -08:00
ac7662c82d
Remove limit of amount of duplicated statistics ( #65641 )
2022-02-04 14:49:45 +01:00
7a2b699371
Update codespell to 2.1.0 ( #64661 )
2022-01-21 22:05:24 +01:00
0a9927d18e
Avoid locking the database for non-SQLite backends ( #63847 )
...
* Avoid locking the database for non-SQLite backends
Currently we only have a lock implementation for SQLite. Just return
success for all other databases as they are not expected to store data
in the config directory and the caller can assume that a backup can
be safely taken.
This fixes `RuntimeError: generator didn't yield` errors when creating
a backup with the current Supervisor dev builds.
2022-01-11 16:17:56 +01:00
a19c95e4bd
Ensure service calls are typed [o-r] ( #62920 )
...
Co-authored-by: epenet <epenet@users.noreply.github.com>
2021-12-28 13:10:17 +01:00
2f8e65a9b0
Store deleted duplicated statistics in .storage ( #62574 )
2021-12-22 14:27:56 +01:00
48b3d6e1c0
Save original + duplicate pairs when deleting duplicated statistics ( #62498 )
2021-12-21 15:24:36 +01:00
cab2a74b5f
Don't pollute config dir with deleted duplicated statistics ( #62489 )
2021-12-21 14:27:35 +01:00
77829e397b
Don't log DB connection string on error ( #61927 )
2021-12-15 20:54:57 +01:00
bceeaec2f8
Remove duplicated statistics rows ( #61146 )
...
* Remove duplicated statistics
* Fix misleading docstring
* Pylint knows best
* Correct test
* Oops
* Prevent insertion of duplicated statistics
* Tweak
* pylint
* Add models_schema_23.py
* Tweak
2021-12-13 14:15:36 +01:00
83989d7b40
Use relative imports [L-R] ( #61575 )
2021-12-13 00:24:46 +01:00
dc5888ab4a
Correct recorder.statistics.get_last_statistics ( #61421 )
2021-12-10 10:09:29 -08:00
a13ae85982
Introduce only_supervisor for @websocket_api.ws_require_user() ( #61298 )
2021-12-08 16:49:35 -08:00
f30eb05870
Refactor recorder queue handling ( #61161 )
...
* Refactor recorder queue handling
* Address pylint's concerns
* Implement workaround for mypy bug
* Address review comments
2021-12-08 16:54:26 +01:00
f0006b92be
Allow to lock SQLite database during backup ( #60874 )
...
* Allow to set CONF_DB_URL
This is useful for test which need a custom DB path.
* Introduce write_lock_db helper to lock SQLite database
* Introduce Websocket API which allows to lock database during backup
* Fix isort
* Avoid mutable default arguments
* Address pylint issues
* Avoid holding executor thread
* Set unlock event in case timeout occures
This makes sure the database is left unlocked even in case of a race
condition.
* Add more unit tests
* Address new pylint errors
* Lower timeout to speedup tests
* Introduce queue overflow test
* Unlock database if necessary
This makes sure that the test runs through in case locking actually
succeeds (and the test fails).
* Make DB_LOCK_TIMEOUT a global
There is no good reason for this to be an argument. The recorder needs
to pick a sensible value.
* Add Websocket Timeout test
* Test lock_database() return
* Update homeassistant/components/recorder/__init__.py
Co-authored-by: Erik Montnemery <erik@montnemery.com>
* Fix format
Co-authored-by: J. Nick Koston <nick@koston.org>
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2021-12-07 13:16:24 +01:00
d980ca7e04
Correct recorder migration._add_columns for PostgreSQL ( #60547 )
2021-11-29 18:33:25 +01:00
18e822b7b6
Bump sqlalchemy to 1.4.27 ( #60383 )
2021-11-25 19:55:17 -08:00
4a5238efa5
Add support for calculating daily and monthly fossil energy consumption ( #59588 )
2021-11-22 09:10:54 -08:00
J. Nick Koston
GitHub Action
Marc Mueller
Dmitry Katsubo
Erik Montnemery
epenet
Joakim Sørensen
Pawel
Franck Nijhof
Christopher Masto
Stefan Agner
Paulus Schoutsen
Michael