bcb8c7ec3c
Add API endpoint get_statistics_metadata ( #68471 )
...
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2022-03-21 21:14:47 -07:00
3150915cb7
Convert unindexed domain queries to entity_id queries ( #68404 )
2022-03-20 01:28:17 -10:00
816695cc96
Avoid selecting attributes in the history api when `no_attributes` is passed ( #68352 )
2022-03-19 23:47:22 -10:00
cf4033b1bc
Simplify time zone setting in tests ( #68330 )
...
* Simplify timezone setting in tests
* Fix typo
* Adjust caldav tests
* Adjust input_datetime tests
* Adjust time_date tests
* Adjust tod tests
* Adjust helper tests
* Adjust recorder tests
* Adjust risco tests
* Adjust aemet tests
* Adjust flux tests
* Adjust forecast_solar tests
* Revert unnecessary change in forecast_solar test
* Adjust climacell tests
* Adjust google tests
* Adjust sensor tests
* Adjust sonarr tests
* Adjust template tests
* Adjust zodiac tests
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2022-03-20 10:25:15 +01:00
7ee647cc78
Fix FOREIGN KEY constraint failed when removing state_attributes ( #68364 )
2022-03-19 14:21:28 -07:00
9215702388
Separate attrs into another table (reduces database size) ( #68224 )
2022-03-18 00:23:13 -10:00
bc862e97ed
Use a dedicated executor pool for database operations ( #68105 )
...
Co-authored-by: Erik Montnemery <erik@montnemery.com>
Co-authored-by: Franck Nijhof <git@frenck.dev>
2022-03-17 23:09:01 -10:00
95f20500ca
Commit any pending changes before running non-EventTasks in the recorder ( #68287 )
2022-03-17 21:33:22 -07:00
d360ac91ca
Disable recorder nightly jobs in tests ( #68188 )
2022-03-15 09:23:52 -07:00
b18096fc54
Remove unused columns from states/events tables ( #68078 )
2022-03-13 23:44:55 -07:00
910b1f1ec8
Speed up deletion of duplicated statistics ( #66014 )
2022-02-07 09:11:52 -08:00
41ab12cb88
Don't use shared session during recorder migration ( #65672 )
2022-02-04 09:55:11 -08:00
ac7662c82d
Remove limit of amount of duplicated statistics ( #65641 )
2022-02-04 14:49:45 +01:00
d057850971
Update typing - collections.abc (3) ( #63947 )
2022-01-12 08:04:17 +01:00
0a9927d18e
Avoid locking the database for non-SQLite backends ( #63847 )
...
* Avoid locking the database for non-SQLite backends
Currently we only have a lock implementation for SQLite. Just return
success for all other databases as they are not expected to store data
in the config directory and the caller can assume that a backup can
be safely taken.
This fixes `RuntimeError: generator didn't yield` errors when creating
a backup with the current Supervisor dev builds.
2022-01-11 16:17:56 +01:00
2f8e65a9b0
Store deleted duplicated statistics in .storage ( #62574 )
2021-12-22 14:27:56 +01:00
48b3d6e1c0
Save original + duplicate pairs when deleting duplicated statistics ( #62498 )
2021-12-21 15:24:36 +01:00
cab2a74b5f
Don't pollute config dir with deleted duplicated statistics ( #62489 )
2021-12-21 14:27:35 +01:00
474ef54477
Fix threading error in recorder tests ( #62187 )
2021-12-17 13:17:48 +01:00
bceeaec2f8
Remove duplicated statistics rows ( #61146 )
...
* Remove duplicated statistics
* Fix misleading docstring
* Pylint knows best
* Correct test
* Oops
* Prevent insertion of duplicated statistics
* Tweak
* pylint
* Add models_schema_23.py
* Tweak
2021-12-13 14:15:36 +01:00
dc5888ab4a
Correct recorder.statistics.get_last_statistics ( #61421 )
2021-12-10 10:09:29 -08:00
a13ae85982
Introduce only_supervisor for @websocket_api.ws_require_user() ( #61298 )
2021-12-08 16:49:35 -08:00
f30eb05870
Refactor recorder queue handling ( #61161 )
...
* Refactor recorder queue handling
* Address pylint's concerns
* Implement workaround for mypy bug
* Address review comments
2021-12-08 16:54:26 +01:00
f0006b92be
Allow to lock SQLite database during backup ( #60874 )
...
* Allow to set CONF_DB_URL
This is useful for test which need a custom DB path.
* Introduce write_lock_db helper to lock SQLite database
* Introduce Websocket API which allows to lock database during backup
* Fix isort
* Avoid mutable default arguments
* Address pylint issues
* Avoid holding executor thread
* Set unlock event in case timeout occures
This makes sure the database is left unlocked even in case of a race
condition.
* Add more unit tests
* Address new pylint errors
* Lower timeout to speedup tests
* Introduce queue overflow test
* Unlock database if necessary
This makes sure that the test runs through in case locking actually
succeeds (and the test fails).
* Make DB_LOCK_TIMEOUT a global
There is no good reason for this to be an argument. The recorder needs
to pick a sensible value.
* Add Websocket Timeout test
* Test lock_database() return
* Update homeassistant/components/recorder/__init__.py
Co-authored-by: Erik Montnemery <erik@montnemery.com>
* Fix format
Co-authored-by: J. Nick Koston <nick@koston.org>
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2021-12-07 13:16:24 +01:00
bcd9f3c05f
Correct end time for monthly statistics summary ( #59551 )
...
* Correct end time for monthly statistics summary
* Add tests
2021-11-12 15:01:40 +01:00
e5ee38d3b0
Adjust recorder tests according to #59087 ( #59143 )
2021-11-05 10:49:58 +01:00
470b01e4ce
Move async_migration_in_progress ( #59087 )
2021-11-05 10:40:56 +01:00
185f7beafc
Improve recorder migration tests ( #59075 )
2021-11-04 20:21:38 -07:00
4c5aca93df
Add recorder status WS API ( #58989 )
...
* Add recorder status WS API
* Rename recorder/status to recorder/info
* Silence pylint
* Improve tests
* Address review comments
* Tweak
* Try to fix tests
* Try to debug flaky tests
* Try to fix tests
* Revert changes to async_migration_in_progress
* Try to fix tests
* Remove debug prints
* Apply suggestions from code review
2021-11-04 16:46:45 +01:00
be4e9f91b6
Change minimum supported SQLite version to 3.31.0 ( #59073 )
2021-11-04 16:34:35 +01:00
dfa50a842a
Simplify recorder PgSQL version checks ( #58533 )
2021-10-27 16:05:40 +02:00
ac5e32d648
Corrections for external statistics ( #58469 )
2021-10-26 14:05:45 +02:00
e9ba5f3b4b
Warn when recorder connects to an unsupported database ( #58161 )
2021-10-26 13:41:59 +02:00
f594bc353b
Add support for external statistics ( #56607 )
...
* Support external statistics
* Update tests
* Lint
* Adjust code after rebase
* Separate external statistic_id with :, add name to metadata
* Adjust tests
* Simplify get_metadata_with_session
* Address review comments
* Allow updating external statistics
* Validate input
* Adjust tests after rebase
* Pylint
* Adjust tests
* Improve test coverage
2021-10-26 10:26:50 +02:00
b301ab25a3
Purge short term statistics ( #58028 )
...
* Purge short term statistics
* Less meep
* Add tests
2021-10-20 16:00:59 +02:00
8ef8838801
Correct detection of row_number support for MariaDB ( #57663 )
2021-10-14 11:19:39 -07:00
0139bfa749
Detect if mysql and sqlite support row_number ( #57475 )
2021-10-11 21:17:18 -07:00
a4d9019ffc
Refactor persistent notification to no longer route all data via a service ( #57157 )
...
* Convert persistent notification tests to async
* Create/dismiss persistent notifications in exposed functions, not service calls
* Fix notify persistent_notification
* Remove setting up persistent_notification
* Drop more setups
* Empty methods
* Undeprecate sync methods because too big task
* Fix setup clearing notifications
* Fix a bunch of tests
* Fix more tests
* Uno mas
* Test persistent notification events
* Clean up stale comment
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-10-07 12:58:00 +02:00
69875cbd11
Improve sensor statistics validation ( #56892 )
2021-10-04 09:47:44 -07:00
8567aa9e13
Evict purged states from recorder's old_state cache ( #56877 )
...
Co-authored-by: J. Nick Koston <nick@koston.org>
2021-10-04 08:21:40 -07:00
40ecf22bac
Remove automatic splitting of net meters from statistics ( #56772 )
2021-09-29 17:20:17 +02:00
00651a4055
Optimize _get_states_with_session ( #56734 )
...
* Optimize _get_states_with_session
* Move custom filters to derived table
* Remove useless derived table
* Filter old states after grouping
* Split query
* Add comments
* Simplify state update period criteria
* Only apply custom filters if we didn't get an include list of entities
Co-authored-by: J. Nick Koston <nick@koston.org>
2021-09-29 17:08:27 +02:00
8ef123259e
Add WS API for updating unit_of_measurement in statistics metadata ( #56184 )
...
* Add WS API for updating statistics metadata
* Update homeassistant/components/recorder/websocket_api.py
Co-authored-by: Bram Kragten <mail@bramkragten.nl>
* Update homeassistant/components/recorder/websocket_api.py
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
* Fix typo
Co-authored-by: Bram Kragten <mail@bramkragten.nl>
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2021-09-28 00:43:29 +02:00
5976f898da
Add WS API for removing statistics for a list of statistic_ids ( #55078 )
...
* Add WS API for removing statistics for a list of statistic_ids
* Refactor according to code review, enable foreign keys support for sqlite
* Adjust tests
* Move clear_statistics WS API to recorder
* Adjust tests after rebase
* Update docstring
* Update homeassistant/components/recorder/websocket_api.py
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
* Adjust tests after rebase
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2021-09-27 23:30:13 +02:00
64393b462d
Add migration for 5-minute statistics ( #56585 )
...
* Add migration for 5-minute statistics
* Tweaks
2021-09-24 09:19:22 +02:00
92253f5192
Minor refactoring of periodic statistics ( #56492 )
2021-09-22 13:31:33 -07:00
6f36419c6f
Improve statistics validation ( #56457 )
2021-09-20 17:54:25 +02:00
8c5efafdd8
Add 5-minute statistics for sensors ( #56006 )
...
* Add 5-minute statistics for sensors
* Address pylint issues
* Black
* Apply suggestion from code review
* Apply suggestions from code review
* Improve tests
2021-09-16 10:57:15 +02:00
d899d15a1e
Add statistics validation ( #56020 )
...
* Add statistics validation
* Remove redundant None-check
* Move validate_statistics WS API to recorder
* Apply suggestion from code review
2021-09-13 13:44:22 +02:00
d2a9f7904a
Include end time of statistics data points in API response ( #56063 )
...
* Include end time of statistics data points in API response
* Correct typing
* Update tests
2021-09-13 10:02:24 +02:00
80fd330479
Add sum_decrease and sum_increase statistics ( #55850 )
2021-09-08 23:35:53 -07:00
27764e9985
Fix handling of imperial units in long term statistics ( #55959 )
2021-09-08 08:08:48 -07:00
22e6ddf8df
Do not let one bad statistic spoil the bunch ( #55942 )
2021-09-08 07:55:40 -07:00
8877f37da0
Fix statistics for sensors setting last_reset ( #55136 )
...
* Re-add state_class total to sensor
* Make energy cost sensor enforce state_class total_increasing
* Drop state_class total
* Only report energy sensor issues once
2021-08-24 17:02:34 +02:00
32e297f4a0
Compile missing statistics ( #54690 )
2021-08-19 22:10:45 -07:00
e1926caeb9
Remove STATE_CLASS_TOTAL and last_reset from sensor ( #54755 )
...
* Remove STATE_CLASS_TOTAL
* Update mill sensor
* Update tests
* Kill last_reset
* Return ATTR_LAST_RESET to utility_meter
* Update energy cost sensor
* Restore last_reset for backwards compatibility
* Re-add and update deprecation warning
* Update tests
* Fix utility_meter
* Update EnergyCostSensor
* Tweak
* Fix rebase mistake
* Fix test
2021-08-18 10:03:27 +02:00
557cc792e9
Fix SQLAlchemy test warnings ( #54116 )
2021-08-08 20:33:47 -07:00
e541bcd54d
Update statistics meta data on entity_id change ( #52755 )
2021-07-14 13:23:11 +02:00
11fd9d9525
Avoid duplicated database queries when fetching statistics ( #52433 )
2021-07-02 15:40:54 +02:00
0476c7f9ee
Normalize temperature statistics to °C ( #52297 )
...
* Normalize temperature statistics to °C
* Fix tests
* Support temperature conversion to and from K, improve tests
* Fix test
* Add tests, pylint
2021-06-30 14:17:58 +02:00
42c4317628
Avoid drift in recorder purge cut-off ( #52135 )
2021-06-25 11:29:38 -10:00
805ef3f90b
Allow fetching multiple statistics ( #51996 )
2021-06-18 21:03:13 +02:00
39c94e8daa
Fix flaky statistics tests ( #51214 )
...
* Fix flaky statistics tests
* Tweak
2021-05-28 22:37:17 +02:00
b339d73109
Weight sensor average statistics by state durations ( #51150 )
...
* Weight sensor average statistics by state durations
* Fix test
2021-05-28 13:16:52 +02:00
aa9b99713c
Add purge_entities service call to recorder ( #48069 )
2021-05-22 10:30:05 -05:00
e16a8063a5
Compile statistics for energy sensors ( #50829 )
...
* Compile statistics for energy sensors
* Update tests
* Rename abs_value to state
* Tweak
* Recreate statistics table
* Pylint
* Try to fix test
* Fix statistics for multiple energy sensors
* Fix energy statistics when last_reset is not set
2021-05-20 13:05:15 +02:00
e7f7e61e88
Ensure a wal checkpoint is scheduled nightly ( #50746 )
2021-05-17 16:27:51 -05:00
72288710ca
Increase the sqlite cache size from ~2MiB to 8MiB ( #50747 )
2021-05-17 19:42:12 +02:00
89dd3292ba
Initial draft of statistics ( #49852 )
2021-05-16 10:23:37 -07:00
973f59e423
Refactor history component ( #50287 )
...
* Refactor history component
* Update tests
* Address review comments
* Correct deprecated functions
2021-05-11 09:21:57 +02:00
bf2d40adfe
Migrate from pytz to python-dateutil ( #49643 )
...
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2021-05-07 22:46:26 -07:00
1c8d9ca68b
Check exception causes for matching strings during recorder migration ( #49999 )
2021-05-02 20:57:42 -07:00
1bd9826684
Handle different entity_id formats ( #49969 )
2021-05-01 20:30:28 -07:00
b27e9e376d
Use StaticPool for recorder and NullPool for all other threads with sqlite3 ( #49693 )
2021-04-26 19:20:31 -10:00
c351098f04
HomeAssistantType —> HomeAssistant for Integrations p* - s* ( #49558 )
2021-04-22 19:58:02 +02:00
c10836fcee
Upgrade to sqlalchemy 1.4.11 ( #49538 )
2021-04-21 20:29:36 -10:00
6d137d2316
Increase recorder test coverage ( #49362 )
...
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-04-19 08:22:38 -07:00
e24f5831a2
Force recorder shutdown at final write event ( #49145 )
...
* Force recorder shutdown at EVENT_HOMEASSISTANT_FINAL_WRITE
* remove unreachable
* remove unreachable
* simplify
* cancel in async
2021-04-19 11:24:17 +02:00
53853f035d
Prevent calling stop or restart services during db upgrade ( #49098 )
2021-04-12 17:18:38 -07:00
9368891b1b
Live db migrations and recovery ( #49036 )
...
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2021-04-11 20:43:54 -10:00
c7e4857d2c
Let recorder deal with event names longer than 32 chars ( #47748 )
2021-04-08 07:08:49 -10:00
08870690a6
Fix a collection of tests with missing asserts ( #48127 )
...
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-03-20 17:23:55 +01:00
fb849b81b5
Rewrite of not a == b occurances ( #48132 )
2021-03-20 01:27:04 +01:00
28c80c1133
Ensure recorder purge tests can handle multiple purge cycle ( #47956 )
...
Since a purge can generate another purge task, we now wait for
three recorder queue completions by default.
2021-03-15 11:19:19 -07:00
61a2460c87
Improve error reporting in recorder purge test ( #47929 )
2021-03-14 19:46:21 -07:00
2178e27fb4
Fix unclean shutdown of recorder test ( #47791 )
2021-03-12 14:17:27 -08:00
92852b9c10
Add apply_filter attribute to recorder.purge service ( #45826 )
2021-03-11 17:03:30 -10:00
10848b9bdf
Recorder improvements ( #47739 )
2021-03-11 07:52:07 -10:00
a060acc2b1
Fix recorder with MSSQL ( #46678 )
...
Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
2021-03-09 11:16:19 -10:00
44293a3738
Add enable and disable services for recorder ( #45778 )
2021-02-24 07:26:05 -06:00
e2fd255a96
Cleanup recorder tests ( #46836 )
2021-02-21 09:52:41 +01:00
9b69549f73
Recover and restart the recorder if the sqlite database encounters corruption while running ( #46612 )
2021-02-19 21:26:24 -10:00
22dbac259b
Ensure recorder shuts down cleanly on restart before startup is finished ( #46604 )
2021-02-19 22:18:21 -08:00
dc26fd5149
Ensure creating an index that already exists is forgiving for postgresql ( #46185 )
...
Unlikely sqlite and mysql, postgresql throws ProgrammingError instead
of InternalError or OperationalError when trying to create an index
that already exists.
2021-02-08 23:22:38 +01:00
65cf2fcb6f
Drop asynctest ( #44746 )
2021-01-01 22:31:56 +01:00
f1dff973dc
Fix broken test test_auto_purge in recorder ( #44687 )
...
* Fix failing test due to edge-of-2021 bug
* Rewrite test_auto_purge to make the intent more clear
2020-12-31 12:04:12 -10:00
2cbb93be43
Always keep the current recorder run when purging ( #43733 )
2020-11-30 09:13:50 +01:00
4ab02cb9bc
Ensure recorder commit can retry after encountering invalid data ( #41426 )
2020-10-08 09:15:25 +02:00
113d738fa2
Reduce orm overhead by grouping object expiration ( #41394 )
2020-10-07 15:35:48 +02:00
1e9e40bf71
Rewrite recorder unittest tests to pytest style test function ( #41264 )
2020-10-06 21:24:13 +02:00
Pawel
J. Nick Koston
Erik Montnemery
Marc Mueller
Stefan Agner
Paulus Schoutsen
PeteBa
jan iversen
Laszlo Magyar
Franck Nijhof
adrian-vlad
Allen Porter
Ariana Hlavaty