Commit Graph

640 Commits (bd6a4d10ea99d757b8b1f19602f4810be8ed2f93)

Author SHA1 Message Date
epenet 6806e85eab
Add missing converters to recorder statistics (#87137) 2023-02-02 09:44:26 +01:00
GitHub Action 0f80932709 [ci skip] Translation update 2023-02-02 00:25:01 +00:00
J. Nick Koston fb55933b0a
Reduce chance of queue overflow during schema migration (#87090)
Co-authored-by: Franck Nijhof <git@frenck.dev>
2023-02-01 18:34:40 +01:00
GitHub Action be73e43989 [ci skip] Translation update 2023-02-01 00:27:42 +00:00
J. Nick Koston 5284837c8f
Add a repair issue when using MariaDB is affected by MDEV-25020 (#87040)
closes https://github.com/home-assistant/core/issues/83787
2023-01-31 20:42:07 +01:00
J. Nick Koston 00118a6f96
Speed up live history setup if there is no pending data to commit (#86942) 2023-01-30 19:00:34 +01:00
J. Nick Koston f874258e7e
Silence spurious warnings about removing ix_states_entity_id with newer installs (#86961)
* Silence spurious warnings about removing ix_states_entity_id with newer installs

https://ptb.discord.com/channels/330944238910963714/427516175237382144/1069648035459641465

* Silence spurious warnings about removing ix_states_entity_id with newer installs

https://ptb.discord.com/channels/330944238910963714/427516175237382144/1069648035459641465
2023-01-30 12:38:33 -05:00
Erik Montnemery 53c5f02ca2
Remove some dead code from recorder (#86697) 2023-01-30 18:37:48 +01:00
J. Nick Koston 4e9bd09d39
Fix old indices not being removed in schema migration leading to slow MySQL queries (#86917)
fixes #83787
2023-01-29 21:33:23 -05:00
J. Nick Koston c612a92cfb
Use python defaults for comparing State, LazyState, and Event objects (#86856)
* Speed up comparing State and Event objects

Use default python implementation for State and Event __hash__ and __eq__

The default implementation compared based on the id() of the object
which is effectively what we want here anyways. These overrides are
left over from the days when these used to be attrs objects

By avoiding implementing these ourselves all of the equality checks
can happen in native code

* tweak

* adjust tests

* write out some more

* fix test to not compare objects

* more test fixes

* more test fixes

* correct stats tests

* fix more tests

* fix more tests

* update sensor recorder tests
2023-01-29 13:31:43 -05:00
J. Nick Koston 58de7b8df0
Fix v32 schema migration when MySQL global.time_zone is configured with non-UTC timezone (#86867)
* Fix v32 schema migration when MySQL timezone is not UTC

* tweak
2023-01-28 22:06:07 -05:00
J. Nick Koston d97a061285
Chunk MariaDB and Postgresql data migration to avoid running out of buffer space (#86680)
* Chunk MariaDB data migration to avoid running out of buffer space

This will make the migration slower but since the innodb_buffer_pool_size
is using the defaul to 128M and not tuned to the db size there is a
risk of running out of buffer space for large databases

* Update homeassistant/components/recorder/migration.py

* hard code since bandit thinks its an injection

* Update homeassistant/components/recorder/migration.py

* guard against manually modified data/corrupt db

* adjust to 10k per chunk

* adjust to 50k per chunk

* memory still just fine at 250k

* but slower

* commit after each chunk to reduce lock pressure

* adjust

* set to 0 if null so we do not loop forever (this should only happen if the data is missing)

* set to 0 if null so we do not loop forever (this should only happen if the data is missing)

* tweak

* tweak

* limit cleanup

* lower limit to give some more buffer

* lower limit to give some more buffer

* where required for sqlite

* sqlite can wipe as many as needed with no limit

* limit on mysql only

* chunk postgres

* fix limit

* tweak

* fix reference

* fix

* tweak for ram

* postgres memory reduction

* defer cleanup

* fix

* same order
2023-01-27 22:39:45 -05:00
Franck Nijhof a79885ceaf
Enable Ruff SIM117 (#86783) 2023-01-27 11:52:49 +01:00
Franck Nijhof d4955a3d87
Replace assert False by raising an error (#86686) 2023-01-26 16:51:43 +01:00
Erik Montnemery fea30c1ce9
Terminate strings at NUL when recording states and events (#86687) 2023-01-26 11:11:03 +01:00
Marc Mueller 45b4b0e990
Import `ParamSpec` from typing [Py310] (#86413)
* Import ParamSpec from typing [Py310]

* Update additional imports
2023-01-23 07:28:43 +01:00
J. Nick Koston 52ea64d1d0
Fix repr for States and Events without a timestamp (#86391) 2023-01-22 08:11:42 -10:00
J. Nick Koston 4c84824ac8
Increase default recorder commit interval to 5 seconds (#86115) 2023-01-21 18:09:00 -10:00
Franck Nijhof 79b52a2b41
Stricter pylint message control (#86154) 2023-01-20 13:47:55 +01:00
Franck Nijhof c5dedb7a79
Code styling tweaks to the recorder integration (#86030) 2023-01-16 19:51:11 +01:00
J. Nick Koston 233332c3a0
Fix fetching history include_start_time_state when timezone is not UTC (#85983) 2023-01-16 09:01:32 +01:00
J. Nick Koston 43fb68ed79
Avoid linear searches for excluded events (#85851)
If the there are a lot of excluded events for the recorder, it
can have a performance impact as the list has to be searched
every time an event fires in HA
2023-01-14 14:25:03 +02:00
Ville Skyttä ae302bbec0
Make use of str.removeprefix and .removesuffix (#85584) 2023-01-13 12:19:38 +01:00
Erik Montnemery b0d4b73874
Add unitless unit converter (#85694)
* Add unitless unit converter

* Adjust type hints

* Adjust tests

* Rename to UnitlessRatioConverter
2023-01-12 09:20:00 +01:00
J. Nick Koston 57239769ba
Only build compressed states once (#85561) 2023-01-09 12:07:32 -10:00
Franck Nijhof fed8f905c8
Correct return type of two raise-only methods in recorder (#85048) 2023-01-03 12:15:45 +01:00
J. Nick Koston 8db086f65b
Bump sqlalchemy to 1.4.45 (#85021)
changelog: https://docs.sqlalchemy.org/en/20/changelog/changelog_14.html#change-1.4.45
2023-01-02 20:46:56 -05:00
J. Nick Koston b8a1537b58
Improve performance of fetching and storing history and events with the database (#84870) 2023-01-02 13:26:08 -10:00
Joakim Plate 987e77780a
Only run garbage collection per module (#84681)
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2023-01-02 09:31:12 -10:00
Franck Nijhof 8819634b61
String formatting and max line length - Part 6 (#84525) 2022-12-24 13:19:51 +01:00
Franck Nijhof f39f3b612a
String formatting and max line length - Part 5 (#84501)
Co-authored-by: jjlawren <jjlawren@users.noreply.github.com>
2022-12-23 16:43:17 +01:00
Franck Nijhof 94755a5773
String formatting and max line length - Part 4 (#84445)
Co-authored-by: jjlawren <jjlawren@users.noreply.github.com>
2022-12-23 13:27:27 +01:00
GitHub Action f19a1147fe [ci skip] Translation update 2022-12-11 00:26:24 +00:00
GitHub Action f25017313a [ci skip] Translation update 2022-12-09 00:24:29 +00:00
Erik Montnemery 587381440f
Fix repairing datetime precision for PostgreSQL (#83351) 2022-12-05 22:33:06 -05:00
GitHub Action 03154e1d83 [ci skip] Translation update 2022-12-03 00:21:50 +00:00
Erik Montnemery f869ce9d06
Validate common statistics DB schema errors on start (#79707)
* Validate common statistics db schema errors on start

* Fix test

* Add tests

* Adjust tests

* Disable statistics schema validation in tests

* Update after rebase
2022-11-29 10:16:08 +01:00
GitHub Action 230b50d099 [ci skip] Translation update 2022-11-29 00:26:01 +00:00
Erik Montnemery 58b3a00b16
Remove unnecessary DB access from statistic_during_period (#82871) 2022-11-28 13:48:06 -05:00
Erik Montnemery a038314d8b
Add display unit to WS recorder/get_statistics_metadata (#82870) 2022-11-28 13:46:57 -05:00
Erik Montnemery 2fe8e95309
Add helper to calculate statistic period start and end (#82493)
* Add helper to calculate statistic period start and end

* Don't parse values in resolve_period

* Add specific test for resolve_period

* Improve typing

* Move to recorder/util.py

* Extract period schema
2022-11-26 13:00:40 -05:00
GitHub Action f3b3193f7a [ci skip] Translation update 2022-11-25 00:24:19 +00:00
Erik Montnemery b94e1e9ef8
Fire events when long term statistics is updated (#82492)
* Fire events when long term statistics is updated

* Allow the new events to be subscribed to by anyone

* Address review comments

* Finish renaming events

* Finish renaming events

* Fix do_adhoc_statistics

* Adjust tests

* Adjust tests
2022-11-24 22:01:36 +01:00
Erik Montnemery 607a0e7697
Reduce size of get_statistics_during_period WS API response (#82131) 2022-11-16 17:36:30 +01:00
Erik Montnemery 9b8f94363c
Fix statistic_during_period for data with holes (#81847) 2022-11-16 12:46:29 +01:00
Franck Nijhof 19e5e671a2
Update sqlalchemy to 1.4.44 (#82129) 2022-11-15 08:51:13 -06:00
Marc Mueller 0c8eeaa643
Update mypy to 0.990 (#81783)
* Update mypy to 0.990

* Remove type ignore - overriding attr with property (13475)

* Remove type ignores - hasattr (13544)

* Adjust type ignore - assignment (13549)

* New error code - type-abstract (13785)

* Disable annotation-unchecked (13851)
2022-11-08 14:41:39 +01:00
Erik Montnemery 68346599d2
Add WS API recorder/statistic_during_period (#80663) 2022-10-27 15:51:09 -04:00
Michael 8175dab7ab
Add week period to recorder statistics api (#80784)
* add week period to get statistics api

* add test
2022-10-25 20:07:28 +02:00
Erik Montnemery beeee8b60e
Use start helper in recorder (#79559) 2022-10-25 10:38:07 +02:00