Commit Graph

668 Commits (580869a9a6440d03342a468d445acf6b0d41dcca)

Author SHA1 Message Date
J. Nick Koston 484027023c
Fix new stats rows creating data in the old created column (#88013) 2023-02-13 21:53:37 -06:00
J. Nick Koston 7b9293941e
The recorder/statistic(s)_during_period websocket API now require the statistic_id(s) field (#87937) 2023-02-12 14:51:10 -06:00
J. Nick Koston 32a6280139
Optimize _sorted_statistics_to_dict to avoid checking which types each loop (#87901)
* Optimize _sorted_statistics_to_dict to avoid checking which types each loop

* tweak

* avoid some more dict lookups
2023-02-11 23:21:40 -05:00
J. Nick Koston 85649ec589
Reduce overhead to see if an entity is recorded (#87912)
A significant chunk of list_statistic_ids was checking if the entity
was recorded because it had to get the recorder instance over and
over
2023-02-11 23:21:16 -05:00
J. Nick Koston e899754919
Bump sqlalchemy to 2.0.3 (#87913)
changes: https://docs.sqlalchemy.org/en/20/changelog/changelog_20.html#change-2.0.3
2023-02-11 23:20:22 -05:00
J. Nick Koston 86a93e9fce
Optimize fetching statistics by avoiding recalculating time boundaries (#87859)
* predictive

* fix conversion error

* fix conversion error

* fix conversion error

* convert day to use new algo

* reduce
2023-02-11 11:41:57 -05:00
J. Nick Koston c35661947a
Optimize fetching statistics by avoiding timestamp conversions (#87845)
* Optimize fetching statistics by avoiding timestamp conversions

* remove accidential unrelated change

* fix test

* recreate so we handle timezone changes
2023-02-10 11:26:46 -05:00
J. Nick Koston ea356ad260
Optimize fetching weekly/monthly/yearly statistics (#87747)
* Optimize fetching statistics

* speed up

* avoid double groupby

* avoid another loop

* tweak flow

* fixes

* tweak

* avoid a new dt object in the cache for week/month

* avoid a new dt object in the cache for week/month
2023-02-09 21:22:32 -05:00
J. Nick Koston 509de02044
Ensure recorder still shuts down if the final commit fails (#87799) 2023-02-09 15:12:40 -06:00
J. Nick Koston abf0c87e40
Migrate statistics to use timestamp columns (#87321) 2023-02-09 12:24:19 -06:00
J. Nick Koston db503c0daa
Always use UTC time for the MariaDB/MySQL session (#87583) 2023-02-09 11:39:45 -06:00
Erik Montnemery 5dfb9cddd6
Sort manifests 8 (#87029) 2023-02-08 20:13:46 +01:00
Erik Montnemery 94519de8dd
Upgrade SQLAlchemy to 2.0.2 (#86436)
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-08 08:17:32 -06:00
epenet a202588fd2
Add return type to json_loads (#85672)
* Add JSON type definitions

* Sample use

* Keep mutable for a follo-up PR (avoid dead code)

* Use list/dict

* Remove JsonObjectType

* Remove reference to Union

* Cleanup

* Improve rest

* Rename json_dict => json_data

* Add docstring

* Add type hint to json_loads

* Add cast

* Move type alias to json helpers

* Cleanup

* Create and use json_loads_object

* Make error more explicit and add tests

* Use JsonObjectType in conversation

* Remove quotes
2023-02-07 17:21:55 +01:00
Marc Mueller b478b4fa16
Add ParamSpec TypeAliases [mypy 1.0] (#87597) 2023-02-07 15:56:26 +01:00
J. Nick Koston 0a1d5c85ad
Optimize history.get_last_state_changes query (#87554)
fixes undefined
2023-02-07 07:50:44 -06:00
J. Nick Koston 3aa744ed19
Fix indent on slow_range_in_select for MySQL/MariaDB (#87581)
We would calculate this value and throw it away
because it was only used on first_connection
2023-02-06 23:59:54 -05:00
Marc Mueller 342b406dc0
Add Self typing (1) [mypy 1.0] (#87598) 2023-02-06 22:29:47 -06:00
Franck Nijhof 939eef3b28
Remove translations from Core (#87543)
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2023-02-06 15:06:43 +01:00
J. Nick Koston c7980bda82
Fix recorder run history during schema migration and startup (#87492)
Fix recorder run history during schema migration

RunHistory.get and RunHistory.current can be called before
RunHistory.start. We need to return a RecorderRuns object
with the recording_start time that will be used when start
it called to ensure history queries still work as expected.

fixes #87112
2023-02-05 22:12:41 -05:00
GitHub Action f475446ebf [ci skip] Translation update 2023-02-06 00:23:07 +00:00
J. Nick Koston c11f3ffa17
Speed up purge time with newer MariaDB versions (#87409)
* Speed up purge time with newer MariaDB versions

* fix

* document

* document

* document

* rename

* self review

* Update homeassistant/components/recorder/util.py

* fixes
2023-02-05 17:58:34 -05:00
Franck Nijhof 3d557b5583
Collection of code styling tweaks (#87381) 2023-02-04 12:52:26 -05:00
Franck Nijhof ca1a12898c
Enable Ruff D212 (#87347) 2023-02-03 23:08:48 +01:00
epenet 810367b757
Fix `can not` typo (#87254) 2023-02-03 11:37:16 +01:00
J. Nick Koston 22698b1cc5
Fix statistics graphs not loading with data_rate, electric_current, voltage, information, and unitless units (#87202)
* Add missing converts to recorder/statistics_during_period API

This was resulting in the stats graphs not loading on the frontend

* its in two places
2023-02-02 14:52:55 -05:00
J. Nick Koston 389fc515a1
Prevent overly large attributes from being stored in the database (#87105) 2023-02-02 11:49:33 -06:00
Franck Nijhof 5e81d28116
Update black to 23.1.0 (#87188) 2023-02-02 18:35:24 +01:00
epenet 6806e85eab
Add missing converters to recorder statistics (#87137) 2023-02-02 09:44:26 +01:00
GitHub Action 0f80932709 [ci skip] Translation update 2023-02-02 00:25:01 +00:00
J. Nick Koston fb55933b0a
Reduce chance of queue overflow during schema migration (#87090)
Co-authored-by: Franck Nijhof <git@frenck.dev>
2023-02-01 18:34:40 +01:00
GitHub Action be73e43989 [ci skip] Translation update 2023-02-01 00:27:42 +00:00
J. Nick Koston 5284837c8f
Add a repair issue when using MariaDB is affected by MDEV-25020 (#87040)
closes https://github.com/home-assistant/core/issues/83787
2023-01-31 20:42:07 +01:00
J. Nick Koston 00118a6f96
Speed up live history setup if there is no pending data to commit (#86942) 2023-01-30 19:00:34 +01:00
J. Nick Koston f874258e7e
Silence spurious warnings about removing ix_states_entity_id with newer installs (#86961)
* Silence spurious warnings about removing ix_states_entity_id with newer installs

https://ptb.discord.com/channels/330944238910963714/427516175237382144/1069648035459641465

* Silence spurious warnings about removing ix_states_entity_id with newer installs

https://ptb.discord.com/channels/330944238910963714/427516175237382144/1069648035459641465
2023-01-30 12:38:33 -05:00
Erik Montnemery 53c5f02ca2
Remove some dead code from recorder (#86697) 2023-01-30 18:37:48 +01:00
J. Nick Koston 4e9bd09d39
Fix old indices not being removed in schema migration leading to slow MySQL queries (#86917)
fixes #83787
2023-01-29 21:33:23 -05:00
J. Nick Koston c612a92cfb
Use python defaults for comparing State, LazyState, and Event objects (#86856)
* Speed up comparing State and Event objects

Use default python implementation for State and Event __hash__ and __eq__

The default implementation compared based on the id() of the object
which is effectively what we want here anyways. These overrides are
left over from the days when these used to be attrs objects

By avoiding implementing these ourselves all of the equality checks
can happen in native code

* tweak

* adjust tests

* write out some more

* fix test to not compare objects

* more test fixes

* more test fixes

* correct stats tests

* fix more tests

* fix more tests

* update sensor recorder tests
2023-01-29 13:31:43 -05:00
J. Nick Koston 58de7b8df0
Fix v32 schema migration when MySQL global.time_zone is configured with non-UTC timezone (#86867)
* Fix v32 schema migration when MySQL timezone is not UTC

* tweak
2023-01-28 22:06:07 -05:00
J. Nick Koston d97a061285
Chunk MariaDB and Postgresql data migration to avoid running out of buffer space (#86680)
* Chunk MariaDB data migration to avoid running out of buffer space

This will make the migration slower but since the innodb_buffer_pool_size
is using the defaul to 128M and not tuned to the db size there is a
risk of running out of buffer space for large databases

* Update homeassistant/components/recorder/migration.py

* hard code since bandit thinks its an injection

* Update homeassistant/components/recorder/migration.py

* guard against manually modified data/corrupt db

* adjust to 10k per chunk

* adjust to 50k per chunk

* memory still just fine at 250k

* but slower

* commit after each chunk to reduce lock pressure

* adjust

* set to 0 if null so we do not loop forever (this should only happen if the data is missing)

* set to 0 if null so we do not loop forever (this should only happen if the data is missing)

* tweak

* tweak

* limit cleanup

* lower limit to give some more buffer

* lower limit to give some more buffer

* where required for sqlite

* sqlite can wipe as many as needed with no limit

* limit on mysql only

* chunk postgres

* fix limit

* tweak

* fix reference

* fix

* tweak for ram

* postgres memory reduction

* defer cleanup

* fix

* same order
2023-01-27 22:39:45 -05:00
Franck Nijhof a79885ceaf
Enable Ruff SIM117 (#86783) 2023-01-27 11:52:49 +01:00
Franck Nijhof d4955a3d87
Replace assert False by raising an error (#86686) 2023-01-26 16:51:43 +01:00
Erik Montnemery fea30c1ce9
Terminate strings at NUL when recording states and events (#86687) 2023-01-26 11:11:03 +01:00
Marc Mueller 45b4b0e990
Import `ParamSpec` from typing [Py310] (#86413)
* Import ParamSpec from typing [Py310]

* Update additional imports
2023-01-23 07:28:43 +01:00
J. Nick Koston 52ea64d1d0
Fix repr for States and Events without a timestamp (#86391) 2023-01-22 08:11:42 -10:00
J. Nick Koston 4c84824ac8
Increase default recorder commit interval to 5 seconds (#86115) 2023-01-21 18:09:00 -10:00
Franck Nijhof 79b52a2b41
Stricter pylint message control (#86154) 2023-01-20 13:47:55 +01:00
Franck Nijhof c5dedb7a79
Code styling tweaks to the recorder integration (#86030) 2023-01-16 19:51:11 +01:00
J. Nick Koston 233332c3a0
Fix fetching history include_start_time_state when timezone is not UTC (#85983) 2023-01-16 09:01:32 +01:00
J. Nick Koston 43fb68ed79
Avoid linear searches for excluded events (#85851)
If the there are a lot of excluded events for the recorder, it
can have a performance impact as the list has to be searched
every time an event fires in HA
2023-01-14 14:25:03 +02:00