Commit Graph

693 Commits (cf369ff1a5965dd86cb02770fc624a7b148b4428)

Author SHA1 Message Date
J. Nick Koston 1bd9767d8c
Handle InnoDB deadlocks during migration (#89073)
* Handle slow InnoDB rollback when encountering duplicates during migration

fixes #89069

* adjust

* fix mock

* tests

* return on success
2023-03-03 22:00:13 -05:00
epenet f2b736fad0
Adjust entity registry access in core platforms (#88944)
* Adjust entity registry access in platforms

* Adjust more core components
2023-03-01 08:02:16 +01:00
J. Nick Koston 69a3738bdb
Fix migration failing when existing data has duplicates (#88712) 2023-02-24 11:41:44 -06:00
J. Nick Koston cbba0fee42
Bump sqlalchemy to 2.0.4 (#88574)
changelog: https://docs.sqlalchemy.org/en/20/changelog/changelog_20.html#change-2.0.4
2023-02-21 21:11:16 -05:00
J. Nick Koston 7929e76ebb
Only build the field_map once in recorder history (#88529) 2023-02-21 13:45:05 -06:00
J. Nick Koston 9c82cfecca
Avoid asking recorder platforms for list_statistic_ids when already complete (#88495)
* Avoid asking recorder platforms for list_statistic_ids when already complete

If we already had all the data needed for list_statistic_ids, we would
still query recorder platforms and throw away the results

* Update homeassistant/components/recorder/statistics.py
2023-02-20 12:39:42 -05:00
J. Nick Koston 1759f58fc1
Ensure new MariaDB/MySQL tables are created using the InnoDB engine (#88468)
Ensure new tables are created using InnoDB

InnoDB is the only supported engine to use with MariaDB
or MySQL as we currently have large keys in the states
table that will not work with MyIASM. Other storage
engines including Aria will likely work fine, but they
are not officially supported.
2023-02-20 08:58:59 -05:00
J. Nick Koston c2b770bcb9
Load pending state attributes and event data ids at startup (#88444)
* Load pending state attributes and event data ids at startup

Since we queue all events to be processed after startup
we can have a thundering herd of queries to prime the
LRUs of event data and state attributes ids. Since we
know we are about to process a chunk of events we can
fetch all the ids in two queries

* lru

* fix hang

* Fix recorder LRU being destroyed if event session is reopened

We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened

* Fix recorder LRU being destroyed if event session is reopened

We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened

* cleanup
2023-02-19 21:26:38 -05:00
J. Nick Koston 3533e77ec9
Fix using MariaDB slow range select workaround with Postgresql (#88459)
* Mark PostgreSQL range select as fast

Currently we were using the slow range select workaround for
PostgreSQL that was original developed for MariaDB but
its actually slower on PostgreSQ

fixes #83253

* Mark PostgreSQL range select as fast

Currently we were using the slow range select workaround for
PostgreSQL that was original developed for MariaDB but
its actually slower on PostgreSQ

fixes #83253
2023-02-19 21:08:43 -05:00
J. Nick Koston 0b311c8c7f
Fix statistics_at_time query not using index (#88460)
* Fix statistics_at_time query not using index

fixes #82411

* fix refactoring error

* fix query so sqlalc does not get confused

* split it

* write as subquery

* reduce

* cleanup

* reduce

* Revert "reduce"

This reverts commit 43b4b55778.
2023-02-19 21:07:31 -05:00
J. Nick Koston eac9ad8437
Fix most recent states query not using the index for group by (#88461)
* Fix most recent states query not using the index for group by

fixes #87851

* Apply suggestions from code review

* reduce
2023-02-19 21:05:45 -05:00
J. Nick Koston 9a6bcc2b63
Fix incorrect created and created_ts columns in statistics database schema (#88462)
* Remove default from created statistics schema

We were still inserting created times because even though
None was passed when creating the object explictly, the
default would still be used

* adjust column

* preserve original pre sql alc 2.0 behavior
2023-02-19 21:00:49 -05:00
J. Nick Koston a9731a7b26
Make the recorder LRU adjustment threadsafe (#88443) 2023-02-19 19:30:08 +01:00
J. Nick Koston 08cdb0e3ee
Fix recorder LRU being destroyed if event session is reopened (#88448) 2023-02-19 19:20:01 +01:00
J. Nick Koston 9df00bd285
Adjust recorder LRU cache to align with the number of entities (#88350)
* Adjust size of recorder LRU based on number of entities

If there are a large number of entities the cache would
get thrashed as there were more state attributes being
recorded than the size of the cache. This meant we had
to go back to the database to do lookups frequently when
an instance has more than 2048 entities that change
frequently

* add a test

* do not actually record 4096 states

* patch target

* patch target
2023-02-19 00:02:36 -05:00
J. Nick Koston d4376b2041
Use a single session to compile statistics (#88405)
We had two session_scope contexts in compile_statistics which causes
a commit to happen twice during the compile.
2023-02-18 20:23:47 -05:00
J. Nick Koston 3ca9f3c0d5
Only build the recorder SupportedDialect enum once (#88402)
Every call to dialect_name was creating a new enum object
in the recorder
2023-02-18 20:21:14 -05:00
Erik Montnemery 1eb20affa1
Update use of sqlalchemy with_variant (#88395) 2023-02-18 09:18:01 -06:00
Erik Montnemery 83e5bf7ae8
Use entity_sources to determine integration in recorder platforms (#88382) 2023-02-18 07:21:41 -06:00
J. Nick Koston 0a7cf2b88c
Speed up formatting history data (#88230) 2023-02-18 10:34:44 +01:00
J. Nick Koston 289bab6f87
Make recorder platform attribute exclude integration aware (#88357) 2023-02-18 10:08:59 +01:00
J. Nick Koston b9f29c27ab
Cleanup stale recorder code (#88275)
- Removes dead commit function
- Small cleanup to debug handling in execute
2023-02-16 23:12:11 -05:00
J. Nick Koston 5fe8829cf6
Speed up compiling hourly statistics (#88225) 2023-02-16 12:55:25 -06:00
J. Nick Koston 6c542bd314
Speed up formatting statistics data (#88228)
* Speed up fetching statistics by using attrgetter

Uses the faster native code vs a lambda
4aeae28671/Modules/_operator.c (L1406)

* avoid dict lookups

* avoid dict lookups

* naming

* split

* Revert "split"

This reverts commit 0ead89603d.

* Revert "Revert "split""

This reverts commit 20014af168.

* tweak

* tweak

* tweak

* tweak

* Revert "tweak"

This reverts commit d5e4aac7c5.

* Revert "tweak"

This reverts commit 0cebae33f8.

* Revert "tweak"

This reverts commit 0184e47e24.

* comment

* comment

* comment

* reduce local vars

* less
2023-02-16 12:34:20 -05:00
epenet ba23816a0c
Inverse json import logic (#88099)
* Fix helpers and util

* Adjust components

* Move back errors

* Add report

* mypy

* mypy

* Assert deprecation messages

* Move test_json_loads_object

* Adjust tests

* Fix rebase

* Adjust pylint plugin

* Fix plugin

* Adjust references

* Adjust backup tests
2023-02-16 11:37:57 +01:00
J. Nick Koston 484027023c
Fix new stats rows creating data in the old created column (#88013) 2023-02-13 21:53:37 -06:00
J. Nick Koston 7b9293941e
The recorder/statistic(s)_during_period websocket API now require the statistic_id(s) field (#87937) 2023-02-12 14:51:10 -06:00
J. Nick Koston 32a6280139
Optimize _sorted_statistics_to_dict to avoid checking which types each loop (#87901)
* Optimize _sorted_statistics_to_dict to avoid checking which types each loop

* tweak

* avoid some more dict lookups
2023-02-11 23:21:40 -05:00
J. Nick Koston 85649ec589
Reduce overhead to see if an entity is recorded (#87912)
A significant chunk of list_statistic_ids was checking if the entity
was recorded because it had to get the recorder instance over and
over
2023-02-11 23:21:16 -05:00
J. Nick Koston e899754919
Bump sqlalchemy to 2.0.3 (#87913)
changes: https://docs.sqlalchemy.org/en/20/changelog/changelog_20.html#change-2.0.3
2023-02-11 23:20:22 -05:00
J. Nick Koston 86a93e9fce
Optimize fetching statistics by avoiding recalculating time boundaries (#87859)
* predictive

* fix conversion error

* fix conversion error

* fix conversion error

* convert day to use new algo

* reduce
2023-02-11 11:41:57 -05:00
J. Nick Koston c35661947a
Optimize fetching statistics by avoiding timestamp conversions (#87845)
* Optimize fetching statistics by avoiding timestamp conversions

* remove accidential unrelated change

* fix test

* recreate so we handle timezone changes
2023-02-10 11:26:46 -05:00
J. Nick Koston ea356ad260
Optimize fetching weekly/monthly/yearly statistics (#87747)
* Optimize fetching statistics

* speed up

* avoid double groupby

* avoid another loop

* tweak flow

* fixes

* tweak

* avoid a new dt object in the cache for week/month

* avoid a new dt object in the cache for week/month
2023-02-09 21:22:32 -05:00
J. Nick Koston 509de02044
Ensure recorder still shuts down if the final commit fails (#87799) 2023-02-09 15:12:40 -06:00
J. Nick Koston abf0c87e40
Migrate statistics to use timestamp columns (#87321) 2023-02-09 12:24:19 -06:00
J. Nick Koston db503c0daa
Always use UTC time for the MariaDB/MySQL session (#87583) 2023-02-09 11:39:45 -06:00
Erik Montnemery 5dfb9cddd6
Sort manifests 8 (#87029) 2023-02-08 20:13:46 +01:00
Erik Montnemery 94519de8dd
Upgrade SQLAlchemy to 2.0.2 (#86436)
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-08 08:17:32 -06:00
epenet a202588fd2
Add return type to json_loads (#85672)
* Add JSON type definitions

* Sample use

* Keep mutable for a follo-up PR (avoid dead code)

* Use list/dict

* Remove JsonObjectType

* Remove reference to Union

* Cleanup

* Improve rest

* Rename json_dict => json_data

* Add docstring

* Add type hint to json_loads

* Add cast

* Move type alias to json helpers

* Cleanup

* Create and use json_loads_object

* Make error more explicit and add tests

* Use JsonObjectType in conversation

* Remove quotes
2023-02-07 17:21:55 +01:00
Marc Mueller b478b4fa16
Add ParamSpec TypeAliases [mypy 1.0] (#87597) 2023-02-07 15:56:26 +01:00
J. Nick Koston 0a1d5c85ad
Optimize history.get_last_state_changes query (#87554)
fixes undefined
2023-02-07 07:50:44 -06:00
J. Nick Koston 3aa744ed19
Fix indent on slow_range_in_select for MySQL/MariaDB (#87581)
We would calculate this value and throw it away
because it was only used on first_connection
2023-02-06 23:59:54 -05:00
Marc Mueller 342b406dc0
Add Self typing (1) [mypy 1.0] (#87598) 2023-02-06 22:29:47 -06:00
Franck Nijhof 939eef3b28
Remove translations from Core (#87543)
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2023-02-06 15:06:43 +01:00
J. Nick Koston c7980bda82
Fix recorder run history during schema migration and startup (#87492)
Fix recorder run history during schema migration

RunHistory.get and RunHistory.current can be called before
RunHistory.start. We need to return a RecorderRuns object
with the recording_start time that will be used when start
it called to ensure history queries still work as expected.

fixes #87112
2023-02-05 22:12:41 -05:00
GitHub Action f475446ebf [ci skip] Translation update 2023-02-06 00:23:07 +00:00
J. Nick Koston c11f3ffa17
Speed up purge time with newer MariaDB versions (#87409)
* Speed up purge time with newer MariaDB versions

* fix

* document

* document

* document

* rename

* self review

* Update homeassistant/components/recorder/util.py

* fixes
2023-02-05 17:58:34 -05:00
Franck Nijhof 3d557b5583
Collection of code styling tweaks (#87381) 2023-02-04 12:52:26 -05:00
Franck Nijhof ca1a12898c
Enable Ruff D212 (#87347) 2023-02-03 23:08:48 +01:00
epenet 810367b757
Fix `can not` typo (#87254) 2023-02-03 11:37:16 +01:00