Commit Graph

334 Commits (4f6a25b470a6779a2a3cbdfd0634e2183278809d)

Author SHA1 Message Date
J. Nick Koston c2b770bcb9
Load pending state attributes and event data ids at startup (#88444)
* Load pending state attributes and event data ids at startup

Since we queue all events to be processed after startup
we can have a thundering herd of queries to prime the
LRUs of event data and state attributes ids. Since we
know we are about to process a chunk of events we can
fetch all the ids in two queries

* lru

* fix hang

* Fix recorder LRU being destroyed if event session is reopened

We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened

* Fix recorder LRU being destroyed if event session is reopened

We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened

* cleanup
2023-02-19 21:26:38 -05:00
J. Nick Koston 3533e77ec9
Fix using MariaDB slow range select workaround with Postgresql (#88459)
* Mark PostgreSQL range select as fast

Currently we were using the slow range select workaround for
PostgreSQL that was original developed for MariaDB but
its actually slower on PostgreSQ

fixes #83253

* Mark PostgreSQL range select as fast

Currently we were using the slow range select workaround for
PostgreSQL that was original developed for MariaDB but
its actually slower on PostgreSQ

fixes #83253
2023-02-19 21:08:43 -05:00
J. Nick Koston a9731a7b26
Make the recorder LRU adjustment threadsafe (#88443) 2023-02-19 19:30:08 +01:00
J. Nick Koston 9df00bd285
Adjust recorder LRU cache to align with the number of entities (#88350)
* Adjust size of recorder LRU based on number of entities

If there are a large number of entities the cache would
get thrashed as there were more state attributes being
recorded than the size of the cache. This meant we had
to go back to the database to do lookups frequently when
an instance has more than 2048 entities that change
frequently

* add a test

* do not actually record 4096 states

* patch target

* patch target
2023-02-19 00:02:36 -05:00
J. Nick Koston 3ca9f3c0d5
Only build the recorder SupportedDialect enum once (#88402)
Every call to dialect_name was creating a new enum object
in the recorder
2023-02-18 20:21:14 -05:00
Erik Montnemery 83e5bf7ae8
Use entity_sources to determine integration in recorder platforms (#88382) 2023-02-18 07:21:41 -06:00
J. Nick Koston 289bab6f87
Make recorder platform attribute exclude integration aware (#88357) 2023-02-18 10:08:59 +01:00
epenet a16e298599
Adjust json imports in tests (#88248) 2023-02-17 19:13:48 +01:00
epenet 185cd61cbd
Add type hints to integration tests (recorder) (#88313) 2023-02-17 18:50:09 +01:00
J. Nick Koston b9f29c27ab
Cleanup stale recorder code (#88275)
- Removes dead commit function
- Small cleanup to debug handling in execute
2023-02-16 23:12:11 -05:00
Franck Nijhof d3ed8de232
Update Ruff to v0.0.247 (#88210)
* Update Ruff to v0.0.247

* Fix new found violations
2023-02-15 16:46:03 -05:00
Franck Nijhof ed79265843
Enable Ruff PT006 (#88165)
* Enable Ruff PT006

* Adjust existing cases

* Fix tests

* Remove unneeded parentheses
2023-02-15 14:09:50 +01:00
epenet efef915fc7
Add mariadb 10.3.32 to database tests (#87741)
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-13 16:28:39 -06:00
Erik Montnemery 4ca0a24f87
Increase test scope of MariaDB + PostgreSQL tests (#87019)
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-13 15:58:13 -06:00
J. Nick Koston 7b9293941e
The recorder/statistic(s)_during_period websocket API now require the statistic_id(s) field (#87937) 2023-02-12 14:51:10 -06:00
J. Nick Koston c35661947a
Optimize fetching statistics by avoiding timestamp conversions (#87845)
* Optimize fetching statistics by avoiding timestamp conversions

* remove accidential unrelated change

* fix test

* recreate so we handle timezone changes
2023-02-10 11:26:46 -05:00
J. Nick Koston cbaf4764e7
Ensure MySQL tests cleanup connections and raise an exception if they do not (#87767)
Co-authored-by: epenet <6771947+epenet@users.noreply.github.com>
Co-authored-by: Erik <erik@montnemery.com>
2023-02-09 16:27:53 -06:00
J. Nick Koston 509de02044
Ensure recorder still shuts down if the final commit fails (#87799) 2023-02-09 15:12:40 -06:00
J. Nick Koston abf0c87e40
Migrate statistics to use timestamp columns (#87321) 2023-02-09 12:24:19 -06:00
Erik Montnemery 2fae82a9ef
Really ensure recorder test fixture is setup before hass fixture (#87796) 2023-02-09 11:43:14 -06:00
J. Nick Koston db503c0daa
Always use UTC time for the MariaDB/MySQL session (#87583) 2023-02-09 11:39:45 -06:00
J. Nick Koston c05a7b29e6
Terminate stale MySQL connections at the end of test runs (#87794) 2023-02-09 11:13:13 -06:00
J. Nick Koston 03b47f1931
Disable test_entity_ids on MariaDB (#87766)
Dropping the database after this test will fail on
MySQL and hang forever because it causes an InnoDB deadlock

```
| 2042 | root   | localhost:52698 | NULL               | Query   |   41 | Waiting for table metadata lock | DROP DATABASE `homeassistant-test` |    0.000 |
```
2023-02-09 15:30:35 +01:00
J. Nick Koston f4a81a8b5b
Disable test_migration_in_progress on MySQL (#87765)
This test was recently enabled on MySQL in
https://github.com/home-assistant/core/pull/87753

Since it the migration is still in progress in the background
when the test ends, it causes deadlock with InnoDB when its dropped
out from under it
2023-02-09 15:27:27 +01:00
J. Nick Koston 483b0cd017
Skip db locking test with postgresql/MySQL (#87760)
* Skip db locking test with postgresql/MySQL

https://github.com/home-assistant/core/pull/87756#issuecomment-1423828389

* switch to the guards instead
2023-02-09 13:58:32 +01:00
J. Nick Koston 77cdde94f3
Turn back off test_shutdown_before_startup_finishes test on MySQL (#87761) 2023-02-09 03:12:10 -06:00
Erik Montnemery 58618b2978
Improve recorder tests to use MariaDB / PostgreSQL (#87756) 2023-02-09 02:18:20 -06:00
Erik Montnemery b0f5bdd504
Improve migration tests to use MariaDB / PostgreSQL database (#87753) 2023-02-09 08:42:32 +01:00
epenet ba85fdcd61
Add type hints to integration tests (q-s) (#87706) 2023-02-08 08:12:54 -08:00
Erik Montnemery 94519de8dd
Upgrade SQLAlchemy to 2.0.2 (#86436)
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-08 08:17:32 -06:00
Erik Montnemery 720f51657d
Add CI job which runs recorder tests on PostgreSQL (#80614)
Co-authored-by: Franck Nijhof <git@frenck.dev>
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-07 09:46:14 -06:00
epenet 59ca7780fa
Add typing to tests with single hass argument (#87631) 2023-02-07 15:01:16 +01:00
J. Nick Koston 0a1d5c85ad
Optimize history.get_last_state_changes query (#87554)
fixes undefined
2023-02-07 07:50:44 -06:00
epenet 4142f0d15d
Add return type to tests without arguments (#87613)
* Add return type to tests without arguments

* Black

* Cancel fixture amends
2023-02-07 14:20:06 +01:00
epenet aa00114c2f
Add return type to async tests without arguments (#87612) 2023-02-07 10:26:56 +01:00
Marc Mueller 342b406dc0
Add Self typing (1) [mypy 1.0] (#87598) 2023-02-06 22:29:47 -06:00
J. Nick Koston c7980bda82
Fix recorder run history during schema migration and startup (#87492)
Fix recorder run history during schema migration

RunHistory.get and RunHistory.current can be called before
RunHistory.start. We need to return a RecorderRuns object
with the recording_start time that will be used when start
it called to ensure history queries still work as expected.

fixes #87112
2023-02-05 22:12:41 -05:00
J. Nick Koston c11f3ffa17
Speed up purge time with newer MariaDB versions (#87409)
* Speed up purge time with newer MariaDB versions

* fix

* document

* document

* document

* rename

* self review

* Update homeassistant/components/recorder/util.py

* fixes
2023-02-05 17:58:34 -05:00
epenet 19e27f4f52
Ensure recorder api aligns with sensor converters (#87257) 2023-02-04 14:28:47 -06:00
J. Nick Koston 389fc515a1
Prevent overly large attributes from being stored in the database (#87105) 2023-02-02 11:49:33 -06:00
Franck Nijhof 5e81d28116
Update black to 23.1.0 (#87188) 2023-02-02 18:35:24 +01:00
epenet 58596d205c
Ensure sensor and recorder converters align (#87138) 2023-02-02 10:36:03 +01:00
J. Nick Koston 1fd58b6cb7
Fix a few cartesian products in recorder tests (#87106)
Fixes
```
recorder/test_init.py:251: SAWarning: SELECT statement has a cartesian product between FROM element(s) "states" and FROM element "state_attributes".  Apply join condition(s) between each element to resolve.
```
2023-02-02 07:47:09 +01:00
J. Nick Koston fb55933b0a
Reduce chance of queue overflow during schema migration (#87090)
Co-authored-by: Franck Nijhof <git@frenck.dev>
2023-02-01 18:34:40 +01:00
J. Nick Koston 5284837c8f
Add a repair issue when using MariaDB is affected by MDEV-25020 (#87040)
closes https://github.com/home-assistant/core/issues/83787
2023-01-31 20:42:07 +01:00
Erik Montnemery 53c5f02ca2
Remove some dead code from recorder (#86697) 2023-01-30 18:37:48 +01:00
J. Nick Koston c612a92cfb
Use python defaults for comparing State, LazyState, and Event objects (#86856)
* Speed up comparing State and Event objects

Use default python implementation for State and Event __hash__ and __eq__

The default implementation compared based on the id() of the object
which is effectively what we want here anyways. These overrides are
left over from the days when these used to be attrs objects

By avoiding implementing these ourselves all of the equality checks
can happen in native code

* tweak

* adjust tests

* write out some more

* fix test to not compare objects

* more test fixes

* more test fixes

* correct stats tests

* fix more tests

* fix more tests

* update sensor recorder tests
2023-01-29 13:31:43 -05:00
Franck Nijhof a79885ceaf
Enable Ruff SIM117 (#86783) 2023-01-27 11:52:49 +01:00
Franck Nijhof e4a78420b8
Enable Ruff PT013 (#86757)
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2023-01-27 09:09:46 +01:00
Erik Montnemery fea30c1ce9
Terminate strings at NUL when recording states and events (#86687) 2023-01-26 11:11:03 +01:00