Commit Graph

77 Commits (415042f356cb77f136d7abbcc93c8bec9fb85c1d)

Author SHA1 Message Date
J. Nick Koston 88893121a1
Increase timeout for test_purge_many_old_events (#93579) 2023-05-25 22:38:44 -05:00
J. Nick Koston 9dcd73b308
Update recorder tests to avoid patching utcnow (#93489) 2023-05-24 17:47:42 -05:00
J. Nick Koston ce3cb5864a
Improve reliablity of recorder purge test (#92641) 2023-05-07 01:28:43 +02:00
J. Nick Koston 6e628d2f06
Add a negative cache to the recorder event type manager (#91770) 2023-04-22 14:09:04 +02:00
J. Nick Koston 49079691d4
Reduce overhead of legacy database columns on new installs (#90246)
* Reduce overhead of legacy database columns on new installs

* Reduce overhead of legacy database columns on new installs

* Reduce overhead of legacy database columns on new installs

* Reduce overhead of legacy database columns on new installs

* not working as expected

* override the type compiler

* override the type compiler

* override the type compiler

* override the type compiler

* Apply suggestions from code review

* pgsql char1

* make entity filter test setup with old schema

* fix some more tests that were mutating state

* fix some more tests that were mutating state

* fix some more tests that were mutating state

* fix more dbstate mutations

* add shim for older tests

* split migration tests

* add coverage for purging legacy data

* tweak

* more fixes

* drop some legacy

* fix another test

* fix a few more

* add casts for postgresql in case someone deletes the schema changes table

* dry

* dry

* dry
2023-04-10 10:08:46 -04:00
J. Nick Koston d0d4ab6056
Require a list of entity ids when fetching history (#90992) 2023-04-08 16:14:44 -10:00
J. Nick Koston aebe4c66a6
Fix cpu thrashing during purge after all legacy events were removed (#89923)
* Fix cpu thrashing during purge after all legacy events were removed

We now remove the the index of of event ids on the states table when its
all NULLs to save space. The purge path needs to avoid checking for legacy
rows to purge if the index has been removed since it will result in a full
table scan each purge cycle that will always find no legacy rows to purge

* one more place

* drop the key constraint as well

* fixes

* more sqlite
2023-03-19 22:04:24 -04:00
J. Nick Koston 99d6b1fa57
Migrate States to use a table manager (#89769) 2023-03-15 16:19:43 -10:00
J. Nick Koston aec2d63302
Add keep_days to recorder.purge_entities (#89726) 2023-03-15 11:13:47 -10:00
J. Nick Koston a244749712
Make StatesMetaManager thread-safe when an entity_id is fully deleted from the database and than re-added (#89732)
* refactor to make StatesMetaManager threadsafe

* refactor to make StatesMetaManager threadsafe

* refactor to make StatesMetaManager threadsafe

* refactor to make StatesMetaManager threadsafe

* reduce

* comments
2023-03-15 08:54:02 -04:00
J. Nick Koston b906d67c1e
Fix filtered purge not removing newer events (#89721) 2023-03-14 16:33:19 -10:00
J. Nick Koston c41f91be89
Deduplicate entity_id in the states table (#89557) 2023-03-12 10:01:58 -10:00
J. Nick Koston 8bd43760b6
Deduplicate event_types in the events table (#89465)
* Deduplicate event_types in the events table

* Deduplicate event_types in the events table

* more fixes

* adjust

* adjust

* fix product

* fix tests

* adjust

* migrate

* migrate

* migrate

* more test fixes

* more test fixes

* fix

* migration test

* adjust

* speed up

* fix index

* fix more tests

* handle db failure

* preload

* tweak

* adjust

* fix stale docs strings, remove dead code

* refactor

* fix slow tests

* coverage

* self join to resolve query performance

* fix typo

* no need for quiet

* no need to drop index already dropped

* remove index that will never be used

* drop index sooner as we no longer use it

* Revert "remove index that will never be used"

This reverts commit 461aad2c52.

* typo
2023-03-11 14:54:55 -05:00
J. Nick Koston 1bd9767d8c
Handle InnoDB deadlocks during migration (#89073)
* Handle slow InnoDB rollback when encountering duplicates during migration

fixes #89069

* adjust

* fix mock

* tests

* return on success
2023-03-03 22:00:13 -05:00
J. Nick Koston c2b770bcb9
Load pending state attributes and event data ids at startup (#88444)
* Load pending state attributes and event data ids at startup

Since we queue all events to be processed after startup
we can have a thundering herd of queries to prime the
LRUs of event data and state attributes ids. Since we
know we are about to process a chunk of events we can
fetch all the ids in two queries

* lru

* fix hang

* Fix recorder LRU being destroyed if event session is reopened

We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened

* Fix recorder LRU being destroyed if event session is reopened

We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened

* cleanup
2023-02-19 21:26:38 -05:00
epenet 185cd61cbd
Add type hints to integration tests (recorder) (#88313) 2023-02-17 18:50:09 +01:00
epenet efef915fc7
Add mariadb 10.3.32 to database tests (#87741)
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-13 16:28:39 -06:00
J. Nick Koston abf0c87e40
Migrate statistics to use timestamp columns (#87321) 2023-02-09 12:24:19 -06:00
Erik Montnemery 720f51657d
Add CI job which runs recorder tests on PostgreSQL (#80614)
Co-authored-by: Franck Nijhof <git@frenck.dev>
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-07 09:46:14 -06:00
J. Nick Koston b8a1537b58
Improve performance of fetching and storing history and events with the database (#84870) 2023-01-02 13:26:08 -10:00
Erik Montnemery f4951a4f31
Add CI job which runs recorder tests on MariaDB (#80586)
Co-authored-by: Franck Nijhof <git@frenck.dev>
2022-10-19 13:04:28 +02:00
Erik Montnemery 31a787558f
Ensure recorder test fixture is setup before hass fixture (#80528)
* Ensure recorder test fixture is setup before hass fixture

* Adjust more tests
2022-10-19 07:58:47 +02:00
Erik Montnemery 5f2b4001f3
Separate recorder database schema from other classes (#72977)
* Separate recorder database schema from other classes

* fix logbook imports

* migrate new tests

* few more

* last one

* fix merge

Co-authored-by: J. Nick Koston <nick@koston.org>
2022-06-07 14:41:43 +02:00
J. Nick Koston d8b0376942
Fix purge of legacy database events that are not state changed (#72815) 2022-06-01 11:56:06 +02:00
J. Nick Koston a70e2a33dc
Fixing purging legacy rows and improve performance (#71916) 2022-05-15 21:25:07 -07:00
J. Nick Koston a8aa0e1cca
Add Estimated Database Size to the recorder system health (#71463) 2022-05-07 21:02:54 -07:00
J. Nick Koston e30940ef2a
Move processing of recorder service call arguments into services.py (#71260) 2022-05-03 13:56:22 -07:00
J. Nick Koston e1be6dd34f
Move recorder services to services.py (#71249) 2022-05-03 10:38:44 -07:00
J. Nick Koston 29bda196b5
Break apart recorder into tasks and core modules (#71222) 2022-05-02 21:53:56 -07:00
J. Nick Koston 5db014666c
Avoid recording state_changed events in the events table (#71165)
* squash

fix mypy

* Update homeassistant/components/recorder/models.py

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>

* pass all columns

* fix commented out code

* reduce logbook query complexity

* merge

* comment

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2022-05-02 02:10:34 -05:00
J. Nick Koston 6ce768465f
Improve purge performance for non-sqlite databases (#70576) 2022-04-26 13:11:57 -07:00
Erik Montnemery 28ebab9c5a
Simplify waiting for recorder in tests (#70647) 2022-04-25 12:04:47 +02:00
Erik Montnemery 982e314de6
Use recorder_mock in tests (#70363)
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2022-04-22 19:29:44 -10:00
J. Nick Koston 3142a106fc
Additional prep work for sqlalchemy 2.0 (#70358) 2022-04-21 12:39:18 -10:00
Erik Montnemery eb3458a3d2
Add MutexPool for recorder tests (#69410)
* Add MutexPool for recorder tests

* Fix get_schema_version

* Update test test_last_run_was_recently_clean

* Update test test_shutdown_before_startup_finishes

* Revert comments in test_write_lock_db

* Make the MutexPool lock a class variable

* Remove stale comment

* Move MutexPool

* Tweak debug prints
2022-04-12 07:41:46 -10:00
J. Nick Koston 3150915cb7
Convert unindexed domain queries to entity_id queries (#68404) 2022-03-20 01:28:17 -10:00
J. Nick Koston 7ee647cc78
Fix FOREIGN KEY constraint failed when removing state_attributes (#68364) 2022-03-19 14:21:28 -07:00
J. Nick Koston 9215702388
Separate attrs into another table (reduces database size) (#68224) 2022-03-18 00:23:13 -10:00
J. Nick Koston b18096fc54
Remove unused columns from states/events tables (#68078) 2022-03-13 23:44:55 -07:00
Erik Montnemery b301ab25a3
Purge short term statistics (#58028)
* Purge short term statistics

* Less meep

* Add tests
2021-10-20 16:00:59 +02:00
Erik Montnemery 8567aa9e13
Evict purged states from recorder's old_state cache (#56877)
Co-authored-by: J. Nick Koston <nick@koston.org>
2021-10-04 08:21:40 -07:00
PeteBa 42c4317628
Avoid drift in recorder purge cut-off (#52135) 2021-06-25 11:29:38 -10:00
PeteBa aa9b99713c
Add purge_entities service call to recorder (#48069) 2021-05-22 10:30:05 -05:00
Erik Montnemery 89dd3292ba
Initial draft of statistics (#49852) 2021-05-16 10:23:37 -07:00
jan iversen c351098f04
HomeAssistantType —> HomeAssistant for Integrations p* - s* (#49558) 2021-04-22 19:58:02 +02:00
J. Nick Koston 6d137d2316
Increase recorder test coverage (#49362)
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-04-19 08:22:38 -07:00
J. Nick Koston 9368891b1b
Live db migrations and recovery (#49036)
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2021-04-11 20:43:54 -10:00
Franck Nijhof 08870690a6
Fix a collection of tests with missing asserts (#48127)
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-03-20 17:23:55 +01:00
Franck Nijhof fb849b81b5
Rewrite of not a == b occurances (#48132) 2021-03-20 01:27:04 +01:00
J. Nick Koston 28c80c1133
Ensure recorder purge tests can handle multiple purge cycle (#47956)
Since a purge can generate another purge task, we now wait for
three recorder queue completions by default.
2021-03-15 11:19:19 -07:00