Commit Graph

70 Commits (a9381d259040d4a02881cc9e203bc8e3946b2146)

Author SHA1 Message Date
J. Nick Koston 2ec1359063
Dynamically size recorder max backlog based on available memory (#90894)
Co-authored-by: Franck Nijhof <git@frenck.dev>
2023-04-18 12:35:49 +02:00
J. Nick Koston 49079691d4
Reduce overhead of legacy database columns on new installs (#90246)
* Reduce overhead of legacy database columns on new installs

* Reduce overhead of legacy database columns on new installs

* Reduce overhead of legacy database columns on new installs

* Reduce overhead of legacy database columns on new installs

* not working as expected

* override the type compiler

* override the type compiler

* override the type compiler

* override the type compiler

* Apply suggestions from code review

* pgsql char1

* make entity filter test setup with old schema

* fix some more tests that were mutating state

* fix some more tests that were mutating state

* fix some more tests that were mutating state

* fix more dbstate mutations

* add shim for older tests

* split migration tests

* add coverage for purging legacy data

* tweak

* more fixes

* drop some legacy

* fix another test

* fix a few more

* add casts for postgresql in case someone deletes the schema changes table

* dry

* dry

* dry
2023-04-10 10:08:46 -04:00
J. Nick Koston a2efe2445a
Fix migration when encountering a NULL entity_id/event_type (#90542)
* Fix migration when encountering a NULL entity_id/event_type

reported in #beta on discord

* simplify
2023-03-30 20:54:13 -04:00
J. Nick Koston 6b0c98045e
Handle garbage in the context_id column during migration (#90544)
* Handle garbage in the context_id column during migration

* Update homeassistant/components/recorder/migration.py

* lint
2023-03-30 20:53:47 -04:00
J. Nick Koston 31c988c4f0
Fix index not being dropped on postgresql databases with a schema prefix (#90144)
* Fix index not being dropped on postgresql databases with a schema prefix

Added logging in case index drops fail so we can tell
why in the future

* coverage
2023-03-22 21:03:26 -04:00
J. Nick Koston 96225bb287
Rename recorder run_history to table_managers.recorder_runs_manager (#90070) 2023-03-21 18:38:33 -10:00
J. Nick Koston a244749712
Make StatesMetaManager thread-safe when an entity_id is fully deleted from the database and than re-added (#89732)
* refactor to make StatesMetaManager threadsafe

* refactor to make StatesMetaManager threadsafe

* refactor to make StatesMetaManager threadsafe

* refactor to make StatesMetaManager threadsafe

* reduce

* comments
2023-03-15 08:54:02 -04:00
J. Nick Koston b9ac6b4a7c
Improve reliability of context id migration (#89609)
* Split context id migration into states and events tasks

Since events can finish much earlier than states we
would keep looking at the table because states as not
done. Make them seperate tasks

* add retry dec

* fix migration happening twice

* another case
2023-03-12 21:41:48 -04:00
J. Nick Koston c41f91be89
Deduplicate entity_id in the states table (#89557) 2023-03-12 10:01:58 -10:00
J. Nick Koston 8bd43760b6
Deduplicate event_types in the events table (#89465)
* Deduplicate event_types in the events table

* Deduplicate event_types in the events table

* more fixes

* adjust

* adjust

* fix product

* fix tests

* adjust

* migrate

* migrate

* migrate

* more test fixes

* more test fixes

* fix

* migration test

* adjust

* speed up

* fix index

* fix more tests

* handle db failure

* preload

* tweak

* adjust

* fix stale docs strings, remove dead code

* refactor

* fix slow tests

* coverage

* self join to resolve query performance

* fix typo

* no need for quiet

* no need to drop index already dropped

* remove index that will never be used

* drop index sooner as we no longer use it

* Revert "remove index that will never be used"

This reverts commit 461aad2c52.

* typo
2023-03-11 14:54:55 -05:00
J. Nick Koston 170a13302c
Reduce overhead to store context ids in the database (#88942) 2023-03-08 14:51:45 -10:00
J. Nick Koston 1bd9767d8c
Handle InnoDB deadlocks during migration (#89073)
* Handle slow InnoDB rollback when encountering duplicates during migration

fixes #89069

* adjust

* fix mock

* tests

* return on success
2023-03-03 22:00:13 -05:00
J. Nick Koston c2b770bcb9
Load pending state attributes and event data ids at startup (#88444)
* Load pending state attributes and event data ids at startup

Since we queue all events to be processed after startup
we can have a thundering herd of queries to prime the
LRUs of event data and state attributes ids. Since we
know we are about to process a chunk of events we can
fetch all the ids in two queries

* lru

* fix hang

* Fix recorder LRU being destroyed if event session is reopened

We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened

* Fix recorder LRU being destroyed if event session is reopened

We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened

* cleanup
2023-02-19 21:26:38 -05:00
epenet 185cd61cbd
Add type hints to integration tests (recorder) (#88313) 2023-02-17 18:50:09 +01:00
Franck Nijhof ed79265843
Enable Ruff PT006 (#88165)
* Enable Ruff PT006

* Adjust existing cases

* Fix tests

* Remove unneeded parentheses
2023-02-15 14:09:50 +01:00
J. Nick Koston cbaf4764e7
Ensure MySQL tests cleanup connections and raise an exception if they do not (#87767)
Co-authored-by: epenet <6771947+epenet@users.noreply.github.com>
Co-authored-by: Erik <erik@montnemery.com>
2023-02-09 16:27:53 -06:00
J. Nick Koston abf0c87e40
Migrate statistics to use timestamp columns (#87321) 2023-02-09 12:24:19 -06:00
Erik Montnemery 2fae82a9ef
Really ensure recorder test fixture is setup before hass fixture (#87796) 2023-02-09 11:43:14 -06:00
J. Nick Koston f4a81a8b5b
Disable test_migration_in_progress on MySQL (#87765)
This test was recently enabled on MySQL in
https://github.com/home-assistant/core/pull/87753

Since it the migration is still in progress in the background
when the test ends, it causes deadlock with InnoDB when its dropped
out from under it
2023-02-09 15:27:27 +01:00
Erik Montnemery b0f5bdd504
Improve migration tests to use MariaDB / PostgreSQL database (#87753) 2023-02-09 08:42:32 +01:00
epenet ba85fdcd61
Add type hints to integration tests (q-s) (#87706) 2023-02-08 08:12:54 -08:00
epenet 59ca7780fa
Add typing to tests with single hass argument (#87631) 2023-02-07 15:01:16 +01:00
epenet 4142f0d15d
Add return type to tests without arguments (#87613)
* Add return type to tests without arguments

* Black

* Cancel fixture amends
2023-02-07 14:20:06 +01:00
Franck Nijhof 5e81d28116
Update black to 23.1.0 (#87188) 2023-02-02 18:35:24 +01:00
epenet 30bf0634fe
Add per-file-ignore to pylint (#86289) 2023-01-22 17:26:24 +01:00
Erik Montnemery 466c4656ca
Refactor recorder migration (#80175)
* Refactor recorder migration

* Improve test coverage
2022-10-13 08:11:54 +02:00
Erik 4a1c40f09b Revert "Refactor recorder migration"
This reverts commit 69e10e5982.
2022-10-12 15:12:12 +02:00
Erik 69e10e5982 Refactor recorder migration 2022-10-12 15:00:28 +02:00
Erik Montnemery 2de273500e
Remove state_unit_of_measurement from metadata DB table (#79370)
* Remove state_unit_of_measurement from metadata DB table

* Adjust test
2022-10-01 18:55:00 +02:00
Erik Montnemery dd20a7ea62
Display statistics in the source's unit (#78031) 2022-09-15 12:01:24 -04:00
Erik Montnemery 0a1fd36e03
Catch up with statistics after DB migration is done (#78469)
* Catch up with statistics after DB migration is done

* Don't access the database from the event loop

* Fix deadlocking test

* Fix test
2022-09-15 07:40:56 +02:00
Erik Montnemery fd6ffef52f
Support non-live database migration (#72433)
* Support non-live database migration

* Tweak startup order, add test

* Address review comments

* Fix typo

* Clarify comment about promoting dependencies

* Tweak

* Fix merge mistake

* Fix some tests

* Fix additional test

* Fix additional test

* Adjust tests

* Improve test coverage
2022-07-22 15:11:34 +02:00
Erik Montnemery 606d544157
Use recorder get_instance function to improve typing (#75567) 2022-07-22 11:58:26 +02:00
Erik Montnemery 5f2b4001f3
Separate recorder database schema from other classes (#72977)
* Separate recorder database schema from other classes

* fix logbook imports

* migrate new tests

* few more

* last one

* fix merge

Co-authored-by: J. Nick Koston <nick@koston.org>
2022-06-07 14:41:43 +02:00
J. Nick Koston eb77f8db85
Complete strict typing for recorder (#71274)
* Complete strict typing for recorder

* update tests

* Update tests/components/recorder/test_migrate.py

Co-authored-by: Martin Hjelmare <marhje52@gmail.com>

* Update tests/components/recorder/test_migrate.py

Co-authored-by: Martin Hjelmare <marhje52@gmail.com>

* Remove the asserts

* remove ignore comments

Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2022-05-04 12:22:50 -05:00
J. Nick Koston 13ce0a7d6a
Fix history using pre v25 queries during v26 migration (#71295) 2022-05-04 08:56:50 -07:00
J. Nick Koston 29bda196b5
Break apart recorder into tasks and core modules (#71222) 2022-05-02 21:53:56 -07:00
J. Nick Koston 1c4a785fb3
Prevent autoflush from happening during attrs lookup (#70768) 2022-04-26 13:04:58 -07:00
J. Nick Koston f073f17040
Refactor tracking of the recorder run history (#70456)
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2022-04-26 09:59:43 -10:00
J. Nick Koston 89e3a48751
Use database executor in migration test (#70774) 2022-04-26 07:42:28 -07:00
Erik Montnemery 28ebab9c5a
Simplify waiting for recorder in tests (#70647) 2022-04-25 12:04:47 +02:00
J. Nick Koston 66f0a3816a
Reduce memory pressure during database migration (#69628) 2022-04-07 18:29:31 -10:00
Erik Montnemery 949b0e1b65
Don't allow in-memory SQLite database (#69616) 2022-04-08 00:43:09 +02:00
J. Nick Koston e911936a0d
Remove direct usage of concurrent.futures from recorder (#68593) 2022-03-24 09:48:49 -10:00
Erik Montnemery 41ab12cb88
Don't use shared session during recorder migration (#65672) 2022-02-04 09:55:11 -08:00
Erik Montnemery e5ee38d3b0
Adjust recorder tests according to #59087 (#59143) 2021-11-05 10:49:58 +01:00
Erik Montnemery 470b01e4ce
Move async_migration_in_progress (#59087) 2021-11-05 10:40:56 +01:00
Erik Montnemery 185f7beafc
Improve recorder migration tests (#59075) 2021-11-04 20:21:38 -07:00
Erik Montnemery 4c5aca93df
Add recorder status WS API (#58989)
* Add recorder status WS API

* Rename recorder/status to recorder/info

* Silence pylint

* Improve tests

* Address review comments

* Tweak

* Try to fix tests

* Try to debug flaky tests

* Try to fix tests

* Revert changes to async_migration_in_progress

* Try to fix tests

* Remove debug prints

* Apply suggestions from code review
2021-11-04 16:46:45 +01:00
Paulus Schoutsen a4d9019ffc
Refactor persistent notification to no longer route all data via a service (#57157)
* Convert persistent notification tests to async

* Create/dismiss persistent notifications in exposed functions, not service calls

* Fix notify persistent_notification

* Remove setting up persistent_notification

* Drop more setups

* Empty methods

* Undeprecate sync methods because too big task

* Fix setup clearing notifications

* Fix a bunch of tests

* Fix more tests

* Uno mas

* Test persistent notification events

* Clean up stale comment

Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-10-07 12:58:00 +02:00