Commit Graph

305 Commits (94519de8dd9fcac684b510904a38c2269c27a042)

Author SHA1 Message Date
J. Nick Koston 13ce0a7d6a
Fix history using pre v25 queries during v26 migration (#71295) 2022-05-04 08:56:50 -07:00
J. Nick Koston e30940ef2a
Move processing of recorder service call arguments into services.py (#71260) 2022-05-03 13:56:22 -07:00
J. Nick Koston e1be6dd34f
Move recorder services to services.py (#71249) 2022-05-03 10:38:44 -07:00
J. Nick Koston 29bda196b5
Break apart recorder into tasks and core modules (#71222) 2022-05-02 21:53:56 -07:00
J. Nick Koston 75026f9fed
Separate recorder logic for state_changed and non-state_changed events (#71204) 2022-05-02 17:22:53 -05:00
J. Nick Koston 188040b8bb
Use lambda_stmt for recorder queries and migrate them to queries module (#71219) 2022-05-02 15:17:21 -07:00
J. Nick Koston 5db014666c
Avoid recording state_changed events in the events table (#71165)
* squash

fix mypy

* Update homeassistant/components/recorder/models.py

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>

* pass all columns

* fix commented out code

* reduce logbook query complexity

* merge

* comment

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2022-05-02 02:10:34 -05:00
J. Nick Koston c23866e5e5
De-duplicate event data into a new event_data table (#71135) 2022-05-01 21:01:17 -05:00
J. Nick Koston ff48720c6a
Remove db schema v23 stats migration tests (#71137) 2022-05-01 16:11:11 -07:00
J. Nick Koston 66a21e0bc3
Add basic system health data to the recorder (#71086) 2022-04-30 11:10:20 -05:00
J. Nick Koston 195811843b
Remove get_state and get_states history api calls (#70830) 2022-04-26 16:05:43 -07:00
J. Nick Koston 6ce768465f
Improve purge performance for non-sqlite databases (#70576) 2022-04-26 13:11:57 -07:00
J. Nick Koston 1c4a785fb3
Prevent autoflush from happening during attrs lookup (#70768) 2022-04-26 13:04:58 -07:00
J. Nick Koston f073f17040
Refactor tracking of the recorder run history (#70456)
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2022-04-26 09:59:43 -10:00
Erik Montnemery 3016b5fbfc
Use recorder fixtures and helpers in tests (#70773) 2022-04-26 06:08:00 -10:00
J. Nick Koston 89e3a48751
Use database executor in migration test (#70774) 2022-04-26 07:42:28 -07:00
Erik Montnemery d045e8678d
Use recorder test fixtures in tests (#70652) 2022-04-25 14:23:52 +02:00
Erik Montnemery 28ebab9c5a
Simplify waiting for recorder in tests (#70647) 2022-04-25 12:04:47 +02:00
J. Nick Koston 9f11063724
Fix state_changes_during_period bakery caching for limit and descending (#70610) 2022-04-24 21:47:21 -07:00
Erik Montnemery 982e314de6
Use recorder_mock in tests (#70363)
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2022-04-22 19:29:44 -10:00
J. Nick Koston 3737b58e85
Avoid fetching metadata multiple times during stat compile (#70397) 2022-04-22 00:25:42 -10:00
J. Nick Koston 3142a106fc
Additional prep work for sqlalchemy 2.0 (#70358) 2022-04-21 12:39:18 -10:00
J. Nick Koston 89807f0d2d
Reduce the number of queries needed to compile statistics (#69731)
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2022-04-15 09:13:29 -07:00
J. Nick Koston c85387290a
Increase recorder queue max backlog to 40k, improve message (#70065) 2022-04-14 11:45:07 -10:00
Erik Montnemery eb3458a3d2
Add MutexPool for recorder tests (#69410)
* Add MutexPool for recorder tests

* Fix get_schema_version

* Update test test_last_run_was_recently_clean

* Update test test_shutdown_before_startup_finishes

* Revert comments in test_write_lock_db

* Make the MutexPool lock a class variable

* Remove stale comment

* Move MutexPool

* Tweak debug prints
2022-04-12 07:41:46 -10:00
J. Nick Koston 72fffde77a
Fix spelling of periodic in recorder (#69658)
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2022-04-07 23:37:02 -10:00
J. Nick Koston 66f0a3816a
Reduce memory pressure during database migration (#69628) 2022-04-07 18:29:31 -10:00
Erik Montnemery 949b0e1b65
Don't allow in-memory SQLite database (#69616) 2022-04-08 00:43:09 +02:00
Erik Montnemery 4a7c978f69
Speedup recorder test test_write_lock_db (#69578) 2022-04-07 15:36:58 +02:00
J. Nick Koston 5c7c09726a
Cleanup recorder history typing (#69408) 2022-04-07 00:09:05 -10:00
J. Nick Koston bde3646293
Always use a commit interval of 0 for the in memory db in tests (#69330) 2022-04-05 12:06:06 -10:00
J. Nick Koston ec131d685e
Auto repack the database on the second sunday of the month (#69314) 2022-04-04 21:39:12 -10:00
J. Nick Koston f5a13fc51b
Add ability to exclude attributes from being recorded by entity domain (#68824) 2022-03-29 20:13:08 -07:00
Joakim Sørensen 32b2d1e5c9
Add backup platform to recorder (#68229) 2022-03-25 23:17:11 -07:00
J. Nick Koston a566d3943c
Fix history queries while the database migration is in progress (#68598) 2022-03-24 09:49:13 -10:00
J. Nick Koston e911936a0d
Remove direct usage of concurrent.futures from recorder (#68593) 2022-03-24 09:48:49 -10:00
Erik Montnemery 61cc8e32f3
Include has_mean + has_sum in statistics metadata WS response (#68546)
* Include has_mean + has_sum in statistics metadata WS response

* Don't include has_mean/has_sum in history/list_statistic_ids

* Adjust tests

* Do include has_mean/has_sum in history/list_statistic_ids
2022-03-24 10:12:01 +01:00
Erik Montnemery b5c5da96ac
Add WS API to adjust incorrect energy statistics (#65147)
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2022-03-22 15:18:30 -07:00
Pawel bcb8c7ec3c
Add API endpoint get_statistics_metadata (#68471)
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2022-03-21 21:14:47 -07:00
J. Nick Koston 3150915cb7
Convert unindexed domain queries to entity_id queries (#68404) 2022-03-20 01:28:17 -10:00
J. Nick Koston 816695cc96
Avoid selecting attributes in the history api when `no_attributes` is passed (#68352) 2022-03-19 23:47:22 -10:00
Erik Montnemery cf4033b1bc
Simplify time zone setting in tests (#68330)
* Simplify timezone setting in tests

* Fix typo

* Adjust caldav tests

* Adjust input_datetime tests

* Adjust time_date tests

* Adjust tod tests

* Adjust helper tests

* Adjust recorder tests

* Adjust risco tests

* Adjust aemet tests

* Adjust flux tests

* Adjust forecast_solar tests

* Revert unnecessary change in forecast_solar test

* Adjust climacell tests

* Adjust google tests

* Adjust sensor tests

* Adjust sonarr tests

* Adjust template tests

* Adjust zodiac tests

Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2022-03-20 10:25:15 +01:00
J. Nick Koston 7ee647cc78
Fix FOREIGN KEY constraint failed when removing state_attributes (#68364) 2022-03-19 14:21:28 -07:00
J. Nick Koston 9215702388
Separate attrs into another table (reduces database size) (#68224) 2022-03-18 00:23:13 -10:00
J. Nick Koston bc862e97ed
Use a dedicated executor pool for database operations (#68105)
Co-authored-by: Erik Montnemery <erik@montnemery.com>
Co-authored-by: Franck Nijhof <git@frenck.dev>
2022-03-17 23:09:01 -10:00
J. Nick Koston 95f20500ca
Commit any pending changes before running non-EventTasks in the recorder (#68287) 2022-03-17 21:33:22 -07:00
Erik Montnemery d360ac91ca
Disable recorder nightly jobs in tests (#68188) 2022-03-15 09:23:52 -07:00
J. Nick Koston b18096fc54
Remove unused columns from states/events tables (#68078) 2022-03-13 23:44:55 -07:00
Erik Montnemery 910b1f1ec8
Speed up deletion of duplicated statistics (#66014) 2022-02-07 09:11:52 -08:00
Erik Montnemery 41ab12cb88
Don't use shared session during recorder migration (#65672) 2022-02-04 09:55:11 -08:00
Erik Montnemery ac7662c82d
Remove limit of amount of duplicated statistics (#65641) 2022-02-04 14:49:45 +01:00
Marc Mueller d057850971
Update typing - collections.abc (3) (#63947) 2022-01-12 08:04:17 +01:00
Stefan Agner 0a9927d18e
Avoid locking the database for non-SQLite backends (#63847)
* Avoid locking the database for non-SQLite backends

Currently we only have a lock implementation for SQLite. Just return
success for all other databases as they are not expected to store data
in the config directory and the caller can assume that a backup can
be safely taken.

This fixes `RuntimeError: generator didn't yield` errors when creating
a backup with the current Supervisor dev builds.
2022-01-11 16:17:56 +01:00
Erik Montnemery 2f8e65a9b0
Store deleted duplicated statistics in .storage (#62574) 2021-12-22 14:27:56 +01:00
Erik Montnemery 48b3d6e1c0
Save original + duplicate pairs when deleting duplicated statistics (#62498) 2021-12-21 15:24:36 +01:00
Erik Montnemery cab2a74b5f
Don't pollute config dir with deleted duplicated statistics (#62489) 2021-12-21 14:27:35 +01:00
Erik Montnemery 474ef54477
Fix threading error in recorder tests (#62187) 2021-12-17 13:17:48 +01:00
Erik Montnemery bceeaec2f8
Remove duplicated statistics rows (#61146)
* Remove duplicated statistics

* Fix misleading docstring

* Pylint knows best

* Correct test

* Oops

* Prevent insertion of duplicated statistics

* Tweak

* pylint

* Add models_schema_23.py

* Tweak
2021-12-13 14:15:36 +01:00
Erik Montnemery dc5888ab4a
Correct recorder.statistics.get_last_statistics (#61421) 2021-12-10 10:09:29 -08:00
Stefan Agner a13ae85982
Introduce only_supervisor for @websocket_api.ws_require_user() (#61298) 2021-12-08 16:49:35 -08:00
Erik Montnemery f30eb05870
Refactor recorder queue handling (#61161)
* Refactor recorder queue handling

* Address pylint's concerns

* Implement workaround for mypy bug

* Address review comments
2021-12-08 16:54:26 +01:00
Stefan Agner f0006b92be
Allow to lock SQLite database during backup (#60874)
* Allow to set CONF_DB_URL

This is useful for test which need a custom DB path.

* Introduce write_lock_db helper to lock SQLite database

* Introduce Websocket API which allows to lock database during backup

* Fix isort

* Avoid mutable default arguments

* Address pylint issues

* Avoid holding executor thread

* Set unlock event in case timeout occures

This makes sure the database is left unlocked even in case of a race
condition.

* Add more unit tests

* Address new pylint errors

* Lower timeout to speedup tests

* Introduce queue overflow test

* Unlock database if necessary

This makes sure that the test runs through in case locking actually
succeeds (and the test fails).

* Make DB_LOCK_TIMEOUT a global

There is no good reason for this to be an argument. The recorder needs
to pick a sensible value.

* Add Websocket Timeout test

* Test lock_database() return

* Update homeassistant/components/recorder/__init__.py

Co-authored-by: Erik Montnemery <erik@montnemery.com>

* Fix format

Co-authored-by: J. Nick Koston <nick@koston.org>
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2021-12-07 13:16:24 +01:00
Erik Montnemery bcd9f3c05f
Correct end time for monthly statistics summary (#59551)
* Correct end time for monthly statistics summary

* Add tests
2021-11-12 15:01:40 +01:00
Erik Montnemery e5ee38d3b0
Adjust recorder tests according to #59087 (#59143) 2021-11-05 10:49:58 +01:00
Erik Montnemery 470b01e4ce
Move async_migration_in_progress (#59087) 2021-11-05 10:40:56 +01:00
Erik Montnemery 185f7beafc
Improve recorder migration tests (#59075) 2021-11-04 20:21:38 -07:00
Erik Montnemery 4c5aca93df
Add recorder status WS API (#58989)
* Add recorder status WS API

* Rename recorder/status to recorder/info

* Silence pylint

* Improve tests

* Address review comments

* Tweak

* Try to fix tests

* Try to debug flaky tests

* Try to fix tests

* Revert changes to async_migration_in_progress

* Try to fix tests

* Remove debug prints

* Apply suggestions from code review
2021-11-04 16:46:45 +01:00
Erik Montnemery be4e9f91b6
Change minimum supported SQLite version to 3.31.0 (#59073) 2021-11-04 16:34:35 +01:00
Erik Montnemery dfa50a842a
Simplify recorder PgSQL version checks (#58533) 2021-10-27 16:05:40 +02:00
Erik Montnemery ac5e32d648
Corrections for external statistics (#58469) 2021-10-26 14:05:45 +02:00
Erik Montnemery e9ba5f3b4b
Warn when recorder connects to an unsupported database (#58161) 2021-10-26 13:41:59 +02:00
Erik Montnemery f594bc353b
Add support for external statistics (#56607)
* Support external statistics

* Update tests

* Lint

* Adjust code after rebase

* Separate external statistic_id with :, add name to metadata

* Adjust tests

* Simplify get_metadata_with_session

* Address review comments

* Allow updating external statistics

* Validate input

* Adjust tests after rebase

* Pylint

* Adjust tests

* Improve test coverage
2021-10-26 10:26:50 +02:00
Erik Montnemery b301ab25a3
Purge short term statistics (#58028)
* Purge short term statistics

* Less meep

* Add tests
2021-10-20 16:00:59 +02:00
Erik Montnemery 8ef8838801
Correct detection of row_number support for MariaDB (#57663) 2021-10-14 11:19:39 -07:00
Erik Montnemery 0139bfa749
Detect if mysql and sqlite support row_number (#57475) 2021-10-11 21:17:18 -07:00
Paulus Schoutsen a4d9019ffc
Refactor persistent notification to no longer route all data via a service (#57157)
* Convert persistent notification tests to async

* Create/dismiss persistent notifications in exposed functions, not service calls

* Fix notify persistent_notification

* Remove setting up persistent_notification

* Drop more setups

* Empty methods

* Undeprecate sync methods because too big task

* Fix setup clearing notifications

* Fix a bunch of tests

* Fix more tests

* Uno mas

* Test persistent notification events

* Clean up stale comment

Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-10-07 12:58:00 +02:00
Erik Montnemery 69875cbd11
Improve sensor statistics validation (#56892) 2021-10-04 09:47:44 -07:00
Erik Montnemery 8567aa9e13
Evict purged states from recorder's old_state cache (#56877)
Co-authored-by: J. Nick Koston <nick@koston.org>
2021-10-04 08:21:40 -07:00
Erik Montnemery 40ecf22bac
Remove automatic splitting of net meters from statistics (#56772) 2021-09-29 17:20:17 +02:00
Erik Montnemery 00651a4055
Optimize _get_states_with_session (#56734)
* Optimize _get_states_with_session

* Move custom filters to derived table

* Remove useless derived table

* Filter old states after grouping

* Split query

* Add comments

* Simplify state update period criteria

* Only apply custom filters if we didn't get an include list of entities

Co-authored-by: J. Nick Koston <nick@koston.org>
2021-09-29 17:08:27 +02:00
Erik Montnemery 8ef123259e
Add WS API for updating unit_of_measurement in statistics metadata (#56184)
* Add WS API for updating statistics metadata

* Update homeassistant/components/recorder/websocket_api.py

Co-authored-by: Bram Kragten <mail@bramkragten.nl>

* Update homeassistant/components/recorder/websocket_api.py

Co-authored-by: Paulus Schoutsen <balloob@gmail.com>

* Fix typo

Co-authored-by: Bram Kragten <mail@bramkragten.nl>
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2021-09-28 00:43:29 +02:00
Erik Montnemery 5976f898da
Add WS API for removing statistics for a list of statistic_ids (#55078)
* Add WS API for removing statistics for a list of statistic_ids

* Refactor according to code review, enable foreign keys support for sqlite

* Adjust tests

* Move clear_statistics WS API to recorder

* Adjust tests after rebase

* Update docstring

* Update homeassistant/components/recorder/websocket_api.py

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>

* Adjust tests after rebase

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2021-09-27 23:30:13 +02:00
Erik Montnemery 64393b462d
Add migration for 5-minute statistics (#56585)
* Add migration for 5-minute statistics

* Tweaks
2021-09-24 09:19:22 +02:00
Erik Montnemery 92253f5192
Minor refactoring of periodic statistics (#56492) 2021-09-22 13:31:33 -07:00
Erik Montnemery 6f36419c6f
Improve statistics validation (#56457) 2021-09-20 17:54:25 +02:00
Erik Montnemery 8c5efafdd8
Add 5-minute statistics for sensors (#56006)
* Add 5-minute statistics for sensors

* Address pylint issues

* Black

* Apply suggestion from code review

* Apply suggestions from code review

* Improve tests
2021-09-16 10:57:15 +02:00
Erik Montnemery d899d15a1e
Add statistics validation (#56020)
* Add statistics validation

* Remove redundant None-check

* Move validate_statistics WS API to recorder

* Apply suggestion from code review
2021-09-13 13:44:22 +02:00
Erik Montnemery d2a9f7904a
Include end time of statistics data points in API response (#56063)
* Include end time of statistics data points in API response

* Correct typing

* Update tests
2021-09-13 10:02:24 +02:00
Erik Montnemery 80fd330479
Add sum_decrease and sum_increase statistics (#55850) 2021-09-08 23:35:53 -07:00
Erik Montnemery 27764e9985
Fix handling of imperial units in long term statistics (#55959) 2021-09-08 08:08:48 -07:00
Erik Montnemery 22e6ddf8df
Do not let one bad statistic spoil the bunch (#55942) 2021-09-08 07:55:40 -07:00
Erik Montnemery 8877f37da0
Fix statistics for sensors setting last_reset (#55136)
* Re-add state_class total to sensor

* Make energy cost sensor enforce state_class total_increasing

* Drop state_class total

* Only report energy sensor issues once
2021-08-24 17:02:34 +02:00
Erik Montnemery 32e297f4a0
Compile missing statistics (#54690) 2021-08-19 22:10:45 -07:00
Erik Montnemery e1926caeb9
Remove STATE_CLASS_TOTAL and last_reset from sensor (#54755)
* Remove STATE_CLASS_TOTAL

* Update mill sensor

* Update tests

* Kill last_reset

* Return ATTR_LAST_RESET to utility_meter

* Update energy cost sensor

* Restore last_reset for backwards compatibility

* Re-add and update deprecation warning

* Update tests

* Fix utility_meter

* Update EnergyCostSensor

* Tweak

* Fix rebase mistake

* Fix test
2021-08-18 10:03:27 +02:00
Paulus Schoutsen 557cc792e9
Fix SQLAlchemy test warnings (#54116) 2021-08-08 20:33:47 -07:00
Erik Montnemery e541bcd54d
Update statistics meta data on entity_id change (#52755) 2021-07-14 13:23:11 +02:00
Erik Montnemery 11fd9d9525
Avoid duplicated database queries when fetching statistics (#52433) 2021-07-02 15:40:54 +02:00
Erik Montnemery 0476c7f9ee
Normalize temperature statistics to °C (#52297)
* Normalize temperature statistics to °C

* Fix tests

* Support temperature conversion to and from K, improve tests

* Fix test

* Add tests, pylint
2021-06-30 14:17:58 +02:00
PeteBa 42c4317628
Avoid drift in recorder purge cut-off (#52135) 2021-06-25 11:29:38 -10:00
Paulus Schoutsen 805ef3f90b
Allow fetching multiple statistics (#51996) 2021-06-18 21:03:13 +02:00
Erik Montnemery 39c94e8daa
Fix flaky statistics tests (#51214)
* Fix flaky statistics tests

* Tweak
2021-05-28 22:37:17 +02:00
Erik Montnemery b339d73109
Weight sensor average statistics by state durations (#51150)
* Weight sensor average statistics by state durations

* Fix test
2021-05-28 13:16:52 +02:00
PeteBa aa9b99713c
Add purge_entities service call to recorder (#48069) 2021-05-22 10:30:05 -05:00
Erik Montnemery e16a8063a5
Compile statistics for energy sensors (#50829)
* Compile statistics for energy sensors

* Update tests

* Rename abs_value to state

* Tweak

* Recreate statistics table

* Pylint

* Try to fix test

* Fix statistics for multiple energy sensors

* Fix energy statistics when last_reset is not set
2021-05-20 13:05:15 +02:00
J. Nick Koston e7f7e61e88
Ensure a wal checkpoint is scheduled nightly (#50746) 2021-05-17 16:27:51 -05:00
J. Nick Koston 72288710ca
Increase the sqlite cache size from ~2MiB to 8MiB (#50747) 2021-05-17 19:42:12 +02:00
Erik Montnemery 89dd3292ba
Initial draft of statistics (#49852) 2021-05-16 10:23:37 -07:00
Erik Montnemery 973f59e423
Refactor history component (#50287)
* Refactor history component

* Update tests

* Address review comments

* Correct deprecated functions
2021-05-11 09:21:57 +02:00
J. Nick Koston bf2d40adfe
Migrate from pytz to python-dateutil (#49643)
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2021-05-07 22:46:26 -07:00
J. Nick Koston 1c8d9ca68b
Check exception causes for matching strings during recorder migration (#49999) 2021-05-02 20:57:42 -07:00
Paulus Schoutsen 1bd9826684
Handle different entity_id formats (#49969) 2021-05-01 20:30:28 -07:00
J. Nick Koston b27e9e376d
Use StaticPool for recorder and NullPool for all other threads with sqlite3 (#49693) 2021-04-26 19:20:31 -10:00
jan iversen c351098f04
HomeAssistantType —> HomeAssistant for Integrations p* - s* (#49558) 2021-04-22 19:58:02 +02:00
J. Nick Koston c10836fcee
Upgrade to sqlalchemy 1.4.11 (#49538) 2021-04-21 20:29:36 -10:00
J. Nick Koston 6d137d2316
Increase recorder test coverage (#49362)
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-04-19 08:22:38 -07:00
J. Nick Koston e24f5831a2
Force recorder shutdown at final write event (#49145)
* Force recorder shutdown at EVENT_HOMEASSISTANT_FINAL_WRITE

* remove unreachable

* remove unreachable

* simplify

* cancel in async
2021-04-19 11:24:17 +02:00
J. Nick Koston 53853f035d
Prevent calling stop or restart services during db upgrade (#49098) 2021-04-12 17:18:38 -07:00
J. Nick Koston 9368891b1b
Live db migrations and recovery (#49036)
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2021-04-11 20:43:54 -10:00
Laszlo Magyar c7e4857d2c
Let recorder deal with event names longer than 32 chars (#47748) 2021-04-08 07:08:49 -10:00
Franck Nijhof 08870690a6
Fix a collection of tests with missing asserts (#48127)
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-03-20 17:23:55 +01:00
Franck Nijhof fb849b81b5
Rewrite of not a == b occurances (#48132) 2021-03-20 01:27:04 +01:00
J. Nick Koston 28c80c1133
Ensure recorder purge tests can handle multiple purge cycle (#47956)
Since a purge can generate another purge task, we now wait for
three recorder queue completions by default.
2021-03-15 11:19:19 -07:00
J. Nick Koston 61a2460c87
Improve error reporting in recorder purge test (#47929) 2021-03-14 19:46:21 -07:00
Erik Montnemery 2178e27fb4
Fix unclean shutdown of recorder test (#47791) 2021-03-12 14:17:27 -08:00
Marc Mueller 92852b9c10
Add apply_filter attribute to recorder.purge service (#45826) 2021-03-11 17:03:30 -10:00
Marc Mueller 10848b9bdf
Recorder improvements (#47739) 2021-03-11 07:52:07 -10:00
J. Nick Koston a060acc2b1
Fix recorder with MSSQL (#46678)
Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
2021-03-09 11:16:19 -10:00
adrian-vlad 44293a3738
Add enable and disable services for recorder (#45778) 2021-02-24 07:26:05 -06:00
J. Nick Koston e2fd255a96
Cleanup recorder tests (#46836) 2021-02-21 09:52:41 +01:00
J. Nick Koston 9b69549f73
Recover and restart the recorder if the sqlite database encounters corruption while running (#46612) 2021-02-19 21:26:24 -10:00
J. Nick Koston 22dbac259b
Ensure recorder shuts down cleanly on restart before startup is finished (#46604) 2021-02-19 22:18:21 -08:00
J. Nick Koston dc26fd5149
Ensure creating an index that already exists is forgiving for postgresql (#46185)
Unlikely sqlite and mysql, postgresql throws ProgrammingError instead
of InternalError or OperationalError when trying to create an index
that already exists.
2021-02-08 23:22:38 +01:00
Franck Nijhof 65cf2fcb6f
Drop asynctest (#44746) 2021-01-01 22:31:56 +01:00
Allen Porter f1dff973dc
Fix broken test test_auto_purge in recorder (#44687)
* Fix failing test due to edge-of-2021 bug

* Rewrite test_auto_purge to make the intent more clear
2020-12-31 12:04:12 -10:00
J. Nick Koston 2cbb93be43
Always keep the current recorder run when purging (#43733) 2020-11-30 09:13:50 +01:00
J. Nick Koston 4ab02cb9bc
Ensure recorder commit can retry after encountering invalid data (#41426) 2020-10-08 09:15:25 +02:00
J. Nick Koston 113d738fa2
Reduce orm overhead by grouping object expiration (#41394) 2020-10-07 15:35:48 +02:00
Ariana Hlavaty 1e9e40bf71
Rewrite recorder unittest tests to pytest style test function (#41264) 2020-10-06 21:24:13 +02:00
J. Nick Koston 20a136e2a1
Avoid event data serialization during recorder that we throw away (#41217)
We currently serialize the event data for state change events
and then replace it because we save the state in the states table.
Since the old state and new state are both contains in the event
the cost of serializing the data has a noticable impact when there
are many state changed events.
2020-10-05 15:08:47 +02:00
J. Nick Koston d9ba32dc3f
Setup recorder model relationships to avoid calling flush (#40467) 2020-09-30 13:11:43 +02:00
J. Nick Koston 557684c3ce
Add ability to disable the sqlite3 quick_check (#39479) 2020-09-02 10:12:56 +02:00
J. Nick Koston 56ddbd87b2
Fix recorder test intermittently failing (#39468) 2020-08-30 13:58:00 -05:00
J. Nick Koston 45401d4308
Fix flapping recorder last run test (#39134) 2020-08-22 09:35:18 +02:00
J. Nick Koston 7c346c2f7c
Skip the sqlite quick_check on clean restarts (#38972) 2020-08-21 14:20:46 +02:00
J. Nick Koston 7878d97588
Use SimpleQueue for recorder (#38967)
Now that python 3.7 is the minimum supported version, we can
use the more efficient SimpleQueue in the recorder as it does
not have to use threading.Lock
2020-08-21 13:54:13 +02:00
Bas Nijholt 0427d87ba4
Bump codespell from v1.16.0 to v1.17.1 and fix new spelling errors (#38663) 2020-08-08 14:41:02 +02:00
J. Nick Koston 60009ec2f9
Use event loop scheduling for tracking time patterns (#38021)
* Use event loop scheduling for tracking time patterns

* make patching of time targetable

* patch time tests since time can tick to match during the test

* fix more tests

* time can only move forward

* time can only move forward

* back to 100% coverage

* simplify since the event loop time cannot move backwards

* simplify some more

* revert simplify

* Revert "revert simplify"

This reverts commit bd42f232f6.

* Revert "simplify some more"

This reverts commit 2a6c57d514.

* Revert "simplify since the event loop time cannot move backwards"

This reverts commit 3b13714ef4.

* Attempt another simplify

* time does not move backwards in the last two

* remove next_time <= now check

* fix previous merge error
2020-07-20 20:18:31 -10:00
J. Nick Koston 1acdb28cdd
Automatically recover when the sqlite3 database is malformed or corrupted (#37949)
* Validate sqlite database on startup and move away if corruption is detected.

* do not switch context in test -- its all sync
2020-07-17 19:07:37 -10:00
J. Nick Koston 4b2ebf5487
Ensure removed entities are not displayed in logbook (#37395) 2020-07-03 23:08:46 -07:00
Franck Nijhof 79f131066c
Ensure recorder data integrity and MySQL lock error handling (#37228) 2020-06-29 16:23:11 -07:00