Commit Graph

51 Commits (13fe837fd2d0c9a1b1cebaaeb67761273ae96395)

Author SHA1 Message Date
Paulus Schoutsen 1bd9826684
Handle different entity_id formats (#49969) 2021-05-01 20:30:28 -07:00
J. Nick Koston b27e9e376d
Use StaticPool for recorder and NullPool for all other threads with sqlite3 (#49693) 2021-04-26 19:20:31 -10:00
jan iversen c351098f04
HomeAssistantType —> HomeAssistant for Integrations p* - s* (#49558) 2021-04-22 19:58:02 +02:00
J. Nick Koston e24f5831a2
Force recorder shutdown at final write event (#49145)
* Force recorder shutdown at EVENT_HOMEASSISTANT_FINAL_WRITE

* remove unreachable

* remove unreachable

* simplify

* cancel in async
2021-04-19 11:24:17 +02:00
J. Nick Koston 9368891b1b
Live db migrations and recovery (#49036)
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2021-04-11 20:43:54 -10:00
Marc Mueller 10848b9bdf
Recorder improvements (#47739) 2021-03-11 07:52:07 -10:00
adrian-vlad 44293a3738
Add enable and disable services for recorder (#45778) 2021-02-24 07:26:05 -06:00
J. Nick Koston e2fd255a96
Cleanup recorder tests (#46836) 2021-02-21 09:52:41 +01:00
J. Nick Koston 9b69549f73
Recover and restart the recorder if the sqlite database encounters corruption while running (#46612) 2021-02-19 21:26:24 -10:00
J. Nick Koston 22dbac259b
Ensure recorder shuts down cleanly on restart before startup is finished (#46604) 2021-02-19 22:18:21 -08:00
Franck Nijhof 65cf2fcb6f
Drop asynctest (#44746) 2021-01-01 22:31:56 +01:00
Allen Porter f1dff973dc
Fix broken test test_auto_purge in recorder (#44687)
* Fix failing test due to edge-of-2021 bug

* Rewrite test_auto_purge to make the intent more clear
2020-12-31 12:04:12 -10:00
J. Nick Koston 4ab02cb9bc
Ensure recorder commit can retry after encountering invalid data (#41426) 2020-10-08 09:15:25 +02:00
Ariana Hlavaty 1e9e40bf71
Rewrite recorder unittest tests to pytest style test function (#41264) 2020-10-06 21:24:13 +02:00
J. Nick Koston d9ba32dc3f
Setup recorder model relationships to avoid calling flush (#40467) 2020-09-30 13:11:43 +02:00
J. Nick Koston 557684c3ce
Add ability to disable the sqlite3 quick_check (#39479) 2020-09-02 10:12:56 +02:00
J. Nick Koston 60009ec2f9
Use event loop scheduling for tracking time patterns (#38021)
* Use event loop scheduling for tracking time patterns

* make patching of time targetable

* patch time tests since time can tick to match during the test

* fix more tests

* time can only move forward

* time can only move forward

* back to 100% coverage

* simplify since the event loop time cannot move backwards

* simplify some more

* revert simplify

* Revert "revert simplify"

This reverts commit bd42f232f6.

* Revert "simplify some more"

This reverts commit 2a6c57d514.

* Revert "simplify since the event loop time cannot move backwards"

This reverts commit 3b13714ef4.

* Attempt another simplify

* time does not move backwards in the last two

* remove next_time <= now check

* fix previous merge error
2020-07-20 20:18:31 -10:00
J. Nick Koston 4b2ebf5487
Ensure removed entities are not displayed in logbook (#37395) 2020-07-03 23:08:46 -07:00
Franck Nijhof 7d74b74570
Fix recorder purging by batch processing purges (#37140) 2020-06-26 10:27:45 -07:00
mdegat01 6c7355785a
Add support for glob matching to entity filters (#36913)
* Added GLOB capability to entityfilter and every place that uses it. All existing tests are passing

* added tests for components affected by glob change

* fixed flake8 error

* mocking the correct listener

* mocking correct bus method in azure test

* tests passing in 3.7 and 3.8

* fixed formatting issue from rebase/conflict

* Checking against glob patterns in more performant way

* perf improvments and reverted unnecessarily adjusted tests

* added new benchmark test around filters

* no longer using get with default in entityfilter

* changed filter name and removed logbook from filter benchmark

* simplified benchmark tests from feedback

* fixed apache tests and returned include exclude schemas to normal

* fixed azure event hub tests to properly go through component logic

* fixed azure test and clean up for other tests

* renaming test files to match standard

* merged mqtt statestream test changes with base

* removed dependency on recorder filter schema from history

* fixed recorder tests after merge and a bunch of lint errors
2020-06-23 20:02:29 -05:00
J. Nick Koston 91e0395c1c
Optimize database indexes for existing queries (#37036)
Cleanup indexes as >50% of the db size was indexes,
many of them unused in any current query

Logbook search was having to filter event_types without
an index:

  Created ix_events_event_type_time_fired
  Dropped ix_events_event_type

States had a redundant keys on composite index:

  Dropped ix_states_entity_id
  Its unused since we have ix_states_entity_id_last_updated

De-duplicate storage of context in states as
its always stored in events and can be found
by joining the state on the event_id.

  Dropped ix_states_context_id
  Dropped ix_states_context_parent_id
  Dropped ix_states_context_user_id

After schema v9:

STATES............................................ 10186       40.9%
EVENTS............................................ 5502        22.1%
IX_STATES_ENTITY_ID_LAST_UPDATED.................. 2177         8.7%
IX_EVENTS_EVENT_TYPE_TIME_FIRED................... 1910         7.7%
IX_EVENTS_CONTEXT_ID.............................. 1592         6.4%
IX_EVENTS_TIME_FIRED.............................. 1383         5.6%
IX_STATES_LAST_UPDATED............................ 1079         4.3%
IX_STATES_EVENT_ID................................ 375          1.5%
IX_EVENTS_CONTEXT_PARENT_ID....................... 347          1.4%
IX_EVENTS_CONTEXT_USER_ID......................... 346          1.4%
IX_RECORDER_RUNS_START_END........................ 1            0.004%
RECORDER_RUNS..................................... 1            0.004%
SCHEMA_CHANGES.................................... 1            0.004%
SQLITE_MASTER..................................... 1            0.004%
2020-06-23 10:57:52 -07:00
J. Nick Koston 2b5e7c2611
Fix recorder stopping after unserializable state (#36937)
* Ensure unserializable states do not collapse recording

* augment test coverage

* fix wal mode being set every time
2020-06-19 12:03:06 -05:00
J. Nick Koston e7d982ee11
Improve db performance of state change events and reduce overall db size (#36883)
* Add old_state_id to states, remove old/new state data from events since it can now be found by a join

* remove state lookup on restart

* Ensure old_state is set for exisitng states
2020-06-17 22:26:41 -05:00
Paulus Schoutsen c33edbe5bb
Use TestCase.addCleanup (#36560) 2020-06-08 12:26:40 -07:00
Paulus Schoutsen 2af984917e
Use asynctest-mock in most places (#35109)
* Use asynctest-mock in most places

* Fix broken patch in pilight
2020-05-03 11:27:19 -07:00
Anders Melchiorsen 6e2cf9663a
Purge recorder database at night (#33646) 2020-04-16 15:11:36 -07:00
Franck Nijhof 906385172a
String formatting improvements for tests (#33663) 2020-04-05 00:26:08 +02:00
J. Nick Koston 3a680bf7b7
Add a commit interval setting to recorder (#32596)
* Add a commit interval setting to recorder

* Make the default every 1s instead of immediate

* See attached py-spy flamegraphs for why 1s

* This avoids disk thrashing during event storms

* Make Home Assistant significantly more responsive on busy systems

* remove debug

* Add commit forces for tests that expect commits to be immediate

* Add commit forces for tests that expect commits to be immediate

* make sure _trigger_db_commit is in the right place (all effective "wait_recording_done" calls)

* De-duplicate wait_recording_done code
2020-03-09 17:43:26 -07:00
Jonathan Østrup 150b376cf9
Add recorder vars db_max_retries and db_retry_wait (#31561)
* added recorder vars db_max_retries and db_retry_wait

* fixed test_recorder_setup_failure

I failed because it was missing the two new variables. I simply added these with default values.

* fixed syntax error in test_recorder_setup_failure

* fixed formatting error in test_init_py for recorder component

* fixed typo in test case

* Updated the way the default keys for db_,max_wait and db_retry_wait is set

Implemented based on suggestions from @springstan

* Updated config_schema call to adhere to Black

* changed conf.get to conf[dict] for var retrieval

* removed 2 blank lines
2020-02-09 17:43:09 +01:00
Ville Skyttä 5216477353
Be consistent with Home Assistant and Hass.io spelling (#30500)
* Be consistent with Home Assistant spelling

* Be consistent with Hass.io spelling
2020-01-05 14:09:17 +02:00
Bas Nijholt 3d64405896 Sort imports according to PEP8 for recorder (#29652) 2019-12-08 18:48:18 +01:00
Paulus Schoutsen 4de97abc3a Black 2019-07-31 12:25:30 -07:00
Paulus Schoutsen 848a2a95a8 Fix recorder defaults (#24399)
* Fix recorder defaults

* Address comment
2019-06-08 18:18:28 -05:00
Ville Skyttä dbd0763f83 Grammar and spelling fixes (#16065) 2018-08-19 22:29:08 +02:00
Anders Melchiorsen 2d36d4d9f3 Set event_id foreign key in recorded states (#12580) 2018-02-21 12:51:20 -08:00
PeteBa 1ffccfc91c Maintain recorder purge schedule (#10279)
* Maintain automated purge schedule

* Updates from review feedback
2017-11-03 08:28:16 -07:00
milanvo 3f9d052218 Add recorder purge service, rework purge timer (#9523)
* Add recorder purge service

* Recorder test to match purge config

* Removed purge timer, move service handler to setup, add service description file

* Tests for recorder purge service

* Recorder purge timer rework, add purge service parameter, tests

* Purge service schema change

* Service description change value range

* First cleanup

* Fix name of config
2017-10-04 14:07:42 +02:00
amigian74 775d45ae5a Exclude filter for event types (#7627)
* add exclude filter for event types to recorder component

* corrected long line (279)

* change source code structure
add test for exclude event types

* code cleanup

* change source code structure

* Update __init__.py

* Update test_init.py
2017-05-24 15:23:52 -07:00
Johann Kellerman fbd0bf77c7 [recorder] Catch more startup errors #6179 (#6192)
* [recorder] Catch more startup errors #6179

* Rebase on new recorder
2017-03-02 22:44:52 -08:00
Paulus Schoutsen 61909e873f Feature/reorg recorder (#6237)
* Re-organize recorder

* Fix history

* Fix history stats

* Fix restore state

* Lint

* Fix session reconfigure

* Move imports around

* Do not start recording till HASS started

* Lint

* Fix logbook

* Fix race condition recorder init

* Better reporting on errors
2017-02-26 14:38:06 -08:00
Johann Kellerman fdc373f27e Restore_state helper to restore entity states from the DB on startup (#4614)
* Restore states

* feedback

* Remove component move into recorder

* space

* helper

* Address my own comments

* Improve test coverage

* Add test for light restore state
2017-02-20 23:40:27 -08:00
Adam Mills be08bf0ef7 [recorder] Add tests for full schema migration (#5831)
* [recorder] Add tests for full schema migration

* Remove leftover code

* Fix duplicate creation of sqlalchemy Index object

* It's that kind of day...

* Improve models_original docstring
2017-02-09 18:17:17 -08:00
Johann Kellerman 490ef6afad WIP: [component/recorder] Refactoring & better handling of SQLAlchemy Sessions (#5607)
* Refactor recorder and Sessions

* Cover #4352

* NO_reset_on_return

* contextmanager

* coverage
2017-02-07 21:47:41 -08:00
Adam Mills 6a64e79d7b [recorder] Index events time_fired to improve logbook performance (#5633)
* Index events time_fired to improve logbook perf.

* Updated implementation to track schema versions

* Added tests for schema migration support logic

* Rename check_schema to migrate_schema
2017-02-02 22:04:14 -05:00
Johann Kellerman 1f31dfe5d3 [recorder] Include & Exclude domain fix & unit tests (#5213)
* Tests & domain fix

* incl/excl combined
2017-01-09 21:53:30 +01:00
Paulus Schoutsen e88b98f5fa Clean up tests (#4209) 2016-11-03 21:58:18 -07:00
Pascal Vizeli d5368f6f78 Async bootstrap / component init (#3991)
* Async bootstrap

* Adress comments

* Fix tests

* More fixes

* Tests fixes
2016-10-27 00:16:23 -07:00
Johann Kellerman afc527ea55 Fix tests (#3387) 2016-09-13 18:17:51 -07:00
Paulus Schoutsen 609d7ebea5 Migrate core from threads to async awesomeness (#3248)
* Add event loop to the core

* Add block_till_done to HA core object

* Fix some tests

* Linting core

* Fix statemachine tests

* Core test fixes

* fix block_till_done to wait for loop and queue to empty

* fix test_core for passing, and correct start/stop/block_till_done

* Fix remote tests

* Fix tests: block_till_done

* Fix linting

* Fix more tests

* Fix final linting

* Fix remote test

* remove unnecessary import

* reduce sleep to avoid slowing down the tests excessively

* fix remaining tests to wait for non-threadsafe operations

* Add async_ doc strings for event loop / coroutine info

* Fix command line test to block for the right timeout

* Fix py3.4.2 loop var access

* Fix SERVICE_CALL_LIMIT being in effect for other tests

* Fix lint errors

* Fix lint error with proper placement

* Fix slave start to not start a timer

* Add asyncio compatible listeners.

* Increase min Python version to 3.4.2

* Move async backports to util

* Add backported async tests

* Fix linting

* Simplify Python version check

* Fix lint

* Remove unneeded try/except and queue listener appproriately.

* Fix tuple vs. list unorderable error on version compare.

* Fix version tests
2016-09-12 19:16:14 -07:00
Johann Kellerman 4cf618334c Update recorder. (#2549)
* Update recorder.

models.py:
 - Use scoped_session in models.py to fix shutdown error
__init__.py:
 - Session _commit & retry method
 - Single session var for purge_data
 - Ensure single _INSTANCE
 - repeat purge every 2 days
 - show correct time in log_error

* _commit

* Restore models to old functionality, swap purge, remove _INSTANCE cleanup from tests, typing ignore Base class

* pylint

* Remove recorder from model unit test
2016-07-23 11:25:17 -07:00