* Include has_mean + has_sum in statistics metadata WS response
* Don't include has_mean/has_sum in history/list_statistic_ids
* Adjust tests
* Do include has_mean/has_sum in history/list_statistic_ids
* Avoid locking the database for non-SQLite backends
Currently we only have a lock implementation for SQLite. Just return
success for all other databases as they are not expected to store data
in the config directory and the caller can assume that a backup can
be safely taken.
This fixes `RuntimeError: generator didn't yield` errors when creating
a backup with the current Supervisor dev builds.
* Allow to set CONF_DB_URL
This is useful for test which need a custom DB path.
* Introduce write_lock_db helper to lock SQLite database
* Introduce Websocket API which allows to lock database during backup
* Fix isort
* Avoid mutable default arguments
* Address pylint issues
* Avoid holding executor thread
* Set unlock event in case timeout occures
This makes sure the database is left unlocked even in case of a race
condition.
* Add more unit tests
* Address new pylint errors
* Lower timeout to speedup tests
* Introduce queue overflow test
* Unlock database if necessary
This makes sure that the test runs through in case locking actually
succeeds (and the test fails).
* Make DB_LOCK_TIMEOUT a global
There is no good reason for this to be an argument. The recorder needs
to pick a sensible value.
* Add Websocket Timeout test
* Test lock_database() return
* Update homeassistant/components/recorder/__init__.py
Co-authored-by: Erik Montnemery <erik@montnemery.com>
* Fix format
Co-authored-by: J. Nick Koston <nick@koston.org>
Co-authored-by: Erik Montnemery <erik@montnemery.com>
* Convert persistent notification tests to async
* Create/dismiss persistent notifications in exposed functions, not service calls
* Fix notify persistent_notification
* Remove setting up persistent_notification
* Drop more setups
* Empty methods
* Undeprecate sync methods because too big task
* Fix setup clearing notifications
* Fix a bunch of tests
* Fix more tests
* Uno mas
* Test persistent notification events
* Clean up stale comment
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Optimize _get_states_with_session
* Move custom filters to derived table
* Remove useless derived table
* Filter old states after grouping
* Split query
* Add comments
* Simplify state update period criteria
* Only apply custom filters if we didn't get an include list of entities
Co-authored-by: J. Nick Koston <nick@koston.org>
* Add WS API for removing statistics for a list of statistic_ids
* Refactor according to code review, enable foreign keys support for sqlite
* Adjust tests
* Move clear_statistics WS API to recorder
* Adjust tests after rebase
* Update docstring
* Update homeassistant/components/recorder/websocket_api.py
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
* Adjust tests after rebase
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
* Re-add state_class total to sensor
* Make energy cost sensor enforce state_class total_increasing
* Drop state_class total
* Only report energy sensor issues once
* Compile statistics for energy sensors
* Update tests
* Rename abs_value to state
* Tweak
* Recreate statistics table
* Pylint
* Try to fix test
* Fix statistics for multiple energy sensors
* Fix energy statistics when last_reset is not set
Unlikely sqlite and mysql, postgresql throws ProgrammingError instead
of InternalError or OperationalError when trying to create an index
that already exists.
We currently serialize the event data for state change events
and then replace it because we save the state in the states table.
Since the old state and new state are both contains in the event
the cost of serializing the data has a noticable impact when there
are many state changed events.
Now that python 3.7 is the minimum supported version, we can
use the more efficient SimpleQueue in the recorder as it does
not have to use threading.Lock
* Use event loop scheduling for tracking time patterns
* make patching of time targetable
* patch time tests since time can tick to match during the test
* fix more tests
* time can only move forward
* time can only move forward
* back to 100% coverage
* simplify since the event loop time cannot move backwards
* simplify some more
* revert simplify
* Revert "revert simplify"
This reverts commit bd42f232f6.
* Revert "simplify some more"
This reverts commit 2a6c57d514.
* Revert "simplify since the event loop time cannot move backwards"
This reverts commit 3b13714ef4.
* Attempt another simplify
* time does not move backwards in the last two
* remove next_time <= now check
* fix previous merge error
* Added GLOB capability to entityfilter and every place that uses it. All existing tests are passing
* added tests for components affected by glob change
* fixed flake8 error
* mocking the correct listener
* mocking correct bus method in azure test
* tests passing in 3.7 and 3.8
* fixed formatting issue from rebase/conflict
* Checking against glob patterns in more performant way
* perf improvments and reverted unnecessarily adjusted tests
* added new benchmark test around filters
* no longer using get with default in entityfilter
* changed filter name and removed logbook from filter benchmark
* simplified benchmark tests from feedback
* fixed apache tests and returned include exclude schemas to normal
* fixed azure event hub tests to properly go through component logic
* fixed azure test and clean up for other tests
* renaming test files to match standard
* merged mqtt statestream test changes with base
* removed dependency on recorder filter schema from history
* fixed recorder tests after merge and a bunch of lint errors
Cleanup indexes as >50% of the db size was indexes,
many of them unused in any current query
Logbook search was having to filter event_types without
an index:
Created ix_events_event_type_time_fired
Dropped ix_events_event_type
States had a redundant keys on composite index:
Dropped ix_states_entity_id
Its unused since we have ix_states_entity_id_last_updated
De-duplicate storage of context in states as
its always stored in events and can be found
by joining the state on the event_id.
Dropped ix_states_context_id
Dropped ix_states_context_parent_id
Dropped ix_states_context_user_id
After schema v9:
STATES............................................ 10186 40.9%
EVENTS............................................ 5502 22.1%
IX_STATES_ENTITY_ID_LAST_UPDATED.................. 2177 8.7%
IX_EVENTS_EVENT_TYPE_TIME_FIRED................... 1910 7.7%
IX_EVENTS_CONTEXT_ID.............................. 1592 6.4%
IX_EVENTS_TIME_FIRED.............................. 1383 5.6%
IX_STATES_LAST_UPDATED............................ 1079 4.3%
IX_STATES_EVENT_ID................................ 375 1.5%
IX_EVENTS_CONTEXT_PARENT_ID....................... 347 1.4%
IX_EVENTS_CONTEXT_USER_ID......................... 346 1.4%
IX_RECORDER_RUNS_START_END........................ 1 0.004%
RECORDER_RUNS..................................... 1 0.004%
SCHEMA_CHANGES.................................... 1 0.004%
SQLITE_MASTER..................................... 1 0.004%
* adj
* time_fired_isoformat
* remove unused code
* tests for processing timestamps
* restore missing import lost in merge conflict
* test for None case
* Add old_state_id to states, remove old/new state data from events since it can now be found by a join
* remove state lookup on restart
* Ensure old_state is set for exisitng states
* Add a commit interval setting to recorder
* Make the default every 1s instead of immediate
* See attached py-spy flamegraphs for why 1s
* This avoids disk thrashing during event storms
* Make Home Assistant significantly more responsive on busy systems
* remove debug
* Add commit forces for tests that expect commits to be immediate
* Add commit forces for tests that expect commits to be immediate
* make sure _trigger_db_commit is in the right place (all effective "wait_recording_done" calls)
* De-duplicate wait_recording_done code
* added recorder vars db_max_retries and db_retry_wait
* fixed test_recorder_setup_failure
I failed because it was missing the two new variables. I simply added these with default values.
* fixed syntax error in test_recorder_setup_failure
* fixed formatting error in test_init_py for recorder component
* fixed typo in test case
* Updated the way the default keys for db_,max_wait and db_retry_wait is set
Implemented based on suggestions from @springstan
* Updated config_schema call to adhere to Black
* changed conf.get to conf[dict] for var retrieval
* removed 2 blank lines
* move imports to top-level in recorder init
* move imports to top-level in recorder migration
* move imports to top-level in recorder models
* move imports to top-level in recorder purge
* move imports to top-level in recorder util
* fix pylint
* Restore states through a JSON store
* Accept entity_id directly in restore state helper
* Keep states stored between runs for a limited time
* Remove warning
Erik Montnemery
J. Nick Koston
Joakim Sørensen
Pawel
Marc Mueller
Stefan Agner
Paulus Schoutsen
PeteBa
jan iversen
Laszlo Magyar
Franck Nijhof
adrian-vlad
Allen Porter
Ariana Hlavaty
Bas Nijholt
mdegat01
Anders Melchiorsen
J. Nick Koston
Jonathan Østrup
Ville Skyttä
bastshoes
Malte Franken
Paulus Schoutsen
Adam Mills
Paulus Schoutsen