Unlikely sqlite and mysql, postgresql throws ProgrammingError instead
of InternalError or OperationalError when trying to create an index
that already exists.
* Add index to old_state_id column for older databases
The schema was updated in #43610 but the index was not
added on migration.
* Handle postgresql missing ondelete
* create index first
By default these tables are created with utf8 which can only hold 3 bytes. This
meant that all emjoi would trigger a MySQLdb._exceptions.OperationalError because
they are 4 bytes.
This will only fix the issue for users who recreate their tables.
* MariaDB doesn't purge #42402
This addresses home-assistant#42402
Relationships within table "states" and between tables "states" and "events " home-assistant#40467 prevent the purge from working correctly. The database increases w/o any purge.
This proposal sets related indices to NULL and permits deleting of rows.
Further explanations can be found here home-assistant#42402
This proposal also allows to purge the tables "events" and "states" in any order.
* Update models.py
Corrected for Black style requirements
* Update homeassistant/components/recorder/models.py
Co-authored-by: J. Nick Koston <nick@koston.org>
* Add the options to foreign key constraints
* purge old states when database gets deleted out from under us
* pylint
Co-authored-by: J. Nick Koston <nick@koston.org>
We can avoid processing the relationship when the old state was already
written to the database which will common case with a commit interval
of 1s. Since we already know the value for old_state_id we can use it
instead of asking sqlalchemy to process the relationship at flush/commit
time which can significantly speed up sqlalchemy's _emit_insert_statements
implementation.
We currently serialize the event data for state change events
and then replace it because we save the state in the states table.
Since the old state and new state are both contains in the event
the cost of serializing the data has a noticable impact when there
are many state changed events.
Disable expire_on_commit for the event writer. Since we never expect the
old_state_id to change in the database, it was never worth the expense of
refetching the id after the commit.
Now that python 3.7 is the minimum supported version, we can
use the more efficient SimpleQueue in the recorder as it does
not have to use threading.Lock
On startup we run an sqlite3 quick_check to verify the database
integrity. In the majority of cases, the quick_check takes under
10 seconds.
On systems with very large databases and very slow disk/cpu,
this can take much longer so we freeze the timeout.
* Added GLOB capability to entityfilter and every place that uses it. All existing tests are passing
* added tests for components affected by glob change
* fixed flake8 error
* mocking the correct listener
* mocking correct bus method in azure test
* tests passing in 3.7 and 3.8
* fixed formatting issue from rebase/conflict
* Checking against glob patterns in more performant way
* perf improvments and reverted unnecessarily adjusted tests
* added new benchmark test around filters
* no longer using get with default in entityfilter
* changed filter name and removed logbook from filter benchmark
* simplified benchmark tests from feedback
* fixed apache tests and returned include exclude schemas to normal
* fixed azure event hub tests to properly go through component logic
* fixed azure test and clean up for other tests
* renaming test files to match standard
* merged mqtt statestream test changes with base
* removed dependency on recorder filter schema from history
* fixed recorder tests after merge and a bunch of lint errors
Cleanup indexes as >50% of the db size was indexes,
many of them unused in any current query
Logbook search was having to filter event_types without
an index:
Created ix_events_event_type_time_fired
Dropped ix_events_event_type
States had a redundant keys on composite index:
Dropped ix_states_entity_id
Its unused since we have ix_states_entity_id_last_updated
De-duplicate storage of context in states as
its always stored in events and can be found
by joining the state on the event_id.
Dropped ix_states_context_id
Dropped ix_states_context_parent_id
Dropped ix_states_context_user_id
After schema v9:
STATES............................................ 10186 40.9%
EVENTS............................................ 5502 22.1%
IX_STATES_ENTITY_ID_LAST_UPDATED.................. 2177 8.7%
IX_EVENTS_EVENT_TYPE_TIME_FIRED................... 1910 7.7%
IX_EVENTS_CONTEXT_ID.............................. 1592 6.4%
IX_EVENTS_TIME_FIRED.............................. 1383 5.6%
IX_STATES_LAST_UPDATED............................ 1079 4.3%
IX_STATES_EVENT_ID................................ 375 1.5%
IX_EVENTS_CONTEXT_PARENT_ID....................... 347 1.4%
IX_EVENTS_CONTEXT_USER_ID......................... 346 1.4%
IX_RECORDER_RUNS_START_END........................ 1 0.004%
RECORDER_RUNS..................................... 1 0.004%
SCHEMA_CHANGES.................................... 1 0.004%
SQLITE_MASTER..................................... 1 0.004%
* adj
* time_fired_isoformat
* remove unused code
* tests for processing timestamps
* restore missing import lost in merge conflict
* test for None case
* Add old_state_id to states, remove old/new state data from events since it can now be found by a join
* remove state lookup on restart
* Ensure old_state is set for exisitng states
* Improve history api performance
A new option "minimal_response" reduces the amount of data
sent between the first and last history states to only the
"last_changed" and "state" fields.
Calling to_native is now avoided where possible and only
done at the end for rows that will be returned in the response.
When sending the `minimal_response` option, the history
api now returns a json response similar to the following
for an entity
Testing:
History API Response time for 1 day
Average of 10 runs with minimal_response
Before: 19.89s. (content length : 3427428)
After: 8.44s (content length: 592199)
```
[{
"attributes": {--TRUNCATED--},
"context": {--TRUNCATED--},
"entity_id": "binary_sensor.powerwall_status",
"last_changed": "2020-05-18T23:20:03.213000+00:00",
"last_updated": "2020-05-18T23:20:03.213000+00:00",
"state": "on"
},
...
{
"last_changed": "2020-05-19T00:41:08Z",
"state": "unavailable"
},
...
{
"attributes": {--TRUNCATED--},
"context": {--TRUNCATED--},
"entity_id": "binary_sensor.powerwall_status",
"last_changed": "2020-05-19T00:42:08.069698+00:00",
"last_updated": "2020-05-19T00:42:08.069698+00:00",
"state": "on"
}]
```
* Remove impossible state check
* Remove another impossible state check
* Update homeassistant/components/history/__init__.py
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
* Reorder to save some indent per review
* Make query response make sense with to_native=False
* Update test for 00:00 to Z change
* Update homeassistant/components/recorder/models.py
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
The database fields are timezoned via DateTime(timezone=True), so the
default value should be timezoned too. When using cockroachdb this is
fatal and results in the recorder crashing.
* Avoid a context switch in the history api
The history api was creating a job to fetch the
states and another job to convert the states to
json. This can be done in a single job which
decreases the overhead of the operation.
* Ensure there is only one sqlalchemy session created per history
query.
Most queries created three sqlalchemy sessions which was
especially slow with sqlite since it opens and closes the
database.
In testing the UI is noticeably faster at generating history
graphs for entites.
* Add additional coverage
* pass hass first to _states_to_json and _get_significant_states
Some providers have set their wait_timeout to 60s
in order to pack as many users as they can on a machine.
The mysql default is 28800 seconds (8 hours)
Since mysql connection build and tear down is relativity
expensive, we want to avoid being disconnected.
We now accommodate this scenario with the following:
1. Raise the mysql session wait_timeout 28800 when we connect
2. The event session now does a 30 second keep alive to
ensure the connection stays open
If the database server disconnects there were exceptions
that were not trapped which would cause the recorder event
loop to collapse. As we never want the loop to end
we trap exceptions broadly.
Fix a bug in the new commit interval setting which caused
it to always commit after 1s
* Add a commit interval setting to recorder
* Make the default every 1s instead of immediate
* See attached py-spy flamegraphs for why 1s
* This avoids disk thrashing during event storms
* Make Home Assistant significantly more responsive on busy systems
* remove debug
* Add commit forces for tests that expect commits to be immediate
* Add commit forces for tests that expect commits to be immediate
* make sure _trigger_db_commit is in the right place (all effective "wait_recording_done" calls)
* De-duplicate wait_recording_done code
* [recorder] Use orjson to parse json faster
* Remove from http manifest
* Bump to orjson 2.5.1
* Empty commit to trigger CI
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
* added recorder vars db_max_retries and db_retry_wait
* fixed test_recorder_setup_failure
I failed because it was missing the two new variables. I simply added these with default values.
* fixed syntax error in test_recorder_setup_failure
* fixed formatting error in test_init_py for recorder component
* fixed typo in test case
* Updated the way the default keys for db_,max_wait and db_retry_wait is set
Implemented based on suggestions from @springstan
* Updated config_schema call to adhere to Black
* changed conf.get to conf[dict] for var retrieval
* removed 2 blank lines
* move imports to top-level in recorder init
* move imports to top-level in recorder migration
* move imports to top-level in recorder models
* move imports to top-level in recorder purge
* move imports to top-level in recorder util
* fix pylint
Franck Nijhof
J. Nick Koston
Marc Mueller
CurrentThread
adrian-vlad
Fabian Affolter
Ville Skyttä
moinmoin-sh
Matthias Alphart
Pascal Vizeli
mdegat01
Alex van den Hoogen
Odin Ugedal
Josef Schlehofer
Anders Melchiorsen
springstan
Paulus Schoutsen
J. Nick Koston
Ruslan Sayfutdinov
Jonathan Østrup
Bas Nijholt
bastshoes
Malte Franken
Franck Nijhof