* Prebake common history queries
The python overhead of to construct
the queries exceeded the database overhead. We now
prebake the queries that get frequently polled.
This reduces the time it takes to update history_stats
sensors and can make quite a difference if there
are a lot of them.
When using the mini-graph-card card, all the entities
on the card being graphed are queried every few seconds
for new states. Previously this would tie up the database if there
are lot of these graphs in the UI.
* Update homeassistant/components/history/__init__.py
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
* Update homeassistant/components/history/__init__.py
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
* cache entity_filter in the lambda
* switch to yield
* Revert "switch to yield"
This reverts commit f8386f4940.
* get_states always returns a list
* query wasnt actually reusable so revert part of the breakout
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
* Added GLOB capability to entityfilter and every place that uses it. All existing tests are passing
* added tests for components affected by glob change
* fixed flake8 error
* mocking the correct listener
* mocking correct bus method in azure test
* tests passing in 3.7 and 3.8
* fixed formatting issue from rebase/conflict
* Checking against glob patterns in more performant way
* perf improvments and reverted unnecessarily adjusted tests
* added new benchmark test around filters
* no longer using get with default in entityfilter
* changed filter name and removed logbook from filter benchmark
* simplified benchmark tests from feedback
* fixed apache tests and returned include exclude schemas to normal
* fixed azure event hub tests to properly go through component logic
* fixed azure test and clean up for other tests
* renaming test files to match standard
* merged mqtt statestream test changes with base
* removed dependency on recorder filter schema from history
* fixed recorder tests after merge and a bunch of lint errors
Cleanup indexes as >50% of the db size was indexes,
many of them unused in any current query
Logbook search was having to filter event_types without
an index:
Created ix_events_event_type_time_fired
Dropped ix_events_event_type
States had a redundant keys on composite index:
Dropped ix_states_entity_id
Its unused since we have ix_states_entity_id_last_updated
De-duplicate storage of context in states as
its always stored in events and can be found
by joining the state on the event_id.
Dropped ix_states_context_id
Dropped ix_states_context_parent_id
Dropped ix_states_context_user_id
After schema v9:
STATES............................................ 10186 40.9%
EVENTS............................................ 5502 22.1%
IX_STATES_ENTITY_ID_LAST_UPDATED.................. 2177 8.7%
IX_EVENTS_EVENT_TYPE_TIME_FIRED................... 1910 7.7%
IX_EVENTS_CONTEXT_ID.............................. 1592 6.4%
IX_EVENTS_TIME_FIRED.............................. 1383 5.6%
IX_STATES_LAST_UPDATED............................ 1079 4.3%
IX_STATES_EVENT_ID................................ 375 1.5%
IX_EVENTS_CONTEXT_PARENT_ID....................... 347 1.4%
IX_EVENTS_CONTEXT_USER_ID......................... 346 1.4%
IX_RECORDER_RUNS_START_END........................ 1 0.004%
RECORDER_RUNS..................................... 1 0.004%
SCHEMA_CHANGES.................................... 1 0.004%
SQLITE_MASTER..................................... 1 0.004%
* adj
* time_fired_isoformat
* remove unused code
* tests for processing timestamps
* restore missing import lost in merge conflict
* test for None case
This builds on #35822
Minimize the amount of data selected from
the database
Testing:
History API Response time for 1 day
Average of 10 runs with minimal_response
Before: 9.47s
After: 4.43s
* Improve history api performance
A new option "minimal_response" reduces the amount of data
sent between the first and last history states to only the
"last_changed" and "state" fields.
Calling to_native is now avoided where possible and only
done at the end for rows that will be returned in the response.
When sending the `minimal_response` option, the history
api now returns a json response similar to the following
for an entity
Testing:
History API Response time for 1 day
Average of 10 runs with minimal_response
Before: 19.89s. (content length : 3427428)
After: 8.44s (content length: 592199)
```
[{
"attributes": {--TRUNCATED--},
"context": {--TRUNCATED--},
"entity_id": "binary_sensor.powerwall_status",
"last_changed": "2020-05-18T23:20:03.213000+00:00",
"last_updated": "2020-05-18T23:20:03.213000+00:00",
"state": "on"
},
...
{
"last_changed": "2020-05-19T00:41:08Z",
"state": "unavailable"
},
...
{
"attributes": {--TRUNCATED--},
"context": {--TRUNCATED--},
"entity_id": "binary_sensor.powerwall_status",
"last_changed": "2020-05-19T00:42:08.069698+00:00",
"last_updated": "2020-05-19T00:42:08.069698+00:00",
"state": "on"
}]
```
* Remove impossible state check
* Remove another impossible state check
* Update homeassistant/components/history/__init__.py
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
* Reorder to save some indent per review
* Make query response make sense with to_native=False
* Update test for 00:00 to Z change
* Update homeassistant/components/recorder/models.py
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
* Avoid a context switch in the history api
The history api was creating a job to fetch the
states and another job to convert the states to
json. This can be done in a single job which
decreases the overhead of the operation.
* Ensure there is only one sqlalchemy session created per history
query.
Most queries created three sqlalchemy sessions which was
especially slow with sqlite since it opens and closes the
database.
In testing the UI is noticeably faster at generating history
graphs for entites.
* Add additional coverage
* pass hass first to _states_to_json and _get_significant_states
* Avoid a context switch in the history api
The history api was creating a job to fetch the
states and another job to convert the states to
json. This can be done in a single job which
decreases the overhead of the operation.
* Revert to original solution to avoid function redefine each call
* Add include_location_attributes
* Add check for different state
* Fix things
* Drop filter
* significant changes only
* Change default behavior
* Remove trailing commas
* Speed up history `get_states`
Adding a boundary of the start of the recorder run the point is in, significantly increases the time of the query. This speeds up the fetching of the history of 1 entity.
* Make single entity query easier
no need for joins with single entity
* Lint
J. Nick Koston
mdegat01
Shulyaka
Ville Skyttä
Franck Nijhof
Paulus Schoutsen
Santobert
springstan
Franck Nijhof
Bram Kragten
Thomas Lovén
cgtobi
Paulus Schoutsen
Joakim Plate
Fabian Affolter