Commit Graph

63 Commits (39aaa383b372012cf16a9699fdd721498822bffd)

Author SHA1 Message Date
Erik Montnemery 8c5efafdd8
Add 5-minute statistics for sensors (#56006)
* Add 5-minute statistics for sensors

* Address pylint issues

* Black

* Apply suggestion from code review

* Apply suggestions from code review

* Improve tests
2021-09-16 10:57:15 +02:00
Jaroslav Hanslík a4a6bf8a85
New icon names based on MDI 6.1.95 (#56085) 2021-09-11 12:34:31 -07:00
Joakim Sørensen 99a62799ae
Allow non-admin users to call history/list_statistic_ids (#54698) 2021-08-16 16:07:11 +02:00
Paulus Schoutsen 557cc792e9
Fix SQLAlchemy test warnings (#54116) 2021-08-08 20:33:47 -07:00
Bram Kragten fcc6ea7497
Add energy integration (#52001)
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
Co-authored-by: Erik <erik@montnemery.com>
2021-07-26 09:37:37 -07:00
Erik Montnemery 0ab999738b
Add statistics meta data table (#52331)
* Add statistics meta data table

* Tweak meta data generation
2021-06-30 13:32:17 +02:00
Erik Montnemery 0ca199d8d0
Add WS API for listing available statistic ids (#51984)
* Add WS API for listing available statistic ids

* Update homeassistant/components/history/__init__.py

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
Co-authored-by: Bram Kragten <mail@bramkragten.nl>
2021-06-18 21:32:30 +02:00
Paulus Schoutsen 805ef3f90b
Allow fetching multiple statistics (#51996) 2021-06-18 21:03:13 +02:00
Erik Montnemery 837220cce4
Add deprecated backwards compatible history.LazyState (#51144) 2021-05-28 11:01:28 +02:00
Bram Kragten 26563e3ea4
Add statistics websocket endpoint (#51044)
Co-authored-by: Erik <erik@montnemery.com>
2021-05-25 08:03:37 -07:00
Erik Montnemery 89dd3292ba
Initial draft of statistics (#49852) 2021-05-16 10:23:37 -07:00
Erik Montnemery 973f59e423
Refactor history component (#50287)
* Refactor history component

* Update tests

* Address review comments

* Correct deprecated functions
2021-05-11 09:21:57 +02:00
Marc Mueller c07646db5d
Update typing syntax (#49480)
* Update typing syntax

* Replace typing imports with ones from collections where possible

* Changes after review
2021-04-20 17:40:41 +02:00
Justin Paupore c4f9489d61
Fix infinite recursion in LazyState (#48719)
If LazyState cannot parse the attributes of its row as JSON, it prints
a message to the logger. Unfortunately, it passes `self` as a format
argument to that message, which causes its `__repr__` method to be
called, which then tries to retrieve `self.attributes` in order to
display them. This leads to an infinite recursion and a crash of the
entire core.

To fix, send the database row to be printed in the log message, rather
than the LazyState object that wraps around it.
2021-04-06 11:39:54 -07:00
Franck Nijhof c1d5638739
Remove HomeAssistantType alias from entity components - Part 2 (#48468) 2021-03-29 13:24:56 -10:00
Marc Mueller 5cdd945f44
Update typing 08 (#48058) 2021-03-18 09:25:40 +01:00
Philip Allgaier 470b0b8b87
Fix historic attributes for input_datetime (#45208) 2021-03-16 22:40:20 -07:00
J. Nick Koston 0da4034179
Ensure history LazyState state value is always a string (#45644)
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2021-01-28 16:05:02 +01:00
J. Nick Koston 6822190772
Fix unmocked https in the test suite (#42316) 2020-10-25 07:16:23 -05:00
J. Nick Koston 9649525fe6
Increase coverage and optimize history skip_initial_state (#42014)
Avoids a database query when all states have not
changed in the time window when skip_initial_state
is set.
2020-10-19 09:53:47 +02:00
J. Nick Koston aada6a1d88
Fix logbook with empty filter and remove unused code (#40565) 2020-09-30 13:10:11 +02:00
J. Nick Koston a19e10c57a
Add entity glob matching support to history (#40387) 2020-09-28 15:58:55 +02:00
Franck Nijhof 1c2ebdf307
Upgrade black to 20.8b1 (#39287) 2020-08-27 13:56:20 +02:00
J. Nick Koston f8d547f2d3
Adjust history as all scripts can now be canceled (#37820) 2020-07-14 22:31:34 +02:00
J. Nick Koston b64ae55c66
Prebake common history queries (#37496)
* Prebake common history queries

The python overhead of to construct
the queries exceeded the database overhead.  We now
prebake the queries that get frequently polled.

This reduces the time it takes to update history_stats
sensors and can make quite a difference if there
are a lot of them.

When using the mini-graph-card card, all the entities
on the card being graphed are queried every few seconds
for new states. Previously this would tie up the database if there
are lot of these graphs in the UI.

* Update homeassistant/components/history/__init__.py

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>

* Update homeassistant/components/history/__init__.py

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>

* cache entity_filter in the lambda

* switch to yield

* Revert "switch to yield"

This reverts commit f8386f4940.

* get_states always returns a list

* query wasnt actually reusable so revert part of the breakout

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2020-07-05 11:03:23 -05:00
J. Nick Koston 045cdee30c
Avoid selecting the states created column for history (#37450)
We never use the data for history.  We should
not select it as it created unneeded overhead.
2020-07-03 23:04:11 -07:00
J. Nick Koston a87c29b5d9
Ensure logbook performs well when filtering is configured (#37292) 2020-07-02 09:12:27 -07:00
J. Nick Koston 30df871787
Improve isoformat timestamp performance for full states (#37105) 2020-06-25 17:11:49 -07:00
J. Nick Koston f4528d0db2
Ensure history states can be copied (#37081)
The filter integration makes a copy of a
state object obtained from history.
2020-06-24 22:43:08 -05:00
mdegat01 6c7355785a
Add support for glob matching to entity filters (#36913)
* Added GLOB capability to entityfilter and every place that uses it. All existing tests are passing

* added tests for components affected by glob change

* fixed flake8 error

* mocking the correct listener

* mocking correct bus method in azure test

* tests passing in 3.7 and 3.8

* fixed formatting issue from rebase/conflict

* Checking against glob patterns in more performant way

* perf improvments and reverted unnecessarily adjusted tests

* added new benchmark test around filters

* no longer using get with default in entityfilter

* changed filter name and removed logbook from filter benchmark

* simplified benchmark tests from feedback

* fixed apache tests and returned include exclude schemas to normal

* fixed azure event hub tests to properly go through component logic

* fixed azure test and clean up for other tests

* renaming test files to match standard

* merged mqtt statestream test changes with base

* removed dependency on recorder filter schema from history

* fixed recorder tests after merge and a bunch of lint errors
2020-06-23 20:02:29 -05:00
J. Nick Koston 5b79c1f9ef
Remove support for deprecated hidden attribute from logbook and history (#37039) 2020-06-23 13:40:39 -07:00
J. Nick Koston 91e0395c1c
Optimize database indexes for existing queries (#37036)
Cleanup indexes as >50% of the db size was indexes,
many of them unused in any current query

Logbook search was having to filter event_types without
an index:

  Created ix_events_event_type_time_fired
  Dropped ix_events_event_type

States had a redundant keys on composite index:

  Dropped ix_states_entity_id
  Its unused since we have ix_states_entity_id_last_updated

De-duplicate storage of context in states as
its always stored in events and can be found
by joining the state on the event_id.

  Dropped ix_states_context_id
  Dropped ix_states_context_parent_id
  Dropped ix_states_context_user_id

After schema v9:

STATES............................................ 10186       40.9%
EVENTS............................................ 5502        22.1%
IX_STATES_ENTITY_ID_LAST_UPDATED.................. 2177         8.7%
IX_EVENTS_EVENT_TYPE_TIME_FIRED................... 1910         7.7%
IX_EVENTS_CONTEXT_ID.............................. 1592         6.4%
IX_EVENTS_TIME_FIRED.............................. 1383         5.6%
IX_STATES_LAST_UPDATED............................ 1079         4.3%
IX_STATES_EVENT_ID................................ 375          1.5%
IX_EVENTS_CONTEXT_PARENT_ID....................... 347          1.4%
IX_EVENTS_CONTEXT_USER_ID......................... 346          1.4%
IX_RECORDER_RUNS_START_END........................ 1            0.004%
RECORDER_RUNS..................................... 1            0.004%
SCHEMA_CHANGES.................................... 1            0.004%
SQLITE_MASTER..................................... 1            0.004%
2020-06-23 10:57:52 -07:00
Shulyaka c28493098a
Add new humidifier entity integration (#28693)
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2020-06-22 17:59:16 -07:00
J. Nick Koston 53a91ece4e
Improve isoformat timestamp performance (#36991)
* adj

* time_fired_isoformat

* remove unused code

* tests for processing timestamps

* restore missing import lost in merge conflict

* test for None case
2020-06-22 12:06:02 -05:00
J. Nick Koston e19c97af7c
Fix history api with no constraints (#36979) 2020-06-22 10:32:46 -05:00
J. Nick Koston e25f216fd6
Reduce state conversion overhead in history by lazy loading properties (#36963) 2020-06-22 16:19:33 +02:00
J. Nick Koston edad387b12
Make recorder execute avoid native conversion by default (#36938) 2020-06-21 21:58:57 -07:00
J. Nick Koston 83e3f680bf
Improve history api performance part 4 (#36783)
This builds on #35822

Minimize the amount of data selected from
the database

Testing:

History API Response time for 1 day
Average of 10 runs with minimal_response

Before: 9.47s
After: 4.43s
2020-06-15 11:02:36 -07:00
J. Nick Koston 9cc20fc6b8
Fix history graphs with mysql/mariadb (#36769) 2020-06-13 21:22:16 -07:00
J. Nick Koston 65e9e4a2f3
Fix history timezone with postgres and other dbs (#36746) 2020-06-13 12:32:01 +02:00
J. Nick Koston 0a6deeb49b
Improve history api performance (#35822)
* Improve history api performance

A new option "minimal_response" reduces the amount of data
sent between the first and last history states to only the
"last_changed" and "state" fields.

Calling to_native is now avoided where possible and only
done at the end for rows that will be returned in the response.

When sending the `minimal_response` option, the history
api now returns a json response similar to the following
for an entity

Testing:

History API Response time for 1 day
Average of 10 runs with minimal_response

Before: 19.89s. (content length : 3427428)
After: 8.44s (content length: 592199)

```
[{
	"attributes": {--TRUNCATED--},
	"context": {--TRUNCATED--},
	"entity_id": "binary_sensor.powerwall_status",
	"last_changed": "2020-05-18T23:20:03.213000+00:00",
	"last_updated": "2020-05-18T23:20:03.213000+00:00",
	"state": "on"
},
...
{
	"last_changed": "2020-05-19T00:41:08Z",
	"state": "unavailable"
},
...
{
	"attributes": {--TRUNCATED--},
	"context": {--TRUNCATED--},
	"entity_id": "binary_sensor.powerwall_status",
	"last_changed": "2020-05-19T00:42:08.069698+00:00",
	"last_updated": "2020-05-19T00:42:08.069698+00:00",
	"state": "on"
}]
```

* Remove impossible state check

* Remove another impossible state check

* Update homeassistant/components/history/__init__.py

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>

* Reorder to save some indent per review

* Make query response make sense with to_native=False

* Update test for 00:00 to Z change

* Update homeassistant/components/recorder/models.py

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2020-05-26 21:53:56 -05:00
Ville Skyttä f8416484f8
More data entry flow and HTTP related type hints (#34430) 2020-05-26 09:28:22 -05:00
J. Nick Koston ebed1de581
Avoid creating multiple sqlalchemy sessions in a single history call (#35721)
* Avoid a context switch in the history api

The history api was creating a job to fetch the
states and another job to convert the states to
json. This can be done in a single job which
decreases the overhead of the operation.

* Ensure there is only one sqlalchemy session created per history
query.

Most queries created three sqlalchemy sessions which was
especially slow with sqlite since it opens and closes the
database.

In testing the UI is noticeably faster at generating history
graphs for entites.

* Add additional coverage

* pass hass first to _states_to_json and _get_significant_states
2020-05-19 07:52:38 +02:00
J. Nick Koston 6885d72180
Avoid a context switch in the history api (#35716)
* Avoid a context switch in the history api

The history api was creating a job to fetch the
states and another job to convert the states to
json. This can be done in a single job which
decreases the overhead of the operation.

* Revert to original solution to avoid function redefine each call
2020-05-18 11:57:16 -05:00
Franck Nijhof 187b6525b4
Removal of extraneous parenthesis (#33637) 2020-04-04 20:08:55 +02:00
Paulus Schoutsen f1d3c0d19b
Remove unused manifest fields (#33595) 2020-04-03 12:58:19 -07:00
Santobert 0186ce7896
Filter the history of device_tracker by location attributes (#33356)
* Add include_location_attributes

* Add check for different state

* Fix things

* Drop filter

* significant changes only

* Change default behavior

* Remove trailing commas
2020-03-30 08:13:22 -07:00
Franck Nijhof bfa0edaf92 Migrate Integration Quality Scale from docs to manifest: internal (#30551) 2020-01-07 20:58:49 +01:00
Franck Nijhof a84741392b
Format all manifests with prettier (#30521) 2020-01-06 21:28:23 +01:00
Paulus Schoutsen 9e51a18845
Make hassfest import detection better (#29932)
* Make hassfest import detection better

* Fix tests
2019-12-16 08:22:20 +01:00