Commit Graph

49 Commits (3d90d6073ec2dbd25ffdacca71ccacf095407b73)

Author SHA1 Message Date
Raman Gupta 3ca69f5568
Raise an exception when event_type exceeds the max length (#48115)
* raise an exception when event_type exceeds the max length that the recorder supports

* add test

* use max length constant in recorder

* update config entry reloaded service name

* remove exception string function because it's not needed

* increase limit to 64 and revert event name change

* fix test

* assert exception args

* fix test

* add comment about migration
2021-04-08 20:46:28 +02:00
Laszlo Magyar c7e4857d2c
Let recorder deal with event names longer than 32 chars (#47748) 2021-04-08 07:08:49 -10:00
Stefan Agner 815db999da
Use microsecond precision for datetime values on MariaDB/MySQL (#48749)
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2021-04-07 09:13:55 +02:00
J. Nick Koston 61a2460c87
Improve error reporting in recorder purge test (#47929) 2021-03-14 19:46:21 -07:00
Marc Mueller 10848b9bdf
Recorder improvements (#47739) 2021-03-11 07:52:07 -10:00
CurrentThread 7c8851264f
Use LONGTEXT column instead of TEXT for MySQL/MariaDB and migrate existing databases (#47026) 2021-03-10 08:12:58 -10:00
J. Nick Koston a060acc2b1
Fix recorder with MSSQL (#46678)
Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
2021-03-09 11:16:19 -10:00
J. Nick Koston 12af87bc6e
Add index to old_state_id column for postgres and older databases (#44757)
* Add index to old_state_id column for older databases

The schema was updated in #43610 but the index was not
added on migration.

* Handle postgresql missing ondelete

* create index first
2021-01-04 10:51:44 +01:00
J. Nick Koston 1162d9a752
Create tables with a charset that can hold all expected data under mysql (#43732)
By default these tables are created with utf8 which can only hold 3 bytes. This
meant that all emjoi would trigger a MySQLdb._exceptions.OperationalError because
they are 4 bytes.

This will only fix the issue for users who recreate their tables.
2020-11-28 23:54:05 +01:00
moinmoin-sh 337b8d279e
Ensure MariaDB/MySQL can be purged and handle states being deleted out from under the recorder (#43610)
* MariaDB doesn't purge #42402

This addresses  home-assistant#42402
Relationships within table "states" and between tables "states" and "events " home-assistant#40467 prevent the purge from working correctly. The database increases w/o any purge.
This proposal sets related indices to NULL and permits deleting of rows.
Further explanations can be found here home-assistant#42402
This proposal also allows to purge the tables "events" and "states" in any order.

* Update models.py

Corrected for Black style requirements

* Update homeassistant/components/recorder/models.py

Co-authored-by: J. Nick Koston <nick@koston.org>

* Add the options to foreign key constraints

* purge old states when database gets deleted out from under us

* pylint

Co-authored-by: J. Nick Koston <nick@koston.org>
2020-11-28 08:42:29 -10:00
J. Nick Koston 4798f37c6e
Convert States to dicts via as_dict only once (#41208)
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2020-10-05 09:18:57 -05:00
J. Nick Koston 20a136e2a1
Avoid event data serialization during recorder that we throw away (#41217)
We currently serialize the event data for state change events
and then replace it because we save the state in the states table.
Since the old state and new state are both contains in the event
the cost of serializing the data has a noticable impact when there
are many state changed events.
2020-10-05 15:08:47 +02:00
J. Nick Koston d9ba32dc3f
Setup recorder model relationships to avoid calling flush (#40467) 2020-09-30 13:11:43 +02:00
J. Nick Koston aada6a1d88
Fix logbook with empty filter and remove unused code (#40565) 2020-09-30 13:10:11 +02:00
J. Nick Koston 7c346c2f7c
Skip the sqlite quick_check on clean restarts (#38972) 2020-08-21 14:20:46 +02:00
Franck Nijhof 7d74b74570
Fix recorder purging by batch processing purges (#37140) 2020-06-26 10:27:45 -07:00
J. Nick Koston 91e0395c1c
Optimize database indexes for existing queries (#37036)
Cleanup indexes as >50% of the db size was indexes,
many of them unused in any current query

Logbook search was having to filter event_types without
an index:

  Created ix_events_event_type_time_fired
  Dropped ix_events_event_type

States had a redundant keys on composite index:

  Dropped ix_states_entity_id
  Its unused since we have ix_states_entity_id_last_updated

De-duplicate storage of context in states as
its always stored in events and can be found
by joining the state on the event_id.

  Dropped ix_states_context_id
  Dropped ix_states_context_parent_id
  Dropped ix_states_context_user_id

After schema v9:

STATES............................................ 10186       40.9%
EVENTS............................................ 5502        22.1%
IX_STATES_ENTITY_ID_LAST_UPDATED.................. 2177         8.7%
IX_EVENTS_EVENT_TYPE_TIME_FIRED................... 1910         7.7%
IX_EVENTS_CONTEXT_ID.............................. 1592         6.4%
IX_EVENTS_TIME_FIRED.............................. 1383         5.6%
IX_STATES_LAST_UPDATED............................ 1079         4.3%
IX_STATES_EVENT_ID................................ 375          1.5%
IX_EVENTS_CONTEXT_PARENT_ID....................... 347          1.4%
IX_EVENTS_CONTEXT_USER_ID......................... 346          1.4%
IX_RECORDER_RUNS_START_END........................ 1            0.004%
RECORDER_RUNS..................................... 1            0.004%
SCHEMA_CHANGES.................................... 1            0.004%
SQLITE_MASTER..................................... 1            0.004%
2020-06-23 10:57:52 -07:00
J. Nick Koston 53a91ece4e
Improve isoformat timestamp performance (#36991)
* adj

* time_fired_isoformat

* remove unused code

* tests for processing timestamps

* restore missing import lost in merge conflict

* test for None case
2020-06-22 12:06:02 -05:00
J. Nick Koston edad387b12
Make recorder execute avoid native conversion by default (#36938) 2020-06-21 21:58:57 -07:00
J. Nick Koston e7d982ee11
Improve db performance of state change events and reduce overall db size (#36883)
* Add old_state_id to states, remove old/new state data from events since it can now be found by a join

* remove state lookup on restart

* Ensure old_state is set for exisitng states
2020-06-17 22:26:41 -05:00
J. Nick Koston b0163b65c6
Use states to avoid decoding logbook state changed events. (#36768)
avg 4.43s -> 1.88s
2020-06-15 13:53:05 -05:00
J. Nick Koston 0a6deeb49b
Improve history api performance (#35822)
* Improve history api performance

A new option "minimal_response" reduces the amount of data
sent between the first and last history states to only the
"last_changed" and "state" fields.

Calling to_native is now avoided where possible and only
done at the end for rows that will be returned in the response.

When sending the `minimal_response` option, the history
api now returns a json response similar to the following
for an entity

Testing:

History API Response time for 1 day
Average of 10 runs with minimal_response

Before: 19.89s. (content length : 3427428)
After: 8.44s (content length: 592199)

```
[{
	"attributes": {--TRUNCATED--},
	"context": {--TRUNCATED--},
	"entity_id": "binary_sensor.powerwall_status",
	"last_changed": "2020-05-18T23:20:03.213000+00:00",
	"last_updated": "2020-05-18T23:20:03.213000+00:00",
	"state": "on"
},
...
{
	"last_changed": "2020-05-19T00:41:08Z",
	"state": "unavailable"
},
...
{
	"attributes": {--TRUNCATED--},
	"context": {--TRUNCATED--},
	"entity_id": "binary_sensor.powerwall_status",
	"last_changed": "2020-05-19T00:42:08.069698+00:00",
	"last_updated": "2020-05-19T00:42:08.069698+00:00",
	"state": "on"
}]
```

* Remove impossible state check

* Remove another impossible state check

* Update homeassistant/components/history/__init__.py

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>

* Reorder to save some indent per review

* Make query response make sense with to_native=False

* Update test for 00:00 to Z change

* Update homeassistant/components/recorder/models.py

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2020-05-26 21:53:56 -05:00
Odin Ugedal 6c4a6568f5
Fix timezone issues for db fields in recorder (#35719)
The database fields are timezoned via DateTime(timezone=True), so the
default value should be timezoned too. When using cockroachdb this is
fatal and results in the recorder crashing.
2020-05-19 13:13:27 -04:00
Paulus Schoutsen 7d8da47309
Revert "Use orjson to parse json faster (#32153)" (#32185)
This reverts commit 2365e2e8cf.
2020-02-25 21:07:16 +01:00
Ruslan Sayfutdinov 2365e2e8cf
Use orjson to parse json faster (#32153)
* [recorder] Use orjson to parse json faster

* Remove from http manifest

* Bump to orjson 2.5.1

* Empty commit to trigger CI

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2020-02-25 10:20:51 -08:00
Bas Nijholt 3d64405896 Sort imports according to PEP8 for recorder (#29652) 2019-12-08 18:48:18 +01:00
Malte Franken 83a709b768 Move imports in recorder component (#27859)
* move imports to top-level in recorder init

* move imports to top-level in recorder migration

* move imports to top-level in recorder models

* move imports to top-level in recorder purge

* move imports to top-level in recorder util

* fix pylint
2019-10-18 10:14:54 -07:00
Paulus Schoutsen 4de97abc3a Black 2019-07-31 12:25:30 -07:00
Paulus Schoutsen 52f337ef00
Allow chaining contexts (#21028)
* Allow chaining contexts

* Add stubbed out migration
2019-03-01 10:08:38 -08:00
Paulus Schoutsen c7f5beb794
history allowed to load states with invalid entity IDs (#20399) 2019-01-24 17:53:01 -08:00
Aleksandr Smirnov 089a2f4e71 Logbook speedup (#18376)
* filter logbook results by entity_id prior to instantiating them

* include by default, pass pep8

* pass pylint

* use entityfilter, update tests
2018-11-19 10:36:00 +01:00
Paulus Schoutsen 7bb5344942
Remove homeassistant.remote (#16099)
* Remove homeassistant.remote

* Use direct import for API

* Fix docstring
2018-08-21 15:49:58 +02:00
Paulus Schoutsen 9512bb9587
Add and restore context in recorder (#15859) 2018-08-10 18:09:01 +02:00
Ville Skyttä b7c336a687 Pylint cleanups (#15626)
* Pylint 2 no-else-return fixes

* Remove unneeded abstract-class-not-used pylint disable
2018-07-23 10:16:05 +02:00
Anders Melchiorsen 339a839dbe Add SQL index to states.event_id (#12825) 2018-03-03 22:54:38 +01:00
Greg Laabs 6e1785173f History query and schema optimizations for huge performance boost (#8748)
* Add DEBUG-level log for db row to native object conversion

This is now the bottleneck (by a large margin) for big history queries, so I'm leaving this log feature in to help diagnose users with a slow history page

* Rewrite of the "first synthetic datapoint" query for multiple entities

The old method was written in a manner that prevented an index from being used in the inner-most GROUP BY statement, causing massive performance issues especially when querying for a large time period.

The new query does have one material change that will cause it to return different results than before: instead of using max(state_id) to get the latest entry, we now get the max(last_updated). This is more appropriate (primary key should not be assumed to be in order of event firing) and allows an index to be used on the inner-most query. I added another JOIN layer to account for cases where there are two entries on the exact same `last_created` for a given entity. In this case we do use `state_id` as a tiebreaker.

For performance reasons the domain filters were also moved to the outermost query, as it's way more efficient to do it there than on the innermost query as before (due to indexing with GROUP BY problems)

The result is a query that only needs to do a filesort on the final result set, which will only be as many rows as there are entities.

* Remove the ORDER BY entity_id when fetching states, and add logging

Having this ORDER BY in the query prevents it from using an index due to the range filter, so it has been removed.

We already do a `groupby` in the `states_to_json` method which accomplishes exactly what the ORDER BY in the query was trying to do anyway, so this change causes no functional difference.

Also added DEBUG-level logging to allow diagnosing a user's slow history page.

* Add DEBUG-level logging for the synthetic-first-datapoint query

For diagnosing a user's slow history page

* Missed a couple instances of `created` that should be `last_updated`

* Remove `entity_id` sorting from state_changes; match significant_update

This is the same change as 09b3498f41 , but applied to the `state_changes_during_period` method which I missed before. This should give the same performance boost to the history sensor component!

* Bugfix in History query used for History Sensor

The date filter was using a different column for the upper and lower bounds. It would work, but it would be slow!

* Update Recorder purge script to use more appropriate columns

Two reasons: 1. the `created` column's meaning is fairly arbitrary and does not represent when an event or state change actually ocurred. It seems more correct to purge based on the event date than the time the database row was written.
2. The new columns are indexed, which will speed up this purge script by orders of magnitude

* Updating db model to match new query optimizations

A few things here: 1. New schema version with a new index and several removed indexes
2. A new method in the migration script to drop old indexes
3. Added an INFO-level log message when a new index will be added, as this can take quite some time on a Raspberry Pi
2017-08-04 23:16:53 -07:00
Paulus Schoutsen 46e030662d Fix pylint 1.7.2 no-else-return issues (#8361)
* Fix pylint 1.7.2 no-else-return issues

* Update tomato.py
2017-07-05 23:30:01 -07:00
Mike Megally 0bf5021c2c Create an index on the states table to help hass startup time (#8255) 2017-07-01 14:10:17 -07:00
Fabian Affolter a4f1f6e724 Update docstrings (#7374)
* Update docstrings

* Update docstrings

* Update docstrings

* Update docstrings

* Update docstrings

* Update docstrings

* Update docstring

* Update docstrings

* Update docstrings

* Fix lint issues

* Update docstrings

* Revert changes in dict
2017-05-02 09:18:47 -07:00
Tim Soderstrom 5dfdb9e481 New indexes for states and recording_runs tables (#6688)
* New indexes for states table

* Added recorder_runs indexes

* Created a new function for compound indexes.

A new function was created because it makes it a little cleaner when creating
a single-field index since one doesn't have to create a list. This is mostly
when creating the name of the index so with a bit more logic it's possible
to combine it into one function. Given how often migration changes are run,
I thought that code bloat was probably a worthy trade-off for now.

* Adjusted indexes, POC for ref indexes by name.

* Corrected lint errors

* Fixed pydocstyle error

* Moved create_index function outside apply_update

* Moved to single line (just barely)
2017-03-23 20:48:31 -07:00
Adam Mills 6a64e79d7b [recorder] Index events time_fired to improve logbook performance (#5633)
* Index events time_fired to improve logbook perf.

* Updated implementation to track schema versions

* Added tests for schema migration support logic

* Rename check_schema to migrate_schema
2017-02-02 22:04:14 -05:00
Fabian Affolter be272ac64a Disable too-many-* (#4107)
* Disable too-many-* and too-few-public-methods

* Remove globally disabled pylint warnings
2016-10-30 22:18:53 +01:00
Paulus Schoutsen 586208b3ed Fix JSON encoder issue in recorder 2016-08-27 07:43:42 +01:00
Paulus Schoutsen e926426af9 Recorder: Increase size of the entity column in states table (#2778)
Fixes https://github.com/home-assistant/home-assistant/issues/2697
2016-08-10 17:40:52 -07:00
Paulus Schoutsen 640a8b5a7f Limit dependencies of HA core (#2762) 2016-08-08 20:21:40 -07:00
Johann Kellerman 4cf618334c Update recorder. (#2549)
* Update recorder.

models.py:
 - Use scoped_session in models.py to fix shutdown error
__init__.py:
 - Session _commit & retry method
 - Single session var for purge_data
 - Ensure single _INSTANCE
 - repeat purge every 2 days
 - show correct time in log_error

* _commit

* Restore models to old functionality, swap purge, remove _INSTANCE cleanup from tests, typing ignore Base class

* pylint

* Remove recorder from model unit test
2016-07-23 11:25:17 -07:00
Paulus Schoutsen 1738db9ccc Update models.py 2016-07-11 12:38:35 -07:00
Paulus Schoutsen e0dd5a8558 Tweak Recorder 2016-07-11 08:56:07 -07:00
rhooper a2e45b8fdd Switch to SQLAlchemy for the Recorder component. Gives the ability t… (#2377)
* Switch to SQLAlchemy for the Recorder component.  Gives the ability to use MySQL or other.

* fixes for failed lint

* add conversion script

* code review fixes and refactor to use to_native() model methods and execute() helper

* move script to homeassistant.scripts module

* style fixes my tox lint/flake8 missed

* move exclusion up
2016-07-02 11:22:51 -07:00