Commit Graph

64 Commits (99481e22580eecda6d3d2196dcf46da747a7a3d3)

Author SHA1 Message Date
Erik Montnemery bceeaec2f8
Remove duplicated statistics rows ()
* Remove duplicated statistics

* Fix misleading docstring

* Pylint knows best

* Correct test

* Oops

* Prevent insertion of duplicated statistics

* Tweak

* pylint

* Add models_schema_23.py

* Tweak
2021-12-13 14:15:36 +01:00
Erik Montnemery d980ca7e04
Correct recorder migration._add_columns for PostgreSQL () 2021-11-29 18:33:25 +01:00
Erik Montnemery 2f3dea30e2
Correct migration to recorder schema 22 () 2021-11-04 10:29:10 +01:00
Erik Montnemery f594bc353b
Add support for external statistics ()
* Support external statistics

* Update tests

* Lint

* Adjust code after rebase

* Separate external statistic_id with :, add name to metadata

* Adjust tests

* Simplify get_metadata_with_session

* Address review comments

* Allow updating external statistics

* Validate input

* Adjust tests after rebase

* Pylint

* Adjust tests

* Improve test coverage
2021-10-26 10:26:50 +02:00
Erik Montnemery e3534eec87
Report orphaned statistics in statistic validation () 2021-10-20 15:57:22 +02:00
Erik Montnemery 8337baa354
Correct migration to recorder schema 18 () 2021-10-06 13:29:42 +02:00
Erik Montnemery dfb3a0c528
Correct database migration to schema version 22 () 2021-09-30 14:11:00 -07:00
Erik Montnemery 6af1a835e6
Optimize statistics generation ()
* Optimize statistics generation

* pylint
2021-09-30 17:14:36 +02:00
Erik Montnemery 40ecf22bac
Remove automatic splitting of net meters from statistics () 2021-09-29 17:20:17 +02:00
Erik Montnemery 52e9f76f94
Tweak DB migration to schema version 21 () 2021-09-29 11:25:50 +02:00
Erik Montnemery fe66d6295c
Improve migration to recorder schema version 21 () 2021-09-27 19:31:40 +02:00
Erik Montnemery 64393b462d
Add migration for 5-minute statistics ()
* Add migration for 5-minute statistics

* Tweaks
2021-09-24 09:19:22 +02:00
Marcin Ciupak b9ffd74db5
Fix recorder Oracle DB models ()
* Fix recorder models for Oracle DB

* Fix StatisticsRuns

* Update migration for Oracle Identity columns.

* Update migration logic

Migration to schema version 22 done only for engine dialect oracle

* Add missing table check in schema 22 migration
2021-09-21 00:38:42 +02:00
Erik Montnemery 8c5efafdd8
Add 5-minute statistics for sensors ()
* Add 5-minute statistics for sensors

* Address pylint issues

* Black

* Apply suggestion from code review

* Apply suggestions from code review

* Improve tests
2021-09-16 10:57:15 +02:00
Erik Montnemery 88dbc6373f
Make sure character set of events, states tables is utf8 ()
* Make sure character set of events, states tables is utf8

* Pylint

* Apply suggestions from code review
2021-09-09 19:26:28 +02:00
Erik Montnemery a47532c69b
Change character set of statistics_meta table to utf8 () 2021-09-09 17:24:20 +02:00
Erik Montnemery 80fd330479
Add sum_decrease and sum_increase statistics () 2021-09-08 23:35:53 -07:00
Erik Montnemery 0624859bf4
Set statistics columns to double precision () 2021-08-24 11:18:59 +02:00
Erik Montnemery 32e297f4a0
Compile missing statistics () 2021-08-19 22:10:45 -07:00
Ville Skyttä 470f2dd73f
Upgrade pyupgrade to 2.21.2, apply its changes () 2021-07-19 10:46:09 +02:00
Franck Nijhof 74029a0948
Fix Statistics recorder migration path by dropping in pairs () 2021-07-05 13:34:40 +02:00
Franck Nijhof 5e9127ef7a
Remove problematic/redudant db migration happning schema 15 () 2021-07-05 13:28:01 +02:00
Franck Nijhof 7f3f6757ea
Fix Statistics recorder migration order () 2021-07-02 20:55:40 +02:00
Erik Montnemery 24ae05b734
Drop statistic_id and source columns from statistics table ()
* Drop statistic_id and source columns from statistics table

* Remove useless double drop of statistics table

* Update homeassistant/components/recorder/models.py

Co-authored-by: Franck Nijhof <git@frenck.dev>

* black

Co-authored-by: Franck Nijhof <git@frenck.dev>
2021-07-02 13:17:00 +02:00
Erik Montnemery 0ab999738b
Add statistics meta data table ()
* Add statistics meta data table

* Tweak meta data generation
2021-06-30 13:32:17 +02:00
J. Nick Koston 06c9a50869
Handle missing options in foreign_key for MSSQL () 2021-06-05 12:13:12 +02:00
Marcin Ciupak caad125b44
Add support for Oracle DB in recorder () 2021-05-22 21:10:27 -05:00
Erik Montnemery e16a8063a5
Compile statistics for energy sensors ()
* Compile statistics for energy sensors

* Update tests

* Rename abs_value to state

* Tweak

* Recreate statistics table

* Pylint

* Try to fix test

* Fix statistics for multiple energy sensors

* Fix energy statistics when last_reset is not set
2021-05-20 13:05:15 +02:00
J. Nick Koston 1c8d9ca68b
Check exception causes for matching strings during recorder migration () 2021-05-02 20:57:42 -07:00
J. Nick Koston c10836fcee
Upgrade to sqlalchemy 1.4.11 () 2021-04-21 20:29:36 -10:00
J. Nick Koston 9368891b1b
Live db migrations and recovery ()
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2021-04-11 20:43:54 -10:00
Laszlo Magyar c7e4857d2c
Let recorder deal with event names longer than 32 chars () 2021-04-08 07:08:49 -10:00
Stefan Agner 815db999da
Use microsecond precision for datetime values on MariaDB/MySQL ()
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2021-04-07 09:13:55 +02:00
CurrentThread 7c8851264f
Use LONGTEXT column instead of TEXT for MySQL/MariaDB and migrate existing databases () 2021-03-10 08:12:58 -10:00
J. Nick Koston dc26fd5149
Ensure creating an index that already exists is forgiving for postgresql ()
Unlikely sqlite and mysql, postgresql throws ProgrammingError instead
of InternalError or OperationalError when trying to create an index
that already exists.
2021-02-08 23:22:38 +01:00
J. Nick Koston 12af87bc6e
Add index to old_state_id column for postgres and older databases ()
* Add index to old_state_id column for older databases

The schema was updated in  but the index was not
added on migration.

* Handle postgresql missing ondelete

* create index first
2021-01-04 10:51:44 +01:00
moinmoin-sh 337b8d279e
Ensure MariaDB/MySQL can be purged and handle states being deleted out from under the recorder ()
* MariaDB doesn't purge 

This addresses  home-assistant#42402
Relationships within table "states" and between tables "states" and "events " home-assistant#40467 prevent the purge from working correctly. The database increases w/o any purge.
This proposal sets related indices to NULL and permits deleting of rows.
Further explanations can be found here home-assistant#42402
This proposal also allows to purge the tables "events" and "states" in any order.

* Update models.py

Corrected for Black style requirements

* Update homeassistant/components/recorder/models.py

Co-authored-by: J. Nick Koston <nick@koston.org>

* Add the options to foreign key constraints

* purge old states when database gets deleted out from under us

* pylint

Co-authored-by: J. Nick Koston <nick@koston.org>
2020-11-28 08:42:29 -10:00
Pascal Vizeli c291d4aa7d
Intelligent timeout handler for setup/bootstrap ()
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
Co-authored-by: J. Nick Koston <nick@koston.org>
2020-08-05 14:58:19 +02:00
Franck Nijhof 53545c984b
Log lines do not end with a full stop () 2020-07-05 23:04:19 +02:00
J. Nick Koston ccb77ba1e9
Handle index already existing on db migration with MySQLdb backend ()
_create_index needed the same check as _add_columns since
the MySQLdb backend throws OperationalError instead
of InternalError in this case
2020-07-03 15:35:02 -07:00
J. Nick Koston 255d706c24
Avoid creating a column in v8 schema that is removed in v9 schema () 2020-06-24 09:56:01 -07:00
J. Nick Koston cc8e0ef942
Handle mysql index/column already exists during migration () 2020-06-24 09:55:13 -07:00
J. Nick Koston 91e0395c1c
Optimize database indexes for existing queries ()
Cleanup indexes as >50% of the db size was indexes,
many of them unused in any current query

Logbook search was having to filter event_types without
an index:

  Created ix_events_event_type_time_fired
  Dropped ix_events_event_type

States had a redundant keys on composite index:

  Dropped ix_states_entity_id
  Its unused since we have ix_states_entity_id_last_updated

De-duplicate storage of context in states as
its always stored in events and can be found
by joining the state on the event_id.

  Dropped ix_states_context_id
  Dropped ix_states_context_parent_id
  Dropped ix_states_context_user_id

After schema v9:

STATES............................................ 10186       40.9%
EVENTS............................................ 5502        22.1%
IX_STATES_ENTITY_ID_LAST_UPDATED.................. 2177         8.7%
IX_EVENTS_EVENT_TYPE_TIME_FIRED................... 1910         7.7%
IX_EVENTS_CONTEXT_ID.............................. 1592         6.4%
IX_EVENTS_TIME_FIRED.............................. 1383         5.6%
IX_STATES_LAST_UPDATED............................ 1079         4.3%
IX_STATES_EVENT_ID................................ 375          1.5%
IX_EVENTS_CONTEXT_PARENT_ID....................... 347          1.4%
IX_EVENTS_CONTEXT_USER_ID......................... 346          1.4%
IX_RECORDER_RUNS_START_END........................ 1            0.004%
RECORDER_RUNS..................................... 1            0.004%
SCHEMA_CHANGES.................................... 1            0.004%
SQLITE_MASTER..................................... 1            0.004%
2020-06-23 10:57:52 -07:00
J. Nick Koston e7d982ee11
Improve db performance of state change events and reduce overall db size ()
* Add old_state_id to states, remove old/new state data from events since it can now be found by a join

* remove state lookup on restart

* Ensure old_state is set for exisitng states
2020-06-17 22:26:41 -05:00
Bas Nijholt 3d64405896 Sort imports according to PEP8 for recorder () 2019-12-08 18:48:18 +01:00
Malte Franken 83a709b768 Move imports in recorder component ()
* move imports to top-level in recorder init

* move imports to top-level in recorder migration

* move imports to top-level in recorder models

* move imports to top-level in recorder purge

* move imports to top-level in recorder util

* fix pylint
2019-10-18 10:14:54 -07:00
Franck Nijhof decf13b948 Use literal string interpolation in core (f-strings) () 2019-08-23 09:53:33 -07:00
Paulus Schoutsen 4de97abc3a Black 2019-07-31 12:25:30 -07:00
Anders Melchiorsen eabb68ad7d Do not warn when creating an empty database () 2019-03-24 12:00:29 -07:00
Paulus Schoutsen 52f337ef00
Allow chaining contexts ()
* Allow chaining contexts

* Add stubbed out migration
2019-03-01 10:08:38 -08:00