* Drop statistic_id and source columns from statistics table
* Remove useless double drop of statistics table
* Update homeassistant/components/recorder/models.py
Co-authored-by: Franck Nijhof <git@frenck.dev>
* black
Co-authored-by: Franck Nijhof <git@frenck.dev>
* Add WS API for listing available statistic ids
* Update homeassistant/components/history/__init__.py
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
Co-authored-by: Bram Kragten <mail@bramkragten.nl>
* Add more MaxLengthExceeded checks
* remove some validations to reduce performance impact
* check length of generated entity ID
* dont check entity ID twice and use single context id length constant
* fix test
* add missing test
* Compile statistics for energy sensors
* Update tests
* Rename abs_value to state
* Tweak
* Recreate statistics table
* Pylint
* Try to fix test
* Fix statistics for multiple energy sensors
* Fix energy statistics when last_reset is not set
* raise an exception when event_type exceeds the max length that the recorder supports
* add test
* use max length constant in recorder
* update config entry reloaded service name
* remove exception string function because it's not needed
* increase limit to 64 and revert event name change
* fix test
* assert exception args
* fix test
* add comment about migration
Unlikely sqlite and mysql, postgresql throws ProgrammingError instead
of InternalError or OperationalError when trying to create an index
that already exists.
* Add index to old_state_id column for older databases
The schema was updated in #43610 but the index was not
added on migration.
* Handle postgresql missing ondelete
* create index first
By default these tables are created with utf8 which can only hold 3 bytes. This
meant that all emjoi would trigger a MySQLdb._exceptions.OperationalError because
they are 4 bytes.
This will only fix the issue for users who recreate their tables.
* MariaDB doesn't purge #42402
This addresses home-assistant#42402
Relationships within table "states" and between tables "states" and "events " home-assistant#40467 prevent the purge from working correctly. The database increases w/o any purge.
This proposal sets related indices to NULL and permits deleting of rows.
Further explanations can be found here home-assistant#42402
This proposal also allows to purge the tables "events" and "states" in any order.
* Update models.py
Corrected for Black style requirements
* Update homeassistant/components/recorder/models.py
Co-authored-by: J. Nick Koston <nick@koston.org>
* Add the options to foreign key constraints
* purge old states when database gets deleted out from under us
* pylint
Co-authored-by: J. Nick Koston <nick@koston.org>
We can avoid processing the relationship when the old state was already
written to the database which will common case with a commit interval
of 1s. Since we already know the value for old_state_id we can use it
instead of asking sqlalchemy to process the relationship at flush/commit
time which can significantly speed up sqlalchemy's _emit_insert_statements
implementation.
We currently serialize the event data for state change events
and then replace it because we save the state in the states table.
Since the old state and new state are both contains in the event
the cost of serializing the data has a noticable impact when there
are many state changed events.
Disable expire_on_commit for the event writer. Since we never expect the
old_state_id to change in the database, it was never worth the expense of
refetching the id after the commit.
Now that python 3.7 is the minimum supported version, we can
use the more efficient SimpleQueue in the recorder as it does
not have to use threading.Lock
On startup we run an sqlite3 quick_check to verify the database
integrity. In the majority of cases, the quick_check takes under
10 seconds.
On systems with very large databases and very slow disk/cpu,
this can take much longer so we freeze the timeout.
* Added GLOB capability to entityfilter and every place that uses it. All existing tests are passing
* added tests for components affected by glob change
* fixed flake8 error
* mocking the correct listener
* mocking correct bus method in azure test
* tests passing in 3.7 and 3.8
* fixed formatting issue from rebase/conflict
* Checking against glob patterns in more performant way
* perf improvments and reverted unnecessarily adjusted tests
* added new benchmark test around filters
* no longer using get with default in entityfilter
* changed filter name and removed logbook from filter benchmark
* simplified benchmark tests from feedback
* fixed apache tests and returned include exclude schemas to normal
* fixed azure event hub tests to properly go through component logic
* fixed azure test and clean up for other tests
* renaming test files to match standard
* merged mqtt statestream test changes with base
* removed dependency on recorder filter schema from history
* fixed recorder tests after merge and a bunch of lint errors
Cleanup indexes as >50% of the db size was indexes,
many of them unused in any current query
Logbook search was having to filter event_types without
an index:
Created ix_events_event_type_time_fired
Dropped ix_events_event_type
States had a redundant keys on composite index:
Dropped ix_states_entity_id
Its unused since we have ix_states_entity_id_last_updated
De-duplicate storage of context in states as
its always stored in events and can be found
by joining the state on the event_id.
Dropped ix_states_context_id
Dropped ix_states_context_parent_id
Dropped ix_states_context_user_id
After schema v9:
STATES............................................ 10186 40.9%
EVENTS............................................ 5502 22.1%
IX_STATES_ENTITY_ID_LAST_UPDATED.................. 2177 8.7%
IX_EVENTS_EVENT_TYPE_TIME_FIRED................... 1910 7.7%
IX_EVENTS_CONTEXT_ID.............................. 1592 6.4%
IX_EVENTS_TIME_FIRED.............................. 1383 5.6%
IX_STATES_LAST_UPDATED............................ 1079 4.3%
IX_STATES_EVENT_ID................................ 375 1.5%
IX_EVENTS_CONTEXT_PARENT_ID....................... 347 1.4%
IX_EVENTS_CONTEXT_USER_ID......................... 346 1.4%
IX_RECORDER_RUNS_START_END........................ 1 0.004%
RECORDER_RUNS..................................... 1 0.004%
SCHEMA_CHANGES.................................... 1 0.004%
SQLITE_MASTER..................................... 1 0.004%
* adj
* time_fired_isoformat
* remove unused code
* tests for processing timestamps
* restore missing import lost in merge conflict
* test for None case
Franck Nijhof
Erik Montnemery
PeteBa
Paulus Schoutsen
J. Nick Koston
Raman Gupta
tkdrob
Marcin Ciupak
jan iversen
Laszlo Magyar
Stefan Agner
Marc Mueller
CurrentThread
adrian-vlad
Fabian Affolter
Ville Skyttä
moinmoin-sh
Matthias Alphart
Pascal Vizeli
mdegat01
Alex van den Hoogen