Commit Graph

240 Commits (8be6605be95d28abe96a335efdd866ed57298e7b)

Author SHA1 Message Date
J. Nick Koston b27e9e376d
Use StaticPool for recorder and NullPool for all other threads with sqlite3 (#49693) 2021-04-26 19:20:31 -10:00
J. Nick Koston 376b787e4d
Skip recorder commit if there is nothing to do (#49614) 2021-04-25 12:05:49 +02:00
jan iversen c351098f04
HomeAssistantType —> HomeAssistant for Integrations p* - s* (#49558) 2021-04-22 19:58:02 +02:00
J. Nick Koston c10836fcee
Upgrade to sqlalchemy 1.4.11 (#49538) 2021-04-21 20:29:36 -10:00
J. Nick Koston 6d137d2316
Increase recorder test coverage (#49362)
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-04-19 08:22:38 -07:00
J. Nick Koston e24f5831a2
Force recorder shutdown at final write event (#49145)
* Force recorder shutdown at EVENT_HOMEASSISTANT_FINAL_WRITE

* remove unreachable

* remove unreachable

* simplify

* cancel in async
2021-04-19 11:24:17 +02:00
Franck Nijhof 055cdc64c0
Add support for IoT class in manifest (#46935) 2021-04-15 10:21:38 +02:00
J. Nick Koston 53853f035d
Prevent calling stop or restart services during db upgrade (#49098) 2021-04-12 17:18:38 -07:00
J. Nick Koston 9368891b1b
Live db migrations and recovery (#49036)
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2021-04-11 20:43:54 -10:00
Raman Gupta 3ca69f5568
Raise an exception when event_type exceeds the max length (#48115)
* raise an exception when event_type exceeds the max length that the recorder supports

* add test

* use max length constant in recorder

* update config entry reloaded service name

* remove exception string function because it's not needed

* increase limit to 64 and revert event name change

* fix test

* assert exception args

* fix test

* add comment about migration
2021-04-08 20:46:28 +02:00
Laszlo Magyar c7e4857d2c
Let recorder deal with event names longer than 32 chars (#47748) 2021-04-08 07:08:49 -10:00
Stefan Agner 815db999da
Use microsecond precision for datetime values on MariaDB/MySQL (#48749)
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2021-04-07 09:13:55 +02:00
Franck Nijhof 3aed84560f
Merge of nested IF-IF cases - O-R (#48371) 2021-03-27 10:38:57 +01:00
J. Nick Koston c820dd4cb5
Have pylint warn when user visible log messages do not start with capital letter or end with a period (#48064)
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-03-19 09:26:36 -05:00
Marc Mueller a3cd1854f6
Update typing 12 (#48073) 2021-03-18 14:31:38 +01:00
J. Nick Koston 61a2460c87
Improve error reporting in recorder purge test (#47929) 2021-03-14 19:46:21 -07:00
J. Nick Koston 33c4eb3434
Log the full exception when the recorder fails to setup (#47770) 2021-03-11 21:52:04 -08:00
Marc Mueller 92852b9c10
Add apply_filter attribute to recorder.purge service (#45826) 2021-03-11 17:03:30 -10:00
Marc Mueller 10848b9bdf
Recorder improvements (#47739) 2021-03-11 07:52:07 -10:00
CurrentThread 7c8851264f
Use LONGTEXT column instead of TEXT for MySQL/MariaDB and migrate existing databases (#47026) 2021-03-10 08:12:58 -10:00
J. Nick Koston a060acc2b1
Fix recorder with MSSQL (#46678)
Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
2021-03-09 11:16:19 -10:00
adrian-vlad 44293a3738
Add enable and disable services for recorder (#45778) 2021-02-24 07:26:05 -06:00
Franck Nijhof ea4bbd771f
Add service names to previously enriched services (#46929)
Co-authored-by: Tobias Sauerwein <cgtobi@users.noreply.github.com>
2021-02-23 14:10:13 +01:00
J. Nick Koston 9b69549f73
Recover and restart the recorder if the sqlite database encounters corruption while running (#46612) 2021-02-19 21:26:24 -10:00
J. Nick Koston 22dbac259b
Ensure recorder shuts down cleanly on restart before startup is finished (#46604) 2021-02-19 22:18:21 -08:00
Franck Nijhof 8b69608242
Add selectors to Browser, Recorder, Shopping List service definitions (#46749) 2021-02-18 17:06:25 +01:00
J. Nick Koston c9df42b69a
Add support for pre-filtering events to the event bus (#46371) 2021-02-14 09:42:55 -10:00
J. Nick Koston dc26fd5149
Ensure creating an index that already exists is forgiving for postgresql (#46185)
Unlikely sqlite and mysql, postgresql throws ProgrammingError instead
of InternalError or OperationalError when trying to create an index
that already exists.
2021-02-08 23:22:38 +01:00
Fabian Affolter 60d4dadcb6
Upgrade sqlalchemy to 1.3.23 (#45845) 2021-02-01 16:03:51 -10:00
J. Nick Koston 12af87bc6e
Add index to old_state_id column for postgres and older databases (#44757)
* Add index to old_state_id column for older databases

The schema was updated in #43610 but the index was not
added on migration.

* Handle postgresql missing ondelete

* create index first
2021-01-04 10:51:44 +01:00
Fabian Affolter 12b7b2098d
Upgrade sqlalchemy to 1.3.22 (#44698) 2020-12-31 14:28:15 -10:00
Ville Skyttä de04a1ed67
Enable more Bandit tests (#44307)
https://bandit.readthedocs.io/en/latest/plugins/index.html#complete-test-plugin-listing
2020-12-19 12:35:13 +01:00
J. Nick Koston 2cbb93be43
Always keep the current recorder run when purging (#43733) 2020-11-30 09:13:50 +01:00
J. Nick Koston 1162d9a752
Create tables with a charset that can hold all expected data under mysql (#43732)
By default these tables are created with utf8 which can only hold 3 bytes. This
meant that all emjoi would trigger a MySQLdb._exceptions.OperationalError because
they are 4 bytes.

This will only fix the issue for users who recreate their tables.
2020-11-28 23:54:05 +01:00
moinmoin-sh 337b8d279e
Ensure MariaDB/MySQL can be purged and handle states being deleted out from under the recorder (#43610)
* MariaDB doesn't purge #42402

This addresses  home-assistant#42402
Relationships within table "states" and between tables "states" and "events " home-assistant#40467 prevent the purge from working correctly. The database increases w/o any purge.
This proposal sets related indices to NULL and permits deleting of rows.
Further explanations can be found here home-assistant#42402
This proposal also allows to purge the tables "events" and "states" in any order.

* Update models.py

Corrected for Black style requirements

* Update homeassistant/components/recorder/models.py

Co-authored-by: J. Nick Koston <nick@koston.org>

* Add the options to foreign key constraints

* purge old states when database gets deleted out from under us

* pylint

Co-authored-by: J. Nick Koston <nick@koston.org>
2020-11-28 08:42:29 -10:00
Franck Nijhof 5dbb5f12eb
Upgrade sqlalchemy to 1.3.20 (#41765) 2020-10-13 08:43:58 -05:00
J. Nick Koston 33b548e247
Reduce ORM overhead when the old state was already written to the database (#41736)
We can avoid processing the relationship when the old state was already
written to the database which will common case with a commit interval
of 1s. Since we already know the value for old_state_id we can use it
instead of asking sqlalchemy to process the relationship at flush/commit
time which can significantly speed up sqlalchemy's _emit_insert_statements
implementation.
2020-10-13 15:34:46 +02:00
Matthias Alphart 93a9a11065
Add config validator helper positive_float (#41640) 2020-10-11 22:04:49 +02:00
J. Nick Koston 1c431aa7bd
Set created field when creating db events and states (#41523)
Avoids thousands of utcnow calls when the session
is commited on busy systems to fill in the field
default.
2020-10-09 09:31:17 +02:00
J. Nick Koston 4ab02cb9bc
Ensure recorder commit can retry after encountering invalid data (#41426) 2020-10-08 09:15:25 +02:00
J. Nick Koston 113d738fa2
Reduce orm overhead by grouping object expiration (#41394) 2020-10-07 15:35:48 +02:00
J. Nick Koston d35e33610a
Resolve memory leak in recorder (#41349)
Avoids a build up of the InstanceState.
2020-10-06 20:12:47 +02:00
J. Nick Koston 4798f37c6e
Convert States to dicts via as_dict only once (#41208)
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2020-10-05 09:18:57 -05:00
J. Nick Koston 20a136e2a1
Avoid event data serialization during recorder that we throw away (#41217)
We currently serialize the event data for state change events
and then replace it because we save the state in the states table.
Since the old state and new state are both contains in the event
the cost of serializing the data has a noticable impact when there
are many state changed events.
2020-10-05 15:08:47 +02:00
J. Nick Koston 6b509fd9db
Prevent sqlalchemy from refetching the old_state_id as it will never change (#40982)
Disable expire_on_commit for the event writer. Since we never expect the
old_state_id to change in the database, it was never worth the expense of
refetching the id after the commit.
2020-10-01 19:57:52 +02:00
J. Nick Koston d9ba32dc3f
Setup recorder model relationships to avoid calling flush (#40467) 2020-09-30 13:11:43 +02:00
J. Nick Koston aada6a1d88
Fix logbook with empty filter and remove unused code (#40565) 2020-09-30 13:10:11 +02:00
J. Nick Koston c19b5c5ac3
Make recorder block_till_done reliable (#40043) 2020-09-14 08:48:29 +02:00
J. Nick Koston 557684c3ce
Add ability to disable the sqlite3 quick_check (#39479) 2020-09-02 10:12:56 +02:00
Franck Nijhof 1c2ebdf307
Upgrade black to 20.8b1 (#39287) 2020-08-27 13:56:20 +02:00