Commit Graph

61 Commits (86fde1a644ef5ecde6fc66b90cd0c090feaa847f)

Author SHA1 Message Date
J. Nick Koston 7ae8bd5137
Remove sqlalchemy lambda_stmt usage from history, logbook, and statistics (#73191) 2022-06-07 16:15:50 -07:00
Erik Montnemery 5f2b4001f3
Separate recorder database schema from other classes (#72977)
* Separate recorder database schema from other classes

* fix logbook imports

* migrate new tests

* few more

* last one

* fix merge

Co-authored-by: J. Nick Koston <nick@koston.org>
2022-06-07 14:41:43 +02:00
J. Nick Koston 4dbc1ed7a7
Add dialect (database engine) and version to recorder system health data (#72339) 2022-05-22 21:29:30 -07:00
Erik Montnemery edd7a3427c
Remove support for databases without ROW_NUMBER (#72092) 2022-05-18 21:52:38 -05:00
Erik Montnemery 037f6947d8
Fail recorder setup with unsupported dialect or version (#70888) 2022-05-18 16:52:46 +02:00
J. Nick Koston f3c582815c
Convert statistics to use lambda_stmt (#71903)
* Convert stats to use lambda_stmt

- Since baked queries are now [deprecated in 1.4](https://docs.sqlalchemy.org/en/14/orm/extensions/baked.html#module-sqlalchemy.ext.baked) the
  next step is to convert these to `lambda_stmt`

https://docs.sqlalchemy.org/en/14/core/connections.html#quick-guidelines-for-lambdas

* Update homeassistant/components/recorder/statistics.py

Co-authored-by: Erik Montnemery <erik@montnemery.com>
2022-05-18 16:22:21 +02:00
J. Nick Koston a4c1bcefb9
Tune sqlite based on configured settings (#72016) 2022-05-17 18:12:15 -05:00
J. Nick Koston 98809675ff
Convert history queries to use lambda_stmt (#71870)
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2022-05-15 10:47:29 -05:00
J. Nick Koston a8f1dda004
Use ciso8601 for parsing MySQLdb datetimes (#71818)
* Use ciso8601 for parsing MySQLDB datetimes

The default parser is this:

5340191feb/MySQLdb/times.py (L66)

* tweak

* tweak

* add coverage for building the MySQLdb connect conv param
2022-05-13 19:26:09 -05:00
J. Nick Koston 6922209ddb
Ensure all mysql tables get optimized (#71538) 2022-05-08 12:15:06 -07:00
J. Nick Koston a8aa0e1cca
Add Estimated Database Size to the recorder system health (#71463) 2022-05-07 21:02:54 -07:00
Marc Mueller cf90e34776
Type recorder retry decorator (#70993) 2022-04-28 21:04:41 +02:00
J. Nick Koston 72fffde77a
Fix spelling of periodic in recorder (#69658)
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2022-04-07 23:37:02 -10:00
J. Nick Koston ec131d685e
Auto repack the database on the second sunday of the month (#69314) 2022-04-04 21:39:12 -10:00
Marc Mueller 53110f8cb7
Update pylint to 2.13.2 (#68704) 2022-03-27 16:08:24 +02:00
Marc Mueller 911b159281
Cleanup after pylint update (#68657) 2022-03-26 00:34:12 +01:00
J. Nick Koston 225f7a989b
Add strict typing for recorder util (#68681) 2022-03-25 15:03:46 -07:00
J. Nick Koston c5c34bc0d7
Typing and code quality for recorder history (#68647) 2022-03-24 17:58:38 -07:00
J. Nick Koston 7deeb92485
Switch sqlalchemy execute to use .all() instead of list() on the iterator (#68540) 2022-03-22 19:24:21 -10:00
Stefan Agner 0a9927d18e
Avoid locking the database for non-SQLite backends (#63847)
* Avoid locking the database for non-SQLite backends

Currently we only have a lock implementation for SQLite. Just return
success for all other databases as they are not expected to store data
in the config directory and the caller can assume that a backup can
be safely taken.

This fixes `RuntimeError: generator didn't yield` errors when creating
a backup with the current Supervisor dev builds.
2022-01-11 16:17:56 +01:00
Erik Montnemery bceeaec2f8
Remove duplicated statistics rows (#61146)
* Remove duplicated statistics

* Fix misleading docstring

* Pylint knows best

* Correct test

* Oops

* Prevent insertion of duplicated statistics

* Tweak

* pylint

* Add models_schema_23.py

* Tweak
2021-12-13 14:15:36 +01:00
Stefan Agner f0006b92be
Allow to lock SQLite database during backup (#60874)
* Allow to set CONF_DB_URL

This is useful for test which need a custom DB path.

* Introduce write_lock_db helper to lock SQLite database

* Introduce Websocket API which allows to lock database during backup

* Fix isort

* Avoid mutable default arguments

* Address pylint issues

* Avoid holding executor thread

* Set unlock event in case timeout occures

This makes sure the database is left unlocked even in case of a race
condition.

* Add more unit tests

* Address new pylint errors

* Lower timeout to speedup tests

* Introduce queue overflow test

* Unlock database if necessary

This makes sure that the test runs through in case locking actually
succeeds (and the test fails).

* Make DB_LOCK_TIMEOUT a global

There is no good reason for this to be an argument. The recorder needs
to pick a sensible value.

* Add Websocket Timeout test

* Test lock_database() return

* Update homeassistant/components/recorder/__init__.py

Co-authored-by: Erik Montnemery <erik@montnemery.com>

* Fix format

Co-authored-by: J. Nick Koston <nick@koston.org>
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2021-12-07 13:16:24 +01:00
Erik Montnemery 470b01e4ce
Move async_migration_in_progress (#59087) 2021-11-05 10:40:56 +01:00
Erik Montnemery be4e9f91b6
Change minimum supported SQLite version to 3.31.0 (#59073) 2021-11-04 16:34:35 +01:00
Erik Montnemery dfa50a842a
Simplify recorder PgSQL version checks (#58533) 2021-10-27 16:05:40 +02:00
Erik Montnemery e9ba5f3b4b
Warn when recorder connects to an unsupported database (#58161) 2021-10-26 13:41:59 +02:00
Erik Montnemery 8ef8838801
Correct detection of row_number support for MariaDB (#57663) 2021-10-14 11:19:39 -07:00
Erik Montnemery 0139bfa749
Detect if mysql and sqlite support row_number (#57475) 2021-10-11 21:17:18 -07:00
Marc Mueller 565a9fea6b
Import Callable from collections.abc (2) (#56776) 2021-09-29 14:06:51 +02:00
Erik Montnemery 5976f898da
Add WS API for removing statistics for a list of statistic_ids (#55078)
* Add WS API for removing statistics for a list of statistic_ids

* Refactor according to code review, enable foreign keys support for sqlite

* Adjust tests

* Move clear_statistics WS API to recorder

* Adjust tests after rebase

* Update docstring

* Update homeassistant/components/recorder/websocket_api.py

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>

* Adjust tests after rebase

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2021-09-27 23:30:13 +02:00
Erik Montnemery 8c5efafdd8
Add 5-minute statistics for sensors (#56006)
* Add 5-minute statistics for sensors

* Address pylint issues

* Black

* Apply suggestion from code review

* Apply suggestions from code review

* Improve tests
2021-09-16 10:57:15 +02:00
Erik Montnemery 32e297f4a0
Compile missing statistics (#54690) 2021-08-19 22:10:45 -07:00
Paulus Schoutsen 557cc792e9
Fix SQLAlchemy test warnings (#54116) 2021-08-08 20:33:47 -07:00
Erik Montnemery 19d3aa71ad
Enable basic type checking for recorder (#52440)
* Enable basic type checking for recorder

* Tweak
2021-07-13 21:21:45 +02:00
Erik Montnemery 0ab999738b
Add statistics meta data table (#52331)
* Add statistics meta data table

* Tweak meta data generation
2021-06-30 13:32:17 +02:00
J. Nick Koston e7f7e61e88
Ensure a wal checkpoint is scheduled nightly (#50746) 2021-05-17 16:27:51 -05:00
J. Nick Koston 72288710ca
Increase the sqlite cache size from ~2MiB to 8MiB (#50747) 2021-05-17 19:42:12 +02:00
Erik Montnemery 89dd3292ba
Initial draft of statistics (#49852) 2021-05-16 10:23:37 -07:00
jan iversen c351098f04
HomeAssistantType —> HomeAssistant for Integrations p* - s* (#49558) 2021-04-22 19:58:02 +02:00
J. Nick Koston c10836fcee
Upgrade to sqlalchemy 1.4.11 (#49538) 2021-04-21 20:29:36 -10:00
J. Nick Koston 6d137d2316
Increase recorder test coverage (#49362)
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-04-19 08:22:38 -07:00
J. Nick Koston 9368891b1b
Live db migrations and recovery (#49036)
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2021-04-11 20:43:54 -10:00
J. Nick Koston c820dd4cb5
Have pylint warn when user visible log messages do not start with capital letter or end with a period (#48064)
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-03-19 09:26:36 -05:00
Marc Mueller 10848b9bdf
Recorder improvements (#47739) 2021-03-11 07:52:07 -10:00
J. Nick Koston 9b69549f73
Recover and restart the recorder if the sqlite database encounters corruption while running (#46612) 2021-02-19 21:26:24 -10:00
J. Nick Koston 22dbac259b
Ensure recorder shuts down cleanly on restart before startup is finished (#46604) 2021-02-19 22:18:21 -08:00
Ville Skyttä de04a1ed67
Enable more Bandit tests (#44307)
https://bandit.readthedocs.io/en/latest/plugins/index.html#complete-test-plugin-listing
2020-12-19 12:35:13 +01:00
J. Nick Koston 557684c3ce
Add ability to disable the sqlite3 quick_check (#39479) 2020-09-02 10:12:56 +02:00
Franck Nijhof 1c2ebdf307
Upgrade black to 20.8b1 (#39287) 2020-08-27 13:56:20 +02:00
J. Nick Koston 7c346c2f7c
Skip the sqlite quick_check on clean restarts (#38972) 2020-08-21 14:20:46 +02:00