7937bfeedb
Deprecate history integration's statistics API ( #78056 )
2022-09-08 16:03:43 -04:00
008ac8d10d
Improve statistics metadata WS API ( #77209 )
2022-08-31 11:30:45 +02:00
8ed689fede
Add new rule to enforce relative imports in pylint ( #77358 )
...
* Add new rule to enforce relative imports in pylint
* Early return
* Adjust components
2022-08-29 08:55:32 +02:00
4333d9a7d1
Fix recorder being imported before deps are installed ( #77460 )
2022-08-28 22:18:35 -05:00
0caf998547
Bump awesomeversion from 22.6.0 to 22.8.0 ( #77436 )
2022-08-28 14:52:23 -04:00
f61edf0778
Fix updating of statistics metadata name ( #77207 )
...
* Fix updating of statistics metadata name
* Fix test
* Test renaming
2022-08-23 17:51:17 +02:00
4a938ec33e
[ci skip] Translation update
2022-08-10 00:23:36 +00:00
4e3db5bb5c
Update sqlalchemy to 1.4.40 ( #76505 )
2022-08-09 09:35:22 -04:00
c4ad6d46ae
[ci skip] Translation update
2022-07-30 00:22:48 +00:00
20b6c4c48e
Fix recorder hanging at start ( #75627 )
2022-07-22 12:37:25 -05:00
fd6ffef52f
Support non-live database migration ( #72433 )
...
* Support non-live database migration
* Tweak startup order, add test
* Address review comments
* Fix typo
* Clarify comment about promoting dependencies
* Tweak
* Fix merge mistake
* Fix some tests
* Fix additional test
* Fix additional test
* Adjust tests
* Improve test coverage
2022-07-22 15:11:34 +02:00
606d544157
Use recorder get_instance function to improve typing ( #75567 )
2022-07-22 11:58:26 +02:00
1d7d2875e1
Add websocket command recorder/import_statistics ( #73937 )
...
* Expose ws_add_external_statistics in websocket API
* Refactor
* Add tests
* Improve test coverage
Co-authored-by: Thibault Cohen <titilambert@users.noreply.github.com>
Co-authored-by: Erik <erik@montnemery.com>
2022-07-21 12:36:49 +02:00
ca1f0909fb
Fix spelling in recorder integration ( #75539 )
2022-07-20 19:13:10 -05:00
1a1eeb2274
Allow for subclass typing with StatisticsBase ( #75476 )
2022-07-20 04:02:03 +02:00
79a0940932
[ci skip] Translation update
2022-07-17 00:25:56 +00:00
332cf3cd2d
Resolve and caches paths for CachingStaticResource in the executor ( #74474 )
2022-07-06 13:49:48 -05:00
4e079c4417
Fix typo in recorder ( #74178 )
2022-06-29 09:50:24 -05:00
a8349a4866
Adjust entity filters to make includes stronger than excludes ( #74080 )
...
* Adjust entity filters to make includes stronger than excludes
Fixes #59080
* adjust test for stronger entity glob includes
* sync with docs
2022-06-28 11:42:51 -05:00
00a79635c1
Revert "Remove sqlalchemy lambda_stmt usage from history, logbook, and statistics ( #73191 )" ( #73917 )
2022-06-23 13:59:55 -05:00
3d59088a62
Bump sqlalchemy to 1.4.38 ( #73916 )
...
Changes: https://docs.sqlalchemy.org/en/14/changelog/changelog_14.html#change-1.4.38
2022-06-23 14:13:42 -04:00
8015bb98a9
Switch recorder and templates to use json helper ( #73876 )
...
- These were using orjson directly, its a bit cleaner
to use the helper so everything is easier to adjust
in the future if we need to change anything about
the loading
2022-06-23 14:32:26 +02:00
8b067e83f7
Initial orjson support take 3 ( #73849 )
...
* Initial orjson support take 2
Still need to work out problem building wheels
--
Redux of #72754 / #32153 Now possible since the following is solved:
ijl/orjson#220 (comment)
This implements orjson where we use our default encoder. This does not implement orjson where `ExtendedJSONEncoder` is used as these areas tend to be called far less frequently. If its desired, this could be done in a followup, but it seemed like a case of diminishing returns (except maybe for large diagnostics files, or traces, but those are not expected to be downloaded frequently).
Areas where this makes a perceptible difference:
- Anything that subscribes to entities (Initial subscribe_entities payload)
- Initial download of registries on first connection / restore
- History queries
- Saving states to the database
- Large logbook queries
- Anything that subscribes to events (appdaemon)
Cavets:
orjson supports serializing dataclasses natively (and much faster) which
eliminates the need to implement `as_dict` in many places
when the data is already in a dataclass. This works
well as long as all the data in the dataclass can also
be serialized. I audited all places where we have an `as_dict`
for a dataclass and found only backups needs to be adjusted (support for `Path` needed to be added for backups). I was a little bit worried about `SensorExtraStoredData` with `Decimal` but it all seems to work out from since it converts it before it gets to the json encoding cc @dgomes
If it turns out to be a problem we can disable this
with option |= [orjson.OPT_PASSTHROUGH_DATACLASS](https://github.com/ijl/orjson#opt_passthrough_dataclass ) and it
will fallback to `as_dict`
Its quite impressive for history queries
<img width="1271" alt="Screen_Shot_2022-05-30_at_23_46_30" src="https://user-images.githubusercontent.com/663432/171145699-661ad9db-d91d-4b2d-9c1a-9d7866c03a73.png ">
* use for views as well
* handle UnicodeEncodeError
* tweak
* DRY
* DRY
* not needed
* fix tests
* Update tests/components/http/test_view.py
* Update tests/components/http/test_view.py
* black
* templates
2022-06-22 21:59:51 +02:00
24bf42cfbe
Update pylint to 2.14.3 ( #73703 )
2022-06-19 16:29:57 +02:00
53b3d2ee87
Guard MySQL size calculation returning None ( #73331 )
2022-06-10 12:49:58 -07:00
7ae8bd5137
Remove sqlalchemy lambda_stmt usage from history, logbook, and statistics ( #73191 )
2022-06-07 16:15:50 -07:00
5f2b4001f3
Separate recorder database schema from other classes ( #72977 )
...
* Separate recorder database schema from other classes
* fix logbook imports
* migrate new tests
* few more
* last one
* fix merge
Co-authored-by: J. Nick Koston <nick@koston.org>
2022-06-07 14:41:43 +02:00
de2e9b6d77
Fix state_changes_during_period history query when no entities are passed ( #73139 )
2022-06-06 12:50:52 -07:00
0b62944148
Mark counter domain as continuous to exclude it from logbook ( #73101 )
2022-06-05 21:25:26 -10:00
c66b000d34
Reduce branching in generated lambda_stmts ( #73042 )
2022-06-05 21:13:31 -07:00
7ac7af094f
Fix missing historical context data in logbook for MySQL and PostgreSQL ( #73011 )
2022-06-04 12:54:10 -07:00
8e8fa0399e
Fix statistics_during_period being incorrectly cached ( #72947 )
2022-06-03 13:04:46 -07:00
5b31414225
Fix misalignments between sql based filtering with the entityfilter based filtering ( #72936 )
2022-06-02 20:52:53 -07:00
9fbde245d0
Fix performance of logbook entity and devices queries with large MySQL databases ( #72898 )
2022-06-02 14:54:06 -07:00
cd590c79e2
Fix migration of MySQL data when InnoDB is not being used ( #72893 )
...
Fixes #72883
2022-06-02 14:01:06 +02:00
f79e5e002b
Ensure recorder shuts down when its startup future is canceled out from under it ( #72866 )
2022-06-01 22:13:09 -07:00
c6e56c26b3
Fix logbook not setting up with an recorder filter that has empty fields ( #72869 )
2022-06-01 22:12:00 -07:00
7746715590
[ci skip] Translation update
2022-06-02 00:27:51 +00:00
d8b0376942
Fix purge of legacy database events that are not state changed ( #72815 )
2022-06-01 11:56:06 +02:00
e874a04319
Bump sqlalchemy to 1.4.37 ( #72809 )
...
Fixes a bug where reconnects might fail with MySQL 8.0.24+
Changelog: https://docs.sqlalchemy.org/en/14/changelog/changelog_14.html#change-1.4.37
2022-06-01 10:56:05 +02:00
0df9cc907a
[ci skip] Translation update
2022-06-01 00:27:00 +00:00
856e1144c9
Ensure the statistics_meta table is using the dynamic row format ( #72784 )
2022-05-31 14:35:28 -07:00
c365454afb
Revert "Initial orjson support ( #72754 )" ( #72789 )
...
This was causing the wheels to fail to build. We need
to workout why when we don't have release pressure
This reverts commit d9d22a9556
2022-05-31 13:51:55 -07:00
d9d22a9556
Initial orjson support ( #72754 )
2022-05-31 12:18:11 -07:00
587fd05603
Make logbook inherit the recorder filter ( #72728 )
2022-05-30 17:34:32 -07:00
362f5720ed
[ci skip] Translation update
2022-05-31 00:23:11 +00:00
3c5b778ee3
[ci skip] Translation update
2022-05-30 00:27:06 +00:00
a43d47fa0b
Escape % and _ in history/logbook entity_globs, and use ? as _ ( #72623 )
...
Co-authored-by: pyos <pyos100500@gmail.com>
2022-05-27 14:38:29 -07:00
34323ce645
Add explict type casts for postgresql filters ( #72615 )
2022-05-27 08:11:33 -10:00
040e120101
Fix recorder system health when the db_url is lacking a hostname ( #72612 )
2022-05-27 10:32:26 -07:00
049c06061c
Fix memory leak when firing state_changed events ( #72571 )
2022-05-26 20:54:26 -07:00
bfa7693d18
Fixes for logbook filtering and add it to the live stream ( #72501 )
2022-05-25 15:17:08 -10:00
3c246b7800
Update mypy to 0.960 ( #72481 )
2022-05-25 11:42:14 -07:00
301341a49f
[ci skip] Translation update
2022-05-25 00:26:18 +00:00
23bd64b7a2
Prevent duplication of statistics metadata ( #71637 )
...
* Prevent duplication of statistics metadata
* Add models_schema_28.py
* Handle entity renaming as a recorder job
* Improve tests
2022-05-24 15:34:46 +02:00
abbfed24a4
[ci skip] Translation update
2022-05-24 00:25:27 +00:00
4dbc1ed7a7
Add dialect (database engine) and version to recorder system health data ( #72339 )
2022-05-22 21:29:30 -07:00
9c3f949165
Add live streaming logbook websocket endpoint ( #72258 )
...
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2022-05-22 14:57:54 -05:00
1001f9e39a
Fix last state in history minimal respones when all the states at the end are skipped ( #72203 )
2022-05-19 20:58:32 -07:00
edd7a3427c
Remove support for databases without ROW_NUMBER ( #72092 )
2022-05-18 21:52:38 -05:00
3a13ffcf13
[ci skip] Translation update
2022-05-19 00:26:11 +00:00
037f6947d8
Fail recorder setup with unsupported dialect or version ( #70888 )
2022-05-18 16:52:46 +02:00
f3c582815c
Convert statistics to use lambda_stmt ( #71903 )
...
* Convert stats to use lambda_stmt
- Since baked queries are now [deprecated in 1.4](https://docs.sqlalchemy.org/en/14/orm/extensions/baked.html#module-sqlalchemy.ext.baked ) the
next step is to convert these to `lambda_stmt`
https://docs.sqlalchemy.org/en/14/core/connections.html#quick-guidelines-for-lambdas
* Update homeassistant/components/recorder/statistics.py
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2022-05-18 16:22:21 +02:00
c4fc84ec1e
Add support for selecting device_ids from the logbook ( #72039 )
...
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2022-05-18 01:58:30 -05:00
1d6659224f
[ci skip] Translation update
2022-05-18 00:22:07 +00:00
a4c1bcefb9
Tune sqlite based on configured settings ( #72016 )
2022-05-17 18:12:15 -05:00
8fcee01db0
Remove unnecessary flush from recorder ( #71910 )
2022-05-16 18:03:02 -05:00
f4a2afeb37
Guard against recorder pool current connection disappearing during global destruction ( #71971 )
2022-05-16 17:48:14 -05:00
a70e2a33dc
Fixing purging legacy rows and improve performance ( #71916 )
2022-05-15 21:25:07 -07:00
089eb9960a
Reduce logbook websocket payload size and parse json attributes via the DBM ( #71895 )
2022-05-15 20:48:57 -07:00
37f81b261d
[ci skip] Translation update
2022-05-16 00:24:28 +00:00
98809675ff
Convert history queries to use lambda_stmt ( #71870 )
...
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2022-05-15 10:47:29 -05:00
1f753ecd88
Relocate sqlalchemy filter builder to recorder/filters.py ( #71883 )
2022-05-14 23:04:23 -07:00
65f44bd80b
Exclude last_changed when same as last_updated for history websocket api ( #71886 )
2022-05-14 23:03:56 -07:00
0584e84c30
Add MySQL index hints to logbook ( #71864 )
...
* Add MySQL index hints to logbook
* fix mysql query planner
2022-05-14 17:01:36 -05:00
ebce5660e3
Sync event timed_fired and the context ulid time ( #71854 )
2022-05-14 12:12:08 -07:00
8c2743bb67
Avoid storing last_changed in the database if its the same as last_updated ( #71843 )
2022-05-14 12:06:31 -07:00
a8f1dda004
Use ciso8601 for parsing MySQLdb datetimes ( #71818 )
...
* Use ciso8601 for parsing MySQLDB datetimes
The default parser is this:
5340191feb/MySQLdb/times.py (L66)
2022-05-13 19:26:09 -05:00
c7e8428daa
[ci skip] Translation update
2022-05-13 00:26:47 +00:00
1d9fb4bca8
Fix process_datetime_to_timestamp and add test coverage ( #71755 )
2022-05-12 17:12:50 -07:00
8ab27f26b9
Use ciso8601 for parsing datetimes with sqlalchemy sqlite dialect ( #71766 )
2022-05-12 17:11:43 -07:00
e2cef55162
Add history/history_during_period websocket endpoint ( #71688 )
2022-05-11 17:52:22 -05:00
3d2b0a17ce
[ci skip] Translation update
2022-05-11 00:22:31 +00:00
68c2b63ca1
Fix issue creation links in log messages ( #71638 )
2022-05-10 15:22:12 +02:00
bf77c000ea
Complete baked query conversion for recorder.history ( #71618 )
2022-05-09 21:00:19 -07:00
3de7ffde54
[ci skip] Translation update
2022-05-10 00:23:19 +00:00
222baa53dd
Make database access in the eventloop raise an exception ( #71547 )
2022-05-09 15:22:08 -05:00
f1dd3b7f89
[ci skip] Translation update
2022-05-09 00:21:41 +00:00
896bf986eb
Speed up nightly database purges with lambda_stmt ( #71537 )
2022-05-08 12:47:12 -07:00
6922209ddb
Ensure all mysql tables get optimized ( #71538 )
2022-05-08 12:15:06 -07:00
a8aa0e1cca
Add Estimated Database Size to the recorder system health ( #71463 )
2022-05-07 21:02:54 -07:00
44d8f2f773
[ci skip] Translation update
2022-05-07 00:20:47 +00:00
d612b9e0b4
Reduce event loop overhead for listeners that already queue ( #71364 )
...
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2022-05-05 23:09:10 -04:00
07706fa62a
[ci skip] Translation update
2022-05-06 00:22:16 +00:00
2b3fc97020
[ci skip] Translation update
2022-05-05 00:22:23 +00:00
eb77f8db85
Complete strict typing for recorder ( #71274 )
...
* Complete strict typing for recorder
* update tests
* Update tests/components/recorder/test_migrate.py
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Update tests/components/recorder/test_migrate.py
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
* Remove the asserts
* remove ignore comments
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2022-05-04 12:22:50 -05:00
13ce0a7d6a
Fix history using pre v25 queries during v26 migration ( #71295 )
2022-05-04 08:56:50 -07:00
e30940ef2a
Move processing of recorder service call arguments into services.py ( #71260 )
2022-05-03 13:56:22 -07:00
e1be6dd34f
Move recorder services to services.py ( #71249 )
2022-05-03 10:38:44 -07:00
29bda196b5
Break apart recorder into tasks and core modules ( #71222 )
2022-05-02 21:53:56 -07:00
Erik Montnemery
epenet
J. Nick Koston
Joakim Sørensen
GitHub Action
Franck Nijhof
Thibault Cohen
Marc Mueller
Paulus Schoutsen
