4a1c40f09b
Revert "Refactor recorder migration"
...
This reverts commit 69e10e5982
2022-10-12 15:12:12 +02:00
69e10e5982
Refactor recorder migration
2022-10-12 15:00:28 +02:00
577f7904b5
Minor improvements of recorder typing ( #80165 )
...
* Minor improvements of recorder typing
* Only allow specifying statistic_ids as lists
2022-10-12 14:59:10 +02:00
9aa6043255
Set character set to utf8mb4 when connecting to MySQL or MariaDB databases ( #79755 )
2022-10-11 14:01:46 +02:00
e8af007124
Disable echo for non SQLite databases ( #80032 )
...
* Disable echo for non SQLite databases
* Add test
2022-10-10 22:42:38 +03:00
2b27cfdabb
Set system & entity integration types ( #79593 )
2022-10-04 10:36:42 -04:00
92ca95ca81
Fix preserving long term statistics when entity_id is changed ( #79556 )
2022-10-03 21:13:48 -04:00
a2e3978d53
Don't normalize units of long term statistics ( #79320 )
...
* Don't normalize units of long term statistics
* Update statistics.py
2022-10-03 21:42:44 +02:00
0fdb7052e9
Add comment in recorder about dropping column ( #79523 )
...
Add comment in recorder
2022-10-03 11:40:11 +02:00
2de273500e
Remove state_unit_of_measurement from metadata DB table ( #79370 )
...
* Remove state_unit_of_measurement from metadata DB table
* Adjust test
2022-10-01 18:55:00 +02:00
3a9ecab98a
Improve iterable typing (1) ( #79295 )
2022-10-01 00:12:39 +02:00
bc2dffabc4
Improve naming of units used in statistics ( #79276 )
2022-09-30 08:38:44 +02:00
5438552d4a
Cleanup unit conversion ( #79197 )
...
* Move UNIT_RATIO to BaseUnitConverter
* Make UNIT_CONVERSION private
* Remove STATISTIC_UNIT_TO_UNIT_CLASS constant
* Cleanup websocket_api
* Imrpove valid_units check
2022-09-28 13:49:46 +02:00
5389ff3253
Add new weight device class ( #79185 )
2022-09-28 12:13:49 +02:00
12e4d18038
Add volume to SensorDeviceClass ( #77960 )
...
* Add volume to SensorDeviceClass
* Adjust recorder
* Adjust tests
* Adjust sensor UNIT_CONVERTERS
* Adjust recorder
* Update strings.json
2022-09-27 19:37:52 +02:00
7c448416e1
Add speed to SensorDeviceClass ( #77953 )
...
* Add speed to SensorDeviceClass
* Adjust recorder
* Adjust tests
* Adjust sensor UNIT_CONVERTERS
* Add tests
* Add websocket tests
* Update strings.json
2022-09-27 17:19:34 +01:00
bfcc18e5b8
Add distance to SensorDeviceClass ( #77951 )
...
* Add distance to SensorDeviceClass
* Adjust recorder
* Adjust tests
* Adjust recorder
* Update __init__.py
* Update test_websocket_api.py
* Update test_websocket_api.py
* Update test_websocket_api.py
* Update strings.json
* Fix tests
* Adjust docstring
2022-09-27 15:34:00 +01:00
2167cf540d
Drop some unused constants from recorder ( #79138 )
2022-09-27 10:52:16 +02:00
c52d0f7495
Support converting statistics to another unit ( #79117 )
2022-09-27 08:44:58 +02:00
e8156adb13
Update mypy to 0.981 ( #79115 )
2022-09-26 16:10:06 -04:00
83b426deb0
Adjust normalization routines in recorder statistics ( #78966 )
2022-09-23 08:27:27 -04:00
ddf56baf7a
Move temperature utility to unit_conversion ( #78960 )
2022-09-22 18:31:50 +02:00
c8491c4404
Move volume utility to unit_conversion ( #78955 )
...
* Move volume utility to unit_conversion
* Split tests
2022-09-22 17:49:45 +02:00
523d8d246b
Move pressure utility to unit_conversion ( #78953 )
2022-09-22 16:44:09 +02:00
0767cdd935
Move energy and power utilites to unit_conversion ( #78950 )
...
* Move energy and power utilites to unit_conversion
* Move tests
2022-09-22 15:39:49 +02:00
6002377d4f
Convert UnitConverter protocol to a class ( #78934 )
...
* Convert UnitConverter protocl to a class
* Remove logic change
* Use TypeVar
* Remove NORMALIZED_UNIT from pressure
* Reduce size of PR
* Reduce some more
* Once more
* Once more
* Remove DEVICE_CLASS
2022-09-22 14:15:22 +02:00
713fb874a8
Add NORMALISED_UNIT to UnitConverter ( #78920 )
...
* Add NORMALISED_UNIT to UnitConverter
* Adjust statistics
* Rename
2022-09-22 08:50:08 +02:00
39315b7fe3
Introduce UnitConverter protocol ( #78888 )
...
* Introduce ConversionUtility
* Use ConversionUtility in number
* Use ConversionUtility in sensor
* Use ConversionUtility in sensor recorder
* Add normalise to ConversionUtility
* Revert changes to recorder.py
* Reduce size of PR
* Adjust recorder statistics
* Rename variable
* Rename
* Apply suggestion
Co-authored-by: Erik Montnemery <erik@montnemery.com>
* Apply suggestion
Co-authored-by: Erik Montnemery <erik@montnemery.com>
* Apply suggestion
Co-authored-by: Erik Montnemery <erik@montnemery.com>
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2022-09-22 07:18:00 +02:00
ca78b1a77d
Add new energy utility ( #78883 )
...
* Add new energy utility
* Adjust STATISTIC_UNIT_TO_VALID_UNITS
2022-09-21 19:40:40 +02:00
cd6697615f
Validate units when importing statistics ( #78891 )
2022-09-21 12:08:53 -04:00
d7382aadfe
Add new power utility ( #78867 )
...
* Add power utility
* Fix tests
2022-09-21 14:48:38 +02:00
dae00c70de
Allow selecting display unit when fetching statistics ( #78578 )
2022-09-20 17:43:57 -04:00
6e4a0b9fdc
Switch recorder to use async_timeout instead of asyncio.wait_for ( #78607 )
2022-09-17 20:40:10 +03:00
dd20a7ea62
Display statistics in the source's unit ( #78031 )
2022-09-15 12:01:24 -04:00
eae384bbf7
Update sqlalchemy to 1.4.41 ( #78507 )
2022-09-15 11:54:18 +02:00
0a1fd36e03
Catch up with statistics after DB migration is done ( #78469 )
...
* Catch up with statistics after DB migration is done
* Don't access the database from the event loop
* Fix deadlocking test
* Fix test
2022-09-15 07:40:56 +02:00
13f250319d
Import websocket api constants from root ( #78250 )
2022-09-12 22:07:58 -04:00
dbc6dda41e
Adjust components to use relative imports ( #78279 )
2022-09-12 23:33:21 +02:00
253d355526
Remove unused mypy ignore statements ( #78292 )
2022-09-12 09:25:11 -04:00
eb28d7188b
Fix DB migration to schema version 29 ( #78037 )
...
* Fix DB migration to schema version 29
* Fix misspelled constants
2022-09-09 08:06:14 +02:00
7937bfeedb
Deprecate history integration's statistics API ( #78056 )
2022-09-08 16:03:43 -04:00
008ac8d10d
Improve statistics metadata WS API ( #77209 )
2022-08-31 11:30:45 +02:00
8ed689fede
Add new rule to enforce relative imports in pylint ( #77358 )
...
* Add new rule to enforce relative imports in pylint
* Early return
* Adjust components
2022-08-29 08:55:32 +02:00
4333d9a7d1
Fix recorder being imported before deps are installed ( #77460 )
2022-08-28 22:18:35 -05:00
0caf998547
Bump awesomeversion from 22.6.0 to 22.8.0 ( #77436 )
2022-08-28 14:52:23 -04:00
f61edf0778
Fix updating of statistics metadata name ( #77207 )
...
* Fix updating of statistics metadata name
* Fix test
* Test renaming
2022-08-23 17:51:17 +02:00
4a938ec33e
[ci skip] Translation update
2022-08-10 00:23:36 +00:00
4e3db5bb5c
Update sqlalchemy to 1.4.40 ( #76505 )
2022-08-09 09:35:22 -04:00
c4ad6d46ae
[ci skip] Translation update
2022-07-30 00:22:48 +00:00
20b6c4c48e
Fix recorder hanging at start ( #75627 )
2022-07-22 12:37:25 -05:00
fd6ffef52f
Support non-live database migration ( #72433 )
...
* Support non-live database migration
* Tweak startup order, add test
* Address review comments
* Fix typo
* Clarify comment about promoting dependencies
* Tweak
* Fix merge mistake
* Fix some tests
* Fix additional test
* Fix additional test
* Adjust tests
* Improve test coverage
2022-07-22 15:11:34 +02:00
606d544157
Use recorder get_instance function to improve typing ( #75567 )
2022-07-22 11:58:26 +02:00
1d7d2875e1
Add websocket command recorder/import_statistics ( #73937 )
...
* Expose ws_add_external_statistics in websocket API
* Refactor
* Add tests
* Improve test coverage
Co-authored-by: Thibault Cohen <titilambert@users.noreply.github.com>
Co-authored-by: Erik <erik@montnemery.com>
2022-07-21 12:36:49 +02:00
ca1f0909fb
Fix spelling in recorder integration ( #75539 )
2022-07-20 19:13:10 -05:00
1a1eeb2274
Allow for subclass typing with StatisticsBase ( #75476 )
2022-07-20 04:02:03 +02:00
79a0940932
[ci skip] Translation update
2022-07-17 00:25:56 +00:00
332cf3cd2d
Resolve and caches paths for CachingStaticResource in the executor ( #74474 )
2022-07-06 13:49:48 -05:00
4e079c4417
Fix typo in recorder ( #74178 )
2022-06-29 09:50:24 -05:00
a8349a4866
Adjust entity filters to make includes stronger than excludes ( #74080 )
...
* Adjust entity filters to make includes stronger than excludes
Fixes #59080
* adjust test for stronger entity glob includes
* sync with docs
2022-06-28 11:42:51 -05:00
00a79635c1
Revert "Remove sqlalchemy lambda_stmt usage from history, logbook, and statistics ( #73191 )" ( #73917 )
2022-06-23 13:59:55 -05:00
3d59088a62
Bump sqlalchemy to 1.4.38 ( #73916 )
...
Changes: https://docs.sqlalchemy.org/en/14/changelog/changelog_14.html#change-1.4.38
2022-06-23 14:13:42 -04:00
8015bb98a9
Switch recorder and templates to use json helper ( #73876 )
...
- These were using orjson directly, its a bit cleaner
to use the helper so everything is easier to adjust
in the future if we need to change anything about
the loading
2022-06-23 14:32:26 +02:00
8b067e83f7
Initial orjson support take 3 ( #73849 )
...
* Initial orjson support take 2
Still need to work out problem building wheels
--
Redux of #72754 / #32153 Now possible since the following is solved:
ijl/orjson#220 (comment)
This implements orjson where we use our default encoder. This does not implement orjson where `ExtendedJSONEncoder` is used as these areas tend to be called far less frequently. If its desired, this could be done in a followup, but it seemed like a case of diminishing returns (except maybe for large diagnostics files, or traces, but those are not expected to be downloaded frequently).
Areas where this makes a perceptible difference:
- Anything that subscribes to entities (Initial subscribe_entities payload)
- Initial download of registries on first connection / restore
- History queries
- Saving states to the database
- Large logbook queries
- Anything that subscribes to events (appdaemon)
Cavets:
orjson supports serializing dataclasses natively (and much faster) which
eliminates the need to implement `as_dict` in many places
when the data is already in a dataclass. This works
well as long as all the data in the dataclass can also
be serialized. I audited all places where we have an `as_dict`
for a dataclass and found only backups needs to be adjusted (support for `Path` needed to be added for backups). I was a little bit worried about `SensorExtraStoredData` with `Decimal` but it all seems to work out from since it converts it before it gets to the json encoding cc @dgomes
If it turns out to be a problem we can disable this
with option |= [orjson.OPT_PASSTHROUGH_DATACLASS](https://github.com/ijl/orjson#opt_passthrough_dataclass ) and it
will fallback to `as_dict`
Its quite impressive for history queries
<img width="1271" alt="Screen_Shot_2022-05-30_at_23_46_30" src="https://user-images.githubusercontent.com/663432/171145699-661ad9db-d91d-4b2d-9c1a-9d7866c03a73.png ">
* use for views as well
* handle UnicodeEncodeError
* tweak
* DRY
* DRY
* not needed
* fix tests
* Update tests/components/http/test_view.py
* Update tests/components/http/test_view.py
* black
* templates
2022-06-22 21:59:51 +02:00
24bf42cfbe
Update pylint to 2.14.3 ( #73703 )
2022-06-19 16:29:57 +02:00
53b3d2ee87
Guard MySQL size calculation returning None ( #73331 )
2022-06-10 12:49:58 -07:00
7ae8bd5137
Remove sqlalchemy lambda_stmt usage from history, logbook, and statistics ( #73191 )
2022-06-07 16:15:50 -07:00
5f2b4001f3
Separate recorder database schema from other classes ( #72977 )
...
* Separate recorder database schema from other classes
* fix logbook imports
* migrate new tests
* few more
* last one
* fix merge
Co-authored-by: J. Nick Koston <nick@koston.org>
2022-06-07 14:41:43 +02:00
de2e9b6d77
Fix state_changes_during_period history query when no entities are passed ( #73139 )
2022-06-06 12:50:52 -07:00
0b62944148
Mark counter domain as continuous to exclude it from logbook ( #73101 )
2022-06-05 21:25:26 -10:00
c66b000d34
Reduce branching in generated lambda_stmts ( #73042 )
2022-06-05 21:13:31 -07:00
7ac7af094f
Fix missing historical context data in logbook for MySQL and PostgreSQL ( #73011 )
2022-06-04 12:54:10 -07:00
8e8fa0399e
Fix statistics_during_period being incorrectly cached ( #72947 )
2022-06-03 13:04:46 -07:00
5b31414225
Fix misalignments between sql based filtering with the entityfilter based filtering ( #72936 )
2022-06-02 20:52:53 -07:00
9fbde245d0
Fix performance of logbook entity and devices queries with large MySQL databases ( #72898 )
2022-06-02 14:54:06 -07:00
cd590c79e2
Fix migration of MySQL data when InnoDB is not being used ( #72893 )
...
Fixes #72883
2022-06-02 14:01:06 +02:00
f79e5e002b
Ensure recorder shuts down when its startup future is canceled out from under it ( #72866 )
2022-06-01 22:13:09 -07:00
c6e56c26b3
Fix logbook not setting up with an recorder filter that has empty fields ( #72869 )
2022-06-01 22:12:00 -07:00
7746715590
[ci skip] Translation update
2022-06-02 00:27:51 +00:00
d8b0376942
Fix purge of legacy database events that are not state changed ( #72815 )
2022-06-01 11:56:06 +02:00
e874a04319
Bump sqlalchemy to 1.4.37 ( #72809 )
...
Fixes a bug where reconnects might fail with MySQL 8.0.24+
Changelog: https://docs.sqlalchemy.org/en/14/changelog/changelog_14.html#change-1.4.37
2022-06-01 10:56:05 +02:00
0df9cc907a
[ci skip] Translation update
2022-06-01 00:27:00 +00:00
856e1144c9
Ensure the statistics_meta table is using the dynamic row format ( #72784 )
2022-05-31 14:35:28 -07:00
c365454afb
Revert "Initial orjson support ( #72754 )" ( #72789 )
...
This was causing the wheels to fail to build. We need
to workout why when we don't have release pressure
This reverts commit d9d22a9556
2022-05-31 13:51:55 -07:00
d9d22a9556
Initial orjson support ( #72754 )
2022-05-31 12:18:11 -07:00
587fd05603
Make logbook inherit the recorder filter ( #72728 )
2022-05-30 17:34:32 -07:00
362f5720ed
[ci skip] Translation update
2022-05-31 00:23:11 +00:00
3c5b778ee3
[ci skip] Translation update
2022-05-30 00:27:06 +00:00
a43d47fa0b
Escape % and _ in history/logbook entity_globs, and use ? as _ ( #72623 )
...
Co-authored-by: pyos <pyos100500@gmail.com>
2022-05-27 14:38:29 -07:00
34323ce645
Add explict type casts for postgresql filters ( #72615 )
2022-05-27 08:11:33 -10:00
040e120101
Fix recorder system health when the db_url is lacking a hostname ( #72612 )
2022-05-27 10:32:26 -07:00
049c06061c
Fix memory leak when firing state_changed events ( #72571 )
2022-05-26 20:54:26 -07:00
bfa7693d18
Fixes for logbook filtering and add it to the live stream ( #72501 )
2022-05-25 15:17:08 -10:00
3c246b7800
Update mypy to 0.960 ( #72481 )
2022-05-25 11:42:14 -07:00
301341a49f
[ci skip] Translation update
2022-05-25 00:26:18 +00:00
23bd64b7a2
Prevent duplication of statistics metadata ( #71637 )
...
* Prevent duplication of statistics metadata
* Add models_schema_28.py
* Handle entity renaming as a recorder job
* Improve tests
2022-05-24 15:34:46 +02:00
abbfed24a4
[ci skip] Translation update
2022-05-24 00:25:27 +00:00
4dbc1ed7a7
Add dialect (database engine) and version to recorder system health data ( #72339 )
2022-05-22 21:29:30 -07:00
9c3f949165
Add live streaming logbook websocket endpoint ( #72258 )
...
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2022-05-22 14:57:54 -05:00
1001f9e39a
Fix last state in history minimal respones when all the states at the end are skipped ( #72203 )
2022-05-19 20:58:32 -07:00
edd7a3427c
Remove support for databases without ROW_NUMBER ( #72092 )
2022-05-18 21:52:38 -05:00
Erik
Franck Nijhof
Marc Mueller
epenet
J. Nick Koston
Joakim Sørensen
GitHub Action
Thibault Cohen
Paulus Schoutsen
