fd6ffef52f
Support non-live database migration ( #72433 )
...
* Support non-live database migration
* Tweak startup order, add test
* Address review comments
* Fix typo
* Clarify comment about promoting dependencies
* Tweak
* Fix merge mistake
* Fix some tests
* Fix additional test
* Fix additional test
* Adjust tests
* Improve test coverage
2022-07-22 15:11:34 +02:00
8015bb98a9
Switch recorder and templates to use json helper ( #73876 )
...
- These were using orjson directly, its a bit cleaner
to use the helper so everything is easier to adjust
in the future if we need to change anything about
the loading
2022-06-23 14:32:26 +02:00
8b067e83f7
Initial orjson support take 3 ( #73849 )
...
* Initial orjson support take 2
Still need to work out problem building wheels
--
Redux of #72754 / #32153 Now possible since the following is solved:
ijl/orjson#220 (comment)
This implements orjson where we use our default encoder. This does not implement orjson where `ExtendedJSONEncoder` is used as these areas tend to be called far less frequently. If its desired, this could be done in a followup, but it seemed like a case of diminishing returns (except maybe for large diagnostics files, or traces, but those are not expected to be downloaded frequently).
Areas where this makes a perceptible difference:
- Anything that subscribes to entities (Initial subscribe_entities payload)
- Initial download of registries on first connection / restore
- History queries
- Saving states to the database
- Large logbook queries
- Anything that subscribes to events (appdaemon)
Cavets:
orjson supports serializing dataclasses natively (and much faster) which
eliminates the need to implement `as_dict` in many places
when the data is already in a dataclass. This works
well as long as all the data in the dataclass can also
be serialized. I audited all places where we have an `as_dict`
for a dataclass and found only backups needs to be adjusted (support for `Path` needed to be added for backups). I was a little bit worried about `SensorExtraStoredData` with `Decimal` but it all seems to work out from since it converts it before it gets to the json encoding cc @dgomes
If it turns out to be a problem we can disable this
with option |= [orjson.OPT_PASSTHROUGH_DATACLASS](https://github.com/ijl/orjson#opt_passthrough_dataclass ) and it
will fallback to `as_dict`
Its quite impressive for history queries
<img width="1271" alt="Screen_Shot_2022-05-30_at_23_46_30" src="https://user-images.githubusercontent.com/663432/171145699-661ad9db-d91d-4b2d-9c1a-9d7866c03a73.png ">
* use for views as well
* handle UnicodeEncodeError
* tweak
* DRY
* DRY
* not needed
* fix tests
* Update tests/components/http/test_view.py
* Update tests/components/http/test_view.py
* black
* templates
2022-06-22 21:59:51 +02:00
5f2b4001f3
Separate recorder database schema from other classes ( #72977 )
...
* Separate recorder database schema from other classes
* fix logbook imports
* migrate new tests
* few more
* last one
* fix merge
Co-authored-by: J. Nick Koston <nick@koston.org>
2022-06-07 14:41:43 +02:00
9fbde245d0
Fix performance of logbook entity and devices queries with large MySQL databases ( #72898 )
2022-06-02 14:54:06 -07:00
c365454afb
Revert "Initial orjson support ( #72754 )" ( #72789 )
...
This was causing the wheels to fail to build. We need
to workout why when we don't have release pressure
This reverts commit d9d22a9556
2022-05-31 13:51:55 -07:00
d9d22a9556
Initial orjson support ( #72754 )
2022-05-31 12:18:11 -07:00
049c06061c
Fix memory leak when firing state_changed events ( #72571 )
2022-05-26 20:54:26 -07:00
bfa7693d18
Fixes for logbook filtering and add it to the live stream ( #72501 )
2022-05-25 15:17:08 -10:00
23bd64b7a2
Prevent duplication of statistics metadata ( #71637 )
...
* Prevent duplication of statistics metadata
* Add models_schema_28.py
* Handle entity renaming as a recorder job
* Improve tests
2022-05-24 15:34:46 +02:00
037f6947d8
Fail recorder setup with unsupported dialect or version ( #70888 )
2022-05-18 16:52:46 +02:00
c4fc84ec1e
Add support for selecting device_ids from the logbook ( #72039 )
...
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2022-05-18 01:58:30 -05:00
a70e2a33dc
Fixing purging legacy rows and improve performance ( #71916 )
2022-05-15 21:25:07 -07:00
089eb9960a
Reduce logbook websocket payload size and parse json attributes via the DBM ( #71895 )
2022-05-15 20:48:57 -07:00
65f44bd80b
Exclude last_changed when same as last_updated for history websocket api ( #71886 )
2022-05-14 23:03:56 -07:00
0584e84c30
Add MySQL index hints to logbook ( #71864 )
...
* Add MySQL index hints to logbook
* fix mysql query planner
2022-05-14 17:01:36 -05:00
ebce5660e3
Sync event timed_fired and the context ulid time ( #71854 )
2022-05-14 12:12:08 -07:00
8c2743bb67
Avoid storing last_changed in the database if its the same as last_updated ( #71843 )
2022-05-14 12:06:31 -07:00
1d9fb4bca8
Fix process_datetime_to_timestamp and add test coverage ( #71755 )
2022-05-12 17:12:50 -07:00
8ab27f26b9
Use ciso8601 for parsing datetimes with sqlalchemy sqlite dialect ( #71766 )
2022-05-12 17:11:43 -07:00
e2cef55162
Add history/history_during_period websocket endpoint ( #71688 )
2022-05-11 17:52:22 -05:00
6922209ddb
Ensure all mysql tables get optimized ( #71538 )
2022-05-08 12:15:06 -07:00
5db014666c
Avoid recording state_changed events in the events table ( #71165 )
...
* squash
fix mypy
* Update homeassistant/components/recorder/models.py
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
* pass all columns
* fix commented out code
* reduce logbook query complexity
* merge
* comment
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2022-05-02 02:10:34 -05:00
c23866e5e5
De-duplicate event data into a new event_data table ( #71135 )
2022-05-01 21:01:17 -05:00
6ce768465f
Improve purge performance for non-sqlite databases ( #70576 )
2022-04-26 13:11:57 -07:00
96d64bd6b7
Avoid selecting last_updated from the db when filtering on last_updated==last_changed ( #70459 )
2022-04-25 08:46:37 -10:00
522a9bb6b1
Exclude supported features and attribution from being recorded in the database ( #69165 )
2022-04-03 13:51:42 +02:00
f5a13fc51b
Add ability to exclude attributes from being recorded by entity domain ( #68824 )
2022-03-29 20:13:08 -07:00
d7634d1cb1
Additional strict typing for additional recorder internals ( #68689 )
...
* Strict typing for additional recorder internals
* revert
* fix refactoring error
2022-03-29 09:45:25 +02:00
53245c6523
Update pylint to 2.13.0 ( #68656 )
2022-03-25 15:14:48 -07:00
c5c34bc0d7
Typing and code quality for recorder history ( #68647 )
2022-03-24 17:58:38 -07:00
3150915cb7
Convert unindexed domain queries to entity_id queries ( #68404 )
2022-03-20 01:28:17 -10:00
0c0df07c52
Avoid hashing attributes when they are already in the cache ( #68395 )
2022-03-19 23:33:37 -07:00
9864090e0b
Cache parsing attr in LazyState ( #68232 )
2022-03-18 07:44:37 -07:00
9215702388
Separate attrs into another table (reduces database size) ( #68224 )
2022-03-18 00:23:13 -10:00
b18096fc54
Remove unused columns from states/events tables ( #68078 )
2022-03-13 23:44:55 -07:00
cb736eaeaf
Add type ignore error codes [recorder] ( #66780 )
2022-02-18 10:37:38 +01:00
bceeaec2f8
Remove duplicated statistics rows ( #61146 )
...
* Remove duplicated statistics
* Fix misleading docstring
* Pylint knows best
* Correct test
* Oops
* Prevent insertion of duplicated statistics
* Tweak
* pylint
* Add models_schema_23.py
* Tweak
2021-12-13 14:15:36 +01:00
f594bc353b
Add support for external statistics ( #56607 )
...
* Support external statistics
* Update tests
* Lint
* Adjust code after rebase
* Separate external statistic_id with :, add name to metadata
* Adjust tests
* Simplify get_metadata_with_session
* Address review comments
* Allow updating external statistics
* Validate input
* Adjust tests after rebase
* Pylint
* Adjust tests
* Improve test coverage
2021-10-26 10:26:50 +02:00
a966714032
Minor cleanup of recorder statistics code ( #55339 )
2021-09-29 11:42:39 -07:00
40ecf22bac
Remove automatic splitting of net meters from statistics ( #56772 )
2021-09-29 17:20:17 +02:00
92253f5192
Minor refactoring of periodic statistics ( #56492 )
2021-09-22 13:31:33 -07:00
b9ffd74db5
Fix recorder Oracle DB models ( #55564 )
...
* Fix recorder models for Oracle DB
* Fix StatisticsRuns
* Update migration for Oracle Identity columns.
* Update migration logic
Migration to schema version 22 done only for engine dialect oracle
* Add missing table check in schema 22 migration
2021-09-21 00:38:42 +02:00
8c5efafdd8
Add 5-minute statistics for sensors ( #56006 )
...
* Add 5-minute statistics for sensors
* Address pylint issues
* Black
* Apply suggestion from code review
* Apply suggestions from code review
* Improve tests
2021-09-16 10:57:15 +02:00
d2a9f7904a
Include end time of statistics data points in API response ( #56063 )
...
* Include end time of statistics data points in API response
* Correct typing
* Update tests
2021-09-13 10:02:24 +02:00
2a8121bdcd
Really change character set of statistics_meta table to utf8 ( #56029 )
2021-09-09 22:55:51 +02:00
a47532c69b
Change character set of statistics_meta table to utf8 ( #56011 )
2021-09-09 17:24:20 +02:00
80fd330479
Add sum_decrease and sum_increase statistics ( #55850 )
2021-09-08 23:35:53 -07:00
7e70252de5
Handle statistics for sensor with changing state class ( #55316 )
2021-08-27 16:18:49 +02:00
6d4a47a53d
Fix double precision float for postgresql ( #55249 )
2021-08-26 10:06:53 +02:00
Erik Montnemery
J. Nick Koston
Marc Mueller
Marcin Ciupak
