80fd330479
Add sum_decrease and sum_increase statistics ( #55850 )
2021-09-08 23:35:53 -07:00
27764e9985
Fix handling of imperial units in long term statistics ( #55959 )
2021-09-08 08:08:48 -07:00
22e6ddf8df
Do not let one bad statistic spoil the bunch ( #55942 )
2021-09-08 07:55:40 -07:00
7e70252de5
Handle statistics for sensor with changing state class ( #55316 )
2021-08-27 16:18:49 +02:00
6d4a47a53d
Fix double precision float for postgresql ( #55249 )
2021-08-26 10:06:53 +02:00
ffbd2d79c8
Generate statistics for all sensors with a supported state_class ( #54882 )
...
* Generate statistics for all sensors
* Fix bugs, add tests
* Address review comments
* Cleanup warnings
* Simplify tests
* Simplify selection of statistics
* Fix tests
2021-08-25 13:00:35 +02:00
8877f37da0
Fix statistics for sensors setting last_reset ( #55136 )
...
* Re-add state_class total to sensor
* Make energy cost sensor enforce state_class total_increasing
* Drop state_class total
* Only report energy sensor issues once
2021-08-24 17:02:34 +02:00
0624859bf4
Set statistics columns to double precision ( #55053 )
2021-08-24 11:18:59 +02:00
8d69475d71
Fix recorder shutdown race and i/o in event loop ( #54979 )
2021-08-21 21:38:02 +02:00
aa7c72a8b5
Bump sqlalchemy to 1.4.23 ( #54980 )
...
- Changelog: https://docs.sqlalchemy.org/en/14/changelog/changelog_14.html#change-1.4.23
2021-08-21 13:00:39 -04:00
32e297f4a0
Compile missing statistics ( #54690 )
2021-08-19 22:10:45 -07:00
2f77b5025c
Add energy validation ( #54567 )
2021-08-18 11:21:51 -07:00
e1926caeb9
Remove STATE_CLASS_TOTAL and last_reset from sensor ( #54755 )
...
* Remove STATE_CLASS_TOTAL
* Update mill sensor
* Update tests
* Kill last_reset
* Return ATTR_LAST_RESET to utility_meter
* Update energy cost sensor
* Restore last_reset for backwards compatibility
* Re-add and update deprecation warning
* Update tests
* Fix utility_meter
* Update EnergyCostSensor
* Tweak
* Fix rebase mistake
* Fix test
2021-08-18 10:03:27 +02:00
e23750b2a4
Add device class `gas` and enable statistics for it ( #54110 )
...
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2021-08-11 09:58:19 -07:00
557cc792e9
Fix SQLAlchemy test warnings ( #54116 )
2021-08-08 20:33:47 -07:00
c875ff8648
Store JSON in database in compact format ( #53364 )
...
* Store JSON in database in compact format
* Fix logbook
2021-07-22 12:05:59 -10:00
470f2dd73f
Upgrade pyupgrade to 2.21.2, apply its changes ( #52987 )
2021-07-19 10:46:09 +02:00
e541bcd54d
Update statistics meta data on entity_id change ( #52755 )
2021-07-14 13:23:11 +02:00
ddc788bf8e
Include future statistics in history/list_statistic_ids ( #52942 )
...
* Include future statistics in history/list_statistic_ids
* Improve tests
2021-07-14 11:54:55 +02:00
19d3aa71ad
Enable basic type checking for recorder ( #52440 )
...
* Enable basic type checking for recorder
* Tweak
2021-07-13 21:21:45 +02:00
c5556a091e
Return empty when listing statistic_ids for unsupported statistic ( #52954 )
2021-07-13 10:35:55 +02:00
0099b54489
Fix recorder purge with sqlite3 < 3.32.0 ( #52929 )
2021-07-12 22:03:13 +02:00
74029a0948
Fix Statistics recorder migration path by dropping in pairs ( #52453 )
2021-07-05 13:34:40 +02:00
5e9127ef7a
Remove problematic/redudant db migration happning schema 15 ( #52541 )
2021-07-05 13:28:01 +02:00
7f3f6757ea
Fix Statistics recorder migration order ( #52449 )
2021-07-02 20:55:40 +02:00
ed25e6fada
Correct recorder table arguments ( #52436 )
2021-07-02 16:28:16 +02:00
11fd9d9525
Avoid duplicated database queries when fetching statistics ( #52433 )
2021-07-02 15:40:54 +02:00
24ae05b734
Drop statistic_id and source columns from statistics table ( #52417 )
...
* Drop statistic_id and source columns from statistics table
* Remove useless double drop of statistics table
* Update homeassistant/components/recorder/models.py
Co-authored-by: Franck Nijhof <git@frenck.dev>
* black
Co-authored-by: Franck Nijhof <git@frenck.dev>
2021-07-02 13:17:00 +02:00
16d2dcbfb2
Improve sensor statistics tests ( #52386 )
2021-07-02 09:51:47 +02:00
d450cda385
Report target unit in statistics meta data ( #52341 )
2021-06-30 16:53:51 +02:00
9d0c4c168e
Convert units when fetching statistics ( #52338 )
2021-06-30 14:53:42 +02:00
0ab999738b
Add statistics meta data table ( #52331 )
...
* Add statistics meta data table
* Tweak meta data generation
2021-06-30 13:32:17 +02:00
42c4317628
Avoid drift in recorder purge cut-off ( #52135 )
2021-06-25 11:29:38 -10:00
0ca199d8d0
Add WS API for listing available statistic ids ( #51984 )
...
* Add WS API for listing available statistic ids
* Update homeassistant/components/history/__init__.py
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
Co-authored-by: Bram Kragten <mail@bramkragten.nl>
2021-06-18 21:32:30 +02:00
805ef3f90b
Allow fetching multiple statistics ( #51996 )
2021-06-18 21:03:13 +02:00
b3a67a2dd7
Bump sqlalchemy to 1.4.17 ( #51593 )
2021-06-08 16:53:51 +02:00
06c9a50869
Handle missing options in foreign_key for MSSQL ( #51503 )
2021-06-05 12:13:12 +02:00
58e37435b3
Make more max lengths constants and add additional MaxLengthExceeded checks ( #50337 )
...
* Add more MaxLengthExceeded checks
* remove some validations to reduce performance impact
* check length of generated entity ID
* dont check entity ID twice and use single context id length constant
* fix test
* add missing test
2021-05-25 19:58:01 +02:00
028a07d86f
Wrap up selectors ( #50794 )
...
Co-authored-by: Franck Nijhof <git@frenck.dev>
2021-05-25 14:45:17 +02:00
caad125b44
Add support for Oracle DB in recorder ( #50090 )
2021-05-22 21:10:27 -05:00
aa9b99713c
Add purge_entities service call to recorder ( #48069 )
2021-05-22 10:30:05 -05:00
e16a8063a5
Compile statistics for energy sensors ( #50829 )
...
* Compile statistics for energy sensors
* Update tests
* Rename abs_value to state
* Tweak
* Recreate statistics table
* Pylint
* Try to fix test
* Fix statistics for multiple energy sensors
* Fix energy statistics when last_reset is not set
2021-05-20 13:05:15 +02:00
e7f7e61e88
Ensure a wal checkpoint is scheduled nightly ( #50746 )
2021-05-17 16:27:51 -05:00
72288710ca
Increase the sqlite cache size from ~2MiB to 8MiB ( #50747 )
2021-05-17 19:42:12 +02:00
89dd3292ba
Initial draft of statistics ( #49852 )
2021-05-16 10:23:37 -07:00
2d5f5bfa9f
Add targets and selectors for services (P-R) ( #50628 )
2021-05-14 22:07:17 -07:00
973f59e423
Refactor history component ( #50287 )
...
* Refactor history component
* Update tests
* Address review comments
* Correct deprecated functions
2021-05-11 09:21:57 +02:00
906de23087
Bump sqlalchemy to 1.4.13 ( #50138 )
2021-05-05 20:08:48 -07:00
1c8d9ca68b
Check exception causes for matching strings during recorder migration ( #49999 )
2021-05-02 20:57:42 -07:00
1bd9826684
Handle different entity_id formats ( #49969 )
2021-05-01 20:30:28 -07:00
b27e9e376d
Use StaticPool for recorder and NullPool for all other threads with sqlite3 ( #49693 )
2021-04-26 19:20:31 -10:00
376b787e4d
Skip recorder commit if there is nothing to do ( #49614 )
2021-04-25 12:05:49 +02:00
c351098f04
HomeAssistantType —> HomeAssistant for Integrations p* - s* ( #49558 )
2021-04-22 19:58:02 +02:00
c10836fcee
Upgrade to sqlalchemy 1.4.11 ( #49538 )
2021-04-21 20:29:36 -10:00
6d137d2316
Increase recorder test coverage ( #49362 )
...
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-04-19 08:22:38 -07:00
e24f5831a2
Force recorder shutdown at final write event ( #49145 )
...
* Force recorder shutdown at EVENT_HOMEASSISTANT_FINAL_WRITE
* remove unreachable
* remove unreachable
* simplify
* cancel in async
2021-04-19 11:24:17 +02:00
055cdc64c0
Add support for IoT class in manifest ( #46935 )
2021-04-15 10:21:38 +02:00
53853f035d
Prevent calling stop or restart services during db upgrade ( #49098 )
2021-04-12 17:18:38 -07:00
9368891b1b
Live db migrations and recovery ( #49036 )
...
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2021-04-11 20:43:54 -10:00
3ca69f5568
Raise an exception when event_type exceeds the max length ( #48115 )
...
* raise an exception when event_type exceeds the max length that the recorder supports
* add test
* use max length constant in recorder
* update config entry reloaded service name
* remove exception string function because it's not needed
* increase limit to 64 and revert event name change
* fix test
* assert exception args
* fix test
* add comment about migration
2021-04-08 20:46:28 +02:00
c7e4857d2c
Let recorder deal with event names longer than 32 chars ( #47748 )
2021-04-08 07:08:49 -10:00
815db999da
Use microsecond precision for datetime values on MariaDB/MySQL ( #48749 )
...
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2021-04-07 09:13:55 +02:00
3aed84560f
Merge of nested IF-IF cases - O-R ( #48371 )
2021-03-27 10:38:57 +01:00
c820dd4cb5
Have pylint warn when user visible log messages do not start with capital letter or end with a period ( #48064 )
...
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-03-19 09:26:36 -05:00
a3cd1854f6
Update typing 12 ( #48073 )
2021-03-18 14:31:38 +01:00
61a2460c87
Improve error reporting in recorder purge test ( #47929 )
2021-03-14 19:46:21 -07:00
33c4eb3434
Log the full exception when the recorder fails to setup ( #47770 )
2021-03-11 21:52:04 -08:00
92852b9c10
Add apply_filter attribute to recorder.purge service ( #45826 )
2021-03-11 17:03:30 -10:00
10848b9bdf
Recorder improvements ( #47739 )
2021-03-11 07:52:07 -10:00
7c8851264f
Use LONGTEXT column instead of TEXT for MySQL/MariaDB and migrate existing databases ( #47026 )
2021-03-10 08:12:58 -10:00
a060acc2b1
Fix recorder with MSSQL ( #46678 )
...
Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
2021-03-09 11:16:19 -10:00
44293a3738
Add enable and disable services for recorder ( #45778 )
2021-02-24 07:26:05 -06:00
ea4bbd771f
Add service names to previously enriched services ( #46929 )
...
Co-authored-by: Tobias Sauerwein <cgtobi@users.noreply.github.com>
2021-02-23 14:10:13 +01:00
9b69549f73
Recover and restart the recorder if the sqlite database encounters corruption while running ( #46612 )
2021-02-19 21:26:24 -10:00
22dbac259b
Ensure recorder shuts down cleanly on restart before startup is finished ( #46604 )
2021-02-19 22:18:21 -08:00
8b69608242
Add selectors to Browser, Recorder, Shopping List service definitions ( #46749 )
2021-02-18 17:06:25 +01:00
c9df42b69a
Add support for pre-filtering events to the event bus ( #46371 )
2021-02-14 09:42:55 -10:00
dc26fd5149
Ensure creating an index that already exists is forgiving for postgresql ( #46185 )
...
Unlikely sqlite and mysql, postgresql throws ProgrammingError instead
of InternalError or OperationalError when trying to create an index
that already exists.
2021-02-08 23:22:38 +01:00
60d4dadcb6
Upgrade sqlalchemy to 1.3.23 ( #45845 )
2021-02-01 16:03:51 -10:00
12af87bc6e
Add index to old_state_id column for postgres and older databases ( #44757 )
...
* Add index to old_state_id column for older databases
The schema was updated in #43610 but the index was not
added on migration.
* Handle postgresql missing ondelete
* create index first
2021-01-04 10:51:44 +01:00
12b7b2098d
Upgrade sqlalchemy to 1.3.22 ( #44698 )
2020-12-31 14:28:15 -10:00
de04a1ed67
Enable more Bandit tests ( #44307 )
...
https://bandit.readthedocs.io/en/latest/plugins/index.html#complete-test-plugin-listing
2020-12-19 12:35:13 +01:00
2cbb93be43
Always keep the current recorder run when purging ( #43733 )
2020-11-30 09:13:50 +01:00
1162d9a752
Create tables with a charset that can hold all expected data under mysql ( #43732 )
...
By default these tables are created with utf8 which can only hold 3 bytes. This
meant that all emjoi would trigger a MySQLdb._exceptions.OperationalError because
they are 4 bytes.
This will only fix the issue for users who recreate their tables.
2020-11-28 23:54:05 +01:00
337b8d279e
Ensure MariaDB/MySQL can be purged and handle states being deleted out from under the recorder ( #43610 )
...
* MariaDB doesn't purge #42402
This addresses home-assistant#42402
Relationships within table "states" and between tables "states" and "events " home-assistant#40467 prevent the purge from working correctly. The database increases w/o any purge.
This proposal sets related indices to NULL and permits deleting of rows.
Further explanations can be found here home-assistant#42402
This proposal also allows to purge the tables "events" and "states" in any order.
* Update models.py
Corrected for Black style requirements
* Update homeassistant/components/recorder/models.py
Co-authored-by: J. Nick Koston <nick@koston.org>
* Add the options to foreign key constraints
* purge old states when database gets deleted out from under us
* pylint
Co-authored-by: J. Nick Koston <nick@koston.org>
2020-11-28 08:42:29 -10:00
5dbb5f12eb
Upgrade sqlalchemy to 1.3.20 ( #41765 )
2020-10-13 08:43:58 -05:00
33b548e247
Reduce ORM overhead when the old state was already written to the database ( #41736 )
...
We can avoid processing the relationship when the old state was already
written to the database which will common case with a commit interval
of 1s. Since we already know the value for old_state_id we can use it
instead of asking sqlalchemy to process the relationship at flush/commit
time which can significantly speed up sqlalchemy's _emit_insert_statements
implementation.
2020-10-13 15:34:46 +02:00
93a9a11065
Add config validator helper positive_float ( #41640 )
2020-10-11 22:04:49 +02:00
1c431aa7bd
Set created field when creating db events and states ( #41523 )
...
Avoids thousands of utcnow calls when the session
is commited on busy systems to fill in the field
default.
2020-10-09 09:31:17 +02:00
4ab02cb9bc
Ensure recorder commit can retry after encountering invalid data ( #41426 )
2020-10-08 09:15:25 +02:00
113d738fa2
Reduce orm overhead by grouping object expiration ( #41394 )
2020-10-07 15:35:48 +02:00
d35e33610a
Resolve memory leak in recorder ( #41349 )
...
Avoids a build up of the InstanceState.
2020-10-06 20:12:47 +02:00
4798f37c6e
Convert States to dicts via as_dict only once ( #41208 )
...
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2020-10-05 09:18:57 -05:00
20a136e2a1
Avoid event data serialization during recorder that we throw away ( #41217 )
...
We currently serialize the event data for state change events
and then replace it because we save the state in the states table.
Since the old state and new state are both contains in the event
the cost of serializing the data has a noticable impact when there
are many state changed events.
2020-10-05 15:08:47 +02:00
6b509fd9db
Prevent sqlalchemy from refetching the old_state_id as it will never change ( #40982 )
...
Disable expire_on_commit for the event writer. Since we never expect the
old_state_id to change in the database, it was never worth the expense of
refetching the id after the commit.
2020-10-01 19:57:52 +02:00
d9ba32dc3f
Setup recorder model relationships to avoid calling flush ( #40467 )
2020-09-30 13:11:43 +02:00
aada6a1d88
Fix logbook with empty filter and remove unused code ( #40565 )
2020-09-30 13:10:11 +02:00
c19b5c5ac3
Make recorder block_till_done reliable ( #40043 )
2020-09-14 08:48:29 +02:00
557684c3ce
Add ability to disable the sqlite3 quick_check ( #39479 )
2020-09-02 10:12:56 +02:00
1c2ebdf307
Upgrade black to 20.8b1 ( #39287 )
2020-08-27 13:56:20 +02:00
Erik Montnemery
J. Nick Koston
Paulus Schoutsen
Bram Kragten
Franck Nijhof
Ville Skyttä
PeteBa
Raman Gupta
tkdrob
Marcin Ciupak
jan iversen
Laszlo Magyar
Stefan Agner
Marc Mueller
CurrentThread
adrian-vlad
Fabian Affolter
moinmoin-sh
Matthias Alphart