* Auto repair incorrect collation on MySQL schema
As we do more union queries in 2023.5.x if there is a mismatch
between collations on tables, they will fail with an error
that is hard for the user to figure out how to fix
`Error executing query: (MySQLdb.OperationalError) (1271, "Illegal mix of collations for operation UNION")`
This was reported in the #beta channel and by PM from others
so the problem is not isolated to a single user
https://discord.com/channels/330944238910963714/427516175237382144/1100908739910963272
* test with ascii since older maraidb versions may not work otherwise
* Revert "test with ascii since older maraidb versions may not work otherwise"
This reverts commit 787fda1aefcd8418a28a8a8f430e7e7232218ef8.t
* older version need to check collation_server because the collation is not reflected if its the default
* Speed up logbook and history queries where ORM rows are not needed
This avoids having sqlalchemy wrap Result in ChunkedIteratorResult
which has additional overhead we do not need for these cases
* more places
* anything that uses _sorted_statistics_to_dict does not need orm rows either
* Fallback to generating a new ULID on migraiton if context is missing or invalid
It was discovered that postgresql will do a full scan if
there is a low cardinality on the index because of missing
context ids. We will now generate a ULID for the timestamp
of the row if the context data is missing or invalid
fixes#91514
* tests
* tweak
* tweak
* preen
* Ensure recorder run shutdown if the run loop raises
If anything goes wrong with the recorder we should
still try to shutdown cleanly
* tweak
* tests
* tests
* handle migraiton failure
* tweak comment
* naming
* order
* order
* order
* reword
* adjust test
* fixes
* threading
* failure case
* fix test
* have to wait for stop because the task blocks on thread join
* delete more code
* tweak
* tweak
* wrappers
* restore lost performance
* restore lost performance
* restore lost performance
* compact
* reduce
* fix refactor
* DRY
* tweak
* delete the start time state injector
* move away the legacy code
* tweak
* adjust
* adjust
* tweak
* ignore impossible
* fix a bug where the first start was changed to the start time when there was no previous history recorded before
* avoid the empty scan most cases
* postgresql
* fixes
* workaround for mariadb < 10.4
* remove unused
* remove unused
* adjust
* bail early
* tweak
* tweak
* fix more tests
* fix recorderrun being init in the future in the test
* run history tests on schema 30 as well
* Revert "run history tests on schema 30 as well"
This reverts commit d798b100ac.
* reduce
* cleanup
* tweak
* reduce
* prune
* adjust
* adjust
* adjust
* reverse later is faster because the index is in forward order and the data size we are reversing is much smaller even if we are in python code
* Revert "reverse later is faster because the index is in forward order and the data size we are reversing is much smaller even if we are in python code"
This reverts commit bf974e103e.
* fix test
* Revert "Revert "reverse later is faster because the index is in forward order and the data size we are reversing is much smaller even if we are in python code""
This reverts commit 119354499e.
* more coverage
* adjust
* fix for table order
* impossible for it to be missing
* remove some more legacy from the all states
* Reduce overhead of legacy database columns on new installs
* Reduce overhead of legacy database columns on new installs
* Reduce overhead of legacy database columns on new installs
* Reduce overhead of legacy database columns on new installs
* not working as expected
* override the type compiler
* override the type compiler
* override the type compiler
* override the type compiler
* Apply suggestions from code review
* pgsql char1
* make entity filter test setup with old schema
* fix some more tests that were mutating state
* fix some more tests that were mutating state
* fix some more tests that were mutating state
* fix more dbstate mutations
* add shim for older tests
* split migration tests
* add coverage for purging legacy data
* tweak
* more fixes
* drop some legacy
* fix another test
* fix a few more
* add casts for postgresql in case someone deletes the schema changes table
* dry
* dry
* dry
* Restart entity id post migration after a restart
If the entity migration finished and Home Assistant was
restarted during the post migration it would never be resumed
which means the old index and space would never be recovered
* add migration resume test
* Add a faster query for get_last_state_changes when the number of states is 1
related issue #90113
* Add a faster query for get_last_state_changes when the number of states is 1
related issue #90113
* coverage
* Apply suggestions from code review
* Add auto repairs for events schema
* Add auto repairs for events schema
* Add auto repairs for events schema
* Add auto repairs for events schema
* Add auto repairs for events schema
* fix bug - wrong table
* Fix index not being dropped on postgresql databases with a schema prefix
Added logging in case index drops fail so we can tell
why in the future
* coverage
* Bump PyJWT to 2.6.0
* fix time being frozen too late which makes the access token creation time in the future
* revert zha change
* fix repairs test
* fix ical test
* Break out statistics schema repairs into a repairs module
A future PR will add repairs for events, states, etc
* reorg
* reorg
* reorg
* reorg
* fixes
* fix patch targets
* name space rename
If a user manually migrated their database to MySQL or PostgresSQL
and incorrectly created the timestamp columns as float we would
fail to correct them to double because when we migrated to use
timestamps for the columns I missed that we needed to change the
columns and types for µs precision
- If the user had previously duplicated data we could end up
picking the next metadata_id and there could be stale rows
in the database that have that metadata_id. This can only happen
from bad manual migrations (which is what this is function
is validating in the first place). To solve this we now insert
data with a future date and look at the latest inserted row
instead of the first.
Example
```
['stored_statistics',
defaultdict(<class 'list'>,
{'recorder.db_test_schema': [{'end': 948589200.0,
'last_reset': None,
'max': None,
'mean': 2021.0,
'min': None,
'start': 948585600.0,
'state': None,
'sum': 394.5068},
{'end': 1601946000.000001,
'last_reset': 1601942400.000001,
'max': 1.000000000000001,
'mean': 1.000000000000001,
'min': 1.000000000000001,
'start': 1601942400.000001,
'state': 1.000000000000001,
'sum': 1.000000000000001}]})]
```
* Fix cpu thrashing during purge after all legacy events were removed
We now remove the the index of of event ids on the states table when its
all NULLs to save space. The purge path needs to avoid checking for legacy
rows to purge if the index has been removed since it will result in a full
table scan each purge cycle that will always find no legacy rows to purge
* one more place
* drop the key constraint as well
* fixes
* more sqlite
* Avoid database executor job to fetch statistic metadata on cache hit
Since we will almost always have a cache hit fetching
statistic meta data we can avoid an executor job
* Avoid database executor job to fetch statistic metadata on cache hit
Since we will almost always have a cache hit fetching
statistic meta data we can avoid an executor job
* Avoid database executor job to fetch statistic metadata on cache hit
Since we will almost always have a cache hit fetching
statistic meta data we can avoid an executor job
* remove exception catch since the threading.excepthook will actually catch this in production
* fix a few missed ones
* threadsafe
* Update homeassistant/components/recorder/table_managers/statistics_meta.py
* coverage and optimistic caching
* refactor to make StatesMetaManager threadsafe
* refactor to make StatesMetaManager threadsafe
* refactor to make StatesMetaManager threadsafe
* refactor to make StatesMetaManager threadsafe
* reduce
* comments
J. Nick Koston
epenet