* Fix cpu thrashing during purge after all legacy events were removed
We now remove the the index of of event ids on the states table when its
all NULLs to save space. The purge path needs to avoid checking for legacy
rows to purge if the index has been removed since it will result in a full
table scan each purge cycle that will always find no legacy rows to purge
* one more place
* drop the key constraint as well
* fixes
* more sqlite
* refactor to make StatesMetaManager threadsafe
* refactor to make StatesMetaManager threadsafe
* refactor to make StatesMetaManager threadsafe
* refactor to make StatesMetaManager threadsafe
* reduce
* comments
Drop duplicated indices from schema
https://docs.percona.com/percona-toolkit/pt-duplicate-key-checker.html
```
% pt-duplicate-key-checker --databases fresh
ALTER TABLE `fresh`.`events` DROP INDEX `ix_events_event_type_id`;
ALTER TABLE `fresh`.`states` DROP INDEX `ix_states_metadata_id`;
ALTER TABLE `fresh`.`statistics` DROP INDEX `ix_statistics_metadata_id`;
ALTER TABLE `fresh`.`statistics_short_term` DROP INDEX `ix_statistics_short_term_metadata_id`;
```
* Split context id migration into states and events tasks
Since events can finish much earlier than states we
would keep looking at the table because states as not
done. Make them seperate tasks
* add retry dec
* fix migration happening twice
* another case
* Deduplicate event_types in the events table
* Deduplicate event_types in the events table
* more fixes
* adjust
* adjust
* fix product
* fix tests
* adjust
* migrate
* migrate
* migrate
* more test fixes
* more test fixes
* fix
* migration test
* adjust
* speed up
* fix index
* fix more tests
* handle db failure
* preload
* tweak
* adjust
* fix stale docs strings, remove dead code
* refactor
* fix slow tests
* coverage
* self join to resolve query performance
* fix typo
* no need for quiet
* no need to drop index already dropped
* remove index that will never be used
* drop index sooner as we no longer use it
* Revert "remove index that will never be used"
This reverts commit 461aad2c52.
* typo
* Fix data migration never finishing when database has invalid datetimes
If there were impossible datetime values in the database (likely
from a manual sqlite to MySQL conversion) the conversion would
never complete
* Update homeassistant/components/recorder/migration.py
* Chunk MariaDB data migration to avoid running out of buffer space
This will make the migration slower but since the innodb_buffer_pool_size
is using the defaul to 128M and not tuned to the db size there is a
risk of running out of buffer space for large databases
* Update homeassistant/components/recorder/migration.py
* hard code since bandit thinks its an injection
* Update homeassistant/components/recorder/migration.py
* guard against manually modified data/corrupt db
* adjust to 10k per chunk
* adjust to 50k per chunk
* memory still just fine at 250k
* but slower
* commit after each chunk to reduce lock pressure
* adjust
* set to 0 if null so we do not loop forever (this should only happen if the data is missing)
* set to 0 if null so we do not loop forever (this should only happen if the data is missing)
* tweak
* tweak
* limit cleanup
* lower limit to give some more buffer
* lower limit to give some more buffer
* where required for sqlite
* sqlite can wipe as many as needed with no limit
* limit on mysql only
* chunk postgres
* fix limit
* tweak
* fix reference
* fix
* tweak for ram
* postgres memory reduction
* defer cleanup
* fix
* same order
* Validate common statistics db schema errors on start
* Fix test
* Add tests
* Adjust tests
* Disable statistics schema validation in tests
* Update after rebase
* Separate recorder database schema from other classes
* fix logbook imports
* migrate new tests
* few more
* last one
* fix merge
Co-authored-by: J. Nick Koston <nick@koston.org>
J. Nick Koston
Erik Montnemery
Franck Nijhof
Dmitry Katsubo