Home-Assistant
/
core
mirror of https://github.com/home-assistant/core.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
core/homeassistant/components
History
Greg Laabs 6e1785173f History query and schema optimizations for huge performance boost (#8748) 
* Add DEBUG-level log for db row to native object conversion

This is now the bottleneck (by a large margin) for big history queries, so I'm leaving this log feature in to help diagnose users with a slow history page

* Rewrite of the "first synthetic datapoint" query for multiple entities

The old method was written in a manner that prevented an index from being used in the inner-most GROUP BY statement, causing massive performance issues especially when querying for a large time period.

The new query does have one material change that will cause it to return different results than before: instead of using max(state_id) to get the latest entry, we now get the max(last_updated). This is more appropriate (primary key should not be assumed to be in order of event firing) and allows an index to be used on the inner-most query. I added another JOIN layer to account for cases where there are two entries on the exact same `last_created` for a given entity. In this case we do use `state_id` as a tiebreaker.

For performance reasons the domain filters were also moved to the outermost query, as it's way more efficient to do it there than on the innermost query as before (due to indexing with GROUP BY problems)

The result is a query that only needs to do a filesort on the final result set, which will only be as many rows as there are entities.

* Remove the ORDER BY entity_id when fetching states, and add logging

Having this ORDER BY in the query prevents it from using an index due to the range filter, so it has been removed.

We already do a `groupby` in the `states_to_json` method which accomplishes exactly what the ORDER BY in the query was trying to do anyway, so this change causes no functional difference.

Also added DEBUG-level logging to allow diagnosing a user's slow history page.

* Add DEBUG-level logging for the synthetic-first-datapoint query

For diagnosing a user's slow history page

* Missed a couple instances of `created` that should be `last_updated`

* Remove `entity_id` sorting from state_changes; match significant_update

This is the same change as 09b3498f41 , but applied to the `state_changes_during_period` method which I missed before. This should give the same performance boost to the history sensor component!

* Bugfix in History query used for History Sensor

The date filter was using a different column for the upper and lower bounds. It would work, but it would be slow!

* Update Recorder purge script to use more appropriate columns

Two reasons: 1. the `created` column's meaning is fairly arbitrary and does not represent when an event or state change actually ocurred. It seems more correct to purge based on the event date than the time the database row was written.
2. The new columns are indexed, which will speed up this purge script by orders of magnitude

* Updating db model to match new query optimizations

A few things here: 1. New schema version with a new index and several removed indexes
2. A new method in the migration script to drop old indexes
3. Added an INFO-level log message when a new index will be added, as this can take quite some time on a Raspberry Pi
 		2017-08-04 23:16:53 -07:00
..
alarm_control_panel Manual alarm with MQTT control (#8257) 2017-07-24 09:06:38 -07:00
automation Remove deprecated automation keywords (#8510) 2017-07-17 22:24:05 +02:00
binary_sensor Remove deprecated sensor_class config options (#8702) 2017-07-29 19:46:27 -04:00
calendar Fix pylint 1.7.2 no-else-return issues (#8361) 2017-07-05 23:30:01 -07:00
camera Enable/Disable Motion detection for Foscam Cameras (#8582) 2017-07-31 23:14:34 -07:00
climate Fixes UnboundLocalError: local variable 'setting' referenced before assignment (#8782) 2017-08-02 09:14:28 +02:00
config Add support for automation config panel (#7509) 2017-05-09 18:44:00 -07:00
cover added invert_state optional parameter (#8695) 2017-07-31 20:24:21 -04:00
device_tracker Fix Lint 2017-07-28 20:53:15 -07:00
emulated_hue Handle manual edits to emulated_hue_ids.json (#8560) 2017-07-19 22:51:50 -07:00
fan Fix Dyson sensors if devices are configured without standby monitoring. Fixes #8569 (#8826) 2017-08-04 14:27:23 -07:00
frontend Update frontend 2 2017-08-04 23:02:07 -07:00
http Fix typos (#7957) 2017-06-08 15:53:12 +02:00
image_processing Update numpy 1.13.1 (#8806) 2017-08-03 11:36:50 +02:00
light Block dependencies that depend on enum34 (#8698) 2017-08-04 23:06:10 -07:00
lock Add bind_hass to components (#8502) 2017-07-16 10:14:46 -07:00
media_player When Sonos gets a tts source - dont't show an image (#8777) 2017-08-03 17:39:11 +03:00
mqtt MQTT Switch: add availability_topic for online/offline status (#8593) 2017-07-27 08:24:15 +02:00
notify Upgrade pyasn1 to 0.3.1 and pyasn1-modules to 0.0.10 (#8787) 2017-08-02 14:14:01 +02:00
recorder History query and schema optimizations for huge performance boost (#8748) 2017-08-04 23:16:53 -07:00
remote Clean up remote component (#8728) 2017-07-31 20:52:39 -07:00
scene Added Lutron Caseta Scene Support (#8690) 2017-07-29 18:07:28 +02:00
sensor Block dependencies that depend on enum34 (#8698) 2017-08-04 23:06:10 -07:00
switch Xiaomi vacuum as platform of new `vacuum` component derived from ToggleEntity, and services (#8623) 2017-08-04 15:27:10 +02:00
telegram_bot Xiaomi vacuum as platform of new `vacuum` component derived from ToggleEntity, and services (#8623) 2017-08-04 15:27:10 +02:00
tts Fix TTS options. #8375 (#8376) 2017-07-06 22:20:49 -07:00
vacuum Xiaomi vacuum as platform of new `vacuum` component derived from ToggleEntity, and services (#8623) 2017-08-04 15:27:10 +02:00
weather Update yweather.py (#8820) 2017-08-04 17:22:38 +02:00
zha zha: Update to bellows 0.3.4 (#8594) 2017-07-23 23:35:49 -07:00
zwave Fix Z-Wave barrier discovery for new API (#8706) 2017-07-30 00:40:56 -04:00
__init__.py Persistent notification import (#8507) 2017-07-16 21:39:38 +02:00
alarmdecoder.py Upgrade the alarmdecoder dependency library from 0.12.1 to 0.12.3. (#8542) 2017-07-19 12:21:39 +02:00
alert.py Fix some issues for PyLint 1.7.2 (#8356) 2017-07-05 20:02:16 -07:00
alexa.py Feature alexa launch request (#8730) 2017-08-01 22:53:36 -07:00
amcrest.py Bumped Amcrest version (#8624) 2017-07-23 23:46:35 -07:00
android_ip_webcam.py Add android ip webcam support for aiohttp2 (#6940) 2017-04-04 21:40:19 -07:00
apcupsd.py Update apcaccess to 0.0.13. Add "Percent Load Capacity" to INFERRED_UNITS. (#8277) 2017-07-02 11:24:07 +02:00
api.py Fix some issues for PyLint 1.7.2 (#8356) 2017-07-05 20:02:16 -07:00
apiai.py Add Intent component (#8434) 2017-07-21 21:38:53 -07:00
apple_tv.py Fix support for multiple Apple TVs (#8539) 2017-07-23 23:35:48 -07:00
arduino.py Supress trackback and upgrade PyMata to 2.14 (#7176) 2017-04-19 12:48:15 +02:00
arlo.py Persistent notification import (#8507) 2017-07-16 21:39:38 +02:00
axis.py Persistent notification import (#8507) 2017-07-16 21:39:38 +02:00
bbb_gpio.py Update docstrings (#7361) 2017-04-29 22:04:49 -07:00
blink.py Blink version bump (#7571) 2017-05-12 20:30:07 -07:00
bloomsky.py Update docstrings (#7361) 2017-04-29 22:04:49 -07:00
browser.py
…
comfoconnect.py Update docstrings (#8244) 2017-06-29 11:44:35 +02:00
configurator.py Persistent notification import (#8507) 2017-07-16 21:39:38 +02:00
conversation.py Upgrade fuzzywuzzy to 0.15.1 (#8671) 2017-07-27 17:22:40 +02:00
datadog.py Add datadog component (#7158) 2017-05-06 00:34:40 +02:00
demo.py Update demo.py 2017-07-18 15:11:17 -07:00
device_sun_light_trigger.py Refactor sun component for correctness (#7295) 2017-05-09 00:03:34 -07:00
digital_ocean.py Use 'hass.data' instead of global (#8245) 2017-06-30 08:46:03 +02:00
discovery.py Make HA discover sabnzbd and add it to the Configurator (#8634) 2017-08-04 23:24:55 +02:00
downloader.py Don't use len(SEQUENCE) as condition value (#7249) 2017-04-23 20:41:09 -07:00
dweet.py Upgrade dweepy to 0.3.0 (#7550) 2017-05-11 22:48:03 +02:00
dyson.py Fix Dyson sensors if devices are configured without standby monitoring. Fixes #8569 (#8826) 2017-08-04 14:27:23 -07:00
ecobee.py update for pypi (#7430) 2017-05-04 07:48:43 +02:00
eight_sleep.py Bump pyEight version to fix 0hr session errors (#7916) 2017-06-05 17:44:13 +02:00
emoncms_history.py Don't use len(SEQUENCE) as condition value (#7249) 2017-04-23 20:41:09 -07:00
enocean.py Update docstrings (#7405) 2017-05-02 22:47:20 +02:00
envisalink.py Bump pyenvisalink to version 2.1 (#7803) 2017-05-29 10:27:36 +02:00
feedreader.py Don't use len(SEQUENCE) as condition value (#7249) 2017-04-23 20:41:09 -07:00
ffmpeg.py Replace executor with async_add_job (#7658) 2017-05-26 08:28:07 -07:00
foursquare.py Update docstrings (#7361) 2017-04-29 22:04:49 -07:00
google.py Persistent notification import (#8507) 2017-07-16 21:39:38 +02:00
graphite.py Update docstrings (#7361) 2017-04-29 22:04:49 -07:00
group.py Polymer 2 (#8815) 2017-08-03 23:46:57 -07:00
hassio.py Hass.io: logo support / timeout handling (#8668) 2017-07-27 12:23:22 -07:00
hdmi_cec.py Xiaomi vacuum as platform of new `vacuum` component derived from ToggleEntity, and services (#8623) 2017-08-04 15:27:10 +02:00
history.py History query and schema optimizations for huge performance boost (#8748) 2017-08-04 23:16:53 -07:00
homematic.py Upgrade pyhomematic to 0.1.30 (#8673) 2017-07-27 17:45:59 +02:00
ifttt.py Update docstrings (#7361) 2017-04-29 22:04:49 -07:00
influxdb.py InfluxDB component improvements (#8633) 2017-08-03 16:26:01 +02:00
input_boolean.py Add bind_hass to components (#8502) 2017-07-16 10:14:46 -07:00
input_select.py Add bind_hass to components (#8502) 2017-07-16 10:14:46 -07:00
input_slider.py Add bind_hass to components (#8502) 2017-07-16 10:14:46 -07:00
insteon_local.py Fix some issues for PyLint 1.7.2 (#8356) 2017-07-05 20:02:16 -07:00
insteon_plm.py python-insteonplm module version bump (#8736) 2017-07-31 14:33:51 +02:00
intent_script.py Add Intent component (#8434) 2017-07-21 21:38:53 -07:00
introduction.py Update docstrings (#7361) 2017-04-29 22:04:49 -07:00
ios.py Start of migration framework, to allow moving of files in the config … (#7740) 2017-06-01 23:50:04 -07:00
isy994.py Update docstrings (#7361) 2017-04-29 22:04:49 -07:00
joaoapps_join.py Update join (#7443) 2017-05-04 13:14:14 -07:00
juicenet.py Add juicenet platform (#7668) 2017-06-05 08:39:31 -07:00
keyboard.py
…
keyboard_remote.py Check if /dev/input/by-id exists (#8601) 2017-07-23 23:47:52 -07:00
kira.py Add Kira component to sensor and remote platforms (#7479) 2017-05-12 21:12:47 -07:00
knx.py Added a service to write to KNX group addressed including documentation (#8491) 2017-07-20 07:01:05 +02:00
lametric.py LaMetric platform and notify module (#8230) 2017-07-07 08:21:06 +02:00
lirc.py Update docstrings (#7361) 2017-04-29 22:04:49 -07:00
litejet.py Update docstrings (#7361) 2017-04-29 22:04:49 -07:00
logbook.py Fix some issues for PyLint 1.7.2 (#8356) 2017-07-05 20:02:16 -07:00
logentries.py Update docstrings (#7361) 2017-04-29 22:04:49 -07:00
logger.py Replace executor with async_add_job (#7658) 2017-05-26 08:28:07 -07:00
lutron.py Fix typo (sending USERNAME instead of PASSWORD) introduced in #7963 (#8433) 2017-07-10 20:52:37 -07:00
lutron_caseta.py Added Lutron Caseta Scene Support (#8690) 2017-07-29 18:07:28 +02:00
mailgun.py Update mailgun (#7984) 2017-06-11 22:19:10 -07:00
maxcube.py Remove superfluous comments and update ordering (#7227) 2017-04-22 21:13:04 +02:00
media_extractor.py Upgrade youtube_dl to 2017.7.23 (#8617) 2017-07-23 20:20:23 +02:00
microsoft_face.py Replace executor with async_add_job (#7658) 2017-05-26 08:28:07 -07:00
mochad.py Add mochad light component (#8476) 2017-08-03 10:51:01 +02:00
modbus.py Modbus fixes to work with pymodbus 1.3.1 (#8365) 2017-07-06 22:30:23 -07:00
mqtt_eventstream.py Update docstrings (#7361) 2017-04-29 22:04:49 -07:00
mysensors.py Upgrade pymysensors to 0.10.0 (#7469) 2017-05-06 19:10:17 -07:00
neato.py Update docstrings (#7361) 2017-04-29 22:04:49 -07:00
nest.py Add service to set nest away/home modes (#7619) 2017-06-04 23:45:24 -07:00
netatmo.py Fixed the Wind sensor following new release of netatmo-api-python (#8030) 2017-06-15 22:14:46 -07:00
nuimo_controller.py Clean up requirements (#7391) 2017-05-02 09:15:02 -07:00
octoprint.py Prevent errors on Octoprint sensors and binary_sensors when Octoprint and/or Printer are off (#8343) 2017-07-06 22:39:11 -07:00
panel_custom.py
…
panel_iframe.py Update docstrings (#7361) 2017-04-29 22:04:49 -07:00
persistent_notification.py Add bind_hass to components (#8502) 2017-07-16 10:14:46 -07:00
pilight.py Update docstrings (#7405) 2017-05-02 22:47:20 +02:00
plant.py Fix some issues for PyLint 1.7.2 (#8356) 2017-07-05 20:02:16 -07:00
prometheus.py prometheus: Fix zwave battery level (#8615) 2017-07-23 23:49:22 -07:00
proximity.py Update docstrings (#7361) 2017-04-29 22:04:49 -07:00
python_script.py Enhance python_script to support "_getitem_" (#8541) 2017-07-19 22:56:24 -07:00
qwikswitch.py Fix some issues for PyLint 1.7.2 (#8356) 2017-07-05 20:02:16 -07:00
raspihats.py Add new raspihats component (#7392) 2017-05-05 00:02:47 -07:00
rest_command.py Update docstrings (#7361) 2017-04-29 22:04:49 -07:00
rflink.py Correct spelling of aliases, deprecate old config options. (#8348) 2017-07-06 15:59:54 +02:00
rfxtrx.py Fix broken status update for lighting4 devices (#8543) 2017-07-23 23:35:49 -07:00
ring.py Persistent notification import (#8507) 2017-07-16 21:39:38 +02:00
rpi_gpio.py Update docstrings (#7361) 2017-04-29 22:04:49 -07:00
rpi_pfio.py Support for the PiFace Digital I/O module (#7494) 2017-05-09 22:36:33 -07:00
rss_feed_template.py Remove ordered_dict validator (#7375) 2017-04-30 10:55:03 -07:00
script.py Add bind_hass to components (#8502) 2017-07-16 10:14:46 -07:00
scsgate.py Update docstrings (#7361) 2017-04-29 22:04:49 -07:00
services.yaml Added a service to write to KNX group addressed including documentation (#8491) 2017-07-20 07:01:05 +02:00
shell_command.py Update docstrings (#7361) 2017-04-29 22:04:49 -07:00
shiftr.py Upgrade paho-mqtt to 1.3.0 (#8125) 2017-06-21 00:32:04 +02:00
shopping_list.py Persist shopping list + clear completed (#8697) 2017-07-29 12:22:52 -07:00
sleepiq.py Update docstrings (#7361) 2017-04-29 22:04:49 -07:00
snips.py Add Intent component (#8434) 2017-07-21 21:38:53 -07:00
spc.py Add support for Vanderbilt SPC alarm panels and attached sensors (#7663) 2017-06-04 23:53:25 -07:00
splunk.py Added host variable to Splunk.py and updated tox tests (#8052) 2017-06-15 20:41:19 -07:00
statsd.py Adding support for mapping keys to value in statsd (#8665) 2017-07-27 08:58:34 +02:00
sun.py Initialize sun with correct values. (#7559) 2017-05-12 16:04:30 -07:00
tado.py pytado moved to pypi (#8298) 2017-07-02 12:54:59 -07:00
tellduslive.py Fix some issues for PyLint 1.7.2 (#8356) 2017-07-05 20:02:16 -07:00
tellstick.py Update docstrings (#7361) 2017-04-29 22:04:49 -07:00
thingspeak.py Upgrade thingspeak to 0.4.1 and use the correct API key (#5906) 2017-02-13 11:25:28 +01:00
tradfri.py Add support for disabling tradfri groups (#7593) 2017-05-16 23:26:57 -07:00
twilio.py Update docstrings (#6795) 2017-03-26 15:50:40 +02:00
updater.py Updater improvements to send option component information (#7720) 2017-06-15 22:29:18 -07:00
upnp.py Persistent notification import (#8507) 2017-07-16 21:39:38 +02:00
velbus.py Velbus (#8076) 2017-07-26 14:03:29 +02:00
velux.py Fixed link to documentation (#8424) 2017-07-10 12:11:16 +02:00
vera.py Bump pyver to fix exception in fetching colours. 2017-07-20 10:21:08 +01:00
verisure.py vsure 1.3.7 (#8321) 2017-07-04 06:26:55 +02:00
volvooncall.py Provide entity_id to avoid sensor mixup (fixes #7636). Use async_dispatcher. Provide icon. (#7946) 2017-06-15 22:28:30 -07:00
wake_on_lan.py New service `send_magic_packet` with new component `wake_on_lan` (#8397) 2017-07-10 18:37:51 -07:00
weblink.py Update docstrings (#7361) 2017-04-29 22:04:49 -07:00
websocket_api.py Fix some issues for PyLint 1.7.2 (#8356) 2017-07-05 20:02:16 -07:00
wemo.py Update docstrings (#7361) 2017-04-29 22:04:49 -07:00
wink.py Wink discovery (#8739) 2017-07-31 20:54:07 -07:00
xiaomi.py Device support for different new sensors of the xiaomi aqara gateway (#8577) 2017-07-21 10:13:42 +02:00
zabbix.py
…
zeroconf.py Upgrade zeroconf to 0.19.1 (#8043) 2017-06-15 21:25:19 +02:00
zigbee.py Update docstrings (#7361) 2017-04-29 22:04:49 -07:00
zone.py Add bind_hass to components (#8502) 2017-07-16 10:14:46 -07:00
zoneminder.py zoneminder: fix incorrect use of logging.exception. (#7675) 2017-05-21 11:11:33 +02:00