Unload index before closing shard

When deleting a shard, the shard is locked and then removed from the
index.  Removal from the index can be slow if there are a lot of
series.  During this time, the shard is still expected to exist by
the meta store and tsdb store so stats collections, queries and writes
could all be run on this shard while it's locked.  This can cause everything
to lock up until the unindexing completes and the shard can be unlocked.

Fixes #7226

CHANGELOG.md

@@ -8,6 +8,16 @@
 - [#7299](https://github.com/influxdata/influxdb/ssues/7299): Ensure fieldsCreated stat available in shard measurement.
 - [#6846](https://github.com/influxdata/influxdb/issues/6846): Read an invalid JSON response as an error in the influx client.
 - [#7110](https://github.com/influxdata/influxdb/issues/7110): Skip past points at the same time in derivative call within a merged series.
+- [#1834](https://github.com/influxdata/influxdb/issues/1834): Drop time when used as a tag or field key.
+- [#7152](https://github.com/influxdata/influxdb/issues/7152): Decrement number of measurements only once when deleting the last series from a measurement.
+- [#7177](https://github.com/influxdata/influxdb/issues/7177): Fix base64 encoding issue with /debug/vars stats.
+- [#7196](https://github.com/influxdata/influxdb/issues/7196): Fix mmap dereferencing, fixes #7183, #7180
+- [#7013](https://github.com/influxdata/influxdb/issues/7013): Fix the dollar sign so it properly handles reserved keywords.
+- [#7297](https://github.com/influxdata/influxdb/issues/7297): Use consistent column output from the CLI for column formatted responses.
+- [#7231](https://github.com/influxdata/influxdb/issues/7231): Duplicate parsing bug in ALTER RETENTION POLICY.
+- [#7285](https://github.com/influxdata/influxdb/issues/7285): Correctly use password-type field in Admin UI. Thanks @dandv!
+- [#2792](https://github.com/influxdata/influxdb/issues/2792): Exceeding max retention policy duration gives incorrect error message
+- [#7226](https://github.com/influxdata/influxdb/issues/7226): Fix database locked up when deleting shards
 
 ## v1.0.0 [2016-09-07]
 

tsdb/shard.go

@@ -262,6 +262,12 @@ func (s *Shard) Open() error {
 	return nil
 }
 
+// UnloadIndex removes all references to this shard from the DatabaseIndex
+func (s *Shard) UnloadIndex() {
+	// Don't leak our shard ID and series keys in the index
+	s.index.RemoveShard(s.id)
+}
+
 // Close shuts down the shard's store.
 func (s *Shard) Close() error {
 	s.mu.Lock()
@@ -282,7 +288,7 @@ func (s *Shard) close() error {
 	}
 
 	// Don't leak our shard ID and series keys in the index
-	s.index.RemoveShard(s.id)
+	s.UnloadIndex()
 
 	err := s.engine.Close()
 	if err == nil {

tsdb/store.go

@@ -357,6 +357,11 @@ func (s *Store) DeleteShard(shardID uint64) error {
 		return nil
 	}
 
+	// Remove the shard from the database indexes before closing the shard.
+	// Closing the shard will do this as well, but it will unload it while
+	// the shard is locked which can block stats collection and other calls.
+	sh.UnloadIndex()
+
 	if err := sh.Close(); err != nil {
 		return err
 	}