Unload index before closing shard

When deleting a shard, the shard is locked and then removed from the
index.  Removal from the index can be slow if there are a lot of
series.  During this time, the shard is still expected to exist by
the meta store and tsdb store so stats collections, queries and writes
could all be run on this shard while it's locked.  This can cause everything
to lock up until the unindexing completes and the shard can be unlocked.

Fixes #7226

CHANGELOG.md

@@ -24,6 +24,7 @@
 - [#7231](https://github.com/influxdata/influxdb/issues/7231): Duplicate parsing bug in ALTER RETENTION POLICY.
 - [#7285](https://github.com/influxdata/influxdb/issues/7285): Correctly use password-type field in Admin UI. Thanks @dandv!
 - [#2792](https://github.com/influxdata/influxdb/issues/2792): Exceeding max retention policy duration gives incorrect error message
+- [#7226](https://github.com/influxdata/influxdb/issues/7226): Fix database locked up when deleting shards
 
 ## v1.0.0 [2016-09-08]
 

tsdb/shard.go

@@ -263,6 +263,12 @@ func (s *Shard) Open() error {
 	return nil
 }
 
+// UnloadIndex removes all references to this shard from the DatabaseIndex
+func (s *Shard) UnloadIndex() {
+	// Don't leak our shard ID and series keys in the index
+	s.index.RemoveShard(s.id)
+}
+
 // Close shuts down the shard's store.
 func (s *Shard) Close() error {
 	s.mu.Lock()
@@ -283,7 +289,7 @@ func (s *Shard) close() error {
 	}
 
 	// Don't leak our shard ID and series keys in the index
-	s.index.RemoveShard(s.id)
+	s.UnloadIndex()
 
 	err := s.engine.Close()
 	if err == nil {

tsdb/store.go

@@ -357,6 +357,11 @@ func (s *Store) DeleteShard(shardID uint64) error {
 		return nil
 	}
 
+	// Remove the shard from the database indexes before closing the shard.
+	// Closing the shard will do this as well, but it will unload it while
+	// the shard is locked which can block stats collection and other calls.
+	sh.UnloadIndex()
+
 	if err := sh.Close(); err != nil {
 		return err
 	}