Commit Graph

204 Commits (986f2eaa7569433d37a420dc19299b3f0d61e4ba)

Author SHA1 Message Date
Jacob Marble ff82e844ae
feat(storage): Add StringIteratorWriter.WrittenN() (#13456) 2019-04-17 11:05:45 -07:00
Jacob Marble 53810fadeb
feat(storage): Implement storage schema RPC de/serializer, merge, APIs (#13409)
* Extend storage service protobuf with TagKeys and TagValues

Co-authored-by: Michael Desa <mjdesa@gmail.com>
Co-authored-by: Jacob Marble <jacobmarble@influxdata.com>

* Extend the reads.Store interface with new TagKeys and TagValues APIs

* Extend readservice.store to implement refactored reads.Store interface

* Implement a StringIterator gRPC writer / serializer

* Implement a StringIterator gRPC reader / deserializer

* Implement a StringIterator merger
2019-04-16 16:01:05 -07:00
Stuart Carnie 5a224c74ea
feat(storage): Stub storage schema APIs
Closes #13241
2019-04-12 12:09:46 -07:00
Jacob Marble f56c42794b
chore(tracing): Cleanup (#13296)
* chore(tracing): Cleanup

* broken test

* fix unused var

* fix test
2019-04-10 19:28:21 -07:00
kelwang be674622c6
Revert "fix(inmem): remove the old inmem implementation" 2019-04-09 14:24:40 -04:00
kelwang d0022dfd5c
Merge pull request #13039 from zhulongcheng/rm-inmem-impl
fix(inmem): remove the old inmem implementation
2019-04-09 13:06:50 -04:00
jlapacik 0cde401678 refactor(storage): update GetSource method of Store interface 2019-04-08 15:59:37 -07:00
jlapacik 8078b915fd refactor(storage): ReadFilter storage operation 2019-04-08 15:59:37 -07:00
zhulongcheng cacd6a8223 fix(inmem): replace inmem.Service with kv.Service 2019-04-08 15:18:38 +08:00
Ben Johnson 307bb6af9c
Improve bulk series file writes. 2019-04-05 14:38:58 -06:00
Jeff Wendling 5d594deb2d
Merge pull request #13094 from influxdata/jmw-protobuf-bump
chore: bump gogo/protobuf and regenerate
2019-04-04 14:35:38 -06:00
jlapacik 2909927ab0
fix(reads): HasFieldValueKey searches for "$", not "_value" (#13138)
Storage converts references to _value in filter predicates to $.
It then considers any predicate that does not reference $ to be
a tag predicate. Tag predicates are used to construct series index
cursors.

This commit fixes a bug where field comparisons were being included
in tag predicates due to the HasFieldValueKey function searching
for comparison expressions referencing _value instead of $. Because
all references to _value had already been replaced. An expression
of the form '$ < 3000' would be considered a tag expression and
therefore would be mistakenly included as a tag predicate.

Fixes #13159.
2019-04-04 11:13:09 -07:00
Jeff Wendling 5dc3e52fd9 chore: bump gogo/protobuf and regenerate
It had been bumped previously from v1.1.1 to v1.2.0 and nothing was
regenerated. This bumps it to v1.2.1 and regenerates.
2019-04-03 15:42:33 -06:00
Jeff Wendling 157c4fcf0c storage: add tests for retention metrics 2019-04-01 17:44:37 -06:00
Jeff Wendling 3e370b0f99 storage: make retention metrics global
In the case that there are multiple engines, we want the retention
metrics from all of them.
2019-04-01 17:44:37 -06:00
Jacob Marble 0b626beb53
chore(storage): Don't log every read request (#12881)
* don't log every read request

* check for err
2019-03-25 13:14:45 -07:00
Stuart Carnie 8abb76cb4e
Merge pull request #12710 from influxdata/sgc/data-gen
Add data generation subcommand to influxd
2019-03-20 11:08:49 -07:00
Lorenzo Affetti 7dbecfa256
fix(storage/reads): release arrow buffers when advancing on table (#12737) 2019-03-20 15:48:59 +01:00
Stuart Carnie fe8b63c10b
chore(gen): Update the gen package to produce 2.0 series keys 2019-03-19 20:35:02 -07:00
Stuart Carnie 07aef958f8
fix(reads): Interface to identify a remote streamed cursor
A cursor can be in process (local) or remote. Remote cursors have
already applied the `hasPoints` check, to reduce network traffic.
Testing whether the cursor has points again here:

https://github.com/influxdata/influxdb/blob/master/storage/reads/reader.go#L221

will always return `false` for a remote cursor that has applied the
NoPoints optimization.

This temporary fix allows the `hasPoints` function to differentiate
a streamCursor and always return true in that case.
2019-03-11 18:40:35 -07:00
Jacob Marble 603a1f26e0 use tracing.StartSpanFromContext 2019-03-07 12:12:31 -07:00
Jacob Marble 9541e861a3 goimports -w -local github.com/influxdata/influxdb 2019-03-07 12:12:31 -07:00
Jacob Marble 92fa813c45 add spans to multiple services 2019-03-07 12:12:31 -07:00
Christopher M. Wolff e28ecdc0e9
refactor(query): make queryd present ProxyQueryService (#12360)
Fixes influxdata/idpe#2014.
2019-03-07 07:32:13 -08:00
Edd Robinson 582ed6834c ddress PR feedback 2019-03-07 09:56:07 +00:00
Edd Robinson 098ec71919 Remove erroneous print 2019-03-07 09:56:07 +00:00
Edd Robinson 8bdf857ddb Fix expected flux cases 2019-03-07 09:56:07 +00:00
Edd Robinson 3f1bec0836 Update emitted keys and tests 2019-03-07 09:56:07 +00:00
Edd Robinson f21be142d1 Storage engine now validates all tags are utf-8
The storage engine will now drop any points that contain invalid tag
data. Special tag keys for the measurement and field key will be
excepted from this validation.
2019-03-07 09:56:07 +00:00
Edd Robinson 9949fef2a5 Only validate tag pairs are valid unicode
We will want to validate that all tag key/value data is valid unicode.
This commit changes the validation helper to only validate provided
tags, since measurements are currently very likely to contain invalid
utf-8 characters.

There are two exceptions to the tag validation: the validation of the
special tag keys for measurements and field keys.
2019-03-07 09:56:07 +00:00
Edd Robinson f029f1645d Change location and value for internal tag keys 2019-03-07 09:56:07 +00:00
Jonathan A. Sternberg 9549cc4d97
fix(storage/reads): track memory allocations when reading from storage (#12404) 2019-03-06 15:29:45 -06:00
Jeff Wendling f53f9cd949 storage: detect conflicting types in a single batch of points
When the WAL was moved up, the validation that happened at the cache
was skipped. This moves the field type validation for a batch of
points up ahead of the WAL again.
2019-03-06 10:30:52 -07:00
Stuart Carnie da24fe6c7f feedback(reads): Clear r.buf before read 2019-03-05 11:45:51 -07:00
Stuart Carnie acc94985e3 fix(reads): Add retry when fetching data from StreamReader
It is possible a StreamReader (gRPC) may return an empty response. This
change adds retry and bail-out support. When a bail-out occurs,
reads.ErrStreamNoData is returned.
2019-03-05 11:45:51 -07:00
Stuart Carnie 28976e9b92 feat: Allow StreamReader to provide cursors.CursorStats
StorageReadClient adapts a gRPC Storage_ReadClient to provide
cursors.CursorStats by reading the trailer after receiving the final
message from the stream.
2019-03-05 11:45:51 -07:00
Stuart Carnie 5f241e5e5e fix(reads): Read Statistics before calling Close 2019-03-05 11:45:51 -07:00
Jacob Marble b9c7ec439e
feat(influxd): Tracing refactor (#12318)
* feat(launcher): Tracing to log disabled by default

* remove traceLogger and use opentracing directly

* add Jaeger tracing

* go vet && go fmt
2019-03-04 11:48:11 -08:00
Jeff Wendling 0fae44e219 storage: fix problems with keeping resources alive
This commit adds the pkg/lifecycle.Resource to help manage opening,
closing, and leasing out references to some resource. A resource
cannot be closed until all acquired references have been released.
If the debug_ref tag is enabled, all resource acquisitions keep
track of the stack trace that created them and have a finalizer
associated with them to print on stderr if they are leaked. It also
registers a handler on SIGUSR2 to dump all of the currently live
resources.

Having resources tracked in a uniform way with a data type allows us
to do more sophisticated tracking with the debug_ref tag, as well.
For example, we could panic the process if a resource cannot be
closed within a certain time frame, or attempt to figure out the
DAG of resource ownership dynamically.

This commit also fixes many issues around resources, correctness
during error scenarios, reporting of errors, idempotency of
close, tracking of memory for some data structures, resource leaks
in tests, and out of order dependency closes in tests.
2019-02-28 10:22:01 -07:00
Jonathan A. Sternberg 70507670c3
feat(storage/reads): add scanned values and bytes metadata to the query (#12156)
This updates influxdb to use the new arbitrary metadata that can be
attached by a source and adds metadata entries for the cursor
statistics.
2019-02-25 14:44:18 -06:00
Michael Desa 0d3d0d4d78
chore(influxdb): add context to storage.PointsWriter 2019-02-25 11:11:20 -05:00
Stuart Carnie 662887c679 feedback: Simplify reader; use constants from influxdb package 2019-02-21 11:18:08 -08:00
Stuart Carnie 01b5fccfbe feat(storage): Enforce single org for series key reads 2019-02-21 11:18:08 -08:00
Alirie Gray 5f524eb92d Rename all occurences of Macro to Variable 2019-02-14 13:21:57 -08:00
Jeff Wendling 3bb765279b storage: respond to review comments 2019-02-04 12:26:26 -07:00
Jeff Wendling 3014733b20 chore: fix staticcheck issues 2019-02-04 10:32:52 -07:00
Jeff Wendling 376b347d56 wal: change deletes to be based on DeleteBucket 2019-02-04 10:32:52 -07:00
Jeff Wendling 7f54e816e3 refactor: have retention use DeleteBucketRange 2019-02-04 10:32:52 -07:00
Jeff Wendling aa12144fc7 storage: replay the WAL through the whole engine 2019-02-04 10:32:52 -07:00
Jeff Wendling 6deced1215 refactor: make the WAL part of snapshots again 2019-02-04 10:32:52 -07:00
Jeff Wendling 2989936d5a refactor: write to the WAL again 2019-02-04 10:32:52 -07:00
Jeff Wendling 2f46937527 refactor: move value package up to tsdb 2019-02-04 10:32:52 -07:00
Jeff Wendling 8dbd98ccbe refactor: change the way the engine opens and closes and reload the cache
Open and Close now proceed as best as they are able to in the presence
of errors and clean up appropriately.
2019-02-04 10:32:52 -07:00
Jeff Wendling d2ddd48eea refactor: hook up metrics and wal to storage engine
It turns out that LastModified and DiskSize are unused, and so it
was easy to change to not care about the WAL.

This hooks up metrics and starts the WAL again.
2019-02-04 10:32:52 -07:00
Jeff Wendling 95de3d52b2 refactor: use concrete WAL in tsm1
At the cost of some nil checks, we don't have to have an interface, defend against
subtle bugs with nils in non-nil interfaces, an empty implementation, etc.

Also, the tsm1 engine is losing the WAL anyway.
2019-02-04 10:32:52 -07:00
Jeff Wendling c9bb55b889 refactor: move the tsm1/wal into the storage/wal package
Because the WAL relies on the tsm1.Value type, we move that into its own
tsm1/value package and set up some aliases forwarding them into tsm1. This
also required adding some methods and changing consumers to avoid the
unexported fields. I imagine this step will be useful one day when we make
the write path more efficient with respect to consuming points.

This commit additionally fixes some issues with generation. The iterator.tmpldata
and generation for array_cursor_* were removed accidentally when removing
iterators, making those generated files stale. Restore that and regenerate.

No change in functionality.
2019-02-04 10:32:52 -07:00
Nathaniel Cook 8372859dee Merge branch 'master' into flux-staging 2019-01-15 08:35:59 -07:00
Nathaniel Cook d0603457b7 refactor(flux): make packages mirror Flux namespaces 2019-01-14 18:00:45 -07:00
Edd Robinson b025d9afa9 Improve efficiency of TSI index series drop
This commit improves the performance of a mass delete on the TSI index
by deleting at the measurement level instead of deleting each series
individually.
2019-01-14 12:46:55 +00:00
Edd Robinson 9e11602b6a Add DeleteBucket benchmark 2019-01-14 12:46:55 +00:00
Nathaniel Cook 1708a41fa7 refactor: update query functions for Flux builtins 2019-01-11 13:11:57 -07:00
Jonathan A. Sternberg a59e6b8d25 refactor: rename DoArrow to Do (#2372)
See influxdata/flux#783 for details.
2019-01-10 10:30:25 -07:00
Mark Rushakoff d73d73c0d4 chore: rename imports from platform to influxdb
I did this with a dumb editor macro, so some comments changed too.

Also rename root package from platform to influxdb.

In interest of minimizing risk, anyone importing the root package has
now aliased it to "platform" so that no changes beyond imports were
necessary in those files.

Lastly, replace the old platform module to local path /dev/null so that
nobody can accidentally reintroduce a platform dependency while
migrating platform code to influxdb.
2019-01-09 20:51:47 -08:00
Jeff Wendling 703c3c15ca Hook up DeleteBucket to the tsm1 engine 2019-01-09 15:24:26 -07:00
Edd Robinson 42ff769f1c Wire up storage.Engine to HTTP BucketService 2019-01-09 15:09:56 +00:00
Edd Robinson e3ae256782 Add storage bucket service
The storage bucket service wraps another bucket service, and invokes
actions on a storage engine based upon the actions taken upon buckets.

Currently, the storage bucket service will delete bucket data from the
storage engine when the bucket is deleted via the bucket service.
2019-01-09 13:35:25 +00:00
Edd Robinson 5a12a3a72e Export access to bucket drop in engine 2019-01-09 13:35:07 +00:00
Nathaniel Cook 89f4525841 build(Makefile): fix various bug with makefiles
Fixes subdir ordering.
Works around issue with stringer not working with Go modules.
Fixes issues with generated code being ignored.

Fixes #2044
2018-12-19 17:02:19 -07:00
Nathaniel Cook d6c0a393b0 Merge branch 'master' into flux-staging 2018-12-19 11:30:55 -07:00
Nathaniel Cook 61e36cbee6 chore(Makefile): add target to check generated files are accurate
A standard Makefile is used now in all subdirs that run go generate.
Make will only generate the file if its source files changed.
The checkgenerate target runs clean to ensure all targets a generated
fresh.
2018-12-18 12:54:17 -07:00
Jonathan A. Sternberg cc93531400 fix(storage): convert the storage table interface to use arrow buffers
The table interface was modified to expose the arrow buffers. The
storage table has now been converted to use this interface with the same
fixes so that it exposes arrow buffers.

The influxql package has also been updated to use the `DoArrow` method
from the `flux.Table` interface.
2018-12-18 09:47:53 -07:00
Ben Johnson 0541d568bd
fix(storage): Extend lock during stats fetch to prevent race 2018-12-13 11:13:20 -07:00
Stuart Carnie 8c296dd20a
fix(storage): Add unit tests to verify nil cursor
When matching series have no data, the NewGroupResultSet cursor should
return nil. Added tests to verify these conditions.
2018-12-12 14:27:11 -07:00
Stuart Carnie f2891c3f59
fix(storage): Fix panic when read request results in empty an group
Moves the check to determine if a series has data for the current time
range into the groupBySort function, which is consistent with the
groupNoneSort function.
2018-12-12 13:22:02 -07:00
Stuart Carnie 8f8311a3ae
chore(gen): Add ingen generator data structures to platform for reuse 2018-12-11 17:45:57 -07:00
Edd Robinson c4b42c72be Add option to disable TSI metrics 2018-12-10 15:02:26 +00:00
Edd Robinson b015757c06 Ensure all tsi1 metrics support multiple instances 2018-12-07 14:32:34 +00:00
Edd Robinson bff655786f Ensure tsdb metrics properly registered 2018-12-07 14:32:34 +00:00
Edd Robinson aa936df138 Ensure all tsm1 metrics support multiple instances 2018-12-07 14:32:34 +00:00
Edd Robinson f9a2f7a017 go fmt 2018-12-07 14:32:34 +00:00
Edd Robinson 7960ccc320 Add TSI index metrics 2018-12-07 14:32:34 +00:00
Edd Robinson d1fe2bc188 Add series file metrics 2018-12-07 14:32:34 +00:00
Edd Robinson 4e67e37ca6 Fix bug with engineID 2018-12-07 14:32:34 +00:00
Edd Robinson 8ca637bd80 Refactor default labels and retention metrics 2018-12-07 14:32:34 +00:00
Edd Robinson 44e5fbae0a Convert WAL stats 2018-12-07 14:32:34 +00:00
Edd Robinson f56bc0853f Convert TSM compaction stats to Prom metrics
This commits converts all the 1.x TSM compaction statistics, which
previously were written to an _internal db, to Prometheus metrics.
2018-12-07 14:32:34 +00:00
Jacob Marble b3e5e9fc31 feat(storage): return cost stats with read response via gRPC trailer 2018-12-05 15:32:41 -08:00
Edd Robinson 9403c1ec8e Ensure error strings not capitalised ST1005 2018-11-30 10:54:24 +00:00
Ben Johnson 0084d4d824
Remove influxdb dependency. 2018-11-29 11:44:22 -07:00
Ben Johnson 868fdbbe03
megacheck 2018-11-28 15:52:11 -07:00
Ben Johnson 1862b4421d
Integrate scanned values statistics tracking. 2018-11-28 15:32:06 -07:00
Mark Rushakoff a06d4dea16 refactor(query): explicitly shut down query controller
The flux query controller was updated to include a Shutdown method a
while ago. Explicitly handle query controller creation and shutdown
where applicable.

In influxd, this ensures that outstanding queries are handled before the
process dies. In tests, this ensures that query controller goroutines
aren't leaked, which drastically simplifies reading full stack traces.

This change also registers query controller metrics with the prometheus
registry in influxd.
2018-11-26 16:13:19 -08:00
Christopher M. Wolff a3ad02288b
Remove use of verbose from control.Config (#1573) 2018-11-26 16:02:41 -07:00
jlapacik 2b52224e84 update flux dependency - add missing Statistics() methods 2018-11-26 13:19:14 -08:00
Stuart Carnie f3d15be61b
chore: Add key order sanity check to StreamReaders 2018-11-21 12:01:36 -07:00
Stuart Carnie 8920dab57a
feedback: Changes per PR feedback 2018-11-21 12:01:36 -07:00
Stuart Carnie 831e2ab943
feat(reads): Implement merge operation for []reads.GroupResultSet
Specifically the currently supported GroupNone and GroupBy operations
2018-11-21 12:01:36 -07:00
Stuart Carnie 268a52148a
feat(reads): Implement merge operation for []reads.ResultSet 2018-11-21 12:01:36 -07:00
Stuart Carnie 7fb015cc7f
chore: Add GroupBy benchmark 2018-11-21 12:01:36 -07:00
Stuart Carnie 26408dc1fa
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err
* add option to Group sort to specify precedence of `nil`
* improve unit tests for GroupResultSet implementation
2018-11-21 12:01:35 -07:00