74d3c2e6d2
feat: Translate DataFusion execution metrics to IOx Spans ( #2529 )
...
* feat: Translate DataFusion execution metrics to IOx Spans
* fix: add end to end test to ensure plumbing is hookedup
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-15 18:14:23 +00:00
1ea4335ff3
fix: report correct DatabaseStateCode ( #2543 )
...
* fix: report correct DatabaseStateCode
* chore: fix lint
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-15 18:02:54 +00:00
bfaba78dc3
refactor: move `predicate` into its own crate
...
Two reasons:
1. I wanna decouple `parquet_file` from `query` (nearly done, needs a
small follow-up PR).
2. `predicate` will have more and more features (like serialization)
which justifies a new home
2021-09-14 17:13:02 +02:00
3f2e46c397
feat: prune old transactions from preserved catalog
2021-09-14 12:08:17 +02:00
f93984cd94
refactor: clarify wording
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-09-14 09:43:55 +02:00
f3bcafcfea
feat: migrate http metrics to metric crate ( #2508 )
...
* feat: migrate http metrics
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-13 18:56:20 +00:00
064fd39ff3
docs: update help text to show command to run the server
2021-09-13 11:42:26 -04:00
e07bc85670
feat: enable tracing for http API ( #2273 ) ( #2498 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-10 11:21:02 +00:00
8a1d759fb7
feat: response classification ( #2273 ) ( #2502 )
...
* feat: response classification (#2273 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-10 09:05:47 +00:00
1effb11ad9
refactor: move delete parseing work out of influxdb_line_protocol crate
2021-09-09 15:27:17 -04:00
00df7b064c
feat: finally have the delete predicate parsed
2021-09-08 17:30:10 -04:00
801cf08be7
feat: auto-creation of sequencers by write buffer
...
For Kafka, that basically means that we create a topic if it doesn't
exist yet.
Closed #2455 .
Fixes #2189 .
2021-09-07 18:24:57 +02:00
dbe4bcff22
chore: merge main to branch
2021-09-07 10:54:59 -04:00
a4b0cbc0e7
feat: migrate jemalloc metrics to `metric` crate ( #2435 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-06 09:18:27 +00:00
9de3b79a90
refactor: more cleanup
2021-09-06 01:45:47 -04:00
de0bd80c3d
refactor: cleanup
2021-09-06 01:07:07 -04:00
4801b2c238
feat: Have the ParseDelete message and its corresponding ProvidedParseDelete struct ready for building delete parser
2021-09-06 00:13:59 -04:00
d8b3208896
refactor: sstart building parser in the client side
2021-09-03 17:56:45 -04:00
e2274a9f41
feat: parser for delete predicate
2021-09-02 17:02:05 -04:00
ce6030a3cb
feat: Wire list deleted databases through gRPC and CLI APIs
2021-09-02 15:48:07 -04:00
4a863993ec
feat: "dump catalog" debug CLI
2021-09-02 08:08:20 +02:00
efcdc8e50f
feat: implement delete parser
2021-09-01 17:31:49 -04:00
e183ecb3e7
Merge branch 'main' into cn/list-but-not-deleted-databases
2021-09-01 13:03:09 +00:00
a4183de411
feat: more progress on the delete flow from grpc API to catalog chunks
2021-08-31 17:42:07 -04:00
79ad48ac3a
chore: rename "labels" to "attributes"
2021-08-31 11:31:15 +02:00
8dc375100a
feat: Wire database deletion through the CLI
...
Fixes #2432 .
2021-08-30 14:14:22 -04:00
e67624dd37
fix: Assert on which error getting a deleted database returns
2021-08-30 11:29:25 -04:00
d688678464
feat: Add an iox_object_store API for writing the tombstone file
...
Connects to #1871 .
2021-08-30 10:42:45 -04:00
2f49e47a23
feat: return DatabaseRules for ListDatabases request ( #2431 )
2021-08-28 10:53:24 +00:00
779b027271
feat: Store only the database rules sent by the client (do not store default values) ( #2430 )
...
* feat: add omit_default to protobuf definition
* feat: Persist only the client provided rules
* fix: Remove race conditions
* fix: merge confit
* refactor: do not use macro
* refactor: restore use of glob import
* fix: review comments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-28 10:26:32 +00:00
499af57299
chore: merge mian to branch and resolve conflicts
2021-08-27 17:51:07 -04:00
b79eaa34d1
refactor: address review comments
2021-08-27 15:53:27 -04:00
589bb31497
fix: make error msg clearer
2021-08-27 15:49:16 +02:00
8591aac761
test: improve error checking
2021-08-27 15:16:13 +02:00
90c60fd181
feat: "persist partition" CLI
2021-08-27 15:16:13 +02:00
6625b3a768
feat: "persist partition" gRPC
2021-08-27 15:16:13 +02:00
fcec394a28
feat: connect up new metrics ( #2428 )
2021-08-27 12:55:35 +00:00
bcd39e225c
feat: Management API for delete
2021-08-26 17:31:21 -04:00
e3e801d29a
feat: propagate span context into storage RPC queries ( #2407 )
...
* feat: propagate span context into storage RPC queries
* refactor: create ExecutionContextProvider trait
* chore: cleanup imports
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-26 17:11:49 +00:00
1773bf5d37
feat: add storage client to influxdb_iox_client ( #2404 )
...
* feat: add storage client to influxdb_iox_client
* chore: fix type_url
* refactor: split storage into separate crate
* chore: fix doctest
* chore: review feedback
* chore: add generated_types cleanup ticket
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-26 10:30:59 +00:00
5d97acb2f3
Merge branch 'main' into crepererum/issue2372
2021-08-25 07:08:15 +00:00
f7792aafe6
feat: query tracing ( #2273 ) ( #2391 )
...
* feat: query tracing (#2273 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-24 17:35:59 +00:00
2c9c191b17
refactor: split out trace http ( #2388 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-24 13:37:20 +00:00
2592d5c382
fix: improve log info wording
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-08-24 15:33:05 +02:00
363d202202
feat: stop application executor in one dedicated place
2021-08-24 14:46:36 +02:00
3fdc0e9a6a
chore: remove orphaned code from trogging ( #2371 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-24 10:12:59 +00:00
a6c9cc2bf2
refactor: rework exec module ( #2384 )
...
* refactor: rework exec module
* chore: update docs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-24 08:39:54 +00:00
0946ffe916
refactor: reuse IOxExecutionContext ( #2373 )
...
* refactor: reuse IOxExecutionContext
* fix: orphaned comment
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-23 15:47:15 +00:00
a20990fc49
feat: Make jemalloc optional
...
Add a way to use the system memory allocator which is required for some
analysis tools like valgrind.
2021-08-23 09:29:49 +02:00
bbbd38bb12
fix(heappy): Prevent re-entering a running heap profile
...
Upgrades heappy to include https://github.com/mkmik/heappy/pull/2
and adapts to the API change.
2021-08-20 14:36:36 +02:00
4a2f0b0a52
fix: Make heappy and nonheappy mutually exclusive
...
Closes #2288
2021-08-20 11:35:07 +02:00
db97069ecd
feat: jaeger span export ( #2273 ) ( #2348 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-20 08:57:16 +00:00
a9b185e508
fix: Remove confusing and unnecessary rust global allocator
...
The jemalloctor rust global allocator only intercepts allocations coming from rust.
We used to enabled the `unprefixed_malloc_on_supported_platforms` on the jemallocator crate
which effectivelly installs jemalloc as a replacement for system malloc so that both rust and C libraries
would use jemalloc. This not only ensures that all libraries use jemalloc for the sake of using
a better allocator, but also allows us to use the jemalloc metrics (exposed by jemalloc-ctl) to figure
out how much memory we are consuming; if we don't enable `unprefixed_malloc_on_supported_platforms` and only install the rust global allocator everything works just fine, except the memory metrics return a lower number because memory allocated by e.g. croaring bitmaps wouldn't be tracked.
However since heappy is is incompatible with `unprefixed_malloc_on_supported_platforms` but effectively plays the same role (by installing a replacement of malloc that ultimately invokes jemalloc) and we
were in a rush last week, we turned on heappy and turned off `unprefixed_malloc_on_supported_platforms`
without doing the full work of figuring out how to tell Cargo that those flags have that mutually-exclusive logic; the followup work has been put in a #2288 .
This PR removes this rust global allocator since it's a source of confusion. Now if we'll accidentally
disable jemalloc for C we'll also disable it for rust which will be immediately visible since we won't
have any memory metrics at all.
2021-08-19 15:14:37 +02:00
04ba139cb2
refactor: split trace crate ( #2343 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-19 11:00:32 +00:00
710f477e35
refactor: remove opentelemetry from observability_deps ( #2192 ) ( #2336 )
2021-08-18 17:24:05 +00:00
5e1cb244f7
feat: make pprof optional ( #2331 )
2021-08-18 15:13:37 +00:00
e62790cf4b
fix: include underlying error message in connection error ( #2326 )
...
* fix: include underlying error message in connection error
* fix: clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-18 11:33:30 +00:00
c0640bf1f9
fix: Make allocs profile record all allocations
...
Rename `frequency` to `interval` and set the default to `1`, which means that
every allocation will be counted.
Before this PR, we were incorrectly reusing the same "frequency" parameter (and related default of 99)
which meant that we were skipping small allocations that.
The sampling interval is a number of bytes that have to cumulatively allocated for a sample to be taken.
For example if the sampling interval is 99, and you're doing a million of 40 bytes allocations, the allocations profile will account for 16MB instead of 40MB. Heappy will adjust the estimate for sampled recordings, but now that feature is not yet implemented.
2021-08-18 13:08:36 +02:00
1d6a8703af
feat: support sinking traces to an OTEL SpanExporter ( #2319 )
...
* feat: support sinking traces to an OTEL SpanExporter
* chore: consistent logging
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* chore: review feedback
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-18 08:05:21 +00:00
efa776bd03
feat: fix log crate output ( #2325 )
...
* feat: fix log crate output
* chore: fix doc
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-17 16:19:22 +00:00
60ea637c38
refactor: remove `drop_non_persisted`
...
This flag should always be `!persist`.
Closes #2212 .
2021-08-17 16:53:45 +02:00
e6ae36722a
Merge branch 'main' into crepererum/issue2292
2021-08-17 11:29:24 +00:00
86df65145c
fix: interpret hex strings in span context ( #2313 )
2021-08-17 09:20:16 +00:00
6847fa97cc
feat: drop partition CLI
2021-08-17 09:44:35 +02:00
fcf2bee443
feat: drop partition gRPC
2021-08-17 09:44:35 +02:00
42d5f9f3a1
feat: skip replay via CLI
2021-08-16 13:47:07 +02:00
e18fa7bdeb
docs: Remove the description of unused commands in cli ( #2290 )
2021-08-16 11:10:09 +00:00
302e2b5353
feat: support jaeger context propagation format ( #2293 )
2021-08-16 10:55:40 +00:00
756f5c6699
feat: add end-to-end tracing test ( #2285 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-14 13:37:05 +00:00
500019cff9
Merge branch 'main' into ntran/heappy2
2021-08-13 17:27:38 +02:00
8650e419f7
feat: add trace collection plumbing ( #2273 ) ( #2274 )
...
* feat: add trace collection plumbing (#2273 )
* chore: fix lints
* chore: use tower directly
* chore: add tests
* chore: review feedback
* refactor: use serde instead of custom display impl
* chore: fix doc
2021-08-13 14:38:31 +00:00
0a065b4968
feat: set header on gRPC requests ( #2283 )
...
* feat: set header on gRPC requests
* feat: always insert header value
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-13 14:13:50 +00:00
5704b7f622
fix: madd order by in select query to have the output deterministic
2021-08-13 09:55:38 -04:00
67753343e9
refactor: remove unecessary gate
2021-08-13 08:18:01 -05:00
697de875ca
refactor: make heappy an optional feature, add to build image
2021-08-13 08:11:32 -05:00
36e84f85ef
chore: Merge branch 'main' into ntran/heappy2
2021-08-13 08:16:58 -04:00
cba03e9c47
refactor: gRPC connection in single location ( #2276 )
...
* refactor: gRPC connection in single location
* chore: log messages
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-13 08:44:33 +00:00
82d86fc857
fix: Use same allocation profile url path as Go
...
Also the name "heappy" is a the codename of a project. It produces a standard allocation
profile.
2021-08-13 10:08:22 +02:00
607a54cd9d
refactor: isolate CLI handling for object store
...
The object store handling is a large part of our CLI and kinda inlined
into the rest of the setup routines. This commit moves the object store
related code into its own module. As a bonus it can now be reused by
other CLI commands (e.g. low-level debug tools).
There are no functional changes here except for slight changes in the
error structure.
2021-08-13 09:23:25 +02:00
4fd5862a9c
fix: turn heappy on
2021-08-12 17:14:03 -04:00
e0448be345
Merge branch 'main' into ntran/heappy
2021-08-12 17:57:16 +00:00
1f0ce5ae5a
refactor: use previous use
2021-08-12 12:21:38 -04:00
4f52a70d5b
refactor: adddress mkm's comments
2021-08-12 12:04:42 -04:00
7d105e9229
docs: fix warnings
2021-08-12 09:30:54 +02:00
8aa9514d93
feat: integrate heappy implemented by mkm
2021-08-11 23:42:50 -04:00
3de6b44e23
build: use new rustdoc lint name ( #2261 )
...
* fix: nocache feature code rot
The MBChunk::snapshot code when using the "nocache" option no longer
compiles - this commit updates it to match the not(nocache) code.
* build: use updated broken_intra_doc_links name
The broken_intra_doc_links lint was renamed
rustdoc::broken_intra_doc_links
https://doc.rust-lang.org/rustdoc/lints.html
2021-08-11 19:48:51 +00:00
2344c28f4e
feat: drain database jobs on shutdown ( #2239 )
...
* feat: drain database jobs on shutdown
* chore: fmt
* chore: review feedback
* chore: use join() not member directly
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-10 16:47:37 +00:00
cd5f4e1755
feat: background worker panic handling ( #2091 ) ( #2234 )
...
* feat: worker panic handling (#2091 )
* chore: add test comments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-10 09:17:56 +00:00
564819d24f
feat: Server own background worker ( #2232 )
...
* feat: Server own background worker
* chore: fix shutdown
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-09 18:01:48 +00:00
559db4529d
refactor: Move DatabaseStore out of query crate ( #2219 )
...
* refactor: Move DatabaseStore out of query crate
* fix: doc links
2021-08-09 12:06:25 +00:00
1f450ef371
feat: add Database abstraction ( #2186 ) ( #2203 )
...
* feat: add Database abstraction
* chore: minor tweaks
* chore: remove redundant test fixture restart
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-08 17:14:23 +00:00
e92e94caad
chore: Update deps (including arrow 5.1.0, tonic -> 0.5, and prost 0.5) ( #2172 )
...
* chore: Update deps (including arrow 5.0.0 --> arrow 5.1.0)
* chore: update all the things
* refactor: Update serving readiness check due to change in Tonic API
* chore: update more deps
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-05 15:57:38 +00:00
98d4c9fca1
feat: switch protobuf write service to canonical definition ( #2182 )
...
* feat: switch protobuf write service to canonical definition
The protobuf definition used for the proto write endpoint was a WIP. Now
that a canonical definition exists at
https://github.com/influxdata/influxdb-pb-data-protocol/ we can switch
to that.
* chore: lint etc
* chore: fix rustdoc nit in proto definition comment
2021-08-04 00:16:49 +00:00
12b8074d9d
fix: doc link
2021-08-03 18:11:21 +02:00
c91cb041e7
fix: env prefix
2021-08-03 17:34:59 +02:00
1276aa4d66
fix: fix skip replay env flag
2021-08-03 17:28:41 +02:00
ffb36cd50c
refactor: extract ApplicationState from Server ( #2167 )
...
* refactor: extract Application from Server
* chore: review feedback
2021-08-03 09:36:55 +00:00
fa67dd4886
Merge branch 'main' into crepererum/issue2169
2021-08-03 07:45:01 +00:00
9864c6b7f1
fix: Return a more helpful error message for no matching sharding rule
...
Fixes #2127 .
2021-08-02 16:49:40 -04:00
c912e91c95
feat: add flag to skip replay
...
Closes #2169 .
2021-08-02 18:14:19 +02:00
Andrew Lamb
Raphael Taylor-Davies
Marco Neumann
Jake Goulding
Nga Tran
Carol (Nichols || Goulding)
kodiakhq[bot]
Marko Mikulicic
Jason
Andrew Lamb
Dom
Jacob Marble