0bd7941a18
fix(REPL): Don't buffer lines until a trailing semicolon is found and add history hinting ( #3630 )
...
* fix(REPL): Don't buffer lines until a trailing semicolon is found
The repl would silently buffer all lines until a trailing semicolon were found which
resulted in some very confusing error messages as I would input invalid commands followed
by a command I thought were valid, except I'd still get an error due to the previous command being buffered.
This uses rustyline's helper feature to detect incomplete input (no trailing semicolon) and makes
it accept multiline input until the input is completed.
I also included some of rustyline's default hint and highlighting while I was at it.
* chore: cargo clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-03 17:11:01 +00:00
b3b2d9b623
feat: catalog setup CLI command ( #3627 )
...
Closes #3509 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-03 14:16:21 +00:00
ab3c7573f5
test: add end to end for read_filter and empty string predicates ( #3619 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-03 14:05:32 +00:00
50cff27b01
chore: remove rdkafka dependency ( #3625 )
...
All features are now covered by rskafka. This also removes the need to
specify a server ID for write buffer consumers. This was only used for
rdkafka since there we needed to specify a consumer group, even though
we did not use any transactions.
2022-02-03 13:33:56 +00:00
3197ea945b
Merge branch 'main' into dom/extract-ns-cache
2022-02-03 12:30:37 +00:00
77b80e7618
fix(InfluxQL): treat null tags as `''` rather than `null` in storagerpc queries ( #3557 )
...
* fix(InfluxQL): treat null tags as `''` rather than `null` in storage rpc queries
* test: add one more case
* fix: Update comment
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-03 12:14:43 +00:00
ce46bbaada
feat: wire up the write buffer to the ingester process ( #3533 )
...
This adds the scaffolding for the ingester server to consume data from Kafka. This ingests data in an in memory structure while creating records in the catalog for any partitions that don't yet exist.
I've removed catalog_update.rs in ingester for now. That was mostly a placeholder and will be going in a combination of handler.rs and data.rs on my next PR which will have some primitive lifecycle wired up.
There's one ugly bit here where the DML write is cloned because it's getting borrowed to output spans and metrics. I'll need to follow up with a refactor to make it so that the DML write's tables can be consumed without it gumming up the metrics stuff.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-03 11:47:28 +00:00
3cc4481616
refactor: extract NamespaceSchema cache
...
Breaks the in-memory cache of NamespaceSchema out into a decoupled type
that can be shared across multiple DML handlers.
2022-02-03 10:01:07 +00:00
a534136ccc
fix: Correct a 'long' argument name so the ingester command can run ( #3621 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-02 20:09:41 +00:00
429d59f1b6
feat: Simplify predicates in the `InfluxRpcFrontend` before using them ( #3588 )
...
* feat: normalize + simplify RPC predicates before using them
* docs: Update predicate/src/rpc_predicate.rs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-02 19:46:57 +00:00
6de8ed4adc
Merge branch 'main' into dom/schema-validation
2022-02-02 16:05:41 +00:00
6da15d9690
chore: cleanup catalog CLI output & args
...
Co-authored-by: Marco Neumann <marco@crepererum.net>
2022-02-02 15:30:19 +00:00
15827a534b
feat: catalog CLI command with update subcommand
2022-02-02 15:30:19 +00:00
39d489d9e7
refactor: enable schema validation
...
Adds the SchemaValidator to the DML handler stack - this adds it into
the request path in router2.
2022-02-02 14:04:14 +00:00
5441682207
feat: add support for parsing predicate
2022-02-02 11:02:33 +00:00
08901c13cd
feat: support parsing timerange
2022-02-02 11:02:33 +00:00
a424d1c912
feat: shell command read_filter
2022-02-02 11:02:33 +00:00
59a2c74352
refactor: reusable ingester/router2 CLI pieces ( #3590 )
...
* refactor: use a single CLI parser for ingester/router2 WB
* refactor: reusable catalog DSN CLI handling
We are going to need DSN handling for the router as well as for the some
admin tools.
* fix: DNS -> DSN
2022-02-01 12:57:58 +00:00
22778a3a80
chore: upgrade rskafka and parking_lot ( #3592 )
2022-02-01 11:50:42 +00:00
b326b62b44
feat: buffer writes when writing to RSKafka ( #3520 )
2022-02-01 10:07:52 +00:00
c633c9bc5c
feat: Wire object store into ingester persistence
2022-01-31 10:36:30 -05:00
c50fc8764d
feat: basic non-panic HTTP/gRPC interface for ingester ( #3583 )
...
Don't panic when K8s requests a health status or someone requests a
non-found HTTP route; or when we just TRY to start up the gRPC service.
2022-01-31 11:13:14 +00:00
36642eb71d
test: add end to end tests that query missing tags ( #3563 )
...
* test: add end to end tests that query missing tags
* fix: add github reference
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-28 18:42:57 +00:00
3659a7f799
refactor: clean up ingester CLI ( #3569 )
...
- use same args/envs names as router2 does
- kafka => write buffer
- add long forms to all CLI args so we don't have to pass positional
arguments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-28 17:58:16 +00:00
4101d16f71
chore: feature flag consistency ( #3574 )
...
* chore: feature flag consistency
* chore: add aarch64-apple-darwin to hakari
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-28 16:38:59 +00:00
a22ca7c3d7
fix: router2 writer buffer topic ( #3555 )
...
- Kafka does not support `_` in topic names, but `-` works, so let's
change the default
- Expose topic config via CLI/env
2022-01-28 10:10:04 +00:00
f24ce03754
fix: provide correct environment variable to change log filter ( #3561 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 20:45:06 +00:00
2062267d0f
chore: Update hashbrown ( #3551 )
...
* chore: Update hashbrown
* fix: hakari
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 15:34:10 +00:00
5447554aee
refactor(router2): DML handler stack ( #3549 )
...
* refactor: composable DmlHandler stack
Changes the DmlHandler trait to allow composition of handler logic in
order to construct the complete request processing pipeline.
* feat: debug log write/delete requests
Log requests hitting the HTTP endpoint at DEBUG.
* refactor: dml_handler -> dml_handlers
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 14:54:27 +00:00
21c1824a7a
refactor: remove table_names from Predicate ( #3545 )
...
* refactor: remove table_names from Predicate
* chore: fix benchmarks
* chore: review feedback
Co-authored-by: Edd Robinson <me@edd.io>
* chore: review feedback
* chore: replace Default::default with InfluxRpcPredicate::default()
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 14:44:49 +00:00
16d584b2ff
feat: Add db_name/namespace to DmlWrite and DmlDelete ( #3531 )
...
* feat: Add db_name/namespace to DmlWrite and DmlDelete
This is required for the new ingester to be able to work with the write buffer. The protobuf that gets serialized over Kafka already includes the database name, it just wasn't getting carried through to the marshaled Dml operation.
* fix: database != namespace, propagation through write buffer
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 14:12:20 +00:00
5488c257d1
chore: Update datafusion, upgrade to arrow/parqet/arrow-flight 8.0.0 ( #3517 )
...
* chore: Update datafusion
* chore: update to arrow 8
* fix: update to use new DataFusion APIs
* fix: update case for sortedness
* fix: cargo hakari
2022-01-27 13:33:27 +00:00
107f39d53c
feat: add trace collector to router2 ( #3529 )
...
* feat: add trace collector to router2
* chore: fmt
2022-01-26 11:51:17 +00:00
6b0f7e6b2b
feat: initialise ShardedWriteBuffer ( #3528 )
...
Initialises a ShardedWriteBuffer for the hard-coded "iox_shared" topic.
Adds the following CLI flags:
* --write-buffer: type of buffer [kafka, rskafka, file]
* --write-buffer-addr: write buffer endpoint address
The server uses these config options to initialise the appropriate write
buffer backend, and configure the TableNamespaceSharder to shard
operations over the set of sequencers exposed by the write buffer.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-26 10:49:34 +00:00
1b6aed063d
feat: add per-partition tracing ( #3532 )
...
* feat: add per-partition tracing
* chore: docs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-26 10:39:21 +00:00
db46ac04d0
feat: support line protocol precision parameter ( #3522 ) ( #3526 )
...
* feat: support line protocol precision parameter (#3522 )
* chore: format imports
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-25 14:12:22 +00:00
bb893510a0
feat: Add scaffolding for ingester server
...
* Adds a new ingester command to start an ingester server
* Moves previous ingester server over to handler
* Skeleton for gRPC and HTTP handlers
2022-01-21 18:02:19 -05:00
9c19cd6cc4
fix: clamp start/end of TimestampRange to min/max valid timestamp values ( #3487 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-20 16:08:00 +00:00
9751301374
refactor: rename `storage_api.rs` end to end test to `influxrpc.rs` for consistency ( #3497 )
...
* refactor: rename `storage_api` end to end test to `influxrpc` for consistency
* fix: fmt
2022-01-20 14:25:13 +00:00
168afb63ad
feat: add `size` methods to DML-related types
...
This will be helpful when we want to batch DML operations in memory
(e.g. when using RSKafka).
This also ensures that `MBChunk` accounts for the column names that
are stored within `MutableBatch`.
2022-01-18 13:52:31 +01:00
40a290f6f7
feat: router2 HTTP handlers
...
Implements the HTTP v2 write API endpoint for router2.
2022-01-17 11:57:28 +00:00
c399e676ca
chore: upgrade clap to v3
2022-01-17 12:12:46 +01:00
89db894df4
fix: serde_json `arbitrary_precision` ( #3458 ) ( #3469 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-17 09:50:54 +00:00
b036db293f
fix: Format `ParenExpression` RPC storage Expression `Node`s ( #3463 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-14 16:02:51 +00:00
36ec6019f9
feat: wire up token to query frontends
2022-01-14 10:26:11 +00:00
dd23056efd
chore: update datafusion, arrow, prost, tonic, pbjson, etc ( #3455 )
...
* chore: update datafusion, arrow, prost, tonic, etc
* fix: update pprof as well
* chore: update hakari
* fix: update pbjson
* chore: update heappy
* fix: hakari
* fix: workaround https://github.com/influxdata/influxdb_iox/issues/3458
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-13 17:07:15 +00:00
430823c148
docs: fix typo
...
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
2022-01-13 15:42:30 +00:00
3d32901877
fix: undected gzip HTTP body truncation
...
When reading the gzip-encoded body of a HTTP request, the stream is read
up until the configured maximum number of allowable bytes, at which
point the body was silently trucated. This could allow fields in
submitted line protocol to be silently lost (amongst other bad things).
This change ensures that truncation results in a RequestSizeExceeded
error.
2022-01-13 14:38:37 +00:00
7fc17203e2
refactor: add router2 server mode
...
Plumbs the router2 crate into IOx's CLI & server-runner framework.
2022-01-12 14:52:47 +00:00
a8cb8755de
feat: new router2 crate
...
This commit adds an almost-empty router2 crate containing enough of a
skeleton to plumb into the IOx CLI/server runner.
2022-01-12 14:43:10 +00:00
f3f6f335a9
chore: upgrade to snafu 0.7 ( #3440 )
2022-01-11 19:22:36 +00:00
b76921d26e
fix(influxrpc): Support _field references in front end conversion ( #3426 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-10 21:54:56 +00:00
6d20ce1f9e
feat: Allow wipe catalog in `ReplayError` and `WriteBufferCreationError` states ( #3425 )
...
* feat: feat: Allow wipe catalog in ReplayError
* fix: comments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-07 17:07:44 +00:00
336ffd1966
refactor: Remove `Result` in QueryDatabase trait (none of the functions can fail) ( #3422 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-06 22:03:08 +00:00
a93ae739a9
feat: Add table_name to Partition API ( #3421 )
2022-01-06 16:38:39 +00:00
f9174c483b
refactor: Extract server::db into its own crate ( #3417 )
...
* refactor: Extract JobRegistry from the server crate
Both the server crate and a db crate that I'm about to extract depend on
JobRegistry, so to avoid making circular dependencies, extract the
JobRegistry to its own crate.
* refactor: Move db out of server into its own crate
Fixes #2821 .
2021-12-23 22:01:17 +00:00
2c3ca0c77c
docs: Add doc comments to all CLI subcommands ( #3414 )
...
* docs: Add doc comments to all CLI subcommands
* docs: Update influxdb_iox/src/commands/database/recover.rs
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-23 12:17:39 +00:00
5aaee1bcf4
fix: Remove some straggling references to writer ID that should be server ID ( #3415 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-22 21:05:38 +00:00
c631dd4267
refactor: address PR feedback
2021-12-21 10:23:12 +00:00
200c32cb32
feat: support all tag key predicates
2021-12-21 10:23:12 +00:00
5b277c8754
test: exercise show tag values
2021-12-17 17:59:30 +00:00
119e579fa3
feat: add support for tag values grouped by measurement and tag key
2021-12-17 17:59:30 +00:00
a68d71ef06
feat: add ability to materialise measurements
2021-12-17 17:59:30 +00:00
ebbf86dc1d
refactor: impl stub
2021-12-17 17:59:30 +00:00
3fde298a99
refactor: handle rpc
2021-12-17 17:59:30 +00:00
9f2694bf1b
test: `objest_store` azure support via Azurite
2021-12-17 09:30:21 +01:00
64f915d860
fix: flaky end to end system_tables test ( #3397 )
2021-12-17 08:13:09 +00:00
5d58b06e64
test: fix some environment variables influencing our tests
2021-12-16 11:17:36 +01:00
d8810074e8
refactor: better test action CLI
2021-12-16 09:29:25 +01:00
f4fde15810
refactor: nicer code
...
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-12-16 09:29:25 +01:00
ed775431b6
fix: treat early server worker exit as proper error
...
Instead of just logging the issue, also make sure the error gets
propagated all the way up to the exit code.
Fixes #3375 .
2021-12-16 09:29:25 +01:00
da0330fc5f
fix: print newline at end of `database list --detailed` ( #3382 )
2021-12-15 19:33:24 +00:00
758b65dd29
feat: Add database initialization state and errors to CLI and remove list_databases_detailed gRPC ( #3377 )
...
* feat: Add database initialization state and errors to CLI:
* fix: do not use optional in protobuf
* fix: clippy
* fix: correct check I broke appeasing clippy
2021-12-15 12:18:41 +00:00
7fe6897c59
refactor: add support for handling `TagValuesGroupedByMeasurementAndTagKeyRequest` ( #3373 )
...
* chore: update Storage service
* refactor: handle rpc
* refactor: log request fields
* refactor: add new RPC to client
* test: basic test of RPC
* refactor: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-14 18:42:12 +00:00
ec6d945cbe
feat: wire up general predicates to measurement_names
2021-12-13 14:01:09 +00:00
c0ba69f09e
chore: marge main to branch and resolve conflict
2021-12-09 15:40:33 -05:00
35370922f3
refactor: make a setup for 2 persisted chunks that can be used in for different places
2021-12-09 15:21:56 -05:00
3cda6b6c0f
refactor: Remove collect_query and replication ( #3348 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-09 19:58:19 +00:00
099e2d4056
chore: Apply suggestions from code review
...
Co-authored-by: Marco Neumann <marco@crepererum.net>
2021-12-09 13:52:42 -05:00
e46708354e
test: add management cli tests
2021-12-09 12:53:45 -05:00
365917c2aa
fix: Organize imports
2021-12-09 08:49:34 -05:00
471c3181bb
feat: Make a top-level influxdb_iox feature for kafka functionality
...
Thread the feature through router and server to the write buffer crate.
Move an end-to-end test that uses Kafka behind the feature flag.
2021-12-09 08:49:34 -05:00
38ce54c1be
Merge branch 'main' into crepererum/issue3336
2021-12-09 09:44:55 +00:00
efbfbb1a0b
feat: compact all object store chunks of a given partition
2021-12-08 16:06:03 -05:00
876a9af35a
fix: limit number of S3 connections
...
Otherwise the whole thing blows up when starting a server that has many
DBs registerd, because we potentially create 1 connection per DB (e.g.
to read out the preserved catalog).
Fixes #3336 .
2021-12-08 19:06:02 +01:00
a8bc5b3b6d
fix: allow setting the same server ID twice
...
This is important for idempotence and simplifies clients and helper
scripts a lot.
2021-12-08 19:03:30 +01:00
35edee6b4f
fix: flaky test migrate_table_files_from_one_server_to_another ( #3339 )
2021-12-08 12:08:25 -05:00
218042784f
feat: add `system.queries `system table ( #3328 )
...
* feat: Add query log and system table
* docs: Add docstrings for test normalization
2021-12-08 16:26:24 +00:00
c6a3765d76
feat: Add force flag to RebuildCatalog ( #3292 )
...
* feat: Add force flag to RebuildCatalog
* fix: small cleanups
* docs: Update comments and add WARNING
2021-12-08 15:36:07 +00:00
3e97e49c35
refactor: "update router" -> "create or update router"
2021-12-08 13:45:55 +01:00
e42c6974cf
fix: remove unused code
2021-12-08 13:45:55 +01:00
af71e2422e
fix: typos
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-12-08 13:45:55 +01:00
b7d6865f87
feat: add router CLI
...
Closes #3331 .
2021-12-08 13:45:55 +01:00
c17a6c10c1
feat: add `GetRouter` gRPC method
2021-12-08 13:45:55 +01:00
577df009d8
fix: test flakyness while waiting for gRPC connection
...
Fixes #3312 .
2021-12-07 17:05:06 +01:00
75cd1d24f4
Merge branch 'main' into ntran/grpc_compact_os_chunks
2021-12-07 03:26:06 +00:00
7073691f04
test: grpc test for compaction os chunks
2021-12-06 22:25:33 -05:00
5316037ffe
fix: fix flay migrate_database_files_from_one_server_to_another and prepare for reuse + fix ( #3314 )
...
* fix: fix flay migrate_database_files_from_one_server_to_another and prepare for reuse + fix
* docs: clarify comments
2021-12-06 18:28:08 +00:00
3834435620
chore: Merge branch 'ntran/grpc_compact_os_chunks' of https://github.com/influxdata/influxdb_iox into ntran/grpc_compact_os_chunks
2021-12-06 12:06:22 -05:00
d8ed8a3bf7
refactor: address review comments
2021-12-06 12:06:07 -05:00
Markus Westerlind
Marco Neumann
Andrew Lamb
kodiakhq[bot]
Paul Dix
Dom Dwyer
Carol (Nichols || Goulding)
Luke Bond
Luke Bond
Edd Robinson
Carol (Nichols || Goulding)
Raphael Taylor-Davies
Paul Dix
Nga Tran