2b76c31157
refactor: make statistics null counts optional ( #4160 )
...
Min/max values and distinct counts are already optional, so let's make
the null counts optional as well. This will be helpful for NG to deal w/
partial statistics (e.g. we only populate stats for the time column).
Note that the total count is still mandatory, but we normally have the
chunk/file-level row count at hand.
2022-03-29 17:47:57 +00:00
196ad50f5a
test: add e2e test for schema CLI ( #4132 )
...
* test: add e2e test for schema CLI
* fix: logical confict
2022-03-25 18:11:03 +00:00
9886ff42cc
refactor: clean up querier public interface
2022-03-25 11:54:52 +01:00
e222acbb48
refactor: use typed `TestConfig` rather than environment variable names for NG end to end tests ( #4126 )
...
* refactor: move environment variable mapping in end to end tests into TestConfig
* fix: clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-24 18:36:47 +00:00
39d9f30f12
refactor: Make `server` an implementation detail ( #4122 )
2022-03-24 15:58:04 +00:00
fb75ef7b82
test: add end to end test for all in one mode, restructure fixture ( #4114 )
...
* test: add end to end test for all in one mode, restructure fixture
* docs: fix typos and clarify schema requrements
2022-03-24 12:53:25 +00:00
c891f3c4f2
fix: Print an error with sample env var variable if unset
2022-03-09 09:55:43 -05:00
8af2f60b59
fix: Run catalog setup as part of end-to-end test setup
2022-03-09 09:55:43 -05:00
93b0cdbcc4
fix: Create the test database as part of ng server fixture startup
2022-03-09 09:55:43 -05:00
e4fb227c6e
feat: Set the catalog URL explicitly in the test config
2022-03-09 09:55:43 -05:00
465fb0272d
fix: Remove unneeded server id const
2022-03-09 09:55:43 -05:00
6d086705a8
fix: Remove ng create_shared server fixture method for now
2022-03-09 09:55:42 -05:00
e315012fe3
fix: Switch NG tests to use TEST_INFLUXDB_IOX_CATALOG_DSN too
2022-03-09 09:55:42 -05:00
a14e642f39
refactor: Extract NG end-to-end tests from OG end-to-end tests
2022-03-09 09:55:42 -05:00
1536bdeca0
test: Add an end-to-end NG test
2022-03-09 09:55:42 -05:00
66a5649258
test: Print out the server type with server log messages
...
So that when you have more than one server running in a test, it's
easier to see which one is saying what
2022-03-09 09:55:42 -05:00
ae45d9f750
test: Add Router2 as a supported ServerFixture type
2022-03-09 09:55:42 -05:00
d2c0acdd46
refactor: Remove serving readiness gate ( #3986 )
...
* refactor: Remove serving_readiness
* fix: remove more
* fix: remove test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-09 12:17:44 +00:00
e304613546
feat: include trace ID in query log ( #3912 ) ( #3923 )
...
* feat: include trace ID in query log (#3912 )
* chore: fmt
* chore: lint
2022-03-03 17:50:49 +00:00
82aa314659
feat: make wait-server-initialized wait for database init ( #3915 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-03 15:37:22 +00:00
c0ebe5a2f1
Merge branch 'main' into dom/fix-feature-flags
2022-03-03 11:48:35 +00:00
6204a9046d
fix: jemalloc feature flag
...
In #2435 we moved the jemalloc metrics around, and missed a few feature
gates to avoid pulling in jemalloc unless it is requested.
2022-03-03 11:48:24 +00:00
0660b514e3
feat: allow persisting all partitions and/or all tables from CLI ( #3874 )
...
* feat: allow persisting all partitions and/or all tables
* chore: use BTreeSet
* feat: non-zero exit code on error continue
* test: test continue-on-error
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-03 11:17:34 +00:00
806a43eb6c
fix: Temporarily remove ingester end-to-end test case
...
This now needs more setup because without any data, the flight query
request returns an error. The setup is proving to be nontrivial.
2022-03-02 15:29:48 -05:00
2a90841715
refactor: Move IngesterQueryRequest to data_types2
...
So that querier doesn't need to depend on ingester.
2022-03-02 13:52:13 -05:00
8f3e44bf76
refactor: Extract a crate for shared data types in the new design
2022-03-02 12:16:15 -05:00
16d86ed05b
feat: Deserialize metadata to get max_sequencer_number
...
And add an end-to-end test for the flight request to the ingester.
2022-03-02 11:50:47 -05:00
141a6087d0
feat: Querier able to send Flight queries to Ingester
...
Fixes #3773 .
2022-03-02 11:50:45 -05:00
286d5f7b2b
feat: add `success` column to system.queries ( #3891 )
...
* feat: add `success` column to system.queries
* refactor: Remove lifetime from QueryCompletedToken and thread through flight
* test: update test to make incomplete query clearer
* refactor: use better patter to set complete
* fix: logical merge conflict
2022-03-02 15:05:06 +00:00
33851be3a5
chore: upgrade Rust to 1.59 ( #3875 )
...
Mostly a few new clippy crates around `flat_map`, `and_then`, and
"underscore locks" (!!!):
https://rust-lang.github.io/rust-clippy/master/index.html#let_underscore_lock
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-28 15:14:19 +00:00
2a842fbb1a
feat: correctly sort data and store in catalog metadata ( #3864 )
...
* feat: respect sort order in ChunkTableProvider (#3214 )
feat: persist sort order in catalog (#3845 )
refactor: owned SortKey (#3845 )
* fix: size tests
* refactor: immutable SortKey
* test: test sort order restart (#3845 )
* chore: explicit None for sort key
* chore: test cleanup
* fix: handling of sort keys containing fields
* chore: remove unused selected_sort_key
* chore: more docs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-25 17:56:27 +00:00
65ab5213e5
chore(deps): Bump clap from 3.0.14 to 3.1.1 ( #3809 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.0.14 to 3.1.1.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.0.14...v3.1.1 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-22 14:51:53 +00:00
1960645055
feat: add wildcard support to persist partition CLI command ( #3790 )
...
* feat: add wildcard support to persist partition
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 14:21:06 +00:00
39c42678d7
feat: trigger persistence if over soft limit and no evictable chunks ( #3791 )
...
* feat: trigger persistence if over soft limit and no evictable chunks
* chore: fmt
* fix: avoid test_full_lifecycle exceeding soft limit
* fix: don't expect chunk to be unloaded
* feat: only trigger if no outstanding persist
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 10:14:27 +00:00
83cba3d2fb
feat: template static router config ( #3781 )
...
* feat: template static router config
* chore: lint and improved failure output
* chore: clarify docs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-18 10:53:10 +00:00
44ee0166a0
fix: start Kafka write buffer stream at "earliest" offset, not at "0" ( #3748 )
2022-02-15 13:36:59 +00:00
26fd5273f0
feat: static database configuration ( #2436 ) ( #3732 )
...
* feat: static database configuration (#2436 )
* chore: fmt
* feat: don't base64 encode UUIDs in ServerConfigFile
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 19:42:49 +00:00
c79050254f
refactor: traitify database configuration ( #2436 ) ( #3730 )
...
* refactor: traitify database configuration (#2436 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-13 09:26:44 +00:00
866777ecd2
feat: static router configuration ( #2436 ) ( #3725 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-11 14:09:37 +00:00
4e3f66ed07
feat: CLI and gRPC APIs for shutting down and restarting databases ( #3720 )
...
* feat: allow catalog wipe and rebuild whilst shutdown
* feat: CLI and gRPC APIs for shutting down and restarting databases
* feat: add ability to skip replay on restart
* fix: test_wipe_persisted_catalog_error_db_exists
* fix: wipe_preserved_catalog
2022-02-11 10:14:43 +00:00
b1190262b7
feat: restartable `Database` ( #3368 ) ( #3711 )
...
* feat: restartable `Database` (#3368 )
* chore: fmt
* fix: wipe_preserved_catalog
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 18:32:05 +00:00
73828323ac
feat: Ingester Flight gRPC API ( #3623 )
...
* feat: Add a way to run ingester with an in-memory catalog from the CLI
If you set the --catalog-dsn string to "mem", rather than using that as
a Postgres connection URL, create an in-memory catalog.
Planning on using this in tests, so not documenting.
* fix: Set default topic to the same value as SHARED_KAFKA_TOPIC
Namely, both should use an underscore. I don't think there's a way to
directly share these values between a constant and an annotation.
* feat: Add a flight API (handshake only) to ingester
* fix: Create partitions if using file-based write buffer
* fix: Change the server fixture to handle ingester server type
For now, the ingester doesn't implement the deployment API. Not sure if
it should or not.
* feat: Start implementing ingester do_get, namely decoding the query
Skip serialization of the predicate for the moment.
* refactor: Rename ingest protos to ingester to match crate name
* refactor: Rename QueryResults to QueryData
* feat: Move ingester flight client to new querier crate
* fix: Off by one error, different starting indexes in sequencers
* fix: Create new CLI argument to pick the catalog type
* fix: Create a CLI option to set the number of topics to auto-create in the write buffer
* fix: Check the arrow flight service's health to tell that the ingester gRPC is up
* fix: Set postgres as the default catalog type
* fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
e2db1df11f
refactor: improve writer buffer consumer interface ( #3631 )
...
* refactor: improve writer buffer consumer interface
The change looks huge but is actually rather simple. To
understand the interface change, let me first explain what we want:
- be able to fetch watermarks for any sequencer
- have streams:
- each streams tracks a sequencer and has an offset state (no read
multiplexing)
- we can seek a stream
- seeking and streaming cannot be done at the same time (that would be
weird and likely leads to many bugs both in write buffer and in the
user code)
- ideally we don't need to create streams of all sequencers but can
choose a subset
Before this change we had one mutable consumer struct where you can get
all streams and watermark functions (this mutable-borrows the consumer)
or you can seek a single stream (this also mutable-borrows the
consumer). This is a bit weird for multiple reasons:
- you cannot seek a single stream without dropping all of them
- the mutable-borrow construct makes it really difficult to pass the
streams into separate threads
- the consumer is boxed (because its mutable) which makes it more
difficult to handle in a large-scale application
What this change does is the following:
- you have an immutable consumer (similar to the producer)
- the consumer offers the following methods:
- get the set of sequencer IDs
- get watermark for any sequencer
- get a stream handler (see next point) for any sequencer
- the stream handler captures the stream state (offset) and provides you
a standard `Stream<_>` interface as well as a seek function.
Mutable-borrows ensure that you cannot use both at the same time.
The stream handler provides you the stream via `handler.stream()`. It
doesn't implement `Stream<_>` itself because the way boxing, dynamic
dispatch work, and pinning interact (i.e. I couldn't get it to work
without the indirection).
As a bonus point (which we don't use however) you can now create
multiple streams for the same sequencer and they all have their own
offset.
* fix: review comments
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-07 12:24:17 +00:00
77b80e7618
fix(InfluxQL): treat null tags as `''` rather than `null` in storagerpc queries ( #3557 )
...
* fix(InfluxQL): treat null tags as `''` rather than `null` in storage rpc queries
* test: add one more case
* fix: Update comment
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-03 12:14:43 +00:00
36642eb71d
test: add end to end tests that query missing tags ( #3563 )
...
* test: add end to end tests that query missing tags
* fix: add github reference
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-28 18:42:57 +00:00
16d584b2ff
feat: Add db_name/namespace to DmlWrite and DmlDelete ( #3531 )
...
* feat: Add db_name/namespace to DmlWrite and DmlDelete
This is required for the new ingester to be able to work with the write buffer. The protobuf that gets serialized over Kafka already includes the database name, it just wasn't getting carried through to the marshaled Dml operation.
* fix: database != namespace, propagation through write buffer
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 14:12:20 +00:00
9c19cd6cc4
fix: clamp start/end of TimestampRange to min/max valid timestamp values ( #3487 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-20 16:08:00 +00:00
9751301374
refactor: rename `storage_api.rs` end to end test to `influxrpc.rs` for consistency ( #3497 )
...
* refactor: rename `storage_api` end to end test to `influxrpc` for consistency
* fix: fmt
2022-01-20 14:25:13 +00:00
168afb63ad
feat: add `size` methods to DML-related types
...
This will be helpful when we want to batch DML operations in memory
(e.g. when using RSKafka).
This also ensures that `MBChunk` accounts for the column names that
are stored within `MutableBatch`.
2022-01-18 13:52:31 +01:00
c399e676ca
chore: upgrade clap to v3
2022-01-17 12:12:46 +01:00
89db894df4
fix: serde_json `arbitrary_precision` ( #3458 ) ( #3469 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-17 09:50:54 +00:00
dd23056efd
chore: update datafusion, arrow, prost, tonic, pbjson, etc ( #3455 )
...
* chore: update datafusion, arrow, prost, tonic, etc
* fix: update pprof as well
* chore: update hakari
* fix: update pbjson
* chore: update heappy
* fix: hakari
* fix: workaround https://github.com/influxdata/influxdb_iox/issues/3458
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-13 17:07:15 +00:00
6d20ce1f9e
feat: Allow wipe catalog in `ReplayError` and `WriteBufferCreationError` states ( #3425 )
...
* feat: feat: Allow wipe catalog in ReplayError
* fix: comments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-07 17:07:44 +00:00
a93ae739a9
feat: Add table_name to Partition API ( #3421 )
2022-01-06 16:38:39 +00:00
5aaee1bcf4
fix: Remove some straggling references to writer ID that should be server ID ( #3415 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-22 21:05:38 +00:00
64f915d860
fix: flaky end to end system_tables test ( #3397 )
2021-12-17 08:13:09 +00:00
d8810074e8
refactor: better test action CLI
2021-12-16 09:29:25 +01:00
ed775431b6
fix: treat early server worker exit as proper error
...
Instead of just logging the issue, also make sure the error gets
propagated all the way up to the exit code.
Fixes #3375 .
2021-12-16 09:29:25 +01:00
758b65dd29
feat: Add database initialization state and errors to CLI and remove list_databases_detailed gRPC ( #3377 )
...
* feat: Add database initialization state and errors to CLI:
* fix: do not use optional in protobuf
* fix: clippy
* fix: correct check I broke appeasing clippy
2021-12-15 12:18:41 +00:00
c0ba69f09e
chore: marge main to branch and resolve conflict
2021-12-09 15:40:33 -05:00
35370922f3
refactor: make a setup for 2 persisted chunks that can be used in for different places
2021-12-09 15:21:56 -05:00
3cda6b6c0f
refactor: Remove collect_query and replication ( #3348 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-09 19:58:19 +00:00
099e2d4056
chore: Apply suggestions from code review
...
Co-authored-by: Marco Neumann <marco@crepererum.net>
2021-12-09 13:52:42 -05:00
e46708354e
test: add management cli tests
2021-12-09 12:53:45 -05:00
365917c2aa
fix: Organize imports
2021-12-09 08:49:34 -05:00
471c3181bb
feat: Make a top-level influxdb_iox feature for kafka functionality
...
Thread the feature through router and server to the write buffer crate.
Move an end-to-end test that uses Kafka behind the feature flag.
2021-12-09 08:49:34 -05:00
efbfbb1a0b
feat: compact all object store chunks of a given partition
2021-12-08 16:06:03 -05:00
a8bc5b3b6d
fix: allow setting the same server ID twice
...
This is important for idempotence and simplifies clients and helper
scripts a lot.
2021-12-08 19:03:30 +01:00
35edee6b4f
fix: flaky test migrate_table_files_from_one_server_to_another ( #3339 )
2021-12-08 12:08:25 -05:00
218042784f
feat: add `system.queries `system table ( #3328 )
...
* feat: Add query log and system table
* docs: Add docstrings for test normalization
2021-12-08 16:26:24 +00:00
c6a3765d76
feat: Add force flag to RebuildCatalog ( #3292 )
...
* feat: Add force flag to RebuildCatalog
* fix: small cleanups
* docs: Update comments and add WARNING
2021-12-08 15:36:07 +00:00
3e97e49c35
refactor: "update router" -> "create or update router"
2021-12-08 13:45:55 +01:00
b7d6865f87
feat: add router CLI
...
Closes #3331 .
2021-12-08 13:45:55 +01:00
c17a6c10c1
feat: add `GetRouter` gRPC method
2021-12-08 13:45:55 +01:00
577df009d8
fix: test flakyness while waiting for gRPC connection
...
Fixes #3312 .
2021-12-07 17:05:06 +01:00
75cd1d24f4
Merge branch 'main' into ntran/grpc_compact_os_chunks
2021-12-07 03:26:06 +00:00
7073691f04
test: grpc test for compaction os chunks
2021-12-06 22:25:33 -05:00
5316037ffe
fix: fix flay migrate_database_files_from_one_server_to_another and prepare for reuse + fix ( #3314 )
...
* fix: fix flay migrate_database_files_from_one_server_to_another and prepare for reuse + fix
* docs: clarify comments
2021-12-06 18:28:08 +00:00
8ad2b5e9fd
fix: In end-to-end tests, use influxdb_iox_client instead of influxdb2_client
2021-12-06 09:37:14 -05:00
3900324d33
fix: Organize imports
2021-12-06 09:37:13 -05:00
1099f5c417
refactor: break out database migration end to end tests ( #3308 )
2021-12-06 12:21:10 +00:00
7dea2ef362
feat: Add force flag to ClaimDatabase ( #3284 )
...
* feat: Add force flag to ClaimDatabase
* fix: Update name of test
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* docs: Update influxdb_iox/tests/end_to_end_cases/management_cli.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2021-12-03 21:11:23 +00:00
f17b670fe9
refactor: cleanup gRPC client error handling ( #3251 ) ( #3277 )
...
* refactor: cleanup gRPC client error handling (#3251 )
* chore: review feedback
* chore: review feedback
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-02 20:56:58 +00:00
2857b6a990
Merge branch 'main' into er/feat/load_chunk_cli
2021-12-02 20:20:56 +00:00
b4ea9887ba
refactor: error name
2021-12-02 20:14:02 +00:00
1a899b939e
fix: Remove redundant closures identified by clippy
...
https://rust-lang.github.io/rust-clippy/master/index.html#redundant_closure
2021-12-02 11:52:02 -05:00
15f9918eda
test: add end to end test
2021-12-02 12:59:52 +00:00
9e45ce4121
feat: cleanup gRPC server errors ( #3251 ) ( #3265 )
...
* feat: add precondition violation enumeration (#3251 )
* chore: review feedback
2021-12-01 13:33:17 +00:00
f1936f0b08
refactor: add resource type enumeration ( #3251 ) ( #3260 )
...
* refactor: add resource type enumeration (#3251 )
* chore: more error details
2021-12-01 10:26:12 +00:00
05064eba62
fix: don't error on delete to non-existent table ( #3259 ) ( #3263 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-01 00:40:05 +00:00
7a02830bf7
fix: GIT version in `--version`
...
The GIT version was always "UNKNOWN", a regression introduced by #2965 .
Also adds a regression test.
2021-11-30 16:06:27 +01:00
c161d1176a
refactor: cleanup Db write path error handling ( #3258 )
2021-11-30 15:05:02 +00:00
1e515a1dec
feat: load RUB from object store ( #3224 ) ( #3250 )
2021-11-30 14:39:52 +00:00
88acf3788e
feat: rebuild catalog ( #3225 ) ( #3253 )
...
* feat: rebuild catalog (#3225 )
* chore: fix doc
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-30 12:17:27 +00:00
f165faf5bc
feat: support forcing persistence of partition irrespective of arrival time ( #3249 )
...
* feat: force persistence of partition irrespective of arrival time
* feat: expose to CLI
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-29 22:07:35 +00:00
837e029ed8
refactor: cleanup database recovery ( #3225 ) ( #3245 )
...
* refactor: cleanup database recovery (#3225 )
* chore: fix errors
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-29 18:49:46 +00:00
4e043ecb55
refactor: remove old routing / sharding config
...
This is superseded by the new router subsystem.
2021-11-29 12:33:48 +01:00
ab4f8cdbf2
test: add end to end tests for Storage/Offsets gRPC call ( #3232 )
2021-11-27 12:01:33 +00:00
7f2e4f4342
refactor: remove write buffer direction
...
The direction was required when a database could read or write from/to a
write buffer. Now it is clear from the usage context of a write buffer
context which of the two applications is meant (databases read, routers
write) so the direction flag is no longer required.
2021-11-26 12:38:40 +01:00
589b7ad3af
fix: typo
...
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2021-11-26 11:40:41 +01:00
e5bfa08910
feat: disallow unsequenced deletes when reading from write buffer
...
Similar to how writes to a databases that reads from a write buffer are
forbidden deletes should be rejected as well.
2021-11-26 11:40:08 +01:00
0e06026fbd
refactor: remove write routing logic from `Server`
...
API users should use `Database` instead, there's no need any longer to mirror
this API in `Server`. Note that `Database` is better than `Db` in this
case, because the former can also check if we're writing from a write
buffer and can easily reject unsequenced writes. The `pub` modifiers
were adjusted to make it impossible to write through `Db` directly.
2021-11-26 09:26:43 +01:00
c2a8baf824
refactor: clean up write buffer end2end tests
...
Remove the ones that are covered by `write_pb.rs` or that are obsolete
(i.e. rely on behavior that only exist in the legacy router code).
Migrate the remaining ones to use an actual router.
2021-11-25 14:18:37 +01:00
d3b1557dbc
refactor: isolate and parametrize some management CLI tests
2021-11-24 11:12:22 +01:00
cae8b8572d
refactor: remove remote API from database node
...
We can easily bring it back in case we need it, but at the moment its
not required for database nodes to manage remotes.
2021-11-24 11:12:22 +01:00
ed54100e60
refactor: convert `write_api` end2end tests to use new router mode
2021-11-24 11:12:22 +01:00
aa636eaed9
refactor: remove methods from `ManagementService` that were moved to `RemoteService`
...
Tests are covered by `remote_api.rs`.
Ref #2980 .
2021-11-24 10:01:32 +01:00
5ecaa6d8a9
refactor: use serialized predicates instead of strings for gRPC deletes
...
IOx is the only consumer of this API so we might just use the serialized
form. Cloud2 uses the HTTP API which supports SQL-like predicates.
Fixes #3192 .
2021-11-24 09:30:32 +01:00
53139edef9
Merge branch 'main' into crepererum/issue2980h
2021-11-24 08:21:07 +00:00
42e3a41825
fix: flaky test_chunk_access_time ( #3197 ) ( #3200 )
...
* fix: flaky test_chunk_access_time (#3197 )
* chore: fix lint
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-23 17:39:57 +00:00
dc6b44818d
refactor: remove methods from `ManagementService` that were moved to `DeploymentService`
...
Tests are covered by `deployment_api.rs`.
Ref #2980 .
2021-11-23 18:04:41 +01:00
c9a5726d7b
Merge branch 'main' into er/refactor/grpc_tag_order
2021-11-23 16:39:32 +00:00
2ef3ab8544
refactor: remove methods from `ManagementService` that were moved to `DeleteService`
...
Tests are covered by `delete_api.rs`.
Ref #2980 .
2021-11-23 16:40:00 +01:00
5b200ee564
feat: impl delete gRPC interface for router
2021-11-23 15:39:53 +01:00
65dc0bd4b3
test: update tests
2021-11-23 14:15:06 +00:00
521a8e3636
test: fix tests
2021-11-22 21:40:41 +00:00
25d55cd08a
feat: Move server config paths beneath 'nodes' ( #3144 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-19 09:54:32 +00:00
f69d37e9a8
fix: Remove database delete/restore entirely
2021-11-17 12:03:11 -05:00
7783e4a7ff
fix: Make delete/restore aliases for release/claim; remove tombstone
...
Fixes #2680
2021-11-17 11:41:08 -05:00
8155747735
feat: add write buffer delete encoding ( #2731 ) ( #3127 )
...
* feat: add write buffer delete encoding (#2731 )
* chore: fix doc
* chore: review feedback
* chore: review feedback
* chore: fmt
* chore: review feedback
2021-11-17 16:12:19 +00:00
d6c6e9a6c7
fix: Default kafka timeout to be shorter than gRPC timeout (60 sec --> 10 sec) ( #3131 )
...
* fix: Default kafka timeout to be shorter than gRPC timeout
* docs: fix link style
2021-11-17 12:19:53 +00:00
bc11244828
feat: Rename database disown/adopt to release/claim ( #3111 )
...
* fix: Rename 'disown' to 'release' database
Connects to #3110
* fix: Rename 'adopt' to 'claim' database
Fixes #3110 .
2021-11-15 20:28:09 +00:00
af28cfa2a6
feat: Add an adopt database API
...
Fixes #2679 .
2021-11-15 09:26:06 -05:00
236edc0e7e
Merge branch 'main' into crepererum/improve_router_client_errors
2021-11-15 11:36:57 +00:00
9868430c12
Merge branch 'main' into crepererum/fix_server_id_on_router_startup
2021-11-15 10:29:21 +00:00
a6d83a3026
feat: WriteBufferReader use DmlOperation ( #2731 ) ( #3096 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-15 10:19:54 +00:00
48d0388ea0
feat: improve `RouterClient` errors
2021-11-15 11:18:46 +01:00
d5d07b6eeb
Merge branch 'main' into crepererum/issue3075b
2021-11-15 08:17:16 +00:00
376471591f
Merge branch 'main' into cn/disown
2021-11-12 19:31:03 +00:00
8bdfb253ce
fix: Remove context from disown operation
2021-11-12 11:10:20 -05:00
815347b18a
fix: Disown in the disown tests, not delete
2021-11-12 09:47:25 -05:00
3220970fea
fix: server IDs for routers on startup
2021-11-12 11:31:52 +01:00
f2c90d9259
tests: allow creation of test servers w/ server IDs set on startup
2021-11-12 11:31:52 +01:00
c1827cf2c3
refactor: `s/writer_id/server_id/g`
2021-11-12 11:31:52 +01:00
7f021f556d
tests: improve `TestConfig` to better support server types
2021-11-12 11:31:52 +01:00
8c1cf00164
feat: improve and test handling of updating a server ID a 2nd time
2021-11-12 11:31:52 +01:00
7ef762a88e
tests: add `test_serving_readiness_router`
2021-11-12 11:31:52 +01:00
84a5df44e3
fix: Let structopt handle parsing Uuid
2021-11-11 14:37:19 -05:00
95c46e58bc
Merge branch 'main' into jpg/restore-uuid-bytes
2021-11-11 15:34:23 +00:00
19d5200d1c
test: Make assertion more concise and avoid an allocation, thank you clippy
2021-11-11 09:56:43 -05:00
3667cb36fa
refactor: align HTTP error handling w/ cloud 2
...
Streamline HTTP error by using the same representation as cloud 2. Also
ensure that only internal server errors result in an error-level log
entry.
Fixes #3075 .
2021-11-11 14:48:50 +01:00
6f268f8260
refactor: extract DML types ( #2731 ) ( #3084 )
...
* refactor: extract DML types (#2731 )
* chore: fmt
2021-11-11 12:34:07 +00:00
7e13cb77ac
feat: add transactions for context to database owner info on disown
2021-11-10 16:49:32 -05:00
489fad0040
feat: Add disown database to the gRPC API
2021-11-10 16:49:31 -05:00
b1e81186d3
feat: Add disown database commands to the database CLI
2021-11-10 16:49:31 -05:00
83dbfe1333
feat: gRPC PB-write interface for new router
2021-11-08 15:49:13 +01:00
de672b7599
fix: Store the restore database UUID as bytes instead of as a string
2021-11-05 15:20:18 -04:00
88868e7496
feat: remove legacy write service from influxdb_iox_client ( #3043 )
...
* feat: remove legacy write service from influxdb_iox_client
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-05 17:26:18 +00:00
c8d80e5c28
fix: Change database paths to be under /dbs/ instead of under /[server id]/
2021-11-05 10:14:06 -04:00
898567e221
feat: migrate server to DbWrite ( #2724 ) ( #3035 )
...
* feat: migrate server to DbWrite (#2724 )
* chore: print perf log output
* fix: don't suppress CI status code
* chore: review feedback
* fix: don't error on empty line protocol write payloads
* fix: test
* fix: test
2021-11-05 11:09:33 +00:00
06dfdc4af8
feat: simplify sharding ( #3031 )
...
* feat: simplify sharding
* chore: fix lint
2021-11-04 16:57:19 +00:00
78101eb52d
feat: implement `RemoteService` for new router mode
2021-11-04 15:35:05 +01:00
4840dc11cb
feat: add basic CRUD operations for router configs
2021-11-04 12:03:14 +01:00
07ba629e2b
feat: migrate write buffer producer to MutableBatch and pbdata ( #2743 ) ( #3021 )
...
* feat: migrate write buffer producer to MutableBatch and pbdata (#2743 )
* fix: Kafka message content type header
* chore: fix doc
2021-11-04 10:20:40 +00:00
0f62bb1686
fix: Distinguish between restoring twice and name collisions on restore
...
And return them in the correct situations.
2021-11-03 15:02:22 -04:00
52d9df5caf
fix: Thread a Not Found error on restore database through all APIs
2021-11-03 15:02:21 -04:00
ac75be6b78
fix: Don't require name for database restore CLI; it's now unused
2021-11-03 15:02:21 -04:00
3e44f17698
fix: Return a useful error on attempt to restore an active database
2021-11-03 15:02:18 -04:00
f6327b1940
fix: Send back the correct error when attempting to restore the same database twice
2021-11-03 15:01:56 -04:00
a0b4b6b8f9
test: Add assertions that ensure UUID doesn't change when DatabaseRules are updated
2021-11-03 15:01:27 -04:00
62e7376394
fix: Restore database should only need UUID, not database name too
2021-11-03 15:01:26 -04:00
1cf697f235
refactor: make server fixture creation less confusing
2021-11-03 16:34:51 +01:00
a91e572b43
feat: implement basic router server type
...
This only implements the deployment service and nothing else yet.
2021-11-03 16:34:51 +01:00
39a7f3069f
feat: add alternative string encodings to pbdata ( #2724 ) ( #3006 )
...
* feat: add alternative string encodings to pbdata (#2724 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-02 18:14:34 +00:00
0d0c0cb42b
refactor: move write buffer configs to new home
...
Write buffer configs will partially be shared by database and router
nodes, so lets move them into a shared home.
2021-11-02 10:17:01 +01:00
011af2d6ba
feat: add `DeploymentService`
...
Ref #2980 .
2021-11-01 17:55:40 +01:00
8c3446ac87
Merge branch 'main' into crepererum/issue2980b
2021-11-01 16:39:32 +00:00
f1a6468e7b
feat: migrate write buffer consumer to use DbWrite ( #2724 ) ( #3003 )
...
* feat: migrate write buffer consumer to use DbWrite (#2724 )
* fix: doc
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-01 16:38:48 +00:00
bcd66c555a
feat: add `RemoteService`
...
Ref #2980 .
2021-11-01 11:38:46 +01:00
fc25e313c6
Merge branch 'main' into cn/iox-obj-sto-refactors
2021-10-29 13:53:26 +00:00
f3edda11e7
fix: Print all delete database output to stdout
2021-10-29 09:46:02 -04:00
83e4514a43
feat: add `DeleteService`
...
Ref #2980 .
2021-10-29 11:57:31 +02:00
8a2410e161
feat: mutable batch write entry ( #2724 ) ( #2973 )
...
* feat: mutable batch write entry (#2724 )
* chore: lint
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-28 20:15:28 +00:00
0bccd35f8c
fix: List details about databases only prints active dbs and their UUIDs
...
We can no longer list deleted databases because the server no longer
knows about them, and we now have UUIDs that are useful to know about,
so change the detailed listing of databases to return UUID instead of
possible deleted at time.
2021-10-28 13:20:29 -04:00
ce55cf401c
feat: Return UUID when creating a database
...
Seems polite and is useful for some test setup
2021-10-28 13:20:29 -04:00
4077d3a20a
fix: Take a UUID when restoring a database
2021-10-28 13:20:29 -04:00
95ba84edca
fix: Return UUID when a database is deleted
...
So that it can be immediately used to restore elsewhere.
2021-10-28 13:20:29 -04:00
e80b902d15
test: Delete tests either irrelevant or covered elsewhere
...
Now that the database paths use UUIDs instead of names+generation ids,
and that IoxObjectStore checks there are no files in a directory before
it'll create a database directory (and has tests for that).
2021-10-28 13:20:28 -04:00
a6e0a069cc
refactor: rename "run mode" to "server type"
2021-10-28 11:16:41 +02:00
6e29cec502
refactor: rename `query` mode to `database`
2021-10-27 09:31:46 +02:00
41bc373fe8
refactor: offer different modes via "run" CLI
...
Running a server is now using `influxdb_iox run MODE [args]`, e.g.
`influxdb_iox run query --server-id 1`. Another mode that will follow
soon is `router`.
The old syntax `influxdb_iox run [args]` (w/o the mode part) is still
supported but a deprecation note will be printed.
2021-10-27 09:31:46 +02:00
a0cdca7781
chore: move `influxdb_iox` into a proper workspace package
2021-10-26 11:02:33 +02:00
Marco Neumann
Andrew Lamb
Carol (Nichols || Goulding)
Raphael Taylor-Davies
kodiakhq[bot]
Dom Dwyer
dependabot[bot]
Carol (Nichols || Goulding)
Andrew Lamb
Paul Dix
Nga Tran
Edd Robinson
Jake Goulding
Carol (Nichols || Goulding)