577df009d8
fix: test flakyness while waiting for gRPC connection
...
Fixes #3312 .
2021-12-07 17:05:06 +01:00
75cd1d24f4
Merge branch 'main' into ntran/grpc_compact_os_chunks
2021-12-07 03:26:06 +00:00
7073691f04
test: grpc test for compaction os chunks
2021-12-06 22:25:33 -05:00
5316037ffe
fix: fix flay migrate_database_files_from_one_server_to_another and prepare for reuse + fix ( #3314 )
...
* fix: fix flay migrate_database_files_from_one_server_to_another and prepare for reuse + fix
* docs: clarify comments
2021-12-06 18:28:08 +00:00
8ad2b5e9fd
fix: In end-to-end tests, use influxdb_iox_client instead of influxdb2_client
2021-12-06 09:37:14 -05:00
3900324d33
fix: Organize imports
2021-12-06 09:37:13 -05:00
1099f5c417
refactor: break out database migration end to end tests ( #3308 )
2021-12-06 12:21:10 +00:00
7dea2ef362
feat: Add force flag to ClaimDatabase ( #3284 )
...
* feat: Add force flag to ClaimDatabase
* fix: Update name of test
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* docs: Update influxdb_iox/tests/end_to_end_cases/management_cli.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2021-12-03 21:11:23 +00:00
f17b670fe9
refactor: cleanup gRPC client error handling ( #3251 ) ( #3277 )
...
* refactor: cleanup gRPC client error handling (#3251 )
* chore: review feedback
* chore: review feedback
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-02 20:56:58 +00:00
2857b6a990
Merge branch 'main' into er/feat/load_chunk_cli
2021-12-02 20:20:56 +00:00
b4ea9887ba
refactor: error name
2021-12-02 20:14:02 +00:00
1a899b939e
fix: Remove redundant closures identified by clippy
...
https://rust-lang.github.io/rust-clippy/master/index.html#redundant_closure
2021-12-02 11:52:02 -05:00
15f9918eda
test: add end to end test
2021-12-02 12:59:52 +00:00
9e45ce4121
feat: cleanup gRPC server errors ( #3251 ) ( #3265 )
...
* feat: add precondition violation enumeration (#3251 )
* chore: review feedback
2021-12-01 13:33:17 +00:00
f1936f0b08
refactor: add resource type enumeration ( #3251 ) ( #3260 )
...
* refactor: add resource type enumeration (#3251 )
* chore: more error details
2021-12-01 10:26:12 +00:00
05064eba62
fix: don't error on delete to non-existent table ( #3259 ) ( #3263 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-01 00:40:05 +00:00
7a02830bf7
fix: GIT version in `--version`
...
The GIT version was always "UNKNOWN", a regression introduced by #2965 .
Also adds a regression test.
2021-11-30 16:06:27 +01:00
c161d1176a
refactor: cleanup Db write path error handling ( #3258 )
2021-11-30 15:05:02 +00:00
1e515a1dec
feat: load RUB from object store ( #3224 ) ( #3250 )
2021-11-30 14:39:52 +00:00
88acf3788e
feat: rebuild catalog ( #3225 ) ( #3253 )
...
* feat: rebuild catalog (#3225 )
* chore: fix doc
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-30 12:17:27 +00:00
f165faf5bc
feat: support forcing persistence of partition irrespective of arrival time ( #3249 )
...
* feat: force persistence of partition irrespective of arrival time
* feat: expose to CLI
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-29 22:07:35 +00:00
837e029ed8
refactor: cleanup database recovery ( #3225 ) ( #3245 )
...
* refactor: cleanup database recovery (#3225 )
* chore: fix errors
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-29 18:49:46 +00:00
4e043ecb55
refactor: remove old routing / sharding config
...
This is superseded by the new router subsystem.
2021-11-29 12:33:48 +01:00
ab4f8cdbf2
test: add end to end tests for Storage/Offsets gRPC call ( #3232 )
2021-11-27 12:01:33 +00:00
7f2e4f4342
refactor: remove write buffer direction
...
The direction was required when a database could read or write from/to a
write buffer. Now it is clear from the usage context of a write buffer
context which of the two applications is meant (databases read, routers
write) so the direction flag is no longer required.
2021-11-26 12:38:40 +01:00
589b7ad3af
fix: typo
...
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2021-11-26 11:40:41 +01:00
e5bfa08910
feat: disallow unsequenced deletes when reading from write buffer
...
Similar to how writes to a databases that reads from a write buffer are
forbidden deletes should be rejected as well.
2021-11-26 11:40:08 +01:00
0e06026fbd
refactor: remove write routing logic from `Server`
...
API users should use `Database` instead, there's no need any longer to mirror
this API in `Server`. Note that `Database` is better than `Db` in this
case, because the former can also check if we're writing from a write
buffer and can easily reject unsequenced writes. The `pub` modifiers
were adjusted to make it impossible to write through `Db` directly.
2021-11-26 09:26:43 +01:00
c2a8baf824
refactor: clean up write buffer end2end tests
...
Remove the ones that are covered by `write_pb.rs` or that are obsolete
(i.e. rely on behavior that only exist in the legacy router code).
Migrate the remaining ones to use an actual router.
2021-11-25 14:18:37 +01:00
d3b1557dbc
refactor: isolate and parametrize some management CLI tests
2021-11-24 11:12:22 +01:00
cae8b8572d
refactor: remove remote API from database node
...
We can easily bring it back in case we need it, but at the moment its
not required for database nodes to manage remotes.
2021-11-24 11:12:22 +01:00
ed54100e60
refactor: convert `write_api` end2end tests to use new router mode
2021-11-24 11:12:22 +01:00
aa636eaed9
refactor: remove methods from `ManagementService` that were moved to `RemoteService`
...
Tests are covered by `remote_api.rs`.
Ref #2980 .
2021-11-24 10:01:32 +01:00
5ecaa6d8a9
refactor: use serialized predicates instead of strings for gRPC deletes
...
IOx is the only consumer of this API so we might just use the serialized
form. Cloud2 uses the HTTP API which supports SQL-like predicates.
Fixes #3192 .
2021-11-24 09:30:32 +01:00
53139edef9
Merge branch 'main' into crepererum/issue2980h
2021-11-24 08:21:07 +00:00
42e3a41825
fix: flaky test_chunk_access_time ( #3197 ) ( #3200 )
...
* fix: flaky test_chunk_access_time (#3197 )
* chore: fix lint
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-23 17:39:57 +00:00
dc6b44818d
refactor: remove methods from `ManagementService` that were moved to `DeploymentService`
...
Tests are covered by `deployment_api.rs`.
Ref #2980 .
2021-11-23 18:04:41 +01:00
c9a5726d7b
Merge branch 'main' into er/refactor/grpc_tag_order
2021-11-23 16:39:32 +00:00
2ef3ab8544
refactor: remove methods from `ManagementService` that were moved to `DeleteService`
...
Tests are covered by `delete_api.rs`.
Ref #2980 .
2021-11-23 16:40:00 +01:00
5b200ee564
feat: impl delete gRPC interface for router
2021-11-23 15:39:53 +01:00
65dc0bd4b3
test: update tests
2021-11-23 14:15:06 +00:00
521a8e3636
test: fix tests
2021-11-22 21:40:41 +00:00
25d55cd08a
feat: Move server config paths beneath 'nodes' ( #3144 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-19 09:54:32 +00:00
f69d37e9a8
fix: Remove database delete/restore entirely
2021-11-17 12:03:11 -05:00
7783e4a7ff
fix: Make delete/restore aliases for release/claim; remove tombstone
...
Fixes #2680
2021-11-17 11:41:08 -05:00
8155747735
feat: add write buffer delete encoding ( #2731 ) ( #3127 )
...
* feat: add write buffer delete encoding (#2731 )
* chore: fix doc
* chore: review feedback
* chore: review feedback
* chore: fmt
* chore: review feedback
2021-11-17 16:12:19 +00:00
d6c6e9a6c7
fix: Default kafka timeout to be shorter than gRPC timeout (60 sec --> 10 sec) ( #3131 )
...
* fix: Default kafka timeout to be shorter than gRPC timeout
* docs: fix link style
2021-11-17 12:19:53 +00:00
bc11244828
feat: Rename database disown/adopt to release/claim ( #3111 )
...
* fix: Rename 'disown' to 'release' database
Connects to #3110
* fix: Rename 'adopt' to 'claim' database
Fixes #3110 .
2021-11-15 20:28:09 +00:00
af28cfa2a6
feat: Add an adopt database API
...
Fixes #2679 .
2021-11-15 09:26:06 -05:00
236edc0e7e
Merge branch 'main' into crepererum/improve_router_client_errors
2021-11-15 11:36:57 +00:00
9868430c12
Merge branch 'main' into crepererum/fix_server_id_on_router_startup
2021-11-15 10:29:21 +00:00
a6d83a3026
feat: WriteBufferReader use DmlOperation ( #2731 ) ( #3096 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-15 10:19:54 +00:00
48d0388ea0
feat: improve `RouterClient` errors
2021-11-15 11:18:46 +01:00
d5d07b6eeb
Merge branch 'main' into crepererum/issue3075b
2021-11-15 08:17:16 +00:00
376471591f
Merge branch 'main' into cn/disown
2021-11-12 19:31:03 +00:00
8bdfb253ce
fix: Remove context from disown operation
2021-11-12 11:10:20 -05:00
815347b18a
fix: Disown in the disown tests, not delete
2021-11-12 09:47:25 -05:00
3220970fea
fix: server IDs for routers on startup
2021-11-12 11:31:52 +01:00
f2c90d9259
tests: allow creation of test servers w/ server IDs set on startup
2021-11-12 11:31:52 +01:00
c1827cf2c3
refactor: `s/writer_id/server_id/g`
2021-11-12 11:31:52 +01:00
7f021f556d
tests: improve `TestConfig` to better support server types
2021-11-12 11:31:52 +01:00
8c1cf00164
feat: improve and test handling of updating a server ID a 2nd time
2021-11-12 11:31:52 +01:00
7ef762a88e
tests: add `test_serving_readiness_router`
2021-11-12 11:31:52 +01:00
84a5df44e3
fix: Let structopt handle parsing Uuid
2021-11-11 14:37:19 -05:00
95c46e58bc
Merge branch 'main' into jpg/restore-uuid-bytes
2021-11-11 15:34:23 +00:00
19d5200d1c
test: Make assertion more concise and avoid an allocation, thank you clippy
2021-11-11 09:56:43 -05:00
3667cb36fa
refactor: align HTTP error handling w/ cloud 2
...
Streamline HTTP error by using the same representation as cloud 2. Also
ensure that only internal server errors result in an error-level log
entry.
Fixes #3075 .
2021-11-11 14:48:50 +01:00
6f268f8260
refactor: extract DML types ( #2731 ) ( #3084 )
...
* refactor: extract DML types (#2731 )
* chore: fmt
2021-11-11 12:34:07 +00:00
7e13cb77ac
feat: add transactions for context to database owner info on disown
2021-11-10 16:49:32 -05:00
489fad0040
feat: Add disown database to the gRPC API
2021-11-10 16:49:31 -05:00
b1e81186d3
feat: Add disown database commands to the database CLI
2021-11-10 16:49:31 -05:00
83dbfe1333
feat: gRPC PB-write interface for new router
2021-11-08 15:49:13 +01:00
de672b7599
fix: Store the restore database UUID as bytes instead of as a string
2021-11-05 15:20:18 -04:00
88868e7496
feat: remove legacy write service from influxdb_iox_client ( #3043 )
...
* feat: remove legacy write service from influxdb_iox_client
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-05 17:26:18 +00:00
c8d80e5c28
fix: Change database paths to be under /dbs/ instead of under /[server id]/
2021-11-05 10:14:06 -04:00
898567e221
feat: migrate server to DbWrite ( #2724 ) ( #3035 )
...
* feat: migrate server to DbWrite (#2724 )
* chore: print perf log output
* fix: don't suppress CI status code
* chore: review feedback
* fix: don't error on empty line protocol write payloads
* fix: test
* fix: test
2021-11-05 11:09:33 +00:00
06dfdc4af8
feat: simplify sharding ( #3031 )
...
* feat: simplify sharding
* chore: fix lint
2021-11-04 16:57:19 +00:00
78101eb52d
feat: implement `RemoteService` for new router mode
2021-11-04 15:35:05 +01:00
4840dc11cb
feat: add basic CRUD operations for router configs
2021-11-04 12:03:14 +01:00
07ba629e2b
feat: migrate write buffer producer to MutableBatch and pbdata ( #2743 ) ( #3021 )
...
* feat: migrate write buffer producer to MutableBatch and pbdata (#2743 )
* fix: Kafka message content type header
* chore: fix doc
2021-11-04 10:20:40 +00:00
0f62bb1686
fix: Distinguish between restoring twice and name collisions on restore
...
And return them in the correct situations.
2021-11-03 15:02:22 -04:00
52d9df5caf
fix: Thread a Not Found error on restore database through all APIs
2021-11-03 15:02:21 -04:00
ac75be6b78
fix: Don't require name for database restore CLI; it's now unused
2021-11-03 15:02:21 -04:00
3e44f17698
fix: Return a useful error on attempt to restore an active database
2021-11-03 15:02:18 -04:00
f6327b1940
fix: Send back the correct error when attempting to restore the same database twice
2021-11-03 15:01:56 -04:00
a0b4b6b8f9
test: Add assertions that ensure UUID doesn't change when DatabaseRules are updated
2021-11-03 15:01:27 -04:00
62e7376394
fix: Restore database should only need UUID, not database name too
2021-11-03 15:01:26 -04:00
1cf697f235
refactor: make server fixture creation less confusing
2021-11-03 16:34:51 +01:00
a91e572b43
feat: implement basic router server type
...
This only implements the deployment service and nothing else yet.
2021-11-03 16:34:51 +01:00
39a7f3069f
feat: add alternative string encodings to pbdata ( #2724 ) ( #3006 )
...
* feat: add alternative string encodings to pbdata (#2724 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-02 18:14:34 +00:00
0d0c0cb42b
refactor: move write buffer configs to new home
...
Write buffer configs will partially be shared by database and router
nodes, so lets move them into a shared home.
2021-11-02 10:17:01 +01:00
011af2d6ba
feat: add `DeploymentService`
...
Ref #2980 .
2021-11-01 17:55:40 +01:00
8c3446ac87
Merge branch 'main' into crepererum/issue2980b
2021-11-01 16:39:32 +00:00
f1a6468e7b
feat: migrate write buffer consumer to use DbWrite ( #2724 ) ( #3003 )
...
* feat: migrate write buffer consumer to use DbWrite (#2724 )
* fix: doc
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-01 16:38:48 +00:00
bcd66c555a
feat: add `RemoteService`
...
Ref #2980 .
2021-11-01 11:38:46 +01:00
fc25e313c6
Merge branch 'main' into cn/iox-obj-sto-refactors
2021-10-29 13:53:26 +00:00
f3edda11e7
fix: Print all delete database output to stdout
2021-10-29 09:46:02 -04:00
83e4514a43
feat: add `DeleteService`
...
Ref #2980 .
2021-10-29 11:57:31 +02:00
8a2410e161
feat: mutable batch write entry ( #2724 ) ( #2973 )
...
* feat: mutable batch write entry (#2724 )
* chore: lint
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-28 20:15:28 +00:00
0bccd35f8c
fix: List details about databases only prints active dbs and their UUIDs
...
We can no longer list deleted databases because the server no longer
knows about them, and we now have UUIDs that are useful to know about,
so change the detailed listing of databases to return UUID instead of
possible deleted at time.
2021-10-28 13:20:29 -04:00
ce55cf401c
feat: Return UUID when creating a database
...
Seems polite and is useful for some test setup
2021-10-28 13:20:29 -04:00
4077d3a20a
fix: Take a UUID when restoring a database
2021-10-28 13:20:29 -04:00
95ba84edca
fix: Return UUID when a database is deleted
...
So that it can be immediately used to restore elsewhere.
2021-10-28 13:20:29 -04:00
e80b902d15
test: Delete tests either irrelevant or covered elsewhere
...
Now that the database paths use UUIDs instead of names+generation ids,
and that IoxObjectStore checks there are no files in a directory before
it'll create a database directory (and has tests for that).
2021-10-28 13:20:28 -04:00
a6e0a069cc
refactor: rename "run mode" to "server type"
2021-10-28 11:16:41 +02:00
6e29cec502
refactor: rename `query` mode to `database`
2021-10-27 09:31:46 +02:00
41bc373fe8
refactor: offer different modes via "run" CLI
...
Running a server is now using `influxdb_iox run MODE [args]`, e.g.
`influxdb_iox run query --server-id 1`. Another mode that will follow
soon is `router`.
The old syntax `influxdb_iox run [args]` (w/o the mode part) is still
supported but a deprecation note will be printed.
2021-10-27 09:31:46 +02:00
a0cdca7781
chore: move `influxdb_iox` into a proper workspace package
2021-10-26 11:02:33 +02:00
Marco Neumann
kodiakhq[bot]
Nga Tran
Andrew Lamb
Carol (Nichols || Goulding)
Raphael Taylor-Davies
Edd Robinson
Carol (Nichols || Goulding)
Jake Goulding
Carol (Nichols || Goulding)