d8ed8a3bf7
refactor: address review comments
2021-12-06 12:06:07 -05:00
65660dace5
Merge branch 'main' into ntran/grpc_compact_os_chunks
2021-12-03 18:16:42 -05:00
5f3706e0ee
feat: grpc call for copact object store chunks
2021-12-03 18:01:28 -05:00
7dea2ef362
feat: Add force flag to ClaimDatabase ( #3284 )
...
* feat: Add force flag to ClaimDatabase
* fix: Update name of test
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* docs: Update influxdb_iox/tests/end_to_end_cases/management_cli.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2021-12-03 21:11:23 +00:00
f17b670fe9
refactor: cleanup gRPC client error handling ( #3251 ) ( #3277 )
...
* refactor: cleanup gRPC client error handling (#3251 )
* chore: review feedback
* chore: review feedback
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-02 20:56:58 +00:00
2857b6a990
Merge branch 'main' into er/feat/load_chunk_cli
2021-12-02 20:20:56 +00:00
b4ea9887ba
refactor: error name
2021-12-02 20:14:02 +00:00
7d4693ed8c
fix: Box contents of an enum to make the variants close in size
...
Identified by clippy
https://rust-lang.github.io/rust-clippy/master/index.html#large_enum_variant
2021-12-02 11:52:02 -05:00
5d0fd1c603
fix: Allow dead code on fields that are now detected as never read
2021-12-02 11:52:01 -05:00
4153311ff5
feat: implement loading chunk CLI
2021-12-02 12:59:52 +00:00
9e45ce4121
feat: cleanup gRPC server errors ( #3251 ) ( #3265 )
...
* feat: add precondition violation enumeration (#3251 )
* chore: review feedback
2021-12-01 13:33:17 +00:00
f1936f0b08
refactor: add resource type enumeration ( #3251 ) ( #3260 )
...
* refactor: add resource type enumeration (#3251 )
* chore: more error details
2021-12-01 10:26:12 +00:00
05064eba62
fix: don't error on delete to non-existent table ( #3259 ) ( #3263 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-01 00:40:05 +00:00
c161d1176a
refactor: cleanup Db write path error handling ( #3258 )
2021-11-30 15:05:02 +00:00
1e515a1dec
feat: load RUB from object store ( #3224 ) ( #3250 )
2021-11-30 14:39:52 +00:00
88acf3788e
feat: rebuild catalog ( #3225 ) ( #3253 )
...
* feat: rebuild catalog (#3225 )
* chore: fix doc
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-30 12:17:27 +00:00
f165faf5bc
feat: support forcing persistence of partition irrespective of arrival time ( #3249 )
...
* feat: force persistence of partition irrespective of arrival time
* feat: expose to CLI
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-29 22:07:35 +00:00
837e029ed8
refactor: cleanup database recovery ( #3225 ) ( #3245 )
...
* refactor: cleanup database recovery (#3225 )
* chore: fix errors
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-29 18:49:46 +00:00
8fe2c64f5a
fix: Implement a dummy Storage.Offsets method
...
Closes https://github.com/influxdata/conductor/issues/766
2021-11-26 16:39:07 +01:00
ac20fecf6f
refactor: remove write buffer producer logic from `Db`
...
As a side effect, writing to database no longer requires `async`.
Closes #2243 .
2021-11-26 12:02:27 +01:00
e5bfa08910
feat: disallow unsequenced deletes when reading from write buffer
...
Similar to how writes to a databases that reads from a write buffer are
forbidden deletes should be rejected as well.
2021-11-26 11:40:08 +01:00
22936abb23
refactor: remove connection manager and resolver from `Server`
2021-11-26 11:00:19 +01:00
0e06026fbd
refactor: remove write routing logic from `Server`
...
API users should use `Database` instead, there's no need any longer to mirror
this API in `Server`. Note that `Database` is better than `Db` in this
case, because the former can also check if we're writing from a write
buffer and can easily reject unsequenced writes. The `pub` modifiers
were adjusted to make it impossible to write through `Db` directly.
2021-11-26 09:26:43 +01:00
6e1f86ca48
Merge branch 'main' into crepererum/remove_routing_from_database_mode_1
2021-11-24 14:15:36 +00:00
9bece6808d
Merge branch 'main' into crepererum/issue3154_b
2021-11-24 13:29:45 +00:00
eca7798308
refactor: update influxdb_iox/src/influxdb_ioxd/server_type/database/rpc/storage/service.rs
2021-11-24 10:47:33 +00:00
3eb5d42fe7
fix: address review comments
2021-11-24 11:37:45 +01:00
311375bdac
refactor: add debug logging for meta queries
2021-11-24 10:31:09 +00:00
539c08125b
feat: HTTP delete support for router
...
Closes #3154 .
2021-11-24 11:13:56 +01:00
cae8b8572d
refactor: remove remote API from database node
...
We can easily bring it back in case we need it, but at the moment its
not required for database nodes to manage remotes.
2021-11-24 11:12:22 +01:00
ed54100e60
refactor: convert `write_api` end2end tests to use new router mode
2021-11-24 11:12:22 +01:00
aa636eaed9
refactor: remove methods from `ManagementService` that were moved to `RemoteService`
...
Tests are covered by `remote_api.rs`.
Ref #2980 .
2021-11-24 10:01:32 +01:00
5ecaa6d8a9
refactor: use serialized predicates instead of strings for gRPC deletes
...
IOx is the only consumer of this API so we might just use the serialized
form. Cloud2 uses the HTTP API which supports SQL-like predicates.
Fixes #3192 .
2021-11-24 09:30:32 +01:00
53139edef9
Merge branch 'main' into crepererum/issue2980h
2021-11-24 08:21:07 +00:00
dc6b44818d
refactor: remove methods from `ManagementService` that were moved to `DeploymentService`
...
Tests are covered by `deployment_api.rs`.
Ref #2980 .
2021-11-23 18:04:41 +01:00
3a7cb119c6
feat: add support for binary tag key format
2021-11-23 17:00:28 +00:00
dde1db40d1
refactor: tidy up db_name move
2021-11-23 17:00:28 +00:00
8784ae0d93
refactor: move entire request message into impl
2021-11-23 17:00:28 +00:00
c9a5726d7b
Merge branch 'main' into er/refactor/grpc_tag_order
2021-11-23 16:39:32 +00:00
2ef3ab8544
refactor: remove methods from `ManagementService` that were moved to `DeleteService`
...
Tests are covered by `delete_api.rs`.
Ref #2980 .
2021-11-23 16:40:00 +01:00
5b200ee564
feat: impl delete gRPC interface for router
2021-11-23 15:39:53 +01:00
df19d67532
test: allow to specify `WriteBufferConfigFactory` when creating a test `RouterServer`
2021-11-23 15:39:53 +01:00
65dc0bd4b3
test: update tests
2021-11-23 14:15:06 +00:00
521a8e3636
test: fix tests
2021-11-22 21:40:41 +00:00
f69d37e9a8
fix: Remove database delete/restore entirely
2021-11-17 12:03:11 -05:00
7783e4a7ff
fix: Make delete/restore aliases for release/claim; remove tombstone
...
Fixes #2680
2021-11-17 11:41:08 -05:00
8155747735
feat: add write buffer delete encoding ( #2731 ) ( #3127 )
...
* feat: add write buffer delete encoding (#2731 )
* chore: fix doc
* chore: review feedback
* chore: review feedback
* chore: fmt
* chore: review feedback
2021-11-17 16:12:19 +00:00
38ca9e1339
fix: capture all panic messages in logs ( #3130 )
2021-11-16 21:59:05 +00:00
553e412226
refactor: DMLOperation write path ( #2731 ) ( #3121 )
...
* refactor: DMLOperation write path (#2731 )
* chore: fmt
* chore: review feedback
2021-11-16 12:42:19 +00:00
bc11244828
feat: Rename database disown/adopt to release/claim ( #3111 )
...
* fix: Rename 'disown' to 'release' database
Connects to #3110
* fix: Rename 'adopt' to 'claim' database
Fixes #3110 .
2021-11-15 20:28:09 +00:00
d759d98612
fix: Update new code with API that changed since branching from main
2021-11-15 14:32:50 -05:00
cc693a780e
Merge branch 'main' into cn+jpg/adopt
2021-11-15 19:22:07 +00:00
3545f6d65a
fix: Pass through error for already-owned database
2021-11-15 14:15:56 -05:00
1d68980e4f
fix: typo
...
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2021-11-15 18:37:31 +01:00
c88930a6a5
refactor: clarify `ServerType` background worker handling
...
Ref #3020 .
2021-11-15 18:28:32 +01:00
3cd7d2eda2
refactor: improve usability of proto conversion traits ( #3109 )
...
* refactor: improve usability of proto conversion traits
* chore: review feedback
2021-11-15 16:10:29 +00:00
af28cfa2a6
feat: Add an adopt database API
...
Fixes #2679 .
2021-11-15 09:26:06 -05:00
9868430c12
Merge branch 'main' into crepererum/fix_server_id_on_router_startup
2021-11-15 10:29:21 +00:00
3d091208af
refactor: move delete predicate into data_types ( #2731 ) ( #3094 )
...
* refactor: move delete predicate into dml (#2731 )
* refactor: move DeletePredicate to data_types
* chore: fix doc
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-15 10:28:58 +00:00
d5d07b6eeb
Merge branch 'main' into crepererum/issue3075b
2021-11-15 08:17:16 +00:00
376471591f
Merge branch 'main' into cn/disown
2021-11-12 19:31:03 +00:00
8bdfb253ce
fix: Remove context from disown operation
2021-11-12 11:10:20 -05:00
3220970fea
fix: server IDs for routers on startup
2021-11-12 11:31:52 +01:00
84a5df44e3
fix: Let structopt handle parsing Uuid
2021-11-11 14:37:19 -05:00
95c46e58bc
Merge branch 'main' into jpg/restore-uuid-bytes
2021-11-11 15:34:23 +00:00
3667cb36fa
refactor: align HTTP error handling w/ cloud 2
...
Streamline HTTP error by using the same representation as cloud 2. Also
ensure that only internal server errors result in an error-level log
entry.
Fixes #3075 .
2021-11-11 14:48:50 +01:00
6f268f8260
refactor: extract DML types ( #2731 ) ( #3084 )
...
* refactor: extract DML types (#2731 )
* chore: fmt
2021-11-11 12:34:07 +00:00
f0136e9791
ci: run clippy for all features
...
This indeed fixes some issues in our object store implementation.
Closes #537 .
2021-11-11 10:48:21 +01:00
7e13cb77ac
feat: add transactions for context to database owner info on disown
2021-11-10 16:49:32 -05:00
fb7bde527f
feat: Implement disown in Server and Database
2021-11-10 16:49:32 -05:00
489fad0040
feat: Add disown database to the gRPC API
2021-11-10 16:49:31 -05:00
b1e81186d3
feat: Add disown database commands to the database CLI
2021-11-10 16:49:31 -05:00
81e5c33a72
fix: Make error handling match a bit nicer to read and more consistent
2021-11-10 16:49:31 -05:00
bebb95830a
fix: Make create database CLI output consistent with delete database CLI output
2021-11-10 16:49:31 -05:00
c80088bf12
feat: HTTP write interface for router
2021-11-10 17:55:46 +01:00
721011ab51
docs: improve wording
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-11-09 13:11:32 +01:00
00b3282180
refactor: partially split up HTTP implementation and tests
...
This should make it easier to to wire up the HTTP write interface for
the router. No functional change intended, this just moves code around.
2021-11-09 12:22:31 +01:00
83dbfe1333
feat: gRPC PB-write interface for new router
2021-11-08 15:49:13 +01:00
b226d629c1
feat: router write logic
2021-11-08 14:09:33 +01:00
de672b7599
fix: Store the restore database UUID as bytes instead of as a string
2021-11-05 15:20:18 -04:00
7d070f8582
feat: remove influxdata.iox.write.v1.WriteService ( #3044 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-05 17:51:01 +00:00
88868e7496
feat: remove legacy write service from influxdb_iox_client ( #3043 )
...
* feat: remove legacy write service from influxdb_iox_client
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-05 17:26:18 +00:00
c8d80e5c28
fix: Change database paths to be under /dbs/ instead of under /[server id]/
2021-11-05 10:14:06 -04:00
898567e221
feat: migrate server to DbWrite ( #2724 ) ( #3035 )
...
* feat: migrate server to DbWrite (#2724 )
* chore: print perf log output
* fix: don't suppress CI status code
* chore: review feedback
* fix: don't error on empty line protocol write payloads
* fix: test
* fix: test
2021-11-05 11:09:33 +00:00
78101eb52d
feat: implement `RemoteService` for new router mode
2021-11-04 15:35:05 +01:00
53422dbd48
Merge branch 'main' into crepererum/router_mode_get_set_cfg
2021-11-04 14:06:44 +00:00
d28749bd93
refactor: remove Entry from Db interfaces ( #2724 ) ( #3027 )
2021-11-04 13:28:47 +00:00
4840dc11cb
feat: add basic CRUD operations for router configs
2021-11-04 12:03:14 +01:00
0f62bb1686
fix: Distinguish between restoring twice and name collisions on restore
...
And return them in the correct situations.
2021-11-03 15:02:22 -04:00
6615001dc7
fix: Extract loading the database name out of the rules file into a function
2021-11-03 15:02:21 -04:00
52d9df5caf
fix: Thread a Not Found error on restore database through all APIs
2021-11-03 15:02:21 -04:00
ac75be6b78
fix: Don't require name for database restore CLI; it's now unused
2021-11-03 15:02:21 -04:00
3e44f17698
fix: Return a useful error on attempt to restore an active database
2021-11-03 15:02:18 -04:00
207d701bdb
fix: Have DatabaseState keep track of the UUID after reading it from object storage
2021-11-03 15:01:27 -04:00
62e7376394
fix: Restore database should only need UUID, not database name too
2021-11-03 15:01:26 -04:00
e3e99c45cf
fix: remove stale comment
2021-11-03 16:35:54 +01:00
a91e572b43
feat: implement basic router server type
...
This only implements the deployment service and nothing else yet.
2021-11-03 16:34:51 +01:00
011af2d6ba
feat: add `DeploymentService`
...
Ref #2980 .
2021-11-01 17:55:40 +01:00
8c3446ac87
Merge branch 'main' into crepererum/issue2980b
2021-11-01 16:39:32 +00:00
bcd66c555a
feat: add `RemoteService`
...
Ref #2980 .
2021-11-01 11:38:46 +01:00
fc25e313c6
Merge branch 'main' into cn/iox-obj-sto-refactors
2021-10-29 13:53:26 +00:00
c02e476175
fix: Split database name from rules at the last possible function
...
When updating database rules, the database name should come from the
rules. Enforce this by only taking the rules as a parameter, rather than
having two arguments that could possibly disagree with each other.
2021-10-29 09:46:23 -04:00
f3edda11e7
fix: Print all delete database output to stdout
2021-10-29 09:46:02 -04:00
83e4514a43
feat: add `DeleteService`
...
Ref #2980 .
2021-10-29 11:57:31 +02:00
2caed5456e
Merge branch 'main' into crepererum/run_modes_boilerplate3
2021-10-29 07:06:12 +00:00
7b938fa521
docs: extend `CommonServerState` docs
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-10-29 09:05:29 +02:00
f340738aa5
docs: explain `influxdb_ioxd::main`
2021-10-29 09:04:39 +02:00
3c22d93e38
fix: remove TODO error handling
2021-10-29 08:59:04 +02:00
8a2410e161
feat: mutable batch write entry ( #2724 ) ( #2973 )
...
* feat: mutable batch write entry (#2724 )
* chore: lint
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-28 20:15:28 +00:00
088cdd8f97
fix: Share comfy_table style and give it a more descriptive name and docs
2021-10-28 16:00:58 -04:00
0bccd35f8c
fix: List details about databases only prints active dbs and their UUIDs
...
We can no longer list deleted databases because the server no longer
knows about them, and we now have UUIDs that are useful to know about,
so change the detailed listing of databases to return UUID instead of
possible deleted at time.
2021-10-28 13:20:29 -04:00
ce55cf401c
feat: Return UUID when creating a database
...
Seems polite and is useful for some test setup
2021-10-28 13:20:29 -04:00
4077d3a20a
fix: Take a UUID when restoring a database
2021-10-28 13:20:29 -04:00
95ba84edca
fix: Return UUID when a database is deleted
...
So that it can be immediately used to restore elsewhere.
2021-10-28 13:20:29 -04:00
09ad30cbfd
fix: For dumping a db catalog, look up db location in server config first
...
Rather than requiring the user of this command to know the database UUID
or otherwise having the knowledge of how to construct a database object
store location in this debugging command.
2021-10-28 13:20:28 -04:00
f160712f5e
fix: Remove the concept of generations
...
From iox_object_store and everywhere that fails to compile as a result.
Also make path handling in iox_object_store a bit more consistent.
2021-10-28 13:20:28 -04:00
8198c1ff2a
refactor: Rename IoxObjectStore constructors to better match what server does with Databases
2021-10-28 13:20:27 -04:00
641c7b3051
refactor: make `influxdb_ioxd` server-type-generic
2021-10-28 17:38:06 +02:00
9996cfc6b3
fix: things I forgot to rename
2021-10-28 16:33:03 +02:00
9cd3dfb624
refactor: make gRPC server code reusable
...
Sadly `tonic` is a bit of a type nightmare (see
<23c1392fb7/tonic/src/transport/server/mod.rs (L452-L470)
2021-10-28 13:19:24 +02:00
a6e0a069cc
refactor: rename "run mode" to "server type"
2021-10-28 11:16:41 +02:00
843551d291
docs: improve
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-10-28 11:16:41 +02:00
c465d0af8f
refactor: abstract HTTP inteface for different run modes
...
First iteration of a common interface for the different server run
modes. This splits the existing HTTP server into a part that can be used
by all run modes (e.g. health checks, metrics endpoint, profiling) and a
part that is specific to the the database run mode.
Note that the interface is not final and might require more iterations,
but we have to start somewhere.
2021-10-28 11:16:41 +02:00
6ebca13ed7
Merge branch 'main' into crepererum/prepare_run_mode_cli
2021-10-27 11:04:07 +00:00
cdc0c63cb7
feat: Add LP batch size histogram metric
...
Part of #2752
2021-10-27 12:14:48 +02:00
6e29cec502
refactor: rename `query` mode to `database`
2021-10-27 09:31:46 +02:00
41bc373fe8
refactor: offer different modes via "run" CLI
...
Running a server is now using `influxdb_iox run MODE [args]`, e.g.
`influxdb_iox run query --server-id 1`. Another mode that will follow
soon is `router`.
The old syntax `influxdb_iox run [args]` (w/o the mode part) is still
supported but a deprecation note will be printed.
2021-10-27 09:31:46 +02:00
a0cdca7781
chore: move `influxdb_iox` into a proper workspace package
2021-10-26 11:02:33 +02:00
Nga Tran
Andrew Lamb
Raphael Taylor-Davies
kodiakhq[bot]
Edd Robinson
Carol (Nichols || Goulding)
Marko Mikulicic
Marco Neumann
Carol (Nichols || Goulding)
Jake Goulding
Carol (Nichols || Goulding)