Commit Graph

1416 Commits (4b47d723b105a87ef1e4c850de07469f5b75336a)

Author SHA1 Message Date
Edd Robinson a08246e3c7
Merge branch 'main' into er/refactor/debug_rub_query 2021-12-02 19:42:46 +00:00
Carol (Nichols || Goulding) 0683f851e9
fix: Derive Default instead of doing an equivalent manual impl
Identified by clippy. The impl did have `#[cfg(test)]` on it, but the
entire struct does too, so this is fine.

https://rust-lang.github.io/rust-clippy/master/index.html#derivable_impls
2021-12-02 11:52:02 -05:00
Carol (Nichols || Goulding) 5d0fd1c603
fix: Allow dead code on fields that are now detected as never read 2021-12-02 11:52:01 -05:00
Edd Robinson 4a4e11f08b refactor: adjust debug logging 2021-12-02 12:19:51 +00:00
Nga Tran f085af034e
refactor: not persist empty chunk resulting from deleting & deduplicating (#3274)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-01 20:57:30 +00:00
kodiakhq[bot] e8edab0b43
Merge branch 'main' into ntran/compact_os_preserved_catalog 2021-12-01 18:26:33 +00:00
Nga Tran de54d2843c chore: remove unused error 2021-12-01 13:24:12 -05:00
Nga Tran 60807c2d95 refactor: address review comments 2021-12-01 13:11:17 -05:00
Nga Tran 254d35a6c2 refactor: address review comment 2021-12-01 12:25:42 -05:00
Nga Tran 878c59c0d2 chore: cleanup 2021-12-01 10:25:35 -05:00
Nga Tran 1ac32f1d90 refactor: avoid lock gap while updating in-memory catalog 2021-12-01 10:24:58 -05:00
Raphael Taylor-Davies 05064eba62
fix: don't error on delete to non-existent table (#3259) (#3263)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-01 00:40:05 +00:00
Nga Tran f53cdca010 feat: handling empty compacted stream 2021-11-30 18:13:36 -05:00
Nga Tran 6bb7a796e7 chore: Merge branch 'main' into ntran/compact_os_preserved_catalog 2021-11-30 16:55:38 -05:00
Nga Tran e8034cdc53 refactor: handle delete predicates for compacting os 2021-11-30 16:45:04 -05:00
Raphael Taylor-Davies c161d1176a
refactor: cleanup Db write path error handling (#3258) 2021-11-30 15:05:02 +00:00
Raphael Taylor-Davies 1e515a1dec
feat: load RUB from object store (#3224) (#3250) 2021-11-30 14:39:52 +00:00
Andrew Lamb 20da62eb87
chore: Update `arrow`, `arrow-flight`, and `parquet` to 6.3.0 (#3254)
* chore: Update `arrow`,  `arrow-flight`, and `parquet` to 6.3.0

* fix: update metadata size
2021-11-30 12:30:17 +00:00
Raphael Taylor-Davies 88acf3788e
feat: rebuild catalog (#3225) (#3253)
* feat: rebuild catalog (#3225)

* chore: fix doc

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-30 12:17:27 +00:00
Nga Tran 7afc20142b test: add tests that do actual compacting chunks 2021-11-29 22:59:34 -05:00
Nga Tran 37c4bb402b feat: update os compacted chunk in both in memory and preserved catalog 2021-11-29 15:17:00 -05:00
Raphael Taylor-Davies 837e029ed8
refactor: cleanup database recovery (#3225) (#3245)
* refactor: cleanup database recovery (#3225)

* chore: fix errors

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-29 18:49:46 +00:00
Raphael Taylor-Davies 0517adf752
feat: test interaction of write buffer and lifecycle (#2811) (#3211)
* feat: test interaction of write buffer and lifecycle (#2811)

* chore: more docs
2021-11-29 14:16:43 +00:00
kodiakhq[bot] 23dffefcf8
Merge branch 'main' into crepererum/remove_routing_from_database_mode_7 2021-11-29 11:38:12 +00:00
Raphael Taylor-Davies 197634ed50
feat: reload chunk back into read buffer (#3209) (#3216)
* feat: reload chunk back into read buffer (#3209)

* chore: fix logical conflict

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-29 11:34:55 +00:00
Marco Neumann 4e043ecb55 refactor: remove old routing / sharding config
This is superseded by the new router subsystem.
2021-11-29 12:33:48 +01:00
dependabot[bot] 3bedf6e62a
chore(deps): bump crc32fast from 1.2.1 to 1.2.2
Bumps [crc32fast](https://github.com/srijs/rust-crc32fast) from 1.2.1 to 1.2.2.
- [Release notes](https://github.com/srijs/rust-crc32fast/releases)
- [Commits](https://github.com/srijs/rust-crc32fast/compare/v1.2.1...v1.2.2)

---
updated-dependencies:
- dependency-name: crc32fast
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-11-29 09:00:01 +00:00
Raphael Taylor-Davies a286960285
refactor: remove delete predicates from read filter interface (#3212)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-26 12:15:29 +00:00
Marco Neumann 7f2e4f4342 refactor: remove write buffer direction
The direction was required when a database could read or write from/to a
write buffer. Now it is clear from the usage context of a write buffer
context which of the two applications is meant (databases read, routers
write) so the direction flag is no longer required.
2021-11-26 12:38:40 +01:00
Marco Neumann ac20fecf6f refactor: remove write buffer producer logic from `Db`
As a side effect, writing to database no longer requires `async`.

Closes #2243.
2021-11-26 12:02:27 +01:00
Marco Neumann e5bfa08910 feat: disallow unsequenced deletes when reading from write buffer
Similar to how writes to a databases that reads from a write buffer are
forbidden deletes should be rejected as well.
2021-11-26 11:40:08 +01:00
Marco Neumann 22936abb23 refactor: remove connection manager and resolver from `Server` 2021-11-26 11:00:19 +01:00
Marco Neumann 0e06026fbd refactor: remove write routing logic from `Server`
API users should use `Database` instead, there's no need any longer to mirror
this API in `Server`. Note that `Database` is better than `Db` in this
case, because the former can also check if we're writing from a write
buffer and can easily reject unsequenced writes. The `pub` modifiers
were adjusted to make it impossible to write through `Db` directly.
2021-11-26 09:26:43 +01:00
Nga Tran 4e7e9f6fc9 chore: Merge branch 'ntran/compact_os_blocks' of https://github.com/influxdata/influxdb_iox into ntran/compact_os_blocks 2021-11-23 16:56:59 -05:00
Nga Tran b2b033f00c chore: rerun fmt after merging 2021-11-23 16:56:38 -05:00
kodiakhq[bot] 1d1622efa3
Merge branch 'main' into ntran/compact_os_blocks 2021-11-23 21:47:05 +00:00
Nga Tran ade7ea6cfb chore: Merge branch 'ntran/compact_os_blocks' of https://github.com/influxdata/influxdb_iox into ntran/compact_os_blocks 2021-11-23 16:44:48 -05:00
Nga Tran a0fb6a4758 chore: cleanup 2021-11-23 16:44:30 -05:00
Nga Tran 4fc09484ed
chore: Apply suggestions from code review
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2021-11-23 16:43:12 -05:00
Nga Tran 4e28b4c43b refactor: address review comments 2021-11-23 16:39:45 -05:00
Nga Tran e64ac10c40 refactor: address review comments and compute max checkpoints 2021-11-23 15:32:46 -05:00
Raphael Taylor-Davies 42e3a41825
fix: flaky test_chunk_access_time (#3197) (#3200)
* fix: flaky test_chunk_access_time (#3197)

* chore: fix lint

Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-23 17:39:57 +00:00
Raphael Taylor-Davies bfa4bfe735
refactor: lifecycle observability cleanup (#2286) (#3189)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-23 15:42:24 +00:00
Marco Neumann f9110eecad test: migrate `DistibutedTest` to use proper router 2021-11-23 15:39:53 +01:00
Nga Tran 124f5bb4c5 chore: make comments clearer 2021-11-22 23:30:18 -05:00
Nga Tran 8788dd6d39 chore: fix comment 2021-11-22 23:24:28 -05:00
Nga Tran 3182e935ed chore: cleanup 2021-11-22 23:22:34 -05:00
Nga Tran 06aba60068 feat: another piece: tests for contiguous chunks and read IOx metadata to get checkpoints 2021-11-22 23:15:20 -05:00
kodiakhq[bot] 29aaafe808
Merge branch 'main' into ntran/compact_os_catalog 2021-11-22 19:19:57 +00:00
Nga Tran 5e45c5a349 refactor: add meessage to assert 2021-11-22 13:28:08 -05:00
Nga Tran 731dee4e91 refactor: address review comments 2021-11-22 13:18:54 -05:00
kodiakhq[bot] d16a7759ca
Merge branch 'main' into cn/workspace-hack 2021-11-22 17:05:31 +00:00
kodiakhq[bot] b5d6e201e6
Merge branch 'main' into ntran/compact_os_framework 2021-11-22 15:17:13 +00:00
Nga Tran 09fd25d28e refactor: address to cleanup-lock issue 2021-11-22 10:15:43 -05:00
Raphael Taylor-Davies 0b55cee5f2
feat: skip applying deletes to persisted chunks during replay (#3125) (#3153)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-22 10:38:25 +00:00
Raphael Taylor-Davies 73d60539ad
refactor: use ChunkGenerator in parquet_catalog (#2209) (#3167)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-22 10:29:33 +00:00
Nga Tran 3b2e5c445b chore: Merge branch 'main' into ntran/compact_os_framework 2021-11-19 16:18:24 -05:00
Nga Tran 40ddcf2a2f refactor: address review comments 2021-11-19 16:08:01 -05:00
Raphael Taylor-Davies ff78f5a4f5
feat: reduce cardinality of job metrics (#3169)
* feat: reduce cardinality of job metrics

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-19 19:05:49 +00:00
Andrew Lamb 76d02deb4c
refactor: form `system.column_chunks` column by column rather than row by row (#3155)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-19 18:34:29 +00:00
Carol (Nichols || Goulding) 9fd4a560f5
feat: Results of running cargo hakari manage-deps 2021-11-19 09:21:57 -05:00
Carol (Nichols || Goulding) 25d55cd08a
feat: Move server config paths beneath 'nodes' (#3144)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-19 09:54:32 +00:00
Raphael Taylor-Davies e32d367e85
feat: flush delete mailbox on persist (#3126) (#3147)
* feat: flush delete mailbox on persist (#3126)

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-19 09:45:29 +00:00
Nga Tran c148251dcb feat: implement step2: compact and persist os chunks 2021-11-18 18:18:55 -05:00
Carol (Nichols || Goulding) a2454b542d
fix: Small cleanups in Cargo.tomls (#3160)
* fix: Add tokio rt-multi-thread feature so cargo test -p client_util compiles

* fix: Alphabetize dependencies

* fix: Add the data_types_conversions feature to get tests passing

* fix: Remove dev dependencies already listed under normal dependencies

* fix: Make sure the workspace is using the new resolver
2021-11-18 22:26:33 +00:00
kodiakhq[bot] c9f02f83e7
Merge branch 'main' into cn/alias-db-commands 2021-11-18 15:13:43 +00:00
Nga Tran ccef3b535a feat: clean up and add comments for next steps 2021-11-18 10:11:51 -05:00
Raphael Taylor-Davies 714fc85c8d
refactor: extract Mailbox type (#3126) (#3142)
* refactor: extract Mailbox type (#3126)

* fix: doc

* chore: review feedback

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-11-18 09:34:06 +00:00
Nga Tran a5c04e5fe4 feat: framework for compact os chunks 2021-11-17 18:12:51 -05:00
Carol (Nichols || Goulding) f69d37e9a8
fix: Remove database delete/restore entirely 2021-11-17 12:03:11 -05:00
Carol (Nichols || Goulding) 7783e4a7ff
fix: Make delete/restore aliases for release/claim; remove tombstone
Fixes #2680
2021-11-17 11:41:08 -05:00
Raphael Taylor-Davies 8155747735
feat: add write buffer delete encoding (#2731) (#3127)
* feat: add write buffer delete encoding (#2731)

* chore: fix doc

* chore: review feedback

* chore: review feedback

* chore: fmt

* chore: review feedback
2021-11-17 16:12:19 +00:00
Andrew Lamb 47acd181c5
chore: Update datafusion + arrow/parquet/arrow-flight 6.2.0 (#3136)
* chore: Update datafusion and arrow

* chore: update arrow/parquet/arrow-flight to 6.2.0

* refactor: Add table_exists to SchemaProvider impl

* fix: clippy

* fix: clippy 2

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-17 14:04:49 +00:00
Marco Neumann e6fdd79a0f feat: emit Kafka stats as metrics instead of logs
This maps a subset of Kafka stats as metrics. The set can -- of course
-- be changed in the future depending on our needs.

Fixes #3100.
2021-11-16 17:18:41 +01:00
Raphael Taylor-Davies 553e412226
refactor: DMLOperation write path (#2731) (#3121)
* refactor: DMLOperation write path (#2731)

* chore: fmt

* chore: review feedback
2021-11-16 12:42:19 +00:00
Carol (Nichols || Goulding) bc11244828
feat: Rename database disown/adopt to release/claim (#3111)
* fix: Rename 'disown' to 'release' database

Connects to #3110

* fix: Rename 'adopt' to 'claim' database

Fixes #3110.
2021-11-15 20:28:09 +00:00
Jake Goulding af28cfa2a6
feat: Add an adopt database API
Fixes #2679.
2021-11-15 09:26:06 -05:00
Raphael Taylor-Davies 3d091208af
refactor: move delete predicate into data_types (#2731) (#3094)
* refactor: move delete predicate into dml (#2731)

* refactor: move DeletePredicate to data_types

* chore: fix doc

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-15 10:28:58 +00:00
Raphael Taylor-Davies a6d83a3026
feat: WriteBufferReader use DmlOperation (#2731) (#3096)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-15 10:19:54 +00:00
Carol (Nichols || Goulding) 48f36fab1f
fix: Box an error so the enum variants don't differ greatly in size 2021-11-12 14:53:07 -05:00
kodiakhq[bot] 376471591f
Merge branch 'main' into cn/disown 2021-11-12 19:31:03 +00:00
Carol (Nichols || Goulding) 46c96940f0
feat: Improve log messages to be more specific
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-11-12 14:21:44 -05:00
Carol (Nichols || Goulding) 63ee5b8317
fix: Shut down databases before deleting or disowning them 2021-11-12 13:37:24 -05:00
Carol (Nichols || Goulding) 56c16414a1
fix: Use shared application time rather than passing in a timestamp 2021-11-12 11:32:02 -05:00
Carol (Nichols || Goulding) 315cc3836b
fix: Combine checks of the database state 2021-11-12 11:17:17 -05:00
Carol (Nichols || Goulding) 8bdfb253ce
fix: Remove context from disown operation 2021-11-12 11:10:20 -05:00
Carol (Nichols || Goulding) 2b8f0b7eeb
fix: Don't hardcode timestamp in restore code 2021-11-11 11:44:19 -05:00
Raphael Taylor-Davies 6f268f8260
refactor: extract DML types (#2731) (#3084)
* refactor: extract DML types (#2731)

* chore: fmt
2021-11-11 12:34:07 +00:00
Carol (Nichols || Goulding) 7e13cb77ac
feat: add transactions for context to database owner info on disown 2021-11-10 16:49:32 -05:00
Carol (Nichols || Goulding) fb7bde527f
feat: Implement disown in Server and Database 2021-11-10 16:49:32 -05:00
Carol (Nichols || Goulding) 489fad0040
feat: Add disown database to the gRPC API 2021-11-10 16:49:31 -05:00
Carol (Nichols || Goulding) e160954b57
refactor: Extract functions for creating and updating owner info to match the fetching function 2021-11-10 16:49:31 -05:00
Carol (Nichols || Goulding) d42c890333
refactor: Extract a function for reading a database's owner file 2021-11-10 16:49:31 -05:00
Carol (Nichols || Goulding) 3dda694520
refactor: Extract a macro for matching errors in tests 2021-11-08 11:14:23 -05:00
Carol (Nichols || Goulding) c8d80e5c28
fix: Change database paths to be under /dbs/ instead of under /[server id]/ 2021-11-05 10:14:06 -04:00
Raphael Taylor-Davies 898567e221
feat: migrate server to DbWrite (#2724) (#3035)
* feat: migrate server to DbWrite (#2724)

* chore: print perf log output

* fix: don't suppress CI status code

* chore: review feedback

* fix: don't error on empty line protocol write payloads

* fix: test

* fix: test
2021-11-05 11:09:33 +00:00
Raphael Taylor-Davies 06dfdc4af8
feat: simplify sharding (#3031)
* feat: simplify sharding

* chore: fix lint
2021-11-04 16:57:19 +00:00
Raphael Taylor-Davies d28749bd93
refactor: remove Entry from Db interfaces (#2724) (#3027) 2021-11-04 13:28:47 +00:00
Raphael Taylor-Davies 07ba629e2b
feat: migrate write buffer producer to MutableBatch and pbdata (#2743) (#3021)
* feat: migrate write buffer producer to MutableBatch and pbdata (#2743)

* fix: Kafka message content type header

* chore: fix doc
2021-11-04 10:20:40 +00:00
Andrew Lamb 46213a5a28 chore: update dependencies 2021-11-03 17:00:20 -04:00
Carol (Nichols || Goulding) 0f62bb1686
fix: Distinguish between restoring twice and name collisions on restore
And return them in the correct situations.
2021-11-03 15:02:22 -04:00
Carol (Nichols || Goulding) 6615001dc7
fix: Extract loading the database name out of the rules file into a function 2021-11-03 15:02:21 -04:00
Carol (Nichols || Goulding) 3e44f17698
fix: Return a useful error on attempt to restore an active database 2021-11-03 15:02:18 -04:00
Carol (Nichols || Goulding) 207d701bdb
fix: Have DatabaseState keep track of the UUID after reading it from object storage 2021-11-03 15:01:27 -04:00
Carol (Nichols || Goulding) b1504f17ed
fix: Detangle UUID from ProvidedDatabaseRules 2021-11-03 15:01:27 -04:00
Carol (Nichols || Goulding) f2cb79bfa0
test: Add more tests around database restore 2021-11-03 15:01:26 -04:00
Carol (Nichols || Goulding) 62e7376394
fix: Restore database should only need UUID, not database name too 2021-11-03 15:01:26 -04:00
kodiakhq[bot] bbdfc47b02
Merge branch 'main' into crepererum/warn_on_failed_wb_reads 2021-11-03 17:34:14 +00:00
Marco Neumann 10de63f9b6 refactor: warn when we receive broken data from write buffer
We need to understand why this is the case and where broken data gets
stuck, because at the moment it seems that some Kafka hiccups can
poison the write buffer but it seems restarting IOx recovers the
situation.
2021-11-03 16:03:42 +01:00
Raphael Taylor-Davies 873ce27b0c
feat: reduce lock metric cardinality (#3013) 2021-11-02 17:57:02 +00:00
Marco Neumann 0d0c0cb42b refactor: move write buffer configs to new home
Write buffer configs will partially be shared by database and router
nodes, so lets move them into a shared home.
2021-11-02 10:17:01 +01:00
Raphael Taylor-Davies f1a6468e7b
feat: migrate write buffer consumer to use DbWrite (#2724) (#3003)
* feat: migrate write buffer consumer to use DbWrite (#2724)

* fix: doc

* chore: fmt

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-01 16:38:48 +00:00
kodiakhq[bot] c724bb2ee8
Merge branch 'main' into dependabot/cargo/tokio-1.13.0 2021-11-01 11:30:33 +00:00
dependabot[bot] c540b40f05
chore(deps): bump tokio from 1.12.0 to 1.13.0
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.12.0...tokio-1.13.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 11:21:59 +00:00
kodiakhq[bot] fd5d053d33
Merge branch 'main' into er/feat/read_buffer/meta_delete 2021-11-01 10:36:01 +00:00
Edd Robinson b1096d2a36
Merge branch 'main' into er/feat/read_buffer/meta_delete 2021-11-01 10:26:10 +00:00
dependabot[bot] a1f978e280
chore(deps): bump tokio-util from 0.6.8 to 0.6.9
Bumps [tokio-util](https://github.com/tokio-rs/tokio) from 0.6.8 to 0.6.9.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.6.8...tokio-util-0.6.9)

---
updated-dependencies:
- dependency-name: tokio-util
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 01:28:41 +00:00
Raphael Taylor-Davies 6ceab054ab
refactor: move DbWrite to mutable_batch (#2986)
* refactor: move DbWrite to mutable_batch

* chore: fix doc

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-29 15:13:05 +00:00
kodiakhq[bot] fc25e313c6
Merge branch 'main' into cn/iox-obj-sto-refactors 2021-10-29 13:53:26 +00:00
Carol (Nichols || Goulding) aefdb00bc2
fix: Remove empty test 2021-10-29 09:47:49 -04:00
Carol (Nichols || Goulding) c02e476175
fix: Split database name from rules at the last possible function
When updating database rules, the database name should come from the
rules. Enforce this by only taking the rules as a parameter, rather than
having two arguments that could possibly disagree with each other.
2021-10-29 09:46:23 -04:00
Andrew Lamb 4675dd5331
docs: Add some doc comments (#2984) 2021-10-29 11:21:39 +00:00
Raphael Taylor-Davies 8a2410e161
feat: mutable batch write entry (#2724) (#2973)
* feat: mutable batch write entry (#2724)

* chore: lint

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-28 20:15:28 +00:00
Carol (Nichols || Goulding) c61806f919
fix: Apply commenting and logging suggestions from code review
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-10-28 15:32:33 -04:00
Carol (Nichols || Goulding) f8b5886110
test: Add more test coverage for new behavior 2021-10-28 13:20:29 -04:00
Carol (Nichols || Goulding) d2bd8f05db
fix: Clean up error messages and comments referencing generations 2021-10-28 13:20:29 -04:00
Carol (Nichols || Goulding) 0bccd35f8c
fix: List details about databases only prints active dbs and their UUIDs
We can no longer list deleted databases because the server no longer
knows about them, and we now have UUIDs that are useful to know about,
so change the detailed listing of databases to return UUID instead of
possible deleted at time.
2021-10-28 13:20:29 -04:00
Carol (Nichols || Goulding) ce55cf401c
feat: Return UUID when creating a database
Seems polite and is useful for some test setup
2021-10-28 13:20:29 -04:00
Carol (Nichols || Goulding) 4077d3a20a
fix: Take a UUID when restoring a database 2021-10-28 13:20:29 -04:00
Carol (Nichols || Goulding) 95ba84edca
fix: Return UUID when a database is deleted
So that it can be immediately used to restore elsewhere.
2021-10-28 13:20:29 -04:00
Carol (Nichols || Goulding) e80b902d15
test: Delete tests either irrelevant or covered elsewhere
Now that the database paths use UUIDs instead of names+generation ids,
and that IoxObjectStore checks there are no files in a directory before
it'll create a database directory (and has tests for that).
2021-10-28 13:20:28 -04:00
Carol (Nichols || Goulding) 990f768cda
fix: Assign a UUID when creating a database 2021-10-28 13:20:28 -04:00
Carol (Nichols || Goulding) f160712f5e
fix: Remove the concept of generations
From iox_object_store and everywhere that fails to compile as a result.

Also make path handling in iox_object_store a bit more consistent.
2021-10-28 13:20:28 -04:00
Carol (Nichols || Goulding) 8198c1ff2a
refactor: Rename IoxObjectStore constructors to better match what server does with Databases 2021-10-28 13:20:27 -04:00
kodiakhq[bot] 04b4eeb6ec
Merge branch 'main' into cn/only-read-server 2021-10-25 15:15:17 +00:00
Marco Neumann bc7244c48e chore: use Rust edition 2021 2021-10-25 10:58:20 +02:00
Marco Neumann cd30e6beab refactor: use `SpanRecorder` instead of hand-rolled logic 2021-10-25 10:16:05 +02:00
Marco Neumann f3cc238d1c feat: create span contexts on write buffer consumer side 2021-10-25 10:04:08 +02:00
Andrew Lamb 52cf1a85b9
fix(metadata): Do not report table_names for tables that have no non-null values that match predicate (#2947)
* fix(metadata): Do not report table_names for tables that have no non-null values that match predicate

* fix: make apply_predicate_to_metadata precise

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-24 11:03:30 +00:00
Carol (Nichols || Goulding) f69a470b90
refactor: Use TestDbBuilder to rebuild db in replay tests (#2950)
* refactor: Use TestDbBuilder to rebuild db in replay tests

* refactor: Organize uses

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-23 10:42:05 +00:00
Carol (Nichols || Goulding) 98e84cd624 feat: Only read the server config to list the databases a server owns 2021-10-22 15:56:46 -04:00
kodiakhq[bot] 9b70001aa4
Merge branch 'main' into cn/remove-list-deleted-databases 2021-10-22 19:36:39 +00:00
Carol (Nichols || Goulding) 948cf92aaa fix: Remove the ability to list deleted databases
Once we're relying on the server config file to know about databases,
IOx shouldn't be listing files in object storage to try and find deleted
databases.

When the disown API is implemented for the floating database design, it
will return the UUID of the database just disowned, so that it can
immediately be used in an adopt API call.
2021-10-22 14:50:06 -04:00
Carol (Nichols || Goulding) b4506c2f72 docs: Clean up some comments
Remove some that weren't accurately describing the code, fix some line
wrapping
2021-10-22 14:34:57 -04:00
Carol (Nichols || Goulding) 739fb075a7 refactor: Organize uses 2021-10-22 14:34:43 -04:00
Carol (Nichols || Goulding) 0c3ace5573 refactor: Call TestDbBuilder.build again instead of making a new builder 2021-10-22 14:32:32 -04:00
Carol (Nichols || Goulding) e838a22f92 refactor: Make TestDbBuilder reusable for rebuilding databases
Many tests want to exercise behavior that happens on reloading or
replay of a database. This changes the way the TestDbBuilder works to
make it possible to call `build` on a builder multiple times to create
databases with the same options to simulate reload of a database without
having to keep track of which pieces need to match.
2021-10-22 14:31:31 -04:00
kodiakhq[bot] f4eb3884f3
Merge branch 'main' into cn/write-db-owner-file 2021-10-20 13:51:45 +00:00
Raphael Taylor-Davies ce0127a1f7
feat: MutableBatch write API (#2090) (#2724) (#2882)
* feat: MutableBatch write API (#2090) (#2724)

* chore: fix lint

* fix: handle dictionaries with unused mappings

* chore: review feedback

* chore: further review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-20 08:44:14 +00:00
Andrew Lamb a82dc6f5f0
chore: Update datafusion + arrow (#2903)
* chore: Update datafusion to latest, arrow to 6.0.0

* fix: Update tests

* fix: bubble internal error

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-19 17:14:08 +00:00