Commit Graph

241 Commits (a72c2a619231f4e4d3200e8e92f6a819d1cddbca)

Author SHA1 Message Date
Carol (Nichols || Goulding) ce6030a3cb feat: Wire list deleted databases through gRPC and CLI APIs 2021-09-02 15:48:07 -04:00
Marco Neumann 81e672db14 feat: add `auto_create_sequencers` write buffer flag 2021-09-02 16:41:35 +02:00
Marco Neumann ecf1f99ddb refactor: more flexible writer buffer config
This allows:

- different types (instead of guessing through the connection URL)
- sequencer counts (not used yet but will be by #2455)
- extensible configs (e.g. to configure Kafka in a more granular way,
  not wired up yet)
- future extensions (since we use a message now instead of a single
  string)

**BREAKING: This requires changes for deployed systems / existing DBs!**
2021-09-02 16:41:35 +02:00
Andrew Lamb 779b027271
feat: Store only the database rules sent by the client (do not store default values) (#2430)
* feat: add omit_default to protobuf definition

* feat: Persist only the client provided rules

* fix: Remove race conditions

* fix: merge confit

* refactor: do not use macro

* refactor: restore use of glob import

* fix: review comments

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-28 10:26:32 +00:00
Nga Tran 499af57299 chore: merge mian to branch and resolve conflicts 2021-08-27 17:51:07 -04:00
Nga Tran b79eaa34d1 refactor: address review comments 2021-08-27 15:53:27 -04:00
Nga Tran bcd39e225c feat: Management API for delete 2021-08-26 17:31:21 -04:00
Carol (Nichols || Goulding) 7cf7fb02ed refactor: Rename database ObjectStore state types to DatabaseObjectStore 2021-08-26 09:14:23 -04:00
Carol (Nichols || Goulding) c7eceac8a3 refactor: Have server determine database generation from object store 2021-08-26 09:14:23 -04:00
Marco Neumann 60ea637c38 refactor: remove `drop_non_persisted`
This flag should always be `!persist`.

Closes #2212.
2021-08-17 16:53:45 +02:00
Marco Neumann 77892a0998 feat: add API to drop entire partitions 2021-08-17 09:44:35 +02:00
Nga Tran fd4ea89454 chore: revert panic commit 2021-08-13 12:51:37 -04:00
Andrew Lamb db5fd935dd
fix: revert "Convert panic --> `warn` on statistics mismatch (#2125)" (#2250)
This reverts commit 7415fa0d05.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-13 11:39:12 +00:00
Edd Robinson f82646e7f8 test: ignore failing tests on aarch64 2021-08-12 11:46:23 +01:00
Dom 3de6b44e23
build: use new rustdoc lint name (#2261)
* fix: nocache feature code rot

The MBChunk::snapshot code when using the "nocache" option no longer
compiles - this commit updates it to match the not(nocache) code.

* build: use updated broken_intra_doc_links name

The broken_intra_doc_links lint was renamed
rustdoc::broken_intra_doc_links

https://doc.rust-lang.org/rustdoc/lints.html
2021-08-11 19:48:51 +00:00
Andrew Lamb 0c300a4abb
fix: Handle all null columns in ColumnSummary::update_from (#2251) 2021-08-10 20:37:00 +00:00
Raphael Taylor-Davies 2344c28f4e
feat: drain database jobs on shutdown (#2239)
* feat: drain database jobs on shutdown

* chore: fmt

* chore: review feedback

* chore: use join() not member directly

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-10 16:47:37 +00:00
Raphael Taylor-Davies 1f450ef371
feat: add Database abstraction (#2186) (#2203)
* feat: add Database abstraction

* chore: minor tweaks

* chore: remove redundant test fixture restart

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-08 17:14:23 +00:00
Andrew Lamb 096281bbc4
fix: set distinct count for all entirely null columns to 1 (#2160)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-30 17:17:10 +00:00
Carol (Nichols || Goulding) 9d15798288 fix: Address or allow Clippy warnings new with Rust 1.54 2021-07-30 09:59:59 -04:00
kodiakhq[bot] 545222303f
Merge branch 'main' into cn/cc-only 2021-07-29 17:18:16 +00:00
Carol (Nichols || Goulding) 8ee02bf5ce docs: Fix copypasta in doc comment 2021-07-28 15:27:23 -04:00
Carol (Nichols || Goulding) a1e2ce77f0 refactor: Remove TableSummaryAndTimes type 2021-07-28 14:12:58 -04:00
Marko Mikulicic fe7f65bfa7
feat(iox): Implement max_active_compactions_cpu_fraction 2021-07-28 17:31:17 +02:00
Carol (Nichols || Goulding) 8add00e761 feat: Make CatalogChunk first/last write times required
Connects to #1927.
2021-07-28 09:22:06 -04:00
Carol (Nichols || Goulding) 7c9a21632b refactor: Organize uses 2021-07-28 09:22:04 -04:00
Dom 7d3b6bf80d feat: DatabaseRules (de)serialisation support
Derives serde::Serialise & Deserialise for the DatabaseRules struct and
children.
2021-07-27 14:29:26 +01:00
kodiakhq[bot] 5551dd3a87
Merge branch 'main' into devnull 2021-07-27 09:57:16 +00:00
Andrew Lamb 7415fa0d05
fix: Convert panic --> `warn` on statistics mismatch (#2125) 2021-07-26 22:11:10 +00:00
Andrew Lamb 5fb3e00f2a
fix: Properly record total_count and null_count in statistics (#2103)
* fix: Properly record total_count and null_count in statistics

* fix: fix statistics calculation in mutable_buffer

* refactor: expose null counts in read_buffer

* refactor: expose null_count in parquet_file

* fix: update server crate tests

* fix: update query_tests tests

* docs: tweak comments

* refactor: Use storage_stats rather than adding `null_count`

* refactor: rename test data field for clarity

* fix: fixup merge conflicts

* refactor: rename initial_non_null_count to initial_total_count

* refactor: caculate null_count as row_count - to_add
2021-07-26 18:13:36 +00:00
Marko Mikulicic 094945a72d
feat: Add '/dev/null' sink 2021-07-26 19:19:11 +02:00
kodiakhq[bot] 009c77d864
Merge branch 'main' into cn/parquet-first-last 2021-07-26 14:59:54 +00:00
Raphael Taylor-Davies 4f8b2fdb69
chore: remove unused module (#2102)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-26 11:02:20 +00:00
Marko Mikulicic e5ee252876
feat: Add kafka sink variant 2021-07-26 11:08:02 +02:00
Marko Mikulicic d58a3ccbc7
refactor: Add sink to routing config
This deprecates the "target" field in the RoutingConfig and replaces it with the "sink"
field, which has a variant that accepts a node group.

This commit is backward compatible in that it will accept existing configs.
The configs will roundtrip to the new format though (i.e. `database get` will render
the sink field).
2021-07-26 11:08:01 +02:00
Marko Mikulicic 16a82ba350
refactor: Generailize sinks: Rename Shard to Sink
The ShardConfig applies matchers that resolve to a shard number.
The config then applies a mapping between shard numbers to targets.
The type that encapsulated the target that a shard points to was also called
a "Shard". This is confusing. This commit changes it to "Sink", i.e. a destination
for traffic to go to. Subsequent commits will expand the definition of a Sink to
encompass different kinds of sinks (like kafka write buffer, "devnull", ...)

This changes only the name of the protobuf message and the related rust types,
it doesn't change any name of the json-rendered protobuf configs.
2021-07-26 11:08:00 +02:00
Raphael Taylor-Davies c595039c81
feat: add row timestamp metrics (#2101)
* feat: add row timestamp metrics

* chore: review feedback
2021-07-23 19:17:11 +00:00
Jake Goulding d928bc84e6 feat: Thread time_of_{first,last}_write through Parquet metadata 2021-07-23 14:07:35 -04:00
Raphael Taylor-Davies 20d06e3225
feat: include more information in system.operations table (#2097)
* feat: include more information in system.operations table

* chore: review feedback

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-22 17:16:09 +00:00
Raphael Taylor-Davies 8c974beba0
feat: add access timestamps to CatalogChunk (#2075) (#2081)
* feat: add access timestamps to CatalogChunk (#2075)

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-22 12:19:30 +00:00
Raphael Taylor-Davies ffe6e62aee
feat: add instant to datetime conversion (#2078)
* feat: add instant to datetime conversion

* chore: review feedback
2021-07-21 11:43:27 +00:00
Paul Dix a4704dd165 chore: update parquet_cache_limit to u64 and 0 for default 2021-07-20 15:41:06 -04:00
Paul Dix 297e059085 feat: add parquet cache size setting to database rules 2021-07-20 15:41:06 -04:00
Raphael Taylor-Davies 8e5d5928cf
feat: compute WriteSummary from PersistenceWindows (#2030) (#2054)
* feat: compute WriteSummary from PersistenceWindows (#2030)

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-20 08:46:52 +00:00
Edd Robinson a852dce450 refactor: default to num_cpu 2021-07-19 14:00:10 +01:00
Edd Robinson dd93b2cdec feat: add db-level compaction limiter 2021-07-19 14:00:10 +01:00
Raphael Taylor-Davies 5fc98c7c56
feat: add failure reporting to TaskTracker (#2031)
* feat: add failure reporting to TaskTracker

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-19 09:17:20 +00:00
Andrew Lamb 3fd6430fb6
fix: rename `estimated_bytes` to `memory_bytes` and expose `object_store_bytes` in ChunkSummary and system.chunks (#2017)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 16:00:24 +00:00
Raphael Taylor-Davies a79c0b4e75
feat: add mub row count threshold to lifecycle rules (#1876) (#2016)
* feat: add mub row count threshold to lifecycle rules (#1876)

* chore: update docstring

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 13:42:17 +00:00
Marco Neumann 956086fa6d feat: add "drop chunk" job type 2021-07-15 12:07:56 +02:00