a14e9ab27c
refactor: rename mutable_buffer::Chunk --> mutable_buffer::MBChunk ( #1711 )
...
* refactor: rename mutable_buffer::Chunk --> mutable_buffer::MBChunk
* fix: fmt
2021-06-14 13:35:20 +00:00
8c21eb79af
refactor: inline Table into mutable_buffer::chunk::Chunk ( #1704 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-14 13:04:06 +00:00
a614fef5bc
chore: remove more unused dependencies ( #1658 )
...
* chore: remove more unused deps
* refactor: move benchmarks into server_benchmarks crate
2021-06-09 10:17:20 +00:00
87297f7db4
Merge branch 'main' into cn/delete
2021-06-07 13:32:42 +00:00
1e7ef193a6
refactor: use field metadata to store influx types ( #1642 )
...
* refactor: use field metadata to store influx types
make SchemaBuilder non-consuming
* chore: remove unused variants
* chore: fix lints
2021-06-07 13:26:39 +00:00
4d6569583e
fix: Partially restore SequencedEntry as Entry+sequencer_id+sequence_num
2021-06-04 14:40:19 -04:00
32c6ed1f34
refactor: More cleanup related to multi-table chunks ( #1604 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-02 17:00:23 +00:00
d8fbb7b410
refactor: Remove last vestiges of multi-table chunks from PartitionChunk API ( #1588 )
...
* refactor: Remove last vestiges of multi-table chunks from PartitionChunk API
* fix: remove test that can no longer fail
* fix: update tests + code review comments
* fix: clippy
* fix: clippy
* fix: restore test_measurement_fields_error test
2021-06-01 16:12:33 +00:00
64328dcf1c
feat: cache schema on catalog chunks too ( #1575 )
2021-06-01 12:42:46 +00:00
00e735ef0d
chore: remove unused dependencies ( #1583 )
2021-05-29 10:31:57 +00:00
db432de137
feat: add distinct count to StatValues ( #1568 )
2021-05-28 17:41:34 +00:00
d8f19348bf
feat: per-column dictionaries in MUB ( #1570 )
...
* feat: per-column dictionaries in MUB
* chore: fmt
* refactor: remove chunk-level dictionary
* chore: remove redundant sort
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-28 13:51:56 +00:00
792bff07d1
feat: only store ChunkSnapshot in Closed state ( #1560 )
...
* feat: only store ChunkSnapshot in Closed state
* chore: review feedback
* feat: record MUB size as closed size
* chore: document column ordering assumption
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-27 10:36:47 +00:00
4fcc04e6c9
chore: enable arrow prettyprint feature ( #1566 )
2021-05-27 10:28:14 +00:00
c03b8a3963
refactor: remove tables from ChunkSnapshot ( #1295 ) ( #1558 )
2021-05-26 10:37:40 +00:00
14ba25f86d
chore: Update datafusion and use released version of arrow crates ( #1546 )
...
* chore: Update datafusion and use released version of arrow crate
* fix: Update for change in API
2021-05-24 15:37:22 +00:00
ee06ca4c7d
refactor: remove MutableChunk::has_table ( #1539 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-22 12:02:42 +00:00
37880ee89a
refactor: store chunk IDs only in catalog ( #1521 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-20 04:07:14 +00:00
c1ea6be101
fix: Add context to panic error on tools ( #1501 )
...
* fix: Add context to panic error on tools
* fix: log new unwrap that has snuck in
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-17 21:05:01 +00:00
f9178dbb5f
feat: push metrics into catalog ( #1488 )
...
* feat: push metrics into catalog
* chore: minor cleanup
* fix: include db labels in chunk metric domains
* chore: fmt
* fix: don't allow dropping moving chunks
* chore: further tweaks
* chore: review feedback
* feat: use new_unregistered() for metric instruments instead of default
* chore: use &[KeyValue] instead of &Vec<KeyValue>
* refactor: make GauageValue non default constructible
2021-05-14 17:37:39 +00:00
cfa8160644
fix: don't test snapshotting when disabled ( #1496 ) ( #1498 )
2021-05-14 16:15:55 +00:00
8ab71be5eb
feat: use PackedStringArray in mutable buffer ( #1476 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-13 15:25:19 +00:00
9d0c3a2b1a
refactor: Remove multi-table per chunk code in MUB ( #1471 )
...
* refactor: Remove multi-table per chunk code in MUB
* fix: clippy
* fix: bench build
* fix: merge conflicts
2021-05-11 17:49:07 +00:00
c85d1574eb
feat: move dictionary and bitset into arrow_utils ( #1459 )
...
* feat: move dictionary and bitset into arrow_utils
* chore: review feedback
* chore: remove redundant dictionary methods
* chore: consistent type parameter name in PackedStringArray
* chore: review comments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-11 16:43:38 +00:00
795f5bfcb7
refactor: make `StatValues::{min,max}` optional + handle NaNs
...
This will allow us to:
- handle all-NULL columns correctly
- be in-line with Parquet (where min/max are optional)
- handle NaNs at least somewhat sane (they do not "poison" stats
anymore)
2021-05-10 17:12:25 +02:00
b5ea71f45f
feat: Expose the storage usage for each column in system.chunk_columns ( #1441 )
...
* feat: Expose the storage usage for each column in system.chunk_columns
* fix: fixup logical conflicts
* refactor: move coalsce logic into the read buffer
* fix: Update system_tables to not use coalese
* fix: Improve comments
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-05-07 12:36:49 +00:00
884baf7329
feat: add column_type and influxdb_column_type, remove row_count from system.columns ( #1415 )
...
* feat: add column_type and influxdb_column_type, remove row_count from system.columns
* fix: update tests
* fix: more test update
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: fmt
* fix: copy/paste type conversion to avoid cross dependency between data_types and internal_types
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-05-06 12:59:30 +00:00
10f89a3e8d
refactor: split entry out into separate crate ( #1428 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 11:36:23 +00:00
ca1c698fd0
chore: update hashbrown ( #1430 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-05 22:32:46 +00:00
411cf134e9
refactor: explode arrow_deps ( #1425 )
...
* refactor: explode arrow_deps
* chore: workaround doctest bug
2021-05-05 16:59:12 +00:00
40b9b09cdc
refactor: rename assert_table_eq to assert_batches_eq ( #1368 )
2021-04-30 10:51:08 +00:00
0788892413
feat: add row_count to system.chunks and Chunk managment API ( #1373 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-29 13:53:25 +00:00
9aefcd216f
fix: Validate that ClockValue is never 0
2021-04-28 13:54:55 -04:00
eddc9319ff
docs: deny broken intradoc links
2021-04-27 13:22:28 +02:00
20117de078
feat: string dictionary encoding ( #1220 ) ( #1262 )
...
* feat: string dictionary encoding (#1220 )
* chore: review comments
* chore: fix lint
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-27 09:36:58 +00:00
272cdb85ce
fix: Use the ServerId type everywhere, for writing, querying, anything
2021-04-26 18:44:32 +00:00
0a835436ac
feat: use bitmasks within MUB ( #1274 ) ( #1289 )
...
* feat: use bitmasks within MUB (#1274 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-26 18:00:16 +00:00
146b48325b
refactor: use bool::then
2021-04-22 21:59:41 +00:00
74c25f541d
feat: fast MUB dictionary arrow conversion ( #1273 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-22 20:31:44 +00:00
fe4fa29930
feat: MUB benchmarks ( #635 ) ( #1271 )
2021-04-22 11:15:32 +00:00
58bbe55fd6
fix: Remove error type not used anywhere
2021-04-21 17:43:03 +00:00
adb96ee758
fix: Make error types more granular; cover field-to-tag and tag-to-field mismatch
2021-04-21 14:49:37 +00:00
4407fdd322
fix: Make line lengths less long
...
rustfmt doesn't touch macro calls 😬
2021-04-21 14:49:37 +00:00
d360a08dff
fix: Remove error type that is no longer used
2021-04-21 14:49:37 +00:00
e08795ce6f
fix: update schema conflicts to display column name
2021-04-21 14:49:37 +00:00
1cf3ab3978
feat: enable disabling MUB snapshot caching ( #1203 ) ( #1264 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-21 08:21:31 +00:00
c9772db01b
fix: Allow this upper case acronym; it could be confusing otherwise
2021-04-19 08:48:11 -04:00
f136931225
fix: Inconsistent ordering lints
2021-04-19 08:48:11 -04:00
40decd0154
fix: fix Rust 2021 compat warning
...
This fixes the following warning:
```text
warning: panic message is not a string literal
--> mutable_buffer/src/table.rs:591:13
|
591 | format!("didn't match returned error: {:?}", response)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `#[warn(non_fmt_panic)]` on by default
= note: this is no longer accepted in Rust 2021
= note: the panic!() macro supports formatting, so there's no need for the format!() macro here
help: remove the `format!(..)` macro call
|
591 | "didn't match returned error: {:?}", response
| -- --
```
2021-04-19 14:00:04 +02:00
529c99c93f
fix: don't clone arrays to make TimestampNanosecondArrays ( #1241 )
...
* fix: avoid clone
* fix: remove another clone
2021-04-16 18:40:22 +00:00
e226b5a820
feat: Use TimestampNanosecondArray for timestamps in IOx ( #1230 )
...
* refactor: Create Arrow arrays using iterators
* feat: use Timestamp64(TimeUnit::Nanosecond) for timestamps
* feat: add support for timestamp array
* fix: update more tests
* fix: remove unecessary code
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-16 15:55:33 +00:00
c9cbc74857
feat: Use a DID newtype instead of u32 for dictionary ids
...
Rationale
---------
We use `u32` throughout the codebase to reference for interned dictionary strings.
We also use `u32` for other reasons and it would be nice to get some help from the compiler
to avoid mixing them up
2021-04-13 22:38:41 +00:00
7ef490694c
refactor: reduce module coupling in mutable buffer ( #1199 )
...
* refactor: reduce module coupling in mutable buffer
* refactor: tweak visibility
* chore: formatting
2021-04-13 20:59:41 +00:00
61cd745ab6
refactor: remove mutable buffer predicate logic ( #1186 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-13 19:09:36 +00:00
55a77914b1
feat: basic snapshot caching ( #1184 )
2021-04-13 17:10:28 +00:00
4d22982b5f
fix: fix broken tests from column refactoring
2021-04-13 12:52:14 +00:00
7e28f8ef66
feat: Implement Entry writing to Db
...
This removes the old ReplicatedWrite structure and implements the writing of an Entry to the Db. I also call out in `server/lib.rs` and in the `Db` where sharding and replication might happen.
I've also added helpers in various places to write line protocol to chunks, tables, and databases. That enabled removing a good amount of code from the test helpers crate.
2021-04-13 12:52:14 +00:00
1997324344
feat: mutable buffer snapshotting ( #1179 )
...
* feat: mutable buffer snapshotting
* chore: review feedback
2021-04-13 12:14:54 +00:00
d67739c873
chore: updates for pr feedback
2021-04-12 18:43:14 +00:00
5893c17905
refactor: PR feedback and change ClockValue to actual type.
2021-04-12 18:43:14 +00:00
31115742ec
feat: Add writing of Entry structures to MB Chunk
...
This adds writing of Entry of a vec of TableWriteBatch to the Mutable Buffer Chunk. This is additional to the previous method of writing via ReplicatedWrite. The next step is to remove the old ReplicatedWrite bits.
Test helpers for parsing line protocol into Entry and writing line protocol directly to Chunks have also been added.
2021-04-12 18:43:14 +00:00
319e66706e
Merge branch 'main' into cn/no-more-test-results
2021-04-07 16:45:32 +00:00
82588d5c72
fix: Don't return Result from test functions
2021-04-07 12:40:00 -04:00
c2355aca6d
feat: add basic memory tracking ( #1125 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-07 15:38:24 +00:00
80d55d0829
chore: rename tracing_deps to observability_deps
...
OpenTelemetry makes this necessary.
2021-04-02 13:14:30 -07:00
0b880d3534
chore: Group all tracing-related crates under one crate for easier upgrade management
2021-04-02 09:54:39 -04:00
569f90d937
feat: Add ability to get PartitionSummary statistics from a Db ( #1090 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-31 14:18:53 +00:00
7154dfd5f6
feat: Add timestamps to ChunkSummary ( #1079 )
...
* refactor: Move timestamps from mutable_buffer::Chunk to catalog::Chunk
* feat: Add timestamps to ChunkSummary
* feat: Add timestamp conversion logic to protobuf types
* test: Add tests
* fix: Update data_types test
* fix: handle negative nanos during conversion
* fix: clippy
* fix: more clippy
* fix: even more clippy
* fix: even more clippy
2021-03-30 19:03:23 +00:00
0b7dea09d5
refactor: Remove `MutableBufferDb` and Partition ( #1068 )
...
* refactor: Move test for missing tags as null to db
* refactor: move last_write_at and create_time to catalog partition
* refactor: remove mutable buffer partition and port tests
* fix: port over partition sorting logic
* fix: remove stale comment
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-30 12:51:37 +00:00
6a48001d13
refactor: Manage storage directly in the Catalog ( #1057 )
...
* refactor: Manage mutable buffer chunks directly
* fix: do not use mutable_buffer for listing table names
2021-03-29 17:55:07 +00:00
1a4c793270
feat: lifecycle rules ( #1045 )
2021-03-29 15:32:36 +00:00
02ae743e8e
refactor: Remove async from Database ( #1063 )
2021-03-29 12:48:12 +00:00
44d67db733
feat: Rework Db to use Catalog for chunk state ( #1043 )
...
* feat: Rework Db to use Catalog for chunk state
* docs: Update server/src/db.rs
* fix: fmt
* fix: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-24 17:57:11 +00:00
c47a252c15
feat: Check in generated WAL flatbuffers code
2021-03-22 09:38:59 -04:00
52e415ae1b
fix: Verify flatbuffers when restoring from a file, and not after that
2021-03-22 09:38:59 -04:00
7c6f543d35
fix: wb::ColumnValue is now a set of associated consts, not a real Rust enum
2021-03-22 09:38:58 -04:00
24af745b23
chore: Upgrade to flatbuffers 0.8
2021-03-22 09:38:58 -04:00
6e1795fda0
refactor: Move some types (not yet exposed to clients) into internal_types ( #1015 )
...
* refactor: Move some types (not yet exposed to clients) into internal_types
* docs: Add README.md explaining the rationale
* refactor: remove some stragglers
* fix: fix benches
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: add clippy lints
* fix: fmt
* docs: Apply suggestions from code review
fix typos
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-19 16:27:57 +00:00
4781317647
feat: Management API + CLI command to create a new chunk (rollover partition) ( #990 )
...
* feat: Management API + CLI command to create a new chunk (rollover partition)
* fix: Update tests/end_to_end_cases/management_api.rs
fix typo
* fix: logical merge conflict
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-15 16:41:18 +00:00
65f7a1ac5b
fix: use consistent crate versions ( #989 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-15 15:42:19 +00:00
6ac7e2c1a7
feat: Add management API and CLI to list chunks ( #968 )
...
* feat: Add management API and CLI to list chunks
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: add comment to protobuf
* fix: fix comment
* fix: fmt, fixup merge errors
* fix: fascinating type dance with prost generated types
* fix: clippy
* fix: move command to influxdb_iox database chunk list
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-03-12 13:56:14 +00:00
488252d9f1
fix: test
2021-03-09 07:52:52 -08:00
09c4f88758
fix: cargo fmt
2021-03-09 07:45:05 -08:00
abb991ffca
fix: add tests to unsigned integer parsing in LP and MUB
2021-03-08 15:29:05 -08:00
ac1b0c04ae
fix(line-protocol): add unsigned integer field type
...
Fixes #904
The line protocol parser was lacking the unsigned integer type, which
suffixes values with `u`. This adds unsigned integer support to the line
protocol parser, and fills a few corresponding gaps in the mutable
buffer.
2021-03-08 09:59:12 -08:00
746373a687
refactor: Remove mutable_buffer crate dependency on query crate ( #927 )
2021-03-05 11:34:27 +00:00
a6965769b4
refactor: Remove impl of `query::PartititionChunk` in mutable_buffer: This PR ( #923 )
2021-03-05 10:51:08 +00:00
b1e5cfedf7
refactor: Move chunk predicate creation from query::Predicate into server crate ( #922 )
...
* refactor: Create `ChunkPredicate` predicate in the Server crate
* refactor: move chunk predicate out of chunk
* fix: remove comment in server/src/db/pred.rs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-04 23:23:43 +00:00
3be5c26f92
refactor: Remove impl of `query::Database` in mutable_buffer ( #914 )
2021-03-04 22:02:42 +00:00
945d2f8d45
refactor: Remove query stuff from mutable buffer ( #912 )
...
* refactor: remove query code from mutable_buffer database.rs
* refactor: remove query code from mutable_buffer table.rs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-04 21:42:56 +00:00
8b1f100df3
feat: make read_group and read_window_aggregate work across chunks ( #905 )
...
* feat: make read_group and read_window_aggregate work across chunks
* refactor: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* refactor: Update query/src/frontend/influxrpc.rs
Improve logic and use strings directly
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: fmt
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-04 17:06:31 +00:00
7d8d00781c
feat: Make read_filter work for mutable buffer and read buffer ( #882 )
...
* feat: port read_filter to InfluxRPCPlanner
* fix: remove commented out vestigal test
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: fmt
* fix: Update arrow_deps/src/util.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-03-01 16:50:29 +00:00
18de3bdcab
chore: merge main into branch
...
Merge branch 'main' into ntran/optimize_column_selection
2021-02-26 15:29:43 -05:00
f37e5846aa
feat: fmt auto fix
2021-02-26 14:56:10 -05:00
eb81975151
feat: Optimize Column Selection
2021-02-26 14:28:46 -05:00
12deacd8a0
refactor: move SeriesSetPlans into its own module ( #878 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-25 23:12:39 +00:00
8fb7651719
feat: Port tag_values to the InfluxRPCPlanner ( #859 )
...
* feat: Port tag_values to the InfluxRPCPlanner
* refactor: merge imports
* refactor: rename column_names to tag_column_names for clarity
* fix: Update query/src/frontend/influxrpc.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* refactor: use ensure!
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* refactor: less silly whitespace
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: code review comments
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-24 23:11:22 +00:00
ed7859e182
refactor: rename tag_column_names --> tag_keys in InfluxRPCPlanner ( #860 )
2021-02-23 17:04:53 +00:00
56c515a063
refactor: Simplify partition sorting code
2021-02-22 11:19:47 -05:00
04e2631f26
refactor: Switch to explicit Arc::clone
2021-02-22 11:10:07 -05:00
Andrew Lamb
kodiakhq[bot]
Raphael Taylor-Davies
Carol (Nichols || Goulding)
Marco Neumann
Edd Robinson
Paul Dix
Marko Mikulicic
Jacob Marble
Jacob Marble
Nga Tran
NGA TRAN