195644da04
docs: document semaphore design in server
2021-06-04 12:52:13 +02:00
402ef0ebde
Merge branch 'main' into crepererum/limit_cleanup_amount
2021-06-04 10:47:33 +00:00
e06d65bb2a
refactor: migrate "DBs initialized" RPC to "server status"
2021-06-04 11:33:41 +02:00
b30d7e2821
feat: move DB loading into background worker
...
Before this change we loaded databases eagerly when a serverID was
passed on startup BEFORE starting up the gRPC server. Since loading
(esp. at its current state without checkpoints and with too many small
parquet files) can take very long, K8s thinks IOx is unhealthy. With
this change we are now loading databases in the server background worker
once a serverID is available. Until then we block all DB-related
interactions including adding new databases (since without inspecting
the object store there is now way we can check if the DB already
exists).
Furthermore we now load database no matter if the serverID was passed on
startup (via CLI or environment variable) or was set later via gRPC
call. Before this change the latter case was somewhat forgotten.
2021-06-04 11:33:41 +02:00
696ebdc4db
feat: recover failed lifecycle actions ( #1099 ) ( #1592 )
...
* feat: recover failed lifecycle actions (#1099 )
* chore: review feedback
* chore: fix logical conflicts
2021-06-03 15:46:33 +00:00
91df8a30e7
feat: limit number of files during storage cleanup
...
Since the number of parquet files can potentially be unbound (aka very
very large) and we do not want to hold the transaction lock for too
long and also want to limit memory consumption of the cleanup routine,
let's limit the number of files that we collect for cleanup.
2021-06-03 17:43:11 +02:00
e583e1fbda
Merge branch 'main' into er/feat/read_buffer/float_int
2021-06-03 14:48:36 +01:00
eaa5b75437
refactor: Make it clear only partition_key and table name pruning happens in catalog ( #1608 )
...
* refactor: Make it clear only partition_key and table name pruning is happening in catalog
* fix: clippy
* fix: Update server/src/db/catalog.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* refactor: use TableNameFilter enum rather than Option
* docs: Add docstring to the `From` implementation
* fix: Update server/src/db/catalog/partition.rs
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Edd Robinson <me@edd.io>
2021-06-03 13:09:09 +00:00
65bfa4dd10
test: fix tests
2021-06-03 12:32:40 +01:00
27b9477aa4
test: fix flaky test
2021-06-03 11:23:29 +02:00
7b2663a38a
test: make tests faster
2021-06-03 11:23:29 +02:00
3c9fd81697
refactor: split overlong line
2021-06-03 11:23:29 +02:00
bbd73e59be
feat: jitter background clean-up job + wait on first job
2021-06-03 11:23:29 +02:00
ce412dbce2
fix: use structured error for background cleanup task reporting
2021-06-03 11:23:29 +02:00
1c764c47a2
Merge branch 'main' into ntran/deduplicate
2021-06-02 17:42:36 +00:00
40bd932fff
refactor: address Andrew's comment
2021-06-02 13:41:46 -04:00
32c6ed1f34
refactor: More cleanup related to multi-table chunks ( #1604 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-02 17:00:23 +00:00
e7a97f3ac1
test: merge main and add more tests for deduplicate work
2021-06-02 12:00:40 -04:00
80f4d84ce8
refactor: isolate DB loading and streamline error handling
...
There are not functional changes here (except that errors look slightly
different) but it should allow for an easier move of the DB loading into
a delayed task.
2021-06-02 13:42:24 +02:00
0e09b20ca8
Merge branch 'main' into crepererum/issue1513-b
2021-06-02 07:08:29 +00:00
40df7def0e
test: ttests for the deduplicate work
2021-06-01 18:06:35 -04:00
60ad929721
refactor: add macro tto compare output of explains
2021-06-01 16:39:14 -04:00
aa867601e5
chore: merge main with DF plan display fix
2021-06-01 16:17:41 -04:00
0ad258bab3
refactor: remove comments since the time function predicates are pushed down after the recent constant folding fix in DF
2021-06-01 16:00:09 -04:00
d8fbb7b410
refactor: Remove last vestiges of multi-table chunks from PartitionChunk API ( #1588 )
...
* refactor: Remove last vestiges of multi-table chunks from PartitionChunk API
* fix: remove test that can no longer fail
* fix: update tests + code review comments
* fix: clippy
* fix: clippy
* fix: restore test_measurement_fields_error test
2021-06-01 16:12:33 +00:00
714a082f3a
refactor: remove chunk state struct nesting
...
Inline structs that are only used for enum variants.
2021-06-01 18:00:16 +02:00
5a4562f1c9
test: test `Chunk::new_open`
2021-06-01 18:00:16 +02:00
f45e61f9ef
test: test chunk lifecycle action handling
2021-06-01 18:00:16 +02:00
50636ca011
refactor: rename `Chunk::{set_closed => freeze}` and add tests
...
This make it clearer what is actually happening. Furthermore, freezing
frozen chunks is now a no-op.
2021-06-01 18:00:16 +02:00
aafc8c4746
Merge branch 'main' into crepererum/fix_catalog_replay_logging
2021-06-01 15:59:42 +00:00
98c2963c28
fix: fix confusing log message during catalog replay
2021-06-01 17:58:38 +02:00
d3711a5591
refactor: Use ParquetExec from DataFusion to read parquet files ( #1580 )
...
* refactor: use ParquetExec to read parquet files
* fix: test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-01 14:44:07 +00:00
64328dcf1c
feat: cache schema on catalog chunks too ( #1575 )
2021-06-01 12:42:46 +00:00
4e7b754098
Merge branch 'main' into crepererum/issue1513-a
2021-06-01 08:23:01 +00:00
6e07a735bd
feat: don't recompute chunk size on every iteration ( #1586 )
2021-05-31 16:19:11 +00:00
73cedd2f88
chore: remove unused dependency ( #1587 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-31 14:22:11 +00:00
991314ebe8
docs: fix `set_writing_to_object_store` docstring
2021-05-31 15:44:29 +02:00
996ce833f1
chore: fix formatting
2021-05-31 15:42:13 +02:00
162a808a8d
refactor: Remove `table_name` from PartitionChunk API ( #1584 )
...
* refactor: Remove `table_name` from PartitionChunk API
* fix: clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-31 12:05:09 +00:00
c658a627ed
refactor: change state structure for chunks
...
This is the first step towards #1513 . However it leaves all consumers
bascially unchanged and also does NOT touch state transitions. These
changes will follow in upcoming PRs.
2021-05-31 11:19:01 +02:00
db432de137
feat: add distinct count to StatValues ( #1568 )
2021-05-28 17:41:34 +00:00
d8f19348bf
feat: per-column dictionaries in MUB ( #1570 )
...
* feat: per-column dictionaries in MUB
* chore: fmt
* refactor: remove chunk-level dictionary
* chore: remove redundant sort
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-28 13:51:56 +00:00
d70d7a63a2
Merge branch 'main' into crepererum/remove_invalid_chunk_state
2021-05-28 10:20:05 +00:00
c6f42cf304
refactor: Remove unnecessary code ( #1573 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-28 10:12:47 +00:00
5cfede51f2
refactor: remove `ChunkState::Invalid`
...
This seems to only exist to fight the borrow checker and we can actually
live without it.
2021-05-28 11:16:06 +02:00
3ae44a0375
refactor: Chunks can have at most one object store path ( #1574 )
...
* refactor: Chunk can have at most one path
* fix: update tests
2021-05-27 19:52:09 +00:00
62147ff0d4
feat: add more explain tests
2021-05-27 12:19:41 -04:00
f3bec93ef1
feat: Cache TableSummary in Catalog rather than computing it on demand ( #1569 )
...
* feat: Cache `TableSummary` in catalog Chunks
* refactor: use consistent table summary
2021-05-27 16:03:05 +00:00
5d342d7779
feat: associate tracker with lifecycle action ( #1099 ) ( #1556 )
...
* feat: associate tracker with lifecycle action (#1099 )
* chore: docs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-27 10:47:35 +00:00
792bff07d1
feat: only store ChunkSnapshot in Closed state ( #1560 )
...
* feat: only store ChunkSnapshot in Closed state
* chore: review feedback
* feat: record MUB size as closed size
* chore: document column ordering assumption
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-27 10:36:47 +00:00
4fcc04e6c9
chore: enable arrow prettyprint feature ( #1566 )
2021-05-27 10:28:14 +00:00
efe077da8f
Merge branch 'main' into crepererum/issue1313
2021-05-26 14:46:18 +00:00
24ec1a472e
fix: do NOT delete parquet files that are reachable by time travel
2021-05-26 12:38:54 +02:00
c03b8a3963
refactor: remove tables from ChunkSnapshot ( #1295 ) ( #1558 )
2021-05-26 10:37:40 +00:00
1fb6af2364
refactor: split DB background loop into lifecycle and cleanup
...
This should prevent one from blocking / stalling the other.
2021-05-26 11:09:30 +02:00
5983336366
refactor: rename `parquet_file::{utils => test_utils}`
2021-05-26 11:09:29 +02:00
dd6bbeec42
feat: add background task to clean up OS
...
Closes #1313 .
2021-05-26 11:04:56 +02:00
cc78b5317d
feat: add method to get all parquet files from catalog state
2021-05-26 11:02:40 +02:00
166851d952
Merge branch 'main' into crepererum/in_file_metadata
2021-05-26 07:39:53 +00:00
bae5e5aee3
feat: Add simpler RoutingConfig
2021-05-25 21:51:54 +02:00
19a2733d30
feat: preserve transaction metadata in parquets
2021-05-25 09:56:12 +02:00
fe8e6301fe
refactor: move `read_schema_from_parquet_metadata` back to `parquet_file::metadata`
...
Let us pool all metadata handling in a single module, which makes it
easier to review.
2021-05-25 09:37:53 +02:00
a4215f0a56
fix: Fix 'acive' jemalloc stat misreporting
2021-05-25 02:55:27 +02:00
018e1e0246
chore: add a comment to trick github to check semantic
2021-05-24 17:25:14 -04:00
40a5d7d4ba
chore: Merge branch 'main' into tran/pushdown_parquet
2021-05-24 16:31:06 -04:00
e72ae81a8e
feat: support predicate pushdown for parquet files
2021-05-24 16:22:52 -04:00
db96286ed7
Merge branch 'main' into er/refactor/scalar_comp
2021-05-24 17:02:14 +00:00
c464ffadad
refactor: remove special case timestamp_range in parquet chunk ( #1543 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-24 16:19:44 +00:00
14ba25f86d
chore: Update datafusion and use released version of arrow crates ( #1546 )
...
* chore: Update datafusion and use released version of arrow crate
* fix: Update for change in API
2021-05-24 15:37:22 +00:00
abe64c6edc
test: uncomment tests to fix
2021-05-24 16:18:53 +01:00
5c5064bdac
fix: Set default line timestamp and default partition time to same value ( #1512 )
...
* refactor: Rearrange to allow injection of the current time in tests
* test: Failing test showing a point can be in the wrong partition
* fix: Only get the default time once per ShardedEntry creation, in router
2021-05-24 14:55:11 +00:00
27e5b8fabf
refactor: Remove multiple table support from Parquet Chunk ( #1541 )
2021-05-24 08:40:31 -04:00
1f70d1f9c8
chore: remove a couple more comments
2021-05-21 17:06:53 -04:00
f113abacb5
feat: more unit & e2e tests plus cleanup and addressing review comments of Andrew and Edd
2021-05-21 16:48:43 -04:00
1093542578
fix: now all tests pass. Next step is cleaning up and addressing review comments
2021-05-21 13:29:20 -04:00
784ef88fcd
chore: merge main to branch and add more tests that expose a wrong result bug on unsigned int
2021-05-21 12:38:06 -04:00
93afc9c213
chore: more tests
2021-05-21 11:39:12 -04:00
5b619733d9
refactor: split lifecycle tracking from chunk state ( #1361 ) ( #1099 ) ( #1397 )
...
* refactor: split lifecycle tracking from chunk state (#1361 ) (#1099 )
* chore: namespace internal errors
* chore: fix logical conflict
* chore: don't remove moving chunk size metric
2021-05-21 09:27:44 +00:00
e44a3a87db
feat: fnow predicate is actuallu pushed down to RUB but there are bugs and not working yet
2021-05-20 16:56:15 -04:00
f028a356f4
Merge branch 'main' into crepererum/issue1382-c
2021-05-20 15:51:47 +00:00
aac00d4fa6
Merge branch 'main' into crepererum/remove_snapshotting
2021-05-20 14:14:58 +00:00
0e37d500eb
feat: remove snapshot feature
...
The parquet files produced by this code path are only semi-specified and
will miss many important metadata aspects that we will require for data
lineage.
2021-05-20 14:59:04 +02:00
462a5590c6
fix: fmt
2021-05-20 14:58:50 +02:00
c908cf0f98
fix: review suggestion
...
Co-authored-by: Edd Robinson <me@edd.io>
2021-05-20 14:40:02 +02:00
aa90329c1f
feat: Add remote_template for simpler remote configuration
2021-05-20 12:45:08 +02:00
7e55544eef
fix: correctly track chunk ID counter during catalog replay
2021-05-20 10:32:40 +02:00
93251f22c7
feat: read perserved catalog during DB startup
...
Closes #1382 .
2021-05-20 10:28:31 +02:00
91d7189e6d
feat: Log cached connections
2021-05-20 10:27:20 +02:00
37880ee89a
refactor: store chunk IDs only in catalog ( #1521 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-20 04:07:14 +00:00
00dacb5394
feat: add tests to verify the correctness as well as the explain of the plan
2021-05-19 17:31:16 -04:00
11561111d5
chore: merge main to branch
2021-05-19 15:11:15 -04:00
087d61f229
feat: Part 1 of predicate push down - Send predicates to MUB, RUB, and Parquet File. Note that MUB has not handled predicates yet
2021-05-19 13:59:51 -04:00
ce2f8351be
fix: Cache outbound gRPC connections
2021-05-19 18:28:45 +02:00
8db26485a4
refactor: empty transaction during catalog creation
...
That involves some refactoring which we are going to need anyway for
hooking up the "read" path of the catalog into the DB startup, namely:
- make `Db::new` require a preserved catalog
- introduce a helper function that can provide that
- as a consequence, all test-creations of a Db are now async
This prepares for #1382 .
2021-05-18 17:42:07 +02:00
c3cc58b2ff
Merge branch 'main' into crepererum/issue1382
2021-05-17 17:57:26 +00:00
4f0e46bcd5
refactor: track ingest metrics in one place ( #1503 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-17 16:25:01 +00:00
18f0a7f614
docs: reference open issue
2021-05-17 14:01:51 +02:00
cdf0ada6a6
test: test preserved catalog <-> Db write wiring
2021-05-17 13:57:31 +02:00
91a45fd380
feat: simplify shutdown ( #1502 )
...
* feat: simplify shutdown
* chore: fix lint
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-17 11:50:14 +00:00
4299371cf2
refactor: remove some code
2021-05-17 12:32:48 +02:00
840c11dab2
feat: wire up catalog preservation write path
...
Required a bit of refactoring:
- Add an extra layer between DB an catalog which is the "preserved
catalog" wrapper. This is required to make the ownership model
somewhat sane, because during the read operations the "preserved
catalog" is going to act on the in-mem catalog.
- Move "parquet file written" logic into binding `preserved catalog <->
catalog state`, so we have a single place where new parquet files are
announced. For now this only works for chunks that are already known
(i.e. the writing->written transation when coming from read buffer),
however in the next PR this will be extended to also handle totally
new parquet files during transaction playback.
**NOTE: This does NOT include the read path yet!**
Issue: #1382 .
2021-05-17 11:33:22 +02:00
07db4932ee
refactor: rename data_types/src/chunk.rs -> data_types/src/chunk_metadata.rs ( #1500 )
2021-05-15 10:18:01 +00:00
f9178dbb5f
feat: push metrics into catalog ( #1488 )
...
* feat: push metrics into catalog
* chore: minor cleanup
* fix: include db labels in chunk metric domains
* chore: fmt
* fix: don't allow dropping moving chunks
* chore: further tweaks
* chore: review feedback
* feat: use new_unregistered() for metric instruments instead of default
* chore: use &[KeyValue] instead of &Vec<KeyValue>
* refactor: make GauageValue non default constructible
2021-05-14 17:37:39 +00:00
fdc8461c7f
Merge branch 'main' into cn/wb-clock
2021-05-14 13:00:06 +00:00
35c2ca17fc
fix: Add ingest_fields_total
...
ingest_lines_total count lines (which apparently are the same as points, quite confusingly)
No yaks harmed in the making of this PR.
(NOTE: the code around metric, especially dealing with happy and error paths is very painful;
to be done in another PR)
2021-05-13 17:55:07 +02:00
9583636748
feat: we now can read parquet files form all kind of object stores
2021-05-12 18:05:34 -04:00
8be95856ab
test: Add a test with multiple threads using a process clock
2021-05-12 13:31:26 -04:00
cecb4afc58
docs: Add some documentation on the assumptions around this design
2021-05-12 13:31:26 -04:00
b3fb61a0b3
refactor: Rename now_nanos to system_clock_now for clarity
2021-05-12 13:31:26 -04:00
425aacc391
refactor: Extract ProcessClock into its own type
2021-05-12 13:31:26 -04:00
b749353d21
refactor: Use a compare_exchange loop instead of Arc Mutex
2021-05-12 10:58:08 -04:00
5dfd152549
test: Use the now_nanos helper function more in tests
2021-05-12 10:58:08 -04:00
f28c9ae04c
docs: Add unit and semantic information about the process clock
2021-05-12 10:58:08 -04:00
513d4731be
feat: Add a process clock to Db and use it for Sequenced Entries
...
Connects to #1157 .
2021-05-12 10:58:06 -04:00
f98807936d
test: Some tests don't call await, so they don't need to be async
2021-05-12 10:57:05 -04:00
696e4e0cfd
fix: ensure metrics not overwriting
2021-05-11 20:57:31 +01:00
4409d2c8af
feat: instrument catalog locks ( #1464 )
...
* feat: instrument catalog locks (#1355 )
* chore: add metrics test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-11 18:59:11 +00:00
9d0c3a2b1a
refactor: Remove multi-table per chunk code in MUB ( #1471 )
...
* refactor: Remove multi-table per chunk code in MUB
* fix: clippy
* fix: bench build
* fix: merge conflicts
2021-05-11 17:49:07 +00:00
d1da954fe4
feat: don't store encoded strings twice in RLE dictionaries ( #1469 )
2021-05-11 15:22:25 +00:00
3622a92c8b
feat: wire in rb column metrics
2021-05-11 13:00:52 +01:00
795f5bfcb7
refactor: make `StatValues::{min,max}` optional + handle NaNs
...
This will allow us to:
- handle all-NULL columns correctly
- be in-line with Parquet (where min/max are optional)
- handle NaNs at least somewhat sane (they do not "poison" stats
anymore)
2021-05-10 17:12:25 +02:00
f037c1281a
feat: Calculate all system tables "on demand" ( #1452 )
...
* feat: compute system.columns table on demand
* feat: compute system.chunk_columns on demand
* feat: compute system.operations on demand
* fix: fixup schemas
* fix: Log errors
* fix: clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-10 14:43:55 +00:00
9f5350a6c5
fix: Load only databases for which a config exists
...
Closes #1450
2021-05-10 13:14:22 +02:00
c6b933eb63
chore: merge main to branch
2021-05-07 18:40:17 -04:00
971500681f
refactor: address Andrew's and Carol's comment
2021-05-07 17:33:19 -04:00
ba015ee4df
refactor: clean up and add comments
2021-05-07 09:31:41 -04:00
eae3fec571
feat: wire up regex UDF as predicate filter expr
2021-05-07 13:44:51 +01:00
b5ea71f45f
feat: Expose the storage usage for each column in system.chunk_columns ( #1441 )
...
* feat: Expose the storage usage for each column in system.chunk_columns
* fix: fixup logical conflicts
* refactor: move coalsce logic into the read buffer
* fix: Update system_tables to not use coalese
* fix: Improve comments
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-05-07 12:36:49 +00:00
9320f59de0
feat: add shard sink indirection ( #1447 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-07 11:04:51 +00:00
d7253c72c0
feat: Only calculate system.chunks table "on demand" ( #1446 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-07 10:26:56 +00:00
febc1538ff
chore: Update Rust version ( #1445 )
...
* chore: Update Rust version
* refactor: Make struct constructor field orderings consistent
Sometimes I changed the struct definition, sometimes changed the struct
construction instance, depending on consistency with code around each
(other similar structs, function argument orders, etc)
More info: https://rust-lang.github.io/rust-clippy/master/index.html#inconsistent_struct_constructor
* refactor: Use flatten where appropriate
One instance is a false positive with a clippy bug.
More info:
- https://rust-lang.github.io/rust-clippy/master/index.html#filter_map_identity
- https://rust-lang.github.io/rust-clippy/master/index.html#manual_flatten
* refactor: Use Option map instead of match
More info: https://rust-lang.github.io/rust-clippy/master/index.html#manual_map
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 22:07:10 +00:00
55bf848bd2
feat: Now we can query directly from files in object store
2021-05-06 18:02:17 -04:00
7f6b11266d
feat: instrument catalog locks ( #1355 ) ( #1439 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 17:09:52 +00:00
44de42906f
refactor: use Arc<str> instead of Arc<String> ( #1442 )
2021-05-06 17:05:08 +00:00
49c0b8b90c
feat: pull-based metrics ( #1355 ) ( #1414 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 15:54:30 +00:00
216903a949
refactor: move protobuf conversion logic to generated_types ( #1437 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 15:49:27 +00:00
884baf7329
feat: add column_type and influxdb_column_type, remove row_count from system.columns ( #1415 )
...
* feat: add column_type and influxdb_column_type, remove row_count from system.columns
* fix: update tests
* fix: more test update
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: fmt
* fix: copy/paste type conversion to avoid cross dependency between data_types and internal_types
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-05-06 12:59:30 +00:00
578dc0db25
feat: Add more logs to shed light on the curious incident with missing metrics in the nighttime
2021-05-06 14:42:48 +02:00
10f89a3e8d
refactor: split entry out into separate crate ( #1428 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 11:36:23 +00:00
a5c92fae8a
chore: merge main to branch
2021-05-05 13:48:42 -04:00
411cf134e9
refactor: explode arrow_deps ( #1425 )
...
* refactor: explode arrow_deps
* chore: workaround doctest bug
2021-05-05 16:59:12 +00:00
4395ede244
Merge branch 'main' into debug-chunk-metrics
2021-05-05 15:43:32 +00:00
2b0d7cfb91
feat: Add debug to update_chunk_state metrics
2021-05-05 17:37:57 +02:00
fcb37a0b1d
feat: more testing scenarios for quering parquet files
2021-05-05 10:57:02 -04:00
4a64e22e64
refactor: Use trait object and deref instead of cloning Arc in tests
2021-05-05 10:55:12 -04:00
e32fa43a53
docs: Add note about implication of write buffer errors
2021-05-05 10:55:12 -04:00
7d5c988fba
feat: Actually route SequencedEntry to the Write Buffer, if present
...
Connects to #1157 .
Rearrange some code and comments to be consistent with the design. Make
some more places not care whether they're getting an owned or borrowed
SequencedEntry.
2021-05-05 10:55:11 -04:00
54c5f984d5
fix: Use stdlib's path manipulation rather than format
...
The syntax highlighting in my editor broke because of the unmatched
double quote, which got me to look a bit closer at this test. These
tests would have failed on Windows.
2021-05-05 10:55:11 -04:00
231abd221f
refactor: Extract a TestDbBuilder
2021-05-05 10:55:11 -04:00
62dfb47825
refactor: Reorganize test imports
2021-05-05 10:55:11 -04:00
9e61b470e7
feat: change MemoryStream to accept multiple record batches
2021-05-05 13:29:16 +02:00
34754ebcdb
refactor: move MemoryStream to arrow_deps
2021-05-05 13:29:16 +02:00
733d502350
refactor: fix tests
2021-05-04 18:38:42 +01:00
9aa144e0f4
feat: add per-stage current chunk storage
2021-05-04 17:43:53 +01:00
3b7c5ac350
fix(storage rpc): do not send back tags with empty values ( #1403 )
2021-05-04 10:35:24 +00:00
b579ef8646
feat: Add jemalloc stats
2021-05-03 12:10:48 +02:00
3c5595d046
Merge branch 'main' into ntran/unload_chunks
2021-04-30 22:02:38 +00:00
979f5f9347
refactor: write buffer to use sequenced entry and new segment
...
This refactors the write buffer to use the sequenced entry structure and the new segment definition. It removes the old replicated write and write_buffer.fbs.
Finally, it updates the SequencedEntry wrapper type around the Flatbuffer structure to be a trait so that SequencedEntry can be initialized from a borrowed Flatbuffer or an owned Vec<u8>.
How writes go into segments in the buffer and any kind of validation will likely have to be updated based on what kinds of guarantees we want to make in the buffer. However, that should probably come after we've rethought the design a bit around the new layout of chunks in the Parquet persistence.
2021-04-30 17:00:23 -04:00
a967ebfabd
refactor: rename closing to closed ( #1396 )
...
* refactor: rename closing to closed
* refactor: further renames
2021-04-30 20:59:45 +00:00
34a3388a49
feat: unload chunks from read buffer but keep them in object store
2021-04-30 16:12:02 -04:00
c2f7e7efea
feat: warn on dropping from open partition ( #1395 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-30 18:56:49 +00:00
ebe671e59e
refactor: include default labels
2021-04-30 14:03:09 +01:00
40b9b09cdc
refactor: rename assert_table_eq to assert_batches_eq ( #1368 )
2021-04-30 10:51:08 +00:00
c9b33c6b7d
chore: Merge branch 'main' into ntran/test_query_parquets
2021-04-29 14:22:34 -04:00
f89d56af8a
chore: merge mian to branch
2021-04-29 14:14:04 -04:00
2d643d4407
fix: set nullability correctly for system tables ( #1387 )
2021-04-29 18:09:27 +00:00
e87973babe
refactor: address review comments
2021-04-29 13:15:43 -04:00
ea476c86be
feat: add chunk size histogram
2021-04-29 17:10:57 +01:00
5ef9be4abd
refactor: use RecordBatch::try_from_iter to make system tables ( #1374 )
...
* refactor: use RecordBatch::try_from_iter to make system tables
* fix: clippy
2021-04-29 15:32:35 +00:00
0788892413
feat: add row_count to system.chunks and Chunk managment API ( #1373 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-29 13:53:25 +00:00
3672a20570
feat: Add ingest_entries_bytes_total counter
2021-04-29 14:03:05 +02:00
262bf446ab
feat: change background worker to use backoff instead of poll ( #1339 ) ( #1347 )
...
* feat: change background worker to use backoff instead (#1339 )
* chore: fix comment
* chore: fix tests
* chore: review comments
* chore: review feedback
2021-04-29 11:10:51 +00:00
b1924d10b4
fix: Only show current database's system.operations table ( #1310 )
...
* test: add test for operations.system_tables
* fix: only show operations for current database
* fix: update test
* fix: improve test
* refactor: filter in Schema provider rather than in job tracker
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-28 21:06:19 +00:00
402d9c748c
chore: cargo fmt
2021-04-28 16:52:52 -04:00
0ea8861a65
chore: fix merge that causes test failures
2021-04-28 16:38:29 -04:00
2a2760bd18
feat: complete tests where data in both RUB and OS
2021-04-28 16:14:07 -04:00
140d96dbea
feat: tests ffor loading data to object store and make sure twe still query read buffer
2021-04-28 15:59:17 -04:00
9aefcd216f
fix: Validate that ClockValue is never 0
2021-04-28 13:54:55 -04:00
dcc732b154
Merge branch 'main' into cn/server-id-fix
2021-04-28 17:19:29 +00:00
2f4d7189ff
fix: Validate ServerId when creating structs from flatbuffers
...
When we get the flatbuffers, we won't have the server ID in addition to
the flatbuffers-- it's in the flatbuffers. But we want to validate the
`ServerId` once when the `SequencedEntry` is created so that its
`server_id` method can assume it has a valid `ServerId`.
2021-04-28 13:06:12 -04:00
c4259d5336
refactor: update benchmarks
2021-04-28 16:11:57 +00:00
a9ef604ef6
perf: avoid using channels for query execution
...
Pre-sized channels get full when the results to send over them are larger than the capacities. This causes significant runtime overhead and slows down query performance.
This commit removes the intermediate channels. The potential downside to this approach is there may be more buffering which could increase memory usage during query and also block a thread for longer periods of time.
2021-04-28 16:11:57 +00:00
d6c0f40630
test: add read_group benchmark
2021-04-28 16:11:57 +00:00
5dfe312867
test: add benchmark for influxrpc read_filter
2021-04-28 16:11:57 +00:00
b6054f9df9
feat: Make background task period configurable
2021-04-28 15:47:31 +02:00
9b57214344
feat: move closing chunks as soon as possible ( #1329 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-28 07:50:15 +00:00
6bdc153361
feat: sort RUB (read buffer) chunks ( #1308 )
...
* feat: sort chunks before upserting to read buffer (#1216 )
* chore: review feedback
* chore: fix merge conflict
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-27 16:50:48 +00:00
7ca1da3fcd
feat: pushdown table and partition key predicates to catalog ( #736 ) ( #1327 )
...
* feat: catalog predicate pushdown (#736 )
* chore: fix lints
* chore: review comments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-27 15:31:47 +00:00
91bccdfca3
ci: pass `--document-private-items` to `cargo doc`
2021-04-27 15:42:07 +02:00
e78a81ee59
Merge branch 'main' into crepererum/ci_doc
2021-04-27 12:03:04 +00:00
c3cb86fc1a
Merge branch 'main' into crepererum/issue1296
2021-04-27 11:58:10 +00:00
eddc9319ff
docs: deny broken intradoc links
2021-04-27 13:22:28 +02:00
20117de078
feat: string dictionary encoding ( #1220 ) ( #1262 )
...
* feat: string dictionary encoding (#1220 )
* chore: review comments
* chore: fix lint
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-27 09:36:58 +00:00
c8594ce743
refactor: chunks shall always have data (= 1 table)
...
Chunks now always have "data" (aka exactly 1 table including
schema/columns). Open chunks can only be created with data. Rollovers do
NOT create open chunks anymore (this is now only done for incoming
data).
Closes #1296 .
2021-04-27 10:28:57 +02:00
35bc09ef7c
refactor: use `FromIter` rather than array builders to create system table arrays ( #1246 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-26 19:47:11 +00:00
272cdb85ce
fix: Use the ServerId type everywhere, for writing, querying, anything
2021-04-26 18:44:32 +00:00
b8face3335
refactor: Organize use statements
2021-04-26 18:44:32 +00:00
67f5ad841d
refactor: Introduce ServerId and CurrentServerId types
2021-04-26 18:44:32 +00:00
0a835436ac
feat: use bitmasks within MUB ( #1274 ) ( #1289 )
...
* feat: use bitmasks within MUB (#1274 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-26 18:00:16 +00:00
e33af0c084
fix: remove wait_for_complete
2021-04-26 16:59:36 +00:00
fcf5c304a6
fix: fixup merge conflicts
2021-04-26 16:59:36 +00:00
348f38c2c9
feat: add status column
2021-04-26 16:59:36 +00:00
9c74074095
feat: List background operations as system.operations table
2021-04-26 16:59:36 +00:00
faec98eab9
refactor: remove time column from row group
2021-04-26 09:51:06 +00:00
dd2bee3343
fix: don't over count new chunks
2021-04-23 20:25:37 +01:00
552409ea70
feat: add metrics for chunk lifecycle
2021-04-23 17:13:43 +00:00
0e3b7bff49
refactor: plumb metrics registry to catalog
2021-04-23 17:13:43 +00:00
eacfe9af6a
refactor: address PR feedback
2021-04-23 15:58:48 +00:00
d3218802ab
refactor: more API metric coverage
2021-04-23 15:58:48 +00:00
97b2369140
refactor: swap existing metrics for THE NEW WAY
2021-04-23 15:58:48 +00:00
f12f1eeef7
refactor: make chunks per-table
...
This changes the hierarchy from
```
database -> partition -> chunk -> table
```
to
```
database -> partition -> table -> chunk
```
Only the high-level APIs are changed for now. The chunk states (like
MutableBuffer and ReadBuffer) still multiplex tables, although they will
always only get a single table assigned (or no table if no data was
presented yet).
Closes #1256 .
2021-04-23 09:18:55 +02:00
74c25f541d
feat: fast MUB dictionary arrow conversion ( #1273 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-22 20:31:44 +00:00
83d6550316
feat: Implement write_entry_downstream
2021-04-21 20:50:46 +00:00
88ca1a5245
fix: Rename wal.fbs to write_buffer.fbs
2021-04-21 17:43:03 +00:00
80995afb70
fix: Change WAL to Write Buffer in comments and documentation
2021-04-21 17:43:03 +00:00
5f024401a4
refactor: Rename data_types::wal module to data_types::write_buffer
2021-04-21 17:43:03 +00:00
46f9360c58
fix: Rename WAL_DIR to WRITE_BUFFER_DIR and redefine it as wb, not wal
2021-04-21 17:43:03 +00:00
8baeaab8ca
refactor: Rename Db.wal_buffer to write_buffer
2021-04-21 17:43:03 +00:00
4bfcb68536
refactor: Remove a WalError that doesn't appear to be used
2021-04-21 17:43:03 +00:00
0677c544bb
refactor: Rename WalBufferRollover to WriteBufferRollover
2021-04-21 17:43:03 +00:00
ee45db402b
refactor: Rename WalBufferConfig to WriteBufferConfig
2021-04-21 17:43:03 +00:00
1cf3ab3978
feat: enable disabling MUB snapshot caching ( #1203 ) ( #1264 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-21 08:21:31 +00:00
508aa8be8c
refactor: satisfy new clippy lints
2021-04-20 17:30:50 +00:00
c5d99ba9af
feat: wire up hard buffer limit
...
Fixes #1255 .
2021-04-20 13:25:31 +02:00
d05585f736
refactor: log chunk migration to rb at info
2021-04-19 16:08:49 +00:00
82c1d94ce1
refactor: Use Option.map where possible
2021-04-19 08:48:11 -04:00
fd0da7e74a
chore: upgrade arrow and Rust
...
See https://github.com/apache/arrow/pull/10082 for upstream PR.
2021-04-19 14:00:04 +02:00
4c23ca8888
feat: full implementation of parquet's read_filter for review
2021-04-16 16:03:24 -04:00
e226b5a820
feat: Use TimestampNanosecondArray for timestamps in IOx ( #1230 )
...
* refactor: Create Arrow arrays using iterators
* feat: use Timestamp64(TimeUnit::Nanosecond) for timestamps
* feat: add support for timestamp array
* fix: update more tests
* fix: remove unecessary code
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-16 15:55:33 +00:00
f092294da3
fix: Use MAX (window end) for timestamps in read group ( #1228 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-16 10:51:38 +00:00
878b1b318e
feat: Initial scaffolding for routing layer
...
Part of #916
Adding first-class concept of ShardId in shard config, fixes #1156
NEXT:
- [ ] implement sharder
- [ ] implement `write_entry_downstream`
- [ ] add tests
2021-04-15 09:02:47 +00:00
4001ec73c9
Merge branch 'main' into ntran/query_parquet_2
2021-04-14 20:33:03 +00:00
4e2d59d9a5
feat: saimplement a few more functions as part of supporting query dfrom parquet files
2021-04-14 16:06:47 -04:00
025f760479
refactor: change sync::RwLock to parking_lot
2021-04-14 19:18:03 +00:00
59ca090aef
feat: Use single db-wide executor for running queries ( #1198 )
...
* refactor: plumb executor into all Db instances
* refactor: Route all query executions through worker pool
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-14 16:46:02 +00:00
5bb34e9a97
refactor: use read_buffer column range for time range
2021-04-14 16:10:24 +00:00
04f594bf94
perf: use column_values in read buffer
2021-04-14 15:13:35 +00:00
f5f768d750
feat: Add a dedicated threadpool for running queries ( #1191 )
...
* feat: use a dedicated tokio threadpool for running queries
* feat: plumb number of executor threads through to command line
thread through command line
* fix: Logical merge conflict
* fix: another logical conflict
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-14 10:48:09 +00:00
f430013141
refactor: rename `Db::load_chunk_to_object_store` to `Db::write_chunk_to_object_store
...
For #1083 .
2021-04-14 12:37:43 +02:00
1c6690ae3b
fix: fix constant use in `server::db::tests::read_from_read_buffer`
2021-04-14 12:37:43 +02:00
26ce520cf1
feat: hook RB->OS into lifecycle
...
Fixes #1083 .
2021-04-14 12:37:43 +02:00
7b0b975433
feat: add Db::load_chunk_to_object_store_in_background
...
For #1083 .
2021-04-14 12:37:43 +02:00
84b2ce4571
docs: extend docs of `Db::load_chunk_to_read_buffer_in_background`
2021-04-14 12:37:43 +02:00
9834c845db
test: add influxrpc tag_values benches
...
The initial benchmarks look like this on my i9 MBP:
```
Data in one open chunk and one closed chunk of mutable buffer/tag0/no_pred 1.00 91.0±2.55ms ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag0/with_pred 1.00 11.5±0.72ms ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag1/no_pred 1.00 120.3±5.10ms ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag1/with_pred 1.00 11.2±0.22ms ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag2/no_pred 1.00 203.2±8.45ms ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag2/with_pred 1.00 11.2±0.21ms ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag0/no_pred 1.00 100.3±3.73ms ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag0/with_pred 1.00 31.2±1.80ms ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag1/no_pred 1.00 126.7±2.29ms ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag1/with_pred 1.00 33.0±1.70ms ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag2/no_pred 1.00 212.0±6.86ms ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag2/with_pred 1.00 18.1±0.99ms ? ?/sec
Data in single open chunk of mutable buffer/tag0/no_pred 1.00 98.7±6.08ms ? ?/sec
Data in single open chunk of mutable buffer/tag0/with_pred 1.00 11.2±0.37ms ? ?/sec
Data in single open chunk of mutable buffer/tag1/no_pred 1.00 118.9±3.97ms ? ?/sec
Data in single open chunk of mutable buffer/tag1/with_pred 1.00 11.7±0.64ms ? ?/sec
Data in single open chunk of mutable buffer/tag2/no_pred 1.00 202.1±8.49ms ? ?/sec
Data in single open chunk of mutable buffer/tag2/with_pred 1.00 11.1±0.27ms ? ?/sec
Data in two read buffer chunks/tag0/no_pred 1.00 109.2±5.20ms ? ?/sec
Data in two read buffer chunks/tag0/with_pred 1.00 44.2±1.83ms ? ?/sec
Data in two read buffer chunks/tag1/no_pred 1.00 132.9±3.79ms ? ?/sec
Data in two read buffer chunks/tag1/with_pred 1.00 41.7±2.43ms ? ?/sec
Data in two read buffer chunks/tag2/no_pred 1.00 222.4±7.00ms ? ?/sec
Data in two read buffer chunks/tag2/with_pred 1.00 27.9±0.92ms ? ?/sec
```
2021-04-14 09:36:39 +00:00
4db3a4b3b5
test: enable writer to split large batches
2021-04-14 09:36:39 +00:00
3cfbfe8ebf
refactor: expose query_tests to benches crate
2021-04-14 09:36:39 +00:00
8e0ee48018
Merge branch 'main' into ntran/query_local_parquet
2021-04-13 22:38:56 +00:00
c9cbc74857
feat: Use a DID newtype instead of u32 for dictionary ids
...
Rationale
---------
We use `u32` throughout the codebase to reference for interned dictionary strings.
We also use `u32` for other reasons and it would be nice to get some help from the compiler
to avoid mixing them up
2021-04-13 22:38:41 +00:00
05bf28ce85
feat: Add 2 main functions table_schema and table_names for Parquet Chunk ato pay a foundation for querying it
2021-04-13 18:23:55 -04:00
150ed4e1d9
refactor: Remove async from `InfluxRPCPlanner` ( #1200 )
...
* refactor: Remove async from InfluxRPCPlanner
* fix: make it compile
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-13 22:17:19 +00:00
Marco Neumann
kodiakhq[bot]
Raphael Taylor-Davies
Edd Robinson
Andrew Lamb
Nga Tran
Marko Mikulicic
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
Paul Dix
Jake Goulding
Andrew Lamb