00dacb5394
feat: add tests to verify the correctness as well as the explain of the plan
2021-05-19 17:31:16 -04:00
51de37e752
chore: run fmt
2021-05-19 15:28:44 -04:00
11561111d5
chore: merge main to branch
2021-05-19 15:11:15 -04:00
1f13842550
chore: modify comments
2021-05-19 14:49:48 -04:00
087d61f229
feat: Part 1 of predicate push down - Send predicates to MUB, RUB, and Parquet File. Note that MUB has not handled predicates yet
2021-05-19 13:59:51 -04:00
41093b54d6
docs: tweak profiling.md ( #1526 )
2021-05-19 17:04:59 +00:00
3f3e3673ab
Merge pull request #1525 from influxdata/cacheremotes
...
fix: Cache outbound gRPC connections
2021-05-19 16:33:16 +00:00
ce2f8351be
fix: Cache outbound gRPC connections
2021-05-19 18:28:45 +02:00
9b42c1a065
docs: Document how to use pprof tool ( #1520 )
2021-05-19 13:57:50 +00:00
7e223780f3
feat: Implement Display for query::predicate to improve debug printing of plans ( #1519 )
...
* feat: Implement Display for query::predicate to improve debug printing of plans
* fix: clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-19 12:38:34 +00:00
08eb4ad775
Merge pull request #1518 from influxdata/pprofrs
...
feat: Add /debug/pprof/profile
2021-05-19 11:15:54 +00:00
40b21dbca1
feat: Add /debug/pprof/profile
2021-05-19 13:08:42 +02:00
8417301c59
Merge pull request #1514 from influxdata/crepererum/issue1382-b
...
refactor: empty transaction during catalog creation
2021-05-18 16:29:51 +00:00
8db26485a4
refactor: empty transaction during catalog creation
...
That involves some refactoring which we are going to need anyway for
hooking up the "read" path of the catalog into the DB startup, namely:
- make `Db::new` require a preserved catalog
- introduce a helper function that can provide that
- as a consequence, all test-creations of a Db are now async
This prepares for #1382 .
2021-05-18 17:42:07 +02:00
0680a5167f
chore: Improve DataFusion plan logging ( #1508 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-18 11:08:06 +00:00
206ffae5fa
Merge pull request #1505 from influxdata/er/feat/read_buffer/byte_trim_arrow
...
feat: extend Byte trimming to all nullable integer encodings
2021-05-17 21:22:52 +00:00
094de69b57
Merge branch 'main' into er/feat/read_buffer/byte_trim_arrow
2021-05-17 22:18:41 +01:00
c1ea6be101
fix: Add context to panic error on tools ( #1501 )
...
* fix: Add context to panic error on tools
* fix: log new unwrap that has snuck in
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-17 21:05:01 +00:00
76ee9793dd
Merge pull request #1482 from influxdata/crepererum/issue1382
...
feat: wire up catalog preservation write path
2021-05-17 18:01:46 +00:00
c3cc58b2ff
Merge branch 'main' into crepererum/issue1382
2021-05-17 17:57:26 +00:00
4f0e46bcd5
refactor: track ingest metrics in one place ( #1503 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-17 16:25:01 +00:00
2963d63b5e
feat: implement byte trimming on nullable encodings
2021-05-17 14:32:55 +01:00
6a72274517
feat: extend implementations to more Arrow arrays
2021-05-17 14:32:55 +01:00
2b98bca9ca
test: allow from slice to be testable
2021-05-17 14:32:55 +01:00
b7ea53f5db
refactor: remove unnecessary from imps
2021-05-17 14:32:55 +01:00
18f0a7f614
docs: reference open issue
2021-05-17 14:01:51 +02:00
cdf0ada6a6
test: test preserved catalog <-> Db write wiring
2021-05-17 13:57:31 +02:00
91a45fd380
feat: simplify shutdown ( #1502 )
...
* feat: simplify shutdown
* chore: fix lint
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-17 11:50:14 +00:00
4299371cf2
refactor: remove some code
2021-05-17 12:32:48 +02:00
68729dd5ee
refactor: avoid string allocation
2021-05-17 12:32:34 +02:00
adcd8132e7
docs: more comments regarding catalog transaction handling
2021-05-17 12:05:08 +02:00
a99d53e771
docs: document `OpenTransaction::handle_action*`
2021-05-17 11:48:51 +02:00
840c11dab2
feat: wire up catalog preservation write path
...
Required a bit of refactoring:
- Add an extra layer between DB an catalog which is the "preserved
catalog" wrapper. This is required to make the ownership model
somewhat sane, because during the read operations the "preserved
catalog" is going to act on the in-mem catalog.
- Move "parquet file written" logic into binding `preserved catalog <->
catalog state`, so we have a single place where new parquet files are
announced. For now this only works for chunks that are already known
(i.e. the writing->written transation when coming from read buffer),
however in the next PR this will be extended to also handle totally
new parquet files during transaction playback.
**NOTE: This does NOT include the read path yet!**
Issue: #1382 .
2021-05-17 11:33:22 +02:00
4fb800c7a6
refactor: make PreservedCatalog easier to integrate
2021-05-17 11:33:22 +02:00
f4d7154746
fix: table summaries must include timestamp as well
2021-05-17 11:33:22 +02:00
ff004c1a51
feat: add `ObjectStore.path_from_dirs_and_filename`
2021-05-17 11:33:22 +02:00
7cced3242f
feat: add a way to parse infos from parquet paths
2021-05-17 11:33:22 +02:00
5969caccb0
feat: return parquet metadata from `write_to_object_store`
2021-05-17 11:33:22 +02:00
07db4932ee
refactor: rename data_types/src/chunk.rs -> data_types/src/chunk_metadata.rs ( #1500 )
2021-05-15 10:18:01 +00:00
133ce12827
chore: update deps ( #1499 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-14 20:22:16 +00:00
f9178dbb5f
feat: push metrics into catalog ( #1488 )
...
* feat: push metrics into catalog
* chore: minor cleanup
* fix: include db labels in chunk metric domains
* chore: fmt
* fix: don't allow dropping moving chunks
* chore: further tweaks
* chore: review feedback
* feat: use new_unregistered() for metric instruments instead of default
* chore: use &[KeyValue] instead of &Vec<KeyValue>
* refactor: make GauageValue non default constructible
2021-05-14 17:37:39 +00:00
cfa8160644
fix: don't test snapshotting when disabled ( #1496 ) ( #1498 )
2021-05-14 16:15:55 +00:00
73c43f1710
Merge pull request #1497 from influxdata/dom/rle-size-target
...
refactor: RLE selector target
2021-05-14 15:34:33 +00:00
db6c7728c7
refactor: use 10% target reduction for RLE
...
Comments say 10% but const was 30% - a 10% computed size reduction
sounds sensible!
2021-05-14 15:08:54 +01:00
874d7a1118
test: run rle_rows test
...
The rle_rows test was missing a #[test] annotation preventing it from
running.
2021-05-14 14:41:17 +01:00
d98d66e718
Merge pull request #1444 from influxdata/cn/wb-clock
...
feat: Add a process clock to Db and use it for Sequenced Entries
2021-05-14 13:04:26 +00:00
fdc8461c7f
Merge branch 'main' into cn/wb-clock
2021-05-14 13:00:06 +00:00
02ae69dffc
Merge pull request #1493 from influxdata/er/feat/read_buffer/num_rle
...
feat: implement Read Buffer run-length encoding for scalars
2021-05-14 12:42:51 +00:00
0d21d9e2e0
refactor: implement from_iter, reduce code!
2021-05-14 13:32:02 +01:00
ac4fa1e527
refactor: update read_buffer/src/column/encoding/scalar/rle.rs
...
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
2021-05-14 13:24:55 +01:00
Nga Tran
Andrew Lamb
kodiakhq[bot]
Marko Mikulicic
Marco Neumann
Edd Robinson
Raphael Taylor-Davies
Dom