d2aadddeef
refactor: Remove unneeded reassignments
2021-07-12 09:59:11 -04:00
15aac65c2c
fix: Arrange use statements so rustfmt can manage their order
2021-07-12 09:59:11 -04:00
670826daf9
refactor: make object_store construction interface consistent ( #1944 )
...
* refactor: make object_store construction interface consistent
* fix: benchmarks
* fix: doc build
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-12 12:56:36 +00:00
3d008f4d27
feat: add API+CLI to unload chunks
...
Closes #1919 .
2021-07-12 14:06:01 +02:00
3dae69ede9
Merge pull request #1949 from influxdata/pd-remove-mutable-linger
...
refactor: remove mutable_linger_seconds from lifecycle
2021-07-11 12:31:48 +00:00
6f2d20cb19
chore: reword comment on late_arrive_window_seconds for clarity
2021-07-11 08:25:31 -04:00
2854b54420
refactor: leave deprecated mutable_linger_seconds in proto
2021-07-10 12:48:50 -04:00
0c8c81a321
refactor: remove mutable_linger_seconds from lifecycle
...
The interplay between mutable_linger_seconds, late_arrive_window and persist_age_threshold_seconds can be tricky to reason about. I realized that the lifecycle rules can be simplified by removing mutable_linger_seconds and instead using late_arrive_window_seconds for the same purpose. Semantically, they basically mean the same thing. We want to give data around this amount of time to arrive before the system persists it, which gives it more of an opportunity to persist non-overlapping data.
When a partition goes cold for writes, after we've waiting past this window, we should compact and persist that partition. This removes one unnecessary knob from the lifecycle configuration and also removes the potential for conflicting configuration options.
2021-07-10 08:04:33 -04:00
7b7a60993d
feat: consider time as a special key
2021-07-09 18:54:22 -04:00
8f4463664c
feat: add super_key function
2021-07-09 15:37:04 -04:00
9534220035
feat: Add any lifecycle_action to system.chunks and API ( #1947 )
2021-07-09 17:38:29 +00:00
698b3773df
feat: Flatbuffers root_type and file_identifier
2021-07-09 16:39:40 +02:00
7af560aa99
feat: Persist lifecycle action ( #1888 )
...
* feat: add split and persist operation
* docs: Improve doc strings
* refactor: use for loop rather than map
* refactor: Make it clear that the lifecycle policy picks the split timestamp
* fix: race condition
* docs: improve comments
* fix: logical merge conflict
* fix: clippy
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2021-07-09 13:21:46 +00:00
1a79bf7e99
refactor: Make aws/azure/gcs optional features and stop compiling 100 dependencies during dev ( #1933 )
...
* feat: make aws, gcp, azure dependencies optional
* fix: only run object store tests if the features are enabled
* fix: clean up testing
* fix: rename step
* fix: add to list of jobs
* fix: remove test with object store
* fix: review comments
2021-07-09 11:38:30 +00:00
dac1e6f5ea
feat: add GIT hash to `--version` ( #1942 )
...
```text
$influxdb_iox --version
influxdb_iox 0.1.0, revision 34dcd99
2021-07-09 11:08:46 +00:00
3cb8f297b1
refactor: encapsulate the ObjectStore implementations in the object store crate ( #1932 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-09 10:38:32 +00:00
34dcd991d3
Merge pull request #1929 from influxdata/crepererum/fix_query_schema_unwrap
...
refactor: pass schema arcs from catalog to query engine (instead of creating them on-demand)
2021-07-09 07:53:16 +00:00
676034b4ae
docs: explain why the path placeholder is there
2021-07-09 09:45:13 +02:00
bc958e2ff0
refactor: use Arcs to pass schemas around
2021-07-09 09:45:12 +02:00
09e611deb7
refactor: lift query schema generation up to caller
...
Do no longer scan chunks during query planning to determine the schema
(except for the lifetime jobs where we have a good reason to do so).
Instead pass the schema down to from whoever is triggering the query.
For real SQL queries, we then just use the the table-wide schemas
introduced in #1913 .
Apart from avoiding schema merges we now also don't crash any longer
when no chunks are left in the table (aka columns are present but all
rows are gone).
Fixes #1768 .
Fixes #1884 .
2021-07-09 09:24:21 +02:00
6ac1420335
test: fix out dir for query tests
2021-07-09 09:16:28 +02:00
6eb52f173e
Merge pull request #1941 from influxdata/pd-remove-mutable-minimum-age
...
refactor: remove unused mutable_minimum_age_seconds lifecycle setting
2021-07-08 23:58:51 +00:00
e41fd2a821
refactor: remove unused mutable_minimum_age_seconds lifecycle setting
...
Closes #1878
2021-07-08 18:34:01 -04:00
078d5f7cfb
Merge pull request #1940 from influxdata/cn/chunk-times
...
feat: Record first and last write times on MUB chunks
2021-07-08 21:06:17 +00:00
c37053ad46
Merge branch 'main' into cn/chunk-times
2021-07-08 20:58:54 +00:00
299269a161
Merge pull request #1879 from influxdata/cn/kafka-read-metrics-and-e2e-tests
...
feat: Collect metrics on failed ingest from Kafka
2021-07-08 20:46:57 +00:00
a2726c7e92
Merge branch 'main' into cn/kafka-read-metrics-and-e2e-tests
2021-07-08 20:40:19 +00:00
b9a6a11b34
feat: Record time of first/last write on MBChunk
2021-07-08 16:39:35 -04:00
22495dd355
fix: Take a TableBatch in the MBChunk constructor
...
Thus ensuring all MBChunks will have data in them.
2021-07-08 16:39:35 -04:00
57501e7f59
refactor: Remove a test function that's only slightly different from another test function
2021-07-08 16:39:35 -04:00
2f9cf857af
fix: MBChunk.write_columns doesn't need to be pub
2021-07-08 16:39:35 -04:00
548c64539e
fix: Wrap lines at 100 chars
2021-07-08 16:39:33 -04:00
74c0a6cb00
fix: Arrange use statements so rustfmt can manage their order
2021-07-08 16:39:02 -04:00
9961cf8008
Merge pull request #1902 from influxdata/ntran/avoid_sort_in_scan
...
feat: store sort_key in the chunk schema after the chunk is sorted
2021-07-08 20:29:38 +00:00
c8126784a8
Merge branch 'main' into ntran/avoid_sort_in_scan
2021-07-08 20:22:18 +00:00
680394b50b
refactor: run fmt
2021-07-08 16:21:42 -04:00
72928aab3d
refactor: Move ChunkLifecycleAction to the data_types crate ( #1939 )
2021-07-08 20:18:33 +00:00
c5733ab4a7
refactor: remove redudant code
2021-07-08 16:11:42 -04:00
6738cb272f
refactor: remove duplicate test
2021-07-08 15:59:25 -04:00
da6249a4df
fix: address reviewers' comments and also fixe a bug they discovered
2021-07-08 15:54:54 -04:00
dd3eff7748
refactor: Always use `row_count` for count of rows in system.* tables ( #1937 )
2021-07-08 19:28:11 +00:00
8b9b369189
chore: Update DataFusion again ( #1930 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-08 15:14:45 +00:00
c6bf0a26f4
feat: Add metrics for when ingesting from the write buffer fails
...
So that we have some way of figuring out what might be going on.
2021-07-08 09:57:51 -04:00
9c722959fb
Merge pull request #1752 from influxdata/cn/kafka-read
...
feat: reading entries from Kafka
2021-07-08 13:39:16 +00:00
dd6303e85d
test: Make test data conform to Kafka partitioning assumptions
2021-07-08 09:31:52 -04:00
80e1dcafe0
feat: Support reading from all Kafka partitions
...
When reading from the Kafka write buffer, subscribe to all partitions in
a topic and start from the smallest offset available, instead of
assuming there will only be 1 partition per topic.
2021-07-08 09:30:59 -04:00
c90ef7b14b
fix: Create one consumer group per server+database
...
This hasn't caused any problems for me yet, but seemed like a good idea
because we want to be sure we don't get any of Kafka's consumer
rebalancing if we have multiple partitions.
2021-07-08 09:28:34 -04:00
e5168936f5
feat: Better error messages through to gRPC API + e2e Kafka Read tests
2021-07-08 09:28:34 -04:00
c53ae41d57
fix: Remove unneeded Option from the reading mock
2021-07-08 09:28:34 -04:00
854c28c41a
feat: Stream messages from Kafka into the database
2021-07-08 09:28:34 -04:00
Carol (Nichols || Goulding)
Andrew Lamb
Marco Neumann
kodiakhq[bot]
Paul Dix
Nga Tran
Marko Mikulicic
Raphael Taylor-Davies