466416016b
Merge pull request #1950 from influxdata/cn/refactor-mock-chunks
...
refactor: Go from 3 mock chunks to 1
2021-07-12 17:52:52 +00:00
fd7705d9b4
Merge branch 'main' into cn/refactor-mock-chunks
2021-07-12 17:46:07 +00:00
07b25ad73c
Merge pull request #1952 from influxdata/crepererum/issue1919
...
feat: add API+CLI to unload chunks
2021-07-12 16:21:37 +00:00
c62c7d78d2
Merge branch 'main' into crepererum/issue1919
2021-07-12 16:14:13 +00:00
c681da1031
refactor: Define the TestChunk methods with macros
2021-07-12 09:59:12 -04:00
4e53a32928
refactor: Completely replace query::provider::overlap::TestChunk with query::test::TestChunk
2021-07-12 09:59:12 -04:00
1698edcc39
refactor: Implement query::provider::overlap::TestChunk in terms of query::test::TestChunk
2021-07-12 09:59:12 -04:00
dc0b97e121
refactor: Completely replace TestChunkMeta with TestChunk
2021-07-12 09:59:12 -04:00
96f9485792
refactor: Move a with_no_stats method to be entirely defined on TestChunk
2021-07-12 09:59:12 -04:00
b4c5a87088
refactor: Rename int field to i64 field to be more consistent
2021-07-12 09:59:12 -04:00
54f7ee8b8d
refactor: Implement TestChunkMeta in terms of TestChunk
...
This is a temporary step to make sure TestChunk does everything
TestChunkMeta needs
2021-07-12 09:59:12 -04:00
ee545ce90e
test: Make _with_stats methods able to optionally take max/min
...
Not used yet, but will be when this is unified with query/src/pruning.rs
2021-07-12 09:59:12 -04:00
b26aae1cb4
test: Add an arg to control whether to add a column summary at all
...
Always true for now, but there are some cases in query/src/pruning.rs
that don't add any column summaries that will use this with `false`.
2021-07-12 09:59:12 -04:00
6cd75bc688
test: Optionally take stats in add_schema_to_table
...
This gets rid of a lookup and construction of default stats that aren't
necessary
2021-07-12 09:59:12 -04:00
e05ca7f98b
fix: Change a method name that says null to not say null
...
The comment and implementation seem to indicate this is creating
non-null data.
2021-07-12 09:59:12 -04:00
4406d8a219
test: Always initialize a TableSummary on TestChunk
2021-07-12 09:59:12 -04:00
22d4040c81
test: Always initialize a Schema for TestChunk
2021-07-12 09:59:12 -04:00
92cb5986f1
test: Initialize a schema on TestChunk to always exist
2021-07-12 09:59:12 -04:00
78f1c4fc80
test: Chunks can only have one table; no need to specify repeatedly
...
This lets us make the name required and always present on TestChunks,
and make the ID optional.
2021-07-12 09:59:12 -04:00
d2aadddeef
refactor: Remove unneeded reassignments
2021-07-12 09:59:11 -04:00
15aac65c2c
fix: Arrange use statements so rustfmt can manage their order
2021-07-12 09:59:11 -04:00
670826daf9
refactor: make object_store construction interface consistent ( #1944 )
...
* refactor: make object_store construction interface consistent
* fix: benchmarks
* fix: doc build
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-12 12:56:36 +00:00
3d008f4d27
feat: add API+CLI to unload chunks
...
Closes #1919 .
2021-07-12 14:06:01 +02:00
3dae69ede9
Merge pull request #1949 from influxdata/pd-remove-mutable-linger
...
refactor: remove mutable_linger_seconds from lifecycle
2021-07-11 12:31:48 +00:00
6f2d20cb19
chore: reword comment on late_arrive_window_seconds for clarity
2021-07-11 08:25:31 -04:00
2854b54420
refactor: leave deprecated mutable_linger_seconds in proto
2021-07-10 12:48:50 -04:00
0c8c81a321
refactor: remove mutable_linger_seconds from lifecycle
...
The interplay between mutable_linger_seconds, late_arrive_window and persist_age_threshold_seconds can be tricky to reason about. I realized that the lifecycle rules can be simplified by removing mutable_linger_seconds and instead using late_arrive_window_seconds for the same purpose. Semantically, they basically mean the same thing. We want to give data around this amount of time to arrive before the system persists it, which gives it more of an opportunity to persist non-overlapping data.
When a partition goes cold for writes, after we've waiting past this window, we should compact and persist that partition. This removes one unnecessary knob from the lifecycle configuration and also removes the potential for conflicting configuration options.
2021-07-10 08:04:33 -04:00
9534220035
feat: Add any lifecycle_action to system.chunks and API ( #1947 )
2021-07-09 17:38:29 +00:00
7af560aa99
feat: Persist lifecycle action ( #1888 )
...
* feat: add split and persist operation
* docs: Improve doc strings
* refactor: use for loop rather than map
* refactor: Make it clear that the lifecycle policy picks the split timestamp
* fix: race condition
* docs: improve comments
* fix: logical merge conflict
* fix: clippy
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2021-07-09 13:21:46 +00:00
1a79bf7e99
refactor: Make aws/azure/gcs optional features and stop compiling 100 dependencies during dev ( #1933 )
...
* feat: make aws, gcp, azure dependencies optional
* fix: only run object store tests if the features are enabled
* fix: clean up testing
* fix: rename step
* fix: add to list of jobs
* fix: remove test with object store
* fix: review comments
2021-07-09 11:38:30 +00:00
dac1e6f5ea
feat: add GIT hash to `--version` ( #1942 )
...
```text
$influxdb_iox --version
influxdb_iox 0.1.0, revision 34dcd99
2021-07-09 11:08:46 +00:00
3cb8f297b1
refactor: encapsulate the ObjectStore implementations in the object store crate ( #1932 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-09 10:38:32 +00:00
34dcd991d3
Merge pull request #1929 from influxdata/crepererum/fix_query_schema_unwrap
...
refactor: pass schema arcs from catalog to query engine (instead of creating them on-demand)
2021-07-09 07:53:16 +00:00
676034b4ae
docs: explain why the path placeholder is there
2021-07-09 09:45:13 +02:00
bc958e2ff0
refactor: use Arcs to pass schemas around
2021-07-09 09:45:12 +02:00
09e611deb7
refactor: lift query schema generation up to caller
...
Do no longer scan chunks during query planning to determine the schema
(except for the lifetime jobs where we have a good reason to do so).
Instead pass the schema down to from whoever is triggering the query.
For real SQL queries, we then just use the the table-wide schemas
introduced in #1913 .
Apart from avoiding schema merges we now also don't crash any longer
when no chunks are left in the table (aka columns are present but all
rows are gone).
Fixes #1768 .
Fixes #1884 .
2021-07-09 09:24:21 +02:00
6ac1420335
test: fix out dir for query tests
2021-07-09 09:16:28 +02:00
6eb52f173e
Merge pull request #1941 from influxdata/pd-remove-mutable-minimum-age
...
refactor: remove unused mutable_minimum_age_seconds lifecycle setting
2021-07-08 23:58:51 +00:00
e41fd2a821
refactor: remove unused mutable_minimum_age_seconds lifecycle setting
...
Closes #1878
2021-07-08 18:34:01 -04:00
078d5f7cfb
Merge pull request #1940 from influxdata/cn/chunk-times
...
feat: Record first and last write times on MUB chunks
2021-07-08 21:06:17 +00:00
c37053ad46
Merge branch 'main' into cn/chunk-times
2021-07-08 20:58:54 +00:00
299269a161
Merge pull request #1879 from influxdata/cn/kafka-read-metrics-and-e2e-tests
...
feat: Collect metrics on failed ingest from Kafka
2021-07-08 20:46:57 +00:00
a2726c7e92
Merge branch 'main' into cn/kafka-read-metrics-and-e2e-tests
2021-07-08 20:40:19 +00:00
b9a6a11b34
feat: Record time of first/last write on MBChunk
2021-07-08 16:39:35 -04:00
22495dd355
fix: Take a TableBatch in the MBChunk constructor
...
Thus ensuring all MBChunks will have data in them.
2021-07-08 16:39:35 -04:00
57501e7f59
refactor: Remove a test function that's only slightly different from another test function
2021-07-08 16:39:35 -04:00
2f9cf857af
fix: MBChunk.write_columns doesn't need to be pub
2021-07-08 16:39:35 -04:00
548c64539e
fix: Wrap lines at 100 chars
2021-07-08 16:39:33 -04:00
74c0a6cb00
fix: Arrange use statements so rustfmt can manage their order
2021-07-08 16:39:02 -04:00
9961cf8008
Merge pull request #1902 from influxdata/ntran/avoid_sort_in_scan
...
feat: store sort_key in the chunk schema after the chunk is sorted
2021-07-08 20:29:38 +00:00
kodiakhq[bot]
Carol (Nichols || Goulding)
Andrew Lamb
Marco Neumann
Paul Dix
Raphael Taylor-Davies