1a79bf7e99
refactor: Make aws/azure/gcs optional features and stop compiling 100 dependencies during dev ( #1933 )
...
* feat: make aws, gcp, azure dependencies optional
* fix: only run object store tests if the features are enabled
* fix: clean up testing
* fix: rename step
* fix: add to list of jobs
* fix: remove test with object store
* fix: review comments
2021-07-09 11:38:30 +00:00
dac1e6f5ea
feat: add GIT hash to `--version` ( #1942 )
...
```text
$influxdb_iox --version
influxdb_iox 0.1.0, revision 34dcd99
2021-07-09 11:08:46 +00:00
3cb8f297b1
refactor: encapsulate the ObjectStore implementations in the object store crate ( #1932 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-09 10:38:32 +00:00
34dcd991d3
Merge pull request #1929 from influxdata/crepererum/fix_query_schema_unwrap
...
refactor: pass schema arcs from catalog to query engine (instead of creating them on-demand)
2021-07-09 07:53:16 +00:00
676034b4ae
docs: explain why the path placeholder is there
2021-07-09 09:45:13 +02:00
bc958e2ff0
refactor: use Arcs to pass schemas around
2021-07-09 09:45:12 +02:00
09e611deb7
refactor: lift query schema generation up to caller
...
Do no longer scan chunks during query planning to determine the schema
(except for the lifetime jobs where we have a good reason to do so).
Instead pass the schema down to from whoever is triggering the query.
For real SQL queries, we then just use the the table-wide schemas
introduced in #1913 .
Apart from avoiding schema merges we now also don't crash any longer
when no chunks are left in the table (aka columns are present but all
rows are gone).
Fixes #1768 .
Fixes #1884 .
2021-07-09 09:24:21 +02:00
6ac1420335
test: fix out dir for query tests
2021-07-09 09:16:28 +02:00
6eb52f173e
Merge pull request #1941 from influxdata/pd-remove-mutable-minimum-age
...
refactor: remove unused mutable_minimum_age_seconds lifecycle setting
2021-07-08 23:58:51 +00:00
e41fd2a821
refactor: remove unused mutable_minimum_age_seconds lifecycle setting
...
Closes #1878
2021-07-08 18:34:01 -04:00
078d5f7cfb
Merge pull request #1940 from influxdata/cn/chunk-times
...
feat: Record first and last write times on MUB chunks
2021-07-08 21:06:17 +00:00
c37053ad46
Merge branch 'main' into cn/chunk-times
2021-07-08 20:58:54 +00:00
299269a161
Merge pull request #1879 from influxdata/cn/kafka-read-metrics-and-e2e-tests
...
feat: Collect metrics on failed ingest from Kafka
2021-07-08 20:46:57 +00:00
a2726c7e92
Merge branch 'main' into cn/kafka-read-metrics-and-e2e-tests
2021-07-08 20:40:19 +00:00
b9a6a11b34
feat: Record time of first/last write on MBChunk
2021-07-08 16:39:35 -04:00
22495dd355
fix: Take a TableBatch in the MBChunk constructor
...
Thus ensuring all MBChunks will have data in them.
2021-07-08 16:39:35 -04:00
57501e7f59
refactor: Remove a test function that's only slightly different from another test function
2021-07-08 16:39:35 -04:00
2f9cf857af
fix: MBChunk.write_columns doesn't need to be pub
2021-07-08 16:39:35 -04:00
548c64539e
fix: Wrap lines at 100 chars
2021-07-08 16:39:33 -04:00
74c0a6cb00
fix: Arrange use statements so rustfmt can manage their order
2021-07-08 16:39:02 -04:00
9961cf8008
Merge pull request #1902 from influxdata/ntran/avoid_sort_in_scan
...
feat: store sort_key in the chunk schema after the chunk is sorted
2021-07-08 20:29:38 +00:00
c8126784a8
Merge branch 'main' into ntran/avoid_sort_in_scan
2021-07-08 20:22:18 +00:00
680394b50b
refactor: run fmt
2021-07-08 16:21:42 -04:00
72928aab3d
refactor: Move ChunkLifecycleAction to the data_types crate ( #1939 )
2021-07-08 20:18:33 +00:00
c5733ab4a7
refactor: remove redudant code
2021-07-08 16:11:42 -04:00
6738cb272f
refactor: remove duplicate test
2021-07-08 15:59:25 -04:00
da6249a4df
fix: address reviewers' comments and also fixe a bug they discovered
2021-07-08 15:54:54 -04:00
dd3eff7748
refactor: Always use `row_count` for count of rows in system.* tables ( #1937 )
2021-07-08 19:28:11 +00:00
8b9b369189
chore: Update DataFusion again ( #1930 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-08 15:14:45 +00:00
c6bf0a26f4
feat: Add metrics for when ingesting from the write buffer fails
...
So that we have some way of figuring out what might be going on.
2021-07-08 09:57:51 -04:00
9c722959fb
Merge pull request #1752 from influxdata/cn/kafka-read
...
feat: reading entries from Kafka
2021-07-08 13:39:16 +00:00
dd6303e85d
test: Make test data conform to Kafka partitioning assumptions
2021-07-08 09:31:52 -04:00
80e1dcafe0
feat: Support reading from all Kafka partitions
...
When reading from the Kafka write buffer, subscribe to all partitions in
a topic and start from the smallest offset available, instead of
assuming there will only be 1 partition per topic.
2021-07-08 09:30:59 -04:00
c90ef7b14b
fix: Create one consumer group per server+database
...
This hasn't caused any problems for me yet, but seemed like a good idea
because we want to be sure we don't get any of Kafka's consumer
rebalancing if we have multiple partitions.
2021-07-08 09:28:34 -04:00
e5168936f5
feat: Better error messages through to gRPC API + e2e Kafka Read tests
2021-07-08 09:28:34 -04:00
c53ae41d57
fix: Remove unneeded Option from the reading mock
2021-07-08 09:28:34 -04:00
854c28c41a
feat: Stream messages from Kafka into the database
2021-07-08 09:28:34 -04:00
ee500f5bda
feat: Support configuring a write buffer for writing OR reading
2021-07-08 09:28:34 -04:00
63d26f6f3f
refactor: Rename KafkaBuffer to KafkaBufferProducer
2021-07-08 09:28:34 -04:00
e5de73133c
feat: Change write buffer connection rule to take either Writing or Reading connection info
...
A database on one IOx server can, exclusively:
- Not interact with Kafka at all
- Send writes to Kafka
- Read writes from Kafka
Notably, a database on a particular server will never write *and* read from Kafka at the same time.
2021-07-08 09:28:34 -04:00
fd4bcc2fa5
refactor: Rename the WriteBuffer trait to be WriteBufferWriting
2021-07-08 09:28:34 -04:00
83e50cfba4
refactor: Rename field to not contain the type
2021-07-08 09:28:34 -04:00
3a329d3705
Merge pull request #1931 from influxdata/crepererum/str_arcs
...
refactor: convert some table name and part. key String to Arcs
2021-07-08 13:28:04 +00:00
69e4786fc7
Merge branch 'main' into crepererum/str_arcs
2021-07-08 13:20:49 +00:00
18893e76e0
refactor: convert some table name and part. key String to Arcs
...
This has the (somewhat nice) side effect that it shrinks the in-mem
catalog a bit as well because nw `ParquetChunk` is a bit smaller making
the chunk stage enum smaller as well.
2021-07-08 14:34:28 +02:00
4c05d7ccfd
Merge pull request #1928 from influxdata/er/refactor/logging
...
refactor: log information for compaction results
2021-07-08 11:57:11 +00:00
7ff8ae4ce5
refactor: tidy up sort key rep
2021-07-08 12:48:41 +01:00
f811bf1e5e
refactor: log compaction activity
2021-07-08 12:48:41 +01:00
86372c8818
refactor: remove unused method
2021-07-08 12:48:41 +01:00
33bc85ad18
feat: Infrastructure for persistence ( #1925 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-08 11:14:38 +00:00
Andrew Lamb
Marco Neumann
kodiakhq[bot]
Paul Dix
Carol (Nichols || Goulding)
Nga Tran
Edd Robinson