Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
6,692 Commits
189 Branches
453 Tags
379 MiB
Commit Graph

6692 Commits (b326b62b44ddd09152c2c69d46f4c9f774ecda7f)

Author SHA1 Message Date
Marco Neumann b326b62b44
feat: buffer writes when writing to RSKafka (#3520) 2022-02-01 10:07:52 +00:00
kodiakhq[bot] bf6691e56f
Merge pull request #3544 from influxdata/cn/persist 
feat: Ingester persistence
 2022-01-31 19:01:21 +00:00
kodiakhq[bot] 8bef2c105c
Merge branch 'main' into cn/persist 2022-01-31 18:50:45 +00:00
Andrew Lamb 7b96a37165
chore: Update datafusion (#3586) 
* chore: update DataFusion to f849968057ddddccc9aa19915ef3ea56bf14d80d

* fix: reduce overhead of creating physical expressions

* chore: use MemTrackingMetrics

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-31 18:15:28 +00:00
Carol (Nichols || Goulding) 4006dc14b3
fix: Correct typo in function name 2022-01-31 10:48:30 -05:00
Carol (Nichols || Goulding) 749989a937
refactor: Simplify type, eliminating empty vec creation 
If there aren't any record batches, there isn't any metadata, and vice
versa. Make this relationship clearer by putting the Option around both
the vec of recordbatches and the metadata.
 2022-01-31 10:48:30 -05:00
Carol (Nichols || Goulding) 093d5acfd4
fix: Unify temporary multiple definitions of IoxMetadata 2022-01-31 10:48:29 -05:00
Carol (Nichols || Goulding) 8f81ce5501
refactor: Share parquet_file::storage code between new and old metadata 2022-01-31 10:36:33 -05:00
Carol (Nichols || Goulding) bf89162fa5
refactor: Move IoxMetadata to parquet_file 2022-01-31 10:36:33 -05:00
Carol (Nichols || Goulding) 0f72a881ef
refactor: Rename Rust struct parquet_file::IoxMetadata to be IoxMetadataOld 2022-01-31 10:36:33 -05:00
Carol (Nichols || Goulding) 1b298bb5bd
refactor: Alias the old proto definitions to make clearer the new ones coming in 2022-01-31 10:36:33 -05:00
Carol (Nichols || Goulding) dd9620da0c
feat: Create a new proto definition for the new design's IoxMetadata 2022-01-31 10:36:32 -05:00
Carol (Nichols || Goulding) 81647f253c
feat: Use IoxMetadata and a list of RecordBatches 2022-01-31 10:36:32 -05:00
Carol (Nichols || Goulding) fef968f75c
fix: Remove catalog insertion; will be handled elsewhere 2022-01-31 10:36:32 -05:00
Carol (Nichols || Goulding) 8b47ad6885
test: Add more tests 2022-01-31 10:36:32 -05:00
Carol (Nichols || Goulding) d413157b99
feat: Extract a fn for creating the parquet file paths and test it 2022-01-31 10:36:32 -05:00
Carol (Nichols || Goulding) 5e0e0d8aa7
feat: Write parquet to object storage in a similar way as parquet_file::Storage 2022-01-31 10:36:32 -05:00
Carol (Nichols || Goulding) ea18c71e6d
feat: Create an object store path for a new parquet file 2022-01-31 10:36:32 -05:00
Carol (Nichols || Goulding) c633c9bc5c
feat: Wire object store into ingester persistence 2022-01-31 10:36:30 -05:00
Nga Tran ac247e4de5
feat: update catalog after persistence (#3581) 
* feat: update catalog after persistence

* test: add a negative test for the update catalog

* chore: add IDs into the messages

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* chore: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* refactor: address review comments

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-31 15:23:16 +00:00
Marco Neumann 74c251febb
feat: allow IOx catalog to setup itself (no SQLx CLI required) (#3584) 
* feat: allow IOx catalog to setup itself (no SQLx CLI required)

* refactor: use SQLx macro instead of hand-rolled build script
 2022-01-31 15:07:38 +00:00
Marco Neumann c50fc8764d
feat: basic non-panic HTTP/gRPC interface for ingester (#3583) 
Don't panic when K8s requests a health status or someone requests a
non-found HTTP route; or when we just TRY to start up the gRPC service.
 2022-01-31 11:13:14 +00:00
Andrew Lamb 62e8b41509
refactor: Start consolidating test scenarios in query_tests (#3580) 
* refactor: Start consolidating test scenarios in query_tests

* fix: reference

* fix: fmt
 2022-01-28 21:12:57 +00:00
Andrew Lamb 36642eb71d
test: add end to end tests that query missing tags (#3563) 
* test: add end to end tests that query missing tags

* fix: add github reference

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-28 18:42:57 +00:00
Marco Neumann 3659a7f799
refactor: clean up ingester CLI (#3569) 
- use same args/envs names as router2 does
- kafka => write buffer
- add long forms to all CLI args so we don't have to pass positional
  arguments

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-28 17:58:16 +00:00
Luke Bond 011b297f28
feat: more benchmarks of router2 (#3575) 2022-01-28 17:44:10 +00:00
Raphael Taylor-Davies 4101d16f71
chore: feature flag consistency (#3574) 
* chore: feature flag consistency

* chore: add aarch64-apple-darwin to hakari

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-28 16:38:59 +00:00
kodiakhq[bot] 03c5400f44
Merge pull request #3572 from influxdata/dom/update-rust 
build: use rust 1.58.1
 2022-01-28 16:06:06 +00:00
kodiakhq[bot] 0b04c1d74e
Merge branch 'main' into dom/update-rust 2022-01-28 15:55:20 +00:00
Dom Dwyer 5438a4afac build: use rust 1.58.1 
Security bugfix release, see:
    https://groups.google.com/g/rustlang-security-announcements/c/R1fZFDhnJVQ?pli=1
 2022-01-28 14:43:22 +00:00
Nga Tran 8735ede74f
feat: IoxMetadata for parquet file (#3547) 
* feat: IoxMetadata for parquet file

* fix: typos

* refactor: address review comments

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-28 14:41:59 +00:00
Raphael Taylor-Davies 442d63e65b
feat: catalog timestamp pruning (#3571) 
* feat: catalog timestamp pruning

* chore: test
 2022-01-28 13:45:13 +00:00
kodiakhq[bot] 7bbc4fa66c
Merge pull request #3568 from influxdata/dom/dml-trait 
refactor: decouple error types for DmlHandler
 2022-01-28 11:57:53 +00:00
kodiakhq[bot] 6ece553e68
Merge branch 'main' into dom/dml-trait 2022-01-28 11:48:51 +00:00
Raphael Taylor-Davies d8685888c8
fix: chunk ordering (#3560) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-28 11:36:08 +00:00
Dom Dwyer b38deaa721 refactor: decouple error types for DmlHandler 
Allows the DmlHandler to return different types for each method.

This enables a DmlHandler implementation decorating an inner handler to
return the inner handler's error directly, avoiding any "wrapper"
errors.
 2022-01-28 11:01:06 +00:00
Dom 1669acc0c2
build: update tokio (#3566) 
Release notes:
    https://github.com/tokio-rs/tokio/releases/tag/tokio-1.16.0
 2022-01-28 10:36:12 +00:00
Marco Neumann a22ca7c3d7
fix: router2 writer buffer topic (#3555) 
- Kafka does not support `_` in topic names, but `-` works, so let's
  change the default
- Expose topic config via CLI/env
 2022-01-28 10:10:04 +00:00
Dom 32d7c4cbfe
refactor: remove InfluxColumnType::IOx (#3565) 
* refactor: remove InfluxColumnType::IOx

Remove unused column variant - see #3554 for context.

* refactor: reserve SEMANTIC_TYPE_IOX name in proto

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-27 21:15:36 +00:00
Andrew Lamb b486258dfb
chore: run cargo update (#3559) 
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-27 21:05:31 +00:00
Dom 9201023ea4
feat: schema validation for MutableBuffer instances (#3554) 
* refactor: Debug bounds on Catalog trait

* feat: validate MutableBatch schema

Changes the schema validation code to validate MutableBatch instances
(coming from a pre-parsed LP write, and non-LP-based writes) instead of
parsed LP lines.

* refactor: Send bound on boxed errors

* refactor: clippy

Allow assert_eq!(bool, bool) for readability.

* refactor: no PartialEq<MB Column> for ColumnSchema

Remove the PartialEq<mutable_buffer::Column> for ColumnSchema - it's
definitely more readable as a method call.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-27 20:55:18 +00:00
Andrew Lamb f24ce03754
fix: provide correct environment variable to change log filter (#3561) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-27 20:45:06 +00:00
Edd Robinson 87af662d3b
refactor: include comments in test cases (#3550) 2022-01-27 20:35:14 +00:00
Luke Bond 4a96e52290
feat: router2 sharder benchmarking (#3558) 
* feat: benchmarking the router2 sharder

* chore: added throughput to sharder benchmarks; vary num buckets
 2022-01-27 18:09:16 +00:00
Raphael Taylor-Davies 5efc42494c
feat: add chunk order to chunk columns table (#3556) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-27 17:14:28 +00:00
Raphael Taylor-Davies d1d45fe818
feat: columnar predicate pruning across `Chunks` (#3553) 
* feat: columnar predicate pruning

* fix: doc

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-27 17:02:46 +00:00
Nga Tran fb33a88dc8
test: Delete application during Ingester's compaction (#3542) 
* test: Delete application during Ingester's compaction

* fix: typos

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

* chore: remove comments

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-27 16:53:17 +00:00
Andrew Lamb 2062267d0f
chore: Update hashbrown (#3551) 
* chore: Update hashbrown

* fix: hakari

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-27 15:34:10 +00:00
Edd Robinson 5befa7922b
refactor: move chunk metrics to module (#3548) 
* refactor: move chunk metrics into module

* refactor: use Metrics as internal name

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-27 15:24:31 +00:00
Dom ce568ab447
build(iox_catalog): remove unused dependencies (#3552) 
* build: don't pull in all of tokio

We already specify the tokio features we need so "full" (all features)
is not necessary.

* build: remove chrono dependency

Appears unused.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-27 15:14:42 +00:00