6843eee1d2
feat: Extract encoding from `RecordBatch` --> `FlightData` from flight implementations ( #6460 )
...
* feat: Extract encoding from `RecordBatch` --> `FlightData` from flight implementations
Refactor existing flight server impl
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: fixup code review comments
* fix: update for more details
* fix: Update names / types
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-04 13:36:16 +00:00
7c6ccdb6d7
fix: Use keys and values functions. Thanks clippy!
2022-12-21 14:32:35 -05:00
adc6fcfb04
feat(catalog): linearise sort key updates
...
Updating the sort key is not commutative and MUST be serialised. The
correctness of the current catalog interface relies on the caller
serialising updates globally, something it cannot reasonably assert in a
distributed system.
This change of the catalog interface pushes this responsibility to the
catalog itself where it can be effectively enforced, and allows a caller
to detect parallel updates to the sort key.
2022-12-20 12:31:00 +01:00
8478d41bcb
chore(deps): Bump paste from 1.0.10 to 1.0.11 ( #6430 )
...
Bumps [paste](https://github.com/dtolnay/paste ) from 1.0.10 to 1.0.11.
- [Release notes](https://github.com/dtolnay/paste/releases )
- [Commits](https://github.com/dtolnay/paste/compare/1.0.10...1.0.11 )
---
updated-dependencies:
- dependency-name: paste
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-19 10:31:05 +00:00
c72734473c
chore(deps): Bump async-trait from 0.1.59 to 0.1.60 ( #6433 )
...
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.59 to 0.1.60.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.59...0.1.60 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-19 10:09:23 +00:00
66c610f7b1
Merge branch 'main' into cn/ingester-persisted-file-count
2022-12-14 14:58:31 +00:00
e108a8b6c9
chore(deps): Bump paste from 1.0.9 to 1.0.10 ( #6384 )
...
Bumps [paste](https://github.com/dtolnay/paste ) from 1.0.9 to 1.0.10.
- [Release notes](https://github.com/dtolnay/paste/releases )
- [Commits](https://github.com/dtolnay/paste/compare/1.0.9...1.0.10 )
---
updated-dependencies:
- dependency-name: paste
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-13 06:03:05 +00:00
1c7f322a4e
feat: Keep track of and report number of Parquet files persisted
...
Per partition and starting over each time the ingester restarts.
Fixes #6334 .
2022-12-12 11:45:00 -05:00
2fd2d05ef6
feat: Identify each run of an ingester with a Uuid
...
And send that UUID in the Flight response for queries to that ingester
run.
Fixes #6333 .
2022-12-08 17:22:52 -05:00
1d38d400f0
chore(deps): Bump object_store from 0.5.1 to 0.5.2 ( #6339 )
...
* chore(deps): Bump object_store from 0.5.1 to 0.5.2
Bumps [object_store](https://github.com/apache/arrow-rs ) from 0.5.1 to 0.5.2.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/compare/object_store_0.5.1...object_store_0.5.2 )
---
updated-dependencies:
- dependency-name: object_store
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-06 07:53:54 +00:00
942a6100b5
fix: check schemas in `pretty_print_batches` ( #6309 )
...
* fix: check schemas in `pretty_print_batches`
I think most users of this function (and `assert_batches_eq`) assume
that all batches have the same schema. If not, `pretty_print_batches`
may either fail producing an actual table (some rows may have more or
less columns) or silently produce a table that looks "alright".
* fix: equalize schemas where it is required/desired
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-02 12:14:16 +00:00
ec2e72d223
test: simplify test executors ( #6312 )
...
Have a single global test executor w/ reasonable defaults. Also don't
require tests to join/await executor shutdowns (most tests forget this
anyways and will get a runtime warning).
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-02 11:38:18 +00:00
fc520e0c0f
refactor: Remove unecessary optimize_record_batch ( #6262 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-29 13:35:46 +00:00
9eafa9dbed
style: consistent import ordering
...
Reorder all imports in the ingester to match a consistent order:
* stdlib
* external crates
* intra-crate imports
This helps prevent merge conflicts & keeps everything tidy.
2022-11-22 14:11:10 +01:00
ee8b728c32
refactor: decouple Shard & BufferTree
...
Splits out the nested tree of namespace -> tables -> partitions
(referred to as the "buffer tree") from the Shard which previously held
the namespace map.
This allows the BufferTree to exist without a shard, or many trees to
exist within a shard, etc.
2022-11-22 14:11:10 +01:00
e4c12fa6a5
fix: slice flight response batches ( #6205 )
...
* fix: slice flight response batches
Same as #6094 but for the Apache Flight interface.
Ref https://github.com/influxdata/idpe/issues/16073 .
* refactor: use `RecordBatch::slice`
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-22 12:25:23 +00:00
04c00bbb62
chore(deps): Bump bytes from 1.2.1 to 1.3.0 ( #6199 )
...
Bumps [bytes](https://github.com/tokio-rs/bytes ) from 1.2.1 to 1.3.0.
- [Release notes](https://github.com/tokio-rs/bytes/releases )
- [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md )
- [Commits](https://github.com/tokio-rs/bytes/commits )
---
updated-dependencies:
- dependency-name: bytes
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-22 08:23:24 +00:00
097f0acb85
refactor: move SequenceNumberRange
...
Moves the SequenceNumberRange type out of "data" and into the root to be
reused outside of the data module. This construct is universally useful
across all the ingester code.
2022-11-21 16:11:55 +01:00
1938c18c50
refactor: decouple DmlSink error type
...
Allows different DmlSink implementations to return different error
types. This allows for small, concise errors that are local to the
DmlSink implementation and specific to it. This helps avoid bloated
"kitchen sink" error types.
2022-11-21 15:29:13 +01:00
64c9d87b9b
refactor: move DmlSink
...
Extracts the DmlSink trait into its own module - it is independent of
the Kafka handler and will be reused.
2022-11-21 15:02:24 +01:00
a9db7581cd
chore(deps): Bump tokio from 1.21.2 to 1.22.0 ( #6183 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.21.2 to 1.22.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.21.2...tokio-1.22.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-21 10:21:24 +00:00
85c8d16680
refactor: add a message to unreachable!()
...
Adds a message to say an impossible thing is impossible.
2022-11-18 17:33:58 +01:00
9dc32f1c16
refactor: remove names from DML init
...
Fixes conflicts introduced by #6170 .
2022-11-18 17:31:56 +01:00
59b3c793d3
Merge branch 'main' into dom/ingester-rpc-write
2022-11-18 16:21:07 +00:00
9351e01068
refactor: log dml apply errors
...
Ensures DML apply errors are recorded in the ingester logs.
2022-11-18 16:48:31 +01:00
16eed699fd
refactor: avoid needless partition key clone
...
Moves the trace! invocation to before the DmlWrite init to avoid having
to clone the partition key.
2022-11-18 16:46:14 +01:00
9751512d44
fix: Insert columns in schema in ingester tests where we have table names
2022-11-18 10:40:40 -05:00
02c3083192
fix: Remove table names from Dml operations
2022-11-18 10:40:38 -05:00
90dd9906f6
feat(ingester): rpc write endpoint
...
Adds a handler implementation of the gRPC WriteService to receive direct
RPC writes from a router.
This code is currently unused.
2022-11-18 16:36:19 +01:00
229e2adbb1
refactor: split gRPC services into modules
...
Splits the everything-grpc-in-one-file into smaller, per-service
modules.
2022-11-18 15:51:54 +01:00
49a9565240
feat: gRPC that creates namespace ( #6103 )
...
* feat: create namespace API call in router
Co-authored-by: Nga Tran <nga-tran@live.com>
* chore: treat retention as ns except in CLI
* fix: overflow in nanosecond calc
* fix: retention test after changing it from hours to ns
* chore: comment clarification in cli; better response type for error in ns API
* fix: correct some rebase mistakes
* chore: merge namespace create & create_with_retention; renamed ns create test helper fn & const
* fix: ns autocreation test was wrong after rebase
* fix: mem catalog has default 1hr retention, accidently removed in rebase
* chore: remove mem catalogs default 1hr retention; make it settable in sets & router
Co-authored-by: Luke Bond <luke.n.bond@gmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-18 13:02:12 +00:00
6f7b1e2e26
feat: reject writes that are outside the retention period ( #6148 )
...
* feat: reject writes that are outside the retention period
* feat: add retention validator into handler stack
* chore: Apply suggestions from code review
Co-authored-by: Dom <dom@itsallbroken.com>
* refactor: address review comments
* test: unit tests fot retention validation
* chore: address review comments
* test: more unit tests and integration tests
* refactor: make time inside retention period for emphemeral_mode test
* fix: 2 hours
Co-authored-by: Dom <dom@itsallbroken.com>
2022-11-17 20:55:58 +00:00
1a49fa4864
Merge branch 'main' into cn/test-refactor
2022-11-17 14:01:36 +00:00
5afe58d4d2
refactor: remove unused errors
...
These error states are no longer possible after several refactors, but
do not cause a "not used" lint because of macro magic.
2022-11-17 13:53:54 +01:00
d4715a9fde
fix: Simplify tests by using and creating more test helpers
...
The most important part of this is creating the DmlWrites in one spot.
2022-11-16 21:48:43 -05:00
4e2b68a7c5
fix: Simplify test by not actually creating a catalog namespace
...
This isn't actually needed for what this test is testing.
2022-11-16 21:06:44 -05:00
b6286767b0
fix: Validating the schema in ingester tests isn't necessary
...
The router validates schemas; schema validation shouldn't be tested in
the ingester
2022-11-16 21:05:51 -05:00
c7b9866483
feat: Have make_write_op take the table name as an argument to be more flexible
2022-11-16 21:05:46 -05:00
d0218fb025
refactor: Simplify tests by using make_write_op helper function
2022-11-16 21:00:10 -05:00
cac241b7ad
refactor: Extract shared test setup for ingester data tests
2022-11-16 21:00:10 -05:00
256ded7e00
fix: Move a NamespaceData test into its module
2022-11-16 21:00:10 -05:00
62851afc27
feat: add querier->ingester circuit breaker ( #6147 )
...
* feat: add log ingester memory pressure persist
* feat: add querier->ingester circuit breaker
Closes #4608 .
* docs: explain high-level circuit breaker
* docs: improve
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* test: add additional test assertion
* refactor: upgrade info to warning log
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2022-11-16 10:50:33 +00:00
c27d3a22d2
fix: Remove namespace argument from test helper function
2022-11-14 16:46:04 -05:00
3943faf998
fix: Remove namespace from DmlWrite and DmlDelete constructors
2022-11-14 16:46:04 -05:00
f78195f7c7
fix: Remove namespace name field from DmlWrite and DmlDelete
...
But leave the argument in their constructors for now.
Not all numbers in tests can be 42, Dom.
2022-11-14 16:46:04 -05:00
c203e8295f
test: Keep track of namespaces by ID in ingester TestContext
2022-11-14 16:46:04 -05:00
6c1e9f04ef
Merge branch 'main' into dom/deferred-table-name
2022-11-14 18:22:46 +00:00
fd898cea2a
docs: Correct grammar and update outdated comment
2022-11-14 13:21:55 -05:00
a969754819
chore(deps): Bump chrono from 0.4.22 to 0.4.23 ( #6129 )
...
* chore(deps): Bump chrono from 0.4.22 to 0.4.23
Bumps [chrono](https://github.com/chronotope/chrono ) from 0.4.22 to 0.4.23.
- [Release notes](https://github.com/chronotope/chrono/releases )
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md )
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.22...v0.4.23 )
---
updated-dependencies:
- dependency-name: chrono
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* refactor: chrono future compat
Integer->timstamp conversions should not silently panic.
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-14 13:34:09 +00:00
413b7c8f4a
refactor: use table name from catalog
...
Changes the TableData within the ingester to utilise a TableNameResolver
to fetch the TableName via the catalog on demand / in the background,
instead of using the table name sent over the write.
This change causes the ingester to perform a catalog query in the
background (or on demand) to resolve the table name. This is a
pre-requisite for removing the table name from the write wire format.
2022-11-14 11:32:22 +01:00
Andrew Lamb
Carol (Nichols || Goulding)
Dom Dwyer
dependabot[bot]
kodiakhq[bot]
Marco Neumann
Nga Tran