aa7a3a7721
fix: ignore fields when considering tag predicates ( #6212 )
...
* fix: ignore fields when considering tag predicates
* chore: update test to not use time column in predicate
* chore: update with review feedback
* chore: update tests to avoid fields refs in RPC preds
This is more like what would be coming off the wire from
Influx RPC.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-28 15:16:55 +00:00
d077222897
refactor: avoid `tokio::spawn` for cache requests ( #6229 )
...
* refactor: avoid `tokio::spawn` for cache requests
During the happy-path, we don't need any tokio task to drive a cache
loader requests because the future issuing the request just act as a
driver. If that future is cancelled, we place the cache request in an
extra task. This avoid latencies due to task overhead and (task) context
switches for most requests. This may remove a millisecond or two from
latency but also makes the whole thing easier to analyze/profile because
we don't spawn a truckload of tasks.
This trick was borrowed from rskafka.
* refactor: split up code
2022-11-28 12:52:43 +00:00
443ec49f24
feat: query exec tracing spans
...
Implement a QueryExec decorator that emits named tracing spans covering
the inner delegate's query_exec() execution.
Captures the result, emitting the error string in the span on failure.
2022-11-28 13:29:18 +01:00
a54326d1ae
refactor: rename Response -> QueryResponse
...
Both more descriptive, and less conflict-y! This seems like a more
sensible name for a system with many Response's.
2022-11-28 13:29:18 +01:00
75fcbc1ae0
chore(deps): Bump zstd-safe from 5.0.2+zstd.1.5.2 to 6.0.2+zstd.1.5.2 ( #6234 )
...
* chore(deps): Bump zstd-safe from 5.0.2+zstd.1.5.2 to 6.0.2+zstd.1.5.2
Bumps [zstd-safe](https://github.com/gyscos/zstd-rs ) from 5.0.2+zstd.1.5.2 to 6.0.2+zstd.1.5.2.
- [Release notes](https://github.com/gyscos/zstd-rs/releases )
- [Commits](https://github.com/gyscos/zstd-rs/commits )
---
updated-dependencies:
- dependency-name: zstd-safe
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-28 11:06:39 +00:00
721dfb9960
chore(deps): Bump insta from 1.21.1 to 1.21.2 ( #6235 )
...
Bumps [insta](https://github.com/mitsuhiko/insta ) from 1.21.1 to 1.21.2.
- [Release notes](https://github.com/mitsuhiko/insta/releases )
- [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md )
- [Commits](https://github.com/mitsuhiko/insta/compare/1.21.1...1.21.2 )
---
updated-dependencies:
- dependency-name: insta
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-11-28 09:42:03 +00:00
9d125b9700
Merge pull request #6233 from influxdata/dom/ingester2-grpc
...
feat(ingester2): gRPC methods & type-erased init
2022-11-25 13:16:07 +00:00
a0ab78298f
feat(ingester2): gRPC methods & type-erased init
...
This commit implements the gRPC direct-write RPC interface (largely
copied from the ingester crate), and adds a much improved RPC query
handler.
Compared to the ingester crate, the query API is now split into two
defined halves - the API handler side, and types necessary to support it
(server/grpc/query.rs) and the Ingester query execution side (a stub in
query/exec.rs). These two halves maintain a separation of concerns, and
are interfaced by an abstract QueryExec trait (in query/trait.rs).
I also added the catalog RPC interface as it is currently exposed on the
ingester, though I am unsure if it is used by anything.
This commit also introduces the "init" module, and the
IngesterRpcInterface trait within it. This trait forms the public
ingester2 crate API, defining the complete set of methods external
crates can expect to utilise in a stable, unchanging and decoupled way.
The IngesterRpcInterface trait also serves as a method of type-erasure
on the underlying handler implementations, avoiding the need to
expose/pub the types, abstractions, and internal implementation details
of the ingester to external crates.
2022-11-25 12:40:01 +01:00
9e4bbd0292
Merge pull request #6228 from influxdata/dom/ingester2
...
feat(ingester): ingester2 init
2022-11-25 10:13:11 +00:00
5843e30ba4
Merge branch 'main' into dom/ingester2
2022-11-25 10:07:10 +00:00
8e33c33721
chore(deps): Bump zstd-safe from 5.0.2+zstd.1.5.2 to 6.0.2+zstd.1.5.2 ( #6224 )
...
* chore(deps): Bump zstd-safe from 5.0.2+zstd.1.5.2 to 6.0.2+zstd.1.5.2
Bumps [zstd-safe](https://github.com/gyscos/zstd-rs ) from 5.0.2+zstd.1.5.2 to 6.0.2+zstd.1.5.2.
- [Release notes](https://github.com/gyscos/zstd-rs/releases )
- [Commits](https://github.com/gyscos/zstd-rs/commits )
---
updated-dependencies:
- dependency-name: zstd-safe
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-25 10:06:56 +00:00
74cf99eac7
chore(deps): Bump zstd from 0.11.2+zstd.1.5.2 to 0.12.0+zstd.1.5.2 ( #6225 )
...
* chore(deps): Bump zstd from 0.11.2+zstd.1.5.2 to 0.12.0+zstd.1.5.2
Bumps [zstd](https://github.com/gyscos/zstd-rs ) from 0.11.2+zstd.1.5.2 to 0.12.0+zstd.1.5.2.
- [Release notes](https://github.com/gyscos/zstd-rs/releases )
- [Commits](https://github.com/gyscos/zstd-rs/commits )
---
updated-dependencies:
- dependency-name: zstd
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-25 10:00:28 +00:00
faf8f33b31
chore(deps): Bump zstd-sys from 2.0.1+zstd.1.5.2 to 2.0.3+zstd.1.5.2 ( #6230 )
...
Bumps [zstd-sys](https://github.com/gyscos/zstd-rs ) from 2.0.1+zstd.1.5.2 to 2.0.3+zstd.1.5.2.
- [Release notes](https://github.com/gyscos/zstd-rs/releases )
- [Commits](https://github.com/gyscos/zstd-rs/commits )
---
updated-dependencies:
- dependency-name: zstd-sys
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-25 09:53:48 +00:00
898085300e
chore(deps): Bump flate2 from 1.0.24 to 1.0.25 ( #6231 )
...
Bumps [flate2](https://github.com/rust-lang/flate2-rs ) from 1.0.24 to 1.0.25.
- [Release notes](https://github.com/rust-lang/flate2-rs/releases )
- [Commits](https://github.com/rust-lang/flate2-rs/commits/1.0.25 )
---
updated-dependencies:
- dependency-name: flate2
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-25 09:47:24 +00:00
522ae6c2a3
docs: document possible FSM panic
...
Document a possible panic if the data within a partition FSM cannot be
converted to an Arrow RecordBatch.
2022-11-25 10:46:44 +01:00
ca1e602a0e
Merge pull request #6232 from influxdata/dependabot/cargo/clap-4.0.27
...
chore(deps): Bump clap from 4.0.26 to 4.0.27
2022-11-25 09:40:24 +00:00
f5fbcdae38
Merge branch 'main' into dom/ingester2
2022-11-25 09:37:05 +00:00
657819b8a0
chore: Run cargo hakari tasks
2022-11-25 01:07:01 +00:00
ac6403d347
chore(deps): Bump clap from 4.0.26 to 4.0.27
...
Bumps [clap](https://github.com/clap-rs/clap ) from 4.0.26 to 4.0.27.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v4.0.26...v4.0.27 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-11-25 01:06:03 +00:00
2604eb935b
chore: identifier parser handles preceding whitespace ( #6223 )
...
* fix: `identifier` parser consumes preceding whitespace
* chore: Update module docs
* feat: Add function to identify when an identifier requires quotes
* feat: Add ability to deref OneOrMore to its vector representation
This feature was used as part the InfluxQL logical planner in IOx
* chore: Feedback, prefer slice `[T]` rather than `Vec<T>`
2022-11-25 00:09:26 +00:00
44eeab7e2b
chore: Run cargo hakari tasks
2022-11-24 14:51:21 +00:00
a66fc0b645
feat(ingester): ingester2 init
...
Adds an ingester2 crate to hold the MVP of the Kafkaless project.
This was necessary due to the tight coupling of the ingester internals
with tests in external crates, and eases the parallel development of two
version of the ingester.
This commit contains various changes from the "ingester" crate, mostly
removing the concept/references to a "shard" or "ShardId" where
possible.
This commit does not copy over all of the "ingester" crate - only those
components that are definitely needed. I will drag across more as
functionality is implemented.
2022-11-24 15:34:02 +01:00
92719f5b2b
Merge pull request #6181 from influxdata/pauldix/wal
...
feat: Add a WAL crate for ingester2 to use
2022-11-24 09:21:55 +00:00
b86eb763b1
fix: Tell buf to fetch to a depth of 100 commits when looking for breaking changes
...
Per <https://docs.buf.build/reference/inputs#other-options >, this is by
default 50 (and I have 51 commits on this PR 😰 )
100 commits should be enough for anyone.
2022-11-23 17:24:50 -05:00
727dd864d2
test: Add tests exercising writing of delete and persist ops
2022-11-23 17:15:57 -05:00
47173e53f0
test: Rotating without writing anything is fine
2022-11-23 17:15:57 -05:00
fe837245e7
test: Add some WAL reader unit-ey tests
2022-11-23 17:15:56 -05:00
be445b5057
fix: Remove API to get multiple SegmentEntry records at once; it's not quite right
...
See #6219 for batching reads.
2022-11-23 17:07:49 -05:00
edd606aa3b
feat: Serialize using protobuf instead of json
2022-11-23 17:07:49 -05:00
a85abadfd5
fix: Define and use a sequence number represented as u64 instead
2022-11-23 17:07:49 -05:00
d7ffa65918
feat: Add an id method to OpenSegmentFileWriter
2022-11-23 17:07:49 -05:00
35576b26b5
refactor: Rename placeholder fnamex to build_segment_path
2022-11-23 17:07:49 -05:00
40dd575de9
feat: Add segment_id metadata to WriteSummary
2022-11-23 17:07:48 -05:00
8618264fe6
fix: Remove unused created_at field
2022-11-23 17:07:48 -05:00
1ba0f193a9
fix: fsync parent directory before returning that new segment file was created
2022-11-23 17:07:48 -05:00
d5b439732d
feat: Implement delete WAL file
2022-11-23 17:07:48 -05:00
a2c25f5191
docs: Improve some descriptions
2022-11-23 17:07:48 -05:00
cdec449963
fix: Return single SequencedWalOps for now, will batch later
2022-11-23 17:07:48 -05:00
db9a43f1b3
fix: Create types for 3 different tasks of a WAL
2022-11-23 17:07:48 -05:00
de9c02caad
fix: Use bytes_written rather than file metadata
2022-11-23 17:07:48 -05:00
1354136b93
fix: fsync and panic if it fails
2022-11-23 17:07:47 -05:00
ee99b93d09
fix: Make checksum private
2022-11-23 17:07:47 -05:00
be2207bcd3
refactor: Rename to clarify closed segment types
2022-11-23 17:07:47 -05:00
4335edc6d1
refactor: Rename to clarify open segment types
2022-11-23 17:07:47 -05:00
70b12a8112
fix: Clarify and simplify the public API
2022-11-23 17:07:47 -05:00
1f5fd43829
fix: cross ps and dot qs
2022-11-23 17:07:47 -05:00
0ae40ba025
fix: move writing to blocking and wire up some things
2022-11-23 17:07:47 -05:00
2e6f269840
refactor: move blocking code to new file
2022-11-23 17:07:47 -05:00
49633842a5
test: Rotate and reading of closed segments
2022-11-23 17:07:46 -05:00
58d9ada90c
feat: Stream the reading
2022-11-23 17:07:46 -05:00
Christopher M. Wolff
Marco Neumann
Dom Dwyer
dependabot[bot]
kodiakhq[bot]
CircleCI[bot]
Stuart Carnie
Carol (Nichols || Goulding)
Jake Goulding