2bb6db3f37
fix: ensure ingester state tracked in querier cache is always in-sync ( #6512 )
...
Fixes #6510 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-10 12:00:05 +00:00
d0f52c89ae
chore(deps): Bump arrow-ord from 29.0.0 to 30.0.1 ( #6543 )
...
* chore(deps): Bump arrow-ord from 29.0.0 to 30.0.1
Bumps [arrow-ord](https://github.com/apache/arrow-rs ) from 29.0.0 to 30.0.1.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/compare/29.0.0...30.0.1 )
---
updated-dependencies:
- dependency-name: arrow-ord
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-10 10:09:11 +00:00
3d86834eec
chore(deps): Bump arrow-string from 29.0.0 to 30.0.1 ( #6544 )
...
* chore(deps): Bump arrow-string from 29.0.0 to 30.0.1
Bumps [arrow-string](https://github.com/apache/arrow-rs ) from 29.0.0 to 30.0.1.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/compare/29.0.0...30.0.1 )
---
updated-dependencies:
- dependency-name: arrow-string
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-10 10:02:08 +00:00
c68049c37a
chore(deps): Bump regex from 1.7.0 to 1.7.1 ( #6546 )
...
Bumps [regex](https://github.com/rust-lang/regex ) from 1.7.0 to 1.7.1.
- [Release notes](https://github.com/rust-lang/regex/releases )
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/regex/compare/1.7.0...1.7.1 )
---
updated-dependencies:
- dependency-name: regex
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-10 09:55:41 +00:00
b49cc2e35e
chore(deps): Bump tokio from 1.24.0 to 1.24.1 ( #6545 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.24.0 to 1.24.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.24.0...tokio-1.24.1 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-10 09:48:44 +00:00
cb86e6d602
Merge pull request #6547 from influxdata/dependabot/cargo/assert_cmd-2.0.8
...
chore(deps): Bump assert_cmd from 2.0.7 to 2.0.8
2023-01-10 09:41:24 +00:00
09746c3aea
chore(deps): Bump assert_cmd from 2.0.7 to 2.0.8
...
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd ) from 2.0.7 to 2.0.8.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases )
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.7...v2.0.8 )
---
updated-dependencies:
- dependency-name: assert_cmd
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-01-10 01:06:02 +00:00
828992c9c5
feat: Ingest replica skeleton ( #6529 )
...
* feat: Update replication.proto
* Remove the PartitionId in the replicate request as a single replicate request can have the data for many partitions.
* Add namespace_id and table_id to persist complete request to make data easier to lookup in buffer.
* feat: Initial ingest_replica skeleton
A bunch of copy pasta here from ingester2, but this takes out a ton of stuff that isn't used in replicas.
Also lays the groundwork for the simpler buffer structure to keep the data and a basic cache for catalog information that will be required.
* feat: update replication.proto GetPartitionBufferResponse
* chore: PR cleanup
* chore: PR cleanup
2023-01-09 16:53:49 +00:00
f63157c04c
Merge pull request #6539 from influxdata/dom/remove-atty
...
refactor: remove atty & bump tokio
2023-01-09 15:47:37 +00:00
7f7f0ca170
chore: bump tokio 1.22 -> 1.24
...
Fixes a bug on windows:
https://github.com/tokio-rs/tokio/security/advisories/GHSA-7rrj-xr53-82p7
2023-01-09 16:25:35 +01:00
6b090fae53
ci: whitelist RUSTSEC-2021-0145
...
atty is used by criterion and mockito, both dev-only dependencies.
2023-01-09 16:16:04 +01:00
fa6b07f3a0
refactor(trogging): remove atty dependency
...
Replace atty with IsTerminal due to:
https://rustsec.org/advisories/RUSTSEC-2021-0145
2023-01-09 16:14:11 +01:00
d1fb98fdd7
Merge pull request #6537 from influxdata/dom/track-seq-num
...
feat(ingester2): track buffered op sequence numbers
2023-01-09 14:52:17 +00:00
d666bf6d22
Merge branch 'main' into dom/track-seq-num
2023-01-09 13:47:40 +00:00
de511df63c
Merge pull request #6538 from influxdata/dom/remove-unused-queryrunner
...
refactor: remove unused QueryExec impl
2023-01-09 13:41:37 +00:00
aab4f6c651
refactor: remove unused QueryExec impl
...
This is completely unused and left over from the initial skeleton.
2023-01-09 14:28:16 +01:00
1f509f47b1
refactor: log number of writes in persist batch
...
Include the number of DML operations applied to the persisted buffer
in the "persisted partition" message.
Partly because I'm intrigued / it's useful information, and partly to
ensure LLVM doesn't get snazzy and dead-code the sequence number
tracking because it was never read.
2023-01-09 13:31:42 +01:00
ca2b8afbb1
refactor(ingester2): track buffer sequence numbers
...
Changes the ingester2 buffer FSM to track the sequence numbers that have
been applied to it.
This is a pre-requisite for replication & correct WAL segment dropping.
2023-01-09 13:27:18 +01:00
0916529cfb
docs: remove outdated monotonicity comment
...
Previously the ingester(1) required ordered writes to be applied, this
requirement has been relaxed, and the asserts (previously) removed in
ingester2.
2023-01-09 13:24:51 +01:00
13f2f36a9c
Merge pull request #6513 from influxdata/dom/seqnum-set
...
feat(data_types): SequenceNumberSet
2023-01-09 10:45:53 +00:00
c313db8dfe
Merge branch 'main' into dom/seqnum-set
2023-01-09 10:33:53 +00:00
f89cc2bd29
chore(deps): Bump arrow-ord from 29.0.0 to 30.0.1 ( #6535 )
...
* chore(deps): Bump arrow-ord from 29.0.0 to 30.0.1
Bumps [arrow-ord](https://github.com/apache/arrow-rs ) from 29.0.0 to 30.0.1.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/compare/29.0.0...30.0.1 )
---
updated-dependencies:
- dependency-name: arrow-ord
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-09 09:53:52 +00:00
e3157c3338
chore(deps): Bump arrow-string from 29.0.0 to 30.0.1 ( #6532 )
...
* chore(deps): Bump arrow-string from 29.0.0 to 30.0.1
Bumps [arrow-string](https://github.com/apache/arrow-rs ) from 29.0.0 to 30.0.1.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/compare/29.0.0...30.0.1 )
---
updated-dependencies:
- dependency-name: arrow-string
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-01-09 09:47:15 +00:00
3fac475b63
chore(deps): Bump base64 from 0.20.0 to 0.21.0 ( #6530 )
...
* chore(deps): Bump base64 from 0.20.0 to 0.21.0
Bumps [base64](https://github.com/marshallpierce/rust-base64 ) from 0.20.0 to 0.21.0.
- [Release notes](https://github.com/marshallpierce/rust-base64/releases )
- [Changelog](https://github.com/marshallpierce/rust-base64/blob/master/RELEASE-NOTES.md )
- [Commits](https://github.com/marshallpierce/rust-base64/compare/v0.20.0...v0.21.0 )
---
updated-dependencies:
- dependency-name: base64
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
* fix: deprecations
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-09 09:39:40 +00:00
dfdc964f64
chore(deps): Bump insta from 1.24.1 to 1.26.0 ( #6534 )
...
* chore(deps): Bump insta from 1.24.1 to 1.26.0
Bumps [insta](https://github.com/mitsuhiko/insta ) from 1.24.1 to 1.26.0.
- [Release notes](https://github.com/mitsuhiko/insta/releases )
- [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md )
- [Commits](https://github.com/mitsuhiko/insta/compare/1.24.1...1.26.0 )
---
updated-dependencies:
- dependency-name: insta
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-09 08:02:08 +00:00
e31c84a794
chore(deps): Bump async-trait from 0.1.60 to 0.1.61 ( #6533 )
...
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.60 to 0.1.61.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.60...0.1.61 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-09 07:44:35 +00:00
29df5d7fcb
fix: create statistics for nulled columns in RecordBatchExec ( #6527 )
2023-01-09 07:37:10 +00:00
b20226797a
fix: make trigger midification in different file ( #6526 )
2023-01-06 20:34:48 +00:00
4031ea1c10
feat: integrate new way to get partition candidates for hot compaction ( #6525 )
...
* feat: integrate new way to get partition candidates for hot compaction
* chore: rename
2023-01-06 18:51:52 +00:00
b856edf826
feat: function to get parttion candidates from partition table ( #6519 )
...
* feat: function to get parttion candidates from partition table
* chore: cleanup
* fix: make new_file_at the same value as created_at
* chore: cleanup
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-06 16:20:45 +00:00
c87e1c8fe2
Merge pull request #6523 from influxdata/alamb-patch-1
...
docs: Update docstring on Schema
2023-01-06 11:23:21 +00:00
9a0db0ec30
Merge branch 'main' into alamb-patch-1
2023-01-06 11:16:52 +00:00
f0141bcb41
fix: fmt
2023-01-06 06:13:40 -05:00
544f5c5bff
fix: Update schema/src/lib.rs
...
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2023-01-06 06:13:04 -05:00
e1036a0c63
refactor: cleanup schema boxing ( #6511 )
...
* refactor: cleanup Schema boxing
* chore: clippy
2023-01-06 10:57:39 +00:00
ba844ddf03
docs: Update docstring on Schema
2023-01-06 05:49:49 -05:00
2037db7f7b
refactor: decouple influxql from SchemaProvider ( #6507 )
...
* refactor: decouple influxql from SchemaProvider
* refactor: reorder arguments
* refactor: use QueryNamespaceMeta
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-06 07:36:29 +00:00
23807df7a9
feat: trigger that updates partition table when a parquet file is created ( #6514 )
...
* feat: trigger that update partition table when a parquet file is created
* chore: simplify epoch of now
2023-01-05 19:57:23 +00:00
3b672c223c
feat(data_types): SequenceNumberSet
...
Adds a space-efficient encoding of a set of SequenceNumber, backed by
roaring bitmaps.
Memory utilisation will change as the number of elements changes,
according to the underlying roaring bitmap design, but is intended to be
"relatively" cheap.
2023-01-05 18:20:48 +01:00
e1d0bcdaad
Merge pull request #6494 from influxdata/dom/replication-proto
...
feat(replication): define replication RPC API
2023-01-05 13:51:16 +00:00
69ce1b8fbf
Merge branch 'main' into dom/replication-proto
2023-01-05 13:45:00 +00:00
9f858d1dd5
Merge pull request #6509 from influxdata/dom/router-startup-offline-ingester
...
fix(router2): drive ingester node (re)-discovery
2023-01-05 13:33:49 +00:00
3f22af9183
Merge branch 'main' into dom/router-startup-offline-ingester
2023-01-05 13:26:54 +00:00
9ab86fa154
fix(router2): drive ingester node (re)-discovery
...
The tonic / tower load-balance implementation discards failed nodes,
even when using a static list - this causes nodes that fail once to
never be retried.
This doesn't happen for the last node for some reason, and leads to all
the load from one router hitting a single ingester instead of load
balancing across all ingesters.
This commit adds a hack to constantly tell the load balancer to probe
all nodes, hopefully causing them to re-discover previously failed
nodes. I don't have the time to do this properly :(
2023-01-05 14:06:29 +01:00
6f4b128285
refactor: improve "Parquet files after filtering" dbg log ( #6502 )
...
- Place IDs last because they may hit the "max line length" limit and be
truncated. The other information should NOT be truncated with it.
- Unpack IDs to integer to remove useless `ParquetFileID(...)` wrappers
in output.
- Print number of files in addition to the actual list to simplify
debugging.
2023-01-05 11:13:33 +00:00
c2f479d370
Merge branch 'main' into dom/replication-proto
2023-01-05 11:00:32 +00:00
0431ef9b71
Merge pull request #6499 from influxdata/dependabot/cargo/arrow-string-30.0.0
...
chore(deps): Bump arrow-string from 29.0.0 to 30.0.0
2023-01-05 11:00:23 +00:00
3eba1153d6
chore(deps): Bump arrow-ord from 29.0.0 to 30.0.0 ( #6501 )
...
* chore(deps): Bump arrow-ord from 29.0.0 to 30.0.0
Bumps [arrow-ord](https://github.com/apache/arrow-rs ) from 29.0.0 to 30.0.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/compare/29.0.0...30.0.0 )
---
updated-dependencies:
- dependency-name: arrow-ord
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-01-05 10:46:12 +00:00
d72835564a
Merge branch 'main' into dependabot/cargo/arrow-string-30.0.0
2023-01-05 10:40:13 +00:00
b0b15de272
Merge pull request #6504 from influxdata/dom/router-startup-offline-ingester
...
fix(router2): lazily connect to ingesters
2023-01-05 10:39:18 +00:00
Marco Neumann
dependabot[bot]
Dom
Paul Dix
Andrew Lamb
Nga Tran
kodiakhq[bot]
Andrew Lamb
Raphael Taylor-Davies