Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
10,616 Commits
212 Branches
453 Tags
355 MiB
Commit Graph

10616 Commits (105e3542991aef5f1654fe260f9fa53a32f622f2)

Author SHA1 Message Date
dependabot[bot] 0114e7ee50
chore(deps): Bump async-trait from 0.1.61 to 0.1.63 (#6660) 
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.61 to 0.1.63.
- [Release notes](https://github.com/dtolnay/async-trait/releases)
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.61...0.1.63)

---
updated-dependencies:
- dependency-name: async-trait
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-01-23 08:41:27 +00:00
Nga Tran 411b3db928
fix: Get shard id from a constant (topic, shard_index) to avoid error of shard_id FK violation (#6658) 
* fix: ake shard_id FK always 1

* fix: use const shard_index to read its ID

* refactor: read shard_id during compactor initiation
 2023-01-22 16:49:06 +00:00
Nga Tran 840923abab
refactor: execute compaction plan (#6654) 
* chore: address review comment of previous PR

* refactor: execute compact plan

* refactor: we will now compact all L0 and L1 files of a partition and split them as needed

* chore: comnents

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-20 22:34:50 +00:00
Christopher M. Wolff 6f39ae342e
feat: create a GapFillExec type (#6641) 
* refactor: make gap fill rule avoid aliasing

* feat: create a GapFillExec type

* refactor: remove unneeded sort node from GapFill rule

* chore: code review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-20 17:44:00 +00:00
Marco Neumann 4f1beba482
feat: filter out L2 files from compaction (#6653) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-20 15:44:13 +00:00
kodiakhq[bot] ec7c7634e4
Merge pull request #6626 from influxdata/cn/test-old-with-new 
test: Add old ingester and old parquet states into the query_tests2 framework
 2023-01-20 15:27:43 +00:00
kodiakhq[bot] 24ca1e6f8c
Merge branch 'main' into cn/test-old-with-new 2023-01-20 15:20:40 +00:00
Marco Neumann 111e582d71
feat: improve compactor2 metrics and logging (#6652) 
Closes #6647.

We can always create tickets for concrete issues/wishes or create
on-demand PRs.
 2023-01-20 15:08:00 +00:00
Nga Tran 8aeded32d6
refactor: reorganize compact_files core function (#6636) 
* refactor: reorganize compact_files core function

* chore: smore more in-progress structure

* refactor: further reorganization for compact_files

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-20 13:41:23 +00:00
Andrew Lamb d808c57cdc
chore: Remove `iox_arrow_flight` (#6621) 
* chore: Remove iox_arrow_flight

* fix: hack around tonic status errors

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-20 11:16:19 +00:00
dependabot[bot] 8ac93a58a6
chore(deps): Bump reqwest from 0.11.13 to 0.11.14 (#6643) 
* chore(deps): Bump reqwest from 0.11.13 to 0.11.14

Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.13 to 0.11.14.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.13...v0.11.14)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-20 07:21:58 +00:00
dependabot[bot] 33045489f9
chore(deps): Bump rustyline from 10.1.0 to 10.1.1 (#6642) 
* chore(deps): Bump rustyline from 10.1.0 to 10.1.1

Bumps [rustyline](https://github.com/kkawakam/rustyline) from 10.1.0 to 10.1.1.
- [Release notes](https://github.com/kkawakam/rustyline/releases)
- [Changelog](https://github.com/kkawakam/rustyline/blob/master/History.md)
- [Commits](https://github.com/kkawakam/rustyline/compare/v10.1.0...v10.1.1)

---
updated-dependencies:
- dependency-name: rustyline
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2023-01-20 07:13:12 +00:00
Carol (Nichols || Goulding) 6afd782b3f
fix: Move query_tests2 into influxdb_iox/tests so that the code rebuilds 2023-01-19 16:44:31 -05:00
Carol (Nichols || Goulding) fb6774e40e
fix: Rather than overloading ChunkStage, make a separate IoxArchitecture enum 2023-01-19 16:44:30 -05:00
Carol (Nichols || Goulding) 8783623a19
docs: This method doesn't block until the data is persisted 2023-01-19 16:44:30 -05:00
Carol (Nichols || Goulding) af203f7a6d
docs: Explain why the tests set the number of query threads 2023-01-19 16:44:30 -05:00
Carol (Nichols || Goulding) bc67ca37a9
fix: Make sure tests using the Kafka architecture WaitForReadable 2023-01-19 16:44:30 -05:00
Carol (Nichols || Goulding) 59914906b6
fix: Only reset persist everything flag if data has been persisted 2023-01-19 16:44:30 -05:00
Carol (Nichols || Goulding) 3dbaeedca6
feat: Try implementing the persist api in a diffferent way 2023-01-19 16:44:30 -05:00
Carol (Nichols || Goulding) f85c8d916e
fix: Hardcode query threads in old query_tests too 2023-01-19 16:44:30 -05:00
Carol (Nichols || Goulding) 81f5f3b75f
feat: Implement the persist service gRPC API on the old ingester for query_tests2 to use 2023-01-19 16:44:30 -05:00
Carol (Nichols || Goulding) dc95438694
feat: Add old ingester and old parquet states into the query_tests2 framework 2023-01-19 16:44:29 -05:00
Andrew Lamb b265036f33
refactor: remove iox_arrow_flight use in ingester_replica (#6634) 
* refactor: remove iox_arrow_flight use in ingester_replica

* chore: fixlu

* fix: update doclink

* fix: Undo rustix downgrade

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-19 18:55:57 +00:00
Andrew Lamb 65c020c9f2
refactor: remove iox_arrow_flight use in `influxdb_iox_client ` and `querier` (#6624) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-19 18:48:23 +00:00
Marco Neumann 810f9096b8
feat: catalog commits for compactor2 (#6638) 2023-01-19 17:37:45 +00:00
Marco Neumann 5c462937ca
refactor: converters for `ParquetFile`<>`ParquetFileParams` (#6637) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-19 16:47:54 +00:00
Marco Neumann df85c7b154
refactor: more flexible filter design (#6635) 
Filters can now inspect ALL files for a partition which may be useful
for limiters. This also moves the "is not empty" part into a filter.

Note that we still can only run ONE compaction job per partition for the
time being, so splitting the files into multiple sub-groups and run a
per-group DataFusion job is currently not possible. It should be a rather easy
addition if we ever want that (probably needs another semaphore of
something to limit the overall job count).
 2023-01-19 16:40:54 +00:00
Marco Neumann eb149eca7c
feat: partition error sink (#6633) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-19 15:34:57 +00:00
Andrew Lamb 5b6d261396
refactor: remove iox_arrow_flight use in ingester2 (#6623) 
* refactor: remove iox_arrow_flight use in ingester2

* fix: Update ingester2/src/server/grpc/query.rs

Co-authored-by: Dom <dom@itsallbroken.com>

* chore: remove unused Error enums

Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-19 15:27:23 +00:00
Andrew Lamb f639bf3e23
chore: refactor ingester to use upstream arrow-flight (#6622) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-19 15:16:13 +00:00
Andrew Lamb 2920db4100
docs: Document semantics of selector functions (#6615) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-19 15:07:49 +00:00
Marco Neumann ad69a554d6
refactor: extract "fetch files" component (#6632) 2023-01-19 13:11:44 +00:00
Marco Neumann e4bf9e85d9
refactor: more components for compactor2 (#6630) 
* refactor: rename "rules" to "components"

* refactor: replace `Trait::name` w/ `impl Display`

* refactor: remodel partitions source

* refactor: add TODO

* refactor: fully components-based partitions source

* test: add tests for partitions sources

* docs: improve

Co-authored-by: Dom <dom@itsallbroken.com>

Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-19 12:29:38 +00:00
Marco Neumann 5e297b4667
refactor: lift up compactor2 CLI args, set mem limit to 8GB (#6631) 
- use a single data structure for CLI args (not two)
- set mem limit default to 8GB (same as querier). We can always tune
  this later, but we should not run with "unlimited" to begin with.
 2023-01-19 12:21:51 +00:00
Marco Neumann 29c3800ad7
test: add log level in test tracing capture (#6629) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-19 11:29:13 +00:00
Nga Tran 9ae03b16d6
feat: invokes catalog functions for compactor2 (#6619) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-19 10:33:57 +00:00
Andrew Lamb 3fa91d682b
chore: uncomment now working tests in query_tests2 (#6625) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-19 09:26:13 +00:00
dependabot[bot] 2db763508d
chore(deps): Bump rustix from 0.36.6 to 0.36.7 (#6627) 
Bumps [rustix](https://github.com/bytecodealliance/rustix) from 0.36.6 to 0.36.7.
- [Release notes](https://github.com/bytecodealliance/rustix/releases)
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.36.6...v0.36.7)

---
updated-dependencies:
- dependency-name: rustix
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-19 09:18:27 +00:00
Luke Bond 11fa648116
chore: clarify cli args for iox gc (#6628) 
Signed-off-by: Luke Bond <luke.n.bond@gmail.com>

Signed-off-by: Luke Bond <luke.n.bond@gmail.com>
 2023-01-19 07:43:29 +00:00
Andrew Lamb 6a4fa11dba
chore: Update to latest datafusion / arrow `31.0.0` (#6618) 
* chore: Update datafusion and arrow/parquet/arrow-flight `31.0.0`

* chore: Update for new API

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-18 20:21:10 +00:00
kodiakhq[bot] 9e530da897
Merge pull request #6497 from influxdata/cn/query-tests-grpc 
feat: Reimagining query_tests sql cases
 2023-01-18 19:10:26 +00:00
kodiakhq[bot] 33168b97f0
Merge branch 'main' into cn/query-tests-grpc 2023-01-18 19:03:51 +00:00
Marco Neumann 380a855aab
feat: basic compactor2 algo layout (#6616) 2023-01-18 18:51:59 +00:00
Carol (Nichols || Goulding) d68a35a853
docs: Explain why the tests set the number of query threads 
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
 2023-01-18 13:31:52 -05:00
Christopher M. Wolff 413e4e4088
feat: create a logical plan node and rule for gap-filling (#6602) 
* feat: create a GapFill logical plan node

* feat: create a GapFill optimizer rule

* chore: code review feedback

* chore: fix issue found after merging main

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-18 17:01:55 +00:00
Marco Neumann e72173d58d
feat: very basic compactor2 skeleton (#6614) 
Sets up crate and wires up the main binary. No tests yet, no algorithm
framework, just the bare minimum.

Also I decided to not offer a gRPC server in `compactor2` at the moment
and hence did not implement any handle/delegate infrastructure. We add
this later if we need it. This also means compactor2 does NOT provide a
catalog service for now.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-18 16:36:40 +00:00
Nga Tran 7a5fdd1d95
feat: function to read partition IDs of all partitions with new writes (#6613) 
* feat: function to read partition IDs of all partitions with new writes

* chore: run fmt

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-18 16:30:15 +00:00
Andrew Lamb 3608d2881b
chore: Update dependencies (#6612) 
* chore: Update dependencies

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2023-01-18 16:19:50 +00:00
Andrew Lamb 8410998408
chore: Update datafusion to Jan 17, 2023 (2 / 2) and arrow/parquet `30.0.1` (#6604) 
* chore: Update datafusion to Jan 9, 2023 (2 / 2) and arrow/parquet `30.0.1`

* chore: Update for changes in arrow ipc

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2023-01-18 15:51:24 +00:00
Carol (Nichols || Goulding) 747bdb452b
fix: Add the standard lints to query_tests2 and fix the new warnings 2023-01-18 10:24:17 -05:00