Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
9,508 Commits
218 Branches
453 Tags
340 MiB
Commit Graph

9508 Commits (474620f4a7cd981f9bf669631da1781e4f04e77f)

Author SHA1 Message Date
Dom Dwyer 3e70dc44a0 refactor(catalog): remove partition_info_by_id() 
This method used to return a subset of partition metadata, and was used
exclusively for persistence in the ingester. It is now no longer
necessary.
 2022-10-13 15:26:36 +02:00
Dom Dwyer 3fbeaa1314 refactor: assert monotonic partition persistence 
Copies the existing monotonic partition persistence check into the
partition too - this ensures that even if the partitions are persisted
in order, they are never marked as persisted OUT of order.
 2022-10-13 15:26:36 +02:00
Dom Dwyer 920f7edf75 refactor: defer querying for table schema 
Do not query for the table schema until it is needed.
 2022-10-13 15:26:36 +02:00
Dom Dwyer e556677192 perf(ingester): remove persist lookup queries 
Removes the catalog queries previously used to look up various
information about the partition/table/namespace that was already in
memory.

As part of this change, the compaction helper function is changed to
accept the inputs it needs, rather than a struct of data from the
catalog - this significantly simplifies testing.

This commit also adds additional context to all log messages in the
persist() fn.
 2022-10-13 15:26:36 +02:00
Dom Dwyer 10d77b0ef7 refactor: use deferred sort key loading 
Changes the persist() implementation in the ingester to load the sort
key using the deferred loading mechanism, instead of on-demand.
 2022-10-13 15:26:36 +02:00
kodiakhq[bot] 8942b9043b
Merge pull request #5851 from influxdata/dom/log-sequence-numbers 
refactor: include sequence numbers in apply() logs
 2022-10-13 12:27:37 +00:00
Dom Dwyer dbcbb5b824 refactor: include sequence numbers in apply() logs 
Include the op sequence number in the error/success apply() log
messages.
 2022-10-13 14:19:02 +02:00
Dom f9b4fde1a9
Merge pull request #5850 from influxdata/dom/cheapen-table-lookup 
perf(ingester): cheapen table lookup
 2022-10-13 12:57:54 +01:00
Dom Dwyer 3e1e4c1f0b refactor: remove Table::get_table_persist_info() 
Remove the now-redundant get_table_persist_info() implementations.
 2022-10-13 13:44:50 +02:00
Dom Dwyer 15e153a74c perf(ingester): cheaper table discovery 
This commit changes the table ID lookup query from an expensive,
JOIN multi-query to a simple, single table, indexed lookup.

As this is on the hot path, this should help with the recovery rate of
the ingesters.
 2022-10-13 13:44:50 +02:00
Andrew Lamb 227ecd4da1
refactor: Move http delete parsing code into router along side other http code (#5841) 2022-10-13 11:08:42 +00:00
Carol (Nichols || Goulding) 442a7ff2a4
revert: "revert: rdkafka/rskafka swapping (#5800)" (#5844) 
* revert: "revert: rdkafka/rskafka swapping (#5800)"

This reverts commit b77c3540e1.

* test: Verify write buffer connection_config is parsed as expected

* test: Failing test reproducing the error seen when deploying rdkafka

* fix: Translate k8s-idpe configs to rdkafka configs
 2022-10-13 09:33:06 +00:00
Andrew Lamb 76b2f8d9d0
chore: update dpes (#5846) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-10-13 09:12:57 +00:00
Andrew Lamb 1a77c509c0
docs: Add some doc comments to the datafusion_util module (#5842) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-10-13 09:05:01 +00:00
Andrew Lamb 027636aad2
chore: remove vestigal DML handling from ioxd_common (#5840) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-10-13 08:57:18 +00:00
kodiakhq[bot] b5973b5575
Merge pull request #5843 from influxdata/dependabot/cargo/clap-4.0.14 
chore(deps): Bump clap from 4.0.7 to 4.0.14
 2022-10-12 17:37:19 +00:00
kodiakhq[bot] e39434ce5c
Merge branch 'main' into dependabot/cargo/clap-4.0.14 2022-10-12 17:29:37 +00:00
Nga Tran 1400bf99e4
refactor: split memory estimation into bytes to store and bytes to stream (#5845) 
* refactor: split memory estimation into bytes to store and bytes to stream

* chore: cleanup

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-10-12 16:59:51 +00:00
dependabot[bot] e20fd95db7
chore(deps): Bump clap from 4.0.7 to 4.0.14 
Bumps [clap](https://github.com/clap-rs/clap) from 4.0.7 to 4.0.14.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.0.7...v4.0.14)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-10-12 16:42:21 +00:00
Andrew Lamb d57c99638c
chore: Update datafusion + `arrow`, `arrow-flight`, and `parquet` to 24.0.0.0 (#5792) 
* chore: Update datafusion + `arrow`, `arrow-flight`, and `parquet` to 24.0.0.0

* fix: Update for coercion, fix explain plans for change in column name display

* chore: Update datafusion lock

* fix: Update for other API changes

* chore: Update to latest datafusion pin

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-10-12 16:19:14 +00:00
Nga Tran f05ca867a5
feat: add file size into estimated memory (#5837) 
* feat: add file size into estimataed memory

* chore: cleanup

* chore: fmt

* chore: Apply suggestions from code review

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

* chore: run fmt after applying review suggestion

* fix: fix tests towork with the change for review suggestion

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-10-12 14:42:53 +00:00
kodiakhq[bot] c9672bdc65
Merge pull request #5761 from influxdata/dependabot/cargo/clap-4.0.2 
chore(deps): Bump clap from 3.2.22 to 4.0.7
 2022-10-12 14:09:54 +00:00
kodiakhq[bot] 266b8f2a58
Merge branch 'main' into dependabot/cargo/clap-4.0.2 2022-10-12 14:01:28 +00:00
dependabot[bot] 7202dddab6
chore(deps): Bump tokio-stream from 0.1.10 to 0.1.11 (#5838) 
Bumps [tokio-stream](https://github.com/tokio-rs/tokio) from 0.1.10 to 0.1.11.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-stream-0.1.10...tokio-stream-0.1.11)

---
updated-dependencies:
- dependency-name: tokio-stream
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-10-12 12:37:24 +00:00
Luke Bond 11900cea4d
chore: add some tracing logs to the ingester (#5839) 2022-10-12 12:10:20 +00:00
Nga Tran b7153862b0
refactor: due to limit in size uplaoed to S3, we need to split output file of cold compaction, too (#5834) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-10-11 17:22:19 +00:00
kodiakhq[bot] 990fa55e28
Merge pull request #5832 from influxdata/dom/move-query-types 
refactor: move query types to query_handler
 2022-10-11 16:14:01 +00:00
Dom Dwyer b294bb98aa refactor: move query types to query_handler 
Moves types that are only used for handling queries to the query_handler
module.
 2022-10-11 17:58:55 +02:00
kodiakhq[bot] 7c9a26849b
Merge pull request #5831 from influxdata/dom/remove-tombstones 
refactor(ingester): remove tombstone support and delete tests from `query_tests`
 2022-10-11 15:22:24 +00:00
kodiakhq[bot] 96ff3b020a
Merge branch 'main' into dom/remove-tombstones 2022-10-11 15:14:54 +00:00
Dom Dwyer c4f542bbe2 refactor(ingester): remove tombstone support 
This commit removes tombstone support from the ingester, and deletes
associated code/helpers/tests. This commit does NOT remove tombstone
support from any other service, but MAY include removing overlapping
test coverage.

This also removes the tombstone support from the Ingester -> Querier RPC
response message.

This has the nice side effect of removing a whole lot of thread spawning
in the ingester tests for the Executor, speeding everything up!
 2022-10-11 13:10:04 +02:00
kodiakhq[bot] a205c01b16
Merge pull request #5830 from influxdata/dom/revert-rdkafka 
revert: rdkafka/rskafka swapping (#5800)
 2022-10-11 11:10:01 +00:00
Dom Dwyer b77c3540e1 revert: rdkafka/rskafka swapping (#5800) 
This reverts commit 33391af973.
 2022-10-11 13:01:10 +02:00
Luke Bond fda1479db0
chore: add trace log to ingester to aid debugging (#5829) 2022-10-11 10:33:42 +00:00
Carol (Nichols || Goulding) 33391af973
feat: Swap Kafka Producer implementation back to rdkafka as diagnosis of latency problem (#5800) 
* feat: Add back rdkafka dependency

* feat: Remove RSKafkaProducer

* feat: Remove write buffer RecordAggregator

* feat: Add back rdkafka producer

Using code from 58a2a0b9c8311303c796495db4f167c99a2ea3aa then getting it
to compile with the latest

* feat: Add a metric around enqueue

* fix: Remove unused imports

* fix: Increase Kafka timeout to 20s

* docs: Clarify that Kafka topics should only be created in test/dev envs

* fix: Remove metrics that aren't needed for this experiment

Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-10-11 09:14:45 +00:00
Dom 2b5ca28374
Merge pull request #5827 from influxdata/dependabot/cargo/object_store-0.5.1 
chore(deps): Bump object_store from 0.5.0 to 0.5.1
 2022-10-11 10:05:31 +01:00
Dom d2467d0b63
Merge branch 'main' into dependabot/cargo/object_store-0.5.1 2022-10-11 09:56:27 +01:00
Dom 2b8958fc03
Merge pull request #5826 from influxdata/dom/table-name-type 
refactor: use TableName, not Arc<str>
 2022-10-11 09:54:29 +01:00
dependabot[bot] 933493fab3
chore(deps): Bump object_store from 0.5.0 to 0.5.1 
Bumps [object_store](https://github.com/apache/arrow-rs) from 0.5.0 to 0.5.1.
- [Release notes](https://github.com/apache/arrow-rs/releases)
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md)
- [Commits](https://github.com/apache/arrow-rs/compare/object_store_0.5.0...object_store_0.5.1)

---
updated-dependencies:
- dependency-name: object_store
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-10-11 01:19:10 +00:00
Dom Dwyer 97c6e0f8ce refactor: use TableName, not Arc<str> 
Adds a type wrapper TableName, internally an Arc<str> to leverage the
type system instead of passing around untyped strings.
 2022-10-10 19:09:43 +02:00
Dom c95bf8ff87
Merge pull request #5807 from influxdata/dom/deferred-sort-key-load 
perf(ingester): deferred sort key load
 2022-10-10 18:09:31 +01:00
Dom c1277fb15d
Merge branch 'main' into dom/deferred-sort-key-load 2022-10-10 16:05:14 +01:00
dependabot[bot] 0eac3812c8
chore(deps): Bump snafu from 0.7.1 to 0.7.2 (#5821) 
Bumps [snafu](https://github.com/shepmaster/snafu) from 0.7.1 to 0.7.2.
- [Release notes](https://github.com/shepmaster/snafu/releases)
- [Changelog](https://github.com/shepmaster/snafu/blob/main/CHANGELOG.md)
- [Commits](https://github.com/shepmaster/snafu/compare/0.7.1...0.7.2)

---
updated-dependencies:
- dependency-name: snafu
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-10-10 13:15:29 +00:00
dependabot[bot] f8bc4d8881
chore(deps): Bump libc from 0.2.134 to 0.2.135 (#5822) 
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.134 to 0.2.135.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.134...0.2.135)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-10-10 13:06:59 +00:00
Dom Dwyer 4518bd49d1 test: constify duration seconds 2022-10-10 14:39:35 +02:00
Dom Dwyer ab78f99ab2 refactor: eager background task abort 
Changes the get() code path to abort the background load task when the
caller will resolve the sort key.

Note that an aborted future will leave the DeferredSortKey without a
background task to fetch the key, and the next caller will have to query
the catalog. Given the rarity of aborted futures, and desire to minimise
catalog load, this seems like a decent trade-off.

This commit also documents the many-readers eager loading problem.
 2022-10-10 14:39:35 +02:00
Dom 3bdcf4dc7a
Merge pull request #5820 from influxdata/dependabot/cargo/serde_json-1.0.86 
chore(deps): Bump serde_json from 1.0.85 to 1.0.86
 2022-10-10 12:22:20 +01:00
dependabot[bot] 2277fcf08a
chore(deps): Bump serde_json from 1.0.85 to 1.0.86 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.85 to 1.0.86.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.85...v1.0.86)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-10-10 01:42:37 +00:00
Carol (Nichols || Goulding) 63a4a1051a
fix: Restore --help flag in iox_data_generator 2022-10-07 17:02:56 -04:00
Andrew Lamb d8a318eb57
docs: Tweak local run guide (#5787) 
Update the instructions on how to run IOx locally

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-10-07 20:34:00 +00:00