Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
5,339 Commits
189 Branches
453 Tags
380 MiB
Commit Graph

641 Commits (e911cf9ac172f09e3716d23bb8d0ec95c9a78c4c)

Author SHA1 Message Date
Raphael Taylor-Davies 9de12745e7
feat: migrate lock metrics to metric crate (#2481) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-09-09 08:56:19 +00:00
Nga Tran 00df7b064c feat: finally have the delete predicate parsed 2021-09-08 17:30:10 -04:00
Nga Tran dbe4bcff22 chore: merge main to branch 2021-09-07 10:54:59 -04:00
kodiakhq[bot] f6e040df3d
Merge branch 'main' into dependabot/cargo/tokio-1.11.0 2021-09-06 09:18:55 +00:00
Raphael Taylor-Davies a4b0cbc0e7
feat: migrate jemalloc metrics to `metric` crate (#2435) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-09-06 09:18:27 +00:00
dependabot[bot] b67610d9b9
chore(deps): bump tokio from 1.10.1 to 1.11.0 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.10.1 to 1.11.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.10.1...tokio-1.11.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-09-06 09:11:38 +00:00
kodiakhq[bot] 06311970fa
Merge branch 'main' into dependabot/cargo/parquet-5.3.0 2021-09-06 09:00:27 +00:00
dependabot[bot] ccfd6ad3da
chore(deps): bump parquet from 5.2.0 to 5.3.0 
Bumps [parquet](https://github.com/apache/arrow-rs) from 5.2.0 to 5.3.0.
- [Release notes](https://github.com/apache/arrow-rs/releases)
- [Changelog](https://github.com/apache/arrow-rs/blob/5.3.0/CHANGELOG.md)
- [Commits](https://github.com/apache/arrow-rs/compare/5.2.0...5.3.0)

---
updated-dependencies:
- dependency-name: parquet
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-09-06 08:52:30 +00:00
kodiakhq[bot] 0fd92c8b45
Merge branch 'main' into dependabot/cargo/structopt-0.3.23 2021-09-06 08:51:32 +00:00
Nga Tran 4801b2c238 feat: Have the ParseDelete message and its corresponding ProvidedParseDelete struct ready for building delete parser 2021-09-06 00:13:59 -04:00
dependabot[bot] b1bb390893
chore(deps): bump parking_lot from 0.11.1 to 0.11.2 
Bumps [parking_lot](https://github.com/Amanieu/parking_lot) from 0.11.1 to 0.11.2.
- [Release notes](https://github.com/Amanieu/parking_lot/releases)
- [Changelog](https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Amanieu/parking_lot/compare/0.11.1...0.11.2)

---
updated-dependencies:
- dependency-name: parking_lot
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-09-06 01:18:24 +00:00
dependabot[bot] 2cf9505029
chore(deps): bump structopt from 0.3.22 to 0.3.23 
Bumps [structopt](https://github.com/TeXitoi/structopt) from 0.3.22 to 0.3.23.
- [Release notes](https://github.com/TeXitoi/structopt/releases)
- [Changelog](https://github.com/TeXitoi/structopt/blob/master/CHANGELOG.md)
- [Commits](https://github.com/TeXitoi/structopt/compare/v0.3.22...v0.3.23)

---
updated-dependencies:
- dependency-name: structopt
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-09-06 01:17:41 +00:00
Nga Tran d8b3208896 refactor: sstart building parser in the client side 2021-09-03 17:56:45 -04:00
Carol (Nichols || Goulding) f47d7e6d92 feat: Return the time the database was deleted 2021-09-02 14:57:27 -04:00
Marco Neumann 4a863993ec feat: "dump catalog" debug CLI 2021-09-02 08:08:20 +02:00
Marco Neumann a5b8c1631b chore: upgrade `opentelemetry[-prometheus]` 
- "labels" are now named "attributes"
- `PrometheusExpoerter` is now created via a builder pattern which
  provides way better defaults (e.g. it construct a sane controller
  itself)
 2021-08-31 10:39:57 +02:00
Marco Neumann 7ff28405ea chore: fix `opentelemetry-jeager` upgrade 2021-08-31 10:39:57 +02:00
dependabot[bot] 29b13ad2a6 chore(deps): bump opentelemetry-jaeger from 0.14.0 to 0.15.0 
Bumps [opentelemetry-jaeger](https://github.com/open-telemetry/opentelemetry-rust) from 0.14.0 to 0.15.0.
- [Release notes](https://github.com/open-telemetry/opentelemetry-rust/releases)
- [Commits](https://github.com/open-telemetry/opentelemetry-rust/compare/v0.14.0...v0.15.0)

---
updated-dependencies:
- dependency-name: opentelemetry-jaeger
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-08-31 10:39:57 +02:00
dependabot[bot] 3e154b3b02
chore(deps): bump serde_json from 1.0.66 to 1.0.67 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.66 to 1.0.67.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.66...v1.0.67)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-08-30 11:28:35 +00:00
dependabot[bot] fee8a986b6
chore(deps): bump handlebars from 3.5.5 to 4.1.2 
Bumps [handlebars](https://github.com/sunng87/handlebars-rust) from 3.5.5 to 4.1.2.
- [Release notes](https://github.com/sunng87/handlebars-rust/releases)
- [Changelog](https://github.com/sunng87/handlebars-rust/blob/master/CHANGELOG.md)
- [Commits](https://github.com/sunng87/handlebars-rust/compare/v3.5.5...v4.1.2)

---
updated-dependencies:
- dependency-name: handlebars
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-08-30 11:18:48 +00:00
dependabot[bot] 52fbaef2df
chore(deps): bump thiserror from 1.0.26 to 1.0.28 
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.26 to 1.0.28.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.26...1.0.28)

---
updated-dependencies:
- dependency-name: thiserror
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-08-30 11:05:11 +00:00
dependabot[bot] 13161de003
chore(deps): bump rustyline from 8.2.0 to 9.0.0 
Bumps [rustyline](https://github.com/kkawakam/rustyline) from 8.2.0 to 9.0.0.
- [Release notes](https://github.com/kkawakam/rustyline/releases)
- [Changelog](https://github.com/kkawakam/rustyline/blob/master/History.md)
- [Commits](https://github.com/kkawakam/rustyline/compare/v8.2.0...v9.0.0)

---
updated-dependencies:
- dependency-name: rustyline
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-08-30 01:19:49 +00:00
Raphael Taylor-Davies fcec394a28
feat: connect up new metrics (#2428) 2021-08-27 12:55:35 +00:00
Raphael Taylor-Davies 3a26f7857f
feat: add new metrics abstraction (#2076) (#2365) 
* feat: add new metrics abstraction (#2076)

* chore: fix lint

* feat: add tests

feat: histogram export

* chore: further docs

* chore: docs

* chore: log on prometheus export error

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-27 11:54:23 +00:00
kodiakhq[bot] 8cf84a24eb
Merge branch 'main' into pd/duration-sampling-interval 2021-08-26 17:52:34 +00:00
Carol (Nichols || Goulding) 82b566a3bb fix: Warn when a non-generation item is found in a database directory 2021-08-26 09:14:23 -04:00
Andrew Lamb f975baba6b
chore: Update datafusion + other deps again (get baseline metrics) (#2422) 
* chore: Update datafusion reference

* chore: cargo update

* fix: update explain tests to show Union

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-26 13:13:00 +00:00
Raphael Taylor-Davies 1773bf5d37
feat: add storage client to influxdb_iox_client (#2404) 
* feat: add storage client to influxdb_iox_client

* chore: fix type_url

* refactor: split storage into separate crate

* chore: fix doctest

* chore: review feedback

* chore: add generated_types cleanup ticket

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-26 10:30:59 +00:00
Andrew Lamb ddf6c6362e
chore: update DataFusion again (#2411) 
* chore: update datafusion ref

* chore: run cargo update

* refactor: Rename concurrency to target_partitions, avoid deprecation warning
 2021-08-26 08:03:13 +00:00
Paul Dix 64fca1ee34 feat: Support sampling interval strings in data generator 
This changes the sampling_interval in the data generator to be a string, supporting things like ns, us, ms, s, m, h and others.
 2021-08-25 17:35:01 -04:00
Raphael Taylor-Davies f7792aafe6
feat: query tracing (#2273) (#2391) 
* feat: query tracing (#2273)

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-24 17:35:59 +00:00
Andrew Lamb 745eaa9248
chore: Update DataFusion + other deps (to get new Metric API) (#2385) 
* chore: Update deps

* refactor: Update IOx to use new DataFusion Metric API

* fix: update Modulus --> Modulo
 2021-08-24 16:07:23 +00:00
Raphael Taylor-Davies 2c9c191b17
refactor: split out trace http (#2388) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-24 13:37:20 +00:00
Raphael Taylor-Davies 3fdc0e9a6a
chore: remove orphaned code from trogging (#2371) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-24 10:12:59 +00:00
dependabot[bot] 7a11b7cfd2
chore(deps): bump ouroboros from 0.8.3 to 0.10.1 (#2374) 
* chore(deps): bump ouroboros from 0.8.3 to 0.10.1

Bumps [ouroboros](https://github.com/joshua-maros/ouroboros) from 0.8.3 to 0.10.1.
- [Release notes](https://github.com/joshua-maros/ouroboros/releases)
- [Commits](https://github.com/joshua-maros/ouroboros/commits)

---
updated-dependencies:
- dependency-name: ouroboros
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* refactor: manual derives for `Entry`

- works better w/ ouroboros
- `Debug` now does not show useless+verbose raw data
- `PartialEq` now only compares deserialized data (which is likely more
  correct)

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-23 17:25:14 +00:00
Raphael Taylor-Davies 58716dee71
refactor: remove mem-qe (#2344) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-23 14:35:12 +00:00
dependabot[bot] 631ef17d31
chore(deps): bump integer-encoding from 1.1.7 to 3.0.2 
Bumps [integer-encoding](https://github.com/dermesser/integer-encoding-rs) from 1.1.7 to 3.0.2.
- [Release notes](https://github.com/dermesser/integer-encoding-rs/releases)
- [Commits](https://github.com/dermesser/integer-encoding-rs/compare/v1.1.7...v3.0.2)

---
updated-dependencies:
- dependency-name: integer-encoding
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-08-23 11:42:16 +00:00
kodiakhq[bot] 4b864e8756
Merge branch 'main' into dependabot/cargo/cloud-storage-0.10.2 2021-08-23 11:32:58 +00:00
dependabot[bot] 37c81fb483
chore(deps): bump cloud-storage from 0.9.0 to 0.10.2 
Bumps [cloud-storage](https://github.com/ThouCheese/cloud-storage-rs) from 0.9.0 to 0.10.2.
- [Release notes](https://github.com/ThouCheese/cloud-storage-rs/releases)
- [Changelog](https://github.com/ThouCheese/cloud-storage-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ThouCheese/cloud-storage-rs/commits)

---
updated-dependencies:
- dependency-name: cloud-storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-08-23 11:26:28 +00:00
kodiakhq[bot] 045989e173
Merge branch 'main' into dependabot/cargo/serde-1.0.128 2021-08-23 11:24:56 +00:00
dependabot[bot] 0d4bc4190f
chore(deps): bump assert_cmd from 1.0.8 to 2.0.0 
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd) from 1.0.8 to 2.0.0.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases)
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v1.0.8...v2.0.0)

---
updated-dependencies:
- dependency-name: assert_cmd
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-08-23 01:17:06 +00:00
dependabot[bot] f11ad0bf1d
chore(deps): bump serde from 1.0.127 to 1.0.128 
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.127 to 1.0.128.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.127...v1.0.128)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-08-23 01:16:38 +00:00
Marko Mikulicic bbbd38bb12
fix(heappy): Prevent re-entering a running heap profile 
Upgrades heappy to include https://github.com/mkmik/heappy/pull/2
and adapts to the API change.
 2021-08-20 14:36:36 +02:00
Marko Mikulicic 4a2f0b0a52
fix: Make heappy and nonheappy mutually exclusive 
Closes #2288
 2021-08-20 11:35:07 +02:00
Raphael Taylor-Davies db97069ecd
feat: jaeger span export (#2273) (#2348) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-20 08:57:16 +00:00
Carol (Nichols || Goulding) 31ead36fc0 test: Don't compare floats for strict equality 2021-08-19 15:02:34 -04:00
Paul Dix d5f01a2a68 refactor: move data generator to IOx repo and fix build 2021-08-19 14:26:15 -04:00
Raphael Taylor-Davies 98627944e7
refactor: make packers a dev-dependency of read buffer (#2345) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-19 11:09:34 +00:00
Raphael Taylor-Davies 04ba139cb2
refactor: split trace crate (#2343) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-19 11:00:32 +00:00
Raphael Taylor-Davies 710f477e35
refactor: remove opentelemetry from observability_deps (#2192) (#2336) 2021-08-18 17:24:05 +00:00
Carol (Nichols || Goulding) 6d5cb9c117 refactor: Extract a ParquetFilePath to handle paths to parquet files in a db's object store 2021-08-18 11:32:39 -04:00
Marko Mikulicic fc473995fa
feat(iox): Upgrade heappy to present free memory events 
Currently heappy is reporting "bytes allocated" and "inuse_bytes" ("bytes allocated - freed bytes").

Since the allocation site and free site can be wildly different, it's very hard to correlate these events
and have a meaningful "inuse_bytes" graph.

I have a plan for how to fix this for good, but in the meantime
I'd like to report raw "freed bytes".
 2021-08-18 13:04:48 +02:00
Raphael Taylor-Davies 1d6a8703af
feat: support sinking traces to an OTEL SpanExporter (#2319) 
* feat: support sinking traces to an OTEL SpanExporter

* chore: consistent logging

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

* chore: review feedback

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-18 08:05:21 +00:00
Raphael Taylor-Davies efa776bd03
feat: fix log crate output (#2325) 
* feat: fix log crate output

* chore: fix doc

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-17 16:19:22 +00:00
Marko Mikulicic 94ff77126e
feat: Enable freed memory tracking in heappy 
In aed37ab50a I fixed the main
problem that rendered free tracking useless: the allocator was actually allocating more bytes than requested; we were tracking the amount of memory requested by the user and not the size of the block that was returned to the user. However, when we were tracking a call to free we used the size of the memory block which was bigger than the amount that was tracked; this led to negative numbers of "in use memory".
 2021-08-17 15:09:28 +02:00
Carol (Nichols || Goulding) bda4458ebd fix: Update all the rusoto crates together 2021-08-16 19:59:26 -04:00
dependabot[bot] 97ed42c42f
build(deps): bump rusoto_credential from 0.46.0 to 0.47.0 
Bumps [rusoto_credential](https://github.com/rusoto/rusoto) from 0.46.0 to 0.47.0.
- [Release notes](https://github.com/rusoto/rusoto/releases)
- [Changelog](https://github.com/rusoto/rusoto/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rusoto/rusoto/compare/mock-v0.46.0...rusoto-v0.47.0)

---
updated-dependencies:
- dependency-name: rusoto_credential
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-08-16 16:35:09 +00:00
dependabot[bot] 1d85a3fef6
build(deps): bump predicates from 1.0.8 to 2.0.2 
Bumps [predicates](https://github.com/assert-rs/predicates-rs) from 1.0.8 to 2.0.2.
- [Release notes](https://github.com/assert-rs/predicates-rs/releases)
- [Changelog](https://github.com/assert-rs/predicates-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/predicates-rs/compare/v1.0.8...v2.0.2)

---
updated-dependencies:
- dependency-name: predicates
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-08-16 15:36:49 +00:00
dependabot[bot] eaca64051d
build(deps): bump env_logger from 0.8.4 to 0.9.0 (#2303) 
Bumps [env_logger](https://github.com/env-logger-rs/env_logger) from 0.8.4 to 0.9.0.
- [Release notes](https://github.com/env-logger-rs/env_logger/releases)
- [Changelog](https://github.com/env-logger-rs/env_logger/blob/main/CHANGELOG.md)
- [Commits](https://github.com/env-logger-rs/env_logger/compare/v0.8.4...v0.9.0)

---
updated-dependencies:
- dependency-name: env_logger
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-08-16 15:35:04 +00:00
Andrew Lamb 0b1c48dae3
chore: update deps (to get arrow-5.2.0) (#2299) 2021-08-16 13:51:40 +00:00
Raphael Taylor-Davies 302e2b5353
feat: support jaeger context propagation format (#2293) 2021-08-16 10:55:40 +00:00
Marko Mikulicic 500019cff9
Merge branch 'main' into ntran/heappy2 2021-08-13 17:27:38 +02:00
Raphael Taylor-Davies 8650e419f7
feat: add trace collection plumbing (#2273) (#2274) 
* feat: add trace collection plumbing (#2273)

* chore: fix lints

* chore: use tower directly

* chore: add tests

* chore: review feedback

* refactor: use serde instead of custom display impl

* chore: fix doc
 2021-08-13 14:38:31 +00:00
Raphael Taylor-Davies 0a065b4968
feat: set header on gRPC requests (#2283) 
* feat: set header on gRPC requests

* feat: always insert header value

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-13 14:13:50 +00:00
kodiakhq[bot] d506da2a1a
Merge branch 'main' into cn/extract-iox-object-store 2021-08-13 13:45:35 +00:00
Nga Tran 36e84f85ef chore: Merge branch 'main' into ntran/heappy2 2021-08-13 08:16:58 -04:00
Nga Tran 4fd5862a9c fix: turn heappy on 2021-08-12 17:14:03 -04:00
Andrew Lamb 9a12517dde
chore: Update datafusion dependencies (#2277) 
* chore: bump datafusion deps

* chore: run cargo update
 2021-08-12 20:16:08 +00:00
Carol (Nichols || Goulding) ae6b0e669b refactor: Extract a database persister type that wraps object store 
Connects to #2193.
 2021-08-12 15:05:32 -04:00
kodiakhq[bot] e0448be345
Merge branch 'main' into ntran/heappy 2021-08-12 17:57:16 +00:00
Nga Tran 4f52a70d5b refactor: adddress mkm's comments 2021-08-12 12:04:42 -04:00
Andrew Lamb 34a1c1674f
chore: remove unused dependency (#2247) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-12 08:57:12 +00:00
Nga Tran 8aa9514d93 feat: integrate heappy implemented by mkm 2021-08-11 23:42:50 -04:00
Raphael Taylor-Davies cd5f4e1755
feat: background worker panic handling (#2091) (#2234) 
* feat: worker panic handling (#2091)

* chore: add test comments

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-10 09:17:56 +00:00
Raphael Taylor-Davies c957d8154f
feat: blocking Freezable (#2224) 
* feat: blocking Freezable

* chore: test
 2021-08-08 19:26:11 +00:00
Marco Neumann 882f89cecf fix: only warn when partition ckpt and DB ckpt mins are out-of-sync 
There are currently a few bugs and semi-understood edge cases that can
lead to this case. So instead of bailing out, just issue a warning.
 2021-08-06 09:48:26 +02:00
Andrew Lamb e69a159ad2
refactor: remove unused opentelemetry dependencies (#2201) 2021-08-05 16:20:22 +00:00
Andrew Lamb e92e94caad
chore: Update deps (including arrow 5.1.0, tonic -> 0.5, and prost 0.5) (#2172) 
* chore: Update deps (including arrow 5.0.0 --> arrow 5.1.0)

* chore: update all the things

* refactor: Update serving readiness check due to change in Tonic API

* chore: update more deps

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-05 15:57:38 +00:00
Andrew Lamb bb8021d9fd
fix: "Can not convert index to usize in dictionary of type creating group by value Int32" (#2151) 
* test: add reproducer for index error

* chore: update datafusion
 2021-08-02 12:20:41 +00:00
Andrew Lamb 7ca31703a4
feat: Attempt to dump a stacktrace to stderr prior to process abort on SIGSEGV/SIGILL/SIGBUS (#2155) 
* feat: Attempt to dump a stacktrace to stdout prior to process abort

* refactor: rewrite signal handling in terms of libc, remove sig dep

* refactor: print to stderr

* fix: Update src/main.rs

* docs: note provenance
 2021-07-30 11:58:28 +00:00
Andrew Lamb 21270b7072
chore: update dependencies (#2152) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-07-29 18:17:39 +00:00
kodiakhq[bot] cd0910c825
Merge branch 'main' into cn/update-croaring 2021-07-28 13:58:24 +00:00
Carol (Nichols || Goulding) 537e031987 fix: Upgrade croaring to get a performance fix 
This release includes https://github.com/saulius/croaring-rs/pull/73 which
includes 6403d44cbe
which should hopefully fix some of the dynamic CPUID issues we're
seeing.
 2021-07-28 09:25:01 -04:00
Marco Neumann e736bc6953 feat: add ingest timestamp to `Sequence` 
This allows us to track wall-clock ingest time for entries that we
receive via write buffer (e.g. Kafka).
 2021-07-28 14:41:18 +02:00
Dom 7d3b6bf80d feat: DatabaseRules (de)serialisation support 
Derives serde::Serialise & Deserialise for the DatabaseRules struct and
children.
 2021-07-27 14:29:26 +01:00
Andrew Lamb 9d408296d8
chore: Update datafusion deps (#2108) 
* chore: Update datafusion deps

* chore: update other deps

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-07-23 21:24:01 +00:00
Raphael Taylor-Davies 446af5eb15
fix: consistent write timestamps (#2104) 
* fix: consistent write timestamps

* chore: fix benchmarks
 2021-07-23 18:04:15 +00:00
Raphael Taylor-Davies 844a025c7c
feat: drop based on LRU (#2075) (#2092) 
* feat: drop based on LRU (#2075)

* chore: review feedback

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-07-23 08:31:28 +00:00
kodiakhq[bot] 71f3f1aba2
Merge branch 'main' into cn/refactorings 2021-07-22 19:44:18 +00:00
Andrew Lamb 01c79f1a1a
fix: Print all timestamps using RFC3339 format (#2098) 
* fix: Use IOx pretty printer rather than arrow pretty printer

* chore: update tests in the query crate

* chore: update influxdb_iox tests

* chore: Update end to end tests

* chore: update query_tests

* chore: update mutable_buffer tests

* refactor: update parquet_file tests

* refactor: update db tests

* chore: update kafka integration test output

* fix: merge conflict
 2021-07-22 19:04:52 +00:00
Andrew Lamb b3d6e3ed7b
fix: Implement pretty printer wity RFC3339 formatting for timestamps (#2096) 
* fix: Implement pretty printer wity RFC3339 formatting for timestamps

* fix: doc strings + fmt
 2021-07-22 17:08:27 +00:00
Carol (Nichols || Goulding) 8d1d877196 feat: Record first/last write times for RUB chunks 2021-07-22 11:35:22 -04:00
Raphael Taylor-Davies 8c974beba0
feat: add access timestamps to CatalogChunk (#2075) (#2081) 
* feat: add access timestamps to CatalogChunk (#2075)

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-07-22 12:19:30 +00:00
Raphael Taylor-Davies ffe6e62aee
feat: add instant to datetime conversion (#2078) 
* feat: add instant to datetime conversion

* chore: review feedback
 2021-07-21 11:43:27 +00:00
Andrew Lamb 387667330a
chore: Update datafusion deps (#2073) 
* chore: Update datafusion deps

* fix: update tests
 2021-07-21 08:27:03 +00:00
Andrew Lamb 1c16988a51
chore: Update datafusion references (#2056) 2021-07-19 18:09:06 +00:00
Carol (Nichols || Goulding) ff6b0ac030
fix: Cargo git deps use rev rather than ref (#2050) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-07-19 13:31:24 +00:00
Edd Robinson a852dce450 refactor: default to num_cpu 2021-07-19 14:00:10 +01:00
Andrew Lamb 4da8a16c18
chore: update to arrow 5.0 and master datafusion (#2049) 
* chore: update to arrow 5.0 and master datafusion

* fix: Update test for change in object size
 2021-07-19 12:49:51 +00:00
Marko Mikulicic 06399e88e0
chore: Add some debug logs to write buffer 2021-07-15 22:18:03 +02:00
Andrew Lamb 74b8bb76e6
chore: Update to correct pre-release version of DF (#2023) 
Co-authored-by: Edd Robinson <me@edd.io>
 2021-07-15 18:13:42 +00:00
Marco Neumann b5428e53a5 refactor: write buffer testing + better mocking 
This refactors the write buffer a bit for:

- **Testing:** Add generic tests for the Kafka and the mocking
  implementation. The same interface can be used easily add new
  implementations (e.g. via Redis, filesystem, ...).
- **Partition on Write:** The caller of the writer operation must now
  specify the partition/sequencer ID. The implicit partitioning of the
  Kafka writer would have lead to broken data since we must never spill
  entries w/ the same primary key over multiple partitions. At the
  moment we will only use partition 0 but we can easily implement
  better logic in the future.
- **Improved Mocking:** The mocked implementation now simulates a system
  that feels more real. Especially the handling around multiple streams
  and "write while read" has been improved. This will be helpful for
  testing and for new features like seeking (during replay). A solid
  realistic mock also helps us to ensure that the tests using the mock
  do not rely on unrealistic behavior too much.
 2021-07-15 17:20:45 +02:00
Raphael Taylor-Davies d71f38f27c
feat: compute PartitionCheckpoint from PersistenceWindows (#2011) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-07-15 12:17:23 +00:00
Raphael Taylor-Davies cbeeb97cff
feat: flush open window on persist (#1985) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-07-14 16:58:20 +00:00
Edd Robinson 0e5276ed20
Merge branch 'main' into alamb/go_go_go_go 2021-07-14 13:56:35 +01:00
Marco Neumann 9cb9ae0874 chore: move write buffer into its own crate 2021-07-14 14:09:18 +02:00
Andrew Lamb 4800b36949 chore: Update IOx to a pre-release version of arrow and datafusion to test out performance improvement 2021-07-13 15:44:57 -04:00
Andrew Lamb ef3269ee1d
chore: Update datafusion deps (#1984) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-07-13 17:29:29 +00:00
Marco Neumann 157a0cc98c chore: update flatbuffers to 2.0 2021-07-13 15:44:45 +02:00
Marco Neumann f5c63b2ae6 chore: update nom to version 6 
Triggered some changes from `Fn` to `FnMut`.
 2021-07-13 15:28:42 +02:00
Marco Neumann f11be17523 chore: update mockito to 0.30 2021-07-13 15:17:28 +02:00
Marco Neumann 2e391deb34 chore: update croaring to 0.5.0 
Upstreame changelog:

- CRoaring updated to 0.3.1
- `-march=native` is not a default for croaring-sys anymore
- Impl Default for `Bitmap` and `Treemap`
 2021-07-13 15:15:41 +02:00
Marco Neumann 5c16bd2085 chore: cargo update 2021-07-13 15:13:37 +02:00
Andrew Lamb 8b9b369189
chore: Update DataFusion again (#1930) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-07-08 15:14:45 +00:00
Andrew Lamb 7602bde850
chore: Update datafusion deps (#1799) 
* chore: Update datafusion deps + rework code

* refactor: remove workaround as it has been contributed upstream

* fix: Update query/src/exec/split.rs

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-07-08 10:58:32 +00:00
Marco Neumann d6cff911b6 test: ensure that query tests don't rebuild all the time 
Beforehand:

```text
❯ env CARGO_LOG=cargo::core::compiler::fingerprint=info cargo test -p query_tests
[2021-07-05T08:52:13Z INFO  cargo::core::compiler::fingerprint] stale: changed "/home/mneumann/src/influxdb_iox/query_tests/cases"
[2021-07-05T08:52:13Z INFO  cargo::core::compiler::fingerprint]           (vs) "/home/mneumann/src/influxdb_iox/target/debug/build/query_tests-0e8f741dfb84437f/output"
[2021-07-05T08:52:13Z INFO  cargo::core::compiler::fingerprint]                FileTime { seconds: 1625474716, nanos: 436081357 } != FileTime { seconds: 1625474752, nanos: 52625167 }
[2021-07-05T08:52:13Z INFO  cargo::core::compiler::fingerprint] fingerprint error for query_tests v0.1.0 (/home/mneumann/src/influxdb_iox/query_tests)/Test/TargetInner { ..: lib_target("query_tests", ["lib"], "/home/mneumann/src/influxdb_iox/query_tests/src/lib.rs", Edition2018) }
[2021-07-05T08:52:13Z INFO  cargo::core::compiler::fingerprint]     err: current filesystem status shows we're outdated
[2021-07-05T08:52:13Z INFO  cargo::core::compiler::fingerprint] fingerprint error for query_tests v0.1.0 (/home/mneumann/src/influxdb_iox/query_tests)/RunCustomBuild/TargetInner { ..: custom_build_target("build-script-build", "/home/mneumann/src/influxdb_iox/query_tests/build.rs", Edition2018) }
[2021-07-05T08:52:13Z INFO  cargo::core::compiler::fingerprint]     err: current filesystem status shows we're outdated
[2021-07-05T08:52:13Z INFO  cargo::core::compiler::fingerprint] fingerprint error for query_tests v0.1.0 (/home/mneumann/src/influxdb_iox/query_tests)/Build/TargetInner { ..: lib_target("query_tests", ["lib"], "/home/mneumann/src/influxdb_iox/query_tests/src/lib.rs", Edition2018) }
[2021-07-05T08:52:13Z INFO  cargo::core::compiler::fingerprint]     err: current filesystem status shows we're outdated
   Compiling query_tests v0.1.0 (/home/mneumann/src/influxdb_iox/query_tests)
```

The issue is that both the input and the test output files are located
under `cases/`. `build.rs` used `cargo:rerun-if-changed=cases` which per
Cargo doc will scan ALL files in that directory. Note that the normal
`exclude` directive in `Cargo.toml` does NOT work, see
https://github.com/rust-lang/cargo/issues/4587 .

So we need to split input and output files into separate directories
(`cases/{in,out}`).
 2021-07-05 15:30:10 +02:00
Marco Neumann 4ca2d3e148 chore: move persistence windows related code into own crate 
The entire persistence windows data structures (including the
checkpoints) have nothing to do with the mutable buffer per se. So lets
move them into their own crate. This also makes `parquet_file` not
longer depend on `mutable_buffer`.
 2021-07-05 10:23:58 +02:00
Marco Neumann cdab1bed05 feat: persist part+db checkpoint in parquets and catalog 
This will be required for replay on server startup.
 2021-07-05 09:42:46 +02:00
Jacob Marble 0779b0d9bd
feat: add gRPC listener for new write protocol (#1842) 
* feat: add gRPC listener for new write protocol

* chore: clippy happy

* chore: lint

* chore: cargo fmt --all

* chore: cargo clippy

* chore: protobuf-lint

* chore: more formatting

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-07-01 16:15:12 +00:00
Marco Neumann 4204127b05 refactor: use protobuf for in-parquet metadata 2021-06-30 16:51:37 +02:00
Andrew Lamb fef160e24f
feat: Implement data driven query_tests and port explain tests (#1814) 
* feat: Implment data driven query testing and port explain tests

* fix: do not fmt the auto generated cases

* refactor: split setup and parser into separate modules

* refactor: Add log to runner, add end to end tests

* docs: fixu cpmments
 2021-06-29 16:09:51 +00:00
Andrew Lamb 5cc773ad80
chore: update deps to get new arrow (#1831) 2021-06-28 20:31:31 +00:00
kodiakhq[bot] 1bde983d66
Merge branch 'main' into cn/kafka-write 2021-06-24 12:42:44 +00:00
Marko Mikulicic 69b0bb1510
feat: Implement grpc-router crate 
This PR implements the main building block for implementing the gRPC StorageService router.
 2021-06-23 17:21:46 +02:00
Carol (Nichols || Goulding) c66f9e5aeb feat: Write entries to Kafka when configured as the write buffer 2021-06-23 10:48:18 -04:00
kodiakhq[bot] 3fd45d987e
Merge branch 'main' into crepererum/fix_server_startup 2021-06-23 07:17:19 +00:00
Andrew Lamb 4c5007f961
fix: Select the correct timestamp for min/max selectors (#1771) 
* test: Reproducer showing that the min/max selectors are order dependent

* fix: pick correct timestamp for first/last selectors

* refactor: remove println

* docs: Fixup comments and add to link to arrow-datafusion/issues/600

* fix: Add debug if timestamp is null

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-06-22 17:53:54 +00:00
Marco Neumann 55c546baff feat: eagerly check object store during CLI `run` 
Instead of waiting for the server ID to be set and then mark the server
as errored, directly check the object store on startup. This is
important so that we fail fast when Istio isn't up and running yet.
 2021-06-22 18:21:30 +02:00
Carol (Nichols || Goulding) b4644e6108 test: Start of Kafka Write Buffer integration tests 2021-06-21 09:41:35 -04:00
Andrew Lamb ab052c0501
fix: fix flaky test by updating datafusion dep (#1758) 
* chore: update DataFusion dependencies

* fix: Re-enable previously flaking tests

This reverts commit c63ad0ea31.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-06-18 20:17:18 +00:00
kodiakhq[bot] 1d8469951f
Merge branch 'main' into smaller-cache 2021-06-18 18:50:10 +00:00
Marko Mikulicic b612c3af4e
chore: Switch to smaller cache dep 2021-06-18 09:43:28 +02:00
Andrew Lamb ec43a87909
chore: Update itertools deps (#1750) 2021-06-17 17:56:44 +00:00
Andrew Lamb c5eea9af6a
feat: Implement DeduplicateExec (#1733) 
* feat: Implement DeduplicateExec

* fix: Doc comments

* fix: fix comment

* fix: Update with arrow ticket references and use datafusion coalsce batches impl

* refactor: rename inner.rs to algo.rs

* docs: Add additional documentation on rationale for last field value

* docs: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* docs: Update query/src/provider/deduplicate/algo.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* docs: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* refactor: do not use pub(crate)

* docs: fix test comments

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
 2021-06-17 14:17:52 +00:00
Paul Dix 802ecacf61 feat: Add persistence windows ingestion tracking 
This adds a new module, persistence_windows, to the mutable buffer crate. Later PRs will add this into the mutable buffer chunk where it can be used to track when the lifecycle for persistence should be triggered.
 2021-06-16 15:28:37 -04:00
Marko Mikulicic 760bcde3f0
feat: Factor out tracing/logging CLI options 
This PR factors out the tracing/logging CLI optinos into the `trogging` utility crate,
so that multiple binaries from the IOx suite (such as conductor) can use the same (and quite complex)
logging/tracing configuration options (flags and env vars).

Closes influxdata/conductor#343
 2021-06-16 00:54:11 +02:00
Raphael Taylor-Davies 38d17a3093
chore: remove unused query dependency (#1731) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-06-15 22:06:13 +00:00
Andrew Lamb 2c8060160f
chore: update datafusion deps (#1721) 
* chore: update datafusion deps

* chore: update deps

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-06-15 19:06:15 +00:00
Raphael Taylor-Davies bf54ab51f2
refactor: split lifecycle into separate crate (#1730) 2021-06-15 15:57:47 +00:00
kodiakhq[bot] 4dc338e1bc
Merge branch 'main' into trogging 2021-06-15 13:04:06 +00:00
Marko Mikulicic bde35cf5be
chore: Pull tracing+logging setup in its own crate 
1. so that it can be reused by other binaries (e.g. conductor)
2. so that it's faster to build when working on it.
 2021-06-15 14:52:04 +02:00
Raphael Taylor-Davies dd422492e2
feat: sort order in schema (#1357) (#1667) 
* feat: sort order in schema (#1357)

* chore: review feedback

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-06-14 18:10:41 +00:00
kodiakhq[bot] fc1b5ea165
Merge branch 'main' into crepererum/parquet_metadata_wrapper 2021-06-14 11:20:39 +00:00
Marco Neumann 518f7c6f15 refactor: wrap upstream parquet MD into struct + clean up interface 
This prevents users from `parquet_file::metadata` to also depend on
`parquet` directly. Furthermore they don't need to important dozend of
functions and can instead just use `IoxParquetMetaData` directly.
 2021-06-14 13:17:01 +02:00
Andrew Lamb 0d8d32fd8f
chore: Update deps to get latest arrow (#1708) 
* chore: Update deps to get latest arrow

* fix: Update to rust 1.52

* fix: clippy
 2021-06-14 11:08:09 +00:00
Marco Neumann 898c638630 feat: wire up catalog checkpointing 
Closes #1381.
 2021-06-14 10:08:32 +02:00
Marko Mikulicic 5a68abaa53
feat: Implement LayeredTracing 
__Rationale__

We currently use the `tracing` framework to output to both log outputs (e.g. stdout for k8s) and distributed tracing collectors (e.g. opentelemetry jaeger).

However, due to a limitation in the `tracing` SDK, we can only have one "filter" level that applies
to both logs and tracing outputs. This is unpractical because tracing collectors are designed
to receive high verbosity data (which will be then sampled within the opentelemetry library),
while logs generally are limited to the DEBUG level on production.

This PR adds a `FilteredLayer` tracing subscriber layer, that wraps a subscriber layer with a independent
filter, which can filter events goint to the wrapper subscriber layer more agressively than the global layer.

This will allow us to emit logs at INFO or DEBUG level while passing all events to opentelemetry at TRACE
level (and opentelemetry SDK will then sample the events so that only a small part will be sent to the
ot collector)

__Note__

This PR just implements the `FilteredLayer` and a test. Another PR will integrate this with
our log/tracing setup code.
 2021-06-11 04:32:47 +02:00
Raphael Taylor-Davies 11b25b3aaf
refactor: swap order of partition and table in in-memory catalog (#1678) 
* refactor: swap order of partition and table in in-memory catalog

* chore: review feedback

* chore: validate panic message

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-06-10 16:40:30 +00:00
Marco Neumann 876f642860 test: add benchmark for catalog loading 2021-06-10 15:42:21 +02:00
Andrew Lamb 29dadba4f3
chore: update dependencies (#1673) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-06-10 13:01:31 +00:00