Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
8,663 Commits
188 Branches
453 Tags
378 MiB
Commit Graph

346 Commits (b21799acaea38be920b1cc5942a3243abd37dbbf)

Author SHA1 Message Date
Carol (Nichols || Goulding) 8f3e44bf76
refactor: Extract a crate for shared data types in the new design 2022-03-02 12:16:15 -05:00
Raphael Taylor-Davies 43ada68f37
chore: reset release codegen-units to default (#3883) 2022-03-01 13:44:14 +00:00
Marco Neumann 9079e6ddb0
feat: backoff retries in ingester (#3841) 
* feat: add `backoff` crate

* feat: backoff retries in ingester

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-23 17:58:16 +00:00
Marco Neumann 459974d1b8
refactor: move `DedicatedExecutor` from `query` to `executor` (#3819) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-23 09:10:43 +00:00
Luke Bond 0f012de70c
feat: adding compactor CLI command and crate 
Closes: #3777
 2022-02-21 12:24:09 +00:00
Luke Bond a66e29e5b3
chore: port sqlx-hotswap-pool over from conductor (#3750) 
* chore: port sqlx-hotswap-pool over from conductor

Co-authored-by: Marko Mikulicic <mkm@influxdata.com>

* chore: workspace hack fixes

* fix: unique schema per test db connection

* fix: adjust search path in catalog pg tests to see if it fixes test schema issue

* fix: actually fixed sqlx hotswap pool test

Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-15 16:18:36 +00:00
Luke Bond 54890525da
feat: gitops adapter ci build (#3692) 
* chore: remove references to perf_image in CI

* chore: adding gitops adapter image build in CI

* chore: gitops adapter bin now same as dir & package so docker build works

* fix: circle config package change after renaming gitops adapter package
 2022-02-10 11:23:06 +00:00
Carol (Nichols || Goulding) 73828323ac
feat: Ingester Flight gRPC API (#3623) 
* feat: Add a way to run ingester with an in-memory catalog from the CLI

If you set the --catalog-dsn string to "mem", rather than using that as
a Postgres connection URL, create an in-memory catalog.

Planning on using this in tests, so not documenting.

* fix: Set default topic to the same value as SHARED_KAFKA_TOPIC

Namely, both should use an underscore. I don't think there's a way to
directly share these values between a constant and an annotation.

* feat: Add a flight API (handshake only) to ingester

* fix: Create partitions if using file-based write buffer

* fix: Change the server fixture to handle ingester server type

For now, the ingester doesn't implement the deployment API. Not sure if
it should or not.

* feat: Start implementing ingester do_get, namely decoding the query

Skip serialization of the predicate for the moment.

* refactor: Rename ingest protos to ingester to match crate name

* refactor: Rename QueryResults to QueryData

* feat: Move ingester flight client to new querier crate

* fix: Off by one error, different starting indexes in sequencers

* fix: Create new CLI argument to pick the catalog type

* fix: Create a CLI option to set the number of topics to auto-create in the write buffer

* fix: Check the arrow flight service's health to tell that the ingester gRPC is up

* fix: Set postgres as the default catalog type

* fix: Return an error rather than panicking if CLI args aren't right
 2022-02-09 19:07:44 +00:00
Luke Bond de2a013786
feat: gitops adapter (#3656) 
* feat: scaffolding of gitops adapter bin crate

* chore: refactor gitops adapter; calls CLI now; status update fixes

* feat: gitops adapter now calls out to CLI once per topic; improved tests

* chore: add mock failure script for gitops adapter

* chore: update workspace-hack

* chore: refactor away unecessary to_string in gitops syncer

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-08 13:27:36 +00:00
Edd Robinson c1f5994660
refactor: move sql parsing -> RPC predicate into own crate (#3604) 
* chore: create crate

* refactor: move module to new crate
 2022-02-02 10:41:57 +00:00
NGA-TRAN ef336b4659 feat: add ingester crate and a few basic data structures for its data lifecycle 2022-01-17 15:38:03 -05:00
Paul Dix 4764e71c54 feat: Add initial iox_catalog skeleton 2022-01-17 09:56:20 -05:00
Dom a8cb8755de feat: new router2 crate 
This commit adds an almost-empty router2 crate containing enough of a
skeleton to plumb into the IOx CLI/server runner.
 2022-01-12 14:43:10 +00:00
Carol (Nichols || Goulding) f9174c483b
refactor: Extract server::db into its own crate (#3417) 
* refactor: Extract JobRegistry from the server crate

Both the server crate and a db crate that I'm about to extract depend on
JobRegistry, so to avoid making circular dependencies, extract the
JobRegistry to its own crate.

* refactor: Move db out of server into its own crate

Fixes #2821.
 2021-12-23 22:01:17 +00:00
Carol (Nichols || Goulding) 183de4519f
feat: Add incremental = true to the quick-release profile 2021-12-08 09:53:19 -05:00
Carol (Nichols || Goulding) 255cf79104
docs: Explain tradeoffs and use cases of the different release profiles 2021-12-08 09:53:19 -05:00
Carol (Nichols || Goulding) 758a17e199
feat: Create a build-time-optimized release build profile 
Fixes #3316.
 2021-12-08 09:53:19 -05:00
Carol (Nichols || Goulding) 994938b2f0
fix: Alphabetize/organize deps 2021-12-06 09:36:30 -05:00
Marco Neumann f2f090bde7 feat: test bench 2021-11-25 18:14:40 +01:00
Carol (Nichols || Goulding) f66cd681c2
feat: Results of using cargo hakari init workspace-hack 2021-11-19 09:20:35 -05:00
Carol (Nichols || Goulding) a2454b542d
fix: Small cleanups in Cargo.tomls (#3160) 
* fix: Add tokio rt-multi-thread feature so cargo test -p client_util compiles

* fix: Alphabetize dependencies

* fix: Add the data_types_conversions feature to get tests passing

* fix: Remove dev dependencies already listed under normal dependencies

* fix: Make sure the workspace is using the new resolver
 2021-11-18 22:26:33 +00:00
kodiakhq[bot] 49cfa046bb
Merge branch 'main' into crepererum/smaller_binaries 2021-11-12 11:17:09 +00:00
Raphael Taylor-Davies 6f268f8260
refactor: extract DML types (#2731) (#3084) 
* refactor: extract DML types (#2731)

* chore: fmt
 2021-11-11 12:34:07 +00:00
Marco Neumann 16fc610433 chore: faster and smaller IOx binaries 
Use `codegen-units = 1`, thin-LTO and debug section compression to make our binary smaller (which is good for deploy and
test times) and faster.

# Summary

The binary size of `influxdb_iox` after building with:

```console
$ cargo build --release --no-default-features --features="aws,gcp,azure,jemalloc_replacing_malloc"
```

The profile was:

```toml
[profile.release]
debug = true
```

The commit was:

```text
89ece8b493
```

The size results are:

| Method                                     | Size  |
| ------------------------------------------ | ----- |
| baseline                                   | 833MB |
| baseline + dbg compression                 | 222MB |
| baseline + strip                           |  49MB |
| codegen-units                              | 520MB |
| codegen-units + strip                      |  40MB |
| codegen-units + dbg compression            | 143MB |
| thin LTO                                   | 715MB |
| thin LTO + strip                           |  49MB |
| thin LTO + dbg compression                 | 199MB |
| codegen-units + thin LTO                   | 449MB |
| codegen-units + thin LTO + strip           |  40MB |
| codegen-units + thin LTO + dbg compression | 130MB |

For the methods that were successfully measured I couldn't really see any compile time differences on my laptop.

# Methods

## Strip
Remove debug symbols. We don't really want this, so this is just to get an idea of the size

```console
$ strip baseline
```

## Debug Sections compression
Debug sections make a large amount of our binary size (a stripped executable is 49MB instead of 833MB). Since we like to
have debug symbols we cannot just strip them. However these symbols are only used for:

- backtrace generation (something went wrong, not BAU)
- profiling
- debugging

So in normal operation and most test scenarios, we're just wasting memory. So we could compress them:

```console
$ objcopy --compress-debug-sections baseline baseline-dbg_compressed
```

There is also elfutils:

```console
$ eu-elfcompress test
```

Elfutils nearly ends up with the same size (220MB instead of 222MB that objcopy achieves), but takes more time and is
probably not worth it.

Note that compressed debug sections exist since many years. The Rust ecosystem supports reading them since over a year,
see:

- <https://github.com/gimli-rs/gimli/issues/195>
- <https://github.com/rust-lang/backtrace-rs/issues/342>

## Codegen Units
The rust compiler parallelizes codegen work. This split into units however means that optimizations are somewhat
limited. This can be change by:

```toml
[profile.release]
...
codegen-units = 1
```

As a nice side effect this should also make our code faster.

## Thin LTO
Get LLVM to run "thin" Link Time Optimization:

```toml
[profile.release]
...
lto = "thin"
```

As a nice side effect this should also make our code faster.

## Fat LTO
Get LLVM to run "fat" Link Time Optimization:

```toml
[profile.release]
...
lto = "fat"
```

There are no results for this because this took a massive amount of memory and CPU time and did not finish on my system.
 2021-11-10 15:49:19 +01:00
Raphael Taylor-Davies 60f0deaf1e
feat: remove flatbuffer entry (#3045) 2021-11-05 20:19:24 +00:00
Raphael Taylor-Davies 405ed6db1b
feat: add mutable_batch_tests (#2724) (#3011) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-03 18:15:35 +00:00
Marco Neumann a91e572b43 feat: implement basic router server type 
This only implements the deployment service and nothing else yet.
 2021-11-03 16:34:51 +01:00
Raphael Taylor-Davies 3010ff4176
feat: write entry to MutableBatch (#2972) 
* feat: write entry to MutableBatch

* feat: error on empty TableBatch

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-27 14:05:22 +00:00
Marco Neumann fc651ea193 chore: move `gprc-router-test-gen` to top level 2021-10-26 11:02:33 +02:00
Marco Neumann a0cdca7781 chore: move `influxdb_iox` into a proper workspace package 2021-10-26 11:02:33 +02:00
Marco Neumann 2833cefc12 test: remove Kafka from end2end tests 
Kafka is now sufficiently tested via the `write_buffer` crate. The
end2end tests can now use the in-memory mock implementation or -- if
servers can only be controlled via CLI -- the file-based implementation.
 2021-10-26 10:09:34 +02:00
Marco Neumann 81ccf85104 refactor: integrate print-cpu into main binary 
It's a less confusing structure and also allows us to have this
information available in our shipped binary.
 2021-10-25 18:29:32 +02:00
Marco Neumann 7a28982e2a fix: fix dependency features for `influxdb_iox` 2021-10-25 11:15:19 +02:00
Marco Neumann bc7244c48e chore: use Rust edition 2021 2021-10-25 10:58:20 +02:00
Raphael Taylor-Davies d2b41c5a13
feat: write pbdata to MutableBatch (#2724) (#2927) 
* feat: write pbdata to MutableBatch (#2724)

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-21 14:32:35 +00:00
Raphael Taylor-Davies f64e749eb9
feat: write line protocol to MutableBatch (#2914) 
* feat: write line protocol to MutableBatch

* chore: fix lint

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-21 08:12:54 +00:00
Andrew Lamb 9974a5364c
chore(security): Replace prettytable with comfy-table (#2905) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-20 10:44:36 +00:00
Andrew Lamb a82dc6f5f0
chore: Update datafusion + arrow (#2903) 
* chore: Update datafusion to latest, arrow to 6.0.0

* fix: Update tests

* fix: bubble internal error

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-19 17:14:08 +00:00
kodiakhq[bot] 0be89715e2
Merge branch 'main' into dependabot/cargo/rdkafka-0.27.0 2021-10-19 12:55:21 +00:00
Marco Neumann 7dafb0ff74 chore: remove unused `influxdb_iox` => `synchronized-writer` dep 2021-10-19 14:46:26 +02:00
dependabot[bot] 32e18b6436
chore(deps): bump rdkafka from 0.26.0 to 0.27.0 
Bumps [rdkafka](https://github.com/fede1024/rust-rdkafka) from 0.26.0 to 0.27.0.
- [Release notes](https://github.com/fede1024/rust-rdkafka/releases)
- [Changelog](https://github.com/fede1024/rust-rdkafka/blob/master/changelog.md)
- [Commits](https://github.com/fede1024/rust-rdkafka/compare/v0.26.0...v0.27.0)

---
updated-dependencies:
- dependency-name: rdkafka
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-19 11:49:34 +00:00
Marco Neumann 157b556d4c
feat: make flight responses streaming (#2876) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-18 15:01:50 +00:00
dependabot[bot] 70555ab33d
chore(deps): bump structopt from 0.3.23 to 0.3.25 (#2877) 
Bumps [structopt](https://github.com/TeXitoi/structopt) from 0.3.23 to 0.3.25.
- [Release notes](https://github.com/TeXitoi/structopt/releases)
- [Changelog](https://github.com/TeXitoi/structopt/blob/master/CHANGELOG.md)
- [Commits](https://github.com/TeXitoi/structopt/compare/v0.3.23...v0.3.25)

---
updated-dependencies:
- dependency-name: structopt
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
 2021-10-18 14:49:15 +00:00
Raphael Taylor-Davies bdd6d67e7a
refactor: split out mutable_batch crate (#2841) 
* refactor: split out mutable_batch crate

* refactor: restore chunk module for better diffs

* chore: fmt

* chore: review feedback

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-14 16:46:54 +00:00
Marco Neumann 28195b9c0c chore: new `parquet_catalog` crate 2021-10-14 14:34:59 +02:00
Raphael Taylor-Davies 0554173684
feat: migrate write buffer to TimeProvider (#2722) (#2804) 
* feat: migrate write buffer to TimeProvider (#2722)

* chore: review feedback

Co-authored-by: Marco Neumann <marco@crepererum.net>

Co-authored-by: Marco Neumann <marco@crepererum.net>
 2021-10-12 10:32:34 +00:00
Raphael Taylor-Davies 7bd32a4916
feat: add time provider abstraction (#2722) (#2792) 
* feat: add time provider abstraction (#2722)

* chore: tests

* chore: more tests

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-11 15:00:29 +00:00
dependabot[bot] 61190a9d08
chore(deps): bump predicates from 2.0.2 to 2.0.3 
Bumps [predicates](https://github.com/assert-rs/predicates-rs) from 2.0.2 to 2.0.3.
- [Release notes](https://github.com/assert-rs/predicates-rs/releases)
- [Changelog](https://github.com/assert-rs/predicates-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/predicates-rs/compare/v2.0.2...v2.0.3)

---
updated-dependencies:
- dependency-name: predicates
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-11 14:32:08 +00:00
Raphael Taylor-Davies afe34751e7
refactor: split out schema crate (#2781) 
* refactor: split out schema crate

* chore: fix doc
 2021-10-11 09:45:08 +00:00
dependabot[bot] 1327735537
chore(deps): bump thiserror from 1.0.29 to 1.0.30 
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.29 to 1.0.30.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.29...1.0.30)

---
updated-dependencies:
- dependency-name: thiserror
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-11 08:01:21 +00:00
dependabot[bot] 01a44899fd
chore(deps): bump assert_cmd from 2.0.1 to 2.0.2 
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd) from 2.0.1 to 2.0.2.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases)
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.1...v2.0.2)

---
updated-dependencies:
- dependency-name: assert_cmd
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-11 01:45:35 +00:00
dependabot[bot] 7a6f381e69
chore(deps): bump arrow-flight from 5.4.0 to 5.5.0 
Bumps [arrow-flight](https://github.com/apache/arrow-rs) from 5.4.0 to 5.5.0.
- [Release notes](https://github.com/apache/arrow-rs/releases)
- [Changelog](https://github.com/apache/arrow-rs/blob/5.5.0/CHANGELOG.md)
- [Commits](https://github.com/apache/arrow-rs/compare/5.4.0...5.5.0)

---
updated-dependencies:
- dependency-name: arrow-flight
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-04 13:27:26 +00:00
kodiakhq[bot] 181145eca1
Merge branch 'main' into dependabot/cargo/arrow-5.5.0 2021-10-04 13:10:42 +00:00
dependabot[bot] 6d9205d87c chore(deps): bump parquet from 5.4.0 to 5.5.0 
Bumps [parquet](https://github.com/apache/arrow-rs) from 5.4.0 to 5.5.0.
- [Release notes](https://github.com/apache/arrow-rs/releases)
- [Changelog](https://github.com/apache/arrow-rs/blob/5.5.0/CHANGELOG.md)
- [Commits](https://github.com/apache/arrow-rs/compare/5.4.0...5.5.0)

---
updated-dependencies:
- dependency-name: parquet
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-04 12:15:11 +01:00
dependabot[bot] d1f5209869
chore(deps): bump arrow from 5.4.0 to 5.5.0 
Bumps [arrow](https://github.com/apache/arrow-rs) from 5.4.0 to 5.5.0.
- [Release notes](https://github.com/apache/arrow-rs/releases)
- [Changelog](https://github.com/apache/arrow-rs/blob/5.5.0/CHANGELOG.md)
- [Commits](https://github.com/apache/arrow-rs/compare/5.4.0...5.5.0)

---
updated-dependencies:
- dependency-name: arrow
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-04 08:55:38 +00:00
Raphael Taylor-Davies 86cee568d5
feat: use upstream pbjson (#2650) 
* feat: use upstream pbjson

* chore: fmt
 2021-09-28 16:29:26 +00:00
Raphael Taylor-Davies daa2ec2f4c
revert: "fix: Revert "feat: remove routerify (#2586)" (#2623)" (#2649) 
This reverts commit 18941fcbff.
 2021-09-28 14:42:17 +00:00
Andrew Lamb 18941fcbff
fix: Revert "feat: remove routerify (#2586)" (#2623) 
This reverts commit 9fba7b0ba0.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-09-24 09:50:24 +00:00
Raphael Taylor-Davies 92e6173f49
feat: implement jaeger-agent protocol directly (#2607) 
* feat: implement jaeger-agent protocol directly

* chore: review feedback

* fix: remove jaeger feature flag
 2021-09-22 17:30:37 +00:00
Raphael Taylor-Davies 46088a7ff1
feat: disable rustyline dirs-next dependency (#2579) 2021-09-20 14:42:07 +00:00
Raphael Taylor-Davies 9fba7b0ba0
feat: remove routerify (#2586) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-09-20 08:57:19 +00:00
dependabot[bot] 1112597c44
chore(deps): bump dirs from 3.0.2 to 4.0.0 
Bumps [dirs](https://github.com/soc/dirs-rs) from 3.0.2 to 4.0.0.
- [Release notes](https://github.com/soc/dirs-rs/releases)
- [Commits](https://github.com/soc/dirs-rs/commits)

---
updated-dependencies:
- dependency-name: dirs
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-09-20 08:28:04 +00:00
Raphael Taylor-Davies 3ffb16daa6
feat: remove parse_duration (#2574) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-09-19 11:27:24 +00:00
Raphael Taylor-Davies 1d55d9a1b5
feat: add pbjson support (#2468) 
* feat: add pbjson support

* chore: fix test
 2021-09-16 07:33:27 +00:00
Raphael Taylor-Davies c66095cad1
feat: remove metrics crate (#2552) 2021-09-15 19:43:33 +00:00
Raphael Taylor-Davies 6e7fa3e574
feat: migrate http ingest metrics (#2542) 
* feat: migrate http ingest metrics

* chore: review feedback

* refactor: RAII entry ingest recorder
 2021-09-15 19:01:10 +00:00
Marco Neumann bfaba78dc3 refactor: move `predicate` into its own crate 
Two reasons:

1. I wanna decouple `parquet_file` from `query` (nearly done, needs a
   small follow-up PR).
2. `predicate` will have more and more features (like serialization)
   which justifies a new home
 2021-09-14 17:13:02 +02:00
Marco Neumann 3f2e46c397 feat: prune old transactions from preserved catalog 2021-09-14 12:08:17 +02:00
Raphael Taylor-Davies e07bc85670
feat: enable tracing for http API (#2273) (#2498) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-09-10 11:21:02 +00:00
dependabot[bot] b67610d9b9
chore(deps): bump tokio from 1.10.1 to 1.11.0 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.10.1 to 1.11.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.10.1...tokio-1.11.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-09-06 09:11:38 +00:00
kodiakhq[bot] 06311970fa
Merge branch 'main' into dependabot/cargo/parquet-5.3.0 2021-09-06 09:00:27 +00:00
dependabot[bot] ccfd6ad3da
chore(deps): bump parquet from 5.2.0 to 5.3.0 
Bumps [parquet](https://github.com/apache/arrow-rs) from 5.2.0 to 5.3.0.
- [Release notes](https://github.com/apache/arrow-rs/releases)
- [Changelog](https://github.com/apache/arrow-rs/blob/5.3.0/CHANGELOG.md)
- [Commits](https://github.com/apache/arrow-rs/compare/5.2.0...5.3.0)

---
updated-dependencies:
- dependency-name: parquet
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-09-06 08:52:30 +00:00
kodiakhq[bot] 0fd92c8b45
Merge branch 'main' into dependabot/cargo/structopt-0.3.23 2021-09-06 08:51:32 +00:00
dependabot[bot] b1bb390893
chore(deps): bump parking_lot from 0.11.1 to 0.11.2 
Bumps [parking_lot](https://github.com/Amanieu/parking_lot) from 0.11.1 to 0.11.2.
- [Release notes](https://github.com/Amanieu/parking_lot/releases)
- [Changelog](https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Amanieu/parking_lot/compare/0.11.1...0.11.2)

---
updated-dependencies:
- dependency-name: parking_lot
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-09-06 01:18:24 +00:00
dependabot[bot] 2cf9505029
chore(deps): bump structopt from 0.3.22 to 0.3.23 
Bumps [structopt](https://github.com/TeXitoi/structopt) from 0.3.22 to 0.3.23.
- [Release notes](https://github.com/TeXitoi/structopt/releases)
- [Changelog](https://github.com/TeXitoi/structopt/blob/master/CHANGELOG.md)
- [Commits](https://github.com/TeXitoi/structopt/compare/v0.3.22...v0.3.23)

---
updated-dependencies:
- dependency-name: structopt
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-09-06 01:17:41 +00:00
Marco Neumann 09a5d27434 chore: ignore massif profiler outputs 2021-09-02 15:58:43 +02:00
Marco Neumann 4a863993ec feat: "dump catalog" debug CLI 2021-09-02 08:08:20 +02:00
Jacob Marble 6ba25728e1
chore: exclude non-source code from package (#2452) 
Before this change, modifications to the following files would trigger a
rebuild on `cargo build`.

```
2,13d1
< .circleci/config.yml
< .circleci/get-deploy-tags.sh
< .editorconfig
< .gitattributes
< .github/ISSUE_TEMPLATE/bug_report.md
< .github/ISSUE_TEMPLATE/feature_request.md
< .github/PULL_REQUEST_TEMPLATE.md
< .github/dependabot.yml
< .github/semantic.yml
< .gitignore
< .kodiak.toml
< CONTRIBUTING.md
17,22d4
< Dockerfile
< Dockerfile.dockerignore
< LICENSE-APACHE
< LICENSE-MIT
< README.md
< buf.yaml
24,76d5
< docker/Dockerfile.ci
< docker/Dockerfile.ci.dockerignore
< docker/Dockerfile.ci.integration
< docker/Dockerfile.ci.integration.dockerignore
< docker/Dockerfile.iox
< docker/Dockerfile.iox.dockerignore
< docker/ci-kafka-docker-compose.yml
< docker/integration_test.sh
< docs/README.md
< docs/catalog_persistence.md
< docs/data_management.md
< docs/drawings/data_lifecycle.monopic
< docs/drawings/data_organization.monopic
< docs/encoding_thoughts.md
< docs/env.example
< docs/images/flame_graph.png
< docs/local_filesystems.md
< docs/metrics.md
< docs/multi_core_tasks.md
< docs/profiling.md
< docs/protobuf.md
< docs/regenerating_flatbuffers.md
< docs/server_startup.md
< docs/sql.md
< docs/style_guide.md
< docs/testing.md
< docs/tracing.md
< docs/valgrind.md
< perf/logs/iox_router.log
< perf/logs/iox_writer.log
< perf/logs/jaeger.log
< perf/logs/kafka.log
< perf/logs/minio.log
< perf/logs/zookeeper.log
< perf/volumes/minio/.minio.sys/buckets/.bloomcycle.bin
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/.bloomcycle.bin/fs.json
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/.usage-cache.bin/fs.json
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/.usage.json/fs.json
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/iox1/.usage-cache.bin/fs.json
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/iox2/.usage-cache.bin/fs.json
< perf/volumes/minio/.minio.sys/buckets/.tracker.bin
< perf/volumes/minio/.minio.sys/buckets/.usage-cache.bin
< perf/volumes/minio/.minio.sys/buckets/.usage.json
< perf/volumes/minio/.minio.sys/buckets/iox1/.metadata.bin
< perf/volumes/minio/.minio.sys/buckets/iox1/.usage-cache.bin
< perf/volumes/minio/.minio.sys/buckets/iox2/.metadata.bin
< perf/volumes/minio/.minio.sys/buckets/iox2/.usage-cache.bin
< perf/volumes/minio/.minio.sys/config/config.json
< perf/volumes/minio/.minio.sys/config/iam/format.json
< perf/volumes/minio/.minio.sys/format.json
< perf/volumes/zookeeper/data/myid
< perf/volumes/zookeeper/data/version-2/log.1
< perf/volumes/zookeeper/data/version-2/snapshot.0
79,87d7
< scripts/edit_db_rules
< scripts/genlp.py
< scripts/git-log-prs
< scripts/grpcurl
< scripts/jq_edit_db_rules
< scripts/logfmts
< scripts/prototxt
< scripts/valgrind.supp
< scripts/valigrind_leak
161d80
< tools/iplan
```
 2021-08-31 17:14:02 +00:00
dependabot[bot] 3e154b3b02
chore(deps): bump serde_json from 1.0.66 to 1.0.67 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.66 to 1.0.67.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.66...v1.0.67)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-08-30 11:28:35 +00:00
dependabot[bot] 52fbaef2df
chore(deps): bump thiserror from 1.0.26 to 1.0.28 
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.26 to 1.0.28.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.26...1.0.28)

---
updated-dependencies:
- dependency-name: thiserror
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-08-30 11:05:11 +00:00
dependabot[bot] 13161de003
chore(deps): bump rustyline from 8.2.0 to 9.0.0 
Bumps [rustyline](https://github.com/kkawakam/rustyline) from 8.2.0 to 9.0.0.
- [Release notes](https://github.com/kkawakam/rustyline/releases)
- [Changelog](https://github.com/kkawakam/rustyline/blob/master/History.md)
- [Commits](https://github.com/kkawakam/rustyline/compare/v8.2.0...v9.0.0)

---
updated-dependencies:
- dependency-name: rustyline
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-08-30 01:19:49 +00:00
Raphael Taylor-Davies fcec394a28
feat: connect up new metrics (#2428) 2021-08-27 12:55:35 +00:00
Raphael Taylor-Davies 3a26f7857f
feat: add new metrics abstraction (#2076) (#2365) 
* feat: add new metrics abstraction (#2076)

* chore: fix lint

* feat: add tests

feat: histogram export

* chore: further docs

* chore: docs

* chore: log on prometheus export error

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-27 11:54:23 +00:00
Raphael Taylor-Davies 1773bf5d37
feat: add storage client to influxdb_iox_client (#2404) 
* feat: add storage client to influxdb_iox_client

* chore: fix type_url

* refactor: split storage into separate crate

* chore: fix doctest

* chore: review feedback

* chore: add generated_types cleanup ticket

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-26 10:30:59 +00:00
Raphael Taylor-Davies 2c9c191b17
refactor: split out trace http (#2388) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-24 13:37:20 +00:00
Raphael Taylor-Davies 3fdc0e9a6a
chore: remove orphaned code from trogging (#2371) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-24 10:12:59 +00:00
Raphael Taylor-Davies 58716dee71
refactor: remove mem-qe (#2344) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-23 14:35:12 +00:00
dependabot[bot] 0d4bc4190f
chore(deps): bump assert_cmd from 1.0.8 to 2.0.0 
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd) from 1.0.8 to 2.0.0.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases)
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v1.0.8...v2.0.0)

---
updated-dependencies:
- dependency-name: assert_cmd
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-08-23 01:17:06 +00:00
Marko Mikulicic bbbd38bb12
fix(heappy): Prevent re-entering a running heap profile 
Upgrades heappy to include https://github.com/mkmik/heappy/pull/2
and adapts to the API change.
 2021-08-20 14:36:36 +02:00
Marko Mikulicic 4a2f0b0a52
fix: Make heappy and nonheappy mutually exclusive 
Closes #2288
 2021-08-20 11:35:07 +02:00
Raphael Taylor-Davies db97069ecd
feat: jaeger span export (#2273) (#2348) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-20 08:57:16 +00:00
Paul Dix d5f01a2a68 refactor: move data generator to IOx repo and fix build 2021-08-19 14:26:15 -04:00
Raphael Taylor-Davies 04ba139cb2
refactor: split trace crate (#2343) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-19 11:00:32 +00:00
Andrew Lamb 5e1cb244f7
feat: make pprof optional (#2331) 2021-08-18 15:13:37 +00:00
Marko Mikulicic a50b1646bd
fix(heappy): Report free as positive numbers 2021-08-18 14:30:41 +02:00
Marko Mikulicic fc473995fa
feat(iox): Upgrade heappy to present free memory events 
Currently heappy is reporting "bytes allocated" and "inuse_bytes" ("bytes allocated - freed bytes").

Since the allocation site and free site can be wildly different, it's very hard to correlate these events
and have a meaningful "inuse_bytes" graph.

I have a plan for how to fix this for good, but in the meantime
I'd like to report raw "freed bytes".
 2021-08-18 13:04:48 +02:00
Raphael Taylor-Davies 1d6a8703af
feat: support sinking traces to an OTEL SpanExporter (#2319) 
* feat: support sinking traces to an OTEL SpanExporter

* chore: consistent logging

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

* chore: review feedback

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-18 08:05:21 +00:00
Raphael Taylor-Davies efa776bd03
feat: fix log crate output (#2325) 
* feat: fix log crate output

* chore: fix doc

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-17 16:19:22 +00:00
Marko Mikulicic 94ff77126e
feat: Enable freed memory tracking in heappy 
In aed37ab50a I fixed the main
problem that rendered free tracking useless: the allocator was actually allocating more bytes than requested; we were tracking the amount of memory requested by the user and not the size of the block that was returned to the user. However, when we were tracking a call to free we used the size of the memory block which was bigger than the amount that was tracked; this led to negative numbers of "in use memory".
 2021-08-17 15:09:28 +02:00
dependabot[bot] 1d85a3fef6
build(deps): bump predicates from 1.0.8 to 2.0.2 
Bumps [predicates](https://github.com/assert-rs/predicates-rs) from 1.0.8 to 2.0.2.
- [Release notes](https://github.com/assert-rs/predicates-rs/releases)
- [Changelog](https://github.com/assert-rs/predicates-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/predicates-rs/compare/v1.0.8...v2.0.2)

---
updated-dependencies:
- dependency-name: predicates
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-08-16 15:36:49 +00:00