61190a9d08
chore(deps): bump predicates from 2.0.2 to 2.0.3
...
Bumps [predicates](https://github.com/assert-rs/predicates-rs ) from 2.0.2 to 2.0.3.
- [Release notes](https://github.com/assert-rs/predicates-rs/releases )
- [Changelog](https://github.com/assert-rs/predicates-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/predicates-rs/compare/v2.0.2...v2.0.3 )
---
updated-dependencies:
- dependency-name: predicates
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-11 14:32:08 +00:00
afe34751e7
refactor: split out schema crate ( #2781 )
...
* refactor: split out schema crate
* chore: fix doc
2021-10-11 09:45:08 +00:00
1327735537
chore(deps): bump thiserror from 1.0.29 to 1.0.30
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.29 to 1.0.30.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.29...1.0.30 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-11 08:01:21 +00:00
01a44899fd
chore(deps): bump assert_cmd from 2.0.1 to 2.0.2
...
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd ) from 2.0.1 to 2.0.2.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases )
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.1...v2.0.2 )
---
updated-dependencies:
- dependency-name: assert_cmd
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-11 01:45:35 +00:00
7a6f381e69
chore(deps): bump arrow-flight from 5.4.0 to 5.5.0
...
Bumps [arrow-flight](https://github.com/apache/arrow-rs ) from 5.4.0 to 5.5.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/5.5.0/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/5.4.0...5.5.0 )
---
updated-dependencies:
- dependency-name: arrow-flight
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-04 13:27:26 +00:00
181145eca1
Merge branch 'main' into dependabot/cargo/arrow-5.5.0
2021-10-04 13:10:42 +00:00
6d9205d87c
chore(deps): bump parquet from 5.4.0 to 5.5.0
...
Bumps [parquet](https://github.com/apache/arrow-rs ) from 5.4.0 to 5.5.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/5.5.0/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/5.4.0...5.5.0 )
---
updated-dependencies:
- dependency-name: parquet
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-04 12:15:11 +01:00
d1f5209869
chore(deps): bump arrow from 5.4.0 to 5.5.0
...
Bumps [arrow](https://github.com/apache/arrow-rs ) from 5.4.0 to 5.5.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/5.5.0/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/5.4.0...5.5.0 )
---
updated-dependencies:
- dependency-name: arrow
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-04 08:55:38 +00:00
86cee568d5
feat: use upstream pbjson ( #2650 )
...
* feat: use upstream pbjson
* chore: fmt
2021-09-28 16:29:26 +00:00
daa2ec2f4c
revert: "fix: Revert "feat: remove routerify ( #2586 )" ( #2623 )" ( #2649 )
...
This reverts commit 18941fcbff
2021-09-28 14:42:17 +00:00
18941fcbff
fix: Revert "feat: remove routerify ( #2586 )" ( #2623 )
...
This reverts commit 9fba7b0ba0
2021-09-24 09:50:24 +00:00
92e6173f49
feat: implement jaeger-agent protocol directly ( #2607 )
...
* feat: implement jaeger-agent protocol directly
* chore: review feedback
* fix: remove jaeger feature flag
2021-09-22 17:30:37 +00:00
46088a7ff1
feat: disable rustyline dirs-next dependency ( #2579 )
2021-09-20 14:42:07 +00:00
9fba7b0ba0
feat: remove routerify ( #2586 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-20 08:57:19 +00:00
1112597c44
chore(deps): bump dirs from 3.0.2 to 4.0.0
...
Bumps [dirs](https://github.com/soc/dirs-rs ) from 3.0.2 to 4.0.0.
- [Release notes](https://github.com/soc/dirs-rs/releases )
- [Commits](https://github.com/soc/dirs-rs/commits )
---
updated-dependencies:
- dependency-name: dirs
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-20 08:28:04 +00:00
3ffb16daa6
feat: remove parse_duration ( #2574 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-19 11:27:24 +00:00
1d55d9a1b5
feat: add pbjson support ( #2468 )
...
* feat: add pbjson support
* chore: fix test
2021-09-16 07:33:27 +00:00
c66095cad1
feat: remove metrics crate ( #2552 )
2021-09-15 19:43:33 +00:00
6e7fa3e574
feat: migrate http ingest metrics ( #2542 )
...
* feat: migrate http ingest metrics
* chore: review feedback
* refactor: RAII entry ingest recorder
2021-09-15 19:01:10 +00:00
bfaba78dc3
refactor: move `predicate` into its own crate
...
Two reasons:
1. I wanna decouple `parquet_file` from `query` (nearly done, needs a
small follow-up PR).
2. `predicate` will have more and more features (like serialization)
which justifies a new home
2021-09-14 17:13:02 +02:00
3f2e46c397
feat: prune old transactions from preserved catalog
2021-09-14 12:08:17 +02:00
e07bc85670
feat: enable tracing for http API ( #2273 ) ( #2498 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-10 11:21:02 +00:00
b67610d9b9
chore(deps): bump tokio from 1.10.1 to 1.11.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.10.1 to 1.11.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.10.1...tokio-1.11.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 09:11:38 +00:00
06311970fa
Merge branch 'main' into dependabot/cargo/parquet-5.3.0
2021-09-06 09:00:27 +00:00
ccfd6ad3da
chore(deps): bump parquet from 5.2.0 to 5.3.0
...
Bumps [parquet](https://github.com/apache/arrow-rs ) from 5.2.0 to 5.3.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/5.3.0/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/5.2.0...5.3.0 )
---
updated-dependencies:
- dependency-name: parquet
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 08:52:30 +00:00
0fd92c8b45
Merge branch 'main' into dependabot/cargo/structopt-0.3.23
2021-09-06 08:51:32 +00:00
b1bb390893
chore(deps): bump parking_lot from 0.11.1 to 0.11.2
...
Bumps [parking_lot](https://github.com/Amanieu/parking_lot ) from 0.11.1 to 0.11.2.
- [Release notes](https://github.com/Amanieu/parking_lot/releases )
- [Changelog](https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md )
- [Commits](https://github.com/Amanieu/parking_lot/compare/0.11.1...0.11.2 )
---
updated-dependencies:
- dependency-name: parking_lot
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 01:18:24 +00:00
2cf9505029
chore(deps): bump structopt from 0.3.22 to 0.3.23
...
Bumps [structopt](https://github.com/TeXitoi/structopt ) from 0.3.22 to 0.3.23.
- [Release notes](https://github.com/TeXitoi/structopt/releases )
- [Changelog](https://github.com/TeXitoi/structopt/blob/master/CHANGELOG.md )
- [Commits](https://github.com/TeXitoi/structopt/compare/v0.3.22...v0.3.23 )
---
updated-dependencies:
- dependency-name: structopt
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 01:17:41 +00:00
09a5d27434
chore: ignore massif profiler outputs
2021-09-02 15:58:43 +02:00
4a863993ec
feat: "dump catalog" debug CLI
2021-09-02 08:08:20 +02:00
6ba25728e1
chore: exclude non-source code from package ( #2452 )
...
Before this change, modifications to the following files would trigger a
rebuild on `cargo build`.
```
2,13d1
< .circleci/config.yml
< .circleci/get-deploy-tags.sh
< .editorconfig
< .gitattributes
< .github/ISSUE_TEMPLATE/bug_report.md
< .github/ISSUE_TEMPLATE/feature_request.md
< .github/PULL_REQUEST_TEMPLATE.md
< .github/dependabot.yml
< .github/semantic.yml
< .gitignore
< .kodiak.toml
< CONTRIBUTING.md
17,22d4
< Dockerfile
< Dockerfile.dockerignore
< LICENSE-APACHE
< LICENSE-MIT
< README.md
< buf.yaml
24,76d5
< docker/Dockerfile.ci
< docker/Dockerfile.ci.dockerignore
< docker/Dockerfile.ci.integration
< docker/Dockerfile.ci.integration.dockerignore
< docker/Dockerfile.iox
< docker/Dockerfile.iox.dockerignore
< docker/ci-kafka-docker-compose.yml
< docker/integration_test.sh
< docs/README.md
< docs/catalog_persistence.md
< docs/data_management.md
< docs/drawings/data_lifecycle.monopic
< docs/drawings/data_organization.monopic
< docs/encoding_thoughts.md
< docs/env.example
< docs/images/flame_graph.png
< docs/local_filesystems.md
< docs/metrics.md
< docs/multi_core_tasks.md
< docs/profiling.md
< docs/protobuf.md
< docs/regenerating_flatbuffers.md
< docs/server_startup.md
< docs/sql.md
< docs/style_guide.md
< docs/testing.md
< docs/tracing.md
< docs/valgrind.md
< perf/logs/iox_router.log
< perf/logs/iox_writer.log
< perf/logs/jaeger.log
< perf/logs/kafka.log
< perf/logs/minio.log
< perf/logs/zookeeper.log
< perf/volumes/minio/.minio.sys/buckets/.bloomcycle.bin
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/.bloomcycle.bin/fs.json
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/.usage-cache.bin/fs.json
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/.usage.json/fs.json
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/iox1/.usage-cache.bin/fs.json
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/iox2/.usage-cache.bin/fs.json
< perf/volumes/minio/.minio.sys/buckets/.tracker.bin
< perf/volumes/minio/.minio.sys/buckets/.usage-cache.bin
< perf/volumes/minio/.minio.sys/buckets/.usage.json
< perf/volumes/minio/.minio.sys/buckets/iox1/.metadata.bin
< perf/volumes/minio/.minio.sys/buckets/iox1/.usage-cache.bin
< perf/volumes/minio/.minio.sys/buckets/iox2/.metadata.bin
< perf/volumes/minio/.minio.sys/buckets/iox2/.usage-cache.bin
< perf/volumes/minio/.minio.sys/config/config.json
< perf/volumes/minio/.minio.sys/config/iam/format.json
< perf/volumes/minio/.minio.sys/format.json
< perf/volumes/zookeeper/data/myid
< perf/volumes/zookeeper/data/version-2/log.1
< perf/volumes/zookeeper/data/version-2/snapshot.0
79,87d7
< scripts/edit_db_rules
< scripts/genlp.py
< scripts/git-log-prs
< scripts/grpcurl
< scripts/jq_edit_db_rules
< scripts/logfmts
< scripts/prototxt
< scripts/valgrind.supp
< scripts/valigrind_leak
161d80
< tools/iplan
```
2021-08-31 17:14:02 +00:00
3e154b3b02
chore(deps): bump serde_json from 1.0.66 to 1.0.67
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.66 to 1.0.67.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.66...v1.0.67 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-30 11:28:35 +00:00
52fbaef2df
chore(deps): bump thiserror from 1.0.26 to 1.0.28
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.26 to 1.0.28.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.26...1.0.28 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-30 11:05:11 +00:00
13161de003
chore(deps): bump rustyline from 8.2.0 to 9.0.0
...
Bumps [rustyline](https://github.com/kkawakam/rustyline ) from 8.2.0 to 9.0.0.
- [Release notes](https://github.com/kkawakam/rustyline/releases )
- [Changelog](https://github.com/kkawakam/rustyline/blob/master/History.md )
- [Commits](https://github.com/kkawakam/rustyline/compare/v8.2.0...v9.0.0 )
---
updated-dependencies:
- dependency-name: rustyline
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-30 01:19:49 +00:00
fcec394a28
feat: connect up new metrics ( #2428 )
2021-08-27 12:55:35 +00:00
3a26f7857f
feat: add new metrics abstraction ( #2076 ) ( #2365 )
...
* feat: add new metrics abstraction (#2076 )
* chore: fix lint
* feat: add tests
feat: histogram export
* chore: further docs
* chore: docs
* chore: log on prometheus export error
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-27 11:54:23 +00:00
1773bf5d37
feat: add storage client to influxdb_iox_client ( #2404 )
...
* feat: add storage client to influxdb_iox_client
* chore: fix type_url
* refactor: split storage into separate crate
* chore: fix doctest
* chore: review feedback
* chore: add generated_types cleanup ticket
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-26 10:30:59 +00:00
2c9c191b17
refactor: split out trace http ( #2388 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-24 13:37:20 +00:00
3fdc0e9a6a
chore: remove orphaned code from trogging ( #2371 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-24 10:12:59 +00:00
58716dee71
refactor: remove mem-qe ( #2344 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-23 14:35:12 +00:00
0d4bc4190f
chore(deps): bump assert_cmd from 1.0.8 to 2.0.0
...
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd ) from 1.0.8 to 2.0.0.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases )
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v1.0.8...v2.0.0 )
---
updated-dependencies:
- dependency-name: assert_cmd
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-23 01:17:06 +00:00
bbbd38bb12
fix(heappy): Prevent re-entering a running heap profile
...
Upgrades heappy to include https://github.com/mkmik/heappy/pull/2
and adapts to the API change.
2021-08-20 14:36:36 +02:00
4a2f0b0a52
fix: Make heappy and nonheappy mutually exclusive
...
Closes #2288
2021-08-20 11:35:07 +02:00
db97069ecd
feat: jaeger span export ( #2273 ) ( #2348 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-20 08:57:16 +00:00
d5f01a2a68
refactor: move data generator to IOx repo and fix build
2021-08-19 14:26:15 -04:00
04ba139cb2
refactor: split trace crate ( #2343 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-19 11:00:32 +00:00
5e1cb244f7
feat: make pprof optional ( #2331 )
2021-08-18 15:13:37 +00:00
a50b1646bd
fix(heappy): Report free as positive numbers
2021-08-18 14:30:41 +02:00
fc473995fa
feat(iox): Upgrade heappy to present free memory events
...
Currently heappy is reporting "bytes allocated" and "inuse_bytes" ("bytes allocated - freed bytes").
Since the allocation site and free site can be wildly different, it's very hard to correlate these events
and have a meaningful "inuse_bytes" graph.
I have a plan for how to fix this for good, but in the meantime
I'd like to report raw "freed bytes".
2021-08-18 13:04:48 +02:00
1d6a8703af
feat: support sinking traces to an OTEL SpanExporter ( #2319 )
...
* feat: support sinking traces to an OTEL SpanExporter
* chore: consistent logging
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* chore: review feedback
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-18 08:05:21 +00:00
efa776bd03
feat: fix log crate output ( #2325 )
...
* feat: fix log crate output
* chore: fix doc
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-17 16:19:22 +00:00
94ff77126e
feat: Enable freed memory tracking in heappy
...
In aed37ab50a
2021-08-17 15:09:28 +02:00
1d85a3fef6
build(deps): bump predicates from 1.0.8 to 2.0.2
...
Bumps [predicates](https://github.com/assert-rs/predicates-rs ) from 1.0.8 to 2.0.2.
- [Release notes](https://github.com/assert-rs/predicates-rs/releases )
- [Changelog](https://github.com/assert-rs/predicates-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/predicates-rs/compare/v1.0.8...v2.0.2 )
---
updated-dependencies:
- dependency-name: predicates
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-16 15:36:49 +00:00
500019cff9
Merge branch 'main' into ntran/heappy2
2021-08-13 17:27:38 +02:00
8650e419f7
feat: add trace collection plumbing ( #2273 ) ( #2274 )
...
* feat: add trace collection plumbing (#2273 )
* chore: fix lints
* chore: use tower directly
* chore: add tests
* chore: review feedback
* refactor: use serde instead of custom display impl
* chore: fix doc
2021-08-13 14:38:31 +00:00
cedaf2d5f2
fix: remove unprefixed_malloc_on_supported_platforms feature from tikv-jemallocator
2021-08-13 08:24:15 -05:00
697de875ca
refactor: make heappy an optional feature, add to build image
2021-08-13 08:11:32 -05:00
4fd5862a9c
fix: turn heappy on
2021-08-12 17:14:03 -04:00
ae6b0e669b
refactor: Extract a database persister type that wraps object store
...
Connects to #2193 .
2021-08-12 15:05:32 -04:00
4f52a70d5b
refactor: adddress mkm's comments
2021-08-12 12:04:42 -04:00
8aa9514d93
feat: integrate heappy implemented by mkm
2021-08-11 23:42:50 -04:00
1f450ef371
feat: add Database abstraction ( #2186 ) ( #2203 )
...
* feat: add Database abstraction
* chore: minor tweaks
* chore: remove redundant test fixture restart
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-08 17:14:23 +00:00
82677d9510
feat: add `jaeger` and `otlp` features, default to off ( #2202 )
...
* feat: add jaeger and otlp flags
* chore: add jaeger and otlp features to CI test and deploy image
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-06 13:26:59 +00:00
e69a159ad2
refactor: remove unused opentelemetry dependencies ( #2201 )
2021-08-05 16:20:22 +00:00
e92e94caad
chore: Update deps (including arrow 5.1.0, tonic -> 0.5, and prost 0.5) ( #2172 )
...
* chore: Update deps (including arrow 5.0.0 --> arrow 5.1.0)
* chore: update all the things
* refactor: Update serving readiness check due to change in Tonic API
* chore: update more deps
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-05 15:57:38 +00:00
7ca31703a4
feat: Attempt to dump a stacktrace to stderr prior to process abort on SIGSEGV/SIGILL/SIGBUS ( #2155 )
...
* feat: Attempt to dump a stacktrace to stdout prior to process abort
* refactor: rewrite signal handling in terms of libc, remove sig dep
* refactor: print to stderr
* fix: Update src/main.rs
* docs: note provenance
2021-07-30 11:58:28 +00:00
4def5bd816
chore: Add print_cpu command line to perf build ( #2114 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-26 15:26:34 +00:00
4da8a16c18
chore: update to arrow 5.0 and master datafusion ( #2049 )
...
* chore: update to arrow 5.0 and master datafusion
* fix: Update test for change in object size
2021-07-19 12:49:51 +00:00
b5428e53a5
refactor: write buffer testing + better mocking
...
This refactors the write buffer a bit for:
- **Testing:** Add generic tests for the Kafka and the mocking
implementation. The same interface can be used easily add new
implementations (e.g. via Redis, filesystem, ...).
- **Partition on Write:** The caller of the writer operation must now
specify the partition/sequencer ID. The implicit partitioning of the
Kafka writer would have lead to broken data since we must never spill
entries w/ the same primary key over multiple partitions. At the
moment we will only use partition 0 but we can easily implement
better logic in the future.
- **Improved Mocking:** The mocked implementation now simulates a system
that feels more real. Especially the handling around multiple streams
and "write while read" has been improved. This will be helpful for
testing and for new features like seeking (during replay). A solid
realistic mock also helps us to ensure that the tests using the mock
do not rely on unrealistic behavior too much.
2021-07-15 17:20:45 +02:00
0e5276ed20
Merge branch 'main' into alamb/go_go_go_go
2021-07-14 13:56:35 +01:00
9cb9ae0874
chore: move write buffer into its own crate
2021-07-14 14:09:18 +02:00
4800b36949
chore: Update IOx to a pre-release version of arrow and datafusion to test out performance improvement
2021-07-13 15:44:57 -04:00
1a79bf7e99
refactor: Make aws/azure/gcs optional features and stop compiling 100 dependencies during dev ( #1933 )
...
* feat: make aws, gcp, azure dependencies optional
* fix: only run object store tests if the features are enabled
* fix: clean up testing
* fix: rename step
* fix: add to list of jobs
* fix: remove test with object store
* fix: review comments
2021-07-09 11:38:30 +00:00
4ca2d3e148
chore: move persistence windows related code into own crate
...
The entire persistence windows data structures (including the
checkpoints) have nothing to do with the mutable buffer per se. So lets
move them into their own crate. This also makes `parquet_file` not
longer depend on `mutable_buffer`.
2021-07-05 10:23:58 +02:00
69b0bb1510
feat: Implement grpc-router crate
...
This PR implements the main building block for implementing the gRPC StorageService router.
2021-06-23 17:21:46 +02:00
55c546baff
feat: eagerly check object store during CLI `run`
...
Instead of waiting for the server ID to be set and then mark the server
as errored, directly check the object store on startup. This is
important so that we fail fast when Istio isn't up and running yet.
2021-06-22 18:21:30 +02:00
b4644e6108
test: Start of Kafka Write Buffer integration tests
2021-06-21 09:41:35 -04:00
ec43a87909
chore: Update itertools deps ( #1750 )
2021-06-17 17:56:44 +00:00
760bcde3f0
feat: Factor out tracing/logging CLI options
...
This PR factors out the tracing/logging CLI optinos into the `trogging` utility crate,
so that multiple binaries from the IOx suite (such as conductor) can use the same (and quite complex)
logging/tracing configuration options (flags and env vars).
Closes influxdata/conductor#343
2021-06-16 00:54:11 +02:00
bf54ab51f2
refactor: split lifecycle into separate crate ( #1730 )
2021-06-15 15:57:47 +00:00
bde35cf5be
chore: Pull tracing+logging setup in its own crate
...
1. so that it can be reused by other binaries (e.g. conductor)
2. so that it's faster to build when working on it.
2021-06-15 14:52:04 +02:00
5a68abaa53
feat: Implement LayeredTracing
...
__Rationale__
We currently use the `tracing` framework to output to both log outputs (e.g. stdout for k8s) and distributed tracing collectors (e.g. opentelemetry jaeger).
However, due to a limitation in the `tracing` SDK, we can only have one "filter" level that applies
to both logs and tracing outputs. This is unpractical because tracing collectors are designed
to receive high verbosity data (which will be then sampled within the opentelemetry library),
while logs generally are limited to the DEBUG level on production.
This PR adds a `FilteredLayer` tracing subscriber layer, that wraps a subscriber layer with a independent
filter, which can filter events goint to the wrapper subscriber layer more agressively than the global layer.
This will allow us to emit logs at INFO or DEBUG level while passing all events to opentelemetry at TRACE
level (and opentelemetry SDK will then sample the events so that only a small part will be sent to the
ot collector)
__Note__
This PR just implements the `FilteredLayer` and a test. Another PR will integrate this with
our log/tracing setup code.
2021-06-11 04:32:47 +02:00
a614fef5bc
chore: remove more unused dependencies ( #1658 )
...
* chore: remove more unused deps
* refactor: move benchmarks into server_benchmarks crate
2021-06-09 10:17:20 +00:00
87297f7db4
Merge branch 'main' into cn/delete
2021-06-07 13:32:42 +00:00
5749a2c119
chore: cleanup legacy TSM -> parquet code ( #1639 )
...
* chore: cleanup legacy parquet code
* chore: remove tests of removed functionality
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-07 12:59:33 +00:00
f4a9a5ae56
fix: Remove write buffer
2021-06-04 14:40:17 -04:00
42f26b609b
refactor: Move `query_tests` and `server_benchmarks` into their own crate --> smaller `server` ( #1628 )
...
* refactor: Separate query_tests into its own crate
* fix: references
* refactor: break out server benchmarks
* fix: Update query_tests/src/lib.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-06-04 17:31:19 +00:00
00e735ef0d
chore: remove unused dependencies ( #1583 )
2021-05-29 10:31:57 +00:00
4fcc04e6c9
chore: enable arrow prettyprint feature ( #1566 )
2021-05-27 10:28:14 +00:00
0a37cb80f7
feat: Use jemalloc everywhere instead of malloc
...
There may be many reasons for the discrepancy in jemalloc reported allocations total sizes and RSS.
One of them is that our binary doesn't use jmalloc for all the allocations.
Turns out that jemallocator only sets the global rust allocator. Any call to `malloc` will still
go throught the system allocator. Presumably those calls come from linked C code,
but it's also not impossible that not all rust code honours the global allocator (I have no idea, but let's see)
2021-05-25 03:42:52 +02:00
14ba25f86d
chore: Update datafusion and use released version of arrow crates ( #1546 )
...
* chore: Update datafusion and use released version of arrow crate
* fix: Update for change in API
2021-05-24 15:37:22 +00:00
40b21dbca1
feat: Add /debug/pprof/profile
2021-05-19 13:08:42 +02:00
ebea554e5a
feat: Report num_cpus seen by IOx on startup ( #1484 )
2021-05-12 13:42:16 +00:00
10f89a3e8d
refactor: split entry out into separate crate ( #1428 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 11:36:23 +00:00
7e2e2c4caf
refactor: initial split of observability_deps ( #1429 )
2021-05-06 08:25:09 +00:00
7e01650c53
Merge branch 'main' into reflection
2021-05-05 20:29:13 +00:00
0d6d94dc00
feat: Enable gRPC reflection
2021-05-05 22:04:47 +02:00
411cf134e9
refactor: explode arrow_deps ( #1425 )
...
* refactor: explode arrow_deps
* chore: workaround doctest bug
2021-05-05 16:59:12 +00:00
b579ef8646
feat: Add jemalloc stats
2021-05-03 12:10:48 +02:00
a64e622f6e
feat: Add interactive SQL repl ( #1332 )
...
* feat: Add interactive SQL repl
* fix: try and fix test
* fix: remove test for prompt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-28 20:41:02 +00:00
dependabot[bot]
Raphael Taylor-Davies
kodiakhq[bot]
Andrew Lamb
Marco Neumann
Jacob Marble
Marko Mikulicic
Paul Dix
Andrew Lamb
Nga Tran
Carol (Nichols || Goulding)
Edd Robinson