06311970fa
Merge branch 'main' into dependabot/cargo/parquet-5.3.0
2021-09-06 09:00:27 +00:00
ccfd6ad3da
chore(deps): bump parquet from 5.2.0 to 5.3.0
...
Bumps [parquet](https://github.com/apache/arrow-rs ) from 5.2.0 to 5.3.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/5.3.0/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/5.2.0...5.3.0 )
---
updated-dependencies:
- dependency-name: parquet
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 08:52:30 +00:00
0fd92c8b45
Merge branch 'main' into dependabot/cargo/structopt-0.3.23
2021-09-06 08:51:32 +00:00
b1bb390893
chore(deps): bump parking_lot from 0.11.1 to 0.11.2
...
Bumps [parking_lot](https://github.com/Amanieu/parking_lot ) from 0.11.1 to 0.11.2.
- [Release notes](https://github.com/Amanieu/parking_lot/releases )
- [Changelog](https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md )
- [Commits](https://github.com/Amanieu/parking_lot/compare/0.11.1...0.11.2 )
---
updated-dependencies:
- dependency-name: parking_lot
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 01:18:24 +00:00
2cf9505029
chore(deps): bump structopt from 0.3.22 to 0.3.23
...
Bumps [structopt](https://github.com/TeXitoi/structopt ) from 0.3.22 to 0.3.23.
- [Release notes](https://github.com/TeXitoi/structopt/releases )
- [Changelog](https://github.com/TeXitoi/structopt/blob/master/CHANGELOG.md )
- [Commits](https://github.com/TeXitoi/structopt/compare/v0.3.22...v0.3.23 )
---
updated-dependencies:
- dependency-name: structopt
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 01:17:41 +00:00
09a5d27434
chore: ignore massif profiler outputs
2021-09-02 15:58:43 +02:00
4a863993ec
feat: "dump catalog" debug CLI
2021-09-02 08:08:20 +02:00
6ba25728e1
chore: exclude non-source code from package ( #2452 )
...
Before this change, modifications to the following files would trigger a
rebuild on `cargo build`.
```
2,13d1
< .circleci/config.yml
< .circleci/get-deploy-tags.sh
< .editorconfig
< .gitattributes
< .github/ISSUE_TEMPLATE/bug_report.md
< .github/ISSUE_TEMPLATE/feature_request.md
< .github/PULL_REQUEST_TEMPLATE.md
< .github/dependabot.yml
< .github/semantic.yml
< .gitignore
< .kodiak.toml
< CONTRIBUTING.md
17,22d4
< Dockerfile
< Dockerfile.dockerignore
< LICENSE-APACHE
< LICENSE-MIT
< README.md
< buf.yaml
24,76d5
< docker/Dockerfile.ci
< docker/Dockerfile.ci.dockerignore
< docker/Dockerfile.ci.integration
< docker/Dockerfile.ci.integration.dockerignore
< docker/Dockerfile.iox
< docker/Dockerfile.iox.dockerignore
< docker/ci-kafka-docker-compose.yml
< docker/integration_test.sh
< docs/README.md
< docs/catalog_persistence.md
< docs/data_management.md
< docs/drawings/data_lifecycle.monopic
< docs/drawings/data_organization.monopic
< docs/encoding_thoughts.md
< docs/env.example
< docs/images/flame_graph.png
< docs/local_filesystems.md
< docs/metrics.md
< docs/multi_core_tasks.md
< docs/profiling.md
< docs/protobuf.md
< docs/regenerating_flatbuffers.md
< docs/server_startup.md
< docs/sql.md
< docs/style_guide.md
< docs/testing.md
< docs/tracing.md
< docs/valgrind.md
< perf/logs/iox_router.log
< perf/logs/iox_writer.log
< perf/logs/jaeger.log
< perf/logs/kafka.log
< perf/logs/minio.log
< perf/logs/zookeeper.log
< perf/volumes/minio/.minio.sys/buckets/.bloomcycle.bin
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/.bloomcycle.bin/fs.json
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/.usage-cache.bin/fs.json
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/.usage.json/fs.json
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/iox1/.usage-cache.bin/fs.json
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/iox2/.usage-cache.bin/fs.json
< perf/volumes/minio/.minio.sys/buckets/.tracker.bin
< perf/volumes/minio/.minio.sys/buckets/.usage-cache.bin
< perf/volumes/minio/.minio.sys/buckets/.usage.json
< perf/volumes/minio/.minio.sys/buckets/iox1/.metadata.bin
< perf/volumes/minio/.minio.sys/buckets/iox1/.usage-cache.bin
< perf/volumes/minio/.minio.sys/buckets/iox2/.metadata.bin
< perf/volumes/minio/.minio.sys/buckets/iox2/.usage-cache.bin
< perf/volumes/minio/.minio.sys/config/config.json
< perf/volumes/minio/.minio.sys/config/iam/format.json
< perf/volumes/minio/.minio.sys/format.json
< perf/volumes/zookeeper/data/myid
< perf/volumes/zookeeper/data/version-2/log.1
< perf/volumes/zookeeper/data/version-2/snapshot.0
79,87d7
< scripts/edit_db_rules
< scripts/genlp.py
< scripts/git-log-prs
< scripts/grpcurl
< scripts/jq_edit_db_rules
< scripts/logfmts
< scripts/prototxt
< scripts/valgrind.supp
< scripts/valigrind_leak
161d80
< tools/iplan
```
2021-08-31 17:14:02 +00:00
3e154b3b02
chore(deps): bump serde_json from 1.0.66 to 1.0.67
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.66 to 1.0.67.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.66...v1.0.67 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-30 11:28:35 +00:00
52fbaef2df
chore(deps): bump thiserror from 1.0.26 to 1.0.28
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.26 to 1.0.28.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.26...1.0.28 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-30 11:05:11 +00:00
13161de003
chore(deps): bump rustyline from 8.2.0 to 9.0.0
...
Bumps [rustyline](https://github.com/kkawakam/rustyline ) from 8.2.0 to 9.0.0.
- [Release notes](https://github.com/kkawakam/rustyline/releases )
- [Changelog](https://github.com/kkawakam/rustyline/blob/master/History.md )
- [Commits](https://github.com/kkawakam/rustyline/compare/v8.2.0...v9.0.0 )
---
updated-dependencies:
- dependency-name: rustyline
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-30 01:19:49 +00:00
fcec394a28
feat: connect up new metrics ( #2428 )
2021-08-27 12:55:35 +00:00
3a26f7857f
feat: add new metrics abstraction ( #2076 ) ( #2365 )
...
* feat: add new metrics abstraction (#2076 )
* chore: fix lint
* feat: add tests
feat: histogram export
* chore: further docs
* chore: docs
* chore: log on prometheus export error
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-27 11:54:23 +00:00
1773bf5d37
feat: add storage client to influxdb_iox_client ( #2404 )
...
* feat: add storage client to influxdb_iox_client
* chore: fix type_url
* refactor: split storage into separate crate
* chore: fix doctest
* chore: review feedback
* chore: add generated_types cleanup ticket
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-26 10:30:59 +00:00
2c9c191b17
refactor: split out trace http ( #2388 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-24 13:37:20 +00:00
3fdc0e9a6a
chore: remove orphaned code from trogging ( #2371 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-24 10:12:59 +00:00
58716dee71
refactor: remove mem-qe ( #2344 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-23 14:35:12 +00:00
0d4bc4190f
chore(deps): bump assert_cmd from 1.0.8 to 2.0.0
...
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd ) from 1.0.8 to 2.0.0.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases )
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v1.0.8...v2.0.0 )
---
updated-dependencies:
- dependency-name: assert_cmd
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-23 01:17:06 +00:00
bbbd38bb12
fix(heappy): Prevent re-entering a running heap profile
...
Upgrades heappy to include https://github.com/mkmik/heappy/pull/2
and adapts to the API change.
2021-08-20 14:36:36 +02:00
4a2f0b0a52
fix: Make heappy and nonheappy mutually exclusive
...
Closes #2288
2021-08-20 11:35:07 +02:00
db97069ecd
feat: jaeger span export ( #2273 ) ( #2348 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-20 08:57:16 +00:00
d5f01a2a68
refactor: move data generator to IOx repo and fix build
2021-08-19 14:26:15 -04:00
04ba139cb2
refactor: split trace crate ( #2343 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-19 11:00:32 +00:00
5e1cb244f7
feat: make pprof optional ( #2331 )
2021-08-18 15:13:37 +00:00
a50b1646bd
fix(heappy): Report free as positive numbers
2021-08-18 14:30:41 +02:00
fc473995fa
feat(iox): Upgrade heappy to present free memory events
...
Currently heappy is reporting "bytes allocated" and "inuse_bytes" ("bytes allocated - freed bytes").
Since the allocation site and free site can be wildly different, it's very hard to correlate these events
and have a meaningful "inuse_bytes" graph.
I have a plan for how to fix this for good, but in the meantime
I'd like to report raw "freed bytes".
2021-08-18 13:04:48 +02:00
1d6a8703af
feat: support sinking traces to an OTEL SpanExporter ( #2319 )
...
* feat: support sinking traces to an OTEL SpanExporter
* chore: consistent logging
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* chore: review feedback
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-18 08:05:21 +00:00
efa776bd03
feat: fix log crate output ( #2325 )
...
* feat: fix log crate output
* chore: fix doc
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-17 16:19:22 +00:00
94ff77126e
feat: Enable freed memory tracking in heappy
...
In aed37ab50a
2021-08-17 15:09:28 +02:00
1d85a3fef6
build(deps): bump predicates from 1.0.8 to 2.0.2
...
Bumps [predicates](https://github.com/assert-rs/predicates-rs ) from 1.0.8 to 2.0.2.
- [Release notes](https://github.com/assert-rs/predicates-rs/releases )
- [Changelog](https://github.com/assert-rs/predicates-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/predicates-rs/compare/v1.0.8...v2.0.2 )
---
updated-dependencies:
- dependency-name: predicates
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-16 15:36:49 +00:00
500019cff9
Merge branch 'main' into ntran/heappy2
2021-08-13 17:27:38 +02:00
8650e419f7
feat: add trace collection plumbing ( #2273 ) ( #2274 )
...
* feat: add trace collection plumbing (#2273 )
* chore: fix lints
* chore: use tower directly
* chore: add tests
* chore: review feedback
* refactor: use serde instead of custom display impl
* chore: fix doc
2021-08-13 14:38:31 +00:00
cedaf2d5f2
fix: remove unprefixed_malloc_on_supported_platforms feature from tikv-jemallocator
2021-08-13 08:24:15 -05:00
697de875ca
refactor: make heappy an optional feature, add to build image
2021-08-13 08:11:32 -05:00
4fd5862a9c
fix: turn heappy on
2021-08-12 17:14:03 -04:00
ae6b0e669b
refactor: Extract a database persister type that wraps object store
...
Connects to #2193 .
2021-08-12 15:05:32 -04:00
4f52a70d5b
refactor: adddress mkm's comments
2021-08-12 12:04:42 -04:00
8aa9514d93
feat: integrate heappy implemented by mkm
2021-08-11 23:42:50 -04:00
1f450ef371
feat: add Database abstraction ( #2186 ) ( #2203 )
...
* feat: add Database abstraction
* chore: minor tweaks
* chore: remove redundant test fixture restart
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-08 17:14:23 +00:00
82677d9510
feat: add `jaeger` and `otlp` features, default to off ( #2202 )
...
* feat: add jaeger and otlp flags
* chore: add jaeger and otlp features to CI test and deploy image
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-06 13:26:59 +00:00
e69a159ad2
refactor: remove unused opentelemetry dependencies ( #2201 )
2021-08-05 16:20:22 +00:00
e92e94caad
chore: Update deps (including arrow 5.1.0, tonic -> 0.5, and prost 0.5) ( #2172 )
...
* chore: Update deps (including arrow 5.0.0 --> arrow 5.1.0)
* chore: update all the things
* refactor: Update serving readiness check due to change in Tonic API
* chore: update more deps
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-05 15:57:38 +00:00
7ca31703a4
feat: Attempt to dump a stacktrace to stderr prior to process abort on SIGSEGV/SIGILL/SIGBUS ( #2155 )
...
* feat: Attempt to dump a stacktrace to stdout prior to process abort
* refactor: rewrite signal handling in terms of libc, remove sig dep
* refactor: print to stderr
* fix: Update src/main.rs
* docs: note provenance
2021-07-30 11:58:28 +00:00
4def5bd816
chore: Add print_cpu command line to perf build ( #2114 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-26 15:26:34 +00:00
4da8a16c18
chore: update to arrow 5.0 and master datafusion ( #2049 )
...
* chore: update to arrow 5.0 and master datafusion
* fix: Update test for change in object size
2021-07-19 12:49:51 +00:00
b5428e53a5
refactor: write buffer testing + better mocking
...
This refactors the write buffer a bit for:
- **Testing:** Add generic tests for the Kafka and the mocking
implementation. The same interface can be used easily add new
implementations (e.g. via Redis, filesystem, ...).
- **Partition on Write:** The caller of the writer operation must now
specify the partition/sequencer ID. The implicit partitioning of the
Kafka writer would have lead to broken data since we must never spill
entries w/ the same primary key over multiple partitions. At the
moment we will only use partition 0 but we can easily implement
better logic in the future.
- **Improved Mocking:** The mocked implementation now simulates a system
that feels more real. Especially the handling around multiple streams
and "write while read" has been improved. This will be helpful for
testing and for new features like seeking (during replay). A solid
realistic mock also helps us to ensure that the tests using the mock
do not rely on unrealistic behavior too much.
2021-07-15 17:20:45 +02:00
0e5276ed20
Merge branch 'main' into alamb/go_go_go_go
2021-07-14 13:56:35 +01:00
9cb9ae0874
chore: move write buffer into its own crate
2021-07-14 14:09:18 +02:00
4800b36949
chore: Update IOx to a pre-release version of arrow and datafusion to test out performance improvement
2021-07-13 15:44:57 -04:00
1a79bf7e99
refactor: Make aws/azure/gcs optional features and stop compiling 100 dependencies during dev ( #1933 )
...
* feat: make aws, gcp, azure dependencies optional
* fix: only run object store tests if the features are enabled
* fix: clean up testing
* fix: rename step
* fix: add to list of jobs
* fix: remove test with object store
* fix: review comments
2021-07-09 11:38:30 +00:00
kodiakhq[bot]
dependabot[bot]
Marco Neumann
Jacob Marble
Raphael Taylor-Davies
Marko Mikulicic
Paul Dix
Andrew Lamb
Andrew Lamb
Nga Tran
Carol (Nichols || Goulding)
Edd Robinson