Marco Neumann
1962fcc229
chore: reduce dependencies and run `cargo update` ( #4906 )
...
* chore: reduce proptest features
* chore: remove `grpc-router`
This crate is currently unused and we don't have immediate plans to use
it. And there's GIT, so it can always be restored.
* chore: `cargo update`
2022-06-20 12:18:28 +00:00
Carol (Nichols || Goulding)
6417e7dc2a
feat: Extract sharder to its own crate
2022-06-15 10:01:45 -04:00
Luke Bond
b76a0080d5
chore: remove unused iox_gitops_adapter ( #4675 )
...
* chore: remove unused iox_gitops_adapter
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-05-24 10:28:43 +00:00
Marco Neumann
52346642a0
ci: fix cargo deny ( #4629 )
...
* ci: fix cargo deny
* chore: downgrade `socket2`, version 0.4.5 was yanked
* chore: rename `query` to `iox_query`
`query` is already taken on crates.io and yanked and I am getting tired
of working around that.
2022-05-18 09:38:35 +00:00
Raphael Taylor-Davies
f2bb0fdf77
feat: update to crates.io object_store version ( #4595 )
...
* feat: update to crates.io object_store version
* chore: Run cargo hakari tasks
* fix: tests
* chore: remove object store integration test plumbing
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-05-13 16:26:07 +00:00
Andrew Lamb
35d80fc512
refactor: rename `iox_catalog_service` to `service_grpc_catalog` for consistency ( #4581 )
...
* refactor: rename `iox_catalog_service` to `service_grpc_catalog`
* fix: fmt
* fix: alphabetize crate
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-13 14:07:58 +00:00
kodiakhq[bot]
542ec97b66
Merge branch 'main' into cn/rename-no-ng
2022-05-13 13:47:48 +00:00
Marco Neumann
cb0a4176fd
refactor: move `querier::cache_system` into its own crate ( #4592 )
2022-05-13 13:12:07 +00:00
Carol (Nichols || Goulding)
48e6e5713d
fix: Rename test_helpers_end_to_end_ng to test_helpers_end_to_end
...
Connects to #4450 .
2022-05-12 16:09:07 -04:00
Andrew Lamb
7eff52d04e
refactor: rename `iox_object_store_service` to `service_grpc_object_store` for consisntency ( #4572 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-12 14:22:18 +00:00
Andrew Lamb
b8cb4c3f2b
feat: Interrogate schema from querier (as well as router) ( #4557 )
...
* refactor: move SchemaService into `service_grpc_schema`
* feat: implement schema gRPC for querier
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-05-10 20:55:58 +00:00
Raphael Taylor-Davies
99b1a9b83f
refactor: split out ObjectStoreMetrics ( #4547 )
...
* refactor: split out ObjectStoreMetrics
* chore: add workspace hack
* fix: compile
2022-05-10 10:56:28 +00:00
Carol (Nichols || Goulding)
fcd4815645
fix: Rename router2 to router
2022-05-06 14:51:52 -04:00
Carol (Nichols || Goulding)
0650a9bb77
fix: Rename ioxd_router2 to ioxd_router
2022-05-06 14:45:39 -04:00
Carol (Nichols || Goulding)
068096e7e1
fix: Rename data_types2 to data_types
2022-05-06 14:45:39 -04:00
Carol (Nichols || Goulding)
e1bef1c218
fix: Remove OG data_types crate
2022-05-06 14:45:39 -04:00
Carol (Nichols || Goulding)
ea46830954
fix: Remove iox_object_store crate; move ParquetFilePath to parquet_file
2022-05-06 14:45:36 -04:00
Carol (Nichols || Goulding)
f8bdb022bc
fix: Remove job_registry crate
2022-05-06 11:35:11 -04:00
Carol (Nichols || Goulding)
b88d071ce7
fix: Remove server
2022-05-06 11:30:36 -04:00
Carol (Nichols || Goulding)
e0bc1801ac
fix: Remove router
2022-05-06 11:30:36 -04:00
Carol (Nichols || Goulding)
ba8191c1eb
fix: Remove persistence_windows
2022-05-06 11:30:35 -04:00
Carol (Nichols || Goulding)
cae32209da
fix: Remove parquet_catalog
2022-05-06 11:30:26 -04:00
Carol (Nichols || Goulding)
2d8656e2e1
fix: Remove mutable_buffer
2022-05-06 11:27:33 -04:00
Carol (Nichols || Goulding)
e63f006398
fix: Remove lifecycle crate
2022-05-06 11:27:33 -04:00
Carol (Nichols || Goulding)
e7de16732d
fix: Remove internal_types
2022-05-06 11:27:33 -04:00
Carol (Nichols || Goulding)
7286b4391a
fix: Remove db crate
2022-05-06 11:27:33 -04:00
Jake Goulding
ece38417bc
feat: Remove the `run database` command
2022-05-06 09:48:30 -04:00
Jake Goulding
b939d80db8
feat: Remove the `run router` command
2022-05-06 09:48:30 -04:00
Carol (Nichols || Goulding)
e015d3bafb
feat: Remove the server_benchmarks crate ( #4506 )
2022-05-02 18:20:01 +00:00
Andrew Lamb
9e91af4501
refactor: Move IOx UDfs into a Function Registry (1/3) ( #4428 )
...
* refactor: Move all UDF implementations to query_function crate
* refactor: Move regex udf to query_functions
* refactor: Move functions out of query
* fix: lints, imports
* chore: Run cargo hakari tasks
* fix: clipy + benches
* fix: reduce borrowing and fix clippy
* fix: moar clippy
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-26 17:30:27 +00:00
二手掉包工程师
4b47d723b1
refactor: Rename time to iox_time ( #4416 )
...
Signed-off-by: hi-rustin <rustin.liu@gmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-26 00:19:59 +00:00
Paul Dix
5bf4550259
feat: add object store service to router ( #4338 )
...
Add method to catalog to get parquet file by object store id.
Add gRPC service for object store to get a file from by its uuid.
Add the object store service to router2 with object store config.
2022-04-16 17:58:31 +00:00
Paul Dix
99cbb28a89
feat: add initial catalog service to router ( #4316 )
...
Create new crate for iox_catalog_service.
Add rpc to return parquet_file records by partition id.
Add CatalogService to router2.
The catalog service will be added to over time to provide access to the catalog over gRPC.
2022-04-14 17:39:18 +00:00
Andrew Lamb
edda409b19
refactor: Extract `ioxd_test`, `ioxd_compactor`, `ioxd_ingester`; remove `ioxd` ( #4210 )
...
* refactor: Extract test, compactor, ingester, and test
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-03 10:42:22 +00:00
Andrew Lamb
833c10c083
feat: return write_token from HTTP writes to router2 ( #4202 )
...
* feat: return write_token from HTTP writes to router2
* fix: Update router2/src/dml_handlers/instrumentation.rs
Co-authored-by: Dom <dom@itsallbroken.com>
* refactor: Use WriteSummary::default more vigorously
* fix: fix typo and add links to follow on issues
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-02 10:34:51 +00:00
Andrew Lamb
532d227d11
refactor: extract router2 into ioxd_router2 ( #4183 )
...
* refactor: extract router2 from ioxd
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-31 17:39:05 +00:00
Andrew Lamb
367e926d35
refactor: extract querier into ioxd_querier ( #4182 )
...
* refactor: extract querier into ioxd_querier
* fix: dep
2022-03-31 16:03:31 +00:00
Andrew Lamb
f2a3dd58b2
refactor: rename `influxdb_ioxd` to `ioxd` ( #4162 )
...
* refactor: rename influxdb_ioxd to ioxd
* fix: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-30 13:31:27 +00:00
Andrew Lamb
4ca52e5ae0
refactor: Extract common, OG database and router out of influxdb_ioxd ( #4149 )
...
* refactor: Extract common, OG database and router out of influxdb_ioxd
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-29 13:07:19 +00:00
Andrew Lamb
fb75ef7b82
test: add end to end test for all in one mode, restructure fixture ( #4114 )
...
* test: add end to end test for all in one mode, restructure fixture
* docs: fix typos and clarify schema requrements
2022-03-24 12:53:25 +00:00
Andrew Lamb
29b89aaca7
refactor: extract influxrpc, flight and testing gRPC out of influxdb_ioxd ( #4106 )
...
* refactor: extract grpc service implementations out of influxdb_ioxd
* chore: Run cargo hakari tasks
* refactor: rename server_common to service_common
* refactor: rename server_grpc_influxrpc to service_grpc_influxrpc
* refactor: rename server_grpc_flight to service_grpc_flight
* refactor: rename server_grpc_testing to service_grpc_testing
* fix: Cargo.toml
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-23 20:14:45 +00:00
Nga Tran
f03ebd79ab
refactor: move querier's test utils to a new crate to get reused by tests in other crates ( #4013 )
...
* refactor: move querier's test utils to a new crate to be able resued by tests in other crates
* chore: remove unused import
2022-03-10 18:17:58 +00:00
Andrew Lamb
d2c0acdd46
refactor: Remove serving readiness gate ( #3986 )
...
* refactor: Remove serving_readiness
* fix: remove more
* fix: remove test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-09 12:17:44 +00:00
Andrew Lamb
d6afda0227
refactor: split `influxdb_ioxd`, `clap_blocks` and `serving_readiness` from influxdb_iox crate ( #3908 )
...
* refactor: split influxdb_ioxd, clap_blocks, and serving_readiness out of influxdb_iox
split out serving readiness, get compiling
* fix: hakari
* fix: hakari again
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-03 16:48:30 +00:00
Carol (Nichols || Goulding)
8f3e44bf76
refactor: Extract a crate for shared data types in the new design
2022-03-02 12:16:15 -05:00
Raphael Taylor-Davies
43ada68f37
chore: reset release codegen-units to default ( #3883 )
2022-03-01 13:44:14 +00:00
Marco Neumann
9079e6ddb0
feat: backoff retries in ingester ( #3841 )
...
* feat: add `backoff` crate
* feat: backoff retries in ingester
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 17:58:16 +00:00
Marco Neumann
459974d1b8
refactor: move `DedicatedExecutor` from `query` to `executor` ( #3819 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 09:10:43 +00:00
Luke Bond
0f012de70c
feat: adding compactor CLI command and crate
...
Closes : #3777
2022-02-21 12:24:09 +00:00
Luke Bond
a66e29e5b3
chore: port sqlx-hotswap-pool over from conductor ( #3750 )
...
* chore: port sqlx-hotswap-pool over from conductor
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
* chore: workspace hack fixes
* fix: unique schema per test db connection
* fix: adjust search path in catalog pg tests to see if it fixes test schema issue
* fix: actually fixed sqlx hotswap pool test
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-15 16:18:36 +00:00
Luke Bond
54890525da
feat: gitops adapter ci build ( #3692 )
...
* chore: remove references to perf_image in CI
* chore: adding gitops adapter image build in CI
* chore: gitops adapter bin now same as dir & package so docker build works
* fix: circle config package change after renaming gitops adapter package
2022-02-10 11:23:06 +00:00
Carol (Nichols || Goulding)
73828323ac
feat: Ingester Flight gRPC API ( #3623 )
...
* feat: Add a way to run ingester with an in-memory catalog from the CLI
If you set the --catalog-dsn string to "mem", rather than using that as
a Postgres connection URL, create an in-memory catalog.
Planning on using this in tests, so not documenting.
* fix: Set default topic to the same value as SHARED_KAFKA_TOPIC
Namely, both should use an underscore. I don't think there's a way to
directly share these values between a constant and an annotation.
* feat: Add a flight API (handshake only) to ingester
* fix: Create partitions if using file-based write buffer
* fix: Change the server fixture to handle ingester server type
For now, the ingester doesn't implement the deployment API. Not sure if
it should or not.
* feat: Start implementing ingester do_get, namely decoding the query
Skip serialization of the predicate for the moment.
* refactor: Rename ingest protos to ingester to match crate name
* refactor: Rename QueryResults to QueryData
* feat: Move ingester flight client to new querier crate
* fix: Off by one error, different starting indexes in sequencers
* fix: Create new CLI argument to pick the catalog type
* fix: Create a CLI option to set the number of topics to auto-create in the write buffer
* fix: Check the arrow flight service's health to tell that the ingester gRPC is up
* fix: Set postgres as the default catalog type
* fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
Luke Bond
de2a013786
feat: gitops adapter ( #3656 )
...
* feat: scaffolding of gitops adapter bin crate
* chore: refactor gitops adapter; calls CLI now; status update fixes
* feat: gitops adapter now calls out to CLI once per topic; improved tests
* chore: add mock failure script for gitops adapter
* chore: update workspace-hack
* chore: refactor away unecessary to_string in gitops syncer
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-08 13:27:36 +00:00
Edd Robinson
c1f5994660
refactor: move sql parsing -> RPC predicate into own crate ( #3604 )
...
* chore: create crate
* refactor: move module to new crate
2022-02-02 10:41:57 +00:00
NGA-TRAN
ef336b4659
feat: add ingester crate and a few basic data structures for its data lifecycle
2022-01-17 15:38:03 -05:00
Paul Dix
4764e71c54
feat: Add initial iox_catalog skeleton
2022-01-17 09:56:20 -05:00
Dom
a8cb8755de
feat: new router2 crate
...
This commit adds an almost-empty router2 crate containing enough of a
skeleton to plumb into the IOx CLI/server runner.
2022-01-12 14:43:10 +00:00
Carol (Nichols || Goulding)
f9174c483b
refactor: Extract server::db into its own crate ( #3417 )
...
* refactor: Extract JobRegistry from the server crate
Both the server crate and a db crate that I'm about to extract depend on
JobRegistry, so to avoid making circular dependencies, extract the
JobRegistry to its own crate.
* refactor: Move db out of server into its own crate
Fixes #2821 .
2021-12-23 22:01:17 +00:00
Carol (Nichols || Goulding)
183de4519f
feat: Add incremental = true to the quick-release profile
2021-12-08 09:53:19 -05:00
Carol (Nichols || Goulding)
255cf79104
docs: Explain tradeoffs and use cases of the different release profiles
2021-12-08 09:53:19 -05:00
Carol (Nichols || Goulding)
758a17e199
feat: Create a build-time-optimized release build profile
...
Fixes #3316 .
2021-12-08 09:53:19 -05:00
Carol (Nichols || Goulding)
994938b2f0
fix: Alphabetize/organize deps
2021-12-06 09:36:30 -05:00
Marco Neumann
f2f090bde7
feat: test bench
2021-11-25 18:14:40 +01:00
Carol (Nichols || Goulding)
f66cd681c2
feat: Results of using cargo hakari init workspace-hack
2021-11-19 09:20:35 -05:00
Carol (Nichols || Goulding)
a2454b542d
fix: Small cleanups in Cargo.tomls ( #3160 )
...
* fix: Add tokio rt-multi-thread feature so cargo test -p client_util compiles
* fix: Alphabetize dependencies
* fix: Add the data_types_conversions feature to get tests passing
* fix: Remove dev dependencies already listed under normal dependencies
* fix: Make sure the workspace is using the new resolver
2021-11-18 22:26:33 +00:00
kodiakhq[bot]
49cfa046bb
Merge branch 'main' into crepererum/smaller_binaries
2021-11-12 11:17:09 +00:00
Raphael Taylor-Davies
6f268f8260
refactor: extract DML types ( #2731 ) ( #3084 )
...
* refactor: extract DML types (#2731 )
* chore: fmt
2021-11-11 12:34:07 +00:00
Marco Neumann
16fc610433
chore: faster and smaller IOx binaries
...
Use `codegen-units = 1`, thin-LTO and debug section compression to make our binary smaller (which is good for deploy and
test times) and faster.
# Summary
The binary size of `influxdb_iox` after building with:
```console
$ cargo build --release --no-default-features --features="aws,gcp,azure,jemalloc_replacing_malloc"
```
The profile was:
```toml
[profile.release]
debug = true
```
The commit was:
```text
89ece8b493
```
The size results are:
| Method | Size |
| ------------------------------------------ | ----- |
| baseline | 833MB |
| baseline + dbg compression | 222MB |
| baseline + strip | 49MB |
| codegen-units | 520MB |
| codegen-units + strip | 40MB |
| codegen-units + dbg compression | 143MB |
| thin LTO | 715MB |
| thin LTO + strip | 49MB |
| thin LTO + dbg compression | 199MB |
| codegen-units + thin LTO | 449MB |
| codegen-units + thin LTO + strip | 40MB |
| codegen-units + thin LTO + dbg compression | 130MB |
For the methods that were successfully measured I couldn't really see any compile time differences on my laptop.
# Methods
## Strip
Remove debug symbols. We don't really want this, so this is just to get an idea of the size
```console
$ strip baseline
```
## Debug Sections compression
Debug sections make a large amount of our binary size (a stripped executable is 49MB instead of 833MB). Since we like to
have debug symbols we cannot just strip them. However these symbols are only used for:
- backtrace generation (something went wrong, not BAU)
- profiling
- debugging
So in normal operation and most test scenarios, we're just wasting memory. So we could compress them:
```console
$ objcopy --compress-debug-sections baseline baseline-dbg_compressed
```
There is also elfutils:
```console
$ eu-elfcompress test
```
Elfutils nearly ends up with the same size (220MB instead of 222MB that objcopy achieves), but takes more time and is
probably not worth it.
Note that compressed debug sections exist since many years. The Rust ecosystem supports reading them since over a year,
see:
- <https://github.com/gimli-rs/gimli/issues/195 >
- <https://github.com/rust-lang/backtrace-rs/issues/342 >
## Codegen Units
The rust compiler parallelizes codegen work. This split into units however means that optimizations are somewhat
limited. This can be change by:
```toml
[profile.release]
...
codegen-units = 1
```
As a nice side effect this should also make our code faster.
## Thin LTO
Get LLVM to run "thin" Link Time Optimization:
```toml
[profile.release]
...
lto = "thin"
```
As a nice side effect this should also make our code faster.
## Fat LTO
Get LLVM to run "fat" Link Time Optimization:
```toml
[profile.release]
...
lto = "fat"
```
There are no results for this because this took a massive amount of memory and CPU time and did not finish on my system.
2021-11-10 15:49:19 +01:00
Raphael Taylor-Davies
60f0deaf1e
feat: remove flatbuffer entry ( #3045 )
2021-11-05 20:19:24 +00:00
Raphael Taylor-Davies
405ed6db1b
feat: add mutable_batch_tests ( #2724 ) ( #3011 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-03 18:15:35 +00:00
Marco Neumann
a91e572b43
feat: implement basic router server type
...
This only implements the deployment service and nothing else yet.
2021-11-03 16:34:51 +01:00
Raphael Taylor-Davies
3010ff4176
feat: write entry to MutableBatch ( #2972 )
...
* feat: write entry to MutableBatch
* feat: error on empty TableBatch
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-27 14:05:22 +00:00
Marco Neumann
fc651ea193
chore: move `gprc-router-test-gen` to top level
2021-10-26 11:02:33 +02:00
Marco Neumann
a0cdca7781
chore: move `influxdb_iox` into a proper workspace package
2021-10-26 11:02:33 +02:00
Marco Neumann
2833cefc12
test: remove Kafka from end2end tests
...
Kafka is now sufficiently tested via the `write_buffer` crate. The
end2end tests can now use the in-memory mock implementation or -- if
servers can only be controlled via CLI -- the file-based implementation.
2021-10-26 10:09:34 +02:00
Marco Neumann
81ccf85104
refactor: integrate print-cpu into main binary
...
It's a less confusing structure and also allows us to have this
information available in our shipped binary.
2021-10-25 18:29:32 +02:00
Marco Neumann
7a28982e2a
fix: fix dependency features for `influxdb_iox`
2021-10-25 11:15:19 +02:00
Marco Neumann
bc7244c48e
chore: use Rust edition 2021
2021-10-25 10:58:20 +02:00
Raphael Taylor-Davies
d2b41c5a13
feat: write pbdata to MutableBatch ( #2724 ) ( #2927 )
...
* feat: write pbdata to MutableBatch (#2724 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-21 14:32:35 +00:00
Raphael Taylor-Davies
f64e749eb9
feat: write line protocol to MutableBatch ( #2914 )
...
* feat: write line protocol to MutableBatch
* chore: fix lint
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-21 08:12:54 +00:00
Andrew Lamb
9974a5364c
chore(security): Replace prettytable with comfy-table ( #2905 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-20 10:44:36 +00:00
Andrew Lamb
a82dc6f5f0
chore: Update datafusion + arrow ( #2903 )
...
* chore: Update datafusion to latest, arrow to 6.0.0
* fix: Update tests
* fix: bubble internal error
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-19 17:14:08 +00:00
kodiakhq[bot]
0be89715e2
Merge branch 'main' into dependabot/cargo/rdkafka-0.27.0
2021-10-19 12:55:21 +00:00
Marco Neumann
7dafb0ff74
chore: remove unused `influxdb_iox` => `synchronized-writer` dep
2021-10-19 14:46:26 +02:00
dependabot[bot]
32e18b6436
chore(deps): bump rdkafka from 0.26.0 to 0.27.0
...
Bumps [rdkafka](https://github.com/fede1024/rust-rdkafka ) from 0.26.0 to 0.27.0.
- [Release notes](https://github.com/fede1024/rust-rdkafka/releases )
- [Changelog](https://github.com/fede1024/rust-rdkafka/blob/master/changelog.md )
- [Commits](https://github.com/fede1024/rust-rdkafka/compare/v0.26.0...v0.27.0 )
---
updated-dependencies:
- dependency-name: rdkafka
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-19 11:49:34 +00:00
Marco Neumann
157b556d4c
feat: make flight responses streaming ( #2876 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-18 15:01:50 +00:00
dependabot[bot]
70555ab33d
chore(deps): bump structopt from 0.3.23 to 0.3.25 ( #2877 )
...
Bumps [structopt](https://github.com/TeXitoi/structopt ) from 0.3.23 to 0.3.25.
- [Release notes](https://github.com/TeXitoi/structopt/releases )
- [Changelog](https://github.com/TeXitoi/structopt/blob/master/CHANGELOG.md )
- [Commits](https://github.com/TeXitoi/structopt/compare/v0.3.23...v0.3.25 )
---
updated-dependencies:
- dependency-name: structopt
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-10-18 14:49:15 +00:00
Raphael Taylor-Davies
bdd6d67e7a
refactor: split out mutable_batch crate ( #2841 )
...
* refactor: split out mutable_batch crate
* refactor: restore chunk module for better diffs
* chore: fmt
* chore: review feedback
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-14 16:46:54 +00:00
Marco Neumann
28195b9c0c
chore: new `parquet_catalog` crate
2021-10-14 14:34:59 +02:00
Raphael Taylor-Davies
0554173684
feat: migrate write buffer to TimeProvider ( #2722 ) ( #2804 )
...
* feat: migrate write buffer to TimeProvider (#2722 )
* chore: review feedback
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: Marco Neumann <marco@crepererum.net>
2021-10-12 10:32:34 +00:00
Raphael Taylor-Davies
7bd32a4916
feat: add time provider abstraction ( #2722 ) ( #2792 )
...
* feat: add time provider abstraction (#2722 )
* chore: tests
* chore: more tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-11 15:00:29 +00:00
dependabot[bot]
61190a9d08
chore(deps): bump predicates from 2.0.2 to 2.0.3
...
Bumps [predicates](https://github.com/assert-rs/predicates-rs ) from 2.0.2 to 2.0.3.
- [Release notes](https://github.com/assert-rs/predicates-rs/releases )
- [Changelog](https://github.com/assert-rs/predicates-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/predicates-rs/compare/v2.0.2...v2.0.3 )
---
updated-dependencies:
- dependency-name: predicates
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-11 14:32:08 +00:00
Raphael Taylor-Davies
afe34751e7
refactor: split out schema crate ( #2781 )
...
* refactor: split out schema crate
* chore: fix doc
2021-10-11 09:45:08 +00:00
dependabot[bot]
1327735537
chore(deps): bump thiserror from 1.0.29 to 1.0.30
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.29 to 1.0.30.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.29...1.0.30 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-11 08:01:21 +00:00
dependabot[bot]
01a44899fd
chore(deps): bump assert_cmd from 2.0.1 to 2.0.2
...
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd ) from 2.0.1 to 2.0.2.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases )
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.1...v2.0.2 )
---
updated-dependencies:
- dependency-name: assert_cmd
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-11 01:45:35 +00:00
dependabot[bot]
7a6f381e69
chore(deps): bump arrow-flight from 5.4.0 to 5.5.0
...
Bumps [arrow-flight](https://github.com/apache/arrow-rs ) from 5.4.0 to 5.5.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/5.5.0/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/5.4.0...5.5.0 )
---
updated-dependencies:
- dependency-name: arrow-flight
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-04 13:27:26 +00:00
kodiakhq[bot]
181145eca1
Merge branch 'main' into dependabot/cargo/arrow-5.5.0
2021-10-04 13:10:42 +00:00
dependabot[bot]
6d9205d87c
chore(deps): bump parquet from 5.4.0 to 5.5.0
...
Bumps [parquet](https://github.com/apache/arrow-rs ) from 5.4.0 to 5.5.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/5.5.0/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/5.4.0...5.5.0 )
---
updated-dependencies:
- dependency-name: parquet
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-04 12:15:11 +01:00
dependabot[bot]
d1f5209869
chore(deps): bump arrow from 5.4.0 to 5.5.0
...
Bumps [arrow](https://github.com/apache/arrow-rs ) from 5.4.0 to 5.5.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/5.5.0/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/5.4.0...5.5.0 )
---
updated-dependencies:
- dependency-name: arrow
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-04 08:55:38 +00:00
Raphael Taylor-Davies
86cee568d5
feat: use upstream pbjson ( #2650 )
...
* feat: use upstream pbjson
* chore: fmt
2021-09-28 16:29:26 +00:00
Raphael Taylor-Davies
daa2ec2f4c
revert: "fix: Revert "feat: remove routerify ( #2586 )" ( #2623 )" ( #2649 )
...
This reverts commit 18941fcbff
.
2021-09-28 14:42:17 +00:00
Andrew Lamb
18941fcbff
fix: Revert "feat: remove routerify ( #2586 )" ( #2623 )
...
This reverts commit 9fba7b0ba0
.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-24 09:50:24 +00:00
Raphael Taylor-Davies
92e6173f49
feat: implement jaeger-agent protocol directly ( #2607 )
...
* feat: implement jaeger-agent protocol directly
* chore: review feedback
* fix: remove jaeger feature flag
2021-09-22 17:30:37 +00:00
Raphael Taylor-Davies
46088a7ff1
feat: disable rustyline dirs-next dependency ( #2579 )
2021-09-20 14:42:07 +00:00
Raphael Taylor-Davies
9fba7b0ba0
feat: remove routerify ( #2586 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-20 08:57:19 +00:00
dependabot[bot]
1112597c44
chore(deps): bump dirs from 3.0.2 to 4.0.0
...
Bumps [dirs](https://github.com/soc/dirs-rs ) from 3.0.2 to 4.0.0.
- [Release notes](https://github.com/soc/dirs-rs/releases )
- [Commits](https://github.com/soc/dirs-rs/commits )
---
updated-dependencies:
- dependency-name: dirs
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-20 08:28:04 +00:00
Raphael Taylor-Davies
3ffb16daa6
feat: remove parse_duration ( #2574 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-19 11:27:24 +00:00
Raphael Taylor-Davies
1d55d9a1b5
feat: add pbjson support ( #2468 )
...
* feat: add pbjson support
* chore: fix test
2021-09-16 07:33:27 +00:00
Raphael Taylor-Davies
c66095cad1
feat: remove metrics crate ( #2552 )
2021-09-15 19:43:33 +00:00
Raphael Taylor-Davies
6e7fa3e574
feat: migrate http ingest metrics ( #2542 )
...
* feat: migrate http ingest metrics
* chore: review feedback
* refactor: RAII entry ingest recorder
2021-09-15 19:01:10 +00:00
Marco Neumann
bfaba78dc3
refactor: move `predicate` into its own crate
...
Two reasons:
1. I wanna decouple `parquet_file` from `query` (nearly done, needs a
small follow-up PR).
2. `predicate` will have more and more features (like serialization)
which justifies a new home
2021-09-14 17:13:02 +02:00
Marco Neumann
3f2e46c397
feat: prune old transactions from preserved catalog
2021-09-14 12:08:17 +02:00
Raphael Taylor-Davies
e07bc85670
feat: enable tracing for http API ( #2273 ) ( #2498 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-10 11:21:02 +00:00
dependabot[bot]
b67610d9b9
chore(deps): bump tokio from 1.10.1 to 1.11.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.10.1 to 1.11.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.10.1...tokio-1.11.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 09:11:38 +00:00
kodiakhq[bot]
06311970fa
Merge branch 'main' into dependabot/cargo/parquet-5.3.0
2021-09-06 09:00:27 +00:00
dependabot[bot]
ccfd6ad3da
chore(deps): bump parquet from 5.2.0 to 5.3.0
...
Bumps [parquet](https://github.com/apache/arrow-rs ) from 5.2.0 to 5.3.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/5.3.0/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/5.2.0...5.3.0 )
---
updated-dependencies:
- dependency-name: parquet
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 08:52:30 +00:00
kodiakhq[bot]
0fd92c8b45
Merge branch 'main' into dependabot/cargo/structopt-0.3.23
2021-09-06 08:51:32 +00:00
dependabot[bot]
b1bb390893
chore(deps): bump parking_lot from 0.11.1 to 0.11.2
...
Bumps [parking_lot](https://github.com/Amanieu/parking_lot ) from 0.11.1 to 0.11.2.
- [Release notes](https://github.com/Amanieu/parking_lot/releases )
- [Changelog](https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md )
- [Commits](https://github.com/Amanieu/parking_lot/compare/0.11.1...0.11.2 )
---
updated-dependencies:
- dependency-name: parking_lot
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 01:18:24 +00:00
dependabot[bot]
2cf9505029
chore(deps): bump structopt from 0.3.22 to 0.3.23
...
Bumps [structopt](https://github.com/TeXitoi/structopt ) from 0.3.22 to 0.3.23.
- [Release notes](https://github.com/TeXitoi/structopt/releases )
- [Changelog](https://github.com/TeXitoi/structopt/blob/master/CHANGELOG.md )
- [Commits](https://github.com/TeXitoi/structopt/compare/v0.3.22...v0.3.23 )
---
updated-dependencies:
- dependency-name: structopt
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 01:17:41 +00:00
Marco Neumann
09a5d27434
chore: ignore massif profiler outputs
2021-09-02 15:58:43 +02:00
Marco Neumann
4a863993ec
feat: "dump catalog" debug CLI
2021-09-02 08:08:20 +02:00
Jacob Marble
6ba25728e1
chore: exclude non-source code from package ( #2452 )
...
Before this change, modifications to the following files would trigger a
rebuild on `cargo build`.
```
2,13d1
< .circleci/config.yml
< .circleci/get-deploy-tags.sh
< .editorconfig
< .gitattributes
< .github/ISSUE_TEMPLATE/bug_report.md
< .github/ISSUE_TEMPLATE/feature_request.md
< .github/PULL_REQUEST_TEMPLATE.md
< .github/dependabot.yml
< .github/semantic.yml
< .gitignore
< .kodiak.toml
< CONTRIBUTING.md
17,22d4
< Dockerfile
< Dockerfile.dockerignore
< LICENSE-APACHE
< LICENSE-MIT
< README.md
< buf.yaml
24,76d5
< docker/Dockerfile.ci
< docker/Dockerfile.ci.dockerignore
< docker/Dockerfile.ci.integration
< docker/Dockerfile.ci.integration.dockerignore
< docker/Dockerfile.iox
< docker/Dockerfile.iox.dockerignore
< docker/ci-kafka-docker-compose.yml
< docker/integration_test.sh
< docs/README.md
< docs/catalog_persistence.md
< docs/data_management.md
< docs/drawings/data_lifecycle.monopic
< docs/drawings/data_organization.monopic
< docs/encoding_thoughts.md
< docs/env.example
< docs/images/flame_graph.png
< docs/local_filesystems.md
< docs/metrics.md
< docs/multi_core_tasks.md
< docs/profiling.md
< docs/protobuf.md
< docs/regenerating_flatbuffers.md
< docs/server_startup.md
< docs/sql.md
< docs/style_guide.md
< docs/testing.md
< docs/tracing.md
< docs/valgrind.md
< perf/logs/iox_router.log
< perf/logs/iox_writer.log
< perf/logs/jaeger.log
< perf/logs/kafka.log
< perf/logs/minio.log
< perf/logs/zookeeper.log
< perf/volumes/minio/.minio.sys/buckets/.bloomcycle.bin
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/.bloomcycle.bin/fs.json
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/.usage-cache.bin/fs.json
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/.usage.json/fs.json
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/iox1/.usage-cache.bin/fs.json
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/iox2/.usage-cache.bin/fs.json
< perf/volumes/minio/.minio.sys/buckets/.tracker.bin
< perf/volumes/minio/.minio.sys/buckets/.usage-cache.bin
< perf/volumes/minio/.minio.sys/buckets/.usage.json
< perf/volumes/minio/.minio.sys/buckets/iox1/.metadata.bin
< perf/volumes/minio/.minio.sys/buckets/iox1/.usage-cache.bin
< perf/volumes/minio/.minio.sys/buckets/iox2/.metadata.bin
< perf/volumes/minio/.minio.sys/buckets/iox2/.usage-cache.bin
< perf/volumes/minio/.minio.sys/config/config.json
< perf/volumes/minio/.minio.sys/config/iam/format.json
< perf/volumes/minio/.minio.sys/format.json
< perf/volumes/zookeeper/data/myid
< perf/volumes/zookeeper/data/version-2/log.1
< perf/volumes/zookeeper/data/version-2/snapshot.0
79,87d7
< scripts/edit_db_rules
< scripts/genlp.py
< scripts/git-log-prs
< scripts/grpcurl
< scripts/jq_edit_db_rules
< scripts/logfmts
< scripts/prototxt
< scripts/valgrind.supp
< scripts/valigrind_leak
161d80
< tools/iplan
```
2021-08-31 17:14:02 +00:00
dependabot[bot]
3e154b3b02
chore(deps): bump serde_json from 1.0.66 to 1.0.67
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.66 to 1.0.67.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.66...v1.0.67 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-30 11:28:35 +00:00
dependabot[bot]
52fbaef2df
chore(deps): bump thiserror from 1.0.26 to 1.0.28
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.26 to 1.0.28.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.26...1.0.28 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-30 11:05:11 +00:00
dependabot[bot]
13161de003
chore(deps): bump rustyline from 8.2.0 to 9.0.0
...
Bumps [rustyline](https://github.com/kkawakam/rustyline ) from 8.2.0 to 9.0.0.
- [Release notes](https://github.com/kkawakam/rustyline/releases )
- [Changelog](https://github.com/kkawakam/rustyline/blob/master/History.md )
- [Commits](https://github.com/kkawakam/rustyline/compare/v8.2.0...v9.0.0 )
---
updated-dependencies:
- dependency-name: rustyline
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-30 01:19:49 +00:00
Raphael Taylor-Davies
fcec394a28
feat: connect up new metrics ( #2428 )
2021-08-27 12:55:35 +00:00
Raphael Taylor-Davies
3a26f7857f
feat: add new metrics abstraction ( #2076 ) ( #2365 )
...
* feat: add new metrics abstraction (#2076 )
* chore: fix lint
* feat: add tests
feat: histogram export
* chore: further docs
* chore: docs
* chore: log on prometheus export error
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-27 11:54:23 +00:00
Raphael Taylor-Davies
1773bf5d37
feat: add storage client to influxdb_iox_client ( #2404 )
...
* feat: add storage client to influxdb_iox_client
* chore: fix type_url
* refactor: split storage into separate crate
* chore: fix doctest
* chore: review feedback
* chore: add generated_types cleanup ticket
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-26 10:30:59 +00:00
Raphael Taylor-Davies
2c9c191b17
refactor: split out trace http ( #2388 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-24 13:37:20 +00:00
Raphael Taylor-Davies
3fdc0e9a6a
chore: remove orphaned code from trogging ( #2371 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-24 10:12:59 +00:00
Raphael Taylor-Davies
58716dee71
refactor: remove mem-qe ( #2344 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-23 14:35:12 +00:00
dependabot[bot]
0d4bc4190f
chore(deps): bump assert_cmd from 1.0.8 to 2.0.0
...
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd ) from 1.0.8 to 2.0.0.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases )
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v1.0.8...v2.0.0 )
---
updated-dependencies:
- dependency-name: assert_cmd
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-23 01:17:06 +00:00
Marko Mikulicic
bbbd38bb12
fix(heappy): Prevent re-entering a running heap profile
...
Upgrades heappy to include https://github.com/mkmik/heappy/pull/2
and adapts to the API change.
2021-08-20 14:36:36 +02:00
Marko Mikulicic
4a2f0b0a52
fix: Make heappy and nonheappy mutually exclusive
...
Closes #2288
2021-08-20 11:35:07 +02:00
Raphael Taylor-Davies
db97069ecd
feat: jaeger span export ( #2273 ) ( #2348 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-20 08:57:16 +00:00
Paul Dix
d5f01a2a68
refactor: move data generator to IOx repo and fix build
2021-08-19 14:26:15 -04:00
Raphael Taylor-Davies
04ba139cb2
refactor: split trace crate ( #2343 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-19 11:00:32 +00:00
Andrew Lamb
5e1cb244f7
feat: make pprof optional ( #2331 )
2021-08-18 15:13:37 +00:00
Marko Mikulicic
a50b1646bd
fix(heappy): Report free as positive numbers
2021-08-18 14:30:41 +02:00
Marko Mikulicic
fc473995fa
feat(iox): Upgrade heappy to present free memory events
...
Currently heappy is reporting "bytes allocated" and "inuse_bytes" ("bytes allocated - freed bytes").
Since the allocation site and free site can be wildly different, it's very hard to correlate these events
and have a meaningful "inuse_bytes" graph.
I have a plan for how to fix this for good, but in the meantime
I'd like to report raw "freed bytes".
2021-08-18 13:04:48 +02:00
Raphael Taylor-Davies
1d6a8703af
feat: support sinking traces to an OTEL SpanExporter ( #2319 )
...
* feat: support sinking traces to an OTEL SpanExporter
* chore: consistent logging
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* chore: review feedback
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-18 08:05:21 +00:00
Raphael Taylor-Davies
efa776bd03
feat: fix log crate output ( #2325 )
...
* feat: fix log crate output
* chore: fix doc
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-17 16:19:22 +00:00
Marko Mikulicic
94ff77126e
feat: Enable freed memory tracking in heappy
...
In aed37ab50a
I fixed the main
problem that rendered free tracking useless: the allocator was actually allocating more bytes than requested; we were tracking the amount of memory requested by the user and not the size of the block that was returned to the user. However, when we were tracking a call to free we used the size of the memory block which was bigger than the amount that was tracked; this led to negative numbers of "in use memory".
2021-08-17 15:09:28 +02:00
dependabot[bot]
1d85a3fef6
build(deps): bump predicates from 1.0.8 to 2.0.2
...
Bumps [predicates](https://github.com/assert-rs/predicates-rs ) from 1.0.8 to 2.0.2.
- [Release notes](https://github.com/assert-rs/predicates-rs/releases )
- [Changelog](https://github.com/assert-rs/predicates-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/predicates-rs/compare/v1.0.8...v2.0.2 )
---
updated-dependencies:
- dependency-name: predicates
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-16 15:36:49 +00:00
Marko Mikulicic
500019cff9
Merge branch 'main' into ntran/heappy2
2021-08-13 17:27:38 +02:00
Raphael Taylor-Davies
8650e419f7
feat: add trace collection plumbing ( #2273 ) ( #2274 )
...
* feat: add trace collection plumbing (#2273 )
* chore: fix lints
* chore: use tower directly
* chore: add tests
* chore: review feedback
* refactor: use serde instead of custom display impl
* chore: fix doc
2021-08-13 14:38:31 +00:00
Andrew Lamb
cedaf2d5f2
fix: remove unprefixed_malloc_on_supported_platforms feature from tikv-jemallocator
2021-08-13 08:24:15 -05:00
Andrew Lamb
697de875ca
refactor: make heappy an optional feature, add to build image
2021-08-13 08:11:32 -05:00
Nga Tran
4fd5862a9c
fix: turn heappy on
2021-08-12 17:14:03 -04:00
Carol (Nichols || Goulding)
ae6b0e669b
refactor: Extract a database persister type that wraps object store
...
Connects to #2193 .
2021-08-12 15:05:32 -04:00
Nga Tran
4f52a70d5b
refactor: adddress mkm's comments
2021-08-12 12:04:42 -04:00
Nga Tran
8aa9514d93
feat: integrate heappy implemented by mkm
2021-08-11 23:42:50 -04:00
Raphael Taylor-Davies
1f450ef371
feat: add Database abstraction ( #2186 ) ( #2203 )
...
* feat: add Database abstraction
* chore: minor tweaks
* chore: remove redundant test fixture restart
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-08 17:14:23 +00:00
Andrew Lamb
82677d9510
feat: add `jaeger` and `otlp` features, default to off ( #2202 )
...
* feat: add jaeger and otlp flags
* chore: add jaeger and otlp features to CI test and deploy image
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-06 13:26:59 +00:00
Andrew Lamb
e69a159ad2
refactor: remove unused opentelemetry dependencies ( #2201 )
2021-08-05 16:20:22 +00:00
Andrew Lamb
e92e94caad
chore: Update deps (including arrow 5.1.0, tonic -> 0.5, and prost 0.5) ( #2172 )
...
* chore: Update deps (including arrow 5.0.0 --> arrow 5.1.0)
* chore: update all the things
* refactor: Update serving readiness check due to change in Tonic API
* chore: update more deps
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-05 15:57:38 +00:00
Andrew Lamb
7ca31703a4
feat: Attempt to dump a stacktrace to stderr prior to process abort on SIGSEGV/SIGILL/SIGBUS ( #2155 )
...
* feat: Attempt to dump a stacktrace to stdout prior to process abort
* refactor: rewrite signal handling in terms of libc, remove sig dep
* refactor: print to stderr
* fix: Update src/main.rs
* docs: note provenance
2021-07-30 11:58:28 +00:00
Andrew Lamb
4def5bd816
chore: Add print_cpu command line to perf build ( #2114 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-26 15:26:34 +00:00
Andrew Lamb
4da8a16c18
chore: update to arrow 5.0 and master datafusion ( #2049 )
...
* chore: update to arrow 5.0 and master datafusion
* fix: Update test for change in object size
2021-07-19 12:49:51 +00:00
Marco Neumann
b5428e53a5
refactor: write buffer testing + better mocking
...
This refactors the write buffer a bit for:
- **Testing:** Add generic tests for the Kafka and the mocking
implementation. The same interface can be used easily add new
implementations (e.g. via Redis, filesystem, ...).
- **Partition on Write:** The caller of the writer operation must now
specify the partition/sequencer ID. The implicit partitioning of the
Kafka writer would have lead to broken data since we must never spill
entries w/ the same primary key over multiple partitions. At the
moment we will only use partition 0 but we can easily implement
better logic in the future.
- **Improved Mocking:** The mocked implementation now simulates a system
that feels more real. Especially the handling around multiple streams
and "write while read" has been improved. This will be helpful for
testing and for new features like seeking (during replay). A solid
realistic mock also helps us to ensure that the tests using the mock
do not rely on unrealistic behavior too much.
2021-07-15 17:20:45 +02:00
Edd Robinson
0e5276ed20
Merge branch 'main' into alamb/go_go_go_go
2021-07-14 13:56:35 +01:00
Marco Neumann
9cb9ae0874
chore: move write buffer into its own crate
2021-07-14 14:09:18 +02:00
Andrew Lamb
4800b36949
chore: Update IOx to a pre-release version of arrow and datafusion to test out performance improvement
2021-07-13 15:44:57 -04:00
Andrew Lamb
1a79bf7e99
refactor: Make aws/azure/gcs optional features and stop compiling 100 dependencies during dev ( #1933 )
...
* feat: make aws, gcp, azure dependencies optional
* fix: only run object store tests if the features are enabled
* fix: clean up testing
* fix: rename step
* fix: add to list of jobs
* fix: remove test with object store
* fix: review comments
2021-07-09 11:38:30 +00:00
Marco Neumann
4ca2d3e148
chore: move persistence windows related code into own crate
...
The entire persistence windows data structures (including the
checkpoints) have nothing to do with the mutable buffer per se. So lets
move them into their own crate. This also makes `parquet_file` not
longer depend on `mutable_buffer`.
2021-07-05 10:23:58 +02:00
Marko Mikulicic
69b0bb1510
feat: Implement grpc-router crate
...
This PR implements the main building block for implementing the gRPC StorageService router.
2021-06-23 17:21:46 +02:00
Marco Neumann
55c546baff
feat: eagerly check object store during CLI `run`
...
Instead of waiting for the server ID to be set and then mark the server
as errored, directly check the object store on startup. This is
important so that we fail fast when Istio isn't up and running yet.
2021-06-22 18:21:30 +02:00
Carol (Nichols || Goulding)
b4644e6108
test: Start of Kafka Write Buffer integration tests
2021-06-21 09:41:35 -04:00
Andrew Lamb
ec43a87909
chore: Update itertools deps ( #1750 )
2021-06-17 17:56:44 +00:00
Marko Mikulicic
760bcde3f0
feat: Factor out tracing/logging CLI options
...
This PR factors out the tracing/logging CLI optinos into the `trogging` utility crate,
so that multiple binaries from the IOx suite (such as conductor) can use the same (and quite complex)
logging/tracing configuration options (flags and env vars).
Closes influxdata/conductor#343
2021-06-16 00:54:11 +02:00
Raphael Taylor-Davies
bf54ab51f2
refactor: split lifecycle into separate crate ( #1730 )
2021-06-15 15:57:47 +00:00
Marko Mikulicic
bde35cf5be
chore: Pull tracing+logging setup in its own crate
...
1. so that it can be reused by other binaries (e.g. conductor)
2. so that it's faster to build when working on it.
2021-06-15 14:52:04 +02:00
Marko Mikulicic
5a68abaa53
feat: Implement LayeredTracing
...
__Rationale__
We currently use the `tracing` framework to output to both log outputs (e.g. stdout for k8s) and distributed tracing collectors (e.g. opentelemetry jaeger).
However, due to a limitation in the `tracing` SDK, we can only have one "filter" level that applies
to both logs and tracing outputs. This is unpractical because tracing collectors are designed
to receive high verbosity data (which will be then sampled within the opentelemetry library),
while logs generally are limited to the DEBUG level on production.
This PR adds a `FilteredLayer` tracing subscriber layer, that wraps a subscriber layer with a independent
filter, which can filter events goint to the wrapper subscriber layer more agressively than the global layer.
This will allow us to emit logs at INFO or DEBUG level while passing all events to opentelemetry at TRACE
level (and opentelemetry SDK will then sample the events so that only a small part will be sent to the
ot collector)
__Note__
This PR just implements the `FilteredLayer` and a test. Another PR will integrate this with
our log/tracing setup code.
2021-06-11 04:32:47 +02:00
Andrew Lamb
a614fef5bc
chore: remove more unused dependencies ( #1658 )
...
* chore: remove more unused deps
* refactor: move benchmarks into server_benchmarks crate
2021-06-09 10:17:20 +00:00
kodiakhq[bot]
87297f7db4
Merge branch 'main' into cn/delete
2021-06-07 13:32:42 +00:00
Raphael Taylor-Davies
5749a2c119
chore: cleanup legacy TSM -> parquet code ( #1639 )
...
* chore: cleanup legacy parquet code
* chore: remove tests of removed functionality
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-07 12:59:33 +00:00
Carol (Nichols || Goulding)
f4a9a5ae56
fix: Remove write buffer
2021-06-04 14:40:17 -04:00
Andrew Lamb
42f26b609b
refactor: Move `query_tests` and `server_benchmarks` into their own crate --> smaller `server` ( #1628 )
...
* refactor: Separate query_tests into its own crate
* fix: references
* refactor: break out server benchmarks
* fix: Update query_tests/src/lib.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-06-04 17:31:19 +00:00
Andrew Lamb
00e735ef0d
chore: remove unused dependencies ( #1583 )
2021-05-29 10:31:57 +00:00
Raphael Taylor-Davies
4fcc04e6c9
chore: enable arrow prettyprint feature ( #1566 )
2021-05-27 10:28:14 +00:00
Marko Mikulicic
0a37cb80f7
feat: Use jemalloc everywhere instead of malloc
...
There may be many reasons for the discrepancy in jemalloc reported allocations total sizes and RSS.
One of them is that our binary doesn't use jmalloc for all the allocations.
Turns out that jemallocator only sets the global rust allocator. Any call to `malloc` will still
go throught the system allocator. Presumably those calls come from linked C code,
but it's also not impossible that not all rust code honours the global allocator (I have no idea, but let's see)
2021-05-25 03:42:52 +02:00
Andrew Lamb
14ba25f86d
chore: Update datafusion and use released version of arrow crates ( #1546 )
...
* chore: Update datafusion and use released version of arrow crate
* fix: Update for change in API
2021-05-24 15:37:22 +00:00
Marko Mikulicic
40b21dbca1
feat: Add /debug/pprof/profile
2021-05-19 13:08:42 +02:00
Andrew Lamb
ebea554e5a
feat: Report num_cpus seen by IOx on startup ( #1484 )
2021-05-12 13:42:16 +00:00
Raphael Taylor-Davies
10f89a3e8d
refactor: split entry out into separate crate ( #1428 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 11:36:23 +00:00
Raphael Taylor-Davies
7e2e2c4caf
refactor: initial split of observability_deps ( #1429 )
2021-05-06 08:25:09 +00:00
kodiakhq[bot]
7e01650c53
Merge branch 'main' into reflection
2021-05-05 20:29:13 +00:00
Marko Mikulicic
0d6d94dc00
feat: Enable gRPC reflection
2021-05-05 22:04:47 +02:00
Raphael Taylor-Davies
411cf134e9
refactor: explode arrow_deps ( #1425 )
...
* refactor: explode arrow_deps
* chore: workaround doctest bug
2021-05-05 16:59:12 +00:00
Marko Mikulicic
b579ef8646
feat: Add jemalloc stats
2021-05-03 12:10:48 +02:00
Andrew Lamb
a64e622f6e
feat: Add interactive SQL repl ( #1332 )
...
* feat: Add interactive SQL repl
* fix: try and fix test
* fix: remove test for prompt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-28 20:41:02 +00:00
Edd Robinson
ea909f45ad
feat: RED metrics
2021-04-23 15:58:48 +00:00
Carol (Nichols || Goulding)
f0b93c5c8c
refactor: Rename the wal crate to write_buffer
2021-04-21 17:43:03 +00:00
Jacob Marble
df9dd15ab8
feat(tracing): improve logs and tracing
...
Logs and traces are emitted via one pipeline. For now, it is not
possible to emit both, but it should be possible in a few weeks, as
tokio/tracing/tracing-subscriber is going through some refactoring recently.
All affected flags are well-documented, and I have tested all but the
OTLP output flags.
chore: clippy happy
chore: revert instrumentation changes
feat: add log format logfmt, log destinations stderr, stdout
chore: clippy happy
2021-04-19 15:48:30 -07:00
Raphael Taylor-Davies
1997324344
feat: mutable buffer snapshotting ( #1179 )
...
* feat: mutable buffer snapshotting
* chore: review feedback
2021-04-13 12:14:54 +00:00
Marko Mikulicic
e76980928b
feat: Implement Update API
2021-04-08 22:25:36 +00:00
Marko Mikulicic
df5349406a
feat: Switch to jemalloc
2021-04-08 14:19:35 +00:00
Andrew Lamb
03c446ed43
feat: add line protocol ingest metrics ( #1119 )
...
* feat: add line protocol ingest metrics
* fix: ensure tests pass correctly when run multi-threaded
* refactor: remove uneeded change
* fix: Add tags to metrics, improve test
* refactor: use global `Counter`
* refactor: move metrics to crate wide struct
* fix: fix test
2021-04-07 21:49:02 +00:00
Raphael Taylor-Davies
9a2e636d8c
refactor: move tracking utilities into separate crate ( #1124 )
2021-04-06 11:43:11 +00:00
Jacob Marble
80d55d0829
chore: rename tracing_deps to observability_deps
...
OpenTelemetry makes this necessary.
2021-04-02 13:14:30 -07:00