e7e9945207
chore: Update datafusion pin ( #6017 )
...
* chore: Update datafusion pin
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-02 11:12:21 +00:00
00953460fb
chore: Update datafusion pin ( #6010 )
2022-10-31 17:14:56 +00:00
ace3c11f12
chore: Update datafusion ( #6004 )
...
* chore: Update datafusion
* chore: change path
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-31 16:16:28 +00:00
b27e2bd7d1
feat: Use workspace dep inheritance for the parquet crate
2022-10-26 10:37:51 -04:00
88c3a1f5e7
feat: Use workspace dep inheritance for the arrow-flight crate
2022-10-26 10:34:54 -04:00
3145e2c05b
feat: Use workspace dep inheritance for the arrow crate
2022-10-26 10:34:29 -04:00
44936f661a
feat: Use workspace dep inheritance for datafusion instead of shim crate
2022-10-26 10:33:56 -04:00
99f36f8889
chore: unpatch clap V3 ( #5971 )
...
Upstream issue was solved by a new release.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-25 09:16:16 +00:00
2e83e04eab
feat: Use workspace package metadata to reduce differences and repetition
2022-10-24 13:04:09 -04:00
4ca869fcd9
fix: textwrap 0.15.1 was yanked ( #5953 )
...
Need to pull an unreleased clap v3 so we no longer depend on a yanked
version of textwrap. Also see https://github.com/clap-rs/clap/issues/4418
2022-10-24 07:19:51 +00:00
76d3fbbe53
refactor: remove old `packers` crate ( #5893 )
...
* refactor: remove old packers code
* refactor: Remove from workspace
2022-10-18 19:00:28 +00:00
9b7a7b5e40
refactor: remove read buffer ( #5891 )
2022-10-18 12:46:17 +00:00
81722dc19b
feat: AST traversal using Visitor pattern ( #5796 )
...
* feat: Partition implementation of Visitable for InfluxQL AST
* feat: Added consistent structures for each clause to simplify visitor
Continued to expand `accept` and `pre` / `post` visit implementations.
* feat: Added insta and tests using snapshots (thanks @crepererum)
The insta crate simplifies the process of validating the combination of
visitor and accept implementations are called and in the correct order.
* chore: Run cargo hakari tasks
* feat: Added remaining snapshot tests
Some tests are failing as some minor type changes must be added along
with the addition of related visitor functions.
* feat: Add types to represent each clause in numerous statements
These clauses permit distinct visit functions on the `Visitor` type.
* chore: Reformat `SELECT`
* chore: Explicitly specify access to export selected types only
This required completing all the missing documentation for the exported
types.
* chore: Update Cargo.lock
* chore: macro to implement common traits and hide 0th tuple element
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-10-13 22:37:49 +00:00
d57c99638c
chore: Update datafusion + `arrow`, `arrow-flight`, and `parquet` to 24.0.0.0 ( #5792 )
...
* chore: Update datafusion + `arrow`, `arrow-flight`, and `parquet` to 24.0.0.0
* fix: Update for coercion, fix explain plans for change in column name display
* chore: Update datafusion lock
* fix: Update for other API changes
* chore: Update to latest datafusion pin
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-12 16:19:14 +00:00
66dbb9541f
chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to 23.0.0, `thrift` to 0.16.0 ( #5694 )
...
* chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to 23.0.0
* chore: Update thrift / remove parquet_format
* fix: Update APIs
* chore: Update lock + Run cargo hakari tasks
* fix: use patched version of arrow-rs to work around https://github.com/apache/arrow-rs/issues/2779
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-27 12:50:54 +00:00
65f1550126
feat: Implement `debug parquet_to_lp` command to convert parquet to line protocol ( #5734 )
...
* feat: add `influxdb_iox debug parquet_to_lp` command
* chore: Run cargo hakari tasks
* fix: update command description
* fix: remove unecessary Result import
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2022-09-26 14:17:27 +00:00
f86d3e31da
chore: Update datafusion + object_store ( #5619 )
...
* chore: Update datafusion pin
* chore: update object_store to 0.5.0
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-13 12:34:54 +00:00
15369d4c52
feat: Implement gRPC binary logger ( #5473 )
...
* feat: Implement gRPC binary logger
* chore: Run cargo hakari tasks
* fix: Apply suggestions from code review
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-01 10:57:02 +00:00
711ba77341
chore: update object_store to test IMDSv1 fallback ( #5509 )
...
* chore: update object_store to test IMDSv1 fallback
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-30 12:31:49 +00:00
66f0b59dbb
revert: remove Azure SDK / bump object_store
...
This reverts commit c2f8efa03a
2022-08-30 10:41:29 +02:00
c2f8efa03a
build: remove Azure SDK / bump object_store
...
Bumps the object_store pin to master to pick up:
https://github.com/apache/arrow-rs/pull/2509
This removes the Azure SDK.
2022-08-29 14:06:51 +02:00
b4e5895d7a
feat: Add influxdb_influxql_parser crate ( #5415 )
...
* feat: Add crate; parse quoted identifiers
* chore: Run cargo hakari tasks
* chore: satisfy linter
* chore: Use `test_helpers::Result`
* feat: Add all InfluxQL keywords
* chore: Update influxdb_influxql_parser/src/lib.rs
Co-authored-by: Marco Neumann <marco@crepererum.net>
* chore: PR feedback
* chore: PR Feedback, remove Result<()>
* chore: Update Cargo.lock
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
2022-08-18 23:09:45 +00:00
7f0ae53d6f
chore: Update to (almost) released object_store 0.4.0 ( #5419 )
...
* chore: update object_store
* chore: update hakari config
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-08-17 13:44:48 +00:00
c5f062bba0
feat: initial commit of schema merge bulk import tool ( #5344 )
...
* feat: initial commit of schema merge bulk import tool
* chore: use observability depds instead of tracing-*
* chore: removed debug printlns
* chore: fix feature decls for cloud providers for import crate
* chore: use println instead of info in import- no need for a simple CLI
* chore: tidy whitespace
* chore: remove unused dep in import
* chore: Run cargo hakari tasks
* chore: removed unimpld import job subcommand
* chore: clarifying comment about custom serialisation code
* chore: clarifying comment about schema merge code in import
* chore: fix wrong comment in import command
* chore: bump object store dep to get bugfix
* chore: rename import schema struct for clarity
* chore: run `cargo hakari generate`
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-10 09:07:38 +00:00
dadcc369b1
chore: update object_store to fix credentials client ( #5359 )
...
* chore: update object_store to fix credentials client
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-08-09 13:17:43 +00:00
ccb45d7bac
chore: update to rusoto-less object_store ( #5342 )
...
* chore: update to rusoto-less object_store
* chore: Run cargo hakari tasks
* chore: further fixes
* chore: document workaround
* chore: review feedback
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-08-09 09:06:03 +00:00
284a3069ce
feat: Kafka client produce() instrumentation
...
Adds a decorator over the underlying kafka client to capture the latency
distribution of the low-level kafka writes, independent of the
aggregation/DML batching framework that sits "above" this client.
The latency measurements include the serialisation overhead, protocol
overhead, and actual network I/O.
2022-08-08 15:24:35 +02:00
e4dc8c2067
refactor: rename garbage collector crates for consistency ( #5196 )
...
* refactor: rename garbage collector crates for consistency
* fix: cargo fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-25 12:44:37 +00:00
f7a0fd43d2
feat: make object store garbage collector into a long-running service ( #5135 )
...
* refactor: remove unused logging config
* chore: remove the object store garbage collector CLI tool
* refactor: accept an object store and catalog
* refactor: make Result type alias public like the error
* refactor: remove public modifier from modules
* refactor: allow shutting down the object store garbage collector
* feat: Introduce the object-store garbage collection server
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-18 21:27:38 +00:00
60df069a10
fix: Bump object_store crate for EKS support
2022-07-13 01:12:29 +02:00
428f41f747
feat: Walking skeleton of the object store GC tool
2022-07-07 09:48:04 -04:00
1962fcc229
chore: reduce dependencies and run `cargo update` ( #4906 )
...
* chore: reduce proptest features
* chore: remove `grpc-router`
This crate is currently unused and we don't have immediate plans to use
it. And there's GIT, so it can always be restored.
* chore: `cargo update`
2022-06-20 12:18:28 +00:00
6417e7dc2a
feat: Extract sharder to its own crate
2022-06-15 10:01:45 -04:00
b76a0080d5
chore: remove unused iox_gitops_adapter ( #4675 )
...
* chore: remove unused iox_gitops_adapter
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-05-24 10:28:43 +00:00
52346642a0
ci: fix cargo deny ( #4629 )
...
* ci: fix cargo deny
* chore: downgrade `socket2`, version 0.4.5 was yanked
* chore: rename `query` to `iox_query`
`query` is already taken on crates.io and yanked and I am getting tired
of working around that.
2022-05-18 09:38:35 +00:00
f2bb0fdf77
feat: update to crates.io object_store version ( #4595 )
...
* feat: update to crates.io object_store version
* chore: Run cargo hakari tasks
* fix: tests
* chore: remove object store integration test plumbing
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-05-13 16:26:07 +00:00
35d80fc512
refactor: rename `iox_catalog_service` to `service_grpc_catalog` for consistency ( #4581 )
...
* refactor: rename `iox_catalog_service` to `service_grpc_catalog`
* fix: fmt
* fix: alphabetize crate
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-13 14:07:58 +00:00
542ec97b66
Merge branch 'main' into cn/rename-no-ng
2022-05-13 13:47:48 +00:00
cb0a4176fd
refactor: move `querier::cache_system` into its own crate ( #4592 )
2022-05-13 13:12:07 +00:00
48e6e5713d
fix: Rename test_helpers_end_to_end_ng to test_helpers_end_to_end
...
Connects to #4450 .
2022-05-12 16:09:07 -04:00
7eff52d04e
refactor: rename `iox_object_store_service` to `service_grpc_object_store` for consisntency ( #4572 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-12 14:22:18 +00:00
b8cb4c3f2b
feat: Interrogate schema from querier (as well as router) ( #4557 )
...
* refactor: move SchemaService into `service_grpc_schema`
* feat: implement schema gRPC for querier
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-05-10 20:55:58 +00:00
99b1a9b83f
refactor: split out ObjectStoreMetrics ( #4547 )
...
* refactor: split out ObjectStoreMetrics
* chore: add workspace hack
* fix: compile
2022-05-10 10:56:28 +00:00
fcd4815645
fix: Rename router2 to router
2022-05-06 14:51:52 -04:00
0650a9bb77
fix: Rename ioxd_router2 to ioxd_router
2022-05-06 14:45:39 -04:00
068096e7e1
fix: Rename data_types2 to data_types
2022-05-06 14:45:39 -04:00
e1bef1c218
fix: Remove OG data_types crate
2022-05-06 14:45:39 -04:00
ea46830954
fix: Remove iox_object_store crate; move ParquetFilePath to parquet_file
2022-05-06 14:45:36 -04:00
f8bdb022bc
fix: Remove job_registry crate
2022-05-06 11:35:11 -04:00
b88d071ce7
fix: Remove server
2022-05-06 11:30:36 -04:00
e0bc1801ac
fix: Remove router
2022-05-06 11:30:36 -04:00
ba8191c1eb
fix: Remove persistence_windows
2022-05-06 11:30:35 -04:00
cae32209da
fix: Remove parquet_catalog
2022-05-06 11:30:26 -04:00
2d8656e2e1
fix: Remove mutable_buffer
2022-05-06 11:27:33 -04:00
e63f006398
fix: Remove lifecycle crate
2022-05-06 11:27:33 -04:00
e7de16732d
fix: Remove internal_types
2022-05-06 11:27:33 -04:00
7286b4391a
fix: Remove db crate
2022-05-06 11:27:33 -04:00
ece38417bc
feat: Remove the `run database` command
2022-05-06 09:48:30 -04:00
b939d80db8
feat: Remove the `run router` command
2022-05-06 09:48:30 -04:00
e015d3bafb
feat: Remove the server_benchmarks crate ( #4506 )
2022-05-02 18:20:01 +00:00
9e91af4501
refactor: Move IOx UDfs into a Function Registry (1/3) ( #4428 )
...
* refactor: Move all UDF implementations to query_function crate
* refactor: Move regex udf to query_functions
* refactor: Move functions out of query
* fix: lints, imports
* chore: Run cargo hakari tasks
* fix: clipy + benches
* fix: reduce borrowing and fix clippy
* fix: moar clippy
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-26 17:30:27 +00:00
4b47d723b1
refactor: Rename time to iox_time ( #4416 )
...
Signed-off-by: hi-rustin <rustin.liu@gmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-26 00:19:59 +00:00
5bf4550259
feat: add object store service to router ( #4338 )
...
Add method to catalog to get parquet file by object store id.
Add gRPC service for object store to get a file from by its uuid.
Add the object store service to router2 with object store config.
2022-04-16 17:58:31 +00:00
99cbb28a89
feat: add initial catalog service to router ( #4316 )
...
Create new crate for iox_catalog_service.
Add rpc to return parquet_file records by partition id.
Add CatalogService to router2.
The catalog service will be added to over time to provide access to the catalog over gRPC.
2022-04-14 17:39:18 +00:00
edda409b19
refactor: Extract `ioxd_test`, `ioxd_compactor`, `ioxd_ingester`; remove `ioxd` ( #4210 )
...
* refactor: Extract test, compactor, ingester, and test
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-03 10:42:22 +00:00
833c10c083
feat: return write_token from HTTP writes to router2 ( #4202 )
...
* feat: return write_token from HTTP writes to router2
* fix: Update router2/src/dml_handlers/instrumentation.rs
Co-authored-by: Dom <dom@itsallbroken.com>
* refactor: Use WriteSummary::default more vigorously
* fix: fix typo and add links to follow on issues
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-02 10:34:51 +00:00
532d227d11
refactor: extract router2 into ioxd_router2 ( #4183 )
...
* refactor: extract router2 from ioxd
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-31 17:39:05 +00:00
367e926d35
refactor: extract querier into ioxd_querier ( #4182 )
...
* refactor: extract querier into ioxd_querier
* fix: dep
2022-03-31 16:03:31 +00:00
f2a3dd58b2
refactor: rename `influxdb_ioxd` to `ioxd` ( #4162 )
...
* refactor: rename influxdb_ioxd to ioxd
* fix: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-30 13:31:27 +00:00
4ca52e5ae0
refactor: Extract common, OG database and router out of influxdb_ioxd ( #4149 )
...
* refactor: Extract common, OG database and router out of influxdb_ioxd
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-29 13:07:19 +00:00
fb75ef7b82
test: add end to end test for all in one mode, restructure fixture ( #4114 )
...
* test: add end to end test for all in one mode, restructure fixture
* docs: fix typos and clarify schema requrements
2022-03-24 12:53:25 +00:00
29b89aaca7
refactor: extract influxrpc, flight and testing gRPC out of influxdb_ioxd ( #4106 )
...
* refactor: extract grpc service implementations out of influxdb_ioxd
* chore: Run cargo hakari tasks
* refactor: rename server_common to service_common
* refactor: rename server_grpc_influxrpc to service_grpc_influxrpc
* refactor: rename server_grpc_flight to service_grpc_flight
* refactor: rename server_grpc_testing to service_grpc_testing
* fix: Cargo.toml
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-23 20:14:45 +00:00
f03ebd79ab
refactor: move querier's test utils to a new crate to get reused by tests in other crates ( #4013 )
...
* refactor: move querier's test utils to a new crate to be able resued by tests in other crates
* chore: remove unused import
2022-03-10 18:17:58 +00:00
d2c0acdd46
refactor: Remove serving readiness gate ( #3986 )
...
* refactor: Remove serving_readiness
* fix: remove more
* fix: remove test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-09 12:17:44 +00:00
d6afda0227
refactor: split `influxdb_ioxd`, `clap_blocks` and `serving_readiness` from influxdb_iox crate ( #3908 )
...
* refactor: split influxdb_ioxd, clap_blocks, and serving_readiness out of influxdb_iox
split out serving readiness, get compiling
* fix: hakari
* fix: hakari again
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-03 16:48:30 +00:00
8f3e44bf76
refactor: Extract a crate for shared data types in the new design
2022-03-02 12:16:15 -05:00
43ada68f37
chore: reset release codegen-units to default ( #3883 )
2022-03-01 13:44:14 +00:00
9079e6ddb0
feat: backoff retries in ingester ( #3841 )
...
* feat: add `backoff` crate
* feat: backoff retries in ingester
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 17:58:16 +00:00
459974d1b8
refactor: move `DedicatedExecutor` from `query` to `executor` ( #3819 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 09:10:43 +00:00
0f012de70c
feat: adding compactor CLI command and crate
...
Closes : #3777
2022-02-21 12:24:09 +00:00
a66e29e5b3
chore: port sqlx-hotswap-pool over from conductor ( #3750 )
...
* chore: port sqlx-hotswap-pool over from conductor
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
* chore: workspace hack fixes
* fix: unique schema per test db connection
* fix: adjust search path in catalog pg tests to see if it fixes test schema issue
* fix: actually fixed sqlx hotswap pool test
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-15 16:18:36 +00:00
54890525da
feat: gitops adapter ci build ( #3692 )
...
* chore: remove references to perf_image in CI
* chore: adding gitops adapter image build in CI
* chore: gitops adapter bin now same as dir & package so docker build works
* fix: circle config package change after renaming gitops adapter package
2022-02-10 11:23:06 +00:00
73828323ac
feat: Ingester Flight gRPC API ( #3623 )
...
* feat: Add a way to run ingester with an in-memory catalog from the CLI
If you set the --catalog-dsn string to "mem", rather than using that as
a Postgres connection URL, create an in-memory catalog.
Planning on using this in tests, so not documenting.
* fix: Set default topic to the same value as SHARED_KAFKA_TOPIC
Namely, both should use an underscore. I don't think there's a way to
directly share these values between a constant and an annotation.
* feat: Add a flight API (handshake only) to ingester
* fix: Create partitions if using file-based write buffer
* fix: Change the server fixture to handle ingester server type
For now, the ingester doesn't implement the deployment API. Not sure if
it should or not.
* feat: Start implementing ingester do_get, namely decoding the query
Skip serialization of the predicate for the moment.
* refactor: Rename ingest protos to ingester to match crate name
* refactor: Rename QueryResults to QueryData
* feat: Move ingester flight client to new querier crate
* fix: Off by one error, different starting indexes in sequencers
* fix: Create new CLI argument to pick the catalog type
* fix: Create a CLI option to set the number of topics to auto-create in the write buffer
* fix: Check the arrow flight service's health to tell that the ingester gRPC is up
* fix: Set postgres as the default catalog type
* fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
de2a013786
feat: gitops adapter ( #3656 )
...
* feat: scaffolding of gitops adapter bin crate
* chore: refactor gitops adapter; calls CLI now; status update fixes
* feat: gitops adapter now calls out to CLI once per topic; improved tests
* chore: add mock failure script for gitops adapter
* chore: update workspace-hack
* chore: refactor away unecessary to_string in gitops syncer
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-08 13:27:36 +00:00
c1f5994660
refactor: move sql parsing -> RPC predicate into own crate ( #3604 )
...
* chore: create crate
* refactor: move module to new crate
2022-02-02 10:41:57 +00:00
ef336b4659
feat: add ingester crate and a few basic data structures for its data lifecycle
2022-01-17 15:38:03 -05:00
4764e71c54
feat: Add initial iox_catalog skeleton
2022-01-17 09:56:20 -05:00
a8cb8755de
feat: new router2 crate
...
This commit adds an almost-empty router2 crate containing enough of a
skeleton to plumb into the IOx CLI/server runner.
2022-01-12 14:43:10 +00:00
f9174c483b
refactor: Extract server::db into its own crate ( #3417 )
...
* refactor: Extract JobRegistry from the server crate
Both the server crate and a db crate that I'm about to extract depend on
JobRegistry, so to avoid making circular dependencies, extract the
JobRegistry to its own crate.
* refactor: Move db out of server into its own crate
Fixes #2821 .
2021-12-23 22:01:17 +00:00
183de4519f
feat: Add incremental = true to the quick-release profile
2021-12-08 09:53:19 -05:00
255cf79104
docs: Explain tradeoffs and use cases of the different release profiles
2021-12-08 09:53:19 -05:00
758a17e199
feat: Create a build-time-optimized release build profile
...
Fixes #3316 .
2021-12-08 09:53:19 -05:00
994938b2f0
fix: Alphabetize/organize deps
2021-12-06 09:36:30 -05:00
f2f090bde7
feat: test bench
2021-11-25 18:14:40 +01:00
f66cd681c2
feat: Results of using cargo hakari init workspace-hack
2021-11-19 09:20:35 -05:00
a2454b542d
fix: Small cleanups in Cargo.tomls ( #3160 )
...
* fix: Add tokio rt-multi-thread feature so cargo test -p client_util compiles
* fix: Alphabetize dependencies
* fix: Add the data_types_conversions feature to get tests passing
* fix: Remove dev dependencies already listed under normal dependencies
* fix: Make sure the workspace is using the new resolver
2021-11-18 22:26:33 +00:00
49cfa046bb
Merge branch 'main' into crepererum/smaller_binaries
2021-11-12 11:17:09 +00:00
6f268f8260
refactor: extract DML types ( #2731 ) ( #3084 )
...
* refactor: extract DML types (#2731 )
* chore: fmt
2021-11-11 12:34:07 +00:00
16fc610433
chore: faster and smaller IOx binaries
...
Use `codegen-units = 1`, thin-LTO and debug section compression to make our binary smaller (which is good for deploy and
test times) and faster.
# Summary
The binary size of `influxdb_iox` after building with:
```console
$ cargo build --release --no-default-features --features="aws,gcp,azure,jemalloc_replacing_malloc"
```
The profile was:
```toml
[profile.release]
debug = true
```
The commit was:
```text
89ece8b493https://github.com/gimli-rs/gimli/issues/195 >
- <https://github.com/rust-lang/backtrace-rs/issues/342 >
## Codegen Units
The rust compiler parallelizes codegen work. This split into units however means that optimizations are somewhat
limited. This can be change by:
```toml
[profile.release]
...
codegen-units = 1
```
As a nice side effect this should also make our code faster.
## Thin LTO
Get LLVM to run "thin" Link Time Optimization:
```toml
[profile.release]
...
lto = "thin"
```
As a nice side effect this should also make our code faster.
## Fat LTO
Get LLVM to run "fat" Link Time Optimization:
```toml
[profile.release]
...
lto = "fat"
```
There are no results for this because this took a massive amount of memory and CPU time and did not finish on my system.
2021-11-10 15:49:19 +01:00
60f0deaf1e
feat: remove flatbuffer entry ( #3045 )
2021-11-05 20:19:24 +00:00
405ed6db1b
feat: add mutable_batch_tests ( #2724 ) ( #3011 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-03 18:15:35 +00:00
a91e572b43
feat: implement basic router server type
...
This only implements the deployment service and nothing else yet.
2021-11-03 16:34:51 +01:00
3010ff4176
feat: write entry to MutableBatch ( #2972 )
...
* feat: write entry to MutableBatch
* feat: error on empty TableBatch
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-27 14:05:22 +00:00
fc651ea193
chore: move `gprc-router-test-gen` to top level
2021-10-26 11:02:33 +02:00
a0cdca7781
chore: move `influxdb_iox` into a proper workspace package
2021-10-26 11:02:33 +02:00
2833cefc12
test: remove Kafka from end2end tests
...
Kafka is now sufficiently tested via the `write_buffer` crate. The
end2end tests can now use the in-memory mock implementation or -- if
servers can only be controlled via CLI -- the file-based implementation.
2021-10-26 10:09:34 +02:00
81ccf85104
refactor: integrate print-cpu into main binary
...
It's a less confusing structure and also allows us to have this
information available in our shipped binary.
2021-10-25 18:29:32 +02:00
7a28982e2a
fix: fix dependency features for `influxdb_iox`
2021-10-25 11:15:19 +02:00
bc7244c48e
chore: use Rust edition 2021
2021-10-25 10:58:20 +02:00
d2b41c5a13
feat: write pbdata to MutableBatch ( #2724 ) ( #2927 )
...
* feat: write pbdata to MutableBatch (#2724 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-21 14:32:35 +00:00
f64e749eb9
feat: write line protocol to MutableBatch ( #2914 )
...
* feat: write line protocol to MutableBatch
* chore: fix lint
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-21 08:12:54 +00:00
9974a5364c
chore(security): Replace prettytable with comfy-table ( #2905 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-20 10:44:36 +00:00
a82dc6f5f0
chore: Update datafusion + arrow ( #2903 )
...
* chore: Update datafusion to latest, arrow to 6.0.0
* fix: Update tests
* fix: bubble internal error
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-19 17:14:08 +00:00
0be89715e2
Merge branch 'main' into dependabot/cargo/rdkafka-0.27.0
2021-10-19 12:55:21 +00:00
7dafb0ff74
chore: remove unused `influxdb_iox` => `synchronized-writer` dep
2021-10-19 14:46:26 +02:00
32e18b6436
chore(deps): bump rdkafka from 0.26.0 to 0.27.0
...
Bumps [rdkafka](https://github.com/fede1024/rust-rdkafka ) from 0.26.0 to 0.27.0.
- [Release notes](https://github.com/fede1024/rust-rdkafka/releases )
- [Changelog](https://github.com/fede1024/rust-rdkafka/blob/master/changelog.md )
- [Commits](https://github.com/fede1024/rust-rdkafka/compare/v0.26.0...v0.27.0 )
---
updated-dependencies:
- dependency-name: rdkafka
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-19 11:49:34 +00:00
157b556d4c
feat: make flight responses streaming ( #2876 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-18 15:01:50 +00:00
70555ab33d
chore(deps): bump structopt from 0.3.23 to 0.3.25 ( #2877 )
...
Bumps [structopt](https://github.com/TeXitoi/structopt ) from 0.3.23 to 0.3.25.
- [Release notes](https://github.com/TeXitoi/structopt/releases )
- [Changelog](https://github.com/TeXitoi/structopt/blob/master/CHANGELOG.md )
- [Commits](https://github.com/TeXitoi/structopt/compare/v0.3.23...v0.3.25 )
---
updated-dependencies:
- dependency-name: structopt
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-10-18 14:49:15 +00:00
bdd6d67e7a
refactor: split out mutable_batch crate ( #2841 )
...
* refactor: split out mutable_batch crate
* refactor: restore chunk module for better diffs
* chore: fmt
* chore: review feedback
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-14 16:46:54 +00:00
28195b9c0c
chore: new `parquet_catalog` crate
2021-10-14 14:34:59 +02:00
0554173684
feat: migrate write buffer to TimeProvider ( #2722 ) ( #2804 )
...
* feat: migrate write buffer to TimeProvider (#2722 )
* chore: review feedback
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: Marco Neumann <marco@crepererum.net>
2021-10-12 10:32:34 +00:00
7bd32a4916
feat: add time provider abstraction ( #2722 ) ( #2792 )
...
* feat: add time provider abstraction (#2722 )
* chore: tests
* chore: more tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-11 15:00:29 +00:00
61190a9d08
chore(deps): bump predicates from 2.0.2 to 2.0.3
...
Bumps [predicates](https://github.com/assert-rs/predicates-rs ) from 2.0.2 to 2.0.3.
- [Release notes](https://github.com/assert-rs/predicates-rs/releases )
- [Changelog](https://github.com/assert-rs/predicates-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/predicates-rs/compare/v2.0.2...v2.0.3 )
---
updated-dependencies:
- dependency-name: predicates
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-11 14:32:08 +00:00
afe34751e7
refactor: split out schema crate ( #2781 )
...
* refactor: split out schema crate
* chore: fix doc
2021-10-11 09:45:08 +00:00
1327735537
chore(deps): bump thiserror from 1.0.29 to 1.0.30
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.29 to 1.0.30.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.29...1.0.30 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-11 08:01:21 +00:00
01a44899fd
chore(deps): bump assert_cmd from 2.0.1 to 2.0.2
...
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd ) from 2.0.1 to 2.0.2.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases )
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.1...v2.0.2 )
---
updated-dependencies:
- dependency-name: assert_cmd
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-11 01:45:35 +00:00
7a6f381e69
chore(deps): bump arrow-flight from 5.4.0 to 5.5.0
...
Bumps [arrow-flight](https://github.com/apache/arrow-rs ) from 5.4.0 to 5.5.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/5.5.0/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/5.4.0...5.5.0 )
---
updated-dependencies:
- dependency-name: arrow-flight
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-04 13:27:26 +00:00
181145eca1
Merge branch 'main' into dependabot/cargo/arrow-5.5.0
2021-10-04 13:10:42 +00:00
6d9205d87c
chore(deps): bump parquet from 5.4.0 to 5.5.0
...
Bumps [parquet](https://github.com/apache/arrow-rs ) from 5.4.0 to 5.5.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/5.5.0/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/5.4.0...5.5.0 )
---
updated-dependencies:
- dependency-name: parquet
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-04 12:15:11 +01:00
d1f5209869
chore(deps): bump arrow from 5.4.0 to 5.5.0
...
Bumps [arrow](https://github.com/apache/arrow-rs ) from 5.4.0 to 5.5.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/5.5.0/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/5.4.0...5.5.0 )
---
updated-dependencies:
- dependency-name: arrow
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-04 08:55:38 +00:00
86cee568d5
feat: use upstream pbjson ( #2650 )
...
* feat: use upstream pbjson
* chore: fmt
2021-09-28 16:29:26 +00:00
daa2ec2f4c
revert: "fix: Revert "feat: remove routerify ( #2586 )" ( #2623 )" ( #2649 )
...
This reverts commit 18941fcbff
2021-09-28 14:42:17 +00:00
18941fcbff
fix: Revert "feat: remove routerify ( #2586 )" ( #2623 )
...
This reverts commit 9fba7b0ba0
2021-09-24 09:50:24 +00:00
92e6173f49
feat: implement jaeger-agent protocol directly ( #2607 )
...
* feat: implement jaeger-agent protocol directly
* chore: review feedback
* fix: remove jaeger feature flag
2021-09-22 17:30:37 +00:00
46088a7ff1
feat: disable rustyline dirs-next dependency ( #2579 )
2021-09-20 14:42:07 +00:00
9fba7b0ba0
feat: remove routerify ( #2586 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-20 08:57:19 +00:00
1112597c44
chore(deps): bump dirs from 3.0.2 to 4.0.0
...
Bumps [dirs](https://github.com/soc/dirs-rs ) from 3.0.2 to 4.0.0.
- [Release notes](https://github.com/soc/dirs-rs/releases )
- [Commits](https://github.com/soc/dirs-rs/commits )
---
updated-dependencies:
- dependency-name: dirs
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-20 08:28:04 +00:00
3ffb16daa6
feat: remove parse_duration ( #2574 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-19 11:27:24 +00:00
1d55d9a1b5
feat: add pbjson support ( #2468 )
...
* feat: add pbjson support
* chore: fix test
2021-09-16 07:33:27 +00:00
c66095cad1
feat: remove metrics crate ( #2552 )
2021-09-15 19:43:33 +00:00
6e7fa3e574
feat: migrate http ingest metrics ( #2542 )
...
* feat: migrate http ingest metrics
* chore: review feedback
* refactor: RAII entry ingest recorder
2021-09-15 19:01:10 +00:00
bfaba78dc3
refactor: move `predicate` into its own crate
...
Two reasons:
1. I wanna decouple `parquet_file` from `query` (nearly done, needs a
small follow-up PR).
2. `predicate` will have more and more features (like serialization)
which justifies a new home
2021-09-14 17:13:02 +02:00
3f2e46c397
feat: prune old transactions from preserved catalog
2021-09-14 12:08:17 +02:00
e07bc85670
feat: enable tracing for http API ( #2273 ) ( #2498 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-10 11:21:02 +00:00
b67610d9b9
chore(deps): bump tokio from 1.10.1 to 1.11.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.10.1 to 1.11.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.10.1...tokio-1.11.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 09:11:38 +00:00
06311970fa
Merge branch 'main' into dependabot/cargo/parquet-5.3.0
2021-09-06 09:00:27 +00:00
ccfd6ad3da
chore(deps): bump parquet from 5.2.0 to 5.3.0
...
Bumps [parquet](https://github.com/apache/arrow-rs ) from 5.2.0 to 5.3.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/5.3.0/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/5.2.0...5.3.0 )
---
updated-dependencies:
- dependency-name: parquet
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 08:52:30 +00:00
0fd92c8b45
Merge branch 'main' into dependabot/cargo/structopt-0.3.23
2021-09-06 08:51:32 +00:00
b1bb390893
chore(deps): bump parking_lot from 0.11.1 to 0.11.2
...
Bumps [parking_lot](https://github.com/Amanieu/parking_lot ) from 0.11.1 to 0.11.2.
- [Release notes](https://github.com/Amanieu/parking_lot/releases )
- [Changelog](https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md )
- [Commits](https://github.com/Amanieu/parking_lot/compare/0.11.1...0.11.2 )
---
updated-dependencies:
- dependency-name: parking_lot
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 01:18:24 +00:00
2cf9505029
chore(deps): bump structopt from 0.3.22 to 0.3.23
...
Bumps [structopt](https://github.com/TeXitoi/structopt ) from 0.3.22 to 0.3.23.
- [Release notes](https://github.com/TeXitoi/structopt/releases )
- [Changelog](https://github.com/TeXitoi/structopt/blob/master/CHANGELOG.md )
- [Commits](https://github.com/TeXitoi/structopt/compare/v0.3.22...v0.3.23 )
---
updated-dependencies:
- dependency-name: structopt
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 01:17:41 +00:00
09a5d27434
chore: ignore massif profiler outputs
2021-09-02 15:58:43 +02:00
4a863993ec
feat: "dump catalog" debug CLI
2021-09-02 08:08:20 +02:00
6ba25728e1
chore: exclude non-source code from package ( #2452 )
...
Before this change, modifications to the following files would trigger a
rebuild on `cargo build`.
```
2,13d1
< .circleci/config.yml
< .circleci/get-deploy-tags.sh
< .editorconfig
< .gitattributes
< .github/ISSUE_TEMPLATE/bug_report.md
< .github/ISSUE_TEMPLATE/feature_request.md
< .github/PULL_REQUEST_TEMPLATE.md
< .github/dependabot.yml
< .github/semantic.yml
< .gitignore
< .kodiak.toml
< CONTRIBUTING.md
17,22d4
< Dockerfile
< Dockerfile.dockerignore
< LICENSE-APACHE
< LICENSE-MIT
< README.md
< buf.yaml
24,76d5
< docker/Dockerfile.ci
< docker/Dockerfile.ci.dockerignore
< docker/Dockerfile.ci.integration
< docker/Dockerfile.ci.integration.dockerignore
< docker/Dockerfile.iox
< docker/Dockerfile.iox.dockerignore
< docker/ci-kafka-docker-compose.yml
< docker/integration_test.sh
< docs/README.md
< docs/catalog_persistence.md
< docs/data_management.md
< docs/drawings/data_lifecycle.monopic
< docs/drawings/data_organization.monopic
< docs/encoding_thoughts.md
< docs/env.example
< docs/images/flame_graph.png
< docs/local_filesystems.md
< docs/metrics.md
< docs/multi_core_tasks.md
< docs/profiling.md
< docs/protobuf.md
< docs/regenerating_flatbuffers.md
< docs/server_startup.md
< docs/sql.md
< docs/style_guide.md
< docs/testing.md
< docs/tracing.md
< docs/valgrind.md
< perf/logs/iox_router.log
< perf/logs/iox_writer.log
< perf/logs/jaeger.log
< perf/logs/kafka.log
< perf/logs/minio.log
< perf/logs/zookeeper.log
< perf/volumes/minio/.minio.sys/buckets/.bloomcycle.bin
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/.bloomcycle.bin/fs.json
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/.usage-cache.bin/fs.json
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/.usage.json/fs.json
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/iox1/.usage-cache.bin/fs.json
<
perf/volumes/minio/.minio.sys/buckets/.minio.sys/buckets/iox2/.usage-cache.bin/fs.json
< perf/volumes/minio/.minio.sys/buckets/.tracker.bin
< perf/volumes/minio/.minio.sys/buckets/.usage-cache.bin
< perf/volumes/minio/.minio.sys/buckets/.usage.json
< perf/volumes/minio/.minio.sys/buckets/iox1/.metadata.bin
< perf/volumes/minio/.minio.sys/buckets/iox1/.usage-cache.bin
< perf/volumes/minio/.minio.sys/buckets/iox2/.metadata.bin
< perf/volumes/minio/.minio.sys/buckets/iox2/.usage-cache.bin
< perf/volumes/minio/.minio.sys/config/config.json
< perf/volumes/minio/.minio.sys/config/iam/format.json
< perf/volumes/minio/.minio.sys/format.json
< perf/volumes/zookeeper/data/myid
< perf/volumes/zookeeper/data/version-2/log.1
< perf/volumes/zookeeper/data/version-2/snapshot.0
79,87d7
< scripts/edit_db_rules
< scripts/genlp.py
< scripts/git-log-prs
< scripts/grpcurl
< scripts/jq_edit_db_rules
< scripts/logfmts
< scripts/prototxt
< scripts/valgrind.supp
< scripts/valigrind_leak
161d80
< tools/iplan
```
2021-08-31 17:14:02 +00:00
3e154b3b02
chore(deps): bump serde_json from 1.0.66 to 1.0.67
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.66 to 1.0.67.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.66...v1.0.67 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-30 11:28:35 +00:00
52fbaef2df
chore(deps): bump thiserror from 1.0.26 to 1.0.28
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.26 to 1.0.28.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.26...1.0.28 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-30 11:05:11 +00:00
13161de003
chore(deps): bump rustyline from 8.2.0 to 9.0.0
...
Bumps [rustyline](https://github.com/kkawakam/rustyline ) from 8.2.0 to 9.0.0.
- [Release notes](https://github.com/kkawakam/rustyline/releases )
- [Changelog](https://github.com/kkawakam/rustyline/blob/master/History.md )
- [Commits](https://github.com/kkawakam/rustyline/compare/v8.2.0...v9.0.0 )
---
updated-dependencies:
- dependency-name: rustyline
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-30 01:19:49 +00:00
fcec394a28
feat: connect up new metrics ( #2428 )
2021-08-27 12:55:35 +00:00
3a26f7857f
feat: add new metrics abstraction ( #2076 ) ( #2365 )
...
* feat: add new metrics abstraction (#2076 )
* chore: fix lint
* feat: add tests
feat: histogram export
* chore: further docs
* chore: docs
* chore: log on prometheus export error
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-27 11:54:23 +00:00
1773bf5d37
feat: add storage client to influxdb_iox_client ( #2404 )
...
* feat: add storage client to influxdb_iox_client
* chore: fix type_url
* refactor: split storage into separate crate
* chore: fix doctest
* chore: review feedback
* chore: add generated_types cleanup ticket
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-26 10:30:59 +00:00
2c9c191b17
refactor: split out trace http ( #2388 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-24 13:37:20 +00:00
3fdc0e9a6a
chore: remove orphaned code from trogging ( #2371 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-24 10:12:59 +00:00
58716dee71
refactor: remove mem-qe ( #2344 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-23 14:35:12 +00:00
0d4bc4190f
chore(deps): bump assert_cmd from 1.0.8 to 2.0.0
...
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd ) from 1.0.8 to 2.0.0.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases )
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v1.0.8...v2.0.0 )
---
updated-dependencies:
- dependency-name: assert_cmd
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-23 01:17:06 +00:00
bbbd38bb12
fix(heappy): Prevent re-entering a running heap profile
...
Upgrades heappy to include https://github.com/mkmik/heappy/pull/2
and adapts to the API change.
2021-08-20 14:36:36 +02:00
4a2f0b0a52
fix: Make heappy and nonheappy mutually exclusive
...
Closes #2288
2021-08-20 11:35:07 +02:00
db97069ecd
feat: jaeger span export ( #2273 ) ( #2348 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-20 08:57:16 +00:00
d5f01a2a68
refactor: move data generator to IOx repo and fix build
2021-08-19 14:26:15 -04:00
04ba139cb2
refactor: split trace crate ( #2343 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-19 11:00:32 +00:00
5e1cb244f7
feat: make pprof optional ( #2331 )
2021-08-18 15:13:37 +00:00
a50b1646bd
fix(heappy): Report free as positive numbers
2021-08-18 14:30:41 +02:00
fc473995fa
feat(iox): Upgrade heappy to present free memory events
...
Currently heappy is reporting "bytes allocated" and "inuse_bytes" ("bytes allocated - freed bytes").
Since the allocation site and free site can be wildly different, it's very hard to correlate these events
and have a meaningful "inuse_bytes" graph.
I have a plan for how to fix this for good, but in the meantime
I'd like to report raw "freed bytes".
2021-08-18 13:04:48 +02:00
1d6a8703af
feat: support sinking traces to an OTEL SpanExporter ( #2319 )
...
* feat: support sinking traces to an OTEL SpanExporter
* chore: consistent logging
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* chore: review feedback
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-18 08:05:21 +00:00
efa776bd03
feat: fix log crate output ( #2325 )
...
* feat: fix log crate output
* chore: fix doc
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-17 16:19:22 +00:00
94ff77126e
feat: Enable freed memory tracking in heappy
...
In aed37ab50a
2021-08-17 15:09:28 +02:00
1d85a3fef6
build(deps): bump predicates from 1.0.8 to 2.0.2
...
Bumps [predicates](https://github.com/assert-rs/predicates-rs ) from 1.0.8 to 2.0.2.
- [Release notes](https://github.com/assert-rs/predicates-rs/releases )
- [Changelog](https://github.com/assert-rs/predicates-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/predicates-rs/compare/v1.0.8...v2.0.2 )
---
updated-dependencies:
- dependency-name: predicates
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-16 15:36:49 +00:00
500019cff9
Merge branch 'main' into ntran/heappy2
2021-08-13 17:27:38 +02:00
8650e419f7
feat: add trace collection plumbing ( #2273 ) ( #2274 )
...
* feat: add trace collection plumbing (#2273 )
* chore: fix lints
* chore: use tower directly
* chore: add tests
* chore: review feedback
* refactor: use serde instead of custom display impl
* chore: fix doc
2021-08-13 14:38:31 +00:00
cedaf2d5f2
fix: remove unprefixed_malloc_on_supported_platforms feature from tikv-jemallocator
2021-08-13 08:24:15 -05:00
697de875ca
refactor: make heappy an optional feature, add to build image
2021-08-13 08:11:32 -05:00
4fd5862a9c
fix: turn heappy on
2021-08-12 17:14:03 -04:00
ae6b0e669b
refactor: Extract a database persister type that wraps object store
...
Connects to #2193 .
2021-08-12 15:05:32 -04:00
4f52a70d5b
refactor: adddress mkm's comments
2021-08-12 12:04:42 -04:00
8aa9514d93
feat: integrate heappy implemented by mkm
2021-08-11 23:42:50 -04:00
1f450ef371
feat: add Database abstraction ( #2186 ) ( #2203 )
...
* feat: add Database abstraction
* chore: minor tweaks
* chore: remove redundant test fixture restart
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-08 17:14:23 +00:00
82677d9510
feat: add `jaeger` and `otlp` features, default to off ( #2202 )
...
* feat: add jaeger and otlp flags
* chore: add jaeger and otlp features to CI test and deploy image
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-06 13:26:59 +00:00
e69a159ad2
refactor: remove unused opentelemetry dependencies ( #2201 )
2021-08-05 16:20:22 +00:00
e92e94caad
chore: Update deps (including arrow 5.1.0, tonic -> 0.5, and prost 0.5) ( #2172 )
...
* chore: Update deps (including arrow 5.0.0 --> arrow 5.1.0)
* chore: update all the things
* refactor: Update serving readiness check due to change in Tonic API
* chore: update more deps
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-05 15:57:38 +00:00
7ca31703a4
feat: Attempt to dump a stacktrace to stderr prior to process abort on SIGSEGV/SIGILL/SIGBUS ( #2155 )
...
* feat: Attempt to dump a stacktrace to stdout prior to process abort
* refactor: rewrite signal handling in terms of libc, remove sig dep
* refactor: print to stderr
* fix: Update src/main.rs
* docs: note provenance
2021-07-30 11:58:28 +00:00
4def5bd816
chore: Add print_cpu command line to perf build ( #2114 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-26 15:26:34 +00:00
4da8a16c18
chore: update to arrow 5.0 and master datafusion ( #2049 )
...
* chore: update to arrow 5.0 and master datafusion
* fix: Update test for change in object size
2021-07-19 12:49:51 +00:00
b5428e53a5
refactor: write buffer testing + better mocking
...
This refactors the write buffer a bit for:
- **Testing:** Add generic tests for the Kafka and the mocking
implementation. The same interface can be used easily add new
implementations (e.g. via Redis, filesystem, ...).
- **Partition on Write:** The caller of the writer operation must now
specify the partition/sequencer ID. The implicit partitioning of the
Kafka writer would have lead to broken data since we must never spill
entries w/ the same primary key over multiple partitions. At the
moment we will only use partition 0 but we can easily implement
better logic in the future.
- **Improved Mocking:** The mocked implementation now simulates a system
that feels more real. Especially the handling around multiple streams
and "write while read" has been improved. This will be helpful for
testing and for new features like seeking (during replay). A solid
realistic mock also helps us to ensure that the tests using the mock
do not rely on unrealistic behavior too much.
2021-07-15 17:20:45 +02:00
0e5276ed20
Merge branch 'main' into alamb/go_go_go_go
2021-07-14 13:56:35 +01:00
9cb9ae0874
chore: move write buffer into its own crate
2021-07-14 14:09:18 +02:00
4800b36949
chore: Update IOx to a pre-release version of arrow and datafusion to test out performance improvement
2021-07-13 15:44:57 -04:00
1a79bf7e99
refactor: Make aws/azure/gcs optional features and stop compiling 100 dependencies during dev ( #1933 )
...
* feat: make aws, gcp, azure dependencies optional
* fix: only run object store tests if the features are enabled
* fix: clean up testing
* fix: rename step
* fix: add to list of jobs
* fix: remove test with object store
* fix: review comments
2021-07-09 11:38:30 +00:00
4ca2d3e148
chore: move persistence windows related code into own crate
...
The entire persistence windows data structures (including the
checkpoints) have nothing to do with the mutable buffer per se. So lets
move them into their own crate. This also makes `parquet_file` not
longer depend on `mutable_buffer`.
2021-07-05 10:23:58 +02:00
69b0bb1510
feat: Implement grpc-router crate
...
This PR implements the main building block for implementing the gRPC StorageService router.
2021-06-23 17:21:46 +02:00
55c546baff
feat: eagerly check object store during CLI `run`
...
Instead of waiting for the server ID to be set and then mark the server
as errored, directly check the object store on startup. This is
important so that we fail fast when Istio isn't up and running yet.
2021-06-22 18:21:30 +02:00
b4644e6108
test: Start of Kafka Write Buffer integration tests
2021-06-21 09:41:35 -04:00
ec43a87909
chore: Update itertools deps ( #1750 )
2021-06-17 17:56:44 +00:00
Andrew Lamb
Carol (Nichols || Goulding)
Marco Neumann
Stuart Carnie
Andrew Lamb
Marko Mikulicic
Raphael Taylor-Davies
Dom Dwyer
Luke Bond
Jake Goulding
kodiakhq[bot]
Carol (Nichols || Goulding)
二手掉包工程师
Paul Dix
Nga Tran
Edd Robinson
dependabot[bot]
Jacob Marble
Nga Tran
Carol (Nichols || Goulding)