Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
7,647 Commits
218 Branches
453 Tags
340 MiB
Commit Graph

1103 Commits (86e8f05ed1172a80a14242feeae4e28252fd3604)

Author SHA1 Message Date
dependabot[bot] 09e69a57b2
chore(deps): Bump cc from 1.0.72 to 1.0.73 (#3793) 
Bumps [cc](https://github.com/alexcrichton/cc-rs) from 1.0.72 to 1.0.73.
- [Release notes](https://github.com/alexcrichton/cc-rs/releases)
- [Commits](https://github.com/alexcrichton/cc-rs/compare/1.0.72...1.0.73)

---
updated-dependencies:
- dependency-name: cc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-21 09:47:47 +00:00
dependabot[bot] d6317a2ffd
chore(deps): Bump kube-runtime from 0.69.0 to 0.69.1 (#3794) 
Bumps [kube-runtime](https://github.com/kube-rs/kube-rs) from 0.69.0 to 0.69.1.
- [Release notes](https://github.com/kube-rs/kube-rs/releases)
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.69.0...0.69.1)

---
updated-dependencies:
- dependency-name: kube-runtime
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-02-21 09:37:52 +00:00
Luke Bond 7969ec682c
feat: catalog now supports dsn-file:// (#3787) 
* feat: catalog now supports dsn-file://

* chore: rename fn in catalog PG hotswap code
 2022-02-18 16:42:55 +00:00
Marco Neumann f54ef92b77
fix: supervise and shutdown ingester background tasks (#3769) 
* fix: supervise and shutdown ingester background tasks

Closes #3761.
Closes #3762.

* docs: improve wording

Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>

* test: join/shutdown handling for ingester

Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
 2022-02-18 09:35:29 +00:00
Dom Dwyer 44e9eaf92b test(iox_catalog): isolated catalog tests 
This commit changes the iox_catalog test harness so that each test is
run in an independent, randomly generated schema to avoid concurrent
tests interfering with each other.

Each test creates a randomly-named schema, grants permissions to the new
schema, runs the full migration stack against the new schema, and then
executes the code under test.
 2022-02-17 14:19:01 +00:00
Marco Neumann b1215805a8
fix: upgrade rskafka for bugfixes and race-free start offsets (#3775) 
This fixes the concerns that were brought up during the review of #3748.
 2022-02-17 11:22:25 +00:00
kodiakhq[bot] d0965bb0b2
Merge branch 'main' into dom/mb-partitioning 2022-02-16 11:30:42 +00:00
Luke Bond a66e29e5b3
chore: port sqlx-hotswap-pool over from conductor (#3750) 
* chore: port sqlx-hotswap-pool over from conductor

Co-authored-by: Marko Mikulicic <mkm@influxdata.com>

* chore: workspace hack fixes

* fix: unique schema per test db connection

* fix: adjust search path in catalog pg tests to see if it fixes test schema issue

* fix: actually fixed sqlx hotswap pool test

Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-15 16:18:36 +00:00
Andrew Lamb a30803e692
chore: Update datafusion, update `arrow`/`parquet`/`arrow-flight` to 9.0 (#3733) 
* chore: Update datafusion

* chore: Update arrow

* fix: missing updates

* chore: Update cargo.lock

* fix: update for smaller parquet size

* fix: update test for smaller parquet files

* test: ensure parquet_file tests write multiple row groups

* fix: update callsite

* fix: Update for tests

* fix: harkari

* fix: use IoxObjectStore::existing

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-15 12:10:24 +00:00
dependabot[bot] 89105ccfab
chore(deps): Bump tokio-util from 0.6.9 to 0.7.0 (#3743) 
Bumps [tokio-util](https://github.com/tokio-rs/tokio) from 0.6.9 to 0.7.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/commits)

---
updated-dependencies:
- dependency-name: tokio-util
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-02-15 11:33:41 +00:00
Dom Dwyer 92218ce8aa feat: write partitioner 
Implements a write partitioning DML handler that splits per-table
MutableBatch instances into per-partition, per-table MutableBatch and
concurrently calls the inner DML handler with each.
 2022-02-15 11:29:32 +00:00
Dom Dwyer e99922d518 refactor: parametrise DML handler input type 
Allow a DML handler to specify the write input type on which it
operates.

This allows us to construct a write handler pipeline that transforms the
request as it passes through the various handlers. We'll use this to
implement a handler that annotates a normal set of table writes with the
partition key, modifying downstream handlers to expect this annotated
input.
 2022-02-15 11:23:45 +00:00
Marco Neumann 4db27eec68
fix: defined behaviour when seeking to an unknown sequence number (#3745) 
* chore: upgrade rskafka

* refactor: less cloning

* fix: defined behaviour when seeking to an unknown sequence number

The new, defined behavior is: "return an error once and then end the
stream".

Co-authored-by: Edd Robinson <me@edd.io>

Co-authored-by: Edd Robinson <me@edd.io>
 2022-02-15 11:08:01 +00:00
dependabot[bot] 1cd51b6071
chore(deps): Bump hyper from 0.14.16 to 0.14.17 (#3744) 
Bumps [hyper](https://github.com/hyperium/hyper) from 0.14.16 to 0.14.17.
- [Release notes](https://github.com/hyperium/hyper/releases)
- [Changelog](https://github.com/hyperium/hyper/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/hyper/compare/v0.14.16...v0.14.17)

---
updated-dependencies:
- dependency-name: hyper
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-15 10:23:46 +00:00
dependabot[bot] 2626e4891d
chore(deps): Bump tracing-subscriber from 0.3.7 to 0.3.8 (#3741) 
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.3.7 to 0.3.8.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.7...tracing-subscriber-0.3.8)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-14 20:51:21 +00:00
dependabot[bot] 60a7f87645
chore(deps): Bump serde_json from 1.0.78 to 1.0.79 (#3739) 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.78 to 1.0.79.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.78...v1.0.79)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-14 20:42:54 +00:00
Carol (Nichols || Goulding) 85aa019f50
feat: Turn protobuf predicates into predicate::Predicate (#3707) 
* feat: Turn protobuf predicates into predicate::Predicate

* fix: Take buf lint's suggestions

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-14 17:56:56 +00:00
Luke Bond d39cf0158b
chore: update gitops adapter k8s deps (via dependabot) (#3736) 
* chore: update gitops adapter k8s deps (via dependabot)

* chore: workspace hack updates

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-14 17:47:15 +00:00
Andrew Lamb b77d43fc22
chore: Update datafusion (#3727) 2022-02-11 21:41:36 +00:00
dependabot[bot] 6c3b208438
chore(deps): bump cache_loader_async from 0.1.2 to 0.2.0 (#3665) 
* chore(deps): bump cache_loader_async from 0.1.2 to 0.2.0

Bumps [cache_loader_async](https://github.com/ZeroTwo-Bot/cache-loader-async-rs) from 0.1.2 to 0.2.0.
- [Release notes](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/releases)
- [Changelog](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/commits/v0.2.0)

---
updated-dependencies:
- dependency-name: cache_loader_async
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix: breaking changes from `cache_loader_async`

* fix: Update workspace hack: tokio full is no longer needed

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
 2022-02-10 17:01:11 +00:00
Luke Bond 54890525da
feat: gitops adapter ci build (#3692) 
* chore: remove references to perf_image in CI

* chore: adding gitops adapter image build in CI

* chore: gitops adapter bin now same as dir & package so docker build works

* fix: circle config package change after renaming gitops adapter package
 2022-02-10 11:23:06 +00:00
dependabot[bot] ad60dc6949
chore(deps): bump httparse from 1.5.1 to 1.6.0 (#3708) 
Bumps [httparse](https://github.com/seanmonstar/httparse) from 1.5.1 to 1.6.0.
- [Release notes](https://github.com/seanmonstar/httparse/releases)
- [Commits](https://github.com/seanmonstar/httparse/compare/v1.5.1...v1.6.0)

---
updated-dependencies:
- dependency-name: httparse
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-10 09:54:42 +00:00
dependabot[bot] f23574bc5f
chore(deps): bump futures from 0.3.19 to 0.3.21 (#3706) 
Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.19 to 0.3.21.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.19...0.3.21)

---
updated-dependencies:
- dependency-name: futures
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-10 09:19:19 +00:00
dependabot[bot] 6a002f09e1
chore(deps): bump clap from 3.0.13 to 3.0.14 (#3698) 
Bumps [clap](https://github.com/clap-rs/clap) from 3.0.13 to 3.0.14.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v3.0.13...v3.0.14)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-09 21:21:24 +00:00
dependabot[bot] 85d44552fa
chore(deps): bump cloud-storage from 0.10.3 to 0.11.0 (#3697) 
Bumps [cloud-storage](https://github.com/ThouCheese/cloud-storage-rs) from 0.10.3 to 0.11.0.
- [Release notes](https://github.com/ThouCheese/cloud-storage-rs/releases)
- [Changelog](https://github.com/ThouCheese/cloud-storage-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ThouCheese/cloud-storage-rs/commits)

---
updated-dependencies:
- dependency-name: cloud-storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
 2022-02-09 20:03:44 +00:00
Andrew Lamb d9f331ba2a
chore: update datafusion, stop repartitioning so aggressively (#3633) 
* chore: update datafusion

* fix: Update to use new datafusion api

* chore: update expected plans

* fix: support zero output partitions

* fix: update test

* fix: Update for new DataFusion API

* fix: newly added system table

* fix: update cargo lock
 2022-02-09 19:53:41 +00:00
dependabot[bot] 803296e8cf
chore(deps): bump sqlparser from 0.13.0 to 0.14.0 (#3696) 
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs) from 0.13.0 to 0.14.0.
- [Release notes](https://github.com/sqlparser-rs/sqlparser-rs/releases)
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.13.0...v0.14.0)

---
updated-dependencies:
- dependency-name: sqlparser
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
 2022-02-09 19:40:03 +00:00
Carol (Nichols || Goulding) 73828323ac
feat: Ingester Flight gRPC API (#3623) 
* feat: Add a way to run ingester with an in-memory catalog from the CLI

If you set the --catalog-dsn string to "mem", rather than using that as
a Postgres connection URL, create an in-memory catalog.

Planning on using this in tests, so not documenting.

* fix: Set default topic to the same value as SHARED_KAFKA_TOPIC

Namely, both should use an underscore. I don't think there's a way to
directly share these values between a constant and an annotation.

* feat: Add a flight API (handshake only) to ingester

* fix: Create partitions if using file-based write buffer

* fix: Change the server fixture to handle ingester server type

For now, the ingester doesn't implement the deployment API. Not sure if
it should or not.

* feat: Start implementing ingester do_get, namely decoding the query

Skip serialization of the predicate for the moment.

* refactor: Rename ingest protos to ingester to match crate name

* refactor: Rename QueryResults to QueryData

* feat: Move ingester flight client to new querier crate

* fix: Off by one error, different starting indexes in sequencers

* fix: Create new CLI argument to pick the catalog type

* fix: Create a CLI option to set the number of topics to auto-create in the write buffer

* fix: Check the arrow flight service's health to tell that the ingester gRPC is up

* fix: Set postgres as the default catalog type

* fix: Return an error rather than panicking if CLI args aren't right
 2022-02-09 19:07:44 +00:00
dependabot[bot] 7d11485c77
chore(deps): bump tracing from 0.1.29 to 0.1.30 (#3677) 
Bumps [tracing](https://github.com/tokio-rs/tracing) from 0.1.29 to 0.1.30.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.29...tracing-0.1.30)

---
updated-dependencies:
- dependency-name: tracing
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-08 16:44:16 +00:00
Paul Dix 59b2141c0b
feat: Add lifecycle manager to ingester (#3645) 
This adds the lifecycle manager to the ingester. It will trigger based on a threshold for max partition size or age or based on keeping total memory under a certain threshold.

It defines a new interface for a persister, which is stubbed out for IngesterData. I'm not sure yet how persistence errors should be handled. The assumption here is that the persister continues to retry persistence forever until it succeeds.

There is one scenario I can think of that may cause this lifecycle manager problems. If a single partition is very high throughput, it could cause things to back up as persistence is not parallelized within a single partition. Any given partition can currently only run one persistence operation at a time. We can address this later.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-08 15:23:40 +00:00
dependabot[bot] 9369152096
chore(deps): bump tracing-core from 0.1.21 to 0.1.22 (#3675) 
Bumps [tracing-core](https://github.com/tokio-rs/tracing) from 0.1.21 to 0.1.22.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-core-0.1.21...tracing-core-0.1.22)

---
updated-dependencies:
- dependency-name: tracing-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-08 15:12:39 +00:00
dependabot[bot] ba87ae2918
chore(deps): bump crc32fast from 1.3.1 to 1.3.2 (#3668) 
Bumps [crc32fast](https://github.com/srijs/rust-crc32fast) from 1.3.1 to 1.3.2.
- [Release notes](https://github.com/srijs/rust-crc32fast/releases)
- [Commits](https://github.com/srijs/rust-crc32fast/compare/v1.3.1...v1.3.2)

---
updated-dependencies:
- dependency-name: crc32fast
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-02-08 14:48:31 +00:00
dependabot[bot] 3c9a27b31b
chore(deps): bump futures-test from 0.3.19 to 0.3.21 (#3667) 
Bumps [futures-test](https://github.com/rust-lang/futures-rs) from 0.3.19 to 0.3.21.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.19...0.3.21)

---
updated-dependencies:
- dependency-name: futures-test
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-02-08 14:35:34 +00:00
dependabot[bot] 9a874186d5
chore(deps): bump libc from 0.2.116 to 0.2.117 (#3666) 
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.116 to 0.2.117.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.116...0.2.117)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-08 13:59:02 +00:00
Marco Neumann 5de4d6203f
refactor: catalog transaction (#3660) 
* refactor: catalog Unit of Work (= transaction)

Setup an inteface to handle Units of Work within our catalog. Previously
both the Postgres and the in-mem backend used "mini-transactions on
demand". Now the caller has a clear way to establish boundaries and
gets read and write isolation. A single `Arc<dyn Catalog>` can create as
many `Box<dyn UnitOfWork>` as you like, but note that depending on the
backend you may not scale infinitely (postgres will likely impose
certain limits and the in-mem backend limits concurrency to 1 to keep
things simple).

* docs: improve wording

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

* refactor: rename Unit of Work to Transaction

* test: improve `test_txn_isolation`

* feat: clearify transaction drop semantics

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-08 13:38:33 +00:00
Luke Bond de2a013786
feat: gitops adapter (#3656) 
* feat: scaffolding of gitops adapter bin crate

* chore: refactor gitops adapter; calls CLI now; status update fixes

* feat: gitops adapter now calls out to CLI once per topic; improved tests

* chore: add mock failure script for gitops adapter

* chore: update workspace-hack

* chore: refactor away unecessary to_string in gitops syncer

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-08 13:27:36 +00:00
Markus Westerlind 0bd7941a18
fix(REPL): Don't buffer lines until a trailing semicolon is found and add history hinting (#3630) 
* fix(REPL): Don't buffer lines until a trailing semicolon is found

The repl would silently buffer all lines until a trailing semicolon were found which
resulted in some very confusing error messages as I would input invalid commands followed
by a command I thought were valid, except I'd still get an error due to the previous command being buffered.

This uses rustyline's helper feature to detect incomplete input (no trailing semicolon) and makes
it accept multiline input until the input is completed.

I also included some of rustyline's default hint and highlighting while I was at it.

* chore: cargo clippy

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-03 17:11:01 +00:00
Marco Neumann 50cff27b01
chore: remove rdkafka dependency (#3625) 
All features are now covered by rskafka. This also removes the need to
specify a server ID for write buffer consumers. This was only used for
rdkafka since there we needed to specify a consumer group, even though
we did not use any transactions.
 2022-02-03 13:33:56 +00:00
Marco Neumann bc4b7f8a5b
test: ensure that rskafka and rdkakfa work together (#3624) 
* chore: upgrade rskafka + enable snappy support

* test: ensure that rskafka and rdkakfa work together

Before removing rdkafka ensure that:

- rskafka can consume existing messages produced by rdkafka so we do not
  need to drain existing topics
- rdkafka can consume new messages produced by rskafka so we can roll
  back

I ran the whole `write_buffer` test suite (including the newly added
tests) using Apache Kafka as well as Redpanda.

* test: ensure we handle consumer offset in error case correctly

* docs: explain test setup

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-03 12:52:42 +00:00
Paul Dix ce46bbaada
feat: wire up the write buffer to the ingester process (#3533) 
This adds the scaffolding for the ingester server to consume data from Kafka. This ingests data in an in memory structure while creating records in the catalog for any partitions that don't yet exist.

I've removed catalog_update.rs in ingester for now. That was mostly a placeholder and will be going in a combination of handler.rs and data.rs on my next PR which will have some primitive lifecycle wired up.

There's one ugly bit here where the DML write is cloned because it's getting borrowed to output spans and metrics. I'll need to follow up with a refactor to make it so that the DML write's tables can be consumed without it gumming up the metrics stuff.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-03 11:47:28 +00:00
kodiakhq[bot] 9079baf19a
Merge branch 'main' into dom/schema-validation 2022-02-02 16:27:35 +00:00
Andrew Lamb 030a2cb4c1
chore: Update datafusion (#3613) 
* chore: Update datafusion

* fix: update for latest DF API

* fix: another API change

* fix: clippy

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-02 16:27:11 +00:00
Dom Dwyer 4744c5804e refactor: remove Dashmap 
Swap Dashmap for a regular RwLock<HashMap<..,>> due to soundness issues:

    https://rustsec.org/advisories/RUSTSEC-2022-0002
 2022-02-02 14:04:53 +00:00
Dom Dwyer 39d489d9e7 refactor: enable schema validation 
Adds the SchemaValidator to the DML handler stack - this adds it into
the request path in router2.
 2022-02-02 14:04:14 +00:00
Dom Dwyer 6598023726 feat: cache NamespaceSchema in validator 
Adds an in-memory cache of table schemas to the SchemaValidator DML
handler.

The cache pulls from the global catalog when observing a column for the
first time, and pushes the column type to set it for subsequent requests
if it does not exist (this pull & push is done by atomically by the
catalog in an "upsert" call).

The in-memory cache is sharded by namespace, with each shard guarded by
an individual lock to minimise contention between readers (the expected
average case) and writers (only when adding new columns/tables).

Relies on the catalog to serialise new column creation and validate
parallel creation requests.
 2022-02-02 13:04:53 +00:00
Dom Dwyer c81f207298 feat: schema validation 
Implements a write schema validation DML handler, denying requests that
conflict with the schema within the global catalog. Additive schema
changes are accepted, incrementally updating the global catalog schema.

Deletes are passed through unchanged and unvalidated.
 2022-02-02 13:04:53 +00:00
Edd Robinson 5441682207 feat: add support for parsing predicate 2022-02-02 11:02:33 +00:00
Edd Robinson c1f5994660
refactor: move sql parsing -> RPC predicate into own crate (#3604) 
* chore: create crate

* refactor: move module to new crate
 2022-02-02 10:41:57 +00:00
Edd Robinson fa546047fb
refactor: update dep with API change (#3596) 2022-02-01 14:53:10 +00:00
Edd Robinson 443fd00c1b
Merge branch 'main' into er/feat/parse_sql_rpc 2022-02-01 13:55:08 +00:00
Marco Neumann 0a2cb36ddf
fix: workspace-hack (#3593) 2022-02-01 12:32:58 +00:00
Marco Neumann 22778a3a80
chore: upgrade rskafka and parking_lot (#3592) 2022-02-01 11:50:42 +00:00
Edd Robinson 175732c3ca feat: basic sqlparser -> RPCNode 2022-02-01 10:26:03 +00:00
Marco Neumann b326b62b44
feat: buffer writes when writing to RSKafka (#3520) 2022-02-01 10:07:52 +00:00
kodiakhq[bot] 8bef2c105c
Merge branch 'main' into cn/persist 2022-01-31 18:50:45 +00:00
Andrew Lamb 7b96a37165
chore: Update datafusion (#3586) 
* chore: update DataFusion to f849968057ddddccc9aa19915ef3ea56bf14d80d

* fix: reduce overhead of creating physical expressions

* chore: use MemTrackingMetrics

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-31 18:15:28 +00:00
Carol (Nichols || Goulding) bf89162fa5
refactor: Move IoxMetadata to parquet_file 2022-01-31 10:36:33 -05:00
Carol (Nichols || Goulding) dd9620da0c
feat: Create a new proto definition for the new design's IoxMetadata 2022-01-31 10:36:32 -05:00
Carol (Nichols || Goulding) 5e0e0d8aa7
feat: Write parquet to object storage in a similar way as parquet_file::Storage 2022-01-31 10:36:32 -05:00
Carol (Nichols || Goulding) c633c9bc5c
feat: Wire object store into ingester persistence 2022-01-31 10:36:30 -05:00
Raphael Taylor-Davies 4101d16f71
chore: feature flag consistency (#3574) 
* chore: feature flag consistency

* chore: add aarch64-apple-darwin to hakari

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-28 16:38:59 +00:00
Nga Tran 8735ede74f
feat: IoxMetadata for parquet file (#3547) 
* feat: IoxMetadata for parquet file

* fix: typos

* refactor: address review comments

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-28 14:41:59 +00:00
Dom 1669acc0c2
build: update tokio (#3566) 
Release notes:
    https://github.com/tokio-rs/tokio/releases/tag/tokio-1.16.0
 2022-01-28 10:36:12 +00:00
Andrew Lamb b486258dfb
chore: run cargo update (#3559) 
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-27 21:05:31 +00:00
Dom 9201023ea4
feat: schema validation for MutableBuffer instances (#3554) 
* refactor: Debug bounds on Catalog trait

* feat: validate MutableBatch schema

Changes the schema validation code to validate MutableBatch instances
(coming from a pre-parsed LP write, and non-LP-based writes) instead of
parsed LP lines.

* refactor: Send bound on boxed errors

* refactor: clippy

Allow assert_eq!(bool, bool) for readability.

* refactor: no PartialEq<MB Column> for ColumnSchema

Remove the PartialEq<mutable_buffer::Column> for ColumnSchema - it's
definitely more readable as a method call.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-27 20:55:18 +00:00
Luke Bond 4a96e52290
feat: router2 sharder benchmarking (#3558) 
* feat: benchmarking the router2 sharder

* chore: added throughput to sharder benchmarks; vary num buckets
 2022-01-27 18:09:16 +00:00
Andrew Lamb 2062267d0f
chore: Update hashbrown (#3551) 
* chore: Update hashbrown

* fix: hakari

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-27 15:34:10 +00:00
Dom ce568ab447
build(iox_catalog): remove unused dependencies (#3552) 
* build: don't pull in all of tokio

We already specify the tokio features we need so "full" (all features)
is not necessary.

* build: remove chrono dependency

Appears unused.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-27 15:14:42 +00:00
Raphael Taylor-Davies 21c1824a7a
refactor: remove table_names from Predicate (#3545) 
* refactor: remove table_names from Predicate

* chore: fix benchmarks

* chore: review feedback

Co-authored-by: Edd Robinson <me@edd.io>

* chore: review feedback

* chore: replace Default::default with InfluxRpcPredicate::default()

Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-27 14:44:49 +00:00
Andrew Lamb 5488c257d1
chore: Update datafusion, upgrade to arrow/parqet/arrow-flight 8.0.0 (#3517) 
* chore: Update datafusion

* chore: update to arrow 8

* fix: update to use new DataFusion APIs

* fix: update case for sortedness

* fix: cargo hakari
 2022-01-27 13:33:27 +00:00
Andrew Lamb 7261571abf
fix: Revert "chore: temporarily hack around datafusion tempfiles" (#3525) 
This reverts commit ae5763c1cb6bb4a98ffe0779a3a35f6daaf10971.

Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-27 12:08:49 +00:00
Carol (Nichols || Goulding) bc44d33108
feat: Implement a snapshot method on DataBuffer (#3518) 
* feat: Implement a snapshot method on DataBuffer

Fixes #3510.

* test: Add a test snapshotting batches with different but compatible schemas

* fix: Simplify min/max sequencer number collection

The first batch should always have the min sequencer number. The last
batch should always have the max sequencer number. The min should always
be less than (or equal to, in case there's only one batch) the max.
 2022-01-26 15:22:51 +00:00
Marco Neumann 2928254c0f
fix: test logging (#3536) 
- Use a more standard way to setup the tracing subsystem (as described
  in tracing-subscriber docs)
- Also capture content from `log` crate
- Play nice w/ Rust's libtest message capture

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-26 10:28:51 +00:00
Dom b846ead320
feat(router2): shard writes/deletes into write buffer (#3499) 
* feat: Sequencer wrapper

This type wraps an underlying WriteBufferWriter implementation, tagging
it with a sequencer ID it should use when enqueuing operations to the
buffer.

* feat: mock sharder

Implements a mock Sharder impl that returns pre-configured responses to
shard(), and captures the input to the call.

* feat: sharded write buffer

Implements sharding of ops into an underlying WriteBuffer.

Writes are sharded by some abstract Sharder impl, collated per shard to
maximise the size of each op (and therefore compression efficiency),
converted into a DML operation and then enqueued in parallel to the
underlying WriteBuffer implementation.

Deletes are modelled as being mapped to a single write buffer shard,
which is the case while we support sharding based on the table &
namespace only. Deletes will be extended to support (potentially)
multiple shards when column overrides are implemented.

* refactor: runtime write buffers

Switch from using static dispatch, to using a runtime specified
WriteBufferWriting implementation.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-25 15:19:48 +00:00
Andrew Lamb 51a0f8a56a
chore: temporarily hack around datafusion tempfiles (#3524) 2022-01-25 12:30:29 +00:00
NGA-TRAN f9c1e80a7f chore: update thread_local 
chore: update thread_local
 2022-01-24 13:37:52 -05:00
NGA-TRAN 797ba459b9 chore: merge main to branch 2022-01-24 12:06:23 -05:00
NGA-TRAN 5f98a07b7f chore: add Corgo.lock 2022-01-24 12:03:02 -05:00
Paul Dix bb893510a0 feat: Add scaffolding for ingester server 
* Adds a new ingester command to start an ingester server
* Moves previous ingester server over to handler
* Skeleton for gRPC and HTTP handlers
 2022-01-21 18:02:19 -05:00
Andrew Lamb 9615feacb3
fix(InfluxQL): Support RegEx with escape sequences not supported by Rust regex (#3502) 
* fix(InfluxQL): Translate unsupported meta characters

* fix: remove debugging

* fix: clippy sacrifice

* docs: Add additional background and rationale for rewriting

* fix: doc link
 2022-01-21 14:40:10 +00:00
kodiakhq[bot] 5b22abf6a2
Merge branch 'main' into crepererum/wb_rskafka 2022-01-20 16:48:10 +00:00
Andrew Lamb 9b6e626626
chore: Update datafusion (and get fix for influxql test failure) (#3484) 
* test: add tests for comparing dictionary arrays

* chore: update datafusion deps

* refactor: Update code for DataFusion API changes

* fix: update test

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-20 14:01:47 +00:00
Marco Neumann 76dd62a6c2 feat: RSKafka-driven write buffer 2022-01-20 12:36:10 +01:00
Dom 36d50d083f feat: sharder trait & impl 
This commit defines the Sharder trait that should allow us to implement
multiple sharding strategies over a defined set of input types (such as
a MutableBatch for writes, DeletePredicate for deletes, etc).

This commit also includes a jump hash implementation that consistently
shards (table name, namespace) tuples to a given shard for all input
types.
 2022-01-20 11:10:37 +00:00
Paul Dix d825dab8e2 fix: hakari workspace hack 2022-01-19 14:48:00 -05:00
Paul Dix 41038721e1 feat: Add parquet file records to iox_catalog 
* Adds ParquetFile and scaffolding to IOx catalog
* Changed the file_location in parquet_file to object_store_id which is a uuid
 2022-01-19 14:14:54 -05:00
NGA-TRAN edb97f51cf refactor: add persisting struct 2022-01-19 12:36:18 -05:00
NGA-TRAN 8a17e1c132 refactor: address review comments 2022-01-19 11:20:20 -05:00
NGA-TRAN e8294d21ec fix: add .lock 2022-01-18 22:02:27 -05:00
NGA-TRAN b57f027e35 refactor: address review comments 2022-01-18 20:57:13 -05:00
NGA-TRAN 367a9fb812 fix: add workspace-hack 2022-01-18 18:10:42 -05:00
NGA-TRAN 125285ae9a feat: commit in order to pull and merge new commit from main 2022-01-18 16:11:25 -05:00
NGA-TRAN 23290fd2ff fix: new data structures suggested by reviewers 2022-01-18 14:04:07 -05:00
NGA-TRAN ef336b4659 feat: add ingester crate and a few basic data structures for its data lifecycle 2022-01-17 15:38:03 -05:00
kodiakhq[bot] 02d45bbae0
Merge branch 'main' into dom/delete-handler 2022-01-17 15:26:33 +00:00
Andrew Lamb 1843476651
chore: Update datafusion deps (#3471) 
* chore: Update datafusion

* refactor: Update to use new Exec plan APIs

* fix: error message

* fix: fixup last bit

* fix: clippy

* fix: doclink

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-17 15:26:19 +00:00
Paul Dix d4d4b4c087 fix: workspace hack after adding iox_catalog crate 2022-01-17 10:04:24 -05:00
Dom aa6f118487 feat: iox_catalog sequencers (#3465) 
* refactor: ensure sequencers are unique

Adds a unique constraint to ensure only one sequencer record exists for
each Kafka (topic, partition).

* test: use DSN from env for integration tests

Removes the hard-coded DSN, instead sourcing it from the DATABASE_URL
environment variable.

* docs: integration testing for iox_catalog

Documents the required steps in order to run the Postgres integration
tests for the iox_catalog crate.

* feat(iox_catalog): create & list sequencers

Adds support for interacting with the "sequencer" table.

* chore: update lockfile

Running cargo in iox_catalog generates a lockfile diff.
 2022-01-17 10:00:31 -05:00
Dom 6f2e10cab6 feat(router2): implement delete API handler 
Adds support for the HTTP v2 delete API endpoint.
 2022-01-17 14:57:36 +00:00
Dom 885c831aff refactor: avoid constructing DmlOperation 
Instead of converting the set of MutableBatches into a DmlOperation to
shard into more DmlOperation instances, the sharder can operate directly
on the MutableBatches.
 2022-01-17 11:57:32 +00:00
Dom 40a290f6f7 feat: router2 HTTP handlers 
Implements the HTTP v2 write API endpoint for router2.
 2022-01-17 11:57:28 +00:00
Dom 80b12d417c feat: abstract DML handler 
Defines the DmlHandler trait responsible for processing a request in
some abstract way, decoupling the HTTP/gRPC request handlers from the
underlying routing logic.
 2022-01-17 11:56:04 +00:00
Marco Neumann c399e676ca chore: upgrade clap to v3 2022-01-17 12:12:46 +01:00
Raphael Taylor-Davies 89db894df4
fix: serde_json `arbitrary_precision` (#3458) (#3469) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-17 09:50:54 +00:00
Andrew Lamb dd23056efd
chore: update datafusion, arrow, prost, tonic, pbjson, etc (#3455) 
* chore: update datafusion, arrow, prost, tonic, etc

* fix: update pprof as well

* chore: update hakari

* fix: update pbjson

* chore: update heappy

* fix: hakari

* fix: workaround https://github.com/influxdata/influxdb_iox/issues/3458

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-13 17:07:15 +00:00
Dom b9bee7f735 build: update workspace-hack 2022-01-12 15:09:06 +00:00
Dom a8cb8755de feat: new router2 crate 
This commit adds an almost-empty router2 crate containing enough of a
skeleton to plumb into the IOx CLI/server runner.
 2022-01-12 14:43:10 +00:00
Andrew Lamb 530240cb5a
chore: Update datafusion dependency (#3443) 2022-01-11 20:08:25 +00:00
Marco Neumann f3f6f335a9
chore: upgrade to snafu 0.7 (#3440) 2022-01-11 19:22:36 +00:00
Marco Neumann 37bb7f2120 chore: `cargo update` 
dependabot currently doesn't work due to
https://github.com/dependabot/dependabot-core/issues/4574

Excluded `quote` due to
https://github.com/dtolnay/quote/issues/204
 2022-01-11 14:57:51 +01:00
Carol (Nichols || Goulding) f9174c483b
refactor: Extract server::db into its own crate (#3417) 
* refactor: Extract JobRegistry from the server crate

Both the server crate and a db crate that I'm about to extract depend on
JobRegistry, so to avoid making circular dependencies, extract the
JobRegistry to its own crate.

* refactor: Move db out of server into its own crate

Fixes #2821.
 2021-12-23 22:01:17 +00:00
Andrew Lamb 527885f7f8
chore: Update datafusion (#3413) 
* chore: Update datafusion and update code to handle timezone aware timestamps

* fix: cargo hakari

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-12-23 14:52:12 +00:00
Andrew Lamb ccba68fe3e
chore: Update datafusion and sqlparser, and arrow (#3385) 
* chore: Update datafusion and sqlparser, and arrow

* fix: cargo hakari

* test: fix metadata size

* fix: update some metadata sizes

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-12-16 12:13:40 +00:00
Marko Mikulicic b0c6713bd7 fix: Don't swallow exit code when hakari test fails 2021-12-14 14:49:14 +01:00
Paul Dix 01e86a031e feat: add regex to bucket writers assignment in data generator 
This adds the ability to specify a regex to match against database names when specifying what agents should write to which buckets in the data generator.

A default has also been added for ratio so that it doesn't need to be specified if only a single database writer is defined.
 2021-12-09 13:39:33 -05:00
Marco Neumann 876a9af35a fix: limit number of S3 connections 
Otherwise the whole thing blows up when starting a server that has many
DBs registerd, because we potentially create 1 connection per DB (e.g.
to read out the preserved catalog).

Fixes #3336.
 2021-12-08 19:06:02 +01:00
Andrew Lamb 218042784f
feat: add `system.queries `system table (#3328) 
* feat: Add query log and system table

* docs: Add docstrings for test normalization
 2021-12-08 16:26:24 +00:00
Andrew Lamb 1156a81567
feat: Add pbjson serialization for storage rpc (#3324) 
* feat: Add pbjson serialization for storage rpc

* chore: update pbjson-build to 0.1.1

Co-authored-by: Raphael Taylor-Davies <r.taylordavies@googlemail.com>
 2021-12-07 16:39:16 +00:00
Carol (Nichols || Goulding) 7499eac067
fix: Disable uuid serde feature; we're not actually serializing any UUIDs 
Connects to #3117.
 2021-12-06 09:37:31 -05:00
Carol (Nichols || Goulding) a23695d470
fix: Remove unused generated_types dependency in iox_data_generator 2021-12-06 09:37:16 -05:00
Carol (Nichols || Goulding) 96afb9ffe9
fix: Remove unused tokio dev-dependency in mutable_buffers 2021-12-06 09:37:16 -05:00
Carol (Nichols || Goulding) e8d9e35321
fix: influxdb_iox tests need the json feature of reqwest 
I think this didn't fail before because influxdb_iox used to use
influxdb2_client, which turned on the json feature. Then the workspace
hack crate also turned on the json feature, so when influxdb_iox stopped
using influxdb2_client, it *still* worked. But excluding the object
store crates from hakari analysis that also turned on the json feature
meant nothing turned it on anymore, so now influxdb_iox needs to turn it
on itself.
 2021-12-06 09:37:15 -05:00
Carol (Nichols || Goulding) 76909b07e4
fix: Exclude optional object-store implementation crates from cargo-hakari 
Aha! cargo-hakari turns on all features, but usually in development all
the specific object storage implementation crates are off.

Excluding them in the hakari config reduces the workspace crate a bit.

Connects to #3117.
 2021-12-06 09:37:15 -05:00
Carol (Nichols || Goulding) 1b0a2eb7c7
fix: Specify nom features in the same way cexpr does 
This removes nom from the workspace-hack crate. I'm not sure what's
going on here as cexpr is setting default-features false but then
explicitly setting the features to be the same as nom's default
features, and cargo-hakari seems to be resolving that differently than
using nom's default features. I think this might be a bug in
cargo-hakari, going to investigate.

Connects to #3117.
 2021-12-06 09:37:14 -05:00
Carol (Nichols || Goulding) 6b940166fe
fix: Remove a deprecated memchr feature from the workspace crate 
This was made possible by upgrading nom, which was using the deprecated
feature name. This will save us from pain in the future when the
deprecated feature is removed.

Unfortunately, this adds nom to the workspace hack crate, but we'll see
if it needs to stay there :)

Connects to #3117.
 2021-12-06 09:37:14 -05:00
Carol (Nichols || Goulding) e4cc04612b
chore: Update bindgen, nom, cexpr, and related crates 
Connects to #3117.

This gets rid of some duplicate versions of crates in our dependency
tree.
 2021-12-06 09:37:14 -05:00
Carol (Nichols || Goulding) bd0a2b36e3
fix: Remove futures-util proc-macro features and update to futures-* 0.3.18 
The `futures-util` crate's `proc-macro-hack` and `proc-macro-nested`
features used to be part of `futures-util`'s default feature sets, so
the workspace-hack crate listed them.

`futures-util` 0.3.18 removed those features entirely.

Cargo won't update to versions of crates that don't have features that
have been specified, so the workspace-hack crate was blocking Cargo from
updating `futures-util` and a few other crates to 0.3.18.

Cargo's docs recommend against removing features in semver-compatible
releases, but nothing really prevents crate authors from doing this.

Connects to #3117.
 2021-12-06 09:37:14 -05:00
Carol (Nichols || Goulding) ddc1504135
fix: Exclude influxdb2_client from the workspace hack crate 
This enables the removal of the futures and futures-io crates.

Connects to #3117.
 2021-12-06 09:37:14 -05:00
Carol (Nichols || Goulding) 8ad2b5e9fd
fix: In end-to-end tests, use influxdb_iox_client instead of influxdb2_client 2021-12-06 09:37:14 -05:00
Carol (Nichols || Goulding) cf6286364e
fix: Exclude criterion from hakari analysis 
Connects to #3117.

Criterion builds clap without default features, which causes clap to be
listed in the workspace hack crate with its default features. In
general, actually building criterion will be rare (and we should move
all benches to separate crates that can be excluded) so it should be
fine to rebuild clap when building benchmarks anyway.
 2021-12-06 09:36:50 -05:00
Carol (Nichols || Goulding) 0fda9608d8
fix: Exclude optional pprof crate from workspace-hack 2021-12-06 09:36:32 -05:00
Edd Robinson 30bc8e5eba
Merge branch 'main' into dependabot/cargo/clap-2.34.0 2021-12-06 10:51:52 +00:00
dependabot[bot] 0b87c1e170
chore(deps): bump clap from 2.33.3 to 2.34.0 
Bumps [clap](https://github.com/clap-rs/clap) from 2.33.3 to 2.34.0.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/commits)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-12-06 09:07:47 +00:00
dependabot[bot] e3aa97f04a
chore(deps): bump tracing-subscriber from 0.3.2 to 0.3.3 
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.3.2 to 0.3.3.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.2...tracing-subscriber-0.3.3)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-12-06 09:07:40 +00:00
dependabot[bot] 472c91d520
chore(deps): bump reqwest from 0.11.6 to 0.11.7 
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.6 to 0.11.7.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.6...v0.11.7)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-12-06 08:55:14 +00:00
dependabot[bot] 99737ba379
chore(deps): bump libc from 0.2.108 to 0.2.109 
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.108 to 0.2.109.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.108...0.2.109)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-12-06 08:42:04 +00:00
dependabot[bot] dd7d5cd7e3
chore(deps): bump handlebars from 4.1.5 to 4.1.6 
Bumps [handlebars](https://github.com/sunng87/handlebars-rust) from 4.1.5 to 4.1.6.
- [Release notes](https://github.com/sunng87/handlebars-rust/releases)
- [Changelog](https://github.com/sunng87/handlebars-rust/blob/master/CHANGELOG.md)
- [Commits](https://github.com/sunng87/handlebars-rust/compare/v4.1.5...v4.1.6)

---
updated-dependencies:
- dependency-name: handlebars
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-12-06 08:31:22 +00:00
dependabot[bot] 573ee632b1
chore(deps): bump crc32fast from 1.2.2 to 1.3.0 
Bumps [crc32fast](https://github.com/srijs/rust-crc32fast) from 1.2.2 to 1.3.0.
- [Release notes](https://github.com/srijs/rust-crc32fast/releases)
- [Commits](https://github.com/srijs/rust-crc32fast/compare/v1.2.2...v1.3.0)

---
updated-dependencies:
- dependency-name: crc32fast
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-12-06 01:22:09 +00:00
Andrew Lamb 9e8639f230
chore: Update DataFusion pin (#3279) 
* chore: Update DataFusion pin

* fix: Update for new DF API

* fix: update plan output

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-12-02 12:42:28 +00:00
Marco Neumann c961454dcd feat: `jaeger-debug-id` from data generator 2021-12-01 14:33:09 +01:00
Andrew Lamb 20da62eb87
chore: Update `arrow`, `arrow-flight`, and `parquet` to 6.3.0 (#3254) 
* chore: Update `arrow`,  `arrow-flight`, and `parquet` to 6.3.0

* fix: update metadata size
 2021-11-30 12:30:17 +00:00
kodiakhq[bot] 068b44334d
Merge branch 'main' into crepererum/dml_shard_new_types 2021-11-29 10:33:49 +00:00
dependabot[bot] d7053cad6b
chore(deps): bump futures-io from 0.3.17 to 0.3.18 (#3241) 
Bumps [futures-io](https://github.com/rust-lang/futures-rs) from 0.3.17 to 0.3.18.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.17...0.3.18)

---
updated-dependencies:
- dependency-name: futures-io
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-29 10:33:26 +00:00
dependabot[bot] eba6ebd67a
chore(deps): bump futures-channel from 0.3.17 to 0.3.18 
Bumps [futures-channel](https://github.com/rust-lang/futures-rs) from 0.3.17 to 0.3.18.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.17...0.3.18)

---
updated-dependencies:
- dependency-name: futures-channel
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-29 09:46:01 +00:00
dependabot[bot] 5d784ba866
chore(deps): bump serde_json from 1.0.71 to 1.0.72 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.71 to 1.0.72.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.71...v1.0.72)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-29 09:28:07 +00:00
dependabot[bot] 0ac4b46ebf
chore(deps): bump futures-task from 0.3.17 to 0.3.18 
Bumps [futures-task](https://github.com/rust-lang/futures-rs) from 0.3.17 to 0.3.18.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.17...0.3.18)

---
updated-dependencies:
- dependency-name: futures-task
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-29 09:13:12 +00:00
dependabot[bot] 3bedf6e62a
chore(deps): bump crc32fast from 1.2.1 to 1.2.2 
Bumps [crc32fast](https://github.com/srijs/rust-crc32fast) from 1.2.1 to 1.2.2.
- [Release notes](https://github.com/srijs/rust-crc32fast/releases)
- [Commits](https://github.com/srijs/rust-crc32fast/compare/v1.2.1...v1.2.2)

---
updated-dependencies:
- dependency-name: crc32fast
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-29 09:00:01 +00:00
dependabot[bot] b46d07b8a5
chore(deps): bump predicates from 2.0.3 to 2.1.0 
Bumps [predicates](https://github.com/assert-rs/predicates-rs) from 2.0.3 to 2.1.0.
- [Release notes](https://github.com/assert-rs/predicates-rs/releases)
- [Changelog](https://github.com/assert-rs/predicates-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/predicates-rs/compare/v2.0.3...v2.1.0)

---
updated-dependencies:
- dependency-name: predicates
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-29 08:40:45 +00:00
dependabot[bot] b9d804c236
chore(deps): bump futures-core from 0.3.17 to 0.3.18 
Bumps [futures-core](https://github.com/rust-lang/futures-rs) from 0.3.17 to 0.3.18.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.17...0.3.18)

---
updated-dependencies:
- dependency-name: futures-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-29 08:29:32 +00:00
dependabot[bot] c729fc6a25
chore(deps): bump rdkafka from 0.27.0 to 0.28.0 
Bumps [rdkafka](https://github.com/fede1024/rust-rdkafka) from 0.27.0 to 0.28.0.
- [Release notes](https://github.com/fede1024/rust-rdkafka/releases)
- [Changelog](https://github.com/fede1024/rust-rdkafka/blob/master/changelog.md)
- [Commits](https://github.com/fede1024/rust-rdkafka/compare/v0.27.0...v0.28.0)

---
updated-dependencies:
- dependency-name: rdkafka
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-29 08:18:02 +00:00
dependabot[bot] c0fe972b20
chore(deps): bump cc from 1.0.71 to 1.0.72 
Bumps [cc](https://github.com/alexcrichton/cc-rs) from 1.0.71 to 1.0.72.
- [Release notes](https://github.com/alexcrichton/cc-rs/releases)
- [Commits](https://github.com/alexcrichton/cc-rs/compare/1.0.71...1.0.72)

---
updated-dependencies:
- dependency-name: cc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-29 01:29:47 +00:00
Marco Neumann ad65687cba refactor: consolidate sharding implementation 
Move sharding implementation from router to `dml` (where the types can
be consumed w/o cloning the data) and only support the new sharding
configs.

The old sharding configs will be removed soon.
 2021-11-26 13:00:36 +01:00
Marco Neumann edf7becd20 fix: address review comments 2021-11-24 12:09:52 +01:00
Marco Neumann f9110eecad test: migrate `DistibutedTest` to use proper router 2021-11-23 15:39:53 +01:00
Andrew Lamb 50e9e02ff7
chore: Update datafusion (#3188) 
* chore: Update datafusion

* fix: Update for LogicalPlan changes

* fix: Update explain plan output

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-23 12:23:23 +00:00
kodiakhq[bot] d16a7759ca
Merge branch 'main' into cn/workspace-hack 2021-11-22 17:05:31 +00:00
Carol (Nichols || Goulding) f3c814e091
fix: Exclude crates conductor depends on from workspace hack 2021-11-22 11:55:38 -05:00
dependabot[bot] a24addc5d7
chore(deps): bump handlebars from 4.1.4 to 4.1.5 (#3176) 
Bumps [handlebars](https://github.com/sunng87/handlebars-rust) from 4.1.4 to 4.1.5.
- [Release notes](https://github.com/sunng87/handlebars-rust/releases)
- [Changelog](https://github.com/sunng87/handlebars-rust/blob/master/CHANGELOG.md)
- [Commits](https://github.com/sunng87/handlebars-rust/compare/v4.1.4...v4.1.5)

---
updated-dependencies:
- dependency-name: handlebars
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-11-22 09:56:58 +00:00
dependabot[bot] ca5a437be1
chore(deps): bump hyper from 0.14.14 to 0.14.15 
Bumps [hyper](https://github.com/hyperium/hyper) from 0.14.14 to 0.14.15.
- [Release notes](https://github.com/hyperium/hyper/releases)
- [Changelog](https://github.com/hyperium/hyper/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/hyper/compare/v0.14.14...v0.14.15)

---
updated-dependencies:
- dependency-name: hyper
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-22 09:02:53 +00:00
dependabot[bot] bd974798d4
chore(deps): bump tracing-subscriber from 0.3.1 to 0.3.2 
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.3.1 to 0.3.2.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.1...tracing-subscriber-0.3.2)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-22 08:51:18 +00:00
dependabot[bot] 72305ee446
chore(deps): bump tower from 0.4.10 to 0.4.11 
Bumps [tower](https://github.com/tower-rs/tower) from 0.4.10 to 0.4.11.
- [Release notes](https://github.com/tower-rs/tower/releases)
- [Commits](https://github.com/tower-rs/tower/compare/tower-0.4.10...tower-0.4.11)

---
updated-dependencies:
- dependency-name: tower
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-22 08:40:20 +00:00
dependabot[bot] b83308183b
chore(deps): bump libc from 0.2.106 to 0.2.108 
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.106 to 0.2.108.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.106...0.2.108)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-22 08:29:09 +00:00
dependabot[bot] 5c6deb81db
chore(deps): bump serde_json from 1.0.70 to 1.0.71 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.70 to 1.0.71.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.70...v1.0.71)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-22 01:21:06 +00:00
Carol (Nichols || Goulding) fa36adc3e3
fix: Update the workspace hack crate that no longer needs cache_loader_async 2021-11-19 14:34:24 -05:00
Carol (Nichols || Goulding) a6908f1613
fix: Exclude tikv-jemalloc-sys from workspace hack as it's mutually exclusive with heappy 2021-11-19 13:46:17 -05:00
Carol (Nichols || Goulding) 2f452268af
fix: Exclude some bench/test crates from the workspace hack 2021-11-19 13:38:27 -05:00
Carol (Nichols || Goulding) 9fd4a560f5
feat: Results of running cargo hakari manage-deps 2021-11-19 09:21:57 -05:00
Carol (Nichols || Goulding) d1c6f76a0e
feat: Results of running cargo hakari generate 2021-11-19 09:21:15 -05:00
Carol (Nichols || Goulding) 7783e4a7ff
fix: Make delete/restore aliases for release/claim; remove tombstone 
Fixes #2680
 2021-11-17 11:41:08 -05:00
Andrew Lamb 47acd181c5
chore: Update datafusion + arrow/parquet/arrow-flight 6.2.0 (#3136) 
* chore: Update datafusion and arrow

* chore: update arrow/parquet/arrow-flight to 6.2.0

* refactor: Add table_exists to SchemaProvider impl

* fix: clippy

* fix: clippy 2

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-17 14:04:49 +00:00
Dom da61966858
build: remove proc-macro2 pin (#3137) 
Seems unused, builds without it!

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-17 13:55:39 +00:00
Marco Neumann 04d8133227 chore: upgrade tokio to 1.14.0 to fix RUSTSEC-2021-0124 2021-11-17 09:44:52 +01:00
Marco Neumann e6fdd79a0f feat: emit Kafka stats as metrics instead of logs 
This maps a subset of Kafka stats as metrics. The set can -- of course
-- be changed in the future depending on our needs.

Fixes #3100.
 2021-11-16 17:18:41 +01:00
Marco Neumann 4e71de508e fix: ensure `ConsistenHasher` is consistent 
The std `DefaultHasher` is NOT guaranteed to stay the same, so let's
directly use the `SipHasher13` which at the moment (2021-11-15) is used
by the standard lib.

Fixes #3063.
 2021-11-15 17:39:17 +01:00
Raphael Taylor-Davies 58f3e2e559
refactor: move delete predicate proto serialization to generated_types (#3108) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-15 12:02:14 +00:00
Raphael Taylor-Davies 3d091208af
refactor: move delete predicate into data_types (#2731) (#3094) 
* refactor: move delete predicate into dml (#2731)

* refactor: move DeletePredicate to data_types

* chore: fix doc

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-15 10:28:58 +00:00
dependabot[bot] 3485d70409
chore(deps): bump serde_json from 1.0.69 to 1.0.70 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.69 to 1.0.70.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.69...v1.0.70)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-15 08:48:44 +00:00
dependabot[bot] 7fb967d088
chore(deps): bump nom from 7.0.0 to 7.1.0 
Bumps [nom](https://github.com/Geal/nom) from 7.0.0 to 7.1.0.
- [Release notes](https://github.com/Geal/nom/releases)
- [Changelog](https://github.com/Geal/nom/blob/main/CHANGELOG.md)
- [Commits](https://github.com/Geal/nom/commits)

---
updated-dependencies:
- dependency-name: nom
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-15 08:34:55 +00:00
Jake Goulding d20b7ca390 fix: Remove unused dependencies 
Thanks, `cargo udeps`!
 2021-11-12 16:42:02 -05:00
Raphael Taylor-Davies da7ba511b2
feat: remove legacy per-line partitioning logic (#3087) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-11 16:50:42 +00:00
Raphael Taylor-Davies 6f268f8260
refactor: extract DML types (#2731) (#3084) 
* refactor: extract DML types (#2731)

* chore: fmt
 2021-11-11 12:34:07 +00:00
Marco Neumann c80088bf12 feat: HTTP write interface for router 2021-11-10 17:55:46 +01:00
Carol (Nichols || Goulding) 3dda694520
refactor: Extract a macro for matching errors in tests 2021-11-08 11:14:23 -05:00
Nga Tran abbfafcabd chore: merge main to branch 2021-11-08 09:28:29 -05:00
dependabot[bot] f6bbf7b617
chore(deps): bump libc from 0.2.106 to 0.2.107 
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.106 to 0.2.107.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.106...0.2.107)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-08 11:18:48 +00:00
dependabot[bot] 9ce86f4f8e
chore(deps): bump nom from 7.0.0 to 7.1.0 
Bumps [nom](https://github.com/Geal/nom) from 7.0.0 to 7.1.0.
- [Release notes](https://github.com/Geal/nom/releases)
- [Changelog](https://github.com/Geal/nom/blob/main/CHANGELOG.md)
- [Commits](https://github.com/Geal/nom/commits)

---
updated-dependencies:
- dependency-name: nom
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-08 11:06:18 +00:00
dependabot[bot] e4d943d79c
chore(deps): bump comfy-table from 4.1.1 to 5.0.0 
Bumps [comfy-table](https://github.com/nukesor/comfy-table) from 4.1.1 to 5.0.0.
- [Release notes](https://github.com/nukesor/comfy-table/releases)
- [Changelog](https://github.com/Nukesor/comfy-table/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nukesor/comfy-table/compare/v4.1.1...v5.0.0)

---
updated-dependencies:
- dependency-name: comfy-table
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-08 10:55:57 +00:00
dependabot[bot] af6694299b
chore(deps): bump handlebars from 4.1.3 to 4.1.4 
Bumps [handlebars](https://github.com/sunng87/handlebars-rust) from 4.1.3 to 4.1.4.
- [Release notes](https://github.com/sunng87/handlebars-rust/releases)
- [Changelog](https://github.com/sunng87/handlebars-rust/blob/master/CHANGELOG.md)
- [Commits](https://github.com/sunng87/handlebars-rust/compare/v4.1.3...v4.1.4)

---
updated-dependencies:
- dependency-name: handlebars
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-08 10:42:46 +00:00
dependabot[bot] 7214fa9d6a
chore(deps): bump serde_json from 1.0.68 to 1.0.69 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.68 to 1.0.69.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.68...v1.0.69)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-08 10:31:33 +00:00
kodiakhq[bot] 6e8517cb1f
Merge branch 'main' into dependabot/cargo/paste-1.0.6 2021-11-08 10:20:56 +00:00
dependabot[bot] 8724f82c5c
chore(deps): bump paste from 1.0.5 to 1.0.6 
Bumps [paste](https://github.com/dtolnay/paste) from 1.0.5 to 1.0.6.
- [Release notes](https://github.com/dtolnay/paste/releases)
- [Commits](https://github.com/dtolnay/paste/compare/1.0.5...1.0.6)

---
updated-dependencies:
- dependency-name: paste
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-08 10:16:07 +00:00
Marco Neumann 11f4a1dee8 feat: add connection management for router 2021-11-08 11:11:52 +01:00
Raphael Taylor-Davies 60f0deaf1e
feat: remove flatbuffer entry (#3045) 2021-11-05 20:19:24 +00:00
Raphael Taylor-Davies 88868e7496
feat: remove legacy write service from influxdb_iox_client (#3043) 
* feat: remove legacy write service from influxdb_iox_client

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-05 17:26:18 +00:00
kodiakhq[bot] 94016324da
Merge branch 'main' into ntran/min_max_string 2021-11-05 15:21:10 +00:00
Nga Tran 8dcd2ae9d4 chore: latest datafusion to get the boolean(null) and/or expr folding bug fix 2021-11-05 11:20:16 -04:00
Raphael Taylor-Davies 898567e221
feat: migrate server to DbWrite (#2724) (#3035) 
* feat: migrate server to DbWrite (#2724)

* chore: print perf log output

* fix: don't suppress CI status code

* chore: review feedback

* fix: don't error on empty line protocol write payloads

* fix: test

* fix: test
 2021-11-05 11:09:33 +00:00
Marco Neumann 92e6acaacd feat: implement sharder for new router 2021-11-05 09:49:33 +01:00
Nga Tran 89699cf0de chore: use latest DataFusion to fix the min/max(dictionary string) bug 2021-11-04 11:04:03 -04:00
Paul Dix bd1679a903 fix: revert to previous Cargo.lock and remove rand seeder 2021-11-04 09:56:58 -04:00
Paul Dix 7044b89453 feat: Refactor Data Generator 
This is a huge commit that refactors the data generator. It removes many of the previous features that didn't quite make sense. The goal of this refactor was to make the data generator capable of representing complex tagsets that have values dependent on each other. It also significantly optimizes things to use far less memory and generate data much faster. Follow on work will update the generation of line protocol to support spaces in tags and their keys, double quotes in strings, and add more examples and documentation.
 2021-11-04 09:56:58 -04:00
Raphael Taylor-Davies 07ba629e2b
feat: migrate write buffer producer to MutableBatch and pbdata (#2743) (#3021) 
* feat: migrate write buffer producer to MutableBatch and pbdata (#2743)

* fix: Kafka message content type header

* chore: fix doc
 2021-11-04 10:20:40 +00:00
Andrew Lamb 46213a5a28 chore: update dependencies 2021-11-03 17:00:20 -04:00
Raphael Taylor-Davies 405ed6db1b
feat: add mutable_batch_tests (#2724) (#3011) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-03 18:15:35 +00:00
Marco Neumann a91e572b43 feat: implement basic router server type 
This only implements the deployment service and nothing else yet.
 2021-11-03 16:34:51 +01:00
Raphael Taylor-Davies 08fcd87337
feat: use protobuf encoding in write buffer (#2724) (#3018) 2021-11-03 15:19:05 +00:00
Raphael Taylor-Davies e444fa4cb2
feat: pbdata encode (#2724) (#3009) 2021-11-02 18:31:53 +00:00
Andrew Lamb 1902c4f8a9
chore: Update DataFusion (#3012) 
* chore: Update DataFusion

* fix: restore Cargo.log

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-02 18:06:21 +00:00
Raphael Taylor-Davies f1a6468e7b
feat: migrate write buffer consumer to use DbWrite (#2724) (#3003) 
* feat: migrate write buffer consumer to use DbWrite (#2724)

* fix: doc

* chore: fmt

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-01 16:38:48 +00:00
dependabot[bot] bf4212e98d
chore(deps): bump pprof from 0.5.0 to 0.5.1 
Bumps [pprof](https://github.com/tikv/pprof-rs) from 0.5.0 to 0.5.1.
- [Release notes](https://github.com/tikv/pprof-rs/releases)
- [Commits](https://github.com/tikv/pprof-rs/commits)

---
updated-dependencies:
- dependency-name: pprof
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-01 11:48:31 +00:00
dependabot[bot] 1b377f0377
chore(deps): bump tokio-stream from 0.1.7 to 0.1.8 
Bumps [tokio-stream](https://github.com/tokio-rs/tokio) from 0.1.7 to 0.1.8.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-stream-0.1.7...tokio-stream-0.1.8)

---
updated-dependencies:
- dependency-name: tokio-stream
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-01 11:38:59 +00:00
dependabot[bot] c540b40f05
chore(deps): bump tokio from 1.12.0 to 1.13.0 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.12.0...tokio-1.13.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-01 11:21:59 +00:00
kodiakhq[bot] 9f04877556
Merge branch 'main' into dependabot/cargo/tokio-util-0.6.9 2021-11-01 10:26:00 +00:00
dependabot[bot] a1f978e280
chore(deps): bump tokio-util from 0.6.8 to 0.6.9 
Bumps [tokio-util](https://github.com/tokio-rs/tokio) from 0.6.8 to 0.6.9.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.6.8...tokio-util-0.6.9)

---
updated-dependencies:
- dependency-name: tokio-util
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-01 01:28:41 +00:00
dependabot[bot] 3ab81b7d7b
chore(deps): bump tracing-subscriber from 0.3.0 to 0.3.1 
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.3.0 to 0.3.1.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.0...tracing-subscriber-0.3.1)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-01 01:28:23 +00:00
Raphael Taylor-Davies 6ceab054ab
refactor: move DbWrite to mutable_batch (#2986) 
* refactor: move DbWrite to mutable_batch

* chore: fix doc

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-29 15:13:05 +00:00
kodiakhq[bot] fc25e313c6
Merge branch 'main' into cn/iox-obj-sto-refactors 2021-10-29 13:53:26 +00:00
Raphael Taylor-Davies 8a2410e161
feat: mutable batch write entry (#2724) (#2973) 
* feat: mutable batch write entry (#2724)

* chore: lint

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-28 20:15:28 +00:00
Carol (Nichols || Goulding) 95ba84edca
fix: Return UUID when a database is deleted 
So that it can be immediately used to restore elsewhere.
 2021-10-28 13:20:29 -04:00
Marco Neumann c465d0af8f refactor: abstract HTTP inteface for different run modes 
First iteration of a common interface for the different server run
modes. This splits the existing HTTP server into a part that can be used
by all run modes (e.g. health checks, metrics endpoint, profiling) and a
part that is specific to the the database run mode.

Note that the interface is not final and might require more iterations,
but we have to start somewhere.
 2021-10-28 11:16:41 +02:00
Raphael Taylor-Davies 3010ff4176
feat: write entry to MutableBatch (#2972) 
* feat: write entry to MutableBatch

* feat: error on empty TableBatch

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-27 14:05:22 +00:00
Marco Neumann 2833cefc12 test: remove Kafka from end2end tests 
Kafka is now sufficiently tested via the `write_buffer` crate. The
end2end tests can now use the in-memory mock implementation or -- if
servers can only be controlled via CLI -- the file-based implementation.
 2021-10-26 10:09:34 +02:00
Marco Neumann 9ceb5800c1 fix: adjust code to `tracing-subscriber` 0.3 2021-10-25 17:01:50 +02:00
dependabot[bot] b7186d0bec chore(deps): bump tracing-subscriber from 0.2.25 to 0.3.0 
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.2.25 to 0.3.0.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.2.25...tracing-subscriber-0.3.0)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-25 17:01:50 +02:00
Raphael Taylor-Davies 820e0d56bb
feat: support multiple tables in lp -> MutableBatch conversion (#2959) 2021-10-25 11:42:29 +00:00
dependabot[bot] bd707fb7ca
chore(deps): bump backtrace from 0.3.61 to 0.3.62 
Bumps [backtrace](https://github.com/rust-lang/backtrace-rs) from 0.3.61 to 0.3.62.
- [Release notes](https://github.com/rust-lang/backtrace-rs/releases)
- [Commits](https://github.com/rust-lang/backtrace-rs/compare/0.3.61...0.3.62)

---
updated-dependencies:
- dependency-name: backtrace
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-25 07:49:24 +00:00
dependabot[bot] 80d6ec6fe9
chore(deps): bump hyper from 0.14.13 to 0.14.14 
Bumps [hyper](https://github.com/hyperium/hyper) from 0.14.13 to 0.14.14.
- [Release notes](https://github.com/hyperium/hyper/releases)
- [Changelog](https://github.com/hyperium/hyper/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/hyper/compare/v0.14.13...v0.14.14)

---
updated-dependencies:
- dependency-name: hyper
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-25 07:39:20 +00:00
dependabot[bot] 2b832fcc5c
chore(deps): bump libc from 0.2.104 to 0.2.105 
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.104 to 0.2.105.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.104...0.2.105)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-25 07:28:31 +00:00
dependabot[bot] fae12cee54
chore(deps): bump tower from 0.4.9 to 0.4.10 
Bumps [tower](https://github.com/tower-rs/tower) from 0.4.9 to 0.4.10.
- [Release notes](https://github.com/tower-rs/tower/releases)
- [Commits](https://github.com/tower-rs/tower/compare/tower-0.4.9...tower-0.4.10)

---
updated-dependencies:
- dependency-name: tower
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-25 07:15:51 +00:00
dependabot[bot] 5bf2389486
chore(deps): bump http-body from 0.4.3 to 0.4.4 
Bumps [http-body](https://github.com/hyperium/http-body) from 0.4.3 to 0.4.4.
- [Release notes](https://github.com/hyperium/http-body/releases)
- [Changelog](https://github.com/hyperium/http-body/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/http-body/commits)

---
updated-dependencies:
- dependency-name: http-body
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-25 01:40:25 +00:00
Raphael Taylor-Davies d2b41c5a13
feat: write pbdata to MutableBatch (#2724) (#2927) 
* feat: write pbdata to MutableBatch (#2724)

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-21 14:32:35 +00:00
Raphael Taylor-Davies 297f995a44
feat: partition MutableBatch rows (#2926) 
* feat: partition MutableBatch rows

* chore: review feedback

* chore: review feedback

* chore: fmt
 2021-10-21 12:30:13 +00:00
Raphael Taylor-Davies 272e43492a
feat: mutable batch filtering (#2920) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-21 12:06:26 +00:00
Raphael Taylor-Davies f64e749eb9
feat: write line protocol to MutableBatch (#2914) 
* feat: write line protocol to MutableBatch

* chore: fix lint

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-21 08:12:54 +00:00
Andrew Lamb 9974a5364c
chore(security): Replace prettytable with comfy-table (#2905) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-20 10:44:36 +00:00
Raphael Taylor-Davies ce0127a1f7
feat: MutableBatch write API (#2090) (#2724) (#2882) 
* feat: MutableBatch write API (#2090) (#2724)

* chore: fix lint

* fix: handle dictionaries with unused mappings

* chore: review feedback

* chore: further review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-20 08:44:14 +00:00
Andrew Lamb a82dc6f5f0
chore: Update datafusion + arrow (#2903) 
* chore: Update datafusion to latest, arrow to 6.0.0

* fix: Update tests

* fix: bubble internal error

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-19 17:14:08 +00:00
Marco Neumann 6ec0bd5bab feat: file-based write write_buffer 
Closes #2849.
 2021-10-19 15:26:43 +02:00
dependabot[bot] c466f16803
chore(deps): bump reqwest from 0.11.5 to 0.11.6 
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.5 to 0.11.6.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.5...v0.11.6)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-19 13:05:58 +00:00
kodiakhq[bot] 0be89715e2
Merge branch 'main' into dependabot/cargo/rdkafka-0.27.0 2021-10-19 12:55:21 +00:00
Marco Neumann 9c9a31a402 chore: remove unused `iox_data_generator` => `tracing-futures` dep 2021-10-19 14:46:26 +02:00
Marco Neumann 93987ff107 chore: remove unused `iox_data_generator` => `packers` dep 2021-10-19 14:46:26 +02:00
Marco Neumann 7dafb0ff74 chore: remove unused `influxdb_iox` => `synchronized-writer` dep 2021-10-19 14:46:26 +02:00
Marco Neumann da85f04b3f chore: remove unused `schema` => `arrow_util` dep 2021-10-19 14:46:25 +02:00
Marco Neumann 99c019bf33 chore: remove unused `server_benchmarks` => `packers` dep 2021-10-19 14:45:56 +02:00
Marco Neumann a364bdfb5f chore: remove unused `query_tests` => `chrono` dep 2021-10-19 14:45:56 +02:00
Marco Neumann d8f35d8ee9 chore: remove unused `parquet_file` => `chrono` dep 2021-10-19 14:45:56 +02:00
Marco Neumann 586d6b6f28 chore: remove unused `parquet_catalog` => `chrono` dep 2021-10-19 14:45:56 +02:00
Marco Neumann 2079fb689e chore: remove unused `generated_types` => `thiserror` dep 2021-10-19 14:45:56 +02:00
dependabot[bot] 32e18b6436
chore(deps): bump rdkafka from 0.26.0 to 0.27.0 
Bumps [rdkafka](https://github.com/fede1024/rust-rdkafka) from 0.26.0 to 0.27.0.
- [Release notes](https://github.com/fede1024/rust-rdkafka/releases)
- [Changelog](https://github.com/fede1024/rust-rdkafka/blob/master/changelog.md)
- [Commits](https://github.com/fede1024/rust-rdkafka/compare/v0.26.0...v0.27.0)

---
updated-dependencies:
- dependency-name: rdkafka
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-19 11:49:34 +00:00