Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
7,796 Commits
218 Branches
453 Tags
340 MiB
Commit Graph

1172 Commits (83aea40c9af2a4f77d257875ae752dd292bad14c)

Author SHA1 Message Date
dependabot[bot] 72a2a68eea
chore(deps): Bump sqlx from 0.5.11 to 0.5.12 (#4333) 
* chore(deps): Bump sqlx from 0.5.11 to 0.5.12

Bumps [sqlx](https://github.com/launchbadge/sqlx) from 0.5.11 to 0.5.12.
- [Release notes](https://github.com/launchbadge/sqlx/releases)
- [Changelog](https://github.com/launchbadge/sqlx/blob/master/CHANGELOG.md)
- [Commits](https://github.com/launchbadge/sqlx/compare/v0.5.11...v0.5.12)

---
updated-dependencies:
- dependency-name: sqlx
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-04-15 15:03:08 +00:00
dependabot[bot] 6ca40334ea
chore(deps): Bump toml from 0.5.8 to 0.5.9 (#4332) 
Bumps [toml](https://github.com/alexcrichton/toml-rs) from 0.5.8 to 0.5.9.
- [Release notes](https://github.com/alexcrichton/toml-rs/releases)
- [Commits](https://github.com/alexcrichton/toml-rs/compare/0.5.8...0.5.9)

---
updated-dependencies:
- dependency-name: toml
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-04-15 14:55:18 +00:00
dependabot[bot] f76cbb31ac
chore(deps): Bump tracing from 0.1.33 to 0.1.34 (#4330) 
Bumps [tracing](https://github.com/tokio-rs/tracing) from 0.1.33 to 0.1.34.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.33...tracing-0.1.34)

---
updated-dependencies:
- dependency-name: tracing
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-04-15 14:44:07 +00:00
kodiakhq[bot] 9df51e634e
Merge branch 'main' into crepererum/improve_ingester_grpc_testing 2022-04-14 21:13:18 +00:00
Paul Dix 99cbb28a89
feat: add initial catalog service to router (#4316) 
Create new crate for iox_catalog_service.
Add rpc to return parquet_file records by partition id.
Add CatalogService to router2.

The catalog service will be added to over time to provide access to the catalog over gRPC.
 2022-04-14 17:39:18 +00:00
Andrew Lamb f938d4d6e3
chore: Update datafusion (and point back to apache) (#4322) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-04-14 15:51:06 +00:00
Marco Neumann de1241db85 test: mock gRPC ingester response for querier 
Add infrastructure to test how the querier processes ingester gRPC
responses w/o performing a full query or end2end test.
 2022-04-14 15:00:35 +02:00
CircleCI[bot] 0a253abd57 chore: Run cargo hakari tasks 2022-04-14 10:42:13 +00:00
Edd Robinson e548ba7b17 test: add property tests for byte trimmer 2022-04-14 11:35:19 +01:00
Nga Tran e97cec3f4d
chore: use Nga's DF branch with debug information for compactor (#4314) 2022-04-13 22:20:16 +00:00
Andrew Lamb 85f3e696e8
refactor: Use declarative steps to reduce duplication in end to end testing (#4301) 
* refactor: Use declarative steps to reduce duplication in end to end testing,

* fix: improve whitespace formatting

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-04-13 16:24:57 +00:00
kodiakhq[bot] 58763bd575
Merge branch 'main' into crepererum/kube_071 2022-04-13 16:05:20 +00:00
dependabot[bot] cb16348c5b chore(deps): Bump kube-runtime from 0.70.0 to 0.71.0 
Bumps [kube-runtime](https://github.com/kube-rs/kube-rs) from 0.70.0 to 0.71.0.
- [Release notes](https://github.com/kube-rs/kube-rs/releases)
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.70.0...0.71.0)

---
updated-dependencies:
- dependency-name: kube-runtime
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-04-13 17:48:08 +02:00
dependabot[bot] 6e24d075a9 chore(deps): Bump kube from 0.70.0 to 0.71.0 
Bumps [kube](https://github.com/kube-rs/kube-rs) from 0.70.0 to 0.71.0.
- [Release notes](https://github.com/kube-rs/kube-rs/releases)
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.70.0...0.71.0)

---
updated-dependencies:
- dependency-name: kube
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-04-13 17:47:21 +02:00
dependabot[bot] e7f89f336a chore(deps): Bump kube-derive from 0.70.0 to 0.71.0 
Bumps [kube-derive](https://github.com/kube-rs/kube-rs) from 0.70.0 to 0.71.0.
- [Release notes](https://github.com/kube-rs/kube-rs/releases)
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.70.0...0.71.0)

---
updated-dependencies:
- dependency-name: kube-derive
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-04-13 17:46:17 +02:00
Marco Neumann ab31f78172 test: improve server restarts 
- Ensure that the restarted server uses the same log file as the old
  one. This is important for debugging because we print out server logs
  for failed tests.
- Try to be polite when shutting down or restarting servers (i.e. try to
  use SIGTERM).
 2022-04-13 17:33:09 +02:00
dependabot[bot] cb8a80dc47
chore(deps): Bump tracing-core from 0.1.24 to 0.1.25 (#4293) 
Bumps [tracing-core](https://github.com/tokio-rs/tracing) from 0.1.24 to 0.1.25.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-core-0.1.24...tracing-core-0.1.25)

---
updated-dependencies:
- dependency-name: tracing-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-04-13 10:39:10 +00:00
dependabot[bot] ec4f9e318e
chore(deps): Bump libc from 0.2.122 to 0.2.123 (#4291) 
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.122 to 0.2.123.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.122...0.2.123)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-04-13 10:27:53 +00:00
Dom Dwyer a4df93e604 chore: workspace-hack update 2022-04-12 13:05:08 +01:00
Dom Dwyer f08c8373aa feat: tokio-console support 
This commit adds an optional feature (disabled by default) to the IOx
server binary named "tokio_console". When enabled, it adds support for
the tokio-console to IOx:

    https://github.com/tokio-rs/console

Enabling this feature drops the compile-time log filter to TRACE, and
enables the necessary dependencies to support the instrumentation needed
for the console to function.

Unfortunately, this feature uses tonic 0.7 (latest) while we use tonic
0.6, so we wind up with two tonic versions being compiled in when this
feature is enabled.
 2022-04-12 13:02:31 +01:00
Andrew Lamb d8de38cdb9
feat: MVP include un-persisted results from the ingester in query results (#4255) 
* feat: Return not-yet-persisted data in query results

* fix: comments from code review

* fix: update for logical merge conflict

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-04-12 11:03:19 +00:00
dependabot[bot] b7cf4f1849
chore(deps): Bump flate2 from 1.0.22 to 1.0.23 (#4276) 
Bumps [flate2](https://github.com/rust-lang/flate2-rs) from 1.0.22 to 1.0.23.
- [Release notes](https://github.com/rust-lang/flate2-rs/releases)
- [Commits](https://github.com/rust-lang/flate2-rs/compare/1.0.22...1.0.23)

---
updated-dependencies:
- dependency-name: flate2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-04-12 10:45:18 +00:00
dependabot[bot] 694ffd2238
chore(deps): Bump httparse from 1.6.0 to 1.7.0 (#4277) 
Bumps [httparse](https://github.com/seanmonstar/httparse) from 1.6.0 to 1.7.0.
- [Release notes](https://github.com/seanmonstar/httparse/releases)
- [Commits](https://github.com/seanmonstar/httparse/compare/v1.6.0...v1.7.0)

---
updated-dependencies:
- dependency-name: httparse
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-04-12 09:37:08 +00:00
Marco Neumann 380cd9bbff
refactor: use a single flight client implementation (#4273) 
"end-user -> querier" and "querier -> ingester" should use a single
Flight client implementation. The difference is just the request and
response metadata.

This changes our default Flight client to use protobuf instead of JSON
for the ticket format.
 2022-04-12 09:08:25 +00:00
Dom d4a8fe30c2
Merge branch 'main' into dom/ingester-sink-impl 2022-04-11 11:47:49 +01:00
dependabot[bot] 91991eaecf
chore(deps): Bump tracing from 0.1.32 to 0.1.33 (#4268) 
Bumps [tracing](https://github.com/tokio-rs/tracing) from 0.1.32 to 0.1.33.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.32...tracing-0.1.33)

---
updated-dependencies:
- dependency-name: tracing
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-04-11 08:35:55 +00:00
dependabot[bot] 3689e17704
chore(deps): Bump croaring from 0.5.1 to 0.5.2 (#4267) 
* chore(deps): Bump croaring from 0.5.1 to 0.5.2

Bumps [croaring](https://github.com/saulius/croaring-rs) from 0.5.1 to 0.5.2.
- [Release notes](https://github.com/saulius/croaring-rs/releases)
- [Commits](https://github.com/saulius/croaring-rs/compare/0.5.1...0.5.2)

---
updated-dependencies:
- dependency-name: croaring
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-04-11 08:22:43 +00:00
dependabot[bot] 69e42bf478
chore(deps): Bump tracing-subscriber from 0.3.10 to 0.3.11 (#4266) 
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.3.10 to 0.3.11.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.10...tracing-subscriber-0.3.11)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-04-11 08:12:30 +00:00
Andrew Lamb 2cdd1951d9
fix: fix pprof (#4261) 2022-04-08 17:46:01 +00:00
Andrew Lamb 34e65c23fa
fix: Update for signature change (#4252) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-04-08 11:21:07 +00:00
Dom Dwyer dce939c580 refactor: use SequencedStreamHandler 
Removes the old stream_in_sequenced_entries() write buffer handler,
replacing it with the SequencedStreamHandler introduced in #4203.

This change will affect the metrics emitted by an ingester as outlined
in #4243.
 2022-04-08 11:28:39 +01:00
dependabot[bot] ba3eb409d9
chore(deps): Bump libc from 0.2.121 to 0.2.122 (#4248) 
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.121 to 0.2.122.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.121...0.2.122)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-04-07 19:40:53 +00:00
dependabot[bot] db162980aa
chore(deps): Bump syn from 1.0.90 to 1.0.91 (#4241) 
Bumps [syn](https://github.com/dtolnay/syn) from 1.0.90 to 1.0.91.
- [Release notes](https://github.com/dtolnay/syn/releases)
- [Commits](https://github.com/dtolnay/syn/compare/1.0.90...1.0.91)

---
updated-dependencies:
- dependency-name: syn
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-04-07 19:33:01 +00:00
Andrew Lamb 1de280e60f
chore: Update datafusion (#4247) 2022-04-06 16:22:44 +00:00
Andrew Lamb c244b03281
feat: Add `SequencerProgress` reporting to ingester (#4238) 
* feat: Add `SequencerProgress` reporting to ingester

* refactor: Use KafkaPartition in write_summary

* fix: Update docstrings

* refactor: Change ingester to use KafkaPartition everywhere

* refactor: add SequencerProgress::combine

* refactor: return new SequencerProgress rather than updating

* fix: distinguish between yes/no/unknown in WriteSummary

* docs: Update data_types2/src/lib.rs

Co-authored-by: Paul Dix <paul@pauldix.net>

Co-authored-by: Paul Dix <paul@pauldix.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-04-06 15:13:21 +00:00
dependabot[bot] 438e739344
chore(deps): Bump parquet from 11.0.0 to 11.1.0 (#4240) 
* chore(deps): Bump parquet from 11.0.0 to 11.1.0

Bumps [parquet](https://github.com/apache/arrow-rs) from 11.0.0 to 11.1.0.
- [Release notes](https://github.com/apache/arrow-rs/releases)
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/apache/arrow-rs/compare/11.0.0...11.1.0)

---
updated-dependencies:
- dependency-name: parquet
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix: Update tests

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
 2022-04-06 14:51:01 +00:00
dependabot[bot] 6607c5e179
chore(deps): Bump arrow-flight from 11.0.0 to 11.1.0 (#4242) 
Bumps [arrow-flight](https://github.com/apache/arrow-rs) from 11.0.0 to 11.1.0.
- [Release notes](https://github.com/apache/arrow-rs/releases)
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/apache/arrow-rs/compare/11.0.0...11.1.0)

---
updated-dependencies:
- dependency-name: arrow-flight
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-04-06 10:04:23 +00:00
dependabot[bot] bea49e7611
chore(deps): Bump arrow from 11.0.0 to 11.1.0 (#4234) 
Bumps [arrow](https://github.com/apache/arrow-rs) from 11.0.0 to 11.1.0.
- [Release notes](https://github.com/apache/arrow-rs/releases)
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/apache/arrow-rs/compare/11.0.0...11.1.0)

---
updated-dependencies:
- dependency-name: arrow
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-04-05 16:54:28 +00:00
Dom Dwyer 083ff1f8e3 refactor: ingest stream handler 
Refactors the stream_in_sequenced_entries() into a new impl in the
SequencedStreamHandler type, decoupling the reading / decoding of ops
from Kafka (and associated error handling) from the "what happens to
those ops" concern to ease testing, encapsulate the specifics of "how to
get an op" and improve flexibility.

This is intended to provide robust error handling within what is
reasonably possible (unexpected errors are always unexpected!) while
retaining the existing metrics and functionality. I've also separated
out code that exists in the current impl specifically to drive tests
from the prod code path, instead driving those behaviours through mocks.

As of this commit, the handler is not used - this commit simply adds the
new impl.
 2022-04-05 11:31:54 +01:00
Dom Dwyer 850308cdc9 feat(tests): future timeout helper 
Adds a timeout test helper for futures - this lets us easily write tests
that await on futures for a bounded duration of time.

Optional feature to avoid dragging tokio into existing consumers of the
test_helpers crate that don't need it.
 2022-04-05 11:30:47 +01:00
Andrew Lamb 5d66cd0a81
feat: Add WriteSummary serialization and deserialization to protobuf (#4232) 
* feat: Add WriteSummary serialization and deserialization to protobuf

* fix: clippy

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-04-05 09:57:32 +00:00
Andrew Lamb 756116b497
chore: update datafusion (#4229) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-04-05 09:36:36 +00:00
dependabot[bot] 276449ee09
chore(deps): Bump pbjson from 0.2.3 to 0.3.0 (#4215) 
Bumps [pbjson](https://github.com/influxdata/pbjson) from 0.2.3 to 0.3.0.
- [Release notes](https://github.com/influxdata/pbjson/releases)
- [Commits](https://github.com/influxdata/pbjson/compare/0.2.3...0.3.0)

---
updated-dependencies:
- dependency-name: pbjson
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-04-04 12:05:46 +00:00
dependabot[bot] 26f6a1721f
chore(deps): Bump tracing-core from 0.1.23 to 0.1.24 (#4217) 
Bumps [tracing-core](https://github.com/tokio-rs/tracing) from 0.1.23 to 0.1.24.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-core-0.1.23...tracing-core-0.1.24)

---
updated-dependencies:
- dependency-name: tracing-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-04-04 11:17:24 +00:00
dependabot[bot] d19b944ba5
chore(deps): Bump tracing-subscriber from 0.3.9 to 0.3.10 (#4222) 
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.3.9 to 0.3.10.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.9...tracing-subscriber-0.3.10)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-04-04 11:09:29 +00:00
dependabot[bot] 4c052be568
chore(deps): Bump sqlparser from 0.15.0 to 0.16.0 (#4219) 
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs) from 0.15.0 to 0.16.0.
- [Release notes](https://github.com/sqlparser-rs/sqlparser-rs/releases)
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.15.0...v0.16.0)

---
updated-dependencies:
- dependency-name: sqlparser
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-04-04 11:01:14 +00:00
dependabot[bot] dc9632114c
chore(deps): Bump pretty_assertions from 1.2.0 to 1.2.1 (#4213) 
Bumps [pretty_assertions](https://github.com/colin-kiegel/rust-pretty-assertions) from 1.2.0 to 1.2.1.
- [Release notes](https://github.com/colin-kiegel/rust-pretty-assertions/releases)
- [Changelog](https://github.com/colin-kiegel/rust-pretty-assertions/blob/main/CHANGELOG.md)
- [Commits](https://github.com/colin-kiegel/rust-pretty-assertions/compare/v1.2.0...v1.2.1)

---
updated-dependencies:
- dependency-name: pretty_assertions
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-04-04 10:53:31 +00:00
dependabot[bot] 36dd6f26a3
chore(deps): Bump pbjson-build from 0.2.3 to 0.3.0 (#4220) 
Bumps [pbjson-build](https://github.com/influxdata/pbjson) from 0.2.3 to 0.3.0.
- [Release notes](https://github.com/influxdata/pbjson/releases)
- [Commits](https://github.com/influxdata/pbjson/compare/0.2.3...0.3.0)

---
updated-dependencies:
- dependency-name: pbjson-build
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-04-04 10:45:31 +00:00
dependabot[bot] 1edd89eb67
chore(deps): Bump clap from 3.1.7 to 3.1.8 (#4221) 
Bumps [clap](https://github.com/clap-rs/clap) from 3.1.7 to 3.1.8.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v3.1.7...v3.1.8)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-04-04 10:36:58 +00:00
Andrew Lamb edda409b19
refactor: Extract `ioxd_test`, `ioxd_compactor`, `ioxd_ingester`; remove `ioxd` (#4210) 
* refactor: Extract test, compactor, ingester, and test

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-04-03 10:42:22 +00:00
Andrew Lamb 833c10c083
feat: return write_token from HTTP writes to router2 (#4202) 
* feat: return write_token from HTTP writes to router2

* fix: Update router2/src/dml_handlers/instrumentation.rs

Co-authored-by: Dom <dom@itsallbroken.com>

* refactor: Use WriteSummary::default more vigorously

* fix: fix typo and add links to follow on issues

Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-04-02 10:34:51 +00:00
dependabot[bot] e8b0655ac8
chore(deps): Bump clap from 3.1.6 to 3.1.7 (#4199) 
Bumps [clap](https://github.com/clap-rs/clap) from 3.1.6 to 3.1.7.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v3.1.6...v3.1.7)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-04-01 09:49:30 +00:00
Andrew Lamb d37af1a7f5
fix: include git sha (again) in release build (#4193) 
* fix: error if git-sha can not be found

* refactor: move main to influxdb_iox

* fix: fmt
 2022-03-31 19:14:21 +00:00
Andrew Lamb 532d227d11
refactor: extract router2 into ioxd_router2 (#4183) 
* refactor: extract router2 from ioxd

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2022-03-31 17:39:05 +00:00
Andrew Lamb 367e926d35
refactor: extract querier into ioxd_querier (#4182) 
* refactor: extract querier into ioxd_querier

* fix: dep
 2022-03-31 16:03:31 +00:00
Andrew Lamb a1df864283
feat: Support 'SHOW NAMESPACES' in sql repl (#4164) 
* feat: Support `SHOW NAMESPACES` in sql repl

* feat: add basic support to clients

* fix: add get_namespaces service test

* fix: proper error handling

* test: end to end test for namespace client

* refactor: Use QuerierDatabase rather than Catalog

* refactor: remove unused function
 2022-03-31 12:57:33 +00:00
dependabot[bot] fc4635a334
chore(deps): Bump lock_api from 0.4.6 to 0.4.7 (#4186) 
Bumps [lock_api](https://github.com/Amanieu/parking_lot) from 0.4.6 to 0.4.7.
- [Release notes](https://github.com/Amanieu/parking_lot/releases)
- [Changelog](https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Amanieu/parking_lot/compare/lock_api-0.4.6...lock_api-0.4.7)

---
updated-dependencies:
- dependency-name: lock_api
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-31 07:45:03 +00:00
Andrew Lamb 22b24bdab3
chore: Update datafusion again (#4148) 
* chore: update datafusoon

* refactor: Update for DataFusion API changes

* chore: TEMP TEMP change df to local copy

* chore: Update to datafusion again

* fix: Update Cargo.lock

* fix: logical conflict
 2022-03-30 16:51:48 +00:00
Andrew Lamb f2a3dd58b2
refactor: rename `influxdb_ioxd` to `ioxd` (#4162) 
* refactor: rename influxdb_ioxd to ioxd

* fix: fmt

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-30 13:31:27 +00:00
dependabot[bot] 9950bcee27
chore(deps): Bump indexmap from 1.8.0 to 1.8.1 (#4166) 
Bumps [indexmap](https://github.com/bluss/indexmap) from 1.8.0 to 1.8.1.
- [Release notes](https://github.com/bluss/indexmap/releases)
- [Changelog](https://github.com/bluss/indexmap/blob/master/RELEASES.md)
- [Commits](https://github.com/bluss/indexmap/compare/1.8.0...1.8.1)

---
updated-dependencies:
- dependency-name: indexmap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-30 08:49:07 +00:00
Andrew Lamb 4ca52e5ae0
refactor: Extract common, OG database and router out of influxdb_ioxd (#4149) 
* refactor: Extract common, OG database and router out of influxdb_ioxd

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2022-03-29 13:07:19 +00:00
Marco Neumann decd018a6a
refactor: remove querier sync loop (#4146) 
Namespaces are now created on demand and contain their full schema.
Tombstones/chunks are created on demand during the query.

Closes #4123.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-29 11:47:23 +00:00
dependabot[bot] e154d77bf5
chore(deps): Bump getrandom from 0.2.5 to 0.2.6 (#4155) 
Bumps [getrandom](https://github.com/rust-random/getrandom) from 0.2.5 to 0.2.6.
- [Release notes](https://github.com/rust-random/getrandom/releases)
- [Changelog](https://github.com/rust-random/getrandom/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-random/getrandom/compare/v0.2.5...v0.2.6)

---
updated-dependencies:
- dependency-name: getrandom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-29 08:53:34 +00:00
dependabot[bot] 17af5fcbd1
chore(deps): Bump tokio-util from 0.7.0 to 0.7.1 (#4154) 
* chore(deps): Bump tokio-util from 0.7.0 to 0.7.1

Bumps [tokio-util](https://github.com/tokio-rs/tokio) from 0.7.0 to 0.7.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.7.0...tokio-util-0.7.1)

---
updated-dependencies:
- dependency-name: tokio-util
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-29 08:39:02 +00:00
dependabot[bot] a83ad1ce2b
chore(deps): Bump syn from 1.0.89 to 1.0.90 (#4150) 
Bumps [syn](https://github.com/dtolnay/syn) from 1.0.89 to 1.0.90.
- [Release notes](https://github.com/dtolnay/syn/releases)
- [Commits](https://github.com/dtolnay/syn/compare/1.0.89...1.0.90)

---
updated-dependencies:
- dependency-name: syn
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-29 08:28:30 +00:00
Nga Tran 80b7e9cce1
feat: delete fully processed tombstones & integration tests for find_and_compact (#4116) 
* feat: remove fully processed tombstones

* test: first few tests

* fix: delete SQL

* fix: test how IN (...) works in PG

* fix: test how IN (?) works in PG

* fix: test how IN (?) works in PG

* fix: dynamically add  IN (?, ?, ...)

* fix: dynamically add  IN (?, ?, ...) & its dynamic values

* fix: add argument directly in the SQL

* test: more tests for catalog read and update functions

* chore: move a subfunction to make it easier to read)

* test: first test for find_can_compact but disabled due to bug

* test: integration tests and a bug fix for find_and_compact

* chore: cleanup

* refactor: address review comments

* fix: put 2 delete processed  tombstones and tombstones in a transaction
 2022-03-28 18:35:54 +00:00
Marco Neumann fb186c6733
refactor: `QueryDatabaseProvider::db` should be async (#4143) 
This is required to fetch querier namespaces on demand.

Ref #4123.
 2022-03-28 11:18:20 +00:00
dependabot[bot] 4f9515ffba
chore(deps): Bump async-trait from 0.1.52 to 0.1.53 (#4141) 
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.52 to 0.1.53.
- [Release notes](https://github.com/dtolnay/async-trait/releases)
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.52...0.1.53)

---
updated-dependencies:
- dependency-name: async-trait
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-28 08:55:24 +00:00
dependabot[bot] e5bbc74f7a
chore(deps): Bump paste from 1.0.6 to 1.0.7 (#4140) 
Bumps [paste](https://github.com/dtolnay/paste) from 1.0.6 to 1.0.7.
- [Release notes](https://github.com/dtolnay/paste/releases)
- [Commits](https://github.com/dtolnay/paste/compare/1.0.6...1.0.7)

---
updated-dependencies:
- dependency-name: paste
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-28 08:44:01 +00:00
Andrew Lamb 58c630d709
chore: Update datafusion (#4133) 
* chore: Update datafusion

* fix: typo

* fix: Update explain plan output

* fix: update Cargo.locl

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-25 15:08:39 +00:00
Andrew Lamb 5c69a3f43b
chore: Update deps: datafusion, arrow/arrow-flight/parquet to 11, zstd to 0.11 (#4119) 
* chore: update datafusion

* chore(deps): Bump arrow from 10.0.0 to 11.0.0

Bumps [arrow](https://github.com/apache/arrow-rs) from 10.0.0 to 11.0.0.
- [Release notes](https://github.com/apache/arrow-rs/releases)
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/apache/arrow-rs/compare/10.0.0...11.0.0)

---
updated-dependencies:
- dependency-name: arrow
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore(deps): Bump arrow-flight from 10.0.0 to 11.0.0

Bumps [arrow-flight](https://github.com/apache/arrow-rs) from 10.0.0 to 11.0.0.
- [Release notes](https://github.com/apache/arrow-rs/releases)
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/apache/arrow-rs/compare/10.0.0...11.0.0)

---
updated-dependencies:
- dependency-name: arrow-flight
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: update parquet to 11.0.0

* fix: error on create schema, test for same

* fix: upgrade zstd

* chore: Run cargo hakari tasks

* fix: fix logical merge conflict

* fix: hakari

* fix: hakari

* fix: update newly introduced dep

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-24 15:27:36 +00:00
Marco Neumann 8ca5c337b2
refactor: port more query tests to NG, some code clean up (#4125) 
* refactor: inline function that is used once

* refactor: generalize multi-chunk creation for NG

* refactor: `TwoMeasurementsManyFieldsTwoChunks` is OG-specific

* refactor: generalize `OneMeasurementTwoChunksDifferentTagSet`

* refactor: port `OneMeasurementFourChunksWithDuplicates` to NG

* refactor: `TwoMeasurementsManyFieldsLifecycle` is OG-specific

* refactor: simplify NG chunk generation

* refactor: port `ThreeDeleteThreeChunks` to NG

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-24 15:07:09 +00:00
Andrew Lamb fb75ef7b82
test: add end to end test for all in one mode, restructure fixture (#4114) 
* test: add end to end test for all in one mode, restructure fixture

* docs: fix typos and clarify schema requrements
 2022-03-24 12:53:25 +00:00
Andrew Lamb 29b89aaca7
refactor: extract influxrpc, flight and testing gRPC out of influxdb_ioxd (#4106) 
* refactor: extract grpc service implementations out of influxdb_ioxd

* chore: Run cargo hakari tasks

* refactor: rename server_common to service_common

* refactor: rename server_grpc_influxrpc to service_grpc_influxrpc

* refactor: rename server_grpc_flight to service_grpc_flight

* refactor: rename server_grpc_testing to service_grpc_testing

* fix: Cargo.toml

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2022-03-23 20:14:45 +00:00
Marco Neumann 51da6dd7fa
feat: store sort key in NG metadata (#4110) 
The sort key is optional and currently only produced by `iox_tests`.
Writing it within the ingester/compactor is tracked by #3968. The sort
key is read by the querier (and this will be verified by the query tests
and is required to merge #4103).

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-23 18:24:46 +00:00
dependabot[bot] 8ee9b793f5
chore(deps): Bump hyper from 0.14.17 to 0.14.18 (#4109) 
Bumps [hyper](https://github.com/hyperium/hyper) from 0.14.17 to 0.14.18.
- [Release notes](https://github.com/hyperium/hyper/releases)
- [Changelog](https://github.com/hyperium/hyper/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/hyper/compare/v0.14.17...v0.14.18)

---
updated-dependencies:
- dependency-name: hyper
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-23 13:12:43 +00:00
dependabot[bot] 36071e2d12
chore(deps): Bump log from 0.4.14 to 0.4.16 (#4108) 
Bumps [log](https://github.com/rust-lang/log) from 0.4.14 to 0.4.16.
- [Release notes](https://github.com/rust-lang/log/releases)
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/log/commits)

---
updated-dependencies:
- dependency-name: log
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-23 13:01:30 +00:00
Marco Neumann 89206e013c
test: run SOME query tests for querier (#4098) 
This includes some type changes to dispatch between OG and NG and allows
some tests to be run against the NG querier. This only contains parquet
files though, so it's somewhat a limited scope.

For #3934.
 2022-03-22 17:39:19 +00:00
Andrew Lamb b83b000590
chore: Update datafusion (#4071) 
* chore: update to datafusion 5936edc2a94d5fb20702a41eab2b80695961b9dc

* chore: Update apis to match datafusion changes
 2022-03-22 13:17:41 +00:00
Marco Neumann 55643945a1
refactor: `querier` w/o `db` (#4063) 
* feat: `TombstoneRepo::list_by_table`

* feat: `ParquetFileRepo::list_by_table_not_to_delete`

* refactor: `querier` w/o `db`

Get the `querier` to work w/o relying on `db`. A few notes:

- Testing is kinda shallow, we really need to get `query_tests` working
  w/ `querier` (see #3934).
- We still run a sync loop for namespaces, tables and schemas. This will
  be a replaced by "update namespace incl. tables and schemas on demand".
  Note however that we cannot fetch single tables and schemas on demand
  at the moment, because DataFusion doesn't implement async schema
  inspection (only `scan` / "give me all the chunks" is async). I think
  that's OK for now and we can address this later.
- There is NO cache for parquet files and tombstones at the moment. For
  correctness, they need to be fetched in a single transaction (or we
  need a kinda tricky sequence number / logical clock tracking) and I am
  not sure yet how this makes sense when we have the ingester data wired
  up and predicates pushed down to the catalog (see next point). So
  let's measure first and then decide on a caching strategy for this.
- Predicates are currently NOT pushed down to the catalog. I'll need to
  figure out how to extract time range from generic DataFusion
  expressions to make that work (it's easier for InfluxRPC queries, but
  they are not tested at the moment, see first point).

Sorry that this commit is kinda huge. I initially planned to only
migrate the chunks away from `db` and leave the tables and schemas for a
follow-up PR, but the DataFusion trait structure (chunks are bound to
their tables) makes this kinda pointless.

Closes #3974.

* docs: explain what we're doing

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

* docs: mention tracking issues

* docs: explain what we're doing

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
 2022-03-21 16:58:00 +00:00
dependabot[bot] a23efce408
chore(deps): Bump kube-derive from 0.69.1 to 0.70.0 (#4073) 
* chore(deps): Bump kube-derive from 0.69.1 to 0.70.0

Bumps [kube-derive](https://github.com/kube-rs/kube-rs) from 0.69.1 to 0.70.0.
- [Release notes](https://github.com/kube-rs/kube-rs/releases)
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.69.1...0.70.0)

---
updated-dependencies:
- dependency-name: kube-derive
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore(deps): Bump kube-runtime from 0.69.1 to 0.70.0

Bumps [kube-runtime](https://github.com/kube-rs/kube-rs) from 0.69.1 to 0.70.0.
- [Release notes](https://github.com/kube-rs/kube-rs/releases)
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.69.1...0.70.0)

---
updated-dependencies:
- dependency-name: kube-runtime
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: upgrade kube to version 0.70

* chore: hakari

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
 2022-03-21 15:32:45 +00:00
Carol (Nichols || Goulding) 933dc69ecf
feat: For each compacted data set, persist new parquet file to object store (#4058) 
* feat: Rearrange skeleton functions for split/persist/catalog update

* feat: Persist compacted files to object storage

Fixes #3951.

* docs: Add comment about batches' schemas
 2022-03-21 14:16:03 +00:00
dependabot[bot] cd36229e27
chore(deps): Bump synchronized-writer from 1.1.10 to 1.1.11 (#4075) 
Bumps [synchronized-writer](https://github.com/magiclen/synchronized-writer) from 1.1.10 to 1.1.11.
- [Release notes](https://github.com/magiclen/synchronized-writer/releases)
- [Commits](https://github.com/magiclen/synchronized-writer/compare/v1.1.10...v1.1.11)

---
updated-dependencies:
- dependency-name: synchronized-writer
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-21 10:53:11 +00:00
dependabot[bot] 66cf34c2e2
chore(deps): Bump tokio-rustls from 0.23.2 to 0.23.3 (#4074) 
Bumps [tokio-rustls](https://github.com/tokio-rs/tls) from 0.23.2 to 0.23.3.
- [Release notes](https://github.com/tokio-rs/tls/releases)
- [Commits](https://github.com/tokio-rs/tls/commits)

---
updated-dependencies:
- dependency-name: tokio-rustls
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-21 10:43:07 +00:00
dependabot[bot] 836aecc7ad
chore(deps): Bump libc from 0.2.120 to 0.2.121 (#4076) 
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.120 to 0.2.121.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.120...0.2.121)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-21 09:28:39 +00:00
Marco Neumann a122b1e2ca
refactor: dyn-typed DB for `query_tests` (#4053) 
To test the `db::Db` as well as the `querier` with the same test
framework, they require a shared interface. Ideally this interface is
dynamically typed instead of static dispatched via generics because:

- `query_tests` already take ages to compile
- we often hold a list of scenarios and a single scenario should (in a
  future PR) be able to represent both OG as well as NG

The vision here is that we basically keep the whole test setup but add
new scenarios which are NG-specific later on.

Now the issue w/ many query-related types is that they are NOT
object-safe because methods that don't take `&self` or they have
associated types that we cannot specify in general for OG and NG at the
same time.

So we need a bunch of wrappers that make dynamic dispatch possible. They
mostly call to an internal "interface" crate which is the actual `dyn`
part. The interface is currently only implemented for OG.

The scenarios currently also only contain OG databases. However,
creating a dynamic interface that can be used in all `query_tests` is
already a huge step.

Note that there are two places where we downcast the dynamic/abstract
database to `db::Db` again:

1. To create one scenario based on another and where we need to
   manipulate `db::Db` with OG-specific semantics.
2. `server_benchmarks`. These contain OG databases only and there is no
   point in benchmarking throw the dynamic dispatch interface because
   prod (`influxdb_ioxd`) also uses static dispatch.

Ref #3934.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-18 10:11:17 +00:00
Dom Dwyer d9900f661b refactor: ingester table & namespace count metrics 
Record the number of tables / namespaces an ingester process has
observed.
 2022-03-17 17:20:30 +00:00
dependabot[bot] 58256c32ff
chore(deps): Bump syn from 1.0.88 to 1.0.89 (#4050) 
Bumps [syn](https://github.com/dtolnay/syn) from 1.0.88 to 1.0.89.
- [Release notes](https://github.com/dtolnay/syn/releases)
- [Commits](https://github.com/dtolnay/syn/compare/1.0.88...1.0.89)

---
updated-dependencies:
- dependency-name: syn
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-17 11:30:42 +00:00
kodiakhq[bot] f4cd5c9ba8
Merge branch 'main' into dom/object-store-metrics 2022-03-15 14:27:19 +00:00
dependabot[bot] cc7f71126c
chore(deps): Bump syn from 1.0.86 to 1.0.88 (#4037) 
Bumps [syn](https://github.com/dtolnay/syn) from 1.0.86 to 1.0.88.
- [Release notes](https://github.com/dtolnay/syn/releases)
- [Commits](https://github.com/dtolnay/syn/compare/1.0.86...1.0.88)

---
updated-dependencies:
- dependency-name: syn
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-15 12:22:18 +00:00
dependabot[bot] e2cf5c63e5
chore(deps): Bump nom from 7.1.0 to 7.1.1 (#4035) 
Bumps [nom](https://github.com/Geal/nom) from 7.1.0 to 7.1.1.
- [Release notes](https://github.com/Geal/nom/releases)
- [Changelog](https://github.com/Geal/nom/blob/main/CHANGELOG.md)
- [Commits](https://github.com/Geal/nom/compare/7.1.0...7.1.1)

---
updated-dependencies:
- dependency-name: nom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-15 11:29:01 +00:00
dependabot[bot] eeb918ca2b
chore(deps): Bump libc from 0.2.119 to 0.2.120 (#4034) 
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.119 to 0.2.120.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.119...0.2.120)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-15 10:56:52 +00:00
dependabot[bot] 10831a0530
chore(deps): Bump reqwest from 0.11.9 to 0.11.10 (#4032) 
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.9 to 0.11.10.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.9...v0.11.10)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-15 10:43:28 +00:00
Andrew Lamb 8f1938a482
chore: Update datafusion (#4022) 
* chore: Update datafusion

* chore: update for change in Expr

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-14 17:24:00 +00:00
Marco Neumann 27efb66237
test: add proptest for `AddressableHeap` (#4025) 
* test: add proptest for `AddressableHeap`

For #3985.

* refactor: simplify code

Co-authored-by: Edd Robinson <me@edd.io>

Co-authored-by: Edd Robinson <me@edd.io>
 2022-03-14 12:38:30 +00:00
dependabot[bot] 3f0f090c4e
chore(deps): Bump pretty_assertions from 1.1.0 to 1.2.0 (#4024) 
Bumps [pretty_assertions](https://github.com/colin-kiegel/rust-pretty-assertions) from 1.1.0 to 1.2.0.
- [Release notes](https://github.com/colin-kiegel/rust-pretty-assertions/releases)
- [Changelog](https://github.com/colin-kiegel/rust-pretty-assertions/blob/main/CHANGELOG.md)
- [Commits](https://github.com/colin-kiegel/rust-pretty-assertions/commits)

---
updated-dependencies:
- dependency-name: pretty_assertions
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-14 10:33:27 +00:00
Nga Tran 5a29d070ea
feat: Implement the compact function for NG Compactor (#4001) 
* feat: initial implementation of compact a given list of overlapped parquet files

* feat: Add QueryableParquetChunk and some refactoring

* feat:  build queryable parquet chunks for parquet files with tombstones

* feat: second half the implementation for Compactor's compact. Tests will be next

* fix: comments for trait funnctions fof QueryChunkMeta

* test: add tests for compactor's compact function

* fix: typos

* refactor: address Jake's review comments

* refactor: address Andrew's comments and add one more test for files in different order in the vector

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-11 20:25:19 +00:00
Dom Dwyer bfc473b8cd feat: object store instrumentation 
Implements a decorator of the ObjectStoreAPI trait, recording:

    * Bytes uploaded / downloaded through the instrumented API
    * Call latencies, broken down by operation & success / error state

All the current implementations that return a Stream from the get
operation actually return a "fake" stream containing all the data in one
go rather than streaming chunks from the upstream. I've instrumented the
Stream to future-proof any actual streaming impls in the future.
 2022-03-11 13:38:48 +00:00
dependabot[bot] bd2824aa59
chore(deps): Bump pprof from 0.6.2 to 0.7.0 (#4004) 
* chore(deps): Bump pprof from 0.6.2 to 0.7.0

Bumps [pprof](https://github.com/tikv/pprof-rs) from 0.6.2 to 0.7.0.
- [Release notes](https://github.com/tikv/pprof-rs/releases)
- [Changelog](https://github.com/tikv/pprof-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tikv/pprof-rs/commits)

---
updated-dependencies:
- dependency-name: pprof
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix: `pprof` features for version 0.7

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-11 11:08:59 +00:00
Nga Tran f03ebd79ab
refactor: move querier's test utils to a new crate to get reused by tests in other crates (#4013) 
* refactor: move querier's test utils to a new crate to be able resued by tests in other crates

* chore: remove unused import
 2022-03-10 18:17:58 +00:00
dependabot[bot] 67209fce7c
chore(deps): Bump clap from 3.1.5 to 3.1.6 (#4010) 
Bumps [clap](https://github.com/clap-rs/clap) from 3.1.5 to 3.1.6.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v3.1.5...v3.1.6)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-10 14:38:53 +00:00
dependabot[bot] e1344f7ca8
chore(deps): Bump sqlparser from 0.14.0 to 0.15.0 (#4007) 
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs) from 0.14.0 to 0.15.0.
- [Release notes](https://github.com/sqlparser-rs/sqlparser-rs/releases)
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.14.0...v0.15.0)

---
updated-dependencies:
- dependency-name: sqlparser
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-10 14:28:48 +00:00
dependabot[bot] 68e09264f2
chore(deps): Bump pbjson from 0.2.2 to 0.2.3 (#4006) 
Bumps [pbjson](https://github.com/influxdata/pbjson) from 0.2.2 to 0.2.3.
- [Release notes](https://github.com/influxdata/pbjson/releases)
- [Commits](https://github.com/influxdata/pbjson/compare/0.2.2...0.2.3)

---
updated-dependencies:
- dependency-name: pbjson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-10 14:20:10 +00:00
dependabot[bot] f2834c4101
chore(deps): Bump handlebars from 4.2.1 to 4.2.2 (#4009) 
Bumps [handlebars](https://github.com/sunng87/handlebars-rust) from 4.2.1 to 4.2.2.
- [Release notes](https://github.com/sunng87/handlebars-rust/releases)
- [Changelog](https://github.com/sunng87/handlebars-rust/blob/master/CHANGELOG.md)
- [Commits](https://github.com/sunng87/handlebars-rust/compare/v4.2.1...v4.2.2)

---
updated-dependencies:
- dependency-name: handlebars
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-10 14:11:19 +00:00
dependabot[bot] c051e944f2
chore(deps): Bump tracing from 0.1.31 to 0.1.32 (#4008) 
Bumps [tracing](https://github.com/tokio-rs/tracing) from 0.1.31 to 0.1.32.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.31...tracing-0.1.32)

---
updated-dependencies:
- dependency-name: tracing
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-10 14:01:00 +00:00
dependabot[bot] cef4090bce
chore(deps): Bump pbjson-types from 0.2.2 to 0.2.3 (#4005) 
Bumps [pbjson-types](https://github.com/influxdata/pbjson) from 0.2.2 to 0.2.3.
- [Release notes](https://github.com/influxdata/pbjson/releases)
- [Commits](https://github.com/influxdata/pbjson/compare/0.2.2...0.2.3)

---
updated-dependencies:
- dependency-name: pbjson-types
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-10 13:52:08 +00:00
dependabot[bot] ed84b453d3
chore(deps): Bump tracing-core from 0.1.22 to 0.1.23 (#4003) 
Bumps [tracing-core](https://github.com/tokio-rs/tracing) from 0.1.22 to 0.1.23.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-core-0.1.22...tracing-core-0.1.23)

---
updated-dependencies:
- dependency-name: tracing-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-10 13:43:15 +00:00
dependabot[bot] 0756642137
chore(deps): Bump pbjson-build from 0.2.2 to 0.2.3 (#4002) 
Bumps [pbjson-build](https://github.com/influxdata/pbjson) from 0.2.2 to 0.2.3.
- [Release notes](https://github.com/influxdata/pbjson/releases)
- [Commits](https://github.com/influxdata/pbjson/compare/0.2.2...0.2.3)

---
updated-dependencies:
- dependency-name: pbjson-build
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-10 13:34:00 +00:00
Andrew Lamb 2c3d30ca32
chore: Update datafusion, arrow, flight and parquet (#4000) 
* chore: Update datafusion, arrow, flight and parquet

* fix: api change

* fix: fmt

* fix: update test metadata size

* fix: Update sizes in parquet test

* fix: more metadata size update
 2022-03-10 12:24:47 +00:00
Carol (Nichols || Goulding) 944f628e29
fix: Remove data_types as a dependency of ng compactor (#3993) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-09 17:03:02 +00:00
Carol (Nichols || Goulding) 93b0cdbcc4
fix: Create the test database as part of ng server fixture startup 2022-03-09 09:55:43 -05:00
Andrew Lamb d2c0acdd46
refactor: Remove serving readiness gate (#3986) 
* refactor: Remove serving_readiness

* fix: remove more

* fix: remove test

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-09 12:17:44 +00:00
Nga Tran 09fba1d2c0
feat: NG Compactor - main function for finding and compacting parquet files (#3973) 
* feat: main function for finding and compacting parquet files

* chore: Apply suggestions from code review

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

* refactor: rename file and struct

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-08 16:34:43 +00:00
Dom Dwyer 3278e35b41 build: bump regex 
Bump regex version due to RUSTSEC-2022-0013, see:

    https://rustsec.org/advisories/RUSTSEC-2022-0013
 2022-03-08 15:37:23 +00:00
Dom Dwyer baae6910f2 build: remove openssl dependency 
Switches a stray reqwest dependency to use rustls.
 2022-03-07 13:47:39 +00:00
kodiakhq[bot] e47bfc40be
Merge branch 'main' into dom/router-grpc-write 2022-03-07 11:13:19 +00:00
dependabot[bot] d3321135eb
chore(deps): Bump clap from 3.1.2 to 3.1.5 (#3957) 
Bumps [clap](https://github.com/clap-rs/clap) from 3.1.2 to 3.1.5.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v3.1.2...v3.1.5)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-07 09:58:46 +00:00
dependabot[bot] 3c58a08431
chore(deps): Bump siphasher from 0.3.9 to 0.3.10 (#3956) 
Bumps [siphasher](https://github.com/jedisct1/rust-siphash) from 0.3.9 to 0.3.10.
- [Release notes](https://github.com/jedisct1/rust-siphash/releases)
- [Commits](https://github.com/jedisct1/rust-siphash/compare/0.3.9...0.3.10)

---
updated-dependencies:
- dependency-name: siphasher
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-07 09:10:09 +00:00
dependabot[bot] 48908054d1
chore(deps): Bump once_cell from 1.9.0 to 1.10.0 (#3955) 
* chore(deps): Bump once_cell from 1.9.0 to 1.10.0

Bumps [once_cell](https://github.com/matklad/once_cell) from 1.9.0 to 1.10.0.
- [Release notes](https://github.com/matklad/once_cell/releases)
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md)
- [Commits](https://github.com/matklad/once_cell/compare/v1.9.0...v1.10.0)

---
updated-dependencies:
- dependency-name: once_cell
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2022-03-07 08:54:20 +00:00
Andrew Lamb e09f39d6a0
chore: Update datafusion (#3943) 
* chore: Update datafusion

* refactor: update for new datafusion

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2022-03-04 19:37:46 +00:00
Andrew Lamb b870b9340b
chore: remove uneeded dependencies (#3929) 
* chore: remove unused deps in compactor

* chore: remove unused deps in influxdb_ioxd

* chore: remove unused deps in object_store

* chore: remove unused deps in server

* fix: object_store needs observability deps when compiled with aws

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-04 17:39:46 +00:00
Dom Dwyer 26c43f0a2c feat: grpc write handler 
Implements the gRPC write handler endpoint.
 2022-03-04 14:51:42 +00:00
Marco Neumann 8d00aaba90
feat: sync chunks in querier (#3911) 
* feat: `ParquetFileRepo::list_by_namespace_not_to_delete`

* feat: `ChunkAddr: Clone`

* test: ensure that querier keeps same partition objects

* test: improve `create_parquet_file` flexibility

* feat: sync chunks in querier

* test: improve `test_parquet_file`
 2022-03-04 08:53:39 +00:00
Andrew Lamb d6afda0227
refactor: split `influxdb_ioxd`, `clap_blocks` and `serving_readiness` from influxdb_iox crate (#3908) 
* refactor: split influxdb_ioxd, clap_blocks, and serving_readiness out of influxdb_iox

split out serving readiness, get compiling

* fix: hakari

* fix: hakari again

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-03 16:48:30 +00:00
kodiakhq[bot] 04a7a957fe
Merge branch 'main' into dom/rustls 2022-03-03 11:59:40 +00:00
Dom Dwyer da145ffbe4 feat: batch upsert of columns to catalog 
Adds ColumnRepo::create_or_get_many() to upsert multiple columns in one
round trip to Postgres.
 2022-03-03 11:17:30 +00:00
Dom Dwyer 46bb107be4 refactor: use rustls 
Removes openssl as a dependency, switching to rustls[1] as the TLS
implementation throughout.

It is important to note that this change brings with it a significant
behavioural difference - rustls does not currently support IP SANs in
certificates (instead only supporting fully-qualified names / DNS) and
this will manifest as a failure to connect to IP endpoints over TLS.
This might be a blocker that prevents us using rustls exclusively, but
there's noe asy way to know without trying it. Fortunately the rustls
project has received funding to work on IP SAN support[2].

[1]: https://github.com/rustls/rustls
[2]: https://www.abetterinternet.org/post/preparing-rustls-for-wider-adoption/
 2022-03-03 11:05:20 +00:00
Dom Dwyer 4785a7d028 refactor: bump Azure SDK 
Update to the latest Azure SDK to pick up rustls support.
 2022-03-03 11:02:20 +00:00
Raphael Taylor-Davies 02a2cfde54
feat: update rskafka (#3805) (#3910) 2022-03-03 10:53:47 +00:00
Carol (Nichols || Goulding) 3f2a58b47f
refactor: pub use data_types from data_types2 
So it's clearer which parts of data_types the NG design is using, and
which types can be cleaned up eventually.
 2022-03-02 13:55:31 -05:00
Carol (Nichols || Goulding) 2a90841715
refactor: Move IngesterQueryRequest to data_types2 
So that querier doesn't need to depend on ingester.
 2022-03-02 13:52:13 -05:00
Carol (Nichols || Goulding) 8f3e44bf76
refactor: Extract a crate for shared data types in the new design 2022-03-02 12:16:15 -05:00
Carol (Nichols || Goulding) 141a6087d0
feat: Querier able to send Flight queries to Ingester 
Fixes #3773.
 2022-03-02 11:50:45 -05:00
Raphael Taylor-Davies 44cfdc7aca
feat: update to latest rskafka logging (#3805) (#3896) 2022-03-02 11:05:03 +00:00
Marco Neumann af57664f53
feat: wire up flight into the querier (#3889) 2022-03-02 09:20:26 +00:00
Marco Neumann 936f51013d
test: querier shutdown and background task handling (#3881) 
This is similar to what we already have for the ingester.
 2022-03-02 08:59:50 +00:00
Raphael Taylor-Davies 143311f63f
feat: additional write buffer logging (#3805) (#3887) 
* feat: additional write buffer logging (#3805)

* fix: assertion direction

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-01 17:08:57 +00:00
Nga Tran 0e0dc500f6
feat: prepare data to send to querier (#3825) 
* feat: changes needed to apply tombstones correctly on the life-cycle ingest bacthes

* refactor: adjust the  design after discussing with Paul

* feat: apply the coming tombstone on all data but persiting one

* chore: fmt

* fix: build on buffer tombstone

* test: delete & write tests for a parition and some cleanup

* feat: No need add processed tombstones for newly created parquet file in the ingester becasue all deletes before that parquet file is created were applied

* chore: cleanup

* feat: intitial implementation for preparing data to send back to the Querier

* feat: full implementation of prepare_data_to_querier

* fix: apply filters for the batches

* chore: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* chore: cleanup

* fix: typos in comments

* fix: typos in comments

* fix: typos in comments

* test: create different scenarios and test them

* chore: fix typos

* test: add tests with deletes

* chore: make pub pub(crate)

* chore: Apply suggestions from code review

Co-authored-by: Jake Goulding <jake.goulding@integer32.com>

* refactor: address review comments

* fix: keep batches in their arrival order

* refactor: not assign unecessary values to enum

* refactor: use bitflags enum

* fix: use bitflags correctly

* chore: Apply suggestions from code review

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

* refactor: avoid using use at the end of the function

* chore: merge main to branch

* fix: fix downgrade versions

* refactor: address review comments

* chore: remove unnecessary comments

* refactor: Make the whole test_utils module test-only and bring paths into module scope

Co-authored-by: Paul Dix <paul@pauldix.net>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
 2022-03-01 01:00:45 +00:00
Marco Neumann 6e2470bf5f
feat: create CatalogChunk in querier (#3862) 
* feat: `NamespaceRepo::get_by_id`

* feat: create `CatalogChunk` in querier
 2022-02-28 17:20:38 +00:00
Marco Neumann b213796c98
feat: sync namespaces in querier (#3865) 
* feat: `NamespaceRepo::list`

* feat: sync namespaces in querier

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-28 15:01:28 +00:00
Marco Neumann 77a0c74830
chore: `cargo update` + less rustsec exceptions (#3873) 
It seems that dependabot doesn't catch with the flood of updates, so
this is a manual update run:

```console
$ cargo update
    Updating crates.io index
    Updating git repository `https://github.com/Azure/azure-sdk-for-rust.git`
    Updating git repository `https://github.com/apache/arrow-datafusion.git`
    Updating git repository `https://github.com/influxdata/rskafka.git`
    Updating git repository `https://github.com/mkmik/heappy`
    Updating anyhow v1.0.53 -> v1.0.55
    Updating autocfg v1.0.1 -> v1.1.0
    Updating clang-sys v1.3.0 -> v1.3.1
    Updating clap_derive v3.1.0 -> v3.1.2
    Updating core-foundation v0.9.2 -> v0.9.3
    Updating crossbeam-epoch v0.9.6 -> v0.9.7
    Updating crossbeam-queue v0.3.3 -> v0.3.4
    Updating crossbeam-utils v0.8.6 -> v0.8.7
    Updating debugid v0.7.2 -> v0.7.3
    Updating fd-lock v3.0.3 -> v3.0.4
    Updating inferno v0.10.10 -> v0.10.12
    Updating io-lifetimes v0.4.4 -> v0.5.3
    Updating kube-client v0.69.0 -> v0.69.1
    Updating kube-core v0.69.0 -> v0.69.1
    Updating libm v0.2.1 -> v0.2.2
    Updating linux-raw-sys v0.0.37 -> v0.0.42
    Removing memmap v0.7.0
      Adding memmap2 v0.5.3
    Updating ntapi v0.3.6 -> v0.3.7
    Updating output_vt100 v0.1.2 -> v0.1.3
    Updating rgb v0.8.31 -> v0.8.32
    Updating rustix v0.32.1 -> v0.33.3
    Updating rustls v0.20.2 -> v0.20.4
    Updating security-framework v2.6.0 -> v2.6.1
    Updating security-framework-sys v2.6.0 -> v2.6.1
    Updating semver v1.0.4 -> v1.0.6
    Updating symbolic-common v8.5.0 -> v8.6.1
    Updating symbolic-demangle v8.5.0 -> v8.6.1
    Updating tower-http v0.2.2 -> v0.2.3
    Updating unicode-segmentation v1.8.0 -> v1.9.0
    Updating zeroize v1.5.2 -> v1.5.3
```

This also allows us to remove a rustsec exception.
 2022-02-28 11:24:32 +00:00
dependabot[bot] a9d52aaef3
chore(deps): Bump mockito from 0.30.0 to 0.31.0 (#3871) 
* chore(deps): Bump mockito from 0.30.0 to 0.31.0

Bumps [mockito](https://github.com/lipanski/mockito) from 0.30.0 to 0.31.0.
- [Release notes](https://github.com/lipanski/mockito/releases)
- [Commits](https://github.com/lipanski/mockito/compare/0.30.0...0.31.0)

---
updated-dependencies:
- dependency-name: mockito
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: disallow `RUSTSEC-2020-0095`

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-28 10:05:24 +00:00
dependabot[bot] c289467239
chore(deps): Bump clap from 3.1.1 to 3.1.2 (#3870) 
Bumps [clap](https://github.com/clap-rs/clap) from 3.1.1 to 3.1.2.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v3.1.1...v3.1.2)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-28 09:53:25 +00:00
Marco Neumann f966f4c7a4
feat: create `ParquetChunk` in querier (#3857) 
Adds a small adapter that is able to produce `ParquetChunk`s for NG.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-25 08:54:16 +00:00
Marco Neumann 49d1be30e7
feat: wire up `ParquetFilePath` for NG (#3853) 
It's a bit of a duck-type hack, but if we wanna just `ParquetFileChunk`
in the new architecture, we somehow need it to accept new-gen paths.
Also path handling should be somewhat centralized since
ingester/compactor/querier all need to construct them. So having a
`ParquetFilePath` that supports both path styles seems to be a
not-to-bad solution. This should obviously be cleaned up in some
not-to-distant future.
 2022-02-24 16:05:38 +00:00
Luke Bond 4731913c44
feat: skeleton of querier CLI (#3843) 
* feat: skeleton of querier CLI

* chore: wrap metrics in opt&arc in querier to satisfy new api

* chore: derive debug in querier handler

* chore: add join handles and their shutdown to nascent querier server

* chore: querier server http unimpl -> 404

* fix: join/shutdown fix in querier; removed unused delegates
 2022-02-24 15:42:56 +00:00
Marco Neumann 9079e6ddb0
feat: backoff retries in ingester (#3841) 
* feat: add `backoff` crate

* feat: backoff retries in ingester

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-23 17:58:16 +00:00
kodiakhq[bot] 3e69a5e1b4
Merge branch 'main' into dom/catalog-instrumentation 2022-02-23 16:14:40 +00:00
Marco Neumann 657ac249e9
feat: track ingester jobs (#3836) 2022-02-23 15:33:47 +00:00
Dom Dwyer a0b43323d6 feat: catalog instrumentation decorator 
Adds a MetricDecorator type that wraps a Catalog impl, delegating calls
to the underlying impl and recording per-method call latency broken down
by success/error state.
 2022-02-23 14:34:26 +00:00