b23fe9d141
chore(deps): Bump syn from 1.0.91 to 1.0.92 ( #4471 )
...
Bumps [syn](https://github.com/dtolnay/syn ) from 1.0.91 to 1.0.92.
- [Release notes](https://github.com/dtolnay/syn/releases )
- [Commits](https://github.com/dtolnay/syn/compare/1.0.91...1.0.92 )
---
updated-dependencies:
- dependency-name: syn
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-29 10:43:25 +00:00
01a6cf1c2a
chore(deps): Bump http from 0.2.6 to 0.2.7 ( #4470 )
...
Bumps [http](https://github.com/hyperium/http ) from 0.2.6 to 0.2.7.
- [Release notes](https://github.com/hyperium/http/releases )
- [Changelog](https://github.com/hyperium/http/blob/master/CHANGELOG.md )
- [Commits](https://github.com/hyperium/http/compare/v0.2.6...v0.2.7 )
---
updated-dependencies:
- dependency-name: http
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-29 09:56:48 +00:00
8e48fcd620
feat: add remote pull partition ( #4433 )
...
Add lookup of partitions by table id to catalog.
Add API to catalog to return partitions by table id.
Add to client to return partitions by table id.
Add CLI to pull remote schema, partition, and parquet files into a local catalog and object store.
2022-04-28 21:04:27 +00:00
7b6754cc03
chore: Update dependencies via `cargo update` ( #4460 )
...
* chore: Update dependencies via `cargo update`
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-04-28 18:48:51 +00:00
63df3ceb6f
feat: share server processes in end to end test ( #4387 )
...
* feat: share server processes in end to end test
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-28 17:51:14 +00:00
5688bd63a3
docs: document for deduplication and sort key ( #4452 )
...
* docs: document for deduplication and sort key
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Luke Bond <luke.n.bond@gmail.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* chore: Update query/src/provider.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* chore: address review comments
* fix: fix comment
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Luke Bond <luke.n.bond@gmail.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-28 16:44:40 +00:00
e433c6c704
refactor: use with_timeout_panic ( #4461 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-28 15:33:58 +00:00
6eed09a926
test: use "real" ingester in query tests ( #4455 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-28 14:39:31 +00:00
3198d75f2f
Merge pull request #4447 from influxdata/cn/aio
...
feat: Make all-in-one mode easier to use
2022-04-28 14:00:55 +00:00
5c89e7b952
fix: Remove max catalog connection config from all-in-one mode
2022-04-28 09:29:03 -04:00
ff268ace6d
fix: Add more docs on write buffer temp dir behavior
2022-04-28 09:29:02 -04:00
c93c26c503
fix: Add reference to TODO ticket
2022-04-28 09:29:02 -04:00
4597c02daa
fix: Add instructions on specifying host when loading data
2022-04-28 09:29:02 -04:00
a024e07147
fix: Small corrections to some comments
2022-04-28 09:29:02 -04:00
df1afa3481
feat: Log what kind of server in startup/shutdown logs
2022-04-28 09:29:02 -04:00
84683056db
feat: Add more logging for all-in-one mode to make it clearer what components are used
2022-04-28 09:29:02 -04:00
d6d50f83c2
feat: Set different catalog config defaults for all-in-one mode
...
Connects to #4399 .
If `--catalog-dsn` is specified, use that Postgres catalog. If
`--catalog-dsn` is not specified, use an in-memory catalog.
2022-04-28 09:29:02 -04:00
941dd12dd1
feat: Set different write buffer config defaults for all-in-one mode
...
Connects to #4399 .
Only file-based write buffer is supported. If `--data-dir` is specified,
store it there, otherwise store it in a temp directory to be ephemeral
2022-04-28 09:29:02 -04:00
0cfd16263c
refactor: Change run_config to logging_config
...
The only spot this method is used actually wants the logging config
2022-04-28 09:29:01 -04:00
06342f9ed8
feat: Set different ingester defaults for all-in-one mode
...
Connects to #4399 .
Manually flatten the arguments to set different defaults, not allow
setting the partition min/max, but still allow customization of the
other arguments.
2022-04-28 09:29:01 -04:00
f8166f7cf4
feat: Update README instructions for NG
...
Connects to #4399 .
2022-04-28 09:29:01 -04:00
6ec0b401e9
fix: Support for computing series across multiple record batches ( #4444 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-28 13:12:27 +00:00
e13d3433ae
feat: Use datafusion serialization code rather than our own copy of it ( #4421 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-28 13:03:34 +00:00
e41785ac3b
Merge pull request #4458 from influxdata/dom/e2e-metrics
...
test: remove hard-coded metric count
2022-04-28 12:48:24 +00:00
246af0c3ca
test: remove hard-coded metric count
...
Prior to this commit, adding any metric to the catalog (and only the
catalog) would cause the end_to_end_ng_cases::metrics::test_metrics test
to fail due to asserting an exact number of metrics observed.
This commit changes the check condition to a more permisive >= rather
than ==.
2022-04-28 11:28:48 +01:00
420c306caa
chore(deps): Bump tokio from 1.17.0 to 1.18.0 ( #4453 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.17.0 to 1.18.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.17.0...tokio-1.18.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-28 08:21:17 +00:00
b0ba0361ed
Merge branch 'main' into dom/schema-cache-warm
2022-04-27 21:24:57 +00:00
115f007317
refactor: Use DataFusion `Expr` instead of our own custom wrapper for `ValueExpr` ( #4440 )
...
* refactor: Use DataFusion `Expr` instead of custom wrapper for BinaryExprs
* fix: apply code review suggestions
* fix: more code review suggestions
2022-04-27 19:20:15 +00:00
59f6556483
fix: do not create empty batches in ingester ( #4443 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-27 17:52:22 +00:00
a195973cfb
chore(deps): Bump clap from 3.1.11 to 3.1.12 ( #4406 )
...
* chore(deps): Bump clap from 3.1.11 to 3.1.12
Bumps [clap](https://github.com/clap-rs/clap ) from 3.1.11 to 3.1.12.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.1.11...v3.1.12 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix: Update tests now that the clap crossed-streams bug has been fixed
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-27 17:43:55 +00:00
d7d4de5144
feat(router2): pre-warm schema cache
...
Fetch all NamespaceSchema from the catalog and pre-cache them during
router initialisation.
This pre-warming happens in 3 queries, and should significantly reduce
the load on the Postgres instance backing the catalog during
deployments/after router crashes/etc.
Pre-warming occurs during init, before the HTTP endpoints are bound, and
therefore before the /health HTTP endpoint can return OK. This ensures
the router pre-warms the cache before it begins receiving traffic.
2022-04-27 17:28:07 +01:00
bb8a19b571
feat(iox_catalog): list_schemas()
...
Adds a function to resolve an atomic snapshot of all NamespaceSchema in
the catalog with minimal query overhead.
2022-04-27 17:23:28 +01:00
874521da8a
feat(iox_catalog): ColumnRepo::list()
...
Allow all columns in the catalog to be fetched.
2022-04-27 17:21:00 +01:00
eb5abce99e
feat(iox_catalog): TableRepo::list()
...
Allow all tables in the catalog to be fetched.
2022-04-27 17:20:53 +01:00
468f576698
fix: do not prune all-NULL columns from sort keys ( #4442 )
2022-04-27 16:06:26 +00:00
bea4556749
chore: Ignore flaky OG end to end test ( #4441 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-27 15:17:46 +00:00
6d2a8256ba
feat: Add window_bounds to IOx Function Registry (2.5/3) ( #4432 )
...
* feat: Add window_bounds to IOx Function Registry
* refactor: Prepare for lib tests
* test: Add explicit tests for plumbing
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-27 15:08:38 +00:00
663944a2a9
chore(deps): Bump all rusoto crates from 0.47.0 to 0.48.0 ( #4439 )
...
* chore(deps): Bump rusoto_credential from 0.47.0 to 0.48.0
Bumps [rusoto_credential](https://github.com/rusoto/rusoto ) from 0.47.0 to 0.48.0.
- [Release notes](https://github.com/rusoto/rusoto/releases )
- [Changelog](https://github.com/rusoto/rusoto/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rusoto/rusoto/compare/rusoto-v0.47.0...mock-v0.48.0 )
---
updated-dependencies:
- dependency-name: rusoto_credential
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): Bump rusoto_core from 0.47.0 to 0.48.0
Bumps [rusoto_core](https://github.com/rusoto/rusoto ) from 0.47.0 to 0.48.0.
- [Release notes](https://github.com/rusoto/rusoto/releases )
- [Changelog](https://github.com/rusoto/rusoto/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rusoto/rusoto/compare/rusoto-v0.47.0...mock-v0.48.0 )
---
updated-dependencies:
- dependency-name: rusoto_core
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): Bump rusoto_s3 from 0.47.0 to 0.48.0
Bumps [rusoto_s3](https://github.com/rusoto/rusoto ) from 0.47.0 to 0.48.0.
- [Release notes](https://github.com/rusoto/rusoto/releases )
- [Changelog](https://github.com/rusoto/rusoto/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rusoto/rusoto/compare/rusoto-v0.47.0...mock-v0.48.0 )
---
updated-dependencies:
- dependency-name: rusoto_s3
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-27 14:58:40 +00:00
bd0bae13ce
fix: extend + harden querier `ensure_schema` ( #4429 )
...
- only convert dictionary types that we really want to convert (instead
of blindly converting all types)
- handle missing / NULL columns
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-27 12:49:59 +00:00
5e91a3b909
chore: Update datafusion ( #4436 )
2022-04-27 11:25:10 +00:00
f0ceca985b
feat: Implement IOx Function Registry for regex_match/regex_not_match ( #4431 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-27 10:33:19 +00:00
b01a5bb0a9
chore(deps): Bump httparse from 1.7.0 to 1.7.1 ( #4434 )
...
Bumps [httparse](https://github.com/seanmonstar/httparse ) from 1.7.0 to 1.7.1.
- [Release notes](https://github.com/seanmonstar/httparse/releases )
- [Commits](https://github.com/seanmonstar/httparse/compare/v1.7.0...v1.7.1 )
---
updated-dependencies:
- dependency-name: httparse
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-27 08:12:23 +00:00
fa2c1febf4
feat: use stored partition sort key to deduplicate data ( #4360 )
...
* feat: use stored sort key to deduplicate data
* refactor: verify if one is a super sort key of the other
* test: unit tests for scan and deduplication plans
* fix: typo
* refactor: refactor and add comments
* feat: cache partition sort key to read during planning as needed
* test: tests for query plans with different overlap groups
* chore: cleanup
* chore: resolve merge conflicts
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-26 20:36:32 +00:00
9e91af4501
refactor: Move IOx UDfs into a Function Registry (1/3) ( #4428 )
...
* refactor: Move all UDF implementations to query_function crate
* refactor: Move regex udf to query_functions
* refactor: Move functions out of query
* fix: lints, imports
* chore: Run cargo hakari tasks
* fix: clipy + benches
* fix: reduce borrowing and fix clippy
* fix: moar clippy
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-26 17:30:27 +00:00
11f87cffdd
fix: memorize max persisted tombstone ( #4430 )
2022-04-26 16:13:09 +00:00
a72544f42d
chore(deps): Bump tower-http from 0.2.5 to 0.3.0 ( #4425 )
...
* chore(deps): Bump tower-http from 0.2.5 to 0.3.0
Bumps [tower-http](https://github.com/tower-rs/tower-http ) from 0.2.5 to 0.3.0.
- [Release notes](https://github.com/tower-rs/tower-http/releases )
- [Commits](https://github.com/tower-rs/tower-http/compare/tower-http-0.2.5...tower-http-0.3.0 )
---
updated-dependencies:
- dependency-name: tower-http
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-26 15:05:42 +00:00
bd600bbac6
refactor: allow ingester to be integrated into query tests ( #4427 )
...
* refactor: improve `IngesterData` public interface
* feat: impl `Debug` for `Test{Namespace,Sequencer}`
* refactor: trait interface for `LifecyleHandle`
This is required to mock the lifecycle for query tests.
* refactor: trait for partitioner
2022-04-26 13:44:30 +00:00
401009a7b6
Merge pull request #4419 from influxdata/dom/http-service-protection-limit
...
feat(router2): http service protection limit
2022-04-26 10:21:57 +00:00
8a208a814a
feat: emit http_request_limit_rejected metric
...
Add the "http_request_limit_rejected" metric that is incremented once
each time a request is dropped due to the simultaneous request service
protection limit being exceeded.
This metric will allow us to effectively alert on router saturation, and
the increase rate/second will help inform us in how much capacity needs
adding.
2022-04-26 11:13:26 +01:00
fb777e7e51
feat(router2): configurable max HTTP requests
...
Adds a CLI / env configuration option controlling the maximum number of
simultaneous HTTP requests in the router.
2022-04-26 11:13:25 +01:00
dependabot[bot]
Paul Dix
Andrew Lamb
Nga Tran
Marco Neumann
kodiakhq[bot]
Carol (Nichols || Goulding)
Dom Dwyer
Carol (Nichols || Goulding)