68e09264f2
chore(deps): Bump pbjson from 0.2.2 to 0.2.3 ( #4006 )
...
Bumps [pbjson](https://github.com/influxdata/pbjson ) from 0.2.2 to 0.2.3.
- [Release notes](https://github.com/influxdata/pbjson/releases )
- [Commits](https://github.com/influxdata/pbjson/compare/0.2.2...0.2.3 )
---
updated-dependencies:
- dependency-name: pbjson
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-10 14:20:10 +00:00
f2834c4101
chore(deps): Bump handlebars from 4.2.1 to 4.2.2 ( #4009 )
...
Bumps [handlebars](https://github.com/sunng87/handlebars-rust ) from 4.2.1 to 4.2.2.
- [Release notes](https://github.com/sunng87/handlebars-rust/releases )
- [Changelog](https://github.com/sunng87/handlebars-rust/blob/master/CHANGELOG.md )
- [Commits](https://github.com/sunng87/handlebars-rust/compare/v4.2.1...v4.2.2 )
---
updated-dependencies:
- dependency-name: handlebars
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-10 14:11:19 +00:00
c051e944f2
chore(deps): Bump tracing from 0.1.31 to 0.1.32 ( #4008 )
...
Bumps [tracing](https://github.com/tokio-rs/tracing ) from 0.1.31 to 0.1.32.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.31...tracing-0.1.32 )
---
updated-dependencies:
- dependency-name: tracing
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-10 14:01:00 +00:00
cef4090bce
chore(deps): Bump pbjson-types from 0.2.2 to 0.2.3 ( #4005 )
...
Bumps [pbjson-types](https://github.com/influxdata/pbjson ) from 0.2.2 to 0.2.3.
- [Release notes](https://github.com/influxdata/pbjson/releases )
- [Commits](https://github.com/influxdata/pbjson/compare/0.2.2...0.2.3 )
---
updated-dependencies:
- dependency-name: pbjson-types
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-10 13:52:08 +00:00
ed84b453d3
chore(deps): Bump tracing-core from 0.1.22 to 0.1.23 ( #4003 )
...
Bumps [tracing-core](https://github.com/tokio-rs/tracing ) from 0.1.22 to 0.1.23.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-core-0.1.22...tracing-core-0.1.23 )
---
updated-dependencies:
- dependency-name: tracing-core
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-10 13:43:15 +00:00
0756642137
chore(deps): Bump pbjson-build from 0.2.2 to 0.2.3 ( #4002 )
...
Bumps [pbjson-build](https://github.com/influxdata/pbjson ) from 0.2.2 to 0.2.3.
- [Release notes](https://github.com/influxdata/pbjson/releases )
- [Commits](https://github.com/influxdata/pbjson/compare/0.2.2...0.2.3 )
---
updated-dependencies:
- dependency-name: pbjson-build
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-10 13:34:00 +00:00
2c3d30ca32
chore: Update datafusion, arrow, flight and parquet ( #4000 )
...
* chore: Update datafusion, arrow, flight and parquet
* fix: api change
* fix: fmt
* fix: update test metadata size
* fix: Update sizes in parquet test
* fix: more metadata size update
2022-03-10 12:24:47 +00:00
12771a7061
fix: allow manual merge messages in semantic commit checker ( #4011 )
2022-03-10 12:14:45 +00:00
dcdd9b079d
chore: Run dependabot more frequently ( #3998 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-10 11:40:09 +00:00
30d1c77d36
feat: querier test system, ground work ( #3991 )
...
* feat: querier test system, ground work
See #3985 for the motivation.
This introduces a cache system for the querier which can later be
extended to support the remaining features listed in #3985 (e.g.
metrics, LRU/TTL).
All current caches are wired up to go throw the new cache system. Once
we move away from (ab)using `db`, the set of caches will be different
but the system will remain.
* test: explain it
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
* refactor: simplify cache result broadcast
* refactor: introduce `Loader` crate
* fix: docs
* docs: explain why we manually drop removed hashmap entries
* docs: fix intra-doc link
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2022-03-10 11:27:24 +00:00
59d6bee531
Merge pull request #3990 from influxdata/dom/log-spam
...
refactor: drop trace log verbosity
2022-03-09 20:18:02 +00:00
cec6af2949
Merge branch 'main' into dom/log-spam
2022-03-09 20:10:05 +00:00
944f628e29
fix: Remove data_types as a dependency of ng compactor ( #3993 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-09 17:03:02 +00:00
a9512e0fbe
Merge pull request #3905 from influxdata/cn/ng-end-to-end
...
test: Add some end-to-end ish NG tests
2022-03-09 16:54:25 +00:00
1f474bfbf0
test: Create the test database before running postgres iox_catalog tests
2022-03-09 10:43:30 -05:00
c891f3c4f2
fix: Print an error with sample env var variable if unset
2022-03-09 09:55:43 -05:00
8af2f60b59
fix: Run catalog setup as part of end-to-end test setup
2022-03-09 09:55:43 -05:00
93b0cdbcc4
fix: Create the test database as part of ng server fixture startup
2022-03-09 09:55:43 -05:00
e4fb227c6e
feat: Set the catalog URL explicitly in the test config
2022-03-09 09:55:43 -05:00
465fb0272d
fix: Remove unneeded server id const
2022-03-09 09:55:43 -05:00
6d086705a8
fix: Remove ng create_shared server fixture method for now
2022-03-09 09:55:42 -05:00
e315012fe3
fix: Switch NG tests to use TEST_INFLUXDB_IOX_CATALOG_DSN too
2022-03-09 09:55:42 -05:00
a14e642f39
refactor: Extract NG end-to-end tests from OG end-to-end tests
2022-03-09 09:55:42 -05:00
1536bdeca0
test: Add an end-to-end NG test
2022-03-09 09:55:42 -05:00
880344494a
fix: Remove reference to AWS from postgres test comment
2022-03-09 09:55:42 -05:00
66a5649258
test: Print out the server type with server log messages
...
So that when you have more than one server running in a test, it's
easier to see which one is saying what
2022-03-09 09:55:42 -05:00
ae45d9f750
test: Add Router2 as a supported ServerFixture type
2022-03-09 09:55:42 -05:00
b6eb61831b
Merge pull request #3944 from influxdata/cn/not-database-url
...
fix: Use more consistent env vars for database connections
2022-03-09 13:42:12 +00:00
caba70f871
Merge branch 'main' into cn/not-database-url
2022-03-09 13:32:02 +00:00
d2c0acdd46
refactor: Remove serving readiness gate ( #3986 )
...
* refactor: Remove serving_readiness
* fix: remove more
* fix: remove test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-09 12:17:44 +00:00
874be0097d
refactor: eliminate unnecessary lock
...
This commit splits the API of the LifecycleManager into two:
* LifecycleManager: singleton responsible for evaluating partitions
and running persist tasks.
* LifecycleHandle: a handle for each sequencer ingester(s) to update
the global LifecycleManager state when applying ops.
This keeps the accessible API & responsibilities of each caller distinct
and allows us to leverage the type system to enforce linearisation of
calls to LifecycleManager::maybe_persist() without resorting to an
(unnecessary) mutex guard for serialisation.
2022-03-09 11:16:44 +00:00
8af9330cd7
refactor: drop trace log verbosity
...
This is very shouty, logging one INFO per HTTP request - the router logs
are almost all this one line repeated.
2022-03-09 10:38:20 +00:00
c6cab3538f
refactor: move parquet chunk's new and decode to parquet_file crate ( #3987 )
2022-03-08 22:04:32 +00:00
96100635c3
feat: ingester seeks kafka partition on initialization ( #3940 )
...
Fixes #3851
2022-03-08 19:53:35 +00:00
09fba1d2c0
feat: NG Compactor - main function for finding and compacting parquet files ( #3973 )
...
* feat: main function for finding and compacting parquet files
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* refactor: rename file and struct
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-08 16:34:43 +00:00
e4e993f925
Merge pull request #3982 from influxdata/dom/ingester-debug-impls
...
refactor: impl Debug for ingester types
2022-03-08 16:00:18 +00:00
235049b7a3
Merge branch 'main' into dom/ingester-debug-impls
2022-03-08 15:50:05 +00:00
e26eabf8ee
Merge pull request #3983 from influxdata/dom/regex-bump
...
build: bump regex
2022-03-08 15:49:10 +00:00
3278e35b41
build: bump regex
...
Bump regex version due to RUSTSEC-2022-0013, see:
https://rustsec.org/advisories/RUSTSEC-2022-0013
2022-03-08 15:37:23 +00:00
384133aac7
refactor: impl Debug for ingester types
...
Derive / implement Debug for all types in the ingester crate to help
with debugging, and add a lint at "warn" level to keep everything in
sync.
2022-03-08 15:19:29 +00:00
357ea52d8d
refactor: decouple router2 setup and creation logic from the config structs, switch to `Arc<dyn ServerType>` ( #3969 )
...
* refactor: Extract router server instantiation into `influxdb_ioxd`
* refactor: complete dyn ServerType
* fix: remove some leftovers
* fix: clippy
* fix: restore startup message order
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-08 14:06:18 +00:00
78b015b923
Merge pull request #3978 from influxdata/dom/ingester-indexes
...
perf: get_table_persist_info indexes for joins
2022-03-08 13:53:48 +00:00
d31576b90c
perf: get_table_persist_info indexes for joins
...
Adds indexes to the JOINed fields to reduce execution cost, as the
TableRepo::get_table_persist_info() is currently by far the most
expensive catalog operation.
2022-03-08 12:12:47 +00:00
77f6153f72
refactor: remove `QueryDatabase::chunk_summaries` ( #3977 )
...
- This is not used by the query engine at all.
- The query engine should not care about ALL chunks but only about the
chunks it gets via `QueryDatabase::chunks` (which includes a table
name and a predicate).
- All other users of that API are NOT really query-related.
2022-03-08 11:34:26 +00:00
5cc1c697fc
refactor: remove `QueryDatabase::partition_addr` ( #3976 )
...
- This was not actually used by the query engine.
- The query engine doesn't have a concept of a "partition", it only
cares about chunks.
- Unbound access to all partitions in the database is quite expensive
(esp. on NG).
2022-03-08 11:17:31 +00:00
a3e952847a
refactor: split `querier::namespace` into submodules ( #3975 )
...
This makes it easier to see what's required to support the query
interface.
2022-03-08 10:49:00 +00:00
db3f1e8db7
feat: wire up tombstones into querier ( #3962 )
...
* feat: `TombstoneRepo::list_by_namespace`
* test: model sequencer properly
* feat: wire up tombstones into querier
Closes #3932 .
* refactor: `override_delete_predicates` => `set_delete_predicates`
2022-03-08 10:06:22 +00:00
7fa17ef1d7
refactor: remove unused error enums ( #3970 )
...
* refactor: remove unused error enums
* fix: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-07 21:40:55 +00:00
337e432a0f
feat: ingester persists on cold partitions ( #3942 )
...
Add configuration and lifecycle to trigger partition persistence if it hasn't received a write in a given number of secods.
Fixes #3869
2022-03-07 18:55:56 +00:00
688c1de5af
feat: add distinct count to chunk summary table ( #3966 )
...
* feat: add distinct count to chunk summary table
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-07 18:16:08 +00:00
dependabot[bot]
Andrew Lamb
Marco Neumann
kodiakhq[bot]
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
Dom Dwyer
Nga Tran
Paul Dix
Raphael Taylor-Davies