edda409b19
refactor: Extract `ioxd_test`, `ioxd_compactor`, `ioxd_ingester`; remove `ioxd` ( #4210 )
...
* refactor: Extract test, compactor, ingester, and test
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-03 10:42:22 +00:00
833c10c083
feat: return write_token from HTTP writes to router2 ( #4202 )
...
* feat: return write_token from HTTP writes to router2
* fix: Update router2/src/dml_handlers/instrumentation.rs
Co-authored-by: Dom <dom@itsallbroken.com>
* refactor: Use WriteSummary::default more vigorously
* fix: fix typo and add links to follow on issues
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-02 10:34:51 +00:00
77ad4a7dad
feat: replace a compactor constant with an CLI config param ( #4204 )
2022-04-01 17:50:43 +00:00
d37af1a7f5
fix: include git sha (again) in release build ( #4193 )
...
* fix: error if git-sha can not be found
* refactor: move main to influxdb_iox
* fix: fmt
2022-03-31 19:14:21 +00:00
532d227d11
refactor: extract router2 into ioxd_router2 ( #4183 )
...
* refactor: extract router2 from ioxd
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-31 17:39:05 +00:00
367e926d35
refactor: extract querier into ioxd_querier ( #4182 )
...
* refactor: extract querier into ioxd_querier
* fix: dep
2022-03-31 16:03:31 +00:00
a384448b92
refactor: rename Sequence::id and Sequence::number field names ( #4190 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-31 15:17:58 +00:00
a1df864283
feat: Support 'SHOW NAMESPACES' in sql repl ( #4164 )
...
* feat: Support `SHOW NAMESPACES` in sql repl
* feat: add basic support to clients
* fix: add get_namespaces service test
* fix: proper error handling
* test: end to end test for namespace client
* refactor: Use QuerierDatabase rather than Catalog
* refactor: remove unused function
2022-03-31 12:57:33 +00:00
04d961e70d
feat: wire up compactor scheduler and config ( #4139 )
...
Add configuration options for compactor for the max size of level 0 files and split percentage.
Add metrics for compaction to track the number of candidates, compactions, and durations.
Add functions to separate identifying partitions to compact from running compaction.
Make compaction run in smaller chunks, specifically per partition.
Update compaction to automatically promote level 0 files that are non-overlapping without waiting some period of time.
Closes #4120
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-30 17:45:24 +00:00
92da65a065
feat: Add end to end tests for querier and schema client ( #4178 )
...
* refactor: split up ingester schema test, add mini cluster
* feat: add schema cli test
* feat: add end to end test for querier
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-30 17:07:32 +00:00
7a437387d9
feat: add KeySortCapability capability ( #4176 )
2022-03-30 15:57:03 +00:00
f2a3dd58b2
refactor: rename `influxdb_ioxd` to `ioxd` ( #4162 )
...
* refactor: rename influxdb_ioxd to ioxd
* fix: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-30 13:31:27 +00:00
2b76c31157
refactor: make statistics null counts optional ( #4160 )
...
Min/max values and distinct counts are already optional, so let's make
the null counts optional as well. This will be helpful for NG to deal w/
partial statistics (e.g. we only populate stats for the time column).
Note that the total count is still mandatory, but we normally have the
chunk/file-level row count at hand.
2022-03-29 17:47:57 +00:00
4ca52e5ae0
refactor: Extract common, OG database and router out of influxdb_ioxd ( #4149 )
...
* refactor: Extract common, OG database and router out of influxdb_ioxd
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-29 13:07:19 +00:00
17af5fcbd1
chore(deps): Bump tokio-util from 0.7.0 to 0.7.1 ( #4154 )
...
* chore(deps): Bump tokio-util from 0.7.0 to 0.7.1
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.7.0 to 0.7.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.7.0...tokio-util-0.7.1 )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-29 08:39:02 +00:00
196ad50f5a
test: add e2e test for schema CLI ( #4132 )
...
* test: add e2e test for schema CLI
* fix: logical confict
2022-03-25 18:11:03 +00:00
9886ff42cc
refactor: clean up querier public interface
2022-03-25 11:54:52 +01:00
e222acbb48
refactor: use typed `TestConfig` rather than environment variable names for NG end to end tests ( #4126 )
...
* refactor: move environment variable mapping in end to end tests into TestConfig
* fix: clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-24 18:36:47 +00:00
39d9f30f12
refactor: Make `server` an implementation detail ( #4122 )
2022-03-24 15:58:04 +00:00
5c69a3f43b
chore: Update deps: datafusion, arrow/arrow-flight/parquet to 11, zstd to 0.11 ( #4119 )
...
* chore: update datafusion
* chore(deps): Bump arrow from 10.0.0 to 11.0.0
Bumps [arrow](https://github.com/apache/arrow-rs ) from 10.0.0 to 11.0.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/10.0.0...11.0.0 )
---
updated-dependencies:
- dependency-name: arrow
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): Bump arrow-flight from 10.0.0 to 11.0.0
Bumps [arrow-flight](https://github.com/apache/arrow-rs ) from 10.0.0 to 11.0.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/10.0.0...11.0.0 )
---
updated-dependencies:
- dependency-name: arrow-flight
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: update parquet to 11.0.0
* fix: error on create schema, test for same
* fix: upgrade zstd
* chore: Run cargo hakari tasks
* fix: fix logical merge conflict
* fix: hakari
* fix: hakari
* fix: update newly introduced dep
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-24 15:27:36 +00:00
fb75ef7b82
test: add end to end test for all in one mode, restructure fixture ( #4114 )
...
* test: add end to end test for all in one mode, restructure fixture
* docs: fix typos and clarify schema requrements
2022-03-24 12:53:25 +00:00
7f2c2fde2c
fix: fix all in one mode argument handling so it can start ( #4115 )
...
* fix: fix all in one mode argument handling
* fix: clippy
* fix: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-23 18:16:22 +00:00
4f5321d19b
feat: add compactor configuration for kafka topic and sequencers ( #4107 )
2022-03-23 14:11:47 +00:00
e109fa4987
feat: schema client and CLI ( #4105 )
...
* feat: schema client and CLI
* chore: clarification in comment in schema command
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-23 13:49:24 +00:00
b83b000590
chore: Update datafusion ( #4071 )
...
* chore: update to datafusion 5936edc2a94d5fb20702a41eab2b80695961b9dc
* chore: Update apis to match datafusion changes
2022-03-22 13:17:41 +00:00
6fb1a9b592
feat(querier): enable object store metrics
2022-03-15 16:32:52 +00:00
f4d836eed7
feat(ingester): enable object store metrics
2022-03-15 16:32:52 +00:00
65273721b6
feat(compactor): enable object store metrics
2022-03-15 16:32:52 +00:00
5585dd3c21
refactor: switch to using DynObjectStore
...
Changes all consumers of the object store to use the dynamically
dispatched DynObjectStore type, instead of using a hardcoded concrete
implementation type.
2022-03-15 16:32:52 +00:00
1d5066c421
refactor: rename ObjectStore -> ObjectStoreImpl
...
Frees up the name for so we can use `dyn ObjectStore` throughout the
code instead of `ObjectStoreApi`.
2022-03-15 16:29:43 +00:00
9b3f946c10
feat: all in 1 IOx NG mode ( #3965 )
...
* feat: Add all_in_one mode
* fix: doc
* docs: fix truncated docs
* refactor: correctly identify PG connections
* refactor: resolve failed merge
Co-authored-by: Dom Dwyer <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-15 16:28:37 +00:00
5a29d070ea
feat: Implement the compact function for NG Compactor ( #4001 )
...
* feat: initial implementation of compact a given list of overlapped parquet files
* feat: Add QueryableParquetChunk and some refactoring
* feat: build queryable parquet chunks for parquet files with tombstones
* feat: second half the implementation for Compactor's compact. Tests will be next
* fix: comments for trait funnctions fof QueryChunkMeta
* test: add tests for compactor's compact function
* fix: typos
* refactor: address Jake's review comments
* refactor: address Andrew's comments and add one more test for files in different order in the vector
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-11 20:25:19 +00:00
cc4875cca0
refactor: decouple ingester setup and creation logic from the config structs ( #4020 )
...
* refactor: decouple ingester setup and creation logic from the config structs
* fix: clippy
* refactor: remove comments
2022-03-11 19:25:50 +00:00
b24ae7d23b
refactor: extract out compactor creation from config ( #4018 )
...
* refactor: extract out compactor creation from config
* fix: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-11 14:46:34 +00:00
29df46975f
refactor: extract out querier creation from config ( #4017 )
...
* refactor: extract out querier creation from config
* fix: clippy
2022-03-11 14:38:09 +00:00
bd2824aa59
chore(deps): Bump pprof from 0.6.2 to 0.7.0 ( #4004 )
...
* chore(deps): Bump pprof from 0.6.2 to 0.7.0
Bumps [pprof](https://github.com/tikv/pprof-rs ) from 0.6.2 to 0.7.0.
- [Release notes](https://github.com/tikv/pprof-rs/releases )
- [Changelog](https://github.com/tikv/pprof-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/tikv/pprof-rs/commits )
---
updated-dependencies:
- dependency-name: pprof
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix: `pprof` features for version 0.7
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-11 11:08:59 +00:00
f03ebd79ab
refactor: move querier's test utils to a new crate to get reused by tests in other crates ( #4013 )
...
* refactor: move querier's test utils to a new crate to be able resued by tests in other crates
* chore: remove unused import
2022-03-10 18:17:58 +00:00
2c3d30ca32
chore: Update datafusion, arrow, flight and parquet ( #4000 )
...
* chore: Update datafusion, arrow, flight and parquet
* fix: api change
* fix: fmt
* fix: update test metadata size
* fix: Update sizes in parquet test
* fix: more metadata size update
2022-03-10 12:24:47 +00:00
c891f3c4f2
fix: Print an error with sample env var variable if unset
2022-03-09 09:55:43 -05:00
8af2f60b59
fix: Run catalog setup as part of end-to-end test setup
2022-03-09 09:55:43 -05:00
93b0cdbcc4
fix: Create the test database as part of ng server fixture startup
2022-03-09 09:55:43 -05:00
e4fb227c6e
feat: Set the catalog URL explicitly in the test config
2022-03-09 09:55:43 -05:00
465fb0272d
fix: Remove unneeded server id const
2022-03-09 09:55:43 -05:00
6d086705a8
fix: Remove ng create_shared server fixture method for now
2022-03-09 09:55:42 -05:00
e315012fe3
fix: Switch NG tests to use TEST_INFLUXDB_IOX_CATALOG_DSN too
2022-03-09 09:55:42 -05:00
a14e642f39
refactor: Extract NG end-to-end tests from OG end-to-end tests
2022-03-09 09:55:42 -05:00
1536bdeca0
test: Add an end-to-end NG test
2022-03-09 09:55:42 -05:00
66a5649258
test: Print out the server type with server log messages
...
So that when you have more than one server running in a test, it's
easier to see which one is saying what
2022-03-09 09:55:42 -05:00
ae45d9f750
test: Add Router2 as a supported ServerFixture type
2022-03-09 09:55:42 -05:00
d2c0acdd46
refactor: Remove serving readiness gate ( #3986 )
...
* refactor: Remove serving_readiness
* fix: remove more
* fix: remove test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-09 12:17:44 +00:00
Andrew Lamb
Nga Tran
Paul Dix
Edd Robinson
Marco Neumann
dependabot[bot]
Luke Bond
Dom Dwyer
Carol (Nichols || Goulding)