20bbb88dc5
refactor: remove table name from `TableSummary` ( #4170 )
...
This allows us to remove the table name from the low-level chunk
representations (like `ParquetFile`, RUB, ...) since table names are
already tracked by the higher-level data structures (e.g. catalog,
catalog chunk) that manage the low-level chunk representations.
This is similar to #4167 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-30 13:24:00 +00:00
bc01121c51
refactor(router2): no Arc for instrumentation
...
Removes needless Arc for the InstrumentationDecorator.
2022-03-30 13:02:21 +00:00
91730d6a1c
feat: schema validation conflict/limit metrics
...
Emit metric counters tracking the number of schema conflicts, and number
of service limit errors observed.
2022-03-30 11:42:58 +00:00
e7ab1908e5
fix: Use a bigger circleci machine for build_dev ( #4171 )
2022-03-30 09:59:07 +00:00
036626a576
refactor: remove partition key from `ParquetChunk` ( #4167 )
...
The parquet chunk is always wrapped into some higher-level data
structure (e.g. a catalog chunk, a partition, ...) that knows exactly
"where" the chunk is located. There is no need for the parquet chunk to
back-reference container-level attributes. In the contrary:
double-bookkeeping makes the code more complex and costs additional
memory.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-30 09:24:56 +00:00
f3aad7a731
feat: Capture more extreme tail latency buckets for dml handler ( #4168 )
2022-03-30 09:16:33 +00:00
9950bcee27
chore(deps): Bump indexmap from 1.8.0 to 1.8.1 ( #4166 )
...
Bumps [indexmap](https://github.com/bluss/indexmap ) from 1.8.0 to 1.8.1.
- [Release notes](https://github.com/bluss/indexmap/releases )
- [Changelog](https://github.com/bluss/indexmap/blob/master/RELEASES.md )
- [Commits](https://github.com/bluss/indexmap/compare/1.8.0...1.8.1 )
---
updated-dependencies:
- dependency-name: indexmap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-30 08:49:07 +00:00
bfd5568acf
fix: make sure the QueryableParquetChunks are always sorted correctly ( #4163 )
...
* fix: make sure the chunks are always sorted correctly
* fix: output
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* refactor: make new function for new chunk id
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-29 21:36:45 +00:00
2b76c31157
refactor: make statistics null counts optional ( #4160 )
...
Min/max values and distinct counts are already optional, so let's make
the null counts optional as well. This will be helpful for NG to deal w/
partial statistics (e.g. we only populate stats for the time column).
Note that the total count is still mandatory, but we normally have the
chunk/file-level row count at hand.
2022-03-29 17:47:57 +00:00
7d947c79d5
refactor: small query tests clean up ( #4156 )
...
* refactor: make NG query test generation more flexible
* refactor: rename OG-specfic query tests
* docs: explain chunk stage generation in NG query tests
* fix: typo
2022-03-29 14:00:34 +00:00
4ca52e5ae0
refactor: Extract common, OG database and router out of influxdb_ioxd ( #4149 )
...
* refactor: Extract common, OG database and router out of influxdb_ioxd
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-29 13:07:19 +00:00
99b6be9c2b
Merge pull request #4131 from influxdata/cn/delete-old
...
feat: Clean up old parquet files in the catalog and object storage
2022-03-29 12:37:06 +00:00
db5cd70c77
fix: logical merge conflict, unused import
2022-03-29 08:29:23 -04:00
79447aed33
fix: Logical merge conflict, missing namespace_id in test setup
2022-03-29 08:28:51 -04:00
5c8a80dca6
fix: Add an index to parquet_file to_delete
2022-03-29 08:15:26 -04:00
4a51c9eda6
feat: Add a garbage collector to be called in a background loop
...
Fixes #3954 .
2022-03-29 08:15:26 -04:00
f3f792fd08
feat: Add namespace_id to the parquet_files table; object store paths need it
2022-03-29 08:15:26 -04:00
a373c90415
refactor: Extract the list_all function to object store
...
I'm about to use this in a third file, so time to extract this.
Make it clear that this is appropriate for tests only.
2022-03-29 08:15:24 -04:00
39a1d1b26f
feat: Delete parquet files marked to be deleted before a specified time
...
Connects to #3954 .
2022-03-29 08:13:06 -04:00
decd018a6a
refactor: remove querier sync loop ( #4146 )
...
Namespaces are now created on demand and contain their full schema.
Tombstones/chunks are created on demand during the query.
Closes #4123 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-29 11:47:23 +00:00
e154d77bf5
chore(deps): Bump getrandom from 0.2.5 to 0.2.6 ( #4155 )
...
Bumps [getrandom](https://github.com/rust-random/getrandom ) from 0.2.5 to 0.2.6.
- [Release notes](https://github.com/rust-random/getrandom/releases )
- [Changelog](https://github.com/rust-random/getrandom/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-random/getrandom/compare/v0.2.5...v0.2.6 )
---
updated-dependencies:
- dependency-name: getrandom
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-29 08:53:34 +00:00
17af5fcbd1
chore(deps): Bump tokio-util from 0.7.0 to 0.7.1 ( #4154 )
...
* chore(deps): Bump tokio-util from 0.7.0 to 0.7.1
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.7.0 to 0.7.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.7.0...tokio-util-0.7.1 )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-29 08:39:02 +00:00
a83ad1ce2b
chore(deps): Bump syn from 1.0.89 to 1.0.90 ( #4150 )
...
Bumps [syn](https://github.com/dtolnay/syn ) from 1.0.89 to 1.0.90.
- [Release notes](https://github.com/dtolnay/syn/releases )
- [Commits](https://github.com/dtolnay/syn/compare/1.0.89...1.0.90 )
---
updated-dependencies:
- dependency-name: syn
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-29 08:28:30 +00:00
ec88df6cb7
feat: cache namespace schemas ( #4142 )
...
* feat: `TestColumn` for IOx tests
* feat: cache namespace schemas
In preparation to remove the querier sync loop completely.
Ref #4123 .
2022-03-29 08:19:21 +00:00
80b7e9cce1
feat: delete fully processed tombstones & integration tests for find_and_compact ( #4116 )
...
* feat: remove fully processed tombstones
* test: first few tests
* fix: delete SQL
* fix: test how IN (...) works in PG
* fix: test how IN (?) works in PG
* fix: test how IN (?) works in PG
* fix: dynamically add IN (?, ?, ...)
* fix: dynamically add IN (?, ?, ...) & its dynamic values
* fix: add argument directly in the SQL
* test: more tests for catalog read and update functions
* chore: move a subfunction to make it easier to read)
* test: first test for find_can_compact but disabled due to bug
* test: integration tests and a bug fix for find_and_compact
* chore: cleanup
* refactor: address review comments
* fix: put 2 delete processed tombstones and tombstones in a transaction
2022-03-28 18:35:54 +00:00
fb186c6733
refactor: `QueryDatabaseProvider::db` should be async ( #4143 )
...
This is required to fetch querier namespaces on demand.
Ref #4123 .
2022-03-28 11:18:20 +00:00
4f9515ffba
chore(deps): Bump async-trait from 0.1.52 to 0.1.53 ( #4141 )
...
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.52 to 0.1.53.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.52...0.1.53 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-28 08:55:24 +00:00
e5bbc74f7a
chore(deps): Bump paste from 1.0.6 to 1.0.7 ( #4140 )
...
Bumps [paste](https://github.com/dtolnay/paste ) from 1.0.6 to 1.0.7.
- [Release notes](https://github.com/dtolnay/paste/releases )
- [Commits](https://github.com/dtolnay/paste/compare/1.0.6...1.0.7 )
---
updated-dependencies:
- dependency-name: paste
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-28 08:44:01 +00:00
cabaf7c0e2
refactor: move object store configuration to TestConfig ( #4128 )
...
* refactor: move object store configuration to TestConfig
* fix: clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-25 18:19:44 +00:00
196ad50f5a
test: add e2e test for schema CLI ( #4132 )
...
* test: add e2e test for schema CLI
* fix: logical confict
2022-03-25 18:11:03 +00:00
58c630d709
chore: Update datafusion ( #4133 )
...
* chore: Update datafusion
* fix: typo
* fix: Update explain plan output
* fix: update Cargo.locl
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-25 15:08:39 +00:00
60983dda12
Merge pull request #4136 from influxdata/dom/service-limit-http-code
...
fix: service limit http code
2022-03-25 15:59:49 +01:00
ee8f436d45
Merge branch 'main' into dom/service-limit-http-code
2022-03-25 14:26:44 +00:00
9e6b139fd5
Merge pull request #4135 from influxdata/crepererum/issue4123a
...
refactor: clean up querier public interface
2022-03-25 13:58:26 +00:00
a8def72f47
Merge branch 'main' into crepererum/issue4123a
2022-03-25 13:49:53 +00:00
8e85846db6
refactor: lowercase error messages
...
Lowercases the error messages in the big iox_catalog Error enum for
better composition of messages (no random capitalisation in
glued-together strings, which is common with wrapped errors).
2022-03-25 11:33:27 +00:00
c6af2f5931
refactor: explicit schema validation errors
...
This commit adds new service limit & schema conflict error variants to
the SchemaError enum, allowing less error-prone matching higher up the
stack and more explicit log messages for both.
HTTP status codes are now correctly reported as 500 for internal catalog
errors during schema validation.
2022-03-25 11:33:27 +00:00
1edc4ca567
refactor: emit HTTP 429 for service limits
...
When a user hits the service limit, emit a "HTTP 429 Too Many Requests"
error with no "Retry-After" header. Reference:
https://docs.influxdata.com/influxdb/cloud/account-management/limits/#api-error-responses
2022-03-25 11:29:32 +00:00
9886ff42cc
refactor: clean up querier public interface
2022-03-25 11:54:52 +01:00
90dfca3c39
Merge pull request #4134 from influxdata/dom/file-write-buffer-loop
...
fix: maybe_auto_create_directories infinite loop
2022-03-25 10:42:44 +00:00
ce08292b89
fix: maybe_auto_create_directories infinite loop
...
Fixes symlinking in maybe_auto_create_directories() - previously it
would create a symlink specifying the target path relative to the
working dir, and not relative to the symlink.
If the working dir != the path in the WriteBufferCreationConfig,
subsequent calls would get stuck in an infinite loop attempting to
resolve the bad symlink.
2022-03-25 10:34:16 +00:00
942c3198f6
refactor: pass through backoff config to querier table
2022-03-25 11:18:45 +01:00
3158aa1d53
Merge pull request #4130 from influxdata/dom/revert-revert-revert
...
fix: propagate shutdown requests & compactor join() fix
2022-03-24 18:44:49 +00:00
ec7226d855
Merge branch 'main' into dom/revert-revert-revert
2022-03-24 18:37:08 +00:00
e222acbb48
refactor: use typed `TestConfig` rather than environment variable names for NG end to end tests ( #4126 )
...
* refactor: move environment variable mapping in end to end tests into TestConfig
* fix: clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-24 18:36:47 +00:00
15a9108135
Merge branch 'main' into dom/revert-revert-revert
2022-03-24 16:38:06 +00:00
39d9f30f12
refactor: Make `server` an implementation detail ( #4122 )
2022-03-24 15:58:04 +00:00
bf782de421
fix: compactor early shutdown
...
The compactor stub code would wait on nothing when the caller waited on
join()-ing the compactor handler, and this meant any caller who blocked
on join() would immediately return.
2022-03-24 15:58:02 +00:00
5c69a3f43b
chore: Update deps: datafusion, arrow/arrow-flight/parquet to 11, zstd to 0.11 ( #4119 )
...
* chore: update datafusion
* chore(deps): Bump arrow from 10.0.0 to 11.0.0
Bumps [arrow](https://github.com/apache/arrow-rs ) from 10.0.0 to 11.0.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/10.0.0...11.0.0 )
---
updated-dependencies:
- dependency-name: arrow
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): Bump arrow-flight from 10.0.0 to 11.0.0
Bumps [arrow-flight](https://github.com/apache/arrow-rs ) from 10.0.0 to 11.0.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/10.0.0...11.0.0 )
---
updated-dependencies:
- dependency-name: arrow-flight
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: update parquet to 11.0.0
* fix: error on create schema, test for same
* fix: upgrade zstd
* chore: Run cargo hakari tasks
* fix: fix logical merge conflict
* fix: hakari
* fix: hakari
* fix: update newly introduced dep
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-24 15:27:36 +00:00
1ca9d28fee
fix: remove unecessary test setup ( #4127 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-24 15:17:17 +00:00
Marco Neumann
Dom Dwyer
Marko Mikulicic
dependabot[bot]
Nga Tran
Andrew Lamb
kodiakhq[bot]
Carol (Nichols || Goulding)