3e98f7ea5c
feat: fill sort_key_ids when partition is inserted and updated ( #8517 )
...
* feat: read null sort_key_ids
* chore: clearer explanation about test strategy
* chore: Apply suggestions from code review
Co-authored-by: Marco Neumann <marco@crepererum.net>
* test: tests that add partition with NULL sort_key_ids
* feat: set sort_key_ids to empty array {} during partition insertion
* feat: initial step to update sort_key_ids
* chore: address review comments
* chore: remove unecessary comments and tests
* fix: typos
* chore: remove unecessary tests
* feat: continue the work of updating sort_key_ids
* fix: chec duplicates for SortedColumnSet
* test: tests for sort ley ids
* test: fix a test
* chore: remove unused comments
* chore: address first half of review comments and removing tests of tests
* chore: address review commnets for fetching colums in ingester
---------
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-21 14:26:57 +00:00
12eac3378f
Merge branch 'main' into cn/import-export-cleanup
2023-08-18 11:06:34 +00:00
d2c71bfe67
chore(deps): Bump thiserror from 1.0.46 to 1.0.47 ( #8519 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.46 to 1.0.47.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.46...1.0.47 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-18 09:02:48 +00:00
8f854f46f2
fix: Add missing negative in log message
2023-08-17 15:12:14 -04:00
c11253d8ad
fix: Break up long line
2023-08-17 15:12:14 -04:00
fc18840bba
fix: Remove some commented-out code
2023-08-17 15:12:13 -04:00
7094189004
chore(deps): Bump tokio from 1.31.0 to 1.32.0 ( #8507 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.31.0 to 1.32.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.31.0...tokio-1.32.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-17 08:06:29 +00:00
33900936da
chore(deps): Bump serde_json from 1.0.104 to 1.0.105
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.104 to 1.0.105.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.104...v1.0.105 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-08-16 11:00:44 +00:00
fff313b80c
chore(deps): Bump thiserror from 1.0.44 to 1.0.46 ( #8496 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.44 to 1.0.46.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.44...1.0.46 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-16 10:54:47 +00:00
34b8585931
chore(deps): Bump tokio from 1.30.0 to 1.31.0 ( #8482 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.30.0 to 1.31.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.30.0...tokio-1.31.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-14 06:32:34 +00:00
9bf1c8c11c
chore: revert fill sort_key_ids
2023-08-11 11:36:27 -04:00
da92a5c9e1
feat: fill catalog `sort_key_ids` for partitions with coming data ( #8462 )
...
* feat: fill catalog sort_key_ids for partition with coming data
* test: sort_key_ids has empty array for newly create partition
* test: name of non-existing column
* chore: add comments to ask Andrew about the code
* chore: make comments clearer
* chore: fix a comment to avoid failure in doc
* chore: add comment for the panic if column name of sort key not found
* fix: during import files the partition has to be created with empty sort key first. Then after its files are created, the partition will be uodated with sort key
* chore: remove no longer needed comments after the bug in build_catalog test is fixed
* chore: address review comments
* refactor: Use ColumnSet type
* chore: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* chore: fix a clippy
---------
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2023-08-10 18:12:40 +00:00
3675043585
chore(deps): Bump tokio from 1.29.1 to 1.30.0 ( #8464 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.29.1 to 1.30.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.29.1...tokio-1.30.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-10 07:50:18 +00:00
4a9e76b8b7
feat: Make parquet_file.partition_id optional in the catalog ( #8339 )
...
* feat: Make parquet_file.partition_id optional in the catalog
This will acquire a short lock on the table in postgres, per:
<https://stackoverflow.com/questions/52760971/will-making-column-nullable-lock-the-table-for-reads >
This allows us to persist data for new partitions and associate the
Parquet file catalog records with the partition records using only the
partition hash ID, rather than both that are used now.
* fix: Support transition partition ID in the catalog service
* fix: Use transition partition ID in import/export
This commit also removes support for the `--partition-id` flag of the
`influxdb_iox remote store get-table` command, which Andrew approved.
The `--partition-id` filter was getting the results of the catalog gRPC
service's query for Parquet files of a table and then keeping only the
files whose partition IDs matched. The gRPC query is no longer returning
the partition ID from the Parquet file table, and really, this command
should instead be using `GetParquetFilesByPartitionId` to only request
what's needed rather than filtering.
* feat: Support looking up Parquet files by either kind of Partition id
Regardless of which is actually stored on the Parquet file record.
That is, say there's a Partition in the catalog with:
Partition {
id: 3,
hash_id: abcdefg,
}
and a Parquet file that has:
ParquetFile {
partition_hash_id: abcdefg,
}
calling `list_by_partition_not_to_delete(PartitionId(3))` should still
return this Parquet file because it is associated with the partition
that has ID 3.
This is important for the compactor, which is currently only dealing in
PartitionIds, and I'd like to keep it that way for now to avoid having
to change Even More in this PR.
* fix: Use and set new partition ID fields everywhere they want to be
---------
Co-authored-by: Dom <dom@itsallbroken.com>
2023-07-31 12:40:56 +00:00
700830d7d3
chore(deps): Bump serde_json from 1.0.103 to 1.0.104 ( #8342 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.103 to 1.0.104.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.103...v1.0.104 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-27 09:12:35 +00:00
faa8d44492
chore(deps): Bump thiserror from 1.0.43 to 1.0.44 ( #8315 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.43 to 1.0.44.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.43...1.0.44 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-24 10:18:44 +00:00
b840ca3f1b
chore(deps): Bump serde_json from 1.0.102 to 1.0.103 ( #8240 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.102 to 1.0.103.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.102...v1.0.103 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-17 11:36:47 +00:00
e73116a122
Merge branch 'main' into cn/query-catalog-with-either-partition-identifier
2023-07-12 14:51:02 +00:00
d74f047b06
chore(deps): Bump serde_json from 1.0.100 to 1.0.102
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.100 to 1.0.102.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.100...v1.0.102 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-07-12 02:00:42 +00:00
c1e42651ec
feat: Abstract over which partition ID type we're using to compare and swap sort keys
2023-07-10 13:39:19 -04:00
048fc32bd5
feat: add `influxdb_iox debug build-catalog` command ( #8067 )
...
* feat: add `influxdb_iox debug build-catalog` command
* fix: tests
* fix: Use info! logs instead of println for status
* fix: Set partition_hash_id as well
* fix: remove leftover code
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-07 18:32:27 +00:00
057ee40cb9
chore(deps): Bump thiserror from 1.0.41 to 1.0.43 ( #8181 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.41 to 1.0.43.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.41...1.0.43 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-07 09:25:12 +00:00
3827257f94
chore(deps): Bump thiserror from 1.0.40 to 1.0.41 ( #8149 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.40 to 1.0.41.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.40...1.0.41 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-07-05 09:25:14 +00:00
7c771ce6f3
chore(deps): Bump serde_json from 1.0.99 to 1.0.100 ( #8147 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.99 to 1.0.100.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.99...v1.0.100 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-05 09:12:11 +00:00
b15c6062a9
chore(deps): Bump tokio from 1.28.2 to 1.29.0 ( #8100 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.28.2 to 1.29.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.28.2...tokio-1.29.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-28 13:18:08 +00:00
24ca0fef30
chore(deps): Bump serde_json from 1.0.97 to 1.0.99 ( #8076 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.97 to 1.0.99.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.97...v1.0.99 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-26 09:25:38 +00:00
123ba32fb9
chore(deps): Bump serde_json from 1.0.96 to 1.0.97 ( #8021 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.96 to 1.0.97.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.96...v1.0.97 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-20 08:53:15 +00:00
928a4d163e
build: remove unused dependencies from crates
...
This commit fixes loads of crates (47!) had unused dependencies, or
mis-configured dependencies (test deps as normal deps).
I added the "unused_crate_dependencies" to all crates to help prevent
this mess from growing again!
https://doc.rust-lang.org/beta/nightly-rustc/rustc_lint_defs/builtin/static.UNUSED_CRATE_DEPENDENCIES.html
This has the minor downside of false-positives when specifying
dev-dependencies for test/bench binaries - these are files in /test or
/benches (not normal tests). This commit includes a workaround,
importing them in lib.rs (gated by a feature flag). I think the
trade-off of better dependency management is worth it!
2023-05-23 14:55:43 +02:00
e97c48b388
feat: download catalog objects as part of `remote store get-table` ( #7779 )
...
* feat: download catalog objects as part of remote store get-table
* fix: fix future_not_send
* fix: reorder import order
* chore: reduce repetition of `Error` in naming
* fix: improve the partition filtering API
* fix: clarify export comment
* fix: make index printing consistent
* fix: improve skip message
* chore: add extra clippy lints to inmport_export crate
* fix: comments
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-18 16:52:43 +00:00
042a6a66d5
refactor: remove old tsm import code ( #7804 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-18 10:51:24 +00:00
c077dec0e0
refactor: rename `import` to `iox_import_export` ( #7799 )
...
* refactor: rename `import` crate to `import_export`
* chore: Consolidate tsm import/export
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-16 17:10:17 +00:00
Nga Tran
kodiakhq[bot]
dependabot[bot]
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
Andrew Lamb
Dom Dwyer