cafe37bd1f
Merge branch 'pd/influxdb3-oss'
2023-09-21 09:15:41 -04:00
0a7ae5603f
feat: make sory_key_ids non-optional ( #8750 )
...
* feat: make sory_key_ids non-optional
* refactor: address review comments
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-09-19 14:22:29 +00:00
ada3d310d9
refactor: Use the newtypes in the catalog functions for updating service limits
2023-09-15 16:23:31 -04:00
5a74d09194
fix: Use the newtypes in NamespaceServiceProtectionLimitsOverride
2023-09-15 16:23:31 -04:00
d56217a8f6
refactor: Define the default limits on the types, not the catalog
2023-09-15 16:23:30 -04:00
c32a04388c
feat: Wrap max tables and max columns per table values in newtypes
2023-09-15 13:09:36 -04:00
fb351ac3e1
test: Add a test encoding expected behavior of validate_or_insert_schema ( #8738 )
...
I was confused about whether validate_or_insert_schema should return all
columns a table has in the catalog if another process has added some.
Dom explained that no, this is by design-- the validate_or_insert_schema
function shouldn't be fetching any extra columns from the catalog, only
inserting missing columns from the diff set being processed during a
write.
The NamespaceCache/gossip system takes care of eventually converging
schemas at a higher level.
To avoid anyone having to go through the understanding path I just did,
encode this expected behavior in a test for future reference.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-09-15 14:36:19 +00:00
a86c77213e
feat: compare sort_key_ids in cas_sort_key ( #8579 )
...
* feat: have ingester's SortKeyState include sort_key_ids
* fix: test failures
* chore: address review comments
* feat: first step to compare sort_key_ids
* feat: compare sort_key_ids in cas_sort_key
* fix: comment typos
* refactor: use direct assert instead of going true a function
* chore: fix typo
* test: add tests and comments
* chore: fix typos
* test: add more test to handle empty sort key
* chore: address review comments
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-09-12 15:10:57 +00:00
fdffa871c3
feat: Optionally specify a table name to get just its schema ( #8650 )
...
Rather than always having to request all of a namespace's schema then
filtering to the one you want. Will make this more consistent with
upserting schema by namespace+table.
Fixes #4997 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-09-06 13:33:25 +00:00
de6f710c31
chore: reland have ingester's SortKeyState include sort_key_ids ( #8678 )
2023-09-06 12:01:07 +00:00
7380d76993
chore: prepare for revert just in case
2023-09-05 17:40:24 -04:00
9af06dee9e
feat: have ingester's SortKeyState include sort_key_ids ( #8556 )
...
* feat: have ingester's SortKeyState include sort_key_ids
* fix: test failures
* chore: address review comments
* chore: address review comments by asding asserts to catch bugs if any
* chore: fix typo
* test: get column IDs for the tests
* refactor: reuse function
* chore: address review comments
2023-09-05 20:41:15 +00:00
77f340f964
chore(deps): Bump thiserror from 1.0.47 to 1.0.48 ( #8658 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.47 to 1.0.48.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.47...1.0.48 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-04 09:21:22 +00:00
1f732595d1
feat: re-introduce sqlx pool "used" counter ( #8624 )
...
This was removed in #8336 because the tracking was broken. However
having an "almost right" metric is still helpful because it helps with
dashboarding (see code comment).
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-31 16:18:36 +00:00
5af22cefe6
fix: add assertion migration for #8616 ( #8623 )
...
We don't think that there are any dirty clusters left, but better be
safe than sorry.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-31 15:07:18 +00:00
12b8095c46
feat: Upgrade to Rust 1.72.0 ( #8589 )
...
* feat: Upgrade to Rust 1.72.0
* fix: Allow a warning about an error we're intentionally creating
This is a test for an error. This lint warns that this code will cause
an error. Thanks lint, that's what we wanted!
* chore: rustfmt 1.72
* fix: Remove unnecessary hashes in raw string literals
Thanks Clippy!
https://rust-lang.github.io/rust-clippy/master/index.html#/needless_raw_string_hashes
Note that there are a number of false negatives with this lint; see
https://github.com/rust-lang/rust-clippy/issues/11420
* fix: Remove unnecessary explicit iteration
Looks like clippy::explicit_iter_loop was improved.
https://rust-lang.github.io/rust-clippy/master/index.html#/explicit_iter_loop
* fix: Allow clippy::manual_try_fold in a few places
Some of these might not be possible to rewrite with try_fold, or at
least not trivially. I don't feel confident enough to change these, in
any case. I think the lint is good to have on for future code though, so
that new code can be written with try_fold.
* fix: Remove useless creation of vectors when an array will do
Mostly in tests. Also fix some long lines.
Thanks Clippy!
https://rust-lang.github.io/rust-clippy/master/index.html#/useless_vec
* fix: Allow a single range in a vec init, which is actually what we want
Looks like Clippy's trying to catch a common mistake here, but for realz
we actually want `Vec<Range<usize>>` not `Vec<usize>`
https://rust-lang.github.io/rust-clippy/master/index.html#/single_range_in_vec_init
* fix: Remove a useless conversion
This looks like removing explicit iteration, but it's actually caught by
useless_conversion.
https://rust-lang.github.io/rust-clippy/master/index.html#/useless_conversion
* fix: Remove redundant pattern matching
Thanks Clippy!
https://rust-lang.github.io/rust-clippy/master/index.html#/redundant_pat
* fix: Allow an unwrap on a literal None in a test
This matches with the other tests better, and also when I tried to
remove the `unwrap_or_default` it changed the JSON sent from something
with an empty value to `null`, so I think the `or_default` part is
actually changing from one `None` to another `None`.
https://rust-lang.github.io/rust-clippy/master/index.html#/unnecessary_literal_unwrap
2023-08-29 05:57:38 +00:00
7b37479cd1
chore(deps): Bump siphasher from 0.3.10 to 1.0.0 ( #8561 )
...
Bumps [siphasher](https://github.com/jedisct1/rust-siphash ) from 0.3.10 to 1.0.0.
- [Commits](https://github.com/jedisct1/rust-siphash/compare/0.3.10...1.0.0 )
---
updated-dependencies:
- dependency-name: siphasher
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-24 09:13:14 +00:00
92c0e0c806
feat: re-land #8491 ( #8543 )
...
Includes the whole migration set again:
1. **create helper index:** This may or may not exist in prod depending
on when #8540 landed. This is OK, spend a bit of work here.
2. **actual migration:** The bit that originally failed, but now w/ an
extra step to migrate empty keys (because the `unnest`+`array_agg`
logic only works for non-empty keys and is a no-op for empty ones).
This is idempotent (i.e. a no-op) if we already finished the
migration.
3. **remove helper index:** Remove the helper index from (1).
4. **enfore `NOT NULL`:** This is a no-op if this was already applied.
Closes #8542 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-24 08:14:32 +00:00
cf0185f2a3
revert: #8491 ( #8540 )
...
This reverts #8491 by nullifying the problematic migration and the
aftermath migrations.
2023-08-22 10:35:52 +00:00
d8447779a2
feat: migrate name-based sort keys to ID based on in PG ( #8491 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-22 09:11:43 +00:00
3e98f7ea5c
feat: fill sort_key_ids when partition is inserted and updated ( #8517 )
...
* feat: read null sort_key_ids
* chore: clearer explanation about test strategy
* chore: Apply suggestions from code review
Co-authored-by: Marco Neumann <marco@crepererum.net>
* test: tests that add partition with NULL sort_key_ids
* feat: set sort_key_ids to empty array {} during partition insertion
* feat: initial step to update sort_key_ids
* chore: address review comments
* chore: remove unecessary comments and tests
* fix: typos
* chore: remove unecessary tests
* feat: continue the work of updating sort_key_ids
* fix: chec duplicates for SortedColumnSet
* test: tests for sort ley ids
* test: fix a test
* chore: remove unused comments
* chore: address first half of review comments and removing tests of tests
* chore: address review commnets for fetching colums in ingester
---------
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-21 14:26:57 +00:00
8263c4f3cf
feat: implement new ingest structure
...
Define new proto for the structure that gets sent from router to ingester and persisted in the ingester WAL.
Create ingest_structure crate with functions to convert from line protocol to new proto structure while validating schema.
Add function to convert new proto structure to RecordBatch.
2023-08-20 17:31:52 -04:00
5d17a99dbb
feat: read null sort_key_ids ( #8489 )
...
* feat: read null sort_key_ids
* chore: clearer explanation about test strategy
* chore: Apply suggestions from code review
Co-authored-by: Marco Neumann <marco@crepererum.net>
* test: tests that add partition with NULL sort_key_ids
* chore: address review comments
* chore: remove unecessary comments and tests
* fix: typos
* chore: remove unecessary tests
* fix: chec duplicates for SortedColumnSet
---------
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-18 14:15:27 +00:00
755d461cca
docs: Clarify requirements of the `list_old_style` function
...
Namely, that it must not have a `LIMIT`.
Co-authored-by: Dom <dom@itsallbroken.com>
2023-08-18 09:21:55 -04:00
99d12ef600
feat: Add a catalog method to get all old-style partitions
...
To be fed to the bloom filter.
2023-08-18 09:19:25 -04:00
d2c71bfe67
chore(deps): Bump thiserror from 1.0.46 to 1.0.47 ( #8519 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.46 to 1.0.47.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.46...1.0.47 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-18 09:02:48 +00:00
7094189004
chore(deps): Bump tokio from 1.31.0 to 1.32.0 ( #8507 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.31.0 to 1.32.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.31.0...tokio-1.32.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-17 08:06:29 +00:00
fff313b80c
chore(deps): Bump thiserror from 1.0.44 to 1.0.46 ( #8496 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.44 to 1.0.46.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.44...1.0.46 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-16 10:54:47 +00:00
34b8585931
chore(deps): Bump tokio from 1.30.0 to 1.31.0 ( #8482 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.30.0 to 1.31.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.30.0...tokio-1.31.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-14 06:32:34 +00:00
4c63338354
chore(deps): Bump async-trait from 0.1.72 to 0.1.73 ( #8481 )
...
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.72 to 0.1.73.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.72...0.1.73 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-14 06:25:33 +00:00
9bf1c8c11c
chore: revert fill sort_key_ids
2023-08-11 11:36:27 -04:00
d38882c746
Merge branch 'main' into cn/cleanups
2023-08-11 13:37:44 +00:00
da92a5c9e1
feat: fill catalog `sort_key_ids` for partitions with coming data ( #8462 )
...
* feat: fill catalog sort_key_ids for partition with coming data
* test: sort_key_ids has empty array for newly create partition
* test: name of non-existing column
* chore: add comments to ask Andrew about the code
* chore: make comments clearer
* chore: fix a comment to avoid failure in doc
* chore: add comment for the panic if column name of sort key not found
* fix: during import files the partition has to be created with empty sort key first. Then after its files are created, the partition will be uodated with sort key
* chore: remove no longer needed comments after the bug in build_catalog test is fixed
* chore: address review comments
* refactor: Use ColumnSet type
* chore: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* chore: fix a clippy
---------
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2023-08-10 18:12:40 +00:00
3675043585
chore(deps): Bump tokio from 1.29.1 to 1.30.0 ( #8464 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.29.1 to 1.30.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.29.1...tokio-1.30.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-10 07:50:18 +00:00
f1fbcca23c
fix: make concurrent index creations idempotent ( #8450 )
...
* fix: make concurrent index creations idempotent
`CREATE INDEX CONCURRENTLY` may fail and leave an invalid index behind.
See:
<https://www.postgresql.org/docs/current/sql-createindex.html#SQL-CREATEINDEX-CONCURRENTLY >
Invalid indexes will NOT be used for queries, however they is also useless and
technically the migration wasn't successful. Since #8407 we detect this
situation. #8394 allows us to fix these migrations. #8434 makes sure
that we don't mess this up (esp. that the "other checksum" pragmas are
correct).
For #7897 .
* fix: multi-line processing in `check_linear_migrations.sh`
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-09 19:16:35 +00:00
2c66fc9652
fix: Break up long lines/strings
2023-08-09 14:03:20 -04:00
c6d607991c
docs: Copyedit migration docs
...
Fix some typos, reword or add some words for clarity, add space after
headings for scanability
2023-08-09 14:02:53 -04:00
f847463108
fix: Remove duplicated 'not' in an error message
2023-08-09 13:34:36 -04:00
21eb57953b
fix: Add missing word to an error message
2023-08-09 13:34:16 -04:00
71428b3274
docs: Correct adjective form in a comment
2023-08-09 13:33:49 -04:00
180e982c73
docs: Fix typo
2023-08-09 13:22:33 -04:00
530acf170c
test: more automatic tests for migrations ( #8457 )
...
* test: test self-tests
* test: more tests for migrations
2023-08-09 12:33:53 +00:00
2156bc7e05
feat: static migrations checks ( #8434 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-07 15:13:28 +00:00
922a3cd292
feat: harden migration version checks ( #8415 )
...
* refactor: clean up
* feat: harden migration version checks
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-07 07:59:24 +00:00
614b64d3e4
feat: retry failed migrations ( #8414 )
2023-08-04 11:15:05 +00:00
9803623543
feat: post-migration sanity checks ( #8407 )
2023-08-04 08:56:22 +00:00
975d9a1ae5
refactor: simplify code, harden PG locking ( #8406 )
...
* test: simplify code
* feat: `IOxMigration::single_transaction`
* refactor: unlock faster
* refactor: harden PG locking
Check that `unlock` is actually successful and add a bunch of tests.
2023-08-04 08:39:01 +00:00
0c9570889b
feat: allow fixing migrations ( #8394 )
...
We need to fix our non-idempotent `CREATE INDEX CONCURRENTLY` migrations
so they are self-healing. This is an essential part of #7897 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-04 08:05:45 +00:00
81b5d80a91
feat(idpe-17935): move filtering of skipped partitions to the scheduler ( #8358 )
...
* catalog.get_in_skipped_compaction() should handle for multiple partitions
* add the ability to perform transformation on sets of partitions (rather than filtering one by one). Start with the transformation to remove skipped partitions, in the scheduler.
* move the env var and cli flag setting, for when to ignore skipped partitions, to the scheduler config.
2023-08-03 11:43:09 -07:00
3fb173ec56
refactor: use errors instead of panic ( #8389 )
...
* refactor: use errors instead of panic
* fix: typo
2023-08-02 13:58:55 +00:00
Paul Dix
Nga Tran
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
dependabot[bot]
Marco Neumann
kodiakhq[bot]
wiedld