1ff11d0856
refactor: Change catalog configuration so it is entirely dsn based / support end to end testing without postgres ( #7736 )
...
* refactor: Change catalog configuration so it is entirely dsn based / support end to end testing without postgres
Restores code from https://github.com/influxdata/influxdb_iox/pull/7708
Revert "revert: PR #7708"
This reverts commit c9cfe05f8d
2023-05-17 13:36:25 +00:00
45628ba4d3
chore: Merge remote-tracking branch 'origin/main' into smith/remove-transactions-main
2023-05-16 08:49:25 -04:00
7268ea5c29
refactor: Extract a test helper function to create a basic table
2023-05-15 14:31:24 -04:00
57bedb1c2d
refactor: Extract a test helper function to create a basic namespace
2023-05-15 14:20:38 -04:00
09dc622078
chore: Merge remote-tracking branch 'origin/main' into smith/remove-transactions-main
2023-05-15 10:06:11 -04:00
5fe8affb18
refactor: accept NamespaceName with Namespace create ( #7774 )
...
Co-authored-by: Dom <dom@itsallbroken.com>
2023-05-15 10:03:55 +00:00
e5ee47c32e
chore: code cleanup
2023-05-11 13:21:15 -05:00
a9ee1bae6c
chore: Merge remote-tracking branch 'origin/main' into smith/remove-transactions-main
2023-05-09 13:26:02 -05:00
e8a480f5f6
fix: Give up ownership of Column when adding to a table
...
To enable reuse of existing allocations rather than borrowing, creating
new allocations, then dropping them.
2023-05-09 14:55:00 +02:00
e8655af52d
fix: Change ColumnsByName::new to enable taking ownership if caller wants to give it
2023-05-09 14:55:00 +02:00
cc41216382
fix: Undo the addition of a TableInfo type; store partition_template on TableSchema
2023-05-09 14:54:59 +02:00
596673d515
refactor: Create a new ColumnsByName type to abstract over TableSchema columns
...
And allow usage of just the columns when that's all that's needed
without leaking the BTreeMap implementation detail everywhere
2023-05-09 14:54:58 +02:00
b5af971190
fix: Make NamespaceSchema::new_empty_from rather than From::from
...
To avoid implying that this constructor caches the tables.
2023-05-09 14:54:58 +02:00
1f1dcc947d
fix: Don't change how the compactor gets the table schema
2023-05-09 14:54:58 +02:00
58d9c40ffd
feat: If namespace or table partition templates are specified, use those
2023-05-09 14:54:57 +02:00
3d5df5574a
fix: Remove vestiges of shards
2023-05-08 20:24:36 -04:00
41d93aea4d
chore: Merge remote-tracking branch 'origin/main' into smith/remove-transactions-main
2023-05-04 14:05:49 -04:00
b0959667d5
fix: Move topic and query pool within iox catalog ( #7734 )
...
Still insert them into the database and associate them with namespaces,
but don't ever query them back out.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-04 13:45:56 +00:00
cb80308082
chore: Merge remote-tracking branch 'origin/main' into smith/remove-transactions-main
2023-05-04 09:02:23 -04:00
667da5eea4
chore: Add test for `dsn-file://` catalog urls ( #7735 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-04 10:48:18 +00:00
621caab2e9
fix: Remove unused parquet_max_sequence_number metadata
2023-05-03 10:57:27 -04:00
c9cfe05f8d
revert: PR #7708
...
This reverts commit 61abb58933
2023-05-02 13:51:30 +02:00
61abb58933
refactor: Change catalog configuration so it is entirely dsn based / support end to end testing without postgres ( #7708 )
...
* refactor: Change catalog configuration so it is entirely dsn based
* docs: Add documentation
* chore: update docs
* chore: review feedback
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-02 10:48:33 +00:00
4cf4433c1e
fix(iox_catalog): include missing column in return
...
The Parquet::create() method was missing the `max_sequence_number`
column in the query response!
2023-04-28 15:36:00 +02:00
3ba6ea771e
fix: missed part in a rebase
2023-04-27 13:15:12 -04:00
1a826a9ea9
chore: Merge remote-tracking branch 'origin/main' into smith/remove-transactions-main
2023-04-27 12:24:19 -04:00
038f8e9ce0
fix: Move shard concepts into only the catalog
...
This still inserts the shard id into the database, always set to the
TRANSITION_SHARD_ID, but never reads it back out again.
2023-04-26 11:42:32 -04:00
bdf7f316d7
chore(deps): Bump tokio from 1.27.0 to 1.28.0 ( #7667 )
...
* chore(deps): Bump tokio from 1.27.0 to 1.28.0
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.27.0 to 1.28.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.27.0...tokio-1.28.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-04-26 12:53:26 +00:00
0b39e6738f
chore: add method for creating transition shard using transactions
2023-04-25 13:40:57 -04:00
60eaa71191
test: add test for create_upgrade_delete
2023-04-25 13:40:55 -04:00
a01ae8f23d
feat: remove transactions from Catalog
2023-04-25 13:39:16 -04:00
06fbab2649
fix(garbage collector): limit catalog update for files to delete ( #7623 )
...
* fix(garbage collector): limit catalog update for files to delete
Impose a 1000 LIMIT on flag_for_delete_by_retention so the garbage
collector's load on the catalog is limited. 1000 is used as the fixed
limit in another catalog DML.
* follow up to requests in #7562
* chore: add test for limit on update
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-25 08:57:23 -07:00
cf051e8091
fix: Remove iox catalog shard tests
2023-04-24 10:08:00 -04:00
8d4c2bfabb
fix: Only ever create the transition shard in the in-memory catalog
...
Tests that use the in-memory catalog are creating different shards that
then creates old-style Parquet file paths, but in production, everything
uses the transition shard now. To make the tests more like production,
only ever create and use the transition shard, and stop checking for
different shard IDs.
2023-04-24 10:08:00 -04:00
584d3ab0e7
fix: Set up the transition shard in the in-mem catalog
2023-04-24 10:08:00 -04:00
a9081fc8a8
fix: Remove uses and tests for RPC write env var
2023-04-24 10:00:09 -04:00
0ac2494c5d
fix: Remove unused partitions_with_recent_created_files method
...
And PartitionParam type.
2023-04-20 14:54:03 -04:00
ba1be8db30
fix: Remove unused count_by_overlaps_with_level_1 function
2023-04-17 20:43:40 -04:00
88fad54f4c
fix: Remove unused count_by_overlaps_with_level_0 function
2023-04-17 20:40:42 -04:00
fefc6e4c27
fix: Remove unused most_cold_files_partitions function
2023-04-17 20:36:13 -04:00
a4a1a2e6b3
fix: Remove unused partitions_with_small_l1_file_count function
2023-04-17 20:33:37 -04:00
2dd79c1741
fix: Remove unused recent_highest_througput_partitions function
2023-04-17 20:29:39 -04:00
f1850c9234
fix: Remove unused level_1 function and TablePartition type
2023-04-17 19:28:50 -04:00
a55e2e5fdb
fix: Remove unused level_0 function
2023-04-17 19:28:49 -04:00
5b5ab7faf3
fix: Remove unused delete_old function
2023-04-17 19:28:49 -04:00
f344b94af7
fix: Remove unused list_by_shard_greater_than function
2023-04-17 19:28:49 -04:00
afaa954270
fix: Remove unused most_recent_n_in_shards function
2023-04-17 19:28:49 -04:00
2d1564d568
fix: Remove unused update_persisted_sequence_number function
2023-04-17 19:28:49 -04:00
d0906f5f38
fix: Remove unused list_by_namespace function
2023-04-17 19:28:49 -04:00
6c06b9fd7a
fix: Remove unused list_type_count_by_table_id and ColumnTypeCount
2023-04-17 19:28:49 -04:00
5e6dbec909
fix: Remove tombstones as they aren't functional currently
2023-04-14 13:36:08 -04:00
a244e5b078
test: Add some tests for CatalogToCompactPartitionsSource's existing behavior
2023-04-12 11:07:43 -04:00
a4a79bc640
test: Add a case where a partition's time is included in the max range
2023-03-31 12:42:27 -04:00
e4d5c777d9
feat: Make catalog method not specific to compacting and take optional end time
2023-03-31 12:36:24 -04:00
9ddca96336
chore(deps): Bump thiserror from 1.0.39 to 1.0.40 ( #7375 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.39 to 1.0.40.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.39...1.0.40 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 10:43:28 +00:00
4eedb7ea77
chore(deps): Bump async-trait from 0.1.66 to 0.1.68 ( #7374 )
...
* chore(deps): Bump async-trait from 0.1.66 to 0.1.68
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.66 to 0.1.68.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.66...0.1.68 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-03-30 10:14:36 +00:00
9cbcdc7672
chore(deps): Bump tokio from 1.26.0 to 1.27.0 ( #7373 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.26.0 to 1.27.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.26.0...tokio-1.27.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-30 09:36:04 +00:00
3a473d2374
chore(deps): Bump dotenvy from 0.15.6 to 0.15.7
...
Bumps [dotenvy](https://github.com/allan2/dotenvy ) from 0.15.6 to 0.15.7.
- [Release notes](https://github.com/allan2/dotenvy/releases )
- [Changelog](https://github.com/allan2/dotenvy/blob/master/CHANGELOG.md )
- [Commits](https://github.com/allan2/dotenvy/compare/v0.15.6...v0.15.7 )
---
updated-dependencies:
- dependency-name: dotenvy
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-03-23 02:02:33 +00:00
f780aba353
test: set max_l0_created_at to reasonable values for the tests and al… ( #7286 )
...
* test: set max_l0_created_at to reasonable values for the tests and also verify it using both test layout and catalog function
* fix: typo
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-21 18:57:10 +00:00
8771dcb645
chore(deps): Bump thiserror from 1.0.38 to 1.0.39 ( #7131 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.38 to 1.0.39.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.38...1.0.39 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-06 10:55:26 +00:00
3689827793
chore(deps): Bump paste from 1.0.11 to 1.0.12 ( #7130 )
...
Bumps [paste](https://github.com/dtolnay/paste ) from 1.0.11 to 1.0.12.
- [Release notes](https://github.com/dtolnay/paste/releases )
- [Commits](https://github.com/dtolnay/paste/compare/1.0.11...1.0.12 )
---
updated-dependencies:
- dependency-name: paste
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-06 10:40:41 +00:00
8f3a9396d0
chore(deps): Bump async-trait from 0.1.64 to 0.1.66 ( #7129 )
...
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.64 to 0.1.66.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.64...0.1.66 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-06 10:13:29 +00:00
c538cac4ef
chore(deps): Bump tokio from 1.25.0 to 1.26.0 ( #7107 )
...
* chore(deps): Bump tokio from 1.25.0 to 1.26.0
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.25.0 to 1.26.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.25.0...tokio-1.26.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-02 09:50:39 +00:00
faae5eb438
chore: Rerun cargo hakari manage-deps
2023-02-27 11:56:15 +01:00
7e31b2638d
fix: Understandable compactor2 config report ( #7028 )
...
* fix: Understandable compactor2 config report
* fix: do not log postgres dsn
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-22 23:43:31 +00:00
fb5aa25c5b
fix: Separate most_recent_n into filtering by shard and not
2023-02-17 12:56:51 -05:00
9351dc6c17
fix: Remove unused method list_by_shard
2023-02-17 12:56:51 -05:00
2d46a364dc
feat: namespace soft-delete support
...
This commit adds initial support for "soft" namespace deletion, where
the actual records & data remain, but are no longer queryable /
writeable.
Soft deletion is eventually consistent - users can expect to continue
writing to and reading from a bucket after issuing a soft delete call,
until the various components either restart, or have their caches
flushed.
The components treat soft-deleted namespaces differently:
* router: ignore soft deleted namespaces
* ingester: accept soft deleted namespaces
* compactor: accept soft deleted namespaces
* querier: ignore soft deleted namespaces
* various gRPC services: ignore soft deleted namespaces
This ensures that the ingester & compactor do not see rows "vanishing"
from the database, and continue to make forward progress.
Writes for the deleted namespace that are buffered in the ingester will
be persisted as normal, allowing us to support "un-delete" operations
where the system is restored to a the state at which the delete was
issued (rather than loosing the buffered data).
Follow-on work is required to ensure GC drops the orphaned parquet files
after the configured GC time, and optimisations such as not compacting
parquet from soft-deleted namespaces seems like a trivial win.
2023-02-13 12:01:35 +01:00
a85dcd745b
refactor(catalog): expose deleted_at on Namespace
...
Add the new catalog column to the Namespace representation/model.
2023-02-10 14:15:01 +01:00
aa74bb6292
test(catalog): independent test state
...
All our catalog tests run as one test, over one database connection.
Prior to this commit, there was no state reset during test execution, so
earlier tests would pollute the state of later tests, making it an
increasingly complex and intermingled set of tests trying to assert
their entities while ignoring other, previously created entities (or
changing the order of test execution in search of the golden ordering
that makes everything great again.)
This is a bit of a hack, and is not how I'd have structured catalog
testing w/ clean state if I was writing it fresh. It is what it is.
This has been driving me mad for SO LONG it's SO BAD <shakes fist>.
2023-02-09 17:27:58 +01:00
61409f062c
refactor(catalog): soft delete namespace column
...
Adds a "deleted_at" column that will indicate the timestamp at which is
was marked as logically deleted.
2023-02-09 11:35:27 +01:00
dcba47ab58
feat: allow the compactor to process all known partitions ( #6887 )
...
* feat: `PartitionRepo::list_ids`
* refactor: `CatalogPartitionsSource` => `CatalogToCompactPartitionsSource`
* feat: allow the compactor to process all known partitions
Closes #6648 .
* docs: improve
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
---------
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2023-02-08 09:32:21 +00:00
9e8785f9c1
Merge branch 'main' into dom/cache-table-limit
2023-02-07 10:04:50 +00:00
eb245d6774
feat: Initial SQLite catalog schema ( #6851 )
...
* feat: Initial SQLite catalog schema
* chore: Run cargo hakari tasks
* feat: impls, many TODOs
* feat: completed `todo!()`'s
* chore: add remaining tests from postgres module
* feat: add SQLite to get_catalog API
* chore: Add docs
* chore: Placate clippy
* chore: Placate clippy
* chore: PR feedback from @domodwyer
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-02-06 22:55:14 +00:00
a633964f2b
feat(catalog): return max table limit in schema
...
The maximum number of tables is part of the Namespace, which is already
loaded in its entirety. This commit copies the value into the
NamespaceSchema, making it available for the router to utilise.
2023-02-06 17:33:55 +01:00
30fea67701
fix: Move variables within format strings. Thanks clippy!
...
Changes made automatically using `cargo clippy --fix`.
2023-02-03 13:06:17 -05:00
2af563ad94
refactor(iox): lower default table limit
...
Drops the default table limit from 10,000 to 500. This will apply to all
new namespaces / org&bucket created after this change has been deployed.
2023-02-03 16:28:38 +01:00
c343ea60b7
feat(catalog): use explicit UTC time zone
...
We use UTC, but that doesn't mean everyone does. Queries that utilise
NOW() will return incorrect results when the server is using a non-UTC
tz, but application-provided UTC timestamps / epochs.
2023-02-02 11:51:25 +01:00
167cde1838
fix(iox): Add transition shard to catalog setup ( #6799 )
2023-02-01 17:04:03 +00:00
6f032b1d57
chore(deps): Bump async-trait from 0.1.63 to 0.1.64 ( #6769 )
...
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.63 to 0.1.64.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.63...0.1.64 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-01-31 10:18:27 +00:00
9279570299
fix: PG `flag_for_delete_by_retention` ( #6762 )
...
* test: failing test
* fix: PG `flag_for_delete_by_retention`
2023-01-30 21:27:12 +00:00
ed7d02a225
chore(deps): Bump tokio from 1.24.2 to 1.25.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.24.2 to 1.25.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/commits/tokio-1.25.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-01-30 01:57:27 +00:00
b8a80869d4
feat: introduce a new way of max_sequence_number for ingester, compactor and querier ( #6692 )
...
* feat: introduce a new way of max_sequence_number for ingester, compactor and querier
* chore: cleanup
* feat: new column max_l0_created_at to order files for deduplication
* chore: cleanup
* chore: debug info for chnaging cpu.parquet
* fix: update test parquet file
Co-authored-by: Marco Neumann <marco@crepererum.net>
2023-01-26 10:52:47 +00:00
06d4a5fe4e
refactor: ignore partitions in table skipped compactions ( #6666 )
...
* refactor: ignore partitions in table skipped compactions
* refactor: continue ignoring partitions in skipped compaction
* test: skip partition
2023-01-23 19:53:05 +00:00
0114e7ee50
chore(deps): Bump async-trait from 0.1.61 to 0.1.63 ( #6660 )
...
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.61 to 0.1.63.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.61...0.1.63 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-01-23 08:41:27 +00:00
5c462937ca
refactor: converters for `ParquetFile`<>`ParquetFileParams` ( #6637 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-19 16:47:54 +00:00
7a5fdd1d95
feat: function to read partition IDs of all partitions with new writes ( #6613 )
...
* feat: function to read partition IDs of all partitions with new writes
* chore: run fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-18 16:30:15 +00:00
550cea8bc5
perf: optimize not to update partitions with newly created level 2 files ( #6590 )
...
* perf: optimize not to update partitions with newly created level 2 files
* chore: cleanup
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-13 14:46:58 +00:00
62c0f3dbdd
feat: have cold compaction work with Compactor2 ( #6542 )
...
* feat: cold
* chore: debug info
* feat: only compact qualified cold partition candidates
* fix: catalog test
* chore: cleanup
* chore: add new config flag for cold partition candidates
* chore: implement display for CompactionType and add tests for max num partitions
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-10 16:42:57 +00:00
b49cc2e35e
chore(deps): Bump tokio from 1.24.0 to 1.24.1 ( #6545 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.24.0 to 1.24.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.24.0...tokio-1.24.1 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-10 09:48:44 +00:00
e31c84a794
chore(deps): Bump async-trait from 0.1.60 to 0.1.61 ( #6533 )
...
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.60 to 0.1.61.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.60...0.1.61 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-09 07:44:35 +00:00
b20226797a
fix: make trigger midification in different file ( #6526 )
2023-01-06 20:34:48 +00:00
b856edf826
feat: function to get parttion candidates from partition table ( #6519 )
...
* feat: function to get parttion candidates from partition table
* chore: cleanup
* fix: make new_file_at the same value as created_at
* chore: cleanup
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-06 16:20:45 +00:00
23807df7a9
feat: trigger that updates partition table when a parquet file is created ( #6514 )
...
* feat: trigger that update partition table when a parquet file is created
* chore: simplify epoch of now
2023-01-05 19:57:23 +00:00
c733f27eea
chore: cleanup
2023-01-04 16:00:42 -05:00
72977bf250
feat: catalog query to select partitions with recently created files
2023-01-04 15:54:46 -05:00
1088baea3d
chore: index for selecting partitions with parquet files created after a given time ( #6496 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-04 18:07:07 +00:00
fb8ead8515
Merge branch 'main' into feat/delete-namespace-api
2023-01-04 11:55:01 +00:00
72aab99951
fix: Remove needless borrow. Thanks clippy!
2022-12-21 14:32:34 -05:00
2f88fc71ce
docs: fix incomplete comment
2022-12-20 12:31:01 +01:00
adc6fcfb04
feat(catalog): linearise sort key updates
...
Updating the sort key is not commutative and MUST be serialised. The
correctness of the current catalog interface relies on the caller
serialising updates globally, something it cannot reasonably assert in a
distributed system.
This change of the catalog interface pushes this responsibility to the
catalog itself where it can be effectively enforced, and allows a caller
to detect parallel updates to the sort key.
2022-12-20 12:31:00 +01:00
c0f2ba09ee
Merge branch 'main' into cn/compactor2
2022-12-19 14:22:56 +00:00
299f0e99f9
chore(deps): Bump thiserror from 1.0.37 to 1.0.38
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.37 to 1.0.38.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.37...1.0.38 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-12-19 10:33:32 +00:00
8478d41bcb
chore(deps): Bump paste from 1.0.10 to 1.0.11 ( #6430 )
...
Bumps [paste](https://github.com/dtolnay/paste ) from 1.0.10 to 1.0.11.
- [Release notes](https://github.com/dtolnay/paste/releases )
- [Commits](https://github.com/dtolnay/paste/compare/1.0.10...1.0.11 )
---
updated-dependencies:
- dependency-name: paste
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-19 10:31:05 +00:00
c72734473c
chore(deps): Bump async-trait from 0.1.59 to 0.1.60 ( #6433 )
...
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.59 to 0.1.60.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.59...0.1.60 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-19 10:09:23 +00:00
dfd979477c
fix: Update warm compaction code to optionally take shard ID
2022-12-16 17:41:57 -05:00
b1b9b3122a
test: Run cold compaction catalog functions first and in a transaction
...
To avoid other tests' state bleeding into this one and this one's state
bleeding into other tests, now that it's testing some queries without
scoping by shard.
2022-12-16 17:28:55 -05:00
d7e75d43ea
fix: Make shard ID optional for compactor queries in RPC write mode
2022-12-16 17:28:53 -05:00
f419e2c378
feat: warm compaction ( #6192 )
...
* feat: warm compaction
chore: add missing warm compaction config
chore: tests for warm compaction
chore: modify count usage in warm compaction sql
chore: catalog test for warm compaction; sql fixes
feat: settable target level for compact w/ budget
chore: tests for warm compaction
chore: clarifying comments in warm compaction test
chore: fixed erroneous comment in catalog test
chore: improve warm compactor test by checking file exists
chore: tests for warm compaction
chore: warm compactor test tidy-ups
* chore: improve test for warm compaction
* chore: fix erroneous comment in warm compaction code
2022-12-16 15:59:45 +00:00
1bc2003cf4
chore: simplify delete namespace sql query
2022-12-16 10:23:50 +00:00
a6036631ad
chore: comment typo in catalog
2022-12-16 10:23:50 +00:00
6263ca234a
chore: delete ns postgres impl, test improvements, fix to mem impl
2022-12-16 10:23:50 +00:00
3659be59c7
feat: delete namespace api mem impl
...
chore: tests for delete namespace; use unique ptn names in tests
2022-12-16 10:23:50 +00:00
e108a8b6c9
chore(deps): Bump paste from 1.0.9 to 1.0.10 ( #6384 )
...
Bumps [paste](https://github.com/dtolnay/paste ) from 1.0.9 to 1.0.10.
- [Release notes](https://github.com/dtolnay/paste/releases )
- [Commits](https://github.com/dtolnay/paste/compare/1.0.9...1.0.10 )
---
updated-dependencies:
- dependency-name: paste
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-13 06:03:05 +00:00
a9db7581cd
chore(deps): Bump tokio from 1.21.2 to 1.22.0 ( #6183 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.21.2 to 1.22.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.21.2...tokio-1.22.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-21 10:21:24 +00:00
7c813c170a
feat: reintroduce compactor first file in partition exception ( #6176 )
...
* feat: compactor ignores max file count for first file
chore: typo in comment in compactor
* feat: restore special first file in partition compaction logic; add limit
* fix: calculation in compaction max file count
chore: clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-18 15:58:59 +00:00
49a9565240
feat: gRPC that creates namespace ( #6103 )
...
* feat: create namespace API call in router
Co-authored-by: Nga Tran <nga-tran@live.com>
* chore: treat retention as ns except in CLI
* fix: overflow in nanosecond calc
* fix: retention test after changing it from hours to ns
* chore: comment clarification in cli; better response type for error in ns API
* fix: correct some rebase mistakes
* chore: merge namespace create & create_with_retention; renamed ns create test helper fn & const
* fix: ns autocreation test was wrong after rebase
* fix: mem catalog has default 1hr retention, accidently removed in rebase
* chore: remove mem catalogs default 1hr retention; make it settable in sets & router
Co-authored-by: Luke Bond <luke.n.bond@gmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-18 13:02:12 +00:00
6f7b1e2e26
feat: reject writes that are outside the retention period ( #6148 )
...
* feat: reject writes that are outside the retention period
* feat: add retention validator into handler stack
* chore: Apply suggestions from code review
Co-authored-by: Dom <dom@itsallbroken.com>
* refactor: address review comments
* test: unit tests fot retention validation
* chore: address review comments
* test: more unit tests and integration tests
* refactor: make time inside retention period for emphemeral_mode test
* fix: 2 hours
Co-authored-by: Dom <dom@itsallbroken.com>
2022-11-17 20:55:58 +00:00
bdff4e8848
fix: Consistently use 'namespace' instead of 'database' in comments and other internal text
2022-11-11 15:46:04 -05:00
a3f2fe489c
refactor: remove retention_duration field from namespace catalog table ( #6124 )
2022-11-11 20:30:42 +00:00
9c4266c503
refactor: first step to remove unused retention_duration ( #6113 )
...
* refactor: first step to remove unused retention_duration
* refactor: remove retenion_duration from update catalog
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-11 15:21:06 +00:00
93e11d4c91
chore: Revert "feat: flag partitions for delete ( #6075 )" ( #6111 )
...
This reverts commit 77a2541172
2022-11-10 17:01:39 +00:00
e81ff1f4d5
chore: Revert "feat: catalog delete old partitions ( #6099 )" ( #6109 )
...
This reverts commit 664b0578e9
2022-11-10 15:31:16 +00:00
664b0578e9
feat: catalog delete old partitions ( #6099 )
...
* feat: catalog delete old partitions
* chore: remove debug println
* chore: remove debug println
* chore: clippy
* chore: sql statement refactor for deleting partitions
* chore: improve delete partition test
* chore: clippy
2022-11-10 10:51:22 +00:00
43687a86d2
fix: Remove lots of needless borrows that Clippy can now identify
...
Except for in generated code that we don't control.
2022-11-09 10:54:18 -05:00
07505c8f72
fix: Remove needless borrows, thanks Clippy!
2022-11-09 10:54:18 -05:00
77a2541172
feat: flag partitions for delete ( #6075 )
...
* feat: flag partition for delete
* fix: compare the right date and time
* chore: Run cargo hakari tasks
* chore: cleanup
* fix: typos
* chore: rust style tidy ups in catalog
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Luke Bond <luke.n.bond@gmail.com>
2022-11-09 12:06:23 +00:00
dfb820615c
feat: deletion flagging in GC based on retention policy ( #6073 )
...
* feat: deletion flagging in GC based on retention policy
* chore: typo in comment
* fix: only soft delete parquet files that aren't yet soft deleted
* fix: guard against flakiness in catalog test
* chore: some better tests for parquet file delete flagging
Co-authored-by: Nga Tran <nga-tran@live.com>
2022-11-08 20:22:35 +00:00
369937d68f
Merge branch 'main' into cn/order-by-insert
2022-11-07 19:18:13 +00:00
74a40cc9bd
fix: Assert that there aren't two columns with the same name in the same batch
...
This shouldn't be possible; let's make sure we know if it happens!
2022-11-07 14:10:12 -05:00
5e05fa52cf
feat: soft delete parquet files based on retention period ( #6070 )
2022-11-07 17:31:29 +00:00
9356f2a1b9
feat: grpc for updating namespace retention period ( #6041 )
...
* refactor: make namespace folder for all namesapce's commands
* feat: WIP for add command to set retention period
* feat: more on updating retention period
* feat: grpc for update namespace retention period
* test: end to end test fpr namespace retention
* fix: lint proto
* chore: cleanup
* chore: kick CI run again
* fix: command hierachy
* chore: fix comments
2022-11-04 20:58:11 +00:00
9b0af96927
docs: Add a link to the idpe issue
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2022-11-04 13:39:25 -04:00
d454c66b4b
fix: Use a HashMap for column lookup instead of Vec ordering
...
The checks for whether a column already exists with a different type
were relying on ordering of the input matching the ordering of the
columns returned from inserting the columns in Postgres.
Rather than trying to match the new ordering that is required to avoid
Postgres deadlocks, switch from a Vec to a HashMap and look up the
column type from the name.
This also reduces some allocations that weren't really needed.
2022-11-04 11:52:37 -04:00
a6634ada19
fix: Add an ORDER BY to the insert to prevent Postgres deadlocks
...
Fixes influxdata/idpe#16298 .
Without this ORDER BY, concurrent writes that add many column records to
this table can deadlock because they grab locks to rows/index entries in
an arbitrary order to check the unique index.
By switching to a consistent order across all requests, inserts won't
get in a deadlock loop waiting for each other.
More info:
- <https://rcoh.svbtle.com/postgres-unique-constraints-can-cause-deadlock >
- <https://dba.stackexchange.com/a/195220/27897 >
2022-11-04 11:52:37 -04:00
498851eaf5
feat: add catalog columns needed for retention policy
2022-11-01 15:35:15 -04:00
2e83e04eab
feat: Use workspace package metadata to reduce differences and repetition
2022-10-24 13:04:09 -04:00
df2ba85661
feat: add the ability to delete a skipped compaction
2022-10-21 15:12:20 -04:00
b5574c07b7
chore(deps): Bump async-trait from 0.1.57 to 0.1.58 ( #5904 )
...
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.57 to 0.1.58.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.57...0.1.58 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-19 09:40:26 +00:00
f3c27c5c71
chore(deps): Bump dotenvy from 0.15.5 to 0.15.6 ( #5881 )
...
Bumps [dotenvy](https://github.com/allan2/dotenvy ) from 0.15.5 to 0.15.6.
- [Release notes](https://github.com/allan2/dotenvy/releases )
- [Changelog](https://github.com/allan2/dotenvy/blob/master/CHANGELOG.md )
- [Commits](https://github.com/allan2/dotenvy/compare/v0.15.5...v0.15.6 )
---
updated-dependencies:
- dependency-name: dotenvy
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-18 07:06:40 +00:00
afdc008855
fix: correct default limits
2022-10-14 16:05:56 +02:00
e4179605df
test: assert default service limit values
...
Adds tests that assert the default service limit values.
2022-10-14 14:46:34 +02:00
46bbee5423
refactor: reduce default column limit
...
Reduces the default number of columns allowed per-table, from 1,000 to
200.
2022-10-14 14:45:48 +02:00
efb964c390
feat: Enforce table column limits from the schema cache ( #5819 )
...
* fix: Avoid some allocations by collecting instead of inserting into a vec
* refactor: Encode that adding columns is for one table at a time
* test: Add another test of column limits
* test: Add below/above limit tests for create_or_get_many
* fix: Explicitly DO NOT check column limits when inserting many columns
* feat: Cache the max_columns_per_table on the NamespaceSchema
* feat: Add a function to validate column limits in-memory
* fix: Provide more useful information when over column limits
* fix: Swap types to remove intermediate allocation
* docs: Explain the interactions of the cache and the column limits
* test: Actually set up test that showcases column limit race condition
* fix: Allow writing to existing columns even if table is over column limit
Co-authored-by: Dom <dom@itsallbroken.com>
2022-10-14 11:34:17 +00:00
3e70dc44a0
refactor(catalog): remove partition_info_by_id()
...
This method used to return a subset of partition metadata, and was used
exclusively for persistence in the ingester. It is now no longer
necessary.
2022-10-13 15:26:36 +02:00
3e1e4c1f0b
refactor: remove Table::get_table_persist_info()
...
Remove the now-redundant get_table_persist_info() implementations.
2022-10-13 13:44:50 +02:00
c4f542bbe2
refactor(ingester): remove tombstone support
...
This commit removes tombstone support from the ingester, and deletes
associated code/helpers/tests. This commit does NOT remove tombstone
support from any other service, but MAY include removing overlapping
test coverage.
This also removes the tombstone support from the Ingester -> Querier RPC
response message.
This has the nice side effect of removing a whole lot of thread spawning
in the ingester tests for the Executor, speeding everything up!
2022-10-11 13:10:04 +02:00
afcb96ae47
perf(ingester): deferred sort key lookup queries
...
This commit carries the SortKey in the PartitionData, and configures the
ingester to use deferred sort key lookups, smearing the lookups across a
fixed period of time after initialising the PartitionData, instead of
querying for the sort key at persist time.
This allows large numbers of PartitionData to be initialised without
causing a equally large spike in catalog load to resolve the sort key -
instead this load is spread out randomly to reduce peak query rps.
2022-10-06 16:39:54 +02:00
d171697fd7
feat: always pick cold partitions in next cycle even if it has been pa… ( #5772 )
...
* fix: always pick cold partitions in next cycle even if it has been partially compacted recently
* fix: comment
* fix: test output
* refactor: using var instead of literal
* fix: consider deleted L0s for recent writes
* chore: cleanup
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-30 15:54:00 +00:00
cd4087e00d
style: add no todo!() or dbg!() lints
...
Some crates had theme, some not - lets be consistent and have the
compiler spot dbg!() and todo!() macro calls - they should never be in
prod code!
2022-09-29 13:10:07 +02:00
Andrew Lamb
Jeffrey Smith II
Carol (Nichols || Goulding)
Kaya Gökalp
Carol (Nichols || Goulding)
Dom Dwyer
dependabot[bot]
Phil Bracikowski
Nga Tran
Marco Neumann
Stuart Carnie
Marko Mikulicic
Luke Bond
kodiakhq[bot]
Jake Goulding