a9ee1bae6c
chore: Merge remote-tracking branch 'origin/main' into smith/remove-transactions-main
2023-05-09 13:26:02 -05:00
e8a480f5f6
fix: Give up ownership of Column when adding to a table
...
To enable reuse of existing allocations rather than borrowing, creating
new allocations, then dropping them.
2023-05-09 14:55:00 +02:00
e8655af52d
fix: Change ColumnsByName::new to enable taking ownership if caller wants to give it
2023-05-09 14:55:00 +02:00
cc41216382
fix: Undo the addition of a TableInfo type; store partition_template on TableSchema
2023-05-09 14:54:59 +02:00
596673d515
refactor: Create a new ColumnsByName type to abstract over TableSchema columns
...
And allow usage of just the columns when that's all that's needed
without leaking the BTreeMap implementation detail everywhere
2023-05-09 14:54:58 +02:00
b5af971190
fix: Make NamespaceSchema::new_empty_from rather than From::from
...
To avoid implying that this constructor caches the tables.
2023-05-09 14:54:58 +02:00
1f1dcc947d
fix: Don't change how the compactor gets the table schema
2023-05-09 14:54:58 +02:00
58d9c40ffd
feat: If namespace or table partition templates are specified, use those
2023-05-09 14:54:57 +02:00
3d5df5574a
fix: Remove vestiges of shards
2023-05-08 20:24:36 -04:00
41d93aea4d
chore: Merge remote-tracking branch 'origin/main' into smith/remove-transactions-main
2023-05-04 14:05:49 -04:00
b0959667d5
fix: Move topic and query pool within iox catalog ( #7734 )
...
Still insert them into the database and associate them with namespaces,
but don't ever query them back out.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-04 13:45:56 +00:00
cb80308082
chore: Merge remote-tracking branch 'origin/main' into smith/remove-transactions-main
2023-05-04 09:02:23 -04:00
667da5eea4
chore: Add test for `dsn-file://` catalog urls ( #7735 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-04 10:48:18 +00:00
621caab2e9
fix: Remove unused parquet_max_sequence_number metadata
2023-05-03 10:57:27 -04:00
c9cfe05f8d
revert: PR #7708
...
This reverts commit 61abb58933
2023-05-02 13:51:30 +02:00
61abb58933
refactor: Change catalog configuration so it is entirely dsn based / support end to end testing without postgres ( #7708 )
...
* refactor: Change catalog configuration so it is entirely dsn based
* docs: Add documentation
* chore: update docs
* chore: review feedback
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-02 10:48:33 +00:00
4cf4433c1e
fix(iox_catalog): include missing column in return
...
The Parquet::create() method was missing the `max_sequence_number`
column in the query response!
2023-04-28 15:36:00 +02:00
3ba6ea771e
fix: missed part in a rebase
2023-04-27 13:15:12 -04:00
1a826a9ea9
chore: Merge remote-tracking branch 'origin/main' into smith/remove-transactions-main
2023-04-27 12:24:19 -04:00
038f8e9ce0
fix: Move shard concepts into only the catalog
...
This still inserts the shard id into the database, always set to the
TRANSITION_SHARD_ID, but never reads it back out again.
2023-04-26 11:42:32 -04:00
bdf7f316d7
chore(deps): Bump tokio from 1.27.0 to 1.28.0 ( #7667 )
...
* chore(deps): Bump tokio from 1.27.0 to 1.28.0
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.27.0 to 1.28.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.27.0...tokio-1.28.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-04-26 12:53:26 +00:00
0b39e6738f
chore: add method for creating transition shard using transactions
2023-04-25 13:40:57 -04:00
60eaa71191
test: add test for create_upgrade_delete
2023-04-25 13:40:55 -04:00
a01ae8f23d
feat: remove transactions from Catalog
2023-04-25 13:39:16 -04:00
06fbab2649
fix(garbage collector): limit catalog update for files to delete ( #7623 )
...
* fix(garbage collector): limit catalog update for files to delete
Impose a 1000 LIMIT on flag_for_delete_by_retention so the garbage
collector's load on the catalog is limited. 1000 is used as the fixed
limit in another catalog DML.
* follow up to requests in #7562
* chore: add test for limit on update
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-25 08:57:23 -07:00
cf051e8091
fix: Remove iox catalog shard tests
2023-04-24 10:08:00 -04:00
8d4c2bfabb
fix: Only ever create the transition shard in the in-memory catalog
...
Tests that use the in-memory catalog are creating different shards that
then creates old-style Parquet file paths, but in production, everything
uses the transition shard now. To make the tests more like production,
only ever create and use the transition shard, and stop checking for
different shard IDs.
2023-04-24 10:08:00 -04:00
584d3ab0e7
fix: Set up the transition shard in the in-mem catalog
2023-04-24 10:08:00 -04:00
a9081fc8a8
fix: Remove uses and tests for RPC write env var
2023-04-24 10:00:09 -04:00
0ac2494c5d
fix: Remove unused partitions_with_recent_created_files method
...
And PartitionParam type.
2023-04-20 14:54:03 -04:00
ba1be8db30
fix: Remove unused count_by_overlaps_with_level_1 function
2023-04-17 20:43:40 -04:00
88fad54f4c
fix: Remove unused count_by_overlaps_with_level_0 function
2023-04-17 20:40:42 -04:00
fefc6e4c27
fix: Remove unused most_cold_files_partitions function
2023-04-17 20:36:13 -04:00
a4a1a2e6b3
fix: Remove unused partitions_with_small_l1_file_count function
2023-04-17 20:33:37 -04:00
2dd79c1741
fix: Remove unused recent_highest_througput_partitions function
2023-04-17 20:29:39 -04:00
f1850c9234
fix: Remove unused level_1 function and TablePartition type
2023-04-17 19:28:50 -04:00
a55e2e5fdb
fix: Remove unused level_0 function
2023-04-17 19:28:49 -04:00
5b5ab7faf3
fix: Remove unused delete_old function
2023-04-17 19:28:49 -04:00
f344b94af7
fix: Remove unused list_by_shard_greater_than function
2023-04-17 19:28:49 -04:00
afaa954270
fix: Remove unused most_recent_n_in_shards function
2023-04-17 19:28:49 -04:00
2d1564d568
fix: Remove unused update_persisted_sequence_number function
2023-04-17 19:28:49 -04:00
d0906f5f38
fix: Remove unused list_by_namespace function
2023-04-17 19:28:49 -04:00
6c06b9fd7a
fix: Remove unused list_type_count_by_table_id and ColumnTypeCount
2023-04-17 19:28:49 -04:00
5e6dbec909
fix: Remove tombstones as they aren't functional currently
2023-04-14 13:36:08 -04:00
a244e5b078
test: Add some tests for CatalogToCompactPartitionsSource's existing behavior
2023-04-12 11:07:43 -04:00
a4a79bc640
test: Add a case where a partition's time is included in the max range
2023-03-31 12:42:27 -04:00
e4d5c777d9
feat: Make catalog method not specific to compacting and take optional end time
2023-03-31 12:36:24 -04:00
9ddca96336
chore(deps): Bump thiserror from 1.0.39 to 1.0.40 ( #7375 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.39 to 1.0.40.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.39...1.0.40 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 10:43:28 +00:00
4eedb7ea77
chore(deps): Bump async-trait from 0.1.66 to 0.1.68 ( #7374 )
...
* chore(deps): Bump async-trait from 0.1.66 to 0.1.68
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.66 to 0.1.68.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.66...0.1.68 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-03-30 10:14:36 +00:00
9cbcdc7672
chore(deps): Bump tokio from 1.26.0 to 1.27.0 ( #7373 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.26.0 to 1.27.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.26.0...tokio-1.27.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-30 09:36:04 +00:00
3a473d2374
chore(deps): Bump dotenvy from 0.15.6 to 0.15.7
...
Bumps [dotenvy](https://github.com/allan2/dotenvy ) from 0.15.6 to 0.15.7.
- [Release notes](https://github.com/allan2/dotenvy/releases )
- [Changelog](https://github.com/allan2/dotenvy/blob/master/CHANGELOG.md )
- [Commits](https://github.com/allan2/dotenvy/compare/v0.15.6...v0.15.7 )
---
updated-dependencies:
- dependency-name: dotenvy
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-03-23 02:02:33 +00:00
f780aba353
test: set max_l0_created_at to reasonable values for the tests and al… ( #7286 )
...
* test: set max_l0_created_at to reasonable values for the tests and also verify it using both test layout and catalog function
* fix: typo
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-21 18:57:10 +00:00
8771dcb645
chore(deps): Bump thiserror from 1.0.38 to 1.0.39 ( #7131 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.38 to 1.0.39.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.38...1.0.39 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-06 10:55:26 +00:00
3689827793
chore(deps): Bump paste from 1.0.11 to 1.0.12 ( #7130 )
...
Bumps [paste](https://github.com/dtolnay/paste ) from 1.0.11 to 1.0.12.
- [Release notes](https://github.com/dtolnay/paste/releases )
- [Commits](https://github.com/dtolnay/paste/compare/1.0.11...1.0.12 )
---
updated-dependencies:
- dependency-name: paste
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-06 10:40:41 +00:00
8f3a9396d0
chore(deps): Bump async-trait from 0.1.64 to 0.1.66 ( #7129 )
...
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.64 to 0.1.66.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.64...0.1.66 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-06 10:13:29 +00:00
c538cac4ef
chore(deps): Bump tokio from 1.25.0 to 1.26.0 ( #7107 )
...
* chore(deps): Bump tokio from 1.25.0 to 1.26.0
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.25.0 to 1.26.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.25.0...tokio-1.26.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-02 09:50:39 +00:00
faae5eb438
chore: Rerun cargo hakari manage-deps
2023-02-27 11:56:15 +01:00
7e31b2638d
fix: Understandable compactor2 config report ( #7028 )
...
* fix: Understandable compactor2 config report
* fix: do not log postgres dsn
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-22 23:43:31 +00:00
fb5aa25c5b
fix: Separate most_recent_n into filtering by shard and not
2023-02-17 12:56:51 -05:00
9351dc6c17
fix: Remove unused method list_by_shard
2023-02-17 12:56:51 -05:00
2d46a364dc
feat: namespace soft-delete support
...
This commit adds initial support for "soft" namespace deletion, where
the actual records & data remain, but are no longer queryable /
writeable.
Soft deletion is eventually consistent - users can expect to continue
writing to and reading from a bucket after issuing a soft delete call,
until the various components either restart, or have their caches
flushed.
The components treat soft-deleted namespaces differently:
* router: ignore soft deleted namespaces
* ingester: accept soft deleted namespaces
* compactor: accept soft deleted namespaces
* querier: ignore soft deleted namespaces
* various gRPC services: ignore soft deleted namespaces
This ensures that the ingester & compactor do not see rows "vanishing"
from the database, and continue to make forward progress.
Writes for the deleted namespace that are buffered in the ingester will
be persisted as normal, allowing us to support "un-delete" operations
where the system is restored to a the state at which the delete was
issued (rather than loosing the buffered data).
Follow-on work is required to ensure GC drops the orphaned parquet files
after the configured GC time, and optimisations such as not compacting
parquet from soft-deleted namespaces seems like a trivial win.
2023-02-13 12:01:35 +01:00
a85dcd745b
refactor(catalog): expose deleted_at on Namespace
...
Add the new catalog column to the Namespace representation/model.
2023-02-10 14:15:01 +01:00
aa74bb6292
test(catalog): independent test state
...
All our catalog tests run as one test, over one database connection.
Prior to this commit, there was no state reset during test execution, so
earlier tests would pollute the state of later tests, making it an
increasingly complex and intermingled set of tests trying to assert
their entities while ignoring other, previously created entities (or
changing the order of test execution in search of the golden ordering
that makes everything great again.)
This is a bit of a hack, and is not how I'd have structured catalog
testing w/ clean state if I was writing it fresh. It is what it is.
This has been driving me mad for SO LONG it's SO BAD <shakes fist>.
2023-02-09 17:27:58 +01:00
61409f062c
refactor(catalog): soft delete namespace column
...
Adds a "deleted_at" column that will indicate the timestamp at which is
was marked as logically deleted.
2023-02-09 11:35:27 +01:00
dcba47ab58
feat: allow the compactor to process all known partitions ( #6887 )
...
* feat: `PartitionRepo::list_ids`
* refactor: `CatalogPartitionsSource` => `CatalogToCompactPartitionsSource`
* feat: allow the compactor to process all known partitions
Closes #6648 .
* docs: improve
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
---------
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2023-02-08 09:32:21 +00:00
9e8785f9c1
Merge branch 'main' into dom/cache-table-limit
2023-02-07 10:04:50 +00:00
eb245d6774
feat: Initial SQLite catalog schema ( #6851 )
...
* feat: Initial SQLite catalog schema
* chore: Run cargo hakari tasks
* feat: impls, many TODOs
* feat: completed `todo!()`'s
* chore: add remaining tests from postgres module
* feat: add SQLite to get_catalog API
* chore: Add docs
* chore: Placate clippy
* chore: Placate clippy
* chore: PR feedback from @domodwyer
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-02-06 22:55:14 +00:00
a633964f2b
feat(catalog): return max table limit in schema
...
The maximum number of tables is part of the Namespace, which is already
loaded in its entirety. This commit copies the value into the
NamespaceSchema, making it available for the router to utilise.
2023-02-06 17:33:55 +01:00
30fea67701
fix: Move variables within format strings. Thanks clippy!
...
Changes made automatically using `cargo clippy --fix`.
2023-02-03 13:06:17 -05:00
2af563ad94
refactor(iox): lower default table limit
...
Drops the default table limit from 10,000 to 500. This will apply to all
new namespaces / org&bucket created after this change has been deployed.
2023-02-03 16:28:38 +01:00
c343ea60b7
feat(catalog): use explicit UTC time zone
...
We use UTC, but that doesn't mean everyone does. Queries that utilise
NOW() will return incorrect results when the server is using a non-UTC
tz, but application-provided UTC timestamps / epochs.
2023-02-02 11:51:25 +01:00
167cde1838
fix(iox): Add transition shard to catalog setup ( #6799 )
2023-02-01 17:04:03 +00:00
6f032b1d57
chore(deps): Bump async-trait from 0.1.63 to 0.1.64 ( #6769 )
...
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.63 to 0.1.64.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.63...0.1.64 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-01-31 10:18:27 +00:00
9279570299
fix: PG `flag_for_delete_by_retention` ( #6762 )
...
* test: failing test
* fix: PG `flag_for_delete_by_retention`
2023-01-30 21:27:12 +00:00
ed7d02a225
chore(deps): Bump tokio from 1.24.2 to 1.25.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.24.2 to 1.25.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/commits/tokio-1.25.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-01-30 01:57:27 +00:00
b8a80869d4
feat: introduce a new way of max_sequence_number for ingester, compactor and querier ( #6692 )
...
* feat: introduce a new way of max_sequence_number for ingester, compactor and querier
* chore: cleanup
* feat: new column max_l0_created_at to order files for deduplication
* chore: cleanup
* chore: debug info for chnaging cpu.parquet
* fix: update test parquet file
Co-authored-by: Marco Neumann <marco@crepererum.net>
2023-01-26 10:52:47 +00:00
06d4a5fe4e
refactor: ignore partitions in table skipped compactions ( #6666 )
...
* refactor: ignore partitions in table skipped compactions
* refactor: continue ignoring partitions in skipped compaction
* test: skip partition
2023-01-23 19:53:05 +00:00
0114e7ee50
chore(deps): Bump async-trait from 0.1.61 to 0.1.63 ( #6660 )
...
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.61 to 0.1.63.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.61...0.1.63 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-01-23 08:41:27 +00:00
5c462937ca
refactor: converters for `ParquetFile`<>`ParquetFileParams` ( #6637 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-19 16:47:54 +00:00
7a5fdd1d95
feat: function to read partition IDs of all partitions with new writes ( #6613 )
...
* feat: function to read partition IDs of all partitions with new writes
* chore: run fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-18 16:30:15 +00:00
550cea8bc5
perf: optimize not to update partitions with newly created level 2 files ( #6590 )
...
* perf: optimize not to update partitions with newly created level 2 files
* chore: cleanup
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-13 14:46:58 +00:00
62c0f3dbdd
feat: have cold compaction work with Compactor2 ( #6542 )
...
* feat: cold
* chore: debug info
* feat: only compact qualified cold partition candidates
* fix: catalog test
* chore: cleanup
* chore: add new config flag for cold partition candidates
* chore: implement display for CompactionType and add tests for max num partitions
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-10 16:42:57 +00:00
b49cc2e35e
chore(deps): Bump tokio from 1.24.0 to 1.24.1 ( #6545 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.24.0 to 1.24.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.24.0...tokio-1.24.1 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-10 09:48:44 +00:00
e31c84a794
chore(deps): Bump async-trait from 0.1.60 to 0.1.61 ( #6533 )
...
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.60 to 0.1.61.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.60...0.1.61 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-09 07:44:35 +00:00
b20226797a
fix: make trigger midification in different file ( #6526 )
2023-01-06 20:34:48 +00:00
b856edf826
feat: function to get parttion candidates from partition table ( #6519 )
...
* feat: function to get parttion candidates from partition table
* chore: cleanup
* fix: make new_file_at the same value as created_at
* chore: cleanup
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-06 16:20:45 +00:00
23807df7a9
feat: trigger that updates partition table when a parquet file is created ( #6514 )
...
* feat: trigger that update partition table when a parquet file is created
* chore: simplify epoch of now
2023-01-05 19:57:23 +00:00
c733f27eea
chore: cleanup
2023-01-04 16:00:42 -05:00
72977bf250
feat: catalog query to select partitions with recently created files
2023-01-04 15:54:46 -05:00
1088baea3d
chore: index for selecting partitions with parquet files created after a given time ( #6496 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-04 18:07:07 +00:00
fb8ead8515
Merge branch 'main' into feat/delete-namespace-api
2023-01-04 11:55:01 +00:00
72aab99951
fix: Remove needless borrow. Thanks clippy!
2022-12-21 14:32:34 -05:00
2f88fc71ce
docs: fix incomplete comment
2022-12-20 12:31:01 +01:00
adc6fcfb04
feat(catalog): linearise sort key updates
...
Updating the sort key is not commutative and MUST be serialised. The
correctness of the current catalog interface relies on the caller
serialising updates globally, something it cannot reasonably assert in a
distributed system.
This change of the catalog interface pushes this responsibility to the
catalog itself where it can be effectively enforced, and allows a caller
to detect parallel updates to the sort key.
2022-12-20 12:31:00 +01:00
c0f2ba09ee
Merge branch 'main' into cn/compactor2
2022-12-19 14:22:56 +00:00
299f0e99f9
chore(deps): Bump thiserror from 1.0.37 to 1.0.38
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.37 to 1.0.38.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.37...1.0.38 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-12-19 10:33:32 +00:00
8478d41bcb
chore(deps): Bump paste from 1.0.10 to 1.0.11 ( #6430 )
...
Bumps [paste](https://github.com/dtolnay/paste ) from 1.0.10 to 1.0.11.
- [Release notes](https://github.com/dtolnay/paste/releases )
- [Commits](https://github.com/dtolnay/paste/compare/1.0.10...1.0.11 )
---
updated-dependencies:
- dependency-name: paste
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-19 10:31:05 +00:00
c72734473c
chore(deps): Bump async-trait from 0.1.59 to 0.1.60 ( #6433 )
...
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.59 to 0.1.60.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.59...0.1.60 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-19 10:09:23 +00:00
dfd979477c
fix: Update warm compaction code to optionally take shard ID
2022-12-16 17:41:57 -05:00
b1b9b3122a
test: Run cold compaction catalog functions first and in a transaction
...
To avoid other tests' state bleeding into this one and this one's state
bleeding into other tests, now that it's testing some queries without
scoping by shard.
2022-12-16 17:28:55 -05:00
Jeffrey Smith II
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
Andrew Lamb
Dom Dwyer
dependabot[bot]
Phil Bracikowski
Nga Tran
Marco Neumann
Stuart Carnie
Marko Mikulicic
Luke Bond
kodiakhq[bot]