4a97620664
refactor: `ParquetFilesSink` ( #6928 )
...
* refactor: pass `PlanIR` by ref
* refactor: `ParquetFilesSink`
2023-02-09 15:53:19 +00:00
0e5f31c576
feat: add log around job semaphore and per-partition ( #6927 )
...
May help w/ debug OOMs.
2023-02-09 15:02:34 +00:00
00fd545a18
Merge pull request #6924 from influxdata/dom/soft-delete-namespaces
...
refactor(catalog): soft delete namespace column
2023-02-09 13:50:55 +00:00
8d62a55743
Merge branch 'main' into dom/soft-delete-namespaces
2023-02-09 13:43:08 +00:00
7ae6dda87c
docs(proto): ingester persist endpoint
...
This endpoint has some serious usability caveats that should be known by
users of this API!
2023-02-09 14:11:37 +01:00
08cf71e0ac
refactor(proto): move PersistService to new file
...
Separate the PersistService into it's own file.
2023-02-09 14:04:46 +01:00
7f4fd1013c
feat: measure scratchpad store ( #6925 )
...
* refactor: create `objec_store` submodule
* feat: add gauge for scratchpad size
2023-02-09 13:03:26 +00:00
e33ac920a5
refactor: introduce IR before creating actual DF plan ( #6922 )
...
* refactor: introduce IR before creating actual DF plan
Let's have an IR that presents a machine-readable form of how output
files may look like.
* docs: improve
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* feat: also log plan type
---------
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2023-02-09 11:31:18 +00:00
f374b022ed
Merge branch 'main' into dom/soft-delete-namespaces
2023-02-09 10:47:53 +00:00
905ffddd74
chore: gitignore pending insta snapshots ( #6923 )
...
You never want to commit these, they are mere local state.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-09 10:47:29 +00:00
cdd8a9c8e8
Merge branch 'main' into dom/soft-delete-namespaces
2023-02-09 10:45:09 +00:00
6327e3d9c0
chore(deps): Bump serde_json from 1.0.92 to 1.0.93 ( #6918 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.92 to 1.0.93.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.92...v1.0.93 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-09 10:39:33 +00:00
13c2633695
Merge branch 'main' into dom/soft-delete-namespaces
2023-02-09 10:37:06 +00:00
61409f062c
refactor(catalog): soft delete namespace column
...
Adds a "deleted_at" column that will indicate the timestamp at which is
was marked as logically deleted.
2023-02-09 11:35:27 +01:00
a66dd37c91
Merge pull request #6889 from influxdata/dom/always-requeue
...
fix(ingester2): always maintain persist interval
2023-02-09 10:31:35 +00:00
d44b6d412f
Merge branch 'main' into dom/always-requeue
2023-02-09 10:21:32 +00:00
4911f37fe5
chore(deps): Bump filetime from 0.2.19 to 0.2.20 ( #6917 )
...
Bumps [filetime](https://github.com/alexcrichton/filetime ) from 0.2.19 to 0.2.20.
- [Release notes](https://github.com/alexcrichton/filetime/releases )
- [Commits](https://github.com/alexcrichton/filetime/compare/0.2.19...0.2.20 )
---
updated-dependencies:
- dependency-name: filetime
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-09 10:13:50 +00:00
f6ad37e719
fix: query annotations and params ( #6449 )
...
* fix: query.dialect.annotations should be a string[]
* feat: Add query.params support for use in flux queries
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-09 00:08:55 +00:00
bd2a72a4b6
test(compactor): Improve file size display in compactor tests ( #6909 )
...
* test(compactor): Improve file size display in compactor tests
* fix: try different display
* fix: update
* fix: use b to show size
2023-02-08 22:35:12 +00:00
18d5924dfd
test: allow testing the compactor w/o any real data ( #6908 )
...
* test: allow testing the compactor w/o any real data
Things that are missing:
- output files have nondeterministic IDs which interferes w/ snapshot
testing. We should probably normalize the IDs somehow.
- time ranges of output files are not captured correctly (because the
mock sink doesn't know how to calculate them)
* fix: Add output assertion
* fix: fmt
* docs: improve
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* fix: fmt
---------
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2023-02-08 19:10:28 +00:00
2f4d901fbe
chore: Update datafusion ( #6893 )
...
* chore: Update datafusion
* chore: Run cargo hakari tasks
* fix: update for api change
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-08 18:27:18 +00:00
43b11b8be8
fix: metric labels ( #6904 )
...
Metric labels are key-value based.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-08 18:18:56 +00:00
5e287ac8e6
refactor: make `TestSetup` immutable ( #6907 )
...
* refactor: split off file creation
* refactor: make `TestSetup` immutable
2023-02-08 13:17:15 +00:00
6f7f608685
refactor: avoid nested test modules ( #6906 )
...
Just call it `tests` and avoid nesting a module in a module.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-08 11:16:36 +00:00
0ecde75af5
chore(deps): Bump object_store from 0.5.3 to 0.5.4 ( #6900 )
...
Bumps [object_store](https://github.com/apache/arrow-rs ) from 0.5.3 to 0.5.4.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/compare/object_store_0.5.3...object_store_0.5.4 )
---
updated-dependencies:
- dependency-name: object_store
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-08 09:40:11 +00:00
dcba47ab58
feat: allow the compactor to process all known partitions ( #6887 )
...
* feat: `PartitionRepo::list_ids`
* refactor: `CatalogPartitionsSource` => `CatalogToCompactPartitionsSource`
* feat: allow the compactor to process all known partitions
Closes #6648 .
* docs: improve
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
---------
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2023-02-08 09:32:21 +00:00
bb825faf06
chore(deps): Bump toml from 0.7.1 to 0.7.2 ( #6899 )
...
Bumps [toml](https://github.com/toml-rs/toml ) from 0.7.1 to 0.7.2.
- [Release notes](https://github.com/toml-rs/toml/releases )
- [Commits](https://github.com/toml-rs/toml/compare/toml-v0.7.1...toml-v0.7.2 )
---
updated-dependencies:
- dependency-name: toml
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-08 09:11:04 +00:00
47b47f225b
chore: add more insta snapshots for initial setup ( #6894 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-07 23:44:59 +00:00
a2945a77a4
fix: Implement EXPLAIN, ORDER BY and default ordering ( #6864 )
...
* chore: Add more tests
* chore: Fix default ordering; implement ORDER BY
* feat: Add EXPLAIN support
* chore: Add additional tests to validate GROUP BY expansion
* chore: More test cases for TZ, and failing log scalar function
2023-02-07 22:18:52 +00:00
d4e07eb6ac
chore: Update target_level_upgrade_split to use insta snapshots for partition verificiation ( #6892 )
2023-02-07 20:36:46 +00:00
ea0fc79340
chore: Update tests in compactor2 to ue insta ( #6891 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-07 19:58:43 +00:00
beb22f0490
Merge pull request #6880 from influxdata/ntran/c2-skip
...
feat: Do not skip partition based on num files and size until we reach compaction plan
2023-02-07 14:19:50 -05:00
c5025d6271
chore: merge main to branch
2023-02-07 14:04:03 -05:00
aab6ac7424
refactor: address review comments
2023-02-07 13:35:11 -05:00
76172daf39
feat: to prevent OOMs/crrash that will lead to skip compaction, let us compact L1 files when they are large enough ( #6877 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-07 17:46:44 +00:00
776ec41384
fix(ingester2): always maintain persist interval
...
This fixes an issue where persistence that does not ever complete blocks
the periodic enqueuing of persist tasks - this leads to the amount of
buffered data in the buffer tree increasing, and the persist queue depth
stays the same instead of draining the buffer.
This is an issue as the queue depth is designed to act as the
back-pressure of the ingester - once the depth exceeds a configurable
limit, further writes are rejected until the queue has drained
sufficiently (50%).
After this commit, stalled persistence (i.e. object store outage) will
not prevent the queue depth from growing, which should enable the
saturation protection to kick in.
2023-02-07 17:48:07 +01:00
db51d77eb8
docs: even more detailed info about skipped partitions
2023-02-07 10:13:58 -05:00
f56023b222
feat: use insta_snapshots ( #6884 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-07 15:08:47 +00:00
4ffd7fcc68
test: persist timer & wal rotation
...
Adds a unit test covering WAL rotation, buffer persistence & WAL file
deletion.
2023-02-07 15:52:11 +01:00
6297ad206f
Merge branch 'main' into ntran/c2-skip
2023-02-07 09:38:23 -05:00
646349c52d
docs: addd comments to explain further the skip
2023-02-07 09:34:40 -05:00
997cca67a3
fix: `--compaction-process-once` should exit ( #6886 )
...
- do not wait for a non-empty partition result (this doesn't make sense
if we are not running endlessly)
- modify entry point to allow the compactor to exit on its own (this is
normally not allowed for other server types)
2023-02-07 13:52:17 +00:00
e179887ede
fix: metric buckets ( #6883 )
2023-02-07 12:07:45 +00:00
3a4364b8f8
fix: partition-done metrics/logs ( #6882 )
...
Ignore partitions that where throttled or filtered due to the "not
unique" combo.
This is in line w/ the "partitions source", so the metric for "partition
in" and "partition out" line up.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-07 11:01:33 +00:00
65e7ff779c
feat: verified test output display for compactor ( #6859 )
...
* feat: verified test output display for compactor
* fix: demacrofy assert_parquet_files
* fix: demacrofiy assert_parquet_files_split
* refactor: compute min/max without sentinels
* fix: clippy
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-07 10:49:38 +00:00
25e1c9d7fa
Merge pull request #6873 from influxdata/dom/cache-table-limit
...
perf(router): cheap table limit enforcement + better errors
2023-02-07 10:13:05 +00:00
9e8785f9c1
Merge branch 'main' into dom/cache-table-limit
2023-02-07 10:04:50 +00:00
6ec44d697a
chore: cleanup
2023-02-06 21:41:35 -05:00
24ebd923a2
chore: Merge branch 'ntran/c2-skip' of https://github.com/influxdata/influxdb_iox into ntran/c2-skip
2023-02-06 21:39:24 -05:00
56ca3186b6
chore: turn test assert on
2023-02-06 21:39:17 -05:00
Marco Neumann
Dom
dependabot[bot]
Cole Abbeduto
Andrew Lamb
Stuart Carnie
Nga Tran