Includes two main components:
* Removal of expired data from `PersistedFiles`.
* Modified `ChunkFilter` that precisely excludes expired data from query
results even if the expired data hasn't been removed from the object
store yet.
---------
Co-authored-by: Michael Gattozzi <mgattozzi@influxdata.com>
This commit adds another sub command to load generator that allows
creating constrained throughput of line protocol data shared between
given number of writers. It uses a very naive approach to generate data
which may contain some duplicates. However it is useful when you need to
generate a very specific amount of data per writer. This approach has
been used to reproduce OOMs observed in perf tests.
This does not create a report like other sub-commands, and it also does
not observe any errors in the writes.
pro PR: https://github.com/influxdata/influxdb_pro/pull/886
WAL replay currently loads _all_ WAL files concurrently running into
OOM. This commit adds a CLI parameter `--wal-replay-concurrency-limit`
that would allow the user to set a lower limit and run WAL replay again.
closes: https://github.com/influxdata/influxdb/issues/26481
* feat: add retention period to catalog
* fix: handle humantime parsing error properly
* refactor: use new iox_http_util types
---------
Co-authored-by: Michael Gattozzi <mgattozzi@influxdata.com>
* refactor: Use iox_http_util::Request instead of hyper::Request
* refactor: Use iox_http_util::RequestBuilder instead of hyper::Request::builder
* refactor: Use iox_http_util::empty_request_body instead of Body::empty
* refactor: Use iox_http_util::bytes_to_request_body instead of Body::from
* refactor: Use http_body::Body instead of hyper::body::HttpBody
* refactor: Use iox_http_util::Response instead of hyper::Response
* refactor: Use iox_http_util::ResponseBuilder instead of hyper::Response::builder
* refactor: Use iox_http_util::empty_response_body instead of Body::empty
* refactor: Use iox_http_util::bytes_to_response_body instead of Body::from
* refactor: Use iox_http_util::stream_results_to_response_body instead of Body::wrap_stream
* refactor: Use the read_body_bytes_for_tests helper fn
Currently when there is an OOM while snapshotting, the process keeps
going without crashing. This behaviour is observed in main (commit:
be25c6f52b). This means the wal files keep
increasing to a point that restarts never can replay all the files.
This is happening because of the distribution of memory, in enterprise
especially there is no need for an ingester to be allocated just 20% for
datafusion memory pool (which runs the snapshot) as parquet cache is not
in use at all. This 20% is too conservative for an ingester, so instead
of redistributing the memory settings based on the mode it's running,
a separate write path executor is introduced in this commit with no
bound on memory (still uses `GreedyMemoryPool` under the hoold with
`usize::MAX` as upper limit). This means write path executor will always
run into OOM and stop the whole process.
Also, it is important to let snapshotting process use as much memory
as it needs as without that, the buffer will keep getting bigger and run
into OOM anyway.
closes: https://github.com/influxdata/influxdb/issues/26422
The current build order:
1. Copy in full source tree
2. Fetch python build dependencies
3. Build
The issue with this is any source tree change causes the python dependencies to
be re-fetched, then the rust components of those deps rebuilt.
Instead, copy in just the .circleci directory, which informs the python build
dependency fetch. After we have the python deps, then copy in the full source
tree and build.
This commit brings in v3 catalog changes that's been made in pro. The
pro PR is https://github.com/influxdata/influxdb_pro/pull/866. There is
no real change to catalog that affects core but it is important that
core and enterprise catalogs versions are in sync so that it's easier to
track changes to catalog across core and enterprise.
Includes:
- catalog log file conversions and v3 log file
- catalog snapshot file conversions and v3 snapshot file
- update deserialization function to return v3 version of log/snapshot
file after applying conversion
- `CatalogSnapshot` and all the types that it depends on that implement
`Snapshot` trait are moved to `mod.rs` as only recent version of
snapshot should implement that trait
* chore: update to latest core
* chore: allow CDLA permissive 2 license
* chore: update insta snapshot for new internal df tables
* test: update assertion in flightsql test
* fix: object store size hinting workaround in clap_blocks
Applied a workaround from upstream to strip size hinting from the object
store get request options. See:
https://github.com/influxdata/influxdb_iox/issues/13771
* fix: query_executor tests use object store size hinting workaround
* fix: insta snapshot test for show system summary command
* chore: update windows- crates for advisories
* chore: update to latest sha on influxdb3_core branch
* chore: update to latest influxdb3_core rev
* refactor: pr feedback
* refactor: do not use object store size hint layer
Instead of using the ObjectStoreStripSizeHint layer, just provide the
configuration to datafusion to disable the use of size hinting from
iox_query.
This is used in IOx and not relevant to Monolith.
* fix: use parquet cache for get_opts requests
* test: that the parquet cache is being hit from write buffer
* chore: Ensure Parquet sort key is serialised with snapshots
* chore: PR feedback, rename state variable to match intent
* chore: Use `Default` trait to implement `TableBuffer::new`
* chore: Fix change in file size with extra metadata
* chore: Add rustdoc for `sort_key` field
* refactor: remove unused Key type from write buffer
The write buffer had a Key variant for handling the experimental v3
write API that was phased out and removed from an earlier iteration
of influxdb3.
* refactor: remove key column type from last cache
* feat: `/ping` API contains versioning headers
Further, the product version can be modified by updating the metadata in
the `influxdb3_process` `Cargo.toml`.
* chore: PR feedback
* chore: placate linter
* fix: do not allow operator token from being deleted
closes: https://github.com/influxdata/influxdb_pro/issues/819
* refactor: address PR feedback
* fix: add a word and clarifying colon
* fix: failing test
---------
Co-authored-by: Peter Barnett <peter.barnett03@gmail.com>
* feat: allow health,ping,metrics to opt out of auth
This commit introduces `--disable-authz <DISABLE_AUTHZ_RESOURCES>`. The
options for `DISABLE_AUTHZ_RESOURCES` are health, ping and metrics. By
default all these resources will be guarded
closes: https://github.com/influxdata/influxdb_pro/issues/774
* chore: update influxdb3/src/commands/helpers.rs
space after comma in help text
Co-authored-by: Trevor Hilton <thilton@influxdata.com>
* chore: update influxdb3/src/help/serve.txt
space after comma in help text
Co-authored-by: Trevor Hilton <thilton@influxdata.com>
* chore: update influxdb3/src/help/serve_all.txt
space after comma in help text
Co-authored-by: Trevor Hilton <thilton@influxdata.com>
* refactor: use statics to reduce clones/copies
---------
Co-authored-by: Trevor Hilton <thilton@influxdata.com>
This removes the `pop_back` methods from the lvc and uses truncate instead
so that it ensures that the cache is at its desired size.
Adjusted logic in the eviction of LVC to try to be a bit more efficient
* feat: support `Basic $TOKEN` for all apis
closes: https://github.com/influxdata/influxdb/issues/25833
* refactor: address PR feedback to return MalformedRequest error when `:` is used more than once in user-pass pair
* refactor: change the message sent back for malformed auth header
wayne warren
praveen-influx
Trevor Hilton
Stuart Carnie
Carol (Nichols || Goulding)
Adrian Thurston
peterbarnett03