* chore: update core deps
- arrow/parquet deps are patched (as in core)
- three specific code changes to cope with changes in core crates
- TransitionPartitionId, use `from_parts` instead of `new`
- arrow buffers can take &[u8] directly without `to_vec()`/`vec!`
(used only in tests)
- `schema` and `influxdb_line_protocol` crates need `v3` feature enabled
* chore: update deny.toml
* chore: formatting and deny toml changes
Unicode-3.0 license is added to allowed licenses list, without it
end up with 19 errors (`zerovec`, `zerovec-derive` etc.)
* chore: address PR feedback
- move enabling v3 feature to root Cargo.toml
- added the upstream PR for datafusion-common that introduced RUSTSEC-2024-0384
* chore: update core deps
- arrow/parquet deps are patched (as in core)
- three specific code changes to cope with changes in core crates
- TransitionPartitionId, use `from_parts` instead of `new`
- arrow buffers can take &[u8] directly without `to_vec()`/`vec!`
(used only in tests)
- `schema` and `influxdb_line_protocol` crates need `v3` feature enabled
* chore: update deny.toml
* chore: formatting and deny toml changes
Unicode-3.0 license is added to allowed licenses list, without it
end up with 19 errors (`zerovec`, `zerovec-derive` etc.)
* chore: address PR feedback
- move enabling v3 feature to root Cargo.toml
- added the upstream PR for datafusion-common that introduced RUSTSEC-2024-0384
`cargo deny` was showing that no crate matched the advisory criteria for this [RUSTSEC advisory](https://rustsec.org/advisories/RUSTSEC-2024-0376.html), so this PR removes the ignore entry.
In addition, the `hashbrown` crate was causing a new audit failure, and updating it required that the `Zlib` license be added to our list of allowed licenses.
No issue for this, but it is blocking another PR at the moment (https://github.com/influxdata/influxdb/pull/25515).
We will need to wait on the RUSTSEC advisory to be resolved upstream,
i.e., by having tonic and hyper upgraded in core, before we can lift this
advisory ignore and use the latest versions of those crates.
This changes our CI to use cargo-nextest which is faster and does not
have issues around global statics. Since it runs each test in it's
own process we don't have to worry about tests stepping on each other's
toes in this regard. It also updates the CI to ignore the current
cargo deny failure as we can't do anything until the arrow crates are
upgraded.
The Persister trait was only implemented by a single type, because the
underlying ObjectStore interface has several ways of being mocked, we
mock that instead of the Persister interface.
This commit removes the Persister trait, and moves its interface/impl
directly on a single Persister type in the persister module of the
influxdb3_write crate.
deny.toml had some incorrect field names in license.exceptions, those
were fixed from 'crate' to 'name'.
The Persister trait was only implemented by a single type, because the
underlying ObjectStore interface has several ways of being mocked, we
mock that instead of the Persister interface.
This commit removes the Persister trait, and moves its interface/impl
directly on a single Persister type in the persister module of the
influxdb3_write crate.
deny.toml had some incorrect field names in license.exceptions, those
were fixed from 'crate' to 'name'.
* fix: wal skip persist and notify if empty buffer (#25211)
* fix: wal skip persist and notify if empty buffer
This fixes the WAL so that it will skip persisting a file and notifying the file notifier if the wal buffer is empty.
* fix: fix last cache persist test
* fix: make ParquetChunk fields and mod chunk pub (#25219)
* fix: make ParquetChunk fields and mod chunk pub
This doesn't affect anything in the OSS version, but these changes are
needed for Pro as part of our compactor work.
* fix: cargo deny failure
* fix: query bugs with buffer (#25213)
* fix: query bugs with buffer
This fixes three different bugs with the buffer. First was that aggregations would fail because projection was pushed down to the in-buffer data that de-duplication needs to be called on. The test in influxdb3/tests/server/query.rs catches that.
I also added a test in write_buffer/mod.rs to ensure that data is correctly queryable when combining with different states: only data in buffer, only data in parquet files, and data across both. This showed two bugs, one where the parquet data was being doubled up (parquet chunks were being created in write buffer mod and in queryable buffer. The second was that the timestamp min max on table buffer would panic if the buffer was empty.
* refactor: PR feedback
* fix: fix wal replay and buffer snapshot
Fixes two problems uncovered by adding to the write_buffer/mod.rs test. Ensures we can replay wal data and that snapshots work properly with replayed data.
* fix: run cargo update to fix audit
* feat: use host identifier prefix in object store paths (#25224)
This enforces the use of a host identifier prefix in all object store
paths (currently, for parquet files, catalog files, and snapshot files).
The persister retains the host identifier prefix, and uses it when
constructing paths.
The WalObjectStore also holds the host identifier prefix, so that it can
use it when saving and loading WAL files.
The influxdb3 binary requires a new argument 'host-id' to be passed that
is used to specify the prefix.
* feat: add `system.parquet_files` table (#25225)
This extends the system tables available with a new `parquet_files` table
which will list the parquet files associated with a given table in a
database.
Queries to system.parquet_files must provide a table_name predicate to
specify the table name of interest.
The files are accessed through the QueryableBuffer.
In addition, a test was added to check success and failure modes of the
new system table query.
Finally, the Persister trait had its associated error type removed. This
was somewhat of a consequence of how I initially implemented this change,
but I felt cleaned the code up a bit, so I kept it in the commit.
* fix: un-pub QueryableBuffer and fix compile errors (#25230)
* refactor: Make Level0Duration part of WAL (#25228)
* refactor: Make Level0Duration part of WAL
I noticed this during some testing and cleanup with other PRs. The WAL had its own level_0_duration and the write buffer had a different one, which would cause some weird problems if they weren't the same. This refactors Level0Duration to be in the WAL and fixes up the tests.
As an added bonus, this surfaced a bug where multiple L0 blocks getting persisted in the same snapshot wasn't supported. So now snapshot details can have many files per table.
* fix: have persisted files always return in descending data time order
* fix: sort record batches for test verification
* fix: main (#25231)
* feat: Add last cache create/delete to WAL (#25233)
* feat: Add last cache create/delete to WAL
This moves the LastCacheDefinition into the WAL so that it can be serialized there. This ended up being a pretty large refactor to get the last cache creation to work through the WAL.
I think I also stumbled on a bug where the last cache wasn't getting initialized from the catalog on reboot so that it wouldn't actually end up caching values. The refactored last cache persistence test in write_buffer/mod.rs surfaced this.
Finally, I also had to update the WAL so that it would persist if there were only catalog updates and no writes.
Fixes#25203
* fix: typos
* feat: Catalog apply_catalog_batch only updates if new (#25236)
* feat: Catalog apply_catalog_batch only updates if new
This updates the Catalog so that when applying a catalog batch it only updates the inner catalog and bumps the sequence number and updated tracker if there are new updates in the batch. Also does validation that the catalog batch schema is compatible with any existing.
Closes#25205
* feat: only persist catalog when updated (#25238)
* chore: ignore sqlx rustsec advisory (#25252)
* feat: Add FileIndex type to influxdb3_index
This commit does two important things:
1. It creates a new influxdb3_index crate under influxdb3_pro to contain
all indexing logic and types that we might create for influxdb3_pro
2. Creates our first index type the FileIndex which is part of #20
Note we're starting off with just file ids as this will let us set up
the logic for creating and working with the `FileIndex` inside of the
compactor first. Later we can add row groups as that logic is a bit more
complicated in nature.
The `FileIndex` contains methods to lookup, insert, and delete items
from the index as needed and an associated test to make sure it works as
expected.
Note that the `FileIndex` is meant to have one created for each database
table that has an index created for it. Later on when it's being
integrated into the compactor a `FileIndex` will be returned per
compaction of a given table. We'll later integrate this into the
`WriteBuffer` for querying as well as adding this to the WAL so that
indexes can be recreated as needed.
---------
Co-authored-by: Paul Dix <paul@pauldix.net>
Co-authored-by: Trevor Hilton <thilton@influxdata.com>
* refactor: Move Catalog into influxdb3_catalog crate
This moves the catalog and its serialization logic into its own crate. This is a precursor to recording more catalog modifications into the WAL.
Fixes#25204
* fix: cargo update
* fix: add version = 2 to deny.toml
* fix: update deny.toml
* fix: add CCO to deny.toml
There are a bunch of dependencies in `Cargo.lock` that are related to
mysql. These are NOT compiled at all, and are also not part of `cargo
tree`. The reason for the inclusion is a bug in cargo:
https://github.com/rust-lang/cargo/issues/10801
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
We don't really need the color feature. Removing it has several
benifits:
- less dependencies
- one less MPL license
- no `atty` (only used by tests but has a security bug)
`time` 0.1 suffers from [RUSTSEC-2020-0071] and many upstream crates
have tried to remove it for years. The last dependency is
1. `chrono-english`
2. `chrono` (default features)
3. `chrono` (oldtime)
4. `time` 0.1
`chrono-english` doesn't seem to be super well maintained, but I
couldn't find a nice replacement for it. Luckily the master branch of
`chrono-english` is already fixed, so let's just directly use that.
[RUSTSEC-2020-0071]: https://rustsec.org/advisories/RUSTSEC-2020-0071
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
I just don't wanna rip out parts of our software stack because
someone gets cold feet.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
XML parsing lib for the Azure SDK is unmaintained and reportedly
contains integer overflow / panic issues in the parsing functionality.
Low risk ignore as it is used when talking to Azure only. The Azure SDK
is in the progress of being removed as a dependency.
* ci: fix cargo deny
* chore: downgrade `socket2`, version 0.4.5 was yanked
* chore: rename `query` to `iox_query`
`query` is already taken on crates.io and yanked and I am getting tired
of working around that.
Adds a config file for cargo-deny[1] that runs the following checks:
* advisory-db[2] RUSTSEC checks for deps (like cargo-audit)
* errors if a dependency has been "yanked" from crates.io
* errors if attempting to use OpenSSL as a dependency.
The RUSTSEC checks copy over the whitelist from the current cargo-audit
config.
[1]: https://github.com/EmbarkStudios/cargo-deny
[2]: https://github.com/rustsec/advisory-db
Trevor Hilton
Jackson Newhouse
praveen-influx
Michael Gattozzi
wayne warren
Paul Dix
Marco Neumann
Carol (Nichols || Goulding)
Dom Dwyer
Andrew Lamb
Raphael Taylor-Davies
Jacob Marble