This commit is a major refactor for the code base. It mainly does four
things:
1. Splits code shared between the internal IOx repository and this one
into it's own repo over at https://github.com/influxdata/influxdb3_core
2. Removes any docs or anything else that did not relate to this project
3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to
declare dependencies and versions to keep all crates in sync and sets
all others to use `<dep>.workspace = true` unless it's an optional
dependency
4. Set the top level Cargo.toml to point to the core crates as git
dependencies
With this any changes specific to Edge will be contained here, updating
deps will be a PR over in `influxdata/influxdb3_core`, and we can prove
out the viability for this model to use for IOx.
Adds `skopeo` and gcloud CLI to CI image. This should eventually replace
our manual installation during the `deploy_releases` step:
d8d097c183/.circleci/config.yml (L493-L527)
This step costs us 20min during CD, which is ridiculous. A follow-up PR
later this week will use the CI image instead of an ad hoc installation.
Also see https://github.com/influxdata/idpe/issues/17098 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
This will break `perf_image` until the new CI image is built due to the
newly required `--all-tags` parameter to `docker push` that isn't
available for the docker version we run on buster.
Kafka is now sufficiently tested via the `write_buffer` crate. The
end2end tests can now use the in-memory mock implementation or -- if
servers can only be controlled via CLI -- the file-based implementation.
This refactors the write buffer a bit for:
- **Testing:** Add generic tests for the Kafka and the mocking
implementation. The same interface can be used easily add new
implementations (e.g. via Redis, filesystem, ...).
- **Partition on Write:** The caller of the writer operation must now
specify the partition/sequencer ID. The implicit partitioning of the
Kafka writer would have lead to broken data since we must never spill
entries w/ the same primary key over multiple partitions. At the
moment we will only use partition 0 but we can easily implement
better logic in the future.
- **Improved Mocking:** The mocked implementation now simulates a system
that feels more real. Especially the handling around multiple streams
and "write while read" has been improved. This will be helpful for
testing and for new features like seeking (during replay). A solid
realistic mock also helps us to ensure that the tests using the mock
do not rely on unrealistic behavior too much.
* ci: verify gpg signatures of dependencies in CI
Verify signatures of bazel and influxdb2.
* fix: harden curl in Dockerfile.ci
Co-authored-by: pierwill <pierwill@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
If the layer is cached, docker build won't print the output of the RUN command.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
This will let us stop installing it in the CI job and will let us run
the influxdb2_client integration tests in a Docker container based on
this image locally as well.
That only installs bazel for the `root` user, but we'd like bazel
installed for the `rust` user as well.
Also clean up the installer and run this all as one Docker RUN command.
Michael Gattozzi
Marco Neumann
Carol (Nichols || Goulding)
Dom Dwyer
Andrew Lamb
Raphael Taylor-Davies
Andrew Lamb
Edd Robinson
Carol (Nichols || Goulding)
pierwill
Marko Mikulicic
Luke Bond