* feat(python): update to python-build-standalone 3.13.2
References:
- https://github.com/influxdata/influxdb/issues/26044
* fix: update fetch-python-standalone.bash to properly set 'executable'
* fix: use PYO3_CONFIG_FILE to find PYTHONHOME.
* fix: add comment about PYO3_CONFIG_FILE.
* fix: remove ensure_pyo3().
* fix: add some sleep so catalog is updated.
---------
Co-authored-by: Jackson Newhouse <jnewhouse@influxdata.com>
* fix(circleci): add librt.so to list of acceptable libraries
* feat(circleci): check for glibc portability
* fix(circleci): remove rpm --nodeps workaround in rpm validate
Now that we have glibc portability in rpm builds, we no longer need the
'rpm --nodeps' workaround and can go back to 'yum localinstall'.
Closes#26011
* chore: update README_processing_engine.md for glibc portability
* feat(ci): fetch and configure for python-build-standalone binaries
* fix: make the process engine usable on windows
* feat(ci): build with python-build-standalone (and drop musl)
* fix(ci): set rpath on Linux and libpath on OSX in ci
* fix: set PYTHONHOME everywhere and PYTHONPATH on Windows
* chore(ci): update to use more recent ci-packager-next
* fix(ci): adjust validate to allow certain dynamically linked libraries
* chore: remove install_influxdb.sh (using install_influxdb3.sh instead)
* chore(install_influxdb3.sh): update for processing engine and release builds
* fix: temporarily use rpm --nodeps until compile with old GLIBC
* feat(ci): build docker with python-build-standalone
* chore: add README_processing_engine.md
* chore: add a few more details to README_processing_engine.md
* fix(ci): use patchelf --set-rpath
Not all patchelf versions support --add-rpath for appending to the
RPATH, but --set-path can be used with a colon-separated list. Use
--set-rpath first for maximum compatibility.
* chore: update README_processing_engine.md for standalone local builds
* fix(Dockerfile): also use patchelf --set-rpath
* chore: update code comment for accuracy
* chore: typos, grammar and formatting change in README_processing_engine.md
* chore: update README_processing_engine.md for Docker arm64 (thanks Jackson)
* fix: bind to correct port for e2e tests
This also fixes up some log messages on server start for naming
* chore: do notpass value in TEST_LOG env var to CI tests
* feat(circleci): add inclusivity checks
* chore(circleci): adjust package-validation for inclusive language
* chore: update tests for inclusive language
This changes our CI to use cargo-nextest which is faster and does not
have issues around global statics. Since it runs each test in it's
own process we don't have to worry about tests stepping on each other's
toes in this regard. It also updates the CI to ignore the current
cargo deny failure as we can't do anything until the arrow crates are
upgraded.
* chore: clean up heappy, pprof, and jemalloc
Setup the use of jemalloc as default allocator using tikv-jemallocator
crate instead of tikv-jemalloc-sys.
Removed heappy and pprof, and also cleaned up all the mutually exclusive
compiler flags for using heappy as the allocator.
* chore: remove heappy from ci
For releases we need to have Docker images and binary images available for the
user to actually run influxdb3. These CI changes will build the binaries on a
release tag and the Docker image as well, test, sign, and publish them and make
them available for download.
Co-Authored-By: Brandon Pfeifer <bpfeifer@influxdata.com>
* feat: /ping API to serve version
The /ping API was added, which is served at GET and
POST methods. The API responds with a JSON body
containing the version and revision of the build.
A new crate was added, influxdb3_process, which
takes the process_info.rs module from the influxdb3
crate, and puts it in a separate crate so that other
crates (influxdb3_server) can depend on it. This was
needed in order to have access to the version and
revision values, which are generated at build time,
in the HTTP API code of influxdb3_server.
A E2E test was added to check that /ping works.
E2E TestServer can now have logs emitted using the
TEST_LOG environment variable.
* chore: Update to Rust 1.77.0
This is a fairly quiet upgrade. The only changes are some lints around
`OpenOptions` that were added to clippy between 1.75 and this version
and they're small changes that either remove unecessary function calls
or add a needed function call.
* fix: cargo-deny by using the --locked flag
This commit re-enables the limits test after making a fix that has it
run <1 second on my laptop vs the old behavior of >=30 seconds. It does
so by constructing one single write_lp request to create 1995 tables
rather than 1995 individual requests that make a table. This is far more
efficient.
This commit is a major refactor for the code base. It mainly does four
things:
1. Splits code shared between the internal IOx repository and this one
into it's own repo over at https://github.com/influxdata/influxdb3_core
2. Removes any docs or anything else that did not relate to this project
3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to
declare dependencies and versions to keep all crates in sync and sets
all others to use `<dep>.workspace = true` unless it's an optional
dependency
4. Set the top level Cargo.toml to point to the core crates as git
dependencies
With this any changes specific to Edge will be contained here, updating
deps will be a PR over in `influxdata/influxdb3_core`, and we can prove
out the viability for this model to use for IOx.
Prior to this change we've had CI fail nightly because we can't push the
image to CI due to permissions issues. The problem is that
influxdata/influxdb_iox is the one that actually has access to push that
data to quay.
This commit removes the nightly build and references to it as this image
is built nightly by the IOx team. If things break we have access to fix
it, but I don't think it'll be an issue.
* chore(deps): Update arrow and datafusion to 49.0.0
This commit copies in our dependency code from influxdb_iox in order for
us to be able to upgrade from a forked version of 46.0.0 to 49.0.0 of
both arrow and datafusion. Most of the important changes were around how
we consumed the crates in influxdb3(_server/_write). Those diffs are
particularly worth looking at as the rest was a straight copy and we
don't touch those crates in our development currently for influxdb3
edge.
* fix: regenerate workspace hack crate
* fix: Protobuf issues with incompatibility labels
* fix: Broken CI yaml
* fix: buf version
* fix: Only check IOx repo
* fix: Remove protobuf lint
* fix: Comment out call to protobuf-lint
We currently don't need or want to deploy influxdb as we're still
building out the Edge product. Maybe later for a demo, but for now it
just breaks CI and so this removes it.
This commit changes the circle-ci config to use influxdb3 rather than
iox in our ci config script as the repo is influxdb not influxdb_iox.
While we could probably strip out a lot more here as a first attempt to
get this to build release images and push them on main this will do just
fine.
* fix: build, upgrade rustc, and deps
This commit upgrades Rust to 1.75.0, the latest release. We also
upgraded our dependencies to stay up to date and to clear out any
uneeded deps from the lockfile. In order to make sure everything works
this also fixes the build by upgrading the workspace-hack crate using
cargo hikari and removing the `workspace.lint` that was in
influxdb3_write that didn't need to be there, probably from a merge
issue.
With this we can build influxdb3 as our default on main, but this alone
is not enough to fix CI and will be addressed in future commits.
* fix: warnings for influxdb3 build
This commit fixes the warnings emitted by `cargo build` when compiling
influxdb3. Mainly it adds needed lifetimes and removes uneccesary
imports and functions calls.
* fix: all of the clippy lints
This for the most part just applies suggested fixes by clippy with a few
exceptions:
- Generated type crates had additional allows added since we can't
control what code gets made
- Things that couldn't be automatically fixed were done so manually in
particular adding a Send bound for traits that created a Future that
should be Send
We also had to fix a build issue by adding a feature for tokio-compat
due to the upgrade of deps. The workspace crate was updated accordingly.
* fix: failing test due to rust panic message change
Inbetween rustc 1.72 and rustc 1.75 the way that error messages were
displayed when panicing changed. One of our tests depended on the output
of that behavior and this commit updates the error message to the new
form so that tests will pass.
* fix: broken cargo doc link
* fix: cargo formatting run
* fix: add workspace-hack to influxdb3 crates
This was the last change needed to make sure that the workspace-hack
crate CI lint would pass.
* fix: remove tests that can not run anymore
We removed iox code from this code base and as a result some tests
cannot be run anymore and so this commit removes them from the code base
so that we can get a green build.
* refactor: isolate docker build to script
* chore: add labels to docker image
* chore: export image as OCI
* chore: print image digest
* fix: convert to OCI BEFORE calculating digest
* fix: use digest of uploaded image, not of the local archive
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
As we've learned in #8048 and #8052, rustflags do NOT stack. Since we
only want to change one specific parameter (the debug feature), use the
env variable that cargo provides us.
**In contrast to the linked PRs, this only changes the test excution. Prod
builds remain untouched.**
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
- move `tokio_unstable` to cargo config, so all we can use it within
our code (e.g. for #7982)
- disable incremental builds for prod docker builds. this was tried
before but got lost at some point because build params weren't passed
to docker correclty
- fix `CARGO_NET_GIT_FETCH_WITH_CLI` for docker builds (env wasn't
passed through)
Jamie Strandboge
Trevor Hilton
Jackson Newhouse
praveen-influx
Paul Dix
Brandon Pfeifer
Michael Gattozzi
Marco Neumann
alespour