* chore: clean up runtime code for python setup
ccd5d22aab introduced working but temporary code for setting up the
python runtime environment. This cleans that up:
* refactor various find_python() functionality into virtualenv.rs
* refactor PYTHONHOME calculation to virtualenv.rs:find_python_install()
* adjust init_pyo3() to temporarily set PYTHONHOME based on
virtualenv.rs:find_python_install() as this is the only place it is
needed (indeed, venv activation scripts try to remove it)
Importantly, virtualenv.rs:find_python_install() tries to find the
python build standalone runtime based on a few heuristics. This function
could be improved in the fullness of time to, eg, be configured via a
build parameter.
Also, virtualenv.rs:find_python() can be used pre and post venv
activation. Before entering the venv, it will use find_python_install()
which is useful for things like setting up the initial venv. After
entering the venv, it will honor VIRTUAL_ENV (as set by
virtualenv.rs:initialize_venv()) to find python, which is important for
install packages with pip and having them installed into the venv.
* chore: update README_processing_engine.md for default venv
* chore: add bug reference for venv migrations with python minor releases
* chore: add security URLs to README_processing_engine.md
* chore: find_python_home() returns Option<PathBuf>. Thanks Jackson Newhouse
* fix: manually set sys.prefix, exec_prefix and sys.path
When activating a venv in the shell, sys.base_prefix and
sys.base_exec_prefix should be set to the installation location while
sys.prefix and sys.exec_prefix should be set to the venv dir.
Unfortunately, when initialize_venv() and init_pyo3() are called, we
can't use Py_InitializeFromConfig() to set any of these and certain
platforms are unable to find python-build-standalone. For now, we'll
temporarily set PYTHONHOME in init_pyo3() to the installation location
to make python work. By setting it at this point in the code, sys.prefix
and sys.exec_prefix end up also being set to the installation location,
which is fine when not under a venv, but is different from when entering
an venv.
To address this, in PYTHON_INIT.call_once() and when VIRTUAL_ENV is set
(which initialize_venv() will have set at this point), manually set
sys.prefix and sys.exec_prefix to what is in VIRTUAL_ENV.
Similarly, when activating a venv in the shell, sys.path is appended to
have the venv's site-packages dir. Previously we were setting PYTHONPATH
in initialize_venv() which ensures that the venv's site-packages dir is
in sys.path, but this ends up having the venv's site-packages dir first
in sys.path. To correct this, don't set PYTHONPATH any more and instead
adjust PYTHON_INIT.call_once() to append the venv's site-packages dir to
sys.path when VIRTUAL_ENV is set.
Finally, when exiting init_pyo3(), unconditionally unset PYTHONHOME when
VIRTUAL_ENV is set (like activation scripts do) and restore/unset when
it isn't.
Prior to these changes, all target incorrectly had the venv's
site-packages first in sys.path and OSX and Windows additionally had an
incorrect sys.prefix and sys.exec_prefix. With these initialization
changes in place, the runtime environment for the plugins is much closer
to that of a shell activated venv.
- breaking change, replaced `--parquet-mem-cache-size-mb` and env var for
it with `--parquet-mem-cache-size` (takes value now in percentage or
MB), now defaults to 20% of total available memory
- force snapshotting is set at 50%
- datafusion mem pool is set to 20%
closes: https://github.com/influxdata/influxdb/issues/26009
* feat(ci): fetch and configure for python-build-standalone binaries
* fix: make the process engine usable on windows
* feat(ci): build with python-build-standalone (and drop musl)
* fix(ci): set rpath on Linux and libpath on OSX in ci
* fix: set PYTHONHOME everywhere and PYTHONPATH on Windows
* chore(ci): update to use more recent ci-packager-next
* fix(ci): adjust validate to allow certain dynamically linked libraries
* chore: remove install_influxdb.sh (using install_influxdb3.sh instead)
* chore(install_influxdb3.sh): update for processing engine and release builds
* fix: temporarily use rpm --nodeps until compile with old GLIBC
* feat(ci): build docker with python-build-standalone
* chore: add README_processing_engine.md
* chore: add a few more details to README_processing_engine.md
* fix(ci): use patchelf --set-rpath
Not all patchelf versions support --add-rpath for appending to the
RPATH, but --set-path can be used with a colon-separated list. Use
--set-rpath first for maximum compatibility.
* chore: update README_processing_engine.md for standalone local builds
* fix(Dockerfile): also use patchelf --set-rpath
* chore: update code comment for accuracy
* chore: typos, grammar and formatting change in README_processing_engine.md
* chore: update README_processing_engine.md for Docker arm64 (thanks Jackson)
* deduplicate QueryParams->QueryRequest and Format->QueryFormat
* move WriteParams into influxdb3_types crate
* DRY up client HTTP request handling code in *RequestBuilder.send
methods.
* DRY up a bunch of other non-Builder http request handling
Partially fixes https://github.com/influxdata/influxdb/issues/24672
* move most HTTP req/resp types into `influxdb3_types` crate
* removes the use of locally-scoped request type structs from the `influxdb3_client` crate
* fix plugin dependency/package install bug
* it looks like the `DELETE` http method was being used where `POST` was expected for `/api/v3/configure/plugin_environment/install_packages` and `/api/v3/configure/plugin_environment/install_requirements`
This commit restructures our tests to look like Enterprise in their
layout. We break cli.rs into it's own module, combine the server tests
and cli tests under one lib.rs file and handle the changes for
visibility and import paths needed to make things work. the packages
tests have been cfged out as a module so that it would not need to be
added on a per test basis. Note that those tests fail locally for me
currently, but it seems like we weren't testing these in CI at the
moment.
There is no issue for this.
* feat: introduce parquet caching in query path
This commit scans the parquet files that will be used in query to check
if they can be cached. There are three conditions to satisfy,
- not cached already
- cache has enough space
- file times overlap with the cache policy times
closes: https://github.com/influxdata/influxdb/issues/25906
* refactor: rename env var
This updates trigger creation to load the plugin file before creating the trigger.
Another small change is to make Github references use filenames and paths identical to what they would be in the plugin-dir. This makes it a little easier to have the plugins repo local and develop against it and then be able to reference the same file later with gh: once it's up on the repo.
This refactors plugins and triggers so that plugins no longer need to be "created". Since plugins exist in either the configured local directory or on the Github repo, a user now only needs to create a trigger and reference the plugin filename.
Closes#25876
This change allows *both* the write and query commands to accept input
via stdin, string, or by a file. With this change larger queries are more
feasible to write as they can now be written in a file and smaller
writes via a string are now possible. This also makes the program work
more like people would expect it to, especially on unix based systems.
This commit also contains three tests to make sure the functionality works
as expected.
Closes#25772Closes#25892
* feat: first stab at locally updating parquet cache
closes: https://github.com/influxdata/influxdb/issues/25887
* refactor: use enums to separate out the modes
This commit introduced the `Immediate` and `Eventual` modes for
fulfilling the cache request. In immediate mode since the data is
readily available to be cached, we can avoid extra requests to object
store.
part of: https://github.com/influxdata/influxdb/issues/25887
This commit does a few key things:
- Removes the 72 hour query and write restrictions in Core
- Limits the queries to a default number of parquet files. We chose 432
as this is about 72 hours using default settings for the gen1
timeblock
- The file limit can be increased, but the help text and error message
when exceeded note that query performance will likely be degraded as
a result.
- We warn users to use smaller time ranges if possible if they hit this
query error
With this we eliminate the hard restriction we have in place, but
instead create a soft one that users can choose to take the performance
hit with. If they can't take that hit then it's recomended that they
upgrade to Enterprise which has the compactor built in to make
performant historical queries.
This updates plugins so that they will reload the code if the local file is modified. Github pugins continue to be loaded only once when they are initially created or loaded on startup.
This will make iterating on plugin development locally much easier.
Closes#25863
* feat: Add request plugin capability
Adds the request plugin type. Triggers can be bound to an API endpoint at /api/v3/engine/<path>. Requests will get yielded to the plugin with the query parameters, request parameters, and request body.
I didn't implement the test endpoint for this plugin type as it seems much more natural for users to save the file and make a new request. Once #25863 is done it'll make it very easy.
Closes#25862
* chore: fix spelling in error message
Although the `format` in the request is used, the value coming
through the header is parsed earlier. So, when that lookup in
the header fails an error is returned (`InvalidMimeType`).
In this commit, there are extra checks to allow the default `Accept`
header values that come from the browser by defaulting it to `json`
closes: https://github.com/influxdata/influxdb/issues/25874
* feat(processing_engine): Add cron plugins and triggers to the processing engine.
* feat(processing_engine): switch from 'cron plugin' to 'schedule plugin', use TimeProvider.
* feat(processing_engine): add test for test scheduled plugin.
* feat: improve plugin logging interface
Updates the plugin log functions so they can take any number of Python objects which will be converted into a single log line string.
Closes#25847
* refactor: udpate on PR feedback
* feat: return better plugin execution errors
This sets up the framework for fleshing out more useful plugin execution errors that get returned to the user during testing. We'll also want to capture these for logging in system tables.
Also fixes a test that was broken in previous commit on time limits. Didn't show up because of the feature flag.
* fix: compile errors without system-py feature
This updates the v1 /query API hanlder to handle InfluxDB v1's unique
query response structure when GROUP BY clauses are provided.
The distinction is in the addition of a "tags" field to the emitted series
data that contains a map of the GROUP BY tags along with their distinct
values associated with the data in the "values" field.
This required splitting the QueryExecutor into two query paths for InfluxQL
and SQL, as this allowed for handling InfluxQL query parsing in advance
of query planning.
A set of snapshot tests were added to check that it all works.
This commit sets InfluxDB 3 Core to have a 72 hour limit for queries and
writes. What this means is that writes that contain historical data
older than 72 hours will be rejected and queries will filter out data
older than 72 hours. Core is intended to be a recent timeseries database
and performance over data older than 72 hours will degrade without a
garbage collector, a core feature of InfluxDB 3 Enterprise. InfluxDB 3
Enterprise does not have this write or query limit in place.
Note that this does *not* mean older data is deleted. Older data is
still accessible in object storage as Parquet files that can still be
used in other services and analyzed with dataframe libraries like pandas
and polars.
This commit does a few things:
- Uses timestamps in the year 2065 for tests as these should not break
for longer than many of us will be working in our lifetimes. This is
only needed for the integration tests as other tests use the
MockProvider for time.
- Filters the buffer and persisted files to only show data newer than
3 days ago
- Fixes the integration tests to work with the fact that writes older
than 3 days are rejected
Jackson Newhouse
wayne
Jamie Strandboge
Michael Gattozzi
praveen-influx
Trevor Hilton
Paul Dix
wayne