This commit restores the old behavior we had where new tags can be added
to a schema. To do this we made tags nullable and brings us in line with
our other products. These changes were made in this PR:
https://github.com/influxdata/influxdb3_core/pull/41.
Changes to accomplish this new behavior were:
- Queries now do not return an empty string for null tags instead they
are returned as null, or in many formats not at all.
- References to v1 for parsing and validating lines were removed as we
only have one path for doing so these days shared amongst all the
write_lp endpoints.
- We fixed failing tests that expected us to not be able to have new
tags or depended on that functionality indirectly
- Tests had their snapshot files updated to reflect that tags are
nullable by default
- Behavior for making a schema and checking whether a column can be null
were updated in a separate repo and integrated here
- The series_key is updated whenever we get a new tag added to the
schema
- New tests were added to show that you can add a new tag and that the
series key is updated as part of that
With the above changes we can now allow tags to be added again by users
like they would expect, especially with v1 and v2 apis and Telegraf
plugins.
* chore: clean up runtime code for python setup
ccd5d22aab introduced working but temporary code for setting up the
python runtime environment. This cleans that up:
* refactor various find_python() functionality into virtualenv.rs
* refactor PYTHONHOME calculation to virtualenv.rs:find_python_install()
* adjust init_pyo3() to temporarily set PYTHONHOME based on
virtualenv.rs:find_python_install() as this is the only place it is
needed (indeed, venv activation scripts try to remove it)
Importantly, virtualenv.rs:find_python_install() tries to find the
python build standalone runtime based on a few heuristics. This function
could be improved in the fullness of time to, eg, be configured via a
build parameter.
Also, virtualenv.rs:find_python() can be used pre and post venv
activation. Before entering the venv, it will use find_python_install()
which is useful for things like setting up the initial venv. After
entering the venv, it will honor VIRTUAL_ENV (as set by
virtualenv.rs:initialize_venv()) to find python, which is important for
install packages with pip and having them installed into the venv.
* chore: update README_processing_engine.md for default venv
* chore: add bug reference for venv migrations with python minor releases
* chore: add security URLs to README_processing_engine.md
* chore: find_python_home() returns Option<PathBuf>. Thanks Jackson Newhouse
* fix: manually set sys.prefix, exec_prefix and sys.path
When activating a venv in the shell, sys.base_prefix and
sys.base_exec_prefix should be set to the installation location while
sys.prefix and sys.exec_prefix should be set to the venv dir.
Unfortunately, when initialize_venv() and init_pyo3() are called, we
can't use Py_InitializeFromConfig() to set any of these and certain
platforms are unable to find python-build-standalone. For now, we'll
temporarily set PYTHONHOME in init_pyo3() to the installation location
to make python work. By setting it at this point in the code, sys.prefix
and sys.exec_prefix end up also being set to the installation location,
which is fine when not under a venv, but is different from when entering
an venv.
To address this, in PYTHON_INIT.call_once() and when VIRTUAL_ENV is set
(which initialize_venv() will have set at this point), manually set
sys.prefix and sys.exec_prefix to what is in VIRTUAL_ENV.
Similarly, when activating a venv in the shell, sys.path is appended to
have the venv's site-packages dir. Previously we were setting PYTHONPATH
in initialize_venv() which ensures that the venv's site-packages dir is
in sys.path, but this ends up having the venv's site-packages dir first
in sys.path. To correct this, don't set PYTHONPATH any more and instead
adjust PYTHON_INIT.call_once() to append the venv's site-packages dir to
sys.path when VIRTUAL_ENV is set.
Finally, when exiting init_pyo3(), unconditionally unset PYTHONHOME when
VIRTUAL_ENV is set (like activation scripts do) and restore/unset when
it isn't.
Prior to these changes, all target incorrectly had the venv's
site-packages first in sys.path and OSX and Windows additionally had an
incorrect sys.prefix and sys.exec_prefix. With these initialization
changes in place, the runtime environment for the plugins is much closer
to that of a shell activated venv.
Removed all of the shellcheck complaints that were showing up in my
editor.
Where possible, I just fixed them directly but I just told it to ignore
this one in the install scripts https://www.shellcheck.net/wiki/SC2059
Happy to go and fix that lint too I just wasn't sure if we prefer the
current approach, since it doesn't seem to be causing any problems.
- breaking change, replaced `--parquet-mem-cache-size-mb` and env var for
it with `--parquet-mem-cache-size` (takes value now in percentage or
MB), now defaults to 20% of total available memory
- force snapshotting is set at 50%
- datafusion mem pool is set to 20%
closes: https://github.com/influxdata/influxdb/issues/26009
While the url is https://www.influxdata.com/d/install_influxdb3.sh, what
is copied over to this url must (currently) be named install_influxdb.sh
from this repo otherwise we get a 404. Thanks Pete Barnett for
troubleshooting.
* feat(ci): fetch and configure for python-build-standalone binaries
* fix: make the process engine usable on windows
* feat(ci): build with python-build-standalone (and drop musl)
* fix(ci): set rpath on Linux and libpath on OSX in ci
* fix: set PYTHONHOME everywhere and PYTHONPATH on Windows
* chore(ci): update to use more recent ci-packager-next
* fix(ci): adjust validate to allow certain dynamically linked libraries
* chore: remove install_influxdb.sh (using install_influxdb3.sh instead)
* chore(install_influxdb3.sh): update for processing engine and release builds
* fix: temporarily use rpm --nodeps until compile with old GLIBC
* feat(ci): build docker with python-build-standalone
* chore: add README_processing_engine.md
* chore: add a few more details to README_processing_engine.md
* fix(ci): use patchelf --set-rpath
Not all patchelf versions support --add-rpath for appending to the
RPATH, but --set-path can be used with a colon-separated list. Use
--set-rpath first for maximum compatibility.
* chore: update README_processing_engine.md for standalone local builds
* fix(Dockerfile): also use patchelf --set-rpath
* chore: update code comment for accuracy
* chore: typos, grammar and formatting change in README_processing_engine.md
* chore: update README_processing_engine.md for Docker arm64 (thanks Jackson)
The distinct cache info for tables was not serialized in the catalog.
This fixes it, but also updates the catalog serialization to use the
snapshot type serialization from the Catalog type all the way down.
The Eq and PartialEq impls were removed from Catalog and InnerCatalog
as they were only used in tests, and wer replaced by pure insta snapshot
tests.
A test was added to check that the distinct cache serializes/deserializes
Fixed a bug in the distinct cache where projection that skipped column
in the cache hierarchy caused a panic.
This simplifies the display of the projection in the DistinctCacheExec
in EXPLAIN output to not include the column index, and only the name.
In #25927 we missed that JSON queries were broken despite having some
tests use the format. This fixes JSON queries such that they now
properly contain a comma between RecordBatches. This commit also
includes tests for the formats that now stream data back (CSV, JSON, and
JSON Lines) so that we won't run into this issue again.
Trevor Hilton
Jackson Newhouse
praveen-influx
Michael Gattozzi
wayne
Jamie Strandboge
Adam Curtis
peterbarnett03
clia