influxdb/README_processing_engine.md

The InfluxDB 3 Processing engine is an embedded Python VM for running code inside the database to process and transform data. This document discusses how the processing engine is built within InfluxDB. For usage instructions, see: https://docs.influxdata.com/influxdb3/core/

See the 'Discussion' section for more information on why the processing engine is implemented the way it is.

Implementation

InfluxDB uses the PYO3 crate to build InfluxDB with an embedded python and the processing engine is enabled by default.

PYO3 will then inspect the system to find a python runtime to build and link against. The resulting influxdb3 binary will be dynamically linked to the libpython that PYO3 found during the build. Eg, on a typical Debian or Ubuntu system, if you install the following, then InfluxDB can be built against the system's python:

# build dependencies
$ sudo apt-get install build-essential pkg-config libssl-dev clang lld \
    git protobuf-compiler python3 python3-dev python3-pip

# runtime dependencies
$ sudo apt-get install python3 python3-pip python3-venv

# build
$ cargo build

The choice of python can be influenced by setting the PYTHONHOME environment variable for cargo build or creating a PYO3_CONFIG_FILE file for more specialized setups (such as 'Official builds', below). For details, see https://pyo3.rs/main/building-and-distribution.html

In order for InfluxDB to successfully use the python it was built against, the same libpython version as well as the full runtime environment of the python install (ie, its standard library) must be available to InfluxDB in a location that it can find it. Building against the system python can be a reasonable choice for users who target their builds to a specific release of an OS as InfluxDB will simply use the installed python from the system.

Official builds

To provide a consistent, robust and maintained python environment for InfluxDB that is portable across a range of operating systems, InfluxData's official InfluxDB is built against a pre-built release of python-build-standalone (a CPython standalone python distribution). For a given release of InfluxDB, official builds will use the same version of python for all install methods and operating systems.

The following operating systems and architectures are currently supported:

• Linux amd64/arm64 (tar.gz, deb and rpm)
• Darwin arm64 (tar.gz)
• Windows amd64 (zip)
• Docker (Linux amd64/arm64)

Due to constraints with python-build-standalone and statically linking, all builds are dynamically linked to python-build-standalone's libpython as well as a few OS-specific libraries. Specifically:

• Linux (seen with ldd and strings on the binary):
  • python-build-standalone is linked against glibc and is compatible with glibc 2.17+
  • influxdb3 is linked against libpython from python-build-standalone as well as glibc (official builds target glibc 2.23+)
• Darwin (seen with otool -L; cross-compiled with osxcross):
  • python-build-standalone is linked against:
    • CoreFoundation.framework/Versions/A/CoreFoundation compatibility version 150.0.0
    • libSystem.B.dylib compatibility version 1.0.0
  • influxdb3 is linked against:
    • CoreFoundation.framework/Versions/A/CoreFoundation compatibility version 150.0.0
    • IOKit.framework/Versions/A/IOKit compatibility version 1.0.0
    • libiconv.2.dylib compatibility version 7.0.0
    • libobjc.A.dylib compatibility version 1.0.0
    • libSystem.B.dylib compatibility version 1.0.0
    • Security.framework/Versions/A/Security compatibility version 1.0.0
    • SystemConfiguration.framework/Versions/A/SystemConfiguration compatibility version 1.0.0
• Windows (seen with dumpbin /HEADERS ... and dumpbin /DEPENDENTS ...):
  • python-build-standalone claims Windows 8/Windows Server 2012 or newer. Specifically, it has:
    • 14.42 linker version
    • 6.00 operating system version
    • 6.00 subsystem version
  • influxdb3 has:
    • 2.40 linker version
    • 4.00 operating system version
    • 5.02 subsystem version

At a high level, the build process for Official builds consists of:

1. downloading an appropriate build of python-build-standalone for the target OS and architecture from https://github.com/astral-sh/python-build-standalone/releases

2. unpacking the python-build-standalone build on disk

3. creating a pyo3 build configuration file to point to the unpacked directory and setting the PYO3_CONFIG_FILE environment variable to point to it. Eg (on Linux):

   implementation=CPython
   version=3.11
   shared=true
   abi3=false
   lib_name=python3.11
   lib_dir=/path/to/python-standalone/python/lib
   executable=/path/to/python-standalone/python/bin/python3.11
   pointer_width=64
   build_flags=
   suppress_build_script_link_lines=false
   

   PYO3 will try to auto-detect the location which can work well with a system python, but not with an unpacked python-build-standalone. While the PYO3_PYTHON environment variable can be used to point to the unpacked directory (eg, PYO3_PYTHON=/path/to/python-standalone/python/bin/python3), this was not sufficient. Defining the build configuration in the PYO3_CONFIG_FILE correctly worked for all supported environments with our current build process

4. run PYO3_CONFIG_FILE=/path/to/pyo3_config_file.txt cargo build to build InfluxDB

5. adjust the library search paths for Linux and Darwin so libpython can found (see 'Discussion', below)

6. create official build artifacts:

• Linux/Darwin tar.gz contain influxdb3 and python/...
• Linux deb and rpm contain /usr/bin/influxdb3 and /usr/lib/influxdb3/python
• Windows zip contains influxdb3, *.dll files from python/... and python/... (see 'Discussion', below)

Licensing information for python-build-standalone as distributed by official builds of InfluxDB can found in the python/LICENSE.md.

With the above, influxdb3 can be run in the normal way. Eg, on Linux:

# unpack tarball to /here
$ tar -C /here --strip-components=1 -zxvf /path/to/build/influxdb3-<VERSION>_linux_amd64.tar.gz

# without processing engine
$ /here/influxdb3 serve ...
$ /here/influxdb3 query ...

# with processing engine (/path/to/plugins/.venv created automatically)
$ mkdir /path/to/plugins
$ /here/influxdb3 serve --plugin-dir /path/to/plugins ...        # server
$ /here/influxdb3 create database foo                            # client
$ /here/influxdb3 test schedule_plugin -d foo testme.py          # client
... <plugins can use whatever is in /path/to/plugins/.venv> ...
$ /here/influxdb3 install package bar                            # client
... <plugins can now 'import bar'> ...
$ /here/influxdb3 test schedule_plugin -d foo testme.py          # client

# with processsing engine and alternate venv (/path/to/other-venv created automatically)
# start server to create/use other-venv
$ /here/influxdb3 serve --plugin-dir /path/to/plugins --virtual-env-location /path/to/other-venv
...
$ /here/influxdb3 test schedule_plugin -d foo testme.py          # client
... <plugins can use whatever is in /path/to/other-venv> ...
$ /here/influxdb3 install package bar                            # client
... <plugins can now 'import bar'> ...
$ /here/influxdb3 test schedule_plugin -d foo testme.py          # client

# with processsing engine and alternate pre-created venv
$ /here/python/bin/python3 -m venv /path/to/another-venv         # create another-venv
$ source /path/to/another-venv/bin/activate
(venv)$ python3 -m pip install foo
(venv)$ python3 -m pip freeze > /path/to/another-venv/requirements.txt
...
(venv)$ deactivate

# start server to use another-venv
$ /here/influxdb3 serve --plugin-dir /path/to/plugins --virtual-env-location /path/to/another-venv
... <plugins can now use whatever is in /path/to/another-venv> ...

$ /here/influxdb3 test schedule_plugin -d foo testme.py          # client
$ /here/influxdb3 install package bar                            # client
... <plugins can now 'import bar' in /path/to/another-venv> ...
$ /here/influxdb3 test schedule_plugin -d foo testme.py          # client

Local development with python-build-standalone

Local development with python-build-standalone currently consists of:

1. download python-build-standalone and unpack it somewhere

   • get from https://github.com/astral-sh/python-build-standalone/releases
   • based on your host OS, choose one of aarch64-apple-darwin-install_only_stripped.tar.gz, aarch64-unknown-linux-gnu-install_only_stripped.tar.gz, x86_64-pc-windows-msvc-shared-install_only_stripped.tar.gz, x86_64-unknown-linux-gnu-install_only_stripped.tar.gz

2. create pyo3_config_file.txt to match the unpacked dir and downloaded python version. Eg, if downloaded and unpacked a 3.11.x version to /tmp/python:

   $ cat ./pyo3_config_file.txt
   implementation=CPython
   version=3.11
   shared=true
   abi3=false
   lib_name=python3.11
   lib_dir=/tmp/python/lib
   executable=/tmp/python/bin/python3.11
   pointer_width=64
   build_flags=
   suppress_build_script_link_lines=false
   

3. build with:

   # note: PYO3_CONFIG_FILE must be an absolute path
   $ PYO3_CONFIG_FILE=${PWD}/pyo3_config_file.txt cargo build --features "aws,gcp,azure,jemalloc_replacing_malloc"
   

4. Linux/OSX: patch up the binary to find libpython:

   # linux
   $ patchelf --set-rpath '$ORIGIN/python/lib' ./target/<profile>/influxdb3
   
   # osx (be sure to match the libpython version with what you downloaded)
   $ install_name_tool -change '/install/lib/libpython3.11.dylib' '@executable_path/python/lib/libpython3.11.dylib' ./target/<profile>/influxdb3
   

5. Linux/OSX: put the python runtime in the expected location (XXX: may be possible at run time to see where the libpython we are using is and adjust the code to base the location of the runtime on that). Eg, if unpacked to /tmp/python:

   $ test -e ./target/<profile>/python || ln -s /tmp/python ./target/<profile>/python
   

6. run with:

   $ mkdir -p /path/to/plugin/dir
   
   # linux and osx (if can't find libpython or the runtime, check previous steps)
   $ ./target/<profile>/influxdb3 ... --plugin-dir /path/to/plugin/dir
   
   # windows requires moving the binary into the python-build-standalone unpack directory
   $ cp ./target/<profile>/influxdb3 \path\to\python-standalone\python
   # run influxdb with
   $ \path\to\python-standalone\python\influxdb3.exe ... --plugin-dir \path\to\plugin\dir
   

Discussion

Why python-build-standalone?

python-build-standalone is designed to be portable, maintained and permissively licensed. It is purpose-built for embedding and being redistributable and has a good upstream maintenance story (https://github.com/astral-sh) with lots of users and a corporate sponsor.

An alternative to using a standalone python distribution is to use the system python. While this can be a reasonable choice on systems where the python version and installation locations can be relied upon, it is not a good choice for official builds since users would have to ensure they had a python installation that met InfluxDB's requirements and because the myriad of operating systems, architectures and installed python versions would be a problem to support.

By choosing python-build-standalone, InfluxDB should deliver a consistent experience across OSes and architectures for all users as well as providing a reasonable maintenance story.

Which builds of python-build-standalone are used?

python-build-standalone provides many different builds. Official InfluxDB builds use the following python-build-standalone recommended builds:

• aarch64-apple-darwin-install_only_stripped.tar.gz
• aarch64-unknown-linux-gnu-install_only_stripped.tar.gz
• x86_64-unknown-linux-gnu-install_only_stripped.tar.gz
• x86_64-pc-windows-msvc-shared-install_only_stripped.tar.gz

How will InfluxData maintain the embedded interpreter?

The https://github.com/astral-sh project performs timely builds of CPython micro-releases for python-build-standalone based on the release cadence of upstream Python. InfluxData need only update the build to pull in the new micro-release for security and maintenance releases. This is done by updating the PBS_DATE and PBS_VERSION environment variables in .circleci/config.yaml. See that file and .circleci/scripts/fetch-python-standalone.bash for details.

astral-sh creates new builds for CPython minor releases as they become available from upstream Python. Updating the official builds to pull in a new minor release is straightforward, but processes for verifying builds of InfluxDB with the new python-build-standalone minor release are TBD.

References:

• https://www.python.org/dev/security/
• https://mail.python.org/mailman3/lists/security-announce.python.org/
• https://mail.python.org/archives/list/security-announce@python.org/latest
• https://github.com/astral-sh/python-build-standalone/releases

How is python-build-standalone licensed?

Release builds of python-build-standalone are permissively licensed and contain no copyleft code.

The licensing information from release builds of python-build-standalone are obtained by extracting the python/PYTHON.json and python/licenses/* files from the <arch>-debug-full.tar.zst (Linux/Darwin) and <arch>-pgo-full.tar.zst release tarballs, placing them in the python/licenses directory of the InfluxDB build and generating a python/LICENSE.md file with provenance information.

Linux builds are dynamically linked against glibc (which is permitted by the LGPL without copyleft attachment). InfluxDB does not statically link against glibc nor does it redistribute libc (et al) in official builds.

Why not just statically link with, eg, MUSL?

In an ideal world, InfluxDB would build against a version of python-build-standalone and statically link against it and not have to worry about dynamic library compatibility. Unfortunately, this is not possible for many reasons:

• static python-build-standalone builds for Darwin are not available and doing so may have license implications
• static python-build-standalone builds for Windows are not stable and considered brittle
• static python-build-standalone builds for Linux/arm64 (aarch64) are not available
• static python-build-standalone builds for Linux/amd64 (x86_64) are available using MUSL libc, but:
  • because they are static, they cannot load compiled Python extensions (aka, 'wheels' that have compiled C, Rust, etc code instead of pure python) outside of the Python standard library, greatly diminishing the utility of the processing engine. This is a limitation of ELF
  • there are historical performance issues with python and MUSL

It is theoretically possible to statically link glibc, but in practice this is technically very problematic and statically linking glibc has copyleft attachment.

What about alpine?

Because MUSL can't be used with python-build-standalone without crippling the InfluxDB processing engine, MUSL builds that are compatible with Alpine are not available at this time. Alpine users can choose one of:

• build InfluxDB locally on Alpine against Alpine's system python
• run official InfluxDB within a chroot that contains glibc
• run official InfluxDB with gcompat (untested)

See https://wiki.alpinelinux.org/wiki/Running_glibc_programs for details.

InfluxData may provide Alpine builds at a future date.

GLIBC portability is a problem. How will you address that?

glibc is designed with portability and uses 'compat symbols' to achieve backward compatibility. Most 3rd party applications for Linux use the system's glibc in some fashion and this is possible because of 'compat symbols' and this has worked very well for many, many years.

In essence, 'compat symbols' let glibc and the linker choose a particular implementation of the function. All symbols in glibc are versioned and when a library function changes in an incompatible way, glibc keeps the old implementation in place (with the old symbol version) while adding the new implementation with a new symbol version. In this manner, if an application is compiled and linked against glibc 2.27, it will only ever lookup symbols that are 2.27 or earlier. When 2.28 comes out, it updates any symbols it needs to 2.28, leaving the rest as they are. When the application linked against 2.27 runs on a system with 2.28, everything is ok since 2.28 will resolve all the 2.27 symbols in the expected way the application needs.

Where portability becomes a problem is when the application is linked against a newer version of glibc than is on the system. If the aforementioned application compiled and linked against 2.27 was run on a system with 2.19, it would fail to run because the symbol versions it is looking up (ie, anything from 2.20 and later) are not available.

Unfortunately for developers seeking portability, compiling and linking against the system's glibc means the application will reference the latest available symbols in that glibc. There is no facility for telling the linker to only use symbols from a particular glibc version and earlier. It's also difficult to tell the linker to use an alternate glibc separate from the system's. As a result, glibc-using software seeking wide Linux portability typically needs to be compiled on an older system with a glibc with the desired version.

python-build-standalone and rust both support systems with glibc 2.17+, which is covers distributions going back to 2014 (CentOS/RHEL 7 (EOL), Debian 8 (Jessie; EOL), Ubuntu 14.04 LTS (EOL), Fedora 21, etc.

How does InfluxDB find the correct libpython and the python runtime?

For the best user experience, users should not have to perform any extra setup to use the InfluxDB processing engine. This is achieved by:

• Using an appropriate PYO3_CONFIG_FILE file during the build (see 'Official builds', above)

• Build artifacts putting the runtime in an expected location (see 'Official builds, above)

• At runtime, ensuring that Linux and Darwin binaries look for the runtime in the expected location. Ideally this would be done with linker arguments at builds time, but current (alpha) builds adjust the library search paths like so:

  # linux
  $ patchelf --set-rpath '$ORIGIN/python/lib:$ORIGIN/../lib/influxdb3/python/lib' target/.../influxdb3
  
  # osx
  $ install_name_tool -change '/install/lib/libpython3.NN.dylib' \
      '@executable_path/python/lib/libpythonNN.dylib' target/.../influxdb3
  $ rcodesign sign target/.../influxdb3  # only with osxcross' install_name_tool
  

  This is required, in part, due to how python-build-standalone is built. When using osxcross's version of install_name_tool, must also use rcodesign from apple-codesign to re-sign the binaries (Apple's install_name_tool does this automatically). Rust may gain support for setting arbitrary rpaths at some point.

• The Windows zip file for the current (alpha) builds has copies of the top-level DLL files from the 'python/' directory alongside influxdb3. Windows requires that the dynamically linked DLLs needed by the application are either in the same directory as the binary or found somewhere in PATH (and open source tooling doesn't seem to support modifying this). For user convenience, the *.dll files are shipped alongside the binary on Windows to avoid having to setup the PATH. Rust believes this shouldn't be handled by rustc. This may be addressed in a future release

There is no pip.exe on Windows. Why?

Historical python-build-standalone didn't ship pip.exe on Windows. From upstream: "The Windows distributions have pip installed however no Scripts/pip.exe, Scripts/pip3.exe, and Scripts/pipX.Y.exe files are provided because the way these executables are built isn't portable. (It might be possible to change how these are built to make them portable.)

To use pip, run python.exe -m pip. (It is generally a best practice to invoke pip via python -m pip on all platforms so you can be explicit about the python executable that pip uses.)"

While newer python-build-standalone releases seem to have addressed this, the recommendation to call with python -m pip still holds true.

What limitations are there?

See https://github.com/influxdata/influxdb/issues?q=is%3Aissue%20state%3Aopen%20label%3Av3