Go to file
Carol (Nichols || Goulding) 37746173d9 feat: Change azure object store to only get config from args, not env 2021-03-04 10:14:42 -05:00
.circleci feat: smaller IOx build images 2021-02-25 15:05:54 +00:00
.github feat: add buf linting in CI 2021-02-23 19:46:45 +00:00
arrow_deps chore: use newly added Arrow's Expr::is_not_null function 2021-03-03 11:46:49 -05:00
benches refactor: consolidate line protocol schema creation into data_types and port code to use it (#663) 2021-01-15 17:29:30 -05:00
data_types feat: implement gRPC API and migrate influxdb_iox_client to use it (#853) 2021-03-02 17:51:46 +00:00
docker ci: Remove nightly-2021-02-12 2021-03-02 16:38:41 +00:00
docs docs: Include INFLUXDB_IOX_OBJECT_STORE in env.example 2021-03-03 11:29:35 +00:00
generated_types feat: add basic gRPC health service (#862) 2021-02-25 13:24:12 +00:00
google_types feat: encoding of standard gRPC error details payloads (#846) 2021-02-22 18:03:17 +00:00
influxdb2_client refactor: add arc clone lint 2021-02-15 12:52:12 +00:00
influxdb_iox_client fix: build of influxdb_iox_client with flight disabled (#925) 2021-03-04 12:10:18 +00:00
influxdb_line_protocol refactor: add arc clone lint 2021-02-15 12:52:12 +00:00
influxdb_tsm refactor: add arc clone lint 2021-02-15 12:52:12 +00:00
ingest refactor: Switch to parking_lot::Mutex 2021-02-22 13:51:31 -05:00
logfmt refactor: Switch to parking_lot::Mutex 2021-02-22 13:51:31 -05:00
mem_qe chore: Clean up Cargo.tomls (#754) 2021-02-04 18:56:02 -05:00
mutable_buffer feat: Make read_filter work for mutable buffer and read buffer (#882) 2021-03-01 16:50:29 +00:00
object_store feat: Change azure object store to only get config from args, not env 2021-03-04 10:14:42 -05:00
packers refactor: add arc clone lint 2021-02-15 12:38:19 +00:00
panic_logging refactor: add arc clone lint 2021-02-15 12:38:19 +00:00
query chore: use newly added Arrow's Expr::is_not_null function 2021-03-03 11:46:49 -05:00
read_buffer refactor: enable clipp Self 2021-03-02 15:51:13 +00:00
server refactor: Use `vec![]` instead of inlined strings, for legibility, in tests (#911) 2021-03-03 15:16:37 +00:00
src feat: Box the Command::Server variant as clippy recommends 2021-03-04 10:14:42 -05:00
test_helpers refactor: Switch to parking_lot::Mutex 2021-02-22 13:51:31 -05:00
tests feat: MVP Management CLI (#907) 2021-03-03 17:37:55 +00:00
tools feat: tool to graphically display plans (#803) 2021-02-15 13:03:12 +00:00
wal refactor: Enable parking lot for dependencies 2021-02-22 13:37:35 -05:00
.editorconfig style: Add editor config 2021-02-02 13:28:02 +01:00
.gitignore feat: Optimize Column Selection 2021-02-26 14:28:46 -05:00
.kodiak.toml chore: Add kodiac config 2021-02-11 00:52:21 +01:00
CONTRIBUTING.md feat: tool to graphically display plans (#803) 2021-02-15 13:03:12 +00:00
Cargo.lock feat: MVP Management CLI (#907) 2021-03-03 17:37:55 +00:00
Cargo.toml feat: MVP Management CLI (#907) 2021-03-03 17:37:55 +00:00
Dockerfile feat: Simplify Dockerfile 2021-03-02 18:06:53 +00:00
Dockerfile.dockerignore feat(docker): add one-command Dockerfile 2021-03-01 15:45:27 -08:00
LICENSE-APACHE fix: Add LICENSE (#430) 2020-11-10 12:10:07 -05:00
LICENSE-MIT fix: Add LICENSE (#430) 2020-11-10 12:10:07 -05:00
README.md feat(docker): add one-command Dockerfile 2021-03-01 15:45:27 -08:00
buf.yaml feat: add basic gRPC health service (#862) 2021-02-25 13:24:12 +00:00
build.rs feat: Include GIT_HASH in startup log message (#829) 2021-02-18 10:50:14 +00:00
rust-toolchain chore: update to latest version of arrow + update code (#486) 2020-11-25 14:46:35 -05:00
rustfmt.toml refactor: consolidate line protocol schema creation into data_types and port code to use it (#663) 2021-01-15 17:29:30 -05:00

README.md

InfluxDB IOx

InfluxDB IOx (short for Iron Oxide, pronounced InfluxDB "eye-ox") is the future core of InfluxDB, an open source time series database. The name is in homage to Rust, the language this project is written in. It is built using Apache Arrow and DataFusion among other things. InfluxDB IOx aims to be:

  • The future core of InfluxDB; supporting industry standard SQL, InfluxQL, and Flux
  • An in-memory columnar store using object storage for persistence
  • A fast analytic database for structured and semi-structured events (like logs and tracing data)
  • A system for defining replication (synchronous, asynchronous, push and pull) and partitioning rules for InfluxDB time series data and tabular analytics data
  • A system supporting real-time subscriptions
  • A processor that can transform and do arbitrary computation on time series and event data as it arrives
  • An analytic database built for data science, supporting Apache Arrow Flight for fast data transfer

Persistence is through Parquet files in object storage. It is a design goal to support integration with other big data systems through object storage and Parquet specifically.

For more details on the motivation behind the project and some of our goals, read through the InfluxDB IOx announcement blog post. If you prefer a video that covers a little bit of InfluxDB history and high level goals for InfluxDB IOx you can watch Paul Dix's announcement talk from InfluxDays NA 2020. For more details on the motivation behind the selection of Apache Arrow, Flight and Parquet, read this.

Project Status

This project is very early and in active development. It isn't yet ready for testing, which is why we're not producing builds or documentation yet. If you're interested in following along with the project, drop into our community Slack channel #influxdb_iox. You can find links to join here.

We're also hosting monthly tech talks and community office hours on the project on the 2nd Wednesday of the month at 8:30 AM Pacific Time. The first InfluxDB IOx Tech Talk is on December 9th and you can find details here.

Quick Start

To compile and run InfluxDB IOx from source, you'll need a Rust compiler and a flatc FlatBuffers compiler.

Build a Docker Image

To build a Docker image in one command:

docker build .

To enable caching, and to respect Dockerfile.dockerignore, we strongly suggest that you enable BuildKit.

  • Use Docker version 18.09 or later
  • Enable BuildKit by default by setting { "features": { "buildkit": true } }
    • ...or run docker build . with env var DOCKER_BUILDKIT=1

Cloning the Repository

Using git, check out the code by cloning this repository. If you use the git command line, this looks like:

git clone git@github.com:influxdata/influxdb_iox.git

Then change into the directory containing the code:

cd influxdb_iox

The rest of the instructions assume you are in this directory.

Installing Rust

The easiest way to install Rust is by using rustup, a Rust version manager. Follow the instructions on the rustup site for your operating system.

By default, rustup will install the latest stable verison of Rust. InfluxDB IOx is currently using a nightly version of Rust to get performance benefits from the unstable simd feature. The exact nightly version is specified in the rust-toolchain file. When you're in the directory containing this repository's code, rustup will look in the rust-toolchain file and automatically install and use the correct Rust version for you. Test this out with:

rustc --version

and you should see a nightly version of Rust!

Installing flatc

InfluxDB IOx uses the FlatBuffer serialization format for its write-ahead log. The flatc compiler reads the schema in generated_types/wal.fbs and generates the corresponding Rust code.

Install flatc >= 1.12.0 with one of these methods as appropriate to your operating system:

Once you have installed the packages, you should be able to run:

flatc --version

and see the version displayed.

You won't have to run flatc directly; once it's available, Rust's Cargo build tool manages the compilation process by calling flatc for you.

Installing clang

An installation of clang is required to build the croaring dependency - if it is not already present, it can typically be installed with the system package manager.

clang --version
Apple clang version 12.0.0 (clang-1200.0.32.27)
Target: x86_64-apple-darwin20.1.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

Specifying Configuration

IOx is designed for running in modern containerized environments. As such, it takes its configuration as environment variables.

You can see a list of the current configuration values by running influxdb_iox --help, as well as the specific subcommand config options such as influxdb_iox server --help.

Should you desire specifying config via a file, you can do so using a .env formatted file in the working directory. You can use the provided example as a template if you want:

cp docs/env.example .env

Compiling and Starting the Server

InfluxDB IOx is built using Cargo, Rust's package manager and build tool.

To compile for development, run:

cargo build

which will create a binary in target/debug that you can run with:

./target/debug/influxdb_iox

You can compile and run with one command by using:

cargo run

When compiling for performance testing, build in release mode by using:

cargo build --release

which will create the corresponding binary in target/release:

./target/release/influxdb_iox

Similarly, you can do this in one step with:

cargo run --release

The server will, by default, start an HTTP API server on port 8080 and a gRPC server on port 8082.

Writing and Reading Data

Each IOx instance requires a writer ID. This can be set three ways:

  • set an environment variable INFLUXDB_IOX_ID=42
  • set a flag --writer-id 42
  • send an HTTP PUT request:
curl --request PUT \
  --url http://localhost:8080/iox/api/v1/id \
  --header 'Content-Type: application/json' \
  --data '{
  "id": 42
  }'

To write data, you need a destination database. This is set via HTTP PUT, identifying the database by org company and bucket sensors:

curl --request PUT \
  --url http://localhost:8080/iox/api/v1/databases/company_sensors \
  --header 'Content-Type: application/json' \
  --data '{
}'

Data can be stored in InfluxDB IOx by sending it in line protocol format to the /api/v2/write endpoint. Data is stored by organization and bucket names. Here's an example using curl with the organization name company and the bucket name sensors that will send the data in the tests/fixtures/lineproto/metrics.lp file in this repository, assuming that you're running the server on the default port:

curl -v "http://127.0.0.1:8080/api/v2/write?org=company&bucket=sensors" --data-binary @tests/fixtures/lineproto/metrics.lp

To query stored data, use the /api/v2/read endpoint with a SQL query. This example will return all data in the company organization's sensors bucket for the processes measurement:

curl -v -G -d 'org=company' -d 'bucket=sensors' --data-urlencode 'sql_query=select * from processes' "http://127.0.0.1:8080/api/v2/read"

Health Checks

The HTTP API exposes a healthcheck endpoint at /health

$ curl http://127.0.0.1:8080/health
OK

The gRPC API implements the gRPC Health Checking Protocol. This can be tested with grpc-health-probe

$ grpc_health_probe -addr 127.0.0.1:8082 -service influxdata.platform.storage.Storage
status: SERVING

Contributing

We welcome community contributions from anyone!

Read our Contributing Guide for instructions on how to make your first contribution.

Architecture and Technical Documenation

There are a variety of technical documents describing various parts of IOx in the docs directory.