Go to file
dependabot[bot] fcd9d9e3e6
chore(deps): Bump hashbrown from 0.13.2 to 0.14.0
Bumps [hashbrown](https://github.com/rust-lang/hashbrown) from 0.13.2 to 0.14.0.
- [Changelog](https://github.com/rust-lang/hashbrown/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/hashbrown/compare/v0.13.2...v0.14.0)

---
updated-dependencies:
- dependency-name: hashbrown
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-06-07 11:52:16 -04:00
.cargo chore: enable frame-pointer (#7471) 2023-04-11 08:14:31 +00:00
.circleci fix: Remove temporary limiting of test threads 2023-05-08 20:24:35 -04:00
.config fix: garbage_collector no longer uses chrono-english so it can use the workspace hack crate 2023-05-01 11:31:42 -04:00
.github chore: exclude arrow and datafusion from dependabot (#6564) 2023-01-11 11:18:34 +00:00
arrow_util chore(deps): Bump comfy-table from 6.2.0 to 7.0.0 (#7944) 2023-06-07 08:19:53 +00:00
authz chore(deps): Bump tokio from 1.28.1 to 1.28.2 (#7885) 2023-05-29 09:11:24 +00:00
backoff build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
cache_system chore: clippy 2023-06-04 06:56:19 +10:00
clap_blocks chore(deps): Bump tempfile from 3.5.0 to 3.6.0 2023-06-07 08:21:40 +00:00
client_util build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
compactor refactor: migrate `iox_query` to use DataFusion statistics (#7908) 2023-06-02 09:18:59 +00:00
compactor_test_utils chore: use workspace dependencies for `object_store` 2023-05-26 07:03:42 -04:00
data_types feat: Validate custom partition templates on their creation 2023-06-07 11:38:12 -04:00
datafusion_util chore(deps): Bump url from 2.3.1 to 2.4.0 (#7925) 2023-06-05 14:24:42 +00:00
dml build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
docker fix: Remove uses of KAFKA_CONNECT, redpanda, and some talk of Kafka 2023-04-12 11:53:23 -04:00
docs chore(docs): Update tracing.md with latest jaeger (#7878) 2023-05-26 17:24:05 +00:00
executor chore(deps): Bump once_cell from 1.17.2 to 1.18.0 2023-06-05 02:03:15 +00:00
flightsql chore: clippy 2023-06-04 06:56:19 +10:00
garbage_collector chore(deps): Bump once_cell from 1.17.2 to 1.18.0 2023-06-05 02:03:15 +00:00
generated_types refactor: replace namespace with database for flight proto (#7910) 2023-06-05 16:53:09 +00:00
grpc-binary-logger build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
grpc-binary-logger-proto build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
grpc-binary-logger-test-proto build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
import_export build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
influxdb2_client chore(deps): Bump url from 2.3.1 to 2.4.0 (#7925) 2023-06-05 14:24:42 +00:00
influxdb_influxql_parser chore: clippy 😬 2023-06-06 16:40:34 +10:00
influxdb_iox chore(deps): Bump nu-ansi-term from 0.47.0 to 0.48.0 (#7936) 2023-06-07 09:02:33 +00:00
influxdb_iox_client chore(deps): Bump comfy-table from 6.2.0 to 7.0.0 (#7944) 2023-06-07 08:19:53 +00:00
influxdb_line_protocol chore(deps): Bump log from 0.4.17 to 0.4.18 (#7884) 2023-05-29 09:17:08 +00:00
influxdb_storage_client build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
influxdb_tsm chore: clippy 🧠 2023-06-04 07:23:11 +10:00
influxrpc_parser build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
ingester chore(deps): Bump tempfile from 3.5.0 to 3.6.0 2023-06-07 08:21:40 +00:00
ingester_query_grpc build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
ingester_test_ctx feat: Make a place to do partition template validation 2023-06-07 11:38:12 -04:00
iox_catalog fix: Use expect rather than returning error in a theoretically impossible case 2023-06-07 11:38:12 -04:00
iox_data_generator feat(parquet): reduce and limit buffering when writing parquet files (#7880) 2023-05-31 13:27:32 +00:00
iox_query chore: Update DataFusion pin (#7916) 2023-06-05 18:38:59 +00:00
iox_query_influxql feat: Push outer query time-range to subqueries 2023-06-06 16:33:01 +10:00
iox_query_influxrpc refactor: re-land #7815. (#7852) 2023-05-24 09:46:49 +00:00
iox_tests refactor: faster PG `ParquetFileRepo` (#7907) 2023-06-01 16:17:28 +00:00
iox_time build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
ioxd_common build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
ioxd_compactor build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
ioxd_garbage_collector build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
ioxd_ingester build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
ioxd_querier chore: use workspace dependencies for `object_store` 2023-05-26 07:03:42 -04:00
ioxd_router chore: use workspace dependencies for `object_store` 2023-05-26 07:03:42 -04:00
ioxd_test build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
logfmt chore(deps): Bump once_cell from 1.17.2 to 1.18.0 2023-06-05 02:03:15 +00:00
metric build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
metric_exporters build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
mutable_batch feat: Validate custom partition templates on their creation 2023-06-07 11:38:12 -04:00
mutable_batch_lp chore(deps): Bump criterion from 0.4.0 to 0.5.0 (#7856) 2023-05-24 09:08:37 +00:00
mutable_batch_pb fix: prevent panics in partitioning logic 2023-06-01 17:44:44 +02:00
mutable_batch_tests chore(deps): Bump criterion from 0.4.0 to 0.5.0 (#7856) 2023-05-24 09:08:37 +00:00
object_store_metrics chore: use workspace dependencies for `object_store` 2023-05-26 07:03:42 -04:00
observability_deps chore: Rerun cargo hakari manage-deps 2023-02-27 11:56:15 +01:00
panic_logging test: fix line number test 2023-05-23 14:55:44 +02:00
parquet_file chore: Update DataFusion pin (#7916) 2023-06-05 18:38:59 +00:00
parquet_to_line_protocol chore: Update DataFusion pin (#7916) 2023-06-05 18:38:59 +00:00
predicate chore: Update DataFusion pin (#7916) 2023-06-05 18:38:59 +00:00
querier refactor: migrate `iox_query` to use DataFusion statistics (#7908) 2023-06-02 09:18:59 +00:00
query_functions feat: `selector_first` with additional args (#7898) 2023-06-02 10:08:21 +00:00
router feat: Validate custom partition templates on their creation 2023-06-07 11:38:12 -04:00
schema build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
scripts fix: update proto path for parquet meta script (#7911) 2023-06-02 15:29:11 +00:00
service_common build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
service_grpc_catalog build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
service_grpc_flight refactor: replace namespace with database for flight proto (#7910) 2023-06-05 16:53:09 +00:00
service_grpc_influxrpc chore(deps): Bump regex from 1.8.3 to 1.8.4 (#7928) 2023-06-05 15:50:26 +00:00
service_grpc_namespace feat: Validate custom partition templates on their creation 2023-06-07 11:38:12 -04:00
service_grpc_object_store chore: use workspace dependencies for `object_store` 2023-05-26 07:03:42 -04:00
service_grpc_schema build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
service_grpc_table feat: Validate custom partition templates on their creation 2023-06-07 11:38:12 -04:00
service_grpc_testing build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
sharder chore(deps): Bump criterion from 0.4.0 to 0.5.0 (#7856) 2023-05-24 09:08:37 +00:00
sqlx-hotswap-pool build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
test_fixtures feat: introduce a new way of max_sequence_number for ingester, compactor and querier (#6692) 2023-01-26 10:52:47 +00:00
test_helpers chore(deps): Bump tempfile from 3.5.0 to 3.6.0 2023-06-07 08:21:40 +00:00
test_helpers_end_to_end chore(deps): Bump tempfile from 3.5.0 to 3.6.0 2023-06-07 08:21:40 +00:00
tools feat: tool to graphically display plans (#803) 2021-02-15 13:03:12 +00:00
trace chore(7618): trace ingester response encoding v2 (#7820) 2023-05-26 09:40:16 +00:00
trace_exporters feat(cli): Automatically send influx-trace-id, and improve help text (#7830) 2023-05-25 09:43:55 +00:00
trace_http build: remove unused dependencies from crates 2023-05-23 14:55:43 +02:00
tracker chore(deps): Bump lock_api from 0.4.9 to 0.4.10 (#7932) 2023-06-06 12:45:25 +00:00
trogging chore: clippy 2023-06-04 07:06:02 +10:00
wal refactor(`wal_inspect`): Make `LineProtoWriter` namespace unaware 2023-05-26 17:12:35 +01:00
wal_inspect Merge branch 'main' into savage/wal-regenerate-lp-catalog-support 2023-06-02 14:23:55 +00:00
workspace-hack chore(deps): Bump lock_api from 0.4.9 to 0.4.10 (#7932) 2023-06-06 12:45:25 +00:00
.editorconfig chore: editor config spacing for shell scripts 2022-12-13 11:12:11 +01:00
.gitattributes feat: implement jaeger-agent protocol directly (#2607) 2021-09-22 17:30:37 +00:00
.gitignore chore: gitignore pending insta snapshots (#6923) 2023-02-09 10:47:29 +00:00
.kodiak.toml chore: Set default to squash 2022-01-25 15:57:10 +01:00
.yamllint.yml ci: run yamllint 2021-11-29 15:11:15 +01:00
CONTRIBUTING.md test: document how to run tests (#4982) 2022-06-30 14:01:35 +00:00
Cargo.lock chore(deps): Bump hashbrown from 0.13.2 to 0.14.0 2023-06-07 11:52:16 -04:00
Cargo.toml chore(deps): Bump hashbrown from 0.13.2 to 0.14.0 2023-06-07 11:52:16 -04:00
Dockerfile chore: reduce prod image layer count (#7539) 2023-04-13 12:06:59 +00:00
Dockerfile.dockerignore fix: bring back GIT revision to our prod images (#3537) 2022-01-26 10:12:43 +00:00
LICENSE-APACHE fix: Add LICENSE (#430) 2020-11-10 12:10:07 -05:00
LICENSE-MIT fix: Add LICENSE (#430) 2020-11-10 12:10:07 -05:00
README.md docs(README): Update link to datafusion (#7821) 2023-05-19 11:14:59 +00:00
buf.yaml ci: run yamllint 2021-11-29 15:11:15 +01:00
deny.toml fix: Add chrono-english to the cargo-deny config to avoid pulling it in again in the future 2023-05-01 11:31:42 -04:00
integration-docker-compose.yml fix: Remove uses of KAFKA_CONNECT, redpanda, and some talk of Kafka 2023-04-12 11:53:23 -04:00
rust-toolchain.toml build: bump Rust version 2023-06-04 06:43:22 +10:00
rustfmt.toml chore: use Rust edition 2021 2021-10-25 10:58:20 +02:00

README.md

InfluxDB IOx

InfluxDB IOx (short for Iron Oxide, pronounced InfluxDB "eye-ox") is the core of InfluxDB, an open source time series database. The name is in homage to Rust, the language this project is written in. It is built using Apache Arrow and DataFusion among other technologies. InfluxDB IOx aims to be:

  • The core of InfluxDB; providing industry standard SQL, InfluxQL, and Flux
  • An in-memory columnar store using object storage for persistence
  • A fast analytic database for structured and semi-structured events (like logs and tracing data)
  • A system for defining replication (synchronous, asynchronous, push and pull) and partitioning rules for InfluxDB time series data and tabular analytics data
  • A system supporting real-time subscriptions
  • A processor that can transform and do arbitrary computation on time series and event data as it arrives
  • An analytic database built for data science, supporting Apache Arrow Flight for fast data transfer

Persistence is through Parquet files in object storage. It is a design goal to support integration with other big data systems through object storage and Parquet specifically.

For more details on the motivation behind the project and some of our goals, read through the InfluxDB IOx announcement blog post. If you prefer a video that covers a little bit of InfluxDB history and high level goals for InfluxDB IOx you can watch Paul Dix's announcement talk from InfluxDays NA 2020. For more details on the motivation behind the selection of Apache Arrow, Flight and Parquet, read this.

Platforms

Our current goal is that the following platforms will be able to run InfluxDB IOx.

  • Linux x86 (x86_64-unknown-linux-gnu)
  • Darwin x86 (x86_64-apple-darwin)
  • Darwin arm (aarch64-apple-darwin)

Project Status

This project is in active development, which is why we're not producing builds yet.

If you would like contact the InfluxDB IOx developers, join the InfluxData Community Slack and look for the #influxdb_iox channel.

We're also hosting monthly tech talks and community office hours on the project on the 2nd Wednesday of the month at 8:30 AM Pacific Time.

Get started

  1. Install dependencies
  2. Clone the repository
  3. Configure the server
  4. Compiling and Running (You can also build a Docker image to run InfluxDB IOx.)
  5. Write and read data
  6. Use the CLI
  7. Use InfluxDB 2.0 API compatibility
  8. Run health checks
  9. Manually call the gRPC API

Install dependencies

To compile and run InfluxDB IOx from source, you'll need the following:

Rust

The easiest way to install Rust is to use rustup, a Rust version manager. Follow the instructions for your operating system on the rustup site.

rustup will check the rust-toolchain file and automatically install and use the correct Rust version for you.

C/C++ Compiler

You need some C/C++ compiler for some non-Rust dependencies like zstd.

lld

If you are building InfluxDB IOx on Linux then you will need to ensure you have installed the lld LLVM linker. Check if you have already installed it by running lld -version.

lld -version
lld is a generic driver.
Invoke ld.lld (Unix), ld64.lld (macOS), lld-link (Windows), wasm-ld (WebAssembly) instead

If lld is not already present, it can typically be installed with the system package manager.

protoc

Prost no longer bundles a protoc binary. For instructions on how to install protoc, refer to the official gRPC documentation.

IOx should then build correctly.

Postgres

The catalog is stored in Postgres (unless you're running in ephemeral mode). Postgres can be installed via Homebrew:

brew install postgresql

then follow the instructions for starting Postgres either at system startup or on-demand.

Clone the repository

Clone this repository using git. If you use the git command line, this looks like:

git clone git@github.com:influxdata/influxdb_iox.git

Then change into the directory containing the code:

cd influxdb_iox

The rest of these instructions assume you are in this directory.

Configure the server

InfluxDB IOx can be configured using either environment variables or a configuration file, making it suitable for deployment in containerized environments.

For a list of configuration options, run influxdb_iox --help, after installing IOx. For configuration options for specific subcommands, run influxdb_iox <subcommand> --help.

To use a configuration file, use a .env file in the working directory. See the provided example configuration file. To use the example configuration file, run:

cp docs/env.example .env

Compiling and Running

InfluxDB IOx is built using Cargo, Rust's package manager and build tool.

To compile for development, run:

cargo build

To compile for release and install the influxdb_iox binary in your path (so you can run influxdb_iox directly) do:

# from within the main `influxdb_iox` checkout
cargo install --path influxdb_iox

This creates a binary at target/debug/influxdb_iox.

Build a Docker image (optional)

Building the Docker image requires:

  • Docker 18.09+
  • BuildKit

To enable BuildKit by default, set { "features": { "buildkit": true } } in the Docker engine configuration, or run docker build withDOCKER_BUILDKIT=1

To build the Docker image:

DOCKER_BUILDKIT=1 docker build .

Local filesystem testing mode

InfluxDB IOx supports testing backed by the local filesystem.

Note

This mode should NOT be used for production systems: it will have poor performance and limited tuning knobs are available.

To run IOx in local testing mode, use:

./target/debug/influxdb_iox
# shorthand for
./target/debug/influxdb_iox run all-in-one

This will start an "all-in-one" IOx server with the following configuration:

  1. File backed catalog (sqlite), object store, and write ahead log (wal) stored under <HOMEDIR>/.influxdb_iox
  2. HTTP v2 api server on port 8080, querier gRPC server on port 8082 and several ports for other internal services.

You can also change the configuration in limited ways, such as choosing a different data directory:

./target/debug/influxdb_iox run all-in-one --data-dir=/tmp/iox_data

Compile and run

Rather than building and running the binary in target, you can also compile and run with one command:

cargo run -- run all-in-one

Release mode for performance testing

To compile for performance testing, build in release mode then use the binary in target/release:

cargo build --release
./target/release/influxdb_iox run all-in-one

You can also compile and run in release mode with one step:

cargo run --release -- run all-in-one

Running tests

You can run tests using:

cargo test --all

See [docs/testing.md] for more information

Write and read data

Data can be written to InfluxDB IOx by sending line protocol format to the /api/v2/write endpoint or using the CLI.

For example, assuming you are running in local mode, this command will send data in the test_fixtures/lineproto/metrics.lp file to the company_sensors namespace.

./target/debug/influxdb_iox -vv write company_sensors test_fixtures/lineproto/metrics.lp --host http://localhost:8080

Note that --host http://localhost:8080 is required as the /v2/api endpoint is hosted on port 8080 while the default is the querier gRPC port 8082.

To query the data stored in the company_sensors namespace:

./target/debug/influxdb_iox query company_sensors "SELECT * FROM cpu LIMIT 10"

Use the CLI

InfluxDB IOx is packaged as a binary with commands to start the IOx server, as well as a CLI interface for interacting with and configuring such servers.

The CLI itself is documented via built-in help which you can access by running influxdb_iox --help

Use InfluxDB 2.0 API compatibility

InfluxDB IOx allows seamless interoperability with InfluxDB 2.0.

Where InfluxDB 2.0 stores data in organizations and buckets, InfluxDB IOx stores data in namespaces. IOx maps organization and bucket pairs to namespaces with the two parts separated by an underscore (_): organization_bucket.

Here's an example using curl to send data into the company_sensors namespace using the InfluxDB 2.0 /api/v2/write API:

curl -v "http://127.0.0.1:8080/api/v2/write?org=company&bucket=sensors" --data-binary @test_fixtures/lineproto/metrics.lp

Run health checks

The HTTP API exposes a healthcheck endpoint at /health

$ curl http://127.0.0.1:8080/health
OK

The gRPC API implements the gRPC Health Checking Protocol. This can be tested with grpc-health-probe:

$ grpc_health_probe -addr 127.0.0.1:8082 -service influxdata.platform.storage.Storage
status: SERVING

Manually call the gRPC API

To manually invoke one of the gRPC APIs, use a gRPC CLI client such as grpcurl. Because the gRPC server library in IOx doesn't provide service reflection, you need to pass the IOx .proto files to your client when making requests. After you install grpcurl, you can use the ./scripts/grpcurl wrapper script to make requests that use the .proto files for you--for example:

Use the list command to list gRPC API services:

./scripts/grpcurl -plaintext 127.0.0.1:8082 list
google.longrunning.Operations
grpc.health.v1.Health
influxdata.iox.authz.v1.IoxAuthorizerService
influxdata.iox.catalog.v1.CatalogService
influxdata.iox.compactor.v1.CompactionService
influxdata.iox.delete.v1.DeleteService
influxdata.iox.ingester.v1.PartitionBufferService
influxdata.iox.ingester.v1.PersistService
influxdata.iox.ingester.v1.ReplicationService
influxdata.iox.ingester.v1.WriteInfoService
influxdata.iox.ingester.v1.WriteService
influxdata.iox.namespace.v1.NamespaceService
influxdata.iox.object_store.v1.ObjectStoreService
influxdata.iox.schema.v1.SchemaService
influxdata.platform.storage.IOxTesting
influxdata.platform.storage.Storage

Use the describe command to view methods for a service:

./scripts/grpcurl -plaintext 127.0.0.1:8082 describe influxdata.iox.namespace.v1.NamespaceService
service NamespaceService {
  ...
  rpc GetNamespaces ( .influxdata.iox.namespace.v1.GetNamespacesRequest ) returns ( .influxdata.iox.namespace.v1.GetNamespacesResponse );
  ...
}

Invoke a method:

./scripts/grpcurl -plaintext 127.0.0.1:8082 influxdata.iox.namespace.v1.NamespaceService.GetNamespaces
{
  "namespaces": [
    {
      "id": "1",
      "name": "company_sensors"
    }
  ]
}

Contributing

We welcome community contributions from anyone!

Read our Contributing Guide for instructions on how to run tests and how to make your first contribution.

Architecture and Technical Documentation

There are a variety of technical documents describing various parts of IOx in the docs directory.