Commit Graph

159 Commits (f78195f7c7327dde3ad07a40a2926044572d38a3)

Author SHA1 Message Date
Nga Tran 1c962b05a4 fix: address review comments 2021-12-17 12:18:28 -05:00
Nga Tran 8be572812d
chore: Apply suggestions from code review
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-12-17 11:23:20 -05:00
Nga Tran d36db7eea2 chore: cleanup 2021-12-15 17:29:33 -05:00
Nga Tran 36b2cb021d fix: typos 2021-12-15 17:26:52 -05:00
Nga Tran e877bc7033 docs: all 3 catalogs 2021-12-15 17:06:12 -05:00
Nga Tran fa98b89524 docs: in-memory catalog 2021-12-14 18:33:29 -05:00
Nga Tran 9d2020b906 fix: typo 2021-12-14 14:14:37 -05:00
Nga Tran 2e9362839f docs: initial verstion to test text color 2021-12-14 14:13:43 -05:00
Nga Tran e6e21746fa fix: remove a dot 2021-12-14 01:08:54 -05:00
Nga Tran 8e085e32f5 docs: link docs 2021-12-14 01:06:06 -05:00
Nga Tran 36e87d7b2e fix: typos 2021-12-14 00:45:12 -05:00
Nga Tran 888a807936 docs: explain LifeCycle 2021-12-14 00:23:34 -05:00
Nga Tran 12cb0b0df2 docs: Figure for Data LifeCycle 2021-12-13 22:45:41 -05:00
Nga Tran f73dc9998b docs: chunk stages 2021-12-13 21:24:18 -05:00
Nga Tran 1ffd5446b1 docs: chunk stypes and chunk stages 2021-12-13 15:30:28 -05:00
Nga Tran 6e44daabdc docs: make figure 1 smaller 2021-12-13 14:22:02 -05:00
Nga Tran 5b920fb877 docs: test footnote 2021-12-13 13:02:26 -05:00
Nga Tran 7c466d2851 docs: first draft of IOx Data Organization and Life Cycle 2021-12-13 12:52:39 -05:00
Nga Tran 6fce12079a docs: update tech talks 2021-12-10 16:07:45 -05:00
Andrew Lamb c6a3765d76
feat: Add force flag to RebuildCatalog (#3292)
* feat: Add force flag to RebuildCatalog

* fix: small cleanups

* docs: Update comments and add WARNING
2021-12-08 15:36:07 +00:00
Andrew Lamb d3be1c7afa
docs: Document how to import parquet files into a local database (#3264)
* docs: Document how to import parquet files into a local database

* docs: Update docs/server_startup.md

Co-authored-by: Edd Robinson <me@edd.io>

Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-01 20:12:53 +00:00
Raphael Taylor-Davies f650962221
refactor: merge CI integration tests (#3049)
* refactor: merge CI integration tests

* chore: update docs
2021-11-10 16:48:16 +00:00
Raphael Taylor-Davies 60f0deaf1e
feat: remove flatbuffer entry (#3045) 2021-11-05 20:19:24 +00:00
Marco Neumann 6e29cec502 refactor: rename `query` mode to `database` 2021-10-27 09:31:46 +02:00
Marco Neumann 41bc373fe8 refactor: offer different modes via "run" CLI
Running a server is now using `influxdb_iox run MODE [args]`, e.g.
`influxdb_iox run query --server-id 1`. Another mode that will follow
soon is `router`.

The old syntax `influxdb_iox run [args]` (w/o the mode part) is still
supported but a deprecation note will be printed.
2021-10-27 09:31:46 +02:00
Marco Neumann 93f6519c34 fix: address review comments 2021-10-26 10:09:34 +02:00
Andrew Lamb 92968222fe
docs: Add tech talk links to docs (#2858)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-15 21:44:41 +00:00
Andrew Lamb b8ae7a8247
docs: Add observability context to documentation (#2814)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-13 13:22:31 +00:00
Raphael Taylor-Davies 826d50f14a
chore: update tracing and logging docs (#2657)
* chore: update tracing and logging docs

* chore: review feedback

* chore: add log-format to basic usage section
2021-09-30 08:46:23 +00:00
Raphael Taylor-Davies db22b4bf60
chore: expand protobuf documentation (#2659)
* chore: expand protobuf documentation

* chore: review feedback

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-29 21:22:03 +00:00
Marco Neumann bb94751d98 docs: add massif guide 2021-09-02 15:58:43 +02:00
Andrew Lamb 07b1722412
docs: Add local filesystem assumptions (#2410)
* docs: Add local filesystem assumptions

* docs: Update docs/local_filesystems.md

Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>

* docs: Clarify local filesystem is different than file based object store

* docs: clarify IOx may delete files

Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-28 10:34:34 +00:00
Marco Neumann 803d49648a feat: valgrind support 2021-08-23 09:30:36 +02:00
Jason Tianyi Wang 4f2503d92c
docs: Update system table schema in docs/sql.md (#2364)
* docs: Update system table schema in docs/sql.md

* docs: fix `Total row count by table` query

* docs: add a period to try and trigger CI

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2021-08-22 10:59:43 +00:00
Marco Neumann 42d5f9f3a1 feat: skip replay via CLI 2021-08-16 13:47:07 +02:00
Nga Tran 0a79afc8c0 docs: metrics 2021-08-10 15:30:17 -04:00
Nga Tran 7191053899 docs: document Metrics 2021-08-06 17:12:00 -04:00
Andrew Lamb 2afc7e56ae
docs: add writeup on how to use IOx server from source (#2220)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-06 21:04:44 +00:00
Andrew Lamb 3fd6430fb6
fix: rename `estimated_bytes` to `memory_bytes` and expose `object_store_bytes` in ChunkSummary and system.chunks (#2017)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 16:00:24 +00:00
Andrew Lamb 3bb32594ba
refactor: rename end-to-end.rs to end_to_end.rs (#2015)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 13:50:32 +00:00
Andrew Lamb 0442991e45
docs: Add links to latest tech talks (#2013)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 13:10:49 +00:00
Marko Mikulicic fba64a41f5
docs: improve profiling docs (#1869)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-02 10:47:13 +00:00
Carol (Nichols || Goulding) 44c2c0c763 docs: Recommend pulling latest docker image first 2021-06-21 09:41:36 -04:00
Carol (Nichols || Goulding) 135d885ca5 docs: Clarify the differences between the two ways to run the Kafka integration tests 2021-06-21 09:41:36 -04:00
Carol (Nichols || Goulding) b4644e6108 test: Start of Kafka Write Buffer integration tests 2021-06-21 09:41:35 -04:00
Marco Neumann cb09762067 docs: document server startup and error recovery 2021-06-21 10:26:52 +02:00
Andrew Lamb 4f81573f5f
docs: add query for computing time ranges to cookbook (#1620)
* docs: add query for computing time ranges to cookbook

* fix: add note about nanosecond conversion

* fix: Update docs/sql.md

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-04 14:59:05 +00:00
Marco Neumann 1cbe4031d3 feat: allow passing a session token to AWS stores 2021-05-27 11:53:12 +02:00
Andrew Lamb 41093b54d6
docs: tweak profiling.md (#1526) 2021-05-19 17:04:59 +00:00
Andrew Lamb 9b42c1a065
docs: Document how to use pprof tool (#1520) 2021-05-19 13:57:50 +00:00
DCjanus 9eb658eac5 feat: s3 compatibility storages support 2021-05-06 23:27:37 +08:00
Andrew Lamb 884baf7329
feat: add column_type and influxdb_column_type, remove row_count from system.columns (#1415)
* feat: add column_type and influxdb_column_type, remove row_count from system.columns

* fix: update tests

* fix: more test update

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: fmt

* fix: copy/paste type conversion to avoid cross dependency between data_types and internal_types

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-05-06 12:59:30 +00:00
Raphael Taylor-Davies 10f89a3e8d
refactor: split entry out into separate crate (#1428)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 11:36:23 +00:00
Marco Neumann 1269b5e3f0 docs: adjust catalog preservation design to implementation
For #1253.
2021-05-03 10:04:35 +02:00
Andrew Lamb c64f25e777
feat: Add `OBSERVER` mode to SQL repl along with SQL primer / cookbook (#1349)
* feat: add OBSERVER command to repl

* fix: fixup test

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-29 11:47:25 +00:00
Marco Neumann 74927527b6 docs: add design docs for catalog persistence
Closes #1219.
2021-04-27 15:18:41 +02:00
Carol (Nichols || Goulding) d8a92a8570 test: Don't run any integration tests unless TEST_INTEGRATION is set 2021-04-22 14:21:50 -04:00
Carol (Nichols || Goulding) 80995afb70 fix: Change WAL to Write Buffer in comments and documentation 2021-04-21 17:43:03 +00:00
Carol (Nichols || Goulding) ce645262c6 fix: Rename LOCAL env var to INFLUXDB_IOX_INTEGRATION_LOCAL 2021-04-21 14:59:08 +00:00
Carol (Nichols || Goulding) 731c586e46 docs: Make a sentence clearer 2021-04-21 14:59:08 +00:00
Carol (Nichols || Goulding) 5f85ad3ba1 docs: Bring markdown about flatbuffers in line with script's docs, mention LOCAL=1 2021-04-21 14:59:08 +00:00
Carol (Nichols || Goulding) 96afb25d1a test: Run influxd in docker by default for integration tests 2021-04-21 14:59:08 +00:00
Andrew Lamb 6d0467277b
docs: update multi core design doc (#1197)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-13 21:56:36 +00:00
Edd Robinson 0f53ba8d9a docs: improve flatc install 2021-04-12 20:16:18 +00:00
Carol (Nichols || Goulding) d84f05b2f5 chore: Revert "Merge pull request #1159 from influxdata/er/flatbuffers"
This reverts commit 0efc1d8174, reversing
changes made to a3e04fecc1.
2021-04-09 17:25:59 +00:00
Edd Robinson 10304bdcd8 docs: improve flatc install 2021-04-08 22:16:29 +01:00
Carol (Nichols || Goulding) ebb6bbd13c test: start of integration tests of influxdb2 client against influxdb 2.0 OSS 2021-04-07 14:09:39 -04:00
Carol (Nichols || Goulding) 39c07829bd chore: Remove outdated documentation about testing iox client crate
Looks like the type of integration testing referenced in the docs
deleted here was removed in 51981c9.
2021-04-07 14:09:39 -04:00
Carol (Nichols || Goulding) 4e25147062
docs: Codify our preference for no return type in test functions (#1116)
* docs: Copy edit existing style guide

I can't help myself.

* docs: Codify our preference for no return type in test functions
2021-04-05 18:12:51 +00:00
Marko Mikulicic cd94544e93 feat: Add prototxt script 2021-03-26 12:40:56 +00:00
Carol (Nichols || Goulding) c47a252c15 feat: Check in generated WAL flatbuffers code 2021-03-22 09:38:59 -04:00
Andrew Lamb f542d6216e fix: Update docs/README.md
Co-authored-by: Paul Dix <paul@influxdata.com>
2021-03-12 20:02:34 +00:00
Andrew Lamb a5a7e21840 fix: space 2021-03-12 20:02:34 +00:00
Andrew Lamb 2530be6111 docs: Add links to tech talks 2021-03-12 20:02:34 +00:00
Marko Mikulicic 334fb149b1
feat: Rename server run command to just run
Closes #976
2021-03-12 18:36:04 +01:00
Carol (Nichols || Goulding) 16df4c542b fix: Rename AZURE_STORAGE_MASTER_KEY to AZURE_STORAGE_ACCESS_KEY 2021-03-04 10:15:46 -05:00
Marko Mikulicic 153751e455 docs: Include INFLUXDB_IOX_OBJECT_STORE in env.example 2021-03-03 11:29:35 +00:00
Carol (Nichols || Goulding) ac88bb98f9 docs: Clarify more about TEST_INTEGRATION in regards to both crates that use it 2021-02-25 15:53:20 -05:00
Carol (Nichols || Goulding) c7ef18337c feat: Consolidate all bucket config into one option/env var
Fixes #869.
2021-02-25 15:53:20 -05:00
Marko Mikulicic 9e521a2ea1 feat: Plug GCS list_with_delimiter impl
And fix env.example. Now GCS can be used to persist snapshots.
2021-02-23 21:06:53 +00:00
Marko Mikulicic 9860def4b2 feat: Add S3 plumbing to iodx config 2021-02-23 14:55:55 +00:00
Andrew Lamb dd0810c18d
docs: Refer to the example env file and clarify comments (#851)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-22 19:42:30 +00:00
Carol (Nichols || Goulding) 54ecaaa7a6 docs: Restore the example .env file and add Azure variables 2021-02-18 16:53:06 -05:00
Andrew Lamb 16f2fbcb6d
docs: Improve style guide about using Snafu (#695) 2021-01-27 14:57:59 -05:00
Andrew Lamb 8c0ce0b2ce
docs: Add query README file and explain some rationale (#648)
* docs: Add query README file

* fix: Apply suggestions from code review

Co-authored-by: Edd Robinson <me@edd.io>

Co-authored-by: Edd Robinson <me@edd.io>
2021-01-12 18:26:32 -05:00
Andrew Lamb 9f0ff678f1
feat: Formalizes the config system for IOx, including tests (#608)
* feat: Create configuration system, port IOx to use it

* docs: Apply suggestions from code review

Co-authored-by: Paul Dix <paul@influxdata.com>

* fix: fix test for setting values

Co-authored-by: Paul Dix <paul@influxdata.com>
2020-12-31 07:02:31 -05:00
Andrew Lamb 47c6763f24
docs: Add documentation about data life cycle (#578) 2020-12-18 13:33:35 -05:00
Andrew Lamb 463a20202e
docs: Add reference to docs directory and a table of contents (#577) 2020-12-18 12:39:38 -05:00
Paul Dix 70ff4267ce
chore: Connect server crate to http routes and server command (#569)
* chore: Connect server crate to http routes and server command

This updates the http_routes and main server to use the Server crate. This is the first step in a larger effort to start hooking up the initial IOx API and get things running end to end with in-memory database, WAL buffer, and object storage.

For the time being, this disables the previous disk based WAL. Or rather, it uses the WriteBufferDb without it. That means that this IOx server has no persistence until later. Because of this, the restart in the end-to-end was removed.

Later PRs will add the WAL buffer and restart logic that loads from object store. We can opt to bring the local disk based WAL back later, but it will likely require some refactoring to work with how the WAL Buffer will operate.
2020-12-17 18:48:41 -05:00
Dom 41f5099691 refactor: compile out trace! level for release builds
Configures the IOx tracing to compile out trace!() level events in the release
binary. This effectively gives contributors three levels of output:

* Important to the user (info & friends)
* Not important for regular running, but needed to debug
* Only useful to devs in a specific part of the system, never seen by user

Documents this behaviour (and general usage guidelines) for contributors.
2020-12-14 12:06:53 +00:00
Dom 80da024212 docs(tracing): add IOx tracing usage doc
Describes the components involved in, and usage of the tracing system in IOx.
2020-12-14 12:06:53 +00:00
Brandon Sov 989d0ecad8 refactor: set valid format for default s3 bucket name example 2020-12-08 23:42:27 -08:00
Andrew Lamb a03eb6e8f7
docs: Add mentions of `unsafe` and platform dependent compilation in style guide (#472) 2020-11-23 11:14:33 -05:00
Carol (Nichols || Goulding) 572ff1947a
Merge pull request #428 from influxdata/back-to-the-future 2020-11-10 12:00:27 -05:00
Carol (Nichols || Goulding) c096682ab8 fix: Improve docs based on feedback from @rbetts 2020-11-10 11:53:44 -05:00
Carol (Nichols || Goulding) 05b60b8fd0 fix: Remove remaining mentions of Delorean 2020-11-10 11:47:42 -05:00
Carol (Nichols || Goulding) 920fd6df3f docs: Improve env example
- Remove `-` so you can use your editor's uncomment feature
- Add the env vars to customize the server addresses
- Document a bit more
2020-11-10 11:23:37 -05:00
Andrew Lamb 0532821ed9
chore: Make the project root neat and tidy (#416)
* chore: move example env file into docs

* chore: move dockerfiles out of root

* chore: move storectl-bench.sh into benches
2020-11-10 10:00:22 -05:00
alamb e3a3a4add3 docs: add note about using synchronous I/O and notes from @stuartcarnie on backpressure 2020-07-13 10:35:28 -04:00
alamb 2dca85069f fix: clarify some points 2020-07-13 10:35:28 -04:00