Commit Graph

187 Commits (dac0db21960c871c298924269d198a8b01849724)

Author SHA1 Message Date
Chunchun Ye 8575f3c022
chore(docs): add `influxdb_iox debug build-catalog` to debug.md (#8273)
* chore(docs): add `influxdb_iox debug build-catalog` to debug.md

chore(docs): wording

chore(docs): rephrase

* chore: remove personal shell prompt and make cmds cleaner

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-20 15:42:19 +00:00
Marco Neumann 1723aa46a6
docs: adjust querier cache docs (#8260)
`parquet_file` cache has a TTL as of #7975.
2023-07-18 15:13:15 +00:00
Marco Neumann 76a67a2c65 docs: add counter-example back
Was removed in #8033 but that made @alamb sad.
2023-06-23 12:53:04 +02:00
Marco Neumann 7322f238fb
docs: query processing (#8033)
* docs: query processing

Closes https://github.com/influxdata/idpe/issues/17770 .

* docs: apply recommendations

Co-authored-by: Stuart Carnie <stuart.carnie@gmail.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>

* docs: improve description of the flight protocol

* docs: link `LogicalPlan`

* docs: link `ExecutionPlan`

* docs: improve wording

* docs: improve query planning docs

---------

Co-authored-by: Stuart Carnie <stuart.carnie@gmail.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2023-06-23 09:13:14 +00:00
Andrew Lamb d3b8fa2c21
chore(docs): Update tracing.md with latest jaeger (#7878)
Update instructions to use latest jaeger version

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-26 17:24:05 +00:00
Marco Neumann d34d23c354
refactor: remove `processed_tombstone` table (#7840)
- the table is unused
- there are no foreign keys or triggers based on this table
- the design is generally not scalable (N*M entries) and tombstones
  should rather have
  a timestamp so we can check if a parquet file includes that
  information or not (or some other form of serialization mechanism)
- it's currently empty in prod (an never was filled w/ data in any
  cluster)
2023-05-22 15:56:23 +00:00
Marco Neumann 31b8813760
feat: hide `system.queries` table from prod by default (#7810)
Introduce a new header called `iox-debug` which when set enables certain
debug features. The first one will be the `system.queries` table which
is a process-local, namespace-scoped query log. In most prod setups this
is only useful for debugging and will confuse the user a lot because
when multiple queries are deployed then the K8s routing decides which
pod/process the users hits. This leads to an inconsistent view. However
the log is still useful for debugging.

This also wires the "debug header set" flag through the Flight ticket,
because JDBC proved (integration tests FTW!) that headers are only
passed to `GetFlightInfo` but not to `DoGet` and the ticket must encode
all the relevant information.

Closes #7119.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-22 12:29:24 +00:00
Andrew Lamb 1ff11d0856
refactor: Change catalog configuration so it is entirely dsn based / support end to end testing without postgres (#7736)
* refactor: Change catalog configuration so it is entirely dsn based / support end to end testing without postgres

Restores code from https://github.com/influxdata/influxdb_iox/pull/7708

Revert "revert: PR #7708"

This reverts commit c9cfe05f8d.

* fix: merge

* fix: Update new test
2023-05-17 13:36:25 +00:00
Carol (Nichols || Goulding) e60f703e95
fix: Rename router2 to router
Including an alias and a test for continuing to support `influxdb_iox
run router2`.
2023-05-09 22:01:39 -04:00
Carol (Nichols || Goulding) 3d5df5574a
fix: Remove vestiges of shards 2023-05-08 20:24:36 -04:00
Carol (Nichols || Goulding) 56916cf942
fix: Rename ingester2 to ingester 2023-05-08 12:03:05 -04:00
Carol (Nichols || Goulding) b0959667d5
fix: Move topic and query pool within iox catalog (#7734)
Still insert them into the database and associate them with namespaces,
but don't ever query them back out.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-04 13:45:56 +00:00
Carol (Nichols || Goulding) 621caab2e9
fix: Remove unused parquet_max_sequence_number metadata 2023-05-03 10:57:27 -04:00
Dom Dwyer c9cfe05f8d
revert: PR #7708
This reverts commit 61abb58933.
2023-05-02 13:51:30 +02:00
Andrew Lamb 61abb58933
refactor: Change catalog configuration so it is entirely dsn based / support end to end testing without postgres (#7708)
* refactor: Change catalog configuration so it is entirely dsn based

* docs: Add documentation

* chore: update docs

* chore: review feedback

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-02 10:48:33 +00:00
kodiakhq[bot] da96239605
Merge branch 'main' into cn/delete-tombstones 2023-04-17 13:59:49 +00:00
Carol (Nichols || Goulding) 5e6dbec909
fix: Remove tombstones as they aren't functional currently 2023-04-14 13:36:08 -04:00
kodiakhq[bot] bc3b69ef3f
Merge branch 'main' into cn/remove-obsolete-docs-infra 2023-04-14 17:14:45 +00:00
Carol (Nichols || Goulding) 8d3e285251
fix: Remove outdated documentation that discusses Kafka 2023-04-14 13:08:21 -04:00
Chunchun Ye fb6b3f66da
feat(flightsql): support `database`, `bucket`, and `bucket-name` as grpc header names (#7511)
* feat: support `database`, `bucket`, and `bucket-name` as grpc header names

* chore: lint

* chore: update doc to accept `database`, `bucket`, and `bucket-name` as parameter names

* chore: update doc to only show `database` as the parameter name

* refactor: consolidate header names into a const vec and update comments on database

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-12 16:58:44 +00:00
Carol (Nichols || Goulding) 2e656766f8
fix: Remove docs about testing object_store crate
This crate has its own repo now.
2023-04-12 11:53:29 -04:00
Carol (Nichols || Goulding) f561d599d2
fix: Remove uses of KAFKA_CONNECT, redpanda, and some talk of Kafka 2023-04-12 11:53:23 -04:00
Marco Neumann a3b662f30b
docs: explain new physical plan construction (#6970)
* docs: explain new physical plan construction

There is no code yet, but this is the rough plan.

Ref #6098.

* docs: clarify wording

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

* docs: typos

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

* docs: add links

* docs: extend reasoning

---------

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2023-02-14 11:13:22 +00:00
Andrew Lamb 7853a19953
feat: JDBC integration tests with FlightSQL (#6693)
* feat: basic JDBC integration test

* fix: do not run test without env set

* docs: add maven link

* refactor: clean up java with switch statement
2023-01-25 22:21:18 +00:00
Andrew Lamb 509c80bc55
docs: document how the garbage collector works (#6682)
* docs: document how the garbage collector works

* fix: Updates

* docs: Update docs/garbage_collector.md

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-25 10:54:43 +00:00
Andrew Lamb e5322b24b9
feat: Add --token CLI argument, improve update docs about writing (#6356)
* feat: Add --token CLI argument, improve update docs about writing

* fix: support environment tokens too

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-12 13:43:15 +00:00
Nga Tran 775f4b5eea
docs: Add 2 new config params that can be adjusted to avoid OOMs (#6307)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-01 22:36:48 +00:00
Nga Tran 55508ea794
docs: data retention (#6245)
* docs: data retention

* docs: address review comments

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-29 15:34:23 +00:00
Carol (Nichols || Goulding) 0d8e756e1b
fix: Update docs on the ingester-querier protocol.
I believe this fixes #6049.
2022-11-14 16:46:03 -05:00
Carol (Nichols || Goulding) bdff4e8848
fix: Consistently use 'namespace' instead of 'database' in comments and other internal text 2022-11-11 15:46:04 -05:00
Nga Tran ea2559a640
chore: update SQL syntax after adding the new column to_delete into table partition (#6123) 2022-11-11 16:21:45 +00:00
Nga Tran fba4408d05
refactor: move `influxdb_iox debug namespace` command to `influxdb_iox namespace` (#6031)
* refactor: move  command to

* docs: update the doc accordingly

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-02 10:57:58 +00:00
Andrew Lamb 643fd58e02
docs: Document new CLI commands (#6013)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-01 16:44:14 +00:00
Carol (Nichols || Goulding) 3143843078
docs: Explain the new debug skipped-compactions CLI 2022-10-21 16:34:54 -04:00
Carol (Nichols || Goulding) f07e59e778
fix: Clean up typos, markdown/code formatting, inconsistencies 2022-10-21 16:32:59 -04:00
Andrew Lamb 1df7a0d4fb
refactor: remove outdated observer sql repl mode (#5918)
* refactor: remove Observer mode from repl

* chore: remove outdated SQL docs

* fix: more update of sql docs
2022-10-19 18:39:05 +00:00
Nga Tran 755fb029f9
docs: replace database with namespace (#5914)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-19 16:17:31 +00:00
Nga Tran 6672404711
docs: part 2 of compactor documents with best practice and guidelines (#5880)
* docs: part 2 of compactor

* fix: typos

* chore: Apply suggestions from code review

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

* docs: address review comments

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-18 13:07:48 +00:00
Nga Tran 6ebad08036
docs: compactor document and how to resolve OOMs (#5868)
* docs: compactor document

* chore: Apply suggestions from code review

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

* docs: fix indent and add an explanation why OOMs won't go away by themselves

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-17 14:25:41 +00:00
Andrew Lamb d8a318eb57
docs: Tweak local run guide (#5787)
Update the instructions on how to run IOx locally

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-07 20:34:00 +00:00
YIXIAO SHI 52ae60bf2e
chore: fix comment typo (#5551)
Co-authored-by: Dom <dom@itsallbroken.com>
2022-09-07 08:49:29 +00:00
Carol (Nichols || Goulding) 74c9529062
fix: Rename KafkaPartition to ShardIndex 2022-08-29 14:07:18 -04:00
Marco Neumann 9851a5e357
docs: extend profiling docs with stuff I've learned (#5313)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-04 16:34:07 +00:00
Stuart Carnie 964062b40c
feat: Add information about profiling using Instruments on macOS (#5275) 2022-08-03 08:45:33 +00:00
Marco Neumann 7d16ac7de0
docs: extend profiling guide (#5205)
* docs: extend profiling guide

More tools.

* chore: fix docs lint for `localhost` links

* docs: do not duplicate tracing docs

* refactor: clean up `lint_docs` and strip anchors from relative links
2022-07-26 09:40:42 +00:00
Marco Neumann bd9107a226
docs: extend profiling docs (#5102) 2022-07-13 08:18:34 +00:00
Andrew Lamb ec3f6a8597
docs: Add guide for running / profiling IOx locally (#5057)
* docs: Add guide for running / profiling IOx locally

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* docs: Add note about why it is an "underground" guide

* docs: add note about running locally, and move catalog prodding after

* docs: add INFLUXDB_IOX_MAX_HTTP_REQUEST_SIZE and clarify settings

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2022-07-11 13:20:58 +00:00
Sam Arnold 9438570ba1
test: document how to run tests (#4982)
* test: document how to run tests

Also fix a few issues for local runs.

* docs: add back one-liner for running end to end tests

* docs: add comment for clap_blocks test

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

* docs: add comment in influxdb_iox/tests/end_to_end_cases/cli.rs

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-06-30 14:01:35 +00:00
Ryan Russell 77a4246432
docs: Readability improvements (#4946)
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-27 21:46:18 +00:00
Andrew Lamb 47def89670
docs: Update tracing.md for NG (#4916)
tracing instructions referred to OG -- update for NG

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-23 09:24:38 +00:00