Commit Graph

386 Commits (252ced7adff817f88eee61ba135464dff65ed876)

Author SHA1 Message Date
Carol (Nichols || Goulding) 252ced7adf
feat: Add row count to the parquet_file record in the catalog (#3847)
Fixes #3842.
2022-02-24 15:20:50 +00:00
Carol (Nichols || Goulding) 71f62eee68
fix: Remove min_time and max_time from IngesterQueryRequest (#3839)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 15:46:31 +00:00
Carol (Nichols || Goulding) 1b9212540b
feat: Send IngesterQueryResponse data back as response of doGet Flight request (#3772)
* fix: Adjust fields of IngesterQueryResponse

* feat: Adjust IngestHandler query method to call prepare_data_to_querier

* feat: Send ingest query result data back through Flight doGet

* feat: Send delete predicates and max sequencer number in metadata

* fix: greater_than_sequence_number should be of type SequenceNumber

* fix: Remove DeletePredicates from IngesterQueryResponse

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-18 17:42:49 +00:00
Raphael Taylor-Davies 83cba3d2fb
feat: template static router config (#3781)
* feat: template static router config

* chore: lint and improved failure output

* chore: clarify docs

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-18 10:53:10 +00:00
Carol (Nichols || Goulding) 90da060156
feat: Add namespace and sequencer id fields to IngesterQueryRequest protobuf (#3766)
Fixes #3753.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-16 19:21:15 +00:00
Raphael Taylor-Davies 26fd5273f0
feat: static database configuration (#2436) (#3732)
* feat: static database configuration (#2436)

* chore: fmt

* feat: don't base64 encode UUIDs in ServerConfigFile

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 19:42:49 +00:00
Carol (Nichols || Goulding) 85aa019f50
feat: Turn protobuf predicates into predicate::Predicate (#3707)
* feat: Turn protobuf predicates into predicate::Predicate

* fix: Take buf lint's suggestions

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 17:56:56 +00:00
Raphael Taylor-Davies 866777ecd2
feat: static router configuration (#2436) (#3725)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-11 14:09:37 +00:00
Raphael Taylor-Davies 4e3f66ed07
feat: CLI and gRPC APIs for shutting down and restarting databases (#3720)
* feat: allow catalog wipe and rebuild whilst shutdown

* feat: CLI and gRPC APIs for shutting down and restarting databases

* feat: add ability to skip replay on restart

* fix: test_wipe_persisted_catalog_error_db_exists

* fix: wipe_preserved_catalog
2022-02-11 10:14:43 +00:00
Raphael Taylor-Davies b1190262b7
feat: restartable `Database` (#3368) (#3711)
* feat: restartable `Database` (#3368)

* chore: fmt

* fix: wipe_preserved_catalog

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 18:32:05 +00:00
Carol (Nichols || Goulding) 73828323ac
feat: Ingester Flight gRPC API (#3623)
* feat: Add a way to run ingester with an in-memory catalog from the CLI

If you set the --catalog-dsn string to "mem", rather than using that as
a Postgres connection URL, create an in-memory catalog.

Planning on using this in tests, so not documenting.

* fix: Set default topic to the same value as SHARED_KAFKA_TOPIC

Namely, both should use an underscore. I don't think there's a way to
directly share these values between a constant and an annotation.

* feat: Add a flight API (handshake only) to ingester

* fix: Create partitions if using file-based write buffer

* fix: Change the server fixture to handle ingester server type

For now, the ingester doesn't implement the deployment API. Not sure if
it should or not.

* feat: Start implementing ingester do_get, namely decoding the query

Skip serialization of the predicate for the moment.

* refactor: Rename ingest protos to ingester to match crate name

* refactor: Rename QueryResults to QueryData

* feat: Move ingester flight client to new querier crate

* fix: Off by one error, different starting indexes in sequencers

* fix: Create new CLI argument to pick the catalog type

* fix: Create a CLI option to set the number of topics to auto-create in the write buffer

* fix: Check the arrow flight service's health to tell that the ingester gRPC is up

* fix: Set postgres as the default catalog type

* fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
Carol (Nichols || Goulding) 093d5acfd4
fix: Unify temporary multiple definitions of IoxMetadata 2022-01-31 10:48:29 -05:00
Carol (Nichols || Goulding) dd9620da0c
feat: Create a new proto definition for the new design's IoxMetadata 2022-01-31 10:36:32 -05:00
Raphael Taylor-Davies 4101d16f71
chore: feature flag consistency (#3574)
* chore: feature flag consistency

* chore: add aarch64-apple-darwin to hakari

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-28 16:38:59 +00:00
Dom 32d7c4cbfe
refactor: remove InfluxColumnType::IOx (#3565)
* refactor: remove InfluxColumnType::IOx

Remove unused column variant - see #3554 for context.

* refactor: reserve SEMANTIC_TYPE_IOX name in proto

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 21:15:36 +00:00
Andrew Lamb 9c19cd6cc4
fix: clamp start/end of TimestampRange to min/max valid timestamp values (#3487)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-20 16:08:00 +00:00
Andrew Lamb dd23056efd
chore: update datafusion, arrow, prost, tonic, pbjson, etc (#3455)
* chore: update datafusion, arrow, prost, tonic, etc

* fix: update pprof as well

* chore: update hakari

* fix: update pbjson

* chore: update heappy

* fix: hakari

* fix: workaround https://github.com/influxdata/influxdb_iox/issues/3458

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-13 17:07:15 +00:00
Andrew Lamb a93ae739a9
feat: Add table_name to Partition API (#3421) 2022-01-06 16:38:39 +00:00
Andrew Lamb 758b65dd29
feat: Add database initialization state and errors to CLI and remove list_databases_detailed gRPC (#3377)
* feat: Add database initialization state and errors to CLI:

* fix: do not use optional in protobuf

* fix: clippy

* fix: correct check I broke appeasing clippy
2021-12-15 12:18:41 +00:00
Edd Robinson 7fe6897c59
refactor: add support for handling `TagValuesGroupedByMeasurementAndTagKeyRequest` (#3373)
* chore: update Storage service

* refactor: handle rpc

* refactor: log request fields

* refactor: add new RPC to client

* test: basic test of RPC

* refactor: fmt

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-14 18:42:12 +00:00
Nga Tran efbfbb1a0b feat: compact all object store chunks of a given partition 2021-12-08 16:06:03 -05:00
Andrew Lamb c6a3765d76
feat: Add force flag to RebuildCatalog (#3292)
* feat: Add force flag to RebuildCatalog

* fix: small cleanups

* docs: Update comments and add WARNING
2021-12-08 15:36:07 +00:00
Marco Neumann c17a6c10c1 feat: add `GetRouter` gRPC method 2021-12-08 13:45:55 +01:00
Raphael Taylor-Davies b0e01edb86
fix: include pbjson code for storage package (#3334) 2021-12-08 11:58:09 +00:00
Andrew Lamb 1156a81567
feat: Add pbjson serialization for storage rpc (#3324)
* feat: Add pbjson serialization for storage rpc

* chore: update pbjson-build to 0.1.1

Co-authored-by: Raphael Taylor-Davies <r.taylordavies@googlemail.com>
2021-12-07 16:39:16 +00:00
Marco Neumann d9e8bafeb3 fix: improve state machine for write buffer creation retries 2021-12-07 10:29:08 +01:00
kodiakhq[bot] 75cd1d24f4
Merge branch 'main' into ntran/grpc_compact_os_chunks 2021-12-07 03:26:06 +00:00
Nga Tran 3834435620 chore: Merge branch 'ntran/grpc_compact_os_chunks' of https://github.com/influxdata/influxdb_iox into ntran/grpc_compact_os_chunks 2021-12-06 12:06:22 -05:00
Nga Tran d8ed8a3bf7 refactor: address review comments 2021-12-06 12:06:07 -05:00
Carol (Nichols || Goulding) 30c4da7ca7
fix: Be consistent with regex version range specification 2021-12-06 09:37:15 -05:00
Nga Tran 65660dace5
Merge branch 'main' into ntran/grpc_compact_os_chunks 2021-12-03 18:16:42 -05:00
Nga Tran 5f3706e0ee feat: grpc call for copact object store chunks 2021-12-03 18:01:28 -05:00
Andrew Lamb 7dea2ef362
feat: Add force flag to ClaimDatabase (#3284)
* feat: Add force flag to ClaimDatabase

* fix: Update name of test

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* docs: Update influxdb_iox/tests/end_to_end_cases/management_cli.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2021-12-03 21:11:23 +00:00
Nga Tran fa5b046b45 chore: cleanup 2021-12-02 14:59:47 -05:00
Nga Tran ffc970a60f feat: add to-be-create-chunk-id for compacting OS chunks in CatalogChunk 2021-12-02 14:49:06 -05:00
kodiakhq[bot] e8edab0b43
Merge branch 'main' into ntran/compact_os_preserved_catalog 2021-12-01 18:26:33 +00:00
Raphael Taylor-Davies 94883cd33b
feat: decode error details payloads (#3251) (#3271) 2021-12-01 15:30:59 +00:00
Raphael Taylor-Davies 9e45ce4121
feat: cleanup gRPC server errors (#3251) (#3265)
* feat: add precondition violation enumeration (#3251)

* chore: review feedback
2021-12-01 13:33:17 +00:00
Raphael Taylor-Davies f1936f0b08
refactor: add resource type enumeration (#3251) (#3260)
* refactor: add resource type enumeration (#3251)

* chore: more error details
2021-12-01 10:26:12 +00:00
Nga Tran 6bb7a796e7 chore: Merge branch 'main' into ntran/compact_os_preserved_catalog 2021-11-30 16:55:38 -05:00
Nga Tran e8034cdc53 refactor: handle delete predicates for compacting os 2021-11-30 16:45:04 -05:00
Raphael Taylor-Davies 1e515a1dec
feat: load RUB from object store (#3224) (#3250) 2021-11-30 14:39:52 +00:00
Raphael Taylor-Davies 88acf3788e
feat: rebuild catalog (#3225) (#3253)
* feat: rebuild catalog (#3225)

* chore: fix doc

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-30 12:17:27 +00:00
Raphael Taylor-Davies f165faf5bc
feat: support forcing persistence of partition irrespective of arrival time (#3249)
* feat: force persistence of partition irrespective of arrival time

* feat: expose to CLI

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-29 22:07:35 +00:00
kodiakhq[bot] 23dffefcf8
Merge branch 'main' into crepererum/remove_routing_from_database_mode_7 2021-11-29 11:38:12 +00:00
Raphael Taylor-Davies 197634ed50
feat: reload chunk back into read buffer (#3209) (#3216)
* feat: reload chunk back into read buffer (#3209)

* chore: fix logical conflict

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-29 11:34:55 +00:00
Marco Neumann 4e043ecb55 refactor: remove old routing / sharding config
This is superseded by the new router subsystem.
2021-11-29 12:33:48 +01:00
Raphael Taylor-Davies 0100bfef79
feat: error if persist_age_threshold less than late arrival (#2005) (#3228)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-29 10:21:12 +00:00
Marko Mikulicic 8fe2c64f5a
fix: Implement a dummy Storage.Offsets method
Closes https://github.com/influxdata/conductor/issues/766
2021-11-26 16:39:07 +01:00
Marco Neumann 7f2e4f4342 refactor: remove write buffer direction
The direction was required when a database could read or write from/to a
write buffer. Now it is clear from the usage context of a write buffer
context which of the two applications is meant (databases read, routers
write) so the direction flag is no longer required.
2021-11-26 12:38:40 +01:00