influxdb/influxdb_iox_client
Carol (Nichols || Goulding) 4a9e76b8b7
feat: Make parquet_file.partition_id optional in the catalog (#8339)
* feat: Make parquet_file.partition_id optional in the catalog

This will acquire a short lock on the table in postgres, per:
<https://stackoverflow.com/questions/52760971/will-making-column-nullable-lock-the-table-for-reads>

This allows us to persist data for new partitions and associate the
Parquet file catalog records with the partition records using only the
partition hash ID, rather than both that are used now.

* fix: Support transition partition ID in the catalog service

* fix: Use transition partition ID in import/export

This commit also removes support for the `--partition-id` flag of the
`influxdb_iox remote store get-table` command, which Andrew approved.

The `--partition-id` filter was getting the results of the catalog gRPC
service's query for Parquet files of a table and then keeping only the
files whose partition IDs matched. The gRPC query is no longer returning
the partition ID from the Parquet file table, and really, this command
should instead be using `GetParquetFilesByPartitionId` to only request
what's needed rather than filtering.

* feat: Support looking up Parquet files by either kind of Partition id

Regardless of which is actually stored on the Parquet file record.

That is, say there's a Partition in the catalog with:

Partition {
    id: 3,
    hash_id: abcdefg,
}

and a Parquet file that has:

ParquetFile {
    partition_hash_id: abcdefg,
}

calling `list_by_partition_not_to_delete(PartitionId(3))` should still
return this Parquet file because it is associated with the partition
that has ID 3.

This is important for the compactor, which is currently only dealing in
PartitionIds, and I'd like to keep it that way for now to avoid having
to change Even More in this PR.

* fix: Use and set new partition ID fields everywhere they want to be

---------

Co-authored-by: Dom <dom@itsallbroken.com>
2023-07-31 12:40:56 +00:00
..
src feat: Make parquet_file.partition_id optional in the catalog (#8339) 2023-07-31 12:40:56 +00:00
Cargo.toml chore(deps): Bump serde_json from 1.0.103 to 1.0.104 (#8342) 2023-07-27 09:12:35 +00:00
README.md fix: default the write cli comment to the http default port. (#7394) 2023-03-31 14:59:24 +00:00

README.md

InfluxDB IOx Client

This is the Rust client library for connecting to InfluxDB IOx.

We're attempting to support all apis as they are added and modified but this client is likely not 100% complete at any time.

Some apis are http (for instance the write) and some are gRPC. See the individual client modules for details.

Example: Using the Write Client

To write to IOx, create a connection and a write client, and then send line protocol. Please see the example on 'Client' struct that will work when running against all-in-one mode.