This makes quite a few major changes to our CLI and how users interact
with it:
1. All commands are now in the form <verb> <noun> this was to make the
commands consistent. We had last-cache as a noun, but serve as a
verb in the top level. Given that we could only create or delete
All noun based commands have been move under a create and delete
command
2. --host short form is now -H not -h which is reassigned to -h/--help
for shorter help text and is in line with what users would expect
for a CLI
3. Only the needed items from clap_blocks have been moved into
`influxdb3_clap_blocks` and any IOx specific references were changed
to InfluxDB 3 specific ones
4. References to InfluxDB 3.0 OSS have been changed to InfluxDB 3 Core
in our CLI tools
5. --dbname has been changed to --database to be consistent with --table
in many commands. The short -d flag still remains. In the create/
delete command for the database however the name of the database is
a positional arg
e.g. `influxbd3 create database foo` rather than
`influxdb3 database create --dbname foo`
6. --table has been removed from the delete/create command for tables
and is now a positional arg much like database
7. clap_blocks was removed as dependency to avoid having IOx specific
env vars
8. --cache-name is now an optional positional arg for last_cache and meta_cache
9. last-cache/meta-cache commands are now last_cache and meta_cache respectively
Unfortunately we have quite a few options to run the software and I
couldn't cut down on them, but at least with this commands and options
will be more discoverable and we have full control over our CLI options
now.
Closes#25646
This removes references to IOx in our docs and comments as when we
started making Monolith last year it was a fork of influxdb_iox.
This was something that we changed in some places, but not all as
time went on. To avoid confusion whenever we more broadly release
this software I've removed references to IOx in our code and docs.
Note however that we still use a lot of IOx code and our system
tables also return iox as part of the name. This is a separate issue
namely #25227. Some things like env vars will also be cleaned up
in #25646Closes#25662
* feat: add seconds to generated load files
This adds seconds to the time string portion of the generated files from
load generation runs. Previously, if the generator was run more than once
in the same minute, latter runs would fail because the results files
already exist.
* refactor: make query/write/system graphs optional based on run
Made the analysis tool have optional graphs based on what was actually
generated.
* refactor: change the time string format in generated load files
* refactor: make end common to load generatino tool
Made the --end argument common to both the query and write load generation
runners.
A panic message was also added in the table buffer where unwraps were
causing panics
* refactor: load gen print statements for consistency
* refactor: query/write load gen arg interface
Refactored the argument interface for the query and write load gen
commands to make them easier to unify in a new `full` command.
In summary:
- remove the query sampling interval
- make short-form of --querier-count 'q' instead of 'Q'
- remove the short-form for --query-format
- remove --spec-path in favour of --querier-spec and --writer-spec
for specifying spec path of the `query` and `write` loads, resp.
* feat: produce error on 0s sampling interval
* refactor: split out query/write command configs
Refactored the query and write command clap configurations to make
them composable for the full command
* refactor: expose query and write runner for composability
Refactored the query and write runners so that they can be
composed into the full runner.
* feat: add the full load generator sub-command
Implement a new sub-command for the load generator: full
This runs both the query and write loads simultaneously, and exposes
the unified CLI of the two commands, respectively.
* chore: cargo update to fix audit
* feat: report system stats in load generator
Added the mechanism to report system stats during load generation. The
following stats are saved in a CSV file:
- cpu_usage
- disk_written_bytes
- disk_read_bytes
- memory
- virtual_memory
This only works when running the load generator against a local instance
of influxdb3, i.e., one that is running on your machine.
Generating system stats is done by passing the --system-stats flag to the
load generator.
* feat: add new clap args for results gen
Added the results_dir and configuration_name args
to the common load generator config which will be
used in generating the results directory structure.
* feat: load gen results directory structure
Write and query load generation runners will now setup files in a
results directory, using a specific structure. Users of the load tool
can specify a `results_dir` to save these results, or the tool will
pick a `results` folder in the current directory, by default.
Results will be saved in files using the following path convention:
results/<s>/<c>/<write|query|system>_<time>.csv
- <s>: spec name
- <c>: configuration name, specified by user with the `config-name`
arg, or by default, will use the revision SHA of the running server
- <write|query|system>: which kind of results file
- <time>: a timestamp in the form 'YYYY-MM-DD-HH-MM'
The setup code was unified for both write and query commands, in
preparation for the creation of a system stats file, as well as for
the capability to run both query and write at the same time, however,
those remain unimplemented as of this commit.
* feat: /ping API support on influxdb3_client::Client
Implement the query load generator. The design follows that of the existing write load generator.
A QuerySpec is defined that will be used by the query command to generate a set of queriers to perform queries against a running server in parallel.
* feat: initial load generator implementation
This adds a load generator as a new crate. Initially it only generates write load, but the scaffolding is there to add a query load generator to complement the write load tool.
This could have been added as a subcommand to the influxdb3 program, but I thought it best to have it separate for now.
It's fairly light on tests and error handling given its an internal tooling CLI. I've added only something very basic to test the line protocol generation and run the actual write command by hand.
I included pretty detailed instructions and some runnable examples.
* refactor: address PR feedback
Michael Gattozzi
Jackson Newhouse
Trevor Hilton
Paul Dix