e3d9454f8a
chore: add two test schemas in data generator examples
...
One schema is a test for many small databases like a bunch of free tier users. The other is for a database with 2k measurments getting moderate ingest load (4.1 MB/sec).
2021-12-10 11:26:42 -05:00
add33e084f
Merge pull request #3350 from influxdata/ntran/test_compact_os_delete_propagation
...
test: propogate delete tests for compact OS chunks
2021-12-10 15:47:48 +00:00
2e3aa2e1ab
Merge branch 'main' into ntran/test_compact_os_delete_propagation
2021-12-10 15:11:11 +00:00
328294c4a6
Merge pull request #3346 from influxdata/pd/add-regex-to-db-writers
...
feat: add regex to bucket writers assignment in data generator
2021-12-10 10:11:26 +00:00
f477def27a
Merge branch 'main' into pd/add-regex-to-db-writers
2021-12-10 10:02:10 +00:00
306c2dcba7
Merge pull request #3349 from influxdata/alamb/update_state_machine
...
docs: Update database startup machine diagram
2021-12-10 10:01:44 +00:00
dc152abb55
Merge branch 'main' into alamb/update_state_machine
2021-12-10 09:51:40 +00:00
7d53fad427
Merge branch 'main' into ntran/test_compact_os_delete_propagation
2021-12-09 17:27:22 -05:00
0d6fefdd09
test: test concurrent deletes and OS compaction
2021-12-09 17:13:18 -05:00
7a56b4847a
Merge pull request #3345 from influxdata/ntran/compact_os_partition
...
feat: compact all object store chunks of a given partition
2021-12-09 20:50:17 +00:00
c0ba69f09e
chore: marge main to branch and resolve conflict
2021-12-09 15:40:33 -05:00
35370922f3
refactor: make a setup for 2 persisted chunks that can be used in for different places
2021-12-09 15:21:56 -05:00
191e743ce0
fix: Update server/src/database.rs
...
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-12-09 15:04:39 -05:00
3cda6b6c0f
refactor: Remove collect_query and replication ( #3348 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-09 19:58:19 +00:00
b2f7306d5a
docs: Update database startup machine diagram
2021-12-09 14:22:29 -05:00
099e2d4056
chore: Apply suggestions from code review
...
Co-authored-by: Marco Neumann <marco@crepererum.net>
2021-12-09 13:52:42 -05:00
8c88e1e52c
refactor: change orgbucket to database in data generator
2021-12-09 13:39:33 -05:00
01e86a031e
feat: add regex to bucket writers assignment in data generator
...
This adds the ability to specify a regex to match against database names when specifying what agents should write to which buckets in the data generator.
A default has also been added for ratio so that it doesn't need to be specified if only a single database writer is defined.
2021-12-09 13:39:33 -05:00
e46708354e
test: add management cli tests
2021-12-09 12:53:45 -05:00
bef1024fbb
Merge pull request #3347 from influxdata/cn/optional-kafka
...
feat: Optionally build Kafka
2021-12-09 16:31:45 +00:00
87d8f4a85f
fix: Return error instead of panicking if Kafka support is requested but not included
...
Also add some tests around this behavior.
2021-12-09 10:04:27 -05:00
bd919b52d6
fix: Add the kafka feature to the perf build
2021-12-09 08:49:34 -05:00
beede9bd4e
feat: Test and build releases with the Kafka feature enabled
2021-12-09 08:49:34 -05:00
365917c2aa
fix: Organize imports
2021-12-09 08:49:34 -05:00
471c3181bb
feat: Make a top-level influxdb_iox feature for kafka functionality
...
Thread the feature through router and server to the write buffer crate.
Move an end-to-end test that uses Kafka behind the feature flag.
2021-12-09 08:49:34 -05:00
8c7b3966de
fix: Organize imports
2021-12-09 08:49:34 -05:00
403dcae93c
feat: Put kafka write_buffer code behind a feature flag
...
Which is off by default. This makes rdkafka optional to minimize
build-time dependencies for users that don't plan on using a Kafka write
buffer.
2021-12-09 08:49:34 -05:00
46b43d3c30
Merge pull request #3342 from influxdata/crepererum/issue3336
...
fix: limit number of S3 connections
2021-12-09 09:53:50 +00:00
38ce54c1be
Merge branch 'main' into crepererum/issue3336
2021-12-09 09:44:55 +00:00
95f907efd8
Merge pull request #3340 from influxdata/crepererum/set_server_id_twice
...
fix: allow setting the same server ID twice
2021-12-09 08:21:08 +00:00
12c3c6b12c
Merge branch 'main' into crepererum/set_server_id_twice
2021-12-09 08:08:01 +00:00
4e86d0ef30
test: propogate delete tests for compact OS chunks
2021-12-08 17:50:51 -05:00
8f81120d54
Merge pull request #3319 from influxdata/cn/quick-release
...
feat: Create a build-time-optimized release build profile
2021-12-08 21:18:56 +00:00
60de5b7c5a
Merge branch 'main' into cn/quick-release
2021-12-08 21:10:10 +00:00
efbfbb1a0b
feat: compact all object store chunks of a given partition
2021-12-08 16:06:03 -05:00
876a9af35a
fix: limit number of S3 connections
...
Otherwise the whole thing blows up when starting a server that has many
DBs registerd, because we potentially create 1 connection per DB (e.g.
to read out the preserved catalog).
Fixes #3336 .
2021-12-08 19:06:02 +01:00
a8bc5b3b6d
fix: allow setting the same server ID twice
...
This is important for idempotence and simplifies clients and helper
scripts a lot.
2021-12-08 19:03:30 +01:00
b0209137e6
Merge pull request #3330 from influxdata/pd/data-generator-many-dbs
...
feat: add ability for data generator to write to many buckets
2021-12-08 17:26:21 +00:00
90a6b255a9
Merge branch 'main' into pd/data-generator-many-dbs
2021-12-08 17:17:58 +00:00
12c40b0f0f
Merge pull request #3329 from influxdata/ntran/compact_os_sql_tests
...
test: sql tests and checkpoint test for compact OS chunks
2021-12-08 17:17:41 +00:00
2c8d17bea8
refactor: change percent to ratio in data generator bucket writers
2021-12-08 12:09:04 -05:00
31aa41e240
feat: add ability for data generator to write to many buckets
...
This adds the ability for the data generator to write to many databases. A new command line argument, `bucket_list`, is added which should be a file name. The file should contain a list of databsaes, one per line, with the structure of <org>_<bucket>. This is a little odd given the data generator expects org and bucket separately, but I expect the file that we'll be using will be database names, which have this format.
The configuration can specify what percentage of the list should get written to by which agents at what sampling interval. This should allow configurations where databases get different levels of ingest and different types (as specified via different agent specs). The structure is a little wonky, but I think it'll get the job done. The next step is to run some perf tests to see how the data generator performs if writing to 10k databases.
2021-12-08 12:09:04 -05:00
713a4aca89
Merge branch 'main' into ntran/compact_os_sql_tests
2021-12-08 17:08:50 +00:00
35edee6b4f
fix: flaky test migrate_table_files_from_one_server_to_another ( #3339 )
2021-12-08 12:08:25 -05:00
6c6d3fe818
chore: Merge branch 'ntran/compact_os_sql_tests' of https://github.com/influxdata/influxdb_iox into ntran/compact_os_sql_tests
2021-12-08 12:07:19 -05:00
43e3e76de2
chore: Merge branch 'main' into ntran/compact_os_sql_tests
2021-12-08 12:07:02 -05:00
a374dfdc8f
Merge branch 'main' into ntran/compact_os_sql_tests
2021-12-08 16:51:41 +00:00
b93b7e9304
refactor: address review comments
2021-12-08 11:50:09 -05:00
428851b467
chore: Apply suggestions from code review
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-12-08 11:42:25 -05:00
218042784f
feat: add `system.queries `system table ( #3328 )
...
* feat: Add query log and system table
* docs: Add docstrings for test normalization
2021-12-08 16:26:24 +00:00
Paul Dix
kodiakhq[bot]
Nga Tran
Andrew Lamb
Andrew Lamb
Carol (Nichols || Goulding)
Marco Neumann