53422dbd48
Merge branch 'main' into crepererum/router_mode_get_set_cfg
2021-11-04 14:06:44 +00:00
433b64ca3b
Merge pull request #2989 from influxdata/pd/data-generator-optimiziation
...
feat: new data generator
2021-11-04 14:06:06 +00:00
f19788ce94
Merge branch 'main' into pd/data-generator-optimiziation
2021-11-04 13:57:50 +00:00
f50f4cb719
fix: typos in data generator full example
2021-11-04 09:56:58 -04:00
bd1679a903
fix: revert to previous Cargo.lock and remove rand seeder
2021-11-04 09:56:58 -04:00
7d90314f2b
fix: data generator example
2021-11-04 09:56:58 -04:00
041957be48
fix: datagenerator PR feedback, implement field count
2021-11-04 09:56:58 -04:00
3b64debec7
fix: fix perf datagen toml for new data generator
2021-11-04 09:56:58 -04:00
7044b89453
feat: Refactor Data Generator
...
This is a huge commit that refactors the data generator. It removes many of the previous features that didn't quite make sense. The goal of this refactor was to make the data generator capable of representing complex tagsets that have values dependent on each other. It also significantly optimizes things to use far less memory and generate data much faster. Follow on work will update the generation of line protocol to support spaces in tags and their keys, double quotes in strings, and add more examples and documentation.
2021-11-04 09:56:58 -04:00
348b91edc4
feat: Add noop option to data generator
...
I needed this feature to be able to see how much memory and resources a given data spec toml would take to run.
2021-11-04 09:56:58 -04:00
32bf4be64c
chore: add benchmark for data generator tag set
2021-11-04 09:56:58 -04:00
db2f8a58fc
feat: Add tag_set and tag_pairs to measurements in Data Generator
...
This adds the ability to specify a tag_set and a collection of tag_pairs to measurements in the data generator. Tag pairs are evaluated once when the generator is created. This avoids re-running handlebars evaluations while generating data for tags that don't change value.
This commit also fixes an issue when printing the generation output to stdout while generating from more than one agent. Previously it would be garbled together.
Follow on PRs will update the tag generation code in measurement specs to be more consistent and optimzised for performance. I'll be removing the restriction of using different options while using tag_set and tag_pairs. I wanted to get this in first to show the structure of what is output.
2021-11-04 09:56:58 -04:00
d28749bd93
refactor: remove Entry from Db interfaces ( #2724 ) ( #3027 )
2021-11-04 13:28:47 +00:00
4840dc11cb
feat: add basic CRUD operations for router configs
2021-11-04 12:03:14 +01:00
07ba629e2b
feat: migrate write buffer producer to MutableBatch and pbdata ( #2743 ) ( #3021 )
...
* feat: migrate write buffer producer to MutableBatch and pbdata (#2743 )
* fix: Kafka message content type header
* chore: fix doc
2021-11-04 10:20:40 +00:00
ca5740e1c9
Merge pull request #3025 from influxdata/alamb/updates
...
chore: update dependencies
2021-11-04 08:32:54 +00:00
46213a5a28
chore: update dependencies
2021-11-03 17:00:20 -04:00
d2c9244d1b
Merge pull request #2990 from influxdata/cn/restore-doesnt-need-name
...
fix: Handle errors and rule updates correctly with UUIDs
2021-11-03 19:14:47 +00:00
0f62bb1686
fix: Distinguish between restoring twice and name collisions on restore
...
And return them in the correct situations.
2021-11-03 15:02:22 -04:00
6615001dc7
fix: Extract loading the database name out of the rules file into a function
2021-11-03 15:02:21 -04:00
52d9df5caf
fix: Thread a Not Found error on restore database through all APIs
2021-11-03 15:02:21 -04:00
ac75be6b78
fix: Don't require name for database restore CLI; it's now unused
2021-11-03 15:02:21 -04:00
3e44f17698
fix: Return a useful error on attempt to restore an active database
2021-11-03 15:02:18 -04:00
f6327b1940
fix: Send back the correct error when attempting to restore the same database twice
2021-11-03 15:01:56 -04:00
a0b4b6b8f9
test: Add assertions that ensure UUID doesn't change when DatabaseRules are updated
2021-11-03 15:01:27 -04:00
207d701bdb
fix: Have DatabaseState keep track of the UUID after reading it from object storage
2021-11-03 15:01:27 -04:00
b1504f17ed
fix: Detangle UUID from ProvidedDatabaseRules
2021-11-03 15:01:27 -04:00
f2cb79bfa0
test: Add more tests around database restore
2021-11-03 15:01:26 -04:00
62e7376394
fix: Restore database should only need UUID, not database name too
2021-11-03 15:01:26 -04:00
405ed6db1b
feat: add mutable_batch_tests ( #2724 ) ( #3011 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-03 18:15:35 +00:00
705fa3cb98
fix: dockerfiles should select database server mode by default ( #3010 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-03 18:06:01 +00:00
e83b8691db
refactor: remove chunk module from mutable_buffer ( #3015 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-03 17:56:52 +00:00
3db4abc17f
Merge pull request #3019 from influxdata/crepererum/warn_on_failed_wb_reads
...
refactor: warn when we receive broken data from write buffer
2021-11-03 17:45:05 +00:00
bbdfc47b02
Merge branch 'main' into crepererum/warn_on_failed_wb_reads
2021-11-03 17:34:14 +00:00
4bebb92591
Merge pull request #3016 from influxdata/crepererum/router_mode_boilerplate
...
feat: implement basic router server type
2021-11-03 15:45:28 +00:00
e3e99c45cf
fix: remove stale comment
2021-11-03 16:35:54 +01:00
1cf697f235
refactor: make server fixture creation less confusing
2021-11-03 16:34:51 +01:00
a91e572b43
feat: implement basic router server type
...
This only implements the deployment service and nothing else yet.
2021-11-03 16:34:51 +01:00
08fcd87337
feat: use protobuf encoding in write buffer ( #2724 ) ( #3018 )
2021-11-03 15:19:05 +00:00
10de63f9b6
refactor: warn when we receive broken data from write buffer
...
We need to understand why this is the case and where broken data gets
stuck, because at the moment it seems that some Kafka hiccups can
poison the write buffer but it seems restarting IOx recovers the
situation.
2021-11-03 16:03:42 +01:00
51c6348e54
refactor: extract codec module for write buffer ( #2724 ) ( #3017 )
2021-11-03 14:07:33 +00:00
e444fa4cb2
feat: pbdata encode ( #2724 ) ( #3009 )
2021-11-02 18:31:53 +00:00
39a7f3069f
feat: add alternative string encodings to pbdata ( #2724 ) ( #3006 )
...
* feat: add alternative string encodings to pbdata (#2724 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-02 18:14:34 +00:00
1902c4f8a9
chore: Update DataFusion ( #3012 )
...
* chore: Update DataFusion
* fix: restore Cargo.log
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-02 18:06:21 +00:00
873ce27b0c
feat: reduce lock metric cardinality ( #3013 )
2021-11-02 17:57:02 +00:00
c42a60beee
Merge pull request #3008 from influxdata/crepererum/issue2980f
...
refactor: decouple new router server configs from old management system
2021-11-02 10:19:54 +00:00
511621d3ca
refactor: use stripped-down shard config for router
...
This only implements table-based sharding. We can add value-based
sharding later.
2021-11-02 11:12:20 +01:00
0d0c0cb42b
refactor: move write buffer configs to new home
...
Write buffer configs will partially be shared by database and router
nodes, so lets move them into a shared home.
2021-11-02 10:17:01 +01:00
6644048eec
refactor: remove partitioning from router config
...
Routers are not live yet so this ain't a breaking change. Also w/ write
buffer format rework that is going to be finished soon we don't need
this.
2021-11-02 10:14:00 +01:00
5e86989990
fix: Only use timestamps in first/last when there is a corresponding … ( #2988 )
...
* fix: Only use timestamps in first/last when there is a corresponding value
* docs: Fix broken English in comment
* docs: clarify expectations in test
2021-11-01 18:54:43 +00:00
kodiakhq[bot]
Paul Dix
Raphael Taylor-Davies
Marco Neumann
Andrew Lamb
Carol (Nichols || Goulding)
Andrew Lamb