Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
influxdb/iox_data_generator/schemas/storage_cardinality_example...

82 lines
2.7 KiB
TOML
Raw Normal View History

feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
name = "storage_cardinality_example"
docs: Improving Readability r2 (#4781) * docs: Improve TOML readability Signed-off-by: Ryan Russell <git@ryanrussell.org> * docs: Fix `jemalloc` link Signed-off-by: Ryan Russell <git@ryanrussell.org> * docs: Readability in `point_generation.rs` Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-06-04 16:46:57 +00:00
# Values are automatically generated before the agents are initialized. They generate tag key/value pairs
feat: Add tag_set and tag_pairs to measurements in Data Generator This adds the ability to specify a tag_set and a collection of tag_pairs to measurements in the data generator. Tag pairs are evaluated once when the generator is created. This avoids re-running handlebars evaluations while generating data for tags that don't change value. This commit also fixes an issue when printing the generation output to stdout while generating from more than one agent. Previously it would be garbled together. Follow on PRs will update the tag generation code in measurement specs to be more consistent and optimzised for performance. I'll be removing the restriction of using different options while using tag_set and tag_pairs. I wanted to get this in first to show the structure of what is output.
2021-09-17 20:00:06 +00:00
# with the name of the value as the tag key and the evaluated template as the value. These pairs
# are Arc wrapped so they can be shared across tagsets and used in the agents as pre-generated data.
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
[[values]]
feat: Add tag_set and tag_pairs to measurements in Data Generator This adds the ability to specify a tag_set and a collection of tag_pairs to measurements in the data generator. Tag pairs are evaluated once when the generator is created. This avoids re-running handlebars evaluations while generating data for tags that don't change value. This commit also fixes an issue when printing the generation output to stdout while generating from more than one agent. Previously it would be garbled together. Follow on PRs will update the tag generation code in measurement specs to be more consistent and optimzised for performance. I'll be removing the restriction of using different options while using tag_set and tag_pairs. I wanted to get this in first to show the structure of what is output.
2021-09-17 20:00:06 +00:00
# the name must not have a . in it, which is used to access children later. Otherwise it's open.
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
name = "role"
feat: Add tag_set and tag_pairs to measurements in Data Generator This adds the ability to specify a tag_set and a collection of tag_pairs to measurements in the data generator. Tag pairs are evaluated once when the generator is created. This avoids re-running handlebars evaluations while generating data for tags that don't change value. This commit also fixes an issue when printing the generation output to stdout while generating from more than one agent. Previously it would be garbled together. Follow on PRs will update the tag generation code in measurement specs to be more consistent and optimzised for performance. I'll be removing the restriction of using different options while using tag_set and tag_pairs. I wanted to get this in first to show the structure of what is output.
2021-09-17 20:00:06 +00:00
# the template can use a number of helpers to get an id, a random string and the name, see below for examples
chore: pr cleanup on data generator
2021-09-13 21:45:01 +00:00
template = "storage"
feat: Add tag_set and tag_pairs to measurements in Data Generator This adds the ability to specify a tag_set and a collection of tag_pairs to measurements in the data generator. Tag pairs are evaluated once when the generator is created. This avoids re-running handlebars evaluations while generating data for tags that don't change value. This commit also fixes an issue when printing the generation output to stdout while generating from more than one agent. Previously it would be garbled together. Follow on PRs will update the tag generation code in measurement specs to be more consistent and optimzised for performance. I'll be removing the restriction of using different options while using tag_set and tag_pairs. I wanted to get this in first to show the structure of what is output.
2021-09-17 20:00:06 +00:00
# this number of tag pairs will be generated. If this is > 1, the id or a random character string should be
# used in the template to ensure that the tag key/value pairs are unique.
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
cardinality = 1
[[values]]
name = "url"
chore: pr cleanup on data generator
2021-09-13 21:45:01 +00:00
template = "http://127.0.0.1:6060/metrics/usage"
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
cardinality = 1
[[values]]
refactor: optimizations to make tag set generation significantly faster
2021-09-09 22:55:04 +00:00
name = "org_id"
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
# Fill in the value with the cardinality counter and 15 random alphanumeric characters
chore: pr cleanup on data generator
2021-09-13 21:45:01 +00:00
template = "{{id}}_{{random 15}}"
feat: Refactor Data Generator This is a huge commit that refactors the data generator. It removes many of the previous features that didn't quite make sense. The goal of this refactor was to make the data generator capable of representing complex tagsets that have values dependent on each other. It also significantly optimizes things to use far less memory and generate data much faster. Follow on work will update the generation of line protocol to support spaces in tags and their keys, double quotes in strings, and add more examples and documentation.
2021-10-26 19:34:35 +00:00
cardinality = 100
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
has_one = ["env"]
[[values]]
name = "env"
chore: pr cleanup on data generator
2021-09-13 21:45:01 +00:00
template = "whatever-environment-{{id}}"
feat: Refactor Data Generator This is a huge commit that refactors the data generator. It removes many of the previous features that didn't quite make sense. The goal of this refactor was to make the data generator capable of representing complex tagsets that have values dependent on each other. It also significantly optimizes things to use far less memory and generate data much faster. Follow on work will update the generation of line protocol to support spaces in tags and their keys, double quotes in strings, and add more examples and documentation.
2021-10-26 19:34:35 +00:00
cardinality = 2
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
[[values]]
refactor: optimizations to make tag set generation significantly faster
2021-09-09 22:55:04 +00:00
name = "bucket_id"
feat: Add tag_set and tag_pairs to measurements in Data Generator This adds the ability to specify a tag_set and a collection of tag_pairs to measurements in the data generator. Tag pairs are evaluated once when the generator is created. This avoids re-running handlebars evaluations while generating data for tags that don't change value. This commit also fixes an issue when printing the generation output to stdout while generating from more than one agent. Previously it would be garbled together. Follow on PRs will update the tag generation code in measurement specs to be more consistent and optimzised for performance. I'll be removing the restriction of using different options while using tag_set and tag_pairs. I wanted to get this in first to show the structure of what is output.
2021-09-17 20:00:06 +00:00
# a bucket belongs to an org. With this, you would be able to access the org.id or org.value in the template
refactor: optimizations to make tag set generation significantly faster
2021-09-09 22:55:04 +00:00
belongs_to = "org_id"
feat: Add tag_set and tag_pairs to measurements in Data Generator This adds the ability to specify a tag_set and a collection of tag_pairs to measurements in the data generator. Tag pairs are evaluated once when the generator is created. This avoids re-running handlebars evaluations while generating data for tags that don't change value. This commit also fixes an issue when printing the generation output to stdout while generating from more than one agent. Previously it would be garbled together. Follow on PRs will update the tag generation code in measurement specs to be more consistent and optimzised for performance. I'll be removing the restriction of using different options while using tag_set and tag_pairs. I wanted to get this in first to show the structure of what is output.
2021-09-17 20:00:06 +00:00
# each bucket will have a unique id, which is used here to guarantee uniqueness even across orgs. We also
# have a random 15 character alphanumeric sequence to pad out the value length.
chore: pr cleanup on data generator
2021-09-13 21:45:01 +00:00
template = "{{id}}_{{random 15}}"
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
# For each org, 3 buckets will be generated
cardinality = 3
[[values]]
name = "partition_id"
chore: pr cleanup on data generator
2021-09-13 21:45:01 +00:00
template = "{{id}}"
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
cardinality = 10
feat: Add tag_set and tag_pairs to measurements in Data Generator This adds the ability to specify a tag_set and a collection of tag_pairs to measurements in the data generator. Tag pairs are evaluated once when the generator is created. This avoids re-running handlebars evaluations while generating data for tags that don't change value. This commit also fixes an issue when printing the generation output to stdout while generating from more than one agent. Previously it would be garbled together. Follow on PRs will update the tag generation code in measurement specs to be more consistent and optimzised for performance. I'll be removing the restriction of using different options while using tag_set and tag_pairs. I wanted to get this in first to show the structure of what is output.
2021-09-17 20:00:06 +00:00
# makes a tagset so every bucket appears in every partition. The other tags are descriptive and don't
docs: Improving Readability r2 (#4781) * docs: Improve TOML readability Signed-off-by: Ryan Russell <git@ryanrussell.org> * docs: Fix `jemalloc` link Signed-off-by: Ryan Russell <git@ryanrussell.org> * docs: Readability in `point_generation.rs` Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-06-04 16:46:57 +00:00
# increase the cardinality beyond count(bucket) * count(partition). Later this example will use the
feat: Add tag_set and tag_pairs to measurements in Data Generator This adds the ability to specify a tag_set and a collection of tag_pairs to measurements in the data generator. Tag pairs are evaluated once when the generator is created. This avoids re-running handlebars evaluations while generating data for tags that don't change value. This commit also fixes an issue when printing the generation output to stdout while generating from more than one agent. Previously it would be garbled together. Follow on PRs will update the tag generation code in measurement specs to be more consistent and optimzised for performance. I'll be removing the restriction of using different options while using tag_set and tag_pairs. I wanted to get this in first to show the structure of what is output.
2021-09-17 20:00:06 +00:00
# agent and measurement generation to take this base tagset and increase cardinality on a per-agent basis.
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
[[tag_sets]]
name = "bucket_set"
for_each = [
refactor: optimizations to make tag set generation significantly faster
2021-09-09 22:55:04 +00:00
"role",
"url",
"org_id",
"org_id.env",
"org_id.bucket_id",
"partition_id",
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
]
refactor: change orgbucket to database in data generator
2021-12-09 18:39:18 +00:00
[[database_writers]]
database_ratio = 1.0
feat: add ability for data generator to write to many buckets This adds the ability for the data generator to write to many databases. A new command line argument, `bucket_list`, is added which should be a file name. The file should contain a list of databsaes, one per line, with the structure of <org>_<bucket>. This is a little odd given the data generator expects org and bucket separately, but I expect the file that we'll be using will be database names, which have this format. The configuration can specify what percentage of the list should get written to by which agents at what sampling interval. This should allow configurations where databases get different levels of ingest and different types (as specified via different agent specs). The structure is a little wonky, but I think it'll get the job done. The next step is to run some perf tests to see how the data generator performs if writing to 10k databases.
2021-12-07 21:54:40 +00:00
agents = [{name = "sender", sampling_interval = "10s"}]
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
[[agents]]
refactor: Add agent name to data generator (#3297) This is work leading up to giving the data generator the ability to write to many databases. The plan is to specify which agents databases will use to write data.
2021-12-05 16:21:04 +00:00
name = "sender"
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
[[agents.measurements]]
name = "storage_usage_bucket_cardinality"
feat: Add tag_set and tag_pairs to measurements in Data Generator This adds the ability to specify a tag_set and a collection of tag_pairs to measurements in the data generator. Tag pairs are evaluated once when the generator is created. This avoids re-running handlebars evaluations while generating data for tags that don't change value. This commit also fixes an issue when printing the generation output to stdout while generating from more than one agent. Previously it would be garbled together. Follow on PRs will update the tag generation code in measurement specs to be more consistent and optimzised for performance. I'll be removing the restriction of using different options while using tag_set and tag_pairs. I wanted to get this in first to show the structure of what is output.
2021-09-17 20:00:06 +00:00
# each sampling will have all the tag sets from this collection in addition to the tags and tag_pairs specified
tag_set = "bucket_set"
# for each agent, this specific measurement will be decorated with these additional tags.
tag_pairs = [
{key = "node_id", template = "{{agent.id}}"},
{key = "hostname", template = "{{agent.id}}"},
{key = "host", template = "storage-{{agent.id}}"},
]
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
[[agents.measurements.fields]]
name = "gauge"
i64_range = [1, 8147240]