Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
influxdb/iox_data_generator/schemas/storage_cardinality_example...

81 lines
2.7 KiB
TOML
Raw Normal View History

feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
name = "storage_cardinality_example"
base_seed = "this is a demo"
feat: Add tag_set and tag_pairs to measurements in Data Generator This adds the ability to specify a tag_set and a collection of tag_pairs to measurements in the data generator. Tag pairs are evaluated once when the generator is created. This avoids re-running handlebars evaluations while generating data for tags that don't change value. This commit also fixes an issue when printing the generation output to stdout while generating from more than one agent. Previously it would be garbled together. Follow on PRs will update the tag generation code in measurement specs to be more consistent and optimzised for performance. I'll be removing the restriction of using different options while using tag_set and tag_pairs. I wanted to get this in first to show the structure of what is output.
2021-09-17 20:00:06 +00:00
# Values are automatically generated before the agents are intialized. They generate tag key/value pairs
# with the name of the value as the tag key and the evaluated template as the value. These pairs
# are Arc wrapped so they can be shared across tagsets and used in the agents as pre-generated data.
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
[[values]]
feat: Add tag_set and tag_pairs to measurements in Data Generator This adds the ability to specify a tag_set and a collection of tag_pairs to measurements in the data generator. Tag pairs are evaluated once when the generator is created. This avoids re-running handlebars evaluations while generating data for tags that don't change value. This commit also fixes an issue when printing the generation output to stdout while generating from more than one agent. Previously it would be garbled together. Follow on PRs will update the tag generation code in measurement specs to be more consistent and optimzised for performance. I'll be removing the restriction of using different options while using tag_set and tag_pairs. I wanted to get this in first to show the structure of what is output.
2021-09-17 20:00:06 +00:00
# the name must not have a . in it, which is used to access children later. Otherwise it's open.
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
name = "role"
feat: Add tag_set and tag_pairs to measurements in Data Generator This adds the ability to specify a tag_set and a collection of tag_pairs to measurements in the data generator. Tag pairs are evaluated once when the generator is created. This avoids re-running handlebars evaluations while generating data for tags that don't change value. This commit also fixes an issue when printing the generation output to stdout while generating from more than one agent. Previously it would be garbled together. Follow on PRs will update the tag generation code in measurement specs to be more consistent and optimzised for performance. I'll be removing the restriction of using different options while using tag_set and tag_pairs. I wanted to get this in first to show the structure of what is output.
2021-09-17 20:00:06 +00:00
# the template can use a number of helpers to get an id, a random string and the name, see below for examples
chore: pr cleanup on data generator
2021-09-13 21:45:01 +00:00
template = "storage"
feat: Add tag_set and tag_pairs to measurements in Data Generator This adds the ability to specify a tag_set and a collection of tag_pairs to measurements in the data generator. Tag pairs are evaluated once when the generator is created. This avoids re-running handlebars evaluations while generating data for tags that don't change value. This commit also fixes an issue when printing the generation output to stdout while generating from more than one agent. Previously it would be garbled together. Follow on PRs will update the tag generation code in measurement specs to be more consistent and optimzised for performance. I'll be removing the restriction of using different options while using tag_set and tag_pairs. I wanted to get this in first to show the structure of what is output.
2021-09-17 20:00:06 +00:00
# this number of tag pairs will be generated. If this is > 1, the id or a random character string should be
# used in the template to ensure that the tag key/value pairs are unique.
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
cardinality = 1
[[values]]
name = "url"
chore: pr cleanup on data generator
2021-09-13 21:45:01 +00:00
template = "http://127.0.0.1:6060/metrics/usage"
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
cardinality = 1
[[values]]
refactor: optimizations to make tag set generation significantly faster
2021-09-09 22:55:04 +00:00
name = "org_id"
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
# Fill in the value with the cardinality counter and 15 random alphanumeric characters
chore: pr cleanup on data generator
2021-09-13 21:45:01 +00:00
template = "{{id}}_{{random 15}}"
feat: Add tag_set and tag_pairs to measurements in Data Generator This adds the ability to specify a tag_set and a collection of tag_pairs to measurements in the data generator. Tag pairs are evaluated once when the generator is created. This avoids re-running handlebars evaluations while generating data for tags that don't change value. This commit also fixes an issue when printing the generation output to stdout while generating from more than one agent. Previously it would be garbled together. Follow on PRs will update the tag generation code in measurement specs to be more consistent and optimzised for performance. I'll be removing the restriction of using different options while using tag_set and tag_pairs. I wanted to get this in first to show the structure of what is output.
2021-09-17 20:00:06 +00:00
cardinality = 2
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
has_one = ["env"]
[[values]]
name = "env"
chore: pr cleanup on data generator
2021-09-13 21:45:01 +00:00
template = "whatever-environment-{{id}}"
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
cardinality = 10
[[values]]
refactor: optimizations to make tag set generation significantly faster
2021-09-09 22:55:04 +00:00
name = "bucket_id"
feat: Add tag_set and tag_pairs to measurements in Data Generator This adds the ability to specify a tag_set and a collection of tag_pairs to measurements in the data generator. Tag pairs are evaluated once when the generator is created. This avoids re-running handlebars evaluations while generating data for tags that don't change value. This commit also fixes an issue when printing the generation output to stdout while generating from more than one agent. Previously it would be garbled together. Follow on PRs will update the tag generation code in measurement specs to be more consistent and optimzised for performance. I'll be removing the restriction of using different options while using tag_set and tag_pairs. I wanted to get this in first to show the structure of what is output.
2021-09-17 20:00:06 +00:00
# a bucket belongs to an org. With this, you would be able to access the org.id or org.value in the template
refactor: optimizations to make tag set generation significantly faster
2021-09-09 22:55:04 +00:00
belongs_to = "org_id"
feat: Add tag_set and tag_pairs to measurements in Data Generator This adds the ability to specify a tag_set and a collection of tag_pairs to measurements in the data generator. Tag pairs are evaluated once when the generator is created. This avoids re-running handlebars evaluations while generating data for tags that don't change value. This commit also fixes an issue when printing the generation output to stdout while generating from more than one agent. Previously it would be garbled together. Follow on PRs will update the tag generation code in measurement specs to be more consistent and optimzised for performance. I'll be removing the restriction of using different options while using tag_set and tag_pairs. I wanted to get this in first to show the structure of what is output.
2021-09-17 20:00:06 +00:00
# each bucket will have a unique id, which is used here to guarantee uniqueness even across orgs. We also
# have a random 15 character alphanumeric sequence to pad out the value length.
chore: pr cleanup on data generator
2021-09-13 21:45:01 +00:00
template = "{{id}}_{{random 15}}"
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
# For each org, 3 buckets will be generated
cardinality = 3
[[values]]
name = "partition_id"
chore: pr cleanup on data generator
2021-09-13 21:45:01 +00:00
template = "{{id}}"
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
cardinality = 10
feat: Add tag_set and tag_pairs to measurements in Data Generator This adds the ability to specify a tag_set and a collection of tag_pairs to measurements in the data generator. Tag pairs are evaluated once when the generator is created. This avoids re-running handlebars evaluations while generating data for tags that don't change value. This commit also fixes an issue when printing the generation output to stdout while generating from more than one agent. Previously it would be garbled together. Follow on PRs will update the tag generation code in measurement specs to be more consistent and optimzised for performance. I'll be removing the restriction of using different options while using tag_set and tag_pairs. I wanted to get this in first to show the structure of what is output.
2021-09-17 20:00:06 +00:00
# makes a tagset so every bucket appears in every partition. The other tags are descriptive and don't
# increase the cardiality beyond count(bucket) * count(partition). Later this example will use the
# agent and measurement generation to take this base tagset and increase cardinality on a per-agent basis.
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
[[tag_sets]]
name = "bucket_set"
for_each = [
refactor: optimizations to make tag set generation significantly faster
2021-09-09 22:55:04 +00:00
"role",
"url",
"org_id",
"org_id.env",
"org_id.bucket_id",
"partition_id",
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
]
[[agents]]
name = "metric-scraper"
feat: Add tag_set and tag_pairs to measurements in Data Generator This adds the ability to specify a tag_set and a collection of tag_pairs to measurements in the data generator. Tag pairs are evaluated once when the generator is created. This avoids re-running handlebars evaluations while generating data for tags that don't change value. This commit also fixes an issue when printing the generation output to stdout while generating from more than one agent. Previously it would be garbled together. Follow on PRs will update the tag generation code in measurement specs to be more consistent and optimzised for performance. I'll be removing the restriction of using different options while using tag_set and tag_pairs. I wanted to get this in first to show the structure of what is output.
2021-09-17 20:00:06 +00:00
# create this many agents
count = 3
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
[[agents.measurements]]
name = "storage_usage_bucket_cardinality"
feat: Add tag_set and tag_pairs to measurements in Data Generator This adds the ability to specify a tag_set and a collection of tag_pairs to measurements in the data generator. Tag pairs are evaluated once when the generator is created. This avoids re-running handlebars evaluations while generating data for tags that don't change value. This commit also fixes an issue when printing the generation output to stdout while generating from more than one agent. Previously it would be garbled together. Follow on PRs will update the tag generation code in measurement specs to be more consistent and optimzised for performance. I'll be removing the restriction of using different options while using tag_set and tag_pairs. I wanted to get this in first to show the structure of what is output.
2021-09-17 20:00:06 +00:00
# each sampling will have all the tag sets from this collection in addition to the tags and tag_pairs specified
tag_set = "bucket_set"
# for each agent, this specific measurement will be decorated with these additional tags.
tag_pairs = [
{key = "node_id", template = "{{agent.id}}"},
{key = "hostname", template = "{{agent.id}}"},
{key = "host", template = "storage-{{agent.id}}"},
]
feat: Add pre-generated values and tag sets to data generator This adds the ability to pre-generate values and tag sets in the data generator. This makes it easy to have tags that depend on other tag values (like buckets in an org) and have tag values that have one associated tag (like if something is in production or staging environment). Follow on work will add the actual generation to the agent spec. An added bonus of these pre-generated values is that generating samples won't require any sort of template evaluation for all of the tags in the tag sets. Only unique values (like trace_id or span_id) would need to be generated during sampling generation.
2021-09-02 18:35:40 +00:00
[[agents.measurements.fields]]
name = "gauge"
i64_range = [1, 8147240]