Commit Graph

238 Commits (933a14e16f0232c5ec883644b7363a59da7bcd90)

Author SHA1 Message Date
Nicolas Hennion 717d93ff96
chore: fix typo in config.sample.toml (#21125) 2022-04-14 13:26:21 -05:00
davidby-influx eb3bc7069f
feat: configurable DELETE concurrency (#23055)
Currently, deletion of series or measurements are
serialized. This new feature will add
max-concurrent-deletes to the [data] section of the
 configuration file. Legal values are any positive
 number, defaulting to 1, the current behavior.

 closes https://github.com/influxdata/influxdb/issues/23054
2022-01-13 11:04:57 -08:00
davidby-influx fd1d51690d
feat: optionally dump queries to log on SIGTERM (#22638)
Dump all active queries to the log when a SIGTERM
is received and the termination-query-log flag is
true in the coordinator section of the config. The
default is false.
2021-10-07 14:02:38 -07:00
Sam Arnold df46a86326
fix(tls): Enable configuration of TLS 1.3 (#20939)
Co-authored-by: Gunnar Aasen <gunnar.r.aasen@gmail.com>
2021-03-12 16:35:28 -05:00
Sam Arnold dd3baf6d4a
feat: measurement metrics by login (#20687)
After turning on authentication and both forms of ingress metrics:

"ingress": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"cq","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":76}},
"ingress:1": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"database","rp":"monitor"},"values":{"pointsWritten":76,"valuesWritten":152}},
"ingress:2": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"httpd","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":874}},
"ingress:3": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"ingress","rp":"monitor"},"values":{"pointsWritten":534,"valuesWritten":1068}},
"ingress:4": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"localStore","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":76}},
"ingress:5": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"queryExecutor","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":190}},
"ingress:6": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"runtime","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":570}},
"ingress:7": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"shard","rp":"monitor"},"values":{"pointsWritten":76,"valuesWritten":836}},
"ingress:8": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"subscriber","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":114}},
"ingress:9": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_cache","rp":"monitor"},"values":{"pointsWritten":76,"valuesWritten":684}},
"ingress:10": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_engine","rp":"monitor"},"values":{"pointsWritten":76,"valuesWritten":2204}},
"ingress:11": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_filestore","rp":"monitor"},"values":{"pointsWritten":76,"valuesWritten":152}},
"ingress:12": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_wal","rp":"monitor"},"values":{"pointsWritten":76,"valuesWritten":304}},
"ingress:13": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"write","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":342}},
"ingress:14": {"name":"ingress","tags":{"db":"telegraf","login":"admin","measurement":"cpu","rp":"autogen"},"values":{"pointsWritten":1,"valuesWritten":1}},
"ingress:15": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"cpu","rp":"autogen"},"values":{"pointsWritten":1316,"valuesWritten":13160}},
"ingress:16": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"disk","rp":"autogen"},"values":{"pointsWritten":642,"valuesWritten":4494}},
"ingress:17": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"diskio","rp":"autogen"},"values":{"pointsWritten":214,"valuesWritten":2354}},
"ingress:18": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"mem","rp":"autogen"},"values":{"pointsWritten":107,"valuesWritten":963}},
"ingress:19": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"processes","rp":"autogen"},"values":{"pointsWritten":107,"valuesWritten":856}},
"ingress:20": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"swap","rp":"autogen"},"values":{"pointsWritten":214,"valuesWritten":642}},
"ingress:21": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"system","rp":"autogen"},"values":{"pointsWritten":321,"valuesWritten":749}},

Only by login:

"ingress": {"name":"ingress","tags":{"login":"_systemuser_monitor"},"values":{"pointsWritten":42,"valuesWritten":354}},
"ingress:1": {"name":"ingress","tags":{"login":"admin"},"values":{"pointsWritten":1,"valuesWritten":1}},
"ingress:2": {"name":"ingress","tags":{"login":"telegraf"},"values":{"pointsWritten":3547,"valuesWritten":28246}},

Notice writes by users 'telegraf', '_systemuser_monitor', and 'admin'.
2021-02-04 11:52:53 -05:00
davidby-influx 9e33be2619
fix(error): SELECT INTO doesn't return error with unsupported value (#20429)
When a SELECT INTO query generates an illegal value that cannot be inserted,
like +/- Inf, it should return an error, rather than failing silently.
This adds a boolean parameter to the [data] section of influxdb.conf:
* strict-error-handling
When false, the default, the old behavior is preserved.  When true,
unsupported values will return an error from SELECT INTO queries

Fixes https://github.com/influxdata/influxdb/issues/20426
2020-12-30 18:22:43 -08:00
Ayan George 04536858d7
Merge branch 'master-1.x' into new-http-headers 2020-06-30 11:03:04 -04:00
Ayan George dde8231d5c feat(http): Allow user supplied HTTP headers
This patch adds the [http.headers] subsection to the configuration file
that allows users to supply headers that will be returned in all HTTP
responses.

Applying this patch will:

* Add code to implement new configuration items.
* Add test to ensure configuration is properly parsed.
* Add test to ensure http response headers are set
* Update sample configuration file
2020-06-30 10:59:25 -04:00
Tristan Su 57ea78e984
fix(httpd): add option to authenticate prometheus remote read (#18429) 2020-06-23 15:03:19 -07:00
Gunnar Aasen 0c4703aaa3 feat(httpd): Add option to authenticate debug/pprof and ping endpoints 2019-09-20 16:32:22 -07:00
Ahmed Ashour 926fa7b079 Fix typos (#11494) 2019-02-21 08:28:01 -08:00
Edd Robinson efdddbb31a Allow TSI bitset cache size to be configured
This commit adds a config option to the tsdb Config allowing the size of
the bitset cached in the TSI index to be specified.

Setting the cache size to 0 will disable the cache.
2019-01-24 17:41:45 +00:00
Stuart Carnie c47a3ea2af feat(flux): Add support for optionally logging Flux queries
New configuration:

```toml
[http]
flux-log-enabled = true
```

will produce log entries similar to:

```
ts=2019-01-11T14:47:46.124347Z lvl=info msg="Executed Flux query" log_id=0CwLysq0000 service=httpd compiler_type=flux response_size=1467 query="from(bucket:\"test\") |> range(start: -5000h) |> limit(n:5)" stat_total_duration=3.949ms stat_compile_duration=3.183ms stat_queue_duration=0.026ms stat_plan_duration=0.055ms stat_requeue_duration=0.000ms stat_execute_duration=0.668ms stat_max_allocated=3200 stat_concurrency=1
```
2019-01-11 09:09:26 -07:00
Stuart Carnie 6affa7c7f5 chore(flux): Remove Flux gRPC handler and update config.sample.toml 2018-11-05 14:26:37 -07:00
Stuart Carnie 317fd4982f chore(flux): Document flux-enabled config option 2018-11-05 14:26:37 -07:00
Jonathan A. Sternberg b3472a54ee
Add an access log filter for the access log
The access log filter allows the access log to be filtered by a status
code pattern. The pattern is a list of strings of the form `NXX`. At
least one number must be specified and up to 2 Xs can be used. That
means the filter can be an exact status code or it can be a range of
them. For example, `500` would only match the 500 http status code while
`5XX` would match any status code beginning with the number 5
(categorized as server errors). The pattern `50X` would also be
accepted. Both uppercase and lowercase Xs are allowed.

Multiple filters can be specified and the log line will be printed if
one of them matches. If there are no filters specified, all status codes
are printed.
2018-11-05 11:46:26 -06:00
Ben Johnson 2d266ca186
Merge pull request #9801 from influxdata/bj-validate-write
Add option for unicode validation.
2018-08-20 03:44:41 -10:00
Zach Goldstein 0ef3752a1a Add configuration parameter to expose rate limit for TSM compaction.
Closes: 9938
2018-08-07 10:05:36 -04:00
Edd Robinson 282d265dd4 Add mmap madvice kernel value config option
This PR adds a configuration option that can be used to inform the
kernel that we intent to page in much of the TSM files.

This madvise value has been problematic in the past when its been set,
so this option defaults to off. It may be useful to some users with slow
disks.
2018-08-03 14:07:46 +01:00
Jeff Wendling f4c3c04f8f respond to review comments 2018-08-02 16:58:43 -06:00
Jeff Wendling 767b991089 implement using tls configuration 2018-08-02 14:29:07 -06:00
Gunnar Aasen 8870512e6b Update example config with UDP precision option
Also add a test for precision in the UDP configuration.
2018-07-09 10:07:05 -07:00
Ben Johnson 8b44e3142c
Add optional pprof http endpoint immediately on startup.
This commit adds `debug-pprof-enabled` which will start the default
`net/http/pprof` endpoint and bind against `localhost:6060`. This
will help to debug startup performance issues.
2018-05-23 14:32:15 -06:00
Ben Johnson 8a74c6759f
Add http write throttling.
This commit adds throttling to the HTTP write endpoints based on
queue depth and, optionally, timeout. Two queues exist: `enqueued`
and `current`. The `current` queue is the number of concurrent
requests that can be processed. The `enqueued` queue limits the
maximum number of requests that can be waiting to be processed.

If the timeout is exceeded or the `enqueued` queue is full then
a `"503 Service unavailable"` code is returned and the error is
logged.

By default these options are turned off.
2018-05-21 13:08:24 -06:00
Ben Johnson 58aed93fe6
Add option for unicode validation. 2018-05-02 11:16:55 -06:00
Jonathan A. Sternberg a7e1da5f86 Add suppress-write-log option to disable the write log when the log is enabled 2018-04-23 12:45:48 -05:00
Ben Johnson 92d38414f2
Add adjustable TSI log file size.
This commit adds the `max-index-log-file-size` configuration flag so
that users can restrict the maximum size of log files before compaction.
The default limit was also lowered from `5MB` to `1MB`. The original
size was set before we partitioned the index so the change reflects this.
2018-04-02 11:47:59 -06:00
Adam Petrovic d3e02032df Fixed typo in config comments 2018-03-12 11:28:20 +11:00
Jonathan A. Sternberg 426b6ee151 Update sample config with information on logging levels
Update the query executor to log a message using the style guide. The
actual message is static and the query is now in a context field.
2018-02-21 11:25:38 -06:00
Jonathan A. Sternberg 37bcfc05bc Fix the logging sample config
The original PR that modified this was modified without changing the
sample config so this makes the sample config more accurate.
2018-02-20 15:55:11 -06:00
Stuart Carnie 4f90592316 Update config help docs 2018-02-15 09:39:50 -07:00
Stuart Carnie 584e7ac09a Added option to write HTTP request logs to separate file. 2018-02-14 23:11:01 -07:00
Stuart Carnie 1303030184 Suppress the InfluxDB banner if the log output is not a TTY 2018-02-14 15:23:58 -07:00
Jonathan A. Sternberg 2efc9ddbac Add an option to suppress the initial logo
For any systems that want to read the log file in the specific format,
the logo being printed on restart may not be good for those parsers
since the log parser would have to be aware of the logos existance or
capable of just ignoring lines it couldn't parse.

This gives an option to disable the printed logo if required.

Like other logging options, this will fail if the configuration file
itself is invalid.
2018-02-14 10:11:02 -05:00
Jonathan A. Sternberg ccf2e549b8 Add logging configuration to the main configuration file
Make logfmt the default log output format with json and a more user
friendly console into possible options.
2018-02-14 10:11:02 -05:00
Jason Wilder 6e3602c937 Revert "Increase cache-snapshot-memory-size default"
This reverts commit 171b427a1b.
2017-12-13 13:18:08 -07:00
Jason Wilder e584cb6842 Increase cache-snapshot-memory-size default
With the recent changes to compactions and snapshotting, the current
default can create lots of small level 1 TSM files.  This increases
the default in order to create larger level 1 files and less disk
utilization.
2017-12-06 09:39:03 -07:00
Stuart Carnie 40ffa49a62 rename config section to ifql
Renaming for discoverability for the initial release of ifql
2017-11-13 11:05:02 -08:00
Jonathan A. Sternberg 87ed89ee74 Implement pull request feedback for human readable sizes 2017-11-01 13:08:51 -05:00
Andrew Hare ecb3952fa9 Allow human-readable byte sizes in config
Update support in the `toml` package for parsing human-readble byte sizes.
Supported size suffixes are "k" or "K" for kibibytes, "m" or "M" for
mebibytes, and "g" or "G" for gibibytes. If a size suffix isn't specified
then bytes are assumed.

In the config, `cache-max-memory-size` and `cache-snapshot-memory-size` are
now typed as `toml.Size` and support the new syntax.
2017-11-01 11:09:09 -05:00
Jason Wilder db204f3eb7 Default concurrent compactions to 50% of available cores 2017-09-21 12:48:11 -06:00
Adam e4b4cfb7a1 8426 Updated collectd section in default config file 2017-06-23 09:25:36 -04:00
Stuart Carnie 932edd90b2 Merge branch 'master' into sgc-8188 2017-06-14 10:55:06 +10:00
Jonathan A. Sternberg f7382982fd Change the default stats interval to 1 second instead of 10 seconds 2017-06-12 13:15:25 -05:00
Jonathan A. Sternberg 9c0d31cb36 Update the comment for `max-select-point`
Point out that this limit is only checked every 10 seconds.
2017-06-12 13:07:44 -05:00
Stuart Carnie 2de52834f0 CQ statistics written to monitor database, addresses #8188
* off by default, enabled by `query-stats-enabled`
* writes to cq_query measurement of configured monitor database
* see CHANGELOG for schema of individual points
2017-06-10 09:20:38 +08:00
Stuart Carnie 98f2050bcb Update config sample and CHANGELOG 2017-06-05 22:05:00 +08:00
Tobias Schaefer 7d665ba560 Fix typos in comments. 2017-05-28 07:30:55 +02:00
Mark Rushakoff 3da9fded59 Clean up sample config file
Fixes #7767, #7760.
2017-05-12 17:03:24 -07:00
Jonathan A. Sternberg dea02009e0 Small edits to the etc/config.sample.toml file 2017-05-10 10:56:34 -05:00