Commit Graph

670 Commits (jdstrand/update-golang-jwt)

Author SHA1 Message Date
WeblWabl 2ffb108a27
feat(logging): Add startup logging for shard counts (#25378) (#25507)
* feat(logging): Add startup logging for shard counts (#25378)
This PR adds a check to see how many shards are remaining
vs how many shards are opened. This change displays the percent
completed too.

closes influxdata/feature-requests#476

(cherry picked from commit 3c87f52)

closes https://github.com/influxdata/influxdb/issues/25506
2024-11-01 09:20:35 -05:00
Mark Rushakoff f2898d1992 Wipe out workspace in preparation for v2 merge
"Knock knock."

"Who's there?"

"InfluxDB Veet."

...
2019-01-11 10:38:50 -08:00
Stuart Carnie 216cfa329f feat(flux): Add user authentication and authorization support
* Add AuthorizeDatabase API to QueryAuthorizer to verify a user has
  appropriate access to the specified database
* Update serverFluxQuery handler to require a meta.User when auth is
  enabled
* update Flux createFromSource and createBucketsSource dependencies to
  require Authorizer when auth is enabled in configuration
* update createFromSource to verify read permissions for each bucket
  specified in a Flux query
* update BucketsDecoder, which implements the buckets() Flux function,
  to return buckets that the user has read or write permissions to
* add unit tests to verify authentication is required for Flux HTTP
  requests when auth is enabled in configuration
2018-12-19 14:46:12 -07:00
Stuart Carnie 6affa7c7f5 chore(flux): Remove Flux gRPC handler and update config.sample.toml 2018-11-05 14:26:37 -07:00
Stuart Carnie 5bcb952b6f chore(cli): Improve APIs to increase reuse in Enterprise 2018-11-02 09:16:42 -07:00
Jeff Wendling 69dc031a75 Use platform for most of the read service code
This commit deletes most of the code to service reads from influxdb
and pulls it in from platform instead.

Of note, the models.Tag and models.Tags types are now aliases to the
platform models.Tag and models.Tags types. Additionally, many types
in the tsdb package relating to cursors are also aliases to the same
types in the platform cursors package.

This updates the platform and flux repos to the current master in the
Gopkg.lock.
2018-10-10 11:20:25 -06:00
Stuart Carnie 62b9791da8 feat(flux): Add initial support for executing Flux queries 2018-09-17 22:38:27 -07:00
Stuart Carnie 59bd822af9 refactor(storage): Export subset of APIs to facilitate reuse
* the protocol service definition, ReadRequest and ReadResponse is
  reused across projects, rather than requiring redefinition.

* the ReadRequest protocol buffer definition removes the concept of a
  database and retention policy, replacing it with a field named
  ReadSource of type google.protobuf.Any. OSS requests will use the
  ReadSource message structure defined in local to this package, which
  defines fields to represent a Database and RetentionPolicy. Other
  implementations can provide their own data structure allowing the
  remainder of the ReadRequest to be reused.

* The RPC service and Store are expected to be redefined to handle their
  specific requirements for resolving a ReadSource

* ResultSet and GroupResultSet are interfaces representing non-grouping
  and grouping read behavior respectively. Calling NewResultSet or
  NewGroupResultSet will construct instances of these types

* The ResponseWriter type is exported to deal with serialization of
  the ResultSet and GroupResultSet types
2018-08-24 11:56:04 -07:00
Jonathan A. Sternberg beae42a090
Merge pull request #10151 from influxdata/js-480-config-flux-section
Modify the `ifql` section to `flux`
2018-08-08 11:26:33 -05:00
Jonathan A. Sternberg be83e81102 Modify the `ifql` section to `flux`
Mark the breaking changes in the changelog.
2018-08-02 15:50:23 -05:00
Jeff Wendling f21d1773af move tls config into pkg
this makes it so you don't have to depend on cmd/influxd/run in
order to share the code with the parsing and stuff.
2018-08-02 14:29:07 -06:00
Jeff Wendling 767b991089 implement using tls configuration 2018-08-02 14:29:07 -06:00
Jeff Wendling 903bd6c14a add tls configuration 2018-08-02 11:21:14 -06:00
Edd Robinson 524f400836 Make testing of handler easier 2018-06-12 15:54:18 +01:00
Paul Dix 4f7b93342c Update Prometheus read/write to use new storage query layer.
* Update Prometheus remote write to use metric name as measurement name and value as the field name.
* Update Prometheus remote read to use the storage.Read method to bypass the InfluxQL query engine.
2018-06-12 15:54:18 +01:00
Ben Johnson 8b44e3142c
Add optional pprof http endpoint immediately on startup.
This commit adds `debug-pprof-enabled` which will start the default
`net/http/pprof` endpoint and bind against `localhost:6060`. This
will help to debug startup performance issues.
2018-05-23 14:32:15 -06:00
Jacob Marble 3cfbc33c0e Implement SHOW STATS FOR 'indexes' 2018-05-10 11:33:52 -07:00
Jonathan A. Sternberg dff34939f8
Merge pull request #9028 from cbushko/lint-the-c-dirs
Fix linting on cmd files
2018-04-23 10:41:05 -05:00
Mark Rushakoff fbddcf7cad Move ApplyEnvOverrides to toml package
This code has been duplicated to other projects and its implementations
have grown out of sync. Now the code can live as a package-level
function rather than a method coupled with particular structs.
2018-03-14 15:23:46 -07:00
Jonathan A. Sternberg 5d8d460034 Support setting the log level through the environment variable 2018-03-12 11:10:32 -05:00
Jonathan A. Sternberg ed27a00255
Merge pull request #9524 from influxdata/js-remove-query-prefix
Rename some of the structs and interfaces in the query package
2018-03-07 18:00:37 -06:00
Stuart Carnie 044ca2ac89 ensure ENV overrides are applied when configuring logging
Additionally, the config file is only parsed once.
2018-03-03 07:38:15 -07:00
Jonathan A. Sternberg de4390ae83 Rename some of the structs and interfaces in the query package
Remove the `Query` prefix from some structs and interfaces. They were
there so when the query engine was in the same package as influxql,
these would be differentiated. Now that the package name is query, the
extra prefix seems redundant.
2018-03-02 09:44:12 -06:00
Jonathan A. Sternberg 2bbd96768d Update logging calls to take advantage of structured logging
Includes a style guide that details the basics of how to log.
2018-02-20 10:04:19 -06:00
Stuart Carnie 1303030184 Suppress the InfluxDB banner if the log output is not a TTY 2018-02-14 15:23:58 -07:00
Jonathan A. Sternberg 2bb5b68833 Add an auto choice that will choose the most appropriate logger
This restricts the logging format options so console cannot be chosen.
This way, logs written to a non-TTY will never be the console format.
2018-02-14 11:56:29 -05:00
Jonathan A. Sternberg 2efc9ddbac Add an option to suppress the initial logo
For any systems that want to read the log file in the specific format,
the logo being printed on restart may not be good for those parsers
since the log parser would have to be aware of the logos existance or
capable of just ignoring lines it couldn't parse.

This gives an option to disable the printed logo if required.

Like other logging options, this will fail if the configuration file
itself is invalid.
2018-02-14 10:11:02 -05:00
Jonathan A. Sternberg ccf2e549b8 Add logging configuration to the main configuration file
Make logfmt the default log output format with json and a more user
friendly console into possible options.
2018-02-14 10:11:02 -05:00
Stuart Carnie 41dc96ca91 restore `MetaClient`, which is needed by store
* Switch from an anonymous type to avoid false positives with
  `megacheck`
2018-02-08 12:13:13 -07:00
Edd Robinson 13a1505ba2 Cleanup cmd package 2018-01-21 12:08:25 -08:00
Edd Robinson 6a66b5faf0 Cleanup services package 2018-01-21 10:52:37 -08:00
Jonathan A. Sternberg b775ad3d5d Expand unit test code coverage in services that were undercovered
This expands code coverage for the following packages:
* monitor (3.5% -> 86.9%)
* services/precreator (31.6% -> 83.8%)
* services/retention (83.0% -> 84.9%)
* services/snapshotter (0.0% -> 82.1%)
* tcp (48.7% -> 60.0%)
2017-11-28 15:44:35 -06:00
Jason Wilder 48e21e6fc8
Merge pull request #9084 from influxdata/jw-delete-time
Handle high cardinality deletes in TSM engine
2017-11-13 14:39:54 -07:00
Stuart Carnie 40ffa49a62 rename config section to ifql
Renaming for discoverability for the initial release of ifql
2017-11-13 11:05:02 -08:00
Jason Wilder ca4998f29e Fix pid test to ignore local data
If you have lots of data stored locally, this test takes a while to
complete since it loads it all up from the users home dir.
2017-11-13 08:50:07 -07:00
Jonathan A. Sternberg 0b7c56bcd8 Update the zap logger dependency
The previous sha was taken from a revision on a devel branch that I
thought would continue staying in the tree after it was merged. That
revision was rebased away and the API was changed for the logger.

This updates the usage of the logger and adds a simple package for
constructing the base logger.

The 1.0 version of zap changed the format of the default console logger
so this change moves over to this new logger instead of attempting to
retain backwards compatibility with the old format.
2017-11-10 16:27:16 -06:00
Jonathan A. Sternberg e13ad18160 Use the isolated temp dir for the test pid server 2017-11-07 10:40:46 -06:00
Edd Robinson 3e0bf925f8 Use random port in PID file test 2017-11-06 19:35:17 +00:00
Jonathan A. Sternberg 9dc60962c7 Handle utf16 files when reading the configuration file
Windows computers may produce a utf16 file from the command line that
contains a byte-order-mark. Along with handling the utf8
byte-order-mark, this also handles the utf16 for better Windows
compatibility.
2017-10-30 22:48:44 -05:00
Curt Bushko 37c38326a4 fix linting on cmd 2017-10-26 21:15:27 -04:00
Stuart Carnie 415ed14c53 storage service
* storage service is disabled by default
* default port 8082
* RPC interface generated using yarpc via service.proto
2017-10-25 13:38:07 -07:00
Jonathan A. Sternberg 75b8d9cd4f Code cleanup for removing the pid file 2017-10-20 09:17:16 -05:00
Jonathan A. Sternberg 2005a8c8e5 Remove the pidfile after the server has exited 2017-10-19 16:33:59 -05:00
Mark Rushakoff f3f1cc1064 Initial integration tests for config settings 2017-10-11 17:16:42 -07:00
Jonathan A. Sternberg 9a2357c2c0 Separate the query engine into a separate package
This change provides a clear separation between the query engine
mechanics and the query language so that the language can be parsed and
dealt with separate from the query engine itself.
2017-08-16 13:38:43 -05:00
Edd Robinson 8934aacebb Adds mutex profile
Mutex profiles are now available via the:

  - /debug/pprof/mutex
  - /debug/pprof/all

endpoints.
2017-08-04 14:36:39 +01:00
Adam 2259ada8c3 adds a new header key/value X-Influxdb-Build that has value OSS if called from open-source build, and ENT if called from enterprise. This commit sets the value for the OSS case, and also creates the proper flag 2017-07-07 10:59:26 -04:00
Jason Wilder 72eeacaed4 Append subscribers service after it's open
The Points channel is nil until after the subscriber service is opened.
If it is append before it's opened, the PointsWriter holds onto the
old reference.
2017-06-21 09:28:59 -06:00
Stuart Carnie 932edd90b2 Merge branch 'master' into sgc-8188 2017-06-14 10:55:06 +10:00
Jason Wilder 206fd857c3 Merge pull request #8481 from influxdata/jw-subscriber
Decouple Subscriber from PointsWriter
2017-06-13 15:55:43 -06:00