Commit Graph

228 Commits (1ffe13894dba95f5a103f03bee1016b033bf88f2)

Author SHA1 Message Date
Ayan George 2529799abb
refactor: Change ToLower comparisons to EqualFold (#18147)
When comparing strings in a case-insensitive way, strings.EqualFold() is
(almost?) always faster than comparing the results of strings.ToLower().

In addition, strings.EqualFold() never causes an allocation.

This patch replaces case-insensitive string comparisons that use
strings.ToLower() with a strings.EqualFold() call.
2020-05-18 19:46:59 -04:00
ricky c1e11e732e fix(meta): remove double index increment of set meta 2020-01-14 11:11:43 +08:00
Stuart Carnie 216cfa329f feat(flux): Add user authentication and authorization support
* Add AuthorizeDatabase API to QueryAuthorizer to verify a user has
  appropriate access to the specified database
* Update serverFluxQuery handler to require a meta.User when auth is
  enabled
* update Flux createFromSource and createBucketsSource dependencies to
  require Authorizer when auth is enabled in configuration
* update createFromSource to verify read permissions for each bucket
  specified in a Flux query
* update BucketsDecoder, which implements the buckets() Flux function,
  to return buckets that the user has read or write permissions to
* add unit tests to verify authentication is required for Flux HTTP
  requests when auth is enabled in configuration
2018-12-19 14:46:12 -07:00
David Norton 8575378ae8 fix #10110: limit db & rp name length to 255 2018-12-17 16:58:28 -05:00
Tanya Gordeeva 9d711d9417 httpd/meta: use open auth when unrestricted
This changes adds additional auth shortcutting, primarily for enterprise
usecases with simple FGA setups. OSS users won't see any changes.
2018-08-20 12:55:40 -07:00
Edd Robinson 9eece563b1 Simplify loops 2018-08-05 15:16:33 +01:00
Jeff Wendling e6aec771b0 fix(tsdb): attempt to work on docker on windows
multiple users have attempted to run influxdb in a docker container
with a windows host and a volume mounted from windows. that causes
problems because it apparently uses samba/cifs which does not
support fsync on directories. this patchset will, if it receives an EINVAL
on directory fsync, as is what appears to happen on samba/cifs, then it
will ignore it. this should help.

fixes #9833.
fixes #9630.
2018-06-01 14:57:18 -06:00
Stuart Carnie a8692a9e24 services/meta: improve readability of Contains function, add unit tests 2018-04-19 18:05:55 -07:00
Stuart Carnie d135aecf02 Generate trace logs for a number of significant influx operations
* tsdb Store.Open traces all events related to opening files
    * op.name : tsdb.open
* retention policy shard deletions
    * op.name : retention.delete_check
* all TSM compaction strategies
    * op.name : tsm1.compact_group
* series file compactions
    * op.name : series_partition.compaction
* continuous query execution (if logging enabled)
    * op.name : continuous_querier.execute
* TSI log file compaction
    * op_name: index.tsi.compact_log_file
* TSI level compaction
    * op.name: index.tsi.compact_to_level
2018-02-21 15:08:49 -07:00
Jonathan A. Sternberg 2bbd96768d Update logging calls to take advantage of structured logging
Includes a style guide that details the basics of how to log.
2018-02-20 10:04:19 -06:00
Mark Rushakoff f7fc6a6501 Remove IsAdmin() method from meta.User interface
The method was only called in tests.
2018-02-13 16:59:22 -08:00
Andrew Hare d21ebfe531 Do not report an error when dropping a CQ on a non-existent DB/RP
This makes the behvior similar to other places in the DB where we
don't return an error when we try to drop an object from a non-
existent database.
2018-02-08 13:28:47 -06:00
Edd Robinson 6a66b5faf0 Cleanup services package 2018-01-21 10:52:37 -08:00
Adam 938db68198
Update restore functionality to run in online mode, consume Enterprise backup files. (#9207)
* Live Restore + Enterprise data format compatability

* Extended ImportData to import all DB's if no db name given

* Added a new enterprise data test, and backup command now prints the backup file paths at conclusion

* Added whole-system backup test

* Update to use protobuf in all enterprise data cases

* Update to test to do cross-testing with enterprise version

* incremental enterprise backup format support
2018-01-10 13:59:18 -05:00
Andrew Hare d7e328050c
Merge branch 'master' into ah-truncate-shards 2017-11-28 17:25:31 -07:00
Jonathan A. Sternberg 0b7c56bcd8 Update the zap logger dependency
The previous sha was taken from a revision on a devel branch that I
thought would continue staying in the tree after it was merged. That
revision was rebased away and the API was changed for the logger.

This updates the usage of the logger and adds a simple package for
constructing the base logger.

The 1.0 version of zap changed the format of the default console logger
so this change moves over to this new logger instead of attempting to
retain backwards compatibility with the old format.
2017-11-10 16:27:16 -06:00
Stuart Carnie f3d45ba301 influxdata/influxdb/influxql -> influxdata/influxql 2017-10-30 14:40:26 -07:00
Andrew Hare 13c3808aff Add test 2017-10-19 15:57:16 -06:00
Andrew Hare e6aa5023eb Create a command to truncated shard groups 2017-10-16 20:34:26 -06:00
Joe LeGasse a95647b720 cleanup: remove poor usage of ',ok' with maps
There are several places in the code where comma-ok map retrieval was
being used poorly. Some were benign, like checking existence before
issuing an unconditional delete with no cleanup. Others were potentially
far more serious: assuming that if 'ok' was true, then the resulting
pointer retrieved from the map would be non-nil. `nil` is a perfectly
valid value to store in a map of pointers, and the comma-ok syntax is
meant for when membership is distinct from having a non-zero value.
There was only one or two cases that I saw that being used correctly for
maps of pointers.
2017-08-30 09:49:31 -04:00
Jonathan A. Sternberg 9a2357c2c0 Separate the query engine into a separate package
This change provides a clear separation between the query engine
mechanics and the query language so that the language can be parsed and
dealt with separate from the query engine itself.
2017-08-16 13:38:43 -05:00
Edd Robinson e1a5ee4ede Ensure privileges can't be set on non-existent DB 2017-06-26 14:15:52 +01:00
Stuart Carnie 47f97ea134 use parsed measurement and models.Tags 2017-05-26 13:21:59 -07:00
Joe LeGasse 815f740f4c initial fga work
wip

wip

fix tests / build
2017-05-26 13:16:27 -07:00
David Norton c345f5d700 remove unused NewShardOwner func 2017-05-25 11:37:08 -04:00
Edd Robinson 255992f5ec Reduce cost of admin user check
This commits adds a caching mechanism to the Data object, such that
when large numbers of users exist in the system, the cost of determining
if there is at least one admin user will be low.

To ensure that previously marshalled Data objects contain the correct
cached admin user value, we exhaustively determine if there is an admin
user present whenever we unmarshal a Data object.
2017-03-20 12:04:03 +00:00
Mark Rushakoff 535cf597f1 Report subset of config values in SHOW DIAGNOSTICS
This includes hand-selected config settings that are safe to expose and
not expected to include any kind of secrets.

Fixes #7821
2017-03-14 11:34:19 -07:00
Mark Rushakoff 601cbcd084 Merge branch '1.2' into mr-merge-12 2017-02-17 16:14:22 -08:00
Jonathan A. Sternberg 2fe48d6781 Rename zap import back to github.com/uber-go/zap
They rebased a revision we were previously relying upon that allowed us
to use the vanity name so we are reverting back to an older version with
the old import path.
2017-02-17 17:17:22 -06:00
Mark Rushakoff 53699aa24f Allow non-admin users to execute SHOW DATABASES
This commit introduces a new interface type, influxql.Authorizer, that
is passed as part of a statement's execution context and determines
whether the context is permitted to access a given database. In the
future, the Authorizer interface may be expanded to other resources
besides databases. In this commit, the Authorizer interface is
specifically used to determine which databases are returned when
executing SHOW DATABASES.

When HTTP authentication is enabled, the existing meta.UserInfo struct
implements Authorizer, meaning admin users can SHOW every database, and
non-admin users can SHOW only databases for which they have read and/or
write permission.

When HTTP authentication is disabled, all databases are visible through
SHOW DATABASES.

This addresses a long-standing issue where Chronograf or Grafana would
be unable to list databases if the logged-in user did not have admin
privileges.

Fixes #4785.
2017-02-13 08:59:16 -08:00
Edd Robinson 91ee34b111 Merge pull request #7837 from influxdata/er-tidy
General tidy up and subtle bug fixes
2017-01-26 13:43:07 +00:00
Edd Robinson 63562d9fd5 Fixes #7860.
Fix a regression introduced in #7449.

This commit ensures that create database with retention policy will work
correctly.
2017-01-23 19:01:13 +00:00
Edd Robinson 4442cd7efa Fixes #7860.
Fix a regression introduced in #7449.

This commit ensures that create database with retention policy will work
correctly.
2017-01-23 17:57:32 +00:00
Edd Robinson 9d30ee0a6b Fix subtle bugs and remove dead code from services 2017-01-17 09:47:34 -08:00
Joe LeGasse 2db0250b22 Add db/rp name validation
This change adds some very basic name validation with the following
plain-english description: names must be non-zero sequence of printable
characters that do not contain slashes ('/' or '\') and are not equal to
either "." or "..".

The intent is that, since we currently just use database and retention
policy names directly as path elements, these rules will hopefully leave
us with names that should be at least close to valid directory names.

Ideally, we would restrict names even further or not use them as path
elements directly, but this should be a step towards the former without
restricting names "too much"
2017-01-12 17:38:10 -05:00
Jason Wilder ccdb68841b Merge pull request #7820 from influxdata/cjl-7689-sgi-unix-epoch-zero-time
fix marshaling of zero time
2017-01-12 12:34:48 -07:00
Cory LaNou c6dc164537
fix marshaling of zero time 2017-01-11 12:00:18 -06:00
Vladimir Lopes f05df2a263 Fix panic when pruning shard groups
* Fix #7812 - Panic when pruning shard groups

* Update CHANGELOG.md
2017-01-11 14:56:40 +00:00
Mark Rushakoff a135906b43 Merge pull request #7747 from influxdata/mr-lint-cleanup
Miscellaneous lint cleanup
2017-01-10 08:22:00 -08:00
Jonathan A. Sternberg d7c8c7ca4f Support subquery execution in the query language
This adds query syntax support for subqueries and adds support to the
query engine to execute queries on subqueries.

Subqueries act as a source for another query. It is the equivalent of
writing the results of a query to a temporary database, executing
a query on that temporary database, and then deleting the database
(except this is all performed in-memory).

The syntax is like this:

    SELECT sum(derivative) FROM (SELECT derivative(mean(value)) FROM cpu GROUP BY *)

This will execute derivative and then sum the result of those derivatives.
Another example:

    SELECT max(min) FROM (SELECT min(value) FROM cpu GROUP BY host)

This would let you find the maximum minimum value of each host.

There is complete freedom to mix subqueries with auxiliary fields. The only
caveat is that the following two queries:

    SELECT mean(value) FROM cpu
    SELECT mean(value) FROM (SELECT value FROM cpu)

Have different performance characteristics. The first will calculate
`mean(value)` at the shard level and will be faster, especially when it comes to
clustered setups. The second will process the mean at the top level and will not
include that optimization.
2017-01-07 13:00:48 -06:00
Mark Rushakoff 6a94d200c8 Merge remote-tracking branch 'influx/master' into mr-godoc 2017-01-04 13:27:36 -08:00
Mark Rushakoff 07b87f2630 Miscellaneous lint cleanup 2017-01-03 09:47:32 -08:00
Mark Rushakoff 959c445a88 Fix broken return statements swallowing errors
There was no comment on either case specifying that the `return nil`
was deliberate instead of `return err`, so I'm assuming these were
typos. I added tests to conserve the error-returning behavior.
2017-01-03 08:50:34 -08:00
Mark Rushakoff 218fc3890d Update godoc for services
The admin service was deliberately skipped due to it being deprecated.
2016-12-30 18:03:01 -08:00
Cory LaNou 572da8985c enforce minimum shard duration when creating retention policies 2016-12-20 09:11:43 -06:00
Jonathan A. Sternberg ec57108520 Use proper uber-go/zap import path
It looks like the real import path to the project is go.uber.org/zap
instead of github.com/uber-go/zap since the example in the project
references that path.
2016-12-15 08:54:14 -06:00
Jonathan A. Sternberg 21502a39e8 Switch logging to use structured logging everywhere
The logging library has been switched to use uber-go/zap. While the
logging has been changed to use structured logging, this commit does not
change any of the logging statements to take advantage of the new
structured log or new log levels. Those changes will come in future
commits.
2016-12-14 10:45:15 -06:00
Allen Petersen da9941b9a9 Reject invalid subscription urls
The url must have a scheme of udp,http,https and a port number.
CREATE SUBSCRIPTION will fail if there are invalid destinations.

Additionally Service.createSubscription fail invalid destinations are detected.

Fixes #7615
2016-12-07 12:54:41 -08:00
Edd Robinson aff2d7ce64 Ensure user privs dropped along with db (#7677) 2016-12-02 18:23:35 +00:00
Cory LaNou d54d32b9d4
prune shards in meta data 2016-12-01 11:22:16 -06:00