Commit Graph

111 Commits (2fe52d8edc124057c75658fd3238c4074feb6404)

Author SHA1 Message Date
Sam Arnold 956915296a
fix: use defer for unlock (#22929) 2021-11-24 14:43:46 -05:00
ricky c1e11e732e fix(meta): remove double index increment of set meta 2020-01-14 11:11:43 +08:00
Jeff Wendling e6aec771b0 fix(tsdb): attempt to work on docker on windows
multiple users have attempted to run influxdb in a docker container
with a windows host and a volume mounted from windows. that causes
problems because it apparently uses samba/cifs which does not
support fsync on directories. this patchset will, if it receives an EINVAL
on directory fsync, as is what appears to happen on samba/cifs, then it
will ignore it. this should help.

fixes #9833.
fixes #9630.
2018-06-01 14:57:18 -06:00
Stuart Carnie d135aecf02 Generate trace logs for a number of significant influx operations
* tsdb Store.Open traces all events related to opening files
    * op.name : tsdb.open
* retention policy shard deletions
    * op.name : retention.delete_check
* all TSM compaction strategies
    * op.name : tsm1.compact_group
* series file compactions
    * op.name : series_partition.compaction
* continuous query execution (if logging enabled)
    * op.name : continuous_querier.execute
* TSI log file compaction
    * op_name: index.tsi.compact_log_file
* TSI level compaction
    * op.name: index.tsi.compact_to_level
2018-02-21 15:08:49 -07:00
Jonathan A. Sternberg 2bbd96768d Update logging calls to take advantage of structured logging
Includes a style guide that details the basics of how to log.
2018-02-20 10:04:19 -06:00
Edd Robinson 6a66b5faf0 Cleanup services package 2018-01-21 10:52:37 -08:00
Andrew Hare d7e328050c
Merge branch 'master' into ah-truncate-shards 2017-11-28 17:25:31 -07:00
Jonathan A. Sternberg 0b7c56bcd8 Update the zap logger dependency
The previous sha was taken from a revision on a devel branch that I
thought would continue staying in the tree after it was merged. That
revision was rebased away and the API was changed for the logger.

This updates the usage of the logger and adds a simple package for
constructing the base logger.

The 1.0 version of zap changed the format of the default console logger
so this change moves over to this new logger instead of attempting to
retain backwards compatibility with the old format.
2017-11-10 16:27:16 -06:00
Stuart Carnie f3d45ba301 influxdata/influxdb/influxql -> influxdata/influxql 2017-10-30 14:40:26 -07:00
Andrew Hare e6aa5023eb Create a command to truncated shard groups 2017-10-16 20:34:26 -06:00
Joe LeGasse 815f740f4c initial fga work
wip

wip

fix tests / build
2017-05-26 13:16:27 -07:00
Edd Robinson 255992f5ec Reduce cost of admin user check
This commits adds a caching mechanism to the Data object, such that
when large numbers of users exist in the system, the cost of determining
if there is at least one admin user will be low.

To ensure that previously marshalled Data objects contain the correct
cached admin user value, we exhaustively determine if there is an admin
user present whenever we unmarshal a Data object.
2017-03-20 12:04:03 +00:00
Mark Rushakoff 601cbcd084 Merge branch '1.2' into mr-merge-12 2017-02-17 16:14:22 -08:00
Jonathan A. Sternberg 2fe48d6781 Rename zap import back to github.com/uber-go/zap
They rebased a revision we were previously relying upon that allowed us
to use the vanity name so we are reverting back to an older version with
the old import path.
2017-02-17 17:17:22 -06:00
Edd Robinson 91ee34b111 Merge pull request #7837 from influxdata/er-tidy
General tidy up and subtle bug fixes
2017-01-26 13:43:07 +00:00
Edd Robinson 63562d9fd5 Fixes #7860.
Fix a regression introduced in #7449.

This commit ensures that create database with retention policy will work
correctly.
2017-01-23 19:01:13 +00:00
Edd Robinson 4442cd7efa Fixes #7860.
Fix a regression introduced in #7449.

This commit ensures that create database with retention policy will work
correctly.
2017-01-23 17:57:32 +00:00
Edd Robinson 9d30ee0a6b Fix subtle bugs and remove dead code from services 2017-01-17 09:47:34 -08:00
Vladimir Lopes f05df2a263 Fix panic when pruning shard groups
* Fix #7812 - Panic when pruning shard groups

* Update CHANGELOG.md
2017-01-11 14:56:40 +00:00
Jonathan A. Sternberg d7c8c7ca4f Support subquery execution in the query language
This adds query syntax support for subqueries and adds support to the
query engine to execute queries on subqueries.

Subqueries act as a source for another query. It is the equivalent of
writing the results of a query to a temporary database, executing
a query on that temporary database, and then deleting the database
(except this is all performed in-memory).

The syntax is like this:

    SELECT sum(derivative) FROM (SELECT derivative(mean(value)) FROM cpu GROUP BY *)

This will execute derivative and then sum the result of those derivatives.
Another example:

    SELECT max(min) FROM (SELECT min(value) FROM cpu GROUP BY host)

This would let you find the maximum minimum value of each host.

There is complete freedom to mix subqueries with auxiliary fields. The only
caveat is that the following two queries:

    SELECT mean(value) FROM cpu
    SELECT mean(value) FROM (SELECT value FROM cpu)

Have different performance characteristics. The first will calculate
`mean(value)` at the shard level and will be faster, especially when it comes to
clustered setups. The second will process the mean at the top level and will not
include that optimization.
2017-01-07 13:00:48 -06:00
Mark Rushakoff 6a94d200c8 Merge remote-tracking branch 'influx/master' into mr-godoc 2017-01-04 13:27:36 -08:00
Mark Rushakoff 959c445a88 Fix broken return statements swallowing errors
There was no comment on either case specifying that the `return nil`
was deliberate instead of `return err`, so I'm assuming these were
typos. I added tests to conserve the error-returning behavior.
2017-01-03 08:50:34 -08:00
Mark Rushakoff 218fc3890d Update godoc for services
The admin service was deliberately skipped due to it being deprecated.
2016-12-30 18:03:01 -08:00
Jonathan A. Sternberg ec57108520 Use proper uber-go/zap import path
It looks like the real import path to the project is go.uber.org/zap
instead of github.com/uber-go/zap since the example in the project
references that path.
2016-12-15 08:54:14 -06:00
Jonathan A. Sternberg 21502a39e8 Switch logging to use structured logging everywhere
The logging library has been switched to use uber-go/zap. While the
logging has been changed to use structured logging, this commit does not
change any of the logging statements to take advantage of the new
structured log or new log levels. Those changes will come in future
commits.
2016-12-14 10:45:15 -06:00
Cory LaNou d54d32b9d4
prune shards in meta data 2016-12-01 11:22:16 -06:00
Cory LaNou 6e290040bb
remove SetDefaultRetentionPolicy method from meta.Client 2016-11-03 09:39:41 -05:00
Cory LaNou cd272ce6c3 fix retention policy creation inconsistencies 2016-11-03 09:09:43 -05:00
Jason Wilder bc00490b92 Don't clone the cached meta store
The cacheData is read-only and does not need to be cloned.  Cloning
it for read operations can be expensive with lots of shards/database/RPs/etc.
2016-09-28 10:35:59 -06:00
Jonathan A. Sternberg 23f2d50ecb Use defaults from `meta` package for `CREATE DATABASE`
Instead of having the parser set the defaults, the command will set the
defaults so that the constants for that are actually used. This way we
can also identify which things the user provided and which ones we are
filling with default values.

This allows the meta client to be able to make smarter decisions when
determining if the user requested a conflict or if the requested
capabilities match with what is currently available. If you just say
`CREATE DATABASE WITH NAME myrp`, the user doesn't really care what the
duration of the retention policy is and just wants to use the default.
Now, we can use that information to determine if an existing retention
policy would conflict with what the user requested rather than returning
an error if a default value ever gets changed since the meta client
command can communicate intent more easily.
2016-08-30 13:23:49 -05:00
Jonathan A. Sternberg c05c7f6360 Revert "limit shard concurrency"
This reverts commit 6c7d56d4bc.
2016-08-29 12:39:52 -05:00
Ben Johnson 6c7d56d4bc
limit shard concurrency
This commit limits queries to only process one shard at a time.
However, within a shard, multiple series can still be processed in
parallel. Shard iterators are lazily instantiated during query
execution to limit the amount of memory a given query uses.
2016-08-05 09:45:57 -06:00
David Norton e3328117a9 fix #7081: hardcode auto gen RP name 2016-07-27 16:49:46 -04:00
David Norton b1079c9080 revert previous DefaultRetentionPolicyName change 2016-07-27 11:38:42 -04:00
David Norton 8fbfdc5616 fix #7068: use DefaultRetentionPolicyName from cfg
Use DefaultRetentionPolicyName from the config instead of passing
through meta data.
2016-07-26 18:53:55 -04:00
Jonathan A. Sternberg b8e52ce39a Merge pull request #6889 from influxdata/js-update-config-options
Update help and remove unused config options from the configuration file
2016-06-23 11:49:10 -05:00
kun 8aa2fe8ede fix ClusterID generation bug 2016-06-23 14:52:58 +08:00
Jonathan A. Sternberg 22173acb70 Update help and remove unused config options from the configuration file
Normalize the output for the various help options so they all follow the
same format and display all relevant options.

Removing some of the unused config options from the configuration file
and updating the help documentation. Removing some remaining references
to clustering within the open source version.
2016-06-21 14:06:05 -05:00
Jonathan A. Sternberg 497db2a6d3 Removing dead code from every package except influxql
The tsdb package had a substantial amount of dead code related to the
old query engine still in there. It is no longer used, so it was removed
since it was left unmaintained. There is likely still more code that is
the same, but wasn't found as part of this code cleanup.

influxql has dead code show up because of the code generation so it is
not included in this pruning.
2016-06-20 22:41:07 -05:00
Jonathan A. Sternberg b8e22d9d79 Merge pull request #6586 from influxdata/js-3733-rename-default-retention-policy
Modify the default retention policy name and make it configurable
2016-06-06 15:05:29 -05:00
Rubycut a849f42cc2 Lock auth properly, code suggested by @jwilder 2016-06-04 09:05:10 +02:00
Jonathan A. Sternberg baaa782c95 Modify the default retention policy name and make it configurable
The default retention policy name is changed to "autogen" instead of
"default" since it ends up being ambiguous when we tell a user to check
the default retention policy, it is uncertain if we are referring to the
default retention policy (which can be changed) or the retention policy
with the name "default".

Now the automatically generated retention policy name is "autogen".

The default retention policy is now also configurable through the
configuration file so an administrator can customize what they think
should be the default.

Fixes #3733.
2016-05-24 09:51:23 -04:00
Jason Wilder adddb0e6b8 Reduce lock contention when creating shard group 2016-05-09 16:10:38 -06:00
Seif Lotfy ced79acac5 Remove redundant error return from (Client) Database(name string) and (Client) Databases() 2016-04-30 17:04:38 -05:00
Stephen Gutekanst 9dc09c5257 Make logging output location more programmatically configurable (#6213)
This has various benefits:

- Users embedding InfluxDB within other Go programs can specify a different logger / prefix easily.
- More consistent with code used elsewhere in InfluxDB (e.g. services, other `run.Server.*` fields, etc).
- This is also more efficient, because it means `executeQuery` no longer allocates a single `*log.Logger` each time it is called.
2016-04-20 21:07:08 +01:00
Chris Ramón a4dd61a2c7 adds shard duration normalization before comparing retention policies 2016-04-13 13:09:20 -05:00
Cory LaNou 7eaf7f8855 CreateShardGroup was incrementing meta data index even when it was idempotent 2016-04-07 11:39:45 -05:00
mvadu 94524a7699 Fix failure on Windows
Windows does not allow a file to be renamed while its in use. f.Sync does not close the handle. It needs to be explicitly closed before renaming.
2016-04-06 07:32:37 +05:30
Cory LaNou 9bd60c3e07 remove node.json as well, change where we check for raft.db in startup 2016-03-31 17:15:58 -05:00
Cory LaNou 6c6fe87a40 add error message if detecting raft.db file 2016-03-31 17:15:42 -05:00