Commit Graph

1263 Commits (627cd9d486b98af01f02f0915319185a81b46c87)

Author SHA1 Message Date
Paul Dix c9d82ad0ad Wire up meta service functionality
* Add dir, hostname, and bind address to top level config since it applies to services other than meta
* Add enabled flags to example toml for data and meta services
* Wire up add/remove raft peers and meta servers to meta service
* Update DROP SERVER to be either DROP META SERVER or DROP DATA SERVER
* Bring over statement executor from old meta package
* Start meta service client implementation
* Update meta service test to use the client
* Wire up node ID/meta server storage information
2016-01-21 15:28:32 -05:00
Cory LaNou 9ec7a710c9 some misc refactoring on influxd startup 2016-01-21 15:28:32 -05:00
Cory LaNou 8d878fff91 buildable meta -> services/meta 2016-01-21 15:28:32 -05:00
Jason Wilder b51c6a935f Improve backup/restore logging and help commands 2016-01-20 13:46:14 -07:00
Joe LeGasse 8791011b10 Validate metadata backup blob
Fixes #5349
2016-01-19 10:30:08 -08:00
Joe LeGasse dfc59e8a13 Ensure the backup directory exists.
Fixes #5350
2016-01-15 14:38:33 -08:00
Philip O'Toole 82e16a1899 Merge pull request #5309 from influxdata/c_usage
Unit test scrubbing values during conversion
2016-01-07 14:59:39 -08:00
Philip O'Toole 653dba3759 Merge pull request #5129 from e-dard/fix-precision-flag
Ensure precision flag is respected in CLI
2016-01-07 14:48:57 -08:00
Jason Wilder 16829dedc8 Merge pull request #5308 from influxdata/jw-compact-large
Use block size and file size before skipping a file during compactions
2016-01-07 15:48:03 -07:00
Philip O'Toole cd2de5f25d Unit test scrubbing values during conversion 2016-01-07 14:33:42 -08:00
Edd Robinson c6d32bd1bb Fix #5127 2016-01-07 22:30:46 +00:00
Philip O'Toole 2d2c315356 Dump points per TSM point written 2016-01-07 13:58:23 -08:00
Jason Wilder 0264d77545 Fix block sizes reported by influx_inspect 2016-01-07 14:51:59 -07:00
Philip O'Toole 13d2b69c29 Dump conversion factor on completion 2016-01-07 13:50:56 -08:00
Philip O'Toole bab2a00e5d Track bytes written by conversion tool 2016-01-07 13:50:45 -08:00
Philip O'Toole 9b02808e6c ShardInfos now returns size 2016-01-07 13:36:25 -08:00
Philip O'Toole 067cc71923 Count number of fieldless points filtered 2016-01-06 11:52:42 -08:00
Philip O'Toole ff96e17ddb Skip points without fields 2016-01-06 11:45:20 -08:00
Philip O'Toole 9c916b0b76 Best-effort cleanup of converter. 2016-01-05 19:27:27 -08:00
Philip O'Toole 2a547b0db3 Increment sequence, not generation
Avoid having all the new files getting picked up by the compaction
planner on startup.
2016-01-05 19:27:22 -08:00
Philip O'Toole fbb3e861ca Clearer database backup message 2016-01-05 17:51:52 -08:00
Philip O'Toole 075ef45ae1 Gather conversion stats 2016-01-05 16:55:22 -08:00
Philip O'Toole cac96113c0 Merge pull request #5278 from influxdata/c_fixes
Skip bz1 bolt files without any points
2016-01-05 19:09:49 -05:00
Jason Wilder 90292dd429 Merge pull request #5279 from influxdata/jw-compaction-memory
Reduce allocations during TSM compactions
2016-01-05 16:50:18 -07:00
Philip O'Toole 140f54a01d Skip bolt files without any points 2016-01-05 15:19:56 -08:00
Jason Wilder dd90824eb5 Fix go vet in restore.go 2016-01-05 09:37:44 -07:00
Philip O'Toole d53674e2cc Add note about new tsm1 directory permissions
[ci skip]
2016-01-04 15:23:05 -08:00
Philip O'Toole ed10978b71 Increase default max TSM file size 2016-01-04 13:58:43 -08:00
Philip O'Toole 250c10f126 Tweak influx_tsm help output 2016-01-04 13:58:12 -08:00
Philip O'Toole 8212bc82b9 Correct typo in influx_tsm help 2016-01-04 13:55:36 -08:00
Jonathan A. Sternberg c825ff7bae Merge pull request #5203 from influxdb/js-fix-use-test-panic
Add a mock client to the cli test for Use
2016-01-04 13:54:03 -05:00
Paul Dix ee233c849a Merge pull request #5224 from influxdb/pd-backup-restore
Implement backup/restore for TSM.
2015-12-31 08:56:12 -05:00
Paul Dix 49d480cb0c Fix races in backup/restore 2015-12-31 08:42:01 -05:00
Paul Dix 9cede5fb71 Address PR comments 2015-12-30 18:06:51 -05:00
Paul Dix 26e1c6464a Update backup to address PR comments 2015-12-30 18:06:51 -05:00
Paul Dix 59fbd371fc Implement backup/restore for TSM.
This changes backup and restore to work for TSM. It breaks it for b1 and bz1, but since those are getting removed it's ok.

The backup runs against any host that is specified and can backup either the metasstore, a database, specific retention policy, or a specific shard. It can also take incremental backups with the `since` flag, which will only backup TSM files that have been created since that timestamp.

The backup is safe to run online. However, for shards that are still hot for writes, they won't be able to create new TSM files while the backup for that single shard runs. If the backup isn't too large and the write throughput isn't too high this shouldn't be a problem since the writes will just go into the WAL cache.
2015-12-30 18:06:50 -05:00
Michael Desa bf1673f466 Merge pull request #5239 from influxdb/md-add-db
Add flag to specify db and clarify flag descriptions
2015-12-30 10:59:31 -08:00
Michael Desa 7c025d8497 Change db flag message 2015-12-29 13:12:05 -08:00
Philip O'Toole eaec514ca0 b*1 to tsm1 shard converter 2015-12-29 15:31:07 -05:00
Jonathan A. Sternberg 6b546cb766 Remove calls of os.Exit from influx cli Run method and fix influx tests
One of the first unit tests in the cli tests called the Run method.
Since the Run method called os.Exit, it reported the unit tests as
succeeded. When parallel is set to 1, this skips _all_ unit tests after
the first one. When parallel is set to a higher value, unit tests run by
other processes still get run.

This changes the Run method to return an error (if one occurred). This
error can then be printed out and a bad exit status can be used to exit
the program from the main program instead.  That causes the unit tests
to run correctly regardless of how many parallel processes are running.

Also added an additional option to the CLI called `IgnoreSignals`. If
this is set to true, then signals are not registered with the process.
Setting signals doesn't really work in unit tests so it's good to ensure
they don't get set in the first place.

In addition to fixing the influx cli tests, this adds a mock client to
the cli test for Use. PR #5183 added a validation for `use` to only be
able to select public databases so `_internal` couldn't be chosen. To
implement this, the `SHOW DATABASES` command was used by the internal
client.

Some of the unit tests in `cli_test.go` don't set the client to
anything. `TestParseCommand_Use` previously didn't, but now it needs to
have a client in the unit test with an empty test server.
2015-12-29 14:58:54 -05:00
Michael Desa ebd9b9978e Add flag to specify db and clarify flag descriptions 2015-12-29 11:31:28 -08:00
Mike Glazer c049ebfa43 Support connecting to InfluxDB with the client without HTTPS Verification
The V2 client code supports this, however, the V1 client code didn't,
and the argument to support this just had to be added to the CLI
2015-12-29 09:03:16 -08:00
Jason Wilder a38c95ec85 Update compactions to run concurrently
This has a few changes in it (unfortuantely).  The main change is to run compactions
concurrently.  While implementing this, a few query and performance bugs showed up that
are also fixed by this commit.
2015-12-23 18:01:11 -07:00
Philip O'Toole 767a7f5131 'broker' and 'data' are obsolete 2015-12-23 15:27:47 -05:00
Philip O'Toole 2a101dc379 Add profiling flags to help output 2015-12-23 15:22:50 -05:00
Sebastian Borza 26bc778b40 adding quit and Ctrl-D support to influx cli
update comment based on feedback
2015-12-23 11:54:41 -05:00
Paulo Pires 3165c707fc Small improvements to USE command as per PR comments. 2015-12-22 08:55:32 +00:00
Paulo Pires 3f4949019d USE databasename will work only for existing databases. Fixes #5174 2015-12-21 09:34:40 +00:00
Jason Wilder 4b93f6c72d Merge pull request #5142 from influxdb/jw-tsm-migrate
Optimize TSM compaction
2015-12-18 10:17:34 -07:00
nick.grange 99f90770da Fixed code format using go fmt. 2015-12-18 00:28:58 +11:00
nick.grange 33de878956 Fixed #5078 to support Inserts in non-interactive mode.
Changed non-interactive mode to send everything through the CLI's parser the same way the interactive mode works.
Added multiline support for -execute flag.
2015-12-17 23:42:30 +11:00
Michael Desa ff12e83e7f Merge pull request #5148 from influxdb/jz-influx-stress-documentation
Add README.md for influx_stress
2015-12-16 17:16:22 -08:00
Jack Zampolin a507a8dd8e Add README.md for influx_stress 2015-12-16 16:20:37 -08:00
Jason Wilder 93d93797da Fix snappy decode error w/ influx_inspect 2015-12-16 11:28:13 -07:00
dgnorton d89e233567 Merge pull request #5100 from influxdb/dgn-fix-4303
fix #4303: don't drop from multiple databases
2015-12-15 07:27:05 -05:00
Jack Zampolin 7d0d9b75d9 Add backoff to retry function
Set backoff constant

Wait to parse flags

Remove unnecessary logging

Fix retry logic
2015-12-14 13:46:40 -08:00
David Norton 69f22803d2 fix #4303: duplicate url.Values 2015-12-14 12:28:01 -05:00
David Norton 3014fb90e4 fix #4303: don't drop from multiple databases 2015-12-12 13:54:23 -05:00
Tim Raymond 718fb92cf5 Remove registration, add 'usage-client'
Server registration and stats reporting has been removed from what was
once http://enterprise.influxdata.com. The app that lived there, now
runs at http://usage.influxdata.com, so that the subdomain can
eventually be repurposed. Because we also want to repurpose the
`enterprise-client` repo, we have also renamed that to `usage-client`.
InfluxDB no longer needs the `registration` service now, since all of
the endpoints it communicates with simply discard the data provided to
them.
2015-12-10 15:58:34 -05:00
Philip O'Toole def5e06150 Merge pull request #5064 from mengjinglei/fix-parens-math
fix issue #5054
2015-12-10 08:44:48 -08:00
mengjingleli 149b93fd4b add a test case 2015-12-10 13:51:43 +08:00
Nathaniel Cook b7000c80dd count with fill(none) will drop 0 valued intervals 2015-12-09 15:20:47 -07:00
mengjingleli 3a901abbd8 fix issue #5054 2015-12-09 15:27:33 +08:00
Michael Desa 59169dcdc4 Implement WriteHTTPHandler 2015-12-08 16:09:12 -08:00
Michael Desa 364a699416 Add broadcast channel for stress 2015-12-08 14:29:22 -08:00
Paul Dix 40e606cb14 Merge pull request #5003 from influxdb/jw-compaction
Update compaction planning
2015-12-05 16:49:54 -05:00
Jason Wilder 41b24995a7 Compcation fixes 2015-12-05 12:19:28 -07:00
Michael Desa 14b5ed3001 Merge pull request #4961 from influxdb/md-stress-cluster
Add cluster support to stress test
2015-12-04 15:20:31 -08:00
Philip O'Toole 4f69936564 Add more math unit tests to server-level tests 2015-12-04 11:23:35 -08:00
Jason Wilder c54a3da0ca Implement delete series/measurement 2015-12-04 09:10:26 -07:00
Cory LaNou 477110f0b7 add option to disable logging for meta service. useful for testing 2015-12-03 11:11:36 -06:00
Cory LaNou 0c2588ed43 cluster retention auto create test 2015-12-03 08:08:57 -06:00
Cory LaNou 52da1c76d6 cluster retention policy tests 2015-12-03 08:08:56 -06:00
Cory LaNou 96c230d968 allow for configurable cluster creation 2015-12-03 08:08:56 -06:00
Cory LaNou ab57bde115 more tests, some refactoring 2015-12-03 08:08:56 -06:00
Cory LaNou 91da96eb34 fix race in test harness 2015-12-03 08:08:56 -06:00
Cory LaNou 6e845839c8 fix flakey test harness 2015-12-03 08:08:56 -06:00
Cory LaNou 967a53cabd start sharing integration tests for cluster tests 2015-12-03 08:08:56 -06:00
David Norton 3208564192 fix RP not found error messages 2015-12-03 08:04:09 -06:00
David Norton 343f00d1f2 fix go vet errors 2015-12-03 08:04:09 -06:00
David Norton 76d291e88e beginning of cluster test harness 2015-12-03 08:04:09 -06:00
Michael Desa f9fa366861 Add querying across cluster
Add support for precision
2015-12-02 14:49:23 -08:00
Michael Desa b0f27beb12 Add support for writing to a cluster 2015-11-30 10:22:12 -08:00
Paulo Pires 8e2868ed60 Added support for setting retention policy on database creation.
Fixes #2676
2015-11-27 20:02:55 +00:00
Jason Wilder a6541937f8 Add dumptsmdev to influx_inspect
Allow inspecting the updated TSM format.
2015-11-24 08:50:13 -07:00
Mark Bates 98632778dd %s will convert a float to a less than useful string. %v is better. 2015-11-20 15:43:35 -05:00
Mark Bates c71e323d63 ServerID and ClusterID need to be strings when being sent to Enterprise. 2015-11-20 15:22:27 -05:00
Philip O'Toole 1e9e7c2d94 Merge pull request #4807 from flisky/master
FIX #4279 CLI support multiline
2015-11-18 19:03:34 -08:00
Jason Wilder c4a6490d32 Update influx_inspect to use new DecodeBlock interface 2015-11-17 23:24:09 -07:00
Michael Desa 9a5120b2a3 Merge pull request #4742 from influxdb/md-new-stress
WIP: Refactor Stress tool
2015-11-17 14:42:51 -08:00
Michael Desa eb0e760763 Update stress config examples 2015-11-17 14:36:10 -08:00
Michael Desa b86c0d31d7 Refactor stress tool
Add StressTest type and auxillary interfaces

Add config structs

Move generator to config

Add utility methods used in stress

Add basic components for a stress test

Add touches

Add configuration options

Add unified results handlers

Add final print out of results

Add Success function to response type

Add query support

Send query results

Add comments to run.go

Change Basic to BasicWriter

Add basic query

Add incomplete README

Abstract out response handling

Change plugin to basic

Add responseHandler type

Add additional parameter to Query function

Add todo comments and cleanup main

Lower hard coded value

Add flag for profiling

Fix race condition

Wait at the right place

Chane point from struct to interface

Improve generic write throughput

Reorganize

Fastest State

Add toml config

Add test server

Add basic working version of config file

Move config file logic into its own file

Fix broken config file

Add query count to stress config

Add support for concurrency and batch interval

Reorder config option

Remove unneeded init

Remove old stress package

Move new stress code into stress directory

Rework influx_stress tool

Do something reasonable if no config is given

Remove unneeded comments

Add tests for stress package

Add comments and reorganize code

Add more comments

Count lines posted correctly

Add NewConfig method

Fix style issues

Add backticks to flag description

Fix grammar

Remove `StartTimer` calls where appropriate

Fix comment language

Change Reader to Querier

Reorder defer

Fix issues bought up by golint

Add more comments

Add more detailed Readme

Increase counter appropriately

Add return errors where appropriate

Add test coverage

Move `now()` from QueryClient to QueryGenerator
2015-11-17 12:16:31 -08:00
Cory LaNou 6309ec9a68 fix time inclusive comparison 2015-11-17 10:43:27 -06:00
Cory LaNou 824d7a1d9b Add interface for heap to support Reverse for `order by desc` 2015-11-16 15:10:55 -06:00
Yin Jifeng 2f3fc70514 FIX #4279 CLI support multiline 2015-11-16 22:59:21 +08:00
Cory LaNou bb00645a84 Merge pull request #4768 from pires/4719-cli_history_refactor
CLI history skips blank lines.
2015-11-14 07:25:58 -06:00
Paulo Pires a8fa170f29 Removed bogus comment. 2015-11-14 11:12:31 +00:00
Philip O'Toole 912684b8d6 WHERE fields must be decoded during aggregates
This change ensures that if there are any fields in the WHERE clause of
an aggregate that are different from the fields in the SELECT clause,
that the cursors also decode those fields. Otherwise WHERE clauses of
the form 'SELECT f(w) FROM x WHERE y=z' will return incorrect results

Fixes issue #4701.
2015-11-13 15:18:54 -08:00
Philip O'Toole 172cc43732 Merge pull request #4766 from aneshas/cli-help-contrib
Changed help command output from inside the cli
2015-11-13 11:17:18 -08:00
Paulo Pires aa6ec0cc6f Added more tests to increase coverage. Refs #2313 2015-11-13 11:38:35 +00:00