Commit Graph

406 Commits (4d377b2fb099eb00a5bb99e80bccc97484dd996d)

Author SHA1 Message Date
Mark Rushakoff 43b7617316 test(task): check task status in more methods
This TODO was meant to be removed as part #1179, but was mistakenly
overlooked.
2018-10-24 11:04:07 -07:00
Mark Rushakoff 75c5d91cec feat(task): include task status when listing tasks 2018-10-24 09:38:02 -07:00
Adam 98be8bcd32
moved to_http and to_kafka to flux and update to latest flux (#1175) 2018-10-24 12:33:43 -04:00
Lyon Hill 19b7709bad
Add tasks to the influx cli (#1163) 2018-10-23 11:51:13 -06:00
Mark Rushakoff 8c0aec7975 fix(task): propagate status updates when modifying tasks
This also changes the backend.Store API to remove the EnableTask and
DisableTask methods, merging their functionality into ModifyTask, which
has been named to UpdateTask to keep closer to the CRUD acronym.
2018-10-19 16:49:18 -07:00
Mark Rushakoff 3cf88009c4 test(task): fix incorrect test assertion
We changed the semantics of finding a task that doesn't exist, from
returning `nil, nil`, to `nil, ErrTaskNotFound`; this test didn't
reflect that change until this commit.
2018-10-15 10:24:23 -07:00
Jonathan A. Sternberg 5c8318d181
Merge pull request #1080 from influxdata/js-go-generate-with-go-run
refactor: modify tooling to take advantage of go run
2018-10-15 11:27:13 -05:00
Mark Rushakoff ad4942fe36 refactor(task): expose status when finding from TaskService
And consistently return ErrTaskNotFound, rather than nil, nil, when
searching an ID that doesn't exist.
2018-10-15 09:20:47 -07:00
Mark Rushakoff cb0b54cfd9 feat(task): allow setting task status during creation
This renames TaskEnabled and TaskDisabled to TaskActive and
TaskInactive, to keep in line with the swagger and other parts of the
system. But I left the EnableTask and DisableTask methods on the Store
interface for now. I could see eliminating those methods if we adjust
the signature of the UpdateTask method.
2018-10-15 09:20:47 -07:00
Mark Rushakoff 56f32b4780 refactor(task): extract CreateTaskRequest struct
We're about to add another parameter, so condensing the arguments into a
single struct makes sense at this point.
2018-10-15 09:20:47 -07:00
Jonathan A. Sternberg e9600b1f0b
refactor: modify tooling to take advantage of go run
The generate commands have been modified to take advantage of the new
functionality in Go 1.11 that allows `go run` to execute a package
instead of individual files.

This functionality combined with Go modules allows us to execute a
package directly out of our pinned dependencies rather than accidentally
picking up another binary outside of the build environment.

This also simplifies the Makefile because they no longer have to be
responsible for installing the correct tooling since the Go command
takes care of that logic. It also makes it so that the Makefiles with
file generation can now be invoked from their appropriate subdirectories
so they are contained within the directory itself rather than relying on
values in the top level Makefile.

It is now possible to generate all files within this project by using:

    go generate ./...

Or the Makefile can continue to be used.

This commit also copies over the special copy of `tmpl` that the storage
engine uses within the influxdb repository. It was never copied over so
using `go generate` on these packages did not work.
2018-10-15 11:01:27 -05:00
Lyon Hill e643c434f6
Allow the scheduler to Update tasks. (#1058)
* Allow the scheduler to Update tasks.

* update the coordinator to use the new update action

* remove unclean code
2018-10-12 08:24:22 -06:00
Chris Goller 2bea48f15a refactor(testing): rename MustIDFromString to MustIDBase16 2018-10-11 09:54:19 -05:00
Chris Goller ada7632f98 fix(task/backend): remove extra name by org 2018-10-11 09:54:18 -05:00
Leonardo Di Donato 6725ae6f64 feat(platform/id): method to create an invalid id
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
2018-10-11 09:54:18 -05:00
Leonardo Di Donato 016c7d04cb fix(task/servicetest): porting servicetest to uint64 IDs
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-Authored-By: Lorenzo Fontana <lo@linux.com>
2018-10-11 09:54:18 -05:00
Leonardo Di Donato bfee4df046 fix(task/backend/bolt): using uint64 IDs
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-Authored-By: Lorenzo Fontana <lo@linux.com>
2018-10-11 09:54:18 -05:00
Leonardo Di Donato 5d4d8a7c04 fix(platform/task/backend/executor): porting tests to uint64 IDs 2018-10-11 09:54:18 -05:00
Leonardo Di Donato 5db4276a71 fix(platform/id): improvements on code using uint64 IDs
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-Authored-By: Lorenzo Fontana <lo@linux.com>
2018-10-11 09:54:18 -05:00
Leonardo Di Donato 6587d1b13d fix(task/backend/coordinator): porting test to uint64 IDs 2018-10-11 09:54:18 -05:00
Leonardo Di Donato 0a75951684 fix(task/backend/storetest): porting test functions to uint64 IDs 2018-10-11 09:54:18 -05:00
Leonardo Di Donato ca7ba33d27 fix(task/mock): porting to uint64 IDs 2018-10-11 09:54:18 -05:00
Leonardo Di Donato 9104771408 fix(task/backend): porting to uint64 IDs
Co-Authored-By: Lorenzo Fontana <lo@linux.com>
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
2018-10-11 09:54:18 -05:00
Leonardo Di Donato f5dfe0aecc fix(task/backend): create task returns a platform.ID + inmem store
refactoring

Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-Authored-by: Lorenzo Fontana <lo@linux.com>
2018-10-11 09:54:18 -05:00
Leonardo Di Donato 8d7f06cf4b feat(platform/id): using uint64 for platform.IDs
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-Authored-by: Lorenzo Fontana <lo@linux.com>
2018-10-11 09:54:18 -05:00
Jonathan A. Sternberg e7e17d6972
refactor: move the logger package from influxdb to platform
We are moving the necessary code for 2.0 from the influxdb 1.X
repository to the platform 2.0 repository. The logger is an unnecessary
dependency on the old influxdb that is making life more complicated.
2018-10-10 15:49:07 -05:00
Lyon Hill 610faf18e7
Update the scheduler to add a few task logs (#1029) 2018-10-10 12:37:11 -06:00
Mark Rushakoff 35383074f3 chore(task): remove task name uniqueness requirement 2018-10-09 11:46:12 -07:00
Lyon Hill f081ce95f0
allow run shows to work with an actual org id. (#981)
* allow run shows to work with an actual org id.
also necessary for using actual run lists
2018-10-08 14:07:08 -06:00
Brandon Farmer c1c4dc3393 Properly handle spaces in PATH with make tasks 2018-10-08 09:53:22 -07:00
Michael Desa 42ba7ffe11 feat(http): add authentication handler middleware 2018-10-02 14:11:44 -04:00
Michael Desa 1a0c29ad34 feat(platform): add authorizer interface
This iterface is supposed to be something that both sessions and
authorizations can share so that other components can authorize requests
as they see fit.
2018-10-02 14:11:44 -04:00
docmerlin (j. Emrys Landivar) 25e965d7ae scheduler->TickScheduler 2018-10-01 14:59:02 -05:00
docmerlin (j. Emrys Landivar) 4502b4a47f add requestedAt to task log 2018-09-27 12:46:57 -05:00
Mark Rushakoff 1524768aed test(task): add test for reusing task names after deletion 2018-09-26 14:34:05 -07:00
Lyon Hill 7ff3361e78
fix(task): Allow task scheduler to be stopped and started. (#882) 2018-09-26 00:01:21 -06:00
Chris Goller a64b86399f chore(task/backend): regenerate meta.proto with protc 3.6.1 2018-09-25 22:18:36 -05:00
Chris Goller f2132168b1 fix(task/backend): copy gogo.proto into repo 2018-09-25 22:18:00 -05:00
Mark Rushakoff fe4b676d06 fix(task/backend): avoid data race in in-memory run log
Return reference to new Run instances from ListRuns, to avoid a data
race on the original element.

Fixes #842.
2018-09-17 09:51:14 -07:00
Lyon Hill c117f0d8ca
fix(task): fix a potential race condition in meta (#836) 2018-09-15 01:29:05 -06:00
Lyon Hill eb260436d6
fix(task): remove race condition when finishing a inmem run (#829)
* fix(task): remove race condition when finishing a inmem run
2018-09-13 10:11:59 -06:00
Lyon Hill 8cc55b3a37
fix(task): race condition in creating and finding meta data fixed (#828) 2018-09-13 09:11:25 -06:00
Nathaniel Cook 672e2d5fe7 chore: Updates to be able to remove platform as a dependency of Flux 2018-09-12 10:18:54 -06:00
Lyon Hill 54cb6dfaa9
fix(task): allow schedulers to recover in progress tasks. (#810)
* fix(task): allow schedulers to recover in progress tasks.
2018-09-11 09:32:59 -06:00
Mark Rushakoff 7bbd087980 chore(task): clean up log writer API
Introduce a RunLogBase type that encapsulates the base information
shared across multiple logs for the same run. This has the information
previously part of the API (Task and RunID), and includes the addition
of a RunScheduledFor timestamp.

Also remove the RunScheduled value for RunStatus, as it was never used
in the scheduler.
2018-09-07 09:35:55 -07:00
Stuart Carnie ac75af2f58 refactor: Migrate query package to influxdata/flux repository 2018-09-06 11:13:48 -07:00
Lyon Hill 5bcc9063d7
feat(taskd): create a FindTaskByIDWithMeta function. (#781)
Now we can retrieve both task data as well as meta in one call.
2018-09-06 11:40:32 -06:00
Mark Rushakoff bbcbaee9fd chore(task): rename adaptertest to servicetest
This package assists with testing platform.TaskService. The fact that we
use task.PlatformAdapter is just a detail.

This is just a rename, plus one change to support caller-defined
construction of a TaskService (necessary for porting the tests to
enterprise).
2018-09-05 14:38:45 -07:00
Mark Rushakoff 9a469fd8ad chore(task): log on tick
This will help give operators a better idea that the system is not
actually idle.
2018-09-05 12:06:58 -07:00
Lyon Hill deb25f5c8d
fix(taskd): Stop panic in options (#774)
* fix(taskd): Stop panic in options

When type asserting a value we need to ensure it is the type we expect
before we assert the value.
2018-09-05 11:31:11 -06:00
Mark Rushakoff 479f3dbb3d test(task): add platform adapter tests
Also fix a handful of segfaults caused by improperly retained byte slices
from bolt, and combine two view transactions into one to avoid  data
race when a delete happened between them.
2018-09-05 10:06:55 -07:00
Lyon Hill 5be44d7fac
fix(taskd): Make WithTicker not block. (#765)
A mistake in the WithTicker method caused it to block.
This was not evident because until recently we used a cron styler timer.
Removing the block should allow the task deamon to start up properly.
2018-09-04 10:16:40 -06:00
Nathaniel Cook 46a32e874c chore: Update 'from' to match spec
Removes the `db` parameter and used only bucket and bucketID.
2018-08-30 14:51:51 -06:00
Mark Rushakoff d4753e2ed6 chore(platform): remove Last Run field from Task
Closes #626.
2018-08-30 11:02:17 -07:00
Chris Goller 8fa72fa35b feat(auth): allow authorizations to be enabled/disabled 2018-08-29 13:07:38 -05:00
Mark Rushakoff a8018532b2 test(task): fix two flaky tests
In TestScheduler_RunLog, the run log gets updated asynchronously so we
need to poll to ensure we don't read a stale value.

In TestScheduler_CreateNextRunOnTick, we were previously reading the
one entry out of two from a map, so it was often the entry we weren't
interested in.
2018-08-29 10:02:45 -07:00
Mark Rushakoff caa4539d0e fix(task): fix data race in scheduler 2018-08-28 08:11:16 -07:00
Mark Rushakoff f72ed76a47 feat(task): handle manual runs in scheduler 2018-08-26 20:34:05 -07:00
Mark Rushakoff ac03771c62 feat(task): handle manual runs in StoreTaskMeta.CreateNextRun 2018-08-26 20:34:05 -07:00
Mark Rushakoff 4573c16367 feat(task): add ManuallyRunTimeRange to Store 2018-08-26 20:34:05 -07:00
Lyon Hill 816a11c9a0
refactor(task): add org id to log reader calls (#684)
* refactor(task): add org id to log reader calls
2018-08-23 12:30:59 -06:00
Mark Rushakoff 813ba2f221 chore(task): rename StoreTaskMeta.LastCompleted to LatestCompleted
LastCompleted implies they could be out of order, so LatestCompleted is
more accurate.
2018-08-22 10:36:03 -07:00
Lyon Hill f718b9cdca
feat(task): create validation layer for TaskService (#591)
* feat(task): create validation layer for TaskService
2018-08-20 15:15:04 -06:00
Mark Rushakoff 9149c6508d hotfix(task): skip long test on Jenkins or with -short 2018-08-17 14:34:13 -07:00
Mark Rushakoff 66e97c938d pr: address feedback 2018-08-17 13:45:19 -07:00
Mark Rushakoff de577125a8 refactor(task): remove CreateRun method from Store
This method has been superseded by CreateNextRun.
2018-08-17 11:49:09 -07:00
Mark Rushakoff 11fe3acf05 refactor(task): use CreateNextRun in scheduler 2018-08-17 11:08:23 -07:00
Mark Rushakoff 64ad4bd1c2 refactor(task): add CreateNextRun method to Store interface
CreateNextRun centralizes StoreTaskMeta modification and therefore
simplifies Store functions a bit.

Store.CreateNextRun isn't called anywhere yet. The next step here is to
use CreateNextRun from the scheduler and remove Store.CreateRun.
2018-08-17 11:08:23 -07:00
Mark Rushakoff 05a930ecb3 refactor(task): add CreateNextRun method to StoreTaskMeta 2018-08-17 11:08:23 -07:00
Mark Rushakoff cd9b9d65a4 refactor(task): set Delay on StoreTaskMeta 2018-08-17 11:08:23 -07:00
Mark Rushakoff c2c4de05b0 chore(task): update StoreTaskMeta for manually requested runs 2018-08-17 11:08:23 -07:00
Mark Rushakoff d0cd454a55 refactor(task): add EffectiveCronString method to Options
The logic used to be only in the scheduler, but now Options exposes it.
This is another step towards making StoreTaskMeta schedule-aware to
enable creating manually requested runs.
2018-08-15 16:15:36 -07:00
Mark Rushakoff 769791d366 refactor(task): add FinishRun method to StoreTaskMeta
This logic was repeated in multiple places, so now it can live in a
single place. This also sets a precedent going forward, for adding
methods to the StoreTaskMeta type. We'll likely be adding more methods
to support manual runs of a task.
2018-08-15 09:45:25 -07:00
Mark Rushakoff a5371de0d0 fix(task): undo accidental type rename 2018-08-14 14:34:20 -07:00
Jade McGough 70ccb78c5d chore(task): update run schema 2018-08-14 13:33:05 -07:00
Mark Rushakoff 0759cd28e6 feat(task): allow specifying when task scheduling should start
More specifically, introduce a `scheduleAfter` argument to
Store.CreateTask. The previous behavior was leaving a new task's meta
LatestComplete value set to zero, which meant that the first run of a
schedule would start from 1970. Now, it's set to time.Now unless
otherwise specified.

Updates #595.
2018-08-13 10:47:08 -07:00
docmerlin (j. Emrys Landivar) 5b52f75f7e task name modification and uniqueness 2018-08-10 18:24:41 -05:00
Mark Rushakoff 42ce671724 chore: only set one run_id field on runner logs
We were previously doing `r.logger = r.logger.With(...)`, which ended up
duplicating the provided field in the log output. Now ensure that we
only set the run_id once in a logger.
2018-08-09 13:43:00 -07:00
Nathaniel Cook 5bde0b5be6 fix: Update query services to use Request type
Moves idpe.QueryService into platform/query.ProxyQueryService
Splits the Request into ProxyRequest and Request.

Changes query.QueryService and query.AsyncQueryService to use a Request
type. This means that the Compiler interface is consumed by the service
to abstract out transpilation vs Flux compilation vs raw spec.

The transpiler handler is removed.

There are separate http handlers and service implementations for each of
the three query services.

Query logging types are moved into platform.

The ResultIterator now expects Cancel to always be called.

The fluxd binary exposes the query endpoint specified in the swagger
file.
2018-08-08 15:31:35 -06:00
Lyon Hill e1ecceaca0 refactor(task): make setting now on every execution
By switching to passing a spec it allows us more control
2018-08-08 12:50:24 -06:00
Lyon Hill 3873ac0c65
Merge pull request #578 from influxdata/task/connect
feat(task): connect the api to tasks
2018-08-07 16:57:38 -06:00
Lyon Hill 212210eaec make necessary changes 2018-08-07 16:51:33 -06:00
Lyon Hill 02d1b6507b feat(task): connect the api to tasks
We need a coordination layer to facilitate the communication in the api to the task service.
The api was also not connected in main to the http service.
2018-08-07 15:53:17 -06:00
Mark Rushakoff 24ea772ee4 test(task): cache parsed options in test
This is to work around slow parsing (#484).

This change saves about 2 seconds on tests for task/backend and
task/backend/bolt. It saves around 45 seconds on enterprise tests.
2018-08-07 12:46:50 -07:00
Lyon Hill a647f10c14
Merge pull request #560 from influxdata/task/enable
feat(task): allow tasks to be enabled and disabled.
2018-08-07 09:26:45 -06:00
Lyon Hill 63b9adf86c feat(task): allow tasks to be enabled and disabled. 2018-08-06 15:24:34 -06:00
Mark Rushakoff 4745f27bef fix(task/backend): return useful error when validating empty script
After we started parsing options inside of the task validation methods,
we accidentally shadowed the "missing script" error when the script was
empty. This restores the behavior of telling the user "missing script"
rather than a cryptic syntax error.
2018-08-03 17:13:15 -07:00
Lyon Hill 7f6d51f45d add in comments 2018-08-03 14:27:07 -06:00
Lyon Hill 80fe53f2b6 refactor(task): Move task meta into the backend
Task meta belongs with the rest of the task backend content.
2018-08-03 14:27:07 -06:00
Lyon Hill 0661d05b1f update mock scheduler 2018-08-01 13:36:47 -06:00
Lyon Hill 3ba8784fed fix bad commit with merge issue 2018-08-01 13:36:47 -06:00
Lyon Hill 6bba50a2a1 minor cleanup 2018-08-01 13:32:54 -06:00
Lyon Hill 69761a98f7 feat(task): update the scheduler and logwriter interface
Preperatory change that should enable us to build more complex log writers.
2018-08-01 13:32:54 -06:00
Michael Desa 8c87c9d132 revert #442 2018-08-01 14:54:32 -04:00
Leonardo Di Donato bdaee706d6 Fixing last changes obtained from master to make them use uint64 IDs
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
2018-08-01 18:20:59 +02:00
Leonardo Di Donato 5addb88eb5 MustIDFromString no more requires test instance
It panics now.
2018-08-01 18:20:59 +02:00
Leonardo Di Donato 02a80a0665 Refactoring bolt usage of IDs (tasks)
It should be no more necessary to perform operations directly on ID bytes (pad vs unpad).
2018-08-01 18:20:59 +02:00
Leonardo Di Donato b67d3123e2 Refactoring executor tests' usage of IDs (tasks) 2018-08-01 18:20:59 +02:00
Leonardo Di Donato d2e764200e Refactoring task backend usage of IDs (tasks) 2018-08-01 18:20:59 +02:00
Leonardo Di Donato 14da8b1319 Refactoring storetest package-s usage of IDs (tasks) 2018-08-01 18:20:59 +02:00
Leonardo Di Donato 3651350cc5 Refactoring mock usage of IDs (task package) 2018-08-01 18:20:59 +02:00
Leonardo Di Donato a688eb84b3 Starting refactoring task package usage of IDs 2018-08-01 18:20:59 +02:00
docmerlin (j. Emrys Landivar) d61abc0721 Delete users and orgs 2018-07-31 11:57:09 -05:00
Lyon Hill 67c47aba33 minor fixes 2018-07-31 10:23:35 -06:00
Lyon Hill 84d36e6987 feat(tasks): add get meta to the store interface. 2018-07-31 09:47:12 -06:00
Mark Rushakoff 090d337bf0 feat(task): migrate task code from enterprise
Co-authored-by: j. Emrys Landivar <landivar@gmail.com>
Co-authored-by: Lyon Hill <lyondhill@gmail.com>
2018-07-26 15:16:56 -07:00