Commit Graph

116 Commits (aeeef803c03abb02e0816897aeb50edd1ea2db01)

Author SHA1 Message Date
Mark Rushakoff d9e10ff652 test(task): deflake scheduler test 2018-11-01 10:37:14 -07:00
Mark Rushakoff 39e115cebe fix(task): fix closure mistake when reclaiming tasks
Also improve some the coordinator tests a little bit and improve the
executor logging while we're here.
2018-10-31 19:00:02 -07:00
Nathaniel Cook 2a1d76eb2d chore: Update flux dependency for the type-inference changes 2018-10-31 17:11:11 -06:00
Mark Rushakoff a7e6a6634e test(task): deflake TaskService shared test
What happened here is kind of a long story.

The TaskService tests exercise different parts of the TaskService
concurrently, in hopes of exercising code that may be racy. This worked
fine so long as we were able to use arbitrary user and organization IDs
picked during test.

Then we added tests for the http-client-backed TaskService. The instance
of http.TaskService takes a single token, so the TaskService tests were
updated to use an optional override to get the user and org ID from the
TaskService implementation.

That meant our parallel tests were using one user and org ID in some
cases. One particular test to exercise concurrency would create tasks
and then retrieve them and delete one. Because this test was filtering
by user and org, sometimes it would delete a task created in a different
test.

The fix was to maintain a whitelist of tasks created in that particular
test, so that it wouldn't delete tasks created in a different test.
2018-10-30 18:52:21 -07:00
docmerlin (j. Emrys Landivar) 64883a5dad Use snowflake for bolt task ids 2018-10-30 18:20:59 -05:00
Mark Rushakoff cd80e41220 feat(task): add ability to retry runs
This also adjusts the TaskService interface's RetryRun method to accept
a task ID rather than an org ID. Internally, we still look up runs by
organization, and maybe that will change later, but this is a more
natural way for clients to look it up.
2018-10-30 09:20:40 -07:00
docmerlin (j. Emrys Landivar) 6e2620418c Now you can cancel runs 2018-10-29 16:08:50 -05:00
Mark Rushakoff 3fb86665eb test(task): fix check for setting task RequestedAt 2018-10-26 09:49:45 -07:00
Lyon Hill 751a53dd24
Allow coordinator to pick up existing tasks on startup (#1187) 2018-10-25 13:50:41 -06:00
Mark Rushakoff 8442acd437 refactor(task): emit "ticked" log messages at debug level
The ticked message every second is more annoying than helpful as-is.
Drop it to debug level so it's available if needed.
2018-10-25 11:23:28 -07:00
Mark Rushakoff 43b7617316 test(task): check task status in more methods
This TODO was meant to be removed as part #1179, but was mistakenly
overlooked.
2018-10-24 11:04:07 -07:00
Mark Rushakoff 75c5d91cec feat(task): include task status when listing tasks 2018-10-24 09:38:02 -07:00
Adam 98be8bcd32
moved to_http and to_kafka to flux and update to latest flux (#1175) 2018-10-24 12:33:43 -04:00
Lyon Hill 19b7709bad
Add tasks to the influx cli (#1163) 2018-10-23 11:51:13 -06:00
Mark Rushakoff 8c0aec7975 fix(task): propagate status updates when modifying tasks
This also changes the backend.Store API to remove the EnableTask and
DisableTask methods, merging their functionality into ModifyTask, which
has been named to UpdateTask to keep closer to the CRUD acronym.
2018-10-19 16:49:18 -07:00
Mark Rushakoff 3cf88009c4 test(task): fix incorrect test assertion
We changed the semantics of finding a task that doesn't exist, from
returning `nil, nil`, to `nil, ErrTaskNotFound`; this test didn't
reflect that change until this commit.
2018-10-15 10:24:23 -07:00
Jonathan A. Sternberg 5c8318d181
Merge pull request #1080 from influxdata/js-go-generate-with-go-run
refactor: modify tooling to take advantage of go run
2018-10-15 11:27:13 -05:00
Mark Rushakoff ad4942fe36 refactor(task): expose status when finding from TaskService
And consistently return ErrTaskNotFound, rather than nil, nil, when
searching an ID that doesn't exist.
2018-10-15 09:20:47 -07:00
Mark Rushakoff cb0b54cfd9 feat(task): allow setting task status during creation
This renames TaskEnabled and TaskDisabled to TaskActive and
TaskInactive, to keep in line with the swagger and other parts of the
system. But I left the EnableTask and DisableTask methods on the Store
interface for now. I could see eliminating those methods if we adjust
the signature of the UpdateTask method.
2018-10-15 09:20:47 -07:00
Mark Rushakoff 56f32b4780 refactor(task): extract CreateTaskRequest struct
We're about to add another parameter, so condensing the arguments into a
single struct makes sense at this point.
2018-10-15 09:20:47 -07:00
Jonathan A. Sternberg e9600b1f0b
refactor: modify tooling to take advantage of go run
The generate commands have been modified to take advantage of the new
functionality in Go 1.11 that allows `go run` to execute a package
instead of individual files.

This functionality combined with Go modules allows us to execute a
package directly out of our pinned dependencies rather than accidentally
picking up another binary outside of the build environment.

This also simplifies the Makefile because they no longer have to be
responsible for installing the correct tooling since the Go command
takes care of that logic. It also makes it so that the Makefiles with
file generation can now be invoked from their appropriate subdirectories
so they are contained within the directory itself rather than relying on
values in the top level Makefile.

It is now possible to generate all files within this project by using:

    go generate ./...

Or the Makefile can continue to be used.

This commit also copies over the special copy of `tmpl` that the storage
engine uses within the influxdb repository. It was never copied over so
using `go generate` on these packages did not work.
2018-10-15 11:01:27 -05:00
Lyon Hill e643c434f6
Allow the scheduler to Update tasks. (#1058)
* Allow the scheduler to Update tasks.

* update the coordinator to use the new update action

* remove unclean code
2018-10-12 08:24:22 -06:00
Chris Goller 2bea48f15a refactor(testing): rename MustIDFromString to MustIDBase16 2018-10-11 09:54:19 -05:00
Chris Goller ada7632f98 fix(task/backend): remove extra name by org 2018-10-11 09:54:18 -05:00
Leonardo Di Donato 6725ae6f64 feat(platform/id): method to create an invalid id
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
2018-10-11 09:54:18 -05:00
Leonardo Di Donato 016c7d04cb fix(task/servicetest): porting servicetest to uint64 IDs
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-Authored-By: Lorenzo Fontana <lo@linux.com>
2018-10-11 09:54:18 -05:00
Leonardo Di Donato bfee4df046 fix(task/backend/bolt): using uint64 IDs
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-Authored-By: Lorenzo Fontana <lo@linux.com>
2018-10-11 09:54:18 -05:00
Leonardo Di Donato 5d4d8a7c04 fix(platform/task/backend/executor): porting tests to uint64 IDs 2018-10-11 09:54:18 -05:00
Leonardo Di Donato 5db4276a71 fix(platform/id): improvements on code using uint64 IDs
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-Authored-By: Lorenzo Fontana <lo@linux.com>
2018-10-11 09:54:18 -05:00
Leonardo Di Donato 6587d1b13d fix(task/backend/coordinator): porting test to uint64 IDs 2018-10-11 09:54:18 -05:00
Leonardo Di Donato 0a75951684 fix(task/backend/storetest): porting test functions to uint64 IDs 2018-10-11 09:54:18 -05:00
Leonardo Di Donato ca7ba33d27 fix(task/mock): porting to uint64 IDs 2018-10-11 09:54:18 -05:00
Leonardo Di Donato 9104771408 fix(task/backend): porting to uint64 IDs
Co-Authored-By: Lorenzo Fontana <lo@linux.com>
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
2018-10-11 09:54:18 -05:00
Leonardo Di Donato f5dfe0aecc fix(task/backend): create task returns a platform.ID + inmem store
refactoring

Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-Authored-by: Lorenzo Fontana <lo@linux.com>
2018-10-11 09:54:18 -05:00
Leonardo Di Donato 8d7f06cf4b feat(platform/id): using uint64 for platform.IDs
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-Authored-by: Lorenzo Fontana <lo@linux.com>
2018-10-11 09:54:18 -05:00
Jonathan A. Sternberg e7e17d6972
refactor: move the logger package from influxdb to platform
We are moving the necessary code for 2.0 from the influxdb 1.X
repository to the platform 2.0 repository. The logger is an unnecessary
dependency on the old influxdb that is making life more complicated.
2018-10-10 15:49:07 -05:00
Lyon Hill 610faf18e7
Update the scheduler to add a few task logs (#1029) 2018-10-10 12:37:11 -06:00
Mark Rushakoff 35383074f3 chore(task): remove task name uniqueness requirement 2018-10-09 11:46:12 -07:00
Lyon Hill f081ce95f0
allow run shows to work with an actual org id. (#981)
* allow run shows to work with an actual org id.
also necessary for using actual run lists
2018-10-08 14:07:08 -06:00
Brandon Farmer c1c4dc3393 Properly handle spaces in PATH with make tasks 2018-10-08 09:53:22 -07:00
Michael Desa 42ba7ffe11 feat(http): add authentication handler middleware 2018-10-02 14:11:44 -04:00
Michael Desa 1a0c29ad34 feat(platform): add authorizer interface
This iterface is supposed to be something that both sessions and
authorizations can share so that other components can authorize requests
as they see fit.
2018-10-02 14:11:44 -04:00
docmerlin (j. Emrys Landivar) 25e965d7ae scheduler->TickScheduler 2018-10-01 14:59:02 -05:00
docmerlin (j. Emrys Landivar) 4502b4a47f add requestedAt to task log 2018-09-27 12:46:57 -05:00
Mark Rushakoff 1524768aed test(task): add test for reusing task names after deletion 2018-09-26 14:34:05 -07:00
Lyon Hill 7ff3361e78
fix(task): Allow task scheduler to be stopped and started. (#882) 2018-09-26 00:01:21 -06:00
Chris Goller a64b86399f chore(task/backend): regenerate meta.proto with protc 3.6.1 2018-09-25 22:18:36 -05:00
Chris Goller f2132168b1 fix(task/backend): copy gogo.proto into repo 2018-09-25 22:18:00 -05:00
Mark Rushakoff fe4b676d06 fix(task/backend): avoid data race in in-memory run log
Return reference to new Run instances from ListRuns, to avoid a data
race on the original element.

Fixes #842.
2018-09-17 09:51:14 -07:00
Lyon Hill c117f0d8ca
fix(task): fix a potential race condition in meta (#836) 2018-09-15 01:29:05 -06:00