Commit Graph

98 Commits (aeeef803c03abb02e0816897aeb50edd1ea2db01)

Author SHA1 Message Date
Mark Rushakoff d9e10ff652 test(task): deflake scheduler test 2018-11-01 10:37:14 -07:00
Mark Rushakoff 39e115cebe fix(task): fix closure mistake when reclaiming tasks
Also improve some the coordinator tests a little bit and improve the
executor logging while we're here.
2018-10-31 19:00:02 -07:00
docmerlin (j. Emrys Landivar) 64883a5dad Use snowflake for bolt task ids 2018-10-30 18:20:59 -05:00
Mark Rushakoff cd80e41220 feat(task): add ability to retry runs
This also adjusts the TaskService interface's RetryRun method to accept
a task ID rather than an org ID. Internally, we still look up runs by
organization, and maybe that will change later, but this is a more
natural way for clients to look it up.
2018-10-30 09:20:40 -07:00
docmerlin (j. Emrys Landivar) 6e2620418c Now you can cancel runs 2018-10-29 16:08:50 -05:00
Lyon Hill 751a53dd24
Allow coordinator to pick up existing tasks on startup (#1187) 2018-10-25 13:50:41 -06:00
Mark Rushakoff 8442acd437 refactor(task): emit "ticked" log messages at debug level
The ticked message every second is more annoying than helpful as-is.
Drop it to debug level so it's available if needed.
2018-10-25 11:23:28 -07:00
Mark Rushakoff 75c5d91cec feat(task): include task status when listing tasks 2018-10-24 09:38:02 -07:00
Adam 98be8bcd32
moved to_http and to_kafka to flux and update to latest flux (#1175) 2018-10-24 12:33:43 -04:00
Mark Rushakoff 8c0aec7975 fix(task): propagate status updates when modifying tasks
This also changes the backend.Store API to remove the EnableTask and
DisableTask methods, merging their functionality into ModifyTask, which
has been named to UpdateTask to keep closer to the CRUD acronym.
2018-10-19 16:49:18 -07:00
Jonathan A. Sternberg 5c8318d181
Merge pull request #1080 from influxdata/js-go-generate-with-go-run
refactor: modify tooling to take advantage of go run
2018-10-15 11:27:13 -05:00
Mark Rushakoff ad4942fe36 refactor(task): expose status when finding from TaskService
And consistently return ErrTaskNotFound, rather than nil, nil, when
searching an ID that doesn't exist.
2018-10-15 09:20:47 -07:00
Mark Rushakoff cb0b54cfd9 feat(task): allow setting task status during creation
This renames TaskEnabled and TaskDisabled to TaskActive and
TaskInactive, to keep in line with the swagger and other parts of the
system. But I left the EnableTask and DisableTask methods on the Store
interface for now. I could see eliminating those methods if we adjust
the signature of the UpdateTask method.
2018-10-15 09:20:47 -07:00
Mark Rushakoff 56f32b4780 refactor(task): extract CreateTaskRequest struct
We're about to add another parameter, so condensing the arguments into a
single struct makes sense at this point.
2018-10-15 09:20:47 -07:00
Jonathan A. Sternberg e9600b1f0b
refactor: modify tooling to take advantage of go run
The generate commands have been modified to take advantage of the new
functionality in Go 1.11 that allows `go run` to execute a package
instead of individual files.

This functionality combined with Go modules allows us to execute a
package directly out of our pinned dependencies rather than accidentally
picking up another binary outside of the build environment.

This also simplifies the Makefile because they no longer have to be
responsible for installing the correct tooling since the Go command
takes care of that logic. It also makes it so that the Makefiles with
file generation can now be invoked from their appropriate subdirectories
so they are contained within the directory itself rather than relying on
values in the top level Makefile.

It is now possible to generate all files within this project by using:

    go generate ./...

Or the Makefile can continue to be used.

This commit also copies over the special copy of `tmpl` that the storage
engine uses within the influxdb repository. It was never copied over so
using `go generate` on these packages did not work.
2018-10-15 11:01:27 -05:00
Lyon Hill e643c434f6
Allow the scheduler to Update tasks. (#1058)
* Allow the scheduler to Update tasks.

* update the coordinator to use the new update action

* remove unclean code
2018-10-12 08:24:22 -06:00
Chris Goller 2bea48f15a refactor(testing): rename MustIDFromString to MustIDBase16 2018-10-11 09:54:19 -05:00
Chris Goller ada7632f98 fix(task/backend): remove extra name by org 2018-10-11 09:54:18 -05:00
Leonardo Di Donato 6725ae6f64 feat(platform/id): method to create an invalid id
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
2018-10-11 09:54:18 -05:00
Leonardo Di Donato bfee4df046 fix(task/backend/bolt): using uint64 IDs
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-Authored-By: Lorenzo Fontana <lo@linux.com>
2018-10-11 09:54:18 -05:00
Leonardo Di Donato 5d4d8a7c04 fix(platform/task/backend/executor): porting tests to uint64 IDs 2018-10-11 09:54:18 -05:00
Leonardo Di Donato 5db4276a71 fix(platform/id): improvements on code using uint64 IDs
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-Authored-By: Lorenzo Fontana <lo@linux.com>
2018-10-11 09:54:18 -05:00
Leonardo Di Donato 6587d1b13d fix(task/backend/coordinator): porting test to uint64 IDs 2018-10-11 09:54:18 -05:00
Leonardo Di Donato 0a75951684 fix(task/backend/storetest): porting test functions to uint64 IDs 2018-10-11 09:54:18 -05:00
Leonardo Di Donato 9104771408 fix(task/backend): porting to uint64 IDs
Co-Authored-By: Lorenzo Fontana <lo@linux.com>
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
2018-10-11 09:54:18 -05:00
Leonardo Di Donato f5dfe0aecc fix(task/backend): create task returns a platform.ID + inmem store
refactoring

Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-Authored-by: Lorenzo Fontana <lo@linux.com>
2018-10-11 09:54:18 -05:00
Leonardo Di Donato 8d7f06cf4b feat(platform/id): using uint64 for platform.IDs
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-Authored-by: Lorenzo Fontana <lo@linux.com>
2018-10-11 09:54:18 -05:00
Jonathan A. Sternberg e7e17d6972
refactor: move the logger package from influxdb to platform
We are moving the necessary code for 2.0 from the influxdb 1.X
repository to the platform 2.0 repository. The logger is an unnecessary
dependency on the old influxdb that is making life more complicated.
2018-10-10 15:49:07 -05:00
Lyon Hill 610faf18e7
Update the scheduler to add a few task logs (#1029) 2018-10-10 12:37:11 -06:00
Mark Rushakoff 35383074f3 chore(task): remove task name uniqueness requirement 2018-10-09 11:46:12 -07:00
Lyon Hill f081ce95f0
allow run shows to work with an actual org id. (#981)
* allow run shows to work with an actual org id.
also necessary for using actual run lists
2018-10-08 14:07:08 -06:00
Brandon Farmer c1c4dc3393 Properly handle spaces in PATH with make tasks 2018-10-08 09:53:22 -07:00
docmerlin (j. Emrys Landivar) 25e965d7ae scheduler->TickScheduler 2018-10-01 14:59:02 -05:00
docmerlin (j. Emrys Landivar) 4502b4a47f add requestedAt to task log 2018-09-27 12:46:57 -05:00
Mark Rushakoff 1524768aed test(task): add test for reusing task names after deletion 2018-09-26 14:34:05 -07:00
Lyon Hill 7ff3361e78
fix(task): Allow task scheduler to be stopped and started. (#882) 2018-09-26 00:01:21 -06:00
Chris Goller a64b86399f chore(task/backend): regenerate meta.proto with protc 3.6.1 2018-09-25 22:18:36 -05:00
Chris Goller f2132168b1 fix(task/backend): copy gogo.proto into repo 2018-09-25 22:18:00 -05:00
Mark Rushakoff fe4b676d06 fix(task/backend): avoid data race in in-memory run log
Return reference to new Run instances from ListRuns, to avoid a data
race on the original element.

Fixes #842.
2018-09-17 09:51:14 -07:00
Lyon Hill c117f0d8ca
fix(task): fix a potential race condition in meta (#836) 2018-09-15 01:29:05 -06:00
Lyon Hill eb260436d6
fix(task): remove race condition when finishing a inmem run (#829)
* fix(task): remove race condition when finishing a inmem run
2018-09-13 10:11:59 -06:00
Lyon Hill 8cc55b3a37
fix(task): race condition in creating and finding meta data fixed (#828) 2018-09-13 09:11:25 -06:00
Nathaniel Cook 672e2d5fe7 chore: Updates to be able to remove platform as a dependency of Flux 2018-09-12 10:18:54 -06:00
Lyon Hill 54cb6dfaa9
fix(task): allow schedulers to recover in progress tasks. (#810)
* fix(task): allow schedulers to recover in progress tasks.
2018-09-11 09:32:59 -06:00
Mark Rushakoff 7bbd087980 chore(task): clean up log writer API
Introduce a RunLogBase type that encapsulates the base information
shared across multiple logs for the same run. This has the information
previously part of the API (Task and RunID), and includes the addition
of a RunScheduledFor timestamp.

Also remove the RunScheduled value for RunStatus, as it was never used
in the scheduler.
2018-09-07 09:35:55 -07:00
Stuart Carnie ac75af2f58 refactor: Migrate query package to influxdata/flux repository 2018-09-06 11:13:48 -07:00
Lyon Hill 5bcc9063d7
feat(taskd): create a FindTaskByIDWithMeta function. (#781)
Now we can retrieve both task data as well as meta in one call.
2018-09-06 11:40:32 -06:00
Mark Rushakoff 9a469fd8ad chore(task): log on tick
This will help give operators a better idea that the system is not
actually idle.
2018-09-05 12:06:58 -07:00
Mark Rushakoff 479f3dbb3d test(task): add platform adapter tests
Also fix a handful of segfaults caused by improperly retained byte slices
from bolt, and combine two view transactions into one to avoid  data
race when a delete happened between them.
2018-09-05 10:06:55 -07:00
Lyon Hill 5be44d7fac
fix(taskd): Make WithTicker not block. (#765)
A mistake in the WithTicker method caused it to block.
This was not evident because until recently we used a cron styler timer.
Removing the block should allow the task deamon to start up properly.
2018-09-04 10:16:40 -06:00