* Delete unused configs
* Delete unused test steps
* Reorder our main config file to start with top-level pipelines, and to group conceptually-related jobs
* Introduce new jobs to pull dependencies, to reduce duplicated work across test/lint/build steps
* Rename our caches to bust our current huge sizes
* Use unique names for caches per-step, to avoid conflicts
* Fix result collection for JS tests
* Create Dockerfile for running build & release in CI
* Update CircleCI config to use new Docker image
* Update goreleaser to add a linux ARM64 release
* chore: update circleci config to use golang:1.15 images
* fix: Update Go version and use temporary override for Arrow
The Arrow override is to fix race detection problem. Once the
Arrow PR merges, this override can be removed.
Co-authored-by: Stuart Carnie <stuart.carnie@gmail.com>
- Update CIrcleCI configuration to start release process on an RC build
- Update .goreleaser.yml:
- Start building armel and armhf binaries and rpm and debian packages.
- Generate sha256 checksum file.
- launcher.go: do not use `max` module to escape integeroverflow problem for armel and armhf builds
- Start using `v0.142.0` of goreleaser
- Added pre and post install/uninstall scripts for rpm amd deb packages
Installing `osxcross` to `/opt` requires superuser permissions. This is
built as root in flux, but requires `sudo` for circleCI. Additionally,
there was a problem with `o64-clang` not being on the `PATH` that we
would have discovered the next time we tried to release.
It was a request from boonito QA to start running nightly builds 2 hours earlier, instead of 7am UTC to start running nightly builds at 5am UTC (10pm PDT) to give them time to debug test failures.
* chore(litmus): run integration tests on master:
- aafter successful E2E run after master build, run integration tests
modified: .circleci/config.yml
modified: etc/litmus_fail_notify.sh
modified: etc/litmus_success_notify.sh
* chore(litmus): litmus_integration run should depend on litmus_daily test run
modified: .circleci/config.yml
This adds support for using pkg-config to build libflux inside of the
flux dependency. The build can occur by either installing `pkg-config`
into your path or the `env` script can be used to invoke it from the go
modules.
* ci: run go race tests in parallel over 8 containers
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: add timing hint for splitting go tests
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: remove spurious grep
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: mod=readonly should ignore unknown internal/promqltests path
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: echo the packages tested
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: remove readonly experiment
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: filter promql internal tests
The promql internal tests has a special go.mod that shows
up in go list. That directory cannot be tested.
Signed-off-by: Chris Goller <goller@gmail.com>
* build: split go linting from tests
Signed-off-by: Chris Goller <goller@gmail.com>
* build: split js linting from tests
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: run go tests without verbose option
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: parallelize eslint with 8 containers
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: cache node modules for eslint
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: lint circle yaml
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: cd into ui directory for node caching
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: note eslint parallelization
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: update yarn cache directory to global
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: cache jstest yarn downloads
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: note parallelization of jest tests
Signed-off-by: Chris Goller <goller@gmail.com>
Signed-off-by: Lorenzo Affetti <lorenzo.affetti@gmail.com>
Signed-off-by: Julius Volz <julius.volz@gmail.com>
move to internal
update flux to v0.50
Revert "move to internal"
This reverts commit bcd4caffbd44135f1dbeac4163cb2a22a751f45a.
promtests/internal --> internal/promtests
* chore: add new types for NotiicationRules
* build(client): remove client and build each build
* chore: update gitignore to exclude generated client
* fix: allow client dir to be empty
* docs: add README to client dir
* fix(build): add autogen client to eslintignore
* build: gen client for ci js tests
Binary files can be whitelisted with the special line 'Adds-Binary:
<path/to/binary/file>' included in the commit message body. Deliberately
checked-in binary files should be very rare.
With minimal effort, this could probably be repurposed as a local
postcommit hook.
* ci(litmus): do not run litmus tests on forked PRs
- add a step to check if PR is from fork. If it is, then stop execution of the step and successfully return.
- Fix for https://github.com/influxdata/influxdb/issues/11965
* feat(kv:inmem:bolt): implement user service in a kv
* refactor(kv): use consistent func receiver name
* feat(kv): add initial basic auth service
* refactor(passwords): move auth interface into own file
* refactor(passwords): rename basic auth files to passwords
* refactor(passwords): rename from BasicAuth to Passwords
* refactor(kv): copy bolt user test into kv
Co-authored-by: Michael Desa <mjdesa@gmail.com>
* feat(kv): add inmem testing to kv store
* fix(kv): remove extra user index initialization
* feat(kv): attempt at making errors nice
* fix(http): return not found error if filter is invalid
* fix(http): s/platform/influxdb/ for user service
* fix(http): s/platform/influxdb/ for user service
* feat(kv): initial port of telegraf configs to kv
* feat(kv): first pass at migrating bolt org service to kv
* feat(kv): first pass at bucket service
* feat(kv): first pass at migrating kvlog to kv package
* feat(kv): add resource op logs
* feat(kv): first pass at user resource mapping migration
* feat(kv): add urm usage to bucket and org services
* feat(kv): first pass at kv authz service
* feat(kv): add cascading auth delete for users
* feat(kv): first pass d authorizer.OrganizationService in kv
* feat(cmd/influxd/launcher): user kv services where appropriate
* fix(kv): initialize authorizations
* fix(influxdb): use same buckets while slowly migrating stuff
* fix(kv): make staticcheck pass
* feat(kv): add dashboards to kv
review: make suggestions from pr review
fix: use common bucket names for bolt/kv stores
* test(kv): add complete password test coverage
* chore(kv): fixes for staticcheck
* feat(kv): implement labels generically on kv
* feat(kv): implement macro service
* feat(kv): add source service
* feat(kv): add session service
* feat(kv): add kv secret service
* refactor(kv): update telegraf and urm with error messages
* feat(kv): add lookup service
* feat(kv): add kv onboarding service
* refactor(kv): update telegraf to avoid repetition
* feat(cmd/influxd): use kv lookup service
* feat(kv): add telegraf to lookup service
* feat(cmd/influxd): use kv telegraf service
* feat(kv): initial port of scrapers in bolt to kv
* feat(kv): update scraper error messaging
* feat(cmd/influxd): add kv scraper
* feat(kv): add inmem backend tests
* refactor(kv): copy paste errors
* refactor(kv): add code to password errors
* fix(testing): update error messages for incorrect passwords
* feat(kv:inmem:bolt): implement user service in a kv
* refactor(kv): use consistent func receiver name
* refactor(kv): copy bolt user test into kv
Co-authored-by: Michael Desa <mjdesa@gmail.com>
* feat(kv): add inmem testing to kv store
* fix(kv): remove extra user index initialization
* feat(kv): attempt at making errors nice
* fix(http): return not found error if filter is invalid
* fix(http): s/platform/influxdb/ for user service
* feat(kv): first pass at migrating bolt org service to kv
* feat(kv): first pass at bucket service
* feat(kv): first pass at migrating kvlog to kv package
* feat(kv): add resource op logs
* feat(kv): first pass at user resource mapping migration
* feat(kv): add urm usage to bucket and org services
* feat(kv): first pass at kv authz service
* feat(kv): add cascading auth delete for users
* feat(kv): first pass d authorizer.OrganizationService in kv
* feat(cmd/influxd/launcher): user kv services where appropriate
* feat(kv): add initial basic auth service
* refactor(passwords): move auth interface into own file
* refactor(passwords): rename basic auth files to passwords
* fix(http): s/platform/influxdb/ for user service
* fix(kv): initialize authorizations
* fix(influxdb): use same buckets while slowly migrating stuff
* fix(kv): make staticcheck pass
* feat(kv): add dashboards to kv
review: make suggestions from pr review
fix: use common bucket names for bolt/kv stores
* feat(kv): implement labels generically on kv
* refactor(passwords): rename from BasicAuth to Passwords
* test(kv): add complete password test coverage
* chore(kv): fixes for staticcheck
* feat(kv): implement macro service
* feat(kv): add source service
* feat(kv): add session service
* feat(kv): initial port of telegraf configs to kv
* feat(kv): initial port of scrapers in bolt to kv
* feat(kv): add kv secret service
* refactor(kv): update telegraf and urm with error messages
* feat(kv): add lookup service
* feat(kv): add kv onboarding service
* refactor(kv): update telegraf to avoid repetition
* feat(cmd/influxd): use kv lookup service
* feat(kv): add telegraf to lookup service
* feat(cmd/influxd): use kv telegraf service
* feat(kv): update scraper error messaging
* feat(cmd/influxd): add kv scraper
* feat(kv): add inmem backend tests
* refactor(kv): copy paste errors
* refactor(kv): add code to password errors
* fix(testing): update error messages for incorrect passwords
* feat(http): initial support for flushing all key/values from kv store
* feat(kv): rename macro to variable
* feat(cmd/influxd/launcher): user kv services where appropriate
* refactor(passwords): rename from BasicAuth to Passwords
* feat(kv): implement macro service
* test(ui): introduce cypress
* test(ui): introduce first typescript test
* test(ui/e2e): add ci job
* chore: update gitignore to ignore test outputs
* feat(inmem): in memory influxdb
* test(e2e): adding pinger that checks if influxdb is alive
* hackathon
* hack
* hack
* hack
* hack
* Revert "feat(inmem): in memory influxdb"
This reverts commit 30ddf032003e704643b07ce80df61c3299ea7295.
* hack
* hack
* hack
* hack
* hack
* hack
* hack
* hack
* hack
* hack
* hack
* hack
* hack
* chore: lint ignore node_modules
* hack
* hack
* hack
* add user and flush
* hack
* remove unused vars
* hack
* hack
* ci(circle): prefix e2e artifacts
* change test to testid
* update cypress
* moar testid
* fix npm warnings
* remove absolte path
* chore(ci): remove /home/circleci proto mkdir hack
* wip: crud resources e2e
* fix(inmem): use inmem kv store services
* test(dashboard): add first dashboard crud tests
* hack
* undo hack
* fix: use response from setup for orgID
* chore: wip
* add convenience getByTitle function
* test(e2e): ui can create orgs
* test(e2e): add test for org deletion and update
* test(e2e): introduce task creation test
* test(e2e): create and update of buckets on org view
* chore: move types to declaration file
* chore: use route fixture in dashboard tests
* chore(ci): hack back
* test(ui): update snapshots
* chore: package-lock
* chore: remove macros
* fix: launcher rebase issues
* fix: compile errors
* fix: compile errors
* feat(cmd/influxdb): add explicit testing, asset-path, and store flags
Co-authored-by: Andrew Watkins <watts@influxdb.com>
* fix(cmd/influxd): set default HTTP handler and flags
Co-authored-by: Andrew Watkins <watts@influxdb.com>
* build(Makefile): add run-e2e and PHONY
* feat(kv:inmem:bolt): implement user service in a kv
* refactor(kv): use consistent func receiver name
* feat(kv): add initial basic auth service
* refactor(passwords): move auth interface into own file
* refactor(passwords): rename basic auth files to passwords
* refactor(passwords): rename from BasicAuth to Passwords
* refactor(kv): copy bolt user test into kv
Co-authored-by: Michael Desa <mjdesa@gmail.com>
* feat(kv): add inmem testing to kv store
* fix(kv): remove extra user index initialization
* feat(kv): attempt at making errors nice
* fix(http): return not found error if filter is invalid
* fix(http): s/platform/influxdb/ for user service
* fix(http): s/platform/influxdb/ for user service
* feat(kv): initial port of telegraf configs to kv
* feat(kv): initial port of scrapers in bolt to kv
* feat(kv): first pass at migrating bolt org service to kv
* feat(kv): first pass at bucket service
* feat(kv): first pass at migrating kvlog to kv package
* feat(kv): add resource op logs
* feat(kv): first pass at user resource mapping migration
* feat(kv): add urm usage to bucket and org services
* feat(kv): first pass at kv authz service
* feat(kv): add cascading auth delete for users
* feat(kv): first pass d authorizer.OrganizationService in kv
* feat(cmd/influxd/launcher): user kv services where appropriate
* fix(kv): initialize authorizations
* fix(influxdb): use same buckets while slowly migrating stuff
* fix(kv): make staticcheck pass
* feat(kv): add dashboards to kv
review: make suggestions from pr review
fix: use common bucket names for bolt/kv stores
* test(kv): add complete password test coverage
* chore(kv): fixes for staticcheck
* feat(kv): implement labels generically on kv
* feat(kv): implement macro service
* feat(kv): add source service
* feat(kv): add session service
* feat(kv): add kv secret service
* refactor(kv): update telegraf and urm with error messages
* feat(kv): add lookup service
* feat(kv): add kv onboarding service
* refactor(kv): update telegraf to avoid repetition
* feat(cmd/influxd): use kv lookup service
* feat(kv): add telegraf to lookup service
* feat(cmd/influxd): use kv telegraf service
* feat(kv): update scraper error messaging
* feat(cmd/influxd): add kv scraper
* feat(kv): add inmem backend tests
* refactor(kv): copy paste errors
* refactor(kv): add code to password errors
* fix(testing): update error messages for incorrect passwords
* feat(kv): rename macro to variable
* refactor(kv): auth/bucket/org/user unique checks return errors now
* feat(inmem): add way to get all bucket names from store
* feat(inmem): Buckets to return slice of bytes rather than strings
* feat(inmem): add locks around Buckets to avoid races
* feat(cmd/influx): check for unauthorized error in wrapCheckSetup
* chore(e2e): add video and screenshot artifcats to gitignore
* docs(ci): add build instructions for e2e tests
* feat(kv): add id lookup for authorized resources
* enable Litmus Smoke tests on daily PRs
- created a litmus_daily job
- added persist_to_workspace to build job to have access to influxd binary
- added litmus_daily to build workflow
- added two script files, litmus_success_notify.sh and litmus_fail_notify.sh to `etc` directory
- added the above script files to the one to be shared in the workspace
- updated litmus_daily and litmus_nightly jobs to use scripts for notifications.
The project is currently using goreleaser to manage building nightly binaries, docker images and uploads to s3.
The tool has a strict dependency on semantic versioned tags. InfluxDB history does not have a valid semantic tag causing the tool to fail. This commit creates a valid tag for the current time to allow nightly builds to continue.
I did this with a dumb editor macro, so some comments changed too.
Also rename root package from platform to influxdb.
In interest of minimizing risk, anyone importing the root package has
now aliased it to "platform" so that no changes beyond imports were
necessary in those files.
Lastly, replace the old platform module to local path /dev/null so that
nobody can accidentally reintroduce a platform dependency while
migrating platform code to influxdb.
In go1.11, go mod vendor can modify go.sum when go.mod contains replace
directives. This behavior has been fixed in go1.12, so just work around
it for now.
Fixes#2056.
The docker container we're using has 2 vCPUs. -p=8 was working fine for
a while, but diagnosing #1832 looks like the tsi1.test process was
likely getting OOM-killed. Reducing from -p=8 to -p=4 ought to reduce
memory pressure enough to avoid OOM kills, hopefully.
Also remove a stale TODO.
test(platform): run testcontainer integration tests for nightly release
Integration tests for the vault secret service using testcontiners
should not run along with unit tests, however, they should run on some
regular schedule. This commit introduces `make test-integration` which
runs integration tests for vault using testcontainers. The command introduced
relies on docker being available on the host it is executed on.
chore(platform): make go modules tidy
chore: try to fix go mod
chore(platform): remove explicit logrus dependency
chore(platform): run go mod tidy
chore(platform): replace github.com/Sirupsen/logrus with github.com/sirupsen/logrus
chore(platform): update docker dependency
feat(vault): add vault implementation of secret service
test(platform): run testcontainer integration tests for nightly release
Integration tests for the vault secret service using testcontiners
should not run along with unit tests, however, they should run on some
regular schedule. This commit introduces `make test-integration` which
runs integration tests for vault using testcontainers. The command introduced
relies on docker being available on the host it is executed on.
chore(platform): make go modules tidy
chore: try to fix go mod
chore(platform): run go mod tidy
feat(vault): add vault implementation of secret service
chore(platform): make go modules tidy
feat(platform): add Put/Patch/Delete methods on secret service
feat(vault): add Put/Patch/Delete methods on vault secret service
feat(http): add http handler methods for secret service
feat(bolt): add Put/Delete/Patch methods to bolt secret service
feat(testing): add tests for Put/Patch/Delete methods in secret service
feat(mock): add mock secret service
feat(http): add tests for secrets endpoints
feat(http): update swagger for secrets endpoints
chore: run go mod tidy
I looked through the past few days of Circle Go failures, and they all
appeared to be real test errors, indicating that the -p=8 flag most
likely fixed the linker OOMs we were seeing.
If the race-enabled tests continue to OOM, we can try setting -p=2 to
match the number of CPUs actually available on our Circle container.
The environment Go runs in thinks there are 32 CPUs, but according to
https://circleci.com/docs/2.0/configuration-reference/#resource_class
that's a known issue, and only 2 CPUs are actually available.
We're still overallocating with -p=8, so if we continue to get OOMs we
may need to reduce that further.
The one remaining failing static check is in a pigeon-generated promql
file. It looks like megacheck is supposed to ignore generated files by
default, so it's unclear at this time if this is a bug in megacheck, or
if fixing pigeon is the only path forward.
* Simplifiy color type
* Fix type
* Introduce V2 data shape for views
* WIP Split style and parsing to separate Dygraph components
* Add basic dygraph view types
* Add Gauge to v2 view shapes
* Upgrade TypeScript to ^3.0
* Add tsc to Circle build
* Fix Dygraph component paths
* Add testURL to jest config
* Upgrade lodash types
* Remove redundant test linter stetp
* Upgrade to TypeScript ^3
* Remove TableGraph (temporarily)
This updates the dependency manager to use go modules instead of dep so
that we can remove dep as a dependency and we can begin using and giving
feedback on modules within this project. It should simplify dependency
management and make it faster to run the builds too.
The big change is updating the various Makefiles to stop relying on the
vendor directory because it no longer exists. This change creates a
`tools.go` file with the revisions of the tools we use that are currently
in the `required` section of `Gopkg.toml`. We are currently in other
discussions to modify how we handle tool dependency management, but this
change does not change that.
On a cache miss, `yarn install` via `make node_modules` takes about 42
seconds, plus about 19 seconds to upload the cache. See
https://circleci.com/gh/influxdata/platform/1872.
On a cache hit, `make node_modules` takes about 7 seconds to download
the cache plus 19 seconds to `yarn install`. See
https://circleci.com/gh/influxdata/platform/1877. Using the cache here
is a net gain of about 16 seconds on cache hit, which is the most likely
case given a random commit.
Running `dep ensure` via `make vendor`, with a clean dep cache, takes
around 30 seconds.
For run https://circleci.com/gh/influxdata/platform/1838, we can see
that `make vendor` took 33 seconds and saving the dep cache took 38
seconds. The 38 seconds to save the dep cache is paid the first time
that a particular version of Gopkg.lock is pushed. A subsequent run that
fully uses the dep cache costs 14 seconds to restore the dep cache and 5
seconds to run `make vendor`, so this saves about 10 seconds for any run
that doesn't change Gopkg.lock. See
https://circleci.com/gh/influxdata/platform/1840.
With a clean GOCACHE, `make test-go` took about 30 seconds. It's hard to
tell the exact time since `make vendor` was implicitly executed in the
same block.
Now, `make test-go` uses GOCACHE and takes about 5 seconds to restore
the cache, then about 8 seconds to run through a fully cached set of
tests. If any test is actually broken or doesn't compile, this will give
us fast feedback.
But this change also runs `go test -race -count=1`, so that we fully
exercise any possible data races. That currently takes about 19 seconds.
Then finally we save GOCACHE, which has our `go test` results, and a
populated build cache for our test files, including built with the race
detector. Saving that takes about 14 seconds.
That means we took about 30 seconds originally to just run `go test
-count=1`, and now we take (5+8+19+14)=46 seconds to run plain go test
and go test with the race detector. There's probably an argument to be
made for just running with the race detector, but running both gives us
more coverage IMO, and it does allow us to run tests that aren't enabled
on race builds.
See https://circleci.com/gh/influxdata/platform/1840 for a run with full
caching.
Daniel Moran
Gershon Shif
Russ Savage
Stuart Carnie
Stephanie Engel
Zoe Steinkamp
Adrian Thurston
Gavin Cabbage
Jonathan A. Sternberg
karel-rehor
Alex Boatwright
Paul Hummer
jlapacik
Ariel Salem
Chris Goller
Christopher M. Wolff
j. Emrys Landivar (docmerlin)
Yiqun Zhang
Julius Volz
Andrew Watkins
Mark Rushakoff
Brandon Farmer
Kelvin Wang
Michael Desa
Luke Morris
AJ Bond
Mark Rushakoff
ABond
Nathaniel Cook
zhulongcheng
Edd Robinson