influxdb/README.md

178 lines
9.6 KiB
Markdown
Raw Normal View History

# InfluxDB [![CircleCI](https://circleci.com/gh/influxdata/influxdb.svg?style=svg)](https://circleci.com/gh/influxdata/influxdb)
[![Slack Status](https://img.shields.io/badge/slack-join_chat-white.svg?logo=slack&style=social)](https://www.influxdata.com/slack)
InfluxDB is an open source time series platform. This includes APIs for storing and querying data, processing it in the background for ETL or monitoring and alerting purposes, user dashboards, and visualizing and exploring the data and more. The master branch on this repo now represents the latest InfluxDB, which now includes functionality for Kapacitor (background processing) and Chronograf (the UI) all in a single binary.
2019-01-12 22:48:58 +00:00
The list of InfluxDB Client Libraries that are compatible with the latest version can be found in [our documentation](https://docs.influxdata.com/influxdb/latest/tools/client-libraries/).
If you are looking for the 1.x line of releases, there are branches for each minor version as well as a `master-1.x` branch that will contain the code for the next 1.x release. The master-1.x [working branch is here](https://github.com/influxdata/influxdb/tree/master-1.x). The [InfluxDB 1.x Go Client can be found here](https://github.com/influxdata/influxdb1-client).
## Install
2018-05-14 16:14:40 +00:00
We have nightly and versioned Docker images, Debian packages, RPM packages, and tarballs of InfluxDB available at the [InfluxData downloads page](https://portal.influxdata.com/downloads/). We also provide the `influx` command line interface (CLI) client as a separate binary available at the same location.
If you are interested in building from source, see the [building from source](CONTRIBUTING.md#building-from-source) guide for contributors.
## Get Started
For a complete getting started guide, please see our full [online documentation site](https://docs.influxdata.com/influxdb/latest/).
To write and query data or use the API in any way, you'll need to first create a user, credentials, organization and bucket.
Everything in InfluxDB is organized under a concept of an organization. The API is designed to be multi-tenant.
Buckets represent where you store time series data.
They're synonymous with what was previously in InfluxDB 1.x a database and retention policy.
The simplest way to get set up is to point your browser to [http://localhost:8086](http://localhost:8086) and go through the prompts.
You can also get set up from the CLI using the command `influx setup`:
```bash
$ bin/$(uname -s | tr '[:upper:]' '[:lower:]')/influx setup
Welcome to InfluxDB 2.0!
Please type your primary username: marty
Please type your password:
Please type your password again:
Please type your primary organization name.: InfluxData
Please type your primary bucket name.: telegraf
2019-01-22 21:23:51 +00:00
Please type your retention period in hours.
Or press ENTER for infinite.: 72
You have entered:
Username: marty
Organization: InfluxData
Bucket: telegraf
Retention Period: 72 hrs
Confirm? (y/n): y
UserID Username Organization Bucket
033a3f2c5ccaa000 marty InfluxData Telegraf
Your token has been stored in /Users/marty/.influxdbv2/credentials
```
You can run this command non-interactively using the `-f, --force` flag if you are automating the setup.
2019-01-22 21:23:51 +00:00
Some added flags can help:
```bash
$ bin/$(uname -s | tr '[:upper:]' '[:lower:]')/influx setup \
--username marty \
--password F1uxKapacit0r85 \
--org InfluxData \
--bucket telegraf \
--retention 168 \
--token where-were-going-we-dont-need-roads \
--force
```
Once setup is complete, a configuration profile is created to allow you to interact with your local InfluxDB without passing in credentials each time. You can list and manage those profiles using the `influx config` command.
```bash
$ bin/$(uname -s | tr '[:upper:]' '[:lower:]')/influx config
Active Name URL Org
* default http://localhost:8086 InfluxData
```
## Write Data
Write to measurement `m`, with tag `v=2`, in bucket `telegraf`, which belongs to organization `InfluxData`:
```bash
$ bin/$(uname -s | tr '[:upper:]' '[:lower:]')/influx write --bucket telegraf --precision s "m v=2 $(date +%s)"
```
Since you have a default profile set up, you can omit the Organization and Token from the command.
Write the same point using `curl`:
```bash
curl --header "Authorization: Token $(bin/$(uname -s | tr '[:upper:]' '[:lower:]')/influx auth list --json | jq -r '.[0].token')" \
--data-raw "m v=2 $(date +%s)" \
"http://localhost:8086/api/v2/write?org=InfluxData&bucket=telegraf&precision=s"
```
Read that back with a simple Flux query:
```bash
$ bin/$(uname -s | tr '[:upper:]' '[:lower:]')/influx query 'from(bucket:"telegraf") |> range(start:-1h)'
Result: _result
Table: keys: [_start, _stop, _field, _measurement]
_start:time _stop:time _field:string _measurement:string _time:time _value:float
------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ----------------------------
2019-12-30T22:19:39.043918000Z 2019-12-30T23:19:39.043918000Z v m 2019-12-30T23:17:02.000000000Z 2
```
Use the `-r, --raw` option to return the raw flux response from the query. This is useful for moving data from one instance to another as the `influx write` command can accept the Flux response using the `--format csv` option.
## Script with Flux
Flux (previously named IFQL) is an open source functional data scripting language designed for querying, analyzing, and acting on data. Flux supports multiple data source types, including:
- Time series databases (such as InfluxDB)
- Relational SQL databases (such as MySQL and PostgreSQL)
- CSV
The source for Flux is [available on GitHub](https://github.com/influxdata/flux).
To learn more about Flux, see the latest [InfluxData Flux documentation](https://docs.influxdata.com/flux/) and [CTO Paul Dix's presentation](https://speakerdeck.com/pauldix/flux-number-fluxlang-a-new-time-series-data-scripting-language).
## Contribute to the Project
InfluxDB is an [MIT licensed](LICENSE) open source project and we love our community. The fastest way to get something fixed is to open a PR. Check out our [contributing](CONTRIBUTING.md) guide if you're interested in helping out. Also, join us on our [Community Slack Workspace](https://influxdata.com/slack) if you have questions or comments for our engineering teams.
2018-11-21 15:08:34 +00:00
## CI and Static Analysis
2018-11-21 15:29:10 +00:00
### CI
All pull requests will run through CI, which is currently hosted by Circle.
Community contributors should be able to see the outcome of this process by looking at the checks on their PR.
Please fix any issues to ensure a prompt review from members of the team.
The InfluxDB project is used internally in a number of proprietary InfluxData products, and as such, PRs and changes need to be tested internally.
2018-11-21 15:29:10 +00:00
This can take some time, and is not really visible to community contributors.
### Static Analysis
This project uses the following static analysis tools.
Failure during the running of any of these tools results in a failed build.
2018-11-21 15:08:34 +00:00
Generally, code must be adjusted to satisfy these tools, though there are exceptions.
- [go vet](https://golang.org/cmd/vet/) checks for Go code that should be considered incorrect.
- [go fmt](https://golang.org/cmd/gofmt/) checks that Go code is correctly formatted.
- [go mod tidy](https://tip.golang.org/cmd/go/#hdr-Add_missing_and_remove_unused_modules) ensures that the source code and go.mod agree.
2022-06-27 18:45:01 +00:00
- [staticcheck](https://staticcheck.io/docs/) checks for things like: unused code, code that can be simplified, code that is incorrect and code that will have performance issues.
2018-11-21 15:29:10 +00:00
### staticcheck
2018-11-21 15:29:10 +00:00
If your PR fails `staticcheck` it is easy to dig into why it failed, and also to fix the problem.
First, take a look at the error message in Circle under the `staticcheck` build section, e.g.,
```
tsdb/tsm1/encoding.gen.go:1445:24: func BooleanValues.assertOrdered is unused (U1000)
tsdb/tsm1/encoding.go:172:7: receiver name should not be an underscore, omit the name if it is unused (ST1006)
```
Next, go and take a [look here](http://next.staticcheck.io/docs/checks) for some clarification on the error code that you have received, e.g., `U1000`.
The docs will tell you what's wrong, and often what you need to do to fix the issue.
2018-11-21 15:29:10 +00:00
#### Generated Code
Sometimes generated code will contain unused code or occasionally that will fail a different check.
`staticcheck` allows for [entire files](http://next.staticcheck.io/docs/#ignoring-problems) to be ignored, though it's not ideal.
A linter directive, in the form of a comment, must be placed within the generated file.
This is problematic because it will be erased if the file is re-generated.
2018-11-21 15:29:10 +00:00
Until a better solution comes about, below is the list of generated files that need an ignores comment.
If you re-generate a file and find that `staticcheck` has failed, please see this list below for what you need to put back:
| File | Comment |
| :--------------------: | :--------------------------------------------------------------: |
| query/promql/promql.go | //lint:file-ignore SA6001 Ignore all unused code, it's generated |
feat(kv): implemented key/value store with end-to-end integration tests * feat(kv:inmem:bolt): implement user service in a kv * refactor(kv): use consistent func receiver name * feat(kv): add initial basic auth service * refactor(passwords): move auth interface into own file * refactor(passwords): rename basic auth files to passwords * refactor(passwords): rename from BasicAuth to Passwords * refactor(kv): copy bolt user test into kv Co-authored-by: Michael Desa <mjdesa@gmail.com> * feat(kv): add inmem testing to kv store * fix(kv): remove extra user index initialization * feat(kv): attempt at making errors nice * fix(http): return not found error if filter is invalid * fix(http): s/platform/influxdb/ for user service * fix(http): s/platform/influxdb/ for user service * feat(kv): initial port of telegraf configs to kv * feat(kv): first pass at migrating bolt org service to kv * feat(kv): first pass at bucket service * feat(kv): first pass at migrating kvlog to kv package * feat(kv): add resource op logs * feat(kv): first pass at user resource mapping migration * feat(kv): add urm usage to bucket and org services * feat(kv): first pass at kv authz service * feat(kv): add cascading auth delete for users * feat(kv): first pass d authorizer.OrganizationService in kv * feat(cmd/influxd/launcher): user kv services where appropriate * fix(kv): initialize authorizations * fix(influxdb): use same buckets while slowly migrating stuff * fix(kv): make staticcheck pass * feat(kv): add dashboards to kv review: make suggestions from pr review fix: use common bucket names for bolt/kv stores * test(kv): add complete password test coverage * chore(kv): fixes for staticcheck * feat(kv): implement labels generically on kv * feat(kv): implement macro service * feat(kv): add source service * feat(kv): add session service * feat(kv): add kv secret service * refactor(kv): update telegraf and urm with error messages * feat(kv): add lookup service * feat(kv): add kv onboarding service * refactor(kv): update telegraf to avoid repetition * feat(cmd/influxd): use kv lookup service * feat(kv): add telegraf to lookup service * feat(cmd/influxd): use kv telegraf service * feat(kv): initial port of scrapers in bolt to kv * feat(kv): update scraper error messaging * feat(cmd/influxd): add kv scraper * feat(kv): add inmem backend tests * refactor(kv): copy paste errors * refactor(kv): add code to password errors * fix(testing): update error messages for incorrect passwords * feat(kv:inmem:bolt): implement user service in a kv * refactor(kv): use consistent func receiver name * refactor(kv): copy bolt user test into kv Co-authored-by: Michael Desa <mjdesa@gmail.com> * feat(kv): add inmem testing to kv store * fix(kv): remove extra user index initialization * feat(kv): attempt at making errors nice * fix(http): return not found error if filter is invalid * fix(http): s/platform/influxdb/ for user service * feat(kv): first pass at migrating bolt org service to kv * feat(kv): first pass at bucket service * feat(kv): first pass at migrating kvlog to kv package * feat(kv): add resource op logs * feat(kv): first pass at user resource mapping migration * feat(kv): add urm usage to bucket and org services * feat(kv): first pass at kv authz service * feat(kv): add cascading auth delete for users * feat(kv): first pass d authorizer.OrganizationService in kv * feat(cmd/influxd/launcher): user kv services where appropriate * feat(kv): add initial basic auth service * refactor(passwords): move auth interface into own file * refactor(passwords): rename basic auth files to passwords * fix(http): s/platform/influxdb/ for user service * fix(kv): initialize authorizations * fix(influxdb): use same buckets while slowly migrating stuff * fix(kv): make staticcheck pass * feat(kv): add dashboards to kv review: make suggestions from pr review fix: use common bucket names for bolt/kv stores * feat(kv): implement labels generically on kv * refactor(passwords): rename from BasicAuth to Passwords * test(kv): add complete password test coverage * chore(kv): fixes for staticcheck * feat(kv): implement macro service * feat(kv): add source service * feat(kv): add session service * feat(kv): initial port of telegraf configs to kv * feat(kv): initial port of scrapers in bolt to kv * feat(kv): add kv secret service * refactor(kv): update telegraf and urm with error messages * feat(kv): add lookup service * feat(kv): add kv onboarding service * refactor(kv): update telegraf to avoid repetition * feat(cmd/influxd): use kv lookup service * feat(kv): add telegraf to lookup service * feat(cmd/influxd): use kv telegraf service * feat(kv): update scraper error messaging * feat(cmd/influxd): add kv scraper * feat(kv): add inmem backend tests * refactor(kv): copy paste errors * refactor(kv): add code to password errors * fix(testing): update error messages for incorrect passwords * feat(http): initial support for flushing all key/values from kv store * feat(kv): rename macro to variable * feat(cmd/influxd/launcher): user kv services where appropriate * refactor(passwords): rename from BasicAuth to Passwords * feat(kv): implement macro service * test(ui): introduce cypress * test(ui): introduce first typescript test * test(ui/e2e): add ci job * chore: update gitignore to ignore test outputs * feat(inmem): in memory influxdb * test(e2e): adding pinger that checks if influxdb is alive * hackathon * hack * hack * hack * hack * Revert "feat(inmem): in memory influxdb" This reverts commit 30ddf032003e704643b07ce80df61c3299ea7295. * hack * hack * hack * hack * hack * hack * hack * hack * hack * hack * hack * hack * hack * chore: lint ignore node_modules * hack * hack * hack * add user and flush * hack * remove unused vars * hack * hack * ci(circle): prefix e2e artifacts * change test to testid * update cypress * moar testid * fix npm warnings * remove absolte path * chore(ci): remove /home/circleci proto mkdir hack * wip: crud resources e2e * fix(inmem): use inmem kv store services * test(dashboard): add first dashboard crud tests * hack * undo hack * fix: use response from setup for orgID * chore: wip * add convenience getByTitle function * test(e2e): ui can create orgs * test(e2e): add test for org deletion and update * test(e2e): introduce task creation test * test(e2e): create and update of buckets on org view * chore: move types to declaration file * chore: use route fixture in dashboard tests * chore(ci): hack back * test(ui): update snapshots * chore: package-lock * chore: remove macros * fix: launcher rebase issues * fix: compile errors * fix: compile errors * feat(cmd/influxdb): add explicit testing, asset-path, and store flags Co-authored-by: Andrew Watkins <watts@influxdb.com> * fix(cmd/influxd): set default HTTP handler and flags Co-authored-by: Andrew Watkins <watts@influxdb.com> * build(Makefile): add run-e2e and PHONY * feat(kv:inmem:bolt): implement user service in a kv * refactor(kv): use consistent func receiver name * feat(kv): add initial basic auth service * refactor(passwords): move auth interface into own file * refactor(passwords): rename basic auth files to passwords * refactor(passwords): rename from BasicAuth to Passwords * refactor(kv): copy bolt user test into kv Co-authored-by: Michael Desa <mjdesa@gmail.com> * feat(kv): add inmem testing to kv store * fix(kv): remove extra user index initialization * feat(kv): attempt at making errors nice * fix(http): return not found error if filter is invalid * fix(http): s/platform/influxdb/ for user service * fix(http): s/platform/influxdb/ for user service * feat(kv): initial port of telegraf configs to kv * feat(kv): initial port of scrapers in bolt to kv * feat(kv): first pass at migrating bolt org service to kv * feat(kv): first pass at bucket service * feat(kv): first pass at migrating kvlog to kv package * feat(kv): add resource op logs * feat(kv): first pass at user resource mapping migration * feat(kv): add urm usage to bucket and org services * feat(kv): first pass at kv authz service * feat(kv): add cascading auth delete for users * feat(kv): first pass d authorizer.OrganizationService in kv * feat(cmd/influxd/launcher): user kv services where appropriate * fix(kv): initialize authorizations * fix(influxdb): use same buckets while slowly migrating stuff * fix(kv): make staticcheck pass * feat(kv): add dashboards to kv review: make suggestions from pr review fix: use common bucket names for bolt/kv stores * test(kv): add complete password test coverage * chore(kv): fixes for staticcheck * feat(kv): implement labels generically on kv * feat(kv): implement macro service * feat(kv): add source service * feat(kv): add session service * feat(kv): add kv secret service * refactor(kv): update telegraf and urm with error messages * feat(kv): add lookup service * feat(kv): add kv onboarding service * refactor(kv): update telegraf to avoid repetition * feat(cmd/influxd): use kv lookup service * feat(kv): add telegraf to lookup service * feat(cmd/influxd): use kv telegraf service * feat(kv): update scraper error messaging * feat(cmd/influxd): add kv scraper * feat(kv): add inmem backend tests * refactor(kv): copy paste errors * refactor(kv): add code to password errors * fix(testing): update error messages for incorrect passwords * feat(kv): rename macro to variable * refactor(kv): auth/bucket/org/user unique checks return errors now * feat(inmem): add way to get all bucket names from store * feat(inmem): Buckets to return slice of bytes rather than strings * feat(inmem): add locks around Buckets to avoid races * feat(cmd/influx): check for unauthorized error in wrapCheckSetup * chore(e2e): add video and screenshot artifcats to gitignore * docs(ci): add build instructions for e2e tests * feat(kv): add id lookup for authorized resources
2019-02-19 23:47:19 +00:00
#### End-to-End Tests
CI also runs end-to-end tests. These test the integration between the `influxd` server the UI.
Since the UI is used by interal repositories as well as the `influxdb` repository, the
end-to-end tests cannot be run on forked pull requests or run locally. The extent of end-to-end
testing required for forked pull requests will be determined as part of the review process.