From 1554f9b45265570d34bc46e3b2e6f5d68d832afc Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Fri, 13 May 2022 17:06:29 -0500 Subject: [PATCH] fix: async writes and deletes (#4015) - Correct handling of writes and deletes in cloud vs. oss (async vs. sync). - Cleanup redundancy in Troubleshooting. - Closes https://github.com/influxdata/DAR/issues/291 Co-authored-by: lwandzura <51929958+lwandzura@users.noreply.github.com> --- .../influxdb/cloud/write-data/troubleshoot.md | 245 +-------------- .../v2.2/write-data/best-practices/_index.md | 2 +- .../influxdb/v2.2/write-data/delete-data.md | 10 + .../influxdb/v2.2/write-data/troubleshoot.md | 292 ++++++++++++++++-- 4 files changed, 274 insertions(+), 275 deletions(-) diff --git a/content/influxdb/cloud/write-data/troubleshoot.md b/content/influxdb/cloud/write-data/troubleshoot.md index 2a8642254..bfc30ea70 100644 --- a/content/influxdb/cloud/write-data/troubleshoot.md +++ b/content/influxdb/cloud/write-data/troubleshoot.md @@ -17,247 +17,4 @@ related: - influxdb/cloud/account-management/limits --- -Learn how to handle and recover from errors when writing to InfluxDB. - -- [Discover common failure scenarios](#discover-common-failure-scenarios) -- [Review HTTP status codes](#review-http-status-codes) -- [Troubleshoot failures](#troubleshoot-failures) -- [Troubleshoot rejected points](#troubleshoot-rejected-points) - -## Discover common failure scenarios - -Write requests made to InfluxDB may fail for a number of reasons. -Common failure scenarios that return an HTTP `4xx` or `5xx` error status code include the following: - -- API token was invalid. See how to [manage API tokens](/influxdb/cloud/security/tokens/). -- Requests exceeded [service quotas](/influxdb/cloud/account-management/limits/#adjustable-service-quotas). -- Payload size exceeded [global limits](/influxdb/cloud/account-management/limits/#global-limits). -- Client or server reached a timeout threshold. -- Data was not formatted correctly. See how to [find parsing errors](#find-parsing-errors) -- Data did not conform to the [explicit bucket schema](/influxdb/cloud/organizations/buckets/bucket-schema/). - See how to resolve [explicit schema rejections](#resolve-explicit-schema-rejections). - -To find the causes of a specific error, [review HTTP status codes](#review-http-status-codes). - -### Troubleshoot partial writes - -Writes may fail partially or completely even though InfluxDB returns an HTTP `2xx` status code for a valid request. -To resolve partial writes and rejected points, see [troubleshoot failures](#troubleshoot-failures). - -## Review HTTP status codes - -InfluxDB uses conventional HTTP status codes to indicate the success or failure of a request. - -{{% warn %}} -#### Asynchronous writes - -`204` indicates InfluxDB validated the request data format. Because data is written to InfluxDB asynchronously, data may not yet be written to a bucket. If some of your data didn't write to the bucket, see how to [check for rejected points](#review-rejected-points). -{{% /warn %}} - -Write requests return the following status codes: - -| HTTP response code | Message | Description | -| :-------------------------------| :--------------------------------------------------------------- | :------------- | -| `204 "Success"` | | If InfluxDB validated the request data format and accepted the data for writing to the bucket | -| `400 "Bad request"` | `message` contains the first malformed line | If data is malformed | -| `401 "Unauthorized"` | | If the [`Authorization: Token` header](/influxdb/cloud/api-guide/api_intro/#authentication) is missing or malformed or if the [API token](/influxdb/cloud/api-guide/api_intro/#authentication) doesn't have [permission](/influxdb/cloud/security/tokens/) to write to the bucket | -| `404 "Not found"` | requested **resource type**, e.g. "organization", and **resource name** | If a requested resource (e.g. organization or bucket) wasn't found. | -| `413 “Request too large”` | cannot read data: points in batch is too large | If a **write** request exceeds the maximum [global limit](/influxdb/cloud/account-management/limits/#global-limits) | -| `429 “Too many requests”` | `Retry-After` header: xxx (seconds to wait before retrying the request) | If a **read** or **write** request exceeds your plan's [adjustable service quotas](/influxdb/cloud/account-management/limits/#adjustable-service-quotas) or if a **delete** request exceeds the maximum [global limit](/influxdb/cloud/account-management/limits/#global-limits) | -| `500 "Internal server error"` | | Default status for an error | -| `503 “Service unavailable“` | Series cardinality exceeds your plan's service quota | If **series cardinality** exceeds your plan's [adjustable service quotas](/influxdb/cloud/account-management/limits/#adjustable-service-quotas) | - -The `message` property of the response body may contain additional details about the error. - -## Troubleshoot failures - -If you notice data is missing in your bucket, do the following: - -- Check the `message` property in the response body for details about the error, e.g. `partial write error` indicates [rejected points](#troubleshoot-rejected-points). -- Check for [rejected points](#troubleshoot-rejected-points) in your organization's `_monitoring` bucket. -- Verify all lines contain valid syntax, e.g. [line protocol](/influxdb/cloud/reference/syntax/line-protocol/) or [CSV](/influxdb/cloud/reference/syntax/annotated-csv/). -- Verify the data types match the [series](/influxdb/cloud/reference/key-concepts/data-elements/#series) or [bucket schema](/influxdb/cloud/organizations/buckets/bucket-schema/). -- Verify the timestamps match the [precision parameter](/influxdb/cloud/write-data/#timestamp-precision). -- Minimize payload size and network errors by [optimizing writes](/influxdb/cloud/write-data/best-practices/optimize-writes/). - -## Troubleshoot rejected points - -InfluxDB may have rejected points even if the HTTP request returned "Success". -InfluxDB logs rejected data points and associated errors to your organization's `_monitoring` bucket. - -- [Review rejected points](#review-rejected-points) - - [Find parsing errors](#find-parsing-errors) - - [Find data type conflicts and schema rejections](#find-data-type-conflicts-and-schema-rejections) -- [Resolve data type conflicts](#resolve-data-type-conflicts) -- [Resolve explicit schema rejections](#resolve-explicit-schema-rejections) - -### Review rejected points - -To get a log of rejected points, query the [`rejected_points` measurement](/influxdb/cloud/reference/internals/system-buckets/#_monitoring-bucket-schema) in your organization's `_monitoring` bucket. -To more quickly locate `rejected_points`, keep the following in mind: - -- If your line protocol batch contains single lines with multiple [fields](/influxdb/cloud/reference/syntax/line-protocol/#field-set), InfluxDB logs an entry for each point (each unique field) that is rejected. -- Each entry contains a `reason` tag that describes why the point was rejected. -- Entries for [data type conflicts and schema rejections](#find-data-type-conflicts-and-schema-rejections) have a `count` field value of `1`. -- Entries for [parsing errors](#find-parsing-errors) contain an `error` field (and don't contain a `count` field). - -#### rejected_points schema - -| Name | Value | -|:------ |:----- | -| `_measurement`| `rejected_points` | -| `_field` | [`count`](#find-data-type-conflicts-and-schema-rejections) or [`error`](#find-parsing-errors) | -| `_value` | [`1`](#find-data-type-conflicts-and-schema-rejections) or [error details](#find-parsing-errors) | -| `bucket` | ID of the bucket that rejected the point | -| `measurement` | Measurement name of the point | -| `field` | Name of the field that caused the rejection | -| `reason` | Brief description of the problem. See specific reasons in [field type conflicts and schema rejections](#find-field-type-conflicts-and-schema-rejections). | -| `gotType` | Received [field](/influxdb/cloud/reference/key-concepts/data-elements/#field-value) type: `Boolean`, `Float`, `Integer`, `String`, or `UnsignedInteger` | -| `wantType` | Expected [field](/influxdb/cloud/reference/key-concepts/data-elements/#field-value) type: `Boolean`, `Float`, `Integer`, `String`, or `UnsignedInteger` | -| `` | Time the rejected point was logged | - -#### Find parsing errors - -If InfluxDB can't parse a line (e.g. due to syntax problems), the response `message` might not provide details. -To find parsing error details, query `rejected_points` entries that contain the `error` field (instead of the `count` field). - -```js -from(bucket: "_monitoring") - |> range(start: -1h) - |> filter(fn: (r) => r._measurement == "rejected_points") - |> filter(fn: (r) => r._field == "error") -``` - -#### Find data type conflicts and schema rejections - -To find `rejected_points` caused by [data type conflicts](#resolve-data-type-conflicts) or [schema rejections](#resolve-explicit-schema-rejections), -query for the `count` field. - -```js -from(bucket: "_monitoring") - |> range(start: -1h) - |> filter(fn: (r) => r._measurement == "rejected_points") - |> filter(fn: (r) => r._field == "count") -``` - -### Resolve data type conflicts - -When you write to a bucket that has the `implicit` schema type, InfluxDB compares new points to points that have the same [series](/influxdb/cloud/reference/key-concepts/data-elements/#series). -If a point has a field with a different data type than the series, InfluxDB rejects the point and logs a `rejected_points` entry. -The `rejected_points` entry contains one of the following reasons: - -| Reason | Meaning | -|:------ |:------- | -| `type conflict in batch write` | The **batch** contains another point with the same series, but one of the fields has a different value type. | -| `type conflict with existing data` | The **bucket** contains another point with the same series, but one of the fields has a different value type. | - -### Resolve explicit schema rejections - -Buckets with the [`explicit` schema type] use [explicit bucket schemas](/influxdb/cloud/organizations/buckets/bucket-schema/). -When you write to a bucket that uses explicit bucket schemas, -InfluxDB rejects data that don't conform to one of the configured schemas. - -Learn how to interpret `rejected_points` logging for [explicit bucket schemas](/influxdb/cloud/organizations/buckets/bucket-schema/). - -- [Detect a measurement mismatch](#detect-a-measurement-mismatch) -- [Detect a field type mismatch](#detect-a-field-type-mismatch) - -##### Detect a measurement mismatch - -InfluxDB rejects a point if the [measurement](/influxdb/cloud/reference/key-concepts/data-elements/#measurement) doesn't match the **name** of a [bucket schema](/influxdb/cloud/organizations/buckets/bucket-schema/). -The `rejected_points` entry contains the following `reason` tag value: - -| Reason | Meaning | -|:------ |:------- -| `measurement not allowed by schema` | The **bucket** is configured to use explicit schemas and none of the schemas matches the **measurement** of the point. | - -Consider the following [line protocol](/influxdb/cloud/reference/syntax/line-protocol) data. - -``` -airSensors,sensorId=TLM0201 temperature=73.97,humidity=35.23,co=0.48 1637014074 -``` - -The line has an `airSensors` measurement and three fields (`temperature`, `humidity`, and `co`). -If you try to write this data to a bucket that has the [`explicit` schema type](/influxdb/cloud/organizations/buckets/bucket-schema/) and doesn't have an `airSensors` schema, the `/api/v2/write` InfluxDB API returns an error and the following data: - -```json -{ - "code": "invalid", - "message": "3 out of 3 points rejected (check rejected_points in your _monitoring bucket for further information)" -} -``` - -InfluxDB logs three `rejected_points` entries, one for each field. - -| _measurement | _field | _value | field | measurement | reason | -|:----------------|:-------|:-------|:------------|:------------|:----------------------------------| -| rejected_points | count | 1 | humidity | airSensors | measurement not allowed by schema | -| rejected_points | count | 1 | co | airSensors | measurement not allowed by schema | -| rejected_points | count | 1 | temperature | airSensors | measurement not allowed by schema | - -##### Detect a field type mismatch - -InfluxDB rejects a point if the [measurement](/influxdb/cloud/reference/key-concepts/data-elements/#measurement) matches the **name** of a bucket schema and the field data types don't match. -The `rejected_points` entry contains the following reason: - -| Reason | Meaning | -|:------------------------------------|:-----------------------------------------------------------------------------------------------------| -| `field type mismatch with schema` | The point has the same measurement as a configured schema and they have different field value types. | - -Consider a bucket that has the following `airSensors` [`explicit bucket schema`](/influxdb/cloud/organizations/buckets/bucket-schema/): - -```json -{ - "name": "airSensors", - "columns": [ - { - "name": "time", - "type": "timestamp" - }, - { - "name": "sensorId", - "type": "tag" - }, - { - "name": "temperature", - "type": "field", - "dataType": "float" - }, - { - "name": "humidity", - "type": "field", - "dataType": "float" - }, - { - "name": "co", - "type": "field", - "dataType": "float" - } - ] -} -``` - -The following [line protocol](/influxdb/cloud/reference/syntax/line-protocol/) data has an `airSensors` measurement, a `sensorId` tag, and three fields (`temperature`, `humidity`, and `co`). - -``` -airSensors,sensorId=L1 temperature=90.5,humidity=70.0,co=0.2 1637014074 -airSensors,sensorId=L1 temperature="90.5",humidity=70.0,co=0.2 1637014074 -``` - -In the example data, one of the points has a `temperature` field value with the _string_ data type. -However, the `airSensors` schema requires `temperature` to have the _float_ data type. -If you try to write the example data to the `airSensors` bucket schema, -InfluxDB returns a `400` error and a message that describes the result: - -```json -{ - "code": "invalid", - "message": "partial write error (5 accepted): 1 out of 6 points rejected (check rejected_points in your _monitoring bucket for further information)" -} -``` - -InfluxDB logs the following `rejected_points` entry to the `_monitoring` bucket: - -| _measurement | _field | _value | bucket | field | gotType | measurement | reason | wantType | -|:------------------|:-------|:-------|:-------------------|:--------------|:---------|:------------|:----------------------------------|:---------| -| rejected_points | count | 1 | a7d5558b880a93da | temperature | String | airSensors | field type mismatch with schema | Float | +{{% duplicate-oss %}} \ No newline at end of file diff --git a/content/influxdb/v2.2/write-data/best-practices/_index.md b/content/influxdb/v2.2/write-data/best-practices/_index.md index c6b52c3f4..7be5fc078 100644 --- a/content/influxdb/v2.2/write-data/best-practices/_index.md +++ b/content/influxdb/v2.2/write-data/best-practices/_index.md @@ -11,7 +11,7 @@ menu: parent: Write data --- -The following articles walk through recommendations and best practices for writing +The following articles provide recommendations and best practices for writing data to InfluxDB. {{< children >}} diff --git a/content/influxdb/v2.2/write-data/delete-data.md b/content/influxdb/v2.2/write-data/delete-data.md index e5974a19a..7e17762bc 100644 --- a/content/influxdb/v2.2/write-data/delete-data.md +++ b/content/influxdb/v2.2/write-data/delete-data.md @@ -37,8 +37,18 @@ InfluxDB {{< current-version >}} does not support deleting data by field. {{% /oss-only %}} +{{% cloud-only %}} +In InfluxDB Cloud, writes and deletes are asynchronous and eventually consistent. +Once InfluxDB validates your request and queues the delete, it sends a _success_ (HTTP `204` status code) response as an acknowledgement. +To ensure that InfluxDB handles writes and deletes in the order you request them, wait for the acknowledgement before you send the next request. +Once InfluxDB executes a queued delete, the deleted data is no longer queryable, +but will remain on disk until the compaction service runs. +{{% /cloud-only %}} + +{{% oss-only %}} Once a delete request completes successfully, the deleted data is no longer queryable, but will remain on disk until the compaction service runs. +{{% /oss-only %}} ## Delete data using the influx CLI diff --git a/content/influxdb/v2.2/write-data/troubleshoot.md b/content/influxdb/v2.2/write-data/troubleshoot.md index 03e80ff45..11fe32e37 100644 --- a/content/influxdb/v2.2/write-data/troubleshoot.md +++ b/content/influxdb/v2.2/write-data/troubleshoot.md @@ -14,41 +14,58 @@ related: - /influxdb/v2.2/reference/internals - /influxdb/v2.2/reference/cli/influx/write --- -Learn how to handle and recover from errors when writing to InfluxDB. +Learn how to avoid unexpected results and recover from errors when writing to InfluxDB. -- [Discover common failure scenarios](#discover-common-failure-scenarios) -- [Review HTTP status codes](#review-http-status-codes) +- [Handle `write` and `delete` responses](#handle-write-and-delete-responses) - [Troubleshoot failures](#troubleshoot-failures) +{{% cloud-only %}} +- [Troubleshoot rejected points](#troubleshoot-rejected-points) -## Discover common failure scenarios +{{% /cloud-only %}} -Write requests made to InfluxDB may fail for a number of reasons. -Common failure scenarios that return an HTTP `4xx` or `5xx` error status code include the following: +## Handle `write` and `delete` responses -- API token was invalid. . See how to [manage API tokens](/influxdb/v2.0/security/tokens/). -- Payload size was too large. -- Client or server reached a timeout threshold. -- Data was not formatted correctly. +{{% cloud-only %}} -To find the causes of a specific error, [review HTTP status codes](#review-http-status-codes). +In InfluxDB Cloud, writes and deletes are asynchronous and eventually consistent. +Once InfluxDB validates your request and [queues](/influxdb/cloud/reference/internals/durability/#backup-on-write) the write or delete, it sends a _success_ (HTTP `204` status code) response as an acknowledgement. +To ensure that InfluxDB handles writes and deletes in the order you request them, wait for the acknowledgement before you send the next request. +Because writes are asynchronous, keep the following in mind: +- Data might not yet be queryable when you receive an HTTP `204` status code. +- InfluxDB may still reject points after responding with an HTTP `204` status code. -### Troubleshoot partial writes +{{% /cloud-only %}} -Writes may fail partially or completely even though InfluxDB returns an HTTP `2xx` status code for a valid request. -For example, a partial write may occur when InfluxDB writes all points that conform to the bucket schema, but rejects points that have the wrong data type in a field. -To resolve partial writes and rejected points, see [troubleshoot failures](#troubleshoot-failures). +{{% oss-only %}} -## Review HTTP status codes +If InfluxDB OSS successfully writes all the request data to the bucket, then it returns success (HTTP `204` status code). +The first rejected point in a batch causes InfluxDB to reject the entire batch and respond with an [HTTP error status](#review-http-status-codes). + +{{% /oss-only %}} + +### Review HTTP status codes InfluxDB uses conventional HTTP status codes to indicate the success or failure of a request. Write requests return the following status codes: -- `204` **Success**: InfluxDB validated the request data format and accepted the data for writing to the bucket. - {{% note %}} - `204` doesn't indicate a successful write operation since writes are asynchronous. - If some of your data did not write to the bucket, see how to [troubleshoot rejected points](#troubleshoot-rejected-points). - {{% /note %}} +{{% cloud-only %}} +| HTTP response code | Message | Description | +| :-------------------------------| :--------------------------------------------------------------- | :------------- | +| `204 "Success"` | | If InfluxDB validated the request data format and queued the data for writing to the bucket | +| `400 "Bad request"` | `message` contains the first malformed line | If data is malformed | +| `401 "Unauthorized"` | | If the [`Authorization: Token` header](/influxdb/cloud/api-guide/api_intro/#authentication) is missing or malformed or if the [API token](/influxdb/cloud/api-guide/api_intro/#authentication) doesn't have [permission](/influxdb/cloud/security/tokens/) to write to the bucket | +| `404 "Not found"` | requested **resource type**, e.g. "organization", and **resource name** | If a requested resource (e.g. organization or bucket) wasn't found. | +| `413 “Request too large”` | cannot read data: points in batch is too large | If a **write** request exceeds the maximum [global limit](/influxdb/cloud/account-management/limits/#global-limits) | +| `429 “Too many requests”` | `Retry-After` header: xxx (seconds to wait before retrying the request) | If a **read** or **write** request exceeds your plan's [adjustable service quotas](/influxdb/cloud/account-management/limits/#adjustable-service-quotas) or if a **delete** request exceeds the maximum [global limit](/influxdb/cloud/account-management/limits/#global-limits) | +| `500 "Internal server error"` | | Default status for an error | +| `503 “Service unavailable“` | Series cardinality exceeds your plan's service quota | If **series cardinality** exceeds your plan's [adjustable service quotas](/influxdb/cloud/account-management/limits/#adjustable-service-quotas) | + +{{% /cloud-only %}} + +{{% oss-only %}} + +- `204` **Success**: All request data was written to the bucket. - `400` **Bad request**: The [line protocol](/influxdb/v2.2/reference/syntax/line-protocol/) data in the request was malformed. The response body contains the first malformed line in the data. All request data was rejected and not written. - `401` **Unauthorized**: May indicate one of the following: @@ -60,25 +77,240 @@ Write requests return the following status codes: - `500` **Internal server error**: Default HTTP status for an error. - `503` **Service unavailable**: Server is temporarily unavailable to accept writes. The `Retry-After` header describes when to try the write again. +{{% /oss-only %}} + The `message` property of the response body may contain additional details about the error. +If some of your data did not write to the bucket, see how to [troubleshoot rejected points](#troubleshoot-rejected-points). + +{{% cloud-only %}} + +### Troubleshoot partial writes + +Because writes are asynchronous, they may fail partially or completely even though InfluxDB returns an HTTP `2xx` status code for a valid request. +For example, a partial write may occur when InfluxDB writes all points that conform to the bucket schema, but rejects points that have the wrong data type in a field. +To check for writes that fail asynchronously, create a [task](/influxdb/cloud/process-data/manage-tasks/) to [check the _monitoring bucket for rejected points](#review-rejected-points). +To resolve partial writes and rejected points, see [troubleshoot failures](#troubleshoot-failures). + +{{% /cloud-only %}} ## Troubleshoot failures +{{% oss-only %}} + If you notice data is missing in your bucket, do the following: -- Check the `message` property in the response body for details about the error -- Verify all lines contain valid syntax, e.g. [line protocol](/influxdb/v2.2/reference/syntax/line-protocol/) or [CSV](/influxdb/v2.2/reference/syntax/annotated-csv/) -- Verify the data types match other data points with the same series. - For example, did you attempt to write `string` data to an `int` field? +- Check the `message` property in the response body for details about the error. +- If the `message` describes a field error, [troubleshoot rejected points](#troubleshoot-rejected-points). +- Verify all lines contain valid syntax ([line protocol](/influxdb/v2.2/reference/syntax/line-protocol/) or [CSV](/influxdb/v2.2/reference/syntax/annotated-csv/)). - Verify the timestamps match the [precision parameter](/influxdb/v2.2/write-data/#timestamp-precision). -- Minimize payload size and network errors by [optimizing writes](/influxdb/v2.2/write-data/best-practices/optimize-writes/) +- Minimize payload size and network errors by [optimizing writes](/influxdb/v2.2/write-data/best-practices/optimize-writes/). -### Troubleshoot rejected points +{{% /oss-only %}} -InfluxDB may reject points even if the HTTP request returns "Success". -If some of your data did not write to the bucket, check for [field type](/influxdb/v2.2/reference/key-concepts/data-elements/#field-value) differences between the missing data point and other points that have the same [series](/influxdb/v2.2/reference/key-concepts/data-elements/#series). -For example, did you attempt to write `string` data to an `int` field? +{{% cloud-only %}} +If you notice data is missing in your bucket, do the following: + +- Check the `message` property in the response body for details about the error--for example, `partial write error` indicates [rejected points](#troubleshoot-rejected-points). +- Check for [rejected points](#troubleshoot-rejected-points) in your organization's `_monitoring` bucket. +- Verify all lines contain valid syntax ([line protocol](/influxdb/cloud/reference/syntax/line-protocol/) or [CSV](/influxdb/cloud/reference/syntax/annotated-csv/)). See how to [find parsing errors](#find-parsing-errors). +- Verify the data types match the [series](/influxdb/cloud/reference/key-concepts/data-elements/#series) or [bucket schema](/influxdb/cloud/organizations/buckets/bucket-schema/). See how to resolve [explicit schema rejections](#resolve-explicit-schema-rejections). +- Verify the timestamps match the [precision parameter](/influxdb/cloud/write-data/#timestamp-precision). +- Minimize payload size and network errors by [optimizing writes](/influxdb/cloud/write-data/best-practices/optimize-writes/). +{{% /cloud-only %}} + +## Troubleshoot rejected points + +{{% oss-only %}} InfluxDB rejects points for the following reasons: + - The **batch** contains another point with the same series, but one of the fields has a different value type. - The **bucket** contains another point with the same series, but one of the fields has a different value type. + +Check for [field type](/influxdb/v2.2/reference/key-concepts/data-elements/#field-value) differences between the missing data point and other points that have the same [series](/influxdb/v2.2/reference/key-concepts/data-elements/#series)--for example, did you attempt to write `string` data to an `int` field? + +{{% /oss-only %}} + +{{% cloud-only %}} + +InfluxDB may have rejected points even if the HTTP request returned "Success". +InfluxDB logs rejected data points and associated errors to your organization's `_monitoring` bucket. + +- [Review rejected points](#review-rejected-points) + - [Find parsing errors](#find-parsing-errors) + - [Find data type conflicts and schema rejections](#find-data-type-conflicts-and-schema-rejections) +- [Resolve data type conflicts](#resolve-data-type-conflicts) +- [Resolve explicit schema rejections](#resolve-explicit-schema-rejections) + +### Review rejected points + +To get a log of rejected points, query the [`rejected_points` measurement](/influxdb/cloud/reference/internals/system-buckets/#_monitoring-bucket-schema) in your organization's `_monitoring` bucket. +To more quickly locate `rejected_points`, keep the following in mind: + +- If your line protocol batch contains single lines with multiple [fields](/influxdb/cloud/reference/syntax/line-protocol/#field-set), InfluxDB logs an entry for each point (each unique field) that is rejected. +- Each entry contains a `reason` tag that describes why the point was rejected. +- Entries for [data type conflicts and schema rejections](#find-data-type-conflicts-and-schema-rejections) have a `count` field value of `1`. +- Entries for [parsing errors](#find-parsing-errors) contain an `error` field (and don't contain a `count` field). + +#### rejected_points schema + +| Name | Value | +|:------ |:----- | +| `_measurement`| `rejected_points` | +| `_field` | [`count`](#find-data-type-conflicts-and-schema-rejections) or [`error`](#find-parsing-errors) | +| `_value` | [`1`](#find-data-type-conflicts-and-schema-rejections) or [error details](#find-parsing-errors) | +| `bucket` | ID of the bucket that rejected the point | +| `measurement` | Measurement name of the point | +| `field` | Name of the field that caused the rejection | +| `reason` | Brief description of the problem. See specific reasons in [field type conflicts and schema rejections](#find-field-type-conflicts-and-schema-rejections). | +| `gotType` | Received [field](/influxdb/cloud/reference/key-concepts/data-elements/#field-value) type: `Boolean`, `Float`, `Integer`, `String`, or `UnsignedInteger` | +| `wantType` | Expected [field](/influxdb/cloud/reference/key-concepts/data-elements/#field-value) type: `Boolean`, `Float`, `Integer`, `String`, or `UnsignedInteger` | +| `` | Time the rejected point was logged | + +#### Find parsing errors + +If InfluxDB can't parse a line (e.g. due to syntax problems), the response `message` might not provide details. +To find parsing error details, query `rejected_points` entries that contain the `error` field (instead of the `count` field). + +```js +from(bucket: "_monitoring") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "rejected_points") + |> filter(fn: (r) => r._field == "error") +``` + +#### Find data type conflicts and schema rejections + +To find `rejected_points` caused by [data type conflicts](#resolve-data-type-conflicts) or [schema rejections](#resolve-explicit-schema-rejections), +query for the `count` field. + +```js +from(bucket: "_monitoring") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "rejected_points") + |> filter(fn: (r) => r._field == "count") +``` + +### Resolve data type conflicts + +When you write to a bucket that has the `implicit` schema type, InfluxDB compares new points to points that have the same [series](/influxdb/cloud/reference/key-concepts/data-elements/#series). +If a point has a field with a different data type than the series, InfluxDB rejects the point and logs a `rejected_points` entry. +The `rejected_points` entry contains one of the following reasons: + +| Reason | Meaning | +|:------ |:------- | +| `type conflict in batch write` | The **batch** contains another point with the same series, but one of the fields has a different value type. | +| `type conflict with existing data` | The **bucket** contains another point with the same series, but one of the fields has a different value type. | + +### Resolve explicit schema rejections + +Buckets with the [`explicit` schema type] use [explicit bucket schemas](/influxdb/cloud/organizations/buckets/bucket-schema/). +When you write to a bucket that uses explicit bucket schemas, +InfluxDB rejects data that don't conform to one of the configured schemas. + +Learn how to interpret `rejected_points` logging for [explicit bucket schemas](/influxdb/cloud/organizations/buckets/bucket-schema/). + +- [Detect a measurement mismatch](#detect-a-measurement-mismatch) +- [Detect a field type mismatch](#detect-a-field-type-mismatch) + +##### Detect a measurement mismatch + +InfluxDB rejects a point if the [measurement](/influxdb/cloud/reference/key-concepts/data-elements/#measurement) doesn't match the **name** of a [bucket schema](/influxdb/cloud/organizations/buckets/bucket-schema/). +The `rejected_points` entry contains the following `reason` tag value: + +| Reason | Meaning | +|:------ |:------- +| `measurement not allowed by schema` | The **bucket** is configured to use explicit schemas and none of the schemas matches the **measurement** of the point. | + +Consider the following [line protocol](/influxdb/cloud/reference/syntax/line-protocol) data. + +``` +airSensors,sensorId=TLM0201 temperature=73.97,humidity=35.23,co=0.48 1637014074 +``` + +The line has an `airSensors` measurement and three fields (`temperature`, `humidity`, and `co`). +If you try to write this data to a bucket that has the [`explicit` schema type](/influxdb/cloud/organizations/buckets/bucket-schema/) and doesn't have an `airSensors` schema, the `/api/v2/write` InfluxDB API returns an error and the following data: + +```json +{ + "code": "invalid", + "message": "3 out of 3 points rejected (check rejected_points in your _monitoring bucket for further information)" +} +``` + +InfluxDB logs three `rejected_points` entries, one for each field. + +| _measurement | _field | _value | field | measurement | reason | +|:----------------|:-------|:-------|:------------|:------------|:----------------------------------| +| rejected_points | count | 1 | humidity | airSensors | measurement not allowed by schema | +| rejected_points | count | 1 | co | airSensors | measurement not allowed by schema | +| rejected_points | count | 1 | temperature | airSensors | measurement not allowed by schema | + +##### Detect a field type mismatch + +InfluxDB rejects a point if the [measurement](/influxdb/cloud/reference/key-concepts/data-elements/#measurement) matches the **name** of a bucket schema and the field data types don't match. +The `rejected_points` entry contains the following reason: + +| Reason | Meaning | +|:------------------------------------|:-----------------------------------------------------------------------------------------------------| +| `field type mismatch with schema` | The point has the same measurement as a configured schema and they have different field value types. | + +Consider a bucket that has the following `airSensors` [`explicit bucket schema`](/influxdb/cloud/organizations/buckets/bucket-schema/): + +```json +{ + "name": "airSensors", + "columns": [ + { + "name": "time", + "type": "timestamp" + }, + { + "name": "sensorId", + "type": "tag" + }, + { + "name": "temperature", + "type": "field", + "dataType": "float" + }, + { + "name": "humidity", + "type": "field", + "dataType": "float" + }, + { + "name": "co", + "type": "field", + "dataType": "float" + } + ] +} +``` + +The following [line protocol](/influxdb/cloud/reference/syntax/line-protocol/) data has an `airSensors` measurement, a `sensorId` tag, and three fields (`temperature`, `humidity`, and `co`). + +``` +airSensors,sensorId=L1 temperature=90.5,humidity=70.0,co=0.2 1637014074 +airSensors,sensorId=L1 temperature="90.5",humidity=70.0,co=0.2 1637014074 +``` + +In the example data, one of the points has a `temperature` field value with the _string_ data type. +However, the `airSensors` schema requires `temperature` to have the _float_ data type. +If you try to write the example data to the `airSensors` bucket schema, +InfluxDB returns a `400` error and a message that describes the result: + +```json +{ + "code": "invalid", + "message": "partial write error (5 accepted): 1 out of 6 points rejected (check rejected_points in your _monitoring bucket for further information)" +} +``` + +InfluxDB logs the following `rejected_points` entry to the `_monitoring` bucket: + +| _measurement | _field | _value | bucket | field | gotType | measurement | reason | wantType | +|:------------------|:-------|:-------|:-------------------|:--------------|:---------|:------------|:----------------------------------|:---------| +| rejected_points | count | 1 | a7d5558b880a93da | temperature | String | airSensors | field type mismatch with schema | Float | + +{{% /cloud-only %}} \ No newline at end of file