> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ingestly.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Validation
> Validate data with rules, schemas, endpoints, scripts, or AI before passing it downstream.
The validation node checks data against configurable criteria before passing it to downstream nodes. Use it to enforce data quality, ensure required fields are present, verify formats, or run custom validation logic.
Five validation modes are available, from simple field rules to AI-powered checks.
## When to use validation
* You want to catch **bad extractions before delivery**: missing required fields, malformed totals, wrong currency.
* You want to gate runs that should go to [review](/nodes/review) when something looks off. Pair validation `Warn` with a downstream review node.
* You need to enforce a **strict contract** with an external system. Use **Schema** mode with a JSON Schema document.
* You want a programmable check (regex, cross-field math). Use **Script** mode for deterministic logic, **AI** mode when the check is genuinely judgment-based.
* Skip validation when the criteria are simple presence checks already covered by the schema's `required` field on extract.
## Failure actions
Every validation mode uses a **failure action** to control what happens when validation errors are found. The **Validation failures** [Review](/nodes/review) pause mode becomes available as soon as any upstream node is a validation node, with no special failure-action configuration required, so you only choose **Warn** and wire the downstream Review node.
| Action | Behavior |
| -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Fail** | The step fails and the pipeline stops. Errors are reported in the run detail. |
| **Warn** | The step succeeds and the pipeline continues, with validation errors promoted to warnings. When the rules can attribute a failure to specific field paths (**Rules** mode, including expression rules), those paths are recorded on the step output so a downstream [Review](/nodes/review) node configured with the **Validation failures** pause mode pauses the run for a human to correct them. Modes that cannot attribute per-field paths (**Schema**, **Endpoint**, **AI**) just produce warnings. |
## Modes
### Rules
Define field-level validation rules using a visual rule editor. Each rule specifies a **field** path (supports dot notation and array indexing, e.g. `items[0].name`), an **operator**, and an optional **value**. You can also set a custom error message per rule.
#### Operators
| Operator | Group | Description | Value required |
| ---------------- | ---------- | ----------------------------------------------------------- | :------------------: |
| `required` | Presence | Field must exist | No |
| `not_empty` | Presence | Field must not be empty or null | No |
| `equals` | Comparison | Field must equal the value | Yes |
| `not_equals` | Comparison | Field must not equal the value | Yes |
| `gt` | Comparison | Field must be greater than the value | Yes |
| `gte` | Comparison | Field must be greater than or equal to the value | Yes |
| `lt` | Comparison | Field must be less than the value | Yes |
| `lte` | Comparison | Field must be less than or equal to the value | Yes |
| `contains` | String | Field must contain the substring | Yes |
| `not_contains` | String | Field must not contain the substring | Yes |
| `starts_with` | String | Field must start with the value | Yes |
| `ends_with` | String | Field must end with the value | Yes |
| `matches_regex` | String | Field must match the regular expression | Yes |
| `is_email` | Format | Field must be a valid email address | No |
| `is_number` | Format | Field must be a numeric value | No |
| `is_date` | Format | Field must be a valid date | No |
| `is_url` | Format | Field must be a valid URL | No |
| `in_list` | List | Field must be one of the listed values | Yes |
| `not_in_list` | List | Field must not be one of the listed values | Yes |
| `length_min` | Length | Field length must be at least the value | Yes |
| `length_max` | Length | Field length must be at most the value | Yes |
| `length_between` | Length | Field length must be between two values | Yes (two values) |
| `expression` | Expression | A whole-payload arithmetic expression must hold (see below) | Yes (the expression) |
#### Expression rules
The `expression` operator runs a small arithmetic expression against the upstream payload, so you can enforce cross-field invariants that no single-field operator can express. The **Field** input is hidden in the editor; the whole rule is just one expression.
Supported syntax:
| Construct | Example |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Comparison | `subtotal + tax = total`, `discount <= subtotal` |
| Operators | `=`, `==`, `!=`, `>`, `<`, `>=`, `<=` |
| Arithmetic | `+`, `-`, `*`, `/`, parentheses |
| Field reference | `subtotal`, `invoice.total`, `lineItems[0].price` |
| Aggregate functions | `sum({{lineItems[*].amount}}) = total`, `count({{lineItems}}) > 0`, `avg({{lineItems[*].price}}) < 1000`, `min(...)`, `max(...)`, `length({{customerName}}) >= 3` |
`sum`, `avg`, `min`, and `max` require a list of numbers: if their argument plucks a field whose element type is text or yes/no from an array, the pipeline editor flags it as a validation error on the node at save time. `count` accepts a list of any element type, and `length` accepts a list or text.
Numeric values come from JSON numbers directly. Currency-formatted strings like `"$1,234.50"` are also accepted; the evaluator strips common currency markers and thousand separators. If the rule violates, every field referenced by the expression is treated as the affected field (used by the **Warn** action to populate the failing-field list consumed by a downstream Review node's **Validation failures** pause mode).
The expression syntax is independent of the `{{ ... }}` [template syntax](/guides/expressions) used elsewhere. Inside an expression rule, write `subtotal + tax = total`, not `{{subtotal}} + {{tax}} = {{total}}`. The braces are only required around aggregate-function arguments (`sum({{items[*].amount}})`).
### Schema
Validate data against a [JSON Schema](https://json-schema.org/) document. Paste or write your schema in the editor and the node validates the incoming data against it. Useful when you need to enforce a strict contract on the data shape.
### Endpoint
Call an external HTTP endpoint to validate data. The node sends the data to your URL and interprets the response to determine pass or fail.
| Field | Type | Required | Description |
| -------------------- | ------ | -------- | ------------------------------------------------------------ |
| **URL** | string | Yes | The endpoint URL. Supports template substitution. |
| **Method** | select | Yes | HTTP method (GET, POST, PUT, etc.) |
| **Body** | string | No | Request body template |
| **Response mapping** | object | No | Dot-path to `valid` and `errors` fields in the response JSON |
By default, the node looks for `valid` or `isValid` (boolean) and `errors` (string array) at the root of the response body. Use response mapping to point to different paths if your endpoint returns a different shape.
The endpoint request uses a fixed 45-second server timeout; it is not configurable per node. Transient errors (HTTP 429, 502, 503, 504) are retried automatically.
### Script
Write JavaScript validation logic that runs in a sandboxed runtime. Each upstream node is exposed as a JS global with the same shape as `{{nodeName}}` in [templates](/guides/expressions) (`success`, `payload`, `metadata`, `error`). Reference upstream values by node name, for example `extract.payload.email` or `vars.payload.amountLimit`.
**Runtime limits:**
| Limit | Value |
| ----------------- | --------- |
| Execution timeout | 5 seconds |
| Max statements | 10,000 |
| Memory | 16 MB |
The editor provides syntax highlighting, autocomplete, and inline error reporting.
**Return values:**
| Return type | Interpretation |
| ---------------------- | --------------------------------------- |
| `true` | Validation passes |
| `false` | Validation fails (generic error) |
| `"error message"` | Validation fails with the given message |
| `["error1", "error2"]` | Validation fails with multiple errors |
| `undefined` / `null` | Validation passes |
The script must contain a `return` statement. Object return values are not supported.
**Example:**
```javascript theme={null}
const data = extract.payload;
if (!data.email || !data.email.includes('@')) {
return 'Invalid email address';
}
if (data.items && data.items.length === 0) {
return 'At least one item is required';
}
return true;
```
### AI
Use an LLM to validate data against natural language instructions. Write a prompt describing what valid data looks like, and the model returns whether the data passes along with any errors.
| Field | Type | Required | Description |
| ------------- | ------ | -------- | ------------------------------------------------- |
| **Prompt** | text | Yes | Natural language validation instructions |
| **Precision** | select | Yes | Model quality level: `Small`, `Medium`, or `High` |
Higher precision uses a more capable model at a higher credit cost.
## Output format
The validation node passes the upstream data through unchanged as its payload, and the validation diagnostics live in metadata under `metadata.validation`. Downstream nodes reference the data fields directly (no `data` wrapper), for example `{{node.payload.total}}`, and read the diagnostics as `{{node.metadata.validation.isValid}}`.
```json theme={null}
{
"payload": { },
"metadata": {
"validation": {
"isValid": true,
"errors": [],
"warnings": [],
"mode": "rules",
"failureAction": "fail"
}
}
}
```
| Field | Type | Description |
| ----------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------ |
| `payload` | object | The original upstream payload, passed through unchanged. Reference its fields as `{{node.payload.}}`. |
| `metadata.validation.isValid` | boolean | Whether the data passed validation |
| `metadata.validation.errors` | string\[] | Validation error messages (empty when valid) |
| `metadata.validation.warnings` | string\[] | Warning messages (populated when failure action is `warn`) |
| `metadata.validation.mode` | string | The validation mode that was used |
| `metadata.validation.failureAction` | string | The configured failure action (`fail` or `warn`) |
## Inputs and outputs
**Allowed inputs:** extract, merge, transform, filter, split, classify, review, parse, HTTP action, loop, variable, store, prompt.
**Output:** The original data as the payload, with validation diagnostics in `metadata.validation`. When failure action is `fail` and validation errors exist, the step fails and downstream nodes do not execute.
## Common pitfalls
`Fail` halts the pipeline and downstream nodes never run. If you wanted the run to continue but to flag the issue, switch to `Warn` and consume the `metadata.validation.warnings` array downstream.
By default, the endpoint mode looks for `valid` (or `isValid`) and `errors` at the response root. If your service returns a different shape, set **Response mapping** to point at the correct paths or every response will be treated as invalid.
The script must contain a `return` statement. A script that finishes without returning is treated as undefined (which passes validation), so a missing return silently disables the check.
AI validation costs LLM credits per run. If the check is "amount must be a positive number" or "currency must be USD or EUR," use Rules or Script mode for free, deterministic results. Reserve AI mode for genuinely judgment-based checks.
A `required` field marked on the extract schema already gates the field's presence. Don't repeat the same check in validation; use validation for the harder constraints (formats, ranges, cross-field rules).
To send validation failures to a person, set the failure action to **Warn** and add a downstream [Review](/nodes/review) node configured with the **Validation failures** pause mode. Only field-attributable failures (**Rules** mode) populate the field list. **Schema**, **Endpoint**, and **AI** failures produce warnings but no per-field paths, so route those with **Fail** or a manual review trigger instead.
Inside an expression rule, write `subtotal + tax = total`, not `{{subtotal}} + {{tax}} = {{total}}`. The pipeline editor flags template-style expressions at save time.
The pipeline editor checks rule operators and expression types against the upstream schema at save time. A rule that applies `gt` to a non-number, `sum()` to a scalar, or references a field that doesn't exist surfaces as a validation error on the node before you can save. Applying `sum()`, `avg()`, `min()`, or `max()` to a list of non-numbers (a plucked text or yes/no field) is likewise flagged before save, whereas `count()` over the same list is allowed.
## Related
Extract structured data before validating
Add human review for failed validations
Transform validated data to a delivery format
Narrow a document to the pages you validate