---
title: Data
url: https://docs.schemalabs.ai/data
description: Connect sources, sync snapshots, pin datasets for the cached rate, generate synthetic data, run batch jobs, and see what is stored, deleted, and audited.
---

# Data

> Connect sources, sync snapshots, pin datasets for the cached rate, generate synthetic data, run batch jobs, and see what is stored, deleted, and audited.

Data is one group covering connections and datasets, at `/v2/data` and on the platform's Data page. A **connection** is where data comes from. A **dataset** is an ingested or registered table. Each ingest or sync mints an immutable **snapshot** (`dsv_...`), which is what endpoints pin and held-out reports stamp.

## Sources

| Family | Sources | Ref shape |
|---|---|---|
| Upload | CSV, Excel, JSON | `upload` (platform), or inline `tables` in a request |
| Databases | PostgreSQL, MySQL, Supabase, MongoDB, Databricks, Snowflake, Pinecone, Chroma | `postgres://...`, `snowflake://...`, ... |
| Cloud storage | Google Drive, GCP Storage, AWS S3 | `gdrive://...`, `gs://...`, `s3://...` |
| API | REST, GraphQL | `rest+https://...`, `graphql+https://...` |

Small files upload directly on the platform or travel inline as `tables` in a request. Everything else is a connection: registered once, referenced by id or scheme ref.

## Register a connection

cURL:

```bash
curl -X POST https://api.schemalabs.ai/v2/data/connect \
  -H "Authorization: Bearer $SCHEMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "source": "postgres://db.example.com:5432/crm/public/contacts",
        "name": "crm_contacts",
        "credentials": { "user": "schema_reader", "password": "…" },
        "options": { "query": "select * from contacts where deleted_at is null" } }'
```
Python:

```python
import os, requests

SCHEMA_API_KEY = os.environ["SCHEMA_API_KEY"]
r = requests.post("https://api.schemalabs.ai/v2/data/connect",
    headers={"Authorization": f"Bearer {SCHEMA_API_KEY}"},
    json={"source": "s3://example-exports/crm/2026/", "name": "crm_exports",
          "credentials": {"access_key_id": "…", "secret_access_key": "…"}})
```
JavaScript:

```javascript
const SCHEMA_API_KEY = process.env.SCHEMA_API_KEY;
const res = await fetch('https://api.schemalabs.ai/v2/data/connect', {
  method: 'POST',
  headers: { Authorization: `Bearer ${SCHEMA_API_KEY}`, 'Content-Type': 'application/json' },
  body: JSON.stringify({ source: 'rest+https://api.example.com/v1/customers',
    name: 'customers_api', credentials: { bearer: '…' } }),
});
```

The response returns the connection (`conn_...`) and the dataset it produced (`ds_...`) with its first snapshot (`dsv_...`). Reference it anywhere data is accepted: `"data": ["ds_crm01", "snowflake://sales/live"]`. A quick run uses it transiently and registers nothing new; endpoint creation pins it. `GET /v2/data/:id` returns the catalog profile: rows, columns, current snapshot, pin state, and which endpoints pin it. Full family: [Data API](https://docs.schemalabs.ai/api-reference/data).

## Snapshots and sync

Every ingest and every `POST /v2/data/:id/sync` mints a new immutable `dsv_...`. Endpoints and reports reference the exact snapshot they ran on; an endpoint keeps serving its current snapshot until you `refresh` it, even after a sync has minted a newer one. Schedule syncs from your orchestrator, then refresh dependent endpoints (see [Workflows and data stack](https://docs.schemalabs.ai/integrations/data-stack)).

## Pinning

Pinning is a state on a registered dataset, set with `POST /v2/data/:id/pin` and `/unpin`. Pin state controls the cached rate even when no endpoint exists.

| State | Repeat calls over the dataset | Storage | Expiry |
|---|---|---|---|
| Pinned | Cached rate, indefinitely | Counts toward plan storage | Never |
| Unpinned | Cached rate briefly after last use, then fresh | Not counted | Short |

After a sync, the next pass bills added and changed rows fresh, unchanged rows cached, and removed rows nothing. An endpoint keeps serving its pinned snapshot until you refresh it.

Endpoint creation pins its datasets automatically. A dataset pinned by a live endpoint cannot be unpinned or deleted while that endpoint serves; the `409` names the dependent endpoint.

> **Tip: Agents re-querying the same table**
Pin once, then every subsequent quick run that references the dataset bills unchanged rows at the cached rate.

## Batch and large runs

Batch is a service tier, elected per job with `"options": { "processing": "batch" }`. The job bills the batch rate and runs with a completion window, shown on the job's eta. It applies to inference and synthetic generation.

```http
POST /v2/run
{
  "data": ["snowflake://example/warehouse/events_2025"],
  "target": { "column": "converted" },
  "options": { "processing": "batch", "out": "warehouse://schema.outputs" }
}
```

`202` with a job. The bundle is written to `out` (a file or warehouse table) on completion, or retrievable from the job when `out` is omitted and the result fits. A job that misses its window expires unbilled and you resubmit. Batch concurrency per plan: [Rate limits](https://docs.schemalabs.ai/billing#rate-limits).

## Synthetic data

`POST /v2/data/generate { "sector": "hospital operations", "rows": 5000, "name": "hospital_ops_synth" }` produces a synthetic dataset for a sector with realistic per-column ranges, as an ordinary dataset you can run on, pin, or create an endpoint on. Useful when representative data is thin or as a safe test fixture. The response is the new dataset; large generations run as batch jobs. `sector` uses the same vocabulary the sector output returns; if unsure of the exact name, quick-run a small real table and use its `sector.top1.name`. Generation is the one operation where the output is the meter (`C = cells generated`).

## Persistence follows the operation

| Operation | Stored |
|---|---|
| `run`, endpoint calls | No dataset is registered; submitted tables are used for the single pass and discarded. The run's job and run record remain visible in Jobs and Reports. Endpoint logs keep operation events, never row data. |
| Endpoint create, refresh, upgrade | The pinned snapshots, the held-out report with its split seed and partition, the configuration and system prompts. Inline `tables` at creation become an upload-type dataset under Data. |
| Data connect, sync | The dataset snapshot; connection credentials encrypted at rest, write-only. |

Everything stored is visible under Data, deletable immediately, and counted toward storage. Residency and deployment options: [Trust Center](https://www.schemalabs.ai/trust).

## Deletion

| Resource | Policy |
|---|---|
| Datasets and connections | `DELETE /v2/data/:id` is immediate and irreversible. A dataset pinned by a live endpoint is refused with `409 conflict` naming the endpoint: delete the endpoint or refresh it onto other data first. |
| Endpoints | Immediate on `DELETE`. Reports remain in the audit trail; its datasets remain under Data and are unpinned. |
| Reports | Retained for the life of the org. |
| API keys | Revoked immediately with `DELETE /v2/keys/:id`; the key stops working and leaves the listing. |

**To remove data and its influence entirely**: remove the rows at source (or delete the dataset), `sync`, then `refresh` dependent endpoints. Purged rows leave the stored snapshot and the pinned context in one step. Data handling terms are in the [DPA](https://www.schemalabs.ai/dpa).

## Credentials and audit

Connection credentials are encrypted at rest and are write-only; rotate by re-entering. The audit log records connection, sync, key, and deletion events with actor and timestamp.

## Related

[Trust Center](https://www.schemalabs.ai/trust) · [Privacy](https://www.schemalabs.ai/privacy) · [DPA](https://www.schemalabs.ai/dpa) · [Sub-processors](https://www.schemalabs.ai/sub-processors) · [Security](https://www.schemalabs.ai/security)