---
title: Data · API reference
url: https://docs.schemalabs.ai/api-reference/data
description: Register connections and datasets under /v2/data, sync snapshots, pin data for the cached rate, generate synthetic tables, and delete what you no longer need.
---

# Data

> Register connections and datasets under /v2/data, sync snapshots, pin data for the cached rate, generate synthetic tables, and delete what you no longer need.

Connections and datasets are one group under `/v2/data`. A connection is where data comes from; a dataset is an ingested or registered table, auto-classified and profiled. Each ingest or sync mints an immutable snapshot (`dsv_...`), which is what endpoints pin and held-out reports stamp.

Four families of source: upload (CSV, Excel, JSON), databases (PostgreSQL, MySQL, Supabase, MongoDB, Databricks, Snowflake, Pinecone, Chroma), cloud storage (Google Drive, GCP Storage, AWS S3), and APIs (REST, GraphQL). Small files upload directly through the platform; everything else is a registered connection referenced at run or creation time.

## The data objects

`GET /v2/data` returns both kinds. A dataset carries its current snapshot and pin state; a connection carries its scheme ref. Neither carries analytical outputs (sector, profile): those come from a run and live on results and reports.

**Attributes**

- `id` (string): `ds_...` for a dataset, `conn_...` for a connection. Immutable.
- `kind` (string): Object kind. One of: `dataset`, `connection`.
- `name` (string): Display name.
- `source` (string): Scheme ref for connections (`snowflake://sales/live`, `s3://bucket/exports/`, `postgres://...`, `rest+https://...`); `upload` for direct uploads.
- `snapshot` (string | null): Datasets: the current `dsv_...` snapshot id.
- `snapshot_at` (string | null): Timestamp of the current snapshot.
- `rows` (integer): Datasets: row count of the current snapshot.
- `cols` (integer): Datasets: column count.
- `cells` (integer): Datasets: `rows x cols`, the exact metering size of a fresh pass over it.
- `size_bytes` (integer): Stored size, for the storage line.
- `pinned` (boolean): Datasets: pinned datasets bill the cached rate on repeat calls indefinitely and count toward plan storage.
- `pinned_by` (array of strings): Endpoint ids currently serving this dataset. A dataset pinned by a live endpoint stays pinned until that endpoint is deleted or refreshed onto other data.

```json
{
  "id": "ds_crm01",
  "kind": "dataset",
  "name": "crm_contacts",
  "source": "snowflake://example/sales/public/live_accounts",
  "snapshot": "dsv_91f2",
  "snapshot_at": "2026-08-12T09:31:00Z",
  "rows": 22400,
  "cols": 12,
  "cells": 268800,
  "size_bytes": 3145728,
  "pinned": true,
  "pinned_by": [
    "{endpoint_id}"
  ]
}
```

## Operations

### Connect a source

`POST /v2/data/connect` (scope: `manage`)

Registers a connection to a database, cloud bucket, or API, and ingests an initial snapshot.

Credentials are stored encrypted at rest and are write-only; rotation is by re-entering them. Every connection creation and every sync is recorded in the audit log.

**Body** (application/json)

- `source` (string, required): Scheme ref: the provider scheme followed by the provider’s own path to the table or location, for example `snowflake://...`, `postgres://...`, `s3://bucket/prefix/`, `gdrive://folder-id`, `rest+https://host/path`.
- `name` (string): Display name. Defaults to the last path segment.
- `credentials` (object): Provider-specific credentials (for example `user`, `password`, `role`, `warehouse` for Snowflake; `access_key_id`, `secret_access_key` for S3). Never returned.
- `options` (object): Provider options such as `query` (a SQL statement to materialize as the dataset) or `sheet`.

**Returns**

The connection object and the dataset it produced, with the first snapshot.

**Example request (cURL)**

```bash
curl -X POST 'https://api.schemalabs.ai/v2/data/connect' \
  -H "Authorization: Bearer $SCHEMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "source": "snowflake://acme/sales/public/live_accounts",
  "name": "sales_live",
  "credentials": {
    "user": "schema_reader",
    "password": "••••••••",
    "role": "READER",
    "warehouse": "WH_XS"
  }
}'
```

**Example response (201)**

```json
{
  "connection": {
    "id": "conn_3f9a",
    "kind": "connection",
    "name": "sales_live",
    "source": "snowflake://acme/sales/public/live_accounts"
  },
  "dataset": {
    "id": "ds_sales01",
    "kind": "dataset",
    "name": "live_accounts",
    "source": "snowflake://acme/sales/public/live_accounts",
    "snapshot": "dsv_0c44",
    "rows": 51200,
    "cols": 18,
    "cells": 921600,
    "pinned": false,
    "pinned_by": []
  }
}
```

### List data

`GET /v2/data` (scope: `read`)

Lists connections and datasets with their snapshot and pin state.

**Query parameters**

- `limit` (integer, default 20): Page size.
- `cursor` (string): Cursor from a previous page.

**Returns**

A page of data objects and a `next_cursor`.

**Example request (cURL)**

```bash
curl 'https://api.schemalabs.ai/v2/data?limit=20' \
  -H "Authorization: Bearer $SCHEMA_API_KEY"
```

**Example response (200)**

```json
{
  "datasets": [
    {
      "id": "ds_crm01",
      "kind": "dataset",
      "name": "crm_contacts",
      "source": "snowflake://example/sales/public/live_accounts",
      "snapshot": "dsv_91f2",
      "rows": 22400,
      "cols": 12,
      "cells": 268800,
      "pinned": true,
      "pinned_by": [
        "{endpoint_id}"
      ]
    },
    {
      "id": "ds_bill01",
      "kind": "dataset",
      "name": "billing_db",
      "source": "postgres://billing/live",
      "snapshot": "dsv_c4e7",
      "rows": 22400,
      "cols": 6,
      "cells": 134400,
      "pinned": true,
      "pinned_by": [
        "{endpoint_id}"
      ]
    },
    {
      "id": "ds_claims",
      "kind": "dataset",
      "name": "claims_2025",
      "source": "upload",
      "snapshot": "dsv_77e0",
      "rows": 500000,
      "cols": 30,
      "cells": 15000000,
      "pinned": true,
      "pinned_by": [
        "{other_endpoint_id}"
      ]
    },
    {
      "id": "conn_3f9a",
      "kind": "connection",
      "name": "sales_live",
      "source": "snowflake://acme/sales/public/live_accounts",
      "dataset": "ds_sales01",
      "snapshot": "dsv_1d09"
    }
  ],
  "next_cursor": null,
  "total": 4
}
```

### Retrieve data

`GET /v2/data/:id` (scope: `read`)

Returns a dataset’s profile (rows, columns, current snapshot, pin state) or a connection’s status.

**Path parameters**

- `id` (string, required): `ds_...` or `conn_...`.

**Returns**

The data object.

**Example request (cURL)**

```bash
curl 'https://api.schemalabs.ai/v2/data/ds_crm01' \
  -H "Authorization: Bearer $SCHEMA_API_KEY"
```

**Example response (200)**

```json
{
  "id": "ds_crm01",
  "kind": "dataset",
  "name": "crm_contacts",
  "source": "snowflake://example/sales/public/live_accounts",
  "snapshot": "dsv_91f2",
  "snapshot_at": "2026-08-12T09:31:00Z",
  "rows": 22400,
  "cols": 12,
  "cells": 268800,
  "size_bytes": 3145728,
  "pinned": true,
  "pinned_by": [
    "{endpoint_id}"
  ]
}
```

### Sync a connection

`POST /v2/data/:id/sync` (scope: `manage`)

Re-pulls a connection and mints a new immutable snapshot (`dsv_...`) on its dataset. Endpoints pinned to the dataset keep serving the previous snapshot until you refresh them.

**Path parameters**

- `id` (string, required): `conn_...` or the dataset id it feeds.

**Returns**

`202` with a job for large sources, or `200` with the new snapshot for small ones.

**Example request (cURL)**

```bash
curl -X POST 'https://api.schemalabs.ai/v2/data/conn_3f9a/sync' \
  -H "Authorization: Bearer $SCHEMA_API_KEY"
```

**Example response (200)**

```json
{
  "id": "conn_3f9a",
  "dataset": "ds_sales01",
  "snapshot": "dsv_1d09",
  "previous_snapshot": "dsv_0c44",
  "diff": {
    "rows_added": 310,
    "rows_changed": 12,
    "rows_removed": 4,
    "rows_unchanged": 51184
  },
  "synced_at": "2026-08-16T09:00:03Z"
}
```

> A sync itself is not metered. What a subsequent pass pays follows the diff: added and changed rows bill fresh, unchanged rows bill the cached rate, removed rows bill nothing.

### Pin a dataset

`POST /v2/data/:id/pin` (scope: `manage`)

Pins a dataset: repeat calls over it bill the cached rate indefinitely, and it counts toward plan storage. Endpoint creation pins its datasets automatically.

**Path parameters**

- `id` (string, required): `ds_...`.

**Returns**

The dataset object with `pinned: true`.

**Example request (cURL)**

```bash
curl -X POST 'https://api.schemalabs.ai/v2/data/ds_sales01/pin' \
  -H "Authorization: Bearer $SCHEMA_API_KEY"
```

**Example response (200)**

```json
{
  "id": "ds_sales01",
  "kind": "dataset",
  "pinned": true,
  "pinned_by": []
}
```

### Unpin a dataset

`POST /v2/data/:id/unpin` (scope: `manage`)

Unpins a dataset. The cached rate expires shortly after last use; the next pass bills fresh. A dataset pinned by a live endpoint stays pinned (`409` names the endpoint).

**Path parameters**

- `id` (string, required): `ds_...`.

**Returns**

The dataset object with `pinned: false`, or `409 conflict` naming the dependent endpoint.

**Example request (cURL)**

```bash
curl -X POST 'https://api.schemalabs.ai/v2/data/ds_crm01/unpin' \
  -H "Authorization: Bearer $SCHEMA_API_KEY"
```

**Example response (200)**

```json
{
  "id": "ds_crm01",
  "kind": "dataset",
  "pinned": false
}
```

**Pinned by a live endpoint (409)**

```json
{
  "error": {
    "type": "conflict",
    "message": "dataset ds_crm01 is pinned by live endpoint churn ({endpoint_id}); delete the endpoint or refresh it onto other data first",
    "request_id": "req_2c8f1b"
  }
}
```

### Generate synthetic data

`POST /v2/data/generate` (scope: `manage`)

Produces a synthetic tabular dataset for a sector with realistic per-column ranges. The result is an ordinary dataset (`ds_...`), useful for cold-start when representative data is thin.

**Headers**

- `Idempotency-Key` (string): Optional. A unique key for this request. Retrying a POST with the same key and body never creates a duplicate job; the same key with a different body returns `409 conflict`.

**Body** (application/json)

- `sector` (string, required): Sector name, in the same vocabulary the sector output uses, for example `"hospital operations"` or `"consumer financial services"`.
- `rows` (integer, required): Rows to generate.
- `name` (string): Dataset display name.
- `options` (object): `processing: "batch"` runs the generation as a batch job at the batch rate.

**Returns**

The new dataset: id, name, columns, rows, and the processing block. Large generations run as batch jobs and return `202` with a job.

**Example request (cURL)**

```bash
curl -X POST 'https://api.schemalabs.ai/v2/data/generate' \
  -H "Authorization: Bearer $SCHEMA_API_KEY" \
  -H "Idempotency-Key: <unique key>" \
  -H "Content-Type: application/json" \
  -d '{ "sector": "hospital operations", "rows": 5000, "name": "hospital_ops_synth" }'
```

**Example response (200)**

```json
{
  "dataset": "ds_synth03",
  "name": "hospital_ops_synth.csv",
  "sector": "hospital operations",
  "rows": 5000,
  "columns": [
    "patient_id",
    "age",
    "sex",
    "diagnosis_code",
    "lvef_pct",
    "admission_date"
  ],
  "processing": {
    "mode": "realtime"
  }
}
```

> Metering: `C = cells generated` at the synthetic rate; the output grid is the meter.

### Delete data

`DELETE /v2/data/:id` (scope: `delete`)

Removes a dataset or connection immediately and irreversibly.

A dataset pinned by a live endpoint cannot be deleted: the `409 conflict` names the dependency, so delete the endpoint or refresh it onto other data first.

**Path parameters**

- `id` (string, required): `ds_...` or `conn_...`.

**Returns**

The deleted id and status, or `409 conflict` naming dependent endpoints.

**Example request (cURL)**

```bash
curl -X DELETE 'https://api.schemalabs.ai/v2/data/ds_old01' \
  -H "Authorization: Bearer $SCHEMA_API_KEY"
```

**Example response (200)**

```json
{
  "id": "ds_old01",
  "status": "deleted",
  "deleted_at": "2026-08-16T10:11:02Z"
}
```