Data

Data is one group covering connections and datasets, at /v2/data and on the platform’s Data page. A connection is where data comes from. A dataset is an ingested or registered table. Each ingest or sync mints an immutable snapshot (dsv_...), which is what endpoints pin and held-out reports stamp.

Sources

Small files upload directly on the platform or travel inline as tables in a request. Everything else is a connection: registered once, referenced by id or scheme ref.

Register a connection

curl -X POST https://api.schemalabs.ai/v2/data/connect \
  -H "Authorization: Bearer $SCHEMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "source": "postgres://db.example.com:5432/crm/public/contacts",
        "name": "crm_contacts",
        "credentials": { "user": "schema_reader", "password": "…" },
        "options": { "query": "select * from contacts where deleted_at is null" } }'

import os, requests

SCHEMA_API_KEY = os.environ["SCHEMA_API_KEY"]
r = requests.post("https://api.schemalabs.ai/v2/data/connect",
    headers={"Authorization": f"Bearer {SCHEMA_API_KEY}"},
    json={"source": "s3://example-exports/crm/2026/", "name": "crm_exports",
          "credentials": {"access_key_id": "…", "secret_access_key": "…"}})

const SCHEMA_API_KEY = process.env.SCHEMA_API_KEY;
const res = await fetch('https://api.schemalabs.ai/v2/data/connect', {
  method: 'POST',
  headers: { Authorization: `Bearer ${SCHEMA_API_KEY}`, 'Content-Type': 'application/json' },
  body: JSON.stringify({ source: 'rest+https://api.example.com/v1/customers',
    name: 'customers_api', credentials: { bearer: '…' } }),
});

The response returns the connection (conn_...) and the dataset it produced (ds_...) with its first snapshot (dsv_...). Reference it anywhere data is accepted: "data": ["ds_crm01", "snowflake://sales/live"]. A quick run uses it transiently and registers nothing new; endpoint creation pins it. GET /v2/data/:id returns the catalog profile: rows, columns, current snapshot, pin state, and which endpoints pin it. Full family: Data API.

Snapshots and sync

Every ingest and every POST /v2/data/:id/sync mints a new immutable dsv_.... Endpoints and reports reference the exact snapshot they ran on; an endpoint keeps serving its current snapshot until you refresh it, even after a sync has minted a newer one. Schedule syncs from your orchestrator, then refresh dependent endpoints (see Workflows and data stack).

Pinning

Pinning is a state on a registered dataset, set with POST /v2/data/:id/pin and /unpin. Pin state controls the cached rate even when no endpoint exists.

After a sync, the next pass bills added and changed rows fresh, unchanged rows cached, and removed rows nothing. An endpoint keeps serving its pinned snapshot until you refresh it.

Endpoint creation pins its datasets automatically. A dataset pinned by a live endpoint cannot be unpinned or deleted while that endpoint serves; the 409 names the dependent endpoint.

Batch and large runs

Batch is a service tier, elected per job with "options": { "processing": "batch" }. The job bills the batch rate and runs with a completion window, shown on the job’s eta. It applies to inference and synthetic generation.

POST /v2/run
{
  "data": ["snowflake://example/warehouse/events_2025"],
  "target": { "column": "converted" },
  "options": { "processing": "batch", "out": "warehouse://schema.outputs" }
}

202 with a job. The bundle is written to out (a file or warehouse table) on completion, or retrievable from the job when out is omitted and the result fits. A job that misses its window expires unbilled and you resubmit. Batch concurrency per plan: Rate limits.

Synthetic data

POST /v2/data/generate { "sector": "hospital operations", "rows": 5000, "name": "hospital_ops_synth" } produces a synthetic dataset for a sector with realistic per-column ranges, as an ordinary dataset you can run on, pin, or create an endpoint on. Useful when representative data is thin or as a safe test fixture. The response is the new dataset; large generations run as batch jobs. sector uses the same vocabulary the sector output returns; if unsure of the exact name, quick-run a small real table and use its sector.top1.name. Generation is the one operation where the output is the meter (C = cells generated).

Persistence follows the operation

Everything stored is visible under Data, deletable immediately, and counted toward storage. Residency and deployment options: Trust Center.

Deletion

To remove data and its influence entirely: remove the rows at source (or delete the dataset), sync, then refresh dependent endpoints. Purged rows leave the stored snapshot and the pinned context in one step. Data handling terms are in the DPA.

Credentials and audit

Connection credentials are encrypted at rest and are write-only; rotate by re-entering. The audit log records connection, sync, key, and deletion events with actor and timestamp.

Related

Trust Center · Privacy · DPA · Sub-processors · Security