diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index a9e4ca78..2a6854d0 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -183,6 +183,7 @@ Always use the left column. Never use the right. | PowerSync Service | powersync service | | Sync Rules | sync rules | | Sync Streams | sync streams | +| Sync Config | sync config (except in code, e.g. `sync_config`, `sync-config.yaml`) | ## Mintlify Components diff --git a/.github/vale/config/vocabularies/PowerSync/accept.txt b/.github/vale/config/vocabularies/PowerSync/accept.txt index c65bdf97..23d5be5d 100644 --- a/.github/vale/config/vocabularies/PowerSync/accept.txt +++ b/.github/vale/config/vocabularies/PowerSync/accept.txt @@ -6,6 +6,7 @@ agnostically allowlisting autoscaling backpressure +Backblaze baz binlog bitwise diff --git a/architecture/powersync-service.mdx b/architecture/powersync-service.mdx index 792266e0..d9564250 100644 --- a/architecture/powersync-service.mdx +++ b/architecture/powersync-service.mdx @@ -64,9 +64,11 @@ When a change occurs in the source database that affects a certain bucket (based The PowerSync Service persists the bucket state in durable storage: there is a pluggable storage layer for bucket data, and MongoDB and Postgres are currently supported as _bucket storage_ databases. The _bucket storage_ database is separate from the connection to your _source database_ (Postgres, MongoDB, MySQL, SQL Server or Convex). Our cloud-hosting offering (PowerSync Cloud) uses MongoDB Atlas as the _bucket storage_ database. +With [S3 bucket storage](/configuration/powersync-service/s3-bucket-storage) (an optional extension for self-hosted deployments), S3 holds larger blocks of ready-to-sync data while MongoDB tells the PowerSync Service where to find them. This can improve initial sync performance for large datasets and reduce read load on MongoDB. Clients continue to connect only to the PowerSync Service. + Persisting the bucket state in a database is also part of how PowerSync achieves high scalability: it means that the PowerSync Service can have a low memory footprint even as you scale to very large volumes of synced data and users/clients. -The layout of that persisted data follows a storage version. That version can change when you deploy new Sync Streams or Sync Rules. That way, large internal collections can evolve without always requiring expensive upfront migrations across your entire dataset. In most cases you do not need to name the storage version in your sync config. The optional `config.storage_version` field is there for situations where you do need explicit control. Examples include preparing for a Service downgrade that only supports an older storage format, testing different storage formats in a non-production environment, or keeping bucket data on an older stable format while you change other parts of the sync config. For more details, see [Storage version](/sync/advanced/compatibility#storage-version). +The layout of that persisted data follows a storage version. That version can change when you deploy new Sync Streams or Sync Rules. That way, large internal collections can evolve without always requiring expensive upfront migrations across your entire dataset. In most cases you do not need to name the storage version in your Sync Config. The optional `config.storage_version` field is there for situations where you do need explicit control. Examples include preparing for a Service downgrade that only supports an older storage format, testing different storage formats in a non-production environment, or keeping bucket data on an older stable format while you change other parts of the Sync Config. For more details, see [Storage version](/sync/advanced/compatibility#storage-version). ## Replication from the Source Database @@ -85,7 +87,9 @@ When the PowerSync Service replicates data from the source database, it: ### Initial Replication vs. Incremental Replication -Whenever a new version of Sync Streams/Rules is deployed, initial replication takes place by means of taking a snapshot of all tables/collections they reference. +The first Sync Config deployment reads all source rows selected by the config. On later Sync Streams deployments, [incremental reprocessing](/sync/advanced/incremental-reprocessing) (currently in Beta; supported only with a MongoDB source database and MongoDB bucket storage) reuses the prepared data for unchanged streams. This can shorten deployment time, reduce source database load, and prevent clients from downloading unchanged data again. For example, changing one independent stream does not make PowerSync read or send the data for every other stream again. + +Without incremental reprocessing, each deployment reads all rows selected by the new config and prepares a complete new copy of the sync data. Clients continue using the current copy until the new one is ready. After that, data is incrementally replicated using a change data capture stream. The specific mechanism depends on the source database type: Postgres logical replication, MongoDB change streams, the MySQL binlog, SQL Server Change Data Capture, or Convex document deltas. diff --git a/client-sdks/advanced/pre-seeded-sqlite.mdx b/client-sdks/advanced/pre-seeded-sqlite.mdx index c7ecf54f..b431f768 100644 --- a/client-sdks/advanced/pre-seeded-sqlite.mdx +++ b/client-sdks/advanced/pre-seeded-sqlite.mdx @@ -22,7 +22,7 @@ If you're interested in seeing an end-to-end example, we've prepared a demo repo ### Generate a Scoped JWT Token In most cases you'd want to pre-seed the SQLite database with user specific data and not all data from the source database, as you normally would when using PowerSync. For this you would need to generate JWT tokens that include the necessary properties to satisfy the conditions of the queries in your Sync Streams (or legacy Sync Rules). -Let's say we have the following sync config: +Let's say we have the following Sync Config: diff --git a/client-sdks/advanced/sequential-id-mapping.mdx b/client-sdks/advanced/sequential-id-mapping.mdx index 04582ddd..d3d5aa57 100644 --- a/client-sdks/advanced/sequential-id-mapping.mdx +++ b/client-sdks/advanced/sequential-id-mapping.mdx @@ -187,7 +187,7 @@ can move on to updating your Sync Streams/Sync Rules to use the UUID column inst ## Update Sync Streams -As sequential IDs can only be created on the backend source database, we need to use UUIDs in the client. The sync config is updated to use the `uuid` column as the `id` column for the `lists` and `todos` tables, explicitly defining which columns to select so that `list_id` (the integer ID) is no longer exposed to the client. +As sequential IDs can only be created on the backend source database, we need to use UUIDs in the client. The Sync Config is updated to use the `uuid` column as the `id` column for the `lists` and `todos` tables, explicitly defining which columns to select so that `list_id` (the integer ID) is no longer exposed to the client. diff --git a/configuration/powersync-service/s3-bucket-storage.mdx b/configuration/powersync-service/s3-bucket-storage.mdx new file mode 100644 index 00000000..082034d9 --- /dev/null +++ b/configuration/powersync-service/s3-bucket-storage.mdx @@ -0,0 +1,188 @@ +--- +title: "S3 Bucket Storage" +description: "Learn when S3 bucket storage can improve initial sync and how to configure it for self-hosted deployments." +--- + +S3 bucket storage is an optional extension to MongoDB bucket storage for self-hosted deployments. PowerSync keeps the metadata and small blocks in MongoDB, and can move larger blocks of prepared sync data to Amazon S3 or an S3-compatible object store. + +## Why Use S3 Bucket Storage? + +During an initial sync, a new or reinstalled client downloads all the data it subscribes to. This usually transfers much more data than later syncs. Reading large blocks from object storage can improve initial sync performance for large datasets. It can also reduce the amount of bucket data stored in and read from MongoDB. + +This benefit applies when data moves from the PowerSync Service to clients. It does not make the first replication from your source database into PowerSync faster. Your client connections and SDK configuration do not change. + +Because object storage becomes part of the sync path, an object-store outage can temporarily interrupt client sync. PowerSync Client SDKs reconnect and resume sync after access recovers. + + + S3 bucket storage stores PowerSync's internal sync data. To store files uploaded by your application, use the separate [Attachments](/client-sdks/advanced/attachments) pattern. + + +## Availability + +S3 bucket storage is [Beta](/resources/feature-status) in PowerSync Service v1.26.0. You can opt in on self-hosted deployments only. On PowerSync Cloud, PowerSync manages bucket storage for you, so there is nothing to configure. + +To opt in, configure MongoDB bucket storage with both `object_storage` and [storage version 4](/sync/advanced/compatibility#storage-version). Version 4 has a stable storage format, but v1.26.0 continues to use version 2 by default. Selecting version 4 without `object_storage` keeps all bucket data in MongoDB. + +**Postgres bucket storage cannot use this feature.** The MongoDB requirement applies only to the bucket storage database. Your source database can be Postgres or any other supported database. + +## How S3 Bucket Storage Works + +PowerSync prepares source data for client sync and groups it into [buckets](/architecture/powersync-service#bucket-system). MongoDB holds this prepared data as the bucket storage database, and S3 bucket storage offloads the larger blocks from it. + +When you enable S3 bucket storage: + +1. MongoDB keeps the information PowerSync uses to find each block of data. It also keeps small blocks. +2. The object store holds larger blocks. +3. When a client syncs, the PowerSync Service uses MongoDB to find the required blocks, reads them from MongoDB or the object store, and sends them to the client. + +Clients connect only to the PowerSync Service. They never connect directly to your object store. + +## Configure S3 Bucket Storage + +### Requirements + +You need: + +- MongoDB as the [bucket storage database](/configuration/powersync-service/self-hosted-instances#bucket-storage-database). Your source database can be any database that PowerSync supports. +- [Storage version 4](/sync/advanced/compatibility#storage-version) for each Sync Config that should use object storage. +- An Amazon S3 bucket or compatible object store that the PowerSync Service can reach. +- Permission to list the bucket and to read, write, and delete objects under the configured prefix. + +You can use S3 bucket storage with Sync Streams or legacy Sync Rules. It does not depend on which type of Sync Config selects the data. + + + + Create a dedicated bucket, or choose a unique `prefix` for each PowerSync instance. This prevents key collisions and prevents one instance's cleanup from deleting another instance's files. Use separate credentials or access policies if the instances must not be able to read each other's files. + + Disable object versioning on the bucket. PowerSync deletes files that it no longer needs. If versioning is enabled, the object store keeps old file versions and continues charging for their storage. + + Give the PowerSync Service permission to list the bucket and to read, write, and delete objects under the configured prefix. + + + + Add `object_storage` to your MongoDB bucket storage configuration. Set `default_storage_version: 4` so that newly deployed Sync Configs use storage version 4. + + ```yaml service.yaml + storage: + type: mongodb + uri: !env PS_MONGO_STORAGE_URI + default_storage_version: 4 + object_storage: + type: s3 + bucket: powersync-bucket-data + region: us-east-1 + prefix: production + defaults_mode: in-region + ``` + + If you omit `access_key_id` and `secret_access_key`, PowerSync uses the standard AWS credentials available to the Service process. Where possible, give the PowerSync workload an IAM role instead of storing long-lived access keys. + + + + Changing `default_storage_version` does not update Sync Configs that are already deployed. Redeploy each existing Sync Config that should use version 4. + + You can also select the storage version in the Sync Config: + + ```yaml sync-config.yaml + config: + edition: 3 + storage_version: 4 + + streams: + user_lists: + auto_subscribe: true + query: SELECT * FROM lists WHERE owner_id = auth.user_id() + ``` + + The first version 4 deployment reprocesses all data selected by that Sync Config. Clients keep using the current data until the new copy is ready. After the switch, they download the data they subscribe to again. Later changes can use [incremental reprocessing](/sync/advanced/incremental-reprocessing) to reuse the data for unchanged streams when its requirements are met. + + + +## Optional Configuration + + + + For another S3-compatible provider, or when your deployment requires static credentials, load the values from environment variables: + + ```yaml service.yaml + storage: + type: mongodb + uri: !env PS_MONGO_STORAGE_URI + default_storage_version: 4 + object_storage: + type: s3 + bucket: powersync-bucket-data + endpoint: !env PS_OBJECT_STORAGE_ENDPOINT + region: us-east-1 + force_path_style: true + access_key_id: !env PS_OBJECT_STORAGE_ACCESS_KEY_ID + secret_access_key: !env PS_OBJECT_STORAGE_SECRET_ACCESS_KEY + ``` + + Set `access_key_id` and `secret_access_key` together. Omit `endpoint` for Amazon S3. + + + + | Option | Required | What it controls | + | --- | --- | --- | + | `type` | Yes | Must be `s3`. | + | `bucket` | Yes | The bucket that stores PowerSync sync data. | + | `region` | No | The bucket region. PowerSync can also read the region from the AWS environment. | + | `prefix` | No | A folder-like path used only by this PowerSync instance. Do not end it with `/`. | + | `endpoint` | No | The URL of an S3-compatible service. Omit it for Amazon S3. | + | `force_path_style` | No | Changes how the bucket name appears in requests. Some S3-compatible services require `true`; Amazon S3 normally uses `false`. | + | `access_key_id` | No | A static access key. Set it with `secret_access_key`, or omit both to use the credentials available to the process. | + | `secret_access_key` | No | The secret for `access_key_id`. | + | `defaults_mode` | No | Adjusts request timeouts for the network distance to object storage. | + | `concurrency_limit` | No | Limits simultaneous object-storage requests. The default is `16`. Keep the default unless monitoring shows that object storage is the bottleneck. | + | `inline_threshold_bytes` | No | Keeps small blocks in MongoDB instead of creating many small objects. The default is `16384` (16 KiB). Keep the default unless testing shows a clear benefit from changing it. | + + + + S3-compatible providers differ in how they address buckets. Use these as starting values and confirm them against your provider's documentation and your bucket configuration: + + | Provider | `endpoint` | `region` | `force_path_style` | + | --- | --- | --- | --- | + | Amazon S3 | Omit | Bucket region | `false` | + | MinIO with local or basic addressing | MinIO endpoint | Usually `us-east-1` | `true` | + | MinIO with wildcard DNS | MinIO endpoint | Usually `us-east-1` | `false` | + | [Cloudflare R2](https://developers.cloudflare.com/r2/api/s3/api/) | Account endpoint | `auto` | `false` | + | [DigitalOcean Spaces](https://docs.digitalocean.com/products/spaces/reference/s3cmd/#test-the-credentials-with-an-aws-sdk) | Regional endpoint | `us-east-1` | `false` | + | [Backblaze B2](https://www.backblaze.com/docs/cloud-storage-call-the-s3-compatible-api) | Regional endpoint | Account region | `false` | + | [Wasabi](https://docs.wasabi.com/apidocs/rest-api-introduction) | Regional endpoint | Bucket region | `true` (recommended) | + + Wasabi supports both path-style and virtual-hosted-style requests, but recommends path-style requests for broader bucket-name compatibility. Backblaze B2 also supports both styles; use `true` for a bucket name containing periods because virtual-hosted-style HTTPS requests do not support those names. + + + + PowerSync stops stalled object-storage operations so that they do not block sync indefinitely. Each mode controls the connection timeout, per-attempt request timeout, complete operation deadline across AWS SDK retries, and time allowed to wait for a concurrency slot. Choose `defaults_mode` based on where PowerSync runs: + + | Value | Use when | + | --- | --- | + | `in-region` | PowerSync and object storage run in the same region. | + | `standard` | You want the default timeout behavior. | + | `cross-region` | PowerSync and object storage run in different regions. | + | `mobile` | The connection has unusually high latency. | + + If you do not set `defaults_mode`, PowerSync uses `AWS_DEFAULTS_MODE` when available and otherwise uses the `standard` timeout profile. A timeout ends that operation and the current sync request. It does not keep one request open for the duration of an object-store outage. + + + +## Monitoring Object Storage + +Enable the [Prometheus metrics endpoint](/maintenance-ops/self-hosting/monitoring) and monitor: + +- `powersync_object_storage_size_bytes` shows how much object-storage data PowerSync currently tracks. +- `powersync_attributed_object_storage_bytes` shows the amount associated with each active or processing Sync Config. + +Do not add the per-config values together to calculate the bucket size. During a deployment, the current and new configs can use some of the same stored data, so that data appears in both values. + +Completed [sync logs](/maintenance-ops/monitoring-and-alerting#instance-logs) include `ms.s3`, the time spent waiting for object storage. If this value grows while MongoDB timings remain stable, check object-store latency, throttling, and the network path from PowerSync. + +## Cleanup and Teardown + +PowerSync deletes files as their sync data is replaced or removed. Keep the scheduled [compacting job](/maintenance-ops/compacting-buckets) running so it can combine small blocks and remove files that are no longer needed. + +The `teardown TEARDOWN` command deletes PowerSync files under `bucket-data/` within the configured prefix, then drops the MongoDB bucket storage database. + +An interrupted write can occasionally leave an unused file behind. Compare `powersync_object_storage_size_bytes` with the size reported by your object-storage provider if physical storage keeps growing after PowerSync's tracked size has stabilized. diff --git a/configuration/powersync-service/self-hosted-instances.mdx b/configuration/powersync-service/self-hosted-instances.mdx index 5025a2fc..dd136b56 100644 --- a/configuration/powersync-service/self-hosted-instances.mdx +++ b/configuration/powersync-service/self-hosted-instances.mdx @@ -187,6 +187,8 @@ If you are rolling your own Docker environment, you can include this init script - 'sleep 10 && mongosh --host mongo:27017 --eval ''try{rs.status().ok && quit(0)} catch {} rs.initiate({_id: "rs0", version: 1, members: [{ _id: 0, host : "mongo:27017" }]})''' ``` +With storage version 4, you can move larger blocks of ready-to-sync data from MongoDB to an S3-compatible object store. This can improve initial sync performance for large datasets and reduce read load on MongoDB. See [S3 Bucket Storage](/configuration/powersync-service/s3-bucket-storage) for requirements and configuration. + ### Postgres Storage Available since version 1.3.8 of the [`powersync-service`](https://hub.docker.com/r/journeyapps/powersync-service), you can use Postgres as an alternative bucket storage database. @@ -336,4 +338,4 @@ See examples here: - \ No newline at end of file + diff --git a/configuration/source-db/setup.mdx b/configuration/source-db/setup.mdx index c6deef56..2a2d6983 100644 --- a/configuration/source-db/setup.mdx +++ b/configuration/source-db/setup.mdx @@ -380,7 +380,7 @@ DocumentDB is not fully MongoDB-compatible. The following behaviors differ from - **Source database only.** DocumentDB is supported as a source database, not as PowerSync's [storage database](/architecture/powersync-service#bucket-storage) (especially relevant if you are self-hosting PowerSync). We don't currently plan to support it for storage. - **Post-images are not supported.** Set [Post Images](#post-images) to **Off** (`post_images: off`, the default); the `auto_configure` and `read_only` modes fail. This does not reduce correctness: DocumentDB always includes the full current document on update events, so updates and deletes still replicate correctly. - **Collection drop and rename are not replicated.** Already-synced rows remain in PowerSync storage under the old name. To recover, redeploy your Sync Streams to trigger a resync. Regular inserts, updates, and deletes are unaffected. -- **Large initial snapshots may not complete on legacy storage.** DocumentDB retains only a limited amount of change-feed history. On storage versions 1 and 2, a large or busy source can age its resume position out of that window before the initial snapshot finishes, and replication restarts from scratch. Use [Storage Version 3](/sync/advanced/compatibility#storage-version) or later, which consumes the change stream while the snapshot is still running. +- **Large initial snapshots may not complete on legacy storage.** DocumentDB keeps source changes for a limited time. On storage versions 1 and 2, PowerSync waits for the initial scan to finish before reading those changes. On a large or busy source, the earliest required changes can expire before the scan finishes, forcing PowerSync to start over. Use [storage version 4](/sync/advanced/compatibility#storage-version), which reads new changes while the initial scan is running. - **Rows of 15 MiB or larger are not synced.** Such rows are dropped with a logged error. This is a general [PowerSync limit](/resources/performance-and-limits), not DocumentDB-specific, but it is more reachable on DocumentDB because the source permits documents up to 16 MiB and every change event carries the full document. - **Large documents replicate very slowly.** DocumentDB delivers large change events much more slowly than standard MongoDB. The document still replicates correctly, but latency spikes around large documents (including updates that change only a small field), and a single large event delays every change behind it. If your workload includes large documents, validate replication latency against your cluster. - **Streaming updates can be delayed by up to roughly 1 second after an idle period.** DocumentDB does not currently support reliable long-polling on its change stream, so PowerSync adds a short, capped delay between polls when the stream is idle. This is a temporary workaround and is expected to be resolved in an upcoming Azure DocumentDB release. diff --git a/docs.json b/docs.json index 82b2af3c..c7cfbba5 100644 --- a/docs.json +++ b/docs.json @@ -147,7 +147,8 @@ "group": "PowerSync Service Setup", "pages": [ "configuration/powersync-service/cloud-instances", - "configuration/powersync-service/self-hosted-instances" + "configuration/powersync-service/self-hosted-instances", + "configuration/powersync-service/s3-bucket-storage" ] }, { @@ -230,6 +231,7 @@ "sync/advanced/client-id", "sync/advanced/case-sensitivity", "sync/advanced/compatibility", + "sync/advanced/incremental-reprocessing", "sync/advanced/sync-data-by-time", "sync/advanced/schemas-and-connections", "sync/advanced/multiple-client-versions", diff --git a/integrations/neon.mdx b/integrations/neon.mdx index 20720b18..f5f9f67f 100644 --- a/integrations/neon.mdx +++ b/integrations/neon.mdx @@ -141,7 +141,7 @@ PowerSync uses logical replication to sync data from your Neon database. 1. In the PowerSync Dashboard, select your project and instance and go to the **Sync Streams** view (shown as **Sync Rules** if using legacy Sync Rules). -2. Edit the sync config in the editor and replace the contents with the below: +2. Edit the Sync Config in the editor and replace the contents with the below: @@ -190,8 +190,8 @@ PowerSync uses logical replication to sync data from your Neon database. -3. Click **"Validate"** and ensure there are no errors. This validates your sync config against your Postgres database. -4. Click **"Deploy"** to deploy your sync config. +3. Click **"Validate"** and ensure there are no errors. This validates your Sync Config against your Postgres database. +4. Click **"Deploy"** to deploy your Sync Config. - For additional information on PowerSync's Sync Streams, refer to the [Sync Streams](/sync/streams/overview) documentation. diff --git a/intro/setup-guide.mdx b/intro/setup-guide.mdx index 4ecd0a20..fecbc518 100644 --- a/intro/setup-guide.mdx +++ b/intro/setup-guide.mdx @@ -206,7 +206,7 @@ PowerSync is available as a cloud-hosted service (PowerSync Cloud) or can be sel powersync init cloud ``` - This creates a `powersync/` directory with `service.yaml` (instance name, region, connection, auth) and `sync-config.yaml` (sync config). Edit `powersync/service.yaml` to set your instance name and region. You'll configure the database connection in the next step. + This creates a `powersync/` directory with `service.yaml` (instance name, region, connection, auth) and `sync-config.yaml` (Sync Config). Edit `powersync/service.yaml` to set your instance name and region. You'll configure the database connection in the next step. Then create the Cloud instance: @@ -226,11 +226,11 @@ PowerSync is available as a cloud-hosted service (PowerSync Cloud) or can be sel powersync docker configure --database postgres --storage postgres ``` - Docker sets up Postgres for both the source database and bucket storage and creates `powersync/docker/docker-compose.yaml`. Other databases are supported as well, you will learn more about this in the next step. Before starting, replace `powersync/sync-config.yaml` with this minimal sync config: + Docker sets up Postgres for both the source database and bucket storage and creates `powersync/docker/docker-compose.yaml`. Other databases are supported as well, you will learn more about this in the next step. Before starting, replace `powersync/sync-config.yaml` with this minimal Sync Config: ```yaml config: - edition: 2 + edition: 3 streams: todos: @@ -559,18 +559,18 @@ streams: - Edit `powersync/sync-config.yaml` with your sync config, then validate and deploy to the linked Cloud instance: + Edit `powersync/sync-config.yaml` with your Sync Config, then validate and deploy to the linked Cloud instance: ```bash powersync validate powersync deploy ``` - This deploys your full config (connection, auth, and sync config). For subsequent sync-only changes, use `powersync deploy sync-config` instead. + This deploys your full config (connection, auth, and Sync Config). For subsequent sync-only changes, use `powersync deploy sync-config` instead. - Edit `powersync/sync-config.yaml` with your sync config. The default file has a placeholder (`SELECT * FROM todos`). Replace it with your actual table/collection names, then apply the changes: + Edit `powersync/sync-config.yaml` with your Sync Config. The default file has a placeholder (`SELECT * FROM todos`). Replace it with your actual table/collection names, then apply the changes: ```bash powersync validate diff --git a/maintenance-ops/compacting-buckets.mdx b/maintenance-ops/compacting-buckets.mdx index 4d8b8090..66fbb942 100644 --- a/maintenance-ops/compacting-buckets.mdx +++ b/maintenance-ops/compacting-buckets.mdx @@ -24,6 +24,8 @@ When using PowerSync Cloud, compact logs are available in the Dashboard's **Logs For self-hosted setups (PowerSync Open Edition & PowerSync Enterprise Self-Hosted Edition), the `compact` command in the Docker image can be used to compact all buckets. This can be run manually, or on a regular schedule using Kubernetes [CronJob](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/) or similar scheduling functionality. +On storage version 4, the experimental `compact --incremental-only` option runs the lower-cost incremental bucket and parameter-index compacting paths. You can schedule this mode more frequently. It skips bucket compacting for Sync Configs on older storage versions, so keep a regular full `compact` job if your instance still has configs on those versions. + [Defragmenting](/maintenance-ops/compacting-buckets#defragmenting) may still be required. ## Background @@ -149,16 +151,10 @@ You should consider defragmenting your buckets when: ### Defragmenting Strategies -There are manual and automated approaches to defragmenting: +There are two mechanisms with different scopes: -1. **Manual Defragmentation** - - Use the [PowerSync Dashboard](https://dashboard.powersync.com/) to manually trigger defragmentation: Select your project and instance, go to the **Settings** view, and click the **Defragment** button in the "Compact operation history" section - - Best for one-time cleanup or after major data changes +1. **Update source rows, then compact.** Use this when you can identify the rows whose old `PUT` operations block compacting. You can target one table or subset of rows, but each update creates a new operation that clients must download. For recurring maintenance, schedule the updates with a tool such as `pg_cron`: -2. **Scheduled Defragmentation** - - Set up a cron job to regularly update rows - - Recommended for frequently updated tables or tables with large churn - - Example using `pg_cron`: ```sql -- Daily defragmentation for high-churn tables UPDATE audit_logs SET last_updated = now() @@ -168,11 +164,12 @@ There are manual and automated approaches to defragmenting: UPDATE users SET last_updated = now() WHERE last_updated < now() - interval '1 week'; ``` - - This will cause clients to re-sync each updated row, while preventing the number of operations from growing indefinitely. Depending on how often rows in the bucket are modified, the interval can be increased or decreased. + +2. **Force a full reprocess.** In the [PowerSync Dashboard](https://dashboard.powersync.com/), select your project and instance, open **Settings**, and use **Defragment** in the "Compact operation history" section. This action forces a new replication stream even when the deployment could otherwise use incremental reprocessing. It rebuilds all prepared sync data and makes clients download all subscribed data again. Use it for one-time cleanup when you intend to rebuild the entire Sync Config. ### Defragmenting Trade-offs -Defragmenting + compacting as described above can significantly reduce the number of operations in a bucket, at the cost of existing clients needing to re-sync that data. When and how to do this depends on the specific use-case and data update patterns. +Both methods can significantly reduce the operation history, at the cost of existing clients downloading data again. Updating source rows and then compacting lets you control which rows are affected. The Dashboard action rebuilds and replaces the complete replication stream. Key considerations: 1. **Frequency**: More frequent defragmentation means fewer operations per sync but more frequent re-syncs @@ -181,10 +178,10 @@ Key considerations: ## Sync Streams Deployments -Whenever modifications to [Sync Streams](/sync/streams/overview) (or legacy [Sync Rules](/sync/rules/overview)) are deployed, all buckets are re-created from scratch. This has a similar effect to fully defragmenting and compacting all buckets. This was recommended as a workaround before explicit compacting became available ([released July 26, 2024](https://releases.powersync.com/announcements/bucket-compacting)). +With [incremental reprocessing](/sync/advanced/incremental-reprocessing) (currently in Beta; supported only with Sync Streams, a MongoDB source database, and MongoDB bucket storage), PowerSync reuses the prepared data for unchanged streams. This reduces reprocessing during the deployment and prevents clients from downloading unchanged data again. Reused buckets also keep their operation history, so a deployment no longer acts like defragmenting every bucket. -Soon, we will use [incremental reprocessing](https://github.com/orgs/powersync-ja/discussions/349) to process changed definitions only. +Without incremental reprocessing, a deployment reads all data selected by the Sync Config and creates new buckets for it. Clients then download that data again. Do not redeploy a Sync Config for routine history cleanup. Use compacting, targeted source-row updates, or the Dashboard's full-reprocess **Defragment** action according to the scope of cleanup you need. ## Technical Details -See the [documentation](https://github.com/powersync-ja/powersync-service/blob/main/docs/storage/compacting-operations.md) in the `powersync-service` repo for more technical details on compacting. \ No newline at end of file +See the [documentation](https://github.com/powersync-ja/powersync-service/blob/main/docs/storage/compacting-operations.md) in the `powersync-service` repo for more technical details on compacting. diff --git a/maintenance-ops/production-readiness-guide.mdx b/maintenance-ops/production-readiness-guide.mdx index ba159833..d3d4c902 100644 --- a/maintenance-ops/production-readiness-guide.mdx +++ b/maintenance-ops/production-readiness-guide.mdx @@ -292,7 +292,7 @@ During a snapshot, PowerSync warns when less than 50% of the WAL budget remains. #### Managing Replication Slots -Under normal operating conditions when new Sync Streams/Sync Rules are deployed to a PowerSync Service instance, a new replication slot will also be created and used for replication. The old replication slot from the previous version of the sync config will still remain, until reprocessing is completed, at which point the old replication slot will be removed by the PowerSync Service. +Under normal operating conditions when new Sync Streams/Sync Rules are deployed to a PowerSync Service instance, a new replication slot will also be created and used for replication. The old replication slot from the previous version of the Sync Config will still remain, until reprocessing is completed, at which point the old replication slot will be removed by the PowerSync Service. However, in some cases, a replication slot may remain without being used. Usually this happens when a PowerSync Service instance is de-provisioned, stopped intentionally or due to unexpected errors. This results in excessive disk usage due to the continued growth of the WAL. To check which replication slots used by a PowerSync Service are no longer active, the following query can be executed against the source Postgres database: diff --git a/maintenance-ops/self-hosting/diagnostics.mdx b/maintenance-ops/self-hosting/diagnostics.mdx index 80607242..dbd08baa 100644 --- a/maintenance-ops/self-hosting/diagnostics.mdx +++ b/maintenance-ops/self-hosting/diagnostics.mdx @@ -45,9 +45,9 @@ The response `data` object contains: **`connections`** — whether PowerSync can reach the configured source database and any connection-level errors. -**`active_sync_rules`** — the currently serving sync config (Sync Streams or Sync Rules). Contains a `connections[]` array with details about each replication connection including slot name, WAL status, and tables being replicated. Also includes an `errors[]` array for warnings or errors. +**`active_sync_rules`** — the currently serving Sync Config (Sync Streams or Sync Rules). Contains a `connections[]` array with details about each replication connection including slot name, WAL status, and tables being replicated. Also includes an `errors[]` array for warnings or errors. -**`deploying_sync_rules`** — only present while a new sync config is being deployed and the initial replication is in progress. PowerSync runs this process in parallel so clients continue to be served by the existing active config. Once initial replication completes, this section disappears and `active_sync_rules` updates. +**`deploying_sync_rules`** — only present while a new Sync Config is being deployed and the initial replication is in progress. PowerSync runs this process in parallel so clients continue to be served by the existing active config. Once initial replication completes, this section disappears and `active_sync_rules` updates. Each connection in `active_sync_rules.connections[]` includes: diff --git a/maintenance-ops/self-hosting/update-sync-rules.mdx b/maintenance-ops/self-hosting/update-sync-rules.mdx index 323d31b1..8c7f5cd2 100644 --- a/maintenance-ops/self-hosting/update-sync-rules.mdx +++ b/maintenance-ops/self-hosting/update-sync-rules.mdx @@ -3,23 +3,23 @@ title: "Update Sync Streams (Sync Config)" description: "Update Sync Streams/Sync Rules in a self-hosted PowerSync deployment." --- -There are three ways to update your sync config in a self-hosted deployment: +There are three ways to update your Sync Config in a self-hosted deployment: 1. **CLI** — Edit your config and apply with `powersync docker reset` 2. **Config file** — Update your config and restart the service 3. **API endpoint** — Deploy at runtime without restarting - During deployment, existing Sync Streams/Sync Rules continue serving clients while new sync config processes. Clients seamlessly transition once [initial replication](/architecture/powersync-service#initial-replication-vs-incremental-replication) completes. + During deployment, existing Sync Streams/Sync Rules continue serving clients while new Sync Config processes. Clients seamlessly transition once [initial replication](/architecture/powersync-service#initial-replication-vs-incremental-replication) completes. - Run `powersync validate` in the CLI before deploying to catch errors in your sync config without applying changes. + Run `powersync validate` in the CLI before deploying to catch errors in your Sync Config without applying changes. ## Option 1: CLI -If you set up PowerSync using the CLI (`powersync docker`), update your sync config and apply it without a full service restart: +If you set up PowerSync using the CLI (`powersync docker`), update your Sync Config and apply it without a full service restart: @@ -34,13 +34,13 @@ If you set up PowerSync using the CLI (`powersync docker`), update your sync con ```bash powersync docker reset ``` - This restarts the PowerSync Service and applies your updated sync config. + This restarts the PowerSync Service and applies your updated Sync Config. ## Option 2: Config File -Define your sync config in `service.yaml` either inline or via a separate file. See [Self-Hosted Instance Configuration](/configuration/powersync-service/self-hosted-instances) for the full config reference. +Define your Sync Config in `service.yaml` either inline or via a separate file. See [Self-Hosted Instance Configuration](/configuration/powersync-service/self-hosted-instances) for the full config reference. @@ -85,13 +85,13 @@ Define your sync config in `service.yaml` either inline or via a separate file. docker compose restart powersync ``` - Once the service starts up, it will load the updated sync config and begin processing it while continuing to serve the existing config until initial replication completes. + Once the service starts up, it will load the updated Sync Config and begin processing it while continuing to serve the existing config until initial replication completes. ## Option 3: Deploy via API -Deploy sync config at runtime without restarting. Useful for quick iterations during development. +Deploy Sync Config at runtime without restarting. Useful for quick iterations during development. The API is disabled when Sync Streams (or legacy Sync Rules) are defined in `service.yaml`. diff --git a/maintenance-ops/self-hosting/usage-reporting.mdx b/maintenance-ops/self-hosting/usage-reporting.mdx index bd7631e1..453b9ef3 100644 --- a/maintenance-ops/self-hosting/usage-reporting.mdx +++ b/maintenance-ops/self-hosting/usage-reporting.mdx @@ -22,10 +22,17 @@ Below are the data points collected every few minutes and associated with a rand | chunks\_replicated\_total | counter | | operations\_synced\_total | counter | | replication\_storage\_size\_bytes | gauge | +| attributed\_source\_records\_bytes | gauge | | operation\_storage\_size\_bytes | gauge | +| attributed\_bucket\_data\_bytes | gauge | | parameter\_storage\_size\_bytes | gauge | +| attributed\_parameter\_indexes\_bytes | gauge | +| object\_storage\_size\_bytes | gauge | +| attributed\_object\_storage\_bytes | gauge | | concurrent\_connections | gauge | +The four `attributed_*` gauges include `sync_config_id` and `sync_config_state` attributes. The Prometheus endpoint also includes `version_label` when the Sync Config has one. The Service removes `version_label` before sharing telemetry with PowerSync. + To scrape your self-hosted PowerSync Service metrics, please see the [Monitoring](/maintenance-ops/self-hosting/monitoring) docs page for more details. ### Opting Out diff --git a/resources/feature-status.mdx b/resources/feature-status.mdx index 5c53dd0e..37fe9e0f 100644 --- a/resources/feature-status.mdx +++ b/resources/feature-status.mdx @@ -60,6 +60,8 @@ Below is a summary of the current main PowerSync features and their release stat | Postgres | GA | | | | | **PowerSync Service** | | +| Incremental Reprocessing | Beta | +| S3 Bucket Storage | Beta | | Open Edition | GA | | Enterprise Self-Hosted | GA | | Sync Streams | GA | @@ -107,4 +109,4 @@ Below is a summary of the current main PowerSync features and their release stat | CLI | Beta | Also see: -- [PowerSync Roadmap](https://roadmap.powersync.com) \ No newline at end of file +- [PowerSync Roadmap](https://roadmap.powersync.com) diff --git a/resources/usage-and-billing/reducing-usage.mdx b/resources/usage-and-billing/reducing-usage.mdx index 6d7f7f55..b5abc366 100644 --- a/resources/usage-and-billing/reducing-usage.mdx +++ b/resources/usage-and-billing/reducing-usage.mdx @@ -57,7 +57,9 @@ PowerSync processes source changes at the row level. Rewriting a row selected by ### Deployments -Deploying Sync Streams recreates the buckets. Existing clients then sync the new buckets, which can temporarily increase data synced even if the underlying rows did not change. Group related configuration changes when practical instead of deploying many small updates. +With [incremental reprocessing](/sync/advanced/incremental-reprocessing) (currently in Beta; supported only with Sync Streams, a MongoDB source database, and MongoDB bucket storage), changing one independent stream does not make clients download every other stream again. Clients keep their data for unchanged streams and download only the data for new or changed streams. + +Without incremental reprocessing, PowerSync prepares a complete new set of sync data after each deployment. Clients then download all data selected by the new config again. In both cases, bucket storage can grow temporarily while PowerSync prepares the change. Group related changes when practical instead of deploying many small updates. ## Reducing Data Hosted diff --git a/resources/usage-and-billing/usage-and-billing-faq.mdx b/resources/usage-and-billing/usage-and-billing-faq.mdx index 9748208b..710be5a5 100644 --- a/resources/usage-and-billing/usage-and-billing-faq.mdx +++ b/resources/usage-and-billing/usage-and-billing-faq.mdx @@ -174,7 +174,7 @@ Each connected `PowerSyncDatabase` counts as a concurrent client. If one user se Some events increase usage temporarily and don't indicate a problem: - New app installations download all their subscribed data, including the operations history, during the initial sync. Expect higher data synced when onboarding new groups of users. A large operations history makes these initial syncs bigger; see [More Operations Than Rows](#more-operations-than-rows). -- Deploying changes to [Sync Streams](/sync/streams/overview) recreates buckets, and existing clients sync the new buckets. Data hosted can also increase temporarily while the previous and new bucket data exist concurrently. See [Deployments](/resources/usage-and-billing/reducing-usage#deployments). [Incremental reprocessing](https://roadmap.powersync.com/c/85-more-efficient-sync-reprocessing) is planned to limit this work to changed definitions. +- With [incremental reprocessing](/sync/advanced/incremental-reprocessing) (currently in Beta; supported only with Sync Streams, a MongoDB source database, and MongoDB bucket storage), changing one independent stream does not make clients download every other stream again. PowerSync keeps the current sync data while it prepares your changes, so data hosted can still increase temporarily. Without incremental reprocessing, PowerSync prepares a complete new set of sync data and clients download all data selected by the new config again. See [Deployments](/resources/usage-and-billing/reducing-usage#deployments). - Defragmentation reduces what new installations download, but existing clients re-sync each touched row. ## Accident Forgiveness diff --git a/sync/advanced/compatibility.mdx b/sync/advanced/compatibility.mdx index 55672bf6..e3e85def 100644 --- a/sync/advanced/compatibility.mdx +++ b/sync/advanced/compatibility.mdx @@ -1,6 +1,6 @@ --- title: "Compatibility" -description: "Configure compatibility editions and bucket storage format version in PowerSync's sync config." +description: "Configure compatibility editions and bucket storage format version in PowerSync's Sync Config." --- To ensure consistency, it is important that the PowerSync Service does not interpret the same source row in different ways after updating to a new version. @@ -10,7 +10,7 @@ At the same time, we want to fix bugs or other inaccuracies that have accumulate To make this trade‑off explicit, you choose whether to keep the existing behavior or turn on newer fixes that slightly change how data is processed. -Use the `config` block in your sync config YAML to choose the behavior. There are two ways to turn fixes on: +Use the `config` block in your Sync Config YAML to choose the behavior. There are two ways to turn fixes on: 1. Set an `edition` to enable the full set of fixes for that edition. This is the recommended approach for new projects. 2. Toggle individual options for more fine‑grained control. @@ -58,16 +58,18 @@ streams: ## Storage Version -The PowerSync Service stores replicated bucket data in [bucket storage](/architecture/powersync-service#bucket-storage). That data uses a storage version that can evolve when you deploy new Sync Streams or Sync Rules. This versioning approach avoids large upfront migrations on existing bucket data when the Service introduces bigger storage changes. Each time your sync config is deployed and processed, the bucket data written for that deployment uses a specific storage version. +A storage version tells the PowerSync Service how to organize prepared sync data in the [bucket storage database](/architecture/powersync-service#bucket-storage). + +Changing the version does not rewrite the current data in place. When you next deploy the Sync Config, PowerSync prepares a new copy using the selected version. Clients continue using the current copy until the new one is ready. This avoids taking the instance offline for a bucket storage migration. ### Optional `config.storage_version` -You can pin the bucket storage version by setting it under the `config` block: +You can choose the bucket storage version in the `config` block: ```yaml config: edition: 3 - storage_version: 2 # version 2 (stable) and 3 (unstable) are currently supported + storage_version: 4 streams: todos: @@ -76,18 +78,36 @@ streams: ### When to Set `storage_version` Explicitly -In most deployments you can omit `storage_version`. The PowerSync Service then uses the latest stable storage version it supports. You should only set this field if you need more control, e.g.: +When you omit `storage_version`, the PowerSync Service uses its configured default. Self-hosted bucket storage defaults to version 2 unless you set `storage.default_storage_version` in `service.yaml`. PowerSync Cloud manages this setting for you. + +Set `storage_version` when you need to: + +- Use [incremental reprocessing](/sync/advanced/incremental-reprocessing) or [S3 bucket storage](/configuration/powersync-service/s3-bucket-storage), which require version 4. +- Keep an existing Sync Config on version 2 while you prepare for the version 4 deployment. +- Prepare for a Service downgrade. Select a version supported by the older Service, deploy the Sync Config, and wait for the new copy to finish before downgrading. + +For self-hosted deployments, you can make version 4 the default for newly deployed Sync Configs: + +```yaml service.yaml +storage: + type: mongodb + uri: !env PS_MONGO_STORAGE_URI + default_storage_version: 4 +``` + +Changing this setting affects only later Sync Config deployments. Redeploy an existing config to move it to version 4. The first version 4 deployment reads all selected source data and prepares a new copy. -1. Service downgrade: If you need to run an older Service version that only supports up to a given storage version, deploy sync config with that `storage_version`, wait until reprocessing for that deployment has finished, then downgrade the Service. -2. Experiments: Opt into an odd, unstable storage version in non-production environments. -3. Delaying a storage upgrade: Change other sync config while keeping bucket data on an older stable storage version until you are ready for the newer format. +### Supported Production Versions -### Stable and Experimental Versions +| Version | Bucket storage | Behavior | +| --- | --- | --- | +| `1` | MongoDB or Postgres | Legacy format retained for existing deployments. | +| `2` | MongoDB or Postgres | Previous stable format. | +| `4` | MongoDB | Supports S3 bucket storage. With Sync Streams and a MongoDB source database, it also supports incremental reprocessing. Postgres bucket storage does not support version 4. | -The service distinguishes stable and experimental storage versions as follows: +A stable format means future Service versions can continue reading data written with that version until it is officially deprecated. Version 1 predates the stable and experimental numbering convention but remains supported for existing deployments. This is separate from feature maturity: incremental reprocessing and S3 bucket storage are currently in [Beta](/resources/feature-status). -- Even numbers (for example `2`, `4`) denote stable formats. Once a stable version is supported, newer Service releases are expected to keep supporting it until it is officially deprecated. -- Odd numbers (for example `3`) denote unstable formats. The layout may change without notice and support may be removed in a future release. Use odd versions only for testing, not production. +Storage version 3 is experimental: its layout can change without notice and support can be removed in a future release. If you previously set `storage_version: 3`, deploy with version `4`, which uses the same format marked stable. ## Supported Fixes diff --git a/sync/advanced/incremental-reprocessing.mdx b/sync/advanced/incremental-reprocessing.mdx new file mode 100644 index 00000000..c4bfe6a9 --- /dev/null +++ b/sync/advanced/incremental-reprocessing.mdx @@ -0,0 +1,104 @@ +--- +title: "Incremental Reprocessing" +description: "Deploy Sync Config changes with less source database processing and fewer repeated client downloads." +--- + +Without incremental reprocessing, every Sync Config deployment reads all data selected by the config from your source database and prepares a complete new copy of the sync data. On large databases this can take a long time, and clients download all data again even when only a small part of the config changed. + +**Incremental reprocessing** reprocesses only the stream definitions that changed. Unchanged streams keep their prepared sync data, so deployments finish faster, the source database does less work, and clients do not download unchanged data again. + +Incremental reprocessing applies to Sync Config deployments. Normal inserts, updates, and deletes flow through ongoing replication and do not reprocess anything. + +## Availability + +Incremental reprocessing is [Beta](/resources/feature-status) in PowerSync Service v1.26.0. It requires a MongoDB source database, MongoDB bucket storage, [Sync Streams](/sync/streams/overview), and [storage version 4](/sync/advanced/compatibility#storage-version). + +During the Beta, deployments must opt in by selecting storage version 4 in the Sync Config. Version 4 has a stable storage format, but v1.26.0 continues to use version 2 by default. + +```yaml +config: + edition: 3 + storage_version: 4 +``` + +Self-hosted deployments can also set `default_storage_version: 4` in the `storage` section of `service.yaml` to apply it to all newly deployed Sync Configs. There is no separate incremental reprocessing setting. After the first version 4 deployment finishes, later eligible Sync Config deployments use incremental reprocessing automatically. + +**Postgres is not supported.** A Postgres source database cannot use incremental reprocessing, even on PowerSync Cloud. Self-hosted deployments also cannot use it with Postgres bucket storage. These configurations continue to reprocess all data selected by a new Sync Config. + +## What Happens During a Deployment + +PowerSync keeps the current Sync Config active while it prepares your changes: + +1. It compares the new config with the current one. +2. It keeps the existing sync data for unchanged streams. +3. It reads the source rows needed for new or changed streams. +4. It switches to the new config after that data is ready. +5. It deletes sync data that the old config no longer needs. + +Clients keep syncing normally during reprocessing and never see a partly processed config. Processing runs alongside ongoing replication, so the current config also stays up to date while PowerSync prepares the new one. + +## What Gets Reprocessed + +Only the streams that changed: + +- Adding a stream reads only the source data that the new stream selects. +- Removing a stream deletes its sync data without reading anything from the source database. +- Renaming a stream is treated as removing the old stream and adding a new one, so its data is rebuilt. +- Changing a stream's queries reprocesses the data for that stream. Other streams are unaffected. + +Within a changed stream, PowerSync reuses existing data where it can. It compares what each query stores rather than the YAML text, so edits that do not change the stored sync data need no reprocessing. For example, changing how a query reads request parameters only changes how PowerSync evaluates the query for each client: + +```sql +SELECT * FROM projects WHERE user_id = auth.user_id() +-- Changing to auth.jwt() ->> 'owner' needs no reprocessing: +SELECT * FROM projects WHERE user_id = auth.jwt() ->> 'owner' +``` + +Changing which data is stored, such as filtering on `owner_id` instead of `user_id`, changes how rows are grouped for sync, so PowerSync rebuilds that stream's data. + +Two exceptions to keep in mind: + +- Queries in the same stream can share prepared data. Adding or changing one query can rebuild data for other queries in that stream. +- PowerSync favors correctness. When it cannot confirm that a change keeps the stored sync data identical, it rebuilds the affected data. A deployment that reprocesses more than you expect is not an error. [Check the logs](#checking-what-a-deployment-rebuilt) to see what was reused. + +## Checking What a Deployment Rebuilt + +After a deployment, the [replication logs](/maintenance-ops/monitoring-and-alerting#instance-logs) summarize the comparison: + +``` +info: Incremental reprocessing sync config update: +Reused definitions: + - type=bucket_data, id=1, name=user_lists|0 + - type=bucket_data, id=2, name=user_projects|0 +New definitions: + - type=bucket_data, id=6, name=user_tasks|0, tables=tasks +Definitions to drop after switching: + - type=bucket_data, id=5, name=user_tasks|0 +``` + +- `Reused definitions` lists data that PowerSync kept. It does not read that data from the source again. +- `New definitions` lists data that PowerSync must read and prepare. +- `Definitions to drop after switching` lists old data that PowerSync removes after the new config becomes active. + +Definitions are more granular than streams. One stream can appear as several entries. Check these entries when a deployment takes longer or causes more source database load than you expect. + +## What Clients Download + +Clients keep their data for unchanged streams and continue from their existing sync position. After the new config becomes active, they download data only for new or rebuilt streams that they subscribe to. + +PowerSync keeps both the old and new versions of changed data while it prepares the deployment, so bucket storage can grow temporarily. The old data is removed after the switch. + +## When PowerSync Rebuilds Everything + +PowerSync reads all data selected by the Sync Config again when it cannot safely reuse the existing copy. This happens when: + +- Your setup does not meet the requirements above. For example, your source database or bucket storage database is not MongoDB, or you use legacy Sync Rules. +- You move an existing Sync Config to storage version 4 for the first time. +- MongoDB has already deleted source changes that PowerSync still needs to read. +- You explicitly restart replication or use the PowerSync Dashboard's **Defragment** action. Both rebuild all sync data instead of reusing it. + +During a full rebuild, clients keep using the current data until the new copy is ready. After the switch, they download all data they subscribe to again. For routine cleanup of bucket operation history, use [bucket compacting](/maintenance-ops/compacting-buckets) instead of redeploying. + +## Custom Checkpoint Events + +The `event_definitions` used by [Custom Write Checkpoints](/handling-writes/custom-write-checkpoints) follow the same rules as stream queries. An unchanged event keeps its existing checkpoint data, while a new or changed event reads its source table again. Clients continue using checkpoints from the current config until the new config is ready. diff --git a/sync/streams/ctes.mdx b/sync/streams/ctes.mdx index 7a00c416..aace566a 100644 --- a/sync/streams/ctes.mdx +++ b/sync/streams/ctes.mdx @@ -3,7 +3,7 @@ title: "Common Table Expressions (CTEs)" description: "Reuse common query patterns across Sync Streams using Common Table Expressions (CTEs) to simplify configurations and reduce duplication." --- -When streams need reusable filtering logic, you can define it once in a Common Table Expression (CTE) and reference it in queries. CTEs can be defined at the stream level (scoped to one stream) or at the top level of your sync config (shared across all streams). This keeps stream definitions DRY and makes it easier to maintain. For the supported syntax of the `with` block and CTE rules, see [Supported SQL — CTE and WITH syntax](/sync/supported-sql#cte-and-with-syntax). +When streams need reusable filtering logic, you can define it once in a Common Table Expression (CTE) and reference it in queries. CTEs can be defined at the stream level (scoped to one stream) or at the top level of your Sync Config (shared across all streams). This keeps stream definitions DRY and makes it easier to maintain. For the supported syntax of the `with` block and CTE rules, see [Supported SQL — CTE and WITH syntax](/sync/supported-sql#cte-and-with-syntax). ## Why Use CTEs diff --git a/sync/streams/migration.mdx b/sync/streams/migration.mdx index 083a5799..8ba4a17e 100644 --- a/sync/streams/migration.mdx +++ b/sync/streams/migration.mdx @@ -41,7 +41,7 @@ If you want "sync everything upfront" behavior (like Sync Rules), set [`auto_sub - PowerSync Service v1.20.0+ (Cloud instances already meet this) - Latest SDK versions with [Rust-based sync client](https://releases.powersync.com/announcements/improved-sync-performance-in-our-client-sdks) (enabled by default on latest SDKs) -- `config: edition: 3` in your sync config +- `config: edition: 3` in your Sync Config @@ -103,7 +103,7 @@ You can generate a Sync Streams draft from your existing Sync Rules in two ways: 1. **Dashboard:** In the [PowerSync Dashboard](https://dashboard.powersync.com/), use the **Migrate to Sync Streams** button. It converts your Sync Rules into a Sync Streams draft that you can review before deploying. -2. **CLI:** Run `powersync migrate sync-rules` to produce a Sync Streams draft from your current sync config. +2. **CLI:** Run `powersync migrate sync-rules` to produce a Sync Streams draft from your current Sync Config. The output uses `auto_subscribe: true` by default, preserving your existing sync-everything-upfront behavior so no client-side changes are required when you first deploy. @@ -236,7 +236,7 @@ const page2 = await db.syncStream('posts', { page_number: 2 }).subscribe(); ## Client-Side Changes -After updating your sync config, update your client code to use subscriptions: +After updating your Sync Config, update your client code to use subscriptions: ```js // Before (Sync Rules with Client Parameters) diff --git a/sync/supported-sql.mdx b/sync/supported-sql.mdx index 870880d3..a99508eb 100644 --- a/sync/supported-sql.mdx +++ b/sync/supported-sql.mdx @@ -298,7 +298,7 @@ For how to use JOINs in your stream queries (when to use them, patterns, and exa Supported in Sync Streams only. Not available in Sync Rules. -Common Table Expressions (CTEs) can be defined in a `with:` block **inside a stream** (stream-level, scoped to that stream) or at the **top level** of the sync config (global, shared across all streams). Each CTE is a name and a single `SELECT` query. The following rules apply: +Common Table Expressions (CTEs) can be defined in a `with:` block **inside a stream** (stream-level, scoped to that stream) or at the **top level** of the Sync Config (global, shared across all streams). Each CTE is a name and a single `SELECT` query. The following rules apply: - **Stream-level CTEs take precedence over global CTEs.** If a stream defines a CTE with the same name as a global CTE, the stream-level definition is used within that stream. - **Global CTE names must not shadow source table names.** If a global CTE has the same name as a database table or collection, PowerSync reports a validation error. Stream-level CTE names are not subject to this restriction. diff --git a/tools/cli.mdx b/tools/cli.mdx index e011c14f..50b3926a 100644 --- a/tools/cli.mdx +++ b/tools/cli.mdx @@ -27,7 +27,7 @@ description: "Manage PowerSync Cloud and self-hosted instances from the command -The PowerSync CLI lets you manage PowerSync Service instances, deploy sync config (your Sync Streams or Sync Rules), generate client schemas, run diagnostics, and more. It is distributed as the [powersync](https://www.npmjs.com/package/powersync) npm package. +The PowerSync CLI lets you manage PowerSync Service instances, deploy Sync Config (your Sync Streams or Sync Rules), generate client schemas, run diagnostics, and more. It is distributed as the [powersync](https://www.npmjs.com/package/powersync) npm package. The CLI is currently in [beta](/resources/feature-status). We recommend it for @@ -38,7 +38,7 @@ The PowerSync CLI lets you manage PowerSync Service instances, deploy sync confi For a full step-by-step flow using the CLI, use the [Setup Guide](/intro/setup-guide): choose the **CLI (Cloud)** or **CLI (Self-Hosted)** tab in steps 2–5 to configure your instance, connect the - source database, deploy sync config, and generate development tokens. + source database, deploy Sync Config, and generate development tokens. The CLI was overhauled in version 0.9.0. The redesign is based on this [design proposal](https://docs.google.com/document/d/1iqpJF2gog2jB-ZWeN8TBEjcad8aBKNKbue2yJ21q_-s/edit). @@ -122,7 +122,7 @@ You can create instances, deploy and pull config, run all Cloud commands. ``` - Edit `powersync/service.yaml` (name, region, replication, auth) and sync config; use `!env` for secrets. + Edit `powersync/service.yaml` (name, region, replication, auth) and Sync Config; use `!env` for secrets. ```bash @@ -253,7 +253,7 @@ Then use the same commands as any self-hosted instance (`powersync status`, `pow | `powersync pull instance --instance-id=` | Download Cloud config to local files | | `powersync deploy` | Deploy full config to linked Cloud instance | | `powersync deploy service-config` | [Cloud] Deploy only service config | -| `powersync deploy sync-config` | [Cloud] Deploy only sync config | +| `powersync deploy sync-config` | [Cloud] Deploy only Sync Config | | `powersync validate` | Validate config and Sync Streams/Rules | | `powersync edit config` | Open Config Studio (Monaco editor) | | `powersync status` | Instance diagnostics (Cloud and self-hosted) | @@ -270,7 +270,7 @@ Run `powersync --help` or `powersync --help` for flags. Full [command ## Deploying From CI (e.g. GitHub Actions) -You can automate sync config (and full config) deployments using the CLI in CI. Use the config directory as the source of truth: keep `service.yaml` and `sync-config.yaml` in the repo (with secrets via `!env` and CI secrets), then run `powersync deploy` (or `powersync deploy sync-config`). +You can automate Sync Config (and full config) deployments using the CLI in CI. Use the config directory as the source of truth: keep `service.yaml` and `sync-config.yaml` in the repo (with secrets via `!env` and CI secrets), then run `powersync deploy` (or `powersync deploy sync-config`). **Secrets:** Set `PS_ADMIN_TOKEN` to your PowerSync personal access token. If the workflow does not use a linked directory, also set `INSTANCE_ID`. For self-hosted, `API_URL` can specify the PowerSync API base URL. @@ -280,7 +280,7 @@ You can automate sync config (and full config) deployments using the CLI in CI. href="https://github.com/powersync-community/github-actions-demo" horizontal > - Example: deploy sync config on push to main + Example: deploy Sync Config on push to main ## Migrating From the Previous CLI diff --git a/tools/local-development.mdx b/tools/local-development.mdx index 4175d7bd..97a6a03c 100644 --- a/tools/local-development.mdx +++ b/tools/local-development.mdx @@ -110,7 +110,7 @@ storage: # The port which the PowerSync API server will listen on port: 8080 -# Points to the sync config file +# Points to the Sync Config file sync_config: path: sync-config.yaml @@ -159,5 +159,5 @@ docker compose up - [PowerSync CLI](https://github.com/powersync-ja/powersync-cli) — open source CLI; use it to scaffold and run a Docker-based local stack - [self-host-demo](https://github.com/powersync-ja/self-host-demo) — complete working examples with Docker Compose - [Self-Hosted Instance Configuration](/configuration/powersync-service/self-hosted-instances) — full `service.yaml` reference -- [Sync Streams](/sync/streams/overview) — sync config syntax +- [Sync Streams](/sync/streams/overview) — Sync Config syntax - [Generate a Development Token](/intro/setup-guide#5-generate-a-development-token) — for testing without a full auth setup diff --git a/tools/powersync-dashboard.mdx b/tools/powersync-dashboard.mdx index fee3033a..1791b01b 100644 --- a/tools/powersync-dashboard.mdx +++ b/tools/powersync-dashboard.mdx @@ -69,7 +69,7 @@ When you navigate to a specific instance, you'll see a left sidebar with various - **Health** - Overview of its connection health, deploy history, replication status, and recently connected clients - **Database Connections** - Configure and manage the source database connection - **Client Auth** - Configure authentication settings -- **Sync Streams / Sync Rules** - Edit, validate, and deploy your sync config. +- **Sync Streams / Sync Rules** - Edit, validate, and deploy your Sync Config. - **Sync Test** - Test your Sync Streams (or legacy Sync Rules) - **Client SDK Setup** - Generate the [client-side schema](/intro/setup-guide#define-your-client-side-schema) based on your deployed [Sync Streams](/sync/streams/overview) or [Sync Rules](/sync/rules/overview) - **Write API** - Resources for exposing the write API endpoint @@ -87,7 +87,7 @@ In the top bar, you'll see a "Connect" button that provides quick access to your Here are some of the most common tasks you'll perform in the dashboard: -- **Edit and deploy Sync Streams / Sync Rules** - Select your project and instance and go to the **Sync Streams** (or legacy **Sync Rules**) view to edit your sync config, then click **"Validate"** and **"Deploy"** to deploy +- **Edit and deploy Sync Streams / Sync Rules** - Select your project and instance and go to the **Sync Streams** (or legacy **Sync Rules**) view to edit your Sync Config, then click **"Validate"** and **"Deploy"** to deploy - **Generate development token** - Navigate to the **Client Auth** and ensure the **Development tokens** setting is checked. Click the "Connect" button in the top bar and follow instructions to generate a [development token](/configuration/auth/development-tokens). - **Launch the Sync Diagnostics Client** - Navigate to the **Sync Test**, generate a development token and click "Launch" to launch the [Sync Diagnostics Client](/tools/diagnostics-client). - **Copy your instance URL** - Click **Connect** in the top bar and copy the instance URL from the dialog. @@ -130,7 +130,7 @@ A custom role is a set of permissions that you choose. Each permission covers on The access levels are as follows: - **View** gives read-only access to that area. The member can see it but cannot change anything. -- **Deploy** applies only to Instances. It gives View access plus the ability to deploy changes to an instance, such as updating the sync config and database connections. It does not allow creating or deleting instances. +- **Deploy** applies only to Instances. It gives View access plus the ability to deploy changes to an instance, such as updating the Sync Config and database connections. It does not allow creating or deleting instances. - **Manage** gives full access to that area, including creating, editing, and deleting. For Instances, it also includes deploying changes. @@ -142,7 +142,7 @@ The table below lists each permission, its available access levels, and what tho | Permission | Access levels | What it grants | | --- | --- | --- | | Projects | View, Manage | View projects. Manage adds creating, renaming, and deleting projects. | -| Instances | View, Deploy, Manage | View instances. Deploy adds deploying changes such as sync config and database connections. Manage adds creating and deleting instances. | +| Instances | View, Deploy, Manage | View instances. Deploy adds deploying changes such as Sync Config and database connections. Manage adds creating and deleting instances. | | Instance logs | View | View instance logs. | | Alert Rules | View, Manage | View Alert Rules. Manage adds creating, editing, and deleting them. | | Notification Rules | View, Manage | View Notification Rules. Manage adds creating, editing, deleting, and testing them. |