diff --git a/CHANGELOG.md b/CHANGELOG.md index ac398ce..f72713f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,23 @@ All notable changes to the Apify Java client are documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.14.0] - 2026-11-10 + +### Added + +- `ZstdHttpCompressor` for `Content-Encoding: zstd` request bodies (optional `com.github.luben:zstd-jni` dependency). +- `ActorServerRequestOptions.parentRequestId(String)`, sent as the `X-Actor-Request-ID` header so a called multi-tenant Server Actor charges at the pricing tier of the incoming request. + +### Fixed + +- `StreamedLog.stop()` and `close()` no longer interrupt the reader thread, so a log consumer that is running or blocked is not interrupted and does not lose its message. +- `StreamedLog.close()` called from the log consumer no longer deadlocks while another thread is stopping the same `StreamedLog`, and `stop()` or `close()` from an interrupted thread still ends the reader. + +### Changed + +- Updated to Apify OpenAPI specification `v2-2026-11-09T143926Z`: `zstd` is an accepted `Content-Encoding`. +- Documented that `RunChargeOptions.requestId` is ignored by runs of single-tenant Server Actors. + ## [0.13.0] - 2026-11-06 ### Added @@ -485,7 +502,7 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). `setStatusMessage` to the README resources table; noted `getExtra()` on every thin response model. - Fixed a contradiction in `README.md`/`docs/README.md`: both described `docs/examples.md` as "complete, runnable programs," but its own snippets are fragments in the same style as the - resource pages — the complete, runnable programs live under + resource pages; the complete, runnable programs live under `src/test/java/com/apify/client/examples/`. Reworded both to match. Added the missing `WebhookLastDispatch` import to the `com.apify.client.webhook` package-table row. Corrected the "one raw-JSON field" framing in `docs/README.md`'s "Raw JSON values" section: @@ -600,8 +617,8 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). key-value-store, request-queue, task, schedule, webhook, and webhook-dispatch collection clients; and `DatasetClient.iterateItems(...)` for dataset items and `KeyValueStoreClient.iterateKeys(...)` for store keys. The options' `limit` caps the total number of items yielded and `chunkSize` sets - the per-request page size. The non-paginated collections — `ActorVersionCollectionClient.iterate(options)` - and `ActorEnvVarCollectionClient.iterate()` — return the full list in a single fetch (no page size + the per-request page size. The non-paginated collections, `ActorVersionCollectionClient.iterate(options)` + and `ActorEnvVarCollectionClient.iterate()`, return the full list in a single fetch (no page size to tune). ### Changed diff --git a/README.md b/README.md index b468df7..bc9f6b1 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Apify API client for Java -> **Official, but experimental — AI-generated and AI-maintained.** This is an official Apify client, +> **Official, but experimental; AI-generated and AI-maintained.** This is an official Apify client, > but it is experimental: it is generated and maintained by AI. Review the code before relying on it > in production and report issues on the repository. @@ -23,11 +23,11 @@ Maven (Maven Central is a default repository, so no extra configuration is neede com.apify apify-client - 0.13.0 + 0.14.0 ``` -Gradle — ensure `mavenCentral()` is in your `repositories`, then add the dependency: +Gradle; ensure `mavenCentral()` is in your `repositories`, then add the dependency: ```groovy repositories { @@ -35,7 +35,7 @@ repositories { } dependencies { - implementation 'com.apify:apify-client:0.13.0' + implementation 'com.apify:apify-client:0.14.0' } ``` @@ -58,18 +58,18 @@ class HelloApify { } ``` -`ApifyClient.create` takes the token as an explicit argument — it does **not** read `APIFY_TOKEN` (or +`ApifyClient.create` takes the token as an explicit argument; it does **not** read `APIFY_TOKEN` (or any other environment variable) automatically. Read it yourself if you want that, e.g. `ApifyClient.create(System.getenv("APIFY_TOKEN"))`. All public client types live under `com.apify.client`, split by resource into sub-packages (e.g. -`com.apify.client.run.ActorRun`, `com.apify.client.dataset.DatasetListItemsOptions`) — see +`com.apify.client.run.ActorRun`, `com.apify.client.dataset.DatasetListItemsOptions`); see [Resources](#resources) below for the full list; [`docs/README.md`](docs/README.md#imports-and-dependencies) enumerates the model/option-type packages. The Quick start example above is a complete, runnable program (imports, class, `main`); every other snippet in this file, from here on, is a fragment -that assumes a configured `client` and the correct imports for the types it uses — not, by itself, -a complete program — the [resource pages](docs/README.md) show the same kind of fragment per -method. Reading items from a run's default dataset, chaining the two asynchronous calls: +that assumes a configured `client` and the correct imports for the types it uses, not a complete +program. The [resource pages](docs/README.md) show the same kind of fragment per method. Reading +items from a run's default dataset, chaining the two asynchronous calls: ```java client.actor("apify/hello-world").call(null, new ActorStartOptions(), 120L) @@ -79,8 +79,8 @@ client.actor("apify/hello-world").call(null, new ActorStartOptions(), 120L) .join(); ``` -The types used above — `PaginationList` (root package), `DatasetListItemsOptions` -(`com.apify.client.dataset`), and the per-resource clients — are documented on the +The types used above, `PaginationList` (root package), `DatasetListItemsOptions` +(`com.apify.client.dataset`), and the per-resource clients, are documented on the [resource pages](docs/README.md); `ApifyApiException` (`com.apify.client.http`) is covered under [Error handling](#error-handling) below. [`docs/examples.md`](docs/examples.md) has more fragments in the same style (build-and-run, storages, log redirection, and more); the complete, runnable @@ -94,7 +94,7 @@ Every network-calling method returns a (`java.util.concurrent.CompletableFuture`) instead of blocking the calling thread on the HTTP exchange, and every paginated collection's `iterate(...)` returns a [`Flow.Publisher`](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/concurrent/Flow.Publisher.html) -(`java.util.concurrent.Flow.Publisher`) — the JDK's built-in Reactive Streams type — instead of a +(`java.util.concurrent.Flow.Publisher`), the JDK's built-in Reactive Streams type, instead of a blocking `Iterator`, so following a large collection never blocks a thread waiting on the next page's network round-trip. @@ -102,13 +102,13 @@ page's network round-trip. keep a pipeline fully asynchronous end to end. - **Block for a result** with `future.join()` (unchecked) or `future.get()` (checked, `InterruptedException`/`ExecutionException`) when you just want a synchronous-looking call site - (e.g. a `main` method, a test, or a simple script) — every snippet in this documentation that + (e.g. a `main` method, a test, or a simple script); every snippet in this documentation that needs a value calls `.join()` for exactly that reason. - **Consume an `iterate(...)` publisher** either by implementing `Flow.Subscriber` yourself (`onSubscribe` → `request(n)` to pull `n` items, `onNext` per item, `onComplete`/`onError` to finish) for real backpressure, or with the small convenience bridge `com.apify.client.Publishers.collect(publisher)`, which subscribes with unbounded demand and - returns a `CompletableFuture>` — useful when you just want "give me everything" and don't + returns a `CompletableFuture>`: useful when you just want "give me everything" and don't need backpressure: ```java @@ -118,7 +118,7 @@ page's network round-trip. A future/publisher's failure mode is unchanged from before: any `ApifyApiException`/ `ApifyTransportException` this client throws now arrives as the future's exceptional completion (surfaced by `.join()`/`.get()` as a `CompletionException`/`ExecutionException` wrapping the -original, still-unchecked exception) instead of a direct `throw` — see +original, still-unchecked exception) instead of a direct `throw`; see [Error handling](#error-handling) below for the full exception hierarchy, which is otherwise unchanged. @@ -162,7 +162,7 @@ validated at call time (see below), not deferred to `build()`. | `timeoutMedium(Duration)` | `Duration` | `30s` | non-null, strictly `> 0` | | `timeoutLong(Duration)` | `Duration` | `360s` | non-null, strictly `> 0` | | `timeoutMax(Duration)` | `Duration` | `360s` | non-null, strictly `> 0` | -| `compression(HttpCompressor)` | `HttpCompressor` | `BrotliHttpCompressor` (quality 6), or `GzipHttpCompressor` where the brotli native library is unavailable | non-null | +| `compression(HttpCompressor)` | `HttpCompressor` | `BrotliHttpCompressor` (quality 6), or `GzipHttpCompressor` where the brotli native library is unavailable; `ZstdHttpCompressor` is opt-in | non-null | | `header(String, String)` | `String`, `String` | none | non-blank name, non-null value | | `userAgentSuffix(String)` | `String` | none (no suffix appended) | any string | | `httpTransport(HttpTransport)` | `HttpTransport` | `new DefaultHttpTransport()` | non-null | @@ -242,10 +242,27 @@ already did: a started run keeps running (abort it with `RunClient.abort(...)`). Request bodies of at least 1 KiB with a compressible content type are compressed with the configured `HttpCompressor`: `BrotliHttpCompressor` (quality 0-11, default 6; used by default where -its native library is available) or `GzipHttpCompressor` (level 1-9, default 6), or your own -implementation of the interface. A failing compressor fails the request; the client does not fall -back to another coding. A body that already carries a `Content-Encoding`, and already-compressed -content types (images, archives, ...), are sent as they are. +its native library is available), `GzipHttpCompressor` (level 1-9, default 6), `ZstdHttpCompressor` +(level 1-19, default 3), or your own implementation of the interface. A failing compressor fails +the request; the client does not fall back to another coding. A body that already carries a +`Content-Encoding`, and already-compressed content types (images, archives, ...), are sent as they +are. + +`ZstdHttpCompressor` is opt-in and needs the optional `zstd-jni` dependency, which bundles its +native libraries. On the module path, also add `requires com.github.luben.zstd_jni;`. + +```xml + + com.github.luben + zstd-jni + 1.5.7-4 + +``` + +```java +ApifyClient client = + ApifyClient.builder().token("MY-APIFY-TOKEN").compression(new ZstdHttpCompressor()).build(); +``` `header(name, value)` adds a header to every request, such as an integration platform marker. It replaces a built-in header of the same name (`User-Agent`, `Authorization`) and is replaced by a @@ -253,7 +270,7 @@ header a request sets itself. `publicBaseUrl` does not affect ordinary API requests (those always use `baseUrl`); it only changes the origin embedded in the handful of public, directly-fetchable URLs the client builds without a -request — `DatasetClient.createItemsPublicUrl(...)` and `KeyValueStoreClient.getRecordPublicUrl(...)`/ +request: `DatasetClient.createItemsPublicUrl(...)` and `KeyValueStoreClient.getRecordPublicUrl(...)`/ `createKeysPublicUrl(...)` (see [Storages](docs/storages.md)). Set it when the API is reached through one origin (e.g. an internal proxy) but those public URLs must point at a different, externally-reachable one. @@ -272,7 +289,7 @@ ApifyClient withTransport = ApifyClient.builder().token("t").httpTransport(trans `DefaultHttpTransport` also has a `DefaultHttpTransport(Duration connectTimeout)` constructor that builds its own JDK `HttpClient` with a custom connection-establishment timeout (default 10s, `DefaultHttpTransport.DEFAULT_CONNECT_TIMEOUT`) without requiring you to construct the `HttpClient` -yourself — use it when you only want to change the connect timeout and don't otherwise need to +yourself; use it when you only want to change the connect timeout and don't otherwise need to customize the JDK client: ```java @@ -287,7 +304,7 @@ exponential backoff and jitter on `429`, `5xx` and network errors. A custom implementation (`com.apify.client.http.HttpTransport`) must provide the two methods below, both non-blocking. `HttpRequest` and `HttpResponse` below are the JDK's `java.net.http.HttpRequest`/ -`java.net.http.HttpResponse` (not custom client types) — same as `Duration` above: +`java.net.http.HttpResponse` (not custom client types); same as `Duration` above: | Method | Signature | Contract | |---|---|---| @@ -303,11 +320,11 @@ that the API token never follows a redirect to another host. Override both to su `false`, return a redirect response as it is. `DefaultHttpTransport` does, also when it wraps your own JDK `HttpClient` (it then copies the client's settings into one that does not follow redirects). -Every one of the four methods performs exactly one network round-trip — no retrying, no request +Every one of the four methods performs exactly one network round-trip: no retrying, no request mutation (retries, auth, and the `User-Agent` header are handled one layer up, by the client itself), and none may block the calling thread waiting on the network. A non-2xx HTTP status is **not** a transport-level error: complete the future normally with it, as a normal `HttpResponse` carrying its -actual status code. Only fail the future for a genuine transport failure — connection refused, DNS +actual status code. Only fail the future for a genuine transport failure: connection refused, DNS resolution failure, or a timeout. Any exception the future is completed exceptionally with is translated by the client into an `ApifyTransportException`; complete exceptionally with `com.apify.client.http.HttpTimeoutException` specifically for a timeout so @@ -351,27 +368,27 @@ Every exception this client throws for a request/transport failure is an uncheck [Asynchronous API](#asynchronous-api) above), these exceptions arrive as the returned `CompletableFuture`'s exceptional completion rather than a direct `throw`; `.join()` re-throws the original exception wrapped in an unchecked `CompletionException` (`.get()` wraps it in a checked -`ExecutionException` instead) — unwrap `getCause()` to recover the original +`ExecutionException` instead); unwrap `getCause()` to recover the original `ApifyApiException`/`ApifyTransportException`, or use `.handle(...)`/`.exceptionally(...)` to react to it without unwrapping at all: -- `ApifyApiException` — the request reached the API, which answered with a non-success status. The +- `ApifyApiException`: the request reached the API, which answered with a non-success status. The client throws the subclass matching the response's status code, so a `catch` can branch with - `instanceof` instead of comparing `getStatusCode()` by number — `InvalidRequestError` (400), + `instanceof` instead of comparing `getStatusCode()` by number; `InvalidRequestError` (400), `UnauthorizedError` (401), `ForbiddenError` (403), `NotFoundError` (404), `ConflictError` (409), `RateLimitError` (429) or `ServerError` (5xx). Any other status is the plain `ApifyApiException` base class, which every subclass extends, so an existing `catch (ApifyApiException e)` keeps working unchanged. -- `ApifyTransportException` — the request never produced an API response at all (connection +- `ApifyTransportException`: the request never produced an API response at all (connection failure, DNS, timeout, or a local failure preparing the request/response, e.g. compression). `isTimeout()` reports whether the underlying cause was specifically a timeout (backed by `com.apify.client.http.HttpTimeoutException`, part of the `HttpTransport` contract, not any specific transport implementation's own exception type). Note the name collision: this is a distinct type from the JDK's own `java.net.http.HttpTimeoutException` (which - `DefaultHttpTransport` catches internally and translates into this one) — use an explicit, + `DefaultHttpTransport` catches internally and translates into this one); use an explicit, fully-qualified import or a clear alias if a file needs both. -- `ServerActorException` — a call to an Actor's server (see [Server Actors](docs/actors.md#server-actors)) +- `ServerActorException`: a call to an Actor's server (see [Server Actors](docs/actors.md#server-actors)) cannot be made, since the Actor is not a Server Actor, or since the build that serves its server interface does not exist. The API was reached, but it has nothing to send the call to. @@ -388,7 +405,7 @@ resource client's id is rejected with `IllegalArgumentException` when it is empt (any other id is percent-encoded into one path segment), and `DatasetClient.pushItems(...)` throws it for items the API would reject (anything but an object or an array of objects). These are local, no-request-sent failures, not something the API responded with or the transport failed to deliver, which is why -they stay outside the request/transport exception hierarchy above — catch +they stay outside the request/transport exception hierarchy above; catch `IllegalStateException`/`IllegalArgumentException` separately if you call any of them. ```java @@ -420,10 +437,10 @@ try { The public `com.apify.client.Version` class (`import com.apify.client.Version;`) exposes two constants: -- `Version.CLIENT_VERSION` — the semantic version of this client (`0.13.0`). -- `Version.API_SPEC_VERSION` — the version of the [Apify OpenAPI specification](https://docs.apify.com/api/openapi.json) - (its `info.version` field) that this client's endpoints, parameters and models were last generated - and checked against (`v2-2026-11-04T151257Z`). It is a snapshot, not a live compatibility +- `Version.CLIENT_VERSION`: the semantic version of this client (`0.14.0`). +- `Version.API_SPEC_VERSION`: the version of the [Apify OpenAPI specification](https://docs.apify.com/api/openapi.json) + (its `info.version` field) that this client's endpoints, parameters, and models were last generated + and checked against (`v2-2026-11-09T143926Z`). It is a snapshot, not a live compatibility guarantee: the client keeps working against newer, backward-compatible spec revisions, but a feature added to the API after this snapshot has no corresponding method here yet. @@ -487,7 +504,7 @@ Every resource client lives in its own sub-package of `com.apify.client`, named | `store()` | `StoreCollectionClient` | `com.apify.client.store` | Apify Store | | `me()` / `user(id)` | `UserClient` | `com.apify.client.user` | Users | | `log(id)` | `LogClient` | `com.apify.client.log` | Build/run logs | -| `setStatusMessage(message, options)` | — (direct `ApifyClient` method, no resource client) | `com.apify.client.run` (for `SetStatusMessageOptions`) | Sets the status message of the current Actor run (see [docs/README.md](docs/README.md#setting-the-current-runs-status-message)) | +| `setStatusMessage(message, options)` | None (direct `ApifyClient` method, no resource client) | `com.apify.client.run` (for `SetStatusMessageOptions`) | Sets the status message of the current Actor run (see [docs/README.md](docs/README.md#setting-the-current-runs-status-message)) | The HTTP transport contract (`HttpTransport`, `DefaultHttpTransport`) and the exceptions thrown for transport-level failures (`ApifyTransportException`, `HttpTimeoutException`) live in diff --git a/docs/README.md b/docs/README.md index c983e40..3a3a093 100644 --- a/docs/README.md +++ b/docs/README.md @@ -15,7 +15,7 @@ All snippets assume a configured client: ApifyClient client = ApifyClient.create("my-api-token"); ``` -`ApifyClient.create` takes the token as an explicit argument — it does not read `APIFY_TOKEN` +`ApifyClient.create` takes the token as an explicit argument; it does not read `APIFY_TOKEN` automatically. Use `ApifyClient.builder()` for non-default settings (base URL, retries, timeout tiers, compression, default headers, user-agent suffix, custom HTTP transport). `client.withTimeout(...)` returns a client that sends every request with a different timeout (see @@ -25,7 +25,7 @@ Get your API token from the [Apify Console → Settings → API & Integrations](https://console.apify.com/settings/integrations). Every network-calling method returns a `CompletableFuture` (`java.util.concurrent.CompletableFuture`) -rather than blocking the calling thread — see [Asynchronous API](../README.md#asynchronous-api) in +rather than blocking the calling thread; see [Asynchronous API](../README.md#asynchronous-api) in the top-level README. Methods that fetch a single resource complete with an `Optional`: a missing resource is reported by an empty `Optional` rather than an exception. API failures surface as the future's exceptional completion, wrapping an `ApifyApiException` (see @@ -34,14 +34,14 @@ future's exceptional completion, wrapping an `ApifyApiException` (see ## Imports and dependencies Client types are **not** all in one package: `import com.apify.client.*;` only resolves -`ApifyClient`, its builder, and the shared kernel types below — a wildcard import does not reach +`ApifyClient`, its builder, and the shared kernel types below; a wildcard import does not reach into sub-packages in Java. Every resource has its own sub-package for its client(s), model(s) and option types, so import each resource you use from its own package: | Package | Contains | |---|---| | `com.apify.client` (root) | `ApifyClient`, `ApifyClientBuilder`, `Version`, `PaginationList`, `ListOptions`, `StorageListOptions`, `ApifyResource` | -| `com.apify.client.http` | `ApifyClientException`, `ApifyApiException`, `ApifyTransportException`, `HttpTransport`, `DefaultHttpTransport`, `HttpTimeoutException`, `Timeout`, `HttpCompressor`, `BrotliHttpCompressor`, `GzipHttpCompressor` | +| `com.apify.client.http` | `ApifyClientException`, `ApifyApiException`, `ApifyTransportException`, `HttpTransport`, `DefaultHttpTransport`, `HttpTimeoutException`, `Timeout`, `HttpCompressor`, `BrotliHttpCompressor`, `GzipHttpCompressor`, `ZstdHttpCompressor` | | `com.apify.client.actor` | `Actor`, `ActorClient`, `ActorCollectionClient`, `ActorListOptions`, `ActorStartOptions`, `ActorCallOptions`, `ActorBuildOptions`, `ActorStandby`, `ActorVersion`, `SourceCodeFile`, `ActorVersionClient`, `ActorVersionCollectionClient`, `ActorEnvVar`, `ActorEnvVarClient`, `ActorEnvVarCollectionClient`, `ValidateInputOptions`, `ActorStats`, `ActorDefaultRunOptions` | | `com.apify.client.build` | `Build`, `BuildClient`, `BuildCollectionClient`, `BuildActVersion`, `ActorDefinition`, `ActorDefinitionMeta`, `ActorDefinitionStorages`, `BuildMeta`, `BuildStats`, `BuildOptions`, `BuildUsage` | | `com.apify.client.run` | `ActorRun`, `ActorRunStats`, `ActorRunOptions`, `ActorRunMeta`, `ActorRunStandby`, `ActorRunUsage`, `RunClient`, `RunCollectionClient`, `RunListOptions`, `LastRunOptions`, `MetamorphOptions`, `RunChargeOptions`, `RunResurrectOptions`, `SetStatusMessageOptions` | @@ -58,7 +58,7 @@ option types, so import each resource you use from its own package: For example, the [top-level README's dataset snippet](../README.md#quick-start) needs `com.apify.client.ApifyClient`, `com.apify.client.PaginationList`, `com.apify.client.dataset.DatasetListItemsOptions`, `com.apify.client.run.ActorRun`, -`com.apify.client.actor.ActorStartOptions` and `tools.jackson.databind.JsonNode` — five +`com.apify.client.actor.ActorStartOptions` and `tools.jackson.databind.JsonNode`; five different packages for one short snippet. Snippets in these docs also assume the standard-library types they use are imported @@ -86,18 +86,18 @@ the run itself keeps running. A few methods return data whose shape is not modelled by this client and is instead exposed as a Jackson `JsonNode` (or accept an arbitrary `Object` serialized to JSON): -- Read, returning a required `JsonNode` (never absent; a 404 throws rather than returning empty — +- Read, returning a required `JsonNode` (never absent; a 404 throws rather than returning empty; see [error handling](../README.md#error-handling)): `me().monthlyUsage(...)`, `me().limits()` (and the same on `user(id)`), `dataset(id).getStatistics()`, `task(id).getInput()`. - Read, returning `Optional` (empty when the underlying resource has none): `build(id).getOpenApiDefinition()`. - Write: `task(id).updateInput(...)` (itself returning a required `JsonNode`, the updated input) and `me().updateLimits(...)` accept an arbitrary JSON-serializable value, as do - definition/`update`/`create` arguments generally — a `Map`, a `JsonNode`, or your own POJO. + definition/`update`/`create` arguments generally; a `Map`, a `JsonNode`, or your own POJO. - A few typed models still carry one or more raw-JSON fields where the shape is a discriminated union (or otherwise not worth fully modelling): `RequestQueueRequest.getUserData()`, `Webhook.getCondition()`, `Schedule.getActions()` (a `List`), `Task.getInput()` (the - task's stored input, on the `Task` model itself — distinct from the live-fetching + task's stored input, on the `Task` model itself; distinct from the live-fetching `task(id).getInput()` client call listed above), and `ActorRun`, which carries two: `getPricingInfo()` and `getStorageIds()`. @@ -106,22 +106,22 @@ Navigate a `JsonNode` with `node.get("field")`, `node.path("a").asText()`, etc. The request-queue lock/list operations (`listRequests`, `listAndLockHead`, `prolongRequestLock`, `unlockRequests`, `batchDeleteRequests`) return typed models (`RequestsList`, `LockedRequestQueueHead`, `RequestLockInfo`, `UnlockRequestsResult`, `BatchDeleteResult`) rather -than raw `JsonNode` — see [Storages](storages.md#request-queues). +than raw `JsonNode`; see [Storages](storages.md#request-queues). ## What `ApifyResource` is -`ApifyResource` (root package) is the base class every response model extends — both the top-level +`ApifyResource` (root package) is the base class every response model extends; both the top-level resources (`Actor`, `Dataset`, `Schedule`, `ActorRun`, and so on) and the nested value DTOs embedded in them (`ActorRunStats`, `ActorRunMeta`, `BuildStats`, `WebhookStats`, `UserProfile`, `PricingInfo`, and every other nested model in this client). It has no fields of its own beyond the `extra` map described below; it exists purely so every model shares one place to capture unmodelled API fields, -and so code that only needs the common capability (rare — most call sites use a concrete model type +and so code that only needs the common capability (rare; most call sites use a concrete model type directly) can accept `ApifyResource` rather than a specific model. ## Model fields and unmodeled data (`getExtra`) Response models expose the commonly-used fields as typed getters. The API returns more fields than -are modelled; every model — including nested value DTOs, not only the top-level resources — also +are modelled; every model, including nested value DTOs, not only the top-level resources, also carries a `getExtra()` map (`Map`, inherited from `ApifyResource` above) holding any field not mapped to a typed getter, so nothing the API returns is lost, now or as the API grows new fields later. For example a `me()` `User`'s private account details (email, plan, proxy settings, @@ -154,11 +154,11 @@ ActorListOptions options = new ActorListOptions().my(true).limit(10L); PaginationList page = client.actors().list(options).join(); ``` -## Common list options — `ListOptions` +## Common list options: `ListOptions` Most `list` methods (builds, tasks, schedules, webhooks, Actor versions) take the shared `ListOptions`, which carries the standard pagination/ordering controls. Runs additionally take a -`RunListOptions` status filter — `runs().list(ListOptions, RunListOptions)`; see [Runs](runs.md). +`RunListOptions` status filter; `runs().list(ListOptions, RunListOptions)`; see [Runs](runs.md). | Method | Type | Meaning | |---|---|---| @@ -170,7 +170,7 @@ Most `list` methods (builds, tasks, schedules, webhooks, Actor versions) take th PaginationList builds = client.builds().list(new ListOptions().limit(50L).desc(true)).join(); ``` -## Pagination — `PaginationList` +## Pagination: `PaginationList` `list` methods return a `PaginationList` page with `getTotal()`, `getOffset()`, `getLimit()`, `getCount()`, `isDesc()` and `getItems()`. `getCount()` is the number of entries on the page. On a @@ -180,7 +180,7 @@ The next page starts at `offset + Math.min(count, limit)`; the cap matters only can return more items than the rows scanned. Within-storage listers (`listKeys`, `listHead`) return their own page/head containers instead. -## Iteration — `iterate` / `iterateItems` / `iterateKeys` +## Iteration: `iterate` / `iterateItems` / `iterateKeys` Each paginated collection also offers a lazy `Flow.Publisher` (`java.util.concurrent.Flow.Publisher`, the JDK's built-in Reactive Streams type) that fetches pages only as the subscriber signals demand: @@ -190,13 +190,13 @@ the JDK's built-in Reactive Streams type) that fetches pages only as the subscri that is the rows each page covers (`PaginationList.getCount()`, capped at the page limit that was requested). That is the number of items yielded, unless a page's count differs from its items: filtered or unwound dataset items, or a Store list that drops entries after counting them. -`null`/unset — or a non-positive value such as `0` — means no cap, so every item is yielded. (This +`null`/unset, or a non-positive value such as `0`, means no cap, so every item is yielded. (This differs from `list(...)`, which sends `limit=0` to the server verbatim rather than treating it as -unbounded — the iteration behavior matches the reference JS client.) The per-request +unbounded.) The per-request page size is an optional trailing `chunkSize` argument: the per-resource tables below show the `chunkSize` form, and each publisher also has an overload that omits it (using the server's default page size). The page size does not change which items a collection publisher yields; note the one -exception in [Storages](storages.md) — `iterateItems` combined with server-side item filters, where +exception in [Storages](storages.md); `iterateItems` combined with server-side item filters, where the page size can affect the result. Consume a publisher either by implementing `Flow.Subscriber` yourself (see diff --git a/docs/actors.md b/docs/actors.md index 5aa4cb1..f98c38e 100644 --- a/docs/actors.md +++ b/docs/actors.md @@ -48,7 +48,7 @@ Actor created = client.actors().create(Map.of( | `delete()` | Delete the Actor. | | `start(Object input, ActorStartOptions)` | Start a run, returning immediately. Completes with `ActorRun`. The `input` is any JSON-serializable value, or a `byte[]`/`String` that is sent as it is (set the matching `ActorStartOptions.contentType(...)`), or `null`. | | `call(Object input, ActorStartOptions, Long waitSecs)` | Start a run and poll until it finishes (`null` waits indefinitely); does **not** stream the run's log. Completes with `ActorRun`. | -| `call(Object input, ActorCallOptions, Long waitSecs)` | As above, additionally streaming the run's log for the duration of the wait by default (matching the reference client's `call` defaulting `options.log` to `'default'`). Use `ActorCallOptions.disableLogStreaming()` to opt out, or `logOptions(StreamedLogOptions)` for a custom destination. | +| `call(Object input, ActorCallOptions, Long waitSecs)` | As above, additionally streaming the run's log for the duration of the wait by default. Use `ActorCallOptions.disableLogStreaming()` to opt out, or `logOptions(StreamedLogOptions)` for a custom destination. | | `validateInput(Object input)` / `validateInput(Object input, ValidateInputOptions)` | Validate an input against the Actor's input schema. Completes with `boolean`. `ValidateInputOptions` fields (optional): `build` (`String`), `contentType` (`String`). | | `build(String versionNumber, ActorBuildOptions)` | Build a version. Completes with `Build`. | | `defaultBuild(Long waitForFinish)` | Resolve the default build. Completes with `BuildClient`. | @@ -67,19 +67,18 @@ to charge, pay-per-result Actors), `maxTotalChargeUsd(Double)` (maximum total ch pay-per-event Actors), `contentType(String)` (content type of the input body, defaults to `application/json`), `restartOnError(Boolean)` (restart the run if it fails), `forcePermissionLevel(String)` (override the Actor's permission level for this run: -`LIMITED_PERMISSIONS`/`FULL_PERMISSIONS`), `webhooks(List)` — ad-hoc webhook definitions +`LIMITED_PERMISSIONS`/`FULL_PERMISSIONS`), `webhooks(List)`, ad-hoc webhook definitions (each a JSON-serializable `Map`, as in [Webhooks](webhooks.md)) that the client base64-encodes on the -wire — and `waitForResources(boolean)`/`waitForResources(Long maxSecs)`: retry the start, every 10 -seconds, while the account lacks the memory or a concurrent-run slot for the run (`true` retries -until it starts — `ActorStartOptions.WAIT_FOR_RESOURCES_INDEFINITELY`; a bound throws the last such -error once exceeded; omitted, the default, does not retry at all). +wire, and `waitForResources(boolean)`/`waitForResources(Long maxSecs)`: retry the start, every 10 +seconds, while the account lacks the memory or a concurrent-run slot for the run (`true`, or +`ActorStartOptions.WAIT_FOR_RESOURCES_INDEFINITELY`, retries until it starts; a bound throws the +last such error once exceeded; omitted, the default, does not retry at all). `ActorCallOptions` (for the log-streaming `call` overload) mirrors `ActorStartOptions` field for field, but omits `waitForFinish`: that field asks the API to hold the HTTP response open server-side while the run finishes, which is redundant with (and wastes a request slot next to) -`call`'s own client-side `waitSecs` polling. It adds `disableLogStreaming()` (matching the -reference client's `log: null`) and `logOptions(StreamedLogOptions)` (a custom destination/prefix, -matching a custom `Log` instance) — see [Streamed log redirection](runs.md#streamed-log-redirection). +`call`'s own client-side `waitSecs` polling. It adds `disableLogStreaming()` and +`logOptions(StreamedLogOptions)` (a custom destination or prefix); see [Streamed log redirection](runs.md#streamed-log-redirection). `lastRun(String status)` filters only by status; `lastRun(LastRunOptions)` also accepts an origin filter. `LastRunOptions` has fluent setters `status(String)` (e.g. `SUCCEEDED`, `RUNNING`) and @@ -98,17 +97,17 @@ System.out.println(run.getStatus()); ``` `Actor` fields: `getId()`, `getUserId()`, `getName()`, `getUsername()`, `getTitle()`, -`getDescription()`, `isPublic()`, `getCreatedAt()`, `getModifiedAt()`, `getStats()` (`ActorStats` — +`getDescription()`, `isPublic()`, `getCreatedAt()`, `getModifiedAt()`, `getStats()` (`ActorStats`: `getTotalBuilds()`/`getTotalRuns()`/`getTotalUsers()`/`getTotalUsers7Days()`/ `getTotalUsers30Days()`/`getTotalUsers90Days()`/`getTotalMetamorphs()` as `Long`, `getLastRunStartedAt()` as `Instant`), `getVersions()` (`List`), `getPricingInfos()` (`List`; shape depends on pricing model), `getDefaultRunOptions()` -(`ActorDefaultRunOptions` — `getBuild()`, `getTimeoutSecs()`/`getMemoryMbytes()` as `Long`, +(`ActorDefaultRunOptions`: `getBuild()`, `getTimeoutSecs()`/`getMemoryMbytes()` as `Long`, `getRestartOnError()` as `Boolean`), `getTaggedBuilds()` (`JsonNode`; dynamic tag-name keys), `getIsDeprecated()` (`Boolean`), `getDeploymentKey()`, `getSeoTitle()`, `getSeoDescription()`, `getCategories()` (`List`), `getStandbyUrl()` (the URL of the Actor's server, `null` unless Standby mode is enabled), `getActorStandby()` (`ActorStandby`, from -`com.apify.client.actor`, nullable — see [Tasks](tasks.md) for its field list; only on `Actor` does +`com.apify.client.actor`, nullable; see [Tasks](tasks.md) for its field list; only on `Actor` does it additionally expose `getIsEnabled()`), `getActorPermissionLevel()` (e.g. `"OWNER"`), plus `getExtra()` for any unmodelled fields. @@ -134,7 +133,9 @@ Actor's `standbyUrl` and `actorStandby` fields. parameters, converted as for API calls: `null` is left out, a boolean is sent as `1`/`0`, a collection comma-joined), `header(String, String)`/`headers(Map)`, `body(Object)` (a `byte[]` or `String` is sent as it is; anything else is serialized to JSON and sent as `application/json` -unless you set a `Content-Type` header), `compression(HttpCompressor)` and `timeout(Timeout)` +unless you set a `Content-Type` header), `parentRequestId(String)` (see +[Calling a Server Actor from a Server Actor](#calling-a-server-actor-from-a-server-actor)), +`compression(HttpCompressor)` and `timeout(Timeout)` (`Timeout.LONG` by default, since the first request to an idle Actor waits for the platform to start a server run). @@ -197,20 +198,43 @@ you create or update them; `ActorVersion.getActorPath()` and `BuildActVersion.ge it. An update that sends `sourceType`, `sourceFiles` or `tarballUrl` without `actorPath` removes the stored one, so send it again with every upload. +### Calling a Server Actor from a Server Actor + +The platform adds an `X-Actor-Request-ID` header with a unique ID to every request it forwards to a +Server Actor. A multi-tenant Server Actor uses it to charge the user who sent the request, and to +pass that user's pricing tier on to the Server Actors it calls. + +When your multi-tenant Server Actor calls another one while handling a request, pass the incoming +`X-Actor-Request-ID` header value as `parentRequestId`. The client sends it as the same header, so +the called Actor charges at the pricing tier of your incoming request. Use a client created with +the run's `APIFY_TOKEN`. See [Call another multi-tenant Server Actor](https://docs.apify.com/actors/monetize/server-actors#call-another-multi-tenant-server-actor). + +```java +// requestId is the X-Actor-Request-ID header value of the request your Actor is handling. +String requestId = "7Hs2kQw9LxP4mN1vB"; +ApiResponse response = + client + .actor("username/other-actor") + .server() + .request("/search", new ActorServerRequestOptions().param("q", "apify").parentRequestId(requestId)) + .join(); +System.out.println(response.statusCode()); +``` + ## `ActorVersionClient` and `ActorVersionCollectionClient` `client.actor(id).versions()` lists/creates versions; `client.actor(id).version(v)` reads, updates and deletes a single version and exposes its environment variables. -### `ActorVersionCollectionClient` — `client.actor(id).versions()` +### `ActorVersionCollectionClient`: `client.actor(id).versions()` | Method | Description | |---|---| -| `list(ListOptions)` | List the Actor's versions. Completes with `PaginationList`. The endpoint takes no query parameters; `options` (`offset`/`limit`/`desc`) is accepted only for signature consistency with every other `list(ListOptions)` and has no effect — pass `null`. | +| `list(ListOptions)` | List the Actor's versions. Completes with `PaginationList`. The endpoint takes no query parameters; `options` (`offset`/`limit`/`desc`) is accepted only for signature consistency with every other `list(ListOptions)` and has no effect; pass `null`. | | `iterate(ListOptions)` | Lazy `Flow.Publisher` over all versions; `limit` caps the total (`null`/unset or non-positive = all), applied client-side after the single fetch. `offset` has no effect and there is no page size to tune. | | `create(Object version)` | Create a version. Completes with `ActorVersion`. | -### `ActorVersionClient` — `client.actor(id).version(v)` +### `ActorVersionClient`: `client.actor(id).version(v)` | Method | Description | |---|---| @@ -234,7 +258,7 @@ Attach environment variables to a version. `ActorEnvVar` has a `(name, value)` c fluent setters `setName`, `setValue`, `setIsSecret(Boolean)` (when secret, the value is stored encrypted), and matching getters `getName()`, `getValue()`, `getIsSecret()`. -### `ActorEnvVarCollectionClient` — `client.actor(id).version(v).envVars()` +### `ActorEnvVarCollectionClient`: `client.actor(id).version(v).envVars()` | Method | Description | |---|---| @@ -242,7 +266,7 @@ encrypted), and matching getters `getName()`, `getValue()`, `getIsSecret()`. | `iterate()` | `Flow.Publisher` over the variables. The env-var collection is not paginated (all variables are returned at once), so this iterates a single fetched page; provided for API consistency. | | `create(ActorEnvVar)` | Create an environment variable. Completes with `ActorEnvVar`. | -### `ActorEnvVarClient` — `client.actor(id).version(v).envVar(name)` +### `ActorEnvVarClient`: `client.actor(id).version(v).envVar(name)` | Method | Description | |---|---| diff --git a/docs/builds.md b/docs/builds.md index cdc606e..8da32df 100644 --- a/docs/builds.md +++ b/docs/builds.md @@ -23,22 +23,22 @@ builds) and a single build with `client.build(id)`. | `log()` | A `LogClient` for the build's log. | `Build` fields: `getId()`, `getActId()`, `getUserId()`, `getStatus()`, `getStartedAt()`, -`getFinishedAt()`, `getBuildNumber()`, `getActVersion()` (`BuildActVersion` — `getSourceType()`, `getBuildTag()` -(nullable), `getVersionNumber()`, `getGitRepoUrl()`, `getSourceFiles()` (`List` — `getName()`, -`getFormat()`, `getContent()`)), `getActorDefinition()` (`ActorDefinition`, nullable — the Actor's `.actor/actor.json` at build time: +`getFinishedAt()`, `getBuildNumber()`, `getActVersion()` (`BuildActVersion`: `getSourceType()`, `getBuildTag()` +(nullable), `getVersionNumber()`, `getGitRepoUrl()`, `getSourceFiles()` (`List`: `getName()`, +`getFormat()`, `getContent()`)), `getActorDefinition()` (`ActorDefinition`, nullable; the Actor's `.actor/actor.json` at build time: `getActorSpecification()`, `getName()`, `getTitle()`, `getDescription()`, `getVersion()`, `getBuildTag()`, -`getMeta()` (`ActorDefinitionMeta` — `getTemplateId()`, `getGeneratedBy()`), `getEnvironmentVariables()`, +`getMeta()` (`ActorDefinitionMeta`: `getTemplateId()`, `getGeneratedBy()`), `getEnvironmentVariables()`, `getDockerfile()`, `getDockerContextDir()`, `getReadme()`, `getInput()`/`getOutput()` (`JsonNode` schemas), -`getChangelog()`, `getStorages()` (`ActorDefinitionStorages` — `getDataset()`, `getDatasets()`, +`getChangelog()`, `getStorages()` (`ActorDefinitionStorages`: `getDataset()`, `getDatasets()`, `getKeyValueStore()`), `getDefaultMemoryMbytes()` (`JsonNode`, an integer or a dynamic memory expression), `getMinMemoryMbytes()`, `getMaxMemoryMbytes()`, `getUsesStandbyMode()`, `getWebServerSchema()`, -`getWebServerMcpPath()`), `getImageDigest()` (`String`, nullable — the built Docker +`getWebServerMcpPath()`), `getImageDigest()` (`String`, nullable: the built Docker image manifest's digest, without the `sha256:` prefix; compare two builds' digests to tell whether -their image contents differ), `getMeta()` (`BuildMeta` — `getOrigin()`, `getClientIp()`, -`getUserAgent()`), `getStats()` (`BuildStats` — `getDurationMillis()`/`getRunTimeSecs()` as `Long`, -`getComputeUnits()` as `Double`, `getImageSizeBytes()` as `Long`), `getOptions()` (`BuildOptions` — +their image contents differ), `getMeta()` (`BuildMeta`: `getOrigin()`, `getClientIp()`, +`getUserAgent()`), `getStats()` (`BuildStats`: `getDurationMillis()`/`getRunTimeSecs()` as `Long`, +`getComputeUnits()` as `Double`, `getImageSizeBytes()` as `Long`), `getOptions()` (`BuildOptions`: `getUseCache()`/`getBetaPackages()` as `Boolean`, `getMemoryMbytes()`/`getDiskMbytes()` as `Long`), -`getUsage()` / `getUsageUsd()` (`BuildUsage` — `getActorComputeUnits()` as `Double`, the latter as +`getUsage()` / `getUsageUsd()` (`BuildUsage`: `getActorComputeUnits()` as `Double`, the latter as its USD cost), `getUsageTotalUsd()` (`Double`), plus `isTerminal()` and `getExtra()`. The status is one of `READY`, `RUNNING`, `SUCCEEDED`, `FAILED`, `TIMING-OUT`, `TIMED-OUT`, `ABORTING`, `ABORTED`. Every nested model above (`BuildActVersion`, `ActorDefinition`, `ActorDefinitionMeta`, `ActorDefinitionStorages`, `BuildMeta`, `BuildStats`, `BuildOptions`, `BuildUsage`) also has its own @@ -56,18 +56,18 @@ if (finished.isTerminal()) { source* to build, matching one of the version numbers under `client.actor(id).versions()`. `ActorBuildOptions.tag(...)` is unrelated to that: it is the *build tag* (e.g. `"latest"`, `"beta"`) stamped onto the resulting build, which is what `ActorStartOptions.build(...)` / -`Actor.defaultBuild(...)` later resolve by name — a version can be rebuilt many times under the +`Actor.defaultBuild(...)` later resolve by name; a version can be rebuilt many times under the same tag, with each new build replacing which build that tag currently points to. `ActorBuildOptions` fields (all optional): `betaPackages` (`Boolean`; if `true`, use beta versions of Apify packages instead of the latest stable ones), `tag` (`String`; the build tag to apply, e.g. -`"latest"` — see above), `useCache` (`Boolean`; whether to reuse the Docker build cache, default +`"latest"`; see above), `useCache` (`Boolean`; whether to reuse the Docker build cache, default `true`), `waitForFinish` (`Long`; maximum seconds to wait server-side for the build to finish -before the API responds, max 60 — see `defaultBuild` below for the same wait model). +before the API responds, max 60; see `defaultBuild` below for the same wait model). `ActorClient.defaultBuild(Long waitForFinish)` resolves the Actor's currently-tagged default build (the build behind the `"latest"`/`"default"` tag, i.e. what a plain `start`/`call` with no explicit -`build` would run) and completes with a `BuildClient` handle for it — it does not itself return the +`build` would run) and completes with a `BuildClient` handle for it; it does not itself return the `Build` object. `waitForFinish` only bounds how long the *resolving GET* waits server-side for that build to finish before responding (`null`/`0` returns immediately with whatever state the build is currently in); it does not make `defaultBuild` block until the build is done. To actually observe diff --git a/docs/misc.md b/docs/misc.md index 3c868c8..9adaab3 100644 --- a/docs/misc.md +++ b/docs/misc.md @@ -1,6 +1,6 @@ # Store, users & logs -## Apify Store — `client.store()` +## Apify Store: `client.store()` Browse public Actors in the Apify Store. `client.store()` returns a `StoreCollectionClient`. @@ -20,7 +20,7 @@ Actors that allow agentic users), `responseFormat` (the response shape: `full` o ```java // Publishers.collect() drains the whole publisher; fine here since `limit` already caps it. // To stop early without fetching further pages, subscribe with your own Flow.Subscriber instead -// (request(1) at a time, cancel() once you have enough) — see IterateStore.java in the examples. +// (request(1) at a time, cancel() once you have enough); see IterateStore.java in the examples. List items = Publishers.collect(client.store().iterate(new StoreListOptions().search("crawler").limit(20L), 10L)) .join(); @@ -29,12 +29,12 @@ items.forEach(item -> System.out.println(item.getUsername() + "/" + item.getName `ActorStoreListItem` fields: `getId()`, `getName()`, `getUsername()`, `getTitle()`, `getDescription()`, `getStats()` (`ActorStats`; see [Actors](actors.md#actorclient)), -`getCurrentPricingInfo()` (`PricingInfo` — `getPricingModel()`), `getPictureUrl()`, +`getCurrentPricingInfo()` (`PricingInfo`: `getPricingModel()`), `getPictureUrl()`, `getUserPictureUrl()`, `getUrl()`, `getReadmeSummary()`. Any field not covered by a typed getter is still available via the inherited `getExtra()` (see [the docs index](README.md#model-fields-and-unmodeled-data-getextra)). -## Users — `client.me()` / `client.user(id)` +## Users: `client.me()` / `client.user(id)` | Method | Description | |---|---| @@ -58,12 +58,12 @@ me.ifPresent( JsonNode usage = client.me().monthlyUsage().join(); ``` -`User` fields: `getId()`, `getUsername()`, `getProfile()` (`UserProfile` — `getBio()`, `getReadme()`, +`User` fields: `getId()`, `getUsername()`, `getProfile()` (`UserProfile`: `getBio()`, `getReadme()`, `getName()`, `getPictureUrl()`, `getGithubUsername()`, `getWebsiteUrl()`, `getTwitterUsername()`, all nullable). -Only present for `me()`: `getEmail()`, `getProxy()` (`UserProxy` — `getPassword()`, `getGroups()` +Only present for `me()`: `getEmail()`, `getProxy()` (`UserProxy`: `getPassword()`, `getGroups()` as `List`, each with `getName()`/`getDescription()`/`getAvailableCount()` as `Long`), -`getPlan()` (`UserPlan` — `getId()`, `getDescription()`, `getIsEnabled()`, +`getPlan()` (`UserPlan`: `getId()`, `getDescription()`, `getIsEnabled()`, `getMonthlyBasePriceUsd()`/`getMonthlyUsageCreditsUsd()`/`getUsageDiscountPercent()`/ `getMaxMonthlyUsageUsd()`/`getMaxActorMemoryGbytes()`/`getMaxMonthlyActorComputeUnits()`/ `getMaxMonthlyResidentialProxyGbytes()`/`getMaxMonthlyProxySerps()`/ @@ -71,16 +71,16 @@ as `List`, each with `getName()`/`getDescription()`/`getAvailableCou `List`, `getMaxActorCount()`/`getMaxActorTaskCount()`/`getDataRetentionDays()`/ `getTeamAccountSeatCount()` as `Long`, `getAvailableProxyGroups()` as `JsonNode`, `getSupportLevel()`, `getAvailableAddOns()` as `List`), `getEffectivePlatformFeatures()` -(`JsonNode`), `getCreatedAt()` (`Instant`), `getIsPaying()` (`Boolean`). Plus `getExtra()` — a +(`JsonNode`), `getCreatedAt()` (`Instant`), `getIsPaying()` (`Boolean`). Plus `getExtra()`, a `Map` carrying any account field not modelled directly. -## Logs — `client.log(id)` +## Logs: `client.log(id)` Access a build's or run's log directly, or via `client.run(id).log()` / `client.build(id).log()`. | Method | Description | |---|---| -| `get()` / `get(LogOptions)` | The whole log as text. Completes with `Optional` when addressed by an explicit id (`client.log(id)`); on `run.log()`/`build.log()` (no log id of its own), a 404 is ambiguous and throws `NotFoundError` instead — see [Fetching single resources](../README.md#fetching-single-resources). | +| `get()` / `get(LogOptions)` | The whole log as text. Completes with `Optional` when addressed by an explicit id (`client.log(id)`); on `run.log()`/`build.log()` (no log id of its own), a 404 is ambiguous and throws `NotFoundError` instead; see [Fetching single resources](../README.md#fetching-single-resources). | | `getAsBytes()` / `getAsBytes(LogOptions)` | As `get()`, but completes with `Optional` (the raw bytes, not decoded as text). Same 404 handling as `get()`. | | `stream()` / `stream(LogOptions)` | A live `InputStream` over the log (for redirection). Completes with `InputStream`; throws on any error response, including a 404. | diff --git a/docs/runs.md b/docs/runs.md index de5a4f0..5f533a8 100644 --- a/docs/runs.md +++ b/docs/runs.md @@ -16,7 +16,7 @@ Access the run collection with `client.runs()` (or `client.actor(id).runs()` / filter with only non-terminal statuses, declared by `client.runs()` and `client.actor(id).runs()`), declared by every scope's endpoint (`client.runs()`, `client.actor(id).runs()`, `client.task(id).runs()`), and `startedAfter(String)` / `startedBefore(String)` (ISO-8601), declared -only by the account-wide `client.runs()` and the Actor-scoped `client.actor(id).runs()` endpoints — +only by the account-wide `client.runs()` and the Actor-scoped `client.actor(id).runs()` endpoints; the task-scoped `client.task(id).runs()` endpoint's spec does not declare these two parameters at all, so set them only when listing at one of the other two scopes. All parameters a given scope declares compose (e.g. a status filter combined with a start-time window on `client.runs()`). @@ -48,7 +48,7 @@ PaginationList runs = client.runs().list( | `getStreamedLog(StreamedLogOptions)` | As above, with a custom destination / options. | If `waitSecs` elapses before the run reaches a terminal state, `waitForFinish` returns the run's -current (non-terminal) status rather than throwing or blocking further — check `isTerminal()` on +current (non-terminal) status rather than throwing or blocking further; check `isTerminal()` on the result and call `waitForFinish` again (or poll) if you need to keep waiting. `ActorClient`/ `TaskClient`'s `call(...)` overloads (see [Actors](actors.md#actorclient)/[Tasks](tasks.md)) return this same possibly-non-terminal `ActorRun` for the same reason: `call` is `start` followed by @@ -63,17 +63,17 @@ right after a run/build is created), past which it throws that `NotFoundError`. `getStreamedLog()` completes with a `StreamedLog`: a helper that follows the run's live raw log in a background thread and redirects each complete, timestamped message to a destination. It is -`AutoCloseable` — call `start()` (itself asynchronous; it completes once the live stream is open and +`AutoCloseable`; call `start()` (itself asynchronous; it completes once the live stream is open and the background reader thread is running) to begin redirection and `stop()` (or `close()`, via try-with-resources) to end it. With no options, messages go to an SLF4J `Logger` at `INFO` level, prefixed with the Actor name and run id (looked up automatically). `StreamedLogOptions` customizes it: -- `toLog(Consumer)` — send each complete message to your own consumer instead of the +- `toLog(Consumer)`: send each complete message to your own consumer instead of the default logger. -- `prefix(String)` — override the auto-built prefix used by the default logger. -- `fromStart(boolean)` — when `false`, skip log lines produced before redirection started +- `prefix(String)`: override the auto-built prefix used by the default logger. +- `fromStart(boolean)`: when `false`, skip log lines produced before redirection started (default `true`). A run that has not logged anything yet (e.g. still queued for resources, see @@ -82,6 +82,17 @@ A run that has not logged anything yet (e.g. still queued for resources, see rather than silently giving up. If `stop()`/`close()` is called before that happens, it fetches the log in one plain request instead, so whatever was logged by then is still delivered. +The `toLog` consumer runs on the background reader thread, one message at a time. `stop()` and +`close()` wait for the consumer to finish its current message and up to 1 second for a message still +arriving, deliver the last message, and then return. If the calling thread is interrupted, they +close the stream, keep the interrupt flag, and return before the reader has ended, so the last +message may arrive after they return, and `start()` throws until a later `stop()` or `close()` has +joined the reader. + +Calling `close()` from inside the consumer is supported: it signals the stop and returns without +waiting, and the reader ends once the consumer returns. `start()` throws until a later `stop()` or +`close()` from another thread has joined the reader. + ```java RunClient runClient = client.run("RUN_ID"); List collected = new ArrayList<>(); @@ -108,9 +119,8 @@ client.run("RUN_ID").update(Map.of("statusMessage", "half way there", "isStatusM > `task(id).lastRun(...)` targets the `runs/last` path and is intended for reading (`get`, > `dataset()`/`keyValueStore()`/`requestQueue()`, `log()`). The mutating methods (`abort`, `reboot`, > `resurrect`, `metamorph`, `charge`, `update`, `delete`) require a concrete run and have no -> `runs/last` endpoint — resolve the id first (`lastRun(...).get().join()` then `client.run(id)`). The type -> exposes them uniformly (matching the reference client's shared `RunClient`), but calling them on a -> last-run client will fail server-side. +> `runs/last` endpoint; resolve the id first (`lastRun(...).get().join()` then `client.run(id)`). The type +> exposes them uniformly, but calling them on a last-run client fails server-side. `ActorRun` fields include `getId()`, `getActId()`, `getUserId()`, `getStatus()`, `getStatusMessage()`, `getStartedAt()`, `getFinishedAt()`, `getBuildId()`, `getDefaultDatasetId()`, @@ -133,17 +143,17 @@ owning it, e.g. `"ANYONE_WITH_ID_CAN_READ"`, `"RESTRICTED"`, or `null` to follow default), `getChargedEventCounts()` (`Map`; per-event-type charge counts for pay-per-event Actors, `null` otherwise), `getPricingInfo()` (`JsonNode`; shape depends on the Actor's pricing model), `getUsage()` / `getUsageUsd()` (`ActorRunUsage`; per-billable-unit resource -consumption, the latter as its USD cost — see below), `getUsageTotalUsd()` (`Double`; the run's +consumption, the latter as its USD cost; see below), `getUsageTotalUsd()` (`Double`; the run's total cost in USD, combining platform usage and/or event costs depending on pricing model), -`getStats()` (`ActorRunStats`; runtime metrics — `getInputBodyLen()`, `getMigrationCount()`, +`getStats()` (`ActorRunStats`; runtime metrics; `getInputBodyLen()`, `getMigrationCount()`, `getRebootCount()`, `getRestartCount()`, `getResurrectCount()`, `getMemAvgBytes()`/`getMemMaxBytes()`/`getMemCurrentBytes()`, `getCpuAvgUsage()`/`getCpuMaxUsage()`/`getCpuCurrentUsage()`, `getNetRxBytes()`/`getNetTxBytes()`, `getDurationMillis()`, `getRunTimeSecs()`, `getMetamorph()`, `getComputeUnits()`; any other field is still reachable via the inherited `getExtra()`), `getOptions()` (`ActorRunOptions`; the run configuration actually applied, -which may differ from what was requested — see below), `getMeta()` (`ActorRunMeta`; `getOrigin()`, -`getClientIp()`, `getUserAgent()`, `getScheduleId()`, `getScheduledAt()` — how and, if triggered by a +which may differ from what was requested; see below), `getMeta()` (`ActorRunMeta`; `getOrigin()`, +`getClientIp()`, `getUserAgent()`, `getScheduleId()`, `getScheduledAt()`; how and, if triggered by a schedule, when the run was initiated), `getBuildNumber()` (`String`; the build's semver-like number, e.g. `"0.0.36"`), `getExitCode()` (`Integer`; the process exit code once finished), `isContainerServerReady()` (`Boolean`; whether the container's HTTP server can accept @@ -165,8 +175,8 @@ nullable), `getMemoryMbytes()` (`Long`, nullable), `getDiskMbytes()` (`Long`, nu `getRestartOnError()` (`Boolean`, nullable). `RunChargeOptions` (constructed with the required event name) uses plain values: `count(Long)`, -`idempotencyKey(String)` — auto-generated when unset so a transport-retried charge is applied at -most once — and `requestId(String)`, the `X-Actor-Request-ID` of the Server Actor request being +`idempotencyKey(String)`, auto-generated when unset so a transport-retried charge is applied at +most once, and `requestId(String)`, the `X-Actor-Request-ID` of the Server Actor request being charged (required for multi-tenant Server Actor runs, omitted otherwise). ```java diff --git a/docs/storages.md b/docs/storages.md index 2a4fa3a..552dbeb 100644 --- a/docs/storages.md +++ b/docs/storages.md @@ -31,7 +31,7 @@ any field not covered by a typed getter above is still available via the inherit ## Datasets -### `DatasetCollectionClient` — `client.datasets()` +### `DatasetCollectionClient`: `client.datasets()` | Method | Description | |---|---| @@ -42,11 +42,11 @@ any field not covered by a typed getter above is still available via the inherit `StorageListOptions` adds `unnamed(Boolean)` and `ownership(String)` on top of offset/limit/desc. -### `DatasetClient` — `client.dataset(id)` +### `DatasetClient`: `client.dataset(id)` | Method | Description | |---|---| -| `get()` / `update(Object)` / `delete()` | Metadata CRUD. On a client reached through a run/task with no dataset id of its own (e.g. `run.dataset()`), a 404 is ambiguous (parent vs. sub-resource gone) and throws `NotFoundError` instead of resolving to empty/no-op — see [Fetching single resources](../README.md#fetching-single-resources). | +| `get()` / `update(Object)` / `delete()` | Metadata CRUD. On a client reached through a run/task with no dataset id of its own (e.g. `run.dataset()`), a 404 is ambiguous (parent vs. sub-resource gone) and throws `NotFoundError` instead of resolving to empty/no-op; see [Fetching single resources](../README.md#fetching-single-resources). | | `listItems(DatasetListItemsOptions)` | List items. Completes with `PaginationList`. | | `listItems(DatasetListItemsOptions, Class)` | List items decoded into `T`. Completes with `PaginationList`. | | `iterateItems(DatasetListItemsOptions)` / `iterateItems(DatasetListItemsOptions, Long chunkSize)` | A lazy `Flow.Publisher` over all items; the options' `limit` caps the dataset *rows* read (`null`/unset or non-positive = all), the optional `chunkSize` sets the per-request page size. Dataset items have no default page limit, so without `chunkSize` the whole dataset (or all `limit` rows) is loaded in one response: pass `chunkSize` to iterate a large dataset. | @@ -91,14 +91,13 @@ try (InputStream stream = `DatasetListItemsOptions` fields: `offset` (`Long`, number of items to skip), `limit` (`Long`, maximum number of items to return), `desc` (`Boolean`, return items newest-first), `fields` (`List`, restrict the output to these source fields), `outputFields` (`List`, -positionally *renames* the fields chosen by `fields` in the output — the i-th name renames the i-th +positionally *renames* the fields chosen by `fields` in the output; the i-th name renames the i-th `fields` entry, so it only makes sense together with `fields`), `omit` (`List`, exclude these fields from the output), `skipEmpty` (`Boolean`, skip empty items), `skipHidden` (`Boolean`, -skip hidden fields, i.e. those starting with `#`), `clean` (`Boolean`, return only clean — -non-empty, non-hidden — items), `unwind` (`List`, expand these fields so each array element +skip hidden fields, i.e. those starting with `#`), `clean` (`Boolean`, return only clean items: non-empty and non-hidden), `unwind` (`List`, expand these fields so each array element becomes a separate item), `flatten` (`List`, flatten these nested fields into dot-notation keys), `view` (`String`, select a predefined dataset view for field selection), `simplified` -(`Boolean`, return simplified — flattened and cleaned — items), `skipFailedPages` (`Boolean`, skip +(`Boolean`, return flattened and cleaned items), `skipFailedPages` (`Boolean`, skip items that come from failed pages), `signature` (`String`, a pre-shared URL signature granting access without an API token). `downloadItems(...)` returns `byte[]` (the serialized export); `streamItems(...)` takes the same two arguments but returns a live `InputStream` over it instead. @@ -116,7 +115,7 @@ request is sent. The expiry counts from the moment the URL is signed, after the optional `format` sets the serialization the URL serves items in (default `json` when omitted/`null`). -> **Public-URL limitation (matches the JavaScript reference client).** The public-URL builders +> **Public-URL limitation.** The public-URL builders > (`createItemsPublicUrl`, and the key-value-store `getRecordPublicUrl` / `createKeysPublicUrl`) > build the URL from the client's own resource path plus the signature. When called on a *last-run > nested* client (e.g. `client.actor(id).lastRun("SUCCEEDED").dataset().createItemsPublicUrl(...)`) @@ -126,17 +125,17 @@ omitted/`null`). ## Key-value stores -### `KeyValueStoreCollectionClient` — `client.keyValueStores()` +### `KeyValueStoreCollectionClient`: `client.keyValueStores()` `list(StorageListOptions)`, `iterate(StorageListOptions, Long chunkSize)`, `getOrCreate(String)`, and `getOrCreate(String, Object schema)` (the latter sends a creation-time store `schema`), as for datasets. -### `KeyValueStoreClient` — `client.keyValueStore(id)` +### `KeyValueStoreClient`: `client.keyValueStore(id)` | Method | Description | |---|---| -| `get()` / `update(Object)` / `delete()` | Metadata CRUD. On a client reached through a run/task with no store id of its own (e.g. `run.keyValueStore()`), a 404 is ambiguous (parent vs. sub-resource gone) and throws `NotFoundError` instead of resolving to empty/no-op — see [Fetching single resources](../README.md#fetching-single-resources). | +| `get()` / `update(Object)` / `delete()` | Metadata CRUD. On a client reached through a run/task with no store id of its own (e.g. `run.keyValueStore()`), a 404 is ambiguous (parent vs. sub-resource gone) and throws `NotFoundError` instead of resolving to empty/no-op; see [Fetching single resources](../README.md#fetching-single-resources). | | `listKeys(ListKeysOptions)` | List keys. Completes with `KeyValueStoreKeysPage`. | | `iterateKeys(ListKeysOptions)` / `iterateKeys(ListKeysOptions, Long chunkSize)` | A lazy `Flow.Publisher` over all keys, paging with the cursor (`exclusiveStartKey`). Note: here the options' `limit` caps the **total** number of keys yielded (`null`/unset or non-positive = all), whereas for `listKeys`/`createKeysPublicUrl` the same `ListKeysOptions.limit` is a single-request page size. `chunkSize` sets the per-request page size (`null` = server default). | | `recordExists(String key)` | Whether a record exists. Completes with `boolean`. | @@ -167,9 +166,9 @@ rec.ifPresent(r -> System.out.println(new String(r.getValue()))); `SetRecordOptions.contentEncoding(String)` names an encoding already applied to `value` (e.g. `"gzip"` for a pre-compressed file), sent as the `Content-Encoding` header (trimmed first); the client then sends the bytes as they are and does not compress them again, and the API accepts -`gzip`, `br`, `deflate`, or `identity` (matched case-insensitively — `identity` turns the client's -own automatic compression off for this upload). It must not be blank after trimming. The record is -stored exactly as uploaded, but since this client sends no `Accept-Encoding`, a later +`gzip`, `br`, `zstd`, `deflate`, or `identity` (matched case-insensitively; `identity` turns the +client's own automatic compression off for this upload). It must not be blank after trimming. The +record is stored exactly as uploaded, but since this client sends no `Accept-Encoding`, a later `getRecord(...)` transparently receives the decompressed value rather than the stored compressed bytes. @@ -192,23 +191,23 @@ client.keyValueStore("my-store") Both `getRecordPublicUrl` and `createKeysPublicUrl` complete with a `String` URL, signed for private stores and unsigned for public ones. They differ because they sign different things: a single record URL signs the record key directly (there is no expiry to bound), while a key-list URL uses an -expiry-aware storage-content signature — hence only `createKeysPublicUrl` takes an `expiresInSecs` +expiry-aware storage-content signature; hence only `createKeysPublicUrl` takes an `expiresInSecs` (`null` = non-expiring, otherwise positive and at most `4611686018427387`; anything else throws `IllegalArgumentException` before any request is sent). ## Request queues -### `RequestQueueCollectionClient` — `client.requestQueues()` +### `RequestQueueCollectionClient`: `client.requestQueues()` `list(StorageListOptions)`, `iterate(StorageListOptions, Long chunkSize)`, and `getOrCreate(String)`, as for datasets. Unlike datasets/key-value stores, there is no `getOrCreate(String, Object schema)` -overload here — the request-queue creation endpoint does not accept a creation-time schema. +overload here; the request-queue creation endpoint does not accept a creation-time schema. -### `RequestQueueClient` — `client.requestQueue(id)` +### `RequestQueueClient`: `client.requestQueue(id)` | Method | Description | |---|---| -| `get()` / `update(Object)` / `delete()` | Metadata CRUD. On a client reached through a run/task with no queue id of its own (e.g. `run.requestQueue()`), a 404 is ambiguous (parent vs. sub-resource gone) and throws `NotFoundError` instead of resolving to empty/no-op — see [Fetching single resources](../README.md#fetching-single-resources). | +| `get()` / `update(Object)` / `delete()` | Metadata CRUD. On a client reached through a run/task with no queue id of its own (e.g. `run.requestQueue()`), a 404 is ambiguous (parent vs. sub-resource gone) and throws `NotFoundError` instead of resolving to empty/no-op; see [Fetching single resources](../README.md#fetching-single-resources). | | `withClientKey(String)` | A copy that identifies its requests with a stable client key (required for lock operations). | | `listHead(Long limit)` | Requests at the head. Completes with `RequestQueueHead`. | | `addRequest(RequestQueueRequest, boolean forefront)` | Add a request. Completes with `RequestQueueOperationInfo`. | @@ -230,7 +229,7 @@ overload here — the request-queue creation endpoint does not accept a creation > `prolongRequestLock` and `deleteRequestLock` all take a `boolean forefront`. It controls queue > priority: `false` (the common case) processes the request in normal FIFO order; `true` inserts > or moves the request to the *front* of the queue, so it is returned before anything already -> queued — use it for urgent/priority work that should jump the line, e.g. re-queuing a failed +> queued; use it for urgent/priority work that should jump the line, e.g. re-queuing a failed > request for immediate retry or seeding a crawl's very first URLs. ```java @@ -240,10 +239,10 @@ RequestQueueRequest failed = queue.getRequest("REQUEST_ID").join().orElseThrow() queue.updateRequest(failed.setNoRetry(false), true).join(); ``` -> **Naming exception.** Request-queue *requests* are iterated with `paginateRequests(...)` — not an -> `iterate(...)` method — because the request-queue listing is cursor-based rather than +> **Naming exception.** Request-queue *requests* are iterated with `paginateRequests(...)`, not an +> `iterate(...)` method, because the request-queue listing is cursor-based rather than > offset/limit; it always starts from the beginning of the queue (resuming from an explicit -> `exclusiveStartId`/`cursor` is not supported by the publisher — use `listRequests(ListRequestsOptions)` +> `exclusiveStartId`/`cursor` is not supported by the publisher; use `listRequests(ListRequestsOptions)` > directly for that single-page use case). Every other resource uses the > `iterate`/`iterateItems`/`iterateKeys` family. @@ -299,7 +298,7 @@ fields are omitted on the wire): `setId`/`getId` (`String`), `setUrl`/`getUrl` ( `setHandledAt(Instant)`/`getHandledAt()` (`Instant`), `setRetryCount(Integer)`/`getRetryCount()` (`Integer`), `setLoadedUrl(String)`/`getLoadedUrl()` (`String`, the URL actually loaded, after redirects), and -`setErrorMessages(List)`/`getErrorMessages()` (`List`, never `null` — empty when +`setErrorMessages(List)`/`getErrorMessages()` (`List`, never `null`; empty when unset). `getLockExpiresAt()` (read-only, no setter, `Instant`) is populated only on items returned from `listAndLockHead`. diff --git a/docs/tasks.md b/docs/tasks.md index dba8e84..1d41a7c 100644 --- a/docs/tasks.md +++ b/docs/tasks.md @@ -27,7 +27,7 @@ Task task = client.tasks().create(Map.of( | `publish()` / `unpublish()` | Publish/unpublish the task on its public landing page, by setting `isPublic` through `update(Object)`. Both require write permission to both the task and its Actor; `publish()` additionally requires the task's Actor to be public and a configured `publicConfig`. `unpublish()` preserves `publicConfig` so the task can be republished without re-entering it. Complete with the updated `Task`. | | `start(Object input, TaskStartOptions)` | Start a task run (input overrides stored input; `null` uses it). Completes with `ActorRun`. | | `call(Object input, TaskStartOptions, Long waitSecs)` | Start and poll until finished; does **not** stream the run's log. Completes with `ActorRun`. | -| `call(Object input, TaskCallOptions, Long waitSecs)` | As above, additionally streaming the run's log for the duration of the wait by default (matching the reference client's `call` defaulting `options.log` to `'default'`). Use `TaskCallOptions.disableLogStreaming()` to opt out, or `logOptions(StreamedLogOptions)` for a custom destination. | +| `call(Object input, TaskCallOptions, Long waitSecs)` | As above, additionally streaming the run's log for the duration of the wait by default. Use `TaskCallOptions.disableLogStreaming()` to opt out, or `logOptions(StreamedLogOptions)` for a custom destination. | | `getInput()` | The stored input. Completes with `JsonNode`; throws `NotFoundError` if the task is gone (no separate "input absent" state). | | `updateInput(Object)` | Replace the stored input. Completes with `JsonNode`. | | `lastRun(String status)` / `lastRun(LastRunOptions)` | A `RunClient` for the last run (see [`LastRunOptions`](actors.md#actorclient)). | @@ -53,13 +53,13 @@ ActorRun streamed = `Task` fields: `getId()`, `getActId()`, `getUserId()`, `getName()`, `getTitle()`, `getDescription()`, `getCreatedAt()`, `getModifiedAt()`, `getStats()` (`TaskStats`, exposing -`getTotalRuns()`), `getOptions()` (`TaskOptions`: the task's stored default run configuration — +`getTotalRuns()`), `getOptions()` (`TaskOptions`: the task's stored default run configuration: `getBuild()` (`String`), `getTimeoutSecs()` (`Long`), `getMemoryMbytes()` (`Long`), `getRestartOnError()` (`Boolean`)), `getInput()` (a `JsonNode` snapshot of the stored input, from whichever response last returned this `Task` object; prefer `TaskClient.getInput()` above to fetch it fresh on-demand), `getActorStandby()` (`ActorStandby`, from `com.apify.client.actor`, standby-mode configuration overrides for this task, if any), `getIsPublic()` (`Boolean`; derived -from `publicConfig.publishedAt` — use `publish()`/`unpublish()` above to change it), and +from `publicConfig.publishedAt`; use `publish()`/`unpublish()` above to change it), and `getPublicConfig()` (`TaskPublicConfig`, the task's public landing page display configuration, if any). Any field not covered by a typed getter is still available via the inherited `getExtra()` (see @@ -71,9 +71,9 @@ serving standby requests), `getDesiredRequestsPerActorRun()`, `getDisableStandby `getShouldPassActorInput()`. `TaskPublicConfig` fields (all optional; `null` when unset): `getPublishedAt()` (`Instant`; set -when the task is published, `null` when unpublished — read-only, changed via `publish()` / +when the task is published, `null` when unpublished; read-only, changed via `publish()` / `unpublish()`), `getSeoTitle()`, `getSeoDescription()`, `getCategorization()` (not part of the -documented schema; kept for parity with the reference JS client), `getInputSchemaFields()` +documented schema), `getInputSchemaFields()` (`List`), `getDatasetName()`, `getDatasetView()`. ```java diff --git a/docs/webhooks.md b/docs/webhooks.md index 4a8b0cb..ba269ef 100644 --- a/docs/webhooks.md +++ b/docs/webhooks.md @@ -5,7 +5,7 @@ Webhooks notify an external service when specific events occur. Access the colle invocations) are available account-wide via `client.webhookDispatches()` / `client.webhookDispatch(id)` and per-webhook via `client.webhook(id).dispatches()`. -## `WebhookCollectionClient` — `client.webhooks()` +## `WebhookCollectionClient`: `client.webhooks()` The account-wide collection supports both listing and creation. @@ -20,7 +20,7 @@ The account-wide collection supports both listing and creation. `client.actor(id).webhooks()` and `client.task(id).webhooks()` return a `NestedWebhookCollectionClient`. The Apify API only supports **reading** webhooks on those nested paths (`GET /v2/actors/{id}/webhooks`, `GET /v2/actor-tasks/{id}/webhooks`), so this read-only type -exposes `list(ListOptions)` and `iterate(ListOptions, Long chunkSize)` — it has no `create(...)`. To +exposes `list(ListOptions)` and `iterate(ListOptions, Long chunkSize)`; it has no `create(...)`. To create a webhook targeting a specific Actor or task, use `client.webhooks().create(...)` and set the Actor/task in the webhook's `condition`. @@ -57,7 +57,7 @@ System.out.println(dispatch.getId()); `getHeadersTemplate()` are interpolated with values from the triggering event before dispatch), `getCondition()` (`JsonNode`; one of an Actor ID, a task ID or a specific run ID, depending on how the webhook was configured), `isIgnoreSslErrors()`, `isDoNotRetry()`, `getPayloadTemplate()`, -`getHeadersTemplate()`, `getDescription()`, `getLastDispatch()` (`WebhookLastDispatch`, nullable — +`getHeadersTemplate()`, `getDescription()`, `getLastDispatch()` (`WebhookLastDispatch`, nullable: a summary of the most recent dispatch: `getStatus()` one of `"ACTIVE"`/`"SUCCEEDED"`/`"FAILED"`, `getFinishedAt()`/`getRemovedAt()` as `Instant`), and `getStats()` (`WebhookStats`, exposing `getTotalDispatches()`). Any field not covered by a typed getter is still available via the @@ -73,11 +73,11 @@ inherited `getExtra()` (see [the docs index](README.md#model-fields-and-unmodele `WebhookDispatch` fields: `getId()`, `getUserId()`, `getWebhookId()`, `getCreatedAt()` (`Instant`), `getStatus()` (one of `"ACTIVE"`/`"SUCCEEDED"`/`"FAILED"`), `getEventType()` (the event that -triggered the dispatch), `getCalls()` (unmodifiable `List` — each exposing +triggered the dispatch), `getCalls()` (unmodifiable `List`; each exposing `getStartedAt()`/`getFinishedAt()` as `Instant`, `getErrorMessage()`, `getResponseStatus()` -(`Integer`), `getResponseBody()`), `getWebhook()` (`WebhookDispatchWebhookInfo`, nullable — a +(`Integer`), `getResponseBody()`), `getWebhook()` (`WebhookDispatchWebhookInfo`, nullable: a `getRequestUrl()`/`isAdHoc()` summary of the webhook that produced the dispatch), and -`getEventData()` (`WebhookDispatchEventData`, nullable — `getActorRunId()`/`getActorId()`/ +`getEventData()` (`WebhookDispatchEventData`, nullable: `getActorRunId()`/`getActorId()`/ `getActorTaskId()`/`getActorBuildId()`, whichever apply to the triggering event). Any field not covered by a typed getter is still available via the inherited `getExtra()` (see [the docs index](README.md#model-fields-and-unmodeled-data-getextra)). diff --git a/pom.xml b/pom.xml index b8e90ab..8e65c37 100644 --- a/pom.xml +++ b/pom.xml @@ -6,7 +6,7 @@ com.apify apify-client - 0.13.0 + 0.14.0 jar Apify Java Client @@ -42,6 +42,7 @@ 3.2.1 5.10.2 1.23.0 + 1.5.7-4 2.0.13 @@ -79,6 +80,16 @@ true + + + com.github.luben + zstd-jni + ${zstd-jni.version} + true + + org.junit.jupiter junit-jupiter diff --git a/spotbugs-exclude.xml b/spotbugs-exclude.xml index 4c5a74c..d1c344a 100644 --- a/spotbugs-exclude.xml +++ b/spotbugs-exclude.xml @@ -19,7 +19,7 @@ 2. UWF_UNWRITTEN_FIELD / UWF_UNWRITTEN_PUBLIC_OR_PROTECTED_FIELD on Jackson-populated DTOs/response shapes ({@code KeyValueStoreKey}, {@code DeletedRequestInfo}, {@code ActorRunMeta}, {@code TaskOptions}, {@code WebhookLastDispatch}, {@code ActorDefaultRunOptions}, {@code ActorRunStandby}, {@code TaskStats}, {@code BuildActVersion}, - {@code ActorDefinitionMeta}, {@code ActorDefinitionStorages}, + {@code ActorDefinitionMeta}, {@code ActorDefinitionStorages}, {@code SourceCodeFile}, {@code BuildMeta}, {@code BuildOptions}, {@code BuildStats}, {@code PricingInfo}, {@code ProxyGroup}, {@code UserProxy}, {@code WebhookDispatchEventData}, {@code WebhookDispatchWebhookInfo}, the internal error-response mapping in {@code HttpClientCore}): their fields are written by the diff --git a/src/main/java/com/apify/client/ApifyClient.java b/src/main/java/com/apify/client/ApifyClient.java index 2ee3ab6..5c4eb1c 100644 --- a/src/main/java/com/apify/client/ApifyClient.java +++ b/src/main/java/com/apify/client/ApifyClient.java @@ -7,7 +7,6 @@ import com.apify.client.dataset.DatasetClient; import com.apify.client.dataset.DatasetCollectionClient; import com.apify.client.http.DefaultHttpTransport; -import com.apify.client.http.HttpTransport; import com.apify.client.http.Timeout; import com.apify.client.internal.ApiPaths; import com.apify.client.internal.HttpClientCore; @@ -41,10 +40,8 @@ * resource clients via the accessor methods, e.g. {@link #actor(String)}, {@link #dataset(String)}, * {@link #run(String)}. It is safe for concurrent use. * - *

Every network-calling method is asynchronous and non-blocking: it returns a {@link - * java.util.concurrent.CompletableFuture} (or, for a paginated collection's lazy iteration, a - * {@link java.util.concurrent.Flow.Publisher}) instead of blocking the calling thread on the HTTP - * exchange. + *

Every network-calling method is non-blocking and returns a {@link CompletableFuture} (or a + * {@link java.util.concurrent.Flow.Publisher} for lazy iteration of a collection). * *

Quick start

* @@ -57,23 +54,13 @@ * .join(); * } * - *

Architecture

+ *

The HTTP transport is replaceable with {@link ApifyClientBuilder#httpTransport}; the default + * is {@link DefaultHttpTransport}. Authentication, the {@code User-Agent} header, retries and + * timeouts apply to every request. * - *

    - *
  • Public interface: {@link ApifyClient} and the resource clients it returns. - *
  • Replaceable transport: the {@link HttpTransport} interface, with a default {@link - * DefaultHttpTransport}; swap it via {@link ApifyClientBuilder#httpTransport(HttpTransport)}. - *
  • Cross-cutting behaviour (auth, User-Agent, retries with exponential backoff, timeouts) - * lives in the internal HTTP client and is applied to every request, non-blocking end to end. - *
- * - *

Every resource client's constructor is public, but that is an artifact of each living in its - * own resource-scoped package (e.g. {@link ActorClient} in {@code com.apify.client.actor}) rather - * than an invitation to call it directly: they are not meant to be constructed by user code. Always - * obtain one from {@link ApifyClient} or from another resource client's own accessor methods (e.g. - * {@link ActorClient#lastRun}, {@link ActorClient#builds}), which supply the internal wiring (the - * shared HTTP client, base URL, inherited query filters) a direct constructor call would otherwise - * have to reconstruct by hand. + *

Obtain resource clients from this class or from another resource client's accessors, such as + * {@link ActorClient#builds}. Their constructors are public only because they live in separate + * packages, and are not meant to be called directly. */ public final class ApifyClient { @@ -120,27 +107,18 @@ public ApifyClient withTimeout(Timeout timeout) { return new ApifyClient(http.withTimeout(timeout), baseUrl, publicBaseUrl); } - /** - * Returns the {@code User-Agent} header value this client sends. Not part of the public API - * (there is no supported use for an end user to read it back); package-private only so the test - * suite can assert on it. - */ + /** Returns the {@code User-Agent} header value this client sends. Package-private for tests. */ String getUserAgent() { return http.userAgent(); } - /** - * Returns the fully-qualified API base URL this client targets (including the {@code /v2} - * suffix). Not part of the public API, for the same reason as {@link #getUserAgent()}. - */ + /** Returns the API base URL this client targets, including {@code /v2}. For tests. */ String getApiBaseUrl() { return baseUrl; } /** - * Returns the fully-qualified public API base URL this client builds shareable URLs against - * (including the {@code /v2} suffix). Not part of the public API, for the same reason as {@link - * #getUserAgent()}. + * Returns the public API base URL used to build shareable URLs, including {@code /v2}. For tests. */ String getPublicApiBaseUrl() { return publicBaseUrl; @@ -289,10 +267,10 @@ public UserClient user(String id) { /** * Sets the status message of the current Actor run. * - *

This convenience method updates the run identified by the {@code ACTOR_RUN_ID} environment - * variable, so it only works when called from inside an Actor run. Throws {@link - * IllegalStateException} if {@code ACTOR_RUN_ID} is not set. Matches the reference client's - * top-level {@code setStatusMessage}. + *

It updates the run identified by the {@code ACTOR_RUN_ID} environment variable, so it works + * only inside an Actor run. + * + * @throws IllegalStateException if {@code ACTOR_RUN_ID} is not set */ public CompletableFuture setStatusMessage( String message, SetStatusMessageOptions options) { diff --git a/src/main/java/com/apify/client/ApifyClientBuilder.java b/src/main/java/com/apify/client/ApifyClientBuilder.java index d020d00..4223c1e 100644 --- a/src/main/java/com/apify/client/ApifyClientBuilder.java +++ b/src/main/java/com/apify/client/ApifyClientBuilder.java @@ -164,8 +164,8 @@ public ApifyClientBuilder httpTransport(HttpTransport httpTransport) { * Sets how request bodies of at least 1 KiB are compressed (default {@link * com.apify.client.http.BrotliHttpCompressor} where its native library is available, {@link * com.apify.client.http.GzipHttpCompressor} elsewhere). Pass a configured {@code - * BrotliHttpCompressor}/{@code GzipHttpCompressor} to tune the quality, or your own {@link - * HttpCompressor}. + * BrotliHttpCompressor}, {@code GzipHttpCompressor}, or {@code ZstdHttpCompressor} to tune the + * quality, or your own {@link HttpCompressor}. */ public ApifyClientBuilder compression(HttpCompressor compression) { if (compression == null) { @@ -215,12 +215,9 @@ private static Duration requireNonNegative(Duration value, String argName) { } /** - * Validates a non-null, strictly-positive {@link Duration} argument, returning it unchanged. - * Unlike the delay parameters (where zero legitimately means "retry immediately"), a zero {@code - * timeout} is not tolerated: it is passed straight through to {@code HttpRequest.Builder#timeout} - * deep inside {@link DefaultHttpTransport}, which rejects a zero/negative duration at - * request-send time — so a zero value here would build a client that fails on its very first call - * rather than at construction time. + * Returns {@code value} if it is strictly positive. Zero is rejected because {@code + * HttpRequest.Builder#timeout} rejects it at send time, which would fail the first request + * instead of the builder. */ private static Duration requirePositive(Duration value, String argName) { if (value == null || value.isZero() || value.isNegative()) { @@ -295,9 +292,8 @@ private static String trimTrailingSlash(String s) { } /** - * Reports whether the client is running on the Apify platform, by reading the {@code - * APIFY_IS_AT_HOME} environment variable (set to a non-empty value on the platform). Per the - * client requirements the flag is based solely on this variable, matching the JS reference. + * Reports whether the client is running on the Apify platform: the {@code APIFY_IS_AT_HOME} + * environment variable is set to a non-empty value. */ static boolean defaultIsAtHome() { String v = System.getenv(ENV_IS_AT_HOME); @@ -328,11 +324,9 @@ static String buildUserAgent(String suffix, BooleanSupplier isAtHomeFn) { } /** - * Returns the short, lowercase OS token used in the {@code User-Agent}. It matches the - * identifiers emitted by the reference JS client's {@code os.platform()} (Node) — {@code linux}, - * {@code darwin}, {@code win32}, {@code android}, etc. — so all Apify clients report the platform - * uniformly. Java's {@code os.name} system property returns human-readable names ({@code Linux}, - * {@code Mac OS X}, {@code Windows 10}), so it is mapped to the aligned token here. + * Returns the short, lowercase OS token used in the {@code User-Agent}: {@code linux}, {@code + * darwin}, {@code win32}, {@code android}, and so on, as the other Apify clients report it. The + * {@code os.name} property ({@code Mac OS X}, {@code Windows 10}) is mapped to these tokens. */ static String platformToken() { return platformToken(System.getProperty("os.name", ""), System.getProperty("java.vm.name", "")); diff --git a/src/main/java/com/apify/client/ListOptions.java b/src/main/java/com/apify/client/ListOptions.java index d48bf66..296ffc7 100644 --- a/src/main/java/com/apify/client/ListOptions.java +++ b/src/main/java/com/apify/client/ListOptions.java @@ -20,10 +20,8 @@ public ListOptions offset(Long offset) { } /** - * Maximum number of items to return. In {@code list(...)} this is sent verbatim (so {@code 0} - * returns zero items); in {@code iterate(...)} a non-positive/zero {@code limit} means "no cap" — - * iteration yields every item (matching the reference client's falsy-limit-means-unbounded - * intent). + * Maximum number of items to return. {@code list(...)} sends it as it is, so {@code 0} returns no + * items; in {@code iterate(...)} a non-positive limit means no cap. */ public ListOptions limit(Long limit) { this.limit = limit; diff --git a/src/main/java/com/apify/client/PaginationList.java b/src/main/java/com/apify/client/PaginationList.java index 77b7a21..0b9b18e 100644 --- a/src/main/java/com/apify/client/PaginationList.java +++ b/src/main/java/com/apify/client/PaginationList.java @@ -8,9 +8,8 @@ * *

The pagination metadata ({@link #getTotal() total}, {@link #getOffset() offset}, {@link * #getLimit() limit}, {@link #getCount() count}, {@link #isDesc() desc}) accompanies the {@link - * #getItems() items}. Note: {@code total} reflects the API's reported total, which can briefly lag - * immediately after a write (e.g. right after pushing items) because the count is computed - * asynchronously — re-read after a short delay if you need an exact post-write total. + * #getItems() items}. The reported {@code total} can briefly lag right after a write because it is + * computed asynchronously; re-read after a short delay for an exact total. * * @param the item type */ diff --git a/src/main/java/com/apify/client/Version.java b/src/main/java/com/apify/client/Version.java index 8c1d9fe..7ab5ea0 100644 --- a/src/main/java/com/apify/client/Version.java +++ b/src/main/java/com/apify/client/Version.java @@ -13,13 +13,13 @@ public final class Version { * The semantic version of this client library (see SemVer). * Changes to the public interface other than additive ones are considered breaking changes. */ - public static final String CLIENT_VERSION = "0.13.0"; + public static final String CLIENT_VERSION = "0.14.0"; /** * The version of the Apify OpenAPI specification this client was generated and verified against. * Corresponds to the {@code info.version} field of the Apify OpenAPI document. */ - public static final String API_SPEC_VERSION = "v2-2026-11-04T151257Z"; + public static final String API_SPEC_VERSION = "v2-2026-11-09T143926Z"; private Version() {} } diff --git a/src/main/java/com/apify/client/actor/ActorCallOptions.java b/src/main/java/com/apify/client/actor/ActorCallOptions.java index bc8d095..fc6fe2b 100644 --- a/src/main/java/com/apify/client/actor/ActorCallOptions.java +++ b/src/main/java/com/apify/client/actor/ActorCallOptions.java @@ -6,21 +6,13 @@ /** * Configures {@link ActorClient#call(Object, ActorCallOptions, Long)}. * - *

Mirrors {@link ActorStartOptions} (delegating every field to an internal instance) but omits - * {@code waitForFinish}: that field asks the API to hold the HTTP response open server-side while - * the run finishes, which is redundant with (and wastes a request slot next to) {@code call}'s own - * client-side {@code waitSecs} polling. This matches the reference client, whose dedicated {@code - * ActorCallOptions} type is {@code Omit}. + *

It has the options of {@link ActorStartOptions} except {@code waitForFinish}, which {@code + * call} replaces with its own polling. By default {@code call} also streams the run's log to a + * per-run logger while it waits. Use {@link #disableLogStreaming()} to opt out, or {@link + * #logOptions(StreamedLogOptions)} for a custom destination. * - *

By default, {@code call} also streams the run's log to a default per-run logger for the - * duration of the wait (via {@link com.apify.client.run.RunClient#getStreamedLog()}), matching the - * reference client's {@code options.log} defaulting to {@code 'default'}. Use {@link - * #disableLogStreaming()} to opt out, or {@link #logOptions(StreamedLogOptions)} for a custom - * destination. - * - *

{@link #toStartOptions}/{@link #logStreamingEnabledValue}/{@link #logOptionsValue} are - * package-private on purpose — see {@link com.apify.client.internal.RunStartSupport}'s class - * javadoc for why. + *

The package-private accessors are described in {@link + * com.apify.client.internal.RunStartSupport}. */ public final class ActorCallOptions { @@ -98,9 +90,8 @@ public ActorCallOptions waitForResources(Long maxSecs) { } /** - * Disables the default log streaming, matching the reference client's {@code log: null}. {@code - * call} then only starts the run and polls for completion, as if built from {@link - * ActorStartOptions} directly. + * Disables the default log streaming: {@code call} then only starts the run and polls for + * completion. */ public ActorCallOptions disableLogStreaming() { this.logStreamingEnabled = false; diff --git a/src/main/java/com/apify/client/actor/ActorClient.java b/src/main/java/com/apify/client/actor/ActorClient.java index 8228ae3..65c8572 100644 --- a/src/main/java/com/apify/client/actor/ActorClient.java +++ b/src/main/java/com/apify/client/actor/ActorClient.java @@ -91,7 +91,7 @@ public CompletableFuture start(Object input, ActorStartOptions options * still-running run if the wait budget was exhausted). * *

This overload does not stream the run's log; use {@link #call(Object, ActorCallOptions, - * Long)} for that (matching the reference client's default {@code call} behavior). + * Long)} for that. */ public CompletableFuture call(Object input, ActorStartOptions options, Long waitSecs) { return RunStartSupport.call( @@ -106,10 +106,9 @@ public CompletableFuture call(Object input, ActorStartOptions options, /** * Starts the Actor and waits (non-blocking, polling) for it to finish, additionally streaming the - * run's log for the duration of the wait — matching the reference client's {@code call}, whose - * {@code options.log} defaults to {@code 'default'}. {@code waitSecs} bounds the wait; {@code - * null} waits indefinitely. Completes with the finished run (or the still-running run if the wait - * budget was exhausted). + * run's log for the duration of the wait. {@code waitSecs} bounds the wait; {@code null} waits + * indefinitely. Completes with the finished run (or the still-running run if the wait budget was + * exhausted). * *

Log streaming is best-effort: if starting it fails (e.g. the log is not yet available), the * run still starts and is still waited for, just without redirected log output. Use {@link diff --git a/src/main/java/com/apify/client/actor/ActorServerClient.java b/src/main/java/com/apify/client/actor/ActorServerClient.java index b16c200..bbf9590 100644 --- a/src/main/java/com/apify/client/actor/ActorServerClient.java +++ b/src/main/java/com/apify/client/actor/ActorServerClient.java @@ -202,8 +202,8 @@ public CompletableFuture> getOpenApiSpec(Timeout timeout) { * against the server URL, so the request, and the token with it, cannot be sent to another * host. Pass the query as {@link ActorServerRequestOptions#param params}. A fragment ({@code * #...}) is dropped, since it is never sent to the server. - * @param options the method, query, headers, body, compression and timeout; {@code null} sends a - * plain {@code GET} + * @param options the method, query, headers, body, parent request ID, compression and timeout; + * {@code null} sends a plain {@code GET} * @return the response; fails with a {@link ServerActorException} when the Actor is not a Server * Actor, and with a {@link com.apify.client.http.NotFoundError} when the Actor does not exist */ diff --git a/src/main/java/com/apify/client/actor/ActorServerRequestOptions.java b/src/main/java/com/apify/client/actor/ActorServerRequestOptions.java index 3ca1306..410a8af 100644 --- a/src/main/java/com/apify/client/actor/ActorServerRequestOptions.java +++ b/src/main/java/com/apify/client/actor/ActorServerRequestOptions.java @@ -26,10 +26,13 @@ public final class ActorServerRequestOptions { private static final Set METHODS = Set.of("GET", "HEAD", "POST", "PUT", "DELETE", "OPTIONS", "TRACE", "PATCH"); + private static final String PARENT_REQUEST_ID_HEADER = "X-Actor-Request-ID"; + private String method = "GET"; private final Map params = new LinkedHashMap<>(); private final Map headers = new LinkedHashMap<>(); private Object body; + private String parentRequestId; private HttpCompressor compression; private Timeout timeout = Timeout.LONG; @@ -92,6 +95,17 @@ public ActorServerRequestOptions body(Object body) { return this; } + /** + * The {@code X-Actor-Request-ID} header of the request your multi-tenant Server Actor is + * handling. It is sent on as the same header, replacing one set in {@link #headers}, so the + * called Actor charges at that request's pricing tier. Use a client created with the run's {@code + * APIFY_TOKEN}. + */ + public ActorServerRequestOptions parentRequestId(String parentRequestId) { + this.parentRequestId = parentRequestId; + return this; + } + /** * Compresses the request body with {@code compression}, under the same conditions as for API * calls: a body that is large enough and has a content type that is not already compressed, and @@ -125,8 +139,17 @@ Object bodyValue() { return body; } + /** + * The headers to send: the caller's, with the parent request ID header replacing any duplicate. + */ Map headersValue() { - return Collections.unmodifiableMap(headers); + if (parentRequestId == null) { + return Collections.unmodifiableMap(headers); + } + Map merged = new LinkedHashMap<>(headers); + merged.keySet().removeIf(PARENT_REQUEST_ID_HEADER::equalsIgnoreCase); + merged.put(PARENT_REQUEST_ID_HEADER, parentRequestId); + return Collections.unmodifiableMap(merged); } HttpCompressor compressionValue() { diff --git a/src/main/java/com/apify/client/actor/ActorStartOptions.java b/src/main/java/com/apify/client/actor/ActorStartOptions.java index 9f2680a..888fb81 100644 --- a/src/main/java/com/apify/client/actor/ActorStartOptions.java +++ b/src/main/java/com/apify/client/actor/ActorStartOptions.java @@ -9,10 +9,6 @@ /** * Configures starting an Actor run ({@link ActorClient#start}/{@link ActorClient#call}). All fields * are optional. - * - *

{@link #apply}/{@link #contentTypeOrDefault}/{@link #waitForResourcesSecsValue} are - * package-private on purpose — see {@link com.apify.client.internal.RunStartSupport}'s class - * javadoc for why. */ public final class ActorStartOptions { @@ -92,8 +88,8 @@ public ActorStartOptions forcePermissionLevel(String forcePermissionLevel) { } /** - * Ad-hoc webhooks to attach to this run. They are serialized to base64-encoded JSON as the {@code - * webhooks} query parameter, matching the reference clients. + * Ad-hoc webhooks to attach to this run, sent as base64-encoded JSON in the {@code webhooks} + * query parameter. */ public ActorStartOptions webhooks(List webhooks) { this.webhooks = webhooks == null ? null : List.copyOf(webhooks); @@ -103,10 +99,9 @@ public ActorStartOptions webhooks(List webhooks) { /** * Retries the start while the account lacks the memory or a concurrent-run slot for the run (the * API rejects it with {@code actor-memory-limit-exceeded} or {@code - * concurrent-runs-limit-exceeded}, both of which clear as other runs or builds finish), waiting - * 10 seconds between attempts. {@code true} retries until the run starts ({@link - * #WAIT_FOR_RESOURCES_INDEFINITELY}); {@code false} (the default) does not retry — a - * resource-limit error is thrown immediately, like any other error. + * concurrent-runs-limit-exceeded}), waiting 10 seconds between attempts. {@code true} retries + * until the run starts ({@link #WAIT_FOR_RESOURCES_INDEFINITELY}); {@code false} (the default) + * does not retry and throws the error immediately. */ public ActorStartOptions waitForResources(boolean waitForResources) { this.waitForResourcesSecs = waitForResources ? WAIT_FOR_RESOURCES_INDEFINITELY : null; @@ -146,9 +141,8 @@ void apply(QueryParams q) { } /** - * Encodes an ad-hoc webhooks list as base64-encoded JSON, as the API's {@code webhooks} query - * parameter requires. Returns {@code null} for a {@code null} list. Shared by Actor and task - * start options (DRY). + * Encodes an ad-hoc webhooks list as base64-encoded JSON for the {@code webhooks} query + * parameter; returns {@code null} for a {@code null} list. */ public static String encodeWebhooks(List webhooks) { if (webhooks == null) { diff --git a/src/main/java/com/apify/client/actor/ActorVersionCollectionClient.java b/src/main/java/com/apify/client/actor/ActorVersionCollectionClient.java index af0c20a..65f47f8 100644 --- a/src/main/java/com/apify/client/actor/ActorVersionCollectionClient.java +++ b/src/main/java/com/apify/client/actor/ActorVersionCollectionClient.java @@ -20,27 +20,19 @@ public final class ActorVersionCollectionClient { } /** - * Lists the Actor's versions. {@code options} is accepted only for signature consistency with - * every other collection's {@code list(ListOptions)} (matching the reference client, whose {@code - * list()} likewise accepts but never forwards an options argument here) — {@code GET - * .../versions} takes no query parameters at all, so nothing in {@code options} (including {@code - * offset}/{@code limit}/{@code desc}) is sent to the server. Pass {@code null} for the common - * case; use {@link #iterate(ListOptions)}'s {@code limit} to cap the number returned client-side. + * Lists the Actor's versions. The endpoint takes no query parameters, so {@code options} is + * accepted only for consistency with other collections and is never sent. Pass {@code null}; to + * cap the number returned, use the {@code limit} of {@link #iterate(ListOptions)}. */ public CompletableFuture> list(ListOptions options) { return ctx.listResource("", new QueryParams(), ActorVersion.class, Timeout.SHORT); } /** - * Returns a publisher over the Actor's versions. Unlike the paginated collection publishers, this - * fetches eagerly: the single request runs when {@code iterate} is called, not on first demand. - * That's because {@code GET /v2/actors/{actorId}/versions} is not offset/limit paginated at all — - * it takes no pagination parameters and always returns the full version list in one {@code - * {total, items}} response, so routing it through the offset/limit paging engine would loop - * forever (the server returns the same non-empty page at every offset; this is why the sibling - * non-paginated {@code env-vars} collection is also a single-fetch publisher). The options' - * {@code limit} still caps the number yielded ({@code null} or non-positive = all); {@code - * offset} has no effect and there is no page size to tune. + * Returns a publisher over the Actor's versions. The endpoint is not paginated: one request, sent + * when {@code iterate} is called rather than on first demand, returns every version. The {@code + * limit} of {@code options} caps the number yielded ({@code null} or non-positive means all); + * {@code offset} has no effect. */ public Flow.Publisher iterate(ListOptions options) { ListOptions opts = options != null ? options : new ListOptions(); diff --git a/src/main/java/com/apify/client/dataset/DatasetClient.java b/src/main/java/com/apify/client/dataset/DatasetClient.java index 01c3ea1..71a143a 100644 --- a/src/main/java/com/apify/client/dataset/DatasetClient.java +++ b/src/main/java/com/apify/client/dataset/DatasetClient.java @@ -204,7 +204,7 @@ private CompletableFuture> fetchItemsPage( /** * Downloads dataset items serialized in the given format, returning the raw bytes. Unlike {@link * #listItems} (parsed items), this returns the items already serialized to JSON, CSV, XLSX, XML, - * RSS or HTML — useful for exporting. + * RSS or HTML, which is useful for exporting. */ public CompletableFuture downloadItems( DownloadItemsFormat format, DatasetDownloadOptions options) { @@ -253,8 +253,8 @@ private String itemsUrl(DownloadItemsFormat format, DatasetDownloadOptions optio */ public CompletableFuture pushItems(Object items) { byte[] body = items instanceof String json ? serializedItems(json) : serializedItems(items); - // Route through mergedParams like every sibling method, so a context seeded with pinned filters - // (e.g. actor(id).lastRun(...).dataset()) targets the same run's dataset on write as on read. + // mergedParams applies pinned run filters, so a lastRun(...).dataset() client writes to the + // same dataset it reads. String url = ctx.mergedParams(new QueryParams()).applyToUrl(ctx.subUrl("items")); return http.call( "POST", url, body, ResourceContext.CONTENT_TYPE_JSON_CHARSET, Timeout.MEDIUM, true) @@ -355,11 +355,9 @@ public CompletableFuture createItemsPublicUrl( params.addString("signature", sig); } } - // Mirror the JS reference: the public URL is this client's resource path plus the - // signature (over the resolved concrete dataset id) and the explicit options — the - // seeded status/origin filters are deliberately not carried. On a last-run-nested - // client this keeps the unpinned ".../runs/last/dataset" path; see docs/storages.md - // for that limitation. + // The public URL is the resource path, the explicit options and the signature of the + // resolved dataset id. Seeded filters are not carried, so a last-run client keeps the + // unpinned ".../runs/last/dataset" path; see docs/storages.md. return params.applyToUrl(ctx.publicUrl("items")); }); } diff --git a/src/main/java/com/apify/client/dataset/DatasetCollectionClient.java b/src/main/java/com/apify/client/dataset/DatasetCollectionClient.java index 50e2fa8..12feb3c 100644 --- a/src/main/java/com/apify/client/dataset/DatasetCollectionClient.java +++ b/src/main/java/com/apify/client/dataset/DatasetCollectionClient.java @@ -28,8 +28,7 @@ public CompletableFuture getOrCreate(String name) { /** * Gets the dataset with the given name, creating it if it does not exist, applying the given - * {@code schema} on creation. Mirrors the reference client's {@code getOrCreate(name, {schema})}; - * a {@code null} schema behaves like {@link #getOrCreate(String)}. + * {@code schema} on creation. A {@code null} schema behaves like {@link #getOrCreate(String)}. */ public CompletableFuture getOrCreate(String name, Object schema) { return ctx.getOrCreateNamedWithSchema(name, schema, Dataset.class); diff --git a/src/main/java/com/apify/client/http/ApiResponse.java b/src/main/java/com/apify/client/http/ApiResponse.java index a71d955..ffeb0f7 100644 --- a/src/main/java/com/apify/client/http/ApiResponse.java +++ b/src/main/java/com/apify/client/http/ApiResponse.java @@ -8,12 +8,8 @@ * The parsed result of a single API call: the status code, headers and the fully-buffered response * body. * - *

This is a plain, public data carrier - it is legitimately exported, not internal plumbing: - * {@code com.apify.client.internal}'s HTTP core builds it as the result of every call it makes, and - * it is used as the local unwrapped-response type across the resource clients in this module. The - * {@code body} component is exposed raw (no defensive copy) by design; see the {@code - * spotbugs-exclude.xml} header for why that is a deliberate no-copy contract rather than an - * oversight. + *

The {@code body} array is exposed without a defensive copy; see the header of {@code + * spotbugs-exclude.xml}. * * @param statusCode the HTTP status code * @param headers the response headers diff --git a/src/main/java/com/apify/client/http/ApifyApiException.java b/src/main/java/com/apify/client/http/ApifyApiException.java index b82fd9b..e1d97e0 100644 --- a/src/main/java/com/apify/client/http/ApifyApiException.java +++ b/src/main/java/com/apify/client/http/ApifyApiException.java @@ -7,21 +7,14 @@ /** * Thrown for HTTP requests that reach the Apify API but receive a non-success status code. * - *

It mirrors the {@code ApifyApiError} of the reference JavaScript client and exposes the parsed - * error {@link #getType() type}, the human-readable {@link #getMessage() message}, the HTTP {@link - * #getStatusCode() status code}, the number of the final {@link #getAttempt() attempt}, and the - * request {@link #getHttpMethod() method}/{@link #getPath() path}. + *

It exposes the error {@link #getType() type}, the {@link #getMessage() message}, the HTTP + * {@link #getStatusCode() status code}, the final {@link #getAttempt() attempt} number, and the + * request {@link #getHttpMethod() method} and {@link #getPath() path}. * - *

The client throws the subclass matching the response's status code - {@link - * InvalidRequestError} (400), {@link UnauthorizedError} (401), {@link ForbiddenError} (403), {@link - * NotFoundError} (404), {@link ConflictError} (409), {@link RateLimitError} (429) or {@link - * ServerError} (5xx) - so a {@code catch} block can branch with {@code instanceof} instead of - * comparing {@link #getStatusCode()} by number. Any other status is this plain class. Every - * subclass extends this one, so an existing {@code catch (ApifyApiException e)} keeps working - * unchanged. - * - *

It is an unchecked exception so callers are not forced to wrap every call; recover from it - * with a normal {@code try}/{@code catch} where relevant. + *

The client throws the subclass for the status code: {@link InvalidRequestError} (400), {@link + * UnauthorizedError} (401), {@link ForbiddenError} (403), {@link NotFoundError} (404), {@link + * ConflictError} (409), {@link RateLimitError} (429) or {@link ServerError} (5xx). Any other status + * is this plain class. */ public class ApifyApiException extends ApifyClientException { diff --git a/src/main/java/com/apify/client/http/ApifyClientException.java b/src/main/java/com/apify/client/http/ApifyClientException.java index d3b85b3..71f7c58 100644 --- a/src/main/java/com/apify/client/http/ApifyClientException.java +++ b/src/main/java/com/apify/client/http/ApifyClientException.java @@ -3,22 +3,16 @@ /** * Common base of every exception this client throws for a failed API call. * - *

It is unchecked (a {@link RuntimeException}), matching the reference JavaScript client's - * {@code Error}-based, unchecked error model, so callers are not forced to wrap every call in a - * {@code try}/{@code catch}. Three concrete subtypes cover the ways a call can fail: + *

It is unchecked. Three subtypes cover the ways a call can fail: * *

    - *
  • {@link ApifyApiException} — the request reached the API, which answered with a non-success + *
  • {@link ApifyApiException}: the request reached the API, which answered with a non-success * status code. - *
  • {@link ApifyTransportException} — the request never produced an API response at all - * (connection failure, DNS, timeout, or a local failure while preparing the request/response, - * e.g. compression). - *
  • {@link ServerActorException} — a call to an Actor's server cannot be made, since the Actor - * is not a Server Actor or the build that serves its server interface does not exist. + *
  • {@link ApifyTransportException}: the request produced no API response (connection failure, + * DNS, timeout, or a local failure such as compression). + *
  • {@link ServerActorException}: the call to an Actor's server cannot be made, because the + * Actor is not a Server Actor or the build serving its server interface does not exist. *
- * - * Catch this common type to handle all the failure modes uniformly, or catch a specific subtype to - * handle one of them differently. */ public abstract class ApifyClientException extends RuntimeException { diff --git a/src/main/java/com/apify/client/http/DefaultHttpTransport.java b/src/main/java/com/apify/client/http/DefaultHttpTransport.java index cce9f2b..4519393 100644 --- a/src/main/java/com/apify/client/http/DefaultHttpTransport.java +++ b/src/main/java/com/apify/client/http/DefaultHttpTransport.java @@ -42,11 +42,9 @@ public DefaultHttpTransport() { /** Creates a transport with an explicit connection-establishment timeout. */ public DefaultHttpTransport(Duration connectTimeout) { - // Follow redirects (NORMAL, matching the reference clients): some endpoints — e.g. a - // non-attachment key-value-store record GET — answer with a 302 to external storage, which the - // JDK's default Redirect.NEVER would otherwise surface as an error. NORMAL does not follow an - // HTTPS->HTTP downgrade, and the JDK strips the Authorization header on cross-origin hops, so - // the bearer token is not leaked to the redirect target. + // Follow redirects: some endpoints, such as a non-attachment record GET, answer with a 302 to + // external storage. NORMAL does not follow an HTTPS to HTTP downgrade, and the JDK strips the + // Authorization header on cross-origin hops, so the token does not reach the redirect target. this( HttpClient.newBuilder() .connectTimeout(connectTimeout) diff --git a/src/main/java/com/apify/client/http/HttpCompressor.java b/src/main/java/com/apify/client/http/HttpCompressor.java index 6b4a3fe..fe4606d 100644 --- a/src/main/java/com/apify/client/http/HttpCompressor.java +++ b/src/main/java/com/apify/client/http/HttpCompressor.java @@ -8,10 +8,11 @@ * its own; the configured compressor then encodes it and its {@link #contentEncoding()} is sent as * the {@code Content-Encoding} header. Streams are never compressed. * - *

Use {@link BrotliHttpCompressor} (the default, where its native library is available) or - * {@link GzipHttpCompressor}, or implement this interface for another coding. Configure it with - * {@link com.apify.client.ApifyClientBuilder#compression(HttpCompressor)}. Implementations must be - * safe for concurrent use. + *

Use {@link BrotliHttpCompressor} (the default, where its native library is available), {@link + * GzipHttpCompressor}, or {@link ZstdHttpCompressor}, or implement this interface for another + * coding. Configure it with {@link + * com.apify.client.ApifyClientBuilder#compression(HttpCompressor)}. Implementations must be safe + * for concurrent use. */ public interface HttpCompressor { diff --git a/src/main/java/com/apify/client/http/HttpTimeoutException.java b/src/main/java/com/apify/client/http/HttpTimeoutException.java index f4860c0..20a37b6 100644 --- a/src/main/java/com/apify/client/http/HttpTimeoutException.java +++ b/src/main/java/com/apify/client/http/HttpTimeoutException.java @@ -4,17 +4,10 @@ * Signals that an {@link HttpTransport} gave up on a request because it exceeded its timeout, as * opposed to some other transport failure (connection refused, DNS, ...). * - *

Part of the {@link HttpTransport} contract, not tied to any specific transport implementation: - * an implementation that can distinguish "timed out" from other failures should complete its future - * exceptionally with this (wrapping or in place of the underlying exception) so the client can - * apply {@code doNotRetryTimeouts} correctly. A transport implementation that cannot tell the two - * apart may fail with a plain {@link RuntimeException} instead; the call is then simply always - * eligible for a retry. - * - *

Unchecked, matching the rest of the client's async transport contract: a {@link - * java.util.concurrent.CompletableFuture} can only be completed exceptionally with a {@link - * Throwable}, and every other exception this client throws is already unchecked (see {@link - * ApifyClientException}). + *

A transport that can tell a timeout from other failures should complete its future + * exceptionally with this, so the client can apply {@code doNotRetryTimeouts}. A transport that + * cannot may fail with a plain {@link RuntimeException}; the call is then always eligible for a + * retry. */ public class HttpTimeoutException extends RuntimeException { diff --git a/src/main/java/com/apify/client/http/HttpTransport.java b/src/main/java/com/apify/client/http/HttpTransport.java index d703df2..8a3b3f4 100644 --- a/src/main/java/com/apify/client/http/HttpTransport.java +++ b/src/main/java/com/apify/client/http/HttpTransport.java @@ -7,17 +7,15 @@ import java.util.concurrent.CompletableFuture; /** - * The replaceable, non-blocking transport contract of the client. + * The replaceable, non-blocking transport of the client. * - *

Implementations are responsible only for sending a single, fully-prepared request and - * completing the returned future with the raw response — asynchronously, never blocking the calling - * thread on the network round-trip. Authentication, the {@code User-Agent} header, retries and - * (de)serialization are handled by the client, so a transport implementation only needs to perform - * one non-blocking network round-trip. + *

An implementation sends one fully prepared request without blocking the calling thread and + * completes the returned future with the raw response. The client handles authentication, the + * {@code User-Agent} header, retries and (de)serialization. * - *

A non-2xx HTTP status is not an error at this layer — complete the future normally with - * that {@link HttpResponse}. Only a transport-level failure (connection refused, DNS, timeout) - * should complete the future exceptionally. + *

A non-2xx status is not an error here: complete the future normally with the {@link + * HttpResponse}. Only a transport-level failure (connection refused, DNS, timeout) completes it + * exceptionally. * *

Swap the default implementation via {@link ApifyClientBuilder#httpTransport(HttpTransport)} to * share a connection pool, customize TLS/proxy settings, or inject a mock in tests. diff --git a/src/main/java/com/apify/client/http/ZstdHttpCompressor.java b/src/main/java/com/apify/client/http/ZstdHttpCompressor.java new file mode 100644 index 0000000..6cbf6e7 --- /dev/null +++ b/src/main/java/com/apify/client/http/ZstdHttpCompressor.java @@ -0,0 +1,75 @@ +package com.apify.client.http; + +import com.github.luben.zstd.Zstd; +import com.github.luben.zstd.util.Native; + +/** + * An {@link HttpCompressor} that zstd-encodes request bodies ({@code Content-Encoding: zstd}). + * + *

It needs the optional {@code com.github.luben:zstd-jni} dependency; {@link #isAvailable()} + * tells whether it loaded. + */ +public final class ZstdHttpCompressor implements HttpCompressor { + + private static final int MIN_LEVEL = 1; + + /** + * zstd goes up to 22, but levels 20 to 22 use a window above the 8 MB that RFC 9659 requires HTTP + * decoders to support. A record stored at those levels and served with {@code Content-Encoding: + * zstd} could fail to decode in a browser. + */ + private static final int MAX_LEVEL = 19; + + /** zstd's own default: fast, with a ratio close to gzip at its best level. */ + public static final int DEFAULT_LEVEL = 3; + + private final int level; + + /** Whether the zstd-jni library and its native codec loaded. Never throws. */ + public static boolean isAvailable() { + try { + Native.load(); + return true; + } catch (Throwable t) { + // Catch Throwable: a missing class or native library raises LinkageError, not Exception. + return false; + } + } + + /** + * A compressor with the default level, {@value #DEFAULT_LEVEL}. + * + * @throws IllegalStateException if zstd-jni is not available + */ + public ZstdHttpCompressor() { + this(DEFAULT_LEVEL); + } + + /** + * A compressor with the given level. + * + * @param level the compression level from 1 (fastest) to 19 (smallest output) + * @throws IllegalArgumentException if {@code level} is outside 1-19 + * @throws IllegalStateException if zstd-jni is not available + */ + public ZstdHttpCompressor(int level) { + if (level < MIN_LEVEL || level > MAX_LEVEL) { + throw new IllegalArgumentException("zstd level must be between 1 and 19, was " + level); + } + if (!isAvailable()) { + throw new IllegalStateException( + "zstd-jni is not on the classpath or has no native library for this platform"); + } + this.level = level; + } + + @Override + public String contentEncoding() { + return "zstd"; + } + + @Override + public byte[] compress(byte[] data) { + return Zstd.compress(data, level); + } +} diff --git a/src/main/java/com/apify/client/internal/AsyncPaginatedPublisher.java b/src/main/java/com/apify/client/internal/AsyncPaginatedPublisher.java index 2fe01ba..7250da9 100644 --- a/src/main/java/com/apify/client/internal/AsyncPaginatedPublisher.java +++ b/src/main/java/com/apify/client/internal/AsyncPaginatedPublisher.java @@ -8,42 +8,20 @@ import java.util.concurrent.atomic.AtomicLong; /** - * A lazy, backpressure-aware {@link Flow.Publisher} over an offset/limit-paginated list endpoint, - * fetching one page at a time as demand allows. Internal reusable engine shared by every - * offset/limit-paginated collection's {@code iterate} method, so the paging arithmetic lives in one - * place (DRY). (The cursor-based {@code iterateKeys} and the single-fetch {@code versions}/{@code - * env-vars} iterators do not use it.) + * A lazy, backpressure-aware {@link Flow.Publisher} over an offset/limit-paginated list endpoint. + * It fetches one page at a time, as the subscriber signals demand, and supports a single + * subscriber. * - *

This is the client's "reactive" iteration primitive: a {@link java.util.concurrent.Flow} - * ({@code java.util.concurrent.Flow}, the JDK's built-in Reactive Streams interfaces) publisher, - * chosen over a blocking {@link java.util.Iterator} so that following a large paginated collection - * never blocks a thread waiting on the next page's network round-trip. Pages are fetched only as - * the subscriber signals demand via {@link Flow.Subscription#request(long)}. + *

{@code totalLimit} caps the number of rows read across all pages ({@code null} reads + * everything). {@code chunkSize} is the page size ({@code null} lets the API choose); each page + * requests at most the rows left under the cap. A row is an item unless a page's {@link + * PaginationList#getCount() count} differs from the items it returns; all returned items are + * yielded. * - *

Behaviour, mirroring the end-user contract of the reference JS client's iterable {@code - * list()}: - * - *

    - *
  • {@code totalLimit} caps the total number of rows read across all pages ({@code - * null} = read everything). A row is an item unless a page's {@link PaginationList#getCount() - * count} differs from the items it returns; all returned items are yielded. - *
  • {@code chunkSize} is the per-request page size ({@code null} = let the API choose). Each - * page requests {@code min(remaining-under-cap, chunkSize)} rows so the cap is never - * overshot. - *
- * - *

Each page advances the offset by the rows it covered, as {@link PaginationList#getCount()} - * defines them, capped at the page limit that was requested. Iteration stops only when the cap is - * reached or a page covers no rows. It deliberately does not stop on a short page or a - * reported {@code total}: the API clamps an over-large requested page size to its own maximum (so a - * "short" page is common mid-collection), and some endpoints report a {@code total} of {@code 0} or - * a value that lags right after a write (e.g. dataset items) — either signal would truncate - * iteration. Terminating on an empty page costs one extra request at the end but yields the - * complete result in every case. - * - *

Supports a single subscriber (as permitted by the Reactive Streams specification); subscribing - * twice completes the second subscriber's subscription with an error instead of sharing state - * between them. + *

Each page advances the offset by the rows it covered, capped at the requested page limit. + * Iteration stops when the cap is reached or a page covers no rows. It does not stop on a short + * page or on {@code total}: the API clamps large page sizes, and {@code total} can be {@code 0} or + * lag right after a write. The cost is one extra request at the end. */ public final class AsyncPaginatedPublisher implements Flow.Publisher { @@ -104,10 +82,8 @@ private final class Session implements Flow.Subscription { private final AtomicBoolean cancelled = new AtomicBoolean(); private final AtomicBoolean terminated = new AtomicBoolean(); - // Only ever touched from within a `draining`-guarded section, which - by construction below - - // never runs concurrently with itself (the flag admits only one drain loop at a time, and every - // continuation after an async page fetch re-enters through the same guarded path), so no - // separate synchronization on these fields is needed. + // Only touched inside the `draining`-guarded section, which admits one drain loop at a time, + // so these fields need no further synchronization. private List buffer = List.of(); private int pos; private long offset; @@ -125,12 +101,8 @@ private final class Session implements Flow.Subscription { @Override public void request(long n) { if (n <= 0) { - // Deliberately does not go through terminate(): request() is called from outside the - // drain loop (it does not hold the `draining` guard), so - unlike the drain loop's own - // terminal paths - it must not release a guard it may not own. Releasing it here would - // let a concurrent request() elsewhere win the drain() CAS and start a second drain loop - // in parallel with a possibly still-running one, racing on the unsynchronized paging - // fields (mirrors ListPublisher.Session's identical n<=0 handling). + // Not terminate(): request() does not hold the `draining` guard, and releasing it here + // could start a second drain loop that races on the paging fields. if (terminated.compareAndSet(false, true)) { subscriber.onError( new IllegalArgumentException( @@ -160,10 +132,8 @@ private void drain() { } /** - * Emits buffered items while there is demand, fetches the next page when the buffer runs dry - * (or completes the subscriber when the collection is exhausted), and always releases the - * {@code draining} flag exactly once per entry - either directly, or (when a page fetch is - * started) from that fetch's completion callback, which re-enters this same method. + * Emits buffered items while there is demand, then fetches the next page or completes. The + * {@code draining} flag is released once per entry, directly or from the page fetch's callback. */ private void drainLoop() { while (!cancelled.get() && !terminated.get() && requested.get() > 0 && pos < buffer.size()) { @@ -172,19 +142,14 @@ private void drainLoop() { subscriber.onNext(item); } if (cancelled.get() || terminated.get()) { - // `terminated` here means a reentrant request(n<=0) fired onError from within the onNext - // call just above (a subscriber violating Reactive Streams §3.9); stop emitting immediately - // instead of continuing to drain the rest of the buffered demand into a subscriber that has - // already received a terminal signal (RS §1.7). + // `terminated` means a reentrant request(n<=0) delivered onError from onNext above. draining.set(false); return; } if (pos < buffer.size()) { - // Demand exhausted (requested == 0) with items still buffered: stop until more is - // requested. + // Out of demand with items buffered. draining.set(false); - // Re-check demand: a request() on another thread may have arrived between the loop's last - // check and this flag release, which would otherwise be a missed wakeup. + // Re-check: a request() may have arrived between the loop's last check and the release. if (requested.get() > 0 && pos < buffer.size()) { drain(); } @@ -195,13 +160,7 @@ private void drainLoop() { return; } if (fetchInFlight) { - // A fetch from an earlier drain() is still outstanding; its completion will resume this - // loop. With a well-behaved subscriber this is unreachable (the flag is only set/cleared - // within this draining-guarded section, so no concurrent drain can normally observe it - // true here) - but it is a real safety net, not dead code: request(n<=0) no longer - // releases `draining` on its own (see the comment there), yet a subscriber that itself - // races two concurrent request() calls could still reach this branch, and it correctly - // defers to the in-flight fetch's own completion instead of starting a second one. + // A fetch is still outstanding; its completion resumes this loop. draining.set(false); return; } @@ -235,18 +194,14 @@ private void applyPage(PaginationList page) { buffer = page.getItems(); pos = 0; int count = buffer.size(); - // The rows the page covered, which are not the items it returned: see - // PaginationList#getCount. Capped at the rows requested, so an unwound page does not advance - // past rows the next request would never read. + // The rows the page covered (see PaginationList#getCount), capped at the rows requested. long scanned = Math.max(page.getCount(), count); if (requestedLimit != null) { scanned = Math.min(scanned, requestedLimit); } offset += scanned; rowsRead += scanned; - // Stop on the cap or a page that covered nothing (the source is exhausted); never on a short - // *returned* page (the API clamps large page sizes, and filtering can legitimately return - // fewer items than the rows covered) or the reported total (unreliable on some endpoints). + // Stop on the cap or an empty page, never on a short page or the reported total. if (scanned == 0 || (totalLimit != null && rowsRead >= totalLimit)) { exhausted = true; } @@ -265,12 +220,7 @@ private void terminate(Runnable callback) { } } - /** - * Returns the smaller of the two per-page bounds, or {@code null} (server default) when neither - * is set. Both inputs are already positive-or-{@code null}: {@code chunkSize} is normalized in - * the constructor and {@code capRemaining} is only ever {@code > 0} here (the {@code exhausted} - * guard blocks fetching once the cap is reached). - */ + /** Returns the smaller of two positive-or-{@code null} bounds, or {@code null} if both are. */ private static Long minForLimit(Long a, Long b) { if (a == null) { return b; diff --git a/src/main/java/com/apify/client/internal/Extras.java b/src/main/java/com/apify/client/internal/Extras.java index 949c9b2..c877a42 100644 --- a/src/main/java/com/apify/client/internal/Extras.java +++ b/src/main/java/com/apify/client/internal/Extras.java @@ -5,7 +5,7 @@ /** * Helpers for reading typed values out of a model's {@code extra} map ({@link * com.apify.client.ApifyResource#getExtra()}). Shared by every resource client that needs a field - * the model does not (yet) expose as a typed getter (DRY). + * the model does not (yet) expose as a typed getter. */ public final class Extras { diff --git a/src/main/java/com/apify/client/internal/HttpClientCore.java b/src/main/java/com/apify/client/internal/HttpClientCore.java index 983540a..0e5c685 100644 --- a/src/main/java/com/apify/client/internal/HttpClientCore.java +++ b/src/main/java/com/apify/client/internal/HttpClientCore.java @@ -63,9 +63,8 @@ public final class HttpClientCore { private static final int BACKOFF_FACTOR = 2; /** - * Request bodies at or above this size (in bytes) are compressed before being sent. Small bodies - * are left uncompressed because the CPU cost outweighs the transfer saving. Matches the reference - * JS client's 1024-byte threshold. + * Request bodies of at least this many bytes are compressed. Smaller bodies are not worth the CPU + * cost. */ private static final int MIN_COMPRESS_BYTES = 1024; @@ -585,12 +584,9 @@ private CompletableFuture doAttempt( } /** - * Returns {@code min(configured, base * 2^(attempt-1))}: the per-attempt socket timeout. The - * first attempt uses the per-call base timeout; each retry doubles it (so a slow-but-progressing - * connection gets more time) while never exceeding the configured maximum. This mirrors the - * reference client's per-try timeout model ({@code Math.min(timeoutMillis, base * - * 2**(attempt-1))}). Note this bounds each individual attempt, not the cumulative wall-clock time - * across all retries. + * Returns {@code min(configured, base * 2^(attempt-1))}, the timeout of one attempt: the first + * attempt uses the base timeout and each retry doubles it, up to the configured maximum. It + * bounds each attempt, not the total time across retries. */ private Duration attemptTimeout(Duration base, int attempt) { if (base == null) { @@ -679,12 +675,10 @@ private static boolean shouldCompress( /** * Reports whether a request body with the given {@code Content-Type} is worth compressing. - * Images, audio, video and archives already carry their own compression: running them through - * brotli or gzip burns CPU, holds a second full copy of the body in memory, and usually produces - * output slightly larger than the input. A content type that is raw despite such a media type - * (e.g. {@code image/bmp}, {@code audio/wav}) is still compressed, as is a structured-syntax type - * ending in {@code +json}/{@code +xml} (e.g. {@code image/svg+xml}). A missing content type is - * assumed compressible. Matches the reference JS client's {@code isCompressibleContentType}. + * Images, audio, video and archives are already compressed, so compressing them again wastes CPU + * and memory. Raw types under those prefixes (such as {@code image/bmp} or {@code audio/wav}) and + * types ending in {@code +json} or {@code +xml} are still compressed, and so is a missing content + * type. * *

Public so {@code CompressionTest}, outside this non-exported package, can exercise the * classification directly. diff --git a/src/main/java/com/apify/client/internal/Json.java b/src/main/java/com/apify/client/internal/Json.java index da54257..4c16d5b 100644 --- a/src/main/java/com/apify/client/internal/Json.java +++ b/src/main/java/com/apify/client/internal/Json.java @@ -49,7 +49,7 @@ public final class Json { override -> override.setNullHandling(JsonSetter.Value.forValueNulls(Nulls.SKIP))) .changeDefaultPropertyInclusion( incl -> incl.withValueInclusion(JsonInclude.Include.NON_NULL)) - // Bind directly to (private) fields so models need no setters — getters remain the + // Bind directly to (private) fields so models need no setters; getters remain the // public read surface. Any-getters/setters are still honored for the `extra` map. .changeDefaultVisibility( vc -> diff --git a/src/main/java/com/apify/client/internal/ListPublisher.java b/src/main/java/com/apify/client/internal/ListPublisher.java index 7aa6188..c035e1d 100644 --- a/src/main/java/com/apify/client/internal/ListPublisher.java +++ b/src/main/java/com/apify/client/internal/ListPublisher.java @@ -111,10 +111,7 @@ private void drainLoop() { subscriber.onNext(item); } if (cancelled.get() || terminated.get()) { - // `terminated` here means a reentrant request(n<=0) fired onError from within the onNext - // call just above (a subscriber violating Reactive Streams §3.9); stop emitting immediately - // instead of continuing to drain the rest of the buffered demand into a subscriber that has - // already received a terminal signal (RS §1.7). Mirrors AsyncPaginatedPublisher.Session. + // `terminated` means a reentrant request(n<=0) delivered onError from onNext above. draining.set(false); return; } diff --git a/src/main/java/com/apify/client/internal/ResourceContext.java b/src/main/java/com/apify/client/internal/ResourceContext.java index 08c0a12..a3dc385 100644 --- a/src/main/java/com/apify/client/internal/ResourceContext.java +++ b/src/main/java/com/apify/client/internal/ResourceContext.java @@ -17,13 +17,11 @@ import tools.jackson.databind.JavaType; /** - * The resolved context for a resource client: its base URL and the shared HTTP client. The methods - * here implement the CRUD primitives once, so each resource client stays small and consistent - * (DRY). Internal to the client. + * The resolved context for a resource client: its base URL and the shared HTTP client. It + * implements the CRUD primitives the resource clients share. * - *

Every primitive is non-blocking: it returns a {@link CompletableFuture} (or, for a paginated - * collection's lazy iteration, a {@link Flow.Publisher}) rather than blocking the calling thread on - * the underlying HTTP exchange. + *

Every primitive is non-blocking and returns a {@link CompletableFuture} or, for lazy + * iteration, a {@link Flow.Publisher}. */ public final class ResourceContext { @@ -36,22 +34,15 @@ public final class ResourceContext { /** Server-side waitForFinish chunk size (the API caps server waiting at 60 seconds). */ private static final long WAIT_REQUEST_SECS = 60; - /** - * Finite upper bound used when the caller asks to wait "indefinitely" ({@code waitSecs == null}). - * The API will not accept "Infinity", and an unbounded loop can spin forever on a transient 404; - * 999999s (~11.5 days) is effectively indefinite while guaranteeing termination. Mirrors the - * reference client's MAX_WAIT_FOR_FINISH. - */ + /** The wait for {@code waitSecs == null}: about 11.5 days, since the API rejects "Infinity". */ private static final long MAX_WAIT_FOR_FINISH_SECS = 999999; private static final int NOT_FOUND = 404; /** - * Upper bound on how long {@link #waitForFinish} keeps polling a resource that keeps answering - * 404, before giving up and throwing that {@link ApifyApiException}. A resource that was just - * created can 404 briefly (database-replica lag), which this bound rides out; anything longer - * means the id most likely never existed, so polling for the rest of a long (or indefinite) wait - * budget would only delay reporting that. Mirrors the reference client. + * How long {@link #waitForFinish} keeps polling a resource that answers 404 before throwing. A + * just-created resource can 404 briefly because of replica lag; a longer 404 means it most likely + * never existed. */ private static final long MAX_WAIT_FOR_MISSING_RESOURCE_MILLIS = 3000; @@ -189,11 +180,9 @@ public CompletableFuture> getResource( public CompletableFuture> getResource( String subPath, QueryParams params, JavaType dataType, Timeout timeout) { - // ofNullable, not of: an HTTP 200 with body {"data": null} unwraps to null, which is a valid - // "no resource" answer rather than a programming error — never surface it as a raw NPE. - // Explicit witness: getResourceRequired is the *receiver* of .handle below (not an argument - // to another generic call), so unlike a direct `return`, its type parameter cannot be inferred - // from the surrounding target type and must be pinned here. + // {"data": null} in an HTTP 200 unwraps to null, a valid "no resource" answer, so ofNullable + // returns an empty Optional instead of throwing an NPE. + // The explicit is needed because the type cannot be inferred for a method-call receiver. return this.getResourceRequired(subPath, params, dataType, timeout) .handle( (value, error) -> { @@ -296,9 +285,8 @@ public CompletableFuture deleteResource(String subPath) { } /** - * Like {@link #deleteResource}, but - mirroring {@link #getResourceUnlessAmbiguous} - a 404 is - * only treated as a successful no-op when this context addresses its resource by an explicit id. - * Otherwise it is rethrown, since it may mean the parent (not the sub-resource) is gone. + * Like {@link #deleteResource}, but a 404 is a successful no-op only when this context addresses + * its resource by an explicit id. Otherwise it is rethrown, since the parent may be gone. */ public CompletableFuture deleteResourceUnlessAmbiguous(String subPath) { if (ownId) { @@ -370,7 +358,7 @@ public CompletableFuture getOrCreateNamed(String name, Class dataClass * into the {@code {"schema": ...}} request body shape every storage collection's {@code * getOrCreate(name, schema)} sends (a {@code null} schema sends no body, same as {@link * #getOrCreateNamed(String, Class)}). Shared by {@code DatasetCollectionClient}/{@code - * KeyValueStoreCollectionClient} so neither duplicates the wrapping (DRY). + * KeyValueStoreCollectionClient} so neither duplicates the wrapping. */ public CompletableFuture getOrCreateNamedWithSchema( String name, Object schema, Class dataClass) { @@ -380,7 +368,7 @@ public CompletableFuture getOrCreateNamedWithSchema( /** * POST that gets-or-creates a named resource, optionally sending a JSON request body (e.g. a - * storage {@code schema}). A {@code null} body sends no body, matching the plain get-or-create. + * storage {@code schema}). A {@code null} body sends no body. */ public CompletableFuture getOrCreateNamed(String name, Object body, Class dataClass) { QueryParams params = new QueryParams(); @@ -501,8 +489,8 @@ public CompletableFuture getRawRequired( } /** - * Like {@link #getRaw}, but - mirroring {@link #getResourceUnlessAmbiguous} - a 404 is only - * swallowed to {@code null} when this context addresses its resource by an explicit id. + * Like {@link #getRaw}, but a 404 is swallowed to {@code null} only when this context addresses + * its resource by an explicit id. */ public CompletableFuture getRawUnlessAmbiguous( String subPath, QueryParams params, Timeout timeout) { @@ -536,8 +524,7 @@ public CompletableFuture putRaw( /** * As {@link #putRaw(String, QueryParams, byte[], String)}, with an explicit per-request {@code * timeout} and control over whether transport timeouts are retried. Used by key-value-store - * record uploads that expose the reference client's {@code timeoutSecs}/{@code - * doNotRetryTimeouts} write options. + * record uploads. */ public CompletableFuture putRaw( String subPath, @@ -605,10 +592,8 @@ public CompletableFuture waitForFinish( } /** - * The outcome of a single {@code waitForFinish} poll: either a usable fetched value, or "absent" - * - a 404 ({@code notFoundError} set), or a 200 whose {@code data} was {@code null} (tolerated - * the same as "not yet available", not a programming error, matching {@link #getResource}'s own - * null-tolerant handling of that case). + * The outcome of a single {@code waitForFinish} poll: a fetched value, or "absent" because of a + * 404 ({@code notFoundError} set) or a 200 with {@code null} data. */ private record PollOutcome(T value, ApifyApiException notFoundError) { static PollOutcome of(T value) { @@ -722,17 +707,9 @@ private static RuntimeException asRuntimeException(Throwable cause) { } /** - * Encodes a resource id so it is safe to embed in a URL path. Apify uses the {@code - * username~resourcename} form, so every {@code /} in an id is replaced with {@code ~} (not just - * the first: an id with more than one slash must not leave any literal {@code /} that could - * restructure the request path). - * - *

Public (not just intra-package) because {@code RunClient#metamorph} in the {@code .run} - * package needs to apply the same normalization to a resource id supplied as an argument value - * (as opposed to the id embedded in this client's own URL path, which {@link #single} already - * normalizes internally) — a cross-package caller of internal plumbing, same pattern as every - * other member here promoted narrowly for a specific caller (see the package-split rationale in - * this class's other narrowly-public members). + * Encodes a resource id for a URL path: Apify uses the {@code username~resourcename} form, so + * every {@code /} is replaced with {@code ~}. Public because {@code RunClient#metamorph} + * normalizes an id passed as an argument. */ public static String toSafeId(String id) { return id.replace('/', '~'); @@ -742,12 +719,9 @@ public static String toSafeId(String id) { * Percent-encodes a single URL path segment, so that values interpolated into the path (record * keys, request IDs) cannot break out of the segment. * - *

Rejects an empty string or a bare {@code "."}/{@code ".."} segment instead of encoding it: a - * URL parser (client-side, or a proxy/server in front of the API) resolves dot segments - * after percent-decoding, so an encoded {@code "%2E%2E"} would still collapse against - * its neighbours exactly like a literal {@code ".."} once decoded. Rejecting it outright - rather - * than silently encoding something that cannot safely address any record/request - mirrors the - * reference JS client's path-segment sanitization. + *

Rejects an empty string, {@code "."} and {@code ".."} instead of encoding them: a URL parser + * resolves dot segments after percent-decoding, so {@code "%2E%2E"} would collapse like a literal + * {@code ".."}. */ public static String encodePathSegment(String input) { if (input == null || input.isEmpty() || ".".equals(input) || "..".equals(input)) { diff --git a/src/main/java/com/apify/client/internal/RunStartSupport.java b/src/main/java/com/apify/client/internal/RunStartSupport.java index 8f526e8..5cd606b 100644 --- a/src/main/java/com/apify/client/internal/RunStartSupport.java +++ b/src/main/java/com/apify/client/internal/RunStartSupport.java @@ -18,20 +18,14 @@ import org.slf4j.LoggerFactory; /** - * Shared implementation of "start a run" and "call: start, then wait, optionally streaming the log" - * for every client that can start a run from a runnable resource (Actor, Task). + * Shared implementation of starting a run, and of calling it (start, wait, optionally stream the + * log), for Actors and tasks. * - *

{@code ActorStartOptions}/{@code TaskStartOptions}/{@code ActorCallOptions}/{@code - * TaskCallOptions} (in {@code com.apify.client.actor}/{@code com.apify.client.task}) are themselves - * public classes — only the handful of accessor methods this class needs from them ({@code apply}, - * {@code contentTypeOrDefault}, {@code waitForResourcesSecsValue}, {@code toStartOptions}, {@code - * logStreamingEnabledValue}, {@code logOptionsValue}) are package-private, with no public interface - * or shared supertype exposing them. {@code ActorClient}/{@code TaskClient} (same package as their - * respective options types) read those values directly and pass them here as plain values and - * method references (e.g. {@code options::apply} bound as a {@code Consumer}). That - * keeps this one implementation backing both Actor and task calls (instead of each duplicating the - * same ~90 lines) without forcing any of the options builders' internal plumbing — including the - * internal {@link QueryParams} type — into their public API. + *

The accessors this class needs from the start and call options ({@code apply}, {@code + * contentTypeOrDefault}, {@code waitForResourcesSecsValue}, {@code toStartOptions}, {@code + * logStreamingEnabledValue}, {@code logOptionsValue}) are package-private, to keep the internal + * {@link QueryParams} type out of the public API. {@code ActorClient} and {@code TaskClient} read + * them and pass plain values and method references here. */ public final class RunStartSupport { @@ -65,10 +59,10 @@ private RunStartSupport() {} * every start-option query parameter (build, memory, timeout, ...) the caller's options type * configures; {@code contentType} is the input body's content type. * - * @param waitForResourcesSecs mirrors {@link ActorStartOptions#waitForResources(Long)}: {@code - * null} does not retry a start rejected for lack of resources (the default); {@link + * @param waitForResourcesSecs as {@link ActorStartOptions#waitForResources(Long)}: {@code null} + * does not retry a start rejected for lack of resources; {@link * ActorStartOptions#WAIT_FOR_RESOURCES_INDEFINITELY} retries until it succeeds; any other - * value bounds the retrying to that many seconds, after which the last such error is thrown + * value bounds the retrying to that many seconds */ public static CompletableFuture start( ResourceContext ctx, @@ -176,11 +170,8 @@ public static CompletableFuture call( } /** - * Starts a run and waits for it to finish, additionally streaming the run's log for the duration - * of the wait by default, matching the reference client's {@code call} defaulting {@code - * options.log} to {@code 'default'}. Log streaming is best-effort: if starting it fails (e.g. the - * log is not yet available), the run still starts and is still waited for, just without - * redirected log output. + * Starts a run and waits for it to finish, streaming the run's log during the wait by default. + * Log streaming is best-effort: if it fails to start, the run is still started and awaited. * * @param applyParams applies every start-option query parameter the caller's call-options type * configures (delegated from its paired start-options type) diff --git a/src/main/java/com/apify/client/internal/Signatures.java b/src/main/java/com/apify/client/internal/Signatures.java index 986fb00..95ddcb3 100644 --- a/src/main/java/com/apify/client/internal/Signatures.java +++ b/src/main/java/com/apify/client/internal/Signatures.java @@ -8,9 +8,8 @@ import javax.crypto.spec.SecretKeySpec; /** - * Apify storage-content URL signing, byte-for-byte compatible with the platform's - * {@code @apify/utilities} implementation that the reference clients rely on. Internal to the - * client. + * Apify storage-content URL signing, compatible with the platform's {@code @apify/utilities} + * implementation. */ public final class Signatures { @@ -69,8 +68,8 @@ static String toBase62(BigInteger value) { } /** - * Builds a storage-content signature for a resource's public URL, byte-for-byte compatible with - * upstream {@code createStorageContentSignature}. + * Builds a storage-content signature for a resource's public URL, compatible with upstream {@code + * createStorageContentSignature}. * *

It signs the message {@code "{version}.{expiresAtMillis}.{resourceId}"} ({@code * expiresAtMillis} is the absolute expiry in ms, or {@code 0} for a non-expiring URL) with {@link diff --git a/src/main/java/com/apify/client/keyvalue/KeyValueStoreClient.java b/src/main/java/com/apify/client/keyvalue/KeyValueStoreClient.java index 054024f..fb89fc5 100644 --- a/src/main/java/com/apify/client/keyvalue/KeyValueStoreClient.java +++ b/src/main/java/com/apify/client/keyvalue/KeyValueStoreClient.java @@ -105,10 +105,9 @@ public Flow.Publisher iterateKeys(ListKeysOptions options, Lon } /** - * Lazily publishes a store's keys via the cursor-based ({@code exclusiveStartKey}) listing, - * fetching a page only once the subscriber has signalled demand for it. Supports a single - * subscriber, like {@link com.apify.client.internal.AsyncPaginatedPublisher}, whose draining - * design this mirrors (cursor pagination instead of offset/limit is the only real difference). + * Publishes the store's keys page by page through the cursor-based ({@code exclusiveStartKey}) + * listing, as the subscriber signals demand. Like {@link + * com.apify.client.internal.AsyncPaginatedPublisher}, it supports a single subscriber. */ private final class KeysPublisher implements Flow.Publisher { private final Long chunkSize; @@ -287,8 +286,8 @@ public CompletableFuture recordExists(String key) { } /** - * Fetches a record by key, or empty if it does not exist. Like the reference client, it requests - * the record as an attachment so the API returns the raw bytes directly rather than redirecting. + * Fetches a record by key, or empty if it does not exist. The record is requested as an + * attachment, so the API returns the raw bytes instead of redirecting. */ public CompletableFuture> getRecord(String key) { return getRecord(key, new GetRecordOptions().attachment(true)); @@ -323,8 +322,7 @@ public CompletableFuture setRecord(String key, byte[] value, String conten */ public CompletableFuture setRecord( String key, byte[] value, String contentType, SetRecordOptions options) { - // Trimmed before validating/forwarding, matching the reference client's - // z.string().trim().min(1) schema (it sends the trimmed value, not the original). + // The trimmed value is validated and sent. String contentEncoding = options.contentEncodingValue(); if (contentEncoding != null) { contentEncoding = contentEncoding.trim(); @@ -371,9 +369,8 @@ public CompletableFuture getRecordPublicUrl(String key) { params.addString("signature", Signatures.createHmacSignature(secret, key)); } } - // Public URL = resource path + signature only, matching the JS reference. - // Seeded filters are not carried; see DatasetClient.createItemsPublicUrl and - // docs/storages.md for the last-run caveat. + // The public URL is the resource path plus the signature. Seeded filters are not + // carried; see docs/storages.md for the last-run caveat. return params.applyToUrl( ctx.publicUrl("records/" + ResourceContext.encodePathSegment(key))); }); @@ -421,8 +418,8 @@ public CompletableFuture createKeysPublicUrl( Signatures.signStorageContent(secret, store.get().getId(), expiresInSecs)); } } - // Public URL = resource path + the explicit key-listing options + signature, matching - // the JS reference (see the last-run caveat documented in docs/storages.md). + // The public URL is the resource path, the explicit options and the signature (see + // the last-run caveat in docs/storages.md). return params.applyToUrl(ctx.publicUrl("keys")); }); } diff --git a/src/main/java/com/apify/client/keyvalue/KeyValueStoreCollectionClient.java b/src/main/java/com/apify/client/keyvalue/KeyValueStoreCollectionClient.java index 7766807..948db4c 100644 --- a/src/main/java/com/apify/client/keyvalue/KeyValueStoreCollectionClient.java +++ b/src/main/java/com/apify/client/keyvalue/KeyValueStoreCollectionClient.java @@ -28,8 +28,7 @@ public CompletableFuture getOrCreate(String name) { /** * Gets the store with the given name, creating it if it does not exist, applying the given {@code - * schema} on creation. Mirrors the reference client's {@code getOrCreate(name, {schema})}; a - * {@code null} schema behaves like {@link #getOrCreate(String)}. + * schema} on creation. A {@code null} schema behaves like {@link #getOrCreate(String)}. */ public CompletableFuture getOrCreate(String name, Object schema) { return ctx.getOrCreateNamedWithSchema(name, schema, KeyValueStore.class); diff --git a/src/main/java/com/apify/client/keyvalue/SetRecordOptions.java b/src/main/java/com/apify/client/keyvalue/SetRecordOptions.java index 3017815..6e9f798 100644 --- a/src/main/java/com/apify/client/keyvalue/SetRecordOptions.java +++ b/src/main/java/com/apify/client/keyvalue/SetRecordOptions.java @@ -4,8 +4,7 @@ /** * Write options for {@link KeyValueStoreClient#setRecord(String, byte[], String, - * SetRecordOptions)}, mirroring the reference client's {@code timeoutSecs}/{@code - * doNotRetryTimeouts}/{@code contentEncoding}. + * SetRecordOptions)}: the timeout, whether to retry timeouts, and the content encoding. */ public final class SetRecordOptions { private Timeout timeout; @@ -40,18 +39,12 @@ public SetRecordOptions doNotRetryTimeouts(Boolean doNotRetryTimeouts) { /** * Encoding already applied to the record's {@code value}, sent as the {@code Content-Encoding} - * header (trimmed first; it must not be blank). Set it to upload a pre-compressed value (e.g. a - * gzipped file read from disk): the client then sends the bytes as they are and does not compress - * them again. The API accepts {@code gzip}, {@code br}, {@code deflate}, and {@code identity} - * (matched case-insensitively) and stores the record exactly as uploaded; since this client does - * not advertise support for a compressed response ({@code Accept-Encoding}), a later {@link - * KeyValueStoreClient#getRecord(String) getRecord} transparently receives the decompressed value - * rather than the stored compressed bytes. Passing {@code identity} turns the client's own - * automatic compression off for this upload without changing how the client is configured. - * - *

Unlike the reference JS client, this has no corresponding type check: {@code value} here is - * always raw bytes (never a string or an object the client would serialize to JSON), so there is - * no non-binary case to reject. + * header (trimmed; not blank). Use it to upload a pre-compressed value: the client sends the + * bytes as they are and does not compress them again. The API accepts {@code gzip}, {@code br}, + * {@code zstd}, {@code deflate} and {@code identity} (case-insensitive) and stores the record as + * uploaded. Since the client sends no {@code Accept-Encoding}, {@link + * KeyValueStoreClient#getRecord(String) getRecord} returns the decompressed value. {@code + * identity} turns off the client's own compression for this upload. */ public SetRecordOptions contentEncoding(String contentEncoding) { this.contentEncoding = contentEncoding; diff --git a/src/main/java/com/apify/client/log/LogClient.java b/src/main/java/com/apify/client/log/LogClient.java index e8e0def..9515ee5 100644 --- a/src/main/java/com/apify/client/log/LogClient.java +++ b/src/main/java/com/apify/client/log/LogClient.java @@ -59,10 +59,8 @@ public CompletableFuture> get(LogOptions options) { } /** - * As {@link #get()}, but resolves to the log's raw bytes instead of decoding them as text - - * mirroring the reference clients' {@code buffer}/{@code get_as_bytes} option. Useful when the - * log is not UTF-8 text, or the caller wants to avoid the decoding step. See {@link #get()} for - * 404 handling. + * As {@link #get()}, but resolves to the log's raw bytes instead of decoding them as text. See + * {@link #get()} for 404 handling. */ public CompletableFuture> getAsBytes() { return getAsBytes(new LogOptions()); diff --git a/src/main/java/com/apify/client/log/StreamedLog.java b/src/main/java/com/apify/client/log/StreamedLog.java index ce39d38..38d6882 100644 --- a/src/main/java/com/apify/client/log/StreamedLog.java +++ b/src/main/java/com/apify/client/log/StreamedLog.java @@ -15,6 +15,8 @@ import java.util.ArrayList; import java.util.List; import java.util.concurrent.CompletableFuture; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.locks.LockSupport; import java.util.function.Consumer; import java.util.regex.Matcher; import java.util.regex.Pattern; @@ -22,17 +24,14 @@ import org.slf4j.LoggerFactory; /** - * Helper for redirecting a run's (or build's) streamed log to another destination, mirroring the - * reference client's {@code StreamedLog}. + * Redirects a run's or build's streamed log to another destination. * - *

It follows the live raw log stream in the background and forwards each complete, timestamped - * log message to a destination {@link Consumer} (by default a per-run prefixed SLF4J {@link - * Logger}). Messages are parsed on log-line boundaries (the Apify platform prefixes every line with - * an ISO-8601 timestamp), so multi-line messages are kept intact across stream chunks. + *

It follows the live log stream in the background and forwards each complete log message to a + * {@link Consumer} (by default a per-run prefixed SLF4J {@link Logger}). Messages start at a line + * with an ISO-8601 timestamp, so multi-line messages stay intact. * - *

Lifecycle: call {@link #start()} to begin redirection and {@link #stop()} to end it. The class - * is {@link AutoCloseable}, so it can be used in a try-with-resources block; {@link #close()} stops - * redirection if it is still running. + *

Call {@link #start()} to begin and {@link #stop()} to end. {@link #close()} stops redirection + * if it is still running, so the class works in try-with-resources. * *

Typical use: * @@ -71,27 +70,18 @@ public final class StreamedLog implements AutoCloseable { /** Upper bound on the pause between reopens of an empty log stream. */ private static final long EMPTY_STREAM_MAX_RETRY_MILLIS = 5_000; - /** - * How long {@link #stop()}/{@link #close()} wait for a message already in flight before - * force-closing the stream, matching the reference client's {@code STOP_GRACE_MILLIS}. - */ + /** How long {@link #stop()} and {@link #close()} wait for a message in flight before closing. */ private static final long STOP_GRACE_MILLIS = 1_000; private final LogClient logClient; private final Consumer destination; - /** - * Messages older than this instant are dropped ({@code null} means "redirect from the start", so - * nothing is dropped). Set to construction time when {@code fromStart} is {@code false}. - */ + /** Messages older than this are dropped; {@code null} drops nothing. */ private final Instant relevancyTimeLimit; private volatile boolean stopLogging; - /** - * Set once a destination consumer throws, so redirection unwinds on the first failure and - * forwards nothing further (matching the reference client). Reset per {@link #start()}. - */ + /** Set when the destination consumer throws; nothing is forwarded after that. */ private volatile boolean forwardingFailed; private volatile InputStream activeStream; @@ -189,99 +179,117 @@ public synchronized CompletableFuture start() { /** * Stops log redirection, waiting for the background reader thread to finish. * - *

Called from any other thread (the normal case) this joins the reader before returning. - * Called from inside the destination consumer - which runs on the reader thread - it - * cannot join itself, so it only signals the stop and returns immediately; redirection then - * unwinds as the reader returns from the consumer. For stopping from inside the consumer prefer - * {@link #close()}: the reader's final flush can re-invoke the consumer for the last buffered - * message, and a second {@code stop()} on the by-then-stopped helper throws, whereas {@code - * close()} is idempotent. + *

Called from inside the destination consumer, which runs on the reader thread, it cannot join + * itself: it only signals the stop and returns. If the calling thread is interrupted, it closes + * the stream, keeps the interrupt flag, and returns before the reader has ended, so the last + * message may arrive after it returns. {@link #start()} throws until a later {@code stop()} or + * {@link #close()} has joined the reader. * - * @throws IllegalStateException if redirection is not running - including a repeat call from - * within the consumer after an initial self-stop has already cleared the running thread + * @throws IllegalStateException if redirection is not running */ - public synchronized void stop() { - if (streamingThread == null) { - throw new IllegalStateException("Streaming task is not active"); + public void stop() { + StopHandle handle; + synchronized (this) { + if (streamingThread == null) { + throw new IllegalStateException("Streaming task is not active"); + } + handle = signalStop(); } - stopStreaming(); + awaitStopped(handle); } /** * Stops redirection if it is still running; a no-op otherwise. If {@link #start()} is still * opening the log stream, the stream is closed as soon as it opens, no reader is launched and the - * future {@code start()} returned is cancelled. Fully idempotent and safe under a concurrent - * {@link #stop()} or a double {@code close()}: the running-check and the stop happen atomically - * under the monitor, so unlike {@link #stop()} this never throws. + * future {@code start()} returned is cancelled. Waits for the reader thread like {@link #stop()} + * does, and handles an interrupted caller and a call from the consumer the same way. Idempotent + * and safe to call concurrently with {@code stop()}; unlike it, never throws. */ @Override - public synchronized void close() { - if (starting) { - closeRequested = true; + public void close() { + StopHandle handle = null; + synchronized (this) { + if (starting) { + closeRequested = true; + } + if (streamingThread != null) { + handle = signalStop(); + } } - if (streamingThread != null) { - stopStreaming(); + if (handle != null) { + awaitStopped(handle); } } /** - * Signals the reader to stop and joins it (unless called from the reader thread itself, which - * cannot join itself). Callers must hold the monitor and must have already checked that a thread - * is running. + * The reader thread being stopped, its stream, and the timer that closes the stream after the + * grace window. + */ + private record StopHandle( + Thread thread, InputStream stream, CompletableFuture graceTimer) {} + + /** + * Signals the reader to stop. The caller holds the monitor and has checked that a thread is + * running, and joins it with {@link #awaitStopped} after releasing the monitor: the consumer runs + * on the reader thread and may call {@link #close()}, which needs the monitor. */ - private void stopStreaming() { + private StopHandle signalStop() { stopLogging = true; - // Grace window before force-closing the live stream, so a message already in flight (e.g. the - // run's last line, sent right as it finishes) still has a chance to arrive and be processed - // instead of being cut off - matching the reference client's STOP_GRACE_MILLIS. A blocked read - // either completes normally within this window or is unblocked by the forced close below. + // Grace window before closing the stream, so the run's last line still arrives. InputStream streamToForceClose = activeStream; CompletableFuture graceTimer = Async.delay(Duration.ofMillis(STOP_GRACE_MILLIS)) .thenRun(() -> closeQuietly(streamToForceClose)); - // Wakes the reader if it is currently backing off before reopening an empty log stream (a plain - // Thread.sleep - see awaitNonEmptyStream), which neither the stream close above nor the grace - // window otherwise interrupts. It is a wake signal only, not a cancellation the reader must - // honor: sleepAsWakeableBackoff clears it, matching its own javadoc. - streamingThread.interrupt(); - // A thread must never join itself. The destination consumer runs on the streaming thread, so a - // user calling stop()/close() from inside that consumer (e.g. "stop redirecting once I see line - // X") would otherwise join() the current thread on itself and block forever - and because - // stop()/close() are synchronized, that hung thread keeps the monitor, deadlocking every later - // start/stop/close with no exception. Setting stopLogging already unwinds the reader loop - // cooperatively (no read is blocked from inside the consumer's own call stack), so in that - // self-stop case we skip only the join; the field is still cleared below so lifecycle state - // stays consistent. - if (Thread.currentThread() != streamingThread) { - try { - streamingThread.join(); - } catch (InterruptedException e) { - Thread.currentThread().interrupt(); + // Wakes the reader if it is backing off before reopening an empty stream. Not an interrupt: + // that would reach a consumer that is running or blocked in its own call. + LockSupport.unpark(streamingThread); + return new StopHandle(streamingThread, streamToForceClose, graceTimer); + } + + /** + * Joins the stopped reader and clears the state. A stop from the consumer, which runs on the + * reader thread, neither joins nor clears. A join that is interrupted closes the stream, keeps + * the interrupt flag, and does not clear. In both cases {@link #start()} stays rejected until a + * later {@code stop()} or {@code close()} has joined the reader. + */ + private void awaitStopped(StopHandle handle) { + if (Thread.currentThread() == handle.thread()) { + return; + } + boolean joined = true; + try { + handle.thread().join(); + } catch (InterruptedException e) { + joined = false; + Thread.currentThread().interrupt(); + // The grace window cannot be waited out, so end the reader now instead of leaving it blocked. + closeQuietly(handle.stream()); + } + // Cancelling ends the grace window early and keeps a late timer from closing a stream that a + // later start() opened. + handle.graceTimer().cancel(false); + if (joined) { + synchronized (this) { + if (streamingThread == handle.thread()) { + streamingThread = null; + } } } - // The reader has exited (or, in the self-stop case, will unwind on its own without needing the - // forced close) - cancelling here is what makes the grace window end early instead of always - // costing the full STOP_GRACE_MILLIS, and prevents a late timer firing from closing a stream a - // subsequent start() may already have replaced. - graceTimer.cancel(false); - streamingThread = null; } /** Reads the live raw log stream and forwards complete messages to the destination. */ private void streamLog() { - // Bytes after the last newline: an incomplete trailing line kept for the next read so a message - // split across chunk boundaries (including a partial UTF-8 sequence) is not corrupted. Declared - // outside the try so the `finally` block below can still emit it during the final flush. + // Bytes after the last newline, kept for the next read so that a message split across chunks, + // even in the middle of a UTF-8 sequence, is not corrupted. Declared outside the try so the + // final flush can emit it. byte[] lineRemainder = new byte[0]; - // Decoded log text seen so far but not yet split into complete messages. Local to this reader - // (not a shared field) so that if a start() ever races a still-draining reader, the two readers - // never mutate the same non-thread-safe buffer. + // Decoded text not yet split into complete messages. Local, so overlapping readers never share + // it. StringBuilder pending = new StringBuilder(); InputStream stream; try { - // The API serves the log of a run that has not logged anything yet as a stream that ends at - // once; reopen it until it actually delivers a byte, so redirection does not silently give up - // on a run that is still queued or starting. + // The API serves the log of a run that has not logged yet as a stream that ends at once. + // Reopen it until a byte arrives, so a queued or starting run is not given up on. stream = awaitNonEmptyStream(); } catch (IOException e) { if (!stopLogging) { @@ -290,14 +298,7 @@ private void streamLog() { return; } if (stream == null) { - // Stopped before any stream ever delivered a byte: fetch whatever the log holds right now in - // one plain request instead of redirecting nothing at all. Clears a pending interrupt status - // first - stopStreaming()'s wake-up interrupt (see its javadoc) may not yet have been - // consumed - // by a Thread.sleep if it landed between two awaitNonEmptyStream attempts, and it must not - // leak - // into the destination consumer invoked below or into the blocking HTTP call just below that. - Thread.interrupted(); + // Stopped before any byte arrived: fetch the log as it is now. try { String logContent = logClient.get(new LogOptions().raw(true)).join().orElse(""); if (!forwardingFailed) { @@ -308,15 +309,12 @@ private void streamLog() { } return; } - // The stream was opened (possibly reopened by awaitNonEmptyStream) and stored in activeStream; - // own its lifecycle here via try-with-resources so the reader closes it when it exits - // (idempotent - // with stop()'s close, which may close it first to unblock a pending read). + // The reader owns the stream and closes it on exit; stop() may close it first to end a read. try (InputStream liveStream = stream) { byte[] readBuffer = new byte[READ_BUFFER_BYTES]; int read; // Process each chunk before honouring a stop request, so a stop issued right after start - // still redirects whatever the run has already produced (matching the reference client). + // still redirects whatever the run has already produced. while ((read = liveStream.read(readBuffer)) != -1) { byte[] combined = new byte[lineRemainder.length + read]; System.arraycopy(lineRemainder, 0, combined, 0, lineRemainder.length); @@ -336,24 +334,13 @@ private void streamLog() { } } } catch (IOException e) { - // A read error after an explicit stop is expected (the stream was closed under us). Surface - // only genuine, unsolicited failures, and do so without throwing from the background thread. + // A read error after an explicit stop is expected. Report only unsolicited failures. if (!stopLogging) { LOG.warn("Log redirection stopped due to error", e); } } finally { - // Clears a pending interrupt status before the flush below can invoke the destination - // consumer: stopStreaming()'s wake-up interrupt (see its javadoc) is only guaranteed consumed - // when it lands during awaitNonEmptyStream's Thread.sleep; landing instead while this loop - // was - // running would otherwise leak it into user code. - Thread.interrupted(); - // Flush whatever is left when the stream ends OR when a stop closes the stream mid-read: the - // last complete message is held back in `pending` by emitMessages(..., false), and a stop - // unblocks the read via an IOException that skips the loop body. Running the final flush in - // finally guarantees that retained message (and any unterminated trailing line) is still - // delivered on stop, matching the reference client. Skipped once a consumer has thrown: there - // is nothing more to deliver and the flush must not re-invoke the failed consumer. + // Flush on stream end and on a stop mid-read: emitMessages holds back the last message and a + // stop skips the loop body. Skipped once a consumer has thrown. if (!forwardingFailed) { emitMessages(pending, new String(lineRemainder, StandardCharsets.UTF_8), true); } @@ -371,17 +358,13 @@ private InputStream awaitNonEmptyStream() throws IOException { InputStream stream = activeStream; long backoffMillis = EMPTY_STREAM_RETRY_MILLIS; while (true) { - // Always attempt the read before consulting stopLogging - like the main loop below, which - // processes a chunk before honouring a stop request - so a stop issued right after start() - // still drains whatever is already buffered (e.g. a finite stream in tests) instead of - // discarding it. + // Read before checking stopLogging, so a stop right after start() still drains buffered data. byte[] probe = new byte[READ_BUFFER_BYTES]; int read; try { read = stream.read(probe); } catch (IOException e) { - // An explicit stop can close `stream` out from under this read; that is expected, not a - // genuine failure (mirrors the main loop's own IOException handling below). + // An explicit stop closes the stream under this read. if (stopLogging) { return null; } @@ -411,17 +394,9 @@ private InputStream awaitNonEmptyStream() throws IOException { } throw new IOException("failed to reopen log stream", cause); } - // A plain volatile write (no lock needed for a single field): stop()/close() read this same - // field to decide what to force-close, and a `synchronized` publish here could instead - // deadlock against a concurrent stop()/close() - both are `synchronized` methods that may be - // blocked in streamingThread.join() while holding that monitor, which this reader thread - // could - // never then acquire. + // A plain volatile write, so the reader never waits for the monitor here. activeStream = stream; - // A stop() requested while the reopen above was in flight force-closed the *previous* stream - // (the one this method already closed on seeing it empty), not this newly opened one - so it - // would otherwise be handed to the caller with nothing left able to unblock a read that then - // blocks. Closing it here too keeps every reopened stream subject to the same stop signal. + // A stop during the reopen closed the previous stream, not this one, so close it here. if (stopLogging) { closeQuietly(stream); return null; @@ -430,16 +405,14 @@ private InputStream awaitNonEmptyStream() throws IOException { } /** - * Sleeps for {@code millis}, waking early if interrupted and clearing (rather than re-raising) - * the interrupt status - {@link #stopStreaming()} uses interruption purely to wake this wait - * promptly; {@link #stopLogging}, checked by the caller right after, is the actual stop signal, - * so a lingering interrupt status would only risk breaking a later, unrelated blocking call. + * Sleeps for {@code millis}, waking early when {@link #signalStop()} sets {@link #stopLogging}. */ - private static void sleepAsWakeableBackoff(long millis) { - try { - Thread.sleep(millis); - } catch (InterruptedException ignored) { - // Intentionally not restoring the interrupt status - see method javadoc. + private void sleepAsWakeableBackoff(long millis) { + long deadline = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(millis); + long remaining = deadline - System.nanoTime(); + while (!stopLogging && remaining > 0) { + LockSupport.parkNanos(this, remaining); + remaining = deadline - System.nanoTime(); } } @@ -510,10 +483,8 @@ private void emitIfRelevant(String message, String timestamp) { try { destination.accept(message); } catch (RuntimeException e) { - // The destination is a user-supplied consumer running on this background daemon thread; an - // uncaught throw would kill the thread. Match the reference client: stop redirecting on the - // first failure (forwardingFailed unwinds the emit loop and skips the final flush, so the - // consumer is not invoked again) and log a single warning instead of propagating. + // The consumer runs on the background thread: stop at the first failure and log one warning + // instead of propagating. forwardingFailed = true; stopLogging = true; LOG.warn("Log redirection stopped due to error", e); diff --git a/src/main/java/com/apify/client/log/StreamedLogOptions.java b/src/main/java/com/apify/client/log/StreamedLogOptions.java index b346815..8de3d4e 100644 --- a/src/main/java/com/apify/client/log/StreamedLogOptions.java +++ b/src/main/java/com/apify/client/log/StreamedLogOptions.java @@ -6,8 +6,8 @@ /** * Configures {@link RunClient#getStreamedLog(StreamedLogOptions)} log redirection. * - *

Mirrors the reference client's {@code getStreamedLog} options: a destination log ({@code - * toLog}) and whether to include messages produced before redirection started ({@code fromStart}). + *

Sets the destination ({@code toLog}) and whether to include messages produced before + * redirection started ({@code fromStart}). */ public final class StreamedLogOptions { diff --git a/src/main/java/com/apify/client/requestqueue/BatchAddRequestsOptions.java b/src/main/java/com/apify/client/requestqueue/BatchAddRequestsOptions.java index 354ca33..051cb97 100644 --- a/src/main/java/com/apify/client/requestqueue/BatchAddRequestsOptions.java +++ b/src/main/java/com/apify/client/requestqueue/BatchAddRequestsOptions.java @@ -2,18 +2,18 @@ /** * Tuning options for {@link RequestQueueClient#batchAddRequests(java.util.List, boolean, - * BatchAddRequestsOptions)}, mirroring the reference client. Requests the API reports as - * unprocessed (typically due to rate limiting) are automatically retried with exponential backoff. + * BatchAddRequestsOptions)}. Requests the API reports as unprocessed (typically due to rate + * limiting) are automatically retried with exponential backoff. */ public final class BatchAddRequestsOptions { - /** Default number of retry rounds for unprocessed requests (matches the reference client). */ + /** Default number of retry rounds for unprocessed requests. */ static final int DEFAULT_MAX_UNPROCESSED_RETRIES = 3; - /** Default maximum number of batch API calls made in parallel (matches the reference client). */ + /** Default maximum number of batch API calls made in parallel. */ static final int DEFAULT_MAX_PARALLEL = 5; - /** Default minimum delay before retrying unprocessed requests (matches the reference client). */ + /** Default minimum delay before retrying unprocessed requests. */ static final long DEFAULT_MIN_DELAY_BETWEEN_UNPROCESSED_RETRIES_MILLIS = 500; private int maxUnprocessedRequestsRetries = DEFAULT_MAX_UNPROCESSED_RETRIES; diff --git a/src/main/java/com/apify/client/requestqueue/BatchAddResult.java b/src/main/java/com/apify/client/requestqueue/BatchAddResult.java index edb99eb..1454389 100644 --- a/src/main/java/com/apify/client/requestqueue/BatchAddResult.java +++ b/src/main/java/com/apify/client/requestqueue/BatchAddResult.java @@ -22,11 +22,8 @@ public List getUnprocessedRequests() { } /** - * Appends another result's requests into this one (used to merge per-chunk batch results). - * Tolerates {@code other}'s lists being {@code null} (Jackson assigns the field directly on an - * explicit {@code "processedRequests": null}/{@code "unprocessedRequests": null} response, - * bypassing the constructor's empty-list default), mirroring {@link #getProcessedRequests()}/ - * {@link #getUnprocessedRequests()}'s own null-coalescing. + * Appends the requests of another result, which may have {@code null} lists when the API returned + * an explicit {@code null}. */ void merge(BatchAddResult other) { if (other.processedRequests != null) { diff --git a/src/main/java/com/apify/client/requestqueue/RequestQueueClient.java b/src/main/java/com/apify/client/requestqueue/RequestQueueClient.java index 5c70c7d..0207ab1 100644 --- a/src/main/java/com/apify/client/requestqueue/RequestQueueClient.java +++ b/src/main/java/com/apify/client/requestqueue/RequestQueueClient.java @@ -30,29 +30,16 @@ public final class RequestQueueClient { /** The API limit on requests per batch call; larger inputs are split into chunks of this size. */ private static final int MAX_REQUESTS_PER_BATCH = 25; - /** - * The API's request-body payload-size limit for a batch-add call, in bytes (9 MiB), matching the - * reference JS client's {@code MAX_PAYLOAD_SIZE_BYTES}. A batch of up to {@link - * #MAX_REQUESTS_PER_BATCH} requests can still exceed this if the requests are individually large - * (e.g. sizeable {@code userData}), so {@link #batchAddRequests(List, boolean, - * BatchAddRequestsOptions)} additionally splits chunks by cumulative JSON-encoded byte size to - * avoid a 413 (Payload Too Large). - */ + /** The API's payload-size limit for a batch-add call, in bytes (9 MiB). */ private static final long MAX_PAYLOAD_SIZE_BYTES = 9L * 1024 * 1024; - /** - * Safety margin subtracted from {@link #MAX_PAYLOAD_SIZE_BYTES} before slicing by byte size, so a - * chunk sized right at the boundary is not rejected due to minor framing overhead. Matches the - * reference client's {@code SAFETY_BUFFER_PERCENT}. - */ + /** Fraction of {@link #MAX_PAYLOAD_SIZE_BYTES} kept free for framing overhead. */ private static final double PAYLOAD_SAFETY_BUFFER_PERCENT = 0.01 / 100; /** The two bytes of an empty JSON array ({@code []}), the baseline for a byte-size chunk. */ private static final int EMPTY_JSON_ARRAY_BYTES = 2; - /** - * Upper bound on the unprocessed-requests backoff exponent, so {@code 2^attempt} cannot blow up. - */ + /** Upper bound on the backoff exponent for unprocessed requests. */ private static final int MAX_BACKOFF_EXPONENT = 10; private final HttpClientCore http; @@ -221,30 +208,20 @@ public CompletableFuture batchAddRequests( /** * Adds multiple requests to the queue. If {@code forefront} is true, they are added to the front. * - *

The input is automatically split into chunks of at most 25 requests (the API count limit) - * that additionally respect the API's ~9 MiB request-body payload-size limit ({@link - * #MAX_PAYLOAD_SIZE_BYTES}): a chunk of up to 25 requests is further split by cumulative - * JSON-encoded byte size, so a batch of individually large requests (e.g. sizeable {@code - * userData}) cannot 413. Chunks are sent using up to {@link BatchAddRequestsOptions#maxParallel} - * concurrently in-flight chunk requests, and any requests the API leaves unprocessed (typically - * rate-limited) are retried with exponential backoff up to {@link - * BatchAddRequestsOptions#maxUnprocessedRequestsRetries} times. The per-chunk results are merged - * into a single {@link BatchAddResult}. + *

The input is split into chunks of at most 25 requests (the API limit) and of at most {@link + * #MAX_PAYLOAD_SIZE_BYTES} of JSON, with up to {@link BatchAddRequestsOptions#maxParallel} chunks + * in flight. Requests the API leaves unprocessed are retried with exponential backoff up to + * {@link BatchAddRequestsOptions#maxUnprocessedRequestsRetries} times. The chunk results are + * merged into one {@link BatchAddResult}. * - *

This method's returned future never completes exceptionally due to an API/transport failure, - * matching the reference client's contract: any request that could not be confirmed processed — - * whether due to persistent rate-limiting, server errors, or a non-retryable client error (e.g. - * an invalid token or insufficient permissions) — is returned in {@link - * BatchAddResult#getUnprocessedRequests()} instead. A single request whose own JSON encoding - * already exceeds the payload-size limit is rejected up front with {@link - * IllegalArgumentException}, since no chunk size could ever fit it. + *

The future does not fail on an API or transport error: a request that could not be confirmed + * as processed is returned in {@link BatchAddResult#getUnprocessedRequests()}. A single request + * whose JSON exceeds the payload limit is rejected up front with {@link + * IllegalArgumentException}. * - *

Caveat: processed-vs-unprocessed reconciliation after a retry matches requests by - * {@link RequestQueueRequest#getUniqueKey()}. If a request omits {@code uniqueKey} (it is - * nullable), a request that actually succeeded server-side on an earlier attempt can still be - * reported in {@link BatchAddResult#getUnprocessedRequests()} after retries, because there is no - * caller-supplied key to match it back against the API's per-attempt response. Callers that rely - * on {@code batchAddRequests}' return value should set {@code uniqueKey} explicitly. + *

Retries match requests by {@link RequestQueueRequest#getUniqueKey()}. Set {@code uniqueKey} + * explicitly: without it, a request that succeeded on an earlier attempt can still be reported as + * unprocessed. */ public CompletableFuture batchAddRequests( List requests, boolean forefront, BatchAddRequestsOptions options) { @@ -293,11 +270,9 @@ public CompletableFuture batchAddRequests( } /** - * One "worker": repeatedly claims the next not-yet-started chunk (via {@code nextChunkIndex}) and - * processes it, until every chunk has been claimed. Running {@code workerCount} of these - * concurrently bounds how many chunk requests (including their own retries) are in flight at - * once, matching {@link BatchAddRequestsOptions#maxParallel} without needing a thread pool - each - * worker is a chain of async continuations, not a blocked thread. + * One worker: claims the next unstarted chunk through {@code nextChunkIndex} and processes it, + * until all chunks are claimed. Running {@code workerCount} workers bounds the chunk requests in + * flight to {@link BatchAddRequestsOptions#maxParallel}, without a thread pool. */ private CompletableFuture runBatchAddWorker( List> chunks, @@ -320,9 +295,8 @@ private CompletableFuture runBatchAddWorker( /** * Returns the longest leading run of {@code requests} whose combined JSON payload stays under * {@code maxByteLength}, always keeping at least one request so the caller's chunking loop makes - * forward progress. Ports the reference client's {@code sliceArrayByByteLength}. {@code - * startIndex} is the position of {@code requests.get(0)} in the caller's full input list, used - * only to report which request is oversized. + * forward progress. {@code startIndex} is the position of {@code requests.get(0)} in the caller's + * full input list, used only to report which request is oversized. * * @throws IllegalArgumentException if any individual request's own JSON encoding already exceeds * {@code maxByteLength} (no chunk size could ever fit it) @@ -394,13 +368,9 @@ private static byte[] jsonArray(List requests) { /** * Adds one chunk (already sized to the API limit), retrying requests the API leaves unprocessed - * with exponential backoff (a scheduled delay, not a blocked thread; see {@link Async}). The - * returned future never completes exceptionally, matching the reference client's {@code - * _batchAddRequestsWithRetries}: on any failure — a non-retryable 4xx/5xx API response, or a - * transport-level failure (connection error, timeout) that never produced a response at all — the - * remaining requests in the chunk are simply reported as unprocessed rather than surfaced as an - * exception, so the method keeps a single, uniform never-fails contract across both {@link - * ApifyApiException} and {@link com.apify.client.http.ApifyTransportException}. + * with exponential backoff (see {@link Async}). The returned future never completes + * exceptionally: after a non-retryable API error or a transport failure, the remaining requests + * of the chunk are reported as unprocessed. */ private CompletableFuture batchAddChunkWithRetries( List chunk, boolean forefront, BatchAddRequestsOptions options) { @@ -440,10 +410,8 @@ private CompletableFuture batchAddAttempt( processed.addAll(response.getProcessedRequests()); } List stillRemaining = requestsNotYetProcessed(chunk, processed); - // Any client-level failure (response == null, i.e. the batchAddChunk call above - // failed) stops retrying this chunk immediately; whatever has not been confirmed - // processed is reported as unprocessed below, never thrown - matching the reference - // client's contract. + // response == null means the batchAddChunk call failed: stop retrying this chunk and + // report what is not confirmed as unprocessed. boolean stop = response == null || stillRemaining.isEmpty() || attempt >= maxRetries; if (stop) { return CompletableFuture.completedFuture(finish(chunk, processed)); @@ -494,9 +462,8 @@ private static List requestsNotYetProcessed( } /** - * {@code (1 + random) * 2^attempt * minDelay} — exponential backoff with jitter, matching the - * reference. The exponent is capped so a pathologically large retry count cannot overflow the - * delay into an absurd (or negative) duration. + * {@code (1 + random) * 2^attempt * minDelay}: exponential backoff with jitter. The exponent is + * capped so the delay cannot overflow. */ private static Duration backoffDelay(int attempt, long minDelayMillis) { if (minDelayMillis <= 0) { @@ -592,26 +559,22 @@ public CompletableFuture unlockRequests() { } /** - * Returns a lazy, backpressure-aware publisher over all requests in the queue, fetching pages of - * up to {@code pageLimit} requests at a time ({@code null} for the server default). Equivalent to - * {@link #paginateRequests(Long, Long, List) paginateRequests(null, pageLimit, null)} (no total - * cap, no state filter). + * Returns a lazy publisher over all requests in the queue, with pages of up to {@code pageLimit} + * requests ({@code null} for the server default). Equivalent to {@link #paginateRequests(Long, + * Long, List) paginateRequests(null, pageLimit, null)}. */ public Flow.Publisher paginateRequests(Long pageLimit) { return paginateRequests(null, pageLimit, null); } /** - * Returns a lazy, backpressure-aware publisher over the queue's requests via the cursor-based - * listing endpoint. {@code totalLimit} caps the total number of requests yielded across all pages - * ({@code null}/non-positive = unbounded); {@code chunkSize} is the per-request page size ({@code - * null} = server default); {@code filter} restricts to requests in the given states ({@link - * ListRequestsOptions#FILTER_LOCKED}/{@link ListRequestsOptions#FILTER_PENDING}, {@code null} = - * no filter), matching {@link #listRequests(ListRequestsOptions)}'s filter. + * Returns a lazy publisher over the queue's requests, starting from the beginning of the queue. * - *

Always starts from the beginning of the queue; resuming from an explicit {@code - * exclusiveStartId}/{@code cursor} is not supported here (use {@link - * #listRequests(ListRequestsOptions)} directly for that single-page use case). + * @param totalLimit the maximum number of requests yielded, or {@code null} or non-positive for + * all + * @param chunkSize the page size, or {@code null} for the server default + * @param filter the request states to include ({@link ListRequestsOptions#FILTER_LOCKED}, {@link + * ListRequestsOptions#FILTER_PENDING}), or {@code null} for all */ public Flow.Publisher paginateRequests( Long totalLimit, Long chunkSize, List filter) { @@ -619,11 +582,9 @@ public Flow.Publisher paginateRequests( } /** - * Lazily publishes a request queue's requests via the cursor-based listing endpoint, fetching a - * page only once the subscriber has signalled demand. Supports a single subscriber, like {@link - * com.apify.client.internal.AsyncPaginatedPublisher}, whose draining design this mirrors (cursor - * pagination instead of offset/limit, plus the "not yet started" distinction the cursor-based - * termination check needs, are the only real differences). + * Publishes the queue's requests page by page through the cursor-based listing endpoint, as the + * subscriber signals demand. Like {@link com.apify.client.internal.AsyncPaginatedPublisher}, it + * supports a single subscriber. */ private final class RequestsPublisher implements Flow.Publisher { private final Long totalLimit; diff --git a/src/main/java/com/apify/client/requestqueue/RequestQueueRequest.java b/src/main/java/com/apify/client/requestqueue/RequestQueueRequest.java index 245db39..bd8782d 100644 --- a/src/main/java/com/apify/client/requestqueue/RequestQueueRequest.java +++ b/src/main/java/com/apify/client/requestqueue/RequestQueueRequest.java @@ -58,12 +58,12 @@ public RequestQueueRequest setUrl(String url) { } /** - * The deduplication key for the request. Nullable — if unset, the API derives one from the - * request (typically the URL + method). Note that {@link - * RequestQueueClient#batchAddRequests(List, boolean, BatchAddRequestsOptions) batchAddRequests}' - * retry reconciliation matches requests back to the API's response by this field, so a request - * without an explicit {@code uniqueKey} can be falsely reported unprocessed after a retry even - * though it succeeded server-side; set it explicitly if you rely on that return value. + * The deduplication key for the request. Nullable; if unset, the API derives one from the request + * (typically the URL and method). Note that {@link RequestQueueClient#batchAddRequests(List, + * boolean, BatchAddRequestsOptions) batchAddRequests}' retry reconciliation matches requests back + * to the API's response by this field, so a request without an explicit {@code uniqueKey} can be + * falsely reported unprocessed after a retry even though it succeeded server-side; set it + * explicitly if you rely on that return value. */ public String getUniqueKey() { return uniqueKey; diff --git a/src/main/java/com/apify/client/run/LastRunOptions.java b/src/main/java/com/apify/client/run/LastRunOptions.java index 2a1829e..bfaf706 100644 --- a/src/main/java/com/apify/client/run/LastRunOptions.java +++ b/src/main/java/com/apify/client/run/LastRunOptions.java @@ -7,10 +7,7 @@ * Filters which "last" run the {@link ActorClient#lastRun}/{@link TaskClient#lastRun} accessors * resolve to. Leave a field unset to leave that filter unset. * - *

{@code origin} is now a spec-declared query parameter on the {@code runs/last} endpoints - * (alongside {@code status}), matching the reference client's {@code lastRun({status, origin})}. - * The spec also declares {@code waitForFinish} on those endpoints, but the reference client does - * not expose it on {@code lastRun}, so neither do we. + *

{@code waitForFinish}, which the endpoints also accept, is not exposed. */ public final class LastRunOptions { private String status; diff --git a/src/main/java/com/apify/client/run/RunChargeOptions.java b/src/main/java/com/apify/client/run/RunChargeOptions.java index 6c19fc9..2d45e47 100644 --- a/src/main/java/com/apify/client/run/RunChargeOptions.java +++ b/src/main/java/com/apify/client/run/RunChargeOptions.java @@ -26,7 +26,7 @@ public RunChargeOptions count(Long count) { /** * A key that deduplicates the charge across retries. If unset, one is auto-generated as {@code - * "{runId}-{eventName}-{timestampMillis}-{random}"}, matching the reference client. + * "{runId}-{eventName}-{timestampMillis}-{random}"}. */ public RunChargeOptions idempotencyKey(String idempotencyKey) { this.idempotencyKey = idempotencyKey; @@ -36,7 +36,7 @@ public RunChargeOptions idempotencyKey(String idempotencyKey) { /** * The ID of the Server Actor request being charged, taken from the {@code X-Actor-Request-ID} * header of the incoming request. Required for multi-tenant Server Actor runs, where it - * identifies the request the charge belongs to; omit it otherwise. + * identifies the request the charge belongs to. Runs of single-tenant Server Actors ignore it. */ public RunChargeOptions requestId(String requestId) { this.requestId = requestId; diff --git a/src/main/java/com/apify/client/run/RunClient.java b/src/main/java/com/apify/client/run/RunClient.java index 2dd5570..7bcc127 100644 --- a/src/main/java/com/apify/client/run/RunClient.java +++ b/src/main/java/com/apify/client/run/RunClient.java @@ -60,12 +60,8 @@ private RunClient(ApifyClient root, ResourceContext ctx, String id) { } /** - * Builds a {@code RunClient} for a "last run" accessor ({@code runs/last}, nested under an Actor - * or task), with the {@code status}/{@code origin} filters from {@code options} pinned as query - * parameters inherited by every call on the returned client. Shared by {@code - * ActorClient#lastRun}/{@code TaskClient#lastRun} so the construction logic (and the once-off use - * of {@link ResourceContext#seedParams}, which returns a new, still-immutable context rather than - * mutating one in place) lives in a single place (DRY). + * Builds a {@code RunClient} for a {@code runs/last} accessor of an Actor or task, with the + * {@code status} and {@code origin} filters of {@code options} sent on every call. */ public static RunClient lastRun( ApifyClient root, HttpClientCore http, String parentUrl, LastRunOptions options) { @@ -120,11 +116,10 @@ public CompletableFuture abort(Boolean gracefully) { } /** - * Transforms the run into a run of another Actor with a new input. {@code targetActorId} is the - * Actor to metamorph into ({@code username/actor-name} or the plain Actor id; normalized to the - * URL-safe {@code username~actor-name} form before sending, matching the reference client's - * {@code _toSafeId}); {@code input} is the new input: a JSON-serializable value, a {@code byte[]} - * or {@link String} that is sent as it is, or {@code null} for none. + * Transforms the run into a run of another Actor with a new input. {@code targetActorId} is + * {@code username/actor-name} or the plain Actor id; {@code username/actor-name} is sent as + * {@code username~actor-name}. {@code input} is a JSON-serializable value, a {@code byte[]} or + * {@link String} sent as it is, or {@code null} for none. */ public CompletableFuture metamorph( String targetActorId, Object input, MetamorphOptions options) { @@ -166,8 +161,8 @@ public CompletableFuture resurrect(RunResurrectOptions options) { * Charges for a pay-per-event Actor run: records occurrences of a named event. Only meaningful * for runs of pay-per-event Actors. * - *

An idempotency key is always sent (auto-generated if not provided), so a charge that is - * retried by the transport is applied at most once, matching the reference client. + *

An idempotency key is always sent (generated if not provided), so a retried charge is + * applied at most once. */ public CompletableFuture charge(RunChargeOptions options) { String eventName = options.eventNameValue(); @@ -218,10 +213,9 @@ private String generateIdempotencyKey(String eventName) { /** * Polls until the run reaches a terminal state or {@code waitSecs} elapses ({@code null} waits - * indefinitely). Completes with the latest run fetched, whether or not it is terminal: if the - * wait budget runs out first, this completes with the still-running run as of the last poll - * rather than failing or waiting further — check {@link ActorRun#isTerminal()} on the result if - * the distinction matters to the caller. + * indefinitely). Completes with the latest run fetched, terminal or not: if the wait runs out + * first, it completes with the still-running run of the last poll. Check {@link + * ActorRun#isTerminal()} on the result. */ public CompletableFuture waitForFinish(Long waitSecs) { return ctx.waitForFinish(waitSecs, "run", Json.type(ActorRun.class), ActorRun::isTerminal); @@ -349,14 +343,12 @@ public CompletableFuture getStreamedLog(StreamedLogOptions options) } /** - * Builds the per-run prefix used by the default redirection destination, mirroring the reference - * client: the Actor name (looked up from the run) and {@code runId:{id}}, joined with a space and - * followed by {@code " -> "}. Falls back to just the run id when the Actor name is unavailable. + * Builds the prefix of the default redirection destination: the Actor name, {@code runId:{id}}, + * and {@code " -> "}. Falls back to the run id alone when the Actor name is unavailable. */ private CompletableFuture buildDefaultLogPrefix() { - // Fall back to the field `id`: correct for a direct run client, but it is the literal string - // "last" for a `runs/last` accessor (see #lastRun) — replaced with the resolved run's real id - // below as soon as the fetch succeeds. + // The field `id` is the literal "last" for a `runs/last` accessor; the resolved run's id + // replaces it below once the fetch succeeds. return get() .thenCompose( run -> { diff --git a/src/main/java/com/apify/client/task/TaskCallOptions.java b/src/main/java/com/apify/client/task/TaskCallOptions.java index e7097ab..d3ef643 100644 --- a/src/main/java/com/apify/client/task/TaskCallOptions.java +++ b/src/main/java/com/apify/client/task/TaskCallOptions.java @@ -6,21 +6,13 @@ /** * Configures {@link TaskClient#call(Object, TaskCallOptions, Long)}. * - *

Mirrors {@link TaskStartOptions} (delegating every field to an internal instance) but omits - * {@code waitForFinish}: that field asks the API to hold the HTTP response open server-side while - * the run finishes, which is redundant with (and wastes a request slot next to) {@code call}'s own - * client-side {@code waitSecs} polling. This matches the reference client, whose dedicated {@code - * ActorCallOptions} type (shared by tasks) is {@code Omit}. + *

It has the options of {@link TaskStartOptions} except {@code waitForFinish}, which {@code + * call} replaces with its own polling. By default {@code call} also streams the run's log to a + * per-run logger while it waits. Use {@link #disableLogStreaming()} to opt out, or {@link + * #logOptions(StreamedLogOptions)} for a custom destination. * - *

By default, {@code call} also streams the run's log to a default per-run logger for the - * duration of the wait (via {@link com.apify.client.run.RunClient#getStreamedLog()}), matching the - * reference client's {@code options.log} defaulting to {@code 'default'}. Use {@link - * #disableLogStreaming()} to opt out, or {@link #logOptions(StreamedLogOptions)} for a custom - * destination. - * - *

{@link #toStartOptions}/{@link #logStreamingEnabledValue}/{@link #logOptionsValue} are - * package-private on purpose — see {@link com.apify.client.internal.RunStartSupport}'s class - * javadoc for why. + *

The package-private accessors are described in {@link + * com.apify.client.internal.RunStartSupport}. */ public final class TaskCallOptions { @@ -83,9 +75,8 @@ public TaskCallOptions waitForResources(Long maxSecs) { } /** - * Disables the default log streaming, matching the reference client's {@code log: null}. {@code - * call} then only starts the run and polls for completion, as if built from {@link - * TaskStartOptions} directly. + * Disables the default log streaming: {@code call} then only starts the run and polls for + * completion. */ public TaskCallOptions disableLogStreaming() { this.logStreamingEnabled = false; diff --git a/src/main/java/com/apify/client/task/TaskClient.java b/src/main/java/com/apify/client/task/TaskClient.java index cc5e6f6..5206e66 100644 --- a/src/main/java/com/apify/client/task/TaskClient.java +++ b/src/main/java/com/apify/client/task/TaskClient.java @@ -92,7 +92,7 @@ public CompletableFuture start(Object input, TaskStartOptions options) * wait; {@code null} waits indefinitely. * *

This overload does not stream the run's log; use {@link #call(Object, TaskCallOptions, - * Long)} for that (matching the reference client's default {@code call} behavior). + * Long)} for that. */ public CompletableFuture call(Object input, TaskStartOptions options, Long waitSecs) { return RunStartSupport.call( @@ -107,9 +107,8 @@ public CompletableFuture call(Object input, TaskStartOptions options, /** * Starts the task and waits (non-blocking, polling) for it to finish, additionally streaming the - * run's log for the duration of the wait — matching the reference client's {@code call}, whose - * {@code options.log} defaults to {@code 'default'}. {@code waitSecs} bounds the wait; {@code - * null} waits indefinitely. + * run's log for the duration of the wait. {@code waitSecs} bounds the wait; {@code null} waits + * indefinitely. * *

Log streaming is best-effort: if starting it fails (e.g. the log is not yet available), the * run still starts and is still waited for, just without redirected log output. Use {@link diff --git a/src/main/java/com/apify/client/task/TaskPublicConfig.java b/src/main/java/com/apify/client/task/TaskPublicConfig.java index 4f0f25b..031d561 100644 --- a/src/main/java/com/apify/client/task/TaskPublicConfig.java +++ b/src/main/java/com/apify/client/task/TaskPublicConfig.java @@ -12,9 +12,7 @@ * null}. {@code publishedAt} is read-only - use {@link TaskClient#publish()} and {@link * TaskClient#unpublish()} to change the publication state. * - *

{@link #getCategorization()} is not part of the documented {@code TaskPublicConfig} schema in - * the OpenAPI spec, but is kept here for parity with the reference JS client, which also exposes - * it. Every other field below is part of the documented schema. + *

{@link #getCategorization()} is not in the OpenAPI schema of {@code TaskPublicConfig}. */ public final class TaskPublicConfig extends ApifyResource { private Instant publishedAt; diff --git a/src/main/java/com/apify/client/task/TaskStartOptions.java b/src/main/java/com/apify/client/task/TaskStartOptions.java index 25d6bd4..bddc25d 100644 --- a/src/main/java/com/apify/client/task/TaskStartOptions.java +++ b/src/main/java/com/apify/client/task/TaskStartOptions.java @@ -8,13 +8,11 @@ /** * Configures starting a task run ({@link TaskClient#start}/{@link TaskClient#call}). * - *

It mirrors {@link ActorStartOptions} but omits the fields the task run endpoint does not - * accept (the Actor-only {@code contentType} and {@code forcePermissionLevel}), matching the - * reference client. + *

It has the options of {@link ActorStartOptions} except the Actor-only {@code contentType} and + * {@code forcePermissionLevel}, which the task run endpoint does not accept. * - *

{@link #apply}/{@link #contentTypeOrDefault}/{@link #waitForResourcesSecsValue} are - * package-private on purpose — see {@link com.apify.client.internal.RunStartSupport}'s class - * javadoc for why. + *

The package-private accessors are described in {@link + * com.apify.client.internal.RunStartSupport}. */ public final class TaskStartOptions { private String build; @@ -78,10 +76,9 @@ public TaskStartOptions webhooks(List webhooks) { /** * Retries the start while the account lacks the memory or a concurrent-run slot for the run (the * API rejects it with {@code actor-memory-limit-exceeded} or {@code - * concurrent-runs-limit-exceeded}, both of which clear as other runs or builds finish), waiting - * 10 seconds between attempts. {@code true} retries until the run starts ({@link - * ActorStartOptions#WAIT_FOR_RESOURCES_INDEFINITELY}); {@code false} (the default) does not retry - * — a resource-limit error is thrown immediately, like any other error. + * concurrent-runs-limit-exceeded}), waiting 10 seconds between attempts. {@code true} retries + * until the run starts ({@link ActorStartOptions#WAIT_FOR_RESOURCES_INDEFINITELY}); {@code false} + * (the default) does not retry and throws the error immediately. */ public TaskStartOptions waitForResources(boolean waitForResources) { this.waitForResourcesSecs = diff --git a/src/main/java/com/apify/client/user/UserClient.java b/src/main/java/com/apify/client/user/UserClient.java index cc14214..5bc9ff6 100644 --- a/src/main/java/com/apify/client/user/UserClient.java +++ b/src/main/java/com/apify/client/user/UserClient.java @@ -63,9 +63,7 @@ public CompletableFuture limits() { /** Updates the user's account and usage limits. */ public CompletableFuture updateLimits(Object newLimits) { - // Route through the shared PUT primitive for consistency with every other resource client's - // update-style call; the response body (the updated limits) is intentionally not parsed, - // matching the reference client's void-returning updateLimits. + // The response body is not parsed: this call returns nothing. return ctx.putRaw( "limits", new QueryParams(), Json.toBytes(newLimits), ResourceContext.CONTENT_TYPE_JSON); } diff --git a/src/main/java/module-info.java b/src/main/java/module-info.java index d3512e4..6625ef7 100644 --- a/src/main/java/module-info.java +++ b/src/main/java/module-info.java @@ -20,6 +20,7 @@ requires tools.jackson.databind; requires org.slf4j; requires static com.aayushatharva.brotli4j; + requires static com.github.luben.zstd_jni; exports com.apify.client; exports com.apify.client.actor; diff --git a/src/test/java/com/apify/client/ActorServerClientTest.java b/src/test/java/com/apify/client/ActorServerClientTest.java index 2bb7ebb..0aebfb5 100644 --- a/src/test/java/com/apify/client/ActorServerClientTest.java +++ b/src/test/java/com/apify/client/ActorServerClientTest.java @@ -248,6 +248,23 @@ void requestSendsAnAuthorizationHeaderOfTheCallerAsItIs() { assertEquals("Bearer " + TOKEN, request.header("X-Apify-Authorization")); } + @Test + void requestSendsParentRequestIdAsTheActorRequestIdHeaderOverOnePassedInHeaders() { + client() + .actor(ACTOR_ID) + .server() + .request( + "/", + new ActorServerRequestOptions() + .header("x-actor-request-id", "stale") + .parentRequestId("parent-id")) + .join(); + + TestHttpServer.Received request = serverRequests().get(0); + // Exactly one value: the header set with header(...) is replaced, not sent next to it. + assertEquals("parent-id", request.header("X-Actor-Request-ID")); + } + @Test void requestReturnsARedirectOfTheServerAsItIsSoTheTokenNeverReachesItsTarget() throws IOException { diff --git a/src/test/java/com/apify/client/AmbiguousNotFoundTest.java b/src/test/java/com/apify/client/AmbiguousNotFoundTest.java index 5d8451a..8c6f943 100644 --- a/src/test/java/com/apify/client/AmbiguousNotFoundTest.java +++ b/src/test/java/com/apify/client/AmbiguousNotFoundTest.java @@ -11,11 +11,10 @@ import org.junit.jupiter.api.Test; /** - * Hermetic tests for the ambiguous-404 rule (mirrors apify-client-js#1042): a resource - * client addressed by an explicit id swallows a 404 to an empty {@code Optional}/no-op, but a - * client reached through a run/build with no id of its own throws {@link NotFoundError} instead, - * since the 404 could mean either the parent or the sub-resource is gone. + * Hermetic tests for the ambiguous-404 rule: a resource client addressed by an explicit id swallows + * a 404 to an empty {@code Optional}/no-op, but a client reached through a run/build with no id of + * its own throws {@link NotFoundError} instead, since the 404 could mean either the parent or the + * sub-resource is gone. */ class AmbiguousNotFoundTest { diff --git a/src/test/java/com/apify/client/AsyncPaginatedPublisherTest.java b/src/test/java/com/apify/client/AsyncPaginatedPublisherTest.java index 50057c4..923303e 100644 --- a/src/test/java/com/apify/client/AsyncPaginatedPublisherTest.java +++ b/src/test/java/com/apify/client/AsyncPaginatedPublisherTest.java @@ -15,7 +15,7 @@ /** * Hermetic (token-free) tests for the offset/limit async iteration engine. They drive {@link * AsyncPaginatedPublisher} with a stub page fetcher over a synthetic collection, pinning the - * total-cap / chunk-size arithmetic, server-side page-size clamping, and termination — without any + * total-cap / chunk-size arithmetic, server-side page-size clamping, and termination, without any * network or {@code APIFY_TOKEN}. */ class AsyncPaginatedPublisherTest { @@ -150,10 +150,10 @@ void emptyCollectionYieldsNothing() { } /** - * Records onNext/onError deliveries for the reentrant-invalid-request regression tests below. A - * subscriber violating Reactive Streams §3.9 by issuing {@code request(n<=0)} from within {@code - * onNext} must see exactly the one item already being emitted when the violation is reported, - * then a single {@code onError} - never further {@code onNext} calls for the rest of an + * Records onNext/onError deliveries for the reentrant-invalid-request tests below. A subscriber + * violating Reactive Streams §3.9 by issuing {@code request(n<=0)} from within {@code onNext} + * must see exactly the one item already being emitted when the violation is reported, then a + * single {@code onError} - never further {@code onNext} calls for the rest of an * already-fetched/already-fully-in-hand page (RS §1.7: no signals after a terminal one). */ private static final class RecordingSubscriber implements Flow.Subscriber { @@ -184,11 +184,8 @@ public void onComplete() {} @Test void reentrantInvalidRequestStopsEmissionImmediately() { - // Regression test: the drain loop used to check only `cancelled`/demand/buffer-position, never - // `terminated`, so a reentrant request(n<=0) mid-emission still let the loop emit the rest of - // an - // already-fetched page (and even fetch and emit a further page) after onError had already - // fired. + // A reentrant request(n<=0) during emission must stop the drain loop: nothing more is emitted + // or fetched after onError. StubFetcher f = new StubFetcher(5); RecordingSubscriber subscriber = new RecordingSubscriber(); new AsyncPaginatedPublisher<>(null, null, null, f).subscribe(subscriber); @@ -202,7 +199,7 @@ void reentrantInvalidRequestStopsEmissionImmediately() { @Test void listPublisherReentrantInvalidRequestStopsEmissionImmediately() { - // Same regression, for ListPublisher's independent (but structurally identical) drain loop. + // The same, for ListPublisher's drain loop. CompletableFuture> source = CompletableFuture.completedFuture(List.of(0, 1, 2, 3, 4)); RecordingSubscriber subscriber = new RecordingSubscriber(); diff --git a/src/test/java/com/apify/client/BaseUrlNormalizationTest.java b/src/test/java/com/apify/client/BaseUrlNormalizationTest.java index 30643a3..fc39fe9 100644 --- a/src/test/java/com/apify/client/BaseUrlNormalizationTest.java +++ b/src/test/java/com/apify/client/BaseUrlNormalizationTest.java @@ -9,8 +9,7 @@ /** * Offline tests for {@code ApifyClientBuilder}'s {@code baseUrl}/{@code publicBaseUrl} * normalization: trailing slashes are stripped and {@code /v2} is appended, unless the URL's path - * already ends with it. Mirrors the reference JS client's table-driven regression test for the same - * bug (a caller-supplied {@code .../v2} used to become {@code .../v2/v2}). + * already ends with it, so a caller-supplied {@code .../v2} does not become {@code .../v2/v2}. */ class BaseUrlNormalizationTest { diff --git a/src/test/java/com/apify/client/ClientBehaviourRegressionTest.java b/src/test/java/com/apify/client/ClientBehaviourRegressionTest.java index 983e5dc..af28293 100644 --- a/src/test/java/com/apify/client/ClientBehaviourRegressionTest.java +++ b/src/test/java/com/apify/client/ClientBehaviourRegressionTest.java @@ -147,6 +147,9 @@ void chargeRejectsMissingEventName() { assertEquals(0, backend.calls, "no request should be sent for an invalid charge"); } + // The setStatusMessage success path with ACTOR_RUN_ID set is not tested: the JVM environment + // cannot be changed in-process without reflection into JDK internals, and a live test is not an + // Actor run. This test covers the unset case. @Test void setStatusMessageThrowsWhenActorRunIdUnset() { MockTransport backend = MockTransport.ofConstant(200, "{\"data\":{}}"); @@ -157,17 +160,6 @@ void setStatusMessageThrowsWhenActorRunIdUnset() { assertEquals(0, backend.calls, "no request should be sent when ACTOR_RUN_ID is unset"); } - // Documented skip, mirroring this project's existing metamorph/reboot/charge live-skip - // convention (see ActorRunIntegrationTest's class-level comment): the ACTOR_RUN_ID-set success - // path (PUTs statusMessage/isStatusMessageTerminal to the run identified by that env var) is not - // covered here because the JVM's environment map is effectively immutable in-process without - // reflecting into JDK internals (fragile, and guarded by the module system since Java 9), and it - // is not live-testable either (a live integration test isn't itself an Actor run, so - // ACTOR_RUN_ID is never set there). The PUT-body construction it exercises — statusMessage - // always set, isStatusMessageTerminal only when non-null — is otherwise a straight-line, 3-line - // method body with no branch this test suite doesn't already cover elsewhere (RunClient.update's - // PUT is exercised by TaskIntegrationTest#taskCrudFlow and friends). - @Test void metamorphSendsTargetActorIdBuildAndInputBody() { MockTransport backend = MockTransport.ofConstant(200, "{\"data\":{\"id\":\"run123\"}}"); @@ -177,8 +169,7 @@ void metamorphSendsTargetActorIdBuildAndInputBody() { .join(); assertTrue(backend.lastUrl.contains("actor-runs/run123/metamorph"), backend.lastUrl); // targetActorId is normalized to the URL-safe username~actor-name form before sending - // (ResourceContext.toSafeId), matching the reference client's _toSafeId; '~' is then - // percent-encoded like any other query value. + // (ResourceContext.toSafeId); '~' is then percent-encoded like any other query value. assertTrue(backend.lastUrl.contains("targetActorId=apify%7Eother-actor"), backend.lastUrl); assertTrue(backend.lastUrl.contains("build=1.2.3"), backend.lastUrl); assertTrue(backend.lastBody.contains("\"foo\":\"bar\""), backend.lastBody); @@ -552,12 +543,10 @@ void collectionIterateSingleArgDelegatesToServerDefaultPageSize() { void iterateSnapshotsOptionsSoLaterMutationsDoNotLeak() { // The publisher must capture the options (offset/limit AND filters) at call time; mutating the // caller's options object afterwards must not change subsequent page requests. Note this - // backend - // always returns the same non-empty page (it never signals "empty page" / exhaustion - see - // AsyncPaginatedPublisher's class doc on why it never stops on a short page or reported total), - // so the test must not fully drain the publisher (that would fetch pages forever); it only - // needs - // to request a single item to trigger the one page fetch it is asserting on. + // backend always returns the same non-empty page (it never signals "empty page" / exhaustion - + // see AsyncPaginatedPublisher's class doc on why it never stops on a short page or reported + // total), so the test must not fully drain the publisher (that would fetch pages forever); it + // only needs to request a single item to trigger the one page fetch it is asserting on. MockTransport backend = MockTransport.ofConstant( 200, "{\"data\":{\"items\":[{}],\"total\":1,\"offset\":0,\"limit\":0,\"count\":1}}"); @@ -626,9 +615,7 @@ void updateLimitsSendsPutToMeLimits() { @Test void batchAddRequestsNeverThrowsOnNonRetryableClientError() { - // Matches the reference client's `_batchAddRequestsWithRetries`: even a hard 4xx (e.g. bad - // token / insufficient permissions) must NOT be thrown. It is reported as unprocessed instead, - // keeping batchAddRequests' never-throws contract regardless of the failure cause. + // Even a hard 4xx (such as a bad token) is not thrown: it is reported as unprocessed. MockTransport backend = MockTransport.ofConstant( 403, "{\"error\":{\"type\":\"insufficient-permissions\",\"message\":\"no\"}}"); @@ -668,11 +655,10 @@ void batchAddRequestsRunsChunksInParallel() { @Test void batchAddRequestsNeverThrowsOnPersistentTransportFailure() { // Sibling of `batchAddRequestsNeverThrowsOnNonRetryableClientError`: a transport-level failure - // (ApifyTransportException, e.g. connection refused/timeout — the request never reached the - // API at all) must be treated the same as an API-level failure (ApifyApiException). Both are + // (ApifyTransportException, e.g. connection refused or timeout, so the request never reached + // the API) must be treated the same as an API-level failure (ApifyApiException). Both are // ApifyClientException subtypes, and batchAddRequests' never-throws contract must hold across - // both, matching the reference client's `_batchAddRequestsWithRetries`. Sequential path - // (single chunk, default maxParallel). + // both. Sequential path (single chunk, default maxParallel). MockTransport backend = new MockTransport(List.of(MockTransport.networkError())); RequestQueueRequest request = new RequestQueueRequest("https://example.com", "k0"); BatchAddResult result = @@ -690,9 +676,9 @@ void batchAddRequestsNeverThrowsOnPersistentTransportFailure() { @Test void batchAddRequestsNeverThrowsOnPersistentTransportFailureWithParallelChunks() { - // Same as above but exercising the `maxParallel > 1` path, where each chunk fails independently - // — the never-throws contract must hold there too, and every request across every chunk must - // come back via getUnprocessedRequests(). + // Same as above but exercising the `maxParallel > 1` path, where each chunk fails + // independently. The never-throws contract must hold there too, and every request across every + // chunk must come back via getUnprocessedRequests(). MockTransport backend = new MockTransport(List.of(MockTransport.networkError())); List requests = new ArrayList<>(); for (int i = 0; i < 60; i++) { @@ -732,8 +718,7 @@ void waitForFinishReturnsWhenResourceAppearsAfter404() { void waitForFinishToleratesNullDataBodyAsNotYetAvailable() { // A 200 response with a null `data` body (as opposed to a 404) must be tolerated the same way - // treated as "not yet available" and retried - not thrown as an NPE from testing a null value - // for - // terminality. + // for terminality. MockTransport backend = new MockTransport( List.of( @@ -748,8 +733,7 @@ void waitForFinishToleratesNullDataBodyAsNotYetAvailable() { @Test void waitForFinishThrowsWhenResourceNeverAppears() { // If the resource is never fetchable within the budget, the wait must fail loudly rather than - // return a phantom result - with the actual 404 the server gave, matching the reference client, - // not a generic wrapped error. + // return a phantom result, and it must fail with the actual 404, not a generic wrapped error. MockTransport backend = MockTransport.ofConstant( 404, "{\"error\":{\"type\":\"record-not-found\",\"message\":\"missing\"}}"); @@ -764,8 +748,8 @@ void waitForFinishThrowsWhenResourceNeverAppears() { void waitForFinishGivesUpOnSustained404BeforeTheFullWaitBudget() { // A run/build id that never existed 404s forever. waitForFinish must not poll it for the whole // (here effectively indefinite, waitSecs omitted) wait budget - it gives up once the resource - // has - // been missing continuously for a few seconds, bounding how long a bad id can hang the caller. + // has been missing continuously for a few seconds, bounding how long a bad id can hang the + // caller. MockTransport backend = MockTransport.ofConstant( 404, "{\"error\":{\"type\":\"record-not-found\",\"message\":\"missing\"}}"); @@ -1086,7 +1070,7 @@ void setRecordTrimsContentEncodingBeforeForwarding() { assertEquals( "gzip", backend.lastHeaders.firstValue("Content-Encoding").orElse(null), - "the header value must be trimmed, matching the reference client's schema"); + "the header value must be trimmed"); } @Test diff --git a/src/test/java/com/apify/client/ClientMetaTest.java b/src/test/java/com/apify/client/ClientMetaTest.java index 11255a3..cd777a9 100644 --- a/src/test/java/com/apify/client/ClientMetaTest.java +++ b/src/test/java/com/apify/client/ClientMetaTest.java @@ -60,8 +60,8 @@ void userAgentFormat() { @Test void userAgentOsTokenIsShortAndLowercase() { - // The OS token must be a short, lowercase platform identifier (linux/darwin/win32/…), matching - // the reference JS client's os.platform() output — never the human-readable os.name. + // The OS token must be a short, lowercase platform identifier (linux/darwin/win32/…), never the + // human-readable os.name. String token = ApifyClientBuilder.platformToken(); assertFalse(token.isEmpty(), "platform token must not be empty"); assertEquals(token.toLowerCase(Locale.ROOT), token, "token must be lowercase: " + token); diff --git a/src/test/java/com/apify/client/CompressionTest.java b/src/test/java/com/apify/client/CompressionTest.java index 4709c92..7211374 100644 --- a/src/test/java/com/apify/client/CompressionTest.java +++ b/src/test/java/com/apify/client/CompressionTest.java @@ -14,7 +14,9 @@ import com.apify.client.http.BrotliHttpCompressor; import com.apify.client.http.GzipHttpCompressor; import com.apify.client.http.HttpCompressor; +import com.apify.client.http.ZstdHttpCompressor; import com.apify.client.internal.HttpClientCore; +import com.github.luben.zstd.Zstd; import java.io.ByteArrayInputStream; import java.io.IOException; import java.nio.charset.StandardCharsets; @@ -28,11 +30,11 @@ import org.junit.jupiter.api.Test; /** - * Offline tests for request-body compression. Mirrors the reference JS client: bodies of at least - * 1024 bytes are compressed and announced via {@code Content-Encoding}, smaller bodies are sent - * verbatim. The client uses brotli ({@code br}) by default where its native codec is available and - * gzip elsewhere; the compressor is configurable, and its failure fails the request. Both - * compressors are exercised directly, and the live client path via a real request. + * Offline tests for request-body compression: bodies of at least 1024 bytes are compressed and + * announced via {@code Content-Encoding}, smaller bodies are sent verbatim. The client uses brotli + * ({@code br}) by default where its native codec is available and gzip elsewhere; the compressor is + * configurable, and its failure fails the request. Both compressors are exercised directly, and the + * live client path via a real request. */ class CompressionTest { @@ -109,6 +111,44 @@ void gzipCompressorEncodesAsGzipAndRoundTrips() throws IOException { assertArrayEquals(payload, gunzip(compressed)); } + @Test + void zstdCompressorEncodesAsZstdAndRoundTrips() throws IOException { + assertTrue(ZstdHttpCompressor.isAvailable(), "zstd-jni must load on the test platform"); + byte[] payload = payload(4096, (byte) 'a'); + + HttpCompressor compressor = new ZstdHttpCompressor(); + byte[] compressed = compressor.compress(payload); + + assertEquals("zstd", compressor.contentEncoding()); + assertTrue(compressed.length < payload.length, "zstd-compressed body should be smaller"); + assertArrayEquals(payload, Zstd.decompress(compressed, payload.length)); + } + + @Test + void zstdRejectsLevelsOutsideOneToNineteen() { + assertThrows(IllegalArgumentException.class, () -> new ZstdHttpCompressor(0)); + assertThrows(IllegalArgumentException.class, () -> new ZstdHttpCompressor(20)); + } + + @Test + void configuredZstdCompressorIsUsedForLargeBodies() { + MockTransport backend = MockTransport.ofConstant(201, ""); + byte[] payload = payload(4096, (byte) 'a'); + + ApifyClient.builder() + .token("t") + .httpTransport(backend) + .maxRetries(0) + .compression(new ZstdHttpCompressor(19)) + .build() + .keyValueStore(STORE_ID) + .setRecord(RECORD_KEY, payload, CONTENT_TYPE) + .join(); + + assertEquals("zstd", backend.lastHeaders.firstValue("Content-Encoding").orElse(null)); + assertArrayEquals(payload, Zstd.decompress(backend.lastBodyBytes, payload.length)); + } + @Test void gzipDefaultsToLevelSix() throws IOException { assertEquals(6, GzipHttpCompressor.DEFAULT_LEVEL); diff --git a/src/test/java/com/apify/client/ConfigTest.java b/src/test/java/com/apify/client/ConfigTest.java index 56a1330..25b8cdf 100644 --- a/src/test/java/com/apify/client/ConfigTest.java +++ b/src/test/java/com/apify/client/ConfigTest.java @@ -10,7 +10,7 @@ */ class ConfigTest { - /** Mirror of the integration base's resolution logic (kept in sync with IntegrationBase). */ + /** The base URL resolution of IntegrationBase. */ private static String resolveBaseUrl(String apiUrl) { if (apiUrl == null || apiUrl.isEmpty()) { apiUrl = "https://api.apify.com/v2"; diff --git a/src/test/java/com/apify/client/DatasetItemsIteratorTest.java b/src/test/java/com/apify/client/DatasetItemsIteratorTest.java index 2813fce..bf06402 100644 --- a/src/test/java/com/apify/client/DatasetItemsIteratorTest.java +++ b/src/test/java/com/apify/client/DatasetItemsIteratorTest.java @@ -15,9 +15,8 @@ /** * Hermetic (token-free) tests for {@link DatasetClient#iterateItems} and its {@code fetchItemsPage} - * helper — the distinct dataset-items path that parses a bare JSON array body (not a {@code data} - * envelope) and terminates on an empty page. Driven by a {@link MockTransport}, no {@code - * APIFY_TOKEN}. + * helper: the dataset-items path that parses a bare JSON array body (not a {@code data} envelope) + * and terminates on an empty page. Driven by a {@link MockTransport}, no {@code APIFY_TOKEN}. */ class DatasetItemsIteratorTest { @@ -55,7 +54,7 @@ void pagesBareArrayBodyAndStopsOnEmptyPage() { void typedIterateItemsAtDefaultPageSize() { // Typed iteration at the server-default page size uses the 3-arg form with a null chunkSize. // (No (options, Class) overload exists: with a null second argument, the compiler could not - // tell it apart from the (options, Long chunkSize) overload — ambiguous overload resolution.) + // tell it apart from the (options, Long chunkSize) overload.) // Decodes each item into the requested type. MockTransport backend = new MockTransport( @@ -153,11 +152,9 @@ void listItemsScannedCountNullWhenHeaderAbsent() { @Test void advancesByScannedCountNotReturnedCount() { - // Regression for the fully-filtered-page bug (see AsyncPaginatedPublisher.applyPage): a - // server-side filter (clean/skipEmpty/skipHidden) can drop every item of a scanned page. - // Advancing by the *returned* count (0) would stop iteration right there; advancing by the - // *scanned* count (reported via X-Apify-Pagination-Count) must skip past the filtered-out - // window and reach the real data on the next page. + // A server-side filter (clean/skipEmpty/skipHidden) can drop every item of a scanned page. + // Iteration must advance by the scanned count (X-Apify-Pagination-Count), not the returned + // count, to reach the data on the next page. MockTransport backend = new MockTransport( List.of( @@ -182,11 +179,8 @@ void advancesByScannedCountNotReturnedCount() { @Test void advancesByScannedCountEvenWhenUnwindReturnsMoreThanScanned() { - // Regression for the mirror-image bug the naive "trust the header only when it's >= the - // returned count" guard introduced: `unwind` splits one scanned row's array field into several - // output items, so returnedCount > scannedCount is a legitimate, common shape - not a sign the - // header is unpopulated. A guard that falls back to the (larger) returned count here advances - // the offset too far and silently drops the rows in between on the next page. + // `unwind` splits one scanned row into several items, so the returned count can exceed the + // scanned count. The offset must still advance by the scanned count, or rows are skipped. MockTransport backend = new MockTransport( List.of( @@ -257,13 +251,9 @@ void fallsBackToReturnedCountWhenHeaderMissing() { @Test void fallsBackToReturnedCountWhenHeaderIsImplausiblyLow() { - // Regression for a real finding against the live API (see PaginationList.getScannedCount): it - // was observed sending X-Apify-Pagination-Count: 0 on every page of an *unfiltered* listing, - // even though each page genuinely scanned and returned real items - a combination that can - // never - // be a real "nothing scanned" answer (0 scanned rows can't produce items), so it must mean the - // header is not populated for this request/endpoint yet. Trusting it anyway would advance the - // offset by 0 and re-request (or, worse, treat the page as exhausted and stop). + // The API can send X-Apify-Pagination-Count: 0 for an unfiltered page that returned items. + // That is not a real "nothing scanned" answer, so iteration falls back to the returned count + // instead of re-requesting the same offset or stopping. MockTransport backend = new MockTransport( List.of( diff --git a/src/test/java/com/apify/client/ExceptionHierarchyTest.java b/src/test/java/com/apify/client/ExceptionHierarchyTest.java index a5d94b8..fef50cb 100644 --- a/src/test/java/com/apify/client/ExceptionHierarchyTest.java +++ b/src/test/java/com/apify/client/ExceptionHierarchyTest.java @@ -57,7 +57,7 @@ void transportExceptionIsCatchableAsClientException() { /** * {@link HttpClientCore#buildApiError} must construct the subclass matching each status code, so * callers can branch with {@code instanceof} instead of comparing {@code getStatusCode()} by - * number (see the reference JS client's per-status {@code ApifyApiError} subclasses). + * number. */ @Test void buildApiErrorPicksSubclassByStatus() { diff --git a/src/test/java/com/apify/client/NestedDtoForwardCompatTest.java b/src/test/java/com/apify/client/NestedDtoForwardCompatTest.java index bc56e16..a81c4f4 100644 --- a/src/test/java/com/apify/client/NestedDtoForwardCompatTest.java +++ b/src/test/java/com/apify/client/NestedDtoForwardCompatTest.java @@ -25,13 +25,10 @@ import org.junit.jupiter.api.Test; /** - * Guards the fix for the nested-value-DTO data-loss finding: every nested value DTO now extends - * {@link ApifyResource}, so (a) the specific spec fields the finding named as missing typed getters - * are modelled, and (b) any *other* field the API returns that still has no typed getter is - * reachable via {@code getExtra()} rather than silently discarded. Each assertion deserializes a - * literal JSON fixture directly (via the same {@link Json} the client uses for real responses), not - * just a hand-built object, so a regression that drops {@code extends ApifyResource} or a field - * name typo is actually caught here, not only at compile time. + * Every nested value DTO extends {@link ApifyResource}, so its spec fields have typed getters and + * any other field the API returns is kept in {@code getExtra()} instead of being dropped. Each + * assertion deserializes a literal JSON fixture with the same {@link Json} the client uses for + * responses. */ class NestedDtoForwardCompatTest { @@ -184,9 +181,8 @@ void webhookModelsAdHocFlagAndCondition() { } /** - * Every other nested value DTO in the audit (one representative per resource package) also - * preserves fields it has no typed getter for, instead of discarding them — the general fix, not - * just the three fields the finding named explicitly. + * One representative nested value DTO per resource package also keeps fields it has no typed + * getter for. */ @Test void otherAuditedNestedDtosAlsoKeepUnknownFields() { diff --git a/src/test/java/com/apify/client/StreamedLogTest.java b/src/test/java/com/apify/client/StreamedLogTest.java index 7423bfe..b6d7ba8 100644 --- a/src/test/java/com/apify/client/StreamedLogTest.java +++ b/src/test/java/com/apify/client/StreamedLogTest.java @@ -18,16 +18,19 @@ import java.util.List; import java.util.concurrent.CopyOnWriteArrayList; import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicReference; import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; /** * Offline tests for the {@link StreamedLog} log-redirection helper: it must forward complete * timestamped messages to the destination, drop messages older than the relevancy limit when {@code * fromStart} is false, flush a final unterminated line, and enforce its start/stop lifecycle. */ +@Timeout(30) class StreamedLogTest { /** Poll interval used while waiting for a background reader to redirect expected messages. */ @@ -117,10 +120,9 @@ void defaultDestinationDoesNotThrow() throws InterruptedException { void throwingConsumerDoesNotKillDaemonThreadUncaught() throws InterruptedException { // The destination consumer runs on the background daemon reader thread. A user consumer that // throws must not escape as an uncaught exception that silently kills that thread (and, via the - // final flush, throws again). Matching the reference client, redirection must degrade to a - // logged warning. We capture any exception that reaches the reader thread's uncaught handler - // and - // assert none does, while confirming the consumer was actually exercised. + // final flush, throws again). Redirection must degrade to a logged warning. We capture any + // exception that reaches the reader thread's uncaught handler and assert none does, while + // confirming the consumer was actually exercised. AtomicReference uncaught = new AtomicReference<>(); Thread.UncaughtExceptionHandler previous = Thread.getDefaultUncaughtExceptionHandler(); Thread.setDefaultUncaughtExceptionHandler( @@ -152,8 +154,7 @@ void throwingConsumerDoesNotKillDaemonThreadUncaught() throws InterruptedExcepti uncaught.get(), "a throwing destination consumer must not escape as an uncaught daemon-thread exception"); // Two complete lines are streamed, but redirection must unwind on the first throw and forward - // nothing further (no re-invocation for the remaining line or the final flush), matching the - // reference client's single warning. + // nothing further (no re-invocation for the remaining line or the final flush). assertEquals( 1, calls.get(), "the destination consumer must not be invoked again after it throws"); } finally { @@ -200,12 +201,12 @@ void closeWithoutStartIsNoOp() { @Test void deliversLastMessageWhenStoppingLiveStream() { - // Regression test for a live stream that never reaches end-of-stream on its own. The reader - // buffers three complete lines (a, b, c): a and b are emitted immediately while c is retained - // as the possibly-still-growing last message. Then the read blocks. stop() closes the stream, - // which unblocks the read with an IOException; the retained last message (c) must still be - // flushed and delivered. A finite ByteArrayInputStream cannot exercise this because read() - // returns -1 by itself, so we drive a purpose-built blocking stream instead. + // A live stream never reaches end-of-stream on its own. The reader buffers three complete lines + // (a, b, c): a and b are emitted immediately while c is retained as the possibly-still-growing + // last message. Then the read blocks. stop() closes the stream, which unblocks the read with an + // IOException; the retained last message (c) must still be flushed and delivered. A finite + // ByteArrayInputStream cannot exercise this because read() returns -1 by itself, so we drive a + // purpose-built blocking stream instead. String body = "2999-01-01T00:00:00.000Z a\n" + "2999-01-01T00:00:01.000Z b\n" @@ -234,6 +235,179 @@ void deliversLastMessageWhenStoppingLiveStream() { "the retained last message must be delivered after stop()"); } + @Test + void flushesTheLastMessageWithoutAnInterruptPendingOnTheReaderThread() { + BlockingStream stream = new BlockingStream("2999-01-01T00:00:00.000Z a\n"); + MockTransport backend = MockTransport.ofConstant(200, ""); + backend.scriptStream(200, stream); + List interruptedPerMessage = new CopyOnWriteArrayList<>(); + StreamedLog streamedLog = + client(backend) + .run("run123") + .getStreamedLog( + new StreamedLogOptions() + .toLog( + message -> + interruptedPerMessage.add(Thread.currentThread().isInterrupted()))) + .join(); + streamedLog.start().join(); + // The reader has consumed the body and blocks, holding back "a" as the last message. + assertTrue(pollUntil(REDIRECT_POLL_ATTEMPTS, REDIRECT_POLL_MILLIS, stream::isBlocked)); + + streamedLog.stop(); + + assertEquals( + List.of(false), interruptedPerMessage, "the flush must not run the consumer interrupted"); + } + + @Test + void closeFromInsideTheConsumerWhileAnotherThreadClosesDoesNotDeadlock() { + MockTransport backend = MockTransport.ofConstant(200, ""); + backend.scriptStream( + 200, + "2999-01-01T00:00:00.000Z a\n2999-01-01T00:00:01.000Z b\n2999-01-01T00:00:02.000Z c\n"); + List interruptedPerMessage = new CopyOnWriteArrayList<>(); + AtomicReference streamedLog = new AtomicReference<>(); + streamedLog.set( + client(backend) + .run("run123") + .getStreamedLog( + new StreamedLogOptions() + .toLog( + message -> { + interruptedPerMessage.add( + message.substring(message.length() - 1) + + "=" + + Thread.currentThread().isInterrupted()); + if (message.endsWith(" a")) { + streamedLog.get().close(); + } + })) + .join()); + streamedLog.get().start().join(); + + streamedLog.get().close(); + + assertEquals(List.of("a=false", "b=false", "c=false"), interruptedPerMessage); + } + + @Test + void closeFromAnInterruptedThreadStillEndsTheReader() { + BlockingStream stream = new BlockingStream("2999-01-01T00:00:00.000Z a\n"); + MockTransport backend = MockTransport.ofConstant(200, ""); + backend.scriptStream(200, stream); + List collected = new CopyOnWriteArrayList<>(); + StreamedLog streamedLog = + client(backend) + .run("run123") + .getStreamedLog(new StreamedLogOptions().toLog(collected::add)) + .join(); + streamedLog.start().join(); + assertTrue(pollUntil(REDIRECT_POLL_ATTEMPTS, REDIRECT_POLL_MILLIS, stream::isBlocked)); + + Thread.currentThread().interrupt(); + streamedLog.close(); + // Clear the flag that close() restores, so it does not fail the polling below. + assertTrue(Thread.interrupted()); + + assertTrue( + pollUntil(REDIRECT_POLL_ATTEMPTS, REDIRECT_POLL_MILLIS, stream::isClosed), + "close() must end the live stream even when its join is interrupted"); + assertTrue( + pollUntil(REDIRECT_POLL_ATTEMPTS, REDIRECT_POLL_MILLIS, () -> collected.size() == 1), + "the retained last message must still be flushed"); + } + + @Test + void startAfterAStopFromInsideTheConsumerIsRejectedUntilTheReaderHasStopped() + throws InterruptedException { + BlockingStream stream = + new BlockingStream("2999-01-01T00:00:00.000Z a\n2999-01-01T00:00:01.000Z b\n"); + MockTransport backend = MockTransport.ofConstant(200, ""); + backend.scriptStream(200, stream); + CountDownLatch consumerStopped = new CountDownLatch(1); + CountDownLatch releaseConsumer = new CountDownLatch(1); + List collected = new CopyOnWriteArrayList<>(); + AtomicReference streamedLog = new AtomicReference<>(); + streamedLog.set( + client(backend) + .run("run123") + .getStreamedLog( + new StreamedLogOptions() + .toLog( + message -> { + collected.add(message.substring(message.length() - 1)); + if (message.endsWith(" a")) { + streamedLog.get().close(); + consumerStopped.countDown(); + try { + releaseConsumer.await(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + } + })) + .join()); + streamedLog.get().start().join(); + assertTrue(consumerStopped.await(5, TimeUnit.SECONDS)); + + // The reader is still inside the consumer: a start() now would reset the stop flag under it. + assertThrows(IllegalStateException.class, () -> streamedLog.get().start()); + releaseConsumer.countDown(); + streamedLog.get().close(); + + assertEquals(List.of("a", "b"), collected, "the retained message is flushed after the stop"); + assertTrue(stream.isClosed()); + } + + @Test + void stopFromAnotherThreadDoesNotInterruptAConsumerThatIsBlocked() throws InterruptedException { + BlockingStream stream = + new BlockingStream("2999-01-01T00:00:00.000Z a\n2999-01-01T00:00:01.000Z b\n"); + MockTransport backend = MockTransport.ofConstant(200, ""); + backend.scriptStream(200, stream); + CountDownLatch consumerEntered = new CountDownLatch(1); + CountDownLatch releaseConsumer = new CountDownLatch(1); + List outcomePerMessage = new CopyOnWriteArrayList<>(); + StreamedLog streamedLog = + client(backend) + .run("run123") + .getStreamedLog( + new StreamedLogOptions() + .toLog( + message -> { + String name = message.substring(message.length() - 1); + if (name.equals("a")) { + consumerEntered.countDown(); + try { + releaseConsumer.await(); + outcomePerMessage.add("a=ok"); + } catch (InterruptedException e) { + outcomePerMessage.add("a=interrupted"); + } + } else { + outcomePerMessage.add( + name + "=" + Thread.currentThread().isInterrupted()); + } + })) + .join(); + streamedLog.start().join(); + assertTrue(consumerEntered.await(5, TimeUnit.SECONDS)); + Thread stopper = new Thread(streamedLog::stop); + stopper.start(); + // stop() has signaled the reader once it waits for the reader thread to finish. + assertTrue( + pollUntil( + REDIRECT_POLL_ATTEMPTS, + REDIRECT_POLL_MILLIS, + () -> stopper.getState() == Thread.State.WAITING)); + + releaseConsumer.countDown(); + stopper.join(); + + assertEquals(List.of("a=ok", "b=false"), outcomePerMessage); + } + @Test void closeAfterStopIsNoOp() throws InterruptedException { // Sequential idempotency: after an explicit stop() (which nulls the running thread), a trailing @@ -255,12 +429,8 @@ void closeAfterStopIsNoOp() throws InterruptedException { @Test void concurrentCloseNeverThrowsWhileStopRaces() throws InterruptedException { - // Reproduces the concurrent case the fix targets: a stop() and a close() fire simultaneously. - // Because close() does its running-check and stop atomically under the monitor, it must never - // throw regardless of interleaving. (stop() may legitimately throw when it loses the race - its - // explicit-misuse contract - so only close()'s outcome is asserted.) The pre-fix check-then-act - // close() could observe a non-null thread, release the lock, then call a stop() that had - // already nulled the field and threw. Run several rounds to make the interleaving likely. + // A stop() and a close() fire simultaneously. close() never throws; stop() may throw when it + // loses the race, so only close() is asserted. Many rounds make the interleaving likely. for (int round = 0; round < 50; round++) { MockTransport backend = MockTransport.ofConstant(200, ""); backend.scriptStream(200, "2999-01-01T00:00:00.000Z x\n"); @@ -330,13 +500,12 @@ void closeStopsActiveRedirection() throws InterruptedException { @Test void stopFromInsideConsumerDoesNotDeadlock() { // The destination consumer runs on the background reader thread. A user may stop redirection - // from inside it ("stop once I see line X"). Because that calls stopStreaming() on the reader - // thread itself, an unguarded streamingThread.join() would join the thread on itself and hang - // forever - and, since stop() is synchronized, hold the monitor so every later start/stop/close - // deadlocks too. The self-join guard must let the call return so the reader breaks its loop and - // flushes the retained message. If it deadlocked, "b" would never be flushed and the poll below - // would time out with only "a" collected. A blocking stream is required: a finite one would end - // on its own and mask the hang. + // from inside it ("stop once I see line X"). Because that call runs on the reader thread + // itself, an unguarded join would wait for the thread on itself and hang forever. The self-join + // guard must let the call return so the reader breaks its loop and flushes the retained + // message. If it deadlocked, "b" would never be flushed and the poll below would time out with + // only "a" collected. A blocking stream is required: a finite one would end on its own and mask + // the hang. MockTransport backend = MockTransport.ofConstant(200, ""); backend.scriptStream( 200, new BlockingStream("2999-01-01T00:00:00.000Z a\n2999-01-01T00:00:01.000Z b\n")); @@ -351,8 +520,7 @@ void stopFromInsideConsumerDoesNotDeadlock() { .toLog( message -> { collected.add(message); - // Self-stop on the reader thread; fire once (stop() throws after the - // thread reference is cleared). + // Self-stop on the reader thread, once. if (stopRequested.compareAndSet(false, true)) { ref.get().stop(); } @@ -369,10 +537,8 @@ void stopFromInsideConsumerDoesNotDeadlock() { @Test void closeFromInsideConsumerDoesNotDeadlock() { - // Same self-join hazard as stopFromInsideConsumerDoesNotDeadlock, via close() (which routes - // through the same stopStreaming()). close() is idempotent, so the consumer can call it on - // every - // message without guarding. + // Same self-join hazard as stopFromInsideConsumerDoesNotDeadlock, via close(). close() is + // idempotent, so the consumer can call it on every message without guarding. MockTransport backend = MockTransport.ofConstant(200, ""); backend.scriptStream( 200, new BlockingStream("2999-01-01T00:00:00.000Z a\n2999-01-01T00:00:01.000Z b\n")); @@ -422,9 +588,8 @@ void reopensEmptyStreamUntilItDeliversContent() { @Test void stopBeforeAnyBytesStreamedFallsBackToOneShotFetch() { // If stop() is requested before any stream ever delivers a byte, redirection must not simply - // end - // with nothing: it fetches the log in one plain request instead, so whatever was logged by then - // is still delivered. + // end with nothing: it fetches the log in one plain request instead, so whatever was logged by + // then is still delivered. MockTransport backend = MockTransport.ofConstant(200, "fallback content"); backend.scriptStream(200, ""); // every streamed open stays empty - the run never logs anything List collected = new CopyOnWriteArrayList<>(); @@ -440,10 +605,9 @@ void stopBeforeAnyBytesStreamedFallsBackToOneShotFetch() { @Test void stopDuringAnInFlightReopenStillCompletesPromptly() { - // Regression test: stop() must still unblock redirection (not hang) when it races a reopen that - // is already in flight - every scripted open comes back empty, so without draining into the - // "stopped, fetch once" path the reader would otherwise keep reopening for the whole backoff - // schedule (or, pre-fix, could block forever on a newly installed stream nothing then closes). + // stop() must end redirection promptly when it races a reopen that is already in flight. Every + // scripted open is empty, so the reader must take the "stopped, fetch once" path instead of + // reopening for the whole backoff schedule. MockTransport backend = MockTransport.ofConstant(200, ""); backend.scriptStream(200, ""); backend.scriptStream(200, ""); @@ -486,11 +650,20 @@ private static final class BlockingStream extends InputStream { private final byte[] data; private int pos; private final CountDownLatch closed = new CountDownLatch(1); + private volatile boolean blocked; BlockingStream(String data) { this.data = data.getBytes(StandardCharsets.UTF_8); } + boolean isBlocked() { + return blocked; + } + + boolean isClosed() { + return closed.getCount() == 0; + } + @Override public synchronized int read(byte[] b, int off, int len) throws IOException { if (pos < data.length) { @@ -501,6 +674,7 @@ public synchronized int read(byte[] b, int off, int len) throws IOException { } // Payload exhausted: block like a live stream awaiting more bytes, until close() unblocks us. try { + blocked = true; closed.await(); } catch (InterruptedException e) { Thread.currentThread().interrupt(); diff --git a/src/test/java/com/apify/client/TestHttpServer.java b/src/test/java/com/apify/client/TestHttpServer.java index e89ca24..298bc50 100644 --- a/src/test/java/com/apify/client/TestHttpServer.java +++ b/src/test/java/com/apify/client/TestHttpServer.java @@ -124,8 +124,11 @@ private static Received read(InputStream in) throws IOException { Map headers = new TreeMap<>(); for (String line = readLine(in); line != null && !line.isEmpty(); line = readLine(in)) { int colon = line.indexOf(':'); - headers.put( - line.substring(0, colon).toLowerCase(Locale.ROOT), line.substring(colon + 1).trim()); + // Repeated headers are joined with a comma, so a duplicate sent by mistake stays visible. + headers.merge( + line.substring(0, colon).toLowerCase(Locale.ROOT), + line.substring(colon + 1).trim(), + (first, second) -> first + ", " + second); } int length = Integer.parseInt(headers.getOrDefault("content-length", "0")); byte[] body = in.readNBytes(length); diff --git a/src/test/java/com/apify/client/integration/ActorIntegrationTest.java b/src/test/java/com/apify/client/integration/ActorIntegrationTest.java index 79f626c..660ceba 100644 --- a/src/test/java/com/apify/client/integration/ActorIntegrationTest.java +++ b/src/test/java/com/apify/client/integration/ActorIntegrationTest.java @@ -154,7 +154,7 @@ void actorDefaultBuildAndWebhooks() { assertTrue(defaultBuild.get().join().isPresent()); // Read-only nested webhook collection (GET + iterate); the account has none registered for - // this fresh Actor, so this just exercises both calls succeeding against an empty result — + // this fresh Actor, so this just exercises both calls succeeding against an empty result, // not a multi-item case. `NestedWebhookCollectionClient` shares its `list`/`iterate` // implementation with `AbstractWebhookCollectionClient`, whose paging/iteration behavior is // already exercised with real multi-item data (2 created webhooks, paged one at a time) by diff --git a/src/test/java/com/apify/client/integration/ActorRunIntegrationTest.java b/src/test/java/com/apify/client/integration/ActorRunIntegrationTest.java index 8a523f1..d4db57e 100644 --- a/src/test/java/com/apify/client/integration/ActorRunIntegrationTest.java +++ b/src/test/java/com/apify/client/integration/ActorRunIntegrationTest.java @@ -29,17 +29,10 @@ class ActorRunIntegrationTest extends IntegrationBase { - // Note: RunClient.metamorph/reboot/charge are intentionally NOT exercised by a live integration - // test here, matching the project's existing convention for UserIntegrationTest#updateLimits. - // `metamorph` requires a second target Actor under the same account with a compatible input - // schema. `reboot` requires a genuinely *running* container; the store Actor exercised in this - // suite (hello-world) finishes in ~1-2s, so a live reboot test would be either flaky-by- - // construction or need a bespoke slow-running test Actor this suite doesn't otherwise need. - // `charge` only applies to pay-per-event Actors, which none of this suite's fixtures are. All - // three are covered offline instead, against a mock backend, by - // ClientBehaviourRegressionTest#metamorphSendsTargetActorIdBuildAndInputBody, - // #rebootSendsPostToRebootWithNoBody, and #chargeSendsIdempotencyKey (+ its sibling charge* - // tests) respectively. + // metamorph, reboot and charge have no live test: metamorph needs a second Actor with a + // compatible input schema, reboot needs a run that is still running (hello-world finishes in a + // couple of seconds), and charge needs a pay-per-event Actor. ClientBehaviourRegressionTest + // covers them against a mock backend. @Test void listRuns() { @@ -118,9 +111,8 @@ void runActorAndReadOutputs() { client.run(run.getId()).keyValueStore().getRecord("OUTPUT").join(); client.run(run.getId()).requestQueue().listHead(null).join(); - // Nested run-storage metadata GETs (previously untested): confirm the run's default storages - // are reachable as full metadata resources, not just via the item/record/head-level calls - // above. + // Nested run-storage metadata GETs: the run's default storages are reachable as metadata + // resources. assertTrue(client.run(run.getId()).dataset().get().join().isPresent()); assertTrue(client.run(run.getId()).keyValueStore().get().join().isPresent()); assertTrue(client.run(run.getId()).requestQueue().get().join().isPresent()); @@ -137,8 +129,7 @@ void runActorAndReadOutputs() { assertTrue( client.run(run.getId()).requestQueue().getRequest("does-not-exist").join().isEmpty()); - // Typed getters (previously only reachable via getExtra()): verify the API's response - // actually deserializes into them, not just that the code compiles. + // Typed getters: verify the API's response deserializes into them. assertTrue(run.getGeneralAccess() != null); assertTrue(run.getStats() != null); assertTrue(run.getOptions() != null && run.getOptions().getBuild() != null); @@ -192,10 +183,8 @@ void callWithActorCallOptionsStreamsLogByDefault() { ApifyClient client = requireClient(); ActorClient actor = client.actor("apify/hello-world"); List collected = new CopyOnWriteArrayList<>(); - // The log-streaming call() overload (ActorCallOptions), matching the reference client's - // default call(options.log='default') behavior: the run's log is streamed for the duration of - // the wait without any explicit opt-in. runActorAndReadOutputs above exercises the plain - // (non-streaming) ActorStartOptions overload. + // The log-streaming call() overload (ActorCallOptions) streams the run's log during the wait + // without an explicit opt-in. runActorAndReadOutputs above covers the plain overload. ActorRun run = actor .call( @@ -204,14 +193,8 @@ void callWithActorCallOptionsStreamsLogByDefault() { TEST_ACTOR_WAIT_SECS) .join(); assertEquals("SUCCEEDED", run.getStatus()); - // call()'s internal log-streaming lifecycle closes the stream as soon as the run is finished - // (RunStartSupport#callWithLogStreaming), which can race the background reader the same way - // streamedLogRedirection below does for its own explicit StreamedLog. Gate the assertion on - // whether the run actually produced a log at all - via the call()-default variant, which - // (unlike - // streamedLogRedirection's) never rescues via a separate explicit stream, so a genuinely broken - // default-streaming wiring still fails here rather than being masked. - assertCallDefaultStreamedLogNonEmptyIfProduced(client.run(run.getId()), collected); + // The assertion applies only if the run logged anything. + assertStreamedLogNonEmptyIfProduced(client.run(run.getId()), collected); } @Test @@ -265,11 +248,9 @@ void lastRunAccess() { .join(); assertEquals("SUCCEEDED", run.getStatus()); - // `apify/hello-world` is a shared public store Actor: under the account's concurrent-execution - // isolation contract, other runs of that same Actor (e.g. a sibling-language client's suite - // running at the same time against the same test user) can legitimately be "last" between the - // `call` above and the `lastRun` lookup below. Assert presence/status/origin, not identity - // with the run just started. + // `apify/hello-world` is shared, so another run of it by a concurrent test can be "last" + // between the `call` above and the `lastRun` lookup below. Assert presence, status and origin, + // not identity. var lastRun = client.actor("apify/hello-world").lastRun("SUCCEEDED").get().join(); assertTrue(lastRun.isPresent()); assertEquals("SUCCEEDED", lastRun.get().getStatus()); @@ -284,11 +265,8 @@ void lastRunAccess() { assertEquals("SUCCEEDED", byOrigin.get().getStatus()); assertEquals("API", byOrigin.get().getMeta().getOrigin()); - // Last-run-scoped nested storage GETs (previously untested — only `lastRun().get()` itself - // was): `actor(...).lastRun(...)` returns a RunClient, and its nested storage clients must be - // reachable the same way `run(id).()`'s are. Whichever run resolves as "last" (see - // the concurrency note above) is a hello-world run, so its default storages are guaranteed to - // exist. + // Last-run-scoped nested storage GETs. Whichever run is "last" (see the note above) is a + // hello-world run, so its default storages exist. RunClient lastRunClient = client.actor("apify/hello-world").lastRun("SUCCEEDED"); lastRunClient.dataset().listItems(new DatasetListItemsOptions()).join(); lastRunClient.keyValueStore().getRecord("OUTPUT").join(); @@ -308,13 +286,7 @@ void streamedLogRedirection() { runClient.waitForFinish(TEST_ACTOR_WAIT_SECS).join(); } - // The *first* stream above was opened before the run finished, following the live log as the - // run was still writing to it, so the underlying HTTP stream can reach EOF (the container/ - // log-follow connection closing) a hair before the final bytes are flushed to that same - // connection, or before the background reader has pulled any bytes off it at all - no amount - // of client-side waiting after that EOF recovers bytes that were never delivered on it. Gate - // the assertion on the authoritative persisted log (see - // IntegrationBase#assertStreamedLogNonEmptyIfProduced for the full retry/skip rationale). + // The assertion applies only if the run logged anything. assertStreamedLogNonEmptyIfProduced(runClient, collected); } } diff --git a/src/test/java/com/apify/client/integration/IntegrationBase.java b/src/test/java/com/apify/client/integration/IntegrationBase.java index ab0d127..54f7abd 100644 --- a/src/test/java/com/apify/client/integration/IntegrationBase.java +++ b/src/test/java/com/apify/client/integration/IntegrationBase.java @@ -4,13 +4,10 @@ import static org.junit.jupiter.api.Assumptions.assumeTrue; import com.apify.client.ApifyClient; -import com.apify.client.log.StreamedLog; -import com.apify.client.log.StreamedLogOptions; import com.apify.client.run.RunClient; import java.security.SecureRandom; import java.util.List; import java.util.Optional; -import java.util.concurrent.CopyOnWriteArrayList; import java.util.function.BooleanSupplier; /** @@ -20,8 +17,8 @@ * URL is taken from {@code APIFY_API_URL} (which includes the {@code /v2} suffix) and falls back to * {@code https://api.apify.com/v2}. * - *

Tests are designed to run concurrently — including against the same test account from several - * language clients at once — so every test creates uniquely-named resources and cleans them up. + *

Tests are designed to run concurrently, including against the same test account from several + * language clients at once, so every test creates uniquely-named resources and cleans them up. */ public abstract class IntegrationBase { @@ -36,23 +33,6 @@ public abstract class IntegrationBase { */ static final long TEST_ACTOR_WAIT_SECS = 120L; - /** - * Bounded window given to a just-finished run's live log stream to catch up and flush its last - * bytes: {@code streamedLogRedirection}-style assertions can otherwise race the background reader - * thread, which may not have pulled any bytes off the stream yet even though the log content - * itself is already fully available server-side (see {@code - * ActorRunIntegrationTest#streamedLogRedirection}). Shared by every test that needs the same - * catch-up wait, via {@link #STREAM_CATCH_UP_ATTEMPTS} + {@link #pollUntil}. - */ - static final long STREAM_CATCH_UP_TIMEOUT_MILLIS = 15_000; - - /** Poll interval used while waiting out {@link #STREAM_CATCH_UP_TIMEOUT_MILLIS}. */ - static final long STREAM_CATCH_UP_POLL_MILLIS = 250; - - /** {@link #pollUntil} attempt count equivalent to {@link #STREAM_CATCH_UP_TIMEOUT_MILLIS}. */ - static final int STREAM_CATCH_UP_ATTEMPTS = - (int) (STREAM_CATCH_UP_TIMEOUT_MILLIS / STREAM_CATCH_UP_POLL_MILLIS); - /** * {@link #pollUntil} attempt count bounding how long a CRUD-flow test waits for a just-created * resource to surface in its own top-level collection {@code list()}. The list endpoints read @@ -138,105 +118,16 @@ public static boolean pollUntil(int maxAttempts, long backoffMillis, BooleanSupp } /** - * Opens a fresh {@link StreamedLog} against an already-finished run's static log (with {@link - * StreamedLogOptions#fromStart(boolean)} so the run's already-past-relative-to-construction - * messages are not filtered out) and waits up to {@link #STREAM_CATCH_UP_TIMEOUT_MILLIS} for it - * to deliver the (now non-racing) content. Used by {@link #assertStreamedLogNonEmptyIfProduced} - * to retry once when a live-tail stream came up empty despite the run having produced a log: - * opening this stream strictly after the run is already finished means it is a plain GET against - * a static, fully-persisted log, not a live tail, so it cannot race the run's own writer the way - * the original stream could. - */ - static List collectFinishedRunLog(RunClient runClient) { - List retryCollected = new CopyOnWriteArrayList<>(); - try (StreamedLog retryStream = - runClient - .getStreamedLog(new StreamedLogOptions().toLog(retryCollected::add).fromStart(true)) - .join()) { - retryStream.start().join(); - pollUntil( - STREAM_CATCH_UP_ATTEMPTS, STREAM_CATCH_UP_POLL_MILLIS, () -> !retryCollected.isEmpty()); - } - return retryCollected; - } - - /** - * Asserts that {@code collected} - lines captured by an explicit, directly-tested {@link - * StreamedLog} for a now-finished run - is non-empty, but only when the run actually produced log - * output at all, per the authoritative, statically-persisted log ({@code runClient.log().get()}). - * - *

A fast Actor run (the store Actors this suite exercises routinely finish in a couple of - * seconds) can complete before the background reader has pulled any bytes off the live log stream - * yet, even though the log content itself is already fully available server-side once the run is - * done. This method first gives {@code collected} a bounded {@link - * #STREAM_CATCH_UP_TIMEOUT_MILLIS} window to catch up in-place. If it is still empty, it consults - * the authoritative persisted log to find out whether the run produced any log output at all: if - * it did not, there is nothing to have streamed and the assertion is skipped entirely (this is - * not a race, just an Actor that logged nothing); if it did, one more attempt is made via {@link - * #collectFinishedRunLog} (a brand-new, non-racing {@link StreamedLog}) before failing. - * - *

The {@link #collectFinishedRunLog} retry opens its own explicit {@code StreamedLog} against - * the finished run, so it is only safe to use when the test's subject under test is that - * same explicit {@code getStreamedLog} API (as {@code streamedLogRedirection} is) - a genuine - * break in that API still fails the assertion. For a test whose subject is instead a {@code - * call(...)} overload's own internal, default log-streaming wiring, use {@link - * #assertCallDefaultStreamedLogNonEmptyIfProduced} instead: retrying via a separate explicit - * stream there would bypass the very wiring under test, silently masking a genuinely broken - * default (e.g. log streaming not actually enabled by default) behind a green assertion sourced - * from a different code path. + * Asserts that {@code collected}, the lines a {@link StreamedLog} captured for a finished run, is + * non-empty when the run's persisted log has content. A run that logged nothing is not checked. */ static void assertStreamedLogNonEmptyIfProduced(RunClient runClient, List collected) { - pollUntil(STREAM_CATCH_UP_ATTEMPTS, STREAM_CATCH_UP_POLL_MILLIS, () -> !collected.isEmpty()); - - Optional authoritativeLog = runClient.log().get().join(); - if (runProducedLog(authoritativeLog) && collected.isEmpty()) { - collected.addAll(collectFinishedRunLog(runClient)); - } - assertNonEmptyIfLogProduced(authoritativeLog, collected); - } - - /** - * Asserts that {@code collected} - lines captured by a {@code call(...)} overload's own internal, - * default log-streaming wiring for a now-finished run - is non-empty, but only when the run - * actually produced log output at all, per the authoritative, statically-persisted log ({@code - * runClient.log().get()}). - * - *

Like {@link #assertStreamedLogNonEmptyIfProduced}, this gives {@code collected} a bounded - * {@link #STREAM_CATCH_UP_TIMEOUT_MILLIS} window to catch up in-place and skips the assertion - * entirely if the authoritative log shows the run produced no output at all. Unlike that method, - * it does not retry via a separate, explicit {@link StreamedLog} on a mismatch: that - * retry goes through a different code path than the one under test here ({@code call(...)}'s own - * default-streaming wiring, not the standalone {@code getStreamedLog} API), so using it as a - * rescue would let a genuinely broken default (e.g. log streaming silently not enabled) pass by - * being quietly repopulated from the bypassing stream instead of failing. The bounded catch-up - * poll is kept as this variant's sole flake mitigation. - */ - static void assertCallDefaultStreamedLogNonEmptyIfProduced( - RunClient runClient, List collected) { - pollUntil(STREAM_CATCH_UP_ATTEMPTS, STREAM_CATCH_UP_POLL_MILLIS, () -> !collected.isEmpty()); - assertNonEmptyIfLogProduced(runClient.log().get().join(), collected); - } - - /** Whether the authoritative, statically-persisted log shows the run produced any output. */ - private static boolean runProducedLog(Optional authoritativeLog) { - return authoritativeLog.isPresent() && !authoritativeLog.get().isEmpty(); - } - - /** - * Shared assertion backing both {@link #assertStreamedLogNonEmptyIfProduced} and {@link - * #assertCallDefaultStreamedLogNonEmptyIfProduced}: fails if the run produced a non-empty - * authoritative log but nothing was collected, and is a no-op otherwise (including when the run - * produced no log at all, in which case there is nothing to have streamed). - */ - private static void assertNonEmptyIfLogProduced( - Optional authoritativeLog, List collected) { - if (runProducedLog(authoritativeLog)) { + Optional persistedLog = runClient.log().get().join(); + if (persistedLog.isPresent() && !persistedLog.get().isEmpty()) { assertTrue( !collected.isEmpty(), - "run produced a non-empty log (" - + authoritativeLog.get().length() - + " chars) but the streamed collector observed none - log streaming/redirection did" - + " not work"); + "run produced a non-empty log (%d chars) but the streamed collector observed none" + .formatted(persistedLog.get().length())); } } } diff --git a/src/test/java/com/apify/client/integration/IterationIntegrationTest.java b/src/test/java/com/apify/client/integration/IterationIntegrationTest.java index ecf8ad9..4cfa7b2 100644 --- a/src/test/java/com/apify/client/integration/IterationIntegrationTest.java +++ b/src/test/java/com/apify/client/integration/IterationIntegrationTest.java @@ -39,11 +39,11 @@ import tools.jackson.databind.JsonNode; /** - * Iteration coverage for every paginated collection client that exposes an {@code iterate} helper, - * mirroring the reference client's async iterators. Where creation is cheap, tests create a few - * uniquely-named resources and assert iteration finds them (so paging is exercised across pages); - * for resources whose setup is expensive (builds, runs, dispatches) they iterate a bounded slice - * and assert the publisher behaves. All resources are uniquely named for parallel isolation. + * Iteration coverage for every paginated collection client that exposes an {@code iterate} helper. + * Where creation is cheap, tests create a few uniquely-named resources and assert iteration finds + * them (so paging is exercised across pages); for resources whose setup is expensive (builds, runs, + * dispatches) they iterate a bounded slice and assert the publisher behaves. All resources are + * uniquely named for parallel isolation. */ class IterationIntegrationTest extends IntegrationBase { @@ -111,10 +111,10 @@ public void onComplete() { /** * Like {@link #findsAll}, but tolerant of collection LIST-endpoint eventual consistency. A * resource created through a write endpoint is not always immediately reflected in its - * collection's LIST response — the write and the list index converge asynchronously on the - * server, so a create-then-iterate assertion that scans exactly once races that convergence and - * flakes when the just-created entity has not yet propagated. Delegates the retry/backoff shape - * to {@link IntegrationBase#pollUntil}, rebuilding a fresh publisher via {@code newPublisher} on + * collection's LIST response: the write and the list index converge asynchronously on the server, + * so a create-then-iterate assertion that scans exactly once races that convergence and flakes + * when the just-created entity has not yet propagated. Delegates the retry/backoff shape to + * {@link IntegrationBase#pollUntil}, rebuilding a fresh publisher via {@code newPublisher} on * every attempt (up to {@link #ITER_FIND_ATTEMPTS} times, sleeping {@link * #ITER_FIND_BACKOFF_MILLIS} between attempts) and returning as soon as every target id is seen. * An already-consistent account matches on the first pass with no sleeping. diff --git a/src/test/java/com/apify/client/integration/ScheduleIntegrationTest.java b/src/test/java/com/apify/client/integration/ScheduleIntegrationTest.java index eea9704..1d22670 100644 --- a/src/test/java/com/apify/client/integration/ScheduleIntegrationTest.java +++ b/src/test/java/com/apify/client/integration/ScheduleIntegrationTest.java @@ -55,7 +55,7 @@ void getScheduleLog() { // empty (not null/absent) list: the endpoint's "no entries" response must still decode // through the {"data": [...]} envelope rather than erroring or resolving to null. Decoding of // populated entries (message/level/createdAt) is covered against a real wire-format example - // in ClientBehaviourRegressionTest#scheduleGetLogDecodesEnvelopeArray — producing a non-empty + // in ClientBehaviourRegressionTest#scheduleGetLogDecodesEnvelopeArray. Producing a non-empty // log here would require waiting for an actual cron invocation, which isn't practical in a // short-lived integration test. assertEquals(List.of(), client.schedule(sch.getId()).getLog().join()); @@ -77,8 +77,7 @@ void scheduleCrudFlow() { // list() step of the create/get/modify/list/delete flow. assertTrue(client.schedules().list(new ListOptions().limit(5L)).join().getTotal() >= 0); - // Typed getters (previously only reachable via getExtra()): verify the API's response - // actually deserializes into them, not just that the code compiles. + // Typed getters: verify the API's response deserializes into them. assertTrue(sch.isExclusive()); assertTrue(sch.getTimezone() != null && !sch.getTimezone().isEmpty()); assertTrue(sch.getCreatedAt() != null); diff --git a/src/test/java/com/apify/client/integration/TaskIntegrationTest.java b/src/test/java/com/apify/client/integration/TaskIntegrationTest.java index 61d21c5..f0ee867 100644 --- a/src/test/java/com/apify/client/integration/TaskIntegrationTest.java +++ b/src/test/java/com/apify/client/integration/TaskIntegrationTest.java @@ -86,8 +86,7 @@ void taskCrudFlow() { .anyMatch(t -> task.getId().equals(t.getId()))); assertTrue(foundInList, "expected the just-created task to appear in the top-level list"); - // Typed getters (previously only reachable via getExtra()): verify the API's response - // actually deserializes into them, not just that the code compiles. + // Typed getters: verify the API's response deserializes into them. assertEquals("Integration test task", task.getDescription()); assertTrue(task.getStats() != null); assertTrue(task.getOptions() != null && "latest".equals(task.getOptions().getBuild())); @@ -142,9 +141,8 @@ void taskLastRunAndWebhooks() { assertTrue(lastRun.isPresent()); assertEquals(run.getId(), lastRun.get().getId()); - // Last-run-scoped nested storage GETs (previously untested — only `lastRun().get()` itself - // was). This task is exclusively owned by this test, so `lastRun` is deterministically the - // run started above and its default storages are guaranteed to exist. + // Last-run-scoped nested storage GETs. The task is owned by this test, so the last run is the + // one started above. tc.lastRun("SUCCEEDED").dataset().listItems(new DatasetListItemsOptions()).join(); tc.lastRun("SUCCEEDED").keyValueStore().getRecord("OUTPUT").join(); assertTrue(tc.lastRun("SUCCEEDED").log().get().join().isPresent()); @@ -165,9 +163,8 @@ void taskCallStreamsLogByDefault() { try { TaskClient tc = client.task(task.getId()); List collected = new CopyOnWriteArrayList<>(); - // The log-streaming call() overload (TaskCallOptions), matching the reference client's - // default call(options.log='default') behavior: the run's log is streamed for the duration - // of the wait without any explicit opt-in. + // The log-streaming call() overload (TaskCallOptions) streams the run's log during the wait + // without an explicit opt-in. ActorRun run = tc.call( null, @@ -175,16 +172,8 @@ void taskCallStreamsLogByDefault() { TEST_ACTOR_WAIT_SECS) .join(); assertEquals("SUCCEEDED", run.getStatus()); - // As in ActorRunIntegrationTest#callWithActorCallOptionsStreamsLogByDefault: call()'s - // internal log-streaming lifecycle closes the stream as soon as the run finishes, which can - // race the background reader before it has pulled any bytes off the live log stream, even - // though the log content is already fully available server-side. Gate the assertion on - // whether the run actually produced a log at all - via the call()-default variant, which - // never - // rescues via a separate explicit stream, so a genuinely broken default-streaming wiring - // still - // fails here rather than being masked. - assertCallDefaultStreamedLogNonEmptyIfProduced(client.run(run.getId()), collected); + // The assertion applies only if the run logged anything. + assertStreamedLogNonEmptyIfProduced(client.run(run.getId()), collected); } finally { client.task(task.getId()).delete().join(); } diff --git a/src/test/java/com/apify/client/integration/UserIntegrationTest.java b/src/test/java/com/apify/client/integration/UserIntegrationTest.java index 3ebf0e1..658fa59 100644 --- a/src/test/java/com/apify/client/integration/UserIntegrationTest.java +++ b/src/test/java/com/apify/client/integration/UserIntegrationTest.java @@ -63,7 +63,6 @@ void getLimitsAndMonthlyUsageByUserId() { // Note: `me().updateLimits(...)` is intentionally NOT exercised by a live integration test. It // mutates account-global state that cannot be isolated between the concurrent cross-language test // runs the test-requirements mandate (unlike the per-run resources every other test creates), so - // a - // mutating test here would race those runs. Its request behaviour is covered offline instead by + // a mutating test here would race those runs. Its request behaviour is covered offline instead by // ClientBehaviourRegressionTest#updateLimitsSendsPutToMeLimits. } diff --git a/src/test/java/com/apify/client/integration/WebhookIntegrationTest.java b/src/test/java/com/apify/client/integration/WebhookIntegrationTest.java index f2b6b55..8ed1212 100644 --- a/src/test/java/com/apify/client/integration/WebhookIntegrationTest.java +++ b/src/test/java/com/apify/client/integration/WebhookIntegrationTest.java @@ -122,8 +122,7 @@ void webhookCrudFlow() { assertTrue( foundInList, "expected the just-created webhook to appear in the top-level list"); - // Typed getters (previously only reachable via getExtra()): verify the API's response - // actually deserializes into them, not just that the code compiles. + // Typed getters: verify the API's response deserializes into them. assertEquals(task.getId(), wh.getCondition().get("actorTaskId").asString()); assertTrue(wh.getCreatedAt() != null); assertTrue(wh.getModifiedAt() != null); diff --git a/src/test/java/com/apify/client/internal/ResourceContextPathSafetyTest.java b/src/test/java/com/apify/client/internal/ResourceContextPathSafetyTest.java index 9187ef1..2015528 100644 --- a/src/test/java/com/apify/client/internal/ResourceContextPathSafetyTest.java +++ b/src/test/java/com/apify/client/internal/ResourceContextPathSafetyTest.java @@ -7,8 +7,7 @@ /** * Pins {@link ResourceContext#toSafeId} and {@link ResourceContext#encodePathSegment} against - * path-traversal / path-restructuring inputs, mirroring the reference JS client's hardening (apify-client-js#1011). + * path-traversal and path-restructuring inputs. */ class ResourceContextPathSafetyTest {