diff --git a/docs/02_concepts/02_error-handling.md b/docs/02_concepts/02_error-handling.md index a1bc3e0d3..fa36091a2 100644 --- a/docs/02_concepts/02_error-handling.md +++ b/docs/02_concepts/02_error-handling.md @@ -123,6 +123,8 @@ try { Bodies the specification leaves to you aren't validated: dataset items, key-value store records and logs are returned as they are. +Validation also normalizes URL fields, such as `ActorRun.containerUrl` or `Dataset.consoleUrl`. The value is parsed as a URL and you get back its serialization, so an empty path gains a trailing slash and the host is lowercased. For the full list of changes and how to compare such a field with a stored string, see [URL fields are normalized](../04_upgrading/upgrading_v3.md#url-fields-are-normalized). + ## Retries with exponential backoff The client automatically retries requests that fail due to network errors, Apify API internal errors (HTTP 500+), or rate limit errors (HTTP 429). By default, the client retries up to 8 times with exponential backoff starting at 500ms. diff --git a/docs/04_upgrading/upgrading_v3.md b/docs/04_upgrading/upgrading_v3.md index 42464c175..fb8e1dfc6 100644 --- a/docs/04_upgrading/upgrading_v3.md +++ b/docs/04_upgrading/upgrading_v3.md @@ -176,6 +176,46 @@ Two return types change as a result of describing what the endpoints really retu - `ScheduleClient.getLog()` was typed as a `string`, even though the endpoint returns the log as a list of entries. It's now typed as `ScheduleInvoked[]`, each entry carrying `message`, `level` and `createdAt`. - `TaskPublicConfig` now follows the specification: `publishedAt` is optional and read-only, and `categorization`, which the specification doesn't describe, is gone from the type. +## URL fields are normalized + +Fields the specification marks as a URL, such as `ActorRun.containerUrl` or `Dataset.consoleUrl`, are parsed with the [WHATWG `URL`](https://developer.mozilla.org/en-US/docs/Web/API/URL) parser as part of response validation, and the client hands back the parsed URL's serialization. In v2 you got the raw string from the API. In v3 the string can differ, most visibly by an added trailing slash. Normalization also lowercases the host, drops a default port, punycodes an internationalized host, and percent-encodes unsafe characters. Both forms denote the same URL under [RFC 3986](https://datatracker.ietf.org/doc/html/rfc3986#section-6.2). They're just different strings. + +```js +// An empty path becomes '/'. +new URL('https://abc123.runs.apify.net').href; // 'https://abc123.runs.apify.net/' + +// The host is lowercased. +new URL('https://EXAMPLE.com/Path').href; // 'https://example.com/Path' + +// A default port is dropped. +new URL('https://example.com:443/path').href; // 'https://example.com/path' + +// An internationalized host is punycoded. +new URL('https://www.žluty.cz').href; // 'https://www.xn--luty-kbb.cz/' + +// Unsafe characters are percent-encoded. +new URL('https://example.com/a b').href; // 'https://example.com/a%20b' +``` + +Code that compares a stored URL with a URL field has to compare normalized values: + +```js +const run = await client.run('my-run-id').get(); +const storedUrl = 'https://abc123.runs.apify.net'; + +// The raw string no longer matches. +storedUrl === run.containerUrl; // false + +// Normalize the stored side too. +new URL(storedUrl).href === run.containerUrl; // true +``` + +The affected fields are `ActorRun.containerUrl`, `Task.standbyUrl`, `Webhook.requestUrl` (on the full webhook, on a list item, and on the webhook summary a dispatch carries), `Dataset.consoleUrl`, `Dataset.itemsPublicUrl`, `KeyValueStore.consoleUrl`, `KeyValueStore.keysPublicUrl`, `KeyValueStore.recordsPublicUrl`, `KeyValueListItem.recordPublicUrl`, `RequestQueue.consoleUrl`, `ActorStoreList.url`, `ActorStoreList.userPictureUrl`, `UserProfile.pictureUrl`, and `UserProfile.websiteUrl`. Whether a field is normalized depends on its model. The specification doesn't mark `Actor.standbyUrl`, `Actor.pictureUrl`, `ActorStoreList.pictureUrl`, or the `url` of a request queue request as URLs, so those come back exactly as the API sent them. + +A trailing slash appears only on a field the API returns without a path, so on `containerUrl`, `standbyUrl`, `websiteUrl`, and a `Webhook.requestUrl` you registered without one. The rest already carry a path, and normalization leaves it alone. To append to a URL field, use `new URL('status', run.containerUrl)` only when the field ends with a slash: a relative reference replaces the base's last path segment, so on `consoleUrl` it would drop the resource ID. + +A URL field whose value isn't a valid absolute URL now fails response validation and throws `ResponseValidationError`, the same as any other field that doesn't match the specification. + ## `versions().list()` and `envVars().list()` take no options `ActorVersionCollectionClient.list()` and `ActorEnvVarCollectionClient.list()` now take no arguments. Neither endpoint reads `offset`, `limit` or `desc`, and both return every item in one response, so `chunkSize` had nothing to size either. The `ActorVersionCollectionListOptions` and `ActorEnvVarCollectionListOptions` types that declared those four options, deprecated since v2.21.0, are gone from the package. A call that passed an options object no longer compiles. Drop the argument and the call returns the same items as before. diff --git a/scripts/schema_emitter.mts b/scripts/schema_emitter.mts index 85cfb8407..498d40b12 100644 --- a/scripts/schema_emitter.mts +++ b/scripts/schema_emitter.mts @@ -6,7 +6,7 @@ * to another schema is a call of that thunk. Building all of them up front would cost more than importing zod * itself, so instead a schema is built once, by the first response validated against it. * - * The output is meant for validating API *responses*, which shapes three choices that differ from a literal + * The output is meant for validating API *responses*, which shapes four choices that differ from a literal * translation of the specification: * * - Every object is a `z.looseObject()`, so a field the API starts returning before the specification documents @@ -16,11 +16,15 @@ * does not fail every response that carries it. This is the Python client's `Literal[...] | str`. * - `format: date-time` becomes `z.date()`, because the client converts those fields into `Date` objects * before a response is validated, and the generated types say `Date` for the same reason. + * - `format: uri` becomes `z.url({ normalize: true })`, which parses the value as a WHATWG URL and hands back + * its serialization: an empty path gains a `/`, the host is lowercased and punycoded, a default port is dropped + * and unsafe characters are percent-encoded. A value that is not a valid absolute URL is rejected. This is the + * Python client's `pydantic.AnyUrl`, so both clients return the same string for the same field. * * A `default` is deliberately not applied: it says what the server fills in when a request omits the field, not - * what a response carries, so the parsed value stays identical to the response. A `format` other than `date-time` - * (`uri`, `email`, ...) is an annotation rather than a constraint and is not enforced either. Every other constraint - * the specification states -- `required`, `minimum`, `pattern`, ... -- is kept, so a response that violates the + * what a response carries, so the parsed value stays identical to the response. A `format` other than the two + * above (`email`, ...) is an annotation rather than a constraint and is not enforced. Every other constraint the + * specification states -- `required`, `minimum`, `pattern`, ... -- is kept, so a response that violates the * documented contract is reported rather than passed on. * * Only the JSON Schema vocabulary the specification actually uses is supported. Anything else fails generation @@ -269,7 +273,7 @@ class Emitter { private string(node: SchemaNode, path: string): string { if (node.format === 'date-time') return 'z.date()'; - let out = 'z.string()'; + let out = node.format === 'uri' ? 'z.url({ normalize: true })' : 'z.string()'; if (node.minLength !== undefined) out += `.min(${node.minLength})`; if (node.maxLength !== undefined) out += `.max(${node.maxLength})`; if (node.pattern !== undefined) out += `.regex(${regexLiteral(node.pattern, path)})`; diff --git a/src/generated/schemas.ts b/src/generated/schemas.ts index 6c4b75cb4..39f5298c7 100644 --- a/src/generated/schemas.ts +++ b/src/generated/schemas.ts @@ -379,7 +379,7 @@ export const WebhookShort = lazySchema(() => z.looseObject({ condition: WebhookCondition(), ignoreSslErrors: z.boolean(), doNotRetry: z.boolean(), - requestUrl: z.string(), + requestUrl: z.url({ normalize: true }), lastDispatch: ExampleWebhookDispatch().nullable().optional(), stats: WebhookStats().nullable().optional(), })); @@ -649,7 +649,7 @@ export const Run = lazySchema(() => z.looseObject({ }).catchall(z.string()).optional(), }).optional(), buildNumber: z.string().regex(/^([0-9]|[1-9][0-9])\.([0-9]|[1-9][0-9])(\.[1-9][0-9]{0,4})$/).nullable().optional(), - containerUrl: z.string().optional(), + containerUrl: z.url({ normalize: true }).optional(), isContainerServerReady: z.boolean().nullable().optional(), gitBranchName: z.string().nullable().optional(), usage: RunUsage().nullable().optional(), @@ -683,8 +683,8 @@ export const Dataset = lazySchema(() => z.looseObject({ actRunId: z.string().nullable().optional(), fields: z.array(z.string()).nullable().optional(), schema: z.record(z.string(), z.unknown()).nullable().optional(), - consoleUrl: z.string(), - itemsPublicUrl: z.string().optional(), + consoleUrl: z.url({ normalize: true }), + itemsPublicUrl: z.url({ normalize: true }).optional(), urlSigningSecretKey: z.string().nullable().optional(), generalAccess: GeneralAccess().nullable().optional(), stats: DatasetStats().optional(), @@ -767,9 +767,9 @@ export const KeyValueStore = lazySchema(() => z.looseObject({ accessedAt: z.date(), actId: z.string().nullable().optional(), actRunId: z.string().nullable().optional(), - consoleUrl: z.string().optional(), - keysPublicUrl: z.string().optional(), - recordsPublicUrl: z.string().optional(), + consoleUrl: z.url({ normalize: true }).optional(), + keysPublicUrl: z.url({ normalize: true }).optional(), + recordsPublicUrl: z.url({ normalize: true }).optional(), schema: z.record(z.string(), z.unknown()).nullable().optional(), urlSigningSecretKey: z.string().nullable().optional(), generalAccess: GeneralAccess().nullable().optional(), @@ -789,7 +789,7 @@ export const UpdateStoreRequest = lazySchema(() => z.looseObject({ export const KeyValueStoreKey = lazySchema(() => z.looseObject({ key: z.string(), size: z.int(), - recordPublicUrl: z.string(), + recordPublicUrl: z.url({ normalize: true }), })); export const ListOfKeys = lazySchema(() => z.looseObject({ @@ -867,7 +867,7 @@ export const RequestQueue = lazySchema(() => z.looseObject({ handledRequestCount: HandledRequestCount(), pendingRequestCount: PendingRequestCount(), hadMultipleClients: HadMultipleClients(), - consoleUrl: z.string(), + consoleUrl: z.url({ normalize: true }), stats: RequestQueueStats().optional(), generalAccess: GeneralAccess().nullable().optional(), })); @@ -1184,7 +1184,7 @@ export const Task = lazySchema(() => z.looseObject({ input: z.union([TaskInput(), z.array(TaskInput())]).nullable().optional(), title: z.string().nullable().optional(), actorStandby: ActorStandby().nullable().optional(), - standbyUrl: z.string().nullable().optional(), + standbyUrl: z.url({ normalize: true }).nullable().optional(), isPublic: z.boolean().optional(), publicConfig: TaskPublicConfig().nullable().optional(), })); @@ -1215,7 +1215,7 @@ export const Webhook = lazySchema(() => z.looseObject({ condition: WebhookCondition(), ignoreSslErrors: z.boolean(), doNotRetry: z.boolean().nullable().optional(), - requestUrl: z.string().nullable().optional(), + requestUrl: z.url({ normalize: true }).nullable().optional(), payloadTemplate: z.string().nullable().optional(), headersTemplate: z.string().nullable().optional(), description: z.string().nullable().optional(), @@ -1328,7 +1328,7 @@ export const WebhookUpdate = lazySchema(() => z.looseObject({ condition: WebhookCondition().nullable().optional(), ignoreSslErrors: z.boolean().nullable().optional(), doNotRetry: z.boolean().nullable().optional(), - requestUrl: z.string().nullable().optional(), + requestUrl: z.url({ normalize: true }).nullable().optional(), payloadTemplate: z.string().nullable().optional(), headersTemplate: z.string().nullable().optional(), description: z.string().nullable().optional(), @@ -1339,7 +1339,7 @@ export const WebhookUpdate = lazySchema(() => z.looseObject({ export const WebhookDispatchWebhookSummary = lazySchema(() => z.looseObject({ actionType: z.string().optional(), condition: WebhookCondition().optional(), - requestUrl: z.string().nullable().optional(), + requestUrl: z.url({ normalize: true }).nullable().optional(), isAdHoc: z.boolean().optional(), })); @@ -1521,8 +1521,8 @@ export const StoreListActor = lazySchema(() => z.looseObject({ categories: z.array(z.string()).optional(), notice: ActorNotice().optional(), pictureUrl: z.string().nullable().optional(), - userPictureUrl: z.string().nullable().optional(), - url: z.string().nullable().optional(), + userPictureUrl: z.url({ normalize: true }).nullable().optional(), + url: z.url({ normalize: true }).nullable().optional(), stats: ActorStats(), currentPricingInfo: CurrentPricingInfo().optional(), isWhiteListedForAgenticPayments: z.boolean().nullable().optional(), @@ -1544,9 +1544,9 @@ export const ListOfActorsInStoreResponse = lazySchema(() => z.looseObject({ export const Profile = lazySchema(() => z.looseObject({ bio: z.string().nullable().optional(), name: z.string().optional(), - pictureUrl: z.string().nullable().optional(), + pictureUrl: z.url({ normalize: true }).nullable().optional(), githubUsername: z.string().nullable().optional(), - websiteUrl: z.string().nullable().optional(), + websiteUrl: z.url({ normalize: true }).nullable().optional(), twitterUsername: z.string().nullable().optional(), })); diff --git a/src/utils.ts b/src/utils.ts index 7fb0a3476..3fc0c7d3c 100644 --- a/src/utils.ts +++ b/src/utils.ts @@ -56,8 +56,9 @@ const { localeError } = z.locales.en(); /** * Turns a JSON API response into the value a resource method returns: unwraps the `data` envelope, converts the * date fields and validates the result against `schema`, one of the schemas generated from the OpenAPI - * specification. The validated copy is what callers get, so it is exactly what the schema accepted -- unknown - * fields and unknown enum values included, since the schemas let both through. + * specification. The validated copy is what callers get, so it is the schema's output -- unknown fields and unknown + * enum values included, since the schemas let both through, and URL fields normalized, since `z.url()` hands back + * the parsed URL's serialization. * * Throws {@link ResponseValidationError} when the response does not match the specification. * @internal diff --git a/test/response_validation_error.test.ts b/test/response_validation_error.test.ts index 51c916a79..e32274970 100644 --- a/test/response_validation_error.test.ts +++ b/test/response_validation_error.test.ts @@ -104,4 +104,32 @@ describe('response validation in the client', () => { expect(res?.generalAccess).toBeNull(); }); + + test('a URL field comes back normalized, with a lowercased host and a default port dropped', async () => { + mockServer.setResponse({ + body: { data: { ...fixtures.run, containerUrl: 'https://G8KD8KBC5GE8.runs.apify.net:443' } }, + }); + + const res = await client.run('some-run-id').get(); + + expect(res?.containerUrl).toBe('https://g8kd8kbc5ge8.runs.apify.net/'); + }); + + test('normalization leaves the path and the query of a URL field alone, so a signature survives it', async () => { + const url = 'https://g8kd8kbc5ge8.runs.apify.net/Items?signature=a+b%2Fc%3D'; + mockServer.setResponse({ body: { data: { ...fixtures.run, containerUrl: url } } }); + + const res = await client.run('some-run-id').get(); + + expect(res?.containerUrl).toBe(url); + }); + + test('a URL field that is not a URL throws ResponseValidationError', async () => { + mockServer.setResponse({ body: { data: { ...fixtures.run, containerUrl: 'not a url' } } }); + + const call = client.run('some-run-id').get(); + + await expect(call).rejects.toBeInstanceOf(ResponseValidationError); + await expect(call).rejects.toThrow('at `containerUrl`'); + }); }); diff --git a/test/schema_emitter.test.mts b/test/schema_emitter.test.mts index 87d413d32..8630af69a 100644 --- a/test/schema_emitter.test.mts +++ b/test/schema_emitter.test.mts @@ -115,8 +115,15 @@ describe('emitSchema', () => { expect(emitSchema({ type: ['string', 'null'], format: 'date-time' })).toBe('z.date().nullable()'); }); + it('emits uri as a normalizing z.url(), so a URL comes back the way the Python client returns it', () => { + expect(emitSchema({ type: 'string', format: 'uri' })).toBe('z.url({ normalize: true })'); + expect(emitSchema({ type: ['string', 'null'], format: 'uri' })).toBe( + 'z.url({ normalize: true }).nullable()', + ); + }); + it('leaves other string formats as plain strings', () => { - expect(emitSchema({ type: 'string', format: 'uri' })).toBe('z.string()'); + expect(emitSchema({ type: 'string', format: 'email' })).toBe('z.string()'); }); it('emits boolean, null and a schema without a type', () => {