Skip to content

[core][flink][spark] Support OSS blob presigned URLs#8832

Open
jerry-024 wants to merge 7 commits into
apache:masterfrom
jerry-024:support_build_in_function
Open

[core][flink][spark] Support OSS blob presigned URLs#8832
jerry-024 wants to merge 7 commits into
apache:masterfrom
jerry-024:support_build_in_function

Conversation

@jerry-024

@jerry-024 jerry-024 commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Add catalog-aware Flink and Spark SQL functions that convert OSS-backed BlobDescriptor values into short-lived HTTPS GET URLs. Paimon materializes only the descriptor's byte range as a suffix-free OSS object served as application/octet-stream, allowing downstream multimodal services to fetch the exact bytes without loading them into the query engine.

Changes

  • Extend FileIO and Blob with blob presigned URL support, including delegation through resolving, caching, plugin, and REST-token file systems and validation that descriptors belong to the source table.
  • Materialize OSS blob ranges with server-side multipart copy, reuse deterministic fingerprinted objects, and validate cached length, content type, and descriptor fingerprint before generating each URL.
  • Use configured OSS/Jindo clients so credentials and STS tokens are preserved; require HTTPS, map OSS internal endpoints to their public host, and verify the signed URL still targets the expected bucket and object.
  • Add three-argument descriptor_to_presigned_url(source_table, descriptor, validity) and try_descriptor_to_presigned_url(...) functions for Flink and Spark, including catalog binding, literal source-table validation, whole-second interval validation, Spark analyzer resolution, and Flink compatibility support.
  • Package the Jindo OSS dependencies consistently and update the license checker for Java multi-release JAR class entries.
  • Document SQL and Java usage, cache behavior, security guidance, and the suffix-free application/octet-stream behavior.

Testing

  • OSSFileIOTest#testCreateBlobPresignedUrlMaterializesSinglePart
  • Flink DescriptorToPresignedUrlFunctionTest#testTypeInferenceRequiresLiteralTableAndAcceptsInterval
  • Spark DescriptorToPresignedUrlFunctionTest#testBindAndNonDeterministic
  • Relevant module compilation, Checkstyle, Spotless, and Enforcer checks
  • Bailian smoke test using a suffix-free OSS object with application/octet-stream; returned bytes matched the source and the model consumed the URL successfully
  • Full local integration suites could not start in the restricted environment: server-port binding is denied and the local Azul JDK 11 cannot initialize Mockito's Byte Buddy self-attach. These failures occurred during test setup, before business assertions.

Notes

  • This revision replaces the earlier four-argument proposal: callers no longer provide a file extension, and no compatibility overload is retained because the API has not been released.
  • Consumers must inspect the bytes rather than relying on a URL suffix or format-specific content type.
  • source_table must be a non-null literal in database.table form, or catalog.database.table for the function's catalog.
  • Read regular BLOB columns with blob-as-descriptor=true before passing them to the function.
  • Presigned URLs are temporary bearer credentials and must not be logged or persisted.


public JindoFileIO() {}

JindoFileIO(OSSClient blobClient) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @sundapeng to take a review.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JingsongLi @jerry-024 LGTM for the changes to JindoFileIO. Just one minor comment.

@jerry-024 jerry-024 changed the title [core][flink][spark] Support OSS Blob presigned URLs [core][flink][spark] Add presigned URL support for OSS blobs Jul 27, 2026
@jerry-024 jerry-024 changed the title [core][flink][spark] Add presigned URL support for OSS blobs [core][flink][spark] Support OSS blob presigned URLs Jul 27, 2026
endpoint = "https://" + endpoint;
}
String securityToken = options.get(OSS_SECURITY_TOKEN);
OSSClientBuilder builder = new OSSClientBuilder();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better to add region config to OssClientBuilder, fs key is fs.oss.region

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants