Skip to content

Support binary messages over the LongPoll transport - #6907

Open
mandar1045 wants to merge 1 commit into
phoenixframework:mainfrom
mandar1045:fix-longpoll-binary-push
Open

mandar1045 wants to merge 1 commit into
phoenixframework:mainfrom
mandar1045:fix-longpoll-binary-push

Conversation

@mandar1045

Copy link
Copy Markdown

Closes #6906.

Binary pushes and replies over LongPoll were buffered as raw bytes inside the JSON messages list, so non UTF-8 data made the poll fail with a 500 (losing the other buffered messages), and UTF-8 data reached phoenix.js as a broken text frame.

The server now base64 encodes binary messages, and longpoll.js decodes them back into an ArrayBuffer before passing them to the socket, so the serializer handles them the same way as over WebSocket. This mirrors what we already do for client to server binary messages: anything that starts with [ (or { for V1) is text, everything else is base64.

Tests:

  • integration test that pushes and replies with binary over LongPoll (fails on main with {:invalid_byte, 255})
  • JS test for decoding base64 messages; I also changed the placeholder messages in the existing queued message tests to JSON strings, since plain words like "second" are valid base64

@rhcarvalho

Copy link
Copy Markdown
Collaborator

Hi! Thanks! This makes sense from a completeness point of view. Before I review the code in more detail, did you have a particular use case in mind (like uploads were the motivation in the other direction)?

An initial concern I have is that this sort of hides the implications of base64 encoding, making the payload much larger than the binary data.

Depending on the use case, the correct thing to do might be to fetch the binary data out of band.

@mandar1045

Copy link
Copy Markdown
Author

Hi! Thanks! This makes sense from a completeness point of view. Before I review the code in more detail, did you have a particular use case in mind (like uploads were the motivation in the other direction)?

An initial concern I have is that this sort of hides the implications of base64 encoding, making the payload much larger than the binary data.

Depending on the use case, the correct thing to do might be to fetch the binary data out of band.

No app of my own tbh. But pushing binary from channels is pretty common (Lightning does it for Yjs, ex_nvr for video frames), and any app doing that with longpoll enabled breaks on fallback: the poll 500s and the rest of the buffer gets dropped, join reply included.

Fair point on base64. It's only on the longpoll path though, same as uploads. For big stuff out of band is better for sure.

@mandar1045

Copy link
Copy Markdown
Author

If supporting it isn't worth it, I can make it log and skip the binary message so the rest still gets through, and add a docs note.

@SteffenDE

Copy link
Copy Markdown
Member

Since afaik this was never supported and nobody complained about it, I’m not sure if it is really so common.

In any case, let’s wait because #6887 changes a lot of the long polling code and already adds base64 for binary messages when handing over from websockets.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Binary pushes don't work over the LongPoll transport

3 participants