Skip to content

Support chunked text processing via StreamSafe - #120

Open
maratik123 wants to merge 2 commits into
unicode-rs:masterfrom
maratik123:master
Open

maratik123 wants to merge 2 commits into
unicode-rs:masterfrom
maratik123:master

Conversation

@maratik123

Copy link
Copy Markdown

Closes #119

@maratik123

Copy link
Copy Markdown
Author

I want to ask about .ok().expect(...).
Is it ok, or better to add #[derive(Debug)] to StreamSafe to make happy clippy (ok_expect) in future?
Because of that, test will fail on clippy immediatly if somebody will add this trait to StreamSafe.

Comment thread src/stream_safe.rs
@Manishearth

Copy link
Copy Markdown
Member

Derive debug is fine.

@Manishearth Manishearth left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looking good!

Comment thread src/stream_safe.rs
/// on the underlying iterator once. A character it returns is still
/// yielded by the returned iterator.
#[inline]
pub fn try_into_state(mut self) -> Result<StreamSafeState, Self> {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perhaps try_finish_into_state()?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But it is not about to finish. It is about to mine state on exhausted iterator. Here we try to consume iterator to transfer state for creating next iterator. On unsuccessful operation we return iterator (it's up to user what to do with it - to drop or to use).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm we're "finishing" the current iterator. I kind of want the function name to make it clear that this is a finished iterator.

I do agree that we're not really finishing here. Have suggestions?

@maratik123 maratik123 Oct 28, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about try_into_final_state()? The try_into_ part keeps the Arc::try_unwrap-style convention (consume self, hand it back as Err on failure), and "final" says it's the state of a finished iterator without implying that the call finishes anything. If you'd rather have the condition spelled out, into_state_if_exhausted() works too, but we lost here try_ convention. try_into_state_if_exhausted() is too wordly.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

try_into_state_if_exhausted or try_exhausted_into_state maybe. I'm ok if it's a bit wordy, it's a poweruser API

Comment thread src/stream_safe.rs
/// assert!(out.contains('\u{034F}'));
/// ```
#[inline]
pub fn with_state(iter: I, state: StreamSafeState) -> Self {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

new_resume_from_state()?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, "new" and "resume" could confuse user if both used nearby. And what means "resume iterator from state"? For me it cumbersome.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're making a new iterator and resuming from existing state. Open to other name suggestions

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about just resume(iter, state)? It's an associated function returning Self, so it already reads as a constructor without the new_ prefix, and the call site reads naturally: StreamSafe::resume(piece.chars(), state). It also pairs well with the state-taking method.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, that seems fine

Comment thread src/stream_safe.rs

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

StreamSafe: carry the non-starter count across separate inputs (chunked streams)

2 participants