Skip to content

gh-88336: Fix UnicodeDecodeError with multibyte utf8 characters in ShareableList - #26328

Open
junnplus wants to merge 4 commits into
python:mainfrom
junnplus:bpo44170-multibyte-fix
Open

gh-88336: Fix UnicodeDecodeError with multibyte utf8 characters in ShareableList#26328
junnplus wants to merge 4 commits into
python:mainfrom
junnplus:bpo44170-multibyte-fix

Conversation

@junnplus

@junnplus junnplus commented May 24, 2021

Copy link
Copy Markdown

This PR fix UnicodeDecodeError with multibyte utf8 characters in ShareableList.

>> from multiprocessing.shared_memory import ShareableList
>>> strings = ["Boom 💥 💥 💥"]
>>> ShareableList(strings)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/Jun/.pyenv/versions/3.9.0/lib/python3.9/multiprocessing/shared_memory.py", line 479, in __repr__
    return f'{self.__class__.__name__}({list(self)}, name={self.shm.name!r})'
  File "/Users/Jun/.pyenv/versions/3.9.0/lib/python3.9/multiprocessing/shared_memory.py", line 435, in __getitem__
    v = back_transform(v)
  File "/Users/Jun/.pyenv/versions/3.9.0/lib/python3.9/multiprocessing/shared_memory.py", line 277, in <lambda>
    1: lambda value: value.rstrip(b'\x00').decode(_encoding),  # str
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf0 in position 15: unexpected end of data

https://bugs.python.org/issue44170

@the-knights-who-say-ni

Copy link
Copy Markdown

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA).

Recognized GitHub username

We couldn't find a bugs.python.org (b.p.o) account corresponding to the following GitHub usernames:

@junnplus

This might be simply due to a missing "GitHub Name" entry in one's b.p.o account settings. This is necessary for legal reasons before we can look at this contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

You can check yourself to see if the CLA has been received.

Thanks again for the contribution, we look forward to reviewing it!

Comment thread Lib/multiprocessing/shared_memory.py Outdated
if isinstance(value, str) else value)
if len(encoded_value) > allocated_length:
encoded_value = self._encode_value(value)
if len(encoded_value) >= allocated_length:

@junnplus junnplus May 24, 2021

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

>>> from multiprocessing.shared_memory import ShareableList
>>> s1 = ShareableList(['1234567'])
>>> s1.format
'8s'
>>> s2 = ShareableList(['12345678'])
>>> s2.format
'16s'
>>> s1[0] = '12345678'  # Is this behavior expected?
>>> s1.format
'8s'
>>> s3 = ShareableList(s1)
>>> s3.format
'16s'

@github-actions

Copy link
Copy Markdown

This PR is stale because it has been open for 30 days with no activity.

@github-actions github-actions Bot added the stale Stale PR or inactive for long period of time. label Jun 27, 2021
@github-actions github-actions Bot removed the stale Stale PR or inactive for long period of time. label Aug 9, 2022
@github-actions

github-actions Bot commented Apr 9, 2026

Copy link
Copy Markdown

This PR is stale because it has been open for 30 days with no activity.

@github-actions github-actions Bot added the stale Stale PR or inactive for long period of time. label Apr 9, 2026
@github-actions github-actions Bot removed the stale Stale PR or inactive for long period of time. label Aug 11, 2026
@serhiy-storchaka serhiy-storchaka changed the title bpo-44170: Fix UnicodeDecodeError with multibyte utf8 characters in ShareableList gh-88336: Fix UnicodeDecodeError with multibyte utf8 characters in ShareableList Aug 12, 2026
Allocating one extra alignment block for a value whose encoded size is
a multiple of the alignment made a copy of the list wider than the
original.  Round the size up instead, so that it is the same for the
same value, and keep allowing a value which fills the slot exactly:
rejecting it broke setting an 8-byte string in a slot of an integer.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants