Repository navigation
Cross compile linking issue: cannot locate symbol "_Py_NoneStruct" #1077
Description
🐛 Bug Description
I need to cross compile a pyo3 library for android using kivy/buildozer. I've set the LD_LIBRARY_PATH, PYO3_CROSS_INCLUDE_DIR and PYO3_CROSS_LIB_DIR environment variables and am using cargo ndk for the build process and copying the .so to the site_packages directory.
Everything compiles correctly, unfortunately at runtime I get the message cannot locate symbol "_Py_NoneStruct".
🌍 Environment
- Your operating system and version: Ubuntu 20.04 (docker)
- Your python version: 3.8
- How did you install python (e.g. apt or pyenv)? Did you use a virtualenv?: compiled from source
- Your Rust version (
rustc --version): 1.45.0 - Your PyO3 version: 0.11.1
- Have you tried using latest PyO3 master (replace
version = "0.x.y"withgit = "https://github.com/PyO3/pyo3")?: yes
💥 Reproducing
I have a repo for the minimal working example. The necessary commands are shown in the README.md
Activity
davidhewitt commentedon Aug 3, 2020
Thanks for posting. This looks like a packaging issue, possibly in
kivy/python-for-android. Here's an example of a very similar looking upstream bug from 2016 which was closed. kivy/python-for-android#887As they will have much better knowledge than I for diagnosing the buildozer config, I suggest reporting upstream there before I attempt any debugging here.
Progdrasil commentedon Aug 3, 2020
I'll check that out, however I did have a similar issue when compiling the same project in termux I'm not sure if this constitutes a different issue, but it is the same target.
The target is
aarch64-linux-androidProgdrasil commentedon Aug 3, 2020
After reading through the comments it seems what has fixed it was adding the
LDFLAG+="-lpythonX.Ym"flag. I did get a linking issue when removing theextension-modulewhere the linking flags had-lpythonwithout the version. So it could not find the appropriate python library.The referenced comment
edit
I've managed to get it working with the following configuration:
extension-moduleenabled-C link-args=-L<LD_LIBRARY_PATH> -lpythonX.YmAnd it ONLY works with the
mversion of the library. I tested this for targetsaarch64-linux-androidandarmv7-linux-androideabi.Since this is a linking issue with extension-module would it be related to #1040 ?
davidhewitt commentedon Aug 3, 2020
Thanks for the update! Perhaps not #1040 but definitely worth remembering in this area. Let's keep this issue open until we adjust build scripts accordingly.
To figure out what's going on here - so it sounds to me like
libpython.sodoes not exist in the distribution you're compiling for, but presumablylibpython3.8m.sodoes?davidhewitt commentedon Aug 3, 2020
(I'm a bit confused because I thought the last python I'd seen where the library had the
msuffix was 3.6, but I've never found documentation about what the suffix was for, so perhaps there's something else at play here...)programmerjake commentedon Aug 3, 2020
my guess is m is for minimal like in the python3-minimal debian package.
davidhewitt commentedon Aug 3, 2020
Aha looks like it's showing if the ABI has pymalloc support. We can check it with sys.abiflags. However, it was definitely removed in 3.8: https://docs.python.org/3/whatsnew/3.8.html#build-and-c-api-changes
We could fix build scripts to check this for Python 3.5-3.7.
@Progdrasil, you're definitely cross compiling against Python 3.8 when you have this
msuffix?!Progdrasil commentedon Aug 3, 2020
Thats really interesting because i am compiling against 3.8. and both the 'm' version and normal versions are available. Ive tried linking against both and only the 'm' version works at runtime. If i link against the normal version i get the
symbol not founderror at runtimeI am curious to try compiling against 3.7 now. I'll update you when I have
davidhewitt commentedon Aug 4, 2020
After exploring the python-for-android codebase it looks to me like they're generating this
libpython3.8m.soname themselves, so I've opened an upstream patch.There are probably still fixes we can introduce on Python 3.5-3.7 for thisEDIT: looks like we checkmsuffix, where it is a valid identifier.sysconfig.get_config_var('LDVERSION')inbuild.rs, which (at least on my system) does include themsuffix for Python 3.7.Progdrasil commentedon Aug 4, 2020
Yes but that is used for local compilation, not cross compilation as I am doing. Instead you parse the header files and apply the major and minor version. Is there a way to get the configuration of the cross compiled version to see if it is being compiled as the
mvariant?davidhewitt commentedon Aug 4, 2020
Indeed, you're correct and I was thinking about this this morning too. Actually I think that the cross compilation header parsing is both broken and brittle; I'm thinking about whether we need to try a solution using
bindgen.If I were to kick off a PR to use bindgen to detect cross-compile info, would you be willing to help test it? (And maybe even help finish it off, depending on how much time I can find for it...?)
Progdrasil commentedon Aug 4, 2020
Also I succeeded in compiling for 3.7 and running the
sysconfig.get_config_var(LDVERSION)on the target for 3.7 it returned3.7mand when running with 3.8 it returned3.8. However I think your merge request helps because the issue on the target was the fact that I was linking againstlibpython3.8.sowhile the only object that was copied to the target waslibpython3.8m.so.Progdrasil commentedon Aug 4, 2020
Yeah I would be glad to help out! I can also help contribute to it as well
davidhewitt commentedon Aug 4, 2020
Great! I'll try to either get notes or a rough implementation written up tomorrow or soon after.
davidhewitt commentedon Aug 5, 2020
So I took a look at the
bindgenapproach this morning, but it looks like that's not the way to go about this either. A lot of the variables are set by the build system rather than configuration headers, so parsing the headers doesn't give us everything we want.Instead, I started looking at the
sysconfigmodule and saw that we might be able to take another approach. Instead of running the python interpreter to extract sysconfig, I think we might be able to parse the underlying data file. Some examples of the data file on my system:Also interesting to note that the name of the data file has that "m" abiflag again, so it might be possible to infer that from the name of the sysconfigdata script.
Afraid I've got a lot of other things to focus on at the moment, so I'm not sure I have time to write the implementation right now. However, I'd be very willing to help design and review the implmentation if you want to take this on?
If this is potentially nontrivial it could be worth factoring it out into a crate rather than inlining it all into pyo3's
build.rs. I actually already started a crate which might be appropriate https://github.com/davidhewitt/python-config - perhaps if we can get a working draft of this in that crate then we can subsequently move this crate into the pyo3 organisation and update pyo3's build script to use it?Progdrasil commentedon Aug 5, 2020
Sure I can start on this in my free time, might be able to do a bit tomorrow evening
davidhewitt commentedon Aug 5, 2020
👍 many thanks, and no rush at all. Any time you're willing to spend helping out is much appreciated, and I'll try to put some time in where I can! Hopefully over time we'll be able to put together a good cross-compiling support for pyo3 😄
Oh, also, while searching documentation this morning I came across another tidbit which would also contribute to the linking issues you're experiencing: #1082.
Progdrasil commentedon Aug 5, 2020
I'll keep that in mind with the cross compilation.
Load compilation options from _sysconfigdata_*.py file
Load compilation options from _sysconfigdata_*.py file
davidhewitt commentedon Aug 30, 2020
Closed by #1095. (Thanks very much again for your hard work!)