Bug 12611 - borgbackup: Please package pyfuse3
Summary: borgbackup: Please package pyfuse3
Status: CLOSED FIXED
Alias: None
Product: IPFire
Classification: Unclassified
Component: --- (show other bugs)
Version: 2
Hardware: unspecified Unspecified
: Will only affect a few users Minor Usability
Assignee: Adolf Belka
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-04-20 12:12 UTC by Michael Tremer
Modified: 2022-09-16 09:23 UTC (History)
4 users (show)

See Also:


Attachments
borg mount ssh screenshot (9.22 KB, image/png)
2022-07-16 22:20 UTC, Thomas Void
Details
pakfire list python3 (61.22 KB, image/png)
2022-07-16 22:33 UTC, Thomas Void
Details
Build log from pyfuse3 using PyIter_Send in place of _PyGen_Send (6.40 KB, text/x-log)
2022-07-26 11:52 UTC, Adolf Belka
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Tremer 2021-04-20 12:12:59 UTC
We currently have a package called llfuse which does not install any files and therefore does not work.

llfuse is no longer maintained and borgbackup recommends using pyfuse3:

> https://github.com/borgbackup/borg/blob/master/setup.py#L76
Comment 2 Peter Müller 2022-03-02 20:37:45 UTC
Since borgbackup does not strictly depend on either llfuse or pyfuse3, and the former has been dropped in the aforementioned commit, I think this bug can be closed as WONTFIX. Please reopen, if necessary.
Comment 3 Thomas Void 2022-07-16 22:20:58 UTC
Created attachment 1070 [details]
borg mount ssh screenshot

Looks like this breaks borg mount (see attachment)
I haven't been able to check if this is true or just an assumption.
Comment 4 Thomas Void 2022-07-16 22:22:19 UTC
Re-open - see Comment 3
Comment 5 Thomas Void 2022-07-16 22:33:44 UTC
Created attachment 1071 [details]
pakfire list python3

I'm currently on core167 and tried to remove borgbackup, it's dependencies and also python3-llfuse (which interestingly is installed but not listed in pakfie list)
Comment 6 Adolf Belka 2022-07-17 08:58:42 UTC
borgbackup restore only has the option to restore the whole selected archive with additional exclude command to exclude some files.

To be able to restore specific files only from a borg archive you have to mount it as a fuse file system.

python3-llfuse was removed from IPFire in April 2021.

I will pick this up and test out the addition of pyfuse3 and evaluate the borg mount command in my vm testbed with and without pyfuse3 being built in.
Comment 7 Michael Tremer 2022-07-20 15:17:14 UTC
(In reply to Adolf Belka from comment #6)
> python3-llfuse was removed from IPFire in April 2021.

Yes, for this reason:

> Python-LLFUSE is no longer actively developed and just receiving community-contributed maintenance to keep it alive for some time.

From: https://pypi.org/project/llfuse/ - Just for the record.

pyfuse3 seems to have an additional dependency on a package called "trio". So it should be possible to package it.
Comment 8 Adolf Belka 2022-07-25 14:06:08 UTC
So I installed python3-trio and python3-pyfuse3 and ran the build.

The rootfiles for both of those packages were set to have all files included.

I then transferred the trio and pyfuse3 .ipfire packages to my vm clone and installed the files using the tar xvf, ./install.sh route.

Then I installed borgbackup using pakfire on the vm and it installed the other dependencies - packaging, pkgconfig and msgpack

Then I tried borg mount repo mount-point but still got the same failure message.

borg mount not available: no FUSE support, BORG_FUSE_IMPL=pyfuse3,llfuse

I checked in the /usr/lib/python3.10/site-packages/
and there is a directory for trio and for trio-0.21.0-py3.10.egg-info
and for pyfuse3-3.2.1-py3.10.egg-info

The _pyfuse3.py, pyfuse3.cpython-310-x86_64-linux-gnu.so and pyfuse3_asyncio.py files are directly in the site-packages directory and not in their own pyfuse3 directory

Other than the above error message there are no other error messages. The borgbackup, pyfuse3 and trio packages all build without any issue.

I suspect there is something else I need to set to ensure that borgbackup can see the pyfuse package but I can't find out what. There is nothing more in the borgbackup website, as for the installation they just say install everything with pip and there are no clues in the source code readme or install files.
Comment 9 Michael Tremer 2022-07-25 16:07:13 UTC
Can you manually load the pyfuse3 package? Like so:

> [root@ipfire ~]# python3
> Python 3.10.1 (main, Mar 25 2022, 13:22:24) [GCC 11.1.0] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import pyfuse3
Comment 10 Adolf Belka 2022-07-25 16:35:37 UTC
(In reply to Michael Tremer from comment #9)
> Can you manually load the pyfuse3 package? Like so:
> 
> > [root@ipfire ~]# python3
> > Python 3.10.1 (main, Mar 25 2022, 13:22:24) [GCC 11.1.0] on linux
> > Type "help", "copyright", "credits" or "license" for more information.
> > >>> import pyfuse3

I have learnt something new today.

Tried that command and got the following error message

[root@ipfire ahb]# python3
Python 3.10.1 (main, Apr 26 2022, 11:21:49) [GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyfuse3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: /usr/lib/python3.10/site-packages/pyfuse3.cpython-310-x86_64-linux-gnu.so: undefined symbol: _PyGen_Send

I found that error message in the github issues for pyfuse3
https://github.com/libfuse/pyfuse3/issues/52

Unfortunately the issue is still open and the only hope given is that a new release of pyfuse3 is due to fix some issues with python3.11 and the hope is that this _PyGen_Send issue might be fixe3d by that. Can't see any actual basis for that hope other than wishful thinking but will have to wait and see.
Comment 11 Adolf Belka 2022-07-25 16:48:55 UTC
Realised that the github issue on pyfuse3 is actually closed because they said it was related to a docker issue and nothing to do with pyfuse3
Comment 12 Michael Tremer 2022-07-25 16:51:57 UTC
Wait, pyfuse3 is orphaned, too?

> https://github.com/libfuse/pyfuse3/commit/377726d55479bb692501cabefaa17470855a8c11

What the hell is happening out there? Looks like borgbackup needs to look for something else again?!
Comment 13 Adolf Belka 2022-07-25 17:04:04 UTC
(In reply to Michael Tremer from comment #12)
> Wait, pyfuse3 is orphaned, too?
> 
> > https://github.com/libfuse/pyfuse3/commit/377726d55479bb692501cabefaa17470855a8c11
> 
> What the hell is happening out there? Looks like borgbackup needs to look
> for something else again?!

Oh wow. That is not good news. I was thinking about raising an issue on this topic but that is clearly pointless now.
Comment 14 Michael Tremer 2022-07-25 17:21:55 UTC
(In reply to Adolf Belka from comment #13)
> Oh wow. That is not good news. I was thinking about raising an issue on this
> topic but that is clearly pointless now.

There is a ticket: https://github.com/borgbackup/borg/issues/6734

Doesn't seem to have a solution in sight.

So we can now either:

* Package pyfuse3 and hope for the best, or
* Do nothing and wait until this is resolved upstream
Comment 15 Adolf Belka 2022-07-25 17:28:11 UTC
On borgbackup there is an open issue asking for volunteers to take up pyfuse3 maintenance.

https://github.com/borgbackup/borg/issues/6734

That was raised back on May 31st 2022. No responses since then.


I have found a comment from another application, Fiona, that had the same _PyGen_Send failure that it is because python3.10 replaced this with _PyIter_Send

This other application (Fiona) have apparently found that if you are using cython version 0.29.22 or later then the error message is gone.
Comment 16 Adolf Belka 2022-07-25 17:44:43 UTC
_PyGen_Send is in pyfuse3.c three times.

I have no idea if _PyGen_Send can just be replaced with _PyIter_Send or not. Is that worth trying in a build?
Comment 17 Michael Tremer 2022-07-25 18:15:24 UTC
(In reply to Adolf Belka from comment #16)
> I have no idea if _PyGen_Send can just be replaced with _PyIter_Send or not.
> Is that worth trying in a build?

Yes, that should work.
Comment 18 Adolf Belka 2022-07-26 11:52:47 UTC
Created attachment 1073 [details]
Build log from pyfuse3 using PyIter_Send in place of _PyGen_Send

In my earlier comments I referred to _PyIter_Send as the new function in Python3.10 replacing _PyGen_Send. The new function in fact has no leading underscore so the new function is called PyIter_Send

I did a build with the three occurrences of _PyGen_Send replaced by PyIter_Send and the build failed.
The attached log shows the error messages.

Looking up PyIter_Send it shows the use of three arguments for the new function while the usage of the old _PyGen_Send only used two arguments.

So it looks like PyIter_Send is not a drop-in replacement of _PyGen_Send unfortunately.

Not sure if anyone can tell from the log if it just needs a simple change or if it requires a more significant rewrite of pyfuse3
Comment 19 Michael Tremer 2022-07-26 13:24:04 UTC
(In reply to Adolf Belka from comment #18)
> So it looks like PyIter_Send is not a drop-in replacement of _PyGen_Send
> unfortunately.

No, clearly not. Sorry.

> 
> Not sure if anyone can tell from the log if it just needs a simple change or
> if it requires a more significant rewrite of pyfuse3

Not necessarily of pyfuse3, but of Cython. Cython is a code generator for Python which is my src/pyfuse3.c is such a long mess.

I am starting to get really frustrated here, since all those Python bindings for fuse are clearly neglected. And I can understand that nobody wants to maintain something that has rather been put together than cleanly designed and where good Python bindings have been written in native C.

So here we are again at the end of the road of the low hanging fruits:

* We don't have Cython in IPFire and I am not sure whether it is a good idea to package it. It might be a lot of work - I didn't check any dependencies. We could then however re-generate pyfuse3.c and that should work (https://github.com/cython/cython/pull/3921)
* An alternative could be going back to llfuse which builds for IPFire. If pyfuse3 and llfuse are outdated, I don't mind which one we have. Ideally we should't have either, but that doesn't seem to be an option. This is the path that Fedora took for example (they don't package pyfuse3, but they package llfuse).

Which one does sound better to you?
Comment 20 Adolf Belka 2022-07-26 14:01:53 UTC
(In reply to Michael Tremer from comment #19)
> * We don't have Cython in IPFire and I am not sure whether it is a good idea
> to package it. It might be a lot of work - I didn't check any dependencies.
> We could then however re-generate pyfuse3.c and that should work
> (https://github.com/cython/cython/pull/3921)
> * An alternative could be going back to llfuse which builds for IPFire. If
> pyfuse3 and llfuse are outdated, I don't mind which one we have. Ideally we
> should't have either, but that doesn't seem to be an option. This is the
> path that Fedora took for example (they don't package pyfuse3, but they
> package llfuse).
> 
> Which one does sound better to you?

To be honest I think staying with llfuse.

You mention that Fedora have stayed with that, also Arch Linux, who always use the latest packages available normally.

Reading the pyfuse3 readme, the one before the orphaned message, it has the following statement

"pyfuse3 is in beta. Bugs are likely."

At least llfuse has been running for some time and should be a bit more stable against bugs.


I will do a build with llfuse and confirm that it works and hopefully then submit a patch.
Comment 21 Adolf Belka 2022-07-26 17:57:23 UTC
So llfuse is also out of the question.

First build came up with the error that it had not been able to find the fuse.pc pkg-config file for fuse.

I patched the llfuse setup.py file to look for fuse3 instead of fuse.

That then allowed it to find fuse but then the build failed.

Error message is that "this version of the fuse library is not yet supported. Looking in the llfuse.h file it only allows the fuse major version to be 2.


I will now go back to your other option regarding cython.

To clarify, am I right that I need to create a build system that includes cython which would be used to recreate pyfuse3.c which would then be patched into the pyfuse3 source code to build without cython being needed to be present for the build.

If this is correct then I just need to figure out how to make cython recreate pyfuse3.c

If my understanding is not correct then I need some help to figure out how to approach this.
Comment 22 Michael Tremer 2022-07-27 07:15:45 UTC
(In reply to Adolf Belka from comment #21)
> So llfuse is also out of the question.
> 
> First build came up with the error that it had not been able to find the
> fuse.pc pkg-config file for fuse.
> 
> I patched the llfuse setup.py file to look for fuse3 instead of fuse.
> 
> That then allowed it to find fuse but then the build failed.
> 
> Error message is that "this version of the fuse library is not yet
> supported. Looking in the llfuse.h file it only allows the fuse major
> version to be 2.

Great. I wish we could something better here than just try and error.

> I will now go back to your other option regarding cython.

Okay. Let's see how far we can get here :)

> To clarify, am I right that I need to create a build system that includes
> cython which would be used to recreate pyfuse3.c which would then be patched
> into the pyfuse3 source code to build without cython being needed to be
> present for the build.

You don't need to patch anything - just regenerate the C file as shown below.

> If this is correct then I just need to figure out how to make cython
> recreate pyfuse3.c

You run this command in the pyfuse3 source directory:

> python3 setup.py build_cython

And then the usual:

> python3 setup.py build

And so on...
Comment 23 Adolf Belka 2022-07-28 08:02:04 UTC
I installed python3-Cython and made the change to pyfuse3 that you mentioned. After a successful build I ran ./make shell and ran the import test.

There was no longer any problem with an undefined _PyGen_Send

However it flagged up that python3-trio required the attr module. So I installed and built that and re-did the import test and it flagged up that the async_generator module was required.

Built that and then it flagged the attrs module, built that and it then flagged the sniffio module.

All the above modules are required in imports from the trio module.

That is my next step but it will have to wait for a while as I have had a problem with the motherboard on my desktop and I have to replace it and a couple of crashes from that problem looks to have affected the OS on my disks. The root directory is failing to complete an fsck.

My desktop system is where I do my build, so until it is fixed and running again I can't do anything further on this.

Hopefully there are not many more modules required after sniffio. Unfortunately, looking through various files in the trio source code, I have not been able to find anything that highlights a list of what is required. Each of the module imports are from different files in the trio source so it is not a simple case of looking in one file for all the required imports.

Will get back to this as soon as possible.
Comment 24 Michael Tremer 2022-07-28 19:37:20 UTC
(In reply to Adolf Belka from comment #23)
> I installed python3-Cython and made the change to pyfuse3 that you
> mentioned. After a successful build I ran ./make shell and ran the import
> test.
> 
> There was no longer any problem with an undefined _PyGen_Send
> 
> However it flagged up that python3-trio required the attr module. So I
> installed and built that and re-did the import test and it flagged up that
> the async_generator module was required.
> 
> Built that and then it flagged the attrs module, built that and it then
> flagged the sniffio module.
> 
> All the above modules are required in imports from the trio module.

Thanks for working on this.

> That is my next step but it will have to wait for a while as I have had a
> problem with the motherboard on my desktop and I have to replace it and a
> couple of crashes from that problem looks to have affected the OS on my
> disks. The root directory is failing to complete an fsck.

I am starting to believe this ticket is cursed...
Comment 25 Adolf Belka 2022-07-29 11:53:37 UTC
My desktop system was fixed easier than I had anticipated.

After sniffio I had to also add the modules sortedcontainers and outcome.

Then pyfuse3 was successfully imported on my build system using the shell.

Then did a full new build and installed all the extra modules into a vm system. Importing the pyfuse3 module failed again. It also requires the idna module. This is already installed in IPFire but as a package for some other addon. I added it to the dependency tree for python3-trio and pyfuse3 imported successfully.

I then  did a borg mount repo mount-point command on the vm and it successfully worked. I was able to look through all the files I had backed up into my test repo.

Hooray.

I will now put together a patch submission for this.

Previously borgbackup had a total of three other python3 dependencies. Now it will have a total of eleven python3 dependencies!! This doesn't include python3-Cython as that is only required for the build and so the rootfile is in the common section but has all the entries commented out so nothing ends up in the IPFire package.
Comment 26 Adolf Belka 2022-07-29 12:45:40 UTC
Patch set submitted to dev mailing list and patchwork.

https://patchwork.ipfire.org/project/ipfire/list/?series=2981
Comment 27 Michael Tremer 2022-08-01 09:26:56 UTC
(In reply to Adolf Belka from comment #26)
> Patch set submitted to dev mailing list and patchwork.
> 
> https://patchwork.ipfire.org/project/ipfire/list/?series=2981

Great work. Do we always want to ship all the egg-info stuff? I never include that.
Comment 28 Adolf Belka 2022-08-01 09:48:34 UTC
(In reply to Michael Tremer from comment #27)
> 
> Great work. Do we always want to ship all the egg-info stuff? I never
> include that.

The reason I did that is because in the previous borgbackup bug I fixed a couple of CU's ago, I had to uncomment the egg-info lines in the borgbackup rootfile otherwise python failed to find the borgbackup metadata and failed to run.

Searching about the failure of not finding the metadata I found some info about some of the files from the egg-info being needed. When I uncommented those egg-info lines the failed to find metadata error stopped ocurring when trying to start borgbackup.

I had therefore presumed these were needed generally. Maybe it was something special with regard to Borgbackup.

I will do another build run, removing all the egg-info lines from the rootfiles and confirm that the fuse mount command still works successfully. and if yes then I will submit a v2 patch version.
Comment 29 Adolf Belka 2022-08-02 09:30:11 UTC
Rebuilt the patch series without all the egg-info files in the rootfiles. Tested in testbed vm and borg mount command worked as expected.

v2 patch submitted to dev mailing list and patchwork

https://patchwork.ipfire.org/project/ipfire/list/?series=2996
Comment 30 Adolf Belka 2022-08-05 11:42:23 UTC
Patch has been committed to CU170

https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=c02bffe2be9db47c698576e9fdc381559394fb6d
Comment 31 Adolf Belka 2022-08-06 21:46:16 UTC
Updated v3 version of the patch set submitted to deal with the armv6l rootfile having a non-standard architecture naming for the library file created.

https://patchwork.ipfire.org/project/ipfire/list/?series=3011
Comment 32 Adolf Belka 2022-08-08 10:29:07 UTC
v3 patch set has been merged

https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=30ea66cf4b1161d682b489e29096111ddc120c4f