Bug 11098

Summary: Nfs doesn't work in core 100
Product: IPFire Reporter: lars.seidler
Component: ---Assignee: Jonatan Schlag <jonatan.schlag>
Status: CLOSED FIXED QA Contact: Arne.F <arne.fitzenreiter>
Severity: - Unknown -    
Priority: - Unknown - CC: daniel, ebuemir, jonatan.schlag, michael.tremer
Version: 2   
Hardware: unspecified   
OS: Unspecified   
Attachments: Patch v1
Patch v2 (fixes Bug #11098)

Description lars.seidler 2016-04-17 17:23:47 UTC
Hello, 
since Upgrade to Core 100 nfs-server isn't working anymore.

If I try to start it with 

/etc/init.d/nfs-server restart

 the following error occurs: 
[root@ipfire init.d]# ./nfs-server restart Stopping NFS statd... [ OK ] Stopping NFS nfsd... [ FAIL ] Stopping NFS mountd... [ OK ] Refreshing NFS Exported Filesystems... [ OK ] Unmounting NFS Virtual Filesystem... [ OK ] Removing the rpc.statd pid file if it exists Starting NFS mountd... [ OK ] Starting NFS nfsd... [ OK ] Starting NFS statd... [ OK ] Mounting nfsd virtual filesystem... mount: none is already mounted or /proc/fs/nfsd busy none is already mounted on /var/log/rrd [ FAIL ] 

On the target machine, the following error occurs:

 mount.nfs: access denied by server while mounting 192.168.2.1:/mnt/harddisk

 The logfile at the ipFire contains:

 12:56:38 kernel: nfsd: last server has exited, flushing export cache 12:57:53 kernel: svc: failed to register lockdv1 RPC service (errno 97). 12:57:53 kernel: NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transit ion to using nfsdcltrack. 12:57:53 kernel: NFSD: starting 90-second grace period (net c0d99640) 12:59:53 kernel: NFSD: Unable to end grace period: -110 13:01:38 kernel: nfsd: last server has exited, flushing export cache 13:01:41 kernel: svc: failed to register lockdv1 RPC service (errno 97). 13:01:41 kernel: NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transit ion to using nfsdcltrack. 13:01:41 kernel: NFSD: starting 90-second grace period (net c0d99640) 13:03:41 kernel: NFSD: Unable to end grace period: -110
Comment 1 lars.seidler 2016-04-17 17:25:54 UTC
Btw: I hope, the english description will be accepted. The doesn't go netter in it's described in english instead of german. 
I thought, ipfire is a german project. ......
Comment 2 dini 2016-04-17 19:43:47 UTC
Hello,
 I have the same problem since upgrade from 99 to 100.

When i restart the nfs server
i got this on /var/log/messages

Apr 17 17:39:49 ipfire dnsmasq[3072]: reading /var/state/dhcp/dhcpd.leases
Apr 17 17:40:50 ipfire kernel: nfsd: last server has exited, flushing export cache
Apr 17 17:40:51 ipfire rpc.mountd[26960]: Caught signal 15, un-registering and exiting.
Apr 17 17:40:53 ipfire rpc.mountd[25100]: Kernel does not have pseudo root support.
Apr 17 17:40:53 ipfire rpc.mountd[25100]: NFS v4 mounts will be disabled unless fsid=0
Apr 17 17:40:53 ipfire rpc.mountd[25100]: is specfied in /etc/exports file.
Apr 17 17:40:53 ipfire rpc.mountd[25101]: Version 1.2.7 starting
Apr 17 17:40:53 ipfire kernel: svc: failed to register lockdv1 RPC service (errno 97).
Apr 17 17:40:53 ipfire kernel: NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
Apr 17 17:40:53 ipfire kernel: NFSD: starting 90-second grace period (net c0d99640)
Apr 17 17:40:53 ipfire rpc.statd[25127]: Version 1.2.7 starting
Apr 17 17:40:53 ipfire sm-notify[25128]: Version 1.2.7 starting
Apr 17 17:40:53 ipfire sm-notify[25128]: Already notifying clients; Exiting!
Apr 17 17:40:53 ipfire rpc.statd[25127]: Running as root.  chown /var/lib/nfs to choose different user
Comment 3 jakumoto 2016-04-20 19:49:24 UTC
maybe it is the same issue as 3 years ago ...

http://forum.ipfire.org/viewtopic.php?t=7659
Comment 4 dini 2016-04-20 21:47:19 UTC
Maybe is a kernel problem there are something with kernel: grsec.



pr 20 21:44:53 ipfire kernel: svc: failed to register lockdv1 RPC service (errno 97).
Apr 20 21:44:53 ipfire kernel: NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
Apr 20 21:44:53 ipfire kernel: NFSD: starting 90-second grace period (net ffffffff80f59d00)
Apr 20 21:44:53 ipfire rpc.statd[8486]: Version 1.2.7 starting
Apr 20 21:44:53 ipfire sm-notify[8487]: Version 1.2.7 starting
Apr 20 21:44:53 ipfire sm-notify[8487]: Already notifying clients; Exiting!
Apr 20 21:44:53 ipfire rpc.statd[8486]: Running as root.  chown /var/lib/nfs to choose different user
Apr 20 21:45:01 ipfire kernel: rpc.mountd[8462]: segfault at 0 ip 00000000004094e4 sp 00007b4429042ab0 error 4 in rpc.mountd[400000+16000]
Apr 20 21:45:01 ipfire kernel: grsec: From 192.168.0.10: Segmentation fault occurred at            (nil) in /usr/sbin/rpc.mountd[rpc.mountd:8462] uid/euid:0/0 gid/egid:0/0, parent /sbin/init[init:1] uid/euid:0/0 gid/egid:0/0
Apr 20 21:45:01 ipfire kernel: grsec: From 192.168.0.10: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /usr/sbin/rpc.mountd[rpc.mountd:8462] uid/euid:0/0 gid/egid:0/0, parent /sbin/init[init:1] uid/euid:0/0 gid/egid:0/0
Comment 5 Jonatan Schlag 2016-04-21 09:43:52 UTC
Hi,
thanks to all which posted logs and similar things. This helps a lot!

@ jakumoto
Maybe it is the same issue maybe not, but thanks for the hint.

@dini
Thanks a lot for the kernel log.
grsec is not he problem this is the problem:

Apr 20 21:45:01 ipfire kernel: rpc.mountd[8462]: segfault at 0 ip 00000000004094e4 sp 00007b4429042ab0 error 4 in rpc.mountd[400000+16000]

A segfault occur and rpc.mountd stops.

@all
I will try to provide an update later this week for this components. I hope this solve issues.

Regards Jonatan
Comment 6 dini 2016-04-21 15:23:27 UTC
Ok thx Jonatan,

I tried a fresh install Core100  64 Bit.
But the error is the same.

I wait then for the update.

Thx
Comment 7 Jonatan Schlag 2016-04-23 11:48:19 UTC
hi, I am happy to announce that I get NFS working on an IPFire Core 100 x86_64.
I will provide packages and an exact documentation later this day.

Regards Jonatan
Comment 8 lars.seidler 2016-04-23 12:57:58 UTC
Hi Jonatan,

sound pretty!

Does the patch only work on x64 or on x86 also?

Lars
Comment 9 Jonatan Schlag 2016-04-23 14:10:27 UTC
Hi Lars,
the patch should work also on IPFire i586/i686.

Regards Jonatan
Comment 10 Jonatan Schlag 2016-04-23 19:15:12 UTC
Hi,
I uploaded packages for x86_64 to
http://people.ipfire.org/~jschlag/11098/

Installation:
1. Uninstall portmap and nfs
pakfire remove portmap nfs

2.downlaod and copy the packages to /opt/pakfire/tmp
3. For every of the 3 packages do:

tar xvf <package>.ipfire
./install

So for rpcbind it would be

tar xvf rpcbind-0.2.3-3.ipfire
./install

4. Reboot your system
5. NFS should work

I post my example configuration if NFS do not work test this to make sure that no configuration issue exists.

### Example Configuration
1. Create the directory
mkdir -p /var/nfsshare/test
2. This is my etc/exports (Please us for testing only this two lines, comment all others)
################################################################################

/var/nfsshare *(rw,sync,no_subtree_check,fsid=0,crossmnt)
/var/nfsshare/test *(rw,sync,no_subtree_check)

################################################################################

3. reload the NFS server

4. You should be able to mount the share with
mkdir -p /var/test
mount -v -t nfs4  <ip of your ipfire>:/test /var/test

Happy testing :-)

Regards Jonatan
Comment 11 lars.seidler 2016-04-23 19:36:04 UTC
Can I use these files on x86 too? Or are there separate ones for x86?
Comment 12 Jonatan Schlag 2016-04-23 19:53:11 UTC
Hi Lars
I will upload files for x86 tomorrow at 12.00 am.
The build takes 3 -4 hours. So it is not sense full to start the build now :-(, because at 12.00 pm I sleep and so I can not upload the files.

Regards Jonatan
Comment 13 Jonatan Schlag 2016-04-24 12:47:24 UTC
Hi,
I uploaded the packages for x86 or i586/i686 to http://people.ipfire.org/~jschlag/11098/i586/

Happy testing :-).

Regards Jonatan
Comment 14 lars.seidler 2016-04-24 14:04:35 UTC
Jonatan, 
you are great. The nfs works again.

Thank you very much!
Comment 15 Arne.F 2016-04-24 21:40:52 UTC
*** Bug 11097 has been marked as a duplicate of this bug. ***
Comment 16 Arne.F 2016-04-24 21:47:58 UTC
I have tested a bit at weekend. There are two bugs.
One is introduced by 1.2.7-6 and published to testing since 21.12.2015 and no one has reported it. This is why now some people also report that IPFire 2.17 core99 is also affected. (this is a bug in the initskript, the mounting must done before nfsd.)

The second is the segfault of mountd. This only occoured with the new gcc compiler and can easy fixed by update to nfsutils-1.3.3

@Jonatan, why you build libtirpc and exchange rpcbind?
I have simply add --disable-ipv6 and got a working version. Have your solution any benefit because IPFire-2.x doesn't support IPv6.
Comment 17 dini 2016-04-24 22:55:11 UTC
Hi

I have now tested and its ok, now i can mount nfs shares again.

Thx
Comment 18 Jonatan Schlag 2016-04-25 12:59:48 UTC
Hi Arne,

#I have tested a bit at weekend. There are two bugs.
#One is introduced by 1.2.7-6 and published to testing since 21.12.2015 and no #one has reported it. This is why now some people also report that IPFire 2.17 #core99 is also affected. (this is a bug in the initskript, the mounting must #done before nfsd.)

So we both did the same work? You are right there are 2 bugs (I have found a third one but the bug is not so important). I fixed the bug in the initsrcipt and the other bug in the init script (nfsd is not stopped).

#The second is the segfault of mountd. This only occoured with the new gcc #compiler and can easy fixed by update to nfsutils-1.3.3

I fixed also this :-). I attached a patch for this bug, could you review this patch? 

#@Jonatan, why you build libtirpc and exchange rpcbind?
#I have simply add --disable-ipv6 and got a working version. Have your solution #any benefit because IPFire-2.x doesn't support IPv6.

I replaced rpcbind, because I did not really know, what would be the bug, and portmap do not ship the rpcinfo binary, which is very useful for debugging. I just do not remove the change for the packages which I uploaded. 
Otherwise i would suggest replacing portmap by rpcbind because portmap is not maintained any more and do not ship the rpcinfo binary. 
Libtirpc is a dependency for rpcbind. But this change does not concern the bug. So I will do this in a separate patch. :-)

I hope I answer all questions, when not just write back.

Regards Jonatan
Comment 19 Jonatan Schlag 2016-04-25 13:00:57 UTC
Created attachment 436 [details]
Patch v1
Comment 20 Arne.F 2016-04-25 14:34:24 UTC
#So we both did the same work? 
Yes. I have done this in preparation of core101 because there was also a recent kernel change (enable NFS_DEBUG http://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=6fa34327dd38412820e9654df69d48995b02906a ), so i had to check if this maybee the reason.

Please check the rootfile, you should only insert /usr/lib/nfs or subfiles/folders but not both because this was packed twice in the tar.
Comment 21 Jonatan Schlag 2016-04-25 15:24:17 UTC
Hi Arne,
#Please check the rootfile, you should only insert /usr/lib/nfs or #subfiles/folders but not both because this was packed twice in the tar

I checked this and fixed the rootfile, The patch (I attached a second version) should be fine now. Should I send the patch to the mailing list or ...?

Regards Jonatan
Comment 22 Jonatan Schlag 2016-04-25 15:24:54 UTC
Created attachment 437 [details]
Patch v2 (fixes Bug #11098)
Comment 23 Arne.F 2016-04-25 16:35:55 UTC
Yes. Put it to the list.
Comment 24 Jonatan Schlag 2016-04-25 17:11:24 UTC
Hi Arne,
I posted the patch on the list. 

Regards Jonatan
Comment 25 Jonatan Schlag 2016-04-27 15:22:23 UTC
Hi, the fix is in the git tree 

http://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=54cc2c4e5d5abbc1972e9d996117f95d6522ade1

and will hopefully released with Core 102.

Regards Jonatan
Comment 26 Arne.F 2016-04-28 17:12:54 UTC
nfs is an idenpended package it will not released with a core update...

Please test the package from the nightly build. If it work i push it to testing tree and after more feedback to stable.