Can't remove entries in 'extrahd' via the webinterface. Does not check entries for the right syntax, existing drive ID, existing mount point. https://community.ipfire.org/t/extrahd-bug-cant-remove-mount-point/7920
Created attachment 1051 [details] appendix obsolete
I noticed that extrahd.pl is using /etc/fstab file but /etc/fstab is empty Below are example lines: Line 36: my $fstab = "/var/ipfire/extrahd/fstab"; Line 52: system("/bin/cp -f /etc/fstab $fstab"); Line 58: print "Insert $deviceline[0] ($deviceline[1]) --> $deviceline[2] into /etc/fstab!\n"; Line 60: open(FILE, ">>$fstab"); Line 66: system("/bin/cp -f $fstab /etc/fstab"); Line 76: system("/bin/cp -f /etc/fstab $fstab"); Line 77: system("/bin/fgrep -v $ARGV[1] <$fstab >/etc/fstab");
Please delete my comments - entered too early.
I just ran into the same issue. I removed a bad external drive and replaced with a new one but I cannot remove the old drive from ExtraHD. Are the permissions correct? (nobody vs root) ``` [root@ipfire ~] # ls -alR /var/ipfire/extrahd /var/ipfire/extrahd: total 28 drwxr-xr-x 3 nobody nobody 4096 Jan 29 15:40 . drwxr-xr-x 51 root root 4096 Jan 12 21:11 .. drwxr-xr-x 2 root root 4096 Feb 7 2022 bin -rw-r--r-- 1 nobody nobody 57 Jun 24 2022 devices -rw-r--r-- 1 root root 282 Jan 29 15:40 fstab -rw-r--r-- 1 nobody nobody 475 Jan 29 15:40 partitions -rw-r--r-- 1 root root 70 Jan 29 15:40 scan -rw-r--r-- 1 nobody nobody 0 Jun 10 2017 settings /var/ipfire/extrahd/bin: total 12 drwxr-xr-x 2 root root 4096 Feb 7 2022 . drwxr-xr-x 3 nobody nobody 4096 Jan 29 15:40 .. -rwxr-xr-x 1 root root 3808 Feb 7 2022 extrahd.pl [root@ipfire ~] # ```
Some investigation showed, that there is a problem with return/exit values in case of unmount operation. The state is left inconsistent. The device is unmounted, but the files in /var/ipfire/extrahd don't reflect this.
Error found. extrahd.cgi misinterprets the return value of extrahdctrl. 0=success is handled as error. Patch follows.
bump... Is there an update for this?
To me the problem seems to be in the /usr/local/bin/extrahdctrl C program. It doesn’t seem to be returning any error codes. I did this to prove my theory I made a new extrahd.pl with only an error code. ``` [root@ipfireAPU ~] # cat /var/ipfire/extrahd/bin/extrahd.pl #!/usr/bin/perl exit(5); ``` but no matter what extrahdctrl always returns a “zero”. extrahdctrl does not pass the perl error. Michael - can you assist? See: https://community.ipfire.org/t/extrahd-bug-cant-remove-mount-point/7920/31?u=jon
Could I please have the output of: > strace -ff /usr/local/bin/extrahdctrl umount /mnt/harddisk That should help. Generally the binary should return the exit code.
Created attachment 1151 [details] strace output here ya go!
I did another test with extrahd.pl ending with exit 5. Looks like extrahd.pl ends "+++ exited with 5 +++", but extrahdctrl "+++ exited with 0 +++". ``` [root@ipfireAPU ~] # cat /var/ipfire/extrahd/bin/extrahd.pl #!/usr/bin/perl exit(5); [root@ipfireAPU ~] # /var/ipfire/extrahd/bin/extrahd.pl [root@ipfireAPU ~] # echo $? 5 [root@ipfireAPU ~] # strace -ff /usr/local/bin/extrahdctrl umount /mnt/harddisk . . . [pid 12481] rt_sigaction(SIGIO, NULL, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=0}, 8) = 0 [pid 12481] exit_group(5) = ? [pid 12481] +++ exited with 5 +++ <... wait4 resumed>[{WIFEXITED(s) && WEXITSTATUS(s) == 5}], 0, NULL) = 12481 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=12481, si_uid=0, si_status=5, si_utime=0, si_stime=1 /* 0.01 s */} --- exit_group(1280) = ? +++ exited with 0 +++ [root@ipfireAPU ~] # ```
Created attachment 1152 [details] strace output #2 (with perl exit 5)
Created attachment 1158 [details] Proposed patch Hey Jon, I did work on this a couple of weeks ago but completely forgot to follow up on this. So, here is my proposed patch. I will upload a compiled binary for you to test.
Created attachment 1159 [details] extrahdctrl compiled for x86
Hmmm.. it does not work. I downloaded it and changed the download name from `=?UTF-8?Q?extrahdctrl?=` to extrahdctrl. And changed the permissions with `chmod -v 4750 /usr/local/bin/extrahdctrl` And I tested it with my mini perl script: ``` [root@ipfireAPU ~] # cat /var/ipfire/extrahd/bin/extrahd.pl #!/usr/bin/perl exit(5); ``` so far so good! But when I use it with the extrahd.cgi WebGUI page it fails. Adding a new drive causes an error and the first line of text appears in red (instead of green). If I click the trash can on the first line, boom, everything mounts. And the first line disappears. I'll experiment and get some better answers. But right now things seem backwards with the extrahd.pl script or maybe the extrahd.cgi webgui page...
from the top: No drives mounted (except for the normal ipfire drive/partitions). I started with one un-mounted external drive. $ blkid -o list -c /dev/null device fs_type label mount point UUID ----------------------------------------------------------------------------------------------------------------------- /dev/sdb1 ext4 ipfire_bu (not mounted) 4f1513203f-4c42-461b-bd9c-f19324495c618 /dev/sda4 ext4 / 25a5f3aa-5183-11ed-a424-eb259c83c43e5 /dev/sda2 vfat /boot/efi 7FCE-A1B7 /dev/sda3 swap [SWAP] 23b3645154-5183-11ed-ae61-7bfd22d67b195 /dev/sda1 ext4 /boot 2133435fce-5183-11ed-9ab2-0b87d15f415ae From the extrahd.cgi WebGUI page, clicked "add" (the pencil +) to add an external drive. The next line appears in RED near the top of the extrahd.cgi WebGUI page: UUID=4f1513203f-4c42-461b-bd9c-f19324495c618 auto /mnt/harddisk Nothing new in `mount`. Nothing new in `df -h` and nothing new in `blkid -o list -c /dev/null`. No changes to `/etc/fstab` or `/var/ipfire/extrahd/fstab`. Devices at `/var/ipfire/extrahd/devices` did get updated. ``` $ cat /var/ipfire/extrahd/devices UUID=4f1513203f-4c42-461b-bd9c-f19324495c618;auto;/mnt/harddisk; ``` === still testing...
ugh! It looks like you changed `setuid` and I changed extrahdctrl. No wonder nothing worked! I'll be back!
Alright - I have confused myself. Let me know what the binary file represents. As extrahdctrl it does not work. It almost seems like it is not running / launching the `extrahd.pl` script.
(In reply to Jon from comment #18) > Alright - I have confused myself. Let me know what the binary file > represents. It should go to /usr/local/bin/extrahdctrl. You might need to restore the special setuid bit again with: chown root.nobody /usr/local/bin/extrahdctrl chmod 4750 /usr/local/bin/extrahdctrl > As extrahdctrl it does not work. It almost seems like it is not running / > launching the `extrahd.pl` script. In my test it worked. Please send the strace log again when you run this on the command line.
`chown root.nobody /usr/local/bin/extrahdctrl` was a problem (fixed). When the first line turn green does the drive actually mount? I have nothing mounted: ``` $ df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 1.9G 4.0K 1.9G 1% /dev tmpfs 2.0G 0 2.0G 0% /dev/shm tmpfs 2.0G 596K 2.0G 1% /run /dev/sda4 54G 2.6G 49G 6% / /dev/sda1 236M 52M 167M 24% /boot /dev/sda2 32M 270K 32M 1% /boot/efi /var/lock 8.0M 12K 8.0M 1% /var/lock $ mount | grep ^/dev /dev/sda4 on / type ext4 (rw,relatime) /dev/sda1 on /boot type ext4 (rw,nosuid,nodev,noexec,relatime) /dev/sda2 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro) ```
Created attachment 1164 [details] strace results #3 I started over again with a new-ish build (just built about 1 week ago). I am running: IPFire 2.27 (x86_64) - Core-Update 174 on the new build I copied file above (from you) to /usr/local/bin/extrahdctrl. And did the: ``` chown root.nobody /usr/local/bin/extrahdctrl chmod 4750 /usr/local/bin/extrahdctrl ``` It seems like the OP issue (Comment #c0) is still there. There is still an issue unmounting the drive. On the extrahd.cgi WebGUI page I see this: > Error messages > Can't umount /mnt/harddisk1. Maybe the device is in use? > > UUID=b17fc38c-1d4f-43b9-a807-258025f98d6d auto /mnt/harddisk1 And this in `devices`: ``` $ cat /var/ipfire/extrahd/devices UUID=b17fc38c-1d4f-43b9-a807-258025f98d6d;auto;/mnt/harddisk1; ``` Here is the last few lines of strace: ``` [pid 11531] rt_sigaction(SIGIO, NULL, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=0}, 8) = 0 [pid 11531] exit_group(0) = ? [pid 11531] +++ exited with 0 +++ <... wait4 resumed>[{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 11531 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=11531, si_uid=0, si_status=0, si_utime=56 /* 0.56 s */, si_stime=49 /* 0.49 s */} --- exit_group(0) = ? +++ exited with 0 +++ [root@ipfireAPU2 ~]# ``` Strace results are attached
Output of extrahd.pl: See the output of "mount". The exit code is "1" ``` [root@ipfireAPU2 ~]# [root@ipfireAPU2 ~]# /var/ipfire/extrahd/bin/extrahd.pl umount /mnt/harddisk1 \ echo $? fgrep: warning: fgrep is obsolescent; using grep -F fgrep: warning: fgrep is obsolescent; using grep -F Successfully umounted /mnt/harddisk1. [root@ipfireAPU2 ~]# /var/ipfire/extrahd/bin/extrahd.pl umount /mnt/harddisk1 umount: /mnt/harddisk1: not mounted. fgrep: warning: fgrep is obsolescent; using grep -F fgrep: warning: fgrep is obsolescent; using grep -F Successfully umounted /mnt/harddisk1. [root@ipfireAPU2 ~]# echo $? 0 [root@ipfireAPU2 ~]# [root@ipfireAPU2 ~]# /var/ipfire/extrahd/bin/extrahd.pl mount /mnt/harddisk1 Insert UUID=b17fc38c-1d4f-43b9-a807-258025f98d6d (auto) --> /mnt/harddisk1 into /etc/fstab! [root@ipfireAPU2 ~]# [root@ipfireAPU2 ~]# echo $? 1 [root@ipfireAPU2 ~]# [root@ipfireAPU2 ~]# /var/ipfire/extrahd/bin/extrahd.pl umount /mnt/harddisk1 fgrep: warning: fgrep is obsolescent; using grep -F fgrep: warning: fgrep is obsolescent; using grep -F Successfully umounted /mnt/harddisk1. [root@ipfireAPU2 ~]# echo $? 0 [root@ipfireAPU2 ~]# ```
Output of the new extrahdctrl: See the output of "mount". The exit code is "1" [root@ipfireAPU2 ~]# /usr/local/bin/extrahdctrl umount /mnt/harddisk1 umount: /mnt/harddisk1: not mounted. fgrep: warning: fgrep is obsolescent; using grep -F fgrep: warning: fgrep is obsolescent; using grep -F Successfully umounted /mnt/harddisk1. [root@ipfireAPU2 ~]# echo $? 0 [root@ipfireAPU2 ~]# [root@ipfireAPU2 ~]# /usr/local/bin/extrahdctrl mount /mnt/harddisk1 Insert UUID=b17fc38c-1d4f-43b9-a807-258025f98d6d (auto) --> /mnt/harddisk1 into /etc/fstab! [root@ipfireAPU2 ~]# echo $? 1 [root@ipfireAPU2 ~]# [root@ipfireAPU2 ~]# /usr/local/bin/extrahdctrl umount /mnt/harddisk1 fgrep: warning: fgrep is obsolescent; using grep -F fgrep: warning: fgrep is obsolescent; using grep -F Successfully umounted /mnt/harddisk1. [root@ipfireAPU2 ~]# [root@ipfireAPU2 ~]# echo $? 0 [root@ipfireAPU2 ~]#
this *might* fix the extrahd.pl. See line 66. ``` system("/bin/cp -f $fstab /etc/fstab"); `/bin/mount -a`; exit $? >> 8; ```
(In reply to Jon from comment #21) > Here is the last few lines of strace: > ``` > [pid 11531] rt_sigaction(SIGIO, NULL, {sa_handler=SIG_IGN, sa_mask=[], > sa_flags=0}, 8) = 0 > [pid 11531] exit_group(0) = ? > [pid 11531] +++ exited with 0 +++ > <... wait4 resumed>[{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 11531 > --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=11531, si_uid=0, > si_status=0, si_utime=56 /* 0.56 s */, si_stime=49 /* 0.49 s */} --- > exit_group(0) = ? > +++ exited with 0 +++ > [root@ipfireAPU2 ~]# > ``` > > Strace results are attached This looks good to me though. The script returns 0 and therefore the helper binary returns 0, too. So now we need to have a look at that horrible perl script :) I would prefer to rework it so that it no longer modifies /etc/fstab and no longer calls "mount -a".
If I can re-write it as a bash script I'd line to take a shot. It will be like backup.pl (a bash script hidden as a perl).
(In reply to Michael Tremer from comment #25) > > I would prefer to rework it so that it no longer modifies /etc/fstab and no > longer calls "mount -a". If the /etc/fstab is not modified than how does the ExtraHD drive get mounted on reboot?
Just to go backwards for a moment... The changes Michael @ms made to extrahdctrl seem to fix the problems with extrahdctrl. So that one is OK to commit. There is a problem with extrahd.cgi (WebGUI) that Bernhard @bbitsch found and documented here ( adding the "!" ): https://community.ipfire.org/t/extrahd-bug-cant-remove-mount-point/7920/26?u=jon And so that one could be committed also. Making both of these changes allows the extrahd.cgi WebGUI to function as it should. --- Having said that, there are other issues in the extrahd.pl code (lots of little stuff like Michael mentioned in Comment 25). And a small issue with the extrahd.cgi where in certain circumstances you can have multiple drives: ``` $ cat /etc/fstab UUID=21335fce-5183-11ed-9ab2-0b87d5f415ae /boot auto defaults,nodev,noexec,nosuid 1 2 UUID=7FCE-A1B7 /boot/efi auto defaults 1 2 UUID=23b36154-5183-11ed-ae61-7bfd2d67b195 swap swap defaults,pri=1 0 0 UUID=25a5f3aa-5183-11ed-a424-eb259c8c43e5 / auto defaults 1 1 UUID=4f15203f-4c42-461b-bd9c-f1932495c618 /mnt/harddisk auto defaults 0 0 UUID=4f15203f-4c42-461b-bd9c-f1932495c618 /mnt/harddisk1 auto defaults 0 0 UUID=4f15203f-4c42-461b-bd9c-f1932495c618 /mnt/harddisk2 auto defaults 0 0 ``` This may be OK (I just don't know for sure) If IPFire3 is due to be released within the next year, it probably is not worth spending time on the extrahd.pl code. --- Should we release the extrahdctrl commit from Michael? And Bernhard (or I or Michael) can commit the missing "!" in extrahd.cgi ?
Michael - Gentle reminder... Please do the commit for extrahdctrl / setuid.c (if it is OK?).
(In reply to Jon from comment #29) > Michael - Gentle reminder... > > Please do the commit for extrahdctrl / setuid.c (if it is OK?). Thank you for the reminder. Posted to the list: > https://patchwork.ipfire.org/project/ipfire/patch/20230524090841.269580-1-michael.tremer@ipfire.org/
So, I rewrote the shell script: > https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=87718939ca89af170b71d152466f65e85852c4ea This is pretty much a draft that I tested on my system. It will no longer try to modify /etc/fstab (that is never a good idea). In order to mount everything when the system is booting up, we will need to add an initscript. But in the meantime, I would like to ask you to give this a little testing. The script is also trying harder to umount something.
Thank you for the commit! And yes, I will test the extrahd.pl (bash) script
Thank you for creating this! It is so much easier (for me!) to work with BASH then perl. (and this is all about me!) I started with the WebGUI using your extrahdctrl. Most works OK. The WebGUI mounts and un-mounts as it should. Bugs: If I enter "auto" as the mountpoint than extrahd.pl will create a directory (mkdir) at `/srv/web/ipfire/cgi-bin/auto`. If I enter "/auto" as the mountpoint than extrahd.pl will create a directory at `/auto`. I did not test this using `/bin` or `bin`. (or anything similar) Maybe remove the mkdir `--parents` options and place external drives in \mnt or \media only? FYI - I found this with the old extrahd.pl so I knew it existed. === I experimented with extrahd.pl as a stand alone script and NOT using the WebGUI. This command (with no mountpoint) succeeds but should fail. ``` [root@ipfireAPU ~] # /var/ipfire/extrahd/bin/extrahd.pl mount && echo $? 0 ``` If `_mountpoint` is empty than the `mount` command still mounts the drive from the devices file. Same with the un-mount. If `_mountpoint` is empty than the `umount` command still un-mounts the drive. === The only other suggestion I have (for today) is to remove the "Unknown command" echo statement and change tp: echo "Usage: $0 mount <mountpoint> | umount <mountpoint> | scanhd {ide|partitions}" Something similar is fine, I may have the wrong format for a Usage.
(In reply to Jon from comment #33) > Thank you for creating this! It is so much easier (for me!) to work with > BASH then perl. (and this is all about me!) Same for me. > Bugs: > If I enter "auto" as the mountpoint than extrahd.pl will create a directory > (mkdir) at `/srv/web/ipfire/cgi-bin/auto`. Oh this is bad. We should probably catch more invalid stuff in the UI. For example would it be possible to overlay any system mount point like /, /dev, /proc and so on. I am not sure if a blacklist would work well enough. A whitelist to only allow mounting below a certain directory like /media would be nice, but that might break existing installations. I added a check that requires the path to at least be absolute: > https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=3ad8138da571de74ac7bb89451d83130da7ba5a8 > If I enter "/auto" as the mountpoint than extrahd.pl will create a directory > at `/auto`. > > I did not test this using `/bin` or `bin`. (or anything similar) This would be bad and break the system until you (hard) reboot. > Maybe remove the mkdir `--parents` options and place external drives in \mnt > or \media only? Should we do this for new setups? I think so. That would have to be realised in the UI though. > FYI - I found this with the old extrahd.pl so I knew it existed. > > === > > I experimented with extrahd.pl as a stand alone script and NOT using the > WebGUI. > > This command (with no mountpoint) succeeds but should fail. This will now mount all configured mount points and is exactly the command that needs to be executed at boot time. I thought that would make it nice and easy without messing with /etc/fstab. > ``` > [root@ipfireAPU ~] # /var/ipfire/extrahd/bin/extrahd.pl mount && echo $? > 0 > ``` > > If `_mountpoint` is empty than the `mount` command still mounts the drive > from the devices file. > > Same with the un-mount. If `_mountpoint` is empty than the `umount` command > still un-mounts the drive. This is intentional and required to amount everything correctly when the system shuts down. > === > > The only other suggestion I have (for today) is to remove the "Unknown > command" echo statement and change tp: > > echo "Usage: $0 mount <mountpoint> | umount <mountpoint> | scanhd > {ide|partitions}" > > Something similar is fine, I may have the wrong format for a Usage. I was being lazy there and considered this something that should normally not be executed by the user - and so why need this? I should probably add this. I also added a check that skips the umount if nothing is mounted: > https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=957d4103b780d1578373e00c8e43ec6b490327f8
(In reply to Michael Tremer from comment #34) > A whitelist to only > allow mounting below a certain directory like /media would be nice, but that > might break existing installations. In my attempt to script I had include a check similar to: ``` if ! grep -Fqe "/mnt/" -e "/media/" . . . ``` I think this should be included! Otherwise adding "/, /dev, /proc and so on" is too easy. > > I added a check that requires the path to at least be absolute: > > > https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=3ad8138da571de74ac7bb89451d83130da7ba5a8 I added a word without the leading backslash. But it passes on to and the WebGUI shows the device mounted in green. So this test should include a: ``` failed=1 ``` after the log statement. > > > Maybe remove the mkdir `--parents` options and place external drives in \mnt > > or \media only? > > Should we do this for new setups? I think so. yes please. will do mote testing tomorrow...
Created attachment 1179 [details] Multiple UUID mounts (In reply to Jon from comment #28) > > And a small issue with the extrahd.cgi where in certain circumstances you > can have multiple drives: > ``` > $ cat /etc/fstab > UUID=21335fce-5183-11ed-9ab2-0b87d5f415ae /boot auto > defaults,nodev,noexec,nosuid 1 2 > UUID=7FCE-A1B7 /boot/efi auto defaults 1 > 2 > UUID=23b36154-5183-11ed-ae61-7bfd2d67b195 swap swap > defaults,pri=1 0 0 > UUID=25a5f3aa-5183-11ed-a424-eb259c8c43e5 / auto defaults 1 > 1 > UUID=4f15203f-4c42-461b-bd9c-f1932495c618 /mnt/harddisk auto defaults 0 0 > UUID=4f15203f-4c42-461b-bd9c-f1932495c618 /mnt/harddisk1 auto defaults 0 0 > UUID=4f15203f-4c42-461b-bd9c-f1932495c618 /mnt/harddisk2 auto defaults 0 0 > ``` > > This may be OK (I just don't know for sure) > See attached image. Are multiple UUID mounts OK?
maybe something like this to check for multi same UUID: ``` findmnt --source "${device}" ``` If it returns values then the UUID is already mounted.
Added this patch to fix issue with extrahd.cgi file. https://patchwork.ipfire.org/project/ipfire/patch/20230602190116.1790314-1-jon.murphy@ipfire.org/
(In reply to Jon from comment #36) > See attached image. Are multiple UUID mounts OK? No, we simply need to keep an inventory that we check when someone saves something new. I asked Stefan to join us here and help us out with all the Perl CGI stuff as he has the most experience on the team.
Created attachment 1190 [details] Modified extrahd.cgi
Hello, I've spent some time in improving the CGI file and hopefully reduce a lot of errors and problems. Currently IMHO there is still some kind of aestetic issue displaying the added and mounted devices. This code is untouched but I'm open how this could be improved.
Works really well! Just picky items (sorry!) If I pick "ext4" then click Add then the drive mounts as "auto" on the WebGUI. --- In this section: ``` # Check if a valid mount path has been choosen. unless(&is_valid_dir("$extrahdsettings{'PATH'}")) { $errormessage = "$Lang::tr{'extrahd you cant mount'} $extrahdsettings{'DEVICE'} $Lang::tr{'extrahd to root'}."; } ``` ... change "You can't mount /dev/sdb1 to root." to something like this: "You can't mount /dev/sdb1 to invalid directory." --- Super picky: # Copyright (C) 2023 IPFire Team <info@ipfire.org>
Hello Jon, thanks for testing and your feedback. All your mentioned points are done. I've sent the patchset to the development mailing list. Best regards, -Stefan
Patch from Jon merged (thank you!): https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=236e89ae87063bb1883875f76080d3f03c0351d5 Stefan, I'll have a look at your patchset in due course.
Created attachment 1205 [details] Final extrahd.cgi
Created attachment 1206 [details] en.pl Language File for extrahd.cgi
I've attached the final extrahd.cgi and the modified english language file for easy testing.
(In reply to Stefan Schantl from comment #47) > I've attached the final extrahd.cgi and the modified english language file > for easy testing. Thank you!
Just started testing. I noticed the `en.pl` files has the extrahd lines but it also has a few other lines not related to extrahd. Added: 'advproxy wpad notice' => 'Notice: For WPAD/PAC to work properly, furtcher changes need to be made. Please see the <a href="https://wiki.ipfire.org/configuration/network/proxy/extend/wpad" target="_blank">Wiki</a>.', Removed: 'dhcp fixed ip address in dynamic range' => 'Fixed IP Address in dynamic range', 'openvpn cert expires soon' => 'Expires Soon', 'openvpn cert has expired' => 'Expired', I also noticed there are no changes to /etc/fstab and to /var/ipfire/extrahd/fstab. Have those been dropped? If yes - should /var/ipfire/extrahd/fstab be deleted from the build. - should /etc/fstab have the current items changed? Like my currently included external drive. - how does my external drive get re-mounted during a reboot? Also a gentle reminder to update the copyright date; Super picky: # Copyright (C) 2011 IPFire Team <info@ipfire.org>
If I click add (pencil + icon) and the path is blank, the error message is: ``` Error messages . ``` near the top of the page.
Is using `\mnt` as the path valid? Or just considered "bad form"? before `\mnt` is see this (which is normal and correct): ``` [root@ipfireAPU ~] # ls -al /mnt total 12 drwxr-xr-x 3 root root 4096 Jun 17 12:00 . drwxr-xr-x 22 root root 4096 Jul 3 13:14 .. drwxrwxrwx 2 root root 4096 May 22 14:53 ssd [root@ipfireAPU ~] # ``` but after I click add for `\mnt` I see this: ``` [root@ipfireAPU ~] # ls -al /mnt total 52 drwxrwxrwx 8 root root 4096 May 22 18:24 . drwxr-xr-x 22 root root 4096 Jul 3 13:14 .. drwxr-xr-x 191 root root 12288 Jan 17 01:25 IPFireBackups drwx------ 2 root root 16384 Nov 15 2021 lost+found drwxr-xr-x 37 root root 4096 May 22 17:01 snapshot2 drwxr-xr-x 61 root root 4096 Jul 3 12:01 snapshot3 drwxr-xr-x 34 root root 4096 Apr 9 18:01 snapshots drwxr-xr-x 11 root root 4096 Mar 30 22:07 snapshots1 [root@ipfireAPU ~] # ``` and it overwrites (not the right word) my `\mnt` directory and I no longer see `\mnt\ssd`. Hope this made sense!
(In reply to Jon from comment #51) > Is using `\mnt` as the path valid? > Or just considered "bad form"? Sorry - all of these should be `/mnt` All of the above tested on: APU4D4 IPFire 2.27 (x86_64) - Core-Update 175
Hello Jon, thanks for testing - you have found a bug. I'll upload a new file with a fix that "/mnt" and "/media" are not allowed as mount points.
Created attachment 1209 [details] Fixed final extrahd.cgi
(In reply to Jon from comment #49)> > I also noticed there are no changes to /etc/fstab and to > /var/ipfire/extrahd/fstab. Have those been dropped? > > If yes > - should /var/ipfire/extrahd/fstab be deleted from the build. > - should /etc/fstab have the current items changed? Like my currently > included external drive. > - how does my external drive get re-mounted during a reboot? > > Also a gentle reminder to update the copyright date; > Super picky: > # Copyright (C) 2011 IPFire Team <info@ipfire.org> Not sure if you saw these from above...
@Stefan: I would like to merge this patchset into core update 178, but it does not want to apply: > https://patchwork.ipfire.org/project/ipfire/list/?series=3689 Are there any outstanding issues before this can be merged?
(In reply to Michael Tremer from comment #56) > Are there any outstanding issues before this can be merged? I just merged the rebased patchset from Stefan. Thanks for that. What I completely forgot was integrating this into the init process. In the current version that is in next, those shares won't automatically be mounted at boot. That is something I will hopefully fix today. I will also need to come up with some kind of migration process so that we remove any previous entries from /etc/fstab. That is not going to be fun.
(In reply to Michael Tremer from comment #57) > (In reply to Michael Tremer from comment #56) > > Are there any outstanding issues before this can be merged? > > I just merged the rebased patchset from Stefan. Thanks for that. > > What I completely forgot was integrating this into the init process. In the > current version that is in next, those shares won't automatically be mounted > at boot. That is something I will hopefully fix today. > > I will also need to come up with some kind of migration process so that we > remove any previous entries from /etc/fstab. That is not going to be fun. I just pushed those changes into the next branch: > https://patchwork.ipfire.org/project/ipfire/list/?series=3802 Please test. This should conclude the works on ExtraHD 2.0! Thanks to everyone who helped to make it happen.
Just tested out the updated Extra HD page with Core Update 179 Testing on my vm test bed. It seems to not be working with LVM based systems. I had two LVM volumes defined in CU178 (Started in CU177 after the problems with LVM a short while back) In CU178 rebooting the system auto mounts the LVM volumes and they show up as mounted on the Extra HD page. I then upgraded to CU 179 and after the update the dm-0 and dm-1 volumes show up on the Extra HD page but without any mount point defined and no icons to allow to mount them. The raw partitions from the volumes (sda1 and sdb1) show up on the page with an edit pencil icon. The LVM volumes are also no longer mounted and rebooting does not auto mount them as previously. I can manually mount them to the mount point but rebooting then removes them from the mount point and does not attach them back. I will attach screenshots of the system with CU178 and now with CU179 Testing.
Created attachment 1257 [details] Extra HD page on CU178 with two auto mounted LVM volumes
Created attachment 1258 [details] Extra HD page on CU179 with two LVM volumes shown but not mounted.
(In reply to Adolf Belka from comment #61) > Created attachment 1258 [details] > Extra HD page on CU179 with two LVM volumes shown but not mounted. Why are the UUIDs empty on this screenshot? It probably has something to do with the issue.
That's what is showing up. r I just modified my CU178 vm to have one LVM volume based drive using sda1 as the PV, then with a VG and LV and another drive which is just a linux partition, sdb1 Both were mounted on CU178 using the Extra HD page. Rebooting has both still mounted. Then upgraded to CU179 and both of the additional drives are not mounted and there is no way to ask for them to be mounted in the Extra HD page. I have more screenshots of this run through but will only upload them if requested. I think it just shows more of what was in my previous screenshots.
Created attachment 1259 [details] slash image Also, the directory slash for the available drive is the wrong slash. See attached file.
hmmm. and I cannot add a new drive using a backward slash or a forward slash... Tested on: APU4D4 IPFire 2.27 (x86_64) - Core-Update 179 Development Build: master/b5d85855
My bad! (sorry!) the back-slash was left over from an ld test of ExtraHD when I was trying to break things. [root@ipfireAPU ~] # cat /var/ipfire/extrahd/devices UUID=4f15203f-4c42-461b-bd9c-f1932495c618;auto;\mnt; One I cleared out this file then I could add new drives. Sorry again!
I can cause an issue when I try and mount a new available drive with "\mnt" or with "fred" (just a random word). Both cause the status circle to turn red and both can not be deleted with the trash.
I thought for a while that I might have found why I was having my problem. In CU178 I had set up the additional hard disks with mountpoints /home/ahb/data-1 and /home/ahb/data-2 The HD's were mounted at those points and files could be stored etc. In the CU179 Testing extrahd code the mountpoints are restricted to sub directories under data, media and mnt only. So I thought, maybe that was my problem that the code couldn't accept those mountpoints and was not mounting them. So I modified my CU178 vm machine to have the two HD's mounted at /mnt/data-1 and /mnt/data-2 I then upgraded to CU179 Testing and rebooted but everything has stayed looking exactly the same as I had before. The two drives are not mounted and they are not accessible from the ExtraHD WUI page as I showed earlier. I checked through the code and based on that had a look in /var/ipfire/extrahd. There is a file called devices which has the two HD's listed in it with their uuid's and correct mount points and the word auto. UUID=a916ed73-adf2-4531-9f69-b4d040860fbe;auto;/mnt/data-1; UUID=b6c76d0e-c3fe-49a6-b33c-fc085cc515d8;auto;/mnt/data-2; So the entries look correct. The partitions and scan files also look correct as far as I can tell. Looking through extrahd.pl I saw that there should be log messages with the extrahd identifier. I grepped through the /var/log/messages for extrahd and there was nothing. So it seems that there are no log messages for unsuccessful or successful mounting being logged. The fstab file no longer had the additional two drives but a copy of the fstab file with them included is in /var/ipfire/extrahd/fstab My understanding is that the extrahd.cgi/extrahd.pl code should be auto mounting the entries that are in /var/ipfire/extrahd/devices but that does not seem to be happening but also no log messages are to be found. From the code I believe those log messages should be going into /var/log/messages with the identifier extrahd but not a single entry.
Regarding LVM, I don't even get the web page to show me any unmounted logical volumes. I created a logical volume and formatted it with ext4. Did you do anything more than just that?
I created the LVM drive as you described but in a CU178 VM. Then I upgraded to CU179 Testing. I will try the same as you. To create a new LVM drive with ext4 in the CU179 Testing system and see if that is shown or not in the Extra HD screen.
Added a new drive to my CU179 Testing system and created an LVM volume on it and it shows up in the Extra-HD together with the previous two LVM's None have any UUID and none have any option to mount the drive. So I haven't been able to reproduce not finding the dm-x lvm volume at all in the table. I will list the process I did to create the drive to make sure we are doing the same thing. After starting the IPFire vm I ran fdisk on the new drive and created a GPT table and then created a single partition which I made Linux LVM (code 44). Then I ran pvcreate on the new partition. Then I ran vgcreate on that partition. Then I ran lvcreate making 100% of the free space on the volume group the new logical volume. Then I ran mkfs.ext4 on the LVM volume. Then I checked Extra HD.
Created attachment 1263 [details] Extra HD screen with additional LVM volume now shown In this screen shot you can see the new LVM volume dm-2 shown but not able to be mounted as per the previous two. Looking at the whole page I see that it shows sda1, sdb1 and sdc1 with pencils against them looking like they could be mounted. sda1 has dm-0 on it sdb1 has dm-1 on it sdc1 has dm-2 on it - this is the new LVM volume. I tried mounting sda1 to a mount point and that gave the shown error message. I presume it is saying that sda1 is already mounted as it has the dm-0 LVM volume on it.
Okay, I follow now... This is all a big mess. There is a small logic error here, because the code now is searching for block devices (i.e. hard drives) first and then looks at which partitions there are on those block devices. This is usually not how you would configure an LVM volume. It already is its own thing where I normally put a FS on without a partition table. And this is where is all goes wrong. This all requires a kind of larger rewrite in the UI code to make it work. Otherwise it is rather easy to gather call the information together. The question is now what we are going to do about the release. Since things that worked before still kind of work, I am slightly swayed towards just releasing this. There is always a workaround for the LVM people if they really struggle by just putting their volume in /etc/fstab manually. It is not that difficult. For that reason, I would like to move on and ask Stefan to fix this problem in a later release. However, with everyone being stretched out by *this* much at the moment, I don't know when that will be. Can we all live with this solution?
Ah, yes the PV can be created on any block device so that can be a whole disk drive without a partition such as sda but it can also be an MBR or GPT partition such as sda1. Just historically, I have ended up doing all my LVM stuff on top of a GPT partition, mainly because that is what the examples in the Arch Linux wiki use, although they do say you can also just use a whole disk. I am happy to put the LVM stuff manually into fstab. I am only using the Extra HD with my vm setup for testing purposes, so the LVM volumes just contain a couple of test files in just to prove that the LVM stuff is working after there was that problem that Stefan had when I updated LVM2 and the udev files changed so his auto mount didn't work any more.
I cleared the third drive on my vm bacxk to an unpartitioned drive with MBR table. I then created a pv on the whole drive and then a vg/lv It showed up again in Extra HD but still not being mounted and not able to mount so it doesn't look to be different whether a disk partition or a whole disk are used for the LVM definition. I manually added all three LVM's into the fstab and they automatically mount on reboot so that works, even if they don't show up as mounted on the Extra HD page. I think that confirms to go with the CU as it is and the Extra HD page can be worked on later on.
I just ran into another issue: I cannot mount two external drives. I tried with a USB thumb drive and and SSD and bot error with Error messages /dev/sdc1 is allreadv mounted. IPFire 2.27 (x86_64) - Core-Update 179 Development Build: master/b5d85855
(In reply to Jon from comment #76) > I just ran into another issue: I cannot mount two external drives. I tried > with a USB thumb drive and and SSD and bot error with > > Error messages > /dev/sdc1 is allreadv mounted. > > IPFire 2.27 (x86_64) - Core-Update 179 Development Build: master/b5d85855 Is this still performing the operating and just showing a useless error, it does it not mount the other one at all?
(In reply to Michael Tremer from comment #77) > (In reply to Jon from comment #76) > > I just ran into another issue: I cannot mount two external drives. I tried > > with a USB thumb drive and and SSD and bot error with > > > > Error messages > > /dev/sdc1 is allreadv mounted. > > > > IPFire 2.27 (x86_64) - Core-Update 179 Development Build: master/b5d85855 > > Is this still performing the operating and just showing a useless error, it > does it not mount the other one at all? Stefan has posted a fix for this bug here: > https://patchwork.ipfire.org/project/ipfire/patch/20230912182000.4937-1-stefan.schantl@ipfire.org/ I will merge this into master and will then proceed with the release. We will have to fix the LVM issue in a following update as I do not want to further delay this update, and I do not consider this affecting too many people. And if so, there is a workaround. Thanks to everything for their feedback. Keep it coming!
I added the modification from Stefan's fix into my Testing vm system. I then tried to mount a hard disk /dev/sdc1 with ext4 fs. The error message "/dev/sdc1 is allready mounted" was still shown. lsblk shows /dev/sdc1 as not being mounted. I can then manually mount /dev/sdc1 to /mnt/data-3 and then the ExtraHD shows that it has been mounted to that mount point. There is nothing to allow me to unmount it on the ExtraHD page. So Stefan's fix is not sufficient to fix @Jon's bug.
Also my expensive bought 4TB SSD hard disk is no longer mounted and there are many data especially my backup. What do I have to do now so that the hard disk is mounted again under the originals path and and without reformatting of the hard disk is also bad? Unfortunately my SAMBA share is gone and so is my access to the data. Do I now have to run mount /dev/sda /mnt/SSD-4TB on every reboot or can I also enter appropriate values in /etc/fstab? https://community.ipfire.org/t/trouble-with-extra-hd-core-179/10396/5
Created attachment 1348 [details] issue on reboot - cannot mount external drive Tested on: APU4D4 IPFire 2.27 (x86_64) - Core-Update 179 rebooting does not re-mount the external drive. And the external drive cannot be re-mounted via extrahd. Does not do anything. I can get extrahd to mount a external drive by removing the needed drive from /var/ipfire/extrahd/devices. And then all is well until the next reboot. See: https://community.ipfire.org/t/trouble-with-extra-hd-core-179/10396/15?u=jon
*** Bug 13354 has been marked as a duplicate of this bug. ***
This is from Ewald XYZ @ewald and https://community.ipfire.org/t/trouble-with-extra-hd-core-179/10396/5?u=jon "I found some mysteries: /lib/udev/rules.d/61-extrahd.rules contains ACTION=="add", SUBSYSTEM=="block", RUN+="/var/ipfire/extrahd/bin/extrahd.pl udev-event" However, when I am running “/var/ipfire/extrahd/bin/extrahd.pl udev-event”` from the command line, the error message is # /var/ipfire/extrahd/bin/extrahd.pl udev-event /var/ipfire/extrahd/bin/extrahd.pl: Unsupported command: udev-event For me it looks like as if ‘/var/ipfire/extrahd/bin/extrahd.pl’ is still the previous core 178 version: There in no no string ‘udev’ in the file ‘var/ipfire/extrahd/bin/extrahd.pl’."
It looks like this did not make it into CU 179
https://git.ipfire.org/?p=ipfire-2.x.git;a=blobdiff;f=config/extrahd/extrahd.pl;h=cc85228d56f17ad8dc5c69e388a63535af8f5333;hp=145a2a43a1c255ca7f315d4126e4eb7ce5c5f0da;hb=7c9f5f9882af2203e0445e1d9044f4df0bc00265;hpb=fe2fd95a9ef320391a09b91304a97bc68bc0573b
Changing this: /lib/udev/rules.d/61-extrahd.rules: ACTION=="add", SUBSYSTEM=="block", RUN+="/var/ipfire/extrahd/bin/extrahd.pl udev-event" to this: ACTION=="add", SUBSYSTEM=="block", RUN+="/var/ipfire/extrahd/bin/extrahd.pl mount udev-event" Seems to make it work. Still testing.
Good Morning, I've sent a patch to the ML which addresses the missing LVM/RAID support for the ExtraHD CGI. I tested them in an Virtual environment an everything worked well. It also fixes the problem if a filesystem directly is created on a disk without creating any partitions. (See: https://community.ipfire.org/t/trouble-with-extra-hd-core-179/10396/6?u=stevee) The patch can be found here: https://patchwork.ipfire.org/project/ipfire/patch/20230923105455.4960-1-stefan.schantl@ipfire.org/ @Adolf, can you please verify that the patch fixes the LVM problems for you?
Hi @Stefan. Tested out the patch. After applying the patch the extrahd screen showed the LVM volume but the button was in red and the message said it was not mounted. However there was no pencil icon to mount it only the dustbin to remove it but there was nothing to unmount. The problem was that my LVM volumes were previously mounted in /home/xxx/ and the new extrahd only allows mnt, media and another location that I can't remember now. So the info copied from the fstab to devices had a mount location specified that was not allowed. I then cleared the entry from the /var/ipfire/extrahd/devices file and then I got the red button together with the pencil icon. I could then use the pencil icon to mount the drive and the dustbin icon to unmount the drive. So the patch works but users need to have their existing mounts defined with mountpoint points only in the new allowed directory trees. I then rebooted and that came back with the red button and the dustbin icon. So the drives were not mounted on boot but that might need the additional patch from @Arne. I will look at adding Arne's patch into my system later on and see if that remounts successfully after reboot.
I patched in @Arne's code for the udev_event. https://patchwork.ipfire.org/project/ipfire/patch/20230927150408.3190-1-arne_f@ipfire.org/ I then ran reboot. The /dev/sda1 drive was mounted after booting. The /dev/dm-0 LVM volume (/dev/IPFire1/Data1) was not mounted after booting. So Arne's patch of the udev rule works for ordinary partitions but not for LVM volumes. I will also do a test with an unpartitioned drive, ie /dev/sda, to confirm that also works with @Arne's patch.
I created a drive using the whole of a disk, ie /dev/sdb, and formatted it with ext4 and was able to mount it with Extra HD. Then rebooted and the drive was automatically mounted. So the code is working fine with whole drives /dev/sdb and partitions /dev/sdb1 for mounting and being remounted after reboot. LVM volumes on either a whole drive or a partition can be initially mounted but on reboot they end up unmounted and the status can then only be changed via the command line by mounting the volume on the previously used mount point. Then the green button is shown and if the drive is deleted with the dustbin icon then it can be remounted using the pencil icon.
Hello Adolf, thanks for testing and your feedback. I updated my virtual machine to the latest testing version (c180), including a complete reinstall of c178 and c179 to be in sync with upstream again. After applying the CGI patch and Arne's udev patch I rebooted the machine. All ExtraHD drives are mounted - 3 LVM, 1 RAID and a single disk with the filesystem on the plain drive without partition. proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,noexec,size=478684k,nr_inodes=119671,mode=755) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,noexec) tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,mode=755) /dev/vda4 on / type ext4 (rw,relatime) none on /sys/fs/cgroup type cgroup2 (rw,relatime,favordynmods) /dev/vdf on /mnt/test type ext4 (rw,relatime) /dev/md127 on /data type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/vda1 on /boot type ext4 (rw,nosuid,nodev,noexec,relatime) /dev/vda2 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro) /dev/mapper/TEST-BLAH on /media/blah type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/mapper/GROUP-BLUB on /media/blub type ext4 (rw,relatime) /dev/mapper/GROUP-VOLUME on /media/volume type ext4 (rw,relatime) /var/lock on /var/lock type tmpfs (rw,nosuid,nodev,relatime,size=8192k) [root@ipfire ~]# lvscan ACTIVE '/dev/TEST/BLAH' [512.00 MiB] inherit ACTIVE '/dev/GROUP/VOLUME' [1.00 GiB] inherit ACTIVE '/dev/GROUP/BLUB' [512.00 MiB] inherit [root@ipfire ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert BLUB GROUP -wi-ao---- 512.00m VOLUME GROUP -wi-ao---- 1.00g BLAH TEST -wi-ao---- 512.00m
@Adolf, how did you create your LVM groups and volumes? Maybe there a differance can be found why the mount after a reboot works in my vm testbed and not for you....
Run fdisk with the vm drive and create a GPT partition table. Then create a partition the full size of the disk ending with /dev/sda1 pvcreate /dev/sda1 vgcreate IPFire1 /dev/sda1 lvcreate -l 100%FREE IPFire1 -n Data1 mkfs.ext4 /dev/IPFire1/Data1 The second approach I also tried (and gave the same results was:- pvcreate /dev/sda vgcreate IPFire2 /dev/sda lvcreate -l 100%FREE IPFire2 -n Data2 mkfs.ext4 /dev/IPFire2/Data2 How did you create your volumes? Maybe it would be good for me to also install CU178 and update to 179 and then go to CU180 Testing to make sure I am also properly synched. Then add the two patches for extrahd.cgi and extrahd.pl and see if I get the same effect again. Here is the lvs result from the single LVM volume I currently have on my vm system. LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Data1 IPFire1 -wi-ao---- 496.00m lvscan ACTIVE '/dev/IPFire1/Data1' [496.00 MiB] inherit mount proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,noexec,size=479500k,nr_inodes=119875,mode=755) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,noexec) tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,mode=755) /dev/md127p5 on / type ext4 (rw,relatime) none on /sys/fs/cgroup type cgroup2 (rw,relatime,favordynmods) /dev/md127p2 on /boot type ext4 (rw,nosuid,nodev,noexec,relatime) /dev/md127p3 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro) /var/lock on /var/lock type tmpfs (rw,nosuid,nodev,relatime,size=8192k) /dev/sdb on /mnt/data2 type ext4 (rw,relatime) /dev/mapper/IPFire1-Data1 on /mnt/data1 type ext4 (rw,relatime) The raid drive is the main drive with the IPFire installation, hence several partitions. If I reboot then /dev/sdb remounts with no problem. Also the same when I had /dev/sdb1 /dev/mapper/IPFire1-Data1 does not remount on reboot and I have to manually mount it. It can't be done from the ExtraHD WUI page.
Good Morning Adolf, I had my LVM command set from one of the thousands of mini how-to's from the net.... Today I used the latest nightly build (c181) which now also includes the fixes from Arne. Starting with a complete new installation on a new VM. This time with 2 virtual disks for a RAID installation to have a more similar ground system to yours. After that I applied the patched ExtraHD CGI which worked well. Attached 2 additional virtual disks and used exactly your first descriped method (starting from gpt partion table, and your posted command set) to get the LVM volume up. It will be shown correctly in the CGI and can be mounted without any troubles. (I used mountpoint "/data"). Then I created an Ext4 file system directly on the last empty device for testing and mounted it to "/mnt/test" by using the CGI again. I rebooted the virtual machine and everything was mounted properly again.
Created attachment 1351 [details] extrahd.cgi screen after reboot with CU181 (unstable) I am still having the problem. I installed CU178 on the vm system and then updated it to CU179 then changed to Testing and took it to CU180. @Arne's patches are also now in the latest CU180 Testing. I then patched extrahd.cgi with your changes. Same problem happened. After running the patch the drive was not mounted but showed in the cgi with a red button together with the dustbin icon but pressing the dustbin icon did not change anything. I then manually mounted the LVM volume and it showed up with a green button and the dustbin icon. If I then pressed the dustbin icon the drive was removed from the devices file and there was a grey button and the pencil icon. I could then mount the drive from the cgi screen. Then rebooted and the drive was back with the red button and the dustbin icon. I then ran pakfire with the unstable repo which should have given me the same status as when you installed from the latest nightly next build. However the same thing happened with the LVM drive. I then created a new lvm drive from the /dev/sdb drive which had been mounting automatically withy no problem. I wanted to check if the problem was to do with an lvm drive that was previously created vs one created from new at that time. Same thing happened. I mounted the old LVM drive and the new one from the extrahd.cgi screen and then rebooted. Result is the attached screen showing the red button and the dustbin icon. Pressing the dustbin icon doesn't change anything. I am now going to create a new vm and install IPFire onto that. Just to check. I am using VirtualBox for my vm testbed systems. The drives that I have attached to the vm's are dynamically allocated. @Stefan are you using virtualbox and do you have your drives with dynamic allocation or fully pre-allocated?
Hi @Stefan Created a new vm with a new additional disk drive. Again with dynamic allocation. Installed CU179, upgraded to CU180 Testing and patched extrahd.cgi with your patch. Created an lvm volume on the additional drive. Successfully mounted it in the extrahd.cgi page and then rebooted. After reboot the lvm drive is correctly mounted. Looks like there is something weird about the lvm drives I have previously created. So with your patch and @Arne's patches extrahd looks to be working with all types of drives including LVM and mounts them automatically on reboot. Sorry for having caused problems. I will go back to my earlier vm's and see what happens if I attach new drives into the vm and use them.
I went back to my earlier vm's and deleted the additional attached drives and created and attached some new ones and created the LVM volume. Unfortunately the same problem happened. So it looks like the problem is some quirk in the vm image that has been created or cloned over time. So I have now gone back to the newly created vm, defined it to be the same as my previous vm's, restored my backup, including logs and tested out the extrahd with two lvm volumes. They could be controlled from the wui page and stayed mounted after reboots. I will now test doing a clone of this vm and confirm that everything stays working as expected.
Hello Adolf, don't worry about - that's exactly how the development and testing process should go and finally the code is well tested and working. I'm going to put this to Michael so he can merge the patch and hopefully we get this shipped with c180. Thanks for testing and your hard work!
I am glad we managed to get it all working. Making a clone kept the working condition of the LVM volumes, so that is also good. I have also found that making clones of the newly created vm is now much faster than it used to be. So a win all round.
@Michael, please merge the following patch: https://patchwork.ipfire.org/project/ipfire/patch/20230923105455.4960-1-stefan.schantl@ipfire.org/ Thanks in advance, -Stefan
(In reply to Stefan Schantl from comment #100) > @Michael, please merge the following patch: Done! Thank you.
Created attachment 1355 [details] before boot Testing on: APU4D4 IPFire 2.27 (x86_64) - Core-Update 180 Development Build: master/a98abe92 I am not sure if the above version includes comment #c100 below (I do not think it does). I mounted an external USB thumb drive that has two partitions. All mounted and seemed ok. See before_boot.png image. After reboot only one of the two partitions appeared. See after_boot.png image.
Created attachment 1356 [details] after boot
(In reply to Jon from comment #102) > I mounted an external USB thumb drive that has two partitions. All mounted > and seemed ok. > After reboot only one of the two partitions appeared. Hmm, this is definitely not correct. There are two places where these partitions could be mounted: * In the init process after the root filesystem has been mounted. It could happen that the device was not ready (USB sticks, LVM, etc. need a longer time to initialise). * If the device becomes ready later, it will be mounted through udev. So I would say, either way, if a device has two partitions, they are either both ready at init time, or they are both mounted by udev. Could you please try if the device can be mounted at all for me? > /var/ipfire/extrahd/bin/extrahd.pl mount This should mount it if triggered manually. If you then reboot, is it always the same partition that is missing? The udev process can be triggered like this: > udevadm trigger > udevadm settle Does this mount the partition when triggered manually? If not, is it always the same partition?
Keep in mind /dev/sdc is already mounted in an "odd" way. It is not "/dev/sdc1", it is "/dev/sdc" that is mounted. See after_boot.png image. (In reply to Michael Tremer from comment #104) > > Could you please try if the device can be mounted at all for me? > > > /var/ipfire/extrahd/bin/extrahd.pl mount I get this response: [root@ipfireAPU ~] # /var/ipfire/extrahd/bin/extrahd.pl mount mount: /mnt/usb2: /dev/sdc2 already mounted or mount point busy. dmesg(1) may have more information after failed mount system call. [root@ipfireAPU ~] # > > This should mount it if triggered manually. If you then reboot, is it always > the same partition that is missing? Yes, same partition. > The udev process can be triggered like this: > > > udevadm trigger > > udevadm settle > > Does this mount the partition when triggered manually? If not, is it always > the same partition? No, it does not mount with the two udevadm commands above. Yes, the same partition. And it is still mounted via `/dev/sdc` and not `/dev/sdc1`. Could the code be missing the `1`?
I see these lines in the messages log: [root@ipfireAPU ~] # cat /var/log/messages | grep usb Oct 2 12:10:56 ipfireAPU extrahd: Could not mount UUID=E4D8-6E65 to /mnt/usb2: 32 Oct 2 13:00:26 ipfireAPU extrahd: Could not mount UUID=E4D8-6E65 to /mnt/usb2: 32 Oct 2 13:00:35 ipfireAPU extrahd: Could not mount UUID=E4D8-6E65 to /mnt/usb2: 32 Oct 2 13:10:01 ipfireAPU extrahd: Could not mount UUID=E4D8-6E65 to /mnt/usb2: 32 [root@ipfireAPU ~] #
If I unmount the sdc, then I can "mount" the two partitions: [root@ipfireAPU ~] # umount -v /dev/sdc umount: /mnt/usb1 (/dev/sdc) unmounted [root@ipfireAPU ~] # [root@ipfireAPU ~] # mount -v /dev/sdc1 /mnt/usb1 mount: /mnt/usb1: WARNING: source write-protected, mounted read-only. mount: /dev/sdc1 mounted on /mnt/usb1. [root@ipfireAPU ~] # mount -v /dev/sdc2 /mnt/usb2 mount: /dev/sdc2 mounted on /mnt/usb2. [root@ipfireAPU ~] #
The extrahd.cgi patch from @Stefan has been merged into Core Update 180 Testing. I have updated my Core Update 180 Testing system to that latest nightly build and can confirm that my LVM volumes are properly recognised and correctly mounted when rebooting.
I just tried the newer CU 180 test version APU4d4 IPFire 2.27 (x86_64) - Core-Update 180 Development Build: master/729fe58b and I see this issue with ExtraHD: ``` $ mount | grep ^/dev /dev/sda4 on / type ext4 (rw,relatime) /dev/sdc on /mnt/usb1 type iso9660 (ro,relatime,nojoliet,check=s,map=n,blocksize=2048,iocharset=utf8) /dev/sdb1 on /mnt/ssd type ext4 (rw,relatime) /dev/sda1 on /boot type ext4 (rw,nosuid,nodev,noexec,relatime) /dev/sda2 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro) ``` Same as comment 105. I could not tell if this was fixed in one of the recent CU 180 updates or not. Or is set for CU 181.
Core Update 182: still buggy. New Installation (VM Proxmox) and a blank 500GB vdrive. Created partition label + partition with 'parted'. Formatted the partition with 'mkfs' in ext4. Tried to mount via extrahd in /mnt/harddisk. The mount point was created, but the GUI says "not mounted". Also it does not show the partition type 'ext4'. So I did a restart and the boot page showed up a error when trying to mount the partition. Now the GUI says the partition is "not configured". When I try to mount it again it says: "You can't mount /dev/sdb1 to /mnt/harddisk , because there is already a device mounted. "
Also whats the intention of "not configured" and "not mounted"? If a device is not configured, it shows "not mounted". If a device is configured, but however not mounted, it shows up "not configured". This is strange!
If the message is saying that a drive is already mounted on that directory, what do you see if you run the lsblk command. Does it show that directory mounted on a device or not? Also what is the content of the /proc/mounts file?
(In reply to Adolf Belka from comment #112) > If the message is saying that a drive is already mounted on that directory, > what do you see if you run the lsblk command. Does it show that directory > mounted on a device or not? > > Also what is the content of the /proc/mounts file? No. NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 32G 0 disk ├─sda1 8:1 0 512M 0 part /boot ├─sda2 8:2 0 32M 0 part /boot/efi ├─sda3 8:3 0 982.7M 0 part [SWAP] └─sda4 8:4 0 30.5G 0 part / sdb 8:16 0 500G 0 disk └─sdb1 8:17 0 500G 0 part sr0 11:0 1 451M 0 rom
(In reply to Adolf Belka from comment #112) > Also what is the content of the /proc/mounts file? proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,nosuid,noexec,size=1980528k,nr_inodes=495132,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,noexec 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,mode=755 0 0 /dev/sda4 / ext4 rw,relatime 0 0 none /sys/fs/cgroup cgroup2 rw,relatime,favordynmods 0 0 /dev/sda1 /boot ext4 rw,nosuid,nodev,noexec,relatime 0 0 /dev/sda2 /boot/efi vfat rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount->/var/lock /var/lock tmpfs rw,nosuid,nodev,relatime,size=8192k 0 0
So there is definitely nothing mounted but the ExtraHD code seems to think it is mounted. For that to happen there must be somewhere that the code is storing that the drive is mounted and when some hiccup occurs and the mount is unsuccessful the ExtraHD code must still think it is mounted due to the stored info. At least the above is one explanation for the effect that is occurring. Later on I will try and see if I can find if the mount data is recorded somewhere, although the obvious place of the devices file, you have previously mentioned was empty.
@Terry you mentioned that the directory /mnt/harddisk was created so can you try to run the command mount /dev/sdb1 /mnt/harddisk Then check if you can access the files now in that directory. If yes then it means that the mount was successful which means that both the drive and the mount directory are fine. If no then what message did you get when mounting? If it mounts successfully then you can use that for your samba use. You will just need to manually unmount it when a solution to the ExtraHD code is identified.
(In reply to Adolf Belka from comment #116) > @Terry you mentioned that the directory /mnt/harddisk was created so can you > try to run the command > > mount /dev/sdb1 /mnt/harddisk > > Then check if you can access the files now in that directory. > > If yes then it means that the mount was successful which means that both the > drive and the mount directory are fine. > > If no then what message did you get when mounting? > > > > If it mounts successfully then you can use that for your samba use. > > You will just need to manually unmount it when a solution to the ExtraHD > code is identified. The folder still exists and I can mount without any problems. I created a few folders so it works fine.
I have looked through the code and I have found the subroutine that creates the list of mountpoints, which it does from /proc/mounts. The list is updated at two locations in the code and looks to end up in a hash variable with other factors, such as uuid, filesystem etc. The only thing I can think of is if the drive is tried to be mounted, ends up in /proc/mounts and hence gets read into the hash variable and subsequently the mount fails to stay mounted that thge hash variable then thinks that the mountpoint is already used. However rebooting IPFire should re-start the ExtraHD perl code, when the WUI page is selected and this would refresh the mountpoints list from /proc/mounts and therefore should no longer have a problem but if I understand you correctly, once the drive is listed as not being mounted and can't be due to the mountpoint being used, it stays like that even after a reboot. The message "You can't mount /dev/sdb1 to /mnt/harddisk , because there is already a device mounted." occurs if that mountpoint is in variable %mountpoints which is filled from /proc/mounts and filters out any entries not related to a device ie found in /dev/
What happens if you manually unmount the drive, disconnect the drive from IPFire and then reboot IPFire and then re-insert the drive into the usb socket and then check ExtraHD, what does the screen look like at that point. Do you have the pencil icon or the red button icon?
USB Socket? This is a VM. I can't easily plug/unplug the volume. I did a reboot several times and since I cleaned up the "device" file there is no + to mount the drive via ExtraHD GUI anymore, even it's not mounted. Also unless it's mounted, it doesn't recognize the filesystem ext4.
You can see the screenshots in the forum (can't upload pictures here).
Maybe we should check the script version first. I may have an old(er) version than expected.
(In reply to Terry from comment #120) > USB Socket? This is a VM. I can't easily plug/unplug the volume. I did a > reboot several times and since I cleaned up the "device" file there is no + > to mount the drive via ExtraHD GUI anymore, even it's not mounted. > > Also unless it's mounted, it doesn't recognize the filesystem ext4. Ah, a VM. Missed that. Maybe that makes a difference with the commands in ExtraHD. I have used ExtraHD with my VirtualBox vm testbed systems with virtual LVM drives and that all worked with no problems.
(In reply to Terry from comment #122) > Maybe we should check the script version first. I may have an old(er) > version than expected. so I have taken the b2sum hashes for the extrahd.cgi and extrahd.pl files bc87dbb74d3c61844f5fcc2d1d006eb7f1f3eb8288d35088504b04296108677a6936b4d4618cfb84588ecb54c9226f5b09ff2d4a3c53f1742b7681319e3b5d22 /srv/web/ipfire/cgi-bin/extrahd.cgi a75f61f11ab5c653745a7ba8e9daa70d9cab24de9d204925cca74293a6ff366d8ba07068f0ff000397bb09d910b63e587f53b27fb24f7c86e8d0c4e05edefcc5 /var/ipfire/extrahd/bin/extrahd.pl Those are from my CU182 system (physical machine)
Mine is also a LVM volume. I have different hashes SHA-512 4365c06a920ba40b2a91f1ae5080458c48fd6b85d0b4cc052262c9c3c50a810dc75a651ae55a43cb2620aa8aaa28ea15e68b4d345ab0a61c7cf4a8b97413f62b,Z:\extrahd.cgi c64b9329b865bb4b048f46bb66c72910db4345ac629abef143401fa5eb0ea8dfab9c1ff45d7aa1e861333d713613d8604d15ed2aa53b5d7fe51a301b44c0d9cf,Z:\extrahd.pl
b2sum is something unknown to me. My tool can MD5, SHA and CRC32.
The sha512sum values are the same on my system, so the code looks to be as expected, just not doing things as expected. The b2sum hash is available on the IPFire console but so is the sha512sum hash. Anyway the code is the same as on my system.
(In reply to Terry from comment #125) > Mine is also a LVM volume. > If yours is an lvm volume then it is not getting recognised as such because it is showing up as /dev/sdb1 As an LVM drive it should show up as /dev/VG-name/LV-name if I remember correctly on my vm testbed.I will have to have another look at the vm's. They are also updated to CU182.
My bad. The volume files are stored on a lvm. On Proxmox they show up as SCSI drives.
Created another drive with ext3. Same result.
Created attachment 1435 [details] ExtraHD screenshot after usb stick inserted and before mounted At this point the filesystem type is not shown. There is a grey light and a pencil icon.
Created attachment 1436 [details] ExtraHD screenshot after usb stick mounted This shows the screen after I have entered the mount path into the box and then pressed the pencil icon. The light changes from grey to green and the pencil icon changes to a dustbin icon.
(In reply to Adolf Belka from comment #131) > Created attachment 1435 [details] > ExtraHD screenshot after usb stick inserted and before mounted > > At this point the filesystem type is not shown. There is a grey light and a > pencil icon. That's the point. I mentioned it before. My drives show up as not mounted (but configured?) even they are not. That's why I don't have that Add pencil and just the delete Bin.
Created attachment 1437 [details] ExtraHD screenshot after usb stick unmounted This shows the screen after the dustbin icon has been pressed, the button goes grey again and the dustbin icon changes back to the pencil icon. Also the filesystem type is now shown, exfat In your case you end up with the dustbin icon and the red light which indicates that the mount point is already being used. However pressing the dustbin icon does not take you back to the starting position of the pencil icon and the grey light. This all looks like there is a logic mismatch somewhere in the code that once you end up with the red light and the dustbin icon it can never be changed. However, so far I have not found what is going wrong with the code.
Let's see what log messages are occurring from the extrahd actions. It should be possible to see this log info in the System Logs but ExtraHD is not one of the options. I will raise a patch to fix that. Can you provide the output from less /var/log/messages | grep extrahd
(In reply to Adolf Belka from comment #135) > Let's see what log messages are occurring from the extrahd actions. > > It should be possible to see this log info in the System Logs but ExtraHD is > not one of the options. I will raise a patch to fix that. > > Can you provide the output from > > less /var/log/messages | grep extrahd Jan 4 08:56:05 MM-FW-001 extrahd: Skipping invalid mountpoint: harddisk Jan 4 09:09:27 MM-FW-001 extrahd: Could not mount UUID= to /mnt/harddisk: 32 That's all and just before I cleaned up the device file because deleting via GUI didn't work.
Jan 4 08:56:05 MM-FW-001 extrahd: Skipping invalid mountpoint: harddisk When you entered the mountpoint did you enter /mnt/harddisk or just harddisk? Jan 4 09:09:27 MM-FW-001 extrahd: Could not mount UUID= to /mnt/harddisk: 32 This suggests that the code could not find the UUID for the drive as it just has UUID= and there should be the UUID as well. On the screenshot you showed the drive had the UUID shown on the screen so at least for the screen the UUID was known but not for the mounting part. I also note that this log line refers to /mnt/harddisk: and I am not sure about the : plus then there is the entry of 32. I will have to look at the logging code to see what the : and the 32 mean. My extrahd log is the following Dec 31 19:54:07 ipfire extrahd: Successfully mounted UUID=FBF3-5EDA to /mnt/USBPens Dec 31 20:03:03 ipfire extrahd: Successfully umounted UUID=FBF3-5EDA from /mnt/USBPens Jan 4 11:22:47 ipfire extrahd: Successfully mounted UUID=FBF3-5EDA to /mnt/Testing Jan 4 11:23:18 ipfire extrahd: Successfully umounted UUID=FBF3-5EDA from /mnt/Testing Jan 4 11:51:56 ipfire extrahd: Successfully mounted UUID=FBF3-5EDA to /mnt/Testing Jan 4 11:54:30 ipfire extrahd: Successfully umounted UUID=FBF3-5EDA from /mnt/Testing Jan 4 13:33:00 ipfire extrahd: Successfully mounted UUID=FBF3-5EDA to /mnt/Testing Jan 4 13:33:04 ipfire extrahd: Successfully umounted UUID=FBF3-5EDA from /mnt/Testing Jan 4 13:33:11 ipfire extrahd: Successfully mounted UUID=FBF3-5EDA to /mnt/Testing Jan 4 13:34:10 ipfire extrahd: Successfully umounted UUID=FBF3-5EDA from /mnt/Testing Jan 4 15:21:30 ipfire extrahd: Successfully mounted UUID=FBF3-5EDA to /mnt/Testing Jan 4 15:21:35 ipfire extrahd: Successfully umounted UUID=FBF3-5EDA from /mnt/Testing I will try and get some time later today or tomorrow to have a look through the code.
Just to give a few more information, but I thing unimportant: The first log entry was caused because I just defined "harddisk" as mount point. I thought the root directory is fixes /mnt/. The second log entry came up after rebooting... BUT: I did something else wrong (first). At the beginning the drive sdb had no partition, but I was capable to press the mount button via GUI. It created the mount directory, but showed up the status "not mounted". Because of that I did a restart- Finally after some cups of coffee I realized that I had forgotten parted and mkfs, but the mount pencil never came back. However the nice thing of hypervisors is that I can create a many machines as I want and I will just create a new ipfire machine an reproduce the situating by capturing the screen (just in case I can reproduce it).
So this is the look after installation: After that I forgot to create a partition and formatted sdb as ext4. However that worked (don't know why). After that the Add Pencil appeared. I defined "harddisk" as mount point, which failed. However here is the bug. Even the mount point is invalid, it adds it to the "device" file. Also I'm not able to delete it via ExtraHD GUI (Delete Bin has no effect). I did a reboot. Here comes the first difference to the other, original VM: I can't see an error message because of failed mounting during boot. I still can't delete the entry via GUI. I deleted that entry manually in the "device" file. And now here is a other difference to the other, original VM: The GUI is ready to mount the device again. But I just checked the "device" file at the original VM again and... surprise... I could find a line break. That causes a second bug: blocks the functionality of the ExtraHD GUI.
You can find screenshots here: https://community.ipfire.org/t/is-extrahd-broken-in-core-update-181/10797/12