ZFS on Ubuntu/Mint as root guide in exchange for 1 month of 8c/16t Xeon crunching...

[josh1980]

So I'm going to try getting my crunching back up, but I'm not doing it without ZFS on linux. In particular, the OS needs to be on ZFS. I found this guide: https://github.com/zfsonlinux/pkg-zf...oot-Filesystem

But it has problems.

At step 5.5 software-properties is gone. But I can work around that since apt-get kindly tells me what the substitute is.

But 5.6+ is where things go heavily wrong. There is no zfs-grub and I have zero clue how to make it work.

I've got an ESXi server with 16c/32t and one CPU is never active. For whomever can either provide updated working instructions to that link or in this forum my VM will be dedicated to CPU1 for 30 days doing *any* work units your heart desires. Yes, even CEP2.

I've got multiple other machines that need to be assembled and such that I plan to use this winter for "heating", but I'm not going to get into WCG this year without ZFS on linux as root. Too many problems with corruption on hard drives and stuff in past years that took too long to identify. I'm going full steam of all stop on WCG this winter. :P

I know ZFS inside and out, but I don't have a clue about grub and Linux bootloading stuff.

Any takers?

I can help out if you have questions about ZFS, but the real problems are with the grub package install commands. It's which grub package to install, how to setup grub to boot from ZFS (parameters), etc.

The instructions on the link are for Ubuntu 14.04. I plan to use this with Linux Mint 17 (latest Long Term Support version), so working instructions for either OS will work. I can make the conversion from Ubuntu to Mint easily.

[PoppaGeek]

At which command did it fail and what did it say?

[josh1980]

Doing this with an Ubuntu 14.04.01 CD (basically exactly how the manual specifies):


Steps 1 to 4 work perfectly.

Step 5.1-5.4 works fine.

Step 5.5 says that python-properties-common is missing, obsoleted or only available from another source, but the package software-properties-common replaces it. Well, since I was going to install that package anyway I simply remove the python-properties-common from the apt-get install command. Seems harmless enough. It downloads a boatload of dependencies and stuff and appears to be fine.

5.6 is where everything falls apart:

# apt-add-repository --yes ppa:zfs-native/stable

Fine

# apt-add-repository --yes ppa:zfs-native/grub

Fine, but this will be important in a bit...

# apt-get update

Fine

# apt-get install --no-install-recommends linux-image-generic linux-headers-generic

Fine. Another 300MB of packages install without consequence.

There are a few errors like "cannot read table of mounted file systems" which I think is normal and expected for conditions.

# apt-get install ubuntu-zfs

Fine. Another 124MB of packages download and install

# apt-get install grub2-common grub-pc

Things install, but now questions begin.

As part of the install grub-pc asks where to install. Choices are:
/dev/sda (64424MB; ???)
/dev/sda1 (/boot/grub)
/rpool/ROOT/ubuntu-1 (??? MB; ???)

Notice that the last entry has question marks. I think this is a hint of problems to come. The obvious answer is /boot/grub, so that's where I install.

The rest of 5.6 seems to "just work".

6.1: Things are validated to be broken.

# grub-probe /

Expected output is "zfs". Instead I get "grub-probe: error: failed to get canonical path of '/dev/scsi-36xxxxxxxxxxxxxxxxxxxxx-part2".

Ok, now we almost certainly have a problem. That partition is ZFS. So unless I'm mistaken (and I could be), this is grubs way of saying "wtf are you doing to me!? I don't recoginize this garbage."

Step 6.2 says to do an "ls /boot/grub/i386-pc/zfs*" and verify files are in place. Those files are there, so I think I should have ZFS support, no?

Of course, at this point the guide says see the troubleshooting notes for grub below. Go to the bottom of the document to the troubleshooting section iii and it says to install zfs-grub. That package is long gone. It exists only for 12.04 and no 14.04 exists. Note that it says to reinstall zfs-grub. But zfs-grub isn't already installed. Clearly the document was edited without fully reading the whole thing, so it's nonsensical.

6.3 says to run an update-grub. It throws an error that it doesn't know what the scsi-36xxxxxxxxxxxxxxxxxxx-part2 is (the zfs partition). So it still seems to be lost.

6.4 has you run the following:

# grub-install $(readlink -f /dev/disk/by-id/scsi-SATA_disk1)

It says not to reboot until you can an "Installation finished. No error reported." message. That's not what i get (duh?!)

I get:
Installing for i386-pc platform.
grub-install: warning: File system 'ext2' doesn't support embedding.
grub-install :warning: Embedding is not possible. GRUB can only be installed in this setup using blocklists. However, blocklists are UNREALIABLE and their use is discouraged...
grub-install: error: will not proceed with blocklists.

Reading around the internet there *are* people using ZFS as root. Many people asking for help with this, and nobody really answers. I did talk to the guy that had originally wrote the page in IRC and he said he'd get back to me, but he's since fallen off the face of the earth (allegedly there's conferences and work stuff for him right now so he's not available for a while).

At this point i'm at a standstill on what/how to recover from my situation. I have this in a VM right now, with a bunch of snapshots so I can try various things easily. I just don't know what/how to make this work.

So here's what I know (may be inaccurate info as there's conflicting info from ym searches)....

When ZFS on Linux was first getting off the ground grub needed serious recoding to make it work with ZFS. It still has limitations like it can't boot from pools with compression or dedup. No problem and I'm okay with that.

But when grub was being recoded to do zfs it was handled as a separate package. At the time, that was zfs-grub. zfs-grub was last compiled for 12.04. Allegedly the zfs stuff has been merged back into the main grub code, but I have *zero* clue how to validate this works or not (I think it's validated by the fact that the zfs files exist in /boot/grub/i386-pc/zfs*). To boot, some people say you have to install the last 1.xx versions and others say it was added in 2.x. I don't have a clue how to get this info, how to validate it for correctness, nor how I would "get an older version" if I needed it. :P

I was hoping to get a response from you because I think you are the only one in the forum with the necessary Linux experience to have the answer. I'm doing this in a VM right now and if you want I can let you teamviewer into my laptop and play with the VM as much as you want. Right now I take a snapshot after every step, so I have steps 1 to 4 covered. After that I don't take snapshots because I have no doubt that things are very broken.

[josh1980]

One thing I just noticed... The grub error...

# grub-probe /
grub-probe: error: failed to get canonical path of '/dev/scsi-36xxxxxxxxxxxxxxxxxxxxx-part2

It's going to /dev/scsi-blahblahblah.

That's not correct. It should be going to /dev/disk/by-id/scsi-36xxxxxxxxxxxxxxxxxxxxxxxxx-part2 or, alternatively, /dev/sda2. I hear the /dev/disk/by-id is better so that's what I'd expect. So how do I fix this?

[PoppaGeek]

Try:

mount /dev/sda /mnt
chroot /mnt
update-grub2

Does not look like grub is seeing the partition. Hopefully with above it will find it.

Don't know if I can live up to your expectations, not hitting on all cylinders some days. There is one or 2 here that also can be of help.

[josh1980]

Yeah, I can't do that. I get "mount: /dev/sda already mounted or /mnt busy"

Here's the current mounts:

rpool/ROOT/ubuntu-1 on / type zfs (rw,relatime,xattr,noacl)
rpool on /rpool type zfs (rw,relatime,xattr,noacl)
rpool/ROOT on /rpool/ROOT type zfs (rw,relatime,xattr,noacl)
/dev/sda1 on /boot/grub type ext3 (rw,relatime,data=ordered)
udev on /dev type devtmpfs (rw,relatime,size=2012424k,nr_inodes=503106,mode=7 55)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)

I'm not using /mnt nor am I inside it so I don't think it's in use by me. /dev/sda is definitely there though.

[PoppaGeek]

Can you run update-grub2

[josh1980]

Yes. I get the same error as before...

/usr/sbin/grub-probe: error: failed to get canonical path of '/dev/scsi-36xxxxxxxxxxxxxxxxxxxxxxxxxxxx-part2'.

[PoppaGeek]

Ubuntu On a ZFS Root File System for Ubuntu 14.04

Complete instructions. Above for 14.04 changes.
Booting Ubuntu on a ZFS Root File System

[josh1980]

Yeah, i've seen that. But I can't glean any useful info from that link that makes me figure out what I did wrong.

In that link his solution isn't to make it right, it's to go back to the version of grub that was the experimental build from Ubuntu Raring. I tried to do it that way at one point, but it didn't work. Not to mention I'd prefer (but am not required) to stick with the mainstream builds. Last thing I need is for the experimental builds to throw up in 2 months because of some grub bug and I'm left really in trouble because the experimental builds are abandoned. tl;dr the mainstream PPA stuff is what I'd like to stick with if possible, and apparently it's totally possible.

[PoppaGeek]

Sorry I'm not being helpful having a high pain day. Hard to focus.

[josh1980]

I got a caffeine withdrawal headache myself.

No worries PoppaGeek. We can try tomorrow if you have more ideas. I'm just so frustrated over this. I've been trying to get this working since Ubuntu 14.04 came out in April and never got it working.

[PoppaGeek]

The caffeine headache is not so hard to fix.

I understand the frustration. You have your mind set on something and can't get it to work. Been there many times. Maybe start all over again? Maybe ya missed something.

I'll do more reading tomorrow.

[josh1980]

Oh, i've tried over and over. Probably started from scratch 20+ times. Tried VMs. Tried bare metal. I'm convinced the instructions are broken or something is seriously misunderstood by me. Considering how many people claim its broken in the notes for that page I tend to think it's not a "me" thing and that it's not complete.

[PoppaGeek]

Some package probably got updated along the way and the directions were never updated. That is the difficult thing about PPAs. Sometimes they get out of sync with the distro's packages. Very frustrating for those trying to get it to work.

But I am sure you learned from the experience so not all a waste.

I know you have your mind set on ZFS but could you use one disk or partition for OS and a ZFS part or disk for /home and /var/lib/boinc? Easy to re install OS and not bother data.

[josh1980]

Yes, but the problem I've had is that the OS gets corrupted and them I'm back at square one. I wanted a way to know when the OS or any files the machine has are corrupt so I can just do a restore from a known backup.

I don't particularly care of a BOINC WU gets fubared. It'll work itself out on its own. But corrupt OS files... those only get worse.

Edit: And with the zfs property copies=2 I could even recover from occasional flash failure of SSDs.

[jspace]

I'm curious now.
I've only seen root on zfs on 10.04, I think. It might have been 11.04.

Edit:
Eventually got to 5.6 and get a 404 with apt-get update. (Display keeps puking. Got SSH working though)

Ign http://ppa.launchpad.net trusty/main Translation-en
W: Failed to fetch http://ppa.launchpad.net/zfs-native/grub/ubuntu/dists/trusty/main/binary-amd64/Packages  404  Not Found

E: Some index files failed to download. They have been ignored, or old ones used instead.

[D_A]

You might need to change that address to correspond to a different version as "trusty" isn't there.

[jspace]

Yeah, I hadn't looked last night. Figured I would let kernel sources download & go to bed. I'm not an Ubuntu user, so I'm not sure how to change that yet.

josh1980
Did you make any changes to that PPA? It looked like GRUB didn't know about ZFS because support wasn't compiled in. What's the package name? It tells you how to get it in the troubleshooting section under "(iii) GRUB Installation."

[josh1980]

I've changed no PPAs. I'm not 100% convinced that GRUB doesn't know about ZFS. If I get a list of files inside the grub install directory it has files that have zfs in the name. I don't have them handy right now or I'd list them.

To be honest, the whole section "(iii) GRUB installation" is totally and utterly f'ed up because those don't exist for trusty and those commands don't work. Not because of a compatibility issue and such, but because they never worked on 14.04. There is no grub-zfs for trusty as D_A pointed out. But the guide never says to actually install grub-zfs except.. if you get an error. Which is hilarious because it says "verify the zfs enhanced grub PPA is installed". Why would I *verify* a PPA that isn't ever mentioned that it needs to be installed?

Te line that says to do "apt-add-repository --yes ppa:zfs-native/grub" is pointless because there is no trusty. So clearly that step should be removed (replaced with something else???). Of course, I did it anyway.

The funny part is someone fixed the python-software-properties package problem that I noticed a while back 3 days ago. But the guide doesn't work (or I'm an idiot, or both). So why is someone maintaining that kind of minutia when there's things that say "verify the PPA is installed" but you didn't have steps to install it to begin with!

I'm fairly sure that zfs support is in grub despite the commands not working because if I do a directory listing of /boot/grub/i386 after installing grub I find 3 files, zfs.mod, zfsinfo.mod and zfscrypt.mod.

I'm sorry but I don't know what you mean when you ask for the package name. I did the apt-get installs per the steps between 1 and 5.6, so grub-common and grub2 are installed.

per the output of apt-cache policy grub-common and grub-pc I see that:

grub-common is at version 2.02~beta2-9
grub-pc is at version 2.02~beta2-9

Sorry, I'm not particularly familiar with apt-get, so if that's not what you want please let me know and I can run that for you.