It has been a long and interesting weekend of fixing computers.
Adopt the pose: sit cross-legged on the floor surrounded by 9 hard-drives – wait, I need another one, make it 10 hard-drives – and the attendant spaghetti of SATA cables and plastic housings and fragments of case.
Funnily enough the need for screwdrivers has reduced over the years, albeit more than compensated by the cost of a case alone. I’m sure it never used to make for such a sore back either…
Anyway. Amidst the turmoil of fixing my main archive/work/backup server, I discovered a new OS.
For a few years now, I’ve been fond of ZFS – reliable as a brick, convenient as anything to use; I choose my OSes based on their ability to support ZFS, amongst other things. Just a quick
zpool create data /dev/ada1 /dev/ada2 zfs create data/Pictures
and that’s it, a new pool and filesystem created, another 1-liner command to add NFS sharing… Not a format or a mount in sight.
Of course, Linux has not been able to include ZFS in the kernel due to licensing considerations, so the various implementations (custom kernel; user-space FUSE module) have been less than desirable. So I’ve been using FreeBSD as server operating-system of choice. The most convenient way to control a plethora of virtual machines on a FreeBSD host seems to be to use VirtualBox – rather large and clunky nowadays.
However, a couple of weeks ago I stumbled across SmartOS, a new-to-me OS combining ZFS, DTrace and a Solaris/Illumos kernel, with both its own native Zones and Linux’s KVM virtualization.
There have been a few steps in this direction previously – most memorably was Nexenta, an opensolaris/illumos kernel with Debian packaging and GNU toolchain. That was a nice idea, but it lacked virtualization.
So, this weekend, with a storage server box rebuilt (staying with FreeBSD) and a whole new machine on which to experiment, I installed SmartOS.
Overall, it’s the perfect feature blend for running one’s own little cloud server. ZFS remains the filesystem of choice, DTrace has yet to be experimented with, and KVM is a breeze, mostly since Joyent have provided their own OS semi-installed images to work from (think: Docker, but without the Linux-specificity). The vmadm command shares a high-level succinctness with the zfs tools. Just import an image, make a JSON config file describing the guest VM and create an instance and it’s away and running with a VNC interface before you know it.
There’s one quirk that deserves special note so far. If you wish to use a guest VM as a gateway, e.g. via VPN to another network, you have to enable spoofing of IPs and IP forwarding on the private netblocks, in the VM config file.
"allow_dhcp_spoofing": "true",
"allow_ip_spoofing": "true",
"allowed_ips": [ "192.168.99.0/24" ]
[root@78-24-af-39-19-7a ~]# imgadm avail | grep centos-7 5e164fac-286d-11e4-9cf7-b3f73eefcd01 centos-7 20140820 linux 2014-08-20T13:24:52Z 553da8ba-499e-11e4-8bee-5f8dadc234ce centos-7 20141001 linux 2014-10-01T19:08:31Z 1f061f26-6aa9-11e4-941b-ff1a9c437feb centos-7 20141112 linux 2014-11-12T20:18:53Z b1df4936-7a5c-11e4-98ed-dfe1fa3a813a centos-7 20141202 linux 2014-12-02T19:52:06Z 02dbab66-a70a-11e4-819b-b3dc41b361d6 centos-7 20150128 linux 2015-01-28T16:23:36Z 3269b9fa-d22e-11e4-afcc-2b4d49a11805 centos-7 20150324 linux 2015-03-24T14:00:58Z c41bf236-dc75-11e4-88e5-038814c07c11 centos-7 20150406 linux 2015-04-06T15:58:28Z d8e65ea2-1f3e-11e5-8557-6b43e0a88b38 centos-7 20150630 linux 2015-06-30T15:44:09Z [root@78-24-af-39-19-7a ~]# imgadm import d8e65ea2-1f3e-11e5-8557-6b43e0a88b38 Importing d8e65ea2-1f3e-11e5-8557-6b43e0a88b38 (centos-7@20150630) from "https://images.joyent.com" Gather image d8e65ea2-1f3e-11e5-8557-6b43e0a88b38 ancestry Must download and install 1 image (514.3 MiB) Download 1 image [=====================================================>] 100% 514.39MB 564.58KB/s 15m32s Downloaded image d8e65ea2-1f3e-11e5-8557-6b43e0a88b38 (514.3 MiB) ...1f3e-11e5-8557-6b43e0a88b38 [=====================================================>] 100% 514.39MB 38.13MB/s 13s Imported image d8e65ea2-1f3e-11e5-8557-6b43e0a88b38 (centos-7@20150630) [root@78-24-af-39-19-7a ~]# [root@78-24-af-39-19-7a ~]# cat newbox.config { "brand": "kvm", "resolvers": [ "8.8.8.8", "8.8.4.4" ], "ram": "256", "vcpus": "2", "nics": [ { "nic_tag": "admin", "ip": "192.168.5.48", "netmask": "255.255.255.0", "gateway": "192.168.5.1", "model": "virtio", "primary": true, "allow_dhcp_spoofing": "true", "allow_ip_spoofing": "true", "allowed_ips": [ "192.168.99.0/24" ] } ], "disks": [ { "image_uuid": "d8e65ea2-1f3e-11e5-8557-6b43e0a88b38", "boot": true, "model": "virtio" } ], "customer_metadata": { "root_authorized_keys": "ssh-rsa AAAAB3NzaC1y[...]" } } [root@78-24-af-39-19-7a ~]# vmadm create -f newbox.config Successfully created VM d7b00fa6-8aa5-466b-aba4-664913e80a2e [root@78-24-af-39-19-7a ~]# ping -s 192.168.5.48 PING 192.168.5.48: 56 data bytes 64 bytes from 192.168.5.48: icmp_seq=0. time=0.377 ms 64 bytes from 192.168.5.48: icmp_seq=1. time=0.519 ms 64 bytes from 192.168.5.48: icmp_seq=2. time=0.525 ms ... zsh, basalt% ssh root@192.168.5.48 Warning: Permanently added '192.168.5.48' (ECDSA) to the list of known hosts. Last login: Mon Aug 3 16:49:24 2015 from 192.168.5.47 __ . . _| |_ | .-. . . .-. :--. |- |_ _| ;| || |(.-' | | | |__| `--' `-' `;-| `-' ' ' `-' / ; Instance (CentOS 7.1 (1503) 20150630) `-' https://docs.joyent.com/images/linux/centos [root@d7b00fa6-8aa5-466b-aba4-664913e80a2e ~]#
And there we have a new guest VM up and running in less than a minute’s effort.
Infrastructure and development environments recreated from scratch (partly thanks to storing my ~/etc/ in git) in under an hour.
I’m still looking for the perfect distributed filesystem, however…