A bit of a hardware upgrade

Four and a half years ago I bought basalt,. a Lenovo Thinkpad W520 notebook.
Named for being black and geometrically regular, he was not cheap, but then the hardware was chosen for longevity. He did everything – mostly sitting on the desk being a workstation, for both photo-processing and work, with added portability. And he was fast… very fast.

He’s done sterling service, but the time has come for a bit of an upgrade. Since I haven’t done so for a few years now, I built the replacement workstation by hand.

So, after much shopping (mostly on Amazon with a couple of trips to the local Maplin’s), meet rhyolite – named because the red fan LEDs in the case remind me of the pink granite rock in Glen Etive.

basalt rhyolite
RAM  16G 64G
CPU  i7-2720QM @ 2.20GHz i7-6800K  @ 3.40GHz
HD 1TB 2.5″ 2*2T SATA, 64GB SSD, 240GB SSD
Display 15″ 1080p 24″ 4K
Time to process one Olympus Pen-F hi-res ORF file from RAW to TIFF 5min50s 1min6s
Time to rebuild Apache Spark from git source 28min15s 6min28s

It’s funny how oblivious we can be of  machines slowing down and software bloating over time, rather like the proverbial boiling lobster and then when I look up, things can be done 4-5x as fast.

It’s really funny when you copy your entire home directory across wholesale and see what used to be a full-screen maximized Firefox window now occupying barely a quarter of the new display.

How many photo thumbnails can you fit in a 4K display?

There’s only one downside: sometimes I miss having a trackpad in the middle of the keyboard area…

Olympus Pen-F: analysis and hands-on

Mine, my only, my preciousss!

Mine, my only, my preciousss!

There’s only one camera shop left in Perth and yesterday I was their first customer to buy the Olympus Pen-F. It’s not released on Amazon until Monday…

So it’s been the talk of the web. It’s undeniably a beautiful camera, with a rangefinder design reminiscent of its film camera predecessor of the same name. I’ve spent much of the last couple of days putting it through its paces.

I’m pleased to report it has a reasonable sturdy weight to it thanks to the aluminium/magnesium alloy shell. It feels just great in one’s hands. Well, my hands, anyway.

How does it perform?

First, software: it’ll be a little while before Adobe support it, but for us open-source fans, RawTherapee already supports the new .ORF RAW images in both 20-megapixel and hi-resolution 80-megapixel modes. (It seems to be the only open-source raw-converter to do so, at the time of writing.)

Second, I’ve been keeping dark-frame reference images for a few cameras, so I can compare its noise levels against predecessors and competitors. (Method: put the lens-cap on, place camera upside-down on desk, and then at every ISO from 6400 to 100 and below, at various shutter-speeds from 1/250s to 30s, take a couple of photos. A quick Python script reads the images, converts to greyscale and outputs various properties including the average lightness value per pixel.) Results aggregated by camera model and ISO with median-average.)

log-log plot of luminosity noise for various camera models by ISO

log-log plot of luminosity noise for various camera models by ISO

The graph shows 3 things, none surprising but pleasant to see confirmation:

  1. The Lumix GH2 really was awful for sensor quality. Its ISO160 is the Sony NEX-7’s ISO 800.
  2. There is a reasonable progression from Lumix GH2 to Sony NEX-7 to Olympus E-M1 and Pen-F; at native resolution, the E-M1 and Pen-F are much of a muchness for L-channel noise.
  3. The real game-changer is the high-resolution mode on the Pen-F – good, since this is one of the main reasons I bought the camera. This is to be expected: it uses the in-body stabilization to blend 8 images, four of them offset by 1 pixel (so it has full chroma bit-depth at every pixel) and four offset by half a pixel (so it has double the linear luminosity resolution). The result is about 1.5 stops better noise-performance.

Hi-res mode comes with a few limitations: the maximum ISO is 1600; aperture only goes down to f/8; shutter speeds only as long as 8s. You can enable picture modes such as natural, vivid, monochrome etc; however it does not work with bracketing or HDR mode: for HDR, it’s best to work manually.

One interesting point in favour of high-resolution mode: it does not stress the lens’s performance; at no point does it take an image of more than the native resolution, so if the MTF curves of contrast vs spatial frequency are adequate for 20 megapixels, the lens is also good enough for 50-80 (more later).

Hands-on: what’s it like in the field?

I took both current cameras out for a drive around the Highlands of Perthshire this afternoon, the NEX-7 with its kit 18-55mm lens and the Pen-F attached to a new 17mm f/1.8 prime.

Here’s a quick overview of four images: one from the NEX-7, two straight from the Pen-F’s own JPEGs out of camera (just downsized for presentation here) and one converted in RawTherapee:

My first real image from the Pen-F: Amulree and Strathbraan Parish Church at the end of Glen Quaich.

The Olympus images all have the same white-balance; I set the ORF and NEF files to the same whitebalance as the Pen-F chose automatically. It’s interesting that the sky is a proper shade of blue in the converted RAW; this might be a difference of profile between RawTherapee and the camera’s own internal JPEG engine. (Fine by me – I much prefer solid blue skies to cyan ones.)

What about the details?

Now this is a hard question. No doubt the resolution suffers and 80MPel is rather optimistic. Here we compare the NEX-7 against the 20MPel and 80MPel RAW files with no sharpening or detail controls enabled at all:

Obviously the 80MPel image is going to take a lot ofsharpening to bring it up to scratch.

My preferred sharpening algorithm with the NEX-7 has long been deconvolution. However, despite tweaking the radius, strength, dampening and iterations controls a lot, RawTherapee doesn’t seem to have the strength to sharpen the 80MPel hi-res RAW.

The out-of-camera JPEG appears to have a cruder sharpening algorithm with a lot of local contrast, so I switched to unsharp mask. At 80MPel, with enough USM to render the individual stones in the dry stone wall adequately sharp, this reveals serious anti-aliassing issues around straight-line edges:

Pen-F edge-aliassing artifacts

Pen-F edge-aliassing artifacts

These artefacts disappear when downsized to 50MPel in camera. Then again, the in-camera JPEG conversion is very contrasty and over-sharpened, so there’s room for improvement all-round:

In-camera JPEG conversion (20MPel) vs RawTherapee downscaled from 80MPel RAW

In-camera JPEG conversion (20MPel) vs RawTherapee downscaled from 80MPel RAW

Update: by adjusting the USM parameters, I’ve removed much of the artifacts around edges. There are still problems where blades of grass have moved between source frames, but static objects with strong outlines no longer cause such problems.

disable edge-only and halo-detection, just use less USM to start with

disable edge-only and halo-detection, just use less USM to start with

There are other demosaicing algorithms in RawTherapee and other ways of sharpening (newly added wavelet support). Plenty of scope for experimenting to get the best out of this thing.

What’s the verdict?

I absolutely love using it. Where other cameras drive the megapixel race by giving increasingly large images for everything (40MPel+), this camera ticks along at a more than adequate native 20 megapixels for most purposes with the option of taking one’s time to crank out the tripod to make absolutely huge images. This suits me fine – I’ve spent so long doing similar image-stacking/super-resolution trickery as part of the post-processing workflow, it’s great to have the camera do it instead.

The in-body stabilization is excellent. I managed to hand-hold 10 frames at 1/10s at the Falls of Dochart, twice, without a single blurry image.

There are lots of other features to look forward to – I’ve never had the patience to try focus-stacking, but moving it in-camera is very appealing.

With a subtle adjustment to the brighter blues to cater for skies, I could probably live off the camera’s own JPEGs for many purposes.

New lenses and filter adapter rings have been ordered…

Gratuitous sample images

As if the above weren’t nice enough, here are three more images from this afternoon.

Discovering SmartOS

Revolutionize the datacenter: ZFS, DTrace, Zones, KVM

What 22TB looks like.

It has been a long and interesting weekend of fixing computers.
Adopt the pose: sit cross-legged on the floor surrounded by 9 hard-drives – wait, I need another one, make it 10 hard-drives – and the attendant spaghetti of SATA cables and plastic housings and fragments of case.
Funnily enough the need for screwdrivers has reduced over the years, albeit more than compensated by the cost of a case alone. I’m sure it never used to make for such a sore back either…
Anyway. Amidst the turmoil of fixing my main archive/work/backup server, I discovered a new OS.
For a few years now, I’ve been fond of ZFS – reliable as a brick, convenient as anything to use; I choose my OSes based on their ability to support ZFS, amongst other things. Just a quick
zpool create data /dev/ada1 /dev/ada2
zfs create data/Pictures
and that’s it, a new pool and filesystem created, another 1-liner command to add NFS sharing… Not a format or a mount in sight.
Of course, Linux has not been able to include ZFS in the kernel due to licensing considerations, so the various implementations (custom kernel; user-space FUSE module) have been less than desirable. So I’ve been using FreeBSD as server operating-system of choice. The most convenient way to control a plethora of virtual machines on a FreeBSD host seems to be to use VirtualBox – rather large and clunky nowadays.
However, a couple of weeks ago I stumbled across SmartOS, a new-to-me OS combining ZFS, DTrace and a Solaris/Illumos kernel, with both its own native Zones and Linux’s KVM virtualization.
There have been a few steps in this direction previously – most memorably was Nexenta, an opensolaris/illumos kernel with Debian packaging and GNU toolchain. That was a nice idea, but it lacked virtualization.
So, this weekend, with a storage server box rebuilt (staying with FreeBSD) and a whole new machine on which to experiment, I installed SmartOS.
Overall, it’s the perfect feature blend for running one’s own little cloud server. ZFS remains the filesystem of choice, DTrace has yet to be experimented with, and KVM is a breeze, mostly since Joyent have provided their own OS semi-installed images to work from (think: Docker, but without the Linux-specificity). The vmadm command shares a high-level succinctness with the zfs tools. Just import an image, make a JSON config file describing the guest VM and create an instance and it’s away and running with a VNC interface before you know it.
There’s one quirk that deserves special note so far. If you wish to use a guest VM as a gateway, e.g. via VPN to another network, you have to enable spoofing of IPs and IP forwarding on the private netblocks, in the VM config file.
      "allow_dhcp_spoofing": "true",
      "allow_ip_spoofing": "true",
      "allowed_ips": [ "192.168.99.0/24" ]
[root@78-24-af-39-19-7a ~]# imgadm avail | grep centos-7 
5e164fac-286d-11e4-9cf7-b3f73eefcd01 centos-7 20140820 linux 2014-08-20T13:24:52Z 
553da8ba-499e-11e4-8bee-5f8dadc234ce centos-7 20141001 linux 2014-10-01T19:08:31Z 
1f061f26-6aa9-11e4-941b-ff1a9c437feb centos-7 20141112 linux 2014-11-12T20:18:53Z 
b1df4936-7a5c-11e4-98ed-dfe1fa3a813a centos-7 20141202 linux 2014-12-02T19:52:06Z 
02dbab66-a70a-11e4-819b-b3dc41b361d6 centos-7 20150128 linux 2015-01-28T16:23:36Z 
3269b9fa-d22e-11e4-afcc-2b4d49a11805 centos-7 20150324 linux 2015-03-24T14:00:58Z 
c41bf236-dc75-11e4-88e5-038814c07c11 centos-7 20150406 linux 2015-04-06T15:58:28Z 
d8e65ea2-1f3e-11e5-8557-6b43e0a88b38 centos-7 20150630 linux 2015-06-30T15:44:09Z 

[root@78-24-af-39-19-7a ~]# imgadm import d8e65ea2-1f3e-11e5-8557-6b43e0a88b38 Importing d8e65ea2-1f3e-11e5-8557-6b43e0a88b38 (centos-7@20150630) from "https://images.joyent.com" 
Gather image d8e65ea2-1f3e-11e5-8557-6b43e0a88b38 ancestry 
Must download and install 1 image (514.3 MiB) 
Download 1 image [=====================================================>] 100% 514.39MB 564.58KB/s 15m32s 
Downloaded image d8e65ea2-1f3e-11e5-8557-6b43e0a88b38 (514.3 MiB) ...1f3e-11e5-8557-6b43e0a88b38 [=====================================================>] 100% 514.39MB 38.13MB/s 13s 
Imported image d8e65ea2-1f3e-11e5-8557-6b43e0a88b38 (centos-7@20150630) 
[root@78-24-af-39-19-7a ~]# 

[root@78-24-af-39-19-7a ~]# cat newbox.config 
{
  "brand": "kvm",
  "resolvers": [
    "8.8.8.8",
    "8.8.4.4"
  ],
  "ram": "256",
  "vcpus": "2",
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "192.168.5.48",
      "netmask": "255.255.255.0",
      "gateway": "192.168.5.1",
      "model": "virtio",
      "primary": true,
      "allow_dhcp_spoofing": "true",
      "allow_ip_spoofing": "true",
      "allowed_ips": [ "192.168.99.0/24" ]
    }
  ],
  "disks": [
    {
      "image_uuid": "d8e65ea2-1f3e-11e5-8557-6b43e0a88b38",
      "boot": true,
      "model": "virtio"
    }
  ],
"customer_metadata": {
    "root_authorized_keys":
"ssh-rsa AAAAB3NzaC1y[...]"
  }

}
[root@78-24-af-39-19-7a ~]# vmadm create -f newbox.config 
Successfully created VM d7b00fa6-8aa5-466b-aba4-664913e80a2e 
[root@78-24-af-39-19-7a ~]# ping -s 192.168.5.48 
PING 192.168.5.48: 56 data bytes 
64 bytes from 192.168.5.48: icmp_seq=0. time=0.377 ms 
64 bytes from 192.168.5.48: icmp_seq=1. time=0.519 ms 
64 bytes from 192.168.5.48: icmp_seq=2. time=0.525 ms ... 

zsh, basalt% ssh root@192.168.5.48 
Warning: Permanently added '192.168.5.48' (ECDSA) to the list of known hosts.
Last login: Mon Aug  3 16:49:24 2015 from 192.168.5.47
   __        .                   .
 _|  |_      | .-. .  . .-. :--. |-
|_    _|     ;|   ||  |(.-' |  | |
  |__|   `--'  `-' `;-| `-' '  ' `-'
                   /  ;  Instance (CentOS 7.1 (1503) 20150630)
                   `-'   https://docs.joyent.com/images/linux/centos

[root@d7b00fa6-8aa5-466b-aba4-664913e80a2e ~]# 

And there we have a new guest VM up and running in less than a minute’s effort.

Infrastructure and development environments recreated from scratch (partly thanks to storing my ~/etc/ in git) in under an hour.

I’m still looking for the perfect distributed filesystem, however…

Transitions

Panasonic Lumix GH2Nearly 3 years ago I spent the best part of 2 hours one afternoon in PCWorld, looking to kick the Canon habit and vacillating between a Nikon (as I recall, the D3100) and the Panasonic Lumix GH2.

On the one hand, the Nikon had excellent image-quality, but its usability was let-down drastically by the lack of a dedicated ISO button (you could pretend, by reassigning the one “custom” button – so why bother with either?) and it had what I felt was a patronizing user-interface, showing a graphic of a closing lens iris every time one changed the aperture (as though I hadn’t learned anything in the last 10 years spent working on my photography).

On the other hand, the Lumix GH2 had less than stellar image-quality, but the user-interface won me over.

Over the last 2 years, the ergonomics fitted my style like a glove: coming from film, including medium-format with waist-level finders, I find it most natural to operate looking down at the articulated LCD panel in live-view mode. I had sufficient custom presets for two aperture-priority-mode variations of black and white (one square, one 3:2, both with exposure-bracketing set to 3 frames +/-1EV) and a third, a particular colour “film emulation” and manual mode at ISO160 for long exposures, with bracketing set to 5 frames for more extreme HDR. With those 3 modes, I could cover 95% of my subject-matter from woodland closeup to long-exposure seascape and back, at the flip of a dial.

I learned to appreciate its choice of exposure parameters (normally well-considered), and to overcome the sensor’s foibles – it made an excellent test-case for understanding both high-ISO and long-exposure sensor noise and its limited dynamic range increased my familiarity with HDR, panoramas and other multi-exposure-blending techniques (all hail enfuse!). Coupled with the Pentacon 50mm f/1.8 lens, it made for some excellent closeup photos. As a measure of how workable the kit is, I once took every camera I then possessed – including medium- and large-format film – to Arran for a photo-holiday, and never used anything apart from the GH2 for the whole week.

If this all sounds like it’s leading up to something, it is. There is a long-established idea in photographer circles that gear-acquisition-syndrome (GAS), or the buying of new equipment for the sake of it or in order that it might somehow help one take better photos, is delusional. To some extent that’s right, but the flip-side is that any one camera will impose limitations on the shots that can be achieved. So I’ve established the principle that, if one can explain 3 things a camera can allow you to do better, the acquisition is justifiable.

And so I’ve switched. The new camera is a Sony NEX7[Amazon]; even though the model is barely younger than the GH2, it still has a vastly superior sensor that will give me larger images, better dynamic range and narrower depth-of-field. Indeed, at two years old, it’s still punching above its weight despite the pressure from some of the larger dSLRs to have come out since.Wee Waterfall (2)

One of the things I learned from the GH2 is that it always pays to understand one’s equipment. For this reason, the first 100 frames shot on the NEX-7 fell into 4 kinds:

  1. studying noise with varying ISO in a comparatively low-light real-world scene (stuff on bookshelves in the study – good to know how noise and sharpness interplay in both the darkest shadows and midtones)
  2. building a library of dark-frame images at various ISO and shutter-speed combinations (taken with the lens-cap on, for a theoretically black shot – any non-zero pixels are sensor noise)
  3. building a library of lens correction profiles – taking images of a uniform out-of-focus plain wall to compare vignetting at various apertures and focal-lengths on both kit lenses
  4. studying kit-lens sharpness as a function of aperture – discussed previously.

Impressively, I could just load all these images into RawTherapee and easily move them into relevant directories in a couple of right-clicks, and from there I spent the rest of the evening deriving profiles for ISO noise and sharpness with automatic dark-frame-reduction and actually measured vignetting correction – because I know very well how much time it will save me in the future.

Despite having played with film cameras, I’m quite acutely aware of the change in sensor format this time: in moving from prolonged use of micro-4/3rds to APS-C, I can no longer assume that setting the lens to f/8 will give me everything in focus at the lens’s sweetspot, but have to stop-down to f/11 or even further. The tripod has already come into its own…

So there we go.

Oh, and the complete GH2 kit is for sale on ebay, if anyone wants to buy it!
Update 2014-02-02: the complete kit sold on eBay for a very reasonable sum!