Thursday 27 December 2012

Ubuntu 12.10 on an SSD

After having cloned the preloaded disk content to a small old disk freeing up the 500GB hard disk the W530 came with, I also replaced the original disk with a 256MB SSD. This SSD is sold by Lenovo (P/N: 0A65620), and is a rebranded Samsung PM830 self encrypting disk featuring FIPS certified hardware based full disk encryption with AES-256. It allows me to set up an ITCS 300 compliant thinkpad without any impact on performance or battery life imposed by software based full disk encryption such as LUKS. This post will describe the initial steps I took to make the system more SSD-friendly.

Firmware setup

I entered firmware setup and confirmed the disk is initialized in AHCI mode instead of IDE. AHCI stand for Advance Host Controller Interface and is the technical standard defining the operation of SATA host but adapters. It offers features Parallel ATA (IDE) does not offer, such as hot-plugging and native command queueing. Under Linux, AHCI is required to use TRIM support - to be covered later. Many systems ship with IDE legacy compatibility configured by default as Windows XP had installation issues when AHCI is enabled, so it is always useful to double check this setting. In my case, AHCI was selected by default.

I ran memtest, then I returned to the firmware setup and I verified that the firmware of my w530 is up to date. Normally, updating the firmware should be one of the first steps, but one has to make sure the memory modules pass memory check routines before applying any firmware updates - never ever attempt such an update on an unstable system as you could easily brick your box. In my case the firmware seemed to be up to date.

I issued other minor changes such as setting the system time, disabling wake-on-lan and enabling virtualization (both vt-x and vt-d) - these are not related to SSD in any way. I also ensured to boot in UEFI mode instead of BIOS emulation mode.

Before booting from the SSD

I booted an Ubuntu 12.10 live usb stick, quickly ran hdparm and tested the disk via smartmontools. Those who do not know S.M.A.R.T stands for disk self monitoring, analysis and reporting technology should definitely read up on the topic.


sudo hdparm -i /dev/sda # you can see 'Model=SAMSUNG MZ7PC256HAFU-000L7, FwRev=CXM72L1Q'
sudo hdparm -I /dev/sda # more detailed info about the disk
sudo apt-get install aptitude # aptitude is my choice of package management frontend on both servers and desktops. May look old school but is very powerful.
sudo aptitude install smartmontools
smartctl -a /dev/sda | less -S
smartctl -t short /dev/sda # wait until the test completes, check progress/result with the command above
smartctl -t long /dev/sda # same as above

A very short summary on performance and lifetime considerations for SSDs: contrary to traditional rotating hard disks, where the smallest writeable unit is usually one sector, SSDs organize their 4k pages into so called erase blocks. Instead of overwriting a single sector, an SSD must erase and reprogram a full block - typically 512KB or more depending on the SSD. As an immediate consequence, many small writes may sum up to large amount of disk space being erased and rewritten. This is called write amplification. Given that each flash cell can only be rewritten for given number of times (5000 for standard MLC NAND flash), reducing the number or writes and the size of data written will extend the lifetime of your SSD and maintain good performance in the long run. More information on the topic can be found here.

Before launching the installer, I manually created a GPT partition table and the partitions with gdisk. In order to do so, first I had to enable the universe repository - launch 'Software Sources', and enabled the second checkbox. Then I installed the gdisk package with aptitude. I decided to create a small 64MB EFI System partition, 32GB root partition, 4G swap and a separate partition for home taking the remaining unallocated space. Having read up on SSD partitioning and erase block size calculations earlier, I decided to go with 2048 sector aligned partitions - nowadays this is the default in gdisk.


sudo gdisk -l /dev/sda # partition listing after partitioning
Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048          133119   64.0 MiB    EF00  EFI System
   2          133120        67241983   32.0 GiB    0700  Linux filesystem
   3        67241984        75630591   4.0 GiB     8200  Linux swap
   4        75630592       500118158   202.4 GiB   0700  Linux filesystem
sudo mkfs.vfat -F 32 -n EFISYS /dev/sda1 # create the fat32 filesystem

Note that I only formatted the EFI System partition, not the other partitions. Then, I ran the installer, chose "Something else" at the disk setup menu, mapped the partitions I created before. Both root and home will have ext4 filesystem. Installation competed in just a very few minutes. Note that I did not reboot, after installation, but made some additional adjustments to reduce the wear of the ssd. I made sure the ext4 partitions will always be mounted with discard, relatime and an increased commit interval.

  • discard enables TRIM support - the kernel will tell the SSD which blocks are no longer in use, allowing the SSD to optimize and improve both performance and lifetime in the long run.
  • relatime tells the kernel to only update the access time of files (small disk writes) if the previous access time is older than modification or creation time. Many sources recommend noatime instead, but I stick to relatime since it does not break programs like mutt that rely on comparing atime with mtime. See the related discussion for further insights here.
  • commit=120 - when using ext4 filesystem, the kernel will sync in-memory cached data every 5 seconds to disk by default. Increasing this to 2 minutes will boost performance and save many small writes. As the thinkpad is battery backed, I find is a safe option. Even in the case of power loss (or complete kernel lock ups or crashed) the filesystem will not be damaged, I only risk loosing up to 2 minutes of file modifications - thanks to journalling.
Many sources suggest adding these options to your fstab, but instead of doing so, one can specify default mount options on the filesystems themselves with the command tune2fs. This way, mounting the partitions externally - e.g. from a livecd - will automatically pick up and use these mount options unless explicitly overridden. Read the manual for mount and tune2fs. In addition to these, I also wanted to mount a tmpfs over /tmp - allowing it to take up to 25% of my RAM.


sudo su
tune2fs -o discard /dev/sda2 # one could also use -E mount-opts=discard which is newer and less documented
tune2fs -o discard /dev/sda4
# relatime is employed by default on Ubuntu 12.10 so no need to set it up
# default commit interval cannot be specified with the -o option, so enable it in fstab
# 
# mount the root partition and edit
mount /dev/sda1 /mnt; nano /mnt/etc/fstab # add commit=120 to root and home
echo "tmpfs        /tmp        tmpfs        nosuid,nodev,size=25%        0 0" >> /mnt/etc/fstab
umount /mnt

After applying these tweaks, I booted into my freshly installed Ubuntu. On the first boot I experienced a black background with a mouse pointer instead of the lightdm screen. I switched to a virtual terminal (press Ctrl-Alt F1), logged in, installed aptitude, updated packages and powered off. The next boot worked fine. (From time to time, the black screen came back - I fixed it, this will be covered in a separate post.)

Post installation tweaks

One additional tweak I strongly recommend is changing the default I/O scheduler used by Linux kernel. The "Completely Fair Queuing" is optimized for rotating disks where seek time is an important factor. Since SSDs do not have seek times varying based on the on disk location of the data, it is best to avoid the 'CFQ' scheduler and use 'deadline' or 'noop' instead. Many sources suggest hard-coding the scheduler in kernel boot options or scripts, whereas I prefer a dynamic approach that will set 'CFQ' for rotating disks and 'deadline' for non-rotating disks automatically. The following command will create udev rules that automatically assign the right scheduler based on the rotational properety of the disk.


cat <<EOF | sudo tee /etc/udev/rules.d/60-io-scheduler.rules
# set deadline scheduler for non-rotating disks
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="deadline"

# set cfq scheduler for rotating disks
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="cfq"
EOF

Before adding this rule, I recommend checking the current scheduler - it my case it turned out that Ubuntu 12.10 was automatically selecting 'deadline' anyway, so I did not apply the tweak mentioned above on this particular setup.


cat /sys/block/sda/queue/scheduler
noop [deadline] cfq

Last but not least, concerns regarding the swap partition. The old-school approach is to create a swap partition that has at least the size of your memory. I do not want to hibernate - only suspend to ram, and I feel that 16GB of swap is simply overkill. Many sources recommend completely leaving out the swap partition, but as this laptop is going to be a workhorse, I opted for creating a 4G swap partition on the SSD. Recent Linux kernels send TRIM commands automatically if supported by the disk the swap partition is located on. Additionally, I tuned sysctl variables to decrease the swappiness and the tendency of the kernel to swap out directory entry and inode caches. I expect these settings to forbid swapping until really-really necessary.


# transient changes until reboot
echo 1 | sudo tee /proc/sys/vm/swappiness
echo 50 | sudo tee /proc/sys/vm/vfs_cache_pressure

# persistent changes
cat <<EOF | sudo tee -a /etc/sysctl.conf
vm.swappiness=1
vm.vfs_cache_pressure=50
EOF

As a closing note, end-user behaviour also matters much. I try to pay attention to creating transient files under /tmp - if you compile a lot, this matters much. One should find a healthy balance between being SSD-aware and being too paranoid about it. I investigated methods to decrease disk writes caused by syslog - many suggest to mount a tmpfs over /var/log which would mean all your logs are lost when you reboot, making any kind of audit or post mortem debugging impossible. I ended up sticking to the commit=120 mount option after some calculations. You should do the math and enjoy your disk. As with any kind of disk, check the SMART attributes from time to time, and make sure you have take backups on a regular basis.

Monday 24 December 2012

Hard disk passwords and interoperability

Right after unboxing my w530 and changing some settings in the firmware setup menus, before even booting from the hard drive I wanted to clone the preloaded closed source operating system to an old 100GB. I do not want a dual boot box, but a backup of the virgin preloaded system might come handy in the future - e.g. if I decide to pass on the box some day to someone else...

The brand new w530 did not accept the hard disk password I have found on a sticky note attached to the old 100GB sata disk. This disk I have found lying around, it has not been touched for the last 4-5 years - maybe more. I inherited it from a colleague, and to my knowledge at that time I verified the password on the sticky note was working.

I wanted to get the drive working and ready for use - preferably also to check the contents before wiping it. In order to do so, I wanted to collect information about the disk I could use to find out why the password is not working. Based on the SMART self test and error logs, I could have found out how long ago the drive was actively used. Even if I cannot recover the password, I should be able to set a new password - which would mean losing all the contents, but at least having a functional hard disk. More information on ATA passwords and security: https://en.wikipedia.org/wiki/AT_Attachment#HDD_passwords_and_security.

The tool hdparm is a great tool to collect detailed information about the problematic hard disk, and it is also able to issue ATA security commands, such as unlock, secure-erase or change both user and master passwords...

Unfortunately, if this disk is attached, and your password is not accepted after 5 attempts, the system shuts down - it does not give you any chance to boot from a livecd or usb and tinker with hdparm.

My first idea was booting a livecd, then hot-plugging the SATA disk - this would definitely work with a hard disk bay (undock the optical drive, and dock the hard disk bay) but I do not have a bay that is compatible with w530. I tried hot-plugging into the internal slot for the primary hard disk, but that did not work - after attaching the disk, I ran dmesg, but it did not show any event or activity related.

Nevertheless, I do have a hard disk bay compatible with my old t60p, so I booted quickly and investigated how I could safely unplug my optical drive and insert the drive bay with the problematic SATA disk.


$ for D in $(ls /sys/devices/platform/dock.?/type ); do echo $D; cat $D; done;
/sys/devices/platform/dock.0/type
dock_station
/sys/devices/platform/dock.1/type
battery_bay
/sys/devices/platform/dock.2/type
ata_bay
$ cat /sys/devices/platform/dock.2/docked # the output above shows dock.2 is the drive bay
1
$ echo 1 | sudo tee /sys/devices/platform/dock.2/undock
$ dmesg | tail  # shows the optical drive was powered off and is safe to detach
After inserting the bay, I saw a lot of disk read errors - due to the fact that the disk is locked and cannot be accessed without unlocking it with the password. With hdparm I verified the security the disk is locked.

sudo hdparm -I /dev/sdb | grep -A 10 Security

I quickly read https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase and tried to unlock the drive with using the provided password both as a user password and master password, without luck. I carried on with some brute-forcing until I exceeded the number of allowed attempts. This is indicated by the line ‘expired: security count’ in the output of the hdparm command listed above. Undocking and redocking the bay will give another handful of attempts, but this kind of bruteforcing is not very efficient...

From the output of hdparm I also found out the factory master password was not changed “Master password revision code = 65534” which is equal to 0xFFFE in hex. I quickly used google to find out the factory master password for hitachi disks, and multiple sites mentioned 32 spaces but somehow I was not able to unlock or even secure-erase the disk. (If maximum security mode is used - like in my case - the master password cannot be used to unlock a disk locked with a user password, but it should be able to secure erase the disk - however it did not work.)

I have found some information, that some buggy controllers or BIOSes mess up the password under circumstances, and if the 32 bytes dedicated to hold the hard disk password contain non-printable bytes then strange things will happen. Also, depending on the actual BIOS, you may or may not be able to set/input an empty password - especially, Lenovo is mentioned to be unable to unlock a disk locked with an empty password. (Hitting enter at the password prompt might mean no password or empty password as well... I guess this might be the root cause.)

It also became obvious at that point, that some BIOSes might not want to send/store the provided hard disk password to the disk, but might want to scramble/obfuscate/transform it first by design. While this is a reasonable decision from the security point of view, it clearly leads to interoperability problems - putting the same disk into another box and providing the same hard disk password will just not work unless the two boxes use the exact same (including none) scrambling algorithm.

This made me want to boot my old t60p with the hard disk bay inserted at power on time as opposed to hot-plugging it after boot. Magically, the hard disk password was recognized and accepted, and the disk unlocked. Although I could not change the security settings form within the operating system, as the ATA security configuration was frozen by the BIOS, I could easily remove the hard disk password from within the BIOS menu and confirm from within Linux that the change indeed happened. Problem solved.