Lenovo S650 with android 4.4.2 and google authenticator

I have finally found some spare hours to upgrade my Lenovo S650 from the 4.2.2 based S650_S119_131014 ROM to S650_S912_VIBEUI_V1.5_1419_5_ST_1F49 which is based on 4.4.2. As a first step, I have studied the contents of the zip file, especially the updater-script at /META-INF/com/google/android/. It quickly became obvious that I would loose my root access after installing this update, which I wanted to avoid.

The rules I have set for myself do not allow rooting the phone via windows based third party applications (some of which are known to install some souvenirs), and the common path of using the windows-only MediaTek flashing tool in the rooting process is also out of scope for me. For private reasons I decided to not publish information on how to properly root the lenovo ROM via command line, without any closed source, suspicious third party app. Rather than that, let us assume that the original JellyBean system already has root access, and our goal is to retain this during the upgrade.

While setting the immutable flag on the su binary via chattr +i /system/xbin/su would retain root access during minor updates, this is useless if the system partition is formatted. In our case, all partition except sdcard{0,1} would be formatted as it can be seen in the updater-script.

Adding the binary to the itself zip and amending the updates-script with a command to set proper SELinux context, ownership and permissions come as an intuitive solution. This can be achieved by the following line:

set_metadata("/system/xbin/su", "uid", 0, "gid", 0, "mode", 6755, "capabilities", 0x0, "selabel", "u:object_r:su_exec:s0")

One should note that the zip file is a singed jar, and any tinkering would cause signature verification to fail. Although there are various approaches for bypassing signature verification, no magic is needed due to the fact that the current system is already rooted. Custom recovery images allow to install updates without a [valid] signature.

Custom recovery

Readers who have familiarized themselves with the previous post should already know how to 'flash a custom recovery' with a single line.

dd of=/dev/block/mmcblk0 if=recovery.img bs=1024 seek=36352

As an alternative, one can use Mobileuncle MTK Tools to install a custom recovery - if one renames the file to recovery.img and places it into the root of the SD card, the app will automatically pick it up.

I have used TWRP recovery 2.7.0.0. The Lenovo S650 come in two variants, which have different partition layouts. When using pre-built recovery images, make sure to use an image compatible with CN (Chinese) or ROW (Rest Of the World) partition layout according to your device.

The first thing after installing the custom recovery was to test it - taking a TWRP backup of the current system. Actually, testing the backup process is only half of the job, since the recovery process is slightly more important...

Superuser application

While this post is not going to cover the process of rooting, the basic concept of how root permission is granted is outlines. The su binary, unlike in traditional Unices, will not ask for credentials but instead, communicate with a graphical application that will display the details of the 'switch user' request and allow the end user to approve or reject the request via the touchscreen.

As any flaw of the su binary or the superuser application, intentional or unintentional, may critically impact the security of the device, one should consider if this capability is needed at all, and if so, which implementation to chose.

While SuperSU and ChainsDD Superuser are common choices, I decided to go with Koushik Dutta's Superuser. Unlike the other two it is open source, so I have taken some time to go through the source of both the native binary and the app. Additionally it features PIN protection and logging, which typically would not be available in the free versions of other alternatives.

I have downloaded the zip file, and studied the updater-script. I modified the 4.4.2 ROM by adding the binary and the application and changing the scripts to ensure proper metadata is set on the binary.

Google apps

I have downloaded a signed intallable zip of minimal Google Apps for 4.4.2 - this only takes a few megabytes and allows me decide which apps fo google to install later.

The point of no return

I have observed that while the ROM updater-script does not replace the recovery, it installs a script that runs on normal boot and patches the recovery. In worst case, this could mean that I end up with a non-rooted system and my custom recovery, that would allow me to alter the system (install unsigned updates), would be erased and replaced by Lenovo's restricted recovery image. That would have been game over for me.

I have copied the vanilla ROM, the superuser installer and GApps installer zips to the SD card, along with my modified "superuser-included" ROM zip and rebooted to recovery. I performed an advanced wipe selecting user data, cache and dalvik-cache to be erased and then attempted to install the modified ROM.

In the middle of the process, I received an error notifying me that update-binary had failed. The second attempt yielded the same results, leaving the system in an unknown state. I payed attention not to reboot a single time until I was sure the system was properly installed and rooted, or at least the custom recovery image was still in place.

I had to fall back to 'plan B' and installed the vanilla ROM, but instead of rebooting, quickly installed the superuser installer zip as well. Once done, I opened a shell and manually checked the presence and metadata of the su binary and the superuser application. Then I rebooted.

Make space and install GApps

On first boot, I realised there was almost no space on the system partition, so I removed some of the apps like Youyue, GameWorld and Baidu, then took the opportunity to uninstall some preintalled non-system apps like Weibo, QQ and Lakala. I intentionally retained many system apps by Lenovo - in some cases they are really useful and feature rich alternatives to some Google components.

If one is concerned about privacy and decides not to go with Lenovo apps, then the very same person should also be worried about installing any Google component onto his device... Instead of engaging in a religious debate, look at what kind of traffic the various components cause, and block according to your findings.

Once there was 100MB free space on the system partition, I installed the minimal google apps package and rebooted - and singed into my google account. Play store services worked fine, and nothing gave me significant headache... until I tried to use google-authenticator...

Read on, the second part of the series is already available.

Backing up a stock Lenovo S650/MT6582 phone from shell

Right after unboxing my S650, before even booting into normal mode I wanted to clone the preloaded firmware. This device is my first personally owned android smartphone and I wanted to make sure I have a backup of the original state before tinkering or flashing a custom ROM. The rules, as implied by the above statement, were the following:

• Flashing a custom recovery was not allowed.
• The use of third party applications other than those that come preloaded on the Mediatek device was not allowed, as installing those would have 'tainted' the original state.
• Linux only. Running any M$ Windows tool was strictly out of scope.
• More of an unfortunate circumstance than a rule, but I did not have an external microSD card handy, so I had to rely using the internal storage.

I would not position myself as an android expert in any way, as this almost my first encounter of a modern android device. Trying to make use of the skills I have, I have taken a Linux centric approach. After all, my smartphone is just an ARM based mini computer with embedded NAND flash storage running Linux...

Before my phone arrived, I read a few articles on how a full backup could be taken, how recovery works and what the boot process of an android devices is like. This post is not going to cover these basics. My first idea was to boot my phone using a custom recovery image stored on my thinkpad, without actually flashing the recovery image. This involves rebooting the device into bootloader mode and connecting it to my thinkpad via USB.

Factory mode, Bootloader, fastboot and recovery

Having initiated the first boot with the power & volume up buttons pressed, I was confronted with a Chinese menu, which turned out to be not the recovery mode, but the factory mode with options similar to the following:

工厂模式
完整测试
单项测试
测试报告
清除e MMC
版本信息
重启手机 

The only familiar character sequence 'eMMC' suggested that option was responsible for wiping the whole internal storage, while the default selection would reboot the phone. A couple of days later I found a forum post that contained translation of these (or very similar) menu items.

The phone was connected to my thinkpad via a USB cable, but I did not have connectivity to the device, although I would have expected some sort of connection via the Android SDK.

After this, I rebooted into recovery mode via power, volume up & volume down buttons and realised that the stock recovery also does not provide any means to connect to the device from my thinkpad.

Finally I could reboot the device into bootloader mode (also known as fastboot mode). I had to start up the device normally, select 'Media device (MTP)' when configuring the mode of USB connection and enable USB debugging. I have found that 'Camera (PTP)' or 'Mass storage' modes do not allow the execution of Android De bug Bridge (adb) commands.

Once the device was connected, which I confirmed via adb devices, I could reboot the device into bootloader mode via adb reboot bootloader. The devices rebooted and printed that it entered fastboot mode. I confirmed that the output of adb devices does not list the device as ready for connection (as the device was in bootloader mode), and that fastboot devices outputs that the device is present and in fastboot mode.

From within bootloader mode, the command fastboot boot custom-recovery.img is should boot a custom recovery without actually flashing it. I my case, however, the device printed it downloaded the image successfully but I was left with a 'Booting ...' message and a frozen device, that did not do anything for twenty minutes, even did not respond to the power button.

I took out the battery, reinserted it an everything was back to normal. I thought the recovery image I tried to boot was not built for this phone or maybe it was corrupted, so I tried it with multiple images that were told to work with my S650. Nor the CWM, nor the TWRP image seemed to work. I had to take an alternative approach.

Backing up manually, using the shell

Eventually, I rebooted to normal mode and worked towards my goal via adb shell which gave me shell access so I could make use of my linux experience. Some commands essentially require root access - note that rooting an S650 is not covered in this post, but the ability to gain root access is assumed.

First of all, I had to discover how the eMMC storage was partitioned and which parts I need to take a backup of.

$ cat /proc/partitions
major minor  #blocks  name

   7        0      10290 loop0
 253        0     524288 zram0
 179        0    7597184 mmcblk0
 179        1          1 mmcblk0p1
 179        2      10240 mmcblk0p2
 179        3      10240 mmcblk0p3
 179        4       6144 mmcblk0p4
 179        5    1048576 mmcblk0p5
 179        6     131072 mmcblk0p6
 179        7    3145728 mmcblk0p7
 179        8    3202688 mmcblk0p8
 179       64       2048 mmcblk0boot1
 179       32       2048 mmcblk0boot0

$ cat /proc/dumchar_info
Part_Name Size StartAddr Type MapTo
preloader    0x0000000001400000   0x0000000000000000   2   /dev/misc-sd
mbr          0x0000000000080000   0x0000000000000000   2   /dev/block/mmcblk0
ebr1         0x0000000000080000   0x0000000000080000   2   /dev/block/mmcblk0p1
pro_info     0x0000000000300000   0x0000000000100000   2   /dev/block/mmcblk0
nvram        0x0000000000500000   0x0000000000400000   2   /dev/block/mmcblk0
protect_f    0x0000000000a00000   0x0000000000900000   2   /dev/block/mmcblk0p2
protect_s    0x0000000000a00000   0x0000000001300000   2   /dev/block/mmcblk0p3
seccfg       0x0000000000020000   0x0000000001d00000   2   /dev/block/mmcblk0
uboot        0x0000000000060000   0x0000000001d20000   2   /dev/block/mmcblk0
bootimg      0x0000000000600000   0x0000000001d80000   2   /dev/block/mmcblk0
recovery     0x0000000000c00000   0x0000000002380000   2   /dev/block/mmcblk0
sec_ro       0x0000000000600000   0x0000000002f80000   2   /dev/block/mmcblk0p4
misc         0x0000000000080000   0x0000000003580000   2   /dev/block/mmcblk0
logo         0x0000000000300000   0x0000000003600000   2   /dev/block/mmcblk0
ebr2         0x0000000000080000   0x0000000003900000   2   /dev/block/mmcblk0
expdb        0x0000000000a00000   0x0000000003980000   2   /dev/block/mmcblk0
android      0x0000000040000000   0x0000000004380000   2   /dev/block/mmcblk0p5
cache        0x0000000008000000   0x0000000044380000   2   /dev/block/mmcblk0p6
usrdata      0x00000000c0000000   0x000000004c380000   2   /dev/block/mmcblk0p7
fat          0x00000000c37a0000   0x000000010c380000   2   /dev/block/mmcblk0p8
bmtpool      0x0000000001500000   0x00000000febf00a8   2   /dev/block/mmcblk0
Part_Name:Partition name you should open;
Size:size of partition
StartAddr:Start Address of partition;
Type:Type of partition(MTD=1,EMMC=2)
MapTo:actual device you operate

As it could be seen, most of the relevant data is on /dev/block/mmcblk0 and there are critical areas on the flash storage that are not mapped to any partition. An obvious solution was to use the almightly dd command and calculate proper count and skip parameters based on the output above.

It should be noted that the content of the preloader that is mapped to /dev/misc-sd is directly available under /dev/preloader.

dd if=/dev/preloader of=preloader

dd if=/dev/block/mmcblk0 of=mbr.img bs=1024 count=512 skip=0
dd if=/dev/block/mmcblk0 of=ebr1.img bs=1024 count=512 skip=512
dd if=/dev/block/mmcblk0 of=pro_info.img bs=1024 count=3072 skip=1024
dd if=/dev/block/mmcblk0 of=nvram.img bs=1024 count=5120 skip=4096
dd if=/dev/block/mmcblk0 of=protect_f.img bs=1024 count=10240 skip=9216
dd if=/dev/block/mmcblk0 of=protect_s.img bs=1024 count=10240 skip=19456
dd if=/dev/block/mmcblk0 of=seccfg.img bs=1024 count=128 skip=29696
dd if=/dev/block/mmcblk0 of=uboot.img bs=1024 count=384 skip=29824
dd if=/dev/block/mmcblk0 of=bootimg.img bs=1024 count=6144 skip=30208
dd if=/dev/block/mmcblk0 of=recovery.img bs=1024 count=12288 skip=36352
dd if=/dev/block/mmcblk0 of=sec_ro.img bs=1024 count=6144 skip=48640
dd if=/dev/block/mmcblk0 of=misc.img bs=1024 count=512 skip=54784
dd if=/dev/block/mmcblk0 of=logo.img bs=1024 count=3072 skip=55296
dd if=/dev/block/mmcblk0 of=ebr2.img bs=1024 count=512 skip=58368
dd if=/dev/block/mmcblk0 of=expdb.img bs=1024 count=10240 skip=58880
dd if=/dev/block/mmcblk0 of=android.img bs=1024 count=1048576 skip=69120
dd if=/dev/block/mmcblk0 of=cache.img bs=1024 count=131072 skip=1117696

dd if=/dev/block/mmcblk0 bs=1024 count=3145728 skip=1248768 | gzip > usrdata.img.gz
#dd if=/dev/block/mmcblk0 of=fat.img bs=1024 count=3202688 skip=200192

#dd if=/dev/block/mmcblk0 of=bmtpool.img bs=1024 count=21504 skip=4173760.1640625

A final note on the last few commands

At the time of taking the backup, I did not have an external sdcard, so I had to use the FAT partition of the device, which was 3GB in size. Obviously, as an immediate consequence, the related line is commented out, as trying to write the image of the FAT partition to the FAT partition itself would not be the brightest idea, would it? The content of this partition can be simply accessed and copied over to a PC by selecting USB connection mode 'Storage device'.

Further, backing up the usrdata partition, which in size is similar to the partition to which the backup was stored, needed a small trick. Given that the partition was almost empty (confirmed via df), I gzipped the image on the fly to make sure, and ended up with a 315MB compressed image.

I did not find much information on bmtpool. It is not mapped to a partition and has a start address which is not aligned to 512 bytes, so it could not be properly backed up via Windows based SP FlashTool according to multiple sources. Actually, nobody seems to have missed the contents of this section.

Tinkering with the raw backups

Some of the backed up files represent the raw copy of whole partitions, these can be directly loop-mounted on linux. Others such as boot.img or recovery.img need another approach.

Based on my experience with initramfs and kernel images, I initially tried to uncompress them via gunzip and cpio, but looking at them via a hex editor revealed that these are amended by a special header. I have quickly googled up a perl script from clockworkmod which was built for unpacking recovery images, but soon it turned out that MT6582 MediaTek devices use a special format that can be unpacked via this script.

Just out of curiosity, I studies the directory structure of the boot and recovery images and confirmed that my smartphone is indeed just a special linux device...

Restoring manually

Restoring is straightforward. Use the same dd commands, but swap the value of if and if, and replace skip with seek. And do make sure to slightly modify the command for usrdata.img.gz, using gunzip. This one is left as an exercise for the reader.

Changing UIDs of service accounts on Linux

About a month ago I was setting up a proprietary identity management solution on headless Linux servers. During the installation process I tried to keep the number of installed packages and libraries to the bare minimum, to create an as-lean-as-possible setup as opposed to the bloated 'typical' deployments.

This goal in tandem with my choice to use the most up to date version of the target Linux distribution resulted in an 'advanced' installation procedure using the command line. While it was no surprise that the graphical installshield wizard did not start due to missing libraries on the headless server, it surprise me that non-GUI installation failed each time during setting up service accounts.

Having tried various combinations, disabling SELinux, and even creating the service account manually before re-running the installer, it still kept failing. A quick search on the internet did not yield any useful hint, but looking deeper, so I investigated further. Enabled debug logging for the installshield wizard, then extracted the embedded Java installer from the blob, and found that various native binaries are unpacked and executed during the installation process.

One of these binaries was linked against an older version of a shared library (libstdc++.so.5) that was not available in any of the packages any more. As a quick and dirty workaround I quickly created a symlink with that name pointing to libstdc++.so.6, and confirmed this resolved the problem.

The installation was successful, however, it left me with a technical user and group that I did not fancy.

$ grep ldap /etc/passwd /etc/group
/etc/passwd:idsldap:x:502:501::/home/idsldap:/bin/ksh
/etc/group:idsldap:x:501:root

• First and foremost, I prefer technical accounts to have numberic UIDs from the range reserved for system accounts, as opposed to having them created as normal users. This range varies by distribution, RedHat & SuSE based distros reserve UIDs smaller than 500 for system accounts while Debian based ones create normal users with the first available UID after 999.
• If I have the control over the numeric values, I prefer to introduce some semantics into the naming convention, like creating the technical user for an LDAP server with UID 389. This is just a cosmetic point with no real impact, but as I had to change the UID anyway...
• Last but not least, I very strongly dislike if the group which is the dedicated primary group of a service account, uses the same name as the service user but a different numeric ID.

This post will cover the commands I applied to fix the UID and GID of the service user and group. File ownership is automatically fixed, but only for the files under the home directory (and mail spool) of the user, so additional commands were needed to also fix file ownership or files located elsewhere.

$ find / -user idsldap | less
$ usermod -u 389 idsldap
$ find / -user idsldap | less # you see that ownership is only updated under the home directory and mail spool
$ find / -user 502 | less # the old uid is still present on other files
$ find / -user 502 -exec chown idsldap {} \;
$ find / -user 502 | less # almost done, but by default, symbolic links are dereferenced, therefore not affected.
$ find / -user 502 -exec chown -h idsldap {} \;
$ find / -user 502 # none - confirms we are done.

$ find / -group 501 | less
$ groupmod -g 389 idsldap
$ find / -group 501 -exec chgrp -h idsldap {} \;
$ find / -group 501 # confirm we are done
$ find / -group idsldap | less # this gives the same output the first group related command

While most people probably would not put any effort into changing UIDs but rather leave it that way or reinstall from scratch, I do believe that this case was worth investing a bit of extra time.

OpenCL, Python and Ubuntu 12.10 (Part 2)

Using multiple OpenCL devices for computation intensive tasks might turn out to be a bit more challenging. Distributing load between heterogeneous GPUs is straightforward, however, ensuring that one gets the cumulative performance of all the GPUs to the same level as the sum of the individual devices is a bit tricky. At least with the drivers that ship with Ubuntu 12.10.

The current case that motivated this post is tweaking a configuration with multiple AMD GPUs of different generations. There is much controversial information available on whether an X session is required or not, whether to use crossfire cables or not, and whether dummy-plugs are required or not. It also seems to be less certain if and how cards of different generations play together.

Having searched the internet for utilities that help in debugging OpenCL related issues I decided to create my own version, that is not extremely chatty but provides a quick overview of the devices recognised along with their most important performance related parameters. The code is based on a script I found on a forum, extended and customized to match my needs.

#!/usr/bin/python

# 2013-04-03 03:35:03 

import sys
import os
import time
import platform
import imp

def getPyOpenCLPath():
    try:
        file, pathname, descr = imp.find_module('pyopencl')
    except:
        pathname = 'Not found'
    return str(pathname)

path = getPyOpenCLPath()

print 'opencl-info @ %s' % time.asctime()
print 'Operating System: %s %s' % (platform.system(), platform.dist())
print 'Python Version: %s (%s)' % (platform.python_version(), platform.architecture()[0])
print 'PyOpenCL Path: %s' % path

if path == 'Not Found':
    print 'Exiting' 
    sys.exit()

try:
    import pyopencl
    import pyopencl.version
except:
    print 'Unable to load PyOpenCL! OpenCL not supported?'
    sys.exit()
 
print 'PyOpenCL Version: %s' % pyopencl.VERSION_TEXT

try:
    platforms = pyopencl.get_platforms()
except:
    print 'Cannot get platform.'

if len(platforms) == 0:
    print 'No OpenCL platforms found!' 
    sys.exit()

count = 0

for i,p in enumerate(platforms):
    print ''
    print '[cl:%d] %s' % (i, p.name.replace('\x00','').strip())
    for k in ['vendor', 'profile', 'version']:
        print '    %s %s' % ((k + ':').ljust(16), getattr(p,k))
    print ''

    devices = platforms[i].get_devices()
    if len(devices) > 0:
        # Iterate through devices
        for j,d in enumerate(devices):
            count += 1
            print '    [cl:%d:%d] %s' % (i, j, d.name.replace('\x00','').strip())
            print '        type:                %s' % pyopencl.device_type.to_string(d.type)
            print '        memory:              %d MB' % (d.global_mem_size//1024//1024)
            print '        compute units:       %s' % d.max_compute_units
            print '        max clock:           %s MHz' % d.max_clock_frequency
            print '        max work group size: %s' % d.max_work_group_size
            print '        max work item size:  %s' % d.max_work_item_sizes
            # Iterate through device info
            #for name in filter( lambda x: not x.startswith('_'), dir(d)):
            #    try:
            #        print(name + ': '+ str(getattr(d, name)))
            #    except:
            #        print(name + ': (skipped)')

After trials and errors and a bit of tinkering I was able to set up the environment to run fine without an X session, without crossfire cables and without any dummy plug, and yield optimal performance. Documenting the configuration details are beyond the scope of this post, however the script above is provided to help others in OpenCL related debugging and optimisation.

Tracking down TrackPoint issues (Part 2)

This post in a follow up on Tracking down TrackPoint issues. I have noticed few days ago, that hot-plugging a USB mouse "does not work". I normally do not plug a mouse into my thinkpad, so this did not come to my attention so far.

After seeing my cursor would not move after plugging in the mouse, I decided to gather more information, starting with dmesg to verify the device is connected and recognized. Then, I inspected the output of xinput list and found that the mouse is listed as a floating slave. Manually running xinput enable "HID 04b3:3107" fixed the situation, it attached the floating device as a slave to "Virtual core pointer".

⎡ Virtual core pointer                     id=2 [master pointer  (3)]
⎜   ↳ Virtual core XTEST pointer               id=4 [slave  pointer  (2)]
⎜   ↳ SynPS/2 Synaptics TouchPad               id=12 [slave  pointer  (2)]
⎜   ↳ TPPS/2 IBM TrackPoint                    id=14 [slave  pointer  (2)]
⎜   ↳ HID 04b3:3107                            id=15 [slave  pointer  (2)]
⎣ Virtual core keyboard                    id=3 [master keyboard (2)]
    ↳ Virtual core XTEST keyboard              id=5 [slave  keyboard (3)]
    ↳ Power Button                             id=6 [slave  keyboard (3)]
    ↳ Video Bus                                id=7 [slave  keyboard (3)]
    ↳ Video Bus                                id=8 [slave  keyboard (3)]
    ↳ Sleep Button                             id=9 [slave  keyboard (3)]
    ↳ Integrated Camera                        id=10 [slave  keyboard (3)]
    ↳ AT Translated Set 2 keyboard             id=11 [slave  keyboard (3)]
    ↳ ThinkPad Extra Buttons                   id=13 [slave  keyboard (3)]

After restarting my X session - logging out and back in - with the mouse already connected I found it to be enabled and working correctly.

My conclusion was that the device got connected but not enabled after hot-plugging. I revisited my custom hotplug command in .local/bin/touchpad-config (see the previous post referred to above). This script was executed whenever on startup, and then, whenever a device is added or removed. Obviously, the touchpad and trackpoint related commands do not need to be run when an external mouse is connected. They are also needless when any device is disconnected.

I was searching for a way to detect which device is being added or removed and preform different logic based on that and came across this example in the GNOME git repository on github. It clearly demonstrated how device information and action is handed over to the script and allowed me to enhance my script with logging so I could see what was happening under the hood.

Eventually, it turned out that my script was only executed a couple of times on startup, but not invoked at all after that when the mouse was hot-plugged. It boiled down to be caused by some cruft in my Xorg configuration, left over after having explored Optimus related tweaks covered in my other posts. I had to change Option "AutoEnableDevices" "false" in xorg.conf.

Although the original issue was fully resolved by the before mentioned change in to xorg.conf, the sub-optimal .local/bin/touchpad-config was still bugging me so I altered it to only execute the commands relevant for the event that triggered the script. I ended up with the following hotplug-script, which is working flawlessly in my environment:

#!/bin/sh

action=$2
shift 4
device=$@

log () {
 echo "$1" >> /tmp/hotplug-cmd.log
}

log "$action '$device'"

if [ x"$action" = xadded -o x"$action" = xpresent ]; then
 case $device in
  "SynPS/2 Synaptics TouchPad")
                 synclient PalmDetect=1 PalmMinWidth=5 \
    TapButton3=2 HorizTwoFingerScroll=1
   log "configured touchpad"
                 ;;
         "TPPS/2 IBM TrackPoint")
                 xinput disable "<default pointer>" 
                 xinput enable "TPPS/2 IBM TrackPoint"
   log "disabled <default pointer>, enabled trackpoint"
                 ;;
         *)
                 log "nop"
                 ;;
 esac
fi

Note that the script above still includes logging which may be safely removed.

OpenCL, Python and Ubuntu 12.10

Installing PyOpenCL, the python OpenCL wrapper on Ubuntu 12.10 could have been just as easy as installing the according package python-pyopencl but this is currently not possible because of a packaging bug that was reported and confirmed before the release of 12.10 but not fixed since. I am not going to comment on this but document my work around instead.

What is OpenCL?

OpenCL is the short name for Open Computing Language - an open standard that specifies a vendor agnostic language and cross-platform framework for developing code that is can be executed on parallel computing enabled platforms, such as GPUs (graphics cards), FPGAs (field programmable gate arrays) or newer generation multicore CPUs.

The most typical use-case is using the graphics device for non-graphics computation tasks. Typical computation intensive tasks can be broken down into multiple sub-tasks that can be executed independent of each other, in parallel. The many pourpose built processing pipelines, special shift register circuits, shaders and vertext processors inegrated into modern graphics devices are equipped with a multitiude of 256bit registers, and excel at high volume calculation of matrix operations, interpolations and special tranformations (e.g. fourier, laplace). OpenCL enables one to unleash this raw parallel/stream processing power to be used for generic purposes, other than gaming.

Currently, OpenCL support is provided by proprietary drivers for Nvidia GPUs, AMD GPUs and CPUs and Intel CPUs. Although these drivers are not open source components, OpenCL, being an open standard, enables one to avoid coding against vendor specific APIs and create software mostly portable across computing platforms. (One well known example of such vendor specific APIs is provided by Nvidia's Compute Unified Device Architecture - CUDA - but recent CUDA versions expose OpenCL interfaces as well.)

As of the time of this writing, OpenCL is my preferred platform for computation intensive, time critical applications. It is worth to be aware of, and tinker around with it for fun and profit.

What is the bug?

In a nutshell, the package cannot installed, the bug is a package dependency on a non-existent package, namely opencl-icd that is not available in any of the official or partner repositories and also not provided by other package. As the but was reported and confirmed one month before the final release of 12.10 I am rather surprised it has not been fixed yet, especially as fixing the problematic dependency boils down require merely one line to be changed in the package metadata.

I have quickly read up on the format of debian packages to make sure I understand the issue correctly and found the following alternatives for solving the current problem:

• Modify the package python-pyopencl by removing the dependency on opencl-icd. This is a low hanging fruit but would leads to broken packages in the long run as any update to the package would overwrite my local changes. This issue can be dodged by putting the package on hold.
• The strategic solution would be to alter the metadata of packages nvidia-current and fglrx so these packages would provide opencl-icd as a virtual package. This is not my preferred approach for a local workaround, as leads to maintenance issues on package updates. The concerns and workarounds of the previous alternative apply here as well.
• My preferred approach was creating a dummy package python-pyopencl that merely depends on the appropriate drivers but does not contain any real file itself, only metadata. This approach will shield me from package update issues.

Creating and installing the dummy package

$ mkdir /tmp/dummy && cd /tmp/dummy
$ tar czf data.tar.gz * # creating an empty archive, will output some errors
$ tar tzf data.tar.gz   # this verifies the archive is empty and can be read without errors
$ touch md5sums
$ cat <<EOF>> control
Package: opencl-icd
Source: opencl-icd
Version: 1.0.0-1
Architecture: amd64
Maintainer: Tibor Bősze <tibor.boesze@gmail.com>
Installed-Size: 4
Depends: fglrx | nvidia-current | intel-ocl-sdk
Section: universe/libs
Priority: optional
Homepage: http://tuxicate.blogspot.com/
Description: Dummy package
 This is a dummy package to workaround bug 1048036
EOF
$ tar czf control.tar.gz md5sums control
$ echo "2.0" > debian-binary
$ ar rcs opencl-icd_1.0.0-1_amd64.deb debian-binary control.tar.gz data.tar.gz
$ dpkg-deb --verbose --info opencl-icd_1.0.0-1_amd64.deb # verify
$ sudo dpkg -i opencl-icd_1.0.0-1_amd64.deb # install

The package, once installed, will appear under Obsolete and Locally Created Packages in aptitude, so it is easy to track and uninstall if needed. Now, python-pyopencl can be installed and used without issues.

One could modify the dependencies (the line starting with Depends:) as required, I have added Nvidia or AMD Radeon/FireGL graphics drivers, or the Intel OpelCL SDK. The Intel SDK can be downloaded in rpm for Intel, and converted to a deb package with alien.

Optimus and Ubuntu 12.10 (Part 5)

This post is the fifth of a series of posts on tweaking Ubuntu 12.10 to exploit Optimus technology on my Lenovo W530 to the extent I need. Make sure you are familiar with the context and objectives.

As described in Part 1, plymouth issues when Optimus is enabled on the W530 in EFI mode. The issue around missing usplash/plymouth turned out to be connected to the random order in which the kernel initialized the graphics devices.

Part 3 captures the root cause of the symthom: Usplash is hardcoded to render to /dev/fb0 which is fine if the IGD is initialized first by the kernel, but not good if /dev/fb0 is associated with the DIS framebuffer.

One workaround already explained is blacklisting nouveau, and loading it later, after usplash is started on the IGD framebuffer. I seeked more elegant approaches and tinkered with initram to explore the alternatives:

• Forcing proper module load order in initrd:/etc/modules - I found that this file is only processed after the kernel finished autoloading, and cannot load blacklisted modules.
• Crafting custom initram scripts to take care of module loading early in the book process - this approach did not prove effective either.
• Creating an initram script to disable DIS proved to be impossible as debugfs was not mounted at the time of initram script execution.
• Packaging custom udev rules into the initial ramdisk to ensure the desired framebuffer numbering. I settled with this option.

Ensuring consistent framebuffer numbering via udev rules

Traditionally, Unix systems exposed a static set of device files under /dev but current versions of Linux ship with a device manager that can dynamically populate the /dev directory with nodes for the devices present. It allows persistent naming insensitive to the order of hardware initialization or hotplug. This is achieved by udev rules, which match the devices based on their attributes and perform actions such as creating device nodes, symlinks, changing permissions and ownership, or if fact, running arbitrary logic.

This section provides a very brief walk-through on how the rules for the current use-case can be created. First, information about the framebuffer devices had to be gathered, and the properties and attributes for the matching part had to be identified.

$ udevadm info -a -n /dev/fb0 | head -n 24

Udevadm info starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.

  looking at device '/devices/pci0000:00/0000:00:01.0/0000:01:00.0/graphics/fb0':
    KERNEL=="fb0"
    SUBSYSTEM=="graphics"
    DRIVER==""
    ATTR{pan}=="0,0"
    ATTR{name}=="nouveaufb"
    ATTR{mode}==""
    ATTR{console}==""
    ATTR{blank}==""
    ATTR{modes}=="U:1024x768p-0"
    ATTR{state}=="0"
    ATTR{bits_per_pixel}=="32"
    ATTR{cursor}==""
    ATTR{rotate}=="0"
    ATTR{stride}=="4096"
    ATTR{virtual_size}=="1024,768"

$ udevadm info -a -n /dev/fb1 | sed -n '8,24p'
  looking at device '/devices/pci0000:00/0000:00:02.0/graphics/fb1':
    KERNEL=="fb1"
    SUBSYSTEM=="graphics"
    DRIVER==""
    ATTR{pan}=="0,0"
    ATTR{name}=="inteldrmfb"
    ATTR{mode}==""
    ATTR{console}==""
    ATTR{blank}==""
    ATTR{modes}=="U:1920x1080p-0"
    ATTR{state}=="0"
    ATTR{bits_per_pixel}=="32"
    ATTR{cursor}==""
    ATTR{rotate}=="0"
    ATTR{stride}=="7680"
    ATTR{virtual_size}=="1920,1080"

As it can be seen from the listing above, is is sufficient to match devices of the graphics subsystem named by the kernel as "fd0" and "fb1". The attribute "name" can be used to discriminate between the two framebuffers. The goal is create device node /dev/fb0 for the intel framebuffer /dev/fb1 for the nouveau framebuffer respectively. This can be achieved by the following udev rule:

KERNEL=="fb?", SUBSYSTEM=="graphics", ATTR{name}=="nouveaufb", NAME="fb1"
KERNEL=="fb?", SUBSYSTEM=="graphics", ATTR{name}=="inteldrmfb", NAME="fb0"

Normally, one would save these lines to a file in the appropriate folder, say /etc/udev/rules.d/82-explicit-fb-assignment.rules. (The numbering is used to ensure the rule is not overridden, as rules are parsed in lexicographical order.)

In this very case, however, the rule will not yield the desired results. One would see usplash on shutdown, but not on boot. The rule file is located on disk, in the root partition to be precise. It gets mounted after usplash initializes during boot...

Packaging custom udev rules into initrd

As implied by the nature of the use case, it would not suffice to drop the custom udev rule into /etc/udev/rules.d/, it needs to be included in the initial ramdisk. Customization of initial ramdisk content is best done via custom hook scripts - they provide a clean, modular way of achieving the current goal while being minimally intrusive to other parts of the system. The hook scripts will run whenever the a new initrd is created and can contribute files to the ramdisk. They do not become part of the ramdisk themselves.

$ cat <<EOX | sudo tee /etc/initramfs-tools/hooks/explicit-fb-assignment
#!/bin/sh -e

PREREQ="udev"

# Output pre-requisites
prereqs()
{
   echo "$PREREQ"
}

case "$1" in
    prereqs)
   prereqs
   exit 0
   ;;
esac


. /usr/share/initramfs-tools/hook-functions

# Create udev rules to control fb1/fb0 assignment
cat > ${DESTDIR}/lib/udev/rules.d/82-explicit-fb-assignment.rules <<EOF
KERNEL=="fb?", SUBSYSTEM=="graphics", ATTR{name}=="nouveaufb", NAME="fb1"
KERNEL=="fb?", SUBSYSTEM=="graphics", ATTR{name}=="inteldrmfb", NAME="fb0"
EOF

EOX

$ sudo chmod +x /etc/initramfs-tools/hooks/fb_order
# create new initrd for the current kernel
$ sudo update-initramfs -c -k $(uname -r)
# verify on next reboot

The listing above provides the commands for creating the hook and generating a new initrd. Having applied this tweak, usplash/plymouth is fully functional and reliable, as /dev/fb0 always refers to the intel framebuffer device.

Troubleshooting

One has to remember that there are 2 udev daemons launched at different parts of the boot process. One runs off initrd, while the other is spawned after the root filesystem has been mounted. The former uses the rules included in ramdisk, while the later one reads the rules from /{lib|etc}/udev/ruled.d of the root partition, and does not use any rule from the initrd.