HTC One S Android setup

1. hardware
2. phone

This page describes my experiences with the HTC One S, a handy little Android phone I have on a loan by a friend. It's a neat device because it's small and I can wipe it with Cyanogenmod (CM) to have more control over the machine, including running more free software on it and removing the stock proprietary google apps shipped with the phone. It also happens that CM has more recent versions of Android for the phone, which only runs Android 4.1, an unsupported release.

The main downside of the phone is that it doesn't have a SD card socket, so storage is limited to the internal storage area, which is around 16GB. It's possible to get a OTG adapter cable to expand storage, which is mostly useful for backups, as it's pretty inconvenient to have that thing dangling around on the phone. I am also thinking of getting a protective screen.

Obviously, the phone doesn't run exclusively free software, even after all the trouble I go through here: CM itself ships proprietary drivers and the Baseband processor is not free, of course (see phone for a more generic discussion on that).

This is my own version of documentation found elsewhere and gleaned on IRC (thanks to the folks from #cyanogenmod on Freenode). Obviously, the usual warnings apply here: this may void your warranty, destroy your data, brick your phone and make your cat jealous because you wasted 10 hours buzzing around frenetically on your phone.

Flashing a HTC One S with Cyanogenmod

These instructions explain how to replace the stock HTC firmware with one from Cyanogenmod.

Downloads

I do a bunch of downloads first... I apparently have a HTC "ville", so I download the latest nightly.

Then i need to choose which gapps i want. I need to choose one, so i pick the smallest one (pico), the only google apps i'd use being google maps, which i can install later.

Update: turns out I don't really use Google maps, and I can install the app later anyways - it doesn't absolutely need to be part of the base ROM. I want to avoid having proprietary software as much as possible on the phone and Open GAPPS are not free software at all.

I also need TWRP (TeamWin Recovery Project) to load CM onto the phone, so I also downloaded the latest (3.0) release of TWRP for Ville. Note, I had to install cpuspy from f-droid to figure out the CPU (1512MHz), which means it is a S4 processor.

Developer options and USB debugging were already enabled on the phone, but otherwise you can enable them by hitting the "build number" button 7 times in CM.

Rooting the phone

Install instructions from CM require fastboot and adb which is just:

apt-get install android-tools-adb android-tools-fastboot

Go in fastboot mode:

adb reboot bootloader

Then get the token:

[1007]anarcat@angela:~$ sudo fastboot devices
????????????    fastboot
[1008]anarcat@angela:~$ fastboot oem get_identifier_token
...
(bootloader)
(bootloader) < Please cut following message >
(bootloader) <<<< Identifier Token Start >>>>
(bootloader) [... bunch of HEX here]
(bootloader) <<<<< Identifier Token End >>>>>
OKAY [  0.069s]
finished. total time: 0.069s

The devices list was weird, to fix that:

[1009]anarcat@angela:~$ adb kill-server
[1010]anarcat@angela:~$ sudo fastboot devices
HT26PW407343    fastboot
[1011]anarcat@angela:~$

Then I need to go through a byzantine system on HTCdev to unlock the phone. I actually had to register and login, and the forms are all buggy. They tell me to install fastboot again - ignore that, it's just to get the token we had above, which we paste in and then they send us stuff by email again (ugh).

Then I get this unlock file by email:

$ sudo fastboot flash unlocktoken Unlock_code.bin
[sudo] password for anarcat:
< waiting for device >
sending 'unlocktoken' (0 KB)...
OKAY [  0.143s]
writing 'unlocktoken'...
(bootloader) unlock token check successfully
OKAY [  0.006s]
finished. total time: 0.149s

And I lock it back again:

sudo fastboot oem lock

... but i leave it unlocked for now. Besides, i'd have to go into the bootloader again for that. And when I tried to do that later on, I actually wiped the device completely, so let's not do that again...

Installing Recovery ROM

Now it's going through this stupid configuration wizard, which i PLOKTA through to enable debugging mode and again reboot in the bootloader to install TWRP:

$ sudo adb reboot bootloader
$ fastboot flash recovery ~/Downloads/CM/twrp-3.0.0-0-ville.img
< waiting for device >
sending 'recovery' (11896 KB)...
OKAY [  1.629s]
writing 'recovery'...
OKAY [  2.520s]
finished. total time: 4.150s

Then it's back in the bootloader, i choose BOOTLOADER, then RECOVERY. I get into TWRP, and crazy prog music kicks in. Now i click "Keep read only", go into Backup. Keep the defaults (Boot, Data, System) and choose the name Stock ROM <append date> and swipe to backup. Then I need to copy the backup out, for me it was in /sdcard/TWRP/BACKUPS/, so:

$ sudo adb pull /sdcard/TWRP/BACKUPS/
pull: building file list...
pull: /sdcard/TWRP/BACKUPS/HT26PW407343/Stock ROM 2016-03-17--19-17-35/recovery.log -> ./HT26PW407343/Stock ROM 2016-03-17--19-17-35/recovery.log
pull: /sdcard/TWRP/BACKUPS/HT26PW407343/Stock ROM 2016-03-17--19-17-35/system.ext4.win.md5 -> ./HT26PW407343/Stock ROM 2016-03-17--19-17-35/system.ext4.win.md5
pull: /sdcard/TWRP/BACKUPS/HT26PW407343/Stock ROM 2016-03-17--19-17-35/system.info -> ./HT26PW407343/Stock ROM 2016-03-17--19-17-35/system.info
pull: /sdcard/TWRP/BACKUPS/HT26PW407343/Stock ROM 2016-03-17--19-17-35/system.ext4.win -> ./HT26PW407343/Stock ROM 2016-03-17--19-17-35/system.ext4.win
pull: /sdcard/TWRP/BACKUPS/HT26PW407343/Stock ROM 2016-03-17--19-17-35/data.info -> ./HT26PW407343/Stock ROM 2016-03-17--19-17-35/data.info
pull: /sdcard/TWRP/BACKUPS/HT26PW407343/Stock ROM 2016-03-17--19-17-35/data.ext4.win -> ./HT26PW407343/Stock ROM 2016-03-17--19-17-35/data.ext4.win
pull: /sdcard/TWRP/BACKUPS/HT26PW407343/Stock ROM 2016-03-17--19-17-35/boot.emmc.win.md5 -> ./HT26PW407343/Stock ROM 2016-03-17--19-17-35/boot.emmc.win.md5
pull: /sdcard/TWRP/BACKUPS/HT26PW407343/Stock ROM 2016-03-17--19-17-35/boot.emmc.win -> ./HT26PW407343/Stock ROM 2016-03-17--19-17-35/boot.emmc.win
9 files pulled. 0 files skipped.
3145 KB/s (1060868245 bytes in 329.386s)

Installing Cyanogenmod

Then go to Wipe, Advanced wipe, wipe Cache, Dalvik Cache, Data and swipe to wipe.

Then go to Advanced, ADB Sideload, don't check the Wipe Cache and boxes swipe and run:

$ sudo adb sideload cm-12.1-20160316-NIGHTLY-ville.zip
serving: 'cm-12.1-20160316-NIGHTLY-ville.zip'  (~3%)

Do the same with the gapps package chosen above:

sudo adb sideload open_gapps-arm-5.1-pico-20160317.zip

Note: this requires adb 1.0.32, which is only available in stretch. However, the package from stretch installs fine in Jessie.

Also note: a successful sideload should show progress and complete with:

Total xfer: 1.00x

or:

Total xfer: 1.71x

It is possible that the transfer fails for some reason. I have seen errors like this:

$ sudo adb sideload open_gapps-arm-5.1-pico-20160322.zip
serving: 'open_gapps-arm-5.1-pico-20160322.zip'  (~15%)    * failed to read command:

Then TWRP would simply reboot directly in to the regular ROM and give me a blank screen. I would then go back to the bootloader by pressing simultaneously Power and Volume down for around 15 seconds and then release the Power button while keeping Volume down pressed until the bootload comes up (~5-10s?). From there I return to TWRP and sideload again the part that failed.

This is also a good time to load other privileged extensions, like F-Droid:

sudo adb sideload org.fdroid.fdroid.privileged.ota_2040.zip

Once the sideloading is complete, you need to flash the boot.img file in place, by rebooting in the bootloader. Use Reboot, Bootloader from the main menu, not the Reboot button from the sideloading dialog! Then flash the boot image with:

$ unzip cm-12.1-20160316-NIGHTLY-ville.zip boot.img
$ fastboot flash boot boot.img
sending 'boot' (6862 KB)...
OKAY [  1.008s]
writing 'boot'...
OKAY [  1.555s]
finished. total time: 2.563s

And now reboot! The phone takes a while to bootup on first boot, but then it's mostly done.

Oddly enough, it did not erase all the data on the phone, which is something I wanted to happen, so I had to erase all the data again, by hand, through the USB mount interface from the laptop.

My android configuration

Those are things to do when I flash the device, which I seem to screwup so often that I actually had to note this down.

1. Check for updates and install: About phone, CyanogenMod updates if not done automatically There are no updates anymore, tough luck.
2. encrypt the phone (takes ~10 minutes, needs power), see below
3. set lock code (PIN)
4. go through prefs to tweak everything
5. enable privacy guard, including on builtin apps
6. browser: disable a bunch of stuff, enable utf8
7. install f-droid done during sideloading, refresh repos and upgrade instead
8. install and configure apps - note: this can be more easily done by syncing apps with another phone through F-Droid's "nearby" feature
9. import contacts from backups - can be done with nextcloud
10. setup fake GCM screw google
11. configure all installed apps above
12. backup the phone
13. reimport music using git-annex using Subsonic instead

Some of those steps are documented more explicitly below.

Apps install and synchronization

Apps often keep their state only on the phone and don't sync up to servers online, which means we need to backup/restore some things around.

F-Droid has a very nice interface to install "nearby" apps. The "app list" is totally useless in comparison, as the "HTML list" version is unusable: the links are broken and formatting is all out of whack.

Here are the various tweaks required for each app I currently use:

1. anikdroid: syncs easily with central server
2. antenna pod: syncs with gpodder, but better export/import database through syncthing
3. dSub, lichess, Nextcloud, Wallabag, Linphone and VoIP.SMS all need to have their passwords re-set which is annoying, although the latter two are relevant only for "non-GSM" phones
4. OSMand also doesn't have such a good import/export story - all those little settings need to be redone by hand if there's no backup
5. Signal is, of course, a frigging catastrophe as well, althought here is an actually good way to transfer between two phones so the story isn't that bad either
6. syncthing takes care of the rest, namely installing Signal from the APK URL, my Stellarium backup and so on. it also useful for "copy-pasting" (so to speak) passwords around instead of tapping those stupid strings out, including the Signal backup and its secret key
7. contacts can be backed up with nextcloud on one device and restored on the other fairly easily

Backing up phone

To do a phone backup, I reboot into TWRP again and perform a backup with the default settings (Boot, Data, System). I then pull it off TWRP with adb pull:

sudo adb pull /sdcard/TWRP/BACKUPS/HT26PW407343/2016-03-23--13-20-27_cm_ville-userdebug_5.1.1_LMY49H_a105530ecd

This will fail if the phone doesn't have enough free space. Two options:

• OTG dongle: a small adapter that allows you to plug external USB storage in the phone
• adb backup: from a host connected through USB, you can generate an archive of the whole system. this also works through TWRP 3.1 and later if you can't boot the phone properly, with adb backup --twrp. The file format is a little weird see this discussion for details. Also note that the format is different when using TWRP, see that other discussion for details.

I do not believe this makes a backup of the data in sdcard, however, so if user data should also be backed up, the above backup and Music, Podcasts, Pictures and so on can all be pulled at once with:

sudo adb pull /sdcard

It is unclear to me if all the relevant data and configuration settings are saved by TWRP backup, see issue #276 in TWRP for a discussion about this. For now, the pull should work, and rsync could also be used by mounting the drive through USB.

Two more convenient things can be explicitely backed up: the contact and apps lists. For contacts, head for the contacts app, and export: selected (or all) contacts will be saved in a single .vcf file on storage. For apps, I have installed the List My Apps app to export a list of app into a text file. I chose the Market URL list and then reformatted it, but it would be nice to have a better output and not have to copy-paste it into an editor.

Backups are stored on the external drive named Calyx.

Encrypting phone

First set a passphrase or PIN, otherwise things go real weird and you may have to wipe encryption in TWRP. Then go into Security, Encrypt phone and so on. Use logcat in a shell to see progress:

sudo adb root
sudo adb logcat

There is a way to decouple the screen PIN from the crypto password, so you get prompted for the long passphrase on boot while still keeping a usable PIN or pattern. I was told to use the following instructions but haven't tested them:

1. Set Screen Unlock Password
2. Enable USB Debugging in Dev Options
3. Allow Root access to apps and ADB
4. adb root
5. adb shell
6. set a password, hex-encoded (using the ASCII value of each character):
   • ALREADY ENCRYPTED PHONE (PASSWORD NEEDS TO BE IN HEX): vdc cryptfs changepw password HEXED_PASSWORD
   • NOT ENCRYPTED PHONE: vdc cryptfs enablecrypto inplace password HEXED_PASSWORD

There should also be a way to do certain actions when too many PIN entry failures are detected. Locker wipes your phone after a certain number, but it is was not in F-Droid at the time of writing. Furthermore, there was an app named SnooperStopper that did exactly what I needed: it shuts down the phone after a customizable number of failures and controls the crypto passphrase independently from the PIN, without messing around with the above (which needed to be done every time the PIN was changed).

Important: note that by default, the "internal SD card storage" was not encrypted. This means all my Photos, music and much more is not encrypted! It looks like SMSes and contacts are in the private area, but the procedure failed to encrypt the larger part of the storage. This is a known problem which could be because the partition is formatted as FAT, which is not supported by Android's encryption. Yet I have also tried re-formatting the SD card as ext4 and the encryption process still didn't pick it up.

I have tried to re-encrypt the phone, but it just aborted because it was detected as already encrypted. It seems that Android 6.0 is a little better at this so maybe the best approach for now is "wait and see".

Some more links regarding encryption:

• Discussion about the encryption threat model
• Background information regarding encryption in Android
• Possibly relevant issues with encryption:
  • vdc cryptfs changepw is buggy
  • Decryption unsuccessful on N5 running Stock 6.0 with ElementalX 6.03
• More encryption-related links:
  • http://android.stackexchange.com/questions/117405/how-do-you-encrypt-your-device-running-cyanogenmod-12-1
  • http://vogelchr.blogspot.ca/2015/08/full-disk-encryption-with-cyanogenmod.html
  • http://android.stackexchange.com/questions/98228/removing-encryption-from-recovery

Git-annex imports

Assuming it is mounted as a mass storage drive:

    cd mp3
    git init --bare /media/anarcat/32E6-1D00/Music
    cd !$
    git annex init
    git annex reinit 6f812272-18c8-4346-b68a-f57ae50f657e
    cd -
    git annex copy --to htcones --in htcones

I am using a bare repository because i'm having trouble cloning the complete repository, see this bug report for details. I ended up reformatting the partition as ext4 (see below) so I could use a real git-annex repository there. So the above becomes:

[1077]anarcat@angela:cb39412b-d221-4846-a9dd-cdbabd9958f1$ sudo mkdir Music
[1078]anarcat@angela:cb39412b-d221-4846-a9dd-cdbabd9958f1$ sudo chown anarcat Music
[1079]anarcat@angela:cb39412b-d221-4846-a9dd-cdbabd9958f1$ cd Music/
[1082]anarcat@angela:Music$ git init
Dépôt Git vide initialisé dans /media/anarcat/cb39412b-d221-4846-a9dd-cdbabd9958f1/Music/.gi
[1083]anarcat@angela:Music$ git annex init
init  ok
(recording state in git...)
[1084]anarcat@angela:Music130$ git annex reinit 6f812272-18c8-4346-b68a-f57ae50f657e
reinit 6f812272-18c8-4346-b68a-f57ae50f657e ok
[1086]anarcat@angela:Music$ git remote add origin ~/mp3
[1087]anarcat@angela:Music$ git remote update # 12:44
Récupération de origin
warning: no common commits
remote: Décompte des objets: 819312, fait.
remote: Compression des objets: 100% (368189/368189), fait.
Réception d'objets: 100% (819312/819312), 64.97 MiB | 5.19 MiB/s, fait.
remote: Total 819312 (delta 653024), reused 588679 (delta 449859)
Depuis /home/anarcat/mp3
 * [nouvelle branche] git-annex  -> origin/git-annex
 * [nouvelle branche] master     -> origin/master
 * [nouvelle branche] synced/git-annex -> origin/synced/git-annex
 * [nouvelle branche] synced/master -> origin/synced/master
 * [nouvelle étiquette] bak        -> bak

That takes around 3 minutes. Notice how I needed to change the permissions because the files are owned by Android's media_rw user (1023:1023). It is fine, however, because Android shouldn't be allowed to mess with the audio files. Then the actual files (symlinks) need to be created, with:

[1089]anarcat@angela:Music$ git annex merge
merge git-annex (merging origin/git-annex origin/synced/git-annex into git-annex...)
(recording state in git...)

That takes another 2 minutes. Then another two minutes for this:

[1092]anarcat@angela:Music$ git co master
Extraction des fichiers: 100% (22187/22187), fait.
La branche master est paramétrée pour suivre la branche distante master depuis origin.
Déjà sur 'master'

And now i'm free to get the songs I want, for example:

[1099]anarcat@angela:Music130$ date; time git annex get --quiet --in here; date
mercredi 30 mars 2016, 12:53:29 (UTC-0400)
sha256sum: .git/annex/tmp/SHA256E-s4476433--bb954dfe81f3d0906a18e53d02040d1d8f8e78917552e0033b056bbf885710d9.mp3: Aucun fichier ou dossier de ce type
  sha256sum failed
git-annex: .git/annex/tmp/SHA256E-s4476433--bb954dfe81f3d0906a18e53d02040d1d8f8e78917552e0033b056bbf885710d9.mp3: openBinaryFile: does not exist (No such file or directory)
git-annex: get: 1 failed
Command exited with non-zero status 1
156.09user 39.26system 36:44.62elapsed 8%CPU (0avgtext+0avgdata 44900maxresident)k
8775224inputs+5811728outputs (51major+281689minor)pagefaults 0swaps
mercredi 30 mars 2016, 13:30:36 (UTC-0400)

Unfortunately, the above created a huge problem where remotes disappeared from git-annex! I was fortunately able to workaround the issue, and was even able to cram way more music than previously on the SD card, which is great! Unfortunately, the FUSE layer that enables the SD card emulation in that version of Android doesn't support symlinks, which breaks the whole thing. So I went by to direct mode here, which works better on ext4 than FAT, so things actually work as expected now.

Update: it turns out this created its own set of problems too: the direct mode made it so zillions of files were showing up in the music app while not really being there. I tried to reformat to ext4 and use a regular repository, but had the same problems.

I ended up writing a new special remote to transfer files to the phone, as a workaround while we wait for dumb backends (if ever implemented).

The script is called git-annex-remote-dumb and has a few limitations, most notably that it cannot remove files yet. Just installing the script in $PATH and making it executable should make it work.

To prepare the remote:

cd mp3
git annex initremote htconesdumb type=external externaltype=dumb directory=/media/sdb/Music/ encryption=none

To load files, I used:

pmount sdb
sudo chown -R anarcat /media/sdb/Music
git annex copy --to htconesdumb --in htcones
sudo chown -R 1023 /media/sdb/Music
pumount sdb

The content expression may vary of course - in the above i merely replicated the content that was already present on the device before it got wiped.

Also, the location of the mountpoint may change: for example, when mounting with Thunar. In this case, I can use this to change the location of the remote:

git annex enableremote htconesdumb directory=/media/anarcat/57f8f4bc-abf4-655f-bf67-946fc0f9f25b/Music

This makes music way more reliable take up less space and work much more as expected.

An alternative is to use the jmtpfs package to mount the MTP device, which works around the need to chown everything, but is much slower:

jmtpfs -l
jmtpfs ~/mnt

Reformatting internal SD card as ext4

A workaround for the above problems (and a requirement to be able to encrypt the SD card) is to reformat the internal SD card as ext4. Apparently, there is a way to do this with the sdcard command in Android, but I haven't heard much more on how to do that exactly, and searching the web for that string is basically impossible.

So I formatted the thing using mkfs -t ext4 on my laptop, simply. I mounted the partition through a USB cable using TWRP first, made a backup and formatted it.

After a reboot, the camera wouldn't work ("Insert an SD card before using the camera"), because the camera wouldn't be able to write to the card. adb logcat would show:

E/sdcard  (  903): opendir /mnt/media_rw/sdcard0/DCIM failed: Permission denied

The fix was to change the permissions on the whole SD card:

chown -R media_rw:media_rw /mnt/media_rw/sdcard0

That way apps can write to it directly. A more fine-grained approach could have been used - the default permissions are system:system so I could restore that on certain folders I want to protect, which is cool, but for now I left it as is. I have found that this stack exchange question helped me figure that out, even though it was a different problem.

Fake GCM

I used tingle to spoof signatures from the Google Services framework in order to be able to fake the GCM with MicroG. This allows me to have push notifications in the VoIP.ms SMS client and potentially other apps without using proprietary software from Google. I does mean I still use Google's Cloud Messaging, unfortunately, but that's is dependent on specific apps being fixed (e.g. VoIP.ms, Signal).

Running Tingle should look something like this:

[1001]anarcat@angela:dist$ git clone https://github.com/ale5000-git/tingle
Clonage dans 'tingle'...
remote: Counting objects: 656, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 656 (delta 0), reused 0 (delta 0), pack-reused 652
Réception d'objets: 100% (656/656), 11.68 MiB | 2.85 MiB/s, fait.
Résolution des deltas: 100% (382/382), fait.
Vérification de la connectivité... fait.
[1002]anarcat@angela:dist$ cd tingle/
[1010]anarcat@angela:tingle1$ python main.py 
MENU

    1 - Patch file from a device (adb)
    2 - Patch file from the input folder

> 1

 *** OS: Linux 4.6.0-0.bpo.1-amd64 (linux)
 *** Mode: 1
 *** Working dir: /tmp/Tingle-FzBvdm
 *** Selected device: HT26PW407343
 *** Pulling framework from device...
2523 KB/s (5535054 bytes in 2.142s)
 *** Decompressing framework...
      DEBUG: Decompressing framework.jar
 *** Disassembling classes...
      DEBUG: Disassembling framework/classes.dex
 *** Patching...
 *** Patching succeeded.
 *** Reassembling classes...
      DEBUG: Assembling out/classes.dex
 *** Recompressing framework...
      DEBUG: Compressing framework.jar
 *** Copying the patched file in the output folder...
 *** Rooting adbd...
      DEBUG: 
      DEBUG: remount succeeded
 *** Pushing changes to the device...
3957 KB/s (5451374 bytes in 1.345s)
 *** All done! :)

Then the phone needs to be rebooted. To check if the spoof was successful, start the MicroG services app and run the self test. Also, this needs to be ran after each reflash.

Also note that because I am using AdAway, I had to put mtalk.google.com on my whitelist otherwise "problems could occur" (source).

Note that I previously used Needle to perform the same task, but it somewhat failed after a while (the signatures were not spoofed anymore). I disliked the way it was shipping binary files and required running as root. It is (was?) also statically compiled which meant I had to re-enable multi-arch in debian and install the lib32z1 32-bit package. So I figured I would give the fork (tingle) a shot.

Swype keyboard

Running without any GAPPS also means that the "swype" keyboard (gesture-based typing) doesn't work at all even if it is enabled, because a library is missing. I found this tutorial which explains how to install the missing library (libjni_latinimegoogle.so) into /system/lib. Note that you cannot install it directly there from the adb shell, you will get:

$ sudo adb push Optional/swypelibs-lib-arm/common/lib/libjni_latinimegoogle.so /system/lib
failed to copy 'Optional/swypelibs-lib-arm/common/lib/libjni_latinimegoogle.so' to '/system/lib/libjni_latinimegoogle.so': Read-only file system

But the root-enabled file-manager can copy the file in place no problem. It could also be possible to sideload those files in place, but since the above works, I didn't bother.

The file can be found in the swypelibs-lib-arm.tar.xz that is shipped in the open_gapps-arm-5.1-pico-20160623.zip package available from opengapps.org. The file is presumably proprietary software as well, but it seems like a lesser evil.

Update: somehow I don't really use this all that much and stopped adding the proprietary library after the latest upgrade.

F-Droid privileged extension

This allows you to turn of that "allow untrusted sources" checkbox and enables automated upgrades, see the privileged extension project page for more information.

You need to download the .zip file from the privileged extension site and sideload it the usual way:

$ sudo adb reboot recovery
# ... phone reboots in TWRP
# choose "Advanced" -> "Sideload", then swipe
$ sudo adb sideload org.fdroid.fdroid.privileged.ota_2000.zip
# ... will sideload, swipe to reboot

Signal

This section, which should probably become a separate blog post on its own, talks about the Signal software and my experiences with it. It's still largely incomplete but helps me connect the dots again when I need to.

Mixed signals

I have experimented with Signal, a secure messaging and phone app. The problem is there are many forks of Signal, and it's pretty confusing which is what. I updated the LibreSignal FAQ with the details, but basically you have:

• Signal, from WhisperSystems, available from Google Play and Eutopia.cz as (confusingly) LibreSignal
• LibreSignal, also known as the "Websocket" or "JavaJens" fork, which implements GCM-less polling, but lacks server-side support (hence the lack of voice support). available from the Eutopia.cz experimental repository and the f-droid archive.

So from here on, I will use LibreSignal to refer to Eutopia.cz's build of upstream Signal unless it also includes "Websocket" or "JavaJens" in which case it means the (mostly broken) websocket fork. "Signal" is the regular upstream build that is only available through Google Play, which I do not use because I do not want to install that proprietary application on my phone, and can't be bothered manually checking and deploying upgrades.

Note that to use LibreSignal or Signal, you need a working GCM implementation, which can be done by installing opengapps or spoofing signatures, see below. This needs to be done before Signal is installed.

Migrating from LibreSignal to Signal (old procedure)

I eventually tried migrating from LibreSignal to the native Signal client, using gplaycli to download the APK without the Google Play Store, which is somewhat an issue because I won't actually know when/if Signal needs to be updated.

To do that, you need to uninstall LibreSignal and reinstall Signal, because the signatures changed.

1. unregister in LibreSignal (same procedure as unregister from Signal)
2. export text messages
3. uninstall LibreSignal
4. install Signal (I used gplaycli -d org.thoughtcrime.securesms, copied the APK file over using Syncthing and opened it in the file browser)
5. go through the registration process again which at first fails because it doesn't receive the SMS message (even though I actually see it come through the VoIP.ms SMS client), so I need to wait for the 2 minute timeout and do the voice authentication procedure.
6. import text messages

Update: there's a better way to migrate from Signal using Oandbackup. It should be similar to the procedure below, however.

Migrating from LibreSignal to Signal (new procedure)

Since Signal now provides APKs outside of the Google Play Store, I have switched back to using the official Signal packages. The first install (or upgrade from LibreSignal) should look something like this:

1. Download the APK from Signal APK distribution site

2. Verify the signature using apksigner:

   $ apksigner verify --print-certs Signal-website-release-4.1.0.apk 
   Signer #1 certificate DN: CN=Whisper Systems, OU=Research and Development, O=Whisper Systems, L=Pittsburgh, ST=PA, C=US
   Signer #1 certificate SHA-256 digest: 29f34e5f27f211b424bc5bf9d67162c0eafba2da35af35c16416fc446276ba26
   Signer #1 certificate SHA-1 digest: 45989dc9ad8728c2aa9a82fa55503e34a8879374
   Signer #1 certificate MD5 digest: d90db364e32fa3a7bda4c290fb65e310
   WARNING: META-INF/services/com.fasterxml.jackson.core.JsonFactory not protected by signature. Unauthorized modifications to this JAR entry will not be detected. Delete or move the entry outside of META-INF/.
   WARNING: META-INF/services/com.fasterxml.jackson.core.ObjectCodec not protected by signature. Unauthorized modifications to this JAR entry will not be detected. Delete or move the entry outside of META-INF/.
   
   $ diff <(echo 29:F3:4E:5F:27:F2:11:B4:24:BC:5B:F9:D6:71:62:C0 EA:FB:A2:DA:35:AF:35:C1:64:16:FC:44:62:76:BA:26  | sed 's/[: ]//g') <(echo 29f34e5f27f211b424bc5bf9d67162c0eafba2da35af35c16416fc446276ba26 | tr [a-z] [A-Z])
   $
   

3. Copy the APK to your phone, somehow. I used jmtpfs to copy it to the phone using a USB cable.

4. If this is not an upgrade, just install the APK by tapping it in the file manager, and you are done. Otherwise, you first need to make a backup and uninstall Libresignal.
5. Install Oandbackup
6. Backup LibreSignal with Oandbackup, choose data (and not apk) when prompted
7. Uninstall LibreSignal
8. Install Signal by tapping the APK in the file manager
9. DO NOT START SIGNAL! That would create a new identity.
10. Restore from Oandbackup

Why I do not recommend Signal to my fellow users

I am really ambivalent about Signal. I installed the fork, the libre build, and also tried the official build. I still don't feel comfortable recommending it to other users, mainly because of the way the project is managed by Open Whisper Systems (OWS). The specific reasons are multiple:

• OWS have been pretty hostile with other projects wanting to do derivatives of their works. In particular, they have refused to allow federation with other potential Signal servers, going so far as questionning the very idea of federation, and have refused to re-license their work to allow ChatSecure to include Axolotl in the Apple store (although that latter issue is more a problem with the Apple Store than OWS, and was later resolved, see this instructive post from moxie for more background on this)
• Signal doesn't integrate well with my current VoIP-based SMS solution: it assumes your phone can deliver SMS directly, and they refuse to interoperate with other alternative apps
• Signal requires data. Where I live, data plans are still expensive and people regularly keep them turned off. This means that if I send a Signal message to a friend, I can't assume they will receive it instantly: I often go back to my regular SMS app and send a clear-text message for them to turn on their data plan and read my "secure" message, which defeats the whole point of the app. I usually end up just sending the message in the clear instead.
• Signal assumes SMS are a secure way of proving your identity: your identity is mapped, one to one, to your phone number, which creates a whole new class of vulnerabilities like [VoIP-based impersonation][]
• "Controversial" issues (in the eyes of OWS) are locked down to keep discussions from evolving
• Signal requires proprietary Google services and software. In particular, it requires Google Cloud Messaging, which keeps a socket opened from your phone to Google's servers permanently. OWS has consistently refused to implement alternative notification mechanisms. There are ways to use Signal with MicroG instead of the builtin Google Play Services GCM implementation, and download the LibreSignal build from Eutopia.cz instead of the Google play store, but those are not trivial for new users. I think it is bad practice, specifically with security-sensitive software, to encourage people to use proprietary software.
• There is a great alternative for mobile that is federated, and purely free software: XMPP with OMEMO, and great clients: Conversations or ChatSecure (which was renamed to Zom.im although it still doesn't support OMEMO. see below for more alternatives
• Limited desktop interoperability: while it is possible to run the Signal app in a Chrome browser (another Google software dependency, and a heavy one at that), the functionality is limited: no voice support, for example. it takes about 100MB of ram just for that app (not counting chrome itself). A more open protocol and client would see third party clients be implemented...
• lack of videoconferencing support, although that affects most messaging solutions, including XMPP and others for now. only Skype and WebRTC occupy this problem space, AFAIK

Alternatives

The problem space Signal is trying to address is huge, both in terms of actual problems to address (usability, security, etc) than in terms of available solutions. Here's the a short list of non-exhaustive solutions I'm trying to keep an eye on:

• OTR, OTRv4 and OMEMO on XMPP, with Conversations or Zom.im
• Tox
• Whatsapp, Google Allo, Facebook Messenger, which all include Signal-like protocols now
• Skype (yes, for videoconferencing, something none of the above really address)
• WebRTC - only videoconferencing, and no e2e encryption
• IRC (yes, that old fart)
• Matrix.org, Slack, Mattermost, and so many similar others
• ... and more, too many to note down here

I wish I had time to sit down and keep track of all those that are out there. The landscape is horribly fragmented and most of the time, none of those apps talk to each other.

Upgrades

Cyanogenmod frequently checks for upgrades and will offer you to install new versions. Be careful when doing that: boot.img sometimes needs to be flashed as mentioned above. You will know you will need to reflash it when Wifi or other parts of the system don't work.

The above signature spoofing needs to be reinstalled as well, as documented above. The Swype keyboard hack also, although I have stopped doing that recently.

Tricks

adb shell - really useful, gives a shell from your computer, easier to type and copy/paste stuff.

Restoring stock firmware

Not entering the crypto passphrase in TWRP makes it possible to destroy the encrypted partition in Wipe. It turns out that one of my attempt at enabling encryption resulted in the old data being still present.

Boot into TWRP, and push the backup at exactly the right location:

adb push -p HT26PW407343 /sdcard/TWRP/BACKUPS/HT26PW407343

This can also be done with the OTG cable. The backup is in a directory called Stock ROM 2016-03-17--19-17-35 and was last seen on curie, copied over from the calyx external drive, itself moved over from angela.

Go into Restore and select all options to restore from backup.

Reboot into bootloader and flash the old boot sector backup:

$ fastboot flash boot HT26PW407343/Stock\ ROM\ 2016-03-17--19-17-35/boot.emmc.win
sending 'boot' (16384 KB)...
OKAY [  1.857s]
writing 'boot'...
OKAY [  1.557s]
finished. total time: 3.415s

Reboot.

Update:

The HTC dev center only has source code, not official rom files.

Developping for Android

This is more of a general note about android. Obviously, the next step is to start building software for that crazy platform. I was trying to build the Locker app for myself, so I tried to figure out how you build Android apps from source.

I used the SDK straight from Google because the wrappers in Debian do not ship the required android and gradle binaries necessary to build packages just yet. It seems the Debian packaging of Android is not yet complete: I have looked in the google-android-build-tools-installer, android-platform-tools-base and android-system-dev packages without any luck. An alternative would have been to install the libre Android rebuilds, but I discovered that only after being told by a friend about this blog post and mistakenly editing the AndroidTools wiki page, which has since then been fixed.

This documentation says to use android update. fdroid tries to use gradle, which is in the SDK, but needs to be symlinked in the path from gradlew to gradle. And even then, it fails because:

> Gradle version 2.2 is required. Current version is 1.12. If using the gradle wrapper, try editing the distributionUrl in /home/anarcat/src/fdroiddata/build/net.zygotelabs.locker/android/app/gradle/wrapper/gradle-wrapper.properties to gradle-2.2-all.zip

The SDK has a UI in ./tools/android that downloads a bunch of stuff (5GB+!), but still doesn't fix that issue. I have given up on building APKs for now and removed that archive until next time, especially since I found an alternative to Locker that is already in F-Droid.

Note that I have had better experience with the fdroidserver package, but it still depends on the Android APK to do anything, so I could only do some formatting and cleanup on the pull request.

Update: back from the dead

Update: this is back from the dead! I was able to restore wifi somehow. The reset procedure was simple, as detailed below:

sudo fastboot devices
sudo adb reboot recovery
sudo adb sideload cm-12.1-20160822-NIGHTLY-ville.zip
unzip cm-12.1-20160822-NIGHTLY-ville.zip boot.img
sudo fastboot flash boot boot.img

Not sure why I couldn't make this work last I tried. Now, of course, I'm running an old version of Android that's probably insecure. There's no way around this: the last "mod" that supported this phone is Cyanogenmod and that project has died, replaced with LineageOS which barely supports any HTC device whatsoever. But it beats having no phone at all.

I've found nice ways of copying data between the two phones, and I updated the install procedure above accordingly.

Only problem is the Transit app is not compatible with this older Android release.

Another problem is the phone is actually locked to Telus, so I had to do the stupid "network unlock" dance. There are many sites doing this online, so I picked some random one (canadaunlocking.com, found in a startpage search for "android sim unlock") and paid them 10$USD via Paypal for some semblance of security. Worst thing is they have my IMEI number mapped to my Paypal account I guess and/or they steal my money. Other such sites include unlockradar.com (suggested by Quora) and unlockriver.com (found in this youtube video).

As of 2019-02-26T11:50:00EST, no mail had arrived from canadaunlocking.com. Paypal payment was confirmed on 11:39:26EST. They claim to be able to unlock the phone within "5-90 minutes" but the unlocking process actually took a little over 8 hours and was finally shipped at 19:50:43 after being delayed an hour by graylisting. The code, unfortunately, was not working: the unlock failed with "SIM network unlock request unsuccessful". The reason for this is unlocking works only on stock firmware, so I had to dig the original firmware out of my backups and reflash the whole thing (twice) to actually unlock the phone. But a day later, it's done!

Future work

• Restore procedures: wipe the phone with a new firmware and restore all apps - how long does that take?
• Fix encryption: the SD card may just encrypt if we reinstall with the sdcard formatted as ext4 failed. it seems even in ext4, it doesn't actually encrypt the sd card.
• Remove the Google Apps completely Done: Open Gapps is not really open, and features the app store and google play services, which we we can workaround with stuff like Raccoon, gplaycli, Yalp Store or APK Downloader (to download APKs without the Play Store) and MicroG for a free software version of the Google Play services
• Install the official Signal using the above, since we want to make crypted phone calls, until LibreSignal supports Voice calls and the VoIP.ms SMS supports GCM-less push notifications Done, see above.
• Make GPS-less geolocation work. Right now, the GPS works, but it's the only thing that works. There are various location services that can use your Wifi to determine location without talking to Google, which is useful to me when i want to use my phone to plan routes inside the house without having to go outside for the GPS to get a fix. Removing gapps made that work well, along with the fake GSM services described above.
• Install F-Droid using the privileged extension so we don't have to tick the nasty "allow 3rd party APKs"

References

• Cyanogenmod: the free software operating system for Android phones, often abbreviated to CM, now forked into LineageOS
• HTC One S: the device I got for now, see also the XDA developers wiki on the HTC One S
• TWRP (TeamWin Recovery Project): the Recovery ROM I am using (apparently, CM has its own mod which is sometimes recommended over TWRP, unclear. TWRP is also an awesome prog band
• Developer options: the builtin "developer" menu in any Android phone that is sometimes accessible if the phone is somewhat unlocked. Tons of stuff available there.
• adb: desktop client to manage and debug an android phone through USB or the network
• fastboot: a desktop tool to flash and manipulate the bootloader and recovery images
• Libre Android rebuilds: liberated builds of the Android source code
• Debian packaging of Android: Debian packaging effort of parts of the above
• Open Gapps: freely downloadable Google apps that can be flashed on the device on install since they are not part of CM (e.g. Google Play Store, etc). Not free or open source software! Still covered by license agreements and all, and will spy on you.
• MicroG: liberated version of the above. Features free software alternatives for the GCM, Google Play, Google Maps, the "Unified Network Location Provider (UnifiedNlp)" and, eventually, the Google Play store itself. Note that there are still privacy issues with GCM and so on.
• F-Droid: excellent repository of completely free and open source software for Android.
• Aopensource: directory of free software apps, not a repository and links to the Google Play store. Logically, all of those should be on F-Droid...?
• Apps list: the irregularly maintained list of apps i use on the phone, generated with List My Apps