1. Turn on the VNC server by enabling System Preferences -> Sharing ->  Screen Sharing. Even though it provides little security, a VNC password must be set so that OS X will present an authentication scheme that makes sense to a standard VNC client. Enable VNC viewers may control screen with password
   
2. After connecting, you will see a grey linen-backgrounded desktop with nothing in it. Type your user name and password. After logging in, your desktop contents will display!

I don’t like the idea of advertisers, search engines, and social networks building extensive profiles about what I do online (why). A short-list of tools to avoid such tracking:

Prevent Inter-Website Tracking

• Abine’s Do Not Track Plus is nearly a one-stop shop. I wish more details were available about what it does, but the gist is:
• Install and maintain a large number of generic do-not-track-me cookies for many ad networks and tracking services. When content is fetched from these sites, the generic cookie is sent rather than one which is unique to you
• Special handling for social buttons (e.g. Like this on Facebook), in which the button fetched anonymously, but, should you choose to click it, the veil is lifted and the Like associated with your account
• Many ads are blocked from rendering too, which I hadn’t expected. Those that remain are innocuous enough that I do not use Ad Block Plus any more.

Reduce Google Information Gathering

I store some personal information on Google (thanks to Google+, Google Calendar, etc.). I do not want to Google to associate that personal information with all the web searches I do every day. Do Not Track Plus is of limited value here: if you sign in to Google, Do Not Track Plus will be obliged to permit your identity to be sent. Additional steps are needed:

• Don’t search with Google. I prefer Duck Duck Go for most searches, thanks to their Zero-click Info and other goodies.
• For needle-in-the-haystack searches, I find Google often has the best results. Startpage is an anonymous Google Search proxy. Rather than use it directly, I just prefix my Duck Duck Go searches with !sp when needed.
• Straight Google (requires Greasemonkey) prevents Google’s click-tracking. This is less important if you follow the above steps to avoid doing your web searches at google.com. However, they still track links clicked on their other products, which Straight Google can prevent.

Control Intra-Website Tracking

The above steps should take care of attempts to track your movement across the web. However, most websites will still store long-term cookies in your browser to track your history of interaction with that particular website.

• Cookie Whitelist is designed to only allow white-listed cookies from being accepted. In practice, this breaks too many websites. For less hassle, configure as follows:
• Cookie button (the red one): ON. This lets any website set a cookie, but it will be deleted at the end of the session
• For the few websites you wish to remain logged in to (or otherwise personalized) click the green button to whitelist as needed
• Do not accept third-party cookies
• BetterPrivacy is to Flash LSOs (local shared objects, or Flash cookies) what Cookie Whitelist is to regular cookies. Alas with a more confusing set of configuration options.

Note: this post is (obviously?) not about how to avoid your employer/ISP/government monitoring what you do online. To hide what you are doing from someone who has access to all your traffic, you need encryption and proxying. A good first stop to get some encryption is EFF’s HTTPS Everywhere. This goes a long way to prevent the person nearby in the coffee shop from stealing your Facebook account.

Originally published 2012-03-10. Updated 2012-03-14 with intra-website tracking steps.

But, upon click, URLs in Google search results change to go to Google first!

Straight Google removes this tracking from Google URLs across all Google products. Easy to install and no configuration needed, but you must install Greasemonkey first.

% defaults write com.jetbrains.intellij ApplePressAndHoldEnabled -bool false
% defaults write com.jetbrains.rubymine ApplePressAndHoldEnabled -bool false

But how do you figure out what the magic identifier for your application is? Simple: defaults domains will list them all:

defaults domains | gsed -e 's/, /\n/g' | grep jetbrains
com.jetbrains.intellij
com.jetbrains.intellij.ce
com.jetbrains.rubymine
jetbrains.communicator.core

Note that in order to munge the commas into newlines for grep, gsed was required because OS X default sed cannot (easily) insert newlines.

% echo foo,bar,baz | sed -e 's/,/\n/g'
foonbarnbaz

The simple solution is to use GNU sed, which is already installed (as gsed), instead of the default BSD version:

% echo foo,bar,baz | gsed -e 's/,/\n/g'
foo
bar
baz

Alternatively, it is possible (though fiddly) to trick BSD sed into inserting newlines using extquotes.

#!/usr/bin/python
import os
import re
where = '.' # source directory 

ls = os.listdir(where)
for f in ls:
  m = re.search('.*_COMM-([0-9]+).xml', f)
  if m:
    subdir = "%03d" % (int(m.group(1)) / 1000)
    try:
      os.mkdir(subdir)
    except OSError as e:
      pass
    os.rename(f, os.path.join(subdir, f))

Now on to the random selection, again with Python:

#!/usr/bin/python
import os
import random
import re
import sys

if len(sys.argv) > 1:
  where = sys.argv[1]
else:
  where = '.' # source directory 

subdirs = filter(lambda x: re.search('^[0-9]*$', x), os.listdir(where))
subdir = os.path.join(where,random.choice(subdirs))
print os.path.join(subdir,random.choice(os.listdir(subdir)))

A quick shell loop leverages the Python script to grab files and dump into a repository of test data. Works on ZSH, Bash, perhaps others:

for i in {1..250}; do cp $(./pick_a_file.py sub_dir_with_files) /destination/dir/filename_prefix_$(printf "%03d" $i).xml; done;

I often observe this:

1. Delete message on the way to work using K-9 on my phone.
2. Arrive at work and message is gone from my Inbox in Mail.app
3. Come home, download new mail in Thunderbird and see an Inbox full of undead messages.

No amount of re-expunging and re-fetching mail helps. Grepping through the server-side Maildir shows the messages really are gone from the folders in which Thunderbird is still showing them.

It turns out the reason they are still displaying in Thunderbird is mundane client-side index corruption. To clean things up:

1. Right-click on mailbox
2. Choose Properties...
3. Click Repair Folder
4. Rejoice at tidy mailbox

> Array(42)
 => [42]
> Array([1,2,3])
 => [1, 2, 3]
> Array(nil)
 => []
> Array("foo")
 => ["foo"]
> Array({"foo" => "bar", "biz" => "baz"})
 => [["foo", "bar"], ["biz", "baz"]]

What went wrong is that Array() calls the (now deprecated) to_a on each of its arguments. Hash has a custom to_a implementation with different semantics. Instead, do  this:

class Array
  def self.wrap(args)
    return [] unless args
    args.is_a?(Array) ? args : [args]
  end
end

That yields the expected results, even for Hashes:

> Array.wrap(42)
 => [42]
> Array.wrap([1,2,3])
 => [1, 2, 3]
> Array.wrap(nil)
 => []
> Array.wrap("foo")
 => ["foo"]
> Array.wrap({"foo" => "bar", "biz" => "baz"})
 => [{"foo"=>"bar", "biz"=>"baz"}]

Use of is_a? is deliberate; duck-typing in this situation ([:[], :each].all? { |m| args.respond_to? m }) yields unexpected surprises since e.g. String is Enumerable and would not get wrapped.

For further discussion see Ruby-forum thread “shortcut for x = [x] unless x.is_a?(Array)” and StackOverflow “Ruby: Object.to_a replacement“.

<?xml version="1.0" encoding="UTF-8"?>
<foo:Root xmlns:foo="http://www.foo.bar/fnarf/foo">
  <foo:child>
    ...
  </foo:child>
  <foo:child>
    ...
  </foo:child>
</foo:Root>

…xml_split can create many files, each containing:

<?xml version="1.0" encoding="UTF-8"?>
<foo:child>
  ...
</foo:child>

However, this loses the namespace declaration and the enclosing root element. Luckily, a little sed magic can bring those back:

find . -name '*.xml' | xargs -n1 sed -e '1 a\
<foo:Root xmlns:foo="http://www.foo.bar/fnarf/foo">
' -e '$ a\
</foo:Root>
' -i ''

find lists all the files, xargs invokes sed on them one by one (-n1), and sed adds the opening tag with namespace declaration after the first line (1 a) and the closing tag after the last line ($ a). Now each file looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<foo:Root xmlns:foo="http://www.foo.bar/fnarf/foo">
  <foo:child>
    ...
  </foo:child>
</foo:Root>

This evening I was preparing for another recalibration attempt by looking for a way to disable the beeping when power is disconnected. It turns out apctest can disable the alarm. In the process, I noticed apctest can also read and write the battery date. On a whim, I updated the date. And, magic: Merely changing the battery date fixed the reported runtime, charge, and load percentages!

Changing the battery date back to the original value did not bring the bogus readings back. Presumably the behavior is based on dead-reckoning of time elapsed since last battery change rather than any knowledge of what the current date actually is.

A common way to install Ubuntu is with an underlying ext4 file system and eCryptfs encrypted home directories. ext4, like many other file systems, has a maximum filename limit of 255 bytes. eCryptfs creates filenames much longer than the original. Compiled Scala classes tend to have long file names since anonymous classes end up in their own files. Therefore, when compiling Scala projects within eCryptfs on ext4, it is easy to get file name too long errors.

def f(x,y)
  x*y
end

> fArgs = [6,7.0]
=> [6, 7.0]

> f(*fArgs)
=> 42.0

Scala does not have a splat operator per se, but you can achieve the same effect without too much work. Sadly the syntax is different for fixed-arity and variadic functions.

Scala splat for variadic functions

For variadic functions there effectively is a splat operator. If you invoke a variadic function and append :_* to the argument the compiler will perform the splat:

> def g(xs:Int*) = (0 /: xs) (_ + _)
g: (xs: Int*)Int

> val gArgs = List(1,2,3,4)
gArgs: List[Int] = List(1, 2, 3, 4)

> g(gArgs:_*)
res23: Int = 10

Scala splat for fixed-arity functions

> def f(x:Int, y:Double) = x * y
f: (x: Int, y: Double)Double

> val fArgs = (6, 7.0)
fArgs: (Int, Double) = (6,7.0)

> f _ tupled fArgs
res8: Double = 42.0

Magic! The first part, f _, is the syntax for a partially applied function in which none of the arguments have been specified. This works as a mechanism to get a hold of the function object. tupled returns a new function which of arity-1 that takes a single arity-n tuple. It is defined in the Scala Function object,

However, given a List of arguments to pass to f, I’m not sure how to easily convert the List to a Tuple.

p.s. There’s a stackoverflow post about this called “scala tuple unpacking.”

Rebase

First, set rebase for every new upstream branch (Why rebase? It makes your history easier to understand.)

git config --global branch.autosetuprebase always

This is explained in more detail (and with other helpful hints) in Mislav Marohnić’s post A few git tips you didn’t know about.

Tracking

Second, make git push only send the current branch to its matching upstream (aka tracking) branch. (Otherwise the default behavior is to push all branches that have the same name on both ends.)

git config --global push.default upstream

This is covered in some detail in Mark Longhair’s post An asymmetry between git pull and push.

On any existing branches, you can set up tracking by doing an explicit push:

git push -u origin branchname

Default refspec

At this point you should be set according to all the tutorials I came across. In my experience, however, this only works for branches other than master. A plain git push on master yields the error:

fatal: The current branch master has multiple upstream branches, refusing to push.

The solution is to set the default refspec for git push. I’m unclear on why this needed for master but not for other branches.

git config remote.origin.push HEAD

The first concern I came across is that it is important to use a hash algorithm that’s slow (e.g. bcrypt or scrypt). Otherwise it’s too easy to brute-force the master password based on a site password. Suppose site you visit stores your password in plaintext and gets hacked. That breach then compromises your master password, even though only your site-specific password was revealed.

I couldn’t find a JavaScript implementation of scrypt, but I found a JavaScript bcrypt implementation. Better yet, I found a derivative that tidies up the first one, removing dependencies on e.g. ClipperZ, and wraps it in a simple bookmarklet. SuperGenPass provides a much more user-friendly bookmarklet, so I started gearing up to replace it’s MD5 hashing with bcrypt.

But, alas, SuperGenPass (and any other simple bookmarklet) is not secure in the face of a malicious website that contains JavaScript designed to sniff entry of the master password in to the bookmarklet. PwdHash is a browser extension based approach from the Stanford Security Lab designed to combat the weaknesses of the bookmarklet based approach. Their paper, Stronger Password Authentication Using Browser Extensions, is interesting reading and explains a variety of ways to compromise a bookmarklet based approach. PwdHash has already spawned a number of ports to other browsers and mobile devices, but alas they’re all based on prototype code that uses HMAC-MD5 as the hashing algorithm (even though the paper points out PwdHash is a good candidate for a better hashing algorithm).

I was not able to find any PwdHash derivative that used bcrypt. I did find a simple command-line tool based on scrypt, but that’s not great if you don’t have easy access to your own computer.

Solutions like PassPack offer the potential to solve these problems (extension rather than bookmarklet, use of strong encryption rather than weak hashing), but have an Achilles heel of their own: the service provider has the power to decrypt all your passwords. For now I’ll stick with my moinmoin-client-crypt approach.

import actors.Actor
import actors.Actor._

trait FooTraitConfiguration { val times : Int }

trait FooTrait { self:FooTraitConfiguration =>
  case object Ping
  case object Pong

  val a = actor {
    loop {
      react {
        case Ping =>
          self ! Pong
        case Pong =>
          for(_ <- (1 to times)) { print(".") }
          System.out.println("pong.")
  } } }

  def ping = a ! Ping
  def pong = a ! Pong
}

class Foo extends FooTrait with FooTraitConfiguration { @Override val times = 5 }

But, alas, this fails to compile:

error: value ! is not a member of FooTrait with FooTraitConfiguration
self ! Pong

It seems that the self-trait has broken the Actor API! And indeed, it has. Because self-traits are not usually specified with self! It should have been:
this:FooTraitConfiguration =>

The self-type means that within FooTrait the type of this is considered to be FooTrait with FooTraitConfiguration. Using a word other than this additionally sets up an alias to that type for e.g. use within nested classes. And there’s the rub: Actors depend on a method named self which is shadowed when the alias to the type is named self.

Note to self: Don’t use self when specifying self types!

2011-04-11 09:40:12,004 [INFO] [http-3] Hello
2011-04-11 09:40:13,554 [DEBUG] [http-1] Wikipedia
(pronounced /ˌwɪkɨˈpiːdi.ə/ WIK-i-PEE-dee-ə)
2011-04-11 09:40:13,605 [INFO] [http-2] PCC Natural Markets
2011-04-11 09:40:13,688 [INFO] [http-3] World
2011-04-11 09:40:14,015 [INFO] [http-2] began as a food-buying club of 15 families in 1953.
2011-04-11 09:40:16,032 [INFO] [http-1] is a multilingual, web-based, free-content encyclopedia project based on an openly editable model
2011-04-11 09:40:17,775 [INFO] [http-2] Today, it's the largest consumer-owned natural food retail co-operative in the United States.

becomes:

09:40:12,004	Hello
09:40:13,554		Wikipedia
09:40:13,554		(pronounced /ˌwɪkɨˈpiːdi.ə/ WIK-i-PEE-dee-ə)
09:40:13,605			PCC Natural Markets
09:40:13,688	World
09:40:14,015			began as a food-buying club of 15 families in 1953.
09:40:16,032		is a multilingual, web-based, free-content encyclopedia project based on an openly editable model
09:40:17,775			Today, it's the largest consumer-owned natural food retail co-operative in the United States.
            	http-3	http-1	http-2

Just read down the columns vertical for a clear chain of events on each thread. Code below is maintained on GitHub.

#!/usr/bin/env python

# Looks at token in a particular position in each line and indents the line
# differently for each unique identifier found in the file. For example, given
# a log file which contains a thread identifier, contents for each thread will
# be separated out into distinct columns.
#
# Lines not matching the pattern (e.g. stack traces) are presumed to have
# occurred at the time of and belong to the same identifier as the preceding line.
#
# Default pattern is: <date> <stamp> ignored [thread_id] <message>
# Yielding output: <stamp><tabs><message>
#
# An alternate regular expression can be supplied on the command line; it must
# include named capture groups 'stamp', 'id', and 'message'. The default regex
# is: ^\S+ (?P<stamp>\S+) \S+ \[(?P<id>[^\]]+)\] (?P<message>.*)
#
# If the input contains very long lines it can be helpful to truncate them
# beforehand by e.g. piping through awk '{print substr($0,0,400)}'

import sys,re
if len(sys.argv) > 1:
  pattern = re.compile(sys.argv[1])
else:
  pattern = re.compile('^\S+ (?P<stamp>\S+) \S+ \[(?P<id>[^\]]+)\] (?P<message>.*)')

delimiter='\t'
max_level = 1
categories = {}
legend = None
indent = ""
stamp = ""

try:
  for line in [l.strip() for l in sys.stdin]:
    m = pattern.match(line)
    if m:
      stamp,identifier,message = [m.group(x) for x in ['stamp','id','message']]
      indent = categories.get(identifier)
      if not legend:
        legend = " " * len(stamp)
      if not indent:
        indent = delimiter * max_level
        categories[identifier] = indent
        max_level += 1
        legend += delimiter + identifier
      print stamp + indent + message
    else:
      # carry over stamp and indent from previous line
      print stamp + indent + line

  print legend
except IOError:
  pass

Usage:

 <example.log| ./by_time.py 9:40 9:44:15

Code below is maintained on GitHub.

#!/usr/bin/env python

# Selects time range from a log file. Lines with no time (e.g. stack traces)
# are presumed to have occurred at the time of the preceding line.
#
# Assumes first time-like phrase on a line is the timestamp for that line.
#
# Assumes time format is pairs of digits separated by colons with optional , or
# . initiated suffix. E.g. HH:mm:ss,SSS, HH:mm, etc.
#
# Does not strip blank lines; just use awk 'NF>0' for that.

import sys,re
time_pattern = re.compile("(?:^|.*?\D)(\d{1,2}(?::\d{2})+(?:[,.]\d+)?)")
fields_pattern = re.compile("[:,.]")

if len(sys.argv) < 3:
  print >> sys.stderr, "Please specify start and end times (e.g. %s 13:50 14:10:01,101)." % sys.argv[0]
  exit(1)

for item,index in [["start time",1],["end time",2]]:
  if not time_pattern.match(sys.argv[index]):
    raise ValueError("Cannot parse %s: %s" % (item, sys.argv[index]))

start,end = [[int(x) for x in re.split(fields_pattern, s)] for s in sys.argv[1:3]]
too_soon = True

try:
  for line in sys.stdin:
    line = line.strip()
    m = time_pattern.match(line)
    if m:
      t = [int(x) for x in re.split(fields_pattern,m.group(1))]
      if t >= end:
        break
      elif too_soon and t >= start:
        too_soon = False

    if not too_soon:
      print line
except IOError:
  pass

In Firefox 3.6 I had used BarTab, a great plug-in that allowed unloading tabs without closing them to reclaim memory. Alas, it is not yet compatible with Firefox 4.

But it turns out Firefox 4 has a new setting (called Cascaded Session Restore) that provides a reasonable work-around. In about:config set

browser.sessionstore.max_concurrent_tabs=0

…and then, when closed and re-opened, Firefox will not load any tab until you click on it. Now it’s extremely fast to quit, restart, and continue on with reduced memory usage.

After some aborted attempts at translating the JSPWiki source from scratch, I decided the path of least resistance would be to leverage HTML::WikiConverter to translate the HTML output of JSPWiki. This turned out to be time consuming as well. To anyone else going down this path I offer up:

1. A patched version of HTML::WikiConverter-MoinMoin that includes fixes for intra-wiki links, inline images, horizontal rules, and definition lists.
2. A collection of scripts, dubbed JSPWiki-translate-perl, for retrieving HTML from JSPWiki, pre-processing it to make it more palatable for HTML::WikiConverter, and for generating a MoinMoin-style directory layout to contain it.

The original author of HTML::WikiConverter-MoinMoin seems to have abandoned it. I can sympathize; I certainly hope to avoid translating another Wiki any time soon!

moinmoin-client-crypt provides client-side encryption/decryption of MoinMoin wiki pages (or portions thereof). It adds encrypt/decrypt buttons to the edit screen, providing an easy mechanism to secure all or a portion of the content. Encryption is via Chris Veness’ Javascript AES implementation (256 bit key, CTR mode).

Installation involves dropping a couple JavaScript files into the appropriate MoinMoin directory and tweaking the theme init file to reference them. Full functionality with modern and classic themes, perhaps slightly degraded on others. It shouldn’t take much tweaking to adapt to other themes; patches and bug reports are welcome!

The client-side JavaScript approach provides some security if the server were to be seized: the AES ciphertext should be extremely difficult to crack. Also, once the browser is closed on the client side, there should be no trace left of the plaintext. However, if the server were compromised it would be easy to replace moinmoin-client-crypt with a trojan horse if a malicious person were to gain control of the client, they could easily install e.g. a keylogger you have to trust your client machine, your browser, your connection to the server, and the integrity of the server, as explained here by Nate Lawson. The need for client-side security should be obvious; the server and connection must be trusted not to send/inject a modified version of the script.