A bug caused by "make clean"

A Task Coach user reported today that one of the translations (Simplified Chinese) was not working. Not working meaning that instead of the translated texts in the user interface, the original English texts would be shown. A little investigation showed that most translations were OK, but a few were not.

Now, it is necessary to know how Task Coach deals with translations. Task Coach uses Launchpad for translations. Launchpad provides the translations as .po files. These .po files are transformed into Python sources files that are in turn bundled with the different Task Coach installers/packages.

I noticed that a few of these generated Python source files were missing from the folder where the translations are stored. Asking myself how some of these could be missing while others were not, I decided that one possible explanation would be the Makefile not removing all files. So I checked the "clean" target in the Makefile and indeed, it would only remove the "??_??.py" files and not the "??.py" files. That means zh_CN.py would get removed, but nl.py not. So that explains why some translations, such as Simplified Chinese (zh_CN) and Brazilian Portuguese (pt_BR), were missing and others, such as Dutch (nl) and French (fr), not.

The final question is, of course, how to prevent this from happening ever again. We already have a set of release tests. I guess that adding a release test that checks whether all translations are included in the Task Coach installers and packages should do it.

Automating .deb building

Some time ago, Stani helped me to create a Task Coach package in Debian package format (.deb). This is the package format that is also used by Linux distributions derived from Debian, such as Ubuntu. Since I want the release process of Task Coach to be as easy as possible, I decided to automate the package build process as much as possible.

I wrote a distutils command that creates a Debian package from a source distribution, as created by python setup.py sdist. This new distutils command, called bdist_deb, copies the source distribution, unpacks it, adds the necessary Debian control files and compiles the package using the regular packaging tools.

The bdist_deb command takes a large number of parameter since it needs a lot of information to create the .deb. For example, application title, description, version, license information, copyright, author, maintainer, etc. In the case of Task Coach, most of the information is already available in a meta data source file, and easily passed to the bdist_deb command from the distutils setup script.

Since this was specifically developed for Task Coach it is probably not completely generalized. Nevertheless, I hope it provides a starting point for other developers that want to create proper Debian packages for their Python applications.

Check out the bdist_deb command and the invocation of the command in the distutils setup script (called make.py), starting at line 109.

Migrating from CVS to Subversion

I recently migrated the Task Coach source code from its Sourceforge CVS repository to a Subversion repository, also on Sourceforge. I followed the migration instructions at the Sourceforge website and all went smoothly.

Because the cvs2svn script needs the CVSROOT module present, you can only remove it after cvs2svn has run, i.e. you need remove it from the svndump file that cvs2svn creates. svndumpfilter does do that for you. Unfortunately, the CVSROOT module is present in the svndump file multiple times: not only in the trunk folder, but also in all branch and version folders of the svndump. A script to help with finding all those CVSROOT occurences would be nice.

Marcin Zajączkowski helped me out with a shell script. I used it as the basis for a little python script that removes specific folders or files from your svndump file. Use as you like.

---

#!/usr/bin/env python

import os, optparse

usage = '''Usage: %prog dumpfile path [path...]
Remove path(s) from <dumpfile> and write new dump file to <dumpfile>.out'''


class PathRemoverOptionParser(optparse.OptionParser):
    def __init__(self):
        optparse.OptionParser.__init__(self, usage)

    def parse_args(self):
        options, args = optparse.OptionParser.parse_args(self)
        if len(args) < 2:
            self.error('provide both dumpfile and path to remove')
        dumpFile, paths = args[0], args[1:]
        if not os.path.exists(dumpFile):
            self.error('dumpfile (%s) does not exist'%dumpFile)
        return dumpFile, paths


def branches(dumpFile, paths, pathPrefix='Node-path: '):
    ''' Find branches in the dumpFile that contain the paths. '''
    for line in file(dumpFile):
        if line.startswith(pathPrefix):
            for path in paths:
                if line.endswith(path+'\n'):
                    yield line[len(pathPrefix):-1] # yield branch


parser = PathRemoverOptionParser()
dumpFile, paths = parser.parse_args()
excludes = ' '.join(branches(dumpFile, paths))
if not excludes:
    parser.error('path(s) (%s) not found'%', '.join(paths))

os.system('svndumpfilter exclude %s < %s > %s.out'%(excludes, dumpFile, dumpFile))

Installing with or without administrator privileges

To create the installer for Task Coach on Windows, I use the excellent Innosetup tool. I upgraded to the latest version of Innosetup a while ago but didn't notice immediately that the installer created by this new version of Innosetup required the user to have administrator privileges. Because Task Coach is aimed at ordinary users, that is not acceptable.

It took me some time to find out how to have the installer work for both users with and without administrator privileges. I'm recording the solution here so that other developers may benefit from it.

In the registry section of the Innosetup script I included two versions of the lines that associate the ".tsk" extension with Task Coach. The first four lines (*) are used if the user has administrator privileges (Check: IsAdminLoggedOn), but the last four lines are used if the user has no administrator rights (Check: not IsAdminLoggedOn).

(*) I had to split the lines to prevent them from being clipped. The line continuations are indented.

[Registry]
Root: HKCR; Subkey: ".tsk"; ValueType: string; ValueName: "";
   ValueData: "TaskCoach"; Flags: uninsdeletevalue; 
   Check: IsAdminLoggedOn
Root: HKCR; Subkey: "TaskCoach"; ValueType: string; ValueName: ""; 
   ValueData: "Task Coach File"; Flags: uninsdeletekey; 
   Check: IsAdminLoggedOn
Root: HKCR; Subkey: "TaskCoach\DefaultIcon"; ValueType: string; 
    ValueName: ""; ValueData: "{app}\TaskCoach.EXE,0"; 
    Check: IsAdminLoggedOn
Root: HKCR; Subkey: "TaskCoach\shell\open\command"; 
    ValueType: string; ValueName: ""; 
    ValueData: """{app}\TaskCoach.EXE"" ""%%1"""; 
    Check: IsAdminLoggedOn
Root: HKCU; Subkey: "Software\Classes\.tsk"; ValueType: string; 
    ValueName: ""; ValueData: "TaskCoachFile"; 
    Flags: uninsdeletevalue; Check: not IsAdminLoggedOn
Root: HKCU; Subkey: "Software\Classes\TaskCoachFile"; 
    ValueType: string; ValueName: ""; ValueData: "Task Coach File"; 
    Flags: uninsdeletekey; Check: not IsAdminLoggedOn
Root: HKCU; Subkey: "Software\Classes\TaskCoachFile\DefaultIcon"; 
    ValueType: string; ValueName: ""; 
    ValueData: "{app}\TaskCoach.EXE,0"; Check: not IsAdminLoggedOn
Root: HKCU; Subkey: "Software\Classes\TaskCoachFile\shell\open\command"; 
    ValueType: string; ValueName: ""; 
    ValueData: """{app}\TaskCoach.EXE"" ""%%1"""; 
    Check: not IsAdminLoggedOn

Testing translations

A recent bug in Task Coach was caused by one of the translations being incorrect. So, I decided it was time to unittest the translations. For each translated string I wanted to check that certain conditions hold. For example, if the original string has a formatting operator (e.g. '%d' for digits) the translated string should contain the same formatting operator. These tests are relatively simple:

for formatter in '%s', '%d', '%.2f':
  self.assertEqual(self.englishString.count(formatter), 
                   self.translatedString.count(formatter))

The challenge is how to create one unittest for each (language, string)-pair. This is not a good solution:

def testMatchingFormatting(self):
  for language in getLanguages():
    for english, translated in language.dictionary():
      ...

because this unittest stops as soon as one translation is incorrect.

My first thought was that I could use decorators to unfold the loop, but after a few feeble attempts I decided I am not smart enough to wrap my head around decorators. After some more experimenting I ended up with the code below. I put the loop outside the test class and explicitly create a new TestCase class for each (language, string)-pair. This generates a lot of unittests (over 7600 for the current version of Task Coach), but they run in less than 0.5 seconds, so that's a small price to pay for increased test coverage.

import test, i18n, meta, string

class TranslationIntegrityTests(object):
  ''' Unittests for translations. This class is 
      subclassed below for each translated string 
      in each language. '''
      
  def testMatchingFormatting(self):
    for formatter in '%s', '%d', '%.2f':
      self.assertEqual(self.englishString.count(formatter), 
                      self.translatedString.count(formatter))
            
  def testMatchingAccelerators(self):
    # snipped


def getLanguages():
  return [language for language in \
          meta.data.languages.values() \
          if language is not None]
        

def createTestCaseClassName(language, englishString, 
  prefix='TranslationIntegrityTest'):
  ''' Generate a class name for the test case class based 
      on the language and the English string. '''

  # Make sure we only use characters allowed in Python 
  # identifiers:
  englishString = englishString.replace(' ', '_')
  allowableCharacters = string.ascii_letters + \
                        string.digits + '_'
  englishString = ''.join([char for char in englishString \
                           if char in allowableCharacters])
  className = '%s_%s_%s'%(prefix, language, englishString)
  count = 0
  while className in globals(): # Make sure className is unique
      count += 1
      className = '%s_%s_%s_%d'%(prefix, language, 
                                 englishString, count)
  return className


def createTestCaseClass(className, language, englishString, 
                        translatedString):
  class_ = type(className, 
                (TranslationIntegrityTests, test.TestCase), 
                {})
  class_.language = language
  class_.englishString = englishString
  class_.translatedString = translatedString
  return class_


for language in getLanguages():
  translation = __import__('i18n.%s'%language, 
                           fromlist=['dict'])
  for english, translated in translation.dict.iteritems():        
    className = createTestCaseClassName(language, english)
    class_ = createTestCaseClass(className, language, 
                                 english, translated)
    globals()[className] = class_

False positive

A few days ago AVG Anti-virus started reporting a trojan horse in Task Coach (0.63.2). A little investigation with the help of other py2exe users indicates that AVG detects a trojan in a specific part of py2exe. Py2exe is a program that is used to bundle python source code and the python interpreter into an executable that can be easily installed on Windows machines and doesn't require users to install python. It seems that someone wrote a trojan in python and bundled it with py2exe. Apparently, AVG is now triggered by py2exe instead of a signature that is specific for that trojan horse. It probably means that all applications bundled with py2exe are affected as well. What a bummer. But also kind of interesting to see how other applications, that have nothing to do with Task Coach itself, can cause bug reports about Task Coach.

The need for branches

Until now, I have tried to create a new release of Task Coach every two to four weeks. Each release would contain a mixture of bugfixes and new features. The nice thing of this release strategy is that I don't need to create multiple branches in the version control system. And simple is good, right?

However, the feature I'm working on right now (hierarchical categories) turns out to be harder than I thought. At the same time, people have been reporting some bugs on the latest release that are pretty easy to fix. But, since I'm halfway the development of the hierarchical categories feature, the code is not in a releaseable state. So, I cannot release those bugfixes although they are in the version control system. And that is a waste, right?

So, how to resolve this situation? The classical solution is to add a separate branch for the latest release, apply the bugfixes to that branch, and release a bugfix release from that branch. After that, one only needs to merge those bugfixes into the mainline and everything is fine again. Except for the added complexity of having to deal with multiple branches and merging, that is. The alternative solution, the one I have been using so far, is to add functionality in much smaller steps. Steps that are so small that the code is never 'not in a releaseable state' for more than, say, two weeks. Apparently, I have been not applying that strategy successfully lately, leading to the need for branching.

Anyway, I'll try the branching, see how it works out, and then decide on how to proceed in the long run.