The Segfault Trap

burrahobbit

2011-04-11T01:33:00.004+02:00

burrahobbit, a project offering persistent data-structures seen in Clojure in Python, has just had its first release! See http://segfaulthunter.github.com/burrahobbit/ for details. It's based on the pdict.py script in my sandbox but now differs considerably from it by having a comprehensive test-suite and a lot of new features (including but not limited to sets and the binary operators). I highly appreciate any feedback, be it in the form a comment under this post, a bug report or feature request in the GitHub issue tracker or an email sent to my address which can be found in the box to the right.

PersistentTreeMap in Python

2011-03-13T02:47:00.003+01:00

Abstract

Hereinafter I will discuss an implementation of a PersistentTreeMap in Python. PersistentTreeMaps are a language feature of the Clojure programming language and the implementation thus presented ports this feature to Python. It is implemented in pure Python code and is thus not feasible for very large sets of data (it is doubtful that persistent datastructures are feasible at all for very large sets of data).

A word about persistence

So what's all the fuss about persistence about, anyway? A persistent datastructure is an immutable datastructure that (the clue is in the following clause) allows for the user to create many copies; the key to doing this is to make the copies cheap by sharing most of the data between the original and the copy.

Persistent data-structures are of enormous value in multi-threaded programming because data-structures can be assumed not to change their value, this assuring that no other thread may interfere with one's calculation.

Persistence in Python

The only thing that may actually change is the state of the key or value objects stored in the mapping (in fact, the key object should not be mutable as it must implement __hash__ and therefore should not be mutable).

As Python objects are essentially references, the data-structure has no power to prevent data stored in it from being changed (which it need not, because the cheap-copy of the datastructure actually points to the same object and thus is not changed). Thus, looking up a member always returns the same object, but that object's value may have changed with time; this is a restriction inherent to Python.

Concept of persistent HashTreeMaps

A tree structure consists of nodes which have at most as many children as the arity of the tree is; a node without any children is referred to as a leaf of the tree.

A binary tree (i.e. a tree in which each node that is not a leafnode has two children) will serve as an example to convey the concept of a HashTreeMap; a binary tree has several advantages for explanation purposes, most importantly the conciseness of its graphical illustration. Moreover, hashes of objects are assumed to be 3 bits long in this example, which is a quite unrealistic length in the real-world. An implementation with better performance characteristica and premisses that stand in the real-world will be explored afterwards.

There are three types of leaf-nodes: The AssocNode which stores a key to value association, a HashCollisionNode which contains a list of AssocNodes with colliding hashes (hashes which are equal while they key is not) and the NULLNODE which is a dummy node for dead-end leaves. Because there is no state associated to it, only one instance of NULLNODE is required and only this instance may be used for performance reasons.

Before continuing in our example, I need to elaborate the interface each node must export:

assoc(hsh, shift, node)

without(hsh, shift, key)

get(hsh, shift, key)

assoc returns a new tree with the mapping contained in the AssocNode node added (please ignore the additional arguments for the time being, their merits will be discussed later). without returns a new tree with the mapping with the key removed, it will throw a KeyError if the key is not present. get will return the value associated to key or, if no such association exists, raise a KeyError.

When a HashTreeMap is created, the tree only contains one node, being the NULLNODE. Let the key of the association we want to create be "hello", its hash 0b010 and the value "world". The hash of the key is passed to assoc as hsh, an AssocNode containing the association as node and 0 as shift. shift refers to the level of the tree the node whose assoc (or without or get for that matter). The new tree returned now contains the association of "hello" to "world", the old tree is left ontouched.

The insertion of the second association is a bit more tricky. Suppose you want to add an assocation from the key "spam" with the hash 0b001 to the value "eggs". We now need to create a branching point at the 0^th level of the tree. The 0^th branch points to the AssocNode containing the association from "hello" to "world", whereas the other one points to the one associating "spam" with "eggs". To understand this, consider the lowest bit of the hash the index of the branch to take at the first branching point.

Now let us consider one more association. Let the key be "foo", its hash 0b011 and the value "bar". The AssocNode that is the leaf of the 1-branch of the branching point on the 0^th level now needs to be replaced by another branching point that contains the AssocNode for "hello" on the 0-branch and the AssocNode for "foo" on the 1-branch.

Lookup is really straightforward. The tree is walked until a leaf is reached, taking the branch represented by the value of the respective bit at every branching point; this means, the branch to be taken on the 0^th level is decided by the 0^th bit, the branch to be taken on the first level by the next bit, and so forth, until a leaf is eventually reached. If the leaf is a AssocNode, it is checked whether its key equals the one requested, if it is so, the value is returned, otherwise a KeyError is thrown. If the leaf node is a HashCollisionMode, the key of every of the colliding AssocNodes is checked against the requested key, and, if one of them matches, its value returned; if the leaf is a NullNode (which is only possible for empty trees with a binary tree), a KeyError is raised.

Removing an item (done by calling the without method) is done by walking the tree to the item that needs to be removed, and then replacing that with a NullNode (additionally removing any branching points rendered redundant by removing the item).

In the next post, I will elaborate the implementation of a tree maps that is feasible for real-life scenarios by being implemented as a 32-ary tree.

info shred

2011-01-15T00:34:00.004+01:00

The best way to remove something irretrievably is to destroy the media it's on with acid, melt it down, or the like. For cheap removable media like floppy disks, this is the preferred method. However, hard drives are expensive and hard to melt, so the `shred' utility tries to achieve a similar effect non-destructively.

Emphasis mine. The GNU folks have a strange kind of humour.

Mozilla data disclosure 2

2011-01-01T22:50:00.002+01:00

I have received a response with a link to a blog post concerning the incident, which says.

The database included 44,000 inactive accounts using older, md5-based password hashes. We erased all the md5-passwords, rendering the accounts disabled. All current addons.mozilla.org accounts use a more secure SHA-512 password hash with per-user salts. SHA-512 and per user salts has been the standard storage method of password hashes for all active users since April 9th, 2009.

It is important to note that current addons.mozilla.org users and accounts are not at risk. Additionally, this incident did not impact any of Mozilla’s infrastructure. This information was also sent to impacted users by email on December 27th

Mozilla data disclosure

2011-01-01T18:20:00.003+01:00

Hello and happy New Year everyone!

I have just scanned my mailbox for new emails, and one of them particularly caught my attention: Its title was “Important notice about your addons.mozilla.org account”. To be honest, I did not remember I even had an account there, but because they claimed it were important, I read it. Reading it proved to be worthwhile and here are the most important passages of the text:

“Subject: Important notice about your addons.mozilla.org account
The purpose of this email is to notify you about a possible disclosure of your information which occurred on December 17th. […] On this date, we were informed by a 3rd party who discovered a file with individual user records on a public portion of one of our servers. […] This file was placed on this server by mistake and was a partial representation of the users database from addons.mozilla.org. The file included email addresses, first and last names, and an md5 hash representation of your password.”

So effectively they are telling you that they hashed your full name, your email address and the MD5 hash of your password because someone put a file into public that did not belong there. What really puzzled me was not the fact that this awkward incident happened, but rather the wording of the emphasised sentence: It said nothing about whether salting was used to additionally protect the password; is it possible that a company like Mozilla does not consider the user passwords to be sensitive enough to add the additional security layer of salting?

But it is not all bad, they disabled the accounts where data was possibly leaked, investigated on the issue and promise not to let it happen again. The last sentence of the sentence also promises that they may add salting, if they do not already do it.

“The reason we are disclosing this event is because we have removed your existing password from the addons site and are asking you to reset it by going back to the addons site and clicking forgot password. We are also asking you to change your password on other sites in which you use the same password. Since we have effectively erased your password, you don't need to do anything if you do not want to use your account. It is disabled until you perform the password recovery. We have identified the process which allowed this file to be posted publicly and have taken steps to prevent this in the future. We are also evaluating other processes to ensure your information is safe and secure.”

So, curious on whether salting is used at Mozilla's, I decided to contact the address they supplied for questions with the following email.

Subject: Protection of user data

Dear sir or madam!

The email your corporation has sent on 28.12.2010 concerning possible disclosure of user data on addons.mozilla.org has left me with the question of whether the user passwords stored in your database are salted or not. I consider salting of user passwords to be of utmost importance, as it significantly reduces the risk of disclosing
passwords by preventing the attacker to use a rainbow attack. The wording that elaborated which data has been leaked in the abovementioned email (“The file included email addresses, first and last names, and an md5 hash representation of your password.”) has left me with the impression that you do not salt user passwords and thus I would really appreciate if you could give me definite
information about the matter.

Best regards,
Florian Mayer
PS: I will make available publicly the information contained in the response to this email unless explicitly stated otherwise, as I firmly believe people have a right to know how the data they give away is secured and think the principles of openness repeatedly expressed by your corporation require the same.

To be continued …

man dbus-daemon

2010-12-20T01:41:00.004+01:00

Wonderful. This shows only the most motivated and competent people work on dbus:

“Complex globs like "foo.bar.*" aren't allowed for now because they'd be work to implement and maybe encourage sloppy security anyway.”
— man dbus-daemon

Emphasis mine.

Fedora

2010-11-24T11:38:00.003+01:00

Apparently the Fedora Live-CD does not like my Intel Fake RAID (I need to use one because I need Windows-compatibility) whereas the 3.3 GB full-install DVD does. This inconvenience has caused me to question my decision of replacing my Ubuntu system with Fedora, as it does not seem as if it were the sophisticated system I have been looking for either.

GUIs, RPC and everything

2010-11-24T10:45:00.008+01:00

The idea has been in my head for quite some time; but before explaining it please give me some time to explain the circumstances. By recently having bought a smartphone that enables me to write instant messages and using a laptop when away from home, my instant messaging data are now split apart – believe it or not – to four locations: my Linux installation and my Windows installation at home, the laptop and the smartphone. This is especially a problem because you risk losing context if the communication is not actually instant but the location is switched between sending a message and receiving its reply. Plus, the rather unstable connection to the Internet of both the laptop and the smartphone make for great disconnect-connect spams for everyone that has me on their contact list.

So what's the groundbreaking idea and what does the blah about multiple devices have to do with it? The idea is to create an instant messenger where the program logic and the view (i. e. the UI) are only loosely connected via sockets. Whenever I've talked to someone regarding the idea, I was told that X.org already supports to access remote windows; this may be true, but proposing this as a solution is evidence of lack of understanding of the original problem: the problem is not how to share the UI of a program between computers, but how to synchronize multiple UIs so that they have the same state. The difference may sound subtle, but is extremely important. Different systems require different UIs, and thus it is not the right solution to share the view but one should rather share the state of the program in question. My approach is to save all RPC calls done on the server and resend them if a new client connects; this may sound extremely inefficient, but RPC calls that are redundant are not sent if the server marks them as closed. Moreover calls can be labelled as children of other calls and are automatically deleted from the server if the parent is (e. g. if the IM has RPC calls for opening a window, all RPC calls that make text appear therein would be children of the opening call and are thus not resent if the window is already closed).

I hope that more and more programs take on the approach proposed here to conform with the changes the computer industry has been going through leading to a vast range of different devices with radically different requirements to the UI. Decoupling the UI and the logic of native applications is also the next logical step in the direction of cloud-computing (the problems thereof will be not be elaborated as this is a completely different topic and would go beyond the scope of this post).

Work on an IM that uses the techniques elaborated in this post has been started by me today. I will first solve the meta-problem of syncing multiple remote UIs, and afterwards will write an instance messenger on top of it (probably using telepathy as multi IM library).

multiprocessing and Windows

2010-11-16T14:41:00.008+01:00

I had to find out today that multiprocessing does not like to run staticmethods in processes on Windows. It seems it is impossible to pickle staticmethods, for some reason or another:

import multiprocessing

class Foo(object):
   @staticmethod
   def foo():
       pass


class Bar(object):
   def foo(self):
       pass


if __name__ == '__main__':
   bar = Bar()
   # Works.
   multiprocessing.Process(target=bar.foo).start()
   foo = Foo()
   # Fails.
   multiprocessing.Process(target=foo.foo).start()

The corresponding error is

pickle.PicklingError: Can't pickle : it's not the same object as __main__.foo

Dusting the Sandbox #2: Mocking oldstyle lookup on newstyle classes.

2010-10-29T13:46:00.017+02:00

“Dusting the Sandbox” is a series of blogposts where I pick scripts or modules of my sandbox and explain the process of creation and my motives behind it. The Sandbox can be found at Github.

With Python 2.7 being the last non-bugfix release in the 2.x branch of Python, attention is expected to gradually shift to the new 3.x branch. Albeit onloved by a part of the developer community, the discontinuation of implementing new features into the 2.x branch is the most effective way of forcing developers to begin to embrace the new backwards-incompatible version. The one major issue with Python 3.x is that many old, large libraries do not want or cannot, due to lack of workpower, port their code to support the latest release.

If there are any faithful readers of my blog, and you are one of them, you may just experience a déjà vu. Yes - I have already written a post with roughly the same content as above, but: this time, I present a solution that I think is more clever.

These facts have inspired me to once again think of a way to get the old lookup behaviour with Python 3.x, where there are no more oldstyle classes. The piece of code I provided you with in my earlier post led to some runtime overhead for each call of a magic method, whereas this, radically different, approach only implies overhead each time __setattr__ is called (but it does cause a bit higher memory usage). Please again bear in mind that it is not recommended to use the code provided here in production code.

It all started with someone showing me a snippet that did some thing, which I do not remember, but it certainly was clever thing, by replacing the __class__ member of an instance. I instantly thought that I may exploit this to fake old-style behaviour by creating a new class programmatically each time the application developer overrides a magic method. By creating a new class, the new method is in the class struct and thus is consistently accessed, be it from directly Python code or from C code that accesses it in said struct.

The trick is to create a new class using type and replacing the current __class__ member of the instance each time __setattr__ is called with a magic name. The new class is created by the following code, with attr being the function a magic function is being overridden with.

# Licensed under the terms of the X11-License.
d = dict(self.__class__.__dict__)
d.update({name: staticmethod(attr)})
ncls = type(
 self.__class__.__name__ + 'Magic', self.__class__.__bases__, d
)

The static method is necessary because when replacing a method of an instance, the new function is not bound to the object, contrary to a normal function in a class construction that is automatically bound to the object unless it's a staticmethod; it does also work with attributes, because staticmethod makes the argument passed to it the respective instance-member. The next logical step is to write a function determining whether a method is magic or not, it turned out to be the following (explanation about the forbidden set are given in the next paragraph):

# Licensed under the terms of the X11-License.
forbidden = set(('__setattr__', '__class__'))

def is_magic(name):
    return (
        name not in forbidden and name.startswith('__') and name.endswith('__')
    )

The mechanism needing to replace __setattr__, it needs to be treated specially. The same is true for __class__ because that would have us hit infinite recursion (plus __class__ is no method anyway and thus needs not be wrapped).

Now all we need to do is handle the case when we do not handle a "magic" case. If __setattr__ was replaced earlier, use the member of the instance, if it is not, use the __setattr__ method provided by the parent class.

# Licensed under the terms of the X11-License.
import inspect
if (not inspect.ismethod(self.__setattr__) or
    self.__setattr__.im_func != oldstyle):
    # Costum setattr is stored in the instance so that this method is
    # never overriden. Hence __setattr__ is not part of the global
    # magic set.
    self.__setattr__(name, attr)
else:
    super(self.__class__, self).__setattr__(self, name, attr)

Now we combine these two cases (the one where we treat a magic member and the one where we do not) into one function and end up with the final code. Old style behaviour is applied to a class by overriding the __setattr__ method with the oldstyle function provided below.

# Licensed under the terms of the X11-License.
import inspect

forbidden = set(('__setattr__', '__class__'))

def is_magic(name):
    return (
        name not in forbidden and name.startswith('__') and name.endswith('__')
    )


def oldstyle(self, name, attr):
    if is_magic(name):
        d = dict(self.__class__.__dict__)
        d.update({name: staticmethod(attr)})
        ncls = type(
            self.__class__.__name__ + 'Magic', self.__class__.__bases__, d)
        
        self.__class__ = ncls
    else:
        
        if (not inspect.ismethod(self.__setattr__) or
            self.__setattr__.im_func != oldstyle):
            # Costum setattr is stored in the instance so that this method is
            # never overriden. Hence __setattr__ is not part of the global
            # magic set.
            self.__setattr__(name, attr)
        else:
            super(self.__class__, self).__setattr__(self, name, attr)

Now we can easily verify that it works by comparing the behaviour of a normal new-style class and of one with oldstyle as setattr member. Therefore, we create really simple class that allows us to verify whether the member has been replaced or not.

class A(object):
    __setattr__ = oldstyle
    
    def __int__(self):
        return 2

Then we create an instance thereof, convert it to an integer by using the builtin int, replace the member to return any other number and then verify that this number is returned upon passing the object to int.

a = A()
assert int(a) == 2
a.__int__ = lambda: 5
assert int(a) == 5

Removing the "__setattr__ = oldstyle" declaration we can see that normal new-style classes do not support replacing of magic methods in the instance.

Dusting the Sandbox #1: Get nth digit and other funny, but useless, solutions to problems no one will ever face

2010-10-16T01:25:00.019+02:00

“Dusting the Sandbox” is a series of blogposts where I pick scripts or modules of my sandbox and explain the process of creation and my motives behind it. The Sandbox can be found at Github.

Those who are too impatient to read about the creation of the code can skip directly to the complete code.

The other day I've gotten some inspiration on what to code from the #python.de channel on irc.freenode.net, namely a program that gets the n^th digit of a number (it later turned out that we were talking about getting one particular digit of small numbers in the channel, which does not require a sophisticated algorithm but rather yields the best results (in terms of performance) when doing the naive method, which is:

str(nmb)[n]

The solution that yields results for large numbers in a reasonable time is to divide the number by 10^{log(x) - n}.

This may be quite nice, but there is a lot of space for improvement. The first thing that came into my mind was allowing the user to specify how many digits he wants to access (that is, how many of the digits that are in a higher decimal place should be returned). That is easily done by raising the 10 after the modulo operator by the power of the number of digits the user wants to access.

What turned out as a valuable trick throughout the time I have coded so far is to, once a program produces the desired result, take a step back and see how it could be adapted so it can applied to a super-set of problems. The main instrument for doing is, for myself, is to scan the source-code for any constants that are unnecessary and just specialize the problem without adding value. The only constant that can be found in the snipped shown above is 10 (two direct references and one subtle one in `log10`); by replacing all of them with a variable, we can effectively adapt the snippet to work with arbitrary number systems.

Now non-negative indices are wholly implemented; the next step is self-evident: implementing negative indices. This is even easier because we need not know how long the number is (that is what the logarithm is for when accessing via positive indices), we just need to divide by s^{|n| - 1}. The -1 is to compensate for the fact that negative indices start at -1 rather than 0, and because the number does not need to be divided at all (i.e. divided by 1) to access the last digit, we need to subtract 1 from the absolute value of the index.

The next logical thing to implement is slicing. Slicing is a bit more sophisticated, because it involves two indices which might be of a different type in terms of negative or non-negative indices. The way to easily implement this is use the digits parameter of nth we have defined before but not yet used. The two indices need to be processed to one index (which is the larger one, being stop - 1 regardless of the algebraic sign of the index in question [the item with the index given in stop is not included in the slice, just as it isn't with lists, hence stop -1]). With all input sanitizing, that handles the case where some parameters are not given and the case where one index is negative while the other is not, included, we end up with the following code; because there is no way to return an “empty int” (which would be the obvious approach when modeling after the list API which returns an empty list in these cases) when there are no digits between the two indices (which happens if start is greater or equal to stop), we need to throw an expection in this case:

So the final code is:

A funny (but not at all useful, like the rest of the code shown here) addition is to write a wrapper class that supplies __getitem__ on int objects. The gotcha is that you need to inherit from long, rather than from int, and that you must not call long's __init__ in yours, lest you face a deprecation warning.

That concludes this post, I hope you enjoyed it. And yes: I have tried implementing slicing with steps, but my implementation was heavily outperformed by the naive str and slice way (which does not work for all number systems, though).

asynchia 0.1.1

2010-10-16T00:57:00.003+02:00

asynchia 0.1.1 was released yesterday, featuring a host of bugfixes for critical bugs that prevented the library from working in corner-cases and a few feature additions. I have tried to put work into optimizing existing operations that wasted a lot of resources, e.g. by not pre-allocating the amount of memory required for the buffer. Python's bytearrays are used for that purpose, in 2.7 or later memoryview and recv_into is used, but I need to do some more benchmarking to see whether this results in a significant speed-up or not, the data I have yet collected is not unambiguous about that matter.

Check asynchia out at Github

Bitbucket changes

2010-09-29T03:35:00.005+02:00

Albeit me having switched to git for most of my serious development (as it fosters branched development by offering cheap deletable branches; maybe that's possible now with the bookmark extension of hg, but by the time I switched, it was not), I gladly follow the latest development of Bitbucket: Joining forces with Atlassian (to be fair, I have not heard anything of them until now), they now offer unlimited storage, an unlimited number of both private and public repositories and CNAMEs absolutely for free (and also the option to add 5 people to the readers, writers or admins of your private repositories).

I just considered this to be cool enough to bother anyone that might read this with.

Android Application Security

2010-09-16T00:43:00.004+02:00

Sorry for not having written into this Blog for such a long time, but I just did not have any ideas that I thought to be important enough to bother you (if there is anyone out there reading this). I have recently purchased a HTC Wildfire (good device with some shortcomings, but all in all it offers a good price–value ratio, but discussing the pros and cons of the device would miss the point of this post). I have – of course – installed a couple of applications on it (what's the use of a smartphone without those), many of which communicate with the Internet.

Considering I will mostly be using those when I am not home (and thus not have access to a real computer), I have started some analysis to find out which of this applications I can securely use on a public WiFi (or using my mobile internet connection) without fearing to have any data compromised to eavesdroppers (like the WiFi hoster or – if the WiFi is public or the key is known to many people – a third-party that can intercept the datastream); the long and the short of it, I wanted to find out which of the applications sends unencrypted data to the Internet.

I connected my computer (which luckily has two Ethernet cards) to the router with both of them, connected the Android phone to the WiFi of the router and manually set my computer as the gateway, which in turn forwarded all the data back to the router again. Then I started up Wireshark and thus could see all traffic coming from the smartphone, and have drawn the following conclusions:

Google Talk seems safe to use as neither the plaintext of the account credentials nor the one of the messages sent can be found in the data sent.
eBuddy (a multi-IM client inter alia supporting MSN) seems to only encrypt the password when logging in, and definitively does not encrypt the user-name or any messages sent over MSN.

Because my setup did not work as well as I expected it (the Internet was terribly slow on the phone and furthermore Wireshark kept segfaulting), I have not yet analyzed other applications, but if you want me to look at one, do not hesitate to tell me in the comments. Other than that I advise you to set up your email accounts at home over an encrypted connection and manually configure it to use SSL (it appears to favour unencrypted communication when autodetecting your account information from the login data), lest you possibly compromise your login data to the owner of the WiFi or eavesdroppers. Whenever sending sensitive data with the web browser be sure you are using an SSL secured connection to the server.

web--!

2009-12-18T20:33:00.007+01:00

First of all, let me explain the strange title. If you are familiar with C-like programming languages, you may already have guessed its meaning. It's a plea to diverge from the path the web is currently taking.

Seemingly the browser has become the new platform. The extensive use of Javascript faces browsers with challenges they were never designed to meet. None of the components currently used for Web 2.0 applications were designed to be used the way they are. The Web was designed for displaying more or less static pages (which the server may dynamically produce). This results in technical problems and major overheads. Tricks are used to enable stateful communication over the HTTP protocol or dynamic interaction with the web page. Granted, these tricks work most of the time – neglecting the overhead they create for the time being. Still apparently many AJAX-requests seem to be lost in the nirvana of the internet, never to be found again. If you have never seen a half-loaded AJAX-page where the content – which seems to be acquired by AJAX-requests after the page has loaded – is missing, you can count yourself lucky.

Making web applications compatible with browsers poorly implementing the standards takes a lot of the developers time away which could be used in a better way. Even creating static pages which look exactly the same in all browsers (I am looking at you, Internet Explorer) is a lot of work, fiddling about trivialites such as assigning the page a maximum width (no, CSS max-size is not the answer as it is ignored by Internet Explorer). I cannot even imagine how painful it must be to make a page compatible to mobile-phone browsers (maybe that is why most sites offer a seperate mobile version to which you are automatically taken when connecting with a mobile-phone client).

But another important issue which I cannot stress enough is that Web 2.0 applications enable their developers to intesively amass your data and/or data about you. Traditionally, you either uploaded things intended for the public to the internet or stored private data (i.e. backups) at trusted places. Nowadays, pretty much everything is uploaded to anyone who offers the particular service. Whatever you produce using a Web 2.0 application is out of your reach. It may be kept even if you delete it on the front end; it may be gone some day; it may become public, either by mistake or on purpose. You have no influence on what happens with it. You do not even have the security that the application even will resemble its current state in the future; you may find yourself with data in a web-application you must learn to use (again).

The best solution would be to end the browser's unhoped-for streak of popularity and replace it with a technology that it designed for doing what the browser is abused to do – running dynamical programs including complex user interaction. If most applications ran on a single virtual machine, it would have the potential to supersede the browser of the common platform. Good riddance. Java's WebStart is a good attempt of streamlining the deployment process of applications.

2to3 for map "fixed"!

2009-10-27T21:12:00.002+01:00

The issue about the problem with 2to3 and map has been resolved with the following changeset. The fix is generally like saying "Fix your code yourself" rather than fixing the issue. Adding a warning is not an adequate fix for any issue. This fix just helps people who think their code will run after 2to3, but it is not addressing the core problem the slightest bit. Every bug (and every non-feature as the one added in the changeset) in 2to3 is hindering the widespread adoption of Python 3.x; I think the CPython developers should take a second to think about that before commiting such "fixes". At least leave the issue open, hoping for someone to patch it.

The world's biggest hard-drive

2009-10-25T23:01:00.010+01:00

Having ruined my Vista (another story I will blog about soon) I want to re-setup my Windows in order to be able to game. Before doing so, I have to backup my user files of Vista. As these are relatively large (~100G), I had to clear some space on my Linux and created a new partition for it (~150G). Having asked some folks in IRC about what file-system to use, I picked ext2. Copying slowed my system as hell (of course), but df -h verified that there was progress. Suddenly, my tty spit out some weird messages.

EXT2-fs error (device sda6): ext2_new_blocks: Allocating block in system zone - blocks from 35586561, length 1

It wasn't long until the copy operation failed with the following error message.

No space left on device.

Okay.. I thought I had become insane and asked cfdisk about the size of the partition and du -hs about the size of the source files. About 150G and 100G, respectively. So everything was okay. Now the weird part starts. I asked df -h about how much of the partition was used, getting the following (quite amusing - considering my harddrive is 1TB large in total) output.

Filesystem Size Used Avail Use% Mounted on /dev/sda2 92G 6.9G 81G 8% / tmpfs 1.5G 0 1.5G 0% /lib/init/rw varrun 1.5G 120K 1.5G 1% /var/run varlock 1.5G 0 1.5G 0% /var/lock udev 1.5G 152K 1.5G 1% /dev tmpfs 1.5G 832K 1.5G 1% /dev/shm lrm 1.5G 2.2M 1.5G 1% /lib/modules/2.6.28-16-generic/volatile /dev/sda5 184G 139G 36G 80% /home /dev/sda6 16T 16T 0 100% /mnt/media /dev/sda1 489G 258G 231G 53% /media/disk

Note the emphasised part. 16T... This means 16 terrabytes. It is apparent that some bug must have occured. But hang on, the drive gets even larger. Disbelieving, I decided to see what du -hs would say about the directory that is on the device (I would be very thankful for any tips about what the error messages were trying to tell me).

name@gollum:/mnt/media$ du -hs name du: cannot access `name/.gimp-2.6/.idlerc': Stale NFS file handle du: cannot access `name/.gimp-2.6/Anwendungsdaten': Stale NFS file handle du: cannot access `name/.gimp-2.6/AppData/.idlerc': Stale NFS file handle du: cannot access `name/.gimp-2.6/AppData/Anwendungsdaten': Stale NFS file handle du: cannot access `name/.idlerc': Stale NFS file handle du: cannot access `name/Anwendungsdaten': Stale NFS file handle du: cannot access `name/AppData/.gimp-2.6/.idlerc': Stale NFS file handle du: cannot access `name/AppData/.gimp-2.6/Anwendungsdaten': Stale NFS file handle du: cannot access `name/AppData/.idlerc': Stale NFS file handle du: cannot access `name/AppData/Anwendungsdaten': Stale NFS file handle 19T name name@gollum:/mnt/media$ du -hs lost+found 2.0T lost+found

So 19T and 2T, doesn't that make 21T? Either df lied or my maths is really rusty. Right now I'm trying to copy the same data onto an ext3 partition of the same size, hopefully this will work better.

UPDATE: With ext3, it worked fine.

Of Python3's map

2009-10-25T18:28:00.020+01:00

Hello, it's been a long time since the last post, but here we go.

So what is wrong with map? Its semantics have secretly changed between Python 2.x and Python 3.x, without even 2to3 knowing. While the old, non-lazy, map of 2.x stopped when the last sequence was depleted, while the lazy one of 3.x (like 2.x itertools.imap) stops when the first sequence is depleted. The reasonability of this change can be quarreled about, but that is not really the topic here.

The real topic is that not even 2to3 is aware of this subtle change. An easy fix for this would be to translate map(fun, a, b, ...) to list(map(fun, *zip(*itertools.zip_longest(a, b, ...)))) (itertools.izip_longest in 2.6). Another problem with 2to3 and map is that None cannot be given as the function argument with 3.x, but 2to3 is unaware of this. Thus it should translate map(None, ..) to map(lambda *a: a, ..), which it doesn't.

So here is what 2to3 does. It minds neither of the semantic changes.

--- map_test.py (original)
+++ map_test.py (refactored)
@@ -1,1 +1,1 @@
-map(None, [1, 2, 3], [1, 2, 3, 4])
+list(map(None, [1, 2, 3], [1, 2, 3, 4]))

Python 2.x:

>>> map(None, [1, 2, 3], [1, 2, 3, 4])
[(1, 1), (2, 2), (3, 3), (None, 4)]

Python 3.x:

>>> list(map(None, [1, 2, 3], [1, 2, 3, 4]))
Traceback (most recent call last):
File "", line 1, in 
TypeError: 'NoneType' object is not callable
>>> list(map(lambda *a: a, [1, 2, 3], [1, 2, 3, 4]))
[(1, 1), (2, 2), (3, 3)]
>>> list(map(lambda *a: a,
... *zip(*itertools.zip_longest([1, 2, 3], [1, 2, 3, 4]))))
[(1, 1), (2, 2), (3, 3), (None, 4)]

This shows that the proposed fixes work.

Issue created by birkenfeld after discussing the issue with me

Old-style class lookup

2009-08-16T14:31:00.004+02:00

By the wide adoption of Python 3 and the discontinuation of maintenance for 2.x, old-style classes will be gone. This also means that there's no way to easily change the magic-methods of your objects (there is of course a way by manually writing the wrappers and replacing the methods they are wrapping, but I don't count that as being easy any more). So today I have written a metaclass that makes your class mimic the behaviour of old-style classes in 2.x. It does that by automatically generating said wrappers and storing the real methods in a closure.

Here you got the code that allows this and also contains an example that verifies that it works. As said in the doc-string, I wouldn't recommend using this for any purpose other than experimenting.

epoll is awesome

2009-08-06T02:29:00.005+02:00

It really is. Tonight I had the idea of trying out something crazy with epoll. I tried out what happens if you modify the flags of a fd in a thread while the epoll.poll call is blocking the main one. I didn't really expect it to mind the changes done by the thread but - unlike poll - epoll does. This is really nifty and I am sure can be put to good use (though I have to admit I cannot think about one just yet). What perplexes me is that this is not documented - neither in the (rather sparse) Python documentation of select.epoll, nor in the Linux manpage for epoll. All I could find was a "They are thread safe", but that's in an a bit different context.

I'm thinking about creating an API for asynchia that abstracts changes the the socket-map from other threads (you are not expected to understand this if you are not familiar with asynchia).

I have also written code that illustrates how this works with epoll, and how it does not with poll.

Principle of Biggest Surprise

2009-06-27T16:52:00.003+02:00

I've written an article about weird things in Python. You can find it here. Please leave your comments under this post or write an email.

Braid

2009-06-21T21:32:00.004+02:00

This blog isn't really meant to be about games, but this time it has to be.

I recently (on Friday) purchased the indie-game Braid on Steam. I am really excited by it, and have finished it by now. It features really clever gameplay, beautiful music and, in its own way, beautiful graphics. If you care about the storyline of a game, Braid has a very good, but also confusing story. The major downside it has is that it is incredibly short, I had finished it after about 8 hours. But well, I still have the speed-runs to do, and they will not be easy.

It was very well worth the 13 euros.

Parallel computing with Python

2009-06-10T10:54:00.003+02:00

I've been writing a library that streamlines the process of parallelization. It is inspired by Cilk and thus has a really simple design. The name isn't yet decided, but I think I will call it pylk. I'm also trying to write a cluster-library, and once that is finished I will try to add a backend that executes the functions that were spawned on the cluster.

Below is some (commented) example code.

# Use multiprocessing backend.
backend = MPBackend()
# Start the thread that listens for I/O from the processes.
backend.iohandler.start()

# Do some calculations.
a = backend.spawn(reduce, operator.mul, xrange(1, 50000))
b = backend.spawn(reduce, operator.mul, xrange(1, 50000))

# When you need the results, sync them.
a, b = sync(a, b)
# When done, shut down the I/O thread so the application can exit.
backend.iohandler.stop()

Blogger sucks

2009-05-24T20:34:00.001+02:00

Look at the previous post, no more explanations needed.

Unnecessary explanation: That is the HTML I fed it with.

Python threading (and the GIL)

2009-05-24T20:33:00.011+02:00

My shiny new Quad-Core Intel i7 has brought my attention to writing software that scales on multiple processors (the OS even considers the i7 a 8-core because of the HyperThreading technology it comes with).

I like Python, I like it a lot. Still, I have to admit that its threading sucks. It doesn't at all scale to multiple processors (this is due to the GIL). So what does GIL mean anyway? It means that the Python interpreter always only executes one piece of bytecode at a time. This means, that only one thread can execute python code at a time, thus rendering threads for scaleability useless.

Here are some benchmarks on how long it takes to add a lot of numbers (distributed on n threads or processes) on my i7.

Threads	Time (in Seconds)	Difference
1	21.15	-
2	23.17	+9.56%
3	24.83	+7.16%
4	25.13	+1.21%
5	23.67	-5.81%
6	24.90	+5.21%
7	24.74	-0.64%
8	25.36	+2.52%

Processes	Time (in Seconds)	Difference
1	20.36	-
2	10.43	-48.78%
3	6.91	-33.79%
4	5.32	-22.96%
5	5.90	+11.01%
6	5.13	-13.09%
7	4.86	-5.33%
8	4.59	-5.50%

asynchia

2009-03-18T00:55:00.005+01:00

Today I re-wrote the whole library I deleted yesterday. Here I present: asynchia. It's a minimalistic asynchronous network library (it has about 600 lines of code of which about 100 are LGPL-Headers) licensed under the, you guess, GNU LGPL.

I hope you like it and am open for any constructive criticism.

What I learned from deleting: Always use a SCM and always push to a remote location. Plus, commit pretty often so you do not lose so much if you accidentally delete anything.

Always use SCM

2009-03-16T22:17:00.001+01:00

... from the beginning on, or you will learn it the hard way.

I did.

I wrote, and deleted, a async networking library today. Hooray!

Houston we have a problem!

2009-02-22T01:24:00.004+01:00

Okay, let me start this post with a confession: I'm a gamer. Here we go, now it's out.

I think many people don't like it Steam, but I've learned to hate it. Why is that, you may ask? You shall have your answer.

I've decided that I want to try out "Empire: Total War", and the demo was available through Steam. Having played some games that require Steam, I didn't think it'd be a problem, but I was about to be proved wrong. I started downloading, it wasn't fast, but okay. Then, the first time at 40%, the download began to stall. Pause - Resume, nope, didn't fix it. That's why I closed Steam. When I tried reopening it told me it was already open - so I killed the process and the download happily continued. The same thing happened again at 80%, I did my closing and reopening. Then, what I saw was horrible. The download got back to 0%. I gave up for this day.

Today, fresh and full with energy I tried again. In the beginning it went pretty slow, but after trying through many mirrors, I eventually found one that was acceptable (~300KB/s). For those who may be interested, it was the one in the UK. Today I downloaded the demo - there were some problems, but I eventually succeeded. The game's pretty okay, but with my system I have to play it on very low graphics settings.

I guess they have a lot of work to do stabilizing Steam and the servers, as I think there's no excuse for a commercial service that earns money by distributing software to be slower than university servers for open-source Linux distributions.

But another thing happened today. While I was downloading, my mouse began to break. Windows made that "device attached" and "device detached" sounds really often. In classic Windows manner, I disconnected both the mouse and the keyboard and reconnected them. This was a mistake, as Windows demanded me to accept to install the mouse driver because it doesn't comply with some tests. The bad thing was that it did that before installing the keyboard driver, and apparently this question blocked the whole device installation process for all the devices. I had to reboot.

Now comes the weird part. When I tried rebooting, my BIOS hang. I thought resetting it might help (through the jumper on the mainboard), but it did not. When I disconnected my mouse it went fine. So I booted my Arch, connected it. It did not work. This is what dmesg says about it:

usb 3-2.2: new full speed USB device using uhci_hcd and address 8
usb 3-2.2: device descriptor read/64, error 8
usb 3-2.2: device descriptor read/64, error 8
usb 3-2.2: new full speed USB device using uhci_hcd and address 9
usb 3-2.2: device descriptor read/64, error 8
usb 3-2.2: device descriptor read/64, error 8
usb 3-2.2: new full speed USB device using uhci_hcd and address 10
usb 3-2.2: device descriptor read/8, error -71
usb 3-2.2: device descriptor read/8, error -71
usb 3-2.2: new full speed USB device using uhci_hcd and address 11
usb 3-2.2: device descriptor read/8, error -71
usb 3-2.2: device descriptor read/8, error -71
hub 3-2:1.0: unable to enumerate USB device on port 2

Any suggestions of what might be wrong with it are very appreciated. Another mouse works just fine, so I guess my mouse is just broken.

In other news, I had to delete some Java programs in my Archlinux in order to update all packages as there is some confusion that some appear to want openjdk and some jdk and jre.

To sum this post up in one sentence: Everything out there is broken!

QTest

2009-02-02T17:32:00.003+01:00

Hello again. It's been quite some time again since I last posted.

I am going to blog about a new project of mine. I called it QTest and it is a GUI frontend to the unittest testing framework. You give it your TestSuite, it runs the tests in a background process calling back to the GUI so it can display the information.

It uses PyQt4 as the GUI framework. You can have a look at a screenshot if you like. The source is stored in a Mercurial repository which can be found here.

I hope that this is useful to anyone out there and I would be very happy if you could tell me your experiences with it.

As for the usage, it should be explained in the docstring of the module.

Update

2008-12-14T19:15:00.006+01:00

It's been a some time since I last posted here. Still, time didn't stand still. I apologize for my lack of interest in this blog lately, if there are any people out there that like reading it.

I have chosen PyQt as the new toolkit for pypentago, and started working on the new user-interface, I have implemented the Pentago board in C, and have written a basic AI not quite capable of winning, but it's playing semi-decent.

By doing this, I've learned a lot about C and how it works. Often enough I wanted to curse it, but in the end it came out working. I'm using minimax with alpha-beta pruning to determine the best turn, but the current problem is that the function that rates a position of the board isn't very good, it just counts the longest line.

I also took at look at the pictures on my harddrive and uploaded those which I liked to deviantart. Take a look at them if you are interested.

Only for my screencasts, time seemed to have stood still. I haven't touched any of those that I should edit nor have I done a new one. I'll see when I got time to finish those and upload them to ShowMeDo.

Good bye wxPython

2008-11-01T23:44:00.002+01:00

I've decided to give up wxPython and learn another GUI toolkit. This is mostly decided by the amounts of bugs in wxPython. Firstly, it is easy to get it segfaulting, making the search of the error harder and secondly it likes to create unexpected behaviour, like raising an exception when using getattr on an object. I will have to decide between PyQt and PyGTK, but I am still undecided.

Operators

2008-10-11T23:39:00.008+02:00

In Python, most other operators have higher priority than not, this means

not 1 == 2

gives us the result we'd expect, being True. I've had a lengthy discussion on IRC with someone stating that not has the higher priority, which it hasn't. It's very easy to observe in the example above, because not 1 evaluates to False, and so does False == 2. So the result of the example would be False, which it is not. What it really does is evaluate 1 == 2, which is False, and then not takes the opposite, which is True.

Why am I discussing this anyway? Today, I've written a tool that converts not foo == bar to foo != bar, not a in b to a not in b, not a not in b to a in b and so forth. It's about 70 lines of source code that I will never understand when I re-read it, as it is full of regular expressions.

I will be releasing it to public once I've cleaned up the code a bit and decided on which license to use, meaning to choose from MIT and GNU GPL.

Roadmap for "Advanced Python" series

2008-09-25T23:22:00.007+02:00

I've outlined a roadmap for my "Advanced Python" series on ShowMeDo today.
I am going to continue with a brief explanation of the lambda statement, taking a maximum of 2 videos, I would say, then I am going to continue by explaining reduce, map and filter and afterwards I'll show alternatives for them using the list comprehension/generator expression syntax. Eventually I'll also show how to write custom generators.

variable scope to be done
decorators done
lambda post-production
operators module | functools.partial post-production
reduce, map, filter post-production
list comprehensions post-production
generator expressions post-production
generators to be done

E-Voting

2008-09-24T21:22:00.002+02:00

Note: This post is describing the situation in Austria.

No thanks. When E-Voting is introduced, which they are planning to do, I am declaring our democracy as dead. Why is that, you may ask?
Well, let's first look at the brighter side, what are the advantages of E-Voting? Firstly, it may be cheaper, but I am unsure how much maintenance of a computer system secure enough to be capable of being used for a democratic voting, which I think no computer can do. Secondly, people won't need to leave their houses. Fine, I can relate to that.
But now let's look at the disadvantages: secrecy of you vote simple cannot be assured. There has to be some access control to the system, and no matter how good they may do it, afterwards it may be possible to see what you voted. Would you sell your right to vote secretly just for not leaving your house? I doubt it!

But the major problem is transparency. You simply don't know what the system is doing under the hood when you vote for a particular party. It can vote for any party internally, and you never know it.

In contrast to that, when you use the traditional paper method, it is hard to manipulate your vote. When the votes are counted, representatives of all political parties observe the process, and most importantly, you have the right to observe the votes being counted.

Even postal vote is more secure. You put an blank envelope which contains your vote into an envelope that contains your personal information. The personal information is verified, meaning they look whether you are allowed to vote and whether you requested postal voting. After that, the blank envelopes are taken and opened separately. So, at least in theory, you privacy is assured, and I am pretty sure that you are allowed to observe that process, although I am unsure.

Why should I give my democratic rights into the hand of a computer program of which I've got no idea how it works?
It's far to easy to manipulate non-transparent systems to give our basic democratic rights in the hand of them. Don't let laziness restrict your democratic rights.

Decorator screencasts

2008-09-22T23:22:00.005+02:00

I have done two new screencasts today.

Decorators: The Basics
Decorators: Real World Example

Check them out at ShowMeDo.

OO - But not quite

2008-09-20T21:10:00.010+02:00

_inst = Context()
register_handler = _inst.register_handler
register_nostate_handler = _inst.register_nostate_handler
remove_handler = _inst.remove_handler
delete_action = _inst.delete_action
emmit_action = _inst.emmit_action

I have my actions module, as you might know. It exposed a API with only functions, as it was only one state that all of them shared, and I used a global variable for that.
Realizing that this was a mistake, I quickly refactored it to be object-oriented. Now you can create several Contextes, and register and emmit actions in every one of them.
But after all, most people will not even need this, as they will only use one context. So I remembered how it was solved in Python Standard Library's random.py.
It is done exactly the same way as the code I posted above, and I think it's a very good idea.

You can find the whole new code here.

I am planning to do a separate repository for that module, as it turns out to be more useful than I expected it to be.

Yes I am aware of the bug in the gitorious code viewer that makes it write "@@" instead if "@", I already told them in the IRC-Channel.

Black Holes would make great dustbins

2008-09-10T19:39:00.003+02:00

Do you think the guys at CERN would be as kind as to send me some?

Solved 25 projecteuler Problems

2008-09-06T01:19:00.007+02:00

Bravo, segfaulthunter! Now that you have solved 25 problems you have achieved what 78.5% of members have failed to do and have advanced to level 1. Good luck as you continue.

I just solved my 25^th projecteuler problem and got this nice message. Hope I will get to level 2 sometime. Though the problems are getting harder, a lot harder, now.

"Who Can Name the Bigger Number" revised

2008-09-04T23:03:00.003+02:00

I have produced a revised version of Who Can Name the Bigger Number written by Scott Aaronson and have uploaded a PDF of it. It is exactly the same text as the one on his site, only that I have changed the formatting in a more printer-friendly and probably even more readable way.
For me this also was a pretty good introduction into LaTeX which I may use a bit more in the future.
If you haven't read the article yet: read it!
You can find my revised version at http://name.nonlogic.org/bignumbers/bignumbers.pdf.

How I do my screencasts

2008-09-01T14:28:00.007+02:00

I this post I am going show how I am doing my screencasts.
First I think about a topic. This is the easiest problem, as my TODO file is full of them.
Then I think about an example how to explain it, write some code, try it out and if it works I'll use it.
Afterwards I log into my tutorial account and start making the screencast, with recording on. It has to be noted that I have no second monitor showing me the code I am supposed to write, so I pretty much write it from memory. With longer screencasts I might have a paper with notes lying next to me. It is normal that at the first 3 or 4 tries the record will end with [insert swearing word here], because I made a mistake. Whenever I do a mistake, I start all over. I'm no fan of cropping the screencast because it's a lot of work and the output seems odd sometime.
So then after some tries I found out which words I need to look at to pronounce properly(I'm no English native speaker) and which code I need to take care of so I don't get a mistake in.
Once I did a recording that I am comfortable with, I extract its audio using mplayer, amplify it a bit, remove some background noise, et cetera. Then I put it back into the video and encode it using XVid. Then my screencast is done. I upload it to ShowMeDo and wait for people to comment on it.

That's how I do it. It might seem like a bit odd approach, but I think it works out. In my opinion you just have to try it a few times in order to get similar output to what long preparation would get you.

Test Driven Development Screencast

2008-08-31T18:38:00.001+02:00

Today I've done a screencast on test-driven development.
Have a look at it over at ShowMeDo.

findsilence 0.1rc3

2008-08-30T22:52:00.005+02:00

The first release candidate of findsilence was released today.
I forgot posting the other releases to here, but this one I am. The changelog can be found at http://name.nonlogic.org/findsilence/CHANGELOG.

[Download findsilence]

Counting digits

2008-08-29T22:13:00.006+02:00

def digits(x):
""" Return amount of digits of x. """
    return int(math.floor(math.log10(x)) + 1)

Because I needed to create an efficient algorithm to count the amount of digits of a number, I have come up with the one you can read above. It is a bit slower than len(str(x)) for small numbers, but a lot faster with large ones.
Another example on how a good algorithm beats processing power.

Python mathmodule.c patch

2008-08-29T14:41:00.007+02:00

Today I have written a patch for Modules/mathmodule.c of Python 2.6. It removes the inaccuracy of math.log when called with base 10 compared to math.log10. All it does is internally using log10 when log gets 10 as base. It makes the output of math.log much more accurate when used with base 10.
Here is some comparison:
[Unpatched Python] [Patched Python]

I hope the patch is good enough to meet Pythons high standards.
Props go out to Gromit @ freenode for helping me create the patch.

[Python Bug Tracker Issue]

How not to do CLI

2008-08-28T13:33:00.011+02:00

Having concerned myself with audio programming, I found out that some very established tools have a horrible command-line interface. Particularly I am talking about lame here.
It is impossible to just convert multiple files without writing a shell-script. I think that every CLI should give the user the option to use their shells' globbing features. Lame also does a good job at overriding options that users are used to them performing in a different way.
When you see lame -h, would you expect that the -h flag shows you a help, or that it increases the quality of your output file?
Or whats better to read: lame *.wav or for i in *.wav; do lame $i; done? I'd certainly answer Of course that should show me the help! and "lame *.wav" is much more concise! to these questions. And it's not that it would be that hard, at least not with Python and optparse. You just need to iterate over the input files and you are done. for arg in args: [your code here]. That's it. So with Python there is absolutely no excuse not to support multiple input files.
I think that CLI need to be thought about and designed just as much as GUI need to. They need to be powerful yet intuitive to use, and I think lame failed at being either of that. At least with Python, there's no excuse not doing it.

With findsilence I tried to achieve that. It works with just one argument, the input file(s), and it puts the output in the directory output/ in the working directory, if it is not specified using the -o flag. When it gets more than one file, it automatically puts the output of each of them into a separate directory in the output directory, if it only gets one file, it just puts the output into the output directory.
So summarizing I think there are a few things you should note when creating CLI:

Never change behaviour of often used options such as -h, -v, -q
Always accept multiple input files unless you have a good reason not to
Try to minimize the options that are required by keeping sane defaults for unspecified options.
When you have a GUI and a CLI, think about adding a flag to the CLI to start the GUI to not require the user to remember two commands.

findsilence release coming

2008-08-26T23:59:00.003+02:00

I am about to release findsilence 0.1rc1. I have tweaked the values according to three different records, and am just digitizing an old Beatles record for the final test, if that works, I will release.

findsilence improved

2008-08-26T13:59:00.002+02:00

Findsilence, that's the name I gave to my WAV splitting project, has been heavily improved in the last two days since I came back from my vacation.
I have implemented a new algorithm that should find silence more reliably than the old one, and have really improved the user interface. You now see a progressbar while the program is working, get the chance to cancel the progress if you want, and most importantly, it doesn't hang once you've started the progress.

Photograph

2008-08-25T16:56:00.002+02:00

This is one of the photographs I took while I was in Switzerland.
I think it is the best one, and that's why I am posting it now. More may follow.

Back Home

2008-08-25T11:33:00.002+02:00

Today at about 7AM I arrived home at Vienna again. I will sort out the pictures now, maybe will upload the best to Flickr or a similar service.

Holidays

2008-08-12T19:40:00.003+02:00

Today in the evening, I am going to leave towards Switzerland, where I'll arrive tomorrow to spend my holidays. I will return on Monday the 25th of August.
I may have WLAN access somewhere and maybe will post to here when the weather outside is not nice.

New MP3 Player

2008-08-12T00:13:00.002+02:00

Today, as I am leaving for holidays tomorrow, I bought a new MP3 player. As buying hardware that works with Linux is like lottery, I just picked the one I liked best without even considering how it might work with Linux.
But today, I've won the Linux-Lottery. The MP3 player is fully usable by just mounting it and transferring your music to the music folder. All that even though the package explicitly stated that the product does only work with Windows. I already transferred a lot of music onto it so I can listed to it while sitting in the train. I am pretty satisfied with it, the only shortcoming is that it is only chargeable by USB, but that is not that bad as I am taking a laptop there anyway.
The only thing that, I think, is impossible with Linux is making playlists, but as I never use that feature on mobile MP3 devices anyway, it is not that bad.

It was a Sony NWZ-A826, if you are interested in buying an MP3 player that is well-supported under Linux.

Automatic Title splitting

2008-08-11T02:28:00.005+02:00

I have written a Python program that is capable to separate tracks in a long .wav file, as long as there is silence between them. I also supplied it with a nice, minimalistic, User-Interface.
In fact, this is the first User-Interface that I've done with wxPython that really looks the way I planned it to, that resizes properly, if odd-looking, and is, in my opinion, intuitive.
The only problem with it as of now is that the User-Interface and the real splitting code are both executed in the same Thread, thus the User-Interface gets unresponsive once you hit the "Split into Tracks" button. As this is not too hard to do, I will address it in the next time.
I might even release this to the public if anyone has a use for it, but most likely not until I come home from my vacations again.

You might be wondering why I did this program?
We got a turntable capable of directly writing the music it plays onto an USB-Stick. But unfortunately, it doesn't split it into the tracks, but rather writes one big mp3 file that contains the whole record. This is my solution to that, although it is currently unable to directly work on, or at least, convert mp3 files by itself. So for now I will have to manually convert the mp3 to wav. That will change too, but I am still searching a right library for that task. Speaking of which, for the core of this application, I am only using Python builtins, so no external dependencies, but wxPython for the UI, are required.
While coding this, nearly the whole day today, I have found out that the Python documentation on wave and audioop is pretty lacking, so I had to experiment a bit to find out good default values. I might send in some patches for this.

Code Styleguide

2008-08-09T20:54:00.005+02:00

I have been working hard today to get a new styleguide up for pypentago. It describes how the code in pypentago has too look. Despite being similar to PEP8, it also addresses some things that PEP8 does not, like cross-platform file-name operations.
The outcome pretty much pleases me, as it is much longer in detail.

I have written it because at the time of writing there are still 150 PEP8-incompatible variable, class and function-names, despite me having removed some of them in the last days.

It is written using Sphinx and I might add some other developer resources to the online documentation some time later.

If you think I have missed anything in it, feel free to either contact me via email or just leave a comment.

Code Style Guide
PS: I am aware that there is some encoding problem. I will try to resolve that some time.

Gna! SVN repository unbearable

2008-08-07T00:07:00.006+02:00

[ gna/pypentago/trunk ] % svn update
ssh_exchange_identification: Connection closed by remote host
svn: Connection closed unexpectedly

I have been seeing this a lot lately, and when I say a lot, I mean committing fails more often than it succeeds. I often have to wait a minute or so in order to be able to commit, which I think is much too much.
I think I am going to try to find a new home for pypentago. Not that Gna! was bad, but a repository that is down when you try to commit is just unbearable.
This may also be a chance of switching to a distributed version control system, like Git or Mercurial.
At the moment I lean more towards Git, as I think its command-set is more complete, but on the other hand Mercurial is far better when it comes to writing extensions.
The most appealing thing of Git is its rebase command which takes your changes and puts them on top of another branch. While this may be dangerous and bad when you already have your branch shared, I think it is very useful for keeping your local branches up-to-date with the master branch.

UI "Action" Bindings

2008-08-07T00:00:00.002+02:00

Today I have refactored the pypentago User-Interface code to make use of the new action module I have written. I also make heavy use of decorators in it, even using my instance-decorator hack.

I think this is an important step to modularity and I like it much better that way.
On the downside, having used git-svn, the connection failed during the dcommit which messed up git a bit and forced me to svn rm the whole source and copy the new one to it instead.

Who Can Name the Bigger Number?

2008-08-04T02:04:00.004+02:00

You have fifteen seconds. Using standard math notation, English words, or both, name a single whole number—not an infinity—on a blank index card. Be precise enough for any reasonable modern mathematician to determine exactly what number you’ve named, by consulting only your card and, if necessary, the published literature.

I have found an interesting article dealing with numbers so big no human can imagine them.
It also contains an excursus about Turing Machines which I found particularly interesting, because they are the basis of modern computing.

In my opinion it is well-written and understandable, even for people that are not really into maths, such as me. I really enjoyed reading it and think that I learnt a lot of new things from it.

Who Can Name the Bigger Number?

Decorator Magic

2008-08-03T23:27:00.015+02:00

I have been using Python decorators more in the last time, finding the extremely useful. But there are some pitfalls I have experienced.
First one is that when decorating methods, there is no way for the decorator to find out whether it is a method or a function, which is a bit complicated in some cases.
The second one, that is logical though, is that the methods that get passed to the decorator are unbound methods, thus you cannot just store them in a dictionary and execute them later.
When you know which class the method is of, it is enough to just store the name(func.__name__) and get the method in the constructor of the class using getattr(self, func_name).

So what did I need decorators for, anyway?
I was searching for a way to reorganize the network code in pypentago, and seeing all those pointless self.bind("EVENT", self.method) functions, I decided that there has to be a better way.
For me, that better way were decorators, eventually leading the the following form:

@expose("EVENT")
def method(self, evt):
    pass

I think this is much more readable, as that way the code of the method and the corresponding event are not so shattered across multiple sections in the code.
During that process, I have also made great use of the partial function in the functools module. It allows you to store a function with some of the parameters passed, and the other ones left for the time it is executed.

def foo(bar, spam):
    print bar, spam
baz = partial(foo, spam="Spam!")
print baz("Bar!")

This will output "Bar! Spam!". It is really useful when storing functions for execution to occur later.
I think this is pretty useful, I also have create a pretty hacky way to apply decorators to bound methods by storing both the decorator and the method associated with it and decorating it in the __init__ method.

Of KMail

2008-07-30T19:20:00.003+02:00

Today, I have installed KDE4 to look at what has changed. I am mainly a GNOME and xfce user, but I have gotten interested in the new KDE.
So I installed it, opened it up and tried it out a bit. To me it seemed a bit slow and I had some minor graphical problems, meaning that something did not look really good.
Then I began trying out all of those K* programs. When I tried out KMail on my AOL IMAP, it immediately marked all of my email as spam, thus moving it to a local spam folder.
As none of it was really spam, I tried moving it back to the Inbox, which failed. Then I moved it into an empty folder, which took about 15 minutes all in all.

My initial experience with KDE4 therefore is not really a good one. I'll maybe play around with it later, but I won't let it touch my email account for now ;)

Action Handling

2008-07-28T01:22:00.004+02:00

I've just written a small module for pypentago that allows actions to be emitted, and then to be handled in a handler defined by another module. This emphasizes modularity, as it is much easier to write custom user-interfaces this way.
Both the design and the implementation are simple, though I have set a high value on documenting it properly. This leads to more than half of the lines being docstrings, and this is not an overstatement.
Most of the time when developing this I have spent in developing a good design, that allows both the registrar and the emitter to give information to the handler. I have chosen the most obvious, thus most simple approach. The emitter passes one, and exactly one, variable to the handler, that is passed to it as the first parameter. The registrar may provide any amount of additional parameters.
Due to my high usage of wxPython events, I have more and more grown tired of needing to make my handlers accept an event, although they do not use it. This is why I implemented something called "nostate handlers". This means that these handlers do not take information from the emitter, that information I called state. This is useful when trying to bind actions directly to underlying framework functions, such as ones' from wxPython.

[Link to Source Code]

Project Euler

2008-07-15T01:30:00.003+02:00

I have found a really nice website offering you problems that you can solve using computer programming. You can then verify whether the result was true, and if you have an account it will take track of which problems you solved and which are still to be solved for you.
Currently my rating is: Your current rating is 3% genius, having solved 8 out of 202 problems.
I hope you will also try to solve these problems, because it is a good way to practice both programming and mathematics.
The site can be found at http://projecteuler.net/.

Regex cheatsheet

2008-06-30T20:02:00.004+02:00

I have found a neat little cheatsheet for regular expressions on the internet. It can be found here.
Props go to the author of it for investing work and research and then sharing it with the community, it will for sure help me!

unittest screencast

2008-06-28T23:22:00.005+02:00

I have created a screen cast on unittest today. It explains the basics of unit testing and will be extended by another video explaining more complicated uses of it.
I have already uploaded it, but I will turn up the volume and do some other adjustments and then upload it again and to ShowMeDo.
UPDATE: Uploaded xvid version with louder sound: http://www.mediafire.com/?4jkjowghy1q

Screencast on XMLRPC

2008-06-28T12:55:00.004+02:00

The Pocoo Pastebin has made me interested in XMLRPC, because you can post and view pastes using it. I have done some research and now done a short, 5 minute, screencast explaining the most basic features it offers.
If you do not know XMLRPC, it is a way of calling functions on a remote server nearly the same way than if they were in your program directly.

The screencast can be downloaded here and should soon be available at ShowMeDo.

pyometry Python-3.0 compatible

2008-06-19T20:03:00.003+02:00

I have just tried out pyometry.vector in a Python-3.0b1 and it has worked perfectly. This means there most likely will be a nearly seamless conversion from Python-2.x to Python-3.x, at least for the vector module.
[Python-3.0b1 console session]

New Python3.0 string formatting

2008-06-19T15:22:00.004+02:00

I have just tried out the new Python 3.0 string formatting in the alpha release, and I got to say it is really great. I will make great use of it for the pypentago configuration files as I will not have to manually replace {foo} with "foovalue" but rather do string.format(foo="foovalue").

Here you've got a code sample: http://paste2.org/p/40928

Z Shell

2008-06-13T20:19:00.006+02:00

I have once again installed the Z Shell, better known as zsh. I have done this because I got more and more pissed about the bad completion features bash, where you cannot tab through the possible commands.
The feature that lets you display the return code of the last command in the prompt is also great.
All in all I like the zsh much better than the bash as some features are much more advanced in it. It also has a quite helpful first time users configuration program to create a really basic .zshrc.
The bad thing is that rxvt prevents me from having a prompt using colour or such without having to escape all the special characters[1]. But I think this is not the fault of zsh but rather the one of rxvt, as the users in that mailing list describe the same problem using the bash.
I put my .zshrc[2] onto dotfiles if you are interested. I know it is not really advanced so far, but I have most of my stuff defined in .aliases[3] which is loaded by .profile[4]. I have uploaded these two files to dotfiles too.
You might notice that the unp alias in the .aliases file does not need different attributes for different types of files. This is due to tar being able to auto detect the type of a file in new versions, if it does not work for you replace tar with bsdtar, which also supported this in earlier versions.
Some of the ideas for these files also came from other ones on dotfiles.
If you think these files could be improved, which for sure they can, please let me know!

pyometry on gitorious

2008-06-09T19:06:00.006+02:00

I named the package in which I put my vector module pyometry. That name was produced by merging Python and Geometry into pythongeometry. I also think that the name does not sound bad, so I picked it.
Enough of the name, I put this package, called pyometry, onto Gitorious today because I wanted to try out something new. For a change I picked this host, using Git instead of SVN which all hosting sites I used so far provided. So far I like it really good, because it has a simplistic web interface but still has all the features one would need. For source code management, that is, because Gitorious does not offer any features like web-hosting, file-mirroring or bug-tracking.
I think in the beginning I will be fine without those, but maybe I will need to host these somewhere else, if the package is used by some people eventually.

If you like Python and maths, and always waited for some chance to develop something, head over to Gitorious, create yourself an account and clone the mainline branch and eventually file a merge request so I can merge the changes you have done into the mainline branch.

Be ready for some more posts about git in the future, now that I can try it out first handidly.

Vectors in Python

2008-06-09T17:12:00.003+02:00

I have just created a module capable of doing vector calculations for Python. It currently supports calculation cross products, scalar products, unit vectors, the length of vectors and angles between two vectors.
It is pretty easy to use as this code-snippet should show. If there is anything in this that you do not understand, feel free to leave me a comment.
I also tinker with the idea of putting this onto some hosting-site, because I plan extending it so it is capable of doing calculations with planes and lines as well.

If you think you could make use of this, feel free to contact me so I can send you the code I have at the moment.

eric IDE for Python code refactoring

2008-06-06T17:49:00.006+02:00

I have just found a way to refactor Python projects. You can use eric4 using the rope plugin. It allows you to rename methods and classes in your whole project and appears to be working well.
With MS Windows I have experienced some problems with it, but with Archlinux it seems to work fine. I will post again once I have tried it out a bit more.

Python code refactoring

2008-06-04T15:50:00.002+02:00

I have been searching a tool that refactors my Python code, thus renaming my methods, classes and functions throughout the project so I do not have to manually change the names and all references to it.
I have not yet found any tool that really meets my requirements. Most of them are just plain hard to use, or are only integrating into IDEs or editors that I do not use.

Change permissions for NTFS partition

2008-06-01T16:30:00.005+02:00

I have just found out a little hack to change the permissions for a whole NTFS partition, but not for the files in it. It is good if you have a multi-user system and want to deny other users to access your NTFS drive.
First, create /mnt/win/win, then change /mnt/win to the permissions you want, mount the ntfs drive at /mnt/win/win and then create a symlink at /mnt/windows pointing at /mnt/win/win.
This will allow you to control access to this NTFS partition, remember that you cannot change the permissions for anything that is in the parition.

pyPentago userbar

2008-06-01T15:53:00.004+02:00

If someone of you has tried out the pyPentago alpha and liked it, I have created a userbar so you can show that you like it.
You can get the alpha version of pyPentago here, please follow the instructions on how to install it.

New Blog

2008-06-01T14:24:00.000+02:00

I have opened a blog here on Blogger.
I will write about releases of my software or software I am excited about and my thoughts on different things, mostly computers, I think.