Lessons in game design…in a car!

For the greater part of this week I have been trapped in a car driving across the United States for my sister’s wedding. Of course, I had my trusty netbook with me and so I decided to do some programming. After discovering that I had unwittingly installed tons of documentation in my /usr/share/doc folder (don’t you love those computer explorations that actually reveal something?), I started building yet another rendition of Tetris in Python. I did several things differently this time and the code is actually readable and I feel like some structure things I learned/discovered/re-discovered should be shared here.

Game State Management

One problem that has constantly plagued my weak attempts at games has been an inability to do menus and manage whether the game is paused or not in a reasonably simple manner. Of course, I could always hack something together, but I could also write the whole thing in assembly language or maybe write it all as one big function in C. The issue is that if I program myself into a hole, I usually end up giving up a bit unless I am getting paid (then I generally bother thinking before programming so that I don’t waste people’s time). After finishing the game logic and actually getting things to work with the game itself, I remembered a funny tutorial I was once doing for OGRE which introduced a very simple game engine. The concept was one of a “stack” of game states. Using this stack, your main program basically injects input into and then asks a rendering from the state on the top of the stack. The states manipulate the stack, pushing a new state or popping themselves. When the stack is empty, the program exits.

My game state stack works by using an initial state that loads the resources (the render section gives back a loading screen). After it finishes loading the resources, it asks the manager to pop itself from the stack and then push on the main menu state. The menu state is quite simple: It displays several options where each option either does something immediate (like popping the menu from the stack to end the program in the case of the Quit option) or something more complex (like pushing a sub-menu state on to create a new game with some settings). After that there are states for playing the game, viewing the high scores, etc.

The main awesome thing about this structure is that switching states is so easy. Want to pause the game? Just push on a state that doesn’t do anything until it receives a key input or something, after which it pops itself. Feel like doing a complex menu system? The stack keeps track of the “breadcrumbs” so to speak. Never before have I been able to actually create an easily extensible/modification-friendly game menu/management system that doesn’t leave me with a headache later trying to understand it or unravel my spaghetti code.

The second awesome thing is that this helps encourage an inversion of control. Rather than the state itself dictating when it gets input or when it renders something, the program on top of it can choose whether it wants to render that state or whether it wants to give input to that state. I ended up using the render function like a game loop function as well and so if I wanted a state to sort of “pause” execution, I could simply stop telling it to render (usually by just pushing a state on top of it). My main program (in Python) was about 25 lines long and took care of setting up the window, starting the game states, and then just injecting input into the game state and asking the state to render a window. The states themselves could choose how they wanted to render the window or even if they wanted to (that gets into the event bubbling section which is next). It is so much easier to operate that way as the components are loosely tied to one another and there is a clear relationship between them. The program owns the states and the states know nothing about about the program driving them…only that their input comes from somewhere and the renderings they are occasionally asked to do go somewhere.

Event bubbling

When making my Tetris game, I wanted an easy to way to know when to update or redraw the screen rather than always redrawing it. My idea was to create an event system that would “notice” when the game changed somehow: whether a block had moved or maybe the score changed. I remembered of two event systems that I really really liked: C#’s event keyword plus delegate thing and Javascript’s event bubbling system. Some people probably hate both of those, but I have learned to love them. Mainly, its just because I’m new to this sort of thing and am still being dazzled by their plethora of uses. Using some knowledge I gleaned from my database abstraction project about the fun data model special function names in Python, I created the following:

class EventDispatcher(object):
    """
    Event object which operates like C# events

    "event += handler" adds a handler to this event
    "event -= handler" removes a handler from this event
    "event(obj)" calls all of the handlers, passing the passed object

    Events may be temporarily suppressed by using them in a with
    statement. The context returned will be this event object.
    """
    def __init__(self):
        """
        Initializes a new event
        """
        self.__handlers = []
        self.__supress_count = 0
    def __call__(self, e):
        if self.__supress_count &gt; 0:
            return
        for h in self.__handlers:
            h(e)
    def __iadd__(self, other):
        if other not in self.__handlers:
            self.__handlers.append(other)
        return self
    def __isub__(self, other):
        self.__handlers.remove(other)
        return self
    def __enter__(self):
        self.__supress_count += 1
        return self
    def __exit__(self, exc_type, exc_value, traceback):
        self.__supress_count -= 1

class EventDispatcher(object):

"""

Event object which operates like C# events

"event += handler" adds a handler to this event

"event -= handler" removes a handler from this event

"event(obj)" calls all of the handlers, passing the passed object

Events may be temporarily suppressed by using them in a with

statement. The context returned will be this event object.

"""

def __init__(self):

"""

Initializes a new event

"""

self.__handlers = []

self.__supress_count = 0

def __call__(self, e):

if self.__supress_count > 0:

return

for h in self.__handlers:

h(e)

def __iadd__(self, other):

if other not in self.__handlers:

self.__handlers.append(other)

return self

def __isub__(self, other):

self.__handlers.remove(other)

return self

def __enter__(self):

self.__supress_count += 1

return self

def __exit__(self, exc_type, exc_value, traceback):

self.__supress_count -= 1

My first thought when making this was to do it like C# events: super flexible, but no defined “way” to do them. The event dispatcher uses the same syntax as the C# events (+=, -= or __iadd__, __isub__) to add/remove handlers and also to call each handler (() or __call__). It also has the added functionality of optionally suppressing events by using them inside a “with” statement (which may actually be breaking the pattern, but I needed it to avoid some interesting redrawing issues). I would add EventDispatcher objects to represent each type of event that I wanted to catch and then pass an Event object into the event to send it off to the listening functions. However, I ran into an issue with this: Although I was able to cut down the number of events being sent and how far they propagated, I would occasionally lose events. The issue this caused is that my renderer was listening to find out where it should “erase” blocks and where it should “add” blocks and it would sometimes seem to forget to erase some of the blocks. I later discovered that I had simply forgotten to call the event during a certain function which was called periodically to move the Tetris blocks down, but even so, it got my started on the next thing which I feel is better.

Javascript events work by specifying types of events, a “target” or object in the focus of the event, and arguments that get passed along with the event. The events then “bubble” upward through the DOM, firing first for a child and then for the parent of that child until it reaches the top element. The advantage of this is that if one wants to know, for example, if the screen has been clicked, a listener doesn’t have to listen at the lowest leaf element of each branch of the DOM; it can simply listen at the top element and wait for the “click” event to “bubble” upwards through the tree. After my aforementioned issue I initially thought that I was missing events because my structure was flawed, so I ended up re-using the above class to implement event bubbling by doing the following:

class Event(object):
    """
    Instance of an event to be dispatched
    """
    def __init__(self, target, name, *args, **kwargs):
        self.target = target
        self.name = name
        self.args = args
        self.kwargs = kwargs

class EventedObject(object):
    def __init__(self, parent=None):
        self.__parent = parent
        self.event = EventDispatcher()
        self.event += self.__on_event
    def __on_event(self, e):
        if hasattr(self.__parent, 'event'):
            self.__parent.event(e)
    @property
    def parent(self):
        return self.__parent
    @parent.setter
    def parent(self, value):
        l = self.parent
        self.__parent = value
        self.event(Event(self, "parent-changed", current=self.parent, last=l))

class Event(object):

"""

Instance of an event to be dispatched

"""

def __init__(self, target, name, *args, **kwargs):

self.target = target

self.name = name

self.args = args

self.kwargs = kwargs

class EventedObject(object):

def __init__(self, parent=None):

self.__parent = parent

self.event = EventDispatcher()

self.event += self.__on_event

def __on_event(self, e):

if hasattr(self.__parent, 'event'):

self.__parent.event(e)

@property

def parent(self):

return self.__parent

@parent.setter

def parent(self, value):

l = self.parent

self.__parent = value

self.event(Event(self, "parent-changed", current=self.parent, last=l))

The example here is the object from which all of my moving game objects (blocks, polyominoes, the game grid, etc) derive from. It defines a single EventDispatcher, through which Event objects are passed. It listens to its own event and when it hears something, it activates its parent’s event, passing through the same object that it received. The advantage here is that by listening to just one “top” object, all of the events that occurred for the child objects are passed to whatever handler is attached to the top object’s dispatcher. In my specific implementation I had each block send an Event up the pipeline when the were moved, each polyomino send an Event when it was moved or rotated, and the game send an Event when the score, level, or line count was changed. By having my renderer listen to just the game object’s EventDispatcher I was able to intercept all of these events at one location.

The disadvantage with this particular method is that each movement has a potentially high computational cost. All of my events are synchronous since Python doesn’t do true multithreading and I didn’t need a high performance implementation. It’s just method calls, but there is a potential for a stack overflow if either a chain of parents has a loop somewhere or if I simply have too tall of an object tree. If I attach too many listeners to a single EventDispatcher, it will also slow things down.

Another problem I have here has to do with memory leaks which I believe I have (I haven’t tested it and this is entirely in my head, thinking about the issues). Since I am asking the EventDispatcher to add a handler which is a bound method to the object which owns it, there is a loop there. In the event that I forget all references to the EventedObject, the reference count will never decrease to 0 since the EventDispatcher inside the EventedObject still holds a reference to that same EventedObject inside the bound method. I would think that this could cause garbage collection to never happen. Of course, they could make the garbage collector really smart and notice that the dependency tree here is a nice orphan tree detached from the rest and can all be collected. However, if it is a dumb garbage collector, it will probably keep it around. This isn’t a new issue for me: I ran into it with doing something like this on one of my C# projects. However, the way I solved it there was to implement the IDisposable interface and upon disposal, unsubscribe from all events that created a circular dependency. The problem there was worse because there wasn’t a direct exclusive link between the two objects like there is with this one (here one is a property of the other (strong link) and the other only references a bound method to its partner (weak…kinda…link)).

Overall, even though there are those disadvantages, I feel that the advantage gained by having all the events in one place is worth it. In the future I will create “filters” that can be attached to handlers as they are subscribed to avoid calling the handlers for events that don’t match their filter. This is similar to Javascript in that handlers can be used to catch one specific type of event. However, mine differs in that in the spirit of Python duck typing, I decided to make no distinction between types of events outside of a string name that identifies what it is. Since I only had one EventDispatcher per object, it makes sense to only have one type of event that will be fed to its listeners. The individual events can then just be differentiated by a property value. While this feels flaky to me since I usually feel most comfortable with strong typing systems, it seems to be closer to what Python is trying to do.

Conclusion

I eventually will put up this Tetris implementation as a gist or repository on github (probably just a gist unless it gets huge…which it could). So far I have learned a great deal about game design and structure, so this should get interesting as I explore other things like networking and such.

Database Abstraction in Python

As I was recently working on trying out the Flask web framework for Python, I ended up wanting to access my MySQL database. Recently at work I have been using entity framework and I have gotten quite used to having a good database abstraction that allows programmatic creation of SQL. While such frameworks exist in Python, I thought it would interesting to try writing one. This is one great example of getting carried away with a seemingly simple task.

I aimed for these things:

Tables should be represented as objects which each instance of the object representing a row
These objects should be able to generate their own insert, select, and update queries
Querying the database should be accomplished by logical predicates, not by strings
Update queries should be optimized to only update those fields which have changed
The database objects should have support for “immutable” fields that are generated by the database

I also wanted to be able to do relations between tables with foreign keys, but I have decided to stop for now on that. I have a structure outlined, but it isn’t necessary enough at this point since all I wanted was a database abstraction for my simple Flask project. I will probably implement it later.

This can be found as a gist here: https://gist.github.com/kcuzner/5246020

Example

Before going into the code, here is an example of what this abstraction can do as it stands. It directly uses the DbObject and DbQuery-inheriting objects which are shown further down in this post.

from db import *
import hashlib

def salt_password(user, unsalted):
    if user is None:
        return unsalted
    m = hashlib.sha512()
    m.update(user.username)
    m.update(unsalted)
    return m.hexdigest()

class User(DbObject):
    dbo_tablename = "users"
    primary_key = IntColumn("id", allow_none=True, mutable=False)
    username = StringColumn("username", "")
    password = PasswordColumn("password", salt_password, "")
    display_name = StringColumn("display_name", "")
    def __init__(self, **kwargs):
        DbObject.__init__(self, **kwargs)
    @classmethod
    def load(self, cur, username):
        selection = self.select('u')
        selection[0].where(selection[1].username == username)
        result = selection[0].execute(cur)
        if len(result) == 0:
            return None
        else:
            return result[0]
    def match_password(self, password):
        salted = salt_password(self, password)
        return salted == self.password

#assume there is a function get_db defined which returns a PEP-249
#database object
def main():
    db = get_db()
    cur = db.cursor()
    user = User.load(cur, "some username")
    user.password = "a new password!"
    user.save(cur)
    db.commit()

    new_user = User(username="someone", display_name="Their name")
    new_user.password = "A password that will be hashed"
    new_user.save(cur)
    db.commmit()

    print new_user.primary_key # this will now have a database assigned id

from db import *

import hashlib

def salt_password(user, unsalted):

if user is None:

return unsalted

m = hashlib.sha512()

m.update(user.username)

m.update(unsalted)

return m.hexdigest()

class User(DbObject):

dbo_tablename = "users"

primary_key = IntColumn("id", allow_none=True, mutable=False)

username = StringColumn("username", "")

password = PasswordColumn("password", salt_password, "")

display_name = StringColumn("display_name", "")

def __init__(self, **kwargs):

DbObject.__init__(self, **kwargs)

@classmethod

def load(self, cur, username):

selection = self.select('u')

selection[0].where(selection[1].username == username)

result = selection[0].execute(cur)

if len(result) == 0:

return None

else:

return result[0]

def match_password(self, password):

salted = salt_password(self, password)

return salted == self.password

#assume there is a function get_db defined which returns a PEP-249

#database object

def main():

db = get_db()

cur = db.cursor()

user = User.load(cur, "some username")

user.password = "a new password!"

user.save(cur)

db.commit()

new_user = User(username="someone", display_name="Their name")

new_user.password = "A password that will be hashed"

new_user.save(cur)

db.commmit()

print new_user.primary_key # this will now have a database assigned id

This example first loads a user using a DbSelectQuery. The user is then modified and the DbObject-level function save() is used to save it. Next, a new user is created and saved using the same function. After saving, the primary key will have been populated and will be printed.

Change Tracking Columns

I started out with columns. I needed columns that track changes and have a mapping to an SQL column name. I came up with the following:

class ColumnSet(object):
    """
    Object which is updated by ColumnInstances to inform changes
    """
    def __init__(self):
        self.__columns = {} # columns are sorted by name
        i_dict = type(self).__dict__
        for attr in i_dict:
            obj = i_dict[attr]
            if isinstance(obj, Column):
                # we get an instance of this column
                self.__columns[obj.name] = ColumnInstance(obj, self)

    @property
    def mutated(self):
        """
        Returns the mutated columns for this tracker.
        """
        output = []
        for name in self.__columns:
            column = self.get_column(name)
            if column.mutated:
                output.append(column)
        return output

    def get_column(self, name):
        return self.__columns[name]

class ColumnInstance(object):
    """
    Per-instance column data. This is used in ColumnSet objects to hold data
    specific to that particular instance
    """
    def __init__(self, column, owner):
        """
        column: Column object this is created for
        initial: Initial value
        """
        self.__column = column
        self.__owner = owner
        self.update(column.default)

    def update(self, value):
        """
        Updates the value for this instance, resetting the mutated flag
        """
        if value is None and not self.__column.allow_none:
            raise ValueError("'None' is invalid for column '" + 
                             self.__column.name + "'")
        if self.__column.validate(value):
            self.__value = value
            self.__origvalue = value
        else:
            raise ValueError("'" + str(value) + "' is not valid for column '" + 
                             self.__column.name + "'")

    @property
    def column(self):
        return self.__column

    @property
    def owner(self):
        return self.__owner

    @property
    def mutated(self):
        return self.__value != self.__origvalue

    @property
    def value(self):
        return self.__value

    @value.setter
    def value(self, value):
        if value is None and not self.__column.allow_none:
            raise ValueError("'None' is invalid for column '" + 
                             self.__column.name + "'")
        if not self.__column.mutable:
            raise AttributeError("Column '" + self.__column.name + "' is not" +
                                 " mutable")
        if self.__column.validate(value):
            self.__value = value
        else:
            raise ValueError("'" + value + "' is not valid for column '" + 
                             self.__column.name + "'")

class Column(object):
    """
    Column descriptor for a column
    """
    def __init__(self, name, default=None, allow_none=False, mutable=True):
        """
        Initializes a column

        name: Name of the column this maps to
        default: Default value
        allow_none: Whether none (db null) values are allowed
        mutable: Whether this can be mutated by a setter
        """
        self.__name = name
        self.__allow_none = allow_none
        self.__mutable = mutable
        self.__default = default

    def validate(self, value):
        """
        In a child class, this will validate values being set
        """
        raise NotImplementedError

    @property
    def name(self):
        return self.__name

    @property
    def allow_none(self):
        return self.__allow_none

    @property
    def mutable(self):
        return self.__mutable

    @property
    def default(self):
        return self.__default

    def __get__(self, owner, ownertype=None):
        """
        Gets the value for this column for the passed owner
        """
        if owner is None:
            return self
        if not isinstance(owner, ColumnSet):
            raise TypeError("Columns are only allowed on ColumnSets")
        return owner.get_column(self.name).value

    def __set__(self, owner, value):
        """
        Sets the value for this column for the passed owner
        """
        if not isinstance(owner, ColumnSet):
            raise TypeError("Columns are only allowed on ColumnSets")
        owner.get_column(self.name).value = value

class StringColumn(Column):
    def validate(self, value):
        if value is None and self.allow_none:
            print "nonevalue"
            return True
        if isinstance(value, basestring):
            print "isstr"
            return True
        print "not string", value, type(value)
        return False

class IntColumn(Column):
    def validate(self, value):
        if value is None and self.allow_none:
            return True
        if isinstance(value, int) or isinstance(value, long):
            return True
        return False

class PasswordColumn(Column):
    def __init__(self, name, salt_function, default=None, allow_none=False, 
                 mutable=True):
        """
        Create a new password column which uses the specified salt function

        salt_function: a function(self, value) which returns the salted string
        """
        Column.__init__(self, name, default, allow_none, mutable)
        self.__salt_function = salt_function
    def validate(self, value):
        return True
    def __set__(self, owner, value):
        salted = self.__salt_function(owner, value)
        super(PasswordColumn, self).__set__(owner, salted)

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

class ColumnSet(object):

"""

Object which is updated by ColumnInstances to inform changes

"""

def __init__(self):

self.__columns = {} # columns are sorted by name

i_dict = type(self).__dict__

for attr in i_dict:

obj = i_dict[attr]

if isinstance(obj, Column):

# we get an instance of this column

self.__columns[obj.name] = ColumnInstance(obj, self)

@property

def mutated(self):

"""

Returns the mutated columns for this tracker.

"""

output = []

for name in self.__columns:

column = self.get_column(name)

if column.mutated:

output.append(column)

return output

def get_column(self, name):

return self.__columns[name]

class ColumnInstance(object):

"""

Per-instance column data. This is used in ColumnSet objects to hold data

specific to that particular instance

"""

def __init__(self, column, owner):

"""

column: Column object this is created for

initial: Initial value

"""

self.__column = column

self.__owner = owner

self.update(column.default)

def update(self, value):

"""

Updates the value for this instance, resetting the mutated flag

"""

if value is None and not self.__column.allow_none:

raise ValueError("'None' is invalid for column '" +

self.__column.name + "'")

if self.__column.validate(value):

self.__value = value

self.__origvalue = value

else:

raise ValueError("'" + str(value) + "' is not valid for column '" +

self.__column.name + "'")

@property

def column(self):

return self.__column

@property

def owner(self):

return self.__owner

@property

def mutated(self):

return self.__value != self.__origvalue

@property

def value(self):

return self.__value

@value.setter

def value(self, value):

if value is None and not self.__column.allow_none:

raise ValueError("'None' is invalid for column '" +

self.__column.name + "'")

if not self.__column.mutable:

raise AttributeError("Column '" + self.__column.name + "' is not" +

" mutable")

if self.__column.validate(value):

self.__value = value

else:

raise ValueError("'" + value + "' is not valid for column '" +

self.__column.name + "'")

class Column(object):

"""

Column descriptor for a column

"""

def __init__(self, name, default=None, allow_none=False, mutable=True):

"""

Initializes a column

name: Name of the column this maps to

default: Default value

allow_none: Whether none (db null) values are allowed

mutable: Whether this can be mutated by a setter

"""

self.__name = name

self.__allow_none = allow_none

self.__mutable = mutable

self.__default = default

def validate(self, value):

"""

In a child class, this will validate values being set

"""

raise NotImplementedError

@property

def name(self):

return self.__name

@property

def allow_none(self):

return self.__allow_none

@property

def mutable(self):

return self.__mutable

@property

def default(self):

return self.__default

def __get__(self, owner, ownertype=None):

"""

Gets the value for this column for the passed owner

"""

if owner is None:

return self

if not isinstance(owner, ColumnSet):

raise TypeError("Columns are only allowed on ColumnSets")

return owner.get_column(self.name).value

def __set__(self, owner, value):

"""

Sets the value for this column for the passed owner

"""

if not isinstance(owner, ColumnSet):

raise TypeError("Columns are only allowed on ColumnSets")

owner.get_column(self.name).value = value

class StringColumn(Column):

def validate(self, value):

if value is None and self.allow_none:

print "nonevalue"

return True

if isinstance(value, basestring):

print "isstr"

return True

print "not string", value, type(value)

return False

class IntColumn(Column):

def validate(self, value):

if value is None and self.allow_none:

return True

if isinstance(value, int) or isinstance(value, long):

return True

return False

class PasswordColumn(Column):

def __init__(self, name, salt_function, default=None, allow_none=False,

mutable=True):

"""

Create a new password column which uses the specified salt function

salt_function: a function(self, value) which returns the salted string

"""

Column.__init__(self, name, default, allow_none, mutable)

self.__salt_function = salt_function

def validate(self, value):

return True

def __set__(self, owner, value):

salted = self.__salt_function(owner, value)

super(PasswordColumn, self).__set__(owner, salted)

The Column class describes the column and is implemented as a descriptor. Each ColumnSet instance contains multiple columns and holds ColumnInstance objects which hold the individual column per-object properties, such as the value and whether it has been mutated or not. Each column type has a validation function to help screen invalid data from the columns. When a ColumnSet is initiated, it scans itself for columns and at that moment creates its ColumnInstances.

Generation of SQL using logical predicates

The next thing I had to create was the database querying structure. I decided that rather than actually using the ColumnInstance or Column objects, I would use a go-between object that can be assigned a “prefix”. A common thing to do in SQL queries is to rename the tables in the query so that you can reference the same table multiple times or use different tables with the same column names. So, for example if I had a table called posts and I also had a table called users and they both shared a column called ‘last_update’, I could assign a prefix ‘p’ to the post columns and a prefix ‘u’ to the user columns so that the final column name would be ‘p.last_update’ and ‘u.last_update’ for posts and users respectively.

Another thing I wanted to do was avoid the usage of SQL in constructing my queries. This is similar to the way that LINQ works for C#: A predicate is specified and later translated into an SQL query or a series of operations in memory depending on what is going on. So, in Python one of my queries looks like so:

class Table(ColumnSet):
    some_column = StringColumn("column_1", "")
    another = IntColumn("column_2", 0)
a_variable = 5
columns = Table.get_columns('x') # columns with a prefix 'x'
query = DbQuery() # This base class just makes a where statement
query.where((columns.some_column == "4") &amp; (columns.another &gt; a_variable)
print query.sql

class Table(ColumnSet):

some_column = StringColumn("column_1", "")

another = IntColumn("column_2", 0)

a_variable = 5

columns = Table.get_columns('x') # columns with a prefix 'x'

query = DbQuery() # This base class just makes a where statement

query.where((columns.some_column == "4") & (columns.another > a_variable)

print query.sql

This would print out a tuple (" WHERE x.column_1 = %s AND x.column_2 > %s", ["4", 5]). So, how does this work? I used operator overloading to create DbQueryExpression objects. The code is like so:

class DbQueryExpression(object):
    """
    Query expression created from columns, literals, and operators
    """
    def __and__(self, other):
        return DbQueryConjunction(self, other)
    def __or__(self, other):
        return DbQueryDisjunction(self, other)

    def __str__(self):
        raise NotImplementedError
    @property
    def arguments(self):
        raise NotImplementedError

class DbQueryConjunction(DbQueryExpression):
    """
    Query expression joining together a left and right expression with an
    AND statement
    """
    def __init__(self, l, r):
        DbQueryExpression.__ini__(self)
        self.l = l
        self.r = r
    def __str__(self):
        return str(self.l) + " AND " + str(self.r)
    @property
    def arguments(self):
        return self.l.arguments + self.r.arguments

class DbQueryDisjunction(DbQueryExpression):
    """
    Query expression joining together a left and right expression with an
    OR statement
    """
    def __init__(self, l, r):
        DbQueryExpression.__init__(self)
        self.l = l
        self.r = r
    def __str__(self):
        return str(self.r) + " OR " + str(self.r)
    @property
    def arguments(self):
        return self.l.arguments + self.r.arguments

class DbQueryColumnComparison(DbQueryExpression):
    """
    Query expression comparing a combination of a column and/or a value
    """
    def __init__(self, l, op, r):
        DbQueryExpression.__init__(self)
        self.l = l
        self.op = op
        self.r = r
    def __str__(self):
        output = ""
        if isinstance(self.l, DbQueryColumn):
            prefix = self.l.prefix
            if prefix is not None:
                output += prefix + "."
            output += self.l.name
        elif self.l is None:
            output += "NULL"
        else:
            output += "%s"
        output += self.op
        if isinstance(self.r, DbQueryColumn):
            prefix = self.r.prefix
            if prefix is not None:
                output += prefix + "."
            output += self.r.name
        elif self.r is None:
            output += "NULL"
        else:
            output += "%s"
        return output
    @property
    def arguments(self):
        output = []
        if not isinstance(self.l, DbQueryColumn) and self.l is not None:
            output.append(self.l)
        if not isinstance(self.r, DbQueryColumn) and self.r is not None:
            output.append(self.r)
        return output

class DbQueryColumnSet(object):
    """
    Represents a set of columns attached to a specific DbOject type. This
    object dynamically builds itself based on a passed type. The columns
    attached to this set may be used in DbQueries
    """
    def __init__(self, dbo_type, prefix):
        d = dbo_type.__dict__
        self.__columns = {}
        for attr in d:
            obj = d[attr]
            if isinstance(obj, Column):
                column = DbQueryColumn(dbo_type, prefix, obj.name)
                setattr(self, attr, column)
                self.__columns[obj.name] = column
    def __len__(self):
        return len(self.__columns)
    def __getitem__(self, key):
        return self.__columns[key]
    def __iter__(self):
        return iter(self.__columns)

class DbQueryColumn(object):
    """
    Represents a Column object used in a DbQuery
    """
    def __init__(self, dbo_type, prefix, column_name):
        self.dbo_type = dbo_type
        self.name = column_name
        self.prefix = prefix

    def __lt__(self, other):
        return DbQueryColumnComparison(self, "&lt;", other)
    def __le__(self, other):
        return DbQueryColumnComparison(self, "&lt;=", other)
    def __eq__(self, other):
        op = "="
        if other is None:
           op = " IS "
       return DbQueryColumnComparison(self, op, other)
    def __ne__(self, other):
        op = "!="
        if other is None:
            op = " IS NOT "
        return DbQueryColumnComparison(self, op, other)
    def __gt__(self, other):
        return DbQueryColumnComparison(self, "&gt;", other)
    def __ge__(self, other):
        return DbQueryColumnComparison(self, "&gt;=", other)

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

class DbQueryExpression(object):

"""

Query expression created from columns, literals, and operators

"""

def __and__(self, other):

return DbQueryConjunction(self, other)

def __or__(self, other):

return DbQueryDisjunction(self, other)

def __str__(self):

raise NotImplementedError

@property

def arguments(self):

raise NotImplementedError

class DbQueryConjunction(DbQueryExpression):

"""

Query expression joining together a left and right expression with an

AND statement

"""

def __init__(self, l, r):

DbQueryExpression.__ini__(self)

self.l = l

self.r = r

def __str__(self):

return str(self.l) + " AND " + str(self.r)

@property

def arguments(self):

return self.l.arguments + self.r.arguments

class DbQueryDisjunction(DbQueryExpression):

"""

Query expression joining together a left and right expression with an

OR statement

"""

def __init__(self, l, r):

DbQueryExpression.__init__(self)

self.l = l

self.r = r

def __str__(self):

return str(self.r) + " OR " + str(self.r)

@property

def arguments(self):

return self.l.arguments + self.r.arguments

class DbQueryColumnComparison(DbQueryExpression):

"""

Query expression comparing a combination of a column and/or a value

"""

def __init__(self, l, op, r):

DbQueryExpression.__init__(self)

self.l = l

self.op = op

self.r = r

def __str__(self):

output = ""

if isinstance(self.l, DbQueryColumn):

prefix = self.l.prefix

if prefix is not None:

output += prefix + "."

output += self.l.name

elif self.l is None:

output += "NULL"

else:

output += "%s"

output += self.op

if isinstance(self.r, DbQueryColumn):

prefix = self.r.prefix

if prefix is not None:

output += prefix + "."

output += self.r.name

elif self.r is None:

output += "NULL"

else:

output += "%s"

return output

@property

def arguments(self):

output = []

if not isinstance(self.l, DbQueryColumn) and self.l is not None:

output.append(self.l)

if not isinstance(self.r, DbQueryColumn) and self.r is not None:

output.append(self.r)

return output

class DbQueryColumnSet(object):

"""

Represents a set of columns attached to a specific DbOject type. This

object dynamically builds itself based on a passed type. The columns

attached to this set may be used in DbQueries

"""

def __init__(self, dbo_type, prefix):

d = dbo_type.__dict__

self.__columns = {}

for attr in d:

obj = d[attr]

if isinstance(obj, Column):

column = DbQueryColumn(dbo_type, prefix, obj.name)

setattr(self, attr, column)

self.__columns[obj.name] = column

def __len__(self):

return len(self.__columns)

def __getitem__(self, key):

return self.__columns[key]

def __iter__(self):

return iter(self.__columns)

class DbQueryColumn(object):

"""

Represents a Column object used in a DbQuery

"""

def __init__(self, dbo_type, prefix, column_name):

self.dbo_type = dbo_type

self.name = column_name

self.prefix = prefix

def __lt__(self, other):

return DbQueryColumnComparison(self, "<", other)

def __le__(self, other):

return DbQueryColumnComparison(self, "<=", other)

def __eq__(self, other):

op = "="

if other is None:

op = " IS "

return DbQueryColumnComparison(self, op, other)

def __ne__(self, other):

op = "!="

if other is None:

op = " IS NOT "

return DbQueryColumnComparison(self, op, other)

def __gt__(self, other):

return DbQueryColumnComparison(self, ">", other)

def __ge__(self, other):

return DbQueryColumnComparison(self, ">=", other)

The __str__ function and arguments property return recursively generated expressions using the column prefixes (in the case of __str__) and the arguments (in the case of arguments). As can be seen, this supports parameterization of queries. To be honest, this part was the most fun since I was surprised it was so easy to make predicate expressions using a minimum of classes. One thing that I didn’t like, however, was the fact that the boolean and/or operators cannot be overloaded. For that reason I had to use the bitwise operators, so the expressions aren’t entirely correct when being read.

This DbQueryExpression is fed into my DbQuery object which actually does the translation to SQL. In the example above, we saw that I just passed a logical argument into my where function. This actually was a DbQueryExpression since my overloaded operators create DbQueryExpression objects when they are compared. The DbColumnSet object is an dynamically generated object containing the go-between column objects which is created from a DbObject. We will discuss the DbObject a little further down

The DbQuery objects are implemented as follows:

class DbQueryError(Exception):
    """
    Raised when there is an error constructing a query
    """
    def __init__(self, msg):
        self.message = msg
    def __str__(self):
        return self.message

class DbQuery(object):
    """
    Represents a base SQL Query to a database based upon some DbObjects

    All of the methods implemented here are valid on select, update, and
    delete statements.
    """
    def __init__(self, execute_filter=None):
        """
        callback: Function to call when the DbQuery is executed
        """
        self.__where = []
        self.__limit = None
        self.__orderby = []
        self.__execute_filter = execute_filter
    def where(self, expression):
        """Specify an expression to append to the WHERE clause"""
        self.__where.append(expression)
    def limit(self, value=None):
        """Specify the limit to the query"""
        self.__limit = value
    @property
    def sql(self):
        query = ""
        args = []
        if len(self.__where) &gt; 0:
            where = self.__where[0]
            for clause in self.__where[1:]:
                where = where &amp; clause
            args = where.arguments
            query += " WHERE " + str(where)
        if self.__limit is not None:
            query += " LIMIT " + self.__limit
        return query,args
    def execute(self, cur):
        """
        Executes this query on the passed cursor and returns either the result
        of the filter function or the cursor if there is no filter function.
        """
        query = self.sql
        cur.execute(query[0], query[1])
        if self.__execute_filter:
            return self.__execute_filter(self, cur)
        else:
            return cur

class DbSelectQuery(DbQuery):
    """
    Creates a select query to a database based upon DbObjects
    """
    def __init__(self, execute_filter=None):
        DbQuery.__init__(self, execute_filter)
        self.__select = []
        self.__froms = []
        self.__joins = []
        self.__orderby = []
    def select(self, *columns):
        """Specify one or more columns to select"""
        self.__select += columns
    def from_table(self, dbo_type, prefix):
        """Specify a table to select from"""
        self.__froms.append((dbo_type, prefix))
    def join(self, dbo_type, prefix, on):
        """Specify a table to join to"""
        self.__joins.append((dbo_type, prefix, on))
    def orderby(self, *columns):
        """Specify one or more columns to order by"""
        self.__orderby += columns
    @property
    def sql(self):
        query = "SELECT "
        args = []
        if len(self.__select) == 0:
            raise DbQueryError("No selection in DbSelectQuery")
        query += ','.join([col.prefix + "." + 
                 col.name for col in self.__select])
        if len(self.__froms) == 0:
            raise DbQueryError("No FROM clause in DbSelectQuery")
        for table in self.__froms:
            query += " FROM " + table[0].dbo_tablename + " " + table[1]
        if len(self.__joins) &gt; 0:
            for join in self.__joins:
                query += " JOIN " + join[0].dbo_tablename + " " + join[1] + 
                         " ON " + str(join[2])
        query_parent = super(DbSelectQuery, self).sql
        query += query_parent[0]
        args += query_parent[1]
        if len(self.__orderby) &gt; 0:
           query += " ORDER BY " + 
                    ','.join([col.prefix + "." + 
                    col.name for col in self.__orderby])
        return query,args

class DbInsertQuery(DbQuery):
    """
    Creates an insert query to a database based upon DbObjects. This does not
    include any where or limit expressions
    """
    def __init__(self, dbo_type, prefix, execute_filter=None):
        DbQuery.__init__(self, execute_filter)
        self.table = (dbo_type, prefix)
        self.__values = []
    def value(self, column, value):
        self.__values.append((column, value))
    @property
    def sql(self):
        if len(self.__values) == 0:
            raise DbQueryError("No values in insert")
        tablename = self.table[0].dbo_tablename
        query = "INSERT INTO {table} (".format(table=tablename)
        args = [val[1] for val in self.__values 
                if val[0].prefix == self.table[1]]
        query += ",".join([val[0].name for val in self.__values 
                          if val[0].prefix == self.table[1]])
        query += ") VALUES ("
        query += ",".join(["%s" for x in args])
        query += ")"
        return query,args

class DbUpdateQuery(DbQuery):
    """
    Creates an update query to a database based upon DbObjects
    """
    def __init__(self, dbo_type, prefix, execute_filter=None):
        """
        Initialize the update query

        dbo_type: table type to be updating
        prefix: Prefix the columns are known under
        """
        DbQuery.__init__(self, execute_filter)
        self.table = (dbo_type, prefix)
        self.__updates = []
    def update(self, left, right):
        self.__updates.append((left, right))
    @property
    def sql(self):
        if len(self.__updates) == 0:
            raise DbQueryError("No update in DbUpdateQuery")
        query = "UPDATE " + self.table[0].dbo_tablename + " " + self.table[1]
        args = []
        query += " SET "
        for update in self.__updates:
            if isinstance(update[0], DbQueryColumn):
                query += update[0].prefix + "." + update[0].name
            else:
                query += "%s"
                args.append(update[0])
            query += "="
            if isinstance(update[1], DbQueryColumn):
                query += update[1].prefix + "." + update[1].name
            else:
                query += "%s"
                args.append(update[1])
        query_parent = super(DbUpdateQuery, self).sql
        query += query_parent[0]
        args += query_parent[1]
        return query, args

class DbDeleteQuery(DbQuery):
    """
    Creates a delete query for a database based on a DbObject
    """
    def __init__(self, dbo_type, prefix, execute_filter=None):
        DbQuery.__init__(self, execute_filter)
        self.table = (dbo_type, prefix)
    @property
    def sql(self):
        query = "DELETE FROM " + self.table[0].dbo_tablename + " " + 
                self.table[1]
        args = []
        query_parent = super(DbDeleteQuery, self).sql
        query += query_parent[0]
        args += query_parent[1]
        return query, args

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

class DbQueryError(Exception):

"""

Raised when there is an error constructing a query

"""

def __init__(self, msg):

self.message = msg

def __str__(self):

return self.message

class DbQuery(object):

"""

Represents a base SQL Query to a database based upon some DbObjects

All of the methods implemented here are valid on select, update, and

delete statements.

"""

def __init__(self, execute_filter=None):

"""

callback: Function to call when the DbQuery is executed

"""

self.__where = []

self.__limit = None

self.__orderby = []

self.__execute_filter = execute_filter

def where(self, expression):

"""Specify an expression to append to the WHERE clause"""

self.__where.append(expression)

def limit(self, value=None):

"""Specify the limit to the query"""

self.__limit = value

@property

def sql(self):

query = ""

args = []

if len(self.__where) > 0:

where = self.__where[0]

for clause in self.__where[1:]:

where = where & clause

args = where.arguments

query += " WHERE " + str(where)

if self.__limit is not None:

query += " LIMIT " + self.__limit

return query,args

def execute(self, cur):

"""

Executes this query on the passed cursor and returns either the result

of the filter function or the cursor if there is no filter function.

"""

query = self.sql

cur.execute(query[0], query[1])

if self.__execute_filter:

return self.__execute_filter(self, cur)

else:

return cur

class DbSelectQuery(DbQuery):

"""

Creates a select query to a database based upon DbObjects

"""

def __init__(self, execute_filter=None):

DbQuery.__init__(self, execute_filter)

self.__select = []

self.__froms = []

self.__joins = []

self.__orderby = []

def select(self, *columns):

"""Specify one or more columns to select"""

self.__select += columns

def from_table(self, dbo_type, prefix):

"""Specify a table to select from"""

self.__froms.append((dbo_type, prefix))

def join(self, dbo_type, prefix, on):

"""Specify a table to join to"""

self.__joins.append((dbo_type, prefix, on))

def orderby(self, *columns):

"""Specify one or more columns to order by"""

self.__orderby += columns

@property

def sql(self):

query = "SELECT "

args = []

if len(self.__select) == 0:

raise DbQueryError("No selection in DbSelectQuery")

query += ','.join([col.prefix + "." +

col.name for col in self.__select])

if len(self.__froms) == 0:

raise DbQueryError("No FROM clause in DbSelectQuery")

for table in self.__froms:

query += " FROM " + table[0].dbo_tablename + " " + table[1]

if len(self.__joins) > 0:

for join in self.__joins:

query += " JOIN " + join[0].dbo_tablename + " " + join[1] +

" ON " + str(join[2])

query_parent = super(DbSelectQuery, self).sql

query += query_parent[0]

args += query_parent[1]

if len(self.__orderby) > 0:

query += " ORDER BY " +

','.join([col.prefix + "." +

col.name for col in self.__orderby])

return query,args

class DbInsertQuery(DbQuery):

"""

Creates an insert query to a database based upon DbObjects. This does not

include any where or limit expressions

"""

def __init__(self, dbo_type, prefix, execute_filter=None):

DbQuery.__init__(self, execute_filter)

self.table = (dbo_type, prefix)

self.__values = []

def value(self, column, value):

self.__values.append((column, value))

@property

def sql(self):

if len(self.__values) == 0:

raise DbQueryError("No values in insert")

tablename = self.table[0].dbo_tablename

query = "INSERT INTO {table} (".format(table=tablename)

args = [val[1] for val in self.__values

if val[0].prefix == self.table[1]]

query += ",".join([val[0].name for val in self.__values

if val[0].prefix == self.table[1]])

query += ") VALUES ("

query += ",".join(["%s" for x in args])

query += ")"

return query,args

class DbUpdateQuery(DbQuery):

"""

Creates an update query to a database based upon DbObjects

"""

def __init__(self, dbo_type, prefix, execute_filter=None):

"""

Initialize the update query

dbo_type: table type to be updating

prefix: Prefix the columns are known under

"""

DbQuery.__init__(self, execute_filter)

self.table = (dbo_type, prefix)

self.__updates = []

def update(self, left, right):

self.__updates.append((left, right))

@property

def sql(self):

if len(self.__updates) == 0:

raise DbQueryError("No update in DbUpdateQuery")

query = "UPDATE " + self.table[0].dbo_tablename + " " + self.table[1]

args = []

query += " SET "

for update in self.__updates:

if isinstance(update[0], DbQueryColumn):

query += update[0].prefix + "." + update[0].name

else:

query += "%s"

args.append(update[0])

query += "="

if isinstance(update[1], DbQueryColumn):

query += update[1].prefix + "." + update[1].name

else:

query += "%s"

args.append(update[1])

query_parent = super(DbUpdateQuery, self).sql

query += query_parent[0]

args += query_parent[1]

return query, args

class DbDeleteQuery(DbQuery):

"""

Creates a delete query for a database based on a DbObject

"""

def __init__(self, dbo_type, prefix, execute_filter=None):

DbQuery.__init__(self, execute_filter)

self.table = (dbo_type, prefix)

@property

def sql(self):

query = "DELETE FROM " + self.table[0].dbo_tablename + " " +

self.table[1]

args = []

query_parent = super(DbDeleteQuery, self).sql

query += query_parent[0]

args += query_parent[1]

return query, args

Each of the SELECT, INSERT, UPDATE, and DELETE query types inherits from a base DbQuery which does execution and such. I decided to make the DbQuery object take a PEP 249-style cursor object and execute the query itself. My hope is that this will make this a little more portable since, to my knowledge, I didn’t make the queries have any MySQL-specific constructions.

The different query types each implement a variety of statements corresponding to different parts of an SQL query: where(), limit(), orderby(), select(), from_table(), etc. These each take in either a DbQueryColumn (such as is the case with where(), orderby(), select(), etc) or a string to be appended to the query, such as is the case with limit(). I could easily have made limit take in two integers as well, but I was kind of rushing through because I wanted to see if this would even work. The query is built by creating the query object for the basic query type that is desired and then calling its member functions to add things on to the query.

Executing the queries can cause a callback “filter” function to be called which takes in the query and the cursor as arguments. I use this function to create new objects from the data or to update an object. It could probably be used for more clever things as well, but those two cases were my original intent in creating it. If no filter is specified, then the cursor is returned.

Table and row objects

At the highest level of this hierarchy is the DbObject. The DbObject definition actually represents a table in the database with a name and a single primary key column. Each instance represents a row. DbObjects also implement the methods for selecting records of their type and also updating themselves when they are changed. They inherit change tracking from the ColumnSet and use DbQueries to accomplish their querying goals. The code is as follows:

class DbObject(ColumnSet):
    """
    A DbObject is a set of columns linked to a table in the database. This is
    synonomous to a row. The following class attributes must be set:

    dbo_tablename : string table name
    primary_key : Column for the primary key
    """
    def __init__(self, **cols):
        ColumnSet.__init__(self)
        for name in cols:
            c = self.get_column(name)
            c.update(cols[name])

    @classmethod
    def get_query_columns(self, prefix):
        return DbQueryColumnSet(self, prefix)

    @classmethod
    def select(self, prefix):
        """
        Returns a DbSelectQuery set up for this DbObject
        """
        columns = self.get_query_columns(prefix)
        def execute(query, cur):
            output = []
            block = cur.fetchmany()
            while len(block) &gt; 0:
                for row in block:
                    values = {}
                    i = 0
                    for name in columns:
                        values[name] = row[i]
                        i += 1
                    output.append(self(**values))
                block = cur.fetchmany()
            return output
        query = DbSelectQuery(execute)
        query.select(*[columns[name] for name in columns])
        query.from_table(self, prefix)
        return query, columns

    def get_primary_key_name(self):
        return type(self).__dict__['primary_key'].name

    def save(self, cur):
        """
        Saves any changes to this object to the database
        """
        if self.primary_key is None:
            # we need to be saved
            columns = self.get_query_columns('x')
            def execute(query, cur):
                self.get_column(self.get_primary_key_name()
                                ).update(cur.lastrowid)
                selection = []
                for name in columns:
                    if name == self.get_primary_key_name():
                        continue #we have no need to update the primary key
                    column_instance = self.get_column(name)
                    if not column_instance.column.mutable:
                        selection.append(columns[name])
                if len(selection) != 0:
                    # we get to select to get additional computed values
                    def execute2(query, cur):
                        row = cur.fetchone()
                        index = 0
                        for s in selection:
                            self.get_column(s.name).update(row[index])
                            index += 1
                        return True
                    query = DbSelectQuery(execute2)
                    query.select(*selection)
                    query.from_table(type(self), 'x')
                    query.where(columns[self.get_primary_key_name()] == 
                                self.get_column(self.get_primary_key_name()
                                                ).value)
                    return query.execute(cur)
                return True
            query = DbInsertQuery(type(self), 'x', execute)
            for name in columns:
                column_instance = self.get_column(name)
                if not column_instance.column.mutable:
                    continue
                query.value(columns[name], column_instance.value)
            print query.sql
            return query.execute(cur)
        else:
            # we have been modified
            modified = self.mutated
            if len(modified) == 0:
                return True
            columns = self.get_query_columns('x')
            def execute(query, cur):
                for mod in modified:
                    mod.update(mod.value)
                return True
            query = DbUpdateQuery(type(self), 'x', execute)
            for mod in modified:
                query.update(columns[mod.column.name], mod.value)
            query.where(columns[self.get_primary_key_name()] == self.primary_key)
            return query.execute(cur)

100

101

102

class DbObject(ColumnSet):

"""

A DbObject is a set of columns linked to a table in the database. This is

synonomous to a row. The following class attributes must be set:

dbo_tablename : string table name

primary_key : Column for the primary key

"""

def __init__(self, **cols):

ColumnSet.__init__(self)

for name in cols:

c = self.get_column(name)

c.update(cols[name])

@classmethod

def get_query_columns(self, prefix):

return DbQueryColumnSet(self, prefix)

@classmethod

def select(self, prefix):

"""

Returns a DbSelectQuery set up for this DbObject

"""

columns = self.get_query_columns(prefix)

def execute(query, cur):

output = []

block = cur.fetchmany()

while len(block) > 0:

for row in block:

values = {}

i = 0

for name in columns:

values[name] = row[i]

i += 1

output.append(self(**values))

block = cur.fetchmany()

return output

query = DbSelectQuery(execute)

query.select(*[columns[name] for name in columns])

query.from_table(self, prefix)

return query, columns

def get_primary_key_name(self):

return type(self).__dict__['primary_key'].name

def save(self, cur):

"""

Saves any changes to this object to the database

"""

if self.primary_key is None:

# we need to be saved

columns = self.get_query_columns('x')

def execute(query, cur):

self.get_column(self.get_primary_key_name()

).update(cur.lastrowid)

selection = []

for name in columns:

if name == self.get_primary_key_name():

continue #we have no need to update the primary key

column_instance = self.get_column(name)

if not column_instance.column.mutable:

selection.append(columns[name])

if len(selection) != 0:

# we get to select to get additional computed values

def execute2(query, cur):

row = cur.fetchone()

index = 0

for s in selection:

self.get_column(s.name).update(row[index])

index += 1

return True

query = DbSelectQuery(execute2)

query.select(*selection)

query.from_table(type(self), 'x')

query.where(columns[self.get_primary_key_name()] ==

self.get_column(self.get_primary_key_name()

).value)

return query.execute(cur)

return True

query = DbInsertQuery(type(self), 'x', execute)

for name in columns:

column_instance = self.get_column(name)

if not column_instance.column.mutable:

continue

query.value(columns[name], column_instance.value)

print query.sql

return query.execute(cur)

else:

# we have been modified

modified = self.mutated

if len(modified) == 0:

return True

columns = self.get_query_columns('x')

def execute(query, cur):

for mod in modified:

mod.update(mod.value)

return True

query = DbUpdateQuery(type(self), 'x', execute)

for mod in modified:

query.update(columns[mod.column.name], mod.value)

query.where(columns[self.get_primary_key_name()] == self.primary_key)

return query.execute(cur)

DbObjects require that the inheriting classes define two properties: dbo_tablename and primary_key. dbo_tablename is just a string giving the name of the table in the database and primary_key is a Column that will be used as the primary key.

To select records from the database, the select() function can be called from the class. This sets up a DbSelectQuery which will return an array of the DbObject that it is called for when the query is executed.

One fallacy of this structure is that at the moment it assumes that the primary key won’t be None if it has been set. In other words, the way I did it right now does not allow for null primary keys. The reason it does this is because it says that if the primary key hasn’t been set, it needs to generate a DbInsertQuery for the object when save() is called instead of a DbUpdateQuery. Both insert and update queries do not include every field. Immutable fields are always excluded and then later selected or inferred from the cursor object.

256Mb doesn’t do what it used to…

So, this last week I got an email from rackspace saying that my server was thrashing the hard drives and lowering performance of everyone else on that same machine. In consequence, they had rebooted my server for me.

I made a few mistakes in the setup of this first iteration of my server: I didn’t restart it after kernel updates, I ran folding@home on it while running nodejs, and I didn’t have backups turned on. I had it running for well over 200 days without a reboot while there had been a dozen or so kernel updates. When they hard rebooted my server, it wouldn’t respond at all to pings, ssh, or otherwise. In fact, it behaved like the firewalls were shut (hanging on the “waiting” step rather than saying “connection refused”). I ended up having to go into their handy rescue mode and copy out all the files. I only copied my www directory and the mysql binary table files, but as you can see, I was able to restore the server from those.

This gave me an excellent opportunity to actually set up my server correctly. I no longer have to be root to edit my website files (yay!), I have virtual hosts set up in a fashion that makes sense and actually works, and overall performance seems to be improved. From now on, I will be doing the updates less frequently and when I do I will be rebooting the machine. That should fix the problem with breaking everything if a hard reboot happens.

I do pay for the hosting for this, 1.5 cents per hour per 256Mb of RAM with extra for bandwidth. I only have 256Mb and since I don’t make any profit off this server whatsoever at the moment, I plan on keeping it that way for now. Considering that back in the day, 256Mb was a ton of memory, it clearly no longer suffices for running too much on my server (httpd + mysql + nodejs + folding@home = crash and burn).

MMO Asteroids in Node.js…but for real.

Many people saw this april fools joke where the author said that he had created an asteroids MMO using Node.js. In reality, it was completely client side and was a bunch of bots. However, I did find the whole thing rather intriguing and decided to see what I could do with Node.js along this line. Last week I started on the project and this weekend I made enough progress that I can publish v0.0.1. It leaves several things to be desired, including a better game background so that one can tell when the view is shifting and user tracking so that you accumulate your scores over time. It does work and from what I can see its not horribly bad performance. While it certainly won’t be able to handle thousands of clients, I expect that it should be able to handle somewhere between 50-100 before it starts dying. At the moment, its quite limited by the memory on my server and the client side scripting gives the impression of “stuttering” with the dead reckoning system used to make the animations smooth. The stuttering is caused by the linear and angular damping that I have running on the server side not being factored into the projected location on the client side.

For physics I am using Box2Dweb in a Node.js module which may be overkill, but its the simplest Javascript physics engine I could find since I didn’t feel like writing my own. The server keeps track of all of the entities in the game and each client requests a “view” of an area of the room. The client is informed which player is them, but other than that it just sees all players and entities in the same list. The actual rendering function draws the entities onto the canvas dependent upon the type of the object.

I have made the source available on github here: https://github.com/kcuzner/mmo-asteroids

The MMO itself can be viewed here: http://kevincuzner.com:8080/. Note that at times it may be down since I am messing with my firewall right now and I might accidentally close the port. Just leave me a note in the comments and I’ll try to get it working.

Simulation in C++ finally works

After redoing this project several times, I have finally managed to get it working in C++. I gave up for the moment on the GUI since I figured it would be best to get the back end model working first before tackling the GUI. The biggest change I made from my previous paradigm is that I decided to split this into several Qt projects. The model is held in a project called Engine and doesn’t actually depend on Qt. Instead I decided to use Boost so that if I one day decide to ditch Qt I won’t have to throw out my model.

In designing the model I had two principal design concerns: Everything has to implement interfaces and everything has to be interchangeable. By defining everything as an interface (classes with a virtual destructor and all pure virtual methods returning types easily resolved without any additional code) this allows for additional expansion later at the hands of myself or somebody else who decides on a better implementation of any component of the engine. It also allows the Qt plugin system to be able to handle references to any of the types in the engine so that a future plugin interface can use them.

The structure I decided on is similar, but not identical to, the one that I used in the python version of this. The basic unit of a simulation is the Block which implements the IBlock interfaces. There are three child interfaces of IBlock: IEntryBlock, IExitBlock, and IModelBlock. IBlocks define inputs (IBlockInput) and outputs (IBlockOutput) and are contained in Models (IModel interface) which represents units to be simulated. Blocks may define entries and exits (IEntryBlock & IExitBlock) and hold any number of interconnected blocks. Entries with the same name as an Exit will be set to the value of the Exit at the end of each step. This allows for persistent variables in the structure (as opposed to just putting them in code). IBlockOutputs can be attached to multiple IBlockInputs but each IBlockInput may only be attached to one IBlockOutput. Boost Signals2 are used for maintaining the cleanliness of the model so that when an input has its attached output replaced, the formerly attached output is notified. IBlocks also signal when the have an input or output added or removed. Models can also be used as blocks (IModelBlock) inside of other Models, allowing for nested execution and creation of reusable components. When a simulation is running, a block can be re-used multiple times in different “contexts”. An example of this is a model that is used inside an IModelBlock. The model and all of its component blocks are only instantiated once, but the IModel could be used in multiple IModelBlocks which could be placed inside of different parent IModels. For this reason, I created the IContext interface. An IBlock is passed the IContext whenever an option is set, the block is initialized, or the block is executed. The block is expected to store all of its input, output, and stored values inside the context and not actually use any class member variables to store values between steps. In the case of an IModelBlock, it is expected that the block will “own” a child context in a parent context. This is so that each “instance” of the model being executed as a child model will have its own context to store variables in. The root context and root model are tracked by an IEngine. The IEngine takes care of setting the number of steps that constitute a simulation, the size of the delta value between steps, and what step the simulation is on. The IContext does the actual execution and is created by the IEngine (IEngine::getContext()). It is expected that when executing a step the IContext will queue up all the IBlocks in the IModel that have no inputs and that once all the inputs are set for an IBlock it will be queued for execution. A step is finished when there are no longer blocks to execute. To facilitate looping blocks like those that are found in Simulink, the IContext should re-queue a block for execution each time an input is set again after all of its inputs have already been set. While this leads to the possibility of infinite loops, the connection function should try to locate possible loops and stop them from happening.

As for exact implementation, I have created an IEngine called SimpleEngine (creating DefaultContexts) and I plan on creating a ThreadedEngine (creating ThreadeContexts) since the execution of blocks can become parallel at points where an IBlockOutput connects to multiple IBlocks. Other that those specifics, the default implementation is pretty much as specified above. The IModelBlock implementation subscribes to the signals on the IModel which are fired when IEntryBlocks and IExitBlocks are added/removed so that it can keep its IOs in sync.

A gist of how a simple simulation can be set up and run:

[gist id=”4391549″ file=”simulate_basic_test.cpp”]

The next thing on the docket before I start off on creating all of the system blocks (basically, trying to clone the functionality of the blocks listed for Simulink) and making the GUI is to create a set of interfaces for describing the model in files. This is so that I can create an interface to a generic loader allowing for multiple file formats to be loaded (XML, JSON, BSON, something binary, etc) for describing simulations.

Simulink clone…now in C++

I took the plunge and decided to re-implement what I had in Python using C++. I had to change up my structure a bit, but I made the switch because of the following reasons:

The Qt Framework. While there are bindings for Python, I really liked Qt-Creator. Also, Qt is extremely cross platform (at least that is how it seems) and has a large amount of libraries. I’ve been messing around with it now for a bit trying to get a few things to work.
This is redundant of the above, but I really like the Qt Plugin system. After struggling with it for a bit, I finally was able to get a plugin to load. I will explain a little bit below exactly what I plan on doing with these.
As awesome as Python is, it doesn’t support true multithreading (it does multiprocessing…and to do what I wanted to do using multiprocessing would have required me to jump through some hula hoops)
C++ should have the potential to run faster than Python for mathematical things, which in this situation is a good trade off for ease of programming.
Writing the application entirely in C++ with Qt lowers the number of dependencies that would have to be installed on a client machine.

Now, the biggest advantages I see are: Speed and Extensibility. Python is extremely extensible, but it doesn’t have the greatest speed. C++ has the potential to run faster and then by using Qt, the extensibility part was brought in. I also prefer strongly typed systems since they keep me from stepping on my toes programmatically.

By far the coolest part of all this is the Plugins. After discovering that ALL communication between Plugins and the application must be done using interfaces, I realized that if I were to implement the entire thing using interfaces it could be extended to do many awesome things. So far, however, I have only really added interfaces to allow for adding computational “blocks” to the system for use in schematics. The system itself will define no blocks since I have decided to separate the engine from the actual blocks.

I will be posting later a bit about Qt plugins since that is what I have spent the most time on. Google was definitely my friend on that one. Most people it seems just use Qt Plugins for extending Qt itself rather than doing the “low level” extending the application stuff.

In terms of development time, C++ is quite a bit slower for me than Python. However, my potential to write good code is much higher since I am much more familiar with C++ coding conventions and I am more able to clean code while being confident nothing is being broken since in Python, there are no compile-time errors to tell you that you switched the arguments to a function.

Cloning Simulink…in Python

For a while now I have been working on a bench supply. As part of this I have been trying to get a PID controller to work. At first it was simple, but after asking my Dad about it (he does power electronics), he suggested that I use a cascaded PID loop for controlling the voltage and current using the voltage alone. I have sort of a bench model going, but I don’t really want to start construction until I have everything finalized since blowing things up and making mistakes is kind of expensive for my meager college student budget. Tweaking that without a working bench model that I am willing to blow up is kind of hard, so I started trying to figure out how to simulate it. Being partial to simulink (I’ve used it before with some nice pre-built blocks), I wanted to be able to lay it out graphically like control system diagrams usually show and I also wanted to be able to view plots over time. So thus was born my latest project: SimuPy.

Python seemed like an ideal language for this since I wanted it to look nice, be extensible, and be almost universally cross platform. I am relying heavily on the Qt library because it runs on almost anything and it has the ability to use slots and signals on pretty much any object as well. I guess another option would have been Java, but seeing as I don’t like Java that much, Python was what I went with. In addition, I am weighing a couple other options:

At the sacrifice of portability, I could make the interface half in C++ and half Python so that I still keep the extensibility without having the requirement of installing python on the person’s computer to get it to work. The simulation part would then turn into a python library that the program would load.
Write the entire thing in C++ for speed and true multithreading and use Python solely for extensions. This would be a little more difficult since I kind of leverage the dynamic typing thing that Python has going on.

So far, my current structure has everything based on a Model which contains Blocks. In addition, there is a simulation Context which holds information about each Block and where the simulation is in terms of the current step (simulations are stepped over time (dt)). Contexts are also where a Block will store all of its information that it needs to retain during the current step and in the next step. A Block is an operation over time in the flow of the simulation: it could be a simple addition, maybe a derivative or integral, or it could be a full PID controller. Blocks declare themselves to have a number of Input objects and Output objects. Inputs/Outputs are named and have a slot/function called set which sets the value of the input or output. Outputs have a signal called ‘ready’ which inputs connect their ‘set’ slot to. When an output’s ‘set’ method is called, it emits its signal. When an input is set and it sees that all inputs attached to its block are set, it performs a “step” on the block. In addition, there are 3 special blocks: An EntryBlock, ExitBlock, and ModelBlock. Entry and Exit blocks are used in models since a model can have “Entry” and “Exit” points. These points can be used to loop a value from an Exit to an Entry (if they have the same name) or can be used as Input and Output objects if the Model is placed inside a ModelBlock. ModelBlocks are blocks which contain a model which they execute in a child simulation Context to their context. In this way, blocks can be nested. If one creates a Model with 2 Entries and 2 Exits with a pair of those Entries and Exits having the same name and then the Model is attached to a ModelBlock, the ModelBlock will have 1 input and one output to corrispond to the free Entry and Exit on the Model. Models can’t be recursive, but they can be nested so as long as a higher level block doesn’t contain a block which at some child level contains the same higher level block, there can be some sense of re-usability and modularity to a simulation.

Blocks are subclassed into a package called model. The __init__.py file in the model package defines the basic form for a block and then the individual modules in the package define more specific blocks. The blocks then have their constructors cached by reflection so that a block can be constructed by simply naming its name. To extend the blocks available in a simulation, all that must be done is to drop the new module python file into the model folder. I am considering changing this a bit to separate out user-added modules from the “system” modules in kind of the same fashion as I did with the WebSocketServer where I had the files in a folder be loaded into the context of another package.

Simulations are to be stored in an XML format which is going to be more or less human readable and should preserve the look and feel of the simulation. I am still working on the exact format at the moment, but that is the next step.

As for the GUI, I plan on using Qt since it seems the most cross-platform (sorry GTK…Windows needs too much help to load you and PyDev in eclipse doesn’t like the whole introspection thing). I plan on releasing the project under the Apache License (but don’t yet quote me on that or hold me to it…I may choose a different license later once I get more of a feel for how the project would be used). Either way, I plan on publishing the source code on github since it looks like nothing like this really exists in a simple form. Sure, there are clones of Simulink to work with Octave and things like that, but it doesn’t look like there are few, if any, stand-alone applications that do this (except perhaps a paid program called logic.ly, but this should be able to duplicate the functionality of that program as well). I guess it is kind of a niche market since the only people who do this kind of thing usually can afford Simulink and Matlab.

For the record, I do have access to Simulink and Matlab through the University I am attending, but where would the fun be in that?

KnockoutJS and Memory Usage

Recently at work I have been using KnockoutJS for structuring my Javascript. To be honest, it is probably the best thing since jQuery in my opinion in terms of cutting down quantity of code that one must write for an interface. The only problem is, however, that it is really really easy to make a page use a ridiculous amount of memory. After thinking and thinking and trying different things I have realized the proper way to do things with more complex pages.

The KnockoutJS documentation is really great, but it is more geared towards the simple stuff/basics so that you can get started quickly and doesn’t talk much about more complex stuff which leads to comments like the answer here saying that it isn’t so good for complex user interfaces. When things get more complex, like interfacing it with existing applications with different frameworks or handling very very large quantities of data, it doesn’t really say much and kind of leaves one to figure it out on their own. I have one particular project that I was working on which had the capability to display several thousand items in a graph/tree format while calculating multiple inheritance and parentage on several values stored in each item object. In chrome I witnessed this page use 800Mb easily. Firefox it was about the same. Internet explorer got to 1.5Gb before I shut if off. Why was it using so much memory? Here is an example that wouldn’t use a ton of memory, but it would illustrate the error I made:

Example

Javascript (note that this assumes usage of jQuery for things like AJAX):

function ItemModel(id, name) {
    var self = this;
    this.id = id;
    this.name = ko.observable(name);
    this.editing = ko.observable(false);
    this.save = function () {
        //logic that creates a new item if the id is null or just saves the item otherwise
        //through a call to $.ajax
    }
}

function ItemContainerModel(id, name, items) {
    var self = this;
    this.id = id;
    this.name = ko.observable(name);
    this.editing(true);
    this.items = ko.observableArray(items);
    this.save = function () {
        //logic that creates a new item container if the id is null or just saves the item container otherwise
        //through a call to $.ajax
    }
    this.add = function() {
        var aNewItem = new ItemModel(null, null);
        aNewItem.editing(true);
        self.items.push(aNewItem);
    }
    this.remove = function (item) {
        //$.ajax call to the server to remove the item
        self.items.remove(item);
    }
}

function ViewModel() {
    var self = this;
    this.containers = ko.observableArray();
    var blankContainer = new ItemContainerModel(null, null, []);
    this.selected = ko.observable(blankContainer);
    this.add = function () {
        var aNewContainer = new ItemContainerModel(null, null, []);
        aNewContainer.editing(true);
        self.containers.push(aNewContainer);
    }
    this.remove = function(container) {
        //$.ajax call to the server to remove the container
        self.containers.remove(container);
    }
    this.select = function(container) {
        self.selected(container);
    }
}

$(document).ready( function() {
    var vm = new ViewModel();
    ko.applyBindings(vm);
});

function ItemModel(id, name) {

var self = this;

this.id = id;

this.name = ko.observable(name);

this.editing = ko.observable(false);

this.save = function () {

//logic that creates a new item if the id is null or just saves the item otherwise

//through a call to $.ajax

}

function ItemContainerModel(id, name, items) {

var self = this;

this.id = id;

this.name = ko.observable(name);

this.editing(true);

this.items = ko.observableArray(items);

this.save = function () {

//logic that creates a new item container if the id is null or just saves the item container otherwise

//through a call to $.ajax

}

this.add = function() {

var aNewItem = new ItemModel(null, null);

aNewItem.editing(true);

self.items.push(aNewItem);

}

this.remove = function (item) {

//$.ajax call to the server to remove the item

self.items.remove(item);

}

function ViewModel() {

var self = this;

this.containers = ko.observableArray();

var blankContainer = new ItemContainerModel(null, null, []);

this.selected = ko.observable(blankContainer);

this.add = function () {

var aNewContainer = new ItemContainerModel(null, null, []);

aNewContainer.editing(true);

self.containers.push(aNewContainer);

}

this.remove = function(container) {

//$.ajax call to the server to remove the container

self.containers.remove(container);

}

this.select = function(container) {

self.selected(container);

}

$(document).ready( function() {

var vm = new ViewModel();

ko.applyBindings(vm);

});

Now for a really simple view (sorry for lack of styling or the edit capability, but hopefully the point will be clear):

&lt;a data-bind="click: add" href="#"&gt;Add container&lt;/a&gt;
&lt;ul data-bind="foreach: containers"&gt;
    &lt;li&gt;&lt;span data-bind="text: name"&gt;&lt;/span&gt; &lt;a data-bind="click: save" href="#"&gt;Save&lt;/a&gt; &lt;a data-bind="click: $parent.remove" href="#"&gt;Remove&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;div data-bind="with: selected"&gt;
    &lt;a data-bind="click: add" href="#"&gt;Add item&lt;/a&gt;
    &lt;div data-bind="foreach: items"&gt;
        &lt;div data-bind="text: name"&gt;&lt;/div&gt;
        &lt;a data-bind="click: save" href="#"&gt;&lt;/a&gt;
        &lt;a data-bind="click: $parent.remove" href="#"&gt;Remove&lt;/a&gt;
    &lt;/div&gt;
&lt;/div&gt;

<a data-bind="click: add" href="#">Add container</a>

<li><span data-bind="text: name"></span> <a data-bind="click: save" href="#">Save</a> <a data-bind="click: $parent.remove" href="#">Remove</a></li>

</ul>

<a data-bind="click: $parent.remove" href="#">Remove</a>

</div>

The Problem

So, what is the problem here with this model? It works just fine… you can add, remove, save, and display items in a collection of containers. However, if this view was to contain, say, 1000 containers with 1000 items each, what would happen? Well, we would have a lot of memory usage. Now, you could say that would happen no matter what you did and you wouldn’t be wrong. The question here is, how much memory is it going to use? The example above is not nearly the most efficient way of structuring a model and will consume much more memory than is necessary. Here is why:

Note how the saving, adding, and removing functions are implemented. They are declared attached to the this variable inside each object. Now, in languages like C++, C#, or Java, adding functions to an object (that is what attaching the function to the this variable does in Javascript if you aren’t as familiar with objects in Javascript) will not cause increased memory usage generally, but would rather just make the program size larger since the classes would all share the same compiled code. However, Javascript is different.

Javascript uses what are called closures. A closure is a very very powerful tool that allows for intuitive accessing and scoping of variables seen by functions. I won’t go into great detail on the awesome things you can do with these since many others have explained it better than I ever could. Another thing that Javascript does is that it treats functions as “1st class citizens” which essentially means that Javascript sees no difference between a function and a variable. All are alike. This allows you to assign a variable to point to a function (var variable = function () { alert(“hi”); };) so that you could call variable() and it would execute the function as if “variable” was the name of the function.

Now, tying all that together here is what happens: Closures “wrap up” everything in the scope of a function when it is declared so that it has access to all the variables that were able to be seen at that point. By treating functions almost like variables and assigning a function to a variable in the this object, you extend the this object to hold whatever that variable holds. Declaring the functions inline like we see in the add, remove, and save functions while in the scope of the object causes them to become specific to the particular instance of the object. Allow me to explain a bit: Every time that you call ‘new ItemModel(…)’, in addition to creating a new item model, it creates a new function: this.save. Every single ItemModel created has its very own instance of this.save. They don’t share the same function. Now, when we create a new ItemContainerModel, 3 new functions are also created specific to each instance of the ItemContainerModel. That basically means that if we were to create two containers with 3 items each inside we would get 8 functions created (2 for the items, 6 for the containers). In some cases this is very useful since it lets you create custom methods for each oject. To use the example of the item save function, instead of having to access the ‘id’ variable as stored in the object, it could use one of the function parameters in ‘function ItemModel(…)’ inside the save function. This is due to the fact that the closure wrapped up the variables passed into the ItemModel function since they were in scope to the this.save function. By doing this, you could have the this.save function modify something different for each instance of the ItemModel. However, in our situation this is more of an issue than a benefit: We just redundantly created 4 functions that do the exact same thing as 4 other functions that already exist. Each of those functions consumes memory and after a thousand of these objects are made, that usage gets to be quite large.

Solution

How can this be fixed? What we need to do is to reduce the number of anonymous functions that are created. We need to remove the save, add, and remove functions from the ItemModel and ItemContainerModel. As it turns out, the structure of Knockout is geared towards doing something which can save us a lot of memory usage.

When an event binding like ‘click’ is called, the binding will pass an argument into the function which is the model that was being represented for the binding. This allows us to know who called the method. We already see this in use in the example with the remove functions: the first argument was the model that was being referenced by the particular click when it was called. We can use this to fix our problem.

First, we must remove all functions from the models that will be duplicated often. This means that the add, remove, and save functions in the ItemContainer and the save function in the Item models have to go. Next, we create back references so that each contained object outside the viewmodel and its direct children knows who its daddy is. Here is an example:

function ItemModel(id, name, container) {
    //note the addition of the container argument

    //...keep the same variables as before, but remove the this.save stuff

    this.container = container; //add this as our back reference
}

function ItemContainerModel(id, name) {
    //NOTE 1: this didn't need an argument for a back reference. This is because it is a direct child of the root model and
    //since the root model contains the functions dealing with adding and removing containers, it already knows the array to
    //manipulate

    //NOTE 2: the items argument has been removed. This is so that the container can be created before the items and the back
    //reference above can be completed. So, the process for creating a container with items is now: create container, create
    //items with a reference to the container, and then add the items to the container by doing container.items(arrayOfItems);

    //remove all the functions from this model as well
}

function ViewModel() {
    //all the stuff we already had here from the example above stays

    //we add the following:
    this.saveItem = function (item) {
        //instead of using self.id and self.name() when creating our ajax request, we use item.id and item.name()
    }
    this.saveContainer = function(container) {
        //instead of using self.id and self.name() when creating our ajax request, we use item.id and item.name()
    }
    this.addItem = function(container) {
        var aNewItem = new ItemModel(null, null, container);
        aNewItem.editing(true);
        container.items.push(aNewItem);
    }
    this.removeItem = function(item) {
        //create a $.ajax request to remove the item based on its id
        item.container.items.remove(item); //using our back reference, we can remove the item from its parent container
    }
}

function ItemModel(id, name, container) {

//note the addition of the container argument

//...keep the same variables as before, but remove the this.save stuff

this.container = container; //add this as our back reference

}

function ItemContainerModel(id, name) {

//NOTE 1: this didn't need an argument for a back reference. This is because it is a direct child of the root model and

//since the root model contains the functions dealing with adding and removing containers, it already knows the array to

//manipulate

//NOTE 2: the items argument has been removed. This is so that the container can be created before the items and the back

//reference above can be completed. So, the process for creating a container with items is now: create container, create

//items with a reference to the container, and then add the items to the container by doing container.items(arrayOfItems);

//remove all the functions from this model as well

}

function ViewModel() {

//all the stuff we already had here from the example above stays

//we add the following:

this.saveItem = function (item) {

//instead of using self.id and self.name() when creating our ajax request, we use item.id and item.name()

}

this.saveContainer = function(container) {

//instead of using self.id and self.name() when creating our ajax request, we use item.id and item.name()

}

this.addItem = function(container) {

var aNewItem = new ItemModel(null, null, container);

aNewItem.editing(true);

container.items.push(aNewItem);

}

this.removeItem = function(item) {

//create a $.ajax request to remove the item based on its id

item.container.items.remove(item); //using our back reference, we can remove the item from its parent container

}

The view will now look like so (note that the bindings to functions now reference $root: the main ViewModel):

&lt;a data-bind="click: add" href="#"&gt;Add container&lt;/a&gt;
&lt;ul data-bind="foreach: containers"&gt;
    &lt;li&gt;&lt;span data-bind="text: name"&gt;&lt;/span&gt; &lt;a data-bind="click: $root.saveContainer href="#"&gt;Save&lt;/a&gt; &lt;a data-bind="click: $root.remove" href="#"&gt;Remove&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;div data-bind="with: selected"&gt;
    &lt;a data-bind="click: $root.addItem" href="#"&gt;Add item&lt;/a&gt;
    &lt;div data-bind="foreach: items"&gt;
        &lt;div data-bind="text: name"&gt;&lt;/div&gt;
        &lt;a data-bind="click: $root.saveItem" href="#"&gt;&lt;/a&gt;
        &lt;a data-bind="click: $root.removeItem" href="#"&gt;Remove&lt;/a&gt;
    &lt;/div&gt;
&lt;/div&gt;

<a data-bind="click: add" href="#">Add container</a>

<li><span data-bind="text: name"></span> <a data-bind="click: $root.saveContainer href="#">Save</a> <a data-bind="click: $root.remove" href="#">Remove</a></li>

</ul>

<a data-bind="click: $root.removeItem" href="#">Remove</a>

</div>

Now, that wasn’t so hard was it? What we just did was we made it so that we only use memory for the variables and don’t have to create any closures for functions. By moving the individual model functions down to the ViewModel we kept the same functionality as before, did not increase our code size, and significantly reduced memory usage when the model starts to get really big. If we were to create 2 containers with 3 items each, we create no additional functions from the 4 inside the ViewModel. The only memory consumed by each model is the space needed for storing the actual values represented (id, name, etc).

Summary

In summary, to reduce KnockoutJS memory usage consider the following:

Reduce the number of functions inside the scope of each model. Move functions to the lowst possible place in your model tree to avoid unnecessary duplication.
Avoid closures inside heavily duplicated models like the plague. I know I didn’t cover this above, but be careful with computed observables and their functions. It may be better to declare the bulk of a function for a computed observable outside the function and then use it like so: ‘this.aComputedObservable = ko.computed(function () { return aFunctionThatYouCreated(self); });’ where self was earlier declared to be this in the scope of the model itself. That way the computed observable function still has access to the contents of the model while keeping the actual memory usage in the model itself small.
Be very very slim when creating your model classes. Only put data there that will be needed.
Consider pagination or something. If you don’t need 1000 objects displayed at the same time, don’t display 1000 objects at the same time. There is a server there to store the information for a reason.

The first week or two with Arch Linux

After some frustrating times involving Ubuntu 12.04, hibernation, suspending, and random freezing I decided I needed to try something different. Being a Sandy Bridge desktop, my computer naturally seems to have a slight problem with Linux support in general. Don’t get me wrong, I really like my computer and my processor…however, the hardware drivers at times frustrate me. So, at my wits end I decided to do something crazy and take the plunge to a bleeding edge rolling release linux: Arch Linux.

Arch Linux is interesting for me since its the first time I have not been using an operating system with the “version” paradigm. Since its a rolling release it is prone to more problems, but it also gives the advantage of always being up to date. Since my computer’s hardware is relatively new (it has been superseded by Ivy Bridge, but even so its driver support still seems to be being built), I felt that I had more to gain from doing a rolling release where new updates and such would come out (think kernel 3.0 to 3-2…Sandy Bridge processors suddenly got much better power management) almost immediately. So, without further adieu, here are my plusses and minuses (note that this will end up comparing Ubuntu to Arch alot since that’s all I know at the moment):

Plusses:

It was actually very easy to install. Since I have had problems with net installs, I did a core install and then updated it. I practiced several times beforehand on virtual machines, including using existing partitions and such. Although the initial downloads took some time (the lack of curl in the core install was kind of upsetting since I couldn’t use rankmirrors to get better speeds), after that it was pretty fast. Thanks to considerable documentation and a few practice runs, getting an X enviroment set up using Gnome 3 (and gdm…I like the graphical logins) didn’t take long at all. It took a bit of coaxing (read: google + arch wiki) to get things such as networkmanager running and such, but with time I had it all figured out and I managed to get the whole system running more or less stably within a day.
It boots faster than Ubuntu and is more explicit about what exactly it’s doing. I liked the Ubuntu moving logo thing, but I do actually enjoy seeing what the computer is doing when it boots. Coming from pure asthetic reasons, it gives the computer more of a “raw” feel which for some strange masochistic reason, I enjoy. The slowest part is initializing the networks and if that didn’t have to happen the entire system could boot in under 60 seconds after the bios gets done showing off its screen.
The documentation is awesome. Clearly, people have spent lots and lots and lots of time writing the documentation in the wiki for Arch. It certainly made setting up easier since many of the random corner cases were in the troubleshooting section of severl articles and I ended up running into a couple of them. One thing that was easier to set up than in Ubuntu was suspending and hibernating (at least getting it to work reliably). With some help from the forum (see next point) and a few pages of documentationon pm-utils I got suspend, resume, and hibernate (!!!) running. I haven’t even gotten hibernate to work in Windows.
The community is great. I rarely have been able to get a question answered on the Ubuntu forums since they are so conjested. I asked a question on the arch bbs and in less than a day I had a response and was able to do some trial + error and troubleshooting involving the suspend and hibernate functionality of my computer.

Minuses:

The rolling release model breaks things occasionally. Recently, the linux-firmware package was updated and this caused my wireless card to stop working since it could no longer find the drivers. I wasn’t sure why, but I have just downgraded the package and blacklisted it for upgrades. Hopefully that doesn’t kill me later (it probably will), but if it does by then I hope to have figured out what is wrong.
With great power comes great responsibility. The sheer flexibility is great since I don’t have a bunch of extra packages I don’t need, but at the same time when I was practicing with the VMs, I was able to get myself stuck in a hole where the only solution was to re-format the drive. However, ever since a mishap with Ubuntu (the themeing engine changed all my stuff to black on black or white on white for the text) I have separated out my home folder from the system so that I can easily re-format and re-install the system without losing all my stuff (all 132Gb of it).
This isn’t a problem for most people, but it doesn’t access the hard drive as often as other distros. Why is that a con for me? Well, I have a western digital green hard drive which has an automated parking feature which parks the heads after 10 seconds of inactivity. Well, in windows this doesn’t matter, but in linux since it touches the filesystem every 11-15 seconds or so, that results on a LOT of head parkings. Considering that the heads are only rated for 300K cycles and people have reported reaching that in less than a year, it is a real issue. I have a program (wdantiparkd) which writes the hard drive every 7 seconds while watching to see if anything else has written to the hard drive so that it hangs up after 10 minutes rather than 10 seconds. It helps, but it worked better on Ubuntu.

Overall, this experience with Arch has allowed me to become much more familiar with Linux and its guts and slowely but surely I am getting better at fixing issues. If you are considering a switch from your present operating system and already have experience with Linux (especially the command line since that’s what you are stuck with starting out before you install xfce, Gnome, KDE, etc), I would recommend this distribution. Of course, if you get easily frustrated with problems and don’t enjoy solving them, perhaps a little more stability would be something to look for instead.

Here is my desktop as it stands:

Arch Linux with Gnome 3

A Working Demo of the WebSocketServer

I have finally given in and purchased some hosting at rackspace so I could put the websocketserver somewhere. The demo is live and working, so just visit the following URL to try a very super simple chatroom:

http://cuznersoft.com/python-websocket-server/html/demo_chatroom.html

Projects & Libraries

Musings and Projects from Kevin Cuzner

Lessons in game design…in a car!

Game State Management

Event bubbling

Conclusion

Database Abstraction in Python

Example

Change Tracking Columns

Generation of SQL using logical predicates

Table and row objects

256Mb doesn’t do what it used to…

MMO Asteroids in Node.js…but for real.

Simulation in C++ finally works

Simulink clone…now in C++

Cloning Simulink…in Python

KnockoutJS and Memory Usage

Example

The Problem

Solution

Summary

The first week or two with Arch Linux

A Working Demo of the WebSocketServer