Kevin Cuzner's Personal Blog
Electronics, Embedded Systems, and Software are my breakfast, lunch, and dinner.
A New Blog
For the past several years I've struggled with maintaining the wordpress blog on this site. Starting 3 or 4 years ago the blog began to receive sustained stronger than usual traffic from data centers trying to post comments or just brute-forcing the admin login page. This peaked around 2021 when I finally decided to turn off all commenting functionality. This cooled things down except for requiring me to "kick" the server every few days as it ran out of file handles due to some misconfiguration on my part combined with the spam traffic.
This became annoying and I needed to find something that would be even lower maintenance. To that end, I've written my first functioning blogging software since 2006 or so. There is no dynamic content, all of it is statically rendered, and the blog content itself is stored in GitHub with updates managed using GitHub hooks. The content itself is written with ReStructured Text and I think I've created a pretty easy-to-use and low-maintenance blog platform (famous last words). Ask me in a decade if I was actually successful lol.
In addition, I've found that there are newer, cheaper AWS instances that would suit my needs. I started using this instance over 10 years ago and before I shut it down, I had an uptime of almost 2.5 years. It's truly the end of an era:
A new server
So for the past couple months my server has been going on and off due to the fact that rackspace increased their retirements of swapping and such. I made the swap to Amazon EC2 today and so over the next couple weeks we'll see how this works out.
Extreme Attributed Metadata with Autofac
Introduction
If you are anything like me, you love reflection in any programming language. For the last two years or so I have been writing code for work almost exclusively in C# and have found its reflection system to be a pleasure to use. Its simple, can be fast, and can do so much.
I recently started using Autofac at work to help achieve Inversion of Control within our projects. It has honestly been the most life changing C# library (sorry Autofac, jQuery and Knockout still take the cake for "life-changing in all languages") I have ever used and has changed the way I decompose problems when writing programs.
This article will cover some very interesting features of the Autofac Attributed Metadata module. It is a little lengthy, so I have here what will be covered:
- What is autofac?
- Attributed Metadata: The Basics
- The IMetadataProvider interface
- IMetadataProvider: Making a set of objects
- IMetadataProvider: Hierarchical Metadata
What is Autofac?
This post assumes that the reader is at least passingly familiar with Autofac. However, I will make a short introduction: Autofac allows you to "compose" your program structure by "registering" components and then "resolving" them at runtime. The idea is that you define an interface for some object that does "something" and create one or more classes that implement that interface, each accomplishing the "something" in their own way. Your parent class, which needs to have one of those objects for doing that "something" will ask the Autofac container to "resolve" the interface. Autofac will give back either one of your implementations or an IEnumerable of all of your implementations (depending on how you ask it to resolve). The "killer feature" of Autofac, IMO, is being able to use constructor arguments to recursively resolve the "dependencies" of an object. If you want an implementation of an interface passed into your object when it is resolved, just put the interface in the constructor arguments and when your object is resolved by Autofac, Autofac will resolve that interface for you and pass it in to your constructor. Now, this article isn't meant to introduce Autofac, so I would definitely recommend reading up on the subject.
Attributed Metadata: The Basics
One of my most favorite features has been Attributed Metadata. Autofac allows Metadata to be included with objects when they are resolved. Metadata allows one to specify some static parameters that are associated with a particular implementation of something registered with the container. This Metadata is normally created during registration of the particular class and, without this module, must be done "manually". The Attributed Metadata module allows one to use custom attributes to specify the Metadata for the class rather than needing to specify it when the class is registered. This is an absurdly powerful feature which allows for doing some pretty interesting things.
For my example I will use a "extendible" letter formatting program that adds some text to the content of a "letter". I define the following interface:
1interface ILetterFormatter
2{
3 string FormatLetter(string content);
4}
This interface is for something that can "format" a letter in some way. For starters, I will define two implementations:
1class ImpersonalLetterFormatter : ILetterFormatter
2{
3 public string MakeLetter(string content)
4 {
5 return "To Whom It May Concern:nn" + content;
6 }
7}
8
9class PersonalLetterFormatter : ILetterFormatter
10{
11 public string MakeLetter(string content)
12 {
13 return "Dear Individual,nn" + content;
14 }
15}
Now, here is a simple program that will use these formatters:
1class MainClass
2{
3 public static void Main (string[] args)
4 {
5 var builder = new ContainerBuilder();
6
7 //register all ILetterFormatters in this assembly
8 builder.RegisterAssemblyTypes(typeof(MainClass).Assembly)
9 .Where(c => c.IsAssignableTo<ILetterFormatter>())
10 .AsImplementedInterfaces();
11
12 var container = builder.Build();
13
14 using (var scope = container.BeginLifetimeScope())
15 {
16 //resolve all formatters
17 IEnumerable<ILetterFormatter> formatters = scope.Resolve<IEnumerable<ILetterFormatter>>();
18
19 //What do we do now??? So many formatters...which is which?
20 }
21 }
22}
Ok, so we have ran into a problem: We have a list of formatters, but we don't know which is which. There are a couple different solutions:
- Use the "is" test or do a "soft cast" using the "as" operator to a specific type. This is bad because it requires that the resolver know about the specific implementations of the interface (which is what we are trying to avoid)
- Just choose one based on order. This is bad because the resolution order is just as guaranteed as reflection order in C#...which is not guaranteed at all. We can't be sure they will be resolved in the same order each time.
- Use metadata at registration time and resolve it with metadata. The issue here is that if we used RegisterAssemblyTyps like above, it makes registration difficult. Also, once we get any sizable number of things registered with metadata, it becomes unmanageable IMO.
- Use attributed metadata! Example follows...
We define another class:
1[MetadataAttribute]
2sealed class LetterFormatterAttribute : Attribute
3{
4 public string Name { get; private set; }
5
6 public LetterFormatterAttribute(string name)
7 {
8 this.Name = name;
9 }
10}
Marking it with System.ComponetModel.Composition.MetadataAttributeAttribute (no, that's not a typo) will make the Attributed Metadata module place the public properties of the Attribute into the metadata dictionary that is associated with the class at registration time.
We mark the classes as follows:
1[LetterFormatter("Impersonal")]
2class ImpersonalLetterFormatter : ILetterFormatter
3...
4
5[LetterFormatter("Personal")]
6class PersonalLetterFormatter : ILetterFormatter
7...
And then we change the builder to take into account the metadata by asking it to register the Autofac.Extras.Attributed.AttributedMetadataModule. This will cause the Attributed Metadata extensions to scan all of the registered types (past, present, and future) for MetadataAttribute-marked attributes and use the public properties as metadata:
1var builder = new ContainerBuilder();
2
3builder.RegisterModule<AttributedMetadataModule>();
4
5builder.RegisterAssemblyTypes(typeof(MainClass).Assembly)
6 .Where(c => c.IsAssignableTo<ILetterFormatter>())
7 .AsImplementedInterfaces();
Now, when we resolve the ILetterFormatter classes, we can either use Autofac.Features.Meta<TImplementation> or Autofac.Features.Meta<TImplementation, TMetadata>. I'm a personal fan of the "strong" metadata, or the latter. It causes the metadata dictionary to be "forced" into a class rather than just directly accessing the metadata dictionary. This removes any uncertainty about types and such. So, I will create a class that will hold the metadata when the implementations are resolved:
1class LetterMetadata
2{
3 public string Name { get; set; }
4}
It would worthwhile to note that the individual properties must have a value in the metadata dictionary unless the DefaultValue attribute is applied to the property. For example, if I had an integer property called Foo an exception would be thrown when metadata was resolved since I have no corresponding Foo metadata. However, if I put DefaultValue(6) on the Foo property, no exception would be thrown and Foo would be set to 6.
So, we now have the following inside our using statement that controls our scope in the main method:
1//resolve all formatters
2IEnumerable<Meta<ILetterFormatter, LetterMetadata>> formatters = scope.Resolve<IEnumerable<Meta<ILetterFormatter, LetterMetadata>>>();
3
4//we will ask how the letter should be formatted
5Console.WriteLine("Formatters:");
6foreach (var formatter in formatters)
7{
8 Console.Write("- ");
9 Console.WriteLine(formatter.Metadata.Name);
10}
11
12ILetterFormatter chosen = null;
13while (chosen == null)
14{
15 Console.WriteLine("Choose a formatter:");
16 string name = Console.ReadLine();
17 chosen = formatters.Where(f => f.Metadata.Name == name).Select(f => f.Value).FirstOrDefault();
18
19 if (chosen == null)
20 Console.WriteLine(string.Format("Invalid formatter: {0}", name));
21}
22
23//just for kicks, we say the first argument is our letter, so we format it and output it to the console
24Console.WriteLine(chosen.FormatLetter(args[0]));
The IMetadataProvider Interface
So, in the contrived example above, we were able to identify a class based solely on its metadata rather than doing type checking. What's more, we were able to define the metadata through attributes. However, this is old hat for Autofac. This feature has been around for a while.
When I was at work the other day, I needed to be able to handle putting sets of things into metadata (such as a list of strings). Autofac makes no prohibition on this in its metadata dictionary. The dictionary is of the type IDictionary<string, object>, so it can hold pretty much anything, including arbitrary objects. The problem is that the Attributed Metadata module had no way to do this easily. Attributes can only take certain types as constructor arguments and that seriously places a limit on what sort of things could be put into metadata via attributes easily.
I decided to remedy this and after submitting an idea for autofac via a pull request, having some discussion, changing the exact way to accomplish this goal, and fixing things up, my pull request was merged into autofac which resulted in a new feature: The IMetadataProvider interface. This interface provides a way for metadata attributes to control how exactly they produce metadata. By default, the attribute would just have its properties scanned. However, if the attribute implemented the IMetadataProvider interface, a method will be called to get the metadata dictionary rather than doing the property scan. When an IMetadataProvider is found, the GetMetadata(Type targetType) method will be called with the first argument set to the type that is being registered. This allows the IMetadataProvider the opportunity to know which class it is actually applied to; something normally not possible without explicitly passing the attribute a Type in a constructor argument.
To get an idea of what this would look like, here is a metadata attribute which implements this interface:
1[MetadataAttribute]
2class LetterFormatterAttribute : Attribute, IMetadataProvider
3{
4 public string Name { get; private set; }
5
6 public LetterFormatterAttribute(string name)
7 {
8 this.Name = name;
9 }
10
11 #region IMetadataProvider implementation
12
13 public IDictionary<string, object> GetMetadata(Type targetType)
14 {
15 return new Dictionary<string, object>()
16 {
17 { "Name", this.Name }
18 };
19 }
20
21 #endregion
22}
This metadata doesn't do much more than the original. It actually returns exactly what would be created via property scanning. However, this allows much more flexibility in how MetadataAttributes can provide metadata. They can scan the type for other attributes, create arbitrary objects, and many other fun things that I can't even think of.
IMetadataProvider: Making a set of objects
Perhaps the simplest application of this new IMetadataProvider is having the metadata contain a list of objects. Building on our last example, we saw that the "personal" letter formatter just said "Dear Individual" every time. What if we could change that so that there was some way to pass in some "properties" or "options" provided by the caller of the formatting function? We can do this using an IMetadataProvider. We make the following changes:
1class FormatOptionValue
2{
3 public string Name { get; set; }
4 public object Value { get; set; }
5}
6
7interface IFormatOption
8{
9 string Name { get; }
10 string Description { get; }
11}
12
13interface IFormatOptionProvider
14{
15 IFormatOption GetOption();
16}
17
18interface ILetterFormatter
19{
20 string FormatLetter(string content, IEnumerable<FormatOptionValue> options);
21}
22
23[MetadataAttribute]
24sealed class LetterFormatterAttribute : Attribute, IMetadataProvider
25{
26 public string Name { get; private set; }
27
28 public LetterFormatterAttribute(string name)
29 {
30 this.Name = name;
31 }
32
33 public IDictionary<string, object> GetMetadata(Type targetType)
34 {
35 var options = targetType.GetCustomAttributes(typeof(IFormatOptionProvider), true)
36 .Cast<IFormatOptionProvider>()
37 .Select(p => p.GetOption())
38 .ToList();
39
40 return new Dictionary<string, object>()
41 {
42 { "Name", this.Name },
43 { "Options", options }
44 };
45 }
46}
47
48//note the lack of the [MetadataAttribute] here. We don't want autofac to scan this for properties
49[AttributeUsage(AttributeTargets.Class, AllowMultiple = true)]
50sealed class StringOptionAttribute : Attribute, IFormatOptionProvider
51{
52 public string Name { get; private set; }
53
54 public string Description { get; private set; }
55
56 public StringOptionAttribute(string name, string description)
57 {
58 this.Name = name;
59 this.Description = description;
60 }
61
62 public IFormatOption GetOption()
63 {
64 return new StringOption()
65 {
66 Name = this.Name,
67 Description = this.Description
68 };
69 }
70}
71
72public class StringOption : IFormatOption
73{
74 public string Name { get; set; }
75
76 public string Description { get; set; }
77
78 //note that we could easily define other properties that
79 //do not appear in the interface
80}
81
82class LetterMetadata
83{
84 public string Name { get; set; }
85
86 public IEnumerable<IFormatOption> Options { get; set; }
87}
Ok, so this is just a little bit more complicated. There are two changes to pay attention to: Firstly, the FormatLetter function now takes a list of FormatOptionValues. The second change is what enables the caller of FormatLetter to know which options to pass in. The LetterFormatterAttribute now scans the type in order to construct its metadata dictionary by looking for attributes that describe what options it needs. I feel like the usage of this is best illustrated by decorating our PersonalLetterFormatter for it to have some metadata describing the options that it requires:
1[LetterFormatter("Personal")]
2[StringOption(ToOptionName, "Name of the individual to address the letter to")]
3class PersonalLetterFormatter : ILetterFormatter
4{
5 const string ToOptionName = "To";
6
7 public string FormatLetter(string content, IEnumerable<FormatOptionValue> options)
8 {
9 var toName = options.Where(o => o.Name == ToOptionName).Select(o => o.Value).FirstOrDefault() as string;
10 if (toName == null)
11 throw new ArgumentException("The " + ToOptionName + " string option is required");
12
13 return "Dear " + toName + ",nn" + content;
14 }
15}
When the metadata for the PersonalLetterFormatter is resolved, it will contain an IFormatOption which represents the To option. The resolver can attempt to cast the IFormatOption to a StringOption to find out what type it should pass in using the FormatOptionValue.
This can be extended quite easily for other IFormatOptionProviders and IFormatOption pairs, making for a very extensible way to easily declare metadata describing a set of options attached to a class.
IMetadataProvider: Hierarchical Metadata
The last example showed that the IMetadataProvider could be used to scan the class to provide metadata into a structure containing an IEnumerable of objects. It is a short leap to see that this could be used to create hierarchies of arbitrary objects.
For now, I won't provide a full example of how this could be done, but in the future I plan on having a gist or something showing arbitrary metadata hierarchy creation.
Conclusion
I probably use Metadata more than I should in Autofac. With the addition of the IMetadataProvider I feel like its quite easy to define complex metadata and use it with Autofac's natural constructor injection system. Overall, the usage of metadata & reflection in my programs has made them quite a bit more flexible and extendable and I feel like Autofac and its metadata system complement the built in reflection system of C# quite well.
256Mb doesn't do what it used to...
So, this last week I got an email from rackspace saying that my server was thrashing the hard drives and lowering performance of everyone else on that same machine. In consequence, they had rebooted my server for me.
I made a few mistakes in the setup of this first iteration of my server: I didn't restart it after kernel updates, I ran folding@home on it while running nodejs, and I didn't have backups turned on. I had it running for well over 200 days without a reboot while there had been a dozen or so kernel updates. When they hard rebooted my server, it wouldn't respond at all to pings, ssh, or otherwise. In fact, it behaved like the firewalls were shut (hanging on the "waiting" step rather than saying "connection refused"). I ended up having to go into their handy rescue mode and copy out all the files. I only copied my www directory and the mysql binary table files, but as you can see, I was able to restore the server from those.
This gave me an excellent opportunity to actually set up my server correctly. I no longer have to be root to edit my website files (yay!), I have virtual hosts set up in a fashion that makes sense and actually works, and overall performance seems to be improved. From now on, I will be doing the updates less frequently and when I do I will be rebooting the machine. That should fix the problem with breaking everything if a hard reboot happens.
I do pay for the hosting for this, 1.5 cents per hour per 256Mb of RAM with extra for bandwidth. I only have 256Mb and since I don't make any profit off this server whatsoever at the moment, I plan on keeping it that way for now. Considering that back in the day, 256Mb was a ton of memory, it clearly no longer suffices for running too much on my server (httpd + mysql + nodejs + folding@home = crash and burn).
MMO Asteroids in Node.js...but for real.
Many people saw this april fools joke where the author said that he had created an asteroids MMO using Node.js. In reality, it was completely client side and was a bunch of bots. However, I did find the whole thing rather intriguing and decided to see what I could do with Node.js along this line. Last week I started on the project and this weekend I made enough progress that I can publish v0.0.1. It leaves several things to be desired, including a better game background so that one can tell when the view is shifting and user tracking so that you accumulate your scores over time. It does work and from what I can see its not horribly bad performance. While it certainly won't be able to handle thousands of clients, I expect that it should be able to handle somewhere between 50-100 before it starts dying. At the moment, its quite limited by the memory on my server and the client side scripting gives the impression of "stuttering" with the dead reckoning system used to make the animations smooth. The stuttering is caused by the linear and angular damping that I have running on the server side not being factored into the projected location on the client side.
For physics I am using Box2Dweb in a Node.js module which may be overkill, but its the simplest Javascript physics engine I could find since I didn't feel like writing my own. The server keeps track of all of the entities in the game and each client requests a "view" of an area of the room. The client is informed which player is them, but other than that it just sees all players and entities in the same list. The actual rendering function draws the entities onto the canvas dependent upon the type of the object.
I have made the source available on github here: https://github.com/kcuzner/mmo-asteroids
The MMO itself can be viewed here: http://kevincuzner.com:8080/. Note that at times it may be down since I am messing with my firewall right now and I might accidentally close the port. Just leave me a note in the comments and I'll try to get it working.
A Working Demo of the WebSocketServer
I have finally given in and purchased some hosting at rackspace so I could put the websocketserver somewhere. The demo is live and working, so just visit the following URL to try a very super simple chatroom:
http://cuznersoft.com/python-websocket-server/html/demo_chatroom.html
Multiprocessing with the WebSocketServer
After spending way to much time thinking about exactly how to do it, I have got multiprocessing working with the WebSocketServer.
I have learned some interesting things about multiprocessing with Python:
- Basically no complicated objects can be sent over queues.
- Since pickle is used to send the objects, the object must also be serializable
- Methods (i.e. callbacks) cannot be sent either
- Processes can't have new queues added after the initial creation since they aren't picklable, so when the process is created, it has to be given its queues and that's all the queues its ever going to get
So the new model for operation is as follows:
Sadly, because of the lack of ability to share methods between processes, there is a lot of polling going on here. A lot. The service has to poll its queues to see if any clients or packets have come in, the server has to poll all the client socket queues to see if anything came in from them and all the service queues to see if the services want to send anything to the clients, etc. I guess it was a sacrifice that had to be made to be able to use multiprocessing. The advantage gained now is that services have access to their own interpreter and so they aren't all forced to share the same CPU. I would imagine this would improve performance with more intensive servers, but the bottleneck will still be with the actual sending and receiving to the clients since every client on the server still has to share the same thread.
So now, the plan to proceed is as follows:
- Figure out a way to do a pre-forked server so that client polling can be done by more than one process
- Extend the built in classes a bit to simplify service creation and to make it a bit more intuitive.
As always, the source is available at https://github.com/kcuzner/python-websocket-server
Making a WebSocket server
For the past few weeks I have been experimenting a bit with HTML5 WebSockets. I don't normally focus only on software when building something, but this has been an interesting project and has allowed me to learn a lot more about the nitty gritty of sockets and such. I have created a github repository for it (it's my first time using git and I'm loving it) which is here: https://github.com/kcuzner/python-websocket-server
The server I have runs on a port which is considered a dedicated port for WebSocket-based services. The server is written in python and defines a few base classes for implementing a service. The basic structure is as follows:
Each service has its own thread and inherits from a base class which is a thread plus a queue for accepting new clients. The clients are a socket object returned by socket.accept which are wrapped in a class that allows for communication to the socket via queues. The actual communication to sockets is managed by a separate thread that handles all the encoding and decoding to websocket frames. Since adding a client doesn't produce much overhead, this structure potentially could be expanded very easily to handle many many clients.
A few things I plan on adding to the server eventually are:
- Using processes instead of threads for the services. Due to the global interpreter lock, if this is run using CPython (which is what most people use as far as I know and also what comes installed by default on many systems) all the threads will be locked to use the same CPU since the python interpreter can't be used by more than one thread at once (however, it can for processes). The difficult part of all this is that it is hard to pass actual objects between processes and I have to do some serious re-structuring for the code to work without needing to pass objects (such as sockets) to services.
- Creating a better structure for web services (currently only two base classes are really available) including a generalized database binding that is thread safe so that a service could potentially split into many threads while not overwhelming the database connection.
Currently, the repository includes a demo chatroom service for which I should have the client side application done soon and uploaded. Currently it supports multiple chatrooms and multiple users, but there is no authentication really and there are a few features I would like to add (such as being able to see who is in the chatroom).
New Host!
Thanks to my friend Walter Zarnoch, I now have a hosting plan that will work for the next two years while I am gone on a mission for my church. The plan is to basically keep my site up here while I am gone, hopefully getting hits and such from people interested in the crap on here, and then work on it after I get back. As for the future, I might keep this arrangement, but for all I know I might find out about a better arrangement later. Right now we share a hosting plan, splitting the cost 50/50, so its really not too bad at all costwise for a site I won't really be touching for two years.
Anyway, hopefully this works out...and I am sure glad to be off my grandma's website's host.
Not so temporary...
Well I have decided to place my actual website on this server. My home server has failed due to router issues and I will probably be redesigning the entire website in the near future anyway since it uses outdated layout techniques. This server is basically the smith-marsden.org website that I work on for my dad's side of the family. I plan on downloading the contents of my server to here over the weekend so that I can try to clean things up a bit and hopefully get this site back on the map.
I will probably post progress on the new smith-marsden site here since that is currently my main development challenge.
Additional features of the new smith-marsden site:
- Ability to invite other users given to everyone
- RSS feeds on letter submissions
- More information stored so that it is all in one place that is private rather than facebook or somethwere else which could be quite public
- New state of the art layout techniques enhance the website
- AJAX allows pages to submit forms without reloading, making page load times faster and looks really cool.
- The entire thing is made using CakePHP, a robust framework.
- More to come as I think of them and make them...
Recent posts
A New BlogA good workflow and build system with OpenSCAD and Makefiles
AVR TPI Programming With usbasp For Dummies
Writing reusable USB device descriptors with some XML, Python, and C
The IoTree: An internet-connected tree