While testing the Thousand Parsec single player mode on Gentoo the other day, I came across something strange. The wxPython client, when run from a Git working directory, wants to use development versions of libtpproto-py and libtpclient-py. It does this by prepending '.' to sys.path. For some reason, though, import tp.* was still finding the site-packages versions first, which I had installed from my own overlay ebuild. It turns out that setuptools and Portage conspired to install -nspkg.pth files containing Python code that somehow put the site-packages path for the tp module ahead of everything else in sys.path. It seems doing this at all is an undocumented feature, and the current site module docs don't mention it. My evil temporary fix (in the ebuilds) is to simply delete those files and touch an __init.py__ in the tp directory. It seems to work, but I still plan to investigate the cause (maybe the fact that there are multiple libraries sharing a namespace?) and come up with a less hackish fix if possible.

On the bright side, single player mode seems to work well for Linux! The wizard GUI needs vast improvement, but it's functional. As soon as we merge the singleplayer branches of tpclient-pywx and libtpclient-py, we can release a package, and Gentoo will have ebuilds for everything you need to play single player Risk and RFTS. Before we see this, I am probably going to have to get daneel-ai running on Windows so we can test it there before release. We're also going to need packaging for a number of modules on other Linux distributions.

I'm working on stepping up my Python game in general, checking out Expert Python Programming and O'Reilly's Python for Unix and Linux System Administration. A thousand other people have said it better than I, but this is what software development ought to be.

Thousand Parsec single player mode is almost ready! The initial goal is to release tpclient-pywx with single player mode, along with at least one server and one AI client supporting two rulesets, on Windows and Gentoo Linux. We achieved a few of the final steps last month.

First, we have the release of tpserver-cpp 0.6.0. This release includes the new Risk ruleset as well as the administration protocol, both Google Summer of Code 2008 projects. The Gentoo ebuild for tpserver-cpp now pulls in the recently-released tpadmin-cpp. We're currently working on a Windows package for the server.

Next, we have a preliminary release of daneel-ai, also a GSoC project, which implements an AI client for the Risk and RFTS rulesets in pure Python. The Gentoo ebuild installs a script in the path and the XML file necessary for single player mode, so on that platform we're good to go. We plan to have a more solid release and a package for Windows soon.

We're planning to release a version of the Thousand Parsec tpclient-pywx client soon with the new single player wizard. Support for single player mode is also coming in tpserver-cpp and (hopefully) a couple of the AI clients. The target date for the first candidate is October 19.

I've significantly updated the Gentoo overlay for Thousand Parsec recently, stabilizing most of the release ebuilds and adding ebuilds that grab the latest Git versions. Among other things, this means you can grab the as-yet unreleased tpserver-cpp and tpadmin-cpp combo implementing the new administration protocol!

After about a month's break (vacation to BC, followed by a week buried in books, followed by ICDSC), I'm back to both school and Thousand Parsec development.

For the former, attending ICDSC couldn't have come at a better time. I'm just in the process of starting new research, and some of the topics presented there have given me some good perspective on what's important currently. Several researchers (like this guy) employed stereo camera nodes for various applications, lending some credence to my advocacy for them, and also providing some validation for my M.A.Sc. thesis and paper on calibration. A number of groups presented research employing vision graphs and agent-based stuff, both of which are on our menu currently. Finally, there was some interesting talk on the last day about middleware for DSC networks which piqued my interest.

For the latter, my config-branch (remote administration) changes have been merged in tpserver-cpp and family. I'm now going to improve the implementation of the single player back-end functionality in libtpclient-py. A major concern is the compatibility of the bash scripts and motivation for the component developers to actually create them, so I'm going to try to change it to some versatile Python process management code, and have the custom execution stuff live in the description XML file. Hopefully in the process I'll be able to further centralize the work so that writing the extra XML is reasonably easy while still being able to handle a variety of possibilities. I'll also spend some time beefing up the inline documentation throughout the client library.

This Week:

This week's big accomplishment was completing the wizard. From initially getting it to work to a fully functional wizard only took two (long) days of hacking, thanks to good preparation everywhere else.

I also improved the control scripts and XML definitions. The server and ruleset stuff implemented so far works well.

Big sigh of relief. Looks like I covered everything I set out to. While there are undoubtedly some rough edges, given how the slow progress of early August pushed me almost right up to the deadline, there are no major gaps.

From Here On Out:

Two important things remain. The big one is setting up a fully working test with one server, one ruleset, and one AI client. I will do this early tomorrow and document the entire procedure. While I'm waiting for feedback, I'll spend the remaining time improving in-code documentation.

The big trip to British Columbia starts Monday. I'll still be in communication throughout, unsecured wireless networks permitting. :)

Well, I managed an eleventh-hour completion of the single-player wizard, the last major component of my Google Summer of Code project. From here on out, it's wrapping up little loose ends, spit and polish, documentation, and instructions for some sort of proof-of-concept demo.

The wizard works quite well, assuming you have proper XML definitions and control scripts for your servers, rulesets, and AI clients. And that you have the singleplayer branches of tpclient-pywx and libtpclient-py installed. And my branch of tpserver-cpp, if you want to use that server, which is the only one currently supported. And Risk, if you want to use that ruleset, which is the only one on tpserver-cpp currently supported by any AI clients. Which is particularly difficult, since config and risk are two different branches that have yet to be merged.

The control scripts for GenCon and daneel-ai currently don't work, as I have yet to actually install one of these clients and determine how best to start and stop it. Combined with the above issue, this indicates that setting up a working demo is going to be a Herculean task for me, the developer, to say nothing of those evaluating it.

But at least the wizard works.

This Week:

I wrote the Single Player wiki page early this week, and expanded on it several times since. There is a flowchart showing the conceptual steps to starting a single player game in a wizard context, and an accompanying Python script which actually interfaces with the libtpclient-py backend. There are also sample XML and control script files for tpserver-cpp, the Risk ruleset (as if it were a separately installed module), and the two GSoC AI clients (daneel-ai and GenCon).

I finally managed to get an initial single player wizard screen to pop up from the connection window, after nearly a week of struggling with wxPython and XRC. I added a wxWizard base class generator to generate.py and so far it has worked.

I have mixed feelings about this week's progress. I did improve the SinglePlayer module in libtpclient-py significantly, and I know that starting games works quite well from the text-mode Python script from the wiki page. However, my goal of completing the major functionality of the wizard was far from achieved. The technical challenges of coding the GUI portion were greater than I had anticipated, mainly due to a lack of experience with wxPython and XRC.

Next Week:

Now that I have a functional wizard to work with, there are still three major pieces left to accomplish. The first is to get wxWizardPage subclasses working with XRC, which I still haven't figured out how to do. The second is to actually code the libtpclient-py interface and sequencing for the pages, which is not terribly complex but still requires a significant amount of work. The third is figuring out how to dynamically insert fields for parameters into the server and ruleset option pages, which should be similar to the Order stuff already in tpclient-pywx.

Since this will be the last week of GSoC, I also need to make sure this will actually work, and prepare a demonstration using one of the GSoC AI clients and the Risk ruleset. So far I've had success using the backend directly and manually configuring tpclient-pywx, so I am hopeful to get this going with relatively few hiccups once the wizard is complete.

The Thousand Parsec Single Player wiki page, which I started this week, provides information about implementing single player mode in clients (for now, only those using libtpclient-py) as well as adding support in servers, rulesets, and AI clients. The implementation section provides a text-based Python script that emulates the functionality of a wizard.

I am currently struggling with actually implementing the wizard in tpclient-pywx using the wxWizard module with XRC. The wizard itself isn't difficult to implement directly in code, and there is a solid example of that in the wxPython demo. However, implementing it in the tpclient-pywx context using XRC has me completely stumped. Examples online are scarce, with this one only using wxWizardPageSimple and this discussion not really giving much more information. What is really confusing me is the way XRC is being used in tpclient-pywx; there are a bunch of auto-generated .py files I can't make heads or tails of, and I can't figure out how I should be importing my wizard's XRC. And then there's the matter of making dynamic controls appear on some pages (parameters), which I will need to tackle once I get any sort of wizard at all.

This lack of progress is really stressing me out, since it's the last of many parts of my project, and I only have a few days more to wrap it up.

This Week:

Two of three major components of single-player mode have been tackled as planned this week.

The first was specifying the XML and control script formats each server, ruleset module, and AI client should use. I will document it in detail on the wiki soon, but for now, take a look at the most current mock-up I've been using for testing and demonstration.

The second was developing the SinglePlayer module of libtpclient-py (in the singleplayer branch). This builds server, ruleset, and AI client information lists from the aforementioned XML files, presents some useful higher-level functions (more to come, as I develop the GUI part), and starts and stops the external programs using the control scripts with some intelligence.

I learned something new this week. While my C++ coding style lends itself well to making nice atomic commits with few (if any) bugs, evidently my Python coding style lends itself to being sloppy and scatterbrained. Since Python is basically pseudocode with indentation, I seem to eschew the methodical approach I take to more strongly structured languages and go nuts adding, moving, renaming, and removing code until eventually it coalesces into something reasonable. Revision control really shows this: for some fun examples, like when I pushed a commit containing value = value = ..., see previous deltas on SinglePlayer.py.

Overall, I'd say I was productive enough this week. Getting a good idea of what else remains to be done was arguably the most useful aspect.

Next Week:

I hope to make next week the big one for single-player functionality. The final major component, viz. the GUI wizard in tpclient-pywx, should be more or less complete by this time next week. Before I start, however, I also need to document the XML and control script formats, and see about having them installed with various server/ruleset/AI components. There are also various other loose ends to wrap up, and I'll take advantage of any extra time I have to work on those.

In the end, I hope for all of the major portions of my GSoC proposal to be completed next week, so that I can spend the final week cleaning up, documenting, and testing.

Yesterday's thoughts on implementing the single-player wizard in libtpclient-py.

I'm planning to use Python's xml.sax module to parse multiple server/ruleset XML descriptions into a big list-and-dictionary structure. The structure for, say, tpserver-cpp-0.5.1.xml would look something like this (though the final product would, in general, consist of the data from multiple such files):

[ { 'name' : 'tpserver-cpp-0.5.1', 'longname' : 'tpserver-cpp', 'version' : '0.5.1', 'description' : 'Thousand Parsec server in C++.', 'parameters' : [ { 'name' : 'turnlength', 'type' : 'I', 'longname' : 'Turn Length', 'description' : 'Maximum length of a turn (0 for unlimited).', 'default' : 0, 'commandstring' : '--turn_length_over_threshold %d' } ], 'rulesets' : [ { 'name' : 'minisec', 'longname' : 'MiniSec', 'version' : '0.3', 'description' : '', 'parameters' : [] }, { 'name' : 'mtsec', 'longname' : 'MTSec', 'version' : '0.0', 'description' : '', 'parameters' : [] }, { 'name' : 'risk', 'longname' : 'Risk', 'version' : '0.1', 'description' : 'A modified Risk ruleset for tpserver-cpp.', 'parameters' : [ { 'name' : 'map', 'type': 'S', 'longname' : 'Map', 'description' : 'Map file to load.', 'default' : '', 'commandstring' : '--risk_map %s' } ] } ] } ]

Phew.

The other functions would pull data from this beast as necessary to build the wizard form and actually run the server with options. The AI client stuff, of course, would be mostly similar.

I've started mocking up what the XML descriptions and control scripts for Thousand Parsec single-player mode would look like in a /usr/share/tp context. The idea is for the servers, rulesets, and AI clients to provide their own descriptions of what they are and what they need, as well as a common interface that clients can start them with. This way, none of the components are bound to each other's specifics. The design is admittedly very UNIX-centric currently. Feedback, as always, is appreciated.

The client itself (possibly via a client library) populates the single-player wizard based on the XML parameters. Select a server in the Server dropdown box, the Ruleset dropdown box is populated with that server's rulesets; select a ruleset, and the AI Client box is populated with compatible AI clients. A few other standard parameters exist (port, number of AI clients) along with additional parameters brought in from the XML files. The control scripts take care of actually passing this stuff as arguments to the server and AI clients. Interestingly, this entire process doesn't appear to require using the administration protocol at all.

However, I have been banging around this idea of saving and loading games (for post-GSoC implementation), which would definitely need an admin connection to work, along with cooperation from stateful AI clients.

Another relevant post-GSoC idea is adding UNIX IPC socket connections to tpserver-cpp and the protocol libraries, both for player and admin connections. This would be used by the game clients for single-player mode and by the admin clients for local administration. The server XML file and single-player stuff would need to be generalized to specify a connection type rather than defaulting to standard TCP.

This Week:

I completed the libtpproto-py administration components. This turned out to be less ambitious than I anticipated, involving mainly creating an admin connection object, the various command frame objects, and a log message frame object. Everything seems to work so far.

I feel I could have accomplished more this week, but at least I accomplished what I planned. Jumping into a different language and a new set of repositories was bound to be a second slow start in the middle of GSoC for me, but I think I've built up enough momentum now to get the rest of it done in time.

Next Week:

By early next week I hope to have solid plans for two major pieces of the project: a system whereby clients can detect installed servers, rulesets, and AI clients, and an automated administration client for single-player games. I've emailed the developer list with my thoughts on single-player mode implementation and hope to get a broad spectrum of feedback.

Unless the responses effect a paradigm shift on my current plans, I'll try to code the automated administration client (probably in libtpclient-py) next week. This might involve additional work on libtpproto-py as well.

This Week:

I spent some time attempting to fit a couple other students' midterm reports to the layout I designed. I never intended to reuse it, so it consists of a bunch of manually placed text boxes and images. Along with OpenOffice Draw apparently being the black sheep of the family in terms of bugginess, my spirit is beaten, and I give up. I'm going to try to redo the reports in a less masochistic way.

On the bright side, there is now a config branch of libtpproto-py. I have begun adding administration frame functionality to it locally, but I haven't made much progress yet.

I didn't accomplish as much as I'd wanted to this week, with a number of other commitments interfering with my time. Next week, I hope to get back to coding so I can wrap the second part of my project up in this last month of GSoC.

Next Week:

I hope to complete the libtpproto-py administration extension by the end of next week. It seems ambitious, but I need to get back on track. I'm hoping that my experience implementing it in libtpproto-cpp, along with my more recent experience with Python development, will make this go smoothly.

This Week:

I added some small but important bits of functionality to tpadmin-cpp. Notably, libtpproto-cpp now handles log message frames properly, and tpadmin-cpp detects server-end disconnection and cleans up the command list accordingly.

The rest of the week was spent adding comments to the code wherever they seemed particularly important, and working on the midterm report.

Next Week:

It seems I'll be adapting the other students' midterm reports to the Thousand Parsec theme I designed for mine. Unfortunately, it's all manually laid out, so I set myself up for a relatively large amount of work here.

On the coding front, the most logical place to start with the second half of my project -- namely, adding the single-player wizard to tpclient-pywx -- is to duplicate the work I did on libtpproto-cpp in libtpproto-py, adding the administration protocol functionality. I'll see about getting a branch set up for that.

This Week:

First, with llnz's help, I fixed the connection issue in tpadmin-cpp, which had to do with the way libtprl was being used in the EventLoop context.

A ServerCommand class was added to tpadmin-cpp to handle the client end of commands within libtprl. It fills parameters from the command line string, then builds and sends command frames to the server (which also required two additions to the command functionality in libtpproto-cpp).

Once all the major functionality was completed, several bugs were fixed, and some small changes were made to improve the usability and appearance of tpadmin-cpp (such as the debug command which toggles debug messages).

This was a reasonably productive week. Although much time was spent fixing bugs, none proved particularly stubborn. The important outcome is that the administration protocol and basic client are essentially ready for release, though as mentioned in a previous post, there are potentially still some important features to be added.

Next Week:

I intend to complete testing on tpadmin-cpp with the modified tpserver-cpp, to ensure that everything works smoothly before it's merged. At the same time, I'll be poking around for other potential features (if you'd like to see something in particular, let me know).

I'll also start gearing up for the second half of GSoC, where I'll be working on the single-player mode for tpserver-pywx. Aside from the automated server configuration, this involves finding and starting AI clients suitable for the ruleset in use. I'll need to cooperate with the AI client developers to determine how best to integrate them with single-player mode in a common way.

I think the first half of my GSoC project, comprising the administration protocol, the modifications to tpserver-cpp and libtpproto-cpp to support it, and the standalone tpadmin-cpp administration client, is ready for prime time, or at least a half-hour spot at 3:00 AM on the public access channel.

I'll still be hammering it for bugs over the next day or two, but fingers crossed, it looks good right now. There are still a few gaps in functionality that will probably be added at some point; notably, the local command set is only updated at login currently, but eventually the client may want to watch for changes in the server-side list and update.

If you'd like to try it out, compile the config branches of tpserver-cpp and libtpproto-cpp, as well as tpadmin-cpp. Note that this version of tpserver-cpp runs as a daemon, but you can disable that behavior with the -d switch. Issuing open localhost and login admin admin to tpadmin-cpp should give you access to a set of commands on the server (type help to see them all).

I'll greatly appreciate any feedback, especially suggestions and bug reports.

Well, tpadmin-cpp almost works properly now.

The connection problem was due to the console input loop monopolizing the single-threaded event loop, so that the sockets never got checked for data. On Lee's suggestion, this was fixed by making the console a TPSocket that would be scheduled by the event loop.

Another pair of problems in libtpproto-cpp were giving me trouble. The first was an incorrect delete statement that was causing segfaults, the other an incorrect packInt call before packing a string command parameter (I have no idea why I put that in there originally).

Now I just need to figure out why I get a SIGILL every time I try to use the same server command twice...

This Week:

Early in the week, I added an AdminLayer interface to libtpproto-cpp, and a series of other commits completed the functionality over a two-day period. Notably, I created a CommandDescCache for command descriptions. More recently, I added an AdminStatusListener class, so that clients can act when the AdminLayer status changes.

The connect and login functionality from AdminLayer was implemented in tpadmin-cpp shortly after AdminLayer was created, and I later made it more robust. I also added a logger and event loop to the client, the latter of which required some design changes in the client to be used properly. I also added a ClientASL class, to implement the AdminStatusListener, which will be used, among other things, to initially populate the command list on login.

My progress this week was above average in the first couple of days, but slowed to a crawl for most of the week due to a problem with tpadmin-cpp not completing the connection with the server. This means that all did not go well, so my optional goal from last week of starting the command list population was largely unattained. I have as yet been unable to debug the connection issue, but hopefully with Lee's help I will be able to move on very soon.

Next Week:

The first thing to be done is to debug the connection issue in tpadmin-cpp.

Once that is out of the way, I can finish the command list population, which will require much work on tpadmin-cpp and probably some work on libtpproto-cpp. Assuming there are no complications I haven't foreseen, I should be able to get tpadmin-cpp sending commands by the end of next week, completing all major functionality of the remote administration component in the server, protocol library, and client in time for the midterm evaluation.

This Week:

Early in the week I finished adding the various administration frames to libtpproto-cpp in the new config branch. This provides the basic frame packing and unpacking functionality. I have deferred implementation of the CommandUpdate frame since I'm not entirely sure it's necessary.

Next, I started the tpadmin-cpp utility. The basic console functionality is there, but currently it doesn't connect to the server. When I got to this point, I realized that clients using libtpproto-cpp would connect through a high-level interface in that library called GameLayer, rather than implementing their own code for this based on lower-level stuff. So, tpadmin-cpp is now waiting for a similar interface tailored to admin clients.

Overall, I am satisfied with this week's progress. Although I missed a tentative milestone to have tpadmin-cpp connecting to the server (today), I more than made up for it with laying the foundations for other functionality that will put me well ahead by this time next week. My most important accomplishment this week has been mentally tying the ends together; I'm now fully convinced that I can complete the first half of this project on time, because I now know fairly precisely what needs to be done for every remaining piece.

Next Week:

I will revisit libtpproto-cpp and add an AdminLayer interface. This will take care of the connection, authentication, and communications, similar to GameLayer. Pending further investigation into how the existing GameLayer works, I think this should also take care of the command list.

Once the AdminLayer is complete, I will make tpadmin-cpp use it to connect to and authenticate with the server. If all goes well, I should still have time left in the week to add the necessary functionality to get commands from the AdminLayer list into the console list (which is already partially implemented).

On the libtpproto-cpp front, I think I may just be finished, in the programmer's sense of the word (seems finished, but will most likely need plenty more work once I actually try to use it). I added CommandTypesList, Command, and CommandResult frames. Conspicuously missing is the CommandUpdate frame, which I am not really sure I need. No big deal to go back and implement there and in tpserver-cpp if I do.

I've started working on a libtprl-based console, tpadmin-cpp, which will be the CLI server administration client. So far, it doesn't do much. The neat thing is that it shouldn't be too hard to add commands based on the CommandDescriptions from the server; the action() method will just parse the command and send the appropriate Command frame to the server.

Our discussions about whether to include it with tpserver-cpp (in the same repository) have concluded; the decision is that tpadmin-cpp will be its own repository, so the client and server will be available separately but optionally bundled together, and tpserver-cpp will come with an Apache-like control utility that will support a few minimal operations (forcing a config file re-read, generating a turn, etc.).