moved
Ok. I will no longer be posting on livejournal. I have moved my blog to http://thetofu.com
Thanks lj!
Thanks lj!
switching blogs
I have been on livejournal for a while now and I feel it is time to part ways. Where do I go though? I would like to move it back to thetofu.com. I have been looking at what or where to switch and I have not found what I want. So I will look into adapting something or creating something that I can blog with. The following is a list of tentative requirements :
1. The html to be static files.
2. The admin and posting be done via XMPP and Pubsub
3. Not require a relational database. I would use something like couchdb if I have to.
4. Be able to post and manage via an XMPP client and the web.
That is it for now. I will document my progress as I continue to explore. Also, I will give more reasoning for my requirements. :)
1. The html to be static files.
2. The admin and posting be done via XMPP and Pubsub
3. Not require a relational database. I would use something like couchdb if I have to.
4. Be able to post and manage via an XMPP client and the web.
That is it for now. I will document my progress as I continue to explore. Also, I will give more reasoning for my requirements. :)
Birthdays
XMPP or Jabber's 10th Anniversary is coming up. I like the idea of an open day, but anything would be cool. Just a simple gathering in a MUC room would be good, but physical parties would be better! I wish there were more users/advocates of XMPP in Charleston. There is more discussion on this, in the forum, at the following url:
http://jabberforum.org/forumdisplay.php?s=cdf04cdb8cd15917d23e473ab4b60c3d&f=180
Also, ejabberd's 6th birthday is here and badlop made a cool video, check it out.
It is nice to see great projects grow! I am also very excited about the future of XMPP!
http://jabberforum.org/forumdisplay.php?s=cdf04cdb8cd15917d23e473ab4b60c3d&f=180
Also, ejabberd's 6th birthday is here and badlop made a cool video, check it out.
It is nice to see great projects grow! I am also very excited about the future of XMPP!
Palaver's present and future
Today, I made a new release of Palaver. It is an XMPP MUC component, created in python and twisted. The newest version is 0.6. This version has many bug fixes and speed improvements.
New features are memcache support for pgsql backend, and a memcache backend. The memcache backend is a more robust version of the in memory storage. All data is stored in memcached. This allows for palaver restarts without losing all of your data. It is not not as persistent as the other storage mechanisms but if you want a quick and fast MUC then it works, and is a bit better than the in memory dict storage.
I have also been adding MUC support to Wokkel and would like to use that protocol implementation for palaver.
The next release will be dependent on wokkel. So, palaver will then become just the storage mechanism code. Eventually, the wokkel code may make it to twisted words. I think this staging method will allow for quick use and testing. So, by the time it reaches twisted words it will have been used and tested well, beyond the required unit tests.
I am excited about the future of XMPP and Twisted words. The flexibility of twisted is gonna allow for a lot of cool components and servers!
New features are memcache support for pgsql backend, and a memcache backend. The memcache backend is a more robust version of the in memory storage. All data is stored in memcached. This allows for palaver restarts without losing all of your data. It is not not as persistent as the other storage mechanisms but if you want a quick and fast MUC then it works, and is a bit better than the in memory dict storage.
I have also been adding MUC support to Wokkel and would like to use that protocol implementation for palaver.
The next release will be dependent on wokkel. So, palaver will then become just the storage mechanism code. Eventually, the wokkel code may make it to twisted words. I think this staging method will allow for quick use and testing. So, by the time it reaches twisted words it will have been used and tested well, beyond the required unit tests.
I am excited about the future of XMPP and Twisted words. The flexibility of twisted is gonna allow for a lot of cool components and servers!
Twisted Wokkel Bot
A bot is not a new thing, it has been done in IRC for a long time. In IRC, a bot is a program that listens to messages in a channel and responds to them.
Bots are also being used in XMPP and particularly Multi-User Chat. You can see a weather bot in action at the following url :
http://speeqe.com/room/zzchschat@muc.appriss.com/
Using XMPP you can extend a bot to respond to messages that are not direct chats. XMPP is xml so you can send and respond to xml messages. This makes XMPP better than IRC.
At Chesspark bots are used extensively. There is one in particular used for a King of the Hill game. It combines MUC and PubSub XMPP extensions to achieve a fun game, where you knock your opponent off the hill by winning chess games.
There are many ways to create XMPP bots, but I have currently been working on MUC support for Wokkel. From Wokkel's web page : "Wokkel is collection of enhancements on top of the Twisted networking framework, written in Python. It mostly provides a testing ground for enhancements to the Jabber/XMPP protocol implementation as found in Twisted Words, that are meant to eventually move there. " Wokkel is nice for implementing and staging things before they can go into twisted.
Using the MUC client branch, I will show you how you can create a Jabber MUC bot. We have to use the branch because it is still under development and not in trunk yet. Hopefully it will be soon. Note that everything is not fully implemented but there is enough to make a bot.
This bot will be simple and just keep track of users coming in and out of a room. You will be able to see user activity using a last command.
First we check out the developement branch. Then we install it.
After this we will need to create the python class. Will will put it in the client.py file and call it MUCLastBot.
All this does it import the python modules we will be using and extend the muc.MUCClient class.
Next, we need to initialize and add 'last' support and methods to the bot.
This initializes the class with attributes we need and sets up methods to support user activity.
After this, we need to have the bot join a room. This is done with the 'initialized' method in MUCClient. This method is called after the XMPP client authenticates and its observers are initialized.
This code will make the bot join the room after start up and then will configure room if the status shows that the room was just created. It uses the 'join' method to join the room and when the bot joins the room 'initRoom' will be called.
You will notice the decorator 'inlineCallbacks', you can ignore this. This decorator allows the method to call deferred methods and use yield to get the call back result.
You can go to http://www.twistedmatrix.com to learn more. For our purposes we will just ignore it. :)
Next, we need to handle when someone joins and parts the room.
All we do is mark their last activity.
We then need to handle commands and logging messages sent by users.
This method takes a groupchat message and processes it to see if there is a bot command. It also logs last message activity for the user.
So, the command we want to use is 'last', we need that method inorder to complete the bot.
The last command checks the activity for the user given, or if no user is given it will then show the last active user. It will also tell you it does not know the user if it has never seen them in the room.
The '_sendLast' method sends that information back to the room as a groupchat.
The bot is now complete, we will need a conf file to start it up. To do this I will use a .tac file for twisted.
Simply use twistd to run the bot.
That is it, have fun and look forward to more XMPP support in wokkel and twisted!
You can download muc.tac and client.py at the following url:
http://people.chesspark.com/~tofu/wokkel/lastbot/
Bots are also being used in XMPP and particularly Multi-User Chat. You can see a weather bot in action at the following url :
http://speeqe.com/room/zzchschat@muc.appriss.com/
Using XMPP you can extend a bot to respond to messages that are not direct chats. XMPP is xml so you can send and respond to xml messages. This makes XMPP better than IRC.
At Chesspark bots are used extensively. There is one in particular used for a King of the Hill game. It combines MUC and PubSub XMPP extensions to achieve a fun game, where you knock your opponent off the hill by winning chess games.
There are many ways to create XMPP bots, but I have currently been working on MUC support for Wokkel. From Wokkel's web page : "Wokkel is collection of enhancements on top of the Twisted networking framework, written in Python. It mostly provides a testing ground for enhancements to the Jabber/XMPP protocol implementation as found in Twisted Words, that are meant to eventually move there. " Wokkel is nice for implementing and staging things before they can go into twisted.
Using the MUC client branch, I will show you how you can create a Jabber MUC bot. We have to use the branch because it is still under development and not in trunk yet. Hopefully it will be soon. Note that everything is not fully implemented but there is enough to make a bot.
This bot will be simple and just keep track of users coming in and out of a room. You will be able to see user activity using a last command.
First we check out the developement branch. Then we install it.
svn co https://svn.ik.nu/wokkel/branches/wokkel-muc-client-support-24 wokkel python setup.py install
After this we will need to create the python class. Will will put it in the client.py file and call it MUCLastBot.
""" A simple last bot wokkel example """ from twisted.internet import defer import datetime from twisted.words.protocols.jabber import jid from wokkel import muc class MUCLastBot(muc.MUCClient): """ """
All this does it import the python modules we will be using and extend the muc.MUCClient class.
Next, we need to initialize and add 'last' support and methods to the bot.
def __init__(self, server, room, nick):
muc.MUCClient.__init__(self)
self.server = server
self.room = room
self.nick = nick
self.room_jid = jid.internJID(self.room+'@'+self.server+'/'+self.nick)
self.last = {}
self.activity = None
def _getLast(self, nick):
return self.last.get(nick.lower())
def _setLast(self, user):
user.last = datetime.datetime.now()
self.last[user.nick.lower()] = user
self.activity = user
This initializes the class with attributes we need and sets up methods to support user activity.
After this, we need to have the bot join a room. This is done with the 'initialized' method in MUCClient. This method is called after the XMPP client authenticates and its observers are initialized.
def initialized(self):
"""The bot has connected to the xmpp server, now try to join the room.
"""
self.join(self.server, self.room, self.nick).addCallback(self.initRoom)
@defer.inlineCallbacks
def initRoom(self, room):
"""Configure the room if we just created it.
"""
if int(room.status) == muc.STATUS_CODE_CREATED:
config_form = yield self.getConfigureForm(self.room_jid.userhost())
# set config default
config_result = yield self.configure(self.room_jid.userhost())
This code will make the bot join the room after start up and then will configure room if the status shows that the room was just created. It uses the 'join' method to join the room and when the bot joins the room 'initRoom' will be called.
You will notice the decorator 'inlineCallbacks', you can ignore this. This decorator allows the method to call deferred methods and use yield to get the call back result.
You can go to http://www.twistedmatrix.com to learn more. For our purposes we will just ignore it. :)
Next, we need to handle when someone joins and parts the room.
def userJoinedRoom(self, room, user):
"""If a user joined a room, make sure they are in the last dict
"""
self._setLast(user)
def userLeftRoom(self, room, user):
self._setLast(user)
All we do is mark their last activity.
We then need to handle commands and logging messages sent by users.
def receivedGroupChat(self, room, user, body):
# check if this message addresses the bot
cmd = None
user_nick = None
try:
cmd, user_nick = body.split(" ")
except ValueError:
# value error means it was a one word body
cmd = body
cmd = cmd.replace("!", "")
method = getattr(self, 'cmd_'+cmd, None)
if method:
method(room, user_nick)
# log last message
user.last_message = body
self._setLast(user)
This method takes a groupchat message and processes it to see if there is a bot command. It also logs last message activity for the user.
So, the command we want to use is 'last', we need that method inorder to complete the bot.
def cmd_last(self, room, user_nick):
"""
"""
if user_nick is None:
# show last person to do something
self._sendLast(room, self.activity)
else:
u = self._getLast(user_nick)
if u:
self._sendLast(room, u)
else:
self.groupChat(self.room_jid, 'Sorry %s, That person is unknown to me.' % (user.nick,))
def _sendLast(self, room, user):
""" Grab last information from user and room and send it to the room.
"""
last_message = getattr(user,'last_message', '')
last_stamp = getattr(user,'last', '')
if room.inRoster(user):
message = """%s is in this room and said '%s' at %s.""" % (user.nick, last_message, last_stamp)
else:
message = """%s left this room at %s and last said '%s'.""" % (user.nick, last_stamp, last_message)
self.groupChat(self.room_jid, message)
The last command checks the activity for the user given, or if no user is given it will then show the last active user. It will also tell you it does not know the user if it has never seen them in the room.
The '_sendLast' method sends that information back to the room as a groupchat.
The bot is now complete, we will need a conf file to start it up. To do this I will use a .tac file for twisted.
from twisted.application import service
from twisted.words.protocols.jabber import jid
from wokkel.client import XMPPClient
from client import MUCLastBot
application = service.Application("lastbot")
xmppclient = XMPPClient(jid.internJID("test@thetofu.com/lastbot"), "test")
xmppclient.logTraffic = True
mucbot = MUCLastBot('chat.speeqe.com','last', 'LastBot')
mucbot.setHandlerParent(xmppclient)
xmppclient.setServiceParent(application)
Simply use twistd to run the bot.
twisted -y muc.tac
That is it, have fun and look forward to more XMPP support in wokkel and twisted!
You can download muc.tac and client.py at the following url:
http://people.chesspark.com/~tofu/wokkel/lastbot/
Speeqe about it.
One thing I do everyday is group chat via XMPP. There are lots of ways to do this, via many xmpp clients. If I am on my laptop or computer then having a client works out well. I can chat to my hearts content.
Often, a problem that comes about, is if I am away, and I need to communicate via a chat room, what do I do? I could use a remote computer and a command line XMPP client or I can use Speeqe.
Speeqe is a web based chat creator, you can create and theme your own chat rooms! There are many ways to use Speeqe for various different things, but the solution to my problem, I find very useful, is that you can connect to other chat rooms on the federated XMPP network.
To do this you can use the url trick, for example:
Jabber Development Room
http://speeqe.com/room/jdev@conference.jabber.org/
Charleston Linux Users Group
http://speeqe.com/room/csclug@conference.butterfat.net/
You can just start chatting anonymously or login via a XMPP account.
This trick, and the other things you can do with Speeqe, are cool, but best of all it does not end there. Speeqe is open source!
You can use and add features to Speeqe and the possibilities become endless. :)
The neat thing is it already uses open source software. It uses Strophe, Punjab, Palaver, Apache, and Django.
It can be set up to use any XMPP server and any Multi-User chat implementation. It is very flexible.
More examples of setting up and using Speeqe are sure to come!
Look forward to the official release of Speeqe!
Often, a problem that comes about, is if I am away, and I need to communicate via a chat room, what do I do? I could use a remote computer and a command line XMPP client or I can use Speeqe.
Speeqe is a web based chat creator, you can create and theme your own chat rooms! There are many ways to use Speeqe for various different things, but the solution to my problem, I find very useful, is that you can connect to other chat rooms on the federated XMPP network.
To do this you can use the url trick, for example:
Jabber Development Room
http://speeqe.com/room/jdev@conference.jabber.org/
Charleston Linux Users Group
http://speeqe.com/room/csclug@conference.butterfat.net/
You can just start chatting anonymously or login via a XMPP account.
This trick, and the other things you can do with Speeqe, are cool, but best of all it does not end there. Speeqe is open source!
You can use and add features to Speeqe and the possibilities become endless. :)
The neat thing is it already uses open source software. It uses Strophe, Punjab, Palaver, Apache, and Django.
It can be set up to use any XMPP server and any Multi-User chat implementation. It is very flexible.
More examples of setting up and using Speeqe are sure to come!
Look forward to the official release of Speeqe!
Punjab, the BOSH of Choice.
It has been a long while since the last Punjab release. But I am happy to announce that Punjab is now at version 0.13. This release has many changes.
First, there was a massive code clean up and removal. XMLRPC, SOAP, and HTTP Polling are no longer supported and have been removed. Also, versions of Twisted older than 2.5 are no longer supported. Similar, versions of Twisted Words older than 0.5 are not supported.
Second, speed increases have been made to the handling of xml and requests. There have been code improvements and protocol support fixes. There are protocol and unit tests too.
Also, Version 1.0 is close and should be out very soon.
Finally, in order to answer 'Which BOSH server do you need?'. Look for examples and demos of strophe and other software and technology using Punjab! These examples will be on here and the Punjab home page.
First, there was a massive code clean up and removal. XMLRPC, SOAP, and HTTP Polling are no longer supported and have been removed. Also, versions of Twisted older than 2.5 are no longer supported. Similar, versions of Twisted Words older than 0.5 are not supported.
Second, speed increases have been made to the handling of xml and requests. There have been code improvements and protocol support fixes. There are protocol and unit tests too.
Also, Version 1.0 is close and should be out very soon.
Finally, in order to answer 'Which BOSH server do you need?'. Look for examples and demos of strophe and other software and technology using Punjab! These examples will be on here and the Punjab home page.
Baseball and Chess
In sports, or any competition, we always want to know who is the best. Also, we want to know who has the best chance to win the game. Some of us always want our favourite player or team to win. But knowing their chances may help with the expectations that may bring heartache or joy or a wager we may have with a friend.
There are many ways, statisticians and mathematicians have come up with, to determine outcomes and strengths of players or teams. One way, in Chess, is finding a players rating by a rating system. At Chesspark, we use a variation of the Glicko rating system. I have always wondered if this rating system would work well with team sports. So, being that I love baseball, I did some research (google search) on baseball and chess ratings. What follows, is a my first experiments in using glicko atings to determine game outcomes and finding the strength of a team.
First, I could not find any information on baseball and Glicko. I did find some on ELO and baseball. There is a site that does ELO ratings on soccer (football) teams as well. ELO and Glicko are rating systems used in Chess or other games involving two players. Team sports do not necessarily use these methods to determine the outcome of the next game. Baseball has a well known method called the Pythagorean expectation. It uses what the teams have done in previous games to determine the outcome. It uses runs scored and runs allowed. You can use this equation in other sports too. I would like to compare ratings versus the Pythagorean expectation and also investigate combining the two.I gathered data from http://www.retrosheet.org. Wrote a python script to parse the data, and compute the ratings. I will release code in upcoming blog posts. Using the script and data we can calculate the final ratings of teams at the end of the season. The results are interesting.
Glicko Rating results for the Major League Baseball 2007 Season:
NOTE: This is a great example on how close in skill professional baseball teams are.
Now, lets take a game from the 2007 season and determine the outcome via glicko and the same percentage via the Pythagorean expectation. I will use Atlanta Braves games since they are my favourite team. Lets take the 42nd game Atlanta played. This game was versus the Boston Red Sox who was a high rated team all year.
NOTE: This determination is very simplistic and missing some steps, a more in-depth example may follow in other blog posts.
Boston had a record of 29 - 12 (Winning percentage 70.7%)
Atlanta had a record of 25 - 17 (WP 59.52%)
2007-05-19 ATL 1529.0 vs. BOS 1675.0
2007-05-19 ATL 52.562% vs. BOS 72.554%
By rating, Boston should win, they are the strongest. The chance for Atlanta to win is about 47.7%. The outcome of the game was Boston 13 and Atlanta 3. Boston crushed them. Before this, Atlanta split a series with Washington, a very weak team.
By winning percentage, we see that Boston's real percentage is lower than what their percentage should be and we see that Atlanta's real percentage is higher than the determined one. Using that we see that Boston is favoured to win unless luck comes into play.
Both methods show Boston being the favourite, but 10 runs seems too much. :)
Determining the actual chance that Boston will win is a bit more complicated than the above example, with both methods. It is easy to see that using each way shows that Boston is stronger and should win the game, and they did. You can also use other statistical methods to determine how many runs both teams could score and use that to help with determining the outcome. I want to find if I could use ratings to help or replace some of these.I believe ratings is a great indicator of how strong the team is right now (or at the moment) and is a simpler way when looking at strength of schedule and other factors in finding out who will win a series or finish first in the standings.
In posts to follow, I hope to start predicting the outcome of current games, and developing a system to make this easier for me and others. I plan on using XMPP, Pubsub and BOSH. I also want to investigate finding a pitcher or batter's rating using glicko. After that, who knows.
The baseball information used here was obtained free of charge from and is copyrighted by Retrosheet. Interested parties may contact Retrosheet at "www.retrosheet.org".
There are many ways, statisticians and mathematicians have come up with, to determine outcomes and strengths of players or teams. One way, in Chess, is finding a players rating by a rating system. At Chesspark, we use a variation of the Glicko rating system. I have always wondered if this rating system would work well with team sports. So, being that I love baseball, I did some research (google search) on baseball and chess ratings. What follows, is a my first experiments in using glicko atings to determine game outcomes and finding the strength of a team.
First, I could not find any information on baseball and Glicko. I did find some on ELO and baseball. There is a site that does ELO ratings on soccer (football) teams as well. ELO and Glicko are rating systems used in Chess or other games involving two players. Team sports do not necessarily use these methods to determine the outcome of the next game. Baseball has a well known method called the Pythagorean expectation. It uses what the teams have done in previous games to determine the outcome. It uses runs scored and runs allowed. You can use this equation in other sports too. I would like to compare ratings versus the Pythagorean expectation and also investigate combining the two.I gathered data from http://www.retrosheet.org. Wrote a python script to parse the data, and compute the ratings. I will release code in upcoming blog posts. Using the script and data we can calculate the final ratings of teams at the end of the season. The results are interesting.
Glicko Rating results for the Major League Baseball 2007 Season:
====================== teams by rating ========================== team rating rd total wins - losses New York Yankees 1580.0 28.465171 162 94 - 68 Boston Red Sox 1571.0 29.260524 162 96 - 66 Cleveland Indians 1564.0 28.649712 162 96 - 66 Los Angeles Angels of Anaheim 1562.0 28.888047 162 94 - 68 Arizona Diamondbacks 1537.0 28.440955 162 90 - 72 Colorado Rockies 1534.0 28.165939 163 90 - 73 Detroit Tigers 1533.0 28.574597 162 88 - 74 Seattle Mariners 1532.0 28.210795 162 88 - 74 Toronto Blue Jays 1529.0 28.210438 162 83 - 79 Philadelphia Phillies 1529.0 28.447520 162 89 - 73 San Diego Padres 1523.0 28.317698 163 89 - 74 New York Mets 1510.0 28.604631 162 88 - 74 Los Angeles Dodgers 1505.0 28.615645 162 82 - 80 Minnesota Twins 1495.0 28.312780 162 79 - 83 Atlanta Braves 1494.0 28.618743 162 84 - 78 Chicago Cubs 1494.0 28.402241 162 85 - 77 Oakland Athletics 1488.0 28.395109 162 76 - 86 Texas Rangers 1487.0 28.586974 162 75 - 87 Milwaukee Brewers 1474.0 28.445591 162 83 - 79 St. Louis Cardinals 1464.0 28.453097 162 78 - 84 Washington Nationals 1462.0 28.858994 162 73 - 89 San Francisco Giants 1462.0 28.315478 162 71 - 91 Chicago White Sox 1461.0 28.461948 162 72 - 90 Kansas City Royals 1461.0 28.856424 162 69 - 93 Baltimore Orioles 1460.0 28.501870 162 69 - 93 Tampa Bay Rays 1450.0 28.742325 162 66 - 96 Houston Astros 1448.0 28.453125 162 73 - 89 Cincinnati Reds 1442.0 28.818955 162 72 - 90 Florida Marlins 1438.0 28.412856 162 71 - 91 Pittsburgh Pirates 1424.0 28.681253 162 68 - 94
NOTE: This is a great example on how close in skill professional baseball teams are.
Now, lets take a game from the 2007 season and determine the outcome via glicko and the same percentage via the Pythagorean expectation. I will use Atlanta Braves games since they are my favourite team. Lets take the 42nd game Atlanta played. This game was versus the Boston Red Sox who was a high rated team all year.
NOTE: This determination is very simplistic and missing some steps, a more in-depth example may follow in other blog posts.
Boston had a record of 29 - 12 (Winning percentage 70.7%)
Atlanta had a record of 25 - 17 (WP 59.52%)
2007-05-19 ATL 1529.0 vs. BOS 1675.0
2007-05-19 ATL 52.562% vs. BOS 72.554%
By rating, Boston should win, they are the strongest. The chance for Atlanta to win is about 47.7%. The outcome of the game was Boston 13 and Atlanta 3. Boston crushed them. Before this, Atlanta split a series with Washington, a very weak team.
By winning percentage, we see that Boston's real percentage is lower than what their percentage should be and we see that Atlanta's real percentage is higher than the determined one. Using that we see that Boston is favoured to win unless luck comes into play.
Both methods show Boston being the favourite, but 10 runs seems too much. :)
Determining the actual chance that Boston will win is a bit more complicated than the above example, with both methods. It is easy to see that using each way shows that Boston is stronger and should win the game, and they did. You can also use other statistical methods to determine how many runs both teams could score and use that to help with determining the outcome. I want to find if I could use ratings to help or replace some of these.I believe ratings is a great indicator of how strong the team is right now (or at the moment) and is a simpler way when looking at strength of schedule and other factors in finding out who will win a series or finish first in the standings.
In posts to follow, I hope to start predicting the outcome of current games, and developing a system to make this easier for me and others. I plan on using XMPP, Pubsub and BOSH. I also want to investigate finding a pitcher or batter's rating using glicko. After that, who knows.
The baseball information used here was obtained free of charge from and is copyrighted by Retrosheet. Interested parties may contact Retrosheet at "www.retrosheet.org".
Twisted XMPP
It seems talk of an XMPP Server in twisted is picking up. I am looking forward to helping out! http://ralphm.jaiku.com/presence/43726932 http://twistedmatrix.com/pipermail/twisted-python/2008-September/018330.html
WuChess Video Contest
If you have not seen this yet, you really need to check it out. Or even give it a try. ;)
http://www.youtube.com/video_response_view_all?v=3YAIVQ_oh88
http://www.youtube.com/video_response_view_all?v=3YAIVQ_oh88
