Summer of Code

March 27, 2008 at 10:38 AM | categories: python, oldblog | View Comments

Only a few more days to get your application in, and less if you want good feedback. The ideas we're seeing look really good and interesting, but we'd love to hear from more talented developers, and see your really cool ideas. Our ideas page continues to expand with more ideas and the mixture we're seeing is really interesting. If you haven't put in your application yet, why not? :) After all, this year we're after ideas that are completely different from previous years, since the intent is to allow students to produce something really cool they've always wanted to build, such that it grows the component set in interesting ways, as well as the kamaelia cookbook which given all components (from DVB through Open GL through networking) can be used with one another is kinda fun. (as well as useful because the whole thing is naturally multicore friendly) 
Read and Post Comments

Kamaelia and Google Summer of Code

March 21, 2008 at 02:23 PM | categories: python, oldblog | View Comments

Google's summer of code is on again this year, and BBC Research is taking part, and my interest is in Kamaelia based projects. For those that don't know, this is a short summary of what kamaelia is:  Kamaelia's aim is to make it easier and more fun to make software, quickly and maintainably in a way that makes concurrency (eg multicore) easy and fun to work with. The more code that is multicore friendly, the easier we make it for everyone. In this post I cover some links for past years, the sort of students we're looking for, what students have done in past years, and over view of what we'd like to see this year, etc.

Our primary pages for each year are listed below:

Students we're looking for

Please don't prejudge yourself as "I can't do that, I don't have the right experience". Some of the best projects we've seen have come from the most inexperienced of students. Some of the best insights have come from the most naive. If you have a lot of experience, you don't necessarily have the "right" experience, and so lull yourself into a false sense of security. However, fundamentally, if you think the project is interesting, the ideas are interesting or you're inspired by the ideas, please get in touch. Let us make the judgement, rather than prejudge youself out of a fun summer. Yes, we can't take everyone, but that doesn't mean we always take people with lots of experience :-) Your lack of experience can be a fantastic asset, OK? :-)

Previous Years

Previous year students
In previous years we've accepted students who have had a wide range of experience. Some have been pre-university students who are starting university following the summer, other ones we've had have been in the middle of the course. Still others have been graduating and even one post grad student. Lack of knowledge of python isn't something we consider a major hurdle, as long as you put the effort into learning python and doing the mini axon tutorial. (We don't expect any prior knowledge of Kamaelia)

Previous year projects

As an idea of how useful Summer of Code has been all round, a selection of the great work students did last summer as a part of Summer of Code 2006 has:

  • Bittorrent client (created components for seeding and fetching from a swarm including means of chunking live data for "live" bittorrent based streaming)
  • Modular HTTP Server
  • OpenGL capabilities (including, for example, putting pygame components on opengl objects as textures - still able to recieve pixel perfect updates (eg drawing on a 3D surface))
  • Key authentication infrastructures (proof of concept)
  • Speex codec bindings
  • A file handle like interface to backgrounded Kamaelia components
  • Some work on extending & making more user friendly the Kamaelia Web Server (though more work here is welcome)
  • High Level Kamaelia 3D Modelling Components
  • Beginnings of a visual editor for creation & composition of shard components
  • Pygame based scribble pad & textual input and standardised textual output

This year

What we expect

We expect you to only propose a project you're enthusiastic about - we can say alot more about this, but that is actually one of the most important criterion. We all get more out of it that way!

This can be either one off the list, or something you think fits with the project. Kamaelia is a fairly wide ranging project (potentially) due to the goal of making concurrency easy & fun to work with in all contexts. As a result, if you've ever wanted to write that multicore, 3D, P2P game, Kamaelia should be the technology that makes it possible, and we'd view it a valid test of Kamaelia. Then again, if you want to write a program to assist children to learn to read and write (based on the gesture code & speech synthesis) that would be equally cool and useful. Then again, if you want to create systems for working with databases or social software systems, that would be useful as well, since it hits needs of the project. As we say, wide ranging.

There are however expectations we have as a result.
  • We will expect you to have done the MiniAxon tutorial ideally before submitting your application. This teaches you how Kamaelia essentially works under the hood (barring optimisations), and as a result we feel is vitally important.
  • We will expect you to, where practical, discuss your application with us on IRC or email sooner rather than later. We work primarily over IRC, we will expect you to do so too, so start as you mean to go on!
  • Since you're being paid to do this, we expect you to treat it like a full time job. Generally speaking that means 35-40 hours a week, though cultures round the world vary.
  • Since its summer we won't be surprised if you say you're having a holiday, so we expect to see how that is scheduled and planned for.
  • Yes, we know exams happen at different times of the year and we all have to juggle. It is your responsibility to say how you will manage the clash between your studies and your proposed work. (yes, we feel academic work must come first, but if someone's paying you to do something you're taking on a responsibility. It's your responsibility to say how you manage your commitments)
  • We expect you to manage your time and checkin code at least 2-3 times per week - we give you a way of doing this which is safe to do so.
  • We will expect you to attend our normal weekly IRC meeting, unless timezones simply don't work
  • We will expect you to attend a weekly guaranteed mentor time session. This is an hour long session but it's specifically there to allow you to have some guaranteed time with your mentor, rather than anything else.
  • We will expect you to have some plan for maintenance of your code after summer of code is finished. This can come in several forms. One option is to plan to be around after summer of code because you're doing a project you think is personally fascinating/useful. Another option is to aim to write your code with the intent of having it maintainable.
  • We will expect you to also track your work in two ways:
    • To keep a development blog (can use your usuatl one if you like), which you update at least twice a week
    • To start a project task page for your work, and keep that up to date - creating subtask pages as and where needed.
  • If you use code from other sources (eg the python cookbook, reference implementations) we expect you to quote your sources.
  • We may ask you to write an introductory presentation to the project work you have done, in a similar way to what you see on the Kamaelia Grey page. You want people to use your work, right? :-)
  • To have fun :) (What's the point otherwise? :)
ie it's your project and your commitment - we expect you to act that way ! :-)

What to expect from us
Given that freedom of ideas, what do we expect in return? We expect you:
  • Expect us to challenge you. Support you. We want you to achieve your goals, and retain the same enthusiasm for your project at the end as you've got at the end.
  • You can expect guaranteed time specifically set aside to help you discuss your issues/successes
  • By checking in code often, we can and will give you honest feedback on it. This may be hard to take, but we'll try and be gentle :) Bear in mind this works both ways - if you criticise our code, even harshly, we'll thank you for it. (best way to get more feedback :-)
  • We will give you the freedom and support to develop your ideas, and to integrate them with the rest of the project.
  • Whilst we won't spoon feed you - we expect you to read docs for example - we will recognise that you're new to this and help you in finding docs (for example) and in showing you common shortcuts.
Essentially, we want you to achieve your goals, create something fun and useful, we want you to learn how open source works, and we want to have fun at the same time. So we'll do whatever we can to help you, so long as you do whatever you can to help yourself. (which seems a fair deal :)

Ideas for This Year

Our ideas list (Summer of Code 2008) for projects this year keeps on growing, and reflects the broad scope of the project which you can see from the components list. This is divided into two groups - exemplars (interesting useful applications which could be built and would usefully extend Kamaelia at the same time) and core improvements.

Application Ideas

  • Multiwindow pygame based paint program
  • Extend the ER modelling tool to be a full database front end (includes a requirement for Components for working with SQL Databases)
  • Small children friendly version of Compose (our visual system builder)
  • Extend the XMPP implementation code
  • 3D Social Network Visualisation tool
  • A Testing Framework for Kamaelia Systems
Core Improvements
Whilst these are core improvements, we would prefer these to be done in the context of a usecase. (Since whilst it is _possible_ to develop new and useful technology without a usecase, its not amazingly useful since no one can see how to use it. As a result, even the Open GL project for example in the past was required to have examples of usage. Where examples of usage have been weak, the projects have often been weaker. (Which shouldn't be a surprise :-)
  • WSGI-ify the Kamaelia Web Server
  • Tools for working with SQL Databases.
  • Improved Multicore support (We can already do multicore, but it requires merge into the core, and how does it affect things like backplanes and all the various components.)
  • Extensions to the co-ordinating assistant tracker :
  • Change the topology visualiser to work in full 3D
  • Extend Kamaelia's Dirac support to include support for the optimsed version of Dirac called Schoedinger
  • Platform specific improvements
    • Improve Kamaelia's support on Windows (XP and Vista primarily), including how to package up Kamaelia Apps for it. (eg the ER Modelling tool)
    • Improve Kamaelia's support on Mac OS X
    • See how much of Kamaelia can run on IronPython 2
  • Explore how the changes to the way components can be defined can be handled and introspected by Compose.
This list isn't exhaustive by any stretch of the imagination, and whether or not you're interested in our projects or someone elses, have a good summer of code!
Read and Post Comments

Kamaelia Multicore Pipeline. The future is here.

March 16, 2008 at 12:16 AM | categories: python, oldblog | View Comments

As the next step forward, I now have multiprocess support for basic pipeline. Essentially I have the equivalent of "self.link" implemented for multiprocesses. Kinda like the Postman in the MiniAxon tutorial. It's a little messy right now, but works, and does NOT require any changes to existing Kamaelia components. It builds on LikeFile from last years summer of code efforts and pprocess. The core code currently looks like this:
# setup code snipped, and specifically ProcessWrapComponent snipped

from Kamaelia.UI.Pygame.Text import TextDisplayer, Textbox
# Start the pygame based input in one process
X = ProcessWrapComponent(Textbox(position=(20, 340),
                                 text_height=36,
                                 screen_width=900,
                                 screen_height=400,
                                 background_color=(130,0,70),
                                 text_color=(255,255,255)))

# Start the pygame based output in another
Y = ProcessWrapComponent( TextDisplayer(position=(20, 90),
                                        text_height=36,
                                        screen_width=400,
                                        screen_height=540,
                                        background_color=(130,0,70),
                                        text_color=(255,255,255)) )

exchange = pprocess.Exchange()
chan1 = X.activate()
chan2 = Y.activate()

exchange.add(chan1)
exchange.add(chan2)

while 1:
    for chan in exchange.ready(0):
        if chan == chan1:
            D = chan._receive()
            if D[1] == "outbox":
                chan2._send( (D[0], "inbox") )
    time.sleep(0.1)

Clearly most of this would be wrapped up for reuse. The nice thing of course about this, once wrapped for reuse, is that splitting a program across multiple boundaries for reuse will be pretty trivial and simple, and any kamaelia program that wants to do this will find it pretty trivial to implement. Now the nice thing about the example above is that it uses pygame. In pygame, because it builds on SDL, you can only have one window open in your process. Therefore to have 2 windows, you must be using multiple processes. Now as you can see above, the pygame components require no change, and yet this DOES result in multiple windows.

Screenshots:
In this screenshot you see the Textbox on the left and the TextDisplayer on the right. (Both were written as part of summer of code last summer incidentally). You'll see I got a bit excited, but you'll also see a piece of text before I press return.


This screenshot is after I press return:


Note that the data has passed seamlessly from one to the other, across process boundaries and probably CPU boundaries too since this is running on a multicore machine. Here's the ps output:
top - 00:11:41 up 11:51, 13 users,  load average: 1.09, 0.72, 0.66
Tasks: 149 total,   4 running, 145 sleeping,   0 stopped,   0 zombie
Cpu(s): 83.0%us, 16.5%sy,  0.0%ni,  0.3%id,  0.0%wa,  0.2%hi,  0.0%si,  0.0%st
Mem:   2066996k total,  1605500k used,   461496k free,   243348k buffers
Swap:  1959920k total,        0k used,  1959920k free,   789804k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 7727 zathras   25   0 53740  13m 6516 R   96  0.7   0:16.31 MultiPipeline.p
 7729 zathras   25   0 52616  12m 6444 R   93  0.6   0:15.83 MultiPipeline.p

OK, these components are decidedly hogging CPU as well, but as you can see they are both able to do so, and are running happily concurrently and in a not much more awkward way than normal. It's also only more awkward than normal due to the fact I haven't yet implemented the syntactic sugar required to make this work happy...

The multiprocess/core code for the brave is here in subversion.

The future's here. The future's multicore.


Read and Post Comments

Sociable Software (Presentation)

March 03, 2008 at 11:57 PM | categories: python, oldblog | View Comments

Well, I travelled all the way up to barcamp this weekend, and it was well worth it. Whilst I was there I gave a talk called "Sociable Software", as a general over arching description of how Kamaelia deals with concurrency. I'm going to write this up as a proper blog entry as well shortly, but in the meantime I thought adding the slides here would be a good idea.

Read and Post Comments

What is wrong with free software zealots?

February 29, 2008 at 12:12 PM | categories: python, oldblog | View Comments

Now they think they have the right to dictate to me what software I get to choose to run? cf in italics
On Thursday 28 February 2008 15:58:08 Dave Crossland wrote:
>  Even if I choose to use a proprietary program on a open source operating
>  system. Sorry, I'm not wrong,

Sorry, you agree not to share with me, which is wrong.

Sorry Dave, you're wrong. It's perfectly fine. It is preferable to be able to share, but in no way is it wrong to legally buy a copy of something copyrighted and choose to use it according to its license.

If you, like Dave, view it acceptable to tell me what software I can or cannot choose to run, please don't. Being told to go away and push your politics on someone else very often offends. (I'm not sure why they get offended - they're the ones with the offensive behaviour). I make my choice of what to use from the full range of software that's (legally) available to me thank you very much. If you want to restrict my freedom to do so, stay away from me, you're not welcome.

Incidentally, it's attitudes like Dave's that have pushed me further and further away from free software advocates over the years. If you're willing to respect my right to choose from all available software, then you'll probably find we have a lot in common. Advocates fine. Zealots who don't know where the boundaries are? Stay away.

Read and Post Comments

25 years

February 25, 2008 at 12:23 AM | categories: python, oldblog | View Comments

Well, this year will mark the 25th year that I've been writing code, so at some point later in the year I'll probably have some sort of party/drinks relating to that, since it means something (not sure what :-) ) to me. I have to decide a date where that makes sense, maybe some of the following will make sense. This post is a bit self indulgent as a result.

I can't give a particular date, but I suspect the date is probably sometime around easter 1983 - based on some very vague memories. I'd been interested by computers and programming for a couple of years at that point, mainly due to seeing some schematics in Doctor Who weekly saying how to build your own K9. I've found the cover of that magazine here for those who wonder what got me interested.

Anyway, the idea of making a robot dog - especially when you're given schematics, is like a red rag to a 7 year old. Have cardboard boxes, will get started. Anyway over the next couple of years, amongst all the other things little boys do, I read more about robots and so on in kids books and it became more clear that in order to build a robot, you needed to know about computers. As a result the "maker" culture that's sprung up in California amuses me - in a good way (it's a good thing after all) - it was this desire to "make" something that drew me into software in the first place.

It's worth stopping here and considering this: the schematics for K9 looked like something a child could, with effort, actually make. That moved it from the realms of complete fantasy and unreachable into the realms of the possible (which is of course what good science fiction really is). Not only that, it made robots and computers seem like something a child could do, rather than "just" something to be watched. Having seen the schematics again some years later, it is rather beyond the scope of a 7 year old, but that wasn't really the point.

Fast forward a couple of years, and my brother (David) had been interested in programming since they'd done something to the Commodore PET at school which meant they'd been encouraged to fix it. The side effect is that eventually David (then aged 12 I think) promised to explain the BASICs to me.

I still remember the first program I tried to write. I'd been told at that point that "computers have to be told everything, and require very precise instructions and can tell you things". So I wrote down my first program, and the next time we were near Dixons in Lion Yard in Cambridge, (then next to the library) I typed it into one of them (I think a Vic 20).

Unsurprisingly, it didn't actually work because of a rather major missing detail in the above - the fact that computers use special languages for their instructions.

For your amusement, here's that first program (more or less):
"You are a computer
 I am a human
 What are you?"

Since I was 9 at the time at the absolute oldest, I'm going to give myself some credit here - it was a good try, and but with more experience I can see just how demanding a set of conversations that is to a machine with less than 16K of RAM. (I was more or less expecting the machine to pass a subset of the Turing test I now realise, which is an entertaining assumption!)

Fast forward a little while (autumn '83) and my brother had by then taught me the basics of BASIC, what a string was, etc and I was good to do, sans the detail of not actually owning a computer. At that point in time (1983), my Dad got hold of a ZX81 - hardly cutting edge then (the ZX Spectrum existed, in not only 16K but also whizzy 48K version - the ZX81 had a whopping 1K of memory - 0.4K actually usable, and a 16K RAM pack), but it was the first micro I had any access to, beyond playing with the microcomputers in Dixons and Boots. (Looking back, I think those shops were remarkably tolerant in comparison to today!) Heck, they were being almost given away as prizes at the fairground. However, it was a real computer, and it did work.

Co-incidentally, we also moved back to Cambridge - near Waterbeach, and the primary school I was going to had a book club. So I saved up all my pocket money and at that book club bought my first ever book on computer programming, and to this day I still rate it as the single best book ever that introduced programming. It could be updated I think, but would be difficult to improve.

It was targetted at children, and as a result had fun pictures and diagrams - which were all hand drawn & illustrated - but were largely of little robots running around inside a computer. (See, things go full circle sometimes :-). The robots were tasked (literally) with running around moving bits of data from here to there. To sending stuff to the display, putting dots into a big grid to draw pictures. They stood in front of racks of boxes (much like an old fashioned hotel lobby with pigeon holes for keys/messages) where data was put & labelled and so on.

Quite simply it was fantastic :) Not only that, but it was only 48 pages long. Yes. 48 pages.

For those that remember, yes, this was one of the Usborne computer books, and I'm amazed at how dumbed down they've become over the years. (I picked up their "Starting Computers" a few years back to see how it'd "improved"... :-/ ) They were perfectly targetted - they were marketed as part of their "scholastic" series in schools as suitable for 9-12 year olds, and they were perfect. They showed you how to make games. Some how to do graphics. Another taught you electronics projects, and yet another even showed you how to make a robot.

Whilst I still have my book on Computer Graphics and on "how to build your own robot" from those days, I'd mislaid that book: Introduction to Computer Programming - by Brian Smith. However, many years later I'd borrowed from the library an omnibus book "The Beginner's Computer Handbook" - which featured 3 Usborne books in one: "Understanding the Micro", "Introduction to Computer Programming" & "Computer Space Games". That, along with other usborne books of the time were highly responsible for boot strapping my early interest.

By 1984 I'd written a simple "Logo" interpreter. Come 1985 or so, my parents splashed out on a Commodore Plus/4 and for various reasons (mainly fun), I ended up writing a pre-emptive task switching monitor in 6502 assembler for it. (Yep, aged 12 or 13 I wrote an incredibly basic pre-emptive multitasking operating system for an 8 bit micro those who understand what all those words mean.)

Then along came the Amiga, and I kept on building, eventually leading to writing a paint program for my GCSEs - but this was different - you painted notes on a scale - so you literally painted music. That was probably, ironically, one of the few times I've been forced to struggle with a microsoft product - since that was writting in Amiga Basic - which was written by Microsoft. It was also probably one of the last large programs I ever wrote in basic. I've completely forgotten what I did for my 2nd year a level computer science course now. It was some sort of database system.

Part of the reason for that being my last BASIC program is probably due to finding the astounding book "The Cartoon Guide to Computers" by Larry Gonick. Aside from anything else what's perhaps most astounding about it is that it teaches you enough electronics, enough boolean logic, enough about how logic gates work, how a computer is subdivided into its constituent elements, not just in a high level way, but all the way down to how microcode inside a CPU works to implement CPU opcodes that you write for assembler. Stunning book. It was the logical next step after the Usborne books. From then on it was Pascal & C which lead to Amiga E & Standard ML - which taught me object oriented programming and functional programming respectively.

Given my experience of using a pre-emptive multitasking OS, and having written the basics behind it myself before the age of 16, it should come as little surprise that I wasn't particularly impressed with Windows 3.0 when it came out some years later, and the fact it locked up when it shouldn't. Moving from the Amiga to Unix however was pretty natural.

Anyhow, enough digression, I continued to progress, with formal education, but always with a fair amount of play with new ideas. It's easy to think that because I work with computers that somehow I went through the route that some people have of cross training after an unrelated degree, or that I simply did a computer science degree because I couldn't think of anything better to do. However, the reality is, whenever I start something new, learn something new to do with a computer, I'm still learning with that same enthusiasm I had as a 9 year old.

Show me the results of applying text mining to the output of the BBC website, and I jump with glee, not  just wanting it, but knowing how realistic it is to want. Show me multi-touch, and I instantly see that all I'm waiting for is a display that I can actually work with to do something fun with it. Tell me that there's a new speech synthesiser package for Linux that actually works and I integrate it with Kamaelia:
Pipeline(
    Textbox(position=(20, 340),
                     text_height=36,
                     screen_width=900,
                     screen_height=400,
                     background_color=(130,0,70),
                     text_color=(255,255,255)),
    PureTransformer(lambda x: str(x)+"\n" ),
    UnixProcess("while read word; do echo $word | ./espeak -w foo.wav --stdin ; aplay foo.wav ; done")
).run()
Yes, that's a toy. But it's fun. Partly for me, partly for others. But I now wonder why my system doesn't talk as much as it could. After all the above can be integrated with anything, so sooner or later my computer will be talking more. Or it'll find its way into a system that needs to talk.

However, we come back now to full circle. The extremely observant among you will have noticed this above:
"The robots were tasked (literally) with running around moving bits of data from here to there. To sending stuff to the display, putting dots into a big grid to draw pictures. They stood in front of racks of boxes (much like an old fashioned hotel lobby with pigeon holes for keys/messages) where data was put & labelled and so on."
In Kamaelia, my primary metaphor has always been "imagine someone sitting at a desk, with a bunch of intrays(inboxes) and out-trays. They do work based on the data they recieve and send it out the out-trays. Then another person - a postman - comes along and moves the data from the out-trays to the in-trays.". Now clearly this is a metaphor - but there's a few things I've learnt about computers over the years:
  • Metaphors matter.
  • Good metaphors are hard to come up with, perhaps the hardest thing in computer science.
  • That if you can think of a metaphor, you can use it to write code.Therefore, why not pick one that you enjoy?
  • Any good metaphor can be represented by using pictures of humanoid robots.
And that last point brings us to the silly little secret in Kamaelia. When I'm thinking "someone sitting at a desk", I'm picturing a Kids book on Kamaelia with little robots sitting at a desk. When I'm saying "someone comes along to do deliveries", I'm thinking more little robots with roller skates on doing the deliveries between components. When I added in the Co-ordinating Assistant Tracker - the assistant - it was very much added in because there's a very practical metaphor there - that of a team assistant. (You have a team of little robots all working together, and they tell the assistant what they're doing if they're willing to get requests from other robots)

So there you have it, not only am I really wanting to make Kamaelia accessible to the average developer, I'm actually thinking of the average 9-10 year old who wants to be a developer - or perhaps even the 6 1/2 -7 year old who decides "I want to build K9", and at the end it's really due to me remembering how I got here - an interest in building K9, books explaining to kids how robots work, books explaining to kids how to program, the cartoon guide to computing (and underlying electronics, physics and signal theory...), Simula BEGIN, and many more similar books, and hardware/software systems that made these things doable, all in short friendly descriptions, but ones which weren't skin deep.

The books you get today are just awful. 100s (or 1000s) of page tomes on introducing a concept that used to be possible to explain to a 9 year old in a 48 page book with pictures of robots. It must be possible to get back to that stage. Even the "head first" books are awful - they're a nice idea, but just too large to be actually any use IMO. Anyhow, if no-one writes that book, I guess I'll just have to update the usborne books of years ago with something new. (Which, I do actually plan to do - I've scoured amazon and similar for places selling the originals, and getting somewhere there)

The new book will have to deal with video, multimedia, TV, games, 3D, as well as things like web, and all that whizzy stuff, with a nice and easy way of bolting these things together, but in a way that prepares anyone reading it for a world of writing concurrent software. It's always been an underlying agenda in Kamaelia, but this year I think that'll have to happen.

Along with finally making a real K9 :-) (more on that anon)

Read and Post Comments

Javascript turtle

January 25, 2008 at 09:05 AM | categories: python, oldblog | View Comments

A turtle/worm written using javascript: http://languagegame.org/tmp/jstile/exp/worm.html
Read and Post Comments

Register Article on cross platform iPlayer "issues"

January 15, 2008 at 12:42 AM | categories: python, oldblog | View Comments

I tend to understate certain things sometimes. From The Register article: MP accuses BBC chief of illegally championing Microsoft - John Pugh - MP for Southport:
As part of a Public Accounts Committee session on BBC procurement last Tuesday, the director-general was grilled by Liberal Democrat John Pugh MP on the decision to release the download version of the iPlayer for Windows and Internet Explorer only.
...
Pugh followed up the meeting with a letter to Thompson on Wednesday. He wrote: "By guaranteeing full functionality to the products of one software vendor [the BBC] is as a public body handing a commercial advantage to that company - effectively illegal state aid!

"What might be a pragmatic choice for a privately funded company becomes deeply problematic for a public corporation."
The full article is well worth a read - especially for those who think that technology decisions in a public corporation don't have a political angle. (This is why generally R&D has historically aimed on producing technologies and facilities and understanding, with the aim of allowing such decisions to be informed, rather than taking them. At least that's what they keep telling me :-)
Read and Post Comments

Parsing XML in Kamaelia using an expat parser

January 13, 2008 at 03:05 PM | categories: python, oldblog | View Comments

Generally we've used sax for parsing XML, but it's useful to show how to parse XML using a parser like expat that works by calling back into your code. The trick, as usual with anything long running, is to put the thing that does the long running call into a thread and have it emit messages when a callback is called. The following is a minimal example:
import time
import Axon
import xml.parsers.expat
from Kamaelia.Chassis.Pipeline import Pipeline
from Kamaelia.Util.Console import ConsoleEchoer

class Parser(Axon.ThreadedComponent.threadedcomponent):
    data = "<h1> Default </h1>"  # Can be overridden by kwargs as normal

    def start_element(self,name,attrs):
        self.send(("START", name,attrs), "outbox")

    def end_element(self,name):
        self.send(("END", name), "outbox")

    def char_data(self,data):
        data = data.strip()
        self.send(("DATA", data), "outbox")

    def main(self):
        p = xml.parsers.expat.ParserCreate()
        p.StartElementHandler = self.start_element
        p.EndElementHandler = self.end_element
        p.CharacterDataHandler = self.char_data
        p.Parse(self.data, 1)
        time.sleep(1)
        self.send(Axon.Ipc.producerFinished(), "signal")

Pipeline(
    Parser(data="<body><h1>Hello</h1> world <p>Woo</p></body>"),
    ConsoleEchoer(),
).run()
This generates the following output:
('START', u'body', {})('START', u'h1', {})('DATA', u'Hello')('END', u'h1')('DATA', u'world')('START', u'p', {})('DATA', u'Woo')('END', u'p')('END', u'body')
The nice thing about this of course is that this then allows you to test the thing that's taking this information in isolation from the XML handling code. Indeed, it allows for a much simpler test harness overall.
Read and Post Comments

Debugging Kamaelia Systems

January 13, 2008 at 02:56 PM | categories: python, oldblog | View Comments

One thing that is particularly useful when debugging Kamaelia systems - especially servers - is the ability to dump out what's running. Fortunately Kamaelia provides a mechanism to introspect the scheduler, so you can do this as follows:
import time
import Axon
from Kamaelia.Chassis.Pipeline import Pipeline
from Kamaelia.Util.Console import ConsoleEchoer

class PeriodicWakeup(Axon.ThreadedComponent.threadedcomponent):
    interval = 300
    def main(self):
        while 1:
            time.sleep(self.interval)
            self.send("tick", "outbox")

class WakeableIntrospector(Axon.Component.component):
    def main(self):
        while 1:
            Q = [ q.name for q in self.scheduler.listAllThreads() ]
            Q.sort()
            self.send("*debug* THREADS"+ str(Q), "outbox")
            self.scheduler.debuggingon = False
            yield 1
            while not self.dataReady("inbox"):
                self.pause()
                yield 1
            while self.dataReady("inbox"):
                self.recv("inbox")

Pipeline(
    PeriodicWakeup(),
    WakeableIntrospector(),
    ConsoleEchoer(),
).activate()

Note: this construct won't shutdown.
This produces output of the following kind:
*debug* THREADS['Kamaelia.Chassis.Pipeline.Pipeline_11', 'Kamaelia.Chassis.Pipeline.Pipeline_7', 'Kamaelia.Internet.TCPClient.TCPClient_9', '__main__.Listener_10', '__main__.PeriodicWakeup_5', '__main__.WakeableIntrospector_6']
This turns out to be, as you might expect, to be particularly useful.
Read and Post Comments

« Previous Page -- Next Page »