As discussed before, Kamaelia now has multicore support in subversion in the form of ProcessPipeline and ProcessGraphlines, meaning taking any Kamaelia system and making it multicore is a matter of simply pre-pending "Process" to the pipeline set up. That's pretty cool - especially if you're aware of what our normal cookbook code looks like. The question I've really got is where to merge this on the mainline. Given how useful this would be, it strikes me that like ThreadedComponent & LikeFile and Introspection, this ought to live in Axon. However, like Introspection, it probably also ought to be used from inside the Kamaelia Namespace.

As a result, it seems to make sense to do this - Put the core code in:

Axon.Processes

But put the ProcessPipeline & ProcessGraphline component in:

Kamaelia.Chassis.Process

So that you would use them as:

from Kamaelia.Chassis.Process import ProcessPipeline, ProcessGraphline

I'm pretty sure that's a pretty good idea - since it allows updates to the implementation that these use without affecting the interface.
That said, people will probably search for Multicore as well, so is it worth the naming actually being this:

from Kamaelia.Chassis.Multicore import ProcessPipeline, ProcessGraphline

... or simply having that available as an alias ?

Fundamentally this would still be used in the same way as previously described:

from Kamaelia.Chassis.Multicore import ProcessPipeline # this or
from Kamaelia.Chassis.Process import ProcessPipeline   # this?

ProcessPipeline(
                Textbox(position=(20, 340), # Open first pygame window
                                 text_height=36,
                                 screen_width=900,
                                 screen_height=400,
                                 background_color=(130,0,70),
                                 text_color=(255,255,255)),
                TextDisplayer(position=(20, 90), # Open second pygame window
                                        text_height=36,
                                        screen_width=400,
                                        screen_height=540,
                                        background_color=(130,0,70),
                                        text_color=(255,255,255))
                )

But the naming affects where you go hunting for the functionality.

My personal preference is for the former (Kamaelia.Chassis.Process), though I can see user friendliness in naming something based on what people are likely to hunt for being attractive.
Used without care, multiprocess usage would probably hurt performance - since messages between components in separate processes are pickled objects, but generally speaking, we all know you'd take that into account, wouldn't you...?

Thoughts?

New faces bring new ideas. As part of Google Summer of Code this year I asked upfront our students why they didn't want to use the mailing lists the project uses on sourceforge (since it was clear they didn't) - preferring private mails and IRC (as a project we tend to prefer IRC generally nyway). The response we got was unaminous around the fact they didn't want to subscribe due to SF lists being highly spammy, and requested we move over to google groups - so they could also gain better web access and RSS feeds. It was the one answer I hadn't thought of or expected :-)

So I chatted with the other members of the community and they all pretty much agreed, but they hadn't thought of saying anything (which strikes me as a bizarre version of the Bystander Effect :-).

Anyhow, as a result, I've shifted our list over and the new faces on the block are running rampant there, which is really cool. Not only that but it's encouraging students from previous years to start popping their heads back up (so far only on private mail, but I think they can be teased out further), which is really neat :)

Sometimes it's the little details you take for granted that you completely miss...

I just thought it worth sharing in case other people think it's worth asking their new colleagues for any suggestions for making their communities more accessible :-)

People have used the comment or new post feature of my blog recently to leave some useful comments. Rather than have them as top level posts though, I'll drop them as replies to this post. In summary though:

• I had a complaint/bug report from Steve (no surname) about the dates in my RSS feed (which should now be fixed), about double escaping of entities (which is also hopefully fixes). Steve also made a comment saying that he wished he didn't have to enter HTML manually. This strikes me as odd, because I'm typing this using Dojo's editor widget, so I guess that failed disgracefully on his machine. I've seen that on a few machines, but Dojo's generally got the most browser friendly wysiwyg editor, so I'm not sure what's up there.
• I had a request whether Kamaelia's latest release is available for windows as a special bundle. The answer to that really is "it's python, so it should work, but we tend to develop under linux & Mac OS X so you may find some niggles". I certainly haven't created an installer for windows. All that said, we are overdue doing a new release, and /trunk is far and away recommended. Even so, that needs updating as well with recent developments... :-)
• Finally, another anonymous poster made the following comment:
• Came across this today and thought you might be interested in it Michael:
  
  Concurrency/message passing Newsqueak [video.google.com]

Not tested this, but this looks really interesting/promising. The AR Toolkit is really cool, and this appears to be linux & mac os x bindings for it. (Means that using this people should be able to access a webcam easily from both of those OSs using python...)

• http://mgldev.scripps.edu/projects/pyartk/index.html
• http://mgldev.scripps.edu/projects/pyartk/docs.html

We're getting some commons questions regarding GSOC, which boil down to "how do I write a successful application". I've taken the common starting point and generalised my comments, questions and suggestions in the hope of helping those in the crunch. This will also go on the Kamaelia website linked from the GSOC pages. As such this is written on the assumption the reader is someone interested in putting in a GSOC application.

> I want to write a simple kamaelia based [....] for Google Summer of Code,

That's a very clear, definite, good statement. It's incredibly simple to get a hold of, so the questions really arise around "what sort of look/feel do you want to have", and "how do you want the [...] to work" and "how do you want communication between [...] to work" as well as "how do you plan to maintain persistent information as well as dynamic (maybe shared) state". (if any)

If you've sent existing code, this helps a lot with evaluating your idea, especially if it relates directly to your idea. Really, this boosts your application likelihood of acceptance dramatically.

Some general points:

• You need a mechanism for handling initialisation of the system and activities using the system.
• You may need to think about the transition from initialisation to the normal operation. (eg login vs logged in)
• You'll need to think how you manage global state (managing the model)
• You'll need to think how you manage specific state (managing interactions)
• You may need to think about file/definition formats
• If the system is multiuser, with interactions, you'll need to define that mechanism. That could be a protocol, use a pre-existing protocol, or just a mechanism for forwarding internal messages.
• You may want to consider user to user interactions vs global vs localised (eg talking in a room)
• You will need to think how existing components can assist you.

Regarding how you go about development, some points:

• First of all think "what's the simplest possible version that would work?" (single windows GUI, single web page, single screen, single ...)
• Then consider how you extend it in each of the possible directions you're thinking of going, growing organically. (this makes it easier to track your development)
• Consider something really cool as a possible next step, after all, things may be simpler than you expect.

> The system will [...] and [...] and [...] and put [...] and [...] into a [......]

Consider some of the of the implications for each thing you suggest. Do components already exist that can support it? (eg {SQL, webcam, ardunio, twistedintergration} components don't *at present* exist, but would be useful). Recognising what's missing from the component set and how your project helps grow the project is important. Both for us, and for you (since creating components is kinda the point of these projects :). You'll need to think ho your new components will operate and how they'll be used. However DON'T overengineer or think "I'll do this in case someone needs it". If you don't need it, leave it out. If you need it, put it in. You can however sometimes make your life easier by generalisation. Tease these aspects out in your application.

Finally, you need to put the whole thing into the application template covering the various areas we need to assess the project:

• Short one line of what the task is designed to achieve/create.
• A practical, clear result of what will be possible as a result of achieving this task. This is best described in the case of a user story.
• The context in which this task sits. Has this task any history? Is it the result of any previous tasks - either within the project or outside.
• What benefits will be gained by working on this task, and achieving its goals? Speculative as well as certained/realistically expected benefits are valid here.

Regarding the actual application template which is here:

• http://kamaelia.sourceforge.net/SummerOfCode2006Template (yes, this is the same template :-)
  

Some key things:

• We expect you to divide the project deliverables up into two categories "will deliver" and "given sufficient time".
• The Project Details section is where you get to describe lots of the things I raise in terms of questions above. The application character count limit though is worth bearing in mind. (ie detailed, but not too ott maybe)
• Project Schedule is really a guestimate of what you think you will have done by when. It will probably be wrong, but by listing it we can see if you're way ahead of schedule (which can be good - if you have a project with lots of scope for growth - which you do), or if you're really behind, in which case we can re-evaluate the schedule if its for good reasons.
  Key things to think of:
• What do you think you will have done by the mid term evals. For that think "what will I get done by July".
• What do you have confidence of getting done between then and mid-august.

Finally, bear in mind we won't be able to take all applicants, but want to take good applications. If you follow this advice you have a much better chance of success.

Only a few more days to get your application in, and less if you want good feedback. The ideas we're seeing look really good and interesting, but we'd love to hear from more talented developers, and see your really cool ideas. Our ideas page continues to expand with more ideas and the mixture we're seeing is really interesting. If you haven't put in your application yet, why not? :) After all, this year we're after ideas that are completely different from previous years, since the intent is to allow students to produce something really cool they've always wanted to build, such that it grows the component set in interesting ways, as well as the kamaelia cookbook which given all components (from DVB through Open GL through networking) can be used with one another is kinda fun. (as well as useful because the whole thing is naturally multicore friendly) 

Our primary pages for each year are listed below:

• Summer of Code 2008
• Summer of Code 2007
• Summer of Code 2006
• Introductions: Overview, Whiteboard Article, Light Technical Intro, Presentations

Students we're looking for

Please don't prejudge yourself as "I can't do that, I don't have the right experience". Some of the best projects we've seen have come from the most inexperienced of students. Some of the best insights have come from the most naive. If you have a lot of experience, you don't necessarily have the "right" experience, and so lull yourself into a false sense of security. However, fundamentally, if you think the project is interesting, the ideas are interesting or you're inspired by the ideas, please get in touch. Let us make the judgement, rather than prejudge youself out of a fun summer. Yes, we can't take everyone, but that doesn't mean we always take people with lots of experience :-) Your lack of experience can be a fantastic asset, OK? :-)

Previous Years

Previous year students
In previous years we've accepted students who have had a wide range of experience. Some have been pre-university students who are starting university following the summer, other ones we've had have been in the middle of the course. Still others have been graduating and even one post grad student. Lack of knowledge of python isn't something we consider a major hurdle, as long as you put the effort into learning python and doing the mini axon tutorial. (We don't expect any prior knowledge of Kamaelia)

Previous year projects

As an idea of how useful Summer of Code has been all round, a selection of the great work students did last summer as a part of Summer of Code 2006 has:

• Bittorrent client (created components for seeding and fetching from a swarm including means of chunking live data for "live" bittorrent based streaming)
  
• Modular HTTP Server
  
• OpenGL capabilities (including, for example, putting pygame components on opengl objects as textures - still able to recieve pixel perfect updates (eg drawing on a 3D surface))
  
• Key authentication infrastructures (proof of concept)
  
• Speex codec bindings
• A file handle like interface to backgrounded Kamaelia components
• Some work on extending & making more user friendly the Kamaelia Web Server (though more work here is welcome)
• High Level Kamaelia 3D Modelling Components
• Beginnings of a visual editor for creation & composition of shard components
• Pygame based scribble pad & textual input and standardised textual output

This year

What we expect

We expect you to only propose a project you're enthusiastic about - we can say alot more about this, but that is actually one of the most important criterion. We all get more out of it that way!

This can be either one off the list, or something you think fits with the project. Kamaelia is a fairly wide ranging project (potentially) due to the goal of making concurrency easy & fun to work with in all contexts. As a result, if you've ever wanted to write that multicore, 3D, P2P game, Kamaelia should be the technology that makes it possible, and we'd view it a valid test of Kamaelia. Then again, if you want to write a program to assist children to learn to read and write (based on the gesture code & speech synthesis) that would be equally cool and useful. Then again, if you want to create systems for working with databases or social software systems, that would be useful as well, since it hits needs of the project. As we say, wide ranging.

There are however expectations we have as a result.

• We will expect you to have done the MiniAxon tutorial ideally before submitting your application. This teaches you how Kamaelia essentially works under the hood (barring optimisations), and as a result we feel is vitally important.
• We will expect you to, where practical, discuss your application with us on IRC or email sooner rather than later. We work primarily over IRC, we will expect you to do so too, so start as you mean to go on!
  
• Since you're being paid to do this, we expect you to treat it like a full time job. Generally speaking that means 35-40 hours a week, though cultures round the world vary.
• Since its summer we won't be surprised if you say you're having a holiday, so we expect to see how that is scheduled and planned for.
• Yes, we know exams happen at different times of the year and we all have to juggle. It is your responsibility to say how you will manage the clash between your studies and your proposed work. (yes, we feel academic work must come first, but if someone's paying you to do something you're taking on a responsibility. It's your responsibility to say how you manage your commitments)
• We expect you to manage your time and checkin code at least 2-3 times per week - we give you a way of doing this which is safe to do so.
• We will expect you to attend our normal weekly IRC meeting, unless timezones simply don't work
• We will expect you to attend a weekly guaranteed mentor time session. This is an hour long session but it's specifically there to allow you to have some guaranteed time with your mentor, rather than anything else.
  
• We will expect you to have some plan for maintenance of your code after summer of code is finished. This can come in several forms. One option is to plan to be around after summer of code because you're doing a project you think is personally fascinating/useful. Another option is to aim to write your code with the intent of having it maintainable.
• We will expect you to also track your work in two ways:
• To keep a development blog (can use your usuatl one if you like), which you update at least twice a week
• To start a project task page for your work, and keep that up to date - creating subtask pages as and where needed.
• If you use code from other sources (eg the python cookbook, reference implementations) we expect you to quote your sources.
• We may ask you to write an introductory presentation to the project work you have done, in a similar way to what you see on the Kamaelia Grey page. You want people to use your work, right? :-)
  
• To have fun :) (What's the point otherwise? :)

ie it's your project and your commitment - we expect you to act that way ! :-)

What to expect from us
Given that freedom of ideas, what do we expect in return? We expect you:

• Expect us to challenge you. Support you. We want you to achieve your goals, and retain the same enthusiasm for your project at the end as you've got at the end.
• You can expect guaranteed time specifically set aside to help you discuss your issues/successes
• By checking in code often, we can and will give you honest feedback on it. This may be hard to take, but we'll try and be gentle :) Bear in mind this works both ways - if you criticise our code, even harshly, we'll thank you for it. (best way to get more feedback :-)
• We will give you the freedom and support to develop your ideas, and to integrate them with the rest of the project.
• Whilst we won't spoon feed you - we expect you to read docs for example - we will recognise that you're new to this and help you in finding docs (for example) and in showing you common shortcuts.

Essentially, we want you to achieve your goals, create something fun and useful, we want you to learn how open source works, and we want to have fun at the same time. So we'll do whatever we can to help you, so long as you do whatever you can to help yourself. (which seems a fair deal :)

Ideas for This Year

Our ideas list (Summer of Code 2008) for projects this year keeps on growing, and reflects the broad scope of the project which you can see from the components list. This is divided into two groups - exemplars (interesting useful applications which could be built and would usefully extend Kamaelia at the same time) and core improvements.

Application Ideas

• Multiwindow pygame based paint program
• Extend the ER modelling tool to be a full database front end (includes a requirement for Components for working with SQL Databases)
• Small children friendly version of Compose (our visual system builder)
  
• Extend the XMPP implementation code
• 3D Social Network Visualisation tool
• A Testing Framework for Kamaelia Systems

Core Improvements
Whilst these are core improvements, we would prefer these to be done in the context of a usecase. (Since whilst it is _possible_ to develop new and useful technology without a usecase, its not amazingly useful since no one can see how to use it. As a result, even the Open GL project for example in the past was required to have examples of usage. Where examples of usage have been weak, the projects have often been weaker. (Which shouldn't be a surprise :-)

• WSGI-ify the Kamaelia Web Server
• Tools for working with SQL Databases.
• Improved Multicore support (We can already do multicore, but it requires merge into the core, and how does it affect things like backplanes and all the various components.)
  
• Extensions to the co-ordinating assistant tracker :
• Modify the co-ordinating assistant tracker to use the minimal STM code
• Make co-ordinating assistant tracker environments inheritable
• Change the topology visualiser to work in full 3D
  
• Extend Kamaelia's Dirac support to include support for the optimsed version of Dirac called Schoedinger
• Platform specific improvements
  
• Improve Kamaelia's support on Windows (XP and Vista primarily), including how to package up Kamaelia Apps for it. (eg the ER Modelling tool)
  
• Improve Kamaelia's support on Mac OS X
• See how much of Kamaelia can run on IronPython 2
  
• Explore how the changes to the way components can be defined can be handled and introspected by Compose.
  

This list isn't exhaustive by any stretch of the imagination, and whether or not you're interested in our projects or someone elses, have a good summer of code!

As the next step forward, I now have multiprocess support for basic pipeline. Essentially I have the equivalent of "self.link" implemented for multiprocesses. Kinda like the Postman in the MiniAxon tutorial. It's a little messy right now, but works, and does NOT require any changes to existing Kamaelia components. It builds on LikeFile from last years summer of code efforts and pprocess. The core code currently looks like this:

# setup code snipped, and specifically ProcessWrapComponent snipped

from Kamaelia.UI.Pygame.Text import TextDisplayer, Textbox
# Start the pygame based input in one process
X = ProcessWrapComponent(Textbox(position=(20, 340),
                                 text_height=36,
                                 screen_width=900,
                                 screen_height=400,
                                 background_color=(130,0,70),
                                 text_color=(255,255,255)))

# Start the pygame based output in another
Y = ProcessWrapComponent( TextDisplayer(position=(20, 90),
                                        text_height=36,
                                        screen_width=400,
                                        screen_height=540,
                                        background_color=(130,0,70),
                                        text_color=(255,255,255)) )

exchange = pprocess.Exchange()
chan1 = X.activate()
chan2 = Y.activate()

exchange.add(chan1)
exchange.add(chan2)

while 1:
    for chan in exchange.ready(0):
        if chan == chan1:
            D = chan._receive()
            if D[1] == "outbox":
                chan2._send( (D[0], "inbox") )
    time.sleep(0.1)

Clearly most of this would be wrapped up for reuse. The nice thing of course about this, once wrapped for reuse, is that splitting a program across multiple boundaries for reuse will be pretty trivial and simple, and any kamaelia program that wants to do this will find it pretty trivial to implement. Now the nice thing about the example above is that it uses pygame. In pygame, because it builds on SDL, you can only have one window open in your process. Therefore to have 2 windows, you must be using multiple processes. Now as you can see above, the pygame components require no change, and yet this DOES result in multiple windows.

Screenshots:
In this screenshot you see the Textbox on the left and the TextDisplayer on the right. (Both were written as part of summer of code last summer incidentally). You'll see I got a bit excited, but you'll also see a piece of text before I press return.

This screenshot is after I press return:

Note that the data has passed seamlessly from one to the other, across process boundaries and probably CPU boundaries too since this is running on a multicore machine. Here's the ps output:

top - 00:11:41 up 11:51, 13 users,  load average: 1.09, 0.72, 0.66
Tasks: 149 total,   4 running, 145 sleeping,   0 stopped,   0 zombie
Cpu(s): 83.0%us, 16.5%sy,  0.0%ni,  0.3%id,  0.0%wa,  0.2%hi,  0.0%si,  0.0%st
Mem:   2066996k total,  1605500k used,   461496k free,   243348k buffers
Swap:  1959920k total,        0k used,  1959920k free,   789804k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 7727 zathras   25   0 53740  13m 6516 R   96  0.7   0:16.31 MultiPipeline.p
 7729 zathras   25   0 52616  12m 6444 R   93  0.6   0:15.83 MultiPipeline.p

OK, these components are decidedly hogging CPU as well, but as you can see they are both able to do so, and are running happily concurrently and in a not much more awkward way than normal. It's also only more awkward than normal due to the fact I haven't yet implemented the syntactic sugar required to make this work happy...

The multiprocess/core code for the brave is here in subversion.

The future's here. The future's multicore.

Well, I travelled all the way up to barcamp this weekend, and it was well worth it. Whilst I was there I gave a talk called "Sociable Software", as a general over arching description of how Kamaelia deals with concurrency. I'm going to write this up as a proper blog entry as well shortly, but in the meantime I thought adding the slides here would be a good idea.