No sooner do you put a new release out the door and you're reviewing things and planning the next. Since there seems to be a general trend towards numbering releases based on date rather than on (equally arbitrary) release numbers, we've decided to follow that trend. However, we've also planned to fill in the gaps to figure out how to manage shorter release cycles. Also, I don't want to artificially jump from version <1 to version 8, since it would very much say the wrong things.

When we hit version 1.0 I really want certain things in the system which aren't fully there yet, though most of the necessary spikes are in place). As a result, rather than using the versioning of YY.MM, we're going to move to using Y.Y.MM by the end of the year. That implies we hit version 1.0 in early 2010, which is actually only about 14/15 months off, so that seems reasonable. (hence the reason for switching to date based releases)

This means our next release will be 0.7.0, followed by 0.8.12. From then on we'll be moving to 6 weekly release candidate cycles, with a decision made at the beginning of each cycle whether that release candidate will be targetted as a full release, but there will definitely be release candidates. Given the delay between 0.5.0 and 0.6.0, jumping to a 6 weekly cycle seems challenging, so that's part of the reason for aiming for 2 releases by the end of the year (to get into the habit :).

As a result we're currently planning what's going into 0.7.0 now, with a planned feature merge freeze weekend 22nd November, and planned release date of weekend ending 30th Nov. That is intended to be followed up in late december by 0.8.12.

I've started a discussion on the mailing list about these, but the current planned focus for 0.7.0 follows.

• SuSE packaging for Axon & Kamaelia updated from 0.5.0 to K0.6.0 & A1.6.0
• And future packaging changes automated, ideally. (enabling 0.7.0 easily)
• Better graphline shutdown as discussed on the list.
• Tyson's extended version of the file appender,
• Merged of Chong's Google Summer of code project - 3D visualisation work (since this is something I actually need sooner rather than later)
• Packaging up of the whiteboard (pdf) as a standalone app
• Packaging up of the batch file processor (image/video transcoder) as a standalone app (or better packaging than this)
• A mirror of the Ben's Kamaelia AWS code into Kamaelia.Apps.AWS, if it's at a stage where it's ready. (aside from reviews etc)
• Merge of Jason's google summer of code on on extensions/improvements to the HTTP server code, including a better example of the seaside like functionality. (I think Kamaelia Publish itself should probably wait until 0.8.12 or 0.9.X) Probably after it gets a clearer name.
• Perhaps initial work on integration of the STM code into the CAT. (The what? The STM and CAT tools were discussed at pycon uk) Though I suspect this will get started now, and merged in 0.8.12
• 2.6 cleanups (probably based on hinting from 2to3), and work started on a 3.0 autoconversion/build system.
• Other stuff I'd like to see includes : work started on rationalising Kamaelia.File, Full review and merge of UDP_ng code in place of current UDP code, basic "connected" UDP server support (perhaps) (ie such that it can be used as a drop in replacement for TCPServer in ServerCore)

Any suggestions and/or improvements or offers of help welcome :) Whether all this is doable in the time frame is a little speculative at present, but as the project's reaching maturity, it seems appropriate to take a more predictable approach towards releases, even if a release focus is "just" bug fixes, or documentation :)

With great pleasure I'd like to announce the release of Kamaelia 0.6.0 and Axon 1.6.0

Release overview

For the short of time:

Overview: library/framework for concurrency using message passing components as the concurrency metaphor. Consists of a kernel (Axon) and collection of components (Kamaelia). Support for generator, thread & process based components. Targetted towards supporting maintenance, so /accessible/ to novices, but general purpose. Uses concurrency to make life simpler, not harder. Designed as a practical toolkit.

Download:

http://www.kamaelia.org/GetKamaelia
http://www.kamaelia.org/release/Kamaelia-0.6.0.tar.gz
sudo easy_install Kamaelia  

Change Summary:

Major update, multicore, STM, easy concurrency, creation of Kamaelia.Apps namespace for reuse of applications, significant amounts of new functionality, major documentation improvements (including full offline reference docs), support for using Kamaelia components cleanly in non-Kamaelia apps. (ie a clean linear -> concurrent interface (Handle))

Release notes: http://www.kamaelia.org/ReleaseNotes060
Deltas: Kamaelia 0.5.0 -> 0.6.0, Axon 1.5 -> Axon 1.6.0
Last full release: October 2006

Mailing list: http://groups.google.com/group/kamaelia *CHANGED*
New website: http://www.kamaelia.org/Home
Reference: http://www.kamaelia.org/Components
Cookbook: http://www.kamaelia.org/Cookbook

Detailed Version...

What is Kamaelia?

Kamaelia is a library/framework for building systems from simple components that talk to each other. This is primarily targetted at aiding maintenance, but also assists in the creation of systems in the first place. It also means you build naturally concurrent software. It's intended to be powerful, but also accessible by any developer, including novices.

We also find it makes it fun to build and work with concurrent systems.

What sort of systems? Network servers, clients, desktop applications, pygame based games, transcode systems and pipelines, digital TV systems, spam eradicators, teaching tools, and a fair bit more :)

Whilst it comes out of a research project at BBC Research, it is designed as a /practical/ toolkit. This clearly affects the set of components we've created.

In order to do this, Kamaelia is divided into two main namespaces:

• Axon - this provides the core component & concurrency framework.  You use to build components which communicate with one another.
• Kamaelia - this is the collection of components that exist. The vast majority of these come from systems created for a variety of purposes.
  As of this release, a second major carve up of name spaces has been added:
• Kamaelia.Apps - this is where components from some Kamaelia based applications reside. The purpose behind this is to provide an experimental staging ground for new components to migrate into the main Kamaelia namespace. This also means you can use components from other Kamaelia applications sooner rather than later. As a result, these components may be lacking in two main areas - documentation or generality, but putting them here allows for components to migrate to a more generally useful state.
• Kamaelia.{anything else} - this is where components will migrate to when we are happy with the fact they are sufficiently general and useful outside their original application.
  It's worth noting that the bulk of components are in this second category!
  

What's New & Changed?

Kamaelia 0.6.0 represent a update over the 0.5.0 release, and should mark the return to regular releases. (Work has been continuing constantly since the 0.5.0 release, but numbers of releases slowed)

Major to changes reflected in both Axon & Kamaelia:

• New home/website :-)
• http://www.kamaelia.org/Home
• New getting started page:
• http://www.kamaelia.org/GetKamaelia
• Support for easy_install ...
• sudo easy_install Kamaelia
• ... but with caveats that you don't get the docs, tools, or examples that way...
• Large scale documentation improvements
• Results of nightly documentation generation now included in the tar ball.
• Core autogenerated docs:
• http://www.kamaelia.org/Docs/Axon/Axon
• http://www.kamaelia.org/Components

Key changes to Axon - Kamaelia's core:

• Bumped to version 1.6.0
• Support for simplified software transactional memory
• http://www.kamaelia.org/STM
• If you've never heard/understood the term, think (a bit like) "version control for variables"
• Useful if you MUST share data between components.
  
  
• Experimental multicore support
• http://www.kamaelia.org/MulticoreExample
• Largely boils down to put "Process" in front of "Pipeline" to make all the subcomponents of the pipeline run in seperate processes 
• Practical benefit for pygame components - it allows multiwindow pygame apps.
  
  
• Inheritable default values for component initialisers.
• The core aim of this is to allow declarative config for systems rather than something less clear.
•  This allows you to turn this sort of code:
• def ReusableSocketAddrServer(port=100,
                               protocol=EchoProtocol):
      return ServerCore(protocol=protocol,
                        port=port,
                        socketOptions=(socket.SOL_SOCKET,
                                       socket.SO_REUSEADDR, 1))
• Into this:
• class ReusableSocketAttrServer(ServerCore):
      socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
  
• Full discussion here:
• http://www.kamaelia.org/InheritableDefaultValues
  
  
• Added in "Handle" support. This provides two key pieces of functionality:
• The ability to run Kamaelia systems in the background, via:
• from Axon.background import background
  background().start()
• The ability to wrap Kamaelia components or systems in a Handle for use in non-Kamaelia systems.
• from Axon.Handle import Handle
  from Kamaelia.Internet.TCPClient import TCPClient
  conn = Handler(TCPClient("www.kamaelia.org", 80)).activate()
  conn.put("GET / HTTP/1.0\r\n", "inbox")
  conn.put("Host: www.kamaelia.org\r\n", "inbox")
  conn.put("\r\n", "inbox")
• More detail:
• http://www.kamaelia.org/AxonHandle
  
  
• Simplified system shutdown. If you want to close down an entire Kamaelia based system rapidly from inside a component, just do this:
• self.scheduler.stop()
• This puts the scheduler into a "shutting down mode" in which:
• It calls the .stop() method on all the things it's running
• It then shuts down.
• The Kamaelia.Internet components take this as an opportunity to close all the connections they have open cleanly for example.
• Example usage: http://tinyurl.com/AxonShutdown
  
  
• Support for WaitComplete extended, allowing better handling of more complex protocols which are not stateless in a debuggable fashion. It also simplifies working with Pygame, etc.
• http://www.kamaelia.org/WaitComplete
  
  
• As well as unpausing a component when a message is delivered tothe component, it gets unpaused when a message is taken from its outbox. This allows better synchronous behaviour for rate limited in/out-boxes.

Overview of Changes to Kamaelia itself

Key changes to Kamaelia itself:

• Creation of the Kamaelia.Apps namespace
• Shifting of the core code for Kamaelia tools into Kamaelia.Apps
• Significant numbers of new components
• Significant number of bugfixes
• SimpleServer code changed to "ServerCore", representing it's more general structure.
• ... and a lot more besides... :)

In this release there is a slew of extra components and bug fixes, a variety of new tools - from video shot change detection, through to SMTP greylisting, but also perhaps the biggest extra: Multiprocess & hence multicore support (experimental at this stage, but so far so good :) )

OK, here's the dive into the summary of changes. (Full release notes here)

New files in Kamaelia.*

This represents significant amounts of new components and new abilities added into Kamaelia.

• Kamaelia/
• Chassis/Seq.py
• Codec/
• WAV.py YUV4MPEG.py
• Device/DVB/
• SoftDemux.py
• Parse/
• { lots of components for working with DVB information tables }
• Experimental/
• Chassis.py ERParsing.py
• File/
• MaxSpeedFileReader.py UnixProcess2.py
• Internet/
• TimeOutCSA.py UDP_ng.py
• Protocol/
• MimeRequestComponent.py RecoverOrder.py SDP.py
• SimpleVideoCookieServer.py (demo/example)
  
• AIM/
• AIMHarness.py ChatManager.py LoginHandler.py OSCARClient.py
• HTTP/
• HTTPRequestHandler.py
• Handlers/
• Minimal.py SessionExample.py UploadTorrents.py
• IRC/IRCClient.py          
  
• RTP/
• NullPayloadPreFramer.py NullPayloadRTP.py RTCPHeader.py RTPHeader.py RtpPacker.py RTP.py
• UI/
• Pygame/
• Text.py VideoSurface.py
• Util/
• Collate.py FirstOnly.py Max.py OneShot.py PromptedTurnstile.py RangeFilter.py RateChunker.py SequentialTransformer.py Sync.py TagWithSequenceNumber.py TwoWaySplitter.py
• Tokenisation/
•             Simple.py
• Video/
• PixFormatConversion.py DetectShotChanges.py CropAndScale.py
• Visualisation/ER/
• ERLaws.py ERVisualiserServer.py ExtraWindowFurniture.py PAttribute.py PEntity.py PISA.py PRelation.py
• XML/SimpleXMLParser.py
• Support/
• OscarUtil2.py OscarUtil.py DVB/DateTime.py Protocol/IRC.py

New files in Kamaelia.Apps

Functionality in here represents code that can be standardised for use in other apps if there's a desire to do so. Many of these components are reuseable in their current form, though may have limitations.

• Kamaelia/Apps/
• Compose/
• BuildViewer.pyCodeGen.py GUI.py PipeBuild.py PipelineWriter.py
• GUI/
• ArgumentsPanel.py BuilderControlsGUI.py TextOutputGUI.py
• Games4Kids/
• BasicSprite.py MyGamesEventsComponent.py SpriteScheduler.py
• Grey/
• ConcreteMailHandler.py GreyListingPolicy.py MailHandler.py PeriodicWakeup.py Support.py WakeableIntrospector.py
• Whiteboard/
• Audio.py Canvas.py CheckpointSequencer.py CommandConsole.py Entuple.py Options.py Painter.py Palette.py Router.py Routers.py SingleShot.py TagFiltering.py Tokenisation.py TwoWaySplitter.py UI.py
• IRCLogger/Support.py
• Show/GraphSlides.py

Deleted files in Kamaelia.*

A number of files which were deprecated in the last release have been deleted from this release. (See full release notes for details)

Changed files in Kamaelia.*

Largely small improvements, changes to meta data about components, often major documentation improvements - see full release notes for details. Occasional bugfixes. Largest overall change to existing files is improvement of documentation, REsT fixes, and addition of metadata to files.

• Kamaelia/
• Audio/
• Filtering.py RawAudioMixer.py
• Codec/PyMedia/
• Decoder.py Encoder.py
• PyMedia/
• Input.py Output.py Resample.py
• Chassis/
• Carousel.py ConnectedServer.py Graphline.py Pipeline.py Prefab.py
• Codec/
•         Dirac.py RawYUVFramer.py Speex.py
• Device/
• DVB/
• Core.py DemuxerService.py EIT.py NowNext.py PSITables.py Receiver.py Tuner.py
• File/
• BetterReading.py ReadFileAdaptor.py Reading.py UnixProcess.py
• Internet/
• ConnectedSocketAdapter.py Selector.py SingleServer.py TCPClient.py TCPServer.py UDP.py
• Protocol/
• EchoProtocol.py FortuneCookieProtocol.py Framing.py SimpleReliableMulticast.py
• HTTP/
• ErrorPages.py HTTPClient.py HTTPHelpers.py HTTPParser.py HTTPResourceGlue.py HTTPServer.py IcecastClient.py MimeTypes.py
• Torrent/
• TorrentClient.py TorrentMaker.py TorrentPatron.py TorrentService.py
• UI/
• GraphicDisplay.py
• OpenGL/
• Button.py Movement.py SimpleTranslationInteractor.py OpenGLComponent.py
• Pygame/
• Button.py Display.py Image.py KeyEvent.py Multiclick.py Ticker.py VideoOverlay.py
• Util/
• Backplane.py Chooser.py Chunkifier.py ChunkNamer.py Clock.py Comparator.py ConsoleEcho.py Console.py DataSource.py Fanout.py Filter.py Introspector.py MarshallComponent.py Marshalling.py NullSink.py PassThrough.py PureTransformer.py RateFilter.py Splitter.py Stringify.py UnseenOnly.py
• Visualisation/
• Axon/
• AxonVisualiserServer.py ExtraWindowFurniture.py PComponent.py
• PhysicsGraph/
• TopologyViewer.py TopologyViewerServer.py chunks_to_lines.py GridRenderer.py lines_to_tokenlists.py RenderingParticle.py
• Kamaelia/Experimental/Services.py
• Kamaelia/Automata/Behaviours.py
• Kamaelia/Support/
• Deprecate.py
• DVB/
• CRC.py Descriptors.py
• Data/
• bitfieldrec.py Experimental.py Repository.py
• Particles/
• SpatialIndexer.py

Platforms

Kamaelia has been used successfully under both Linux, Windows and Mac OS X. (mostly developed/tested under Linux & Mac OS X)

Where can I get it? & Docs?

Download:

http://www.kamaelia.org/GetKamaelia
http://www.kamaelia.org/release/Kamaelia-0.6.0.tar.gz

Docs:

http://www.kamaelia.org/Docs/Axon/Axon
http://www.kamaelia.org/Components
http://www.kamaelia.org/Cookbook
http://www.kamaelia.org/MiniAxon

Presentations:

http://www.slideshare.net/kamaelian

Get involved: (all locations changed since last major release)

http://www.kamaelia.org/Developers/
http://groups.google.com/group/kamaelia
http://code.google.com/p/kamaelia/

Licensing

Kamaelia is released under the Mozilla tri-license scheme (MPL1.1/GPL2.0/LGPL2.1). See http://www.kamaelia.org/Licensing

Thanks & acknowledgements

Finally, many thanks to everyone who's contributed to this release, especially including: Matt Hammond, Sylvain Hellegouarch, Jinna Lei, Jason Baker, Dave King, Patrick Thomson, Ryan Lothian

As always, Feedback, improvements, corrections & suggestions as usual, very welcome :-)

Just a note for anyone in the london area - if you're interested in kamaelia or what we've learned about making concurrency easy (or much easier) to work with and useful for simplifying common tasks or want hear a talk about it, or just meet up, grab food and chat about kamaelia or python or related stuff, I'll be giving a talk at Linux Expo Live London next week at Olympia in London. Date: 23rd October, time 2:30pm.

Many thanks to Ben Ford for the heads up and release of their code using kamaelia that accesses amazon web services.  His blurb from the email:

Ben Ford wrote:
I've set up a googlecode project with access to the code I've written
to interact with Amazon Web Services here:  http://code.google.com/p/kamaelia-aws/.
I've set up source control here: http://freehg.org/u/ben/kamaelia-aws/.

It's my first time using kamaelia so I'd love to hear any feedback on
the code. Feel free to use it, and let me know what you think.

And from the two links:

Concurrent access to AWS

This project came about when we needed to integrate SQS and S3 into our web application.

It's released here to share with the community in the hope that it's useful and for anyone familiar with kamaelia to pick apart, or use for demo purposes.

What does it give you?

We have several components at varying levels of abstraction.

SQS

Components to handle passing json messages into and out of SQS

S3

Components to upload to S3, download from S3 and delete from S3.

At higher levels we have a few Graphline based components that integrate various different components with a logging module

Assuming no major issues, planning on that going in 0.7.0, due in mid november, mirror to trunk probably this weekend :-)

Interesting project: http://yieldprolog.sourceforge.net/ noting here since it's somewhere I'll come back to, and it strikes me that others would find it interesting too.

A question was raised on the Kamaelia list asking for a demo on how to nest protocols using Kamaelia. In order to show some clarity, and avoid a protocol mismatch I decided to build a protocol which is a composite of the following 3 things:

• Something clumping data into messages
• JSON data being forwarded in serialised form as those messages
• The serialised messages encrypted (naively) using DES encryption.

Rather than repost all the steps leading to that here, you can find the post that does just that here:

• http://groups.google.com/group/kamaelia/msg/387c1209ca99279f

You'd probably want to take the resulting code a little further than I've taken it if you were to use it in production, since (for example) DataChunker is designed to be used with larger & larger numbers of messages. It would be relatively simple to transform this to a pub/sub system/router for sending recieving Javascript fragments, but wanted to leave the example looking relatively simple for now. I may come back to that as a pub/sub demo.

The final example looks like this:

 

#!/usr/bin/python

import cjson
import Axon

from Kamaelia.Chassis.Pipeline import Pipeline
from Kamaelia.Util.Chooser import Chooser
from Kamaelia.Util.Console import ConsoleEchoer

from Kamaelia.Internet.TCPClient import TCPClient
from Kamaelia.Chassis.ConnectedServer import ServerCore

from Kamaelia.Protocol.Framing import DataChunker, DataDeChunker
from Kamaelia.Apps.Grey.PeriodicWakeup import PeriodicWakeup

from Crypto.Cipher import DES

messages = [ {"hello": "world" },
             {"hello": [1,2,3] },
             {"world": [1,2,3] },
             {"world": {"game":"over"} },
           ]*10

class MarshallJSON(Axon.Component.component):
    def main(self):
        while not self.dataReady("control"):
            for j in self.Inbox("inbox"):
                j_encoded = cjson.encode(j)
                self.send(j_encoded, "outbox")
            if not self.anyReady():
                self.pause()
            yield 1

class DeMarshallJSON(Axon.Component.component):
    def main(self):
        while not self.dataReady("control"):
            for j in self.Inbox("inbox"):
                j_decoded = cjson.decode(j)
                self.send(j_decoded, "outbox")
            if not self.anyReady():
                self.pause()
            yield 1

class Encoder(object):
    """Null encoder/base encoder - returns the same string
       for encode/decode"""
    def __init__(self, key, **argd):
        super(Encoder, self).__init__(**argd)
        self.__dict__.update(argd)
        self.key = key
    def encode(self, some_string):
        return some_string
    def decode(self, some_string):
        return some_string

class DES_CRYPT(Encoder):
    def __init__(self, key, **argd):
        super(DES_CRYPT, self).__init__(key, **argd)
        self.key = self.pad_eight(key)[:8]
        self.obj = obj=DES.new(self.key, DES.MODE_ECB)

    def encode(self, some_string):
        padded = self.pad_eight(some_string)
        encrypted = self.obj.encrypt(padded)
        return encrypted

    def decode(self, some_string):
        padded = self.obj.decrypt(some_string)
        decoded = self.unpad(padded)
        return decoded

    def pad_eight(self, some_string):
        X = len(some_string)
        if X % 8 != 0:
            pad_needed = 8-X % 8
        else:
            pad_needed = 8
        pad_needed = 8-(X % 8)
        PAD = pad_needed * chr(pad_needed)
        return some_string+PAD

    def unpad(self, some_string):
        x = ord(some_string[-1])
        return some_string[:-x]

class Encrypter(Axon.Component.component):
    key = "ABCD"
    def main(self):
        crypter = DES_CRYPT(self.key)
        while not self.dataReady("control"):
            for j in self.Inbox("inbox"):
                j_encoded = crypter.encode(j)
                self.send(j_encoded, "outbox")
            if not self.anyReady():
                self.pause()
            yield 1

class Decrypter(Axon.Component.component):
    key = "ABCD"
    def main(self):
        crypter = DES_CRYPT(self.key)
        while not self.dataReady("control"):
            for j in self.Inbox("inbox"):
                j_decoded = crypter.decode(j)
                self.send(j_decoded, "outbox")
            if not self.anyReady():
                self.pause()
            yield 1

def protocol(*args,**argd):
    return Pipeline(
             PeriodicWakeup(message="NEXT", interval=1),
             Chooser(messages),
             MarshallJSON(),
             Encrypter(),       # Encrypt on the way out
             DataChunker(),
        )

def json_client_prefab(ip, port):
    return Pipeline(
             TCPClient(ip, port=port),
             DataDeChunker(),
             Decrypter(),       # Decrypt on the way in
             DeMarshallJSON(),
             ConsoleEchoer(use_repr=True)
        )

ServerCore(protocol=protocol, port=2345).activate()
json_client_prefab("127.0.0.1", 2345).run()
  

Any sufficiently advanced...

The famous lisp quote - "Any sufficiently complicated C or Fortran program contains an ad hoc informally-specified bug-ridden slow implementation of half of Common Lisp" - make me think of something else today. Specifically I've been building a system where users enter data, someone else enters rules (which are data) and the system goes away and infers more data and then more things to do. That made me realise this:

"Any sufficiently complicated data rich system, which responds based on rules stored as data contains an ad hoc informally-specified bug-ridden slow implementation of half of Prolog."

Now, I'm not saying about to suddenly jump ship and just start writing prolog everywhere, but it's (to me) a useful realisation.