Actor Model vs Denial of Service
November 17, 2008 at 03:39 AM | categories: python, oldblog | View CommentsCommunications among actors occur asynchronously: that is, the sending actor does not wait until the message is received before proceeding with computation.The difference between Kamaelia and the Actor Model is in a few places, but possibly the most fundamental is this:
(this summary courtesy of the Wikipedia page on the Actor Model, but the literature supports this view as well)
- With the actor model (as I understand it)
- You have a mailbox where you receive messages. ("recv from inbox")
- communications defaults to unidirectional, though you can build bi-directional
- When you send a message, you know the recipient and send it direct to the recipient. ("send to bob")
- Message sending is asynchronous
- ie sender knows receiver, receiver does not (necessarily) know sender
- This potentially introduces coupling in ways that makes co-ordination languages harder to build
- With Kamaelia's model:
- You receive messages from multiple named inboxes (ala receiving data on stdin, or receiving signals)
- communications defaults to unidirectional, though you can build bi-directional
- You send messages to named outboxes (ala sending to stdout, stderr)
- A higher level co-ordination language (effectively) joins the dots between outboxes to inboxes (generally)
- Message sending defaults to asynchronous, but you can define a "pipewidth" or "max number of messages in transit" to allow synchrony, where needed (such as a producer that produces data faster than the consumer can consume it)
- ie sender does NOT know receiver, receiver does not (necessarily) know sender
- Much like a CPU doesn't know whether it's plugged into a motherboard or a testing harness for example.
- This defaults to needing a co-ordination language - but this encourages greater reusability of components, through a dataflow model
- I say "kamaelia's" model here, but this is pretty similar to hardware, unix pipes, etc.
- Do they do the literal, and solely asynchronous thing? (ie always return immediately without error when you send a message)
- Or do they introduce the ability to add in synchrony where necessary? (block or return an exception when a maximum inbox size is reached)
So, the question I really have is this: if you use or implement an actor model system, do you have any ability in your implementation to be able to say "maximum number of pending incoming messages" ? If you don't, then it is remarkably easy to write code that can break an actor based system by mistake, with problems in dealing with that - making code more complex, and less reusable.
I'd really be interested in hearing both positives and negatives here...
Simple Chat Server in Kamaelia
November 13, 2008 at 11:00 AM | categories: python, oldblog | View CommentsWell, the reason there wasn't an example of that up is because it's a really trivial example in Kamaelia. The most basic version for example looks like this:
from Kamaelia.Util.Backplane import Backplane, PublishTo, SubscribeToA slightly more interesting version, which at least tells clients who they're talking to, and also has slightly better (more explicit) structure is this:
from Kamaelia.Chassis.ConnectedServer import ServerCore
from Kamaelia.Chassis.Pipeline import Pipeline
Backplane("CHAT").activate() # This handles the sharing of data between clients
def ChatClient(*argv, **argd):
return Pipeline(
PublishTo("CHAT"), # Take data from client and publish to all clients
SubscribeTo("CHAT"), # Take data from other clients and send to our client
)
print "Chat server running on port 1501"
ServerCore(protocol = ChatClient, port=1501).run()
#!/usr/bin/pythonTo try this yourself:
from Kamaelia.Util.Backplane import Backplane, PublishTo, SubscribeTo
from Kamaelia.Chassis.ConnectedServer import ServerCore
from Kamaelia.Chassis.Pipeline import Pipeline
from Kamaelia.Util.PureTransformer import PureTransformer
Backplane("CHAT").activate() # This handles the sharing of data between clients
def ChatClient(*argc, **argd):
peer = argd["peer"]
peerport = argd["peerport"]
return Pipeline(
PureTransformer(lambda x: " %s:%s says %s" % (str(peer), str(peerport), str(x))),
PublishTo("CHAT"), # Take data from client and publish to all clients
# ----------------------
SubscribeTo("CHAT"), # Take data from other clients and send to our client
PureTransformer(lambda x: "Chat:"+ str(x).strip()+"\n"),
)
class ChatServer(ServerCore):
protocol = ChatClient
print "Chat server running on port 1501"
ChatServer(port=1501).run()
- sudo easy_install Kamaelia
- curl -O http://kamaelia.googlecode.com/svn/trunk/Sketches/MPS/Examples/SimpleChatServer.py
- ./SimpleChatServer.py
A nice Pygame based client for this looks like this incidentally:
from Kamaelia.Chassis.Pipeline import PipelineTo run that (assuming you have pygame installed):
from Kamaelia.UI.Pygame.Text import Textbox, TextDisplayer
from Kamaelia.Internet.TCPClient import TCPClient
Pipeline(
Textbox(size = (800, 300), position = (100,380)),
TCPClient("127.0.0.1", 1501),
TextDisplayer(size = (800, 300), position = (100,40))
).run()
- sudo easy_install Kamaelia
- curl -O http://kamaelia.googlecode.com/svn/trunk/Sketches/MPS/Examples/SimpleChatClient.py
- ./SimpleChatClient.py
Now there's clearly a lot interesting directions you can take this, but as you can see, this is a relatively simple starting point. For something more complex, there is a basic P2P splitting radio system in our subversion tree. It's just over a 100 lines long for the source side (ie capturing radio off air and serving it), and the client is a similar size (has a playback component rather than capture one). The code for those two examples is here:
The two examples actually contain a lot of common code, so we could extract the common code and have two smaller examples, but like this the files are standalone, which is pretty nice.
Anyway, the point of this post was "what sort of examples would you like to see?" and I'm really interested in any feedback :-)
Have fun :)
A post per day in November
November 02, 2008 at 02:38 PM | categories: python, oldblog | View CommentsThe reason for it being that form is because it is significantly clearer than text.
The world is big enough
October 30, 2008 at 02:35 PM | categories: python, oldblog | View Comments"I believe there is room within Python as a language, and CPython as an implementation of that language - for a virtual buffet of helpful libraries for concurrency and distributed systems."He's absolutely right - if we only have hammers, every solution involves
I'd write more, but fluey at the moment.
Jesse's comment did remind me of this though:
"The best way to have a good idea is to have a lot of ideas." Linus Pauling.It also reminds me of my personal view on things like this - the world would be very boring if we all agreed and nobody ever did anything new or different.
Bar Camp Liverpool
October 26, 2008 at 11:42 PM | categories: python, oldblog | View Commentsthe people who attend, and Pythonistas are an interesting and varied lot, hence posting this here. Key info:
- http://www.barcampliverpool.com/
- 6th-7th December
If there's interest, and if it seems appropriate, could hold a Kamaelia sprint that weekend ?
Planning Kamaelia 0.7.0 release
October 24, 2008 at 12:24 AM | categories: python, oldblog | View CommentsWhen we hit version 1.0 I really want certain things in the system which aren't fully there yet, though most of the necessary spikes are in place). As a result, rather than using the versioning of YY.MM, we're going to move to using Y.Y.MM by the end of the year. That implies we hit version 1.0 in early 2010, which is actually only about 14/15 months off, so that seems reasonable. (hence the reason for switching to date based releases)
This means our next release will be 0.7.0, followed by 0.8.12. From then on we'll be moving to 6 weekly release candidate cycles, with a decision made at the beginning of each cycle whether that release candidate will be targetted as a full release, but there will definitely be release candidates. Given the delay between 0.5.0 and 0.6.0, jumping to a 6 weekly cycle seems challenging, so that's part of the reason for aiming for 2 releases by the end of the year (to get into the habit :).
As a result we're currently planning what's going into 0.7.0 now, with a planned feature merge freeze weekend 22nd November, and planned release date of weekend ending 30th Nov. That is intended to be followed up in late december by 0.8.12.
I've started a discussion on the mailing list about these, but the current planned focus for 0.7.0 follows.
- SuSE packaging for Axon & Kamaelia updated from 0.5.0 to K0.6.0 & A1.6.0
- And future packaging changes automated, ideally. (enabling 0.7.0 easily)
- Better graphline shutdown as discussed on the list.
- Tyson's extended version of the file appender,
- Merged of Chong's Google Summer of code project - 3D visualisation work (since this is something I actually need sooner rather than later)
- Packaging up of the whiteboard (pdf) as a standalone app
- Packaging up of the batch file processor (image/video transcoder) as a standalone app (or better packaging than this)
- A mirror of the Ben's Kamaelia AWS code into Kamaelia.Apps.AWS, if it's at a stage where it's ready. (aside from reviews etc)
- Merge of Jason's google summer of code on on extensions/improvements to the HTTP server code, including a better example of the seaside like functionality. (I think Kamaelia Publish itself should probably wait until 0.8.12 or 0.9.X) Probably after it gets a clearer name.
- Perhaps initial work on integration of the STM code into the CAT. (The what? The STM and CAT tools were discussed at pycon uk) Though I suspect this will get started now, and merged in 0.8.12
- 2.6 cleanups (probably based on hinting from 2to3), and work started on a 3.0 autoconversion/build system.
- Other stuff I'd like to see includes : work started on rationalising Kamaelia.File, Full review and merge of UDP_ng code in place of current UDP code, basic "connected" UDP server support (perhaps) (ie such that it can be used as a drop in replacement for TCPServer in ServerCore)
Any suggestions and/or improvements or offers of help welcome :) Whether all this is doable in the time frame is a little speculative at present, but as the project's reaching maturity, it seems appropriate to take a more predictable approach towards releases, even if a release focus is "just" bug fixes, or documentation :)
New Release - Kamaelia 0.6.0
October 22, 2008 at 12:12 PM | categories: python, oldblog | View CommentsRelease overview
For the short of time:Overview: library/framework for concurrency using message passing components as the concurrency metaphor. Consists of a kernel (Axon) and collection of components (Kamaelia). Support for generator, thread & process based components. Targetted towards supporting maintenance, so /accessible/ to novices, but general purpose. Uses concurrency to make life simpler, not harder. Designed as a practical toolkit.Download:
http://www.kamaelia.org/GetKamaeliaChange Summary:
http://www.kamaelia.org/release/Kamaelia-0.6.0.tar.gz
sudo easy_install Kamaelia
Major update, multicore, STM, easy concurrency, creation of Kamaelia.Apps namespace for reuse of applications, significant amounts of new functionality, major documentation improvements (including full offline reference docs), support for using Kamaelia components cleanly in non-Kamaelia apps. (ie a clean linear -> concurrent interface (Handle))Release notes: http://www.kamaelia.org/ReleaseNotes060
Deltas: Kamaelia 0.5.0 -> 0.6.0, Axon 1.5 -> Axon 1.6.0
Last full release: October 2006
Mailing list: http://groups.google.com/group/kamaelia *CHANGED*
New website: http://www.kamaelia.org/Home
Reference: http://www.kamaelia.org/Components
Cookbook: http://www.kamaelia.org/Cookbook
Detailed Version...
What is Kamaelia?
Kamaelia is a library/framework for building systems from simple components that talk to each other. This is primarily targetted at aiding maintenance, but also assists in the creation of systems in the first place. It also means you build naturally concurrent software. It's intended to be powerful, but also accessible by any developer, including novices.We also find it makes it fun to build and work with concurrent systems.
What sort of systems? Network servers, clients, desktop applications, pygame based games, transcode systems and pipelines, digital TV systems, spam eradicators, teaching tools, and a fair bit more :)
Whilst it comes out of a research project at BBC Research, it is designed as a /practical/ toolkit. This clearly affects the set of components we've created.
In order to do this, Kamaelia is divided into two main namespaces:
- Axon - this provides the core component & concurrency framework. You use to build components which communicate with one another.
- Kamaelia - this is the collection of components that exist. The vast majority of these come from systems created for a variety of purposes.
As of this release, a second major carve up of name spaces has been added: - Kamaelia.Apps - this is where components from some Kamaelia based applications reside. The purpose behind this is to provide an experimental staging ground for new components to migrate into the main Kamaelia namespace. This also means you can use components from other Kamaelia applications sooner rather than later. As a result, these components may be lacking in two main areas - documentation or generality, but putting them here allows for components to migrate to a more generally useful state.
- Kamaelia.{anything else} - this is where components will migrate to when we are happy with the fact they are sufficiently general and useful outside their original application.
It's worth noting that the bulk of components are in this second category!
What's New & Changed?
Kamaelia 0.6.0 represent a update over the 0.5.0 release, and should mark the return to regular releases. (Work has been continuing constantly since the 0.5.0 release, but numbers of releases slowed)Major to changes reflected in both Axon & Kamaelia:
- New home/website :-)
- http://www.kamaelia.org/Home
- New getting started page:
- http://www.kamaelia.org/GetKamaelia
- Support for easy_install ...
- sudo easy_install Kamaelia
- ... but with caveats that you don't get the docs, tools, or examples that way...
- Large scale documentation improvements
- Results of nightly documentation generation now included in the tar ball.
- Core autogenerated docs:
- Bumped to version 1.6.0
- Support for simplified software transactional memory
- If you've never heard/understood the term, think (a bit like) "version control for variables"
- Useful if you MUST share data between components.
- Experimental multicore support
- Largely boils down to put "Process" in front of "Pipeline" to make all the subcomponents of the pipeline run in seperate processes
- Practical benefit for pygame components - it allows multiwindow pygame apps.
- Inheritable default values for component initialisers.
- The core aim of this is to allow declarative config for systems rather than something less clear.
- This allows you to turn this sort of code:
- def ReusableSocketAddrServer(port=100,
protocol=EchoProtocol):
return ServerCore(protocol=protocol,
port=port,
socketOptions=(socket.SOL_SOCKET,
socket.SO_REUSEADDR, 1)) - Into this:
- class ReusableSocketAttrServer(ServerCore):
socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) - Full discussion here:
- Added in "Handle" support. This provides two key pieces of functionality:
- The ability to run Kamaelia systems in the background, via:
- from Axon.background import background
background().start() - The ability to wrap Kamaelia components or systems in a Handle for use in non-Kamaelia systems.
- from Axon.Handle import Handle
from Kamaelia.Internet.TCPClient import TCPClient
conn = Handler(TCPClient("www.kamaelia.org", 80)).activate()
conn.put("GET / HTTP/1.0\r\n", "inbox")
conn.put("Host: www.kamaelia.org\r\n", "inbox")
conn.put("\r\n", "inbox") - More detail:
- Simplified system shutdown. If you want to close down an entire Kamaelia based system rapidly from inside a component, just do this:
- self.scheduler.stop()
- This puts the scheduler into a "shutting down mode" in which:
- It calls the .stop() method on all the things it's running
- It then shuts down.
- The Kamaelia.Internet components take this as an opportunity to close all the connections they have open cleanly for example.
- Example usage: http://tinyurl.com/AxonShutdown
- Support for WaitComplete extended, allowing better handling of more complex protocols which are not stateless in a debuggable fashion. It also simplifies working with Pygame, etc.
- As well as unpausing a component when a message is delivered tothe component, it gets unpaused when a message is taken from its outbox. This allows better synchronous behaviour for rate limited in/out-boxes.
Overview of Changes to Kamaelia itself
Key changes to Kamaelia itself:- Creation of the Kamaelia.Apps namespace
- Shifting of the core code for Kamaelia tools into Kamaelia.Apps
- Significant numbers of new components
- Significant number of bugfixes
- SimpleServer code changed to "ServerCore", representing it's more general structure.
- ... and a lot more besides... :)
OK, here's the dive into the summary of changes. (Full release notes here)
New files in Kamaelia.*
This represents significant amounts of new components and new abilities added into Kamaelia.- Kamaelia/
- Chassis/Seq.py
- Codec/
- WAV.py YUV4MPEG.py
- Device/DVB/
- SoftDemux.py
- Parse/
- { lots of components for working with DVB information tables }
- Experimental/
- Chassis.py ERParsing.py
- File/
- MaxSpeedFileReader.py UnixProcess2.py
- Internet/
- TimeOutCSA.py UDP_ng.py
- Protocol/
- MimeRequestComponent.py RecoverOrder.py SDP.py
- SimpleVideoCookieServer.py (demo/example)
- AIM/
- AIMHarness.py ChatManager.py LoginHandler.py OSCARClient.py
- HTTP/
- HTTPRequestHandler.py
- Handlers/
- Minimal.py SessionExample.py UploadTorrents.py
- IRC/IRCClient.py
- RTP/
- NullPayloadPreFramer.py NullPayloadRTP.py RTCPHeader.py RTPHeader.py RtpPacker.py RTP.py
- UI/
- Pygame/
- Text.py VideoSurface.py
- Util/
- Collate.py FirstOnly.py Max.py OneShot.py PromptedTurnstile.py RangeFilter.py RateChunker.py SequentialTransformer.py Sync.py TagWithSequenceNumber.py TwoWaySplitter.py
- Tokenisation/
- Simple.py
- Video/
- PixFormatConversion.py DetectShotChanges.py CropAndScale.py
- Visualisation/ER/
- ERLaws.py ERVisualiserServer.py ExtraWindowFurniture.py PAttribute.py PEntity.py PISA.py PRelation.py
- XML/SimpleXMLParser.py
- Support/
- OscarUtil2.py OscarUtil.py DVB/DateTime.py Protocol/IRC.py
New files in Kamaelia.Apps
Functionality in here represents code that can be standardised for use in other apps if there's a desire to do so. Many of these components are reuseable in their current form, though may have limitations.- Kamaelia/Apps/
- Compose/
- BuildViewer.pyCodeGen.py GUI.py PipeBuild.py PipelineWriter.py
- GUI/
- ArgumentsPanel.py BuilderControlsGUI.py TextOutputGUI.py
- Games4Kids/
- BasicSprite.py MyGamesEventsComponent.py SpriteScheduler.py
- Grey/
- ConcreteMailHandler.py GreyListingPolicy.py MailHandler.py PeriodicWakeup.py Support.py WakeableIntrospector.py
- Whiteboard/
- Audio.py Canvas.py CheckpointSequencer.py CommandConsole.py Entuple.py Options.py Painter.py Palette.py Router.py Routers.py SingleShot.py TagFiltering.py Tokenisation.py TwoWaySplitter.py UI.py
- IRCLogger/Support.py
- Show/GraphSlides.py
Deleted files in Kamaelia.*
A number of files which were deprecated in the last release have been deleted from this release. (See full release notes for details)Changed files in Kamaelia.*
Largely small improvements, changes to meta data about components, often major documentation improvements - see full release notes for details. Occasional bugfixes. Largest overall change to existing files is improvement of documentation, REsT fixes, and addition of metadata to files.- Kamaelia/
- Audio/
- Filtering.py RawAudioMixer.py
- Codec/PyMedia/
- Decoder.py Encoder.py
- PyMedia/
- Input.py Output.py Resample.py
- Chassis/
- Carousel.py ConnectedServer.py Graphline.py Pipeline.py Prefab.py
- Codec/
- Dirac.py RawYUVFramer.py Speex.py
- Device/
- DVB/
- Core.py DemuxerService.py EIT.py NowNext.py PSITables.py Receiver.py Tuner.py
- File/
- BetterReading.py ReadFileAdaptor.py Reading.py UnixProcess.py
- Internet/
- ConnectedSocketAdapter.py Selector.py SingleServer.py TCPClient.py TCPServer.py UDP.py
- Protocol/
- EchoProtocol.py FortuneCookieProtocol.py Framing.py SimpleReliableMulticast.py
- HTTP/
- ErrorPages.py HTTPClient.py HTTPHelpers.py HTTPParser.py HTTPResourceGlue.py HTTPServer.py IcecastClient.py MimeTypes.py
- Torrent/
- TorrentClient.py TorrentMaker.py TorrentPatron.py TorrentService.py
- UI/
- GraphicDisplay.py
- OpenGL/
- Button.py Movement.py SimpleTranslationInteractor.py OpenGLComponent.py
- Pygame/
- Button.py Display.py Image.py KeyEvent.py Multiclick.py Ticker.py VideoOverlay.py
- Util/
- Backplane.py Chooser.py Chunkifier.py ChunkNamer.py Clock.py Comparator.py ConsoleEcho.py Console.py DataSource.py Fanout.py Filter.py Introspector.py MarshallComponent.py Marshalling.py NullSink.py PassThrough.py PureTransformer.py RateFilter.py Splitter.py Stringify.py UnseenOnly.py
- Visualisation/
- Axon/
- AxonVisualiserServer.py ExtraWindowFurniture.py PComponent.py
- PhysicsGraph/
- TopologyViewer.py TopologyViewerServer.py chunks_to_lines.py GridRenderer.py lines_to_tokenlists.py RenderingParticle.py
- Kamaelia/Experimental/Services.py
- Kamaelia/Automata/Behaviours.py
- Kamaelia/Support/
- Deprecate.py
- DVB/
- CRC.py Descriptors.py
- Data/
- bitfieldrec.py Experimental.py Repository.py
- Particles/
- SpatialIndexer.py
Platforms
Kamaelia has been used successfully under both Linux, Windows and Mac OS X. (mostly developed/tested under Linux & Mac OS X)Where can I get it? & Docs?
Download:http://www.kamaelia.org/GetKamaeliaDocs:
http://www.kamaelia.org/release/Kamaelia-0.6.0.tar.gz
http://www.kamaelia.org/Docs/Axon/AxonPresentations:
http://www.kamaelia.org/Components
http://www.kamaelia.org/Cookbook
http://www.kamaelia.org/MiniAxon
http://www.slideshare.net/kamaelianGet involved: (all locations changed since last major release)
http://www.kamaelia.org/Developers/
http://groups.google.com/group/kamaelia
http://code.google.com/p/kamaelia/
Licensing
Kamaelia is released under the Mozilla tri-license scheme (MPL1.1/GPL2.0/LGPL2.1). See http://www.kamaelia.org/LicensingThanks & acknowledgements
Finally, many thanks to everyone who's contributed to this release, especially including: Matt Hammond, Sylvain Hellegouarch, Jinna Lei, Jason Baker, Dave King, Patrick Thomson, Ryan LothianAs always, Feedback, improvements, corrections & suggestions as usual, very welcome :-)
Kamaelia @ Linux Expo Live, London, Olympia 23rd Oct
October 17, 2008 at 12:48 AM | categories: python, oldblog | View CommentsKamaelia AWS (amazon web services)
October 15, 2008 at 09:57 AM | categories: python, oldblog | View CommentsAnd from the two links:Ben Ford wrote:
It's my first time using kamaelia so I'd love to hear any feedback on
I've set up a googlecode project with access to the code I've written
to interact with Amazon Web Services here: http://code.google.com/p/kamaelia-aws/.
I've set up source control here: http://freehg.org/u/ben/kamaelia-aws/.
the code. Feel free to use it, and let me know what you think.
Assuming no major issues, planning on that going in 0.7.0, due in mid november, mirror to trunk probably this weekend :-)This project came about when we needed to integrate SQS and S3 into our web application.
We have several components at varying levels of abstraction.
Components to handle passing json messages into and out of SQS
Components to upload to S3, download from S3 and delete from S3.
Interesting project
October 08, 2008 at 09:20 AM | categories: python, oldblog | View Comments« Previous Page -- Next Page »