Disabling Comments etc
February 04, 2011 at 10:21 AM | categories: python, oldblog | View Comments
In preparation for a complete replacement of this site, I'm disabling comments and posting after this one. The UI that this site uses has been interesting, but now shows it's age and creaking, people have complained about the editor I put in place for comments (even though it's just a vanilla dojo toolkit editor) and some minor bugs on the server side (which never affected security). The focus of this blog was really to test of a bunch of ideas, some good, some bad. Next iteration will be nicer :-)
Kamaelia 1.0.12.0 Released
December 28, 2010 at 06:43 PM | categories: python, oldblog | View Comments
I'm happy to announce Kamaelia's 4th release of 2010: 1.0.12.0 (Y.Y.M.r) Kamaelia is a component system based around unix-like concurrency/composition & pipelining. There's a strong focus on networked multimedia systems.
Kamaelia's license changed earlier this year to the Apache 2.0 License.
The release is divided up as follows:
Website:
http://www.kamaelia.org/Home.html
Source:
http://code.google.com/p/kamaelia
Tutorial:
http://www.kamaelia.org/PragmaticConcurrency.html
Detail of changes:
http://groups.google.com/group/kamaelia/browse_frm/thread/db45646ce1790233
Download:
http://www.kamaelia.org/release/MonthlyReleases/Kamaelia-1.0.12.0.tar.gz
Overview of Changes in this release:
Mailing list:
http://groups.google.com/group/kamaelia
Have fun :-)
Kamaelia's license changed earlier this year to the Apache 2.0 License.
The release is divided up as follows:
- Axon - the core component framework. Provides safe and secure message based concurrency & composition using generators as limited co-routines, threads, experimental process based support, and software transactional memory. Includes examples.
- Kamaelia - A large Ol' Bucket of components, both application specific and generic. Components vary from network systems, through digital tv, graphics, visualisation, data processing etc. These reflect the work and systems that Kamaelia has been used to build. Includes examples.
- Apps - A collection of applications built using Kamaelia. Whilst Kamaelia includes a collection of examples, these are either releases of internal apps or exemplars created by contributors.
- Bindings - a collection of bindings we maintain as part of Kamaelia, including things like DVB bindings. (Bindings recently changed over to using Cython to make life simpler)
Website:
http://www.kamaelia.org/Home.html
Source:
http://code.google.com/p/kamaelia
Tutorial:
http://www.kamaelia.org/PragmaticConcurrency.html
Detail of changes:
http://groups.google.com/group/kamaelia/browse_frm/thread/db45646ce1790233
Download:
http://www.kamaelia.org/release/MonthlyReleases/Kamaelia-1.0.12.0.tar.gz
Overview of Changes in this release:
- This rolls up (primarily) 3 application and examples branches. The core functionality for these, as ever, is in the main Kamaelia.Apps namespace, meaning these applications and examples are designed for inclusion or extraction into other applications relatively easily. As a result they act as exemplars for things like 3D visualisation, video and audio communications, twitter mining, database interaction and analysis and django integration. They're also useful (and used) as standalone apps in their own right.
- Examples (and application components) added for using the 3D graph visualisation (PyOpenGL based) - one based on visualising collaborations, another based on viewed FOAF networks.
- Whiteboard application extended such that:
- It supports multiway video comms as well as multiway audio comms.
- Adds support for "decks" (collections of slides which can be downloaded, saved, loaded, emailed, encrypted, etc)
- Removes pymedia dependency
- Change audio over to us PyAlsaAudio directly.
- Adds support for calibrated touch screen displays to Pygame Display.
- For example large digital whiteboards in addition to tablets etc.
- Adds in a "Social Bookmarking system" that does the following:
- Harvests a semantic web/RDF data store for realtime search terms (relating to live television broadcast)
- Uses these search terms to search twitter, to identify conversations around the semantic web data.
- Takes the resulting tweets, and stores them in a DB
- Analyses the tweets (including fixing language for analysis using NLTK) for a variety of aspects, storing these in the DB
- Presents the results (graphs of buzz/popularity around the content)
- Additionally the system attempts to identify particularly interesting/notable moments based on audience conversations, and provides links back to the actual broadcast programmes.
- Additionally provides an API for data, generates word clouds etc.
- Front end uses Django and web graph APIs to presnet data.
Mailing list:
http://groups.google.com/group/kamaelia
Have fun :-)
Europython 2010 Videos Now Online
August 14, 2010 at 01:55 PM | categories: python, oldblog | View Comments
Just a brief note to say that all the Europython videos I and helpers recorded are now uploaded and online on blip.tv. Since not everyone is subscribed to mailing lists, please find below the list/summary that I sent to the list. If anyone has any objections to their talk being up, please note I will
take it down. However, please note - it's there because it's great that
you were willing to stand up and talk!
Also, the real thanks have to go to John Pinner, Richard Taylor and Alex Wilmer (and the rest of the crew), without whom this years' Europython wouldn't've been the same. Likewise the same can be said about the many speakers willing to step forward and talk about something they love. And many thanks to Marijn, Richard and Walter too for their fantastic help in producing these videos :-)
Also, the real thanks have to go to John Pinner, Richard Taylor and Alex Wilmer (and the rest of the crew), without whom this years' Europython wouldn't've been the same. Likewise the same can be said about the many speakers willing to step forward and talk about something they love. And many thanks to Marijn, Richard and Walter too for their fantastic help in producing these videos :-)
Europython Community -- Awards Ceremony -- Thankyou Christian Tismer
http://europythonvideos.blip.tv/file/3980661/
Adewale Oshineye TDD on App Engine
http://europythonvideos.blip.tv/file/3980732/
Ali Afshar Glashammer
http://europythonvideos.blip.tv/file/3980733/
Andrew Godwin Fun with Django and Databases
http://europythonvideos.blip.tv/file/3980734/
Austin Bingham Python from the Inside Out
http://europythonvideos.blip.tv/file/3980760/
Bart Demeulenaere Pyradiso
http://europythonvideos.blip.tv/file/3980758/
Bruce Lawson Keynote Open Standards democratising and future proofing the web
http://europythonvideos.blip.tv/file/3980759/
Conference Opening Comments Housekeeping
http://europythonvideos.blip.tv/file/3980755/
Daniel Roseman Advanced Django ORM techniques
http://europythonvideos.blip.tv/file/3980796/
David Read Open Data and coding data.gov.uk
http://europythonvideos.blip.tv/file/3980804/
Denis Bilenko gevent network library
http://europythonvideos.blip.tv/file/3980788/
Donald McCarthy Python and EDA
http://europythonvideos.blip.tv/file/3980791/
Europython Community Awards Ceremony 4 Thanking Christian DryRun
http://europythonvideos.blip.tv/file/3980789/
Geoffrey Bache PyUseCase Testing Python GUIs
http://europythonvideos.blip.tv/file/3980793/
Guido van Rossum Appstats
http://europythonvideos.blip.tv/file/3980802/
Guido van Rossum Keynote
http://europythonvideos.blip.tv/file/3980864/
Henrik Vendelbo Real Time Websites with Python
http://europythonvideos.blip.tv/file/3980849/
Holger Kregel py.test rapid multipurpose testing
http://europythonvideos.blip.tv/file/3980855/
Jonathan Fine JavaScript 4 Pythonistas
http://europythonvideos.blip.tv/file/3980863/
Jonathan Fine JavaScript and MillerColumns
http://europythonvideos.blip.tv/file/3980845/
Jonathan Hartley Hobbyist OpenGL from Python
http://europythonvideos.blip.tv/file/3980846/
Kit Blake Mobi a mobile user agent lib
http://europythonvideos.blip.tv/file/3980862/
Kit Blake Web mobile templating in Silva
http://europythonvideos.blip.tv/file/3980853/
Lennart Regebro Porting to Python 3
http://europythonvideos.blip.tv/file/3980935/
Marc-Andre Lemburg Running Ghana VAT on Python
http://europythonvideos.blip.tv/file/3980937/
Mark Fink Visualizing Software Quality
http://europythonvideos.blip.tv/file/3980933/
Mark Shannon HotPy A comparison
http://europythonvideos.blip.tv/file/3980963/
Matteo Malosio Python Arduino and Mech Music
http://europythonvideos.blip.tv/file/3980961/
Michael Brunton-Spall Open Platform The Guardian API 1 Year On
http://europythonvideos.blip.tv/file/3980959/
Michael Brunton-Spall The Guardian and Appengine
http://europythonvideos.blip.tv/file/3980962/
Michael Foord Unittest New And Improved Part 1 of 2
http://europythonvideos.blip.tv/file/3980985/
Michael Foord Unittest New And Improved Part 2 of 2
http://europythonvideos.blip.tv/file/3980990/
Michael Sparks Arduino and Python
http://europythonvideos.blip.tv/file/3980998/
Nicholas Tollervey Organise a Python code dojo
http://europythonvideos.blip.tv/file/3980996/
Nicholas Tollervey Understanding FluidDB
http://europythonvideos.blip.tv/file/3980986/
Paul Boddie et al Web SIG
http://europythonvideos.blip.tv/file/3981027/
Paul Boddie Web Collaboration and Python
http://europythonvideos.blip.tv/file/3981026/
Peter Howard Aerodynamics and Pianos
http://europythonvideos.blip.tv/file/3981030/
PyPy Status and News Part Overview 1 of 3
http://europythonvideos.blip.tv/file/3981017/
PyPy Status and News Part JIT Compilation 2 of 3
http://europythonvideos.blip.tv/file/3981028/
PyPy Status and News Part cpyext 3 of 3
http://europythonvideos.blip.tv/file/4000720/
Raymond Hettinger Code Clinic 1 of 3
http://europythonvideos.blip.tv/file/4000722/
Raymond Hettinger Code Clinic 2 of 3
http://europythonvideos.blip.tv/file/4000757/
Raymond Hettinger Code Clinic 3 of 3
http://europythonvideos.blip.tv/file/4000761/
Raymond Hettinger Tips and Tricks 1 of 2
http://europythonvideos.blip.tv/file/4000758/
Raymond Hettinger Tips and Tricks 2 of 2
http://europythonvideos.blip.tv/file/4000752/
Richard Barrett Small The Trojan Snake
http://europythonvideos.blip.tv/file/4000778/
Richard Jones Keynote State of Python
http://europythonvideos.blip.tv/file/4000785/
Rob Collins Introduction to SMTPP
http://europythonvideos.blip.tv/file/4000771/
Russel Winder Keynote
http://europythonvideos.blip.tv/file/4000777/
Scott Wilson Flatland Form Processing
http://europythonvideos.blip.tv/file/4000782/
Semen Trygubenko Python and Machine Learning
http://europythonvideos.blip.tv/file/4000783/
Soeren Sonnenburg SHOGUN
http://europythonvideos.blip.tv/file/4000805/
Stefan Schwazer Robust Python Programs
http://europythonvideos.blip.tv/file/4000817/
Steve Holden Awards Ceremony 1 shaky
http://europythonvideos.blip.tv/file/4000798/
Steve Holden Awards Ceremony 2 PSF Community Service Award
http://europythonvideos.blip.tv/file/4000800/
Steve Holden Awards Ceremony 3 Frank Willison Award
http://europythonvideos.blip.tv/file/4000802/
Tomasz Walen Grzegorz Jakacki Codility Testing coders
http://europythonvideos.blip.tv/file/4000812/
Wesley Chun Programming Office with Python
http://europythonvideos.blip.tv/file/4000816/
Zeth What does it all mean
http://europythonvideos.blip.tv/file/4000813/
Lightning Talks:
A better pdb
http://europythonvideos.blip.tv/file/3980837/
Albertas upicasa
http://europythonvideos.blip.tv/file/3980891/
Andreas Klockner PyCUDA
http://europythonvideos.blip.tv/file/3980893/
Ariel Ben Yehuda cfg.parser
http://europythonvideos.blip.tv/file/3980900/
BidForFix.com
http://europythonvideos.blip.tv/file/3980882/
Brett Cannon How to properly package your apps front end code
http://europythonvideos.blip.tv/file/3980888/
Brian Brazil Pycon Ireland
http://europythonvideos.blip.tv/file/3980870/
Care Team Network
http://europythonvideos.blip.tv/file/3980871/
Conference Close
http://europythonvideos.blip.tv/file/3980889/
Ed Crewe From Shell Scripting to Config Mangagement
http://europythonvideos.blip.tv/file/3980890/
Experiences from Python Barcamp Cologne
http://europythonvideos.blip.tv/file/3980898/
Fiona Burrows Write More Games
http://europythonvideos.blip.tv/file/3980894/
Headroid Arduino Robot Face
http://europythonvideos.blip.tv/file/3980892/
Heres What I Think hwit.org
http://europythonvideos.blip.tv/file/3980902/
Jonathan Fine The easiest quiz in the world
http://europythonvideos.blip.tv/file/3980886/
Jonathan Hartley Run Snake Run
http://europythonvideos.blip.tv/file/3980897/
Laurens Van Houtven Python + E == Mont-E
http://europythonvideos.blip.tv/file/3980872/
Luke Leighton A Cry For Help
http://europythonvideos.blip.tv/file/3980885/
Luke Leighton Pyjamas
http://europythonvideos.blip.tv/file/3980869/
Magic Folder File Syncing
http://europythonvideos.blip.tv/file/3980881/
Marc-Andre Lemburg Growing the PSF
http://europythonvideos.blip.tv/file/3980884/
Martijn Faassen How to Fail at Pyweek
http://europythonvideos.blip.tv/file/3980879/
Michael Brunton Spall Python Javascript and Ruby in half an hour
http://europythonvideos.blip.tv/file/3980877/
Michael Sparks Embracing Concurrency
http://europythonvideos.blip.tv/file/3980901/
Moin Moin 2.0
http://europythonvideos.blip.tv/file/3980887/
Monstrum and Mercurial For Legions
http://europythonvideos.blip.tv/file/3980880/
Plone Conference
http://europythonvideos.blip.tv/file/3980895/
Porting Skynet to Python 3
http://europythonvideos.blip.tv/file/3980876/
Pure Python Proxying httplib and urllib2
http://europythonvideos.blip.tv/file/3980874/
Python Status Information 1 of 2
http://europythonvideos.blip.tv/file/3980883/
Python Status Information 2 of 2
http://europythonvideos.blip.tv/file/3980896/
Richard Jones PyWeek
http://europythonvideos.blip.tv/file/3980873/
Richard Jones The Cheese Shop
http://europythonvideos.blip.tv/file/3980867/
Richard Taylor
http://europythonvideos.blip.tv/file/3980899/
Sarah Mount Open Ihm Richard Jones With Gui
http://europythonvideos.blip.tv/file/3980922/
Steve Holden PSF
http://europythonvideos.blip.tv/file/3980917/
ThePythonGameBook.com
http://europythonvideos.blip.tv/file/3980919/
Unladen Swallow
http://europythonvideos.blip.tv/file/3980923/
Zero 14
http://europythonvideos.blip.tv/file/3980925/
Other (aborted) lightning talks:
20100722-2LT-10
http://europythonvideos.blip.tv/file/3980847/
20100722-2LT-5
http://europythonvideos.blip.tv/file/3980850/
20100722-2LT-6
http://europythonvideos.blip.tv/file/3980851/
20100722-LT-11
http://europythonvideos.blip.tv/file/3980854/
Share and Enjoy :-)
If you were 7 again...
June 27, 2010 at 06:28 PM | categories: python, oldblog | View Comments
If you were 7 again, what would you expect to find in a book on beginning programming? I have some thoughts on this, and going to do this, but I'm curious to the thoughts of others.
Python Magazine is dead ?
March 20, 2010 at 09:08 PM | categories: python, oldblog | View Comments
For the past several months the python magazine hasn't sent any new issues out. Indeed, since late last year they ripped out their website and had a banner saying "We're busy building a new python magazine", with a link laughably suggesting that there is more information available. This is after last year them getting several months behind with the magazine. They also said "
Don't worry—your subscription and back issues are safe and will be available when the new site launches.
". That's fine, in theory. However consider:
Since they've now reneged on their half of the sale contract and not delivered, and actually have a good reason to need to get in contact with them, I can't. This means I'm left with 2 choices:
Finally, I get that it's a small publication, that it's one borne out of love, rather than profit (at a guess based on guestimates of costs), but if you're having trouble getting things back started, at least have the decency to tell your subscribers, rather than having content free "information" pages.
After all, a lot can change in 1/2 a year... (Last issue I have is from August 2009...)
(Sorry to anyone who reads this who have nothing to with the python magazine, but if you know someone there, please let me know who is "running" it these days)
- Whilst they may let the grass grow under their feet they haven't bothered telling their subscribers. Paid subscribers.
- They haven't bothered updating their website telling their customers what they're doing.
- Indeed, they appear, from a subscriber point of view, to have simply cut and run.
Since they've now reneged on their half of the sale contract and not delivered, and actually have a good reason to need to get in contact with them, I can't. This means I'm left with 2 choices:
- Either put out a public notice in the hope that it's something that someone there will read, and actually get back in contact to let me know how to contact them
- Contact Visa and say that they're a rogue trader, and that they should be banned from making any further transactions against my card (especially given the last one was done without my explicit consent.
Finally, I get that it's a small publication, that it's one borne out of love, rather than profit (at a guess based on guestimates of costs), but if you're having trouble getting things back started, at least have the decency to tell your subscribers, rather than having content free "information" pages.
After all, a lot can change in 1/2 a year... (Last issue I have is from August 2009...)
(Sorry to anyone who reads this who have nothing to with the python magazine, but if you know someone there, please let me know who is "running" it these days)
Kamaelia components from decorated generators. Pythonic concurrency?
October 04, 2009 at 10:52 PM | categories: python, oldblog | View Comments
A few months ago, there was a thread on the then google group python-concurrency about some standard forms for showing how some libraries deal with concurrent problems. The specific example chosen looked like this:
I quite liked the example for describing how to take this and convert it into a collection of kamaelia components, primarily because by doing so we gain a number of reusable components in this way. For me it was able describing how to move from something rather ad-hoc to something somewhat more generally usable.
For me, the point about Kamaelia is really that it's a component framework aimed at making concurrent problems more tractable & maintainable. Basically so that I can get stuff done quicker, that won't need rewriting completely to use concurrency, which someone else can hack on without needing to come back to me to understand it. In practice though, this also means that I tend to focus on building stuff, rather than asking "is it concurrent?". (Axon kinda ensures that it either is, or is concurrent friendly) This does sometimes also mean I focus on getting the job done, rather than "does this look nice"... Whilst that does matter to me, I do have deadlines like the next person :-)
For example, one thing missing from the above is that when you do something like:
Naturally, being pleased with my version, I blogged about it. Much like anyone else, when I write something it seems like a good idea at the time :-). As sometimes happens, it made it onto reddit with some really nice & honest comments.
And what were those comments? If I had to summarise in one word "ugh!"
Whilst I don't aim for pretty (I aim for safe/correct :), pretty is nice, and pretty is fun. As a result, I've wanted to come back to this.Ugly is no fun :-( . Fun matters :-)
There was also a comment that suggested using decorators to achieve the same goal. However, at that point in time I had a mental block about what that would look like in this case. So I just thought "OK, I agree, can't quite see how to do it". I did recognise though that they're right to say that decorators would improve this case.
In particular the stumbling block is the way python generators are used in the above example is effectively a one way chaining. printer pulls values from grep. grep pulls values from follow. When one of them exits, they all exit. Essentially this is pull based.
In Kamaelia, components can be push, pull or push & pull. Furthermore they can push and pull in as many directions as you need. At the time mapping between the two sensibly it didn't seem tractable to me. Then this morning, as I woke blearily, I realised that the reason why. Essentially the above generator form isn't really directly the same as the shell form - though it is close.
Taking grep, for example, if I do this:
However, if I do this:
In essence, the differences about what's happening here are subtle - in the shell we pass in a symbol which represents which stream needs opening, whereas in the example above, we pass in, effectively, an open stream. Also the shell is very much a combination of push and pull, whereas the generator pipeline above is essentially pull.
This made me realise that rather than activating the generator we want to read from *outside* the generator we're piping into, if we activate the generator *inside* the generator we're piping into, the problem becomes tractable.
For example, if we change this:
Making this small transform allows the above example to be rewritten as kamaelia components like this:
The other nice thing about this approach is that suppose you wanted to define your own generator source like this:
Stepping back, and answering the "why? What does this give you?" question, it becomes more apparent as to why this might be useful when you start monitoring 5 log files at once for POST requests. For example, putting that all together in a single file would look like this:
(assuming you didn't reuse existing components :)
Since this will be going into the next release of Axon - any feedback - especially on naming - would be welcome :-).
(Incidentally, follow/grep have already been added to kamaelia, so this would really be simpler, but it does make an interesting example IMO :-)
#!/bin/shPete Fein also posted an example of this using generators, based on David Beazley's talk on python generators being used as (limited) coroutines:
tail -f /var/log/system.log |grep pants
import timeThe question/challenge raised on the list was essentially "what does this look like in your framework or system?". For some reason, someone saw fit to move the mailing list from google groups, and delete the archives, so I can't point at the thread, but I did repost my answer for what was called "99 bottles" for kamaelia on the python wiki .
import re
def follow(fname):
f = file(fname)
f.seek(0,2) # go to the end
while True:
l = f.readline()
if not l: # no data
time.sleep(.1)
else:
yield l
def grep(lines, pattern):
regex = re.compile(pattern)
for l in lines:
if regex.match(l):
yield l
def printer(lines):
for l in lines:
print l.strip()
f = follow('/var/log/system.log')
g = grep(f, ".*pants.*")
p = printer(g)
for i in p:
pass
I quite liked the example for describing how to take this and convert it into a collection of kamaelia components, primarily because by doing so we gain a number of reusable components in this way. For me it was able describing how to move from something rather ad-hoc to something somewhat more generally usable.
For me, the point about Kamaelia is really that it's a component framework aimed at making concurrent problems more tractable & maintainable. Basically so that I can get stuff done quicker, that won't need rewriting completely to use concurrency, which someone else can hack on without needing to come back to me to understand it. In practice though, this also means that I tend to focus on building stuff, rather than asking "is it concurrent?". (Axon kinda ensures that it either is, or is concurrent friendly) This does sometimes also mean I focus on getting the job done, rather than "does this look nice"... Whilst that does matter to me, I do have deadlines like the next person :-)
For example, one thing missing from the above is that when you do something like:
tail -f /var/log/system.log |grep pantsYou aren't interested in the fact this uses 3 processes - tail, grep & parent process - but the fact that by writing it like this you're able to solve a problem quickly and simply. It also isn't particularly pretty, though I personally I view the shell version as rather elegant.
Naturally, being pleased with my version, I blogged about it. Much like anyone else, when I write something it seems like a good idea at the time :-). As sometimes happens, it made it onto reddit with some really nice & honest comments.
And what were those comments? If I had to summarise in one word "ugh!"
Whilst I don't aim for pretty (I aim for safe/correct :), pretty is nice, and pretty is fun. As a result, I've wanted to come back to this.Ugly is no fun :-( . Fun matters :-)
There was also a comment that suggested using decorators to achieve the same goal. However, at that point in time I had a mental block about what that would look like in this case. So I just thought "OK, I agree, can't quite see how to do it". I did recognise though that they're right to say that decorators would improve this case.
In particular the stumbling block is the way python generators are used in the above example is effectively a one way chaining. printer pulls values from grep. grep pulls values from follow. When one of them exits, they all exit. Essentially this is pull based.
In Kamaelia, components can be push, pull or push & pull. Furthermore they can push and pull in as many directions as you need. At the time mapping between the two sensibly it didn't seem tractable to me. Then this morning, as I woke blearily, I realised that the reason why. Essentially the above generator form isn't really directly the same as the shell form - though it is close.
Taking grep, for example, if I do this:
grep "foo" somefileThen grep will open the file "somefile", read it, and output lines that match the pattern and exit.
However, if I do this:
bla | grep "foo"Then grep will read values from stdin, and output lines which match the pattern. Furthermore, it will pause outputting values when bla stops pushing values into the chain, and exit when bla exits (after finishing processing stdin). ie It essentially has two modes of operating, based on getting a value or having an absent value.
In essence, the differences about what's happening here are subtle - in the shell we pass in a symbol which represents which stream needs opening, whereas in the example above, we pass in, effectively, an open stream. Also the shell is very much a combination of push and pull, whereas the generator pipeline above is essentially pull.
This made me realise that rather than activating the generator we want to read from *outside* the generator we're piping into, if we activate the generator *inside* the generator we're piping into, the problem becomes tractable.
For example, if we change this:
def grep(lines, pattern):To this:
regex = re.compile(pattern)
for l in lines: # Note this requires an activate generator, or another iterable
if regex.match(l):
yield l
def grep(lines, pattern):We gain something that can operate very much like the command line grep. That is, it reads from its equivalent to stdin until stdin is exhausted. To indicated stdin is exhausted it simply yields - ie yields None. The caller can then go off and get more data to feed grep. Alternatively the caller can shutdown this grep at any point in time by throwing in an exception.
"To stop this generator, you need to call it's .throw() method. The wrapper could do this"
regex = re.compile(pattern)
while 1:
for l in lines(): # Note we activate the generator here inside instead
if regex.search(l):
yield l
yield
Making this small transform allows the above example to be rewritten as kamaelia components like this:
import sysThe implementation for both decorators.py and example.py above can both be found here:
import time
import re
import Axon
from Kamaelia.Chassis.Pipeline import Pipeline
from decorators import blockingProducer, TransformerGenComponent
@blockingProducer
def follow(fname):
"To stop this generator, you need to call it's .throw() method. The wrapper could do this"
f = file(fname)
f.seek(0,2) # go to the end
while True:
l = f.readline()
if not l: # no data
time.sleep(.1)
else:
yield l
@TransformerGenComponent
def grep(lines, pattern):
"To stop this generator, you need to call it's .throw() method"
regex = re.compile(pattern)
while 1:
for l in lines():
if regex.search(l):
yield l
yield
@TransformerGenComponent
def printer(lines):
"To stop this generator, you need to call it's .throw() method"
while 1:
for line in lines():
sys.stdout.write(line)
sys.stdout.flush()
yield
Pipeline(
follow('/var/log/system.log'),
grep(None, ".*pants.*"),
printer(None)
).run()
http://code.google.com/p/kamaelia/source/browse/trunk/Sketches/MPS/AxonDecorators/Similarly, if we wanted to use multiple processes, we could rewrite that final pipeline like this:
from Axon.experimental.Process import ProcessPipelineSpecifically the above will use 4 processes. One container process, and 3 subprocesses. (ProcessPipeline would benefit from a rewrite using multiprocess rather than pprocess though)
ProcessPipeline(
follow('/var/log/system.log'),
grep(None, ".*pants.*"),
printer(None)
).run()
The other nice thing about this approach is that suppose you wanted to define your own generator source like this:
def source():You could use that instead of "follow" above like this:
for i in ["hello", "world", "game", "over"]:
yield i
Pipeline(For me, this has a certain symmetry with the change from this
grep(source, ".*pants.*"),
printer(None)
).run()
tail somefile.txt | grep ".*pants.*" | cat -to this:
grep ".*pants.*" source | cat -ie if you pass in an absent value, it processes the standard inbox "inbox", rather than stdin. If you pass in a value, it's assumed to be a generator that needs activating.
Stepping back, and answering the "why? What does this give you?" question, it becomes more apparent as to why this might be useful when you start monitoring 5 log files at once for POST requests. For example, putting that all together in a single file would look like this:
(assuming you didn't reuse existing components :)
import sysNow, I don't particularly like the word pythonic - maybe it is, maybe it isn't - but hopefully this example does look better than perhaps than last time! The biggest area needing work, from my perspective, in this example is the names of the decorators.
import time
import re
import Axon
from Kamaelia.Util.Backplane import Backplane, SubscribeTo, PublishTo
from Kamaelia.Chassis.Pipeline import Pipeline
from decorators import blockingProducer, TransformerGenComponent
@blockingProducer
def follow(fname):
f = file(fname)
f.seek(0,2) # go to the end
while True:
l = f.readline()
if not l: # no data
time.sleep(.1)
else:
yield l
@TransformerGenComponent
def grep(lines, pattern):
regex = re.compile(pattern)
while 1:
for l in lines():
if regex.search(l):
yield l
yield
@TransformerGenComponent
def printer(lines):
while 1:
for line in lines():
sys.stdout.write(line)
sys.stdout.flush()
yield
Backplane("RESULTS").activate()
for logfile in ["com.example.1", "com.example.2", "com.example.3","com.example.4","com.example.5"]:
Pipeline(
follow(logfile+"-access.log"),
grep(None, "POST"),
PublishTo("RESULTS")
).activate()
Pipeline(
SubscribeTo("RESULTS"),
printer(None)
).run()
Since this will be going into the next release of Axon - any feedback - especially on naming - would be welcome :-).
(Incidentally, follow/grep have already been added to kamaelia, so this would really be simpler, but it does make an interesting example IMO :-)
Restarting Python Northwest. 24th Sept
September 14, 2009 at 11:36 PM | categories: python, oldblog | View Comments
A few people will have already noticed some small comments about this, but we're plotting to restart python northwest. Specifically, we're restarting this month.
Details:
How did this happen? I tweeted the idea, a couple of others seconded it, the David Jones pointed out "it easier to arrange for a specific 2 people to meet than it was to e-mail a vague cloud of people and get _any_ 2 to meet anywhere.", so that's where we'll be.
If twitter feedback is anything go by, we're hardly going to be alone, so please come along - the more the merrier :-) Better yet, please reply to this post saying you're coming along!
More generally, assuming this continues, pynw will probably be every third thursday in the month, maybe alternating between technical meets and social ones. (probably topic for discussion :-)
Please forward this to anyone you think may be interested!
See you there!
Details:
- When: Thursday 24th September, 6pm
- Where: Rain Bar, Manchester, UK:
- http://www.rain-bar.co.uk/
- 80 Great Bridgewater Street, Manchester, M1 5JG
- map: http://tinyurl.com/mn4qnl
- Who: Who can come? If you're reading this YOU can (assuming you're sufficiently close :-)
More specifically anyone from beginners, the inexperienced through deeply experienced and all the way back to the plain py-curious.
- What: Suggestion is to start off with a social meet, and chat about stuff we've found interesting/useful/fun with python recently. Topics likely to include robots and audio generation, the recent unconference, and europython.
- Twitter: http://twitter.com/pynw / #pythonnorthwest / #pynw
- Mailing list: http://groups.google.com/group/python-north-west
How did this happen? I tweeted the idea, a couple of others seconded it, the David Jones pointed out "it easier to arrange for a specific 2 people to meet than it was to e-mail a vague cloud of people and get _any_ 2 to meet anywhere.", so that's where we'll be.
If twitter feedback is anything go by, we're hardly going to be alone, so please come along - the more the merrier :-) Better yet, please reply to this post saying you're coming along!
More generally, assuming this continues, pynw will probably be every third thursday in the month, maybe alternating between technical meets and social ones. (probably topic for discussion :-)
Please forward this to anyone you think may be interested!
See you there!
Traffic Server to be Open Source?!
July 07, 2009 at 12:02 PM | categories: python, oldblog | View Comments
If this happens this will be awesome. Traffic Server is some really nice code. It's a large codebase, but it's really cool, and it *scales*. (I used to work at Inktomi, so have been inside the code as well). For those that don't know what it is, it's a very high performance web caching proxy, with a plugin architecture, allowing for the addition of other protocols. It used to support HTTP (& obvious friends), NNTP, RTSP, RTP, WMV, etc.
That's pretty much made my day that has.
That's pretty much made my day that has.
Europython Videos Transcoding
July 07, 2009 at 01:02 AM | categories: python, oldblog | View Comments
Since I've had a few questions about this, a short status update. At Europython last week I was recording all the talks I was attending. Including the lightning talks this means I have video from 55 talks. The video files from the camera are too large for blip.tv, so I'm transcoding them down to a smaller size, before uploading them. Since these 55 talks are spread over nearly 80 files, that naturally takes time.
Fortunately/obviously, I'm automating this, and it'll come as no shock to some that I'm automating it using kamaelia. This automation needs to to be stoppable, since I need to only do this overnight, for practicality reasons.
Anyway, for those curious, this is the code I'm using to do the transcode & upload. You'll note that it saturates my CPU, keeping both cores busy. Also, it's interleaving an IO bound process (ftp upload) with CPU bound - transcode.
It should be fairly clear that this will go as fast as it can, so please be patient :-)
Fortunately/obviously, I'm automating this, and it'll come as no shock to some that I'm automating it using kamaelia. This automation needs to to be stoppable, since I need to only do this overnight, for practicality reasons.
Anyway, for those curious, this is the code I'm using to do the transcode & upload. You'll note that it saturates my CPU, keeping both cores busy. Also, it's interleaving an IO bound process (ftp upload) with CPU bound - transcode.
import os
import re
import Axon
from Kamaelia.Chassis.Graphline import Graphline
from Kamaelia.Chassis.Pipeline import Pipeline
class Find(Axon.Component.component):
path = "."
walktype = "a"
act_like_find = True
def find(self, path = ".", walktype="a"):
if walktype == "a":
addfiles = True
adddirs = True
elif walktype == "f":
addfiles = True
adddirs = False
elif walktype == "d":
adddirs = True
addfiles = False
deque = []
deque.insert(0, (os.path.join(path,x) for x in os.listdir(path)) )
while len(deque)>0:
try:
fullentry = deque[0].next()
if os.path.isdir(fullentry):
if adddirs:
yield fullentry
try:
X= [os.path.join(fullentry,x) for x in os.listdir(fullentry)]
deque.insert(0, iter(X))
except OSError:
if not self.act_like_find:
raise
elif os.path.isfile(fullentry):
if addfiles:
yield fullentry
except StopIteration:
deque.pop(0)
def main(self):
gotShutdown = False
for e in self.find(path = self.path, walktype=self.walktype):
self.send(e, "outbox")
yield 1
if self.dataReady("control"):
gotShutdown = True
break
if not gotShutdown:
self.send(Axon.Ipc.producerFinished(), "signal")
else:
self.send(self.recv("control"), "signal")
class Sort(Axon.Component.component):
def main(self):
dataset = []
while 1:
for i in self.Inbox("inbox"):
dataset.append(i)
if self.dataReady("control"):
break
self.pause()
yield 1
dataset.sort()
for i in dataset:
self.send(i, "outbox")
yield 1
self.send(self.recv("control"), "signal")
class Grep(Axon.Component.component):
pattern = "."
invert = False
def main(self):
match = re.compile(self.pattern)
while 1:
for i in self.Inbox("inbox"):
if match.search(i):
if not self.invert:
self.send(i, "outbox")
else:
if self.invert:
self.send(i, "outbox")
if self.dataReady("control"):
break
self.pause()
yield 1
self.send(self.recv("control"), "signal")
class TwoWayBalancer(Axon.Component.component):
Outboxes=["outbox1", "outbox2", "signal1","signal2"]
def main(self):
c = 0
while 1:
yield 1
for job in self.Inbox("inbox"):
if c == 0:
dest = "outbox1"
else:
dest = "outbox2"
c = (c + 1) % 2
self.send(job, dest)
job = None
if not self.anyReady():
self.pause()
if self.dataReady("control"):
break
R=self.recv("control")
self.send(R, "signal1")
self.send(R, "signal2")
class Transcoder(Axon.ThreadedComponent.threadedcomponent):
command = 'ffmpeg >transcode.log 2>&1 -i "%(SOURCEFILE)s" -s 640x360 -vcodec mpeg4 -acodec copy -vb 1500000 %(ENCODINGNAME)s'
def main(self):
while 1:
for sourcefile in self.Inbox("inbox"):
shortname = os.path.basename(sourcefile)
encoding_name = shortname.replace(".mp4", ".avi")
finalname = sourcefile.replace(".mp4", ".avi")
# Do the actual transcode
print "TRANSCODING", sourcefile, encoding_name
os.system( self.command % {"SOURCEFILE": sourcefile, "ENCODINGNAME":encoding_name})
# file is transcoded, move to done
print "MOVING DONE FILE", sourcefile, os.path.join("done", sourcefile)
os.rename(sourcefile, os.path.join("done", sourcefile))
# Move encoded version to upload queue
upload_name = os.path.join( "to_upload", encoding_name)
print "MOVING TO UPLOAD QUEUE", encoding_name, upload_name
os.rename(encoding_name, upload_name )
# And tell the encoder to upload it please
print "SETTING OFF UPLOAD",upload_name, finalname
self.send( (upload_name, finalname), "outbox")
print "-----------------"
if self.dataReady("control"):
break
self.send(self.recv("control"), "signal")
class Uploader(Axon.ThreadedComponent.threadedcomponent):
command = "ftpput --server=%(HOSTNAME)s --verbose --user=%(USERNAME)s --pass=%(PASSWORD)s --binary --passive %(UPLOADFILE)s"
username = < editted :-) >
password = < editted :-) >
hostname = "ftp.blip.tv"
def main(self):
while 1:
for (upload_name, finalname) in self.Inbox("inbox"):
print "UPLOADING", upload_name
os.system( self.command % {
"HOSTNAME":self.hostname,
"USERNAME":self.username,
"PASSWORD":self.password,
"UPLOADFILE":upload_name,
} )
print "MOVING", upload_name, "TO", os.path.join("encoded", finalname)
os.rename(upload_name, os.path.join("encoded", finalname))
print "-----------------"
if self.dataReady("control"):
break
if not self.anyReady():
self.pause()
self.send(self.recv("control"), "signal")
Graphline(
FILES = Pipeline(
Find(path=".",walktype="f"),
Sort(),
Grep(pattern="(done|encoded|unsorted|transcode.log|to_upload)",
invert = True),
),
SPLIT = TwoWayBalancer(), # Would probably be nicer as a customised PAR chassis
CONSUME1 = Pipeline(
Transcoder(),
Uploader(),
),
CONSUME2 = Pipeline(
Transcoder(),
Uploader(),
),
linkages = {
("FILES","outbox"):("SPLIT","inbox"),
("SPLIT","outbox1"):("CONSUME1","inbox"),
("SPLIT","outbox2"):("CONSUME2","inbox"),
("FILES","signal"):("SPLIT","control"),
("SPLIT","signal1"):("CONSUME1","control"),
("SPLIT","signal2"):("CONSUME2","control"),
}
).run()
It should be fairly clear that this will go as fast as it can, so please be patient :-)
Autoloading in python
June 21, 2009 at 04:14 PM | categories: python, oldblog | View Comments
Before I started using python, I'd used perl for several years, and one thing which I'd liked about perl was its autoload facility. Now in python the closest equivalent that I've seen is __getattr__ for classes, but not __getattr__ for a module. This seemed like a real shame since there are times when autoload can be incredibly useful.
If it seems chaotic, consider the Unix PATH variable. Any time you type a name, the shell looks in lots of locations and runs the first one it finds. That's effectively the same sort of idea as autoloading. Yes, you can do some really nasty magic if you want, but then you can do that with the shell to, and generally people get along find.
Anyway, vaguely curious about it I decided to do some digging around, and came across this post by Leif K Brookes, which suggests this:
Now the reason I'd been thinking about this, is because I'd like to retain the hierarchy of components in Kamaelia that we have at the moment (it's useful for navigating what's where), but given we tend to use them in a similar way to Unix pipelines it's natural to want to be able to do something like:
Now, I haven't done this yet, and wouldn't do it lightly, but the fact that you can actually make autoload functionality work, seems kinda cool, and and a nice opportunity. But I'm also now wondering just how nasty this approach seems to people. After all, Leif describes it as "a bit of a hack", and whilst it's neat, I'm not taking in the positive view. I'm interested in any views on better ways of doing Autoload in python, and also whether people view it as a nice thing at all. (One person's aesthetic is another person's pain after all...)
If it seems chaotic, consider the Unix PATH variable. Any time you type a name, the shell looks in lots of locations and runs the first one it finds. That's effectively the same sort of idea as autoloading. Yes, you can do some really nasty magic if you want, but then you can do that with the shell to, and generally people get along find.
Anyway, vaguely curious about it I decided to do some digging around, and came across this post by Leif K Brookes, which suggests this:
You could wrap it in an object, but that's a bit of a hack.That looked reasonable, so I created a file mymod.py which looks like this:
import sys
class Foo(object):
def __init__(self, wrapped):
self.wrapped = wrapped
def __getattr__(self, name):
try:
return getattr(self.wrapped, name)
except AttributeError:
return 'default'
sys.modules[__name__] = Foo(sys.modules[__name__])
import sysAnd tried using it like this:
def greet(greeting="Hello World"):
print greeting
class mymod_proxy(object):
def __init__(self):
super(mymod_proxy, self).__init__()
self.wrapped = sys.modules["mymod"]
def __getattr__(self, name):
try:
return getattr(self.wrapped, name)
except AttributeError:
def f():
greet(name)
return f
sys.modules["mymod"] = mymod()
~> pythonAnd as you can see, it seems to work as expected/desired.
Python 2.5.1 (r251:54863, Jan 10 2008, 18:01:57)
[GCC 4.2.1 (SUSE Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import mymod
>>> mymod.hello()
hello
>>> from mymod import Hello_World
>>> Hello_World()
Hello_World
Now the reason I'd been thinking about this, is because I'd like to retain the hierarchy of components in Kamaelia that we have at the moment (it's useful for navigating what's where), but given we tend to use them in a similar way to Unix pipelines it's natural to want to be able to do something like:
from Kamaelia import Pipeline, ConsoleReader, ConsoleWriterRather than the more verbose form specifically pulling them in from particular points. Likewise, we don't really want to import every single module in Kamaelia.py, because of the large number of components there (Kamaelia is really a toolbox IMO where things get wired together, and Axon is the tool for making new tools), the majority of which won't be used in ever application!
Pipeline(
ConsoleReader(),
ConsoleWriter(),
).run()
Now, I haven't done this yet, and wouldn't do it lightly, but the fact that you can actually make autoload functionality work, seems kinda cool, and and a nice opportunity. But I'm also now wondering just how nasty this approach seems to people. After all, Leif describes it as "a bit of a hack", and whilst it's neat, I'm not taking in the positive view. I'm interested in any views on better ways of doing Autoload in python, and also whether people view it as a nice thing at all. (One person's aesthetic is another person's pain after all...)
Next Page »