Sparks

Updates: Websites, Actor Flow & Long COVID

Website and (side) Project Updates

The side projects referred to here are exclusively things worked on on my own time, not work. There may be some cross over with work interests, but I don’t be blogging about those here any time soon.

2024/02/18

So, it’s mid-february already. Time has somewhat flown ver the past 3 months. It also feels like many more than 3 months. Such is the nature of life in a pandemic.

I’ve not posted any personal updates on here regarding the pandemic, but the fact there are large gaps should tell you things - life has been difficult and hard. It continues to be difficult and hard.

In the midst of this, life has to go on and part of that is picking up the open source projects and their wider goals. I’m not going to make predictions about things going forward. However, reviewing how things have moved forward, and talking about recent changes to open projects seems like a good thing.

Last July, I posted about the past few years, and talked about some things on my mind then. These included:

  • Actor-like systems (kamaelia, guild, etc)
  • COVID & LongCOVID
  • Modern C++

That led to me looking into Modern C++ to see hoow coroutine support had come on.

Interestingly, these things are still on my mind. This affected what I did last summer, and affected what I did over the christmas period. There’s also something from summer 2022 which I’ll be blogging about here too: miniguild

Let’s work backwards.

Christmas 2020 / Winter 24 Updates

Sparkslabs Website Updates

The most visible change is the extra blogposts. The extra posts being:

  • A repository for storing fragments of code from blogposts. This is handy because it allows the build environments (if any) to be clearly defined in there without affecting an projects the posts rely upon.
  • This came about because it was a nice clean adjunt to the tutorial/benchmarks series of posts around C++ & Python coroutines.
  • Posts musing about C++20, and Modern C++ more generally
  • A post on kubernetes (with a draft post about vagrant/kubernetes sitting in my drafts folder at the moment)

Design back then…

Less obviously - when this blog first started - back in 2006 - the web was a simpler, younger time. It wasn’t actually my first blog - that was my 3rd year project “web log” back in 1997, but this is the longest period of consistent set of posts that I’ve kept visible. Between ’97 and ’06 I used a variety of tools, mainly hand crafted. I then wrote a new engine which I used from 2006 to 2011,

Bear in mind in 2006, none of us used Facebook. Phones were pretty dumb. Neither Android (2008) nor iPhone (2007) were launched or available (dates are product availability dates). Heck, even github [2008] didn’t exist. Git wasn’t mainstream. We all still largely used subversion…

In 2011, I started using “blogofile” which was a nice relatively new, relatively lightweight blogging tool. The blogofile based blog was pretty good and stood me in good stead for many years, but it was fragile to update, and not a very modern look. I mean it was relatively pretty and there’s some aspects - like the archive which I intend to port forward to this iteration, but it’s definitely pre-mobile web. It was also very much a design of it’s time - one based on column grids were very much in vogue, and very much the way to do things. Then. Take a look at 960 if you don’t believe me :-)

This all now looks old.

Current design

The new blog was one I wrote over the christmas period of 2020 to 2021. It was small and lightweight. It took files which were written in markdown and used a couple of libraries (python-markdown and pygments) for basic markup. It also tried to have a “not god-awful” experience on mobile. In hindsight, I’m not convinced that it wasn’t terrible, but it was definitely better than before.

In the summer of 2022 - while there’s no posts here about it, I switched that out the custom rendering in favour of shelling out to pandoc for content generation. That was probably the smartest decision I’ve taken in recent years with regard to the blog.

It fundamentally led to the current design which is this:

  • Markdown source files, suitable for passing directly to pandoc
  • Simplified templates focussed on semantic markdown rather than structural.
  • Flexbox enhanced layout, based on em’s, viewport metrics, and so on.
  • Responsive design - such that on mobile has a relatively compact view to make content more readable with less scrolling, and on larger screens a more natural layout.
  • Better use of CSS with proper media selectors to enable the responsive design, and specifically including a print version for PDF snapshots or just printing.

Visually, it’s not changed dramatically for the desktp view, but on mobile it’s far better than before. Given that that’s where huge amounts of actual reading happen now, this is somewhat overdue.

I’m quite pleased with this right now. In 5 years I’ll probably look back at this and say “what were we thinking”, but at least it’s OK for now.

I’ve been pondering syndicating content to medium for a while. That might happen it might not.

The code is pretty nasty now, and needs refactoring. What’s more likely is it’ll get merged with some of my other software. For now though, it’ll do.

Kamaelia Website

Yep, the Kamaelia Website. Over the christmas period and since (evenings/weekends) I’ve been expending signifcant effort on modernising the Kamaelia website. This is far from done, and continues to be a work in progress.

Kamaelia Website History

The Kamaelia website has gone through a fair number of iterations since it’s first public peek out the door in very late 2004. Yep, just over 19 years ago, The earliest versions were, to say the least, somewhat hacky. The more recent version that’s been in place uses that “modern” column based layout. It stood us in good stead while Kamaelia was an active project both externally and internally at work.

However, Kamaelia switched into maintenance mode around 2012-2013 or so. There were a number of reasons for this. One key one was that an alternate approach to the same ideas “Guild” was proving actively useful. By focussing on a syntactic sugar first approach it made some things quicker and simpler. Yes they wouldn’t be scalable, but the aim of Kamaelia (and guild) was first and foremost on making concurrency manageable in situations where you can’t avoid concurrency. (I always assumed I’d get around to optimisations later - after all the C++ miniaxon proved that was possible)

Kamaelia was based on python 2, and pulled in lots of libraries. While much of the core of Kamaelia still functions correctly, there is definitely “bit-rot” caused by the wide python ecosystem either moving on to python 3 or itself going into bitrot. Some services which Kamaelia has components for connecting to - like AIM closed their doors some years ago. However, the small nature of components lends to stability. For example the core network server code hasn’t needed any minor change since June 2009. The last significant/major change to that code was in 2006.

The last release of Kamaelia was back in 2010, and it was a very stable release. We were still actively using Kamaelia for internal projects for a few years after this. For example the Social Bookmarks app was built around then. The Whiteboard saw greater use for a while around then too. Waypoint was probably the last significant project built using Kamaelia, and was quickly thrown together to support a “playful IOT futures”.

But time marches on. I started experimenting with new ideas using Guild. After a handful of deadends, it stared proving itself useful. One early place it got used with was with the BBC micro:bit prototype. It wasn’t doing anything particularly interesting but it was smoothing spikes in client requests for compilation of children’s python code to C++, in the form of a batch-compiler.

An aside: Dissemination, papers, “Actor-like” systems

Note that some of my interests now listed above, included:

  • Actor-like systems (kamaelia, guild, etc)

Over the course of the pandemic I have read an awfully large number of papers. The vast majority of these involve COVID, LongCOVID, current research findings, hypotheses and research towards treatments. While there aren’t any really effective treatments for LongCOVID right now, I have learnt an awful lot about how to do effective academic paper based research.

In the July 2022, I figured out a way of translating guild’s syntactic sugar to work with C++20. At that point, I realised there was a real opportunity to move things forward.

That summer I realised also that the best way to make this happen would be to do what we did back in the day with Kamaelia. Specifically, to help people get up to speed we got then to work through the MiniAxon tutorial. This allowed them to build their own version. This also led to the creation of a C++ MiniAxon and a ruby based MiniAxon.

It struck me that if there was a “miniguild” python based tutorial, that it would form a ladder of steps to follow when writing a C++ based miniguild. So that miniguild got written that summer.

I raised this at work, but while it seemed potentially useful, there were always higher priority things, and there remain higher priority things there. The ideas have been influential at work, but we agreed late last year that the idea of a C++ based miniguild being developed there is now extremely unlikely. That’s cool, because it frees the idea up to be developed back where it belongs - in the guild repo. (Or at least starting there)

Along the way, I had to explain the concepts in Kamaelia, Guild and similar to others who’d never used the tool/library. Given my newfound academic paper research skills, I decided to apply them to this domain of actor-like, CSP-like libraries targetted at making concurrency easier to use. Specifically things designed around outbox decoupling, late binding, an integrated concurrency safe key value store (now based on STM) and so on.

Despite having now read more than my fair share of papers in that arena I came to the conclusion that what Kamaelia did, what Guild builds upon, was actually fairly novel. It struck me that actually formally documenting what we did with Kamaelia, what we discovered, how it works and so on in the form of peer reviewed papers would be an actively useful thing to do.

After all, the ideas in Kamaelia, Guild and Miniguild continue to grow more relevant with time, and inherent stability in the code is in itself a compelling reason to talk about it more formally.

However, after reading literally dozens of papers on actors, and various other concurrency models, I’m convinced of one thing: Kamaelia isn’t the Actor Model. It’s actor like - but as Ted Leung picked out in his survey of concurrency constructs back in a 2009 keynote at an O’Reilly conference, Kamaelia does do something different and is perhaps best viewed as an actor-variant.

The Actor-Flow Model

I’m pondering calling this model an Actor-Flow Model. (It allows me to also categorise other systems that fall into this category - such as command line shell scripting)

In particular, things both Kamaelia and Guild do:

  • Aim to make concurrency easier to work with, especially for problems which are inherently concurrent

  • Use outboxes and inboxes for explicit decoupling. In implementation terms this is often optimised, but the concept is core to both.

  • Both still use a simple stackless single frame generator as the unit of concurrency. This is embedded in a class to provide context between invocations and to enable external communications.

  • The idea of systems as a communication graph (graphline) or pipeline in simplecases where content can be both pushed and pulled.

  • Use a key-value store (these days in the form of an STM) to enable efficient safe sharing of values that make more sense to be in a repository for querying rather than forcing everything to be comms based.

  • Aim to be debuggable.

Guild extends this to include:

  • The idea of actor-methods, Actor functions, Actor exceptions
  • Syntactic sugar to make things more palatable for experienced developers.

Updates to the Kamaelia Website

With all this in mind, it makes sense to consider writing a paper or two relating to Kamaelia. Sure there were publications in Linux Format UK, Linux Magazine DE, BBC R&D whitepapers, technotes, and lots of talks at conferences, but not formal papers. This really prevented the ideas seeing wider dissemination and take up. (Not that I knew then when younger)

More to the point, an ecosystem of tools built based around the miniguild ideas could be actively extremely useful in life.

I also have some personal (academic) ambitions which one day I’d like to fulfill and these would really require being able to point at publications in this field.

So, with that in mind, suppose papers do get written about Kamaelia, what’s the first thing I would do if I was reading them? I’d go and see if I can find the code, see what runs and see if the arguments are valid. However, if you went to the Kamaelia website in recent years, it would look pretty, but face significant breakage.

Fixing the Kamaelia website strikes me as important for ensuring credibility for the underlying technologies.

So that’s what I’ve been doing. As a short overview:

  • The Kamaelia codebase auto-generated documentation tools have been updated. While the site itself is still “waiting” for these changes to hit the site, there is now a first cut of a 440 page API reference manual to Axon and Kamaelia.
  • Code that doesn’t currently run due to dependency issues (eg things relying on pymedia) now warn the user and refuses to run. The number of libraries affected by this is lower than expected.
  • The old site used to managed by a even older wiki. That wiki hasn’t been used in many years, but was briefly brought back into service locally to generate simple unstyled HTML content without any page furniture. That has been used to generate the first cut of a markdown verson of everything.
  • That markdown has been sanitised. The site content reorganised. The page templates are now based on semantic descriptions. The structure is created/rendered based on semantic CSS, rather than the legacy columns layout. The site looks as close as reasonable to old layout for now. However, this opens the door to a cleaner, much nicer site.
  • The actual content is now rendered using pandoc - leading to greater site consistency.
  • The major broken links (caused by the wiki engine break when a server disappeared) are now fixed.
  • A number of sections have been renamed to what they really are.
  • Semantic versioning is in the process of being introduced.

Along the way, I’ve been thinking of potential interoperability enhancements.

Based on progress to date, there’s probably a couple more months of work to get the site and library “up to date” but it’s a world of improvement relative to even just 6 months ago.

Additionally, I asked for the BBC’s opensource link to Kamaelia to be updated, and this very kindly was done.

One aspiration I do have going forward is a way of auto-wrapping Kamaelia components as Guild components/actors. This would then mean that the time/effort expended in back in the day for this code can continue to be leveraged.

However, before that is getting everything back to a known good level, documented, usable and finally publishable (if there’s merit)

… and that’s the point of the exercise :-)

Summer/Autumn 2023 Updates

Modern C++, Coroutines

Kubernetes/Vagrant

At least one other post is in draft, but coming at some point. (I’ll update this page then since the drafts predate this page!)

Summer 2022 Updates

MiniGuild

Again there is a draft post which I could just publish but I want to refine that first.

There will be a miniguild tutorial forthcoming at some point that builds on this. This may actually go hand in hand with a C++ implementation. That’s yet to be decideded though.

The C++ experiments done that summer would be worth capturing too, so hopefully I’ll get a chance to do that. But at least they’re linked here in case I don’t!

COVID & LongCOVID

Over the past 3.5 - 4 years I’ve amassed quite a collection of papers and references to LongCOVID. I’m intending to reuse the tools created in managing my blog and the Kamaelia website to cross link, cross corellate and document what we know for certain about COVID and how it causes LongCOVID and other sequelae.

What LongCOVID does to indviduals - and how long it causes problems - is horrific.

It spans from the earliest days of the pandemic. Many long haulers infected in March 2020 are still suffering today nearly 4 years on. Vaccines wouldn’t exist for another 6 months or so. Vaccinations would not until 2021. Thankfully vaccination does reduce incidence of LongCOVID. However, over 2 million in the UK are dealing with LongCOVID (based on last available stats).

The numbers becoming long haulers is around 250K per wave. ie 250K - 750K extra long haulers per year. This is driving mass disablement. It’s showing up in ONS stats. It’s a significant drain on personal and national livelihoods.

LongCOVID is very much something I am likely to blog about going forwards - and likely to be mainly based around summarising peer reviewed papers…

… More to come

And there’s likely to be more to come. But this is probably enough for now. When it’ll come? Who knows. But it will :-)

Hopefully this does pique interest and any comments, as always, very welcome.

Updated: 2024/02/18 23:00:00