I've made a new release of pyxie. Largely internal changes.

The focus of this release is the result of the independent intermediate node refactoring project I was undertaking. This was to transform the list oriented data structure for representing a "pure" intermediate form of code into a more stuctured/generalisable format.

The motivation behind this change was the realisation that implementation of end user functions (ie def) would be very difficult with this more structured approach.

The actual release notes (and source releases) are here:

• https://github.com/sparkslabs/pyxie/releases/tag/v0.1.25

Github repo:

• https://github.com/sparkslabs/pyxie

Package on PyPI:

• https://pypi.org/project/pyxie/

Also via PPA for Ubuntu:

• sudo add-apt-repository ppa:sparkslabs/packages
• sudo apt-get update
• apt-get install python-pyxie

What's new user facing

Before I dive into those changes, it's perhaps worth noting some of the other more minor changes, but are more noticeable from a user perspective.

• A collection of docs added in doc/
• A start on documenting the models used in Pyxie
• Licensing on pyxie's output. (Yes, it's what you want, but explicitly so)
• Some new examples focussed on specific aspects of language:
  • if
  • if-else
  • pass
  • print
  • while-break
  • while-continue
• The arduin profile has been extended to support the Adafruit_NeoPixel library
• neopixel example added

Worth noting:

• While there is an example "simplest_function" it is not expected to compile yet

Neopixel example

This is quite nice, and so since it's perhaps more interesting I'm including a snapshot here:

#include <Adafruit_NeoPixel.h>

pin = 6
number_of_pixels = 16
delayval = 500

pixels = Adafruit_NeoPixel(number_of_pixels, pin, NEO_GRB + NEO_KHZ800)

pixels.begin()

while True:
    for i in range(number_of_pixels):
        pixels.setPixelColor(i, pixels.Color(0,150,0))
        pixels.show()
        delay(delayval)

Note: This reuses the fact that #include is a comment in python. It might change at somepoint to be more like this...

from Adafruit_NeoPixel import *

...since that's logically closer, but for now this is a pragmatic approach that works.

Rationale behind changes

Before this change, pyxie used the following data structures, and code phases when parsing, analysing, transforming and performing code generation.

These were not ideal:

• Code phase: Parser - read in the source, parsed it, and created a data structure
  • Data structure: pynodes - these represent the concrete python syntax, and are generated during the code generation phase. These are python objects in a class hierarchy. Once created they are analysed and placed into a context to analyse datatypes.
• Code phase: Analysis. - Analyses the code, and decorates the existing pynodes with type information to understand the programs data and data types. (does not create a new data structure).
• Transform phase: walks the pynode CST, and generates an intermediate data structure intended to represent the program in the abstract independent of the language used. An independent intermediate form if you like.
  • Data structure: independent intermediate form - This is used to model the program in a "pure form" - which isn't constrained by the source language, and contains enough information for the target language output to be generated. That was the theory. In practice it was a nested list of list of lists ... Not ideal. More on this below.
• Code generation phase: This walked the independent intermediate form (the lists of lists), and created an output data structure representing the concrete final program.
• The third/final data structure is intended to represent the final output language. ie it is intended to be a concrete representation of the output language. This final data structure is then capable of creating the output. Again, forms a hierarchy and could be called "CppNodes" (Though they weren't before this change)

The problem here is that while the pynodes are currently well defined (to a large extent) and that the CppNodes are pretty well defined (even if they could be better), the independent intermediate form sucked because it was just nested lists. This meant in practice the code was fragile, difficult to change, and increasingly difficult to work with. In the early days of Pyxie this simplistic structure made sense. However as data analysis becomes more complex and tricky. This mirrors the fact that the same was true in the early days of the parsing side.

So the focus of this sub-project was to replace the intermediate form, and tighten up the differences and make clearer in the code base that we have PyNodes, iiNodes, and CppNodes, where:

• Pynodes - are the current structures, unchanged - note these are currently prefixed Py - PyDefStatement for example.
• CppNodes - are map to objects/hierarchy/etc that represents C++ programs, but made clearer (slightly). These are now prefixed Cpp rather than the previous C_ which some of the objects use.
• iiNodes - represent the independent, intermediate nodes. Since iiNodes need to be used in source code where there are either PyNodes + iiNodes or CppNodes + iiNodes, they have the prefix "ii" enable differentiation.

Since all 3 of these are actually models, the code for these has all moved to sit below pyxie.models.

At some point there will be a need to restructure the code more generally. Parsing, transforming to IINodes and code generation are actually all transforms, and should sit together. That was out of scope for this already relatively major internal change.

Changelog

I could include the changelog, but if you're curious about that, check out the release notes mentioned above.

Future

I'm not going to put a timeline on this - I've made that mistake in the past for pet projects but the following goals exist for future iterations:

• Code structure:
  • Consolidate the structure of CppNodes and make them "cleaner"
  • Similar for iiNodes
  • Shift type inference and variable detection from PyNodes to iiNodes
  • Change iiNodes to be more ECS based
  • Do the analysis/type inference in more of a visitor pattern (ala ECS)
• Types
  • Better implement core datatypes to allow transforms
  • Strings need implementing better
  • Aim for closest match representation of datatypes
  • Allow JSON type structures to exist
    • Implement lists
    • Implement dictionaries
  • Implement tuples
• Language/structural
  • def's with no arguments, and no local variables
  • def's with arguments
  • def's with local variables
  • classes
• Example system goals:
  • It should ideally be possible to run a perceptron style neural network on an Atmega 8A with the aim of making a robot that can follow a line based on said network.

The example system is deliberately ambitious/difficult - at least for a version of python compiled to C++ to run on such a small device.

Some parts of this will go quicker than others, and won't be done strictly in that order.

Most of it however relies upon better internal code structure. (When I started after all it wasn't clear that this would work or be a good approach, so used json objects throughout. As time goes on it becomes clearer that those ad-hoc structures are in the right sort of places, but could do with being more consistent to increase flexibility.

In order to keep releases interesting though, I'll try to add interesting examples at each stage :-)

As usual comments welcome - especially with regard to what examples you might find interesting.

Over the past several months during evenings and weekends I started work on a new python to C++ compiler -- called Pyxie (it's a play on words :). It's currently reached version 0.0.16, and is just beginning to be usable for vaguely fun things.

It's not intended to be a full implementation of python, but rather a "Little Python" that is simple enough to be compiled to C++. "Little Python" will be a well defined subset of python that can be compiled. The reason for this is to allow me to write code in python that compiles to run on embedded systems like Arduino, MSP430 and ARM mbed platforms.

This makes it different from Micropython - which runs a python interpreter on the microcontroller, and PyMite which effectively runs a simplified python VM. Both of those projects assume a device with more memory. Due to the constraints of these devices, the compiler is unlikely to ever support the entirety of the python language.

PLEASE NOTE It is not complete, and almost certainly won't do what you want, yet. If you're looking for a general python to C++ compiler - though not for arduino - take a look at ShedSkin. It is just beginning to be useful though for fun little things with an Arduino (or similar devices based on an Atmel Atmega 32U4), hence this post.

What job does it do?

Pyxie's job is to allow me to write python code, instead of C++ such that it runs on a microcontroller, and to do so efficiently. The reason I want this is to (eventually) use this with cubs and scouts.

• I quite like the idea of describing software in terms of the job it does. Since I came across the approach, I find it clarifies the purpose of a project dramatically.

Website

Pyxie has a site: http://www.sparkslabs.com/pyxie/

It's a work in progress and will become nicer and shinier as Pyxie itself gets nicer and shinier.

Release Schedule

There's no particular release schedule. I try not to have releases that are too large, or too far apart. What's planned to go into a release is done on a release by release basis. What actually has gone in is a reflection of what I've been up to in the evenings/weekends of any given week.

Name

The name is a play on words. Specifically, Python to C++ - can be py2cc or pycc. If you try pronouncing "pycc" it can be "pic", "py cc" or pyc-c". The final one leads to Pixie.

Target Functionality

The guiding principle is that Little Python should look and act like a version of python that was never written. (After all python 1.x, 2.0, 2.5, 3.x, 3.5 are all "python" but simply different iterations) You could almost say that Little Python should look like a simplified "ret-con" version of python.

At present a non-exhaustive high level list of things that are targetted are:

• Duck typing / lack of type declarations (but strong types)
• Whitespace for indentation
• Standard control structures (No else clauses on while/for :) )
• Standard built in types
• Lists and dictionaries
• Easy import/use of external functionality
• User functions (and associated scoping rules)
• Objects, Classes, Introspection, and limited __getattr__ / __setattr__
• Namespaces
• Exceptions
• PEP 255 style generators (ie original and simplest)

Quite a few of these may well be challenging in teeny tiny microcontroller environment, but they're worth targetting.

Status

So what DOES this do at the moment?

At present it supports a very simple subset of python:

• strings*, ints, boo leans, floats
• Variables - and typing via basic type inference
• while, for, if/elif/else
• Parenthised expressions
• Full expression support for ints
• For loops actually implement an iterator protocol under the hood
• The ability to pull in #include's using C++ preprocessor style directives - since they're automatically python comments

It also has 2 compilation profiles:

• Desktop/libc - that is for development and testing
• Arduino Leonardo - that includes things like DF Beetle/etc

This is a simple benchmark for a desktop:

countdown = 2147483647
print "COUNTING DOWN"
while countdown:
    countdown = countdown - 1

print "BLASTOFF"

(That "print" will become a python3 style print BTW - it's currently a python2 style statement while I'm bootstrapping the compiler!)

The arduino "blink" program for the leonardo profile:

led = 13

pinMode(led, OUTPUT)

while True:
  digitalWrite(led, HIGH)
  delay(1000)
  digitalWrite(led, LOW)
  delay(1000)

This is unlikely to ever be a completely general python to C++ compiler, especially in the context of a desktop machine. If you're after that, there are better options (shedskin and Cython spring to mind).

Where to get it -- Ubuntu 14.04LTS (trusty), 15.04 (vivid)

(This is the recommended mechanism)

I build packages for this in my PPA. You can install pyxie as follows -- add my PPA to your ubuntu, update and install the python-pyxie package.

sudo add-apt-repository ppa:sparkslabs/packages
sudo apt-get update
sudo apt-get install python-pyxie

This will automatically pull in the dependencies - PLY and guild

Where to get it -- PyPI

Simplest approach using PyPI:

sudo pip install pyxie

Alternatively go to the Pyxie page and download from there...

• http://pypi.python.org/pypi/pyxie/

... and then do the usual sudo python setup.py install dance.

Dependencies

Uses David Beazeley's PLY package. Either use your package manager, or install from pypi:

• http://pypi.python.org/pypi/ply

At some later point in time the batch compiler will make use of guild - my actor library. That can be found here:

• http://pypi.python.org/pypi/guild

Usage - Default Profile

Given the following python file : benchmark.pyxie

countdown = 2147483647
print "COUNTING DOWN"
while countdown:
    countdown = countdown - 1

print "BLASTOFF"

You compile this as follows:

pyxie compile benchmark.pyxie

That compiles the file using the standard/default profile, and results in a binary (on linux at least) that can be run like this:

./benchmark

Usage - Arduino Profile

Given the following python file : arduino-blink.pyxie

led = 13

pinMode(led, OUTPUT)

while True:
  digitalWrite(led, HIGH)
  delay(1000)
  digitalWrite(led, LOW)
  delay(1000)

You compile this as follows:

pyxie --profile arduino compile arduino-blink.pyxie

That compiles the file using the arduino leonardo profile, and results in a hexfile called this:

arduino-blink.hex

Depending on your actual leondaro type device you can use AVRdude to load the code onto the device. If you're using a bare Atmel 32U4, you can use dfu-programmer to load that hexfile onto your device.

Usage - Summary

More generally:

pyxie -- A little python compiler
Usage:

    pyxie -- show runtime arguments
    pyxie --test run-tests -- Run all tests
    pyxie --test parse-tests -- Just run parse tests
    pyxie --test compile-tests -- Just run compile tests
    pyxie --test parse filename -- Parses a given test given a certain filename
    pyxie parse filename -- parses the given filename, outputs result to console
    pyxie analyse filename -- parses and analyse the given filename, outputs result to console
    pyxie codegen filename -- parses, analyse and generate code for the given filename, outputs result to console. Does not attempt compiling
    pyxie [--profile arduino] compile path/to/filename.suffix -- compiles the given file to path/to/filename
    pyxie [--profile arduino] compile path/to/filename.suffix  path/to/other/filename -- compiles the given file to the destination filename

Open Source

Pyxie is Copyright © 2015 Michael Sparks, and Licensed under the Apache 2 license. It's not public on github at the moment because the internals are still changing somewhat between releases. Once this has stablised I'll open up the git repo.

The code is however on github, so if you'd like early access, let me know. Obviously you can do what you like with the code from pypi within the reasonable limits of the apache 2 license!

It's developed entirely using my own kit/etc too so nothing to do with work. (I mention this primarily because during the Micro:bit project I also built a scrappy python to C++ compiler. That had all sorts of limitations, but made me think "If I was to redo this from scratch, I'd...". This project is a result of that thinking as a result, but as I say nothing to do with work!)

Release History

This is to give some idea of progress it's good (though obviously not swift) progress, with an average of a few weeks between releases. (Mainly because dev depends on spare time:)

• 0.0.1 - (rolled into 0.0.2 - Initial structure)
• 0.0.2 - supports basic assignment
• 0.0.3 - Ability to print & work with a small number of variables
• 0.0.4 - Mixed literals in print statements
• 0.0.5 - Core lexical analysis now matches language spec, including blocks
• 0.0.6 - Character Literals, "plus" expressions, build/test improvements
• 0.0.7 - Structural, testing improvements, infix operators expressions (+ - * / ) for integers, precdence fixes
• 0.0.8 - Internally switch over to using node objects for structure - resulting in better parsing of expressions with variables and better type inference.
• 0.0.9 - Grammar changed to be left, not right recursive. (Fixes precedence in un-bracketed expressions) Added standalone compilation mode - outputs binaries from python code.
• 0.0.10 - Analysis phase to make type inference work better. Lots of related changes. Implementation of expression statements.
• 0.0.11 - Function calls; inclusion of custom C++ headers; empty statements; language spec updates
• 0.0.12 - While loops, break/continue, Website, comparison operators, simple benchmark test
• 0.0.13 - if/elif/else,conditionals/boolean/parenthesised expressions.
• 0.0.14 - For loops implemented. Added clib code, C++ generator implementation, FOR loop style test harness, parsing and basic analysis of of FOR loops using a range interator
• 0.0.15 - clib converted to py clib for adding to build directory
• 0.0.16 - Adds initial Arduino LEONARDO support, improved function call, release build scripts

Supporting this

Please do! There's many ways you can help. Get in touch.

Summary

Pyxie is at an very early stage. It is a simple little python to C++ compiler is at a stage where it can start to be useful for some limited arduino style trinkets.

Kicking the tires, patches and feedback welcome.

So, a fluffy tech-lite blog on this. I've written a more comprehensive one, but I have to save that for another time. Anyway, I can't let this go without posting something on my personal blog. For me, I'll be relatively brief. (Much of this a copy and paste from the official BBC pages, the rest however is personal opinion etc)

Please note though, this is my personal blog. It doesn't represent BBC opinion or anyone else's opinion.

Just over 2 weeks ago, the BBC announced a small device currently nicknamed the BBC Micro Bit. At the launch event, they demonstrated a prototype device:

The official description of it is this:

BBC launches flagship UK-wide initiative to inspire a new generation with digital technology

The Micro Bit

A major BBC project, developed in pioneering partnership with over 25
organisations, will give a personal coding device free to every child in year 7 across the country - 1 million devices in total.
...
Still in development and nicknamed the Micro Bit,* it aims to give children an exciting and engaging introduction to coding, help them realise their early potential and, ultimately, put a new generation back in control of technology. It will be distributed nationwide from autumn 2015.

... The Micro Bit will be a small, wearable device with an LED display that children can programme in a number of ways. It will be a standalone,
entry-level coding device that allows children to pick it up, plug it into a computer and start creating with it immediately.
...
the Micro Bit can even connect and communicate with these other devices, including Arduino, Galileo, Kano and Raspberry Pi, as well as other Micro Bits. This helps a child’s natural learning progression and gives them
even more ways of expressing their creativity.
...
Early feedback from teachers has shown that it encourages independent learning, gives pupils a strong sense of achievement, and can inspire those who are not usually interested in computers to be creative with it.

For me, the key points about the prototype:

• Targetted at children, and their support networks (teachers, parents etc). This is reflected throughout the system. It doesn't underestimate them though. (I work with cubs and scouts in my spare time, and I know they're much willing to try than adults - which means they can often achive more IMO. Especially at scout age)
• Small programmable fun device, instant feedback, with lots of expansion capability with a gentle gradual curve.
• Web based interfaces for coding, as well as offline interfaces
• Graphical, python or C++ code running on the device
• Designed to work well with other things and hook into larger eco-systems where there's lots of users.

(That's all public information, but for me those are really the key points)

Aside from the obvious -- why did the announcement excite me? Simple: While it wasn't my vision - it was Howard Baker's and Jo Claessen's, I implemented that Microbit prototype and system from scratch. While an earlier iteration had been built and used at events, I was asked to build a prototype we could test in schools and to take to partners to take to scale. (I couldn't work from that earlier iteration, but rather worked from the source - Howard and Jo. It was a harder route, but rewarding)

For those who recognise a similarity between this and Dresscode which I blogged about last summer, the similarities you see are co-incidental. (I was clearly on the same wavelength :-) ) While Microbit was in a early stage pre-prototype phase there, I hadn't seen it.

My thinking behind Dresscode though, was part of the reason I jumped at the chance to help out. It also mean though that some aspects I had a head start on though, and in particular, the IOToy and Guild toolsets I developed previously as part of some IOT work in R&D came in handy in building the prototype.(That and the fact I've used the arduino toolset a fair amount before)

Anyway, as I say, while I can't say much about the project, but it was probably the most intense development project I've worked on. Designing & building the hardware from scratch, through to small scale production manufacture in 3 months, concurrent with building the entire software stack to go with it - compilation toolchains, UIs and so on, AND concurrent with that was the development of detailed documentation and specs for partners to build upon. Intense.

In the last month of core development I was joined by 2 colleagues in R&D (Matt Brooks and Paul Golds) who took my spike implementations of web subsystems and fleshed them out and made them work the way Howard and Jo wanted, and the prototype hardware sanity checked and tweaked slightly by Lawrence Archard. There was still a month of development after that, so it was probably the most intense 4 months of development I've ever done. Well worth it though.

(Regarding names, project names change a lot. While I was building our device in even that short 3 months, it got renamed several times, and since! Don't read too much into names :-) Also, while it does fit anywhere else, it was Fiona Iglesias who was the project manager at the time who asked me if I could do this, and showed a great leap of faith and trust when I said "yes" :-) Balancing the agile process I needed to use against the BBC being very process driven (by necessity) was a thankless task!

Finally some closing things. On the day of the launch I posted a couple of photos in my twitter feed, which I'll repeat here.

The first was my way of going "yay". Note the smiley bit :-)

So, look what I made! http://t.co/OTCn14YN7s Earlier revs were built on my kitchen table :-) pic.twitter.com/2k9a2UBOZh

— Michael (@sparks_rd) March 12, 2015

The second related to an earlier iteration:

Earlier iteration: pic.twitter.com/bmQ2G7KK2t

— Michael (@sparks_rd) March 12, 2015

Anyway, exciting times.

What happens next? Dunno - I'm not one of the 25 partners :-)

The prototype is very careful about what technologies it uses under the hood, but then that's because I've got a long history of talking and writing about about why the BBC uses, improves and originates open source. I've not done so recently because I think that argument was won some time ago. The other reason is because actually leaving the door open to release as open source is part of my job spec.

What comes next depends on partners and BBC vagaries. Teach a child to code though, and they won't ever be constrained by a particular product's license, because they can build their own.

Anyway, the design does allow them to change the implementation to suit practicalities. I think Howard and Jo's vision however is in safe hands.

Once again, for anyone who missed it at the top though, this is a personal blog, from a personal viewpoint, and does not in anyway reflect BBC opinion (unless coincidentally :).

I just had to write something to go "Yay!" :-)