I've made a new release of pyxie. Largely internal changes.

The focus of this release is the result of the independent intermediate node refactoring project I was undertaking. This was to transform the list oriented data structure for representing a "pure" intermediate form of code into a more stuctured/generalisable format.

The motivation behind this change was the realisation that implementation of end user functions (ie def) would be very difficult with this more structured approach.

The actual release notes (and source releases) are here:

• https://github.com/sparkslabs/pyxie/releases/tag/v0.1.25

Github repo:

• https://github.com/sparkslabs/pyxie

Package on PyPI:

• https://pypi.org/project/pyxie/

Also via PPA for Ubuntu:

• sudo add-apt-repository ppa:sparkslabs/packages
• sudo apt-get update
• apt-get install python-pyxie

What's new user facing

Before I dive into those changes, it's perhaps worth noting some of the other more minor changes, but are more noticeable from a user perspective.

• A collection of docs added in doc/
• A start on documenting the models used in Pyxie
• Licensing on pyxie's output. (Yes, it's what you want, but explicitly so)
• Some new examples focussed on specific aspects of language:
  • if
  • if-else
  • pass
  • print
  • while-break
  • while-continue
• The arduin profile has been extended to support the Adafruit_NeoPixel library
• neopixel example added

Worth noting:

• While there is an example "simplest_function" it is not expected to compile yet

Neopixel example

This is quite nice, and so since it's perhaps more interesting I'm including a snapshot here:

#include <Adafruit_NeoPixel.h>

pin = 6
number_of_pixels = 16
delayval = 500

pixels = Adafruit_NeoPixel(number_of_pixels, pin, NEO_GRB + NEO_KHZ800)

pixels.begin()

while True:
    for i in range(number_of_pixels):
        pixels.setPixelColor(i, pixels.Color(0,150,0))
        pixels.show()
        delay(delayval)

Note: This reuses the fact that #include is a comment in python. It might change at somepoint to be more like this...

from Adafruit_NeoPixel import *

...since that's logically closer, but for now this is a pragmatic approach that works.

Rationale behind changes

Before this change, pyxie used the following data structures, and code phases when parsing, analysing, transforming and performing code generation.

These were not ideal:

• Code phase: Parser - read in the source, parsed it, and created a data structure
  • Data structure: pynodes - these represent the concrete python syntax, and are generated during the code generation phase. These are python objects in a class hierarchy. Once created they are analysed and placed into a context to analyse datatypes.
• Code phase: Analysis. - Analyses the code, and decorates the existing pynodes with type information to understand the programs data and data types. (does not create a new data structure).
• Transform phase: walks the pynode CST, and generates an intermediate data structure intended to represent the program in the abstract independent of the language used. An independent intermediate form if you like.
  • Data structure: independent intermediate form - This is used to model the program in a "pure form" - which isn't constrained by the source language, and contains enough information for the target language output to be generated. That was the theory. In practice it was a nested list of list of lists ... Not ideal. More on this below.
• Code generation phase: This walked the independent intermediate form (the lists of lists), and created an output data structure representing the concrete final program.
• The third/final data structure is intended to represent the final output language. ie it is intended to be a concrete representation of the output language. This final data structure is then capable of creating the output. Again, forms a hierarchy and could be called "CppNodes" (Though they weren't before this change)

The problem here is that while the pynodes are currently well defined (to a large extent) and that the CppNodes are pretty well defined (even if they could be better), the independent intermediate form sucked because it was just nested lists. This meant in practice the code was fragile, difficult to change, and increasingly difficult to work with. In the early days of Pyxie this simplistic structure made sense. However as data analysis becomes more complex and tricky. This mirrors the fact that the same was true in the early days of the parsing side.

So the focus of this sub-project was to replace the intermediate form, and tighten up the differences and make clearer in the code base that we have PyNodes, iiNodes, and CppNodes, where:

• Pynodes - are the current structures, unchanged - note these are currently prefixed Py - PyDefStatement for example.
• CppNodes - are map to objects/hierarchy/etc that represents C++ programs, but made clearer (slightly). These are now prefixed Cpp rather than the previous C_ which some of the objects use.
• iiNodes - represent the independent, intermediate nodes. Since iiNodes need to be used in source code where there are either PyNodes + iiNodes or CppNodes + iiNodes, they have the prefix "ii" enable differentiation.

Since all 3 of these are actually models, the code for these has all moved to sit below pyxie.models.

At some point there will be a need to restructure the code more generally. Parsing, transforming to IINodes and code generation are actually all transforms, and should sit together. That was out of scope for this already relatively major internal change.

Changelog

I could include the changelog, but if you're curious about that, check out the release notes mentioned above.

Future

I'm not going to put a timeline on this - I've made that mistake in the past for pet projects but the following goals exist for future iterations:

• Code structure:
  • Consolidate the structure of CppNodes and make them "cleaner"
  • Similar for iiNodes
  • Shift type inference and variable detection from PyNodes to iiNodes
  • Change iiNodes to be more ECS based
  • Do the analysis/type inference in more of a visitor pattern (ala ECS)
• Types
  • Better implement core datatypes to allow transforms
  • Strings need implementing better
  • Aim for closest match representation of datatypes
  • Allow JSON type structures to exist
    • Implement lists
    • Implement dictionaries
  • Implement tuples
• Language/structural
  • def's with no arguments, and no local variables
  • def's with arguments
  • def's with local variables
  • classes
• Example system goals:
  • It should ideally be possible to run a perceptron style neural network on an Atmega 8A with the aim of making a robot that can follow a line based on said network.

The example system is deliberately ambitious/difficult - at least for a version of python compiled to C++ to run on such a small device.

Some parts of this will go quicker than others, and won't be done strictly in that order.

Most of it however relies upon better internal code structure. (When I started after all it wasn't clear that this would work or be a good approach, so used json objects throughout. As time goes on it becomes clearer that those ad-hoc structures are in the right sort of places, but could do with being more consistent to increase flexibility.

In order to keep releases interesting though, I'll try to add interesting examples at each stage :-)

As usual comments welcome - especially with regard to what examples you might find interesting.