Python 3.6 and higher support (available for pick-up)

[tekknolagi]

Hi @pfalcon,

Do you have plans to support Python 3.6? What can people do to help?

Cheers.

[pfalcon]

My plans actually to make this module easily adjustable/overridable/subclassable (something like that), so support for different Python versions/variants could be easily added. My primary interest is making a pure-Python, optimizing compiler for MicroPython.

On critical path to that would be cleaning up the existing code, https://github.com/pfalcon/python-compiler/issues/5 .

And fairly speaking, even Python3.5 support is not fully there - why on the level of just bytecode everything is ok, metadata needs more work, that's #1, #3, perhaps #2.

[pfalcon]

@smontanaro, Sorry for bringing it from email, but this ticket should provide the background of my interests/plans for this project.

As mentioned above, my primary interest is in developing pure-Python compiler for then MicroPython, now my own fork https://github.com/pfalcon/pycopy . Pycopy guaranteedly stays with the bytecode, as the primary requirement for Pycopy bytecode is the compactness in size. Where performance is important, Pycopy simply skips wordcode and VM at all, and JIT-generates native code for CPU. (All the above is true for MicroPython too, but I'm not sure where MicroPython goes, whereas the above are invariants for Pycopy).

That's why 3.5 support was chosen for this project, as the last more or less common bytecode version between CPython and MicroPython. As mentioned above, ideally, I'd like to implement as generic and retargetable compiler as possible. But now I'm not sure any longer. For example, I was looking into picking up a Python parser (to AST), and ended up writing my own: https://github.com/pfalcon/pycopy-lib/tree/master/ast . It fully satisfies Pycopy's goals of being small and lightweight (and providing good base for future extensibility), whereas other Python projects oftentimes don't. This codebase also has quite a bunch of cruft, cleaning up which takes considerable time.

So, with that initial experience, I wonder if writing a compiler for Pycopy from scratch would make more sense. I'm not sure yet.

In either case, while the codebase of this project currently generates almost CPython-perfect bytecode, it's not the case for bytecode metadata. The most pressing issue is https://github.com/pfalcon/python-compiler/issues/3. But all the issues mentioned above would need to be resolved first, before moving to support bytecodes of other versions than 3.5. Doing it in another way doesn't make sense IMHO.

[smontanaro]

Thanks @pfalcon. @nascheme pointed me to this project. I'm starting to twist my brain back in the direction of a register-based virtual machine for Python. Neal seemed to think it might be better to start with your compiler, then modifying it to generate wordcode. For my purposes, it seems I should just stick with my fork for now. If it makes sense later, we can discuss merging. Skip

[pfalcon]

@smontanaro, Sounds good. Anything is better than dealing with compilers written in C ;-).

[pfalcon]

Ok, for reference, my own too, I embarked to write a bytecode compiler for Pycopy from scratch. Seems to go well so far, so I probably follow that direction of work.

[pfalcon]

Ok, for reference, my own too, I embarked to write a bytecode compiler for Pycopy from scratch.

Just to confirm, I'm full-on on this path. Progress goes well, but sloooow, where to get more time for this stuff ;-). That means that I unlikely will do further work on this project any time soon. So, interested parties, feel free to pick it up/fork.

[smontanaro]

In my rumination/investigation regarding a register-based virtual machine, I did read up a bit on the new wordcode stuff. Seems it often produces shorter overall bytecode than the original byte-mostly opcode scheme with its variable length instructions, and the consistent instruction size speeds up the evaluation loop a bit. Might be worth looking at.

I've decided to start my work from the existing Python/compile.c. While I need to change the actual code generation, it already does all the other stuff (basic blocks, symbol tables, etc). I don't think much (if any) of that stuff will change. I'll need to tackle register allocation and actual code generation, but I get everything else for free.