[Padre-dev] PPI::Tokenizer in C

[Shmuel Fomberg]

Hi All.

I'm working on a C version of the tokenizer, and got to the amazing 
achievement, that my code can now tokenize "     \n  #  aabbcc d\n" 
correctly.
Yay.

Currently the code is pure C++, but once it will get decent, I'll make 
an XS wrapper.
Thanks to Adam for extra-explaining things, which got me started.

Is there any source control I can put the code in while working on it?

And help is needed. If someone with a bit of knowledge in C/C++ willing 
to take a token to implement, please contact me.

Shmuel.