[Padre-dev] Implementing PPI's Tokenizer in XS?

[Shmuel Fomberg]

Hi Adam.

Adam Kennedy wrote:
> I did create an initial framework for selectively "accelerating" parts
> of PPI, which you can find at PPI::XS
> (http://svn.ali.as/cpan/trunk/PPI-XS). The small amount of code there
> makes it only about 3% faster.
> 
> Steffen Mueller has mentioned though, that the benefits of this
> approach are not likely to be truly enormous, because if we're cherry
> picking a function here and a function there, then there's only so far
> that we can make it faster.
> 
> He recommends instead that a large amounts of code get moved behind
> the XS boundary...

I agree with Steffen. And as much as I can see, moving the tokenizer 
will be the best way to get significant speed boost.
Plus, (as much as I can see) it is possible to clear-cut it from the 
rest of the PPI.

> The tricky part is that PPI::Tokenizer is very complex (although I'm
> sure I could describe the approach it takes well enough to provide a
> good starting point) and it would probably end up being a fairly large
> amount of C, and then would need to be kept in sync with the Perl
> version.

Well, that's what unit tests are for. and I saw that you have many of them.


Shmuel.