[Padre-dev] Implementing PPI's Tokenizer in XS?

[Steffen Mueller]

Hi Adam, hi all,

Adam Kennedy wrote:
> I did create an initial framework for selectively "accelerating" parts
> of PPI, which you can find at PPI::XS
> (http://svn.ali.as/cpan/trunk/PPI-XS). The small amount of code there
> makes it only about 3% faster.
> 
> Steffen Mueller has mentioned though, that the benefits of this
> approach are not likely to be truly enormous, because if we're cherry
> picking a function here and a function there, then there's only so far
> that we can make it faster.

What I tried to say was essentually: The largest benefit from using XS
is when you use it for interfacing only. If you still store all data in
Perl data structures and want to write good, safe C/XS, then you'll
mostly just rewrite the op functions that perl would execute anyway. If
you do it mostly in C, with C data structures, and provide a Perl
interface using XS, then you win a lot of speed.

> He recommends instead that a large amounts of code get moved behind
> the XS boundary...

Right. Except somebody has to write it. And that's a herculean task!

> The tricky part is that PPI::Tokenizer is very complex (although I'm
> sure I could describe the approach it takes well enough to provide a
> good starting point) and it would probably end up being a fairly large
> amount of C, and then would need to be kept in sync with the Perl
> version.

I'm not confident that this is really practical. I can't do it, that's
for sure, but there are *much* better C (and XS) hackers.

Cheers,
Steffen