[Padre-dev] PPI::XS::Tokenizer is dead

[Steffen Mueller]

Hi Shmuel,

Shmuel Fomberg wrote:
> Well, you gave us (as much as I understood) a skeleton for a
> full-object Perl-C++ mapping. which is not what we need. we need an
> object-creating function instead.

That was my first stab. Obviously, you missed my mail since in it I
outlined my *second* stab which creates perl-space objects from the
tokens. But since I'm neither the author of the C++ tokenizer nor of
PPI, I don't know how to convert the C++ objects into Perl objects 
except for the simplest cases:
There is no general 1-to-1 correspondence. This is something that needs
to be worked out between Shmuel and Adam. When someone tells me how
that's supposed to work, then I can write an XS implementation (maybe,
time's in short supply).

Here's my original mail:

> please have a look at the current state of the repository. We can run
this:
> 
> use PPI::XS::Tokenizer; require PPI;
> 
> my $t = PPI::XS::Tokenizer->new("Test");
 > isa_ok($t, 'PPI::XS::Tokenizer');
 > my $token = $t->get_token();
 > ok(defined $token, "Token defined");
 > ok($token->isa("PPI::Token::Word"), "Token is a PPI::Token::Word");
 > is($token->content, 'Test', "Token contains the word 'Test'");
> 
> 
> What's currently blocking is a way to convert the C++ tokens to Perl 
> objects. I implemented a simple XS equivalent of bless( {content =>
> $text} => $class ) for the "simple" tokens.
> 
> How do I know whether a token has more than the "text"? I tried:
> 
> if (theToken->type->isa(isToken_Extended)) { ... }
> 
> But that didn't seem to fire for a HEREDOC. (See t/01basic.t)
> 
> How would one convert an extended C++ token to its corresponding
 > PPI token? I am the wrong guy for this since I know neither
 > which information you store in the C++ object and which information
 > needs to be stored (how?) in the Perl object.
> 
> This latter point is probably something you should discuss with Adam.

Cheers,
Steffen