[Padre-dev] Padre and background tasks

[Steffen Mueller]

Hi Padre people,

I spent the better part of yesterday working on a background-task 
implementation for Padre using threads.

Executive summary: It works if you make sure worker threads are never 
joined/reaped. If that happens, Padre crashes. You can find the 
implementation in branches/. Some documentation on the design and 
implementation can be found in the comments of lib/Padre/TaskManager.pm.

The full story is that things seemed to be working beautifully until I 
added a bit more sophisticated thread pooling: Keep one worker thread 
available at all times and create more up to a maximum if demand 
exceedes the number of threads by some margin. When demand dies down 
again, slowly join one worker after another (every two seconds) until 
back down to the minimum.

What happens is that the first thread that's joined gives a "Scalars 
leaked: 1" warning that I couldn't track down. When the second worker is 
being joined, perl "exits normally" from within the $thread->join() call.

I'm totally at a loss and will not have much time to continue debugging 
during this week. I could use some help. I've been using perl 5.8.8, 
maybe it's better with 5.10.0? My debugging build couldn't run padre at all.

To reproduce the error, do the following:
- Check out branches/padre-with-backgroundthreads
- Install the newest threads, threads::shared and Thread::Queue (just 
run Build.PL to check).
- Run dev.pl
Every two seconds, you will see output like "--1 at 
/home/tsee/padre/branches/padre-with-backgroundthreads/lib/Padre/TaskManager.pm 
line 157.". It means there's one worker thread being held available.
- Enable the ThreadTest plugin
- Run Plugins->Thread Test->Test
This will submit 20 or so short jobs to the taskmanager which will see 
large demand and spawn more threads. When the short jobs are done
The --X numbers will go up quickly and you'll see "in task" warnings 
from the tasks themselves as they're processed. When they're through, 
you'll see something like this:

8 at ... line 94.
--8 at ... line 144.
--8 at ... line 157.
THREAD TERMINATING at ... line 272.

==> The above warning is from the worker thread which acknowledges that 
it's being shut down. It simply does a "return 1".

8 at ...line 94.
--8 at ... line 144.
Joining thread 3 at ... line 148.

==> This is from right before "$thread->join()" is called

Scalars leaked: 1

==> This is from that call.

joined at ... line 150.

==> This is from right after the call. Now, observe what happens with 
the second thread that's being joined:

--7 at ... line 157.
THREAD TERMINATING at ... line 272.
7 at ... line 94.
--7 at ... line 144.
Joining thread 1 at ... line 148.


And then you're back to the shell.

Any help greatly appreciated. I suppose it's a Wx+threads thing.

Best regards,
Steffen