Re: LedgerSMB Scalability Bottleneck-- proposal

["Joshua D. Drake" <..hidden..>]

Chris Travers wrote:
> One of the issues that LedgerSMB can run into in larger environments is 
> that large processes can run into client timeout issues, especially when 
> large amounts of data are passed back and forth.
>
> In one case, I am running into issues where approx. 4k invoices are 
> being paid at once.  Even when these are consolidated into stored 
> procedures as much as possible, the overhead in the db is causing the 
> web pages to time out.
>
> My proposal is to offer a queue system for mass transaction processing.  
> This would allow one to queue thousands of transactions which could then 
> be entered into batches or posted in the background.  A message could be 
> delivered to the user when this completes.

As I understand it, a timeout happens because nothing is happening. E.g; 
 there is no communication between the apache server and the client. 
Why not just have the stored procedures spit out an update as they are 
working?

Processed 100 transactions
          200 transactions
          300 transactions

Until done. If you wanted to offset load, we can use plperl to implement 
some sleep patterns.

Joshua D. Drake