Re: Proposed framework changes for 1.4

[John Locke <..hidden..>]

Hi,

Sounds like a good direction to me. Couple thoughts -- not being that
active in Perl, don't know if these are realistic or not, but just what
comes to mind, feel free to disregard.

- Are there really no other date or number libraries available that do
what we need, to the point we need to create our own? (or are these
primarily wrappers around something else?)

- Ran across this project recently: http://ledger-cli.org/ - a Perl cli
app that does double-entry accounting in text files. I wonder if they've
worked out some good accounting logic we can use, at a minimum -- or
perhaps incorporate into LSMB? I'm thinking with a little collaboration,
ledger-cli might be extended to use LSMB as back end storage, while LSMB
might potentially leverage the work already done in ledger? Have not
looked at the code, have no idea whether this is practical. But an
easier CLI interface sounds very useful, if it's not too much of a
stretch. And might be a good gateway to new users...

- Regarding extensions, I just had to do some upgrades to some other web
apps where customizations and extension code was scattered across a
bunch of libraries. Basically an MVC organization -- but all extension
code had to be split up, with the M in the M, the V in the V, and the C
in the C directories.... I just want to express my extreme distaste for
that file organization, from an ongoing maintenance point of view. Using
a framework for development is great, as long as we can set it up in a
way so that if I want to add some sort of custom module to the site, I
can put all the code for it in one directory.

Extending that last point a bit -- might it be possible to re-organize
the core modules this way, to provide good examples? Instead of having a
stub in the root, a file in scripts, the class in LedgerSMB, etc --
might it be possible to have a modules directory with each module in
it's own subdirectory? modules/AR, modules/AP, modules/HR, etc. And then
inside those, if necessary, mirror the rest of the file structure for
MVC -- modules/AR/scripts, modules/AR/LedgerSMB, modules/AR/UI, etc?

And to take it one step further, what Drupal has done brilliantly is
provided an override mechanism -- if you want to override a core module,
you can simply copy it to a place that gets checked first, then hack
away, leaving the original code intact. Then an upgrade won't clobber
your customizations, and you can much more easily do a simple diff to
see what's changed. LSMB doesn't need to have the multiple paths that
Drupal uses (I don't think) but maybe something as simple as a path
that's checked first for a module - local/ comes to mind. You wanna hack
AR? Copy modules/AR to local/AR and go to it.


Cheers,
John

On 10/18/2011 04:49 AM, Chris Travers wrote:
> Hi all;
>
> INTRODUCTION:
> ================
> Looking towards 1.4 and trying to improve things significantly from
> where they are on 1.3.  In general, I'd hope that with each new
> version, these changes become more incremental (and smaller) until we
> are ready to start on 2.0.  The goals of these proposals are as
> follows:
>
> 1)  To future-proof the LedgerSMB architecture, trying to standardize
> on something solid, building on what works well with the 1.3 codebase
> and addressing shortcomings that come up in relation to the directions
> we have been going.
>
> 2)  To address specific issues we will run into in the areas we are
> re-engineering in 1.4 and later.
>
> 3)  To better support a variety of architectures, such as mod_perl,
> going forward.
>
> All changes in this document are backwards compatible, so unless
> otherwise noted we don't have to rewrite code specifically because of
> this document though we will rewrite for other reasons through the 1.x
> process.
>
> WHAT 1.3 DOES WELL
> ==================
>
> 1.3 is a substantial accomplishment.  We've added real security to an
> inherited codebase, and we've added impressive capabilities at the
> same time, and without serious bugs post-1.3.0 release in the core
> functions (those which most users use).  Major reasons for the
> successes include:
>
> 1)  A focus on declarative logic in the database, both for permissions
> and for data constraints
> 2)  A clear separation of presentation and data logic.  The database
> procedures handle data logic and the presentation is handled by Perl
> and TT.
>
> SHORTCOMINGS OF 1.3
> ====================
> The code during the 1.3 development process evolved quite a bit as
> maintainability issues were uncovered and addressed and coding styles
> evolved.  This makes 1.3 a bit of a mishmash of coding paradigms and
> we can expect some maintenance headaches as a result.  Longer-run we
> need more standardization in particular, and coding practices that are
> more conducive to writing great documentation.  This should make it
> easier for people to follow new code and contribute.
>
> In particular some of the early 1.3 code is relatively unstructured,
> and classes don't have specific properties associated.  This makes the
> code impossible to document and consequently harder to manage and get
> contributions for going forwards.
>
> A second major issue has been the issue of stored procedure discovery
> and the interface between these and the application.  The issue is
> that the stored procedures aren't discoverable enough--- one can
> determine that a specific function name exists, but then invoking it,
> we can't tell what are intrinsic vs extrinsic arguments.  In other
> words, we can't differentiate a property from a (presumably)
> pre-existing objects and those which are expected to be new properties
> supplied directly by the end user.
>
> Finally, I am not entirely happy with the dependency on PostgreSQL
> contrib modules especially since this is changing to a new extension
> system.  This adds needless complexity in areas where we don't
> necessarily need it.
>
> FUTUREPROOFING CONSIDERATIONS
> ==============================
> One of the goals of the LedgerSMB project has been to make the
> LedgerSMB logic reusable not only in other web apps but other
> applications in other environments and even other languages.  The
> emphasis should be on coding conventions at the lowest levels that
> make it possible in the future to implement add-ons in Python, PHP,
> etc...
>
> The second side of this though is continuing the separation of
> presentation and logic.  Support for environments like mod_perl is not
> yet handled, and can't be handled easily with our current codebase.
> Further separation would mean less of a need to change things down the
> road, since the web handling logic could be carefully contained in a
> single module which wouldn't be necessary to include for console
> applications, etc.
>
> SPECIFIC CHALLENGES OF 1.4 AND LATER
> ==================================
> With the move into the financial logic in 1.4, a couple of specific
> challenges appear.  The big ones relate to storage and reporting of
> journal entries, along with GL reports.
>
> On the first, we want to make sure that journal entries are balanced
> and reject entries that are not.  Doing this with the 1.3 framework
> has a number of obstacles, because we can't easily pass the structures
> into the database in a discoverable format.
>
> The second has to do with reporting.  In the current framework we
> cannot add running totals to the db queries gracefully without
> affecting our ability to re-order results.  Adding running totals here
> would be quite useful and I think we should be looking at doing this.
>
> PROPOSALS
> ============
>
>
> Proposal 1:  Use Moose 2.x
> ------------------------------------------
> Using Moose 2.x would give us an ability to declare properties and
> constraints on those properties in a declarative, clear, and
> consistent way.  These can then be easily documented and so forth.  It
> also gives us access to BEFORE/AROUND/AFTER syntax for overriding
> existing methods gracefully.  These methods make customization easier.
>  Moreover, the general approach should make it reasonably possible to
> to quickly and easily convert from web requests to objects, which is
> another strength of 1.3.
>
> Portability:  Debian Stable and Oldstable would require this to be
> installed via CPAN, as would all versions of RHEL and spinoffs at
> present (which are using fairly outdated versions).
>
> Proposal 2:  Require PostgreSQL 8.4
> -------------------------------------------------------
> 8.4 includes a few features that would greatly make our work easier.
> The first is the availability of arrays of complex types.  This means
> it would be possible to pass in all required SQL query parameters as
> data structures, allowing the stored procedure to have all the
> information required in a way which is verifiably meaningful in input.
>
> The second is that it is easy to query values from arrays in 8.4 using
> the unnest() function, making some of the headaches we faced early on
> in 8.1 go away.
>
> Finally, we have the availability of windowing functions.  These are
> functions which are calculated across data windows on the output set,
> such as running totals.  In prior versions of PostgreSQL, there has
> been no graceful method to add something like a running total.
> Starting with 8.4, that has changed, and now these can be added.
>
> Portability:  Supported on all major platforms.
>
> Proposal 3:  Require Plack and go to PSGI
> -----------------------------------------------------------------
> PSGI is an interface for writing web applications which is designed to
> be portable between frameworks (for example fcgi, mod_perl, cgi).  I'd
> recommend going this direction for new code as it will allow us to
> serve more environments than we can get currently.
>
> I'd recommend we go with Plack as an application framework for web
> interfaces from here on out.  It's becoming fairly common, and I think
> it would offer us benefits here in terms of portability and
> consistency we can't get by sticking with CGI::Simple modules.
>
> Portability:  Supported on RHEL 5+, Debian Stable, etc.
>
> Proposal 4:  New Namespaces
> ----------------------------------------------
> I propose the new namespaces:
>
> LedgerSMB::SODA would provide query mapping and error
> processing/exception raising services
>
> LedgerSMB::WebApp would provide http request and response handling
> services, as well as global storage for a request, locale, and user
> object.
>
> LedgerSMB::PGDate would provide input/output formatting for dates and times
>
> LedgerSMB::PGNumeric would provide input/output formatting for numbers.
>
> Proposal 5:  Exception Handling
> -----------------------------------------------
> I'd like to use Try::Tiny for all new exception handling, and use
> actual exceptions for errors instead of the functional approach we
> have today.
>
> Proposal 6:  Stored procedure interface
> ---------------------------------------------------------
> Procedure arguments would be named with prefixes, either arg_ or prop_
> (for new functions) or in_ for old functions.  Args starting with
> prop_ would be assumed to be object properties, while those prefixed
> by arg_would be assumed to be extrinsic to any existing objects.  in_
> would be handled as like arg_ for interop with 1.3-style functions.
>
> Additionally the new interface would have the ability to add window
> specs, order_by lists, and aggregates across data windows to stored
> procedure calls.  This would mean among other things you could add
> running totals to stored procedure calls easily.
>
>
> Proposal 7:  Allow access to LedgerSMB/Scripts directory
> -----------------------------------------------------------------------------------
> The idea here is to allow easier inheritance and specification of
> inheritance in perl workflow scripts.  Long-run we could have a URL
> handler that could address this automatically, but for now, this is a
> baby step in the right direction.
>
> Any feedback?
>
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure contains a
> definitive record of customers, application performance, security
> threats, fraudulent activity and more. Splunk takes this data and makes
> sense of it. Business sense. IT sense. Common sense.
> http://p.sf.net/sfu/splunk-d2d-oct
> _______________________________________________
> Ledger-smb-devel mailing list
> ..hidden..
> https://lists.sourceforge.net/lists/listinfo/ledger-smb-devel
>
> !DSPAM:4e9d681587861581119912!
>