Proposed framework changes for 1.4

[Chris Travers <..hidden..>]

Hi all;

INTRODUCTION:
================
Looking towards 1.4 and trying to improve things significantly from
where they are on 1.3.  In general, I'd hope that with each new
version, these changes become more incremental (and smaller) until we
are ready to start on 2.0.  The goals of these proposals are as
follows:

1)  To future-proof the LedgerSMB architecture, trying to standardize
on something solid, building on what works well with the 1.3 codebase
and addressing shortcomings that come up in relation to the directions
we have been going.

2)  To address specific issues we will run into in the areas we are
re-engineering in 1.4 and later.

3)  To better support a variety of architectures, such as mod_perl,
going forward.

All changes in this document are backwards compatible, so unless
otherwise noted we don't have to rewrite code specifically because of
this document though we will rewrite for other reasons through the 1.x
process.

WHAT 1.3 DOES WELL
==================

1.3 is a substantial accomplishment.  We've added real security to an
inherited codebase, and we've added impressive capabilities at the
same time, and without serious bugs post-1.3.0 release in the core
functions (those which most users use).  Major reasons for the
successes include:

1)  A focus on declarative logic in the database, both for permissions
and for data constraints
2)  A clear separation of presentation and data logic.  The database
procedures handle data logic and the presentation is handled by Perl
and TT.

SHORTCOMINGS OF 1.3
====================
The code during the 1.3 development process evolved quite a bit as
maintainability issues were uncovered and addressed and coding styles
evolved.  This makes 1.3 a bit of a mishmash of coding paradigms and
we can expect some maintenance headaches as a result.  Longer-run we
need more standardization in particular, and coding practices that are
more conducive to writing great documentation.  This should make it
easier for people to follow new code and contribute.

In particular some of the early 1.3 code is relatively unstructured,
and classes don't have specific properties associated.  This makes the
code impossible to document and consequently harder to manage and get
contributions for going forwards.

A second major issue has been the issue of stored procedure discovery
and the interface between these and the application.  The issue is
that the stored procedures aren't discoverable enough--- one can
determine that a specific function name exists, but then invoking it,
we can't tell what are intrinsic vs extrinsic arguments.  In other
words, we can't differentiate a property from a (presumably)
pre-existing objects and those which are expected to be new properties
supplied directly by the end user.

Finally, I am not entirely happy with the dependency on PostgreSQL
contrib modules especially since this is changing to a new extension
system.  This adds needless complexity in areas where we don't
necessarily need it.

FUTUREPROOFING CONSIDERATIONS
==============================
One of the goals of the LedgerSMB project has been to make the
LedgerSMB logic reusable not only in other web apps but other
applications in other environments and even other languages.  The
emphasis should be on coding conventions at the lowest levels that
make it possible in the future to implement add-ons in Python, PHP,
etc...

The second side of this though is continuing the separation of
presentation and logic.  Support for environments like mod_perl is not
yet handled, and can't be handled easily with our current codebase.
Further separation would mean less of a need to change things down the
road, since the web handling logic could be carefully contained in a
single module which wouldn't be necessary to include for console
applications, etc.

SPECIFIC CHALLENGES OF 1.4 AND LATER
==================================
With the move into the financial logic in 1.4, a couple of specific
challenges appear.  The big ones relate to storage and reporting of
journal entries, along with GL reports.

On the first, we want to make sure that journal entries are balanced
and reject entries that are not.  Doing this with the 1.3 framework
has a number of obstacles, because we can't easily pass the structures
into the database in a discoverable format.

The second has to do with reporting.  In the current framework we
cannot add running totals to the db queries gracefully without
affecting our ability to re-order results.  Adding running totals here
would be quite useful and I think we should be looking at doing this.

PROPOSALS
============


Proposal 1:  Use Moose 2.x
------------------------------------------
Using Moose 2.x would give us an ability to declare properties and
constraints on those properties in a declarative, clear, and
consistent way.  These can then be easily documented and so forth.  It
also gives us access to BEFORE/AROUND/AFTER syntax for overriding
existing methods gracefully.  These methods make customization easier.
 Moreover, the general approach should make it reasonably possible to
to quickly and easily convert from web requests to objects, which is
another strength of 1.3.

Portability:  Debian Stable and Oldstable would require this to be
installed via CPAN, as would all versions of RHEL and spinoffs at
present (which are using fairly outdated versions).

Proposal 2:  Require PostgreSQL 8.4
-------------------------------------------------------
8.4 includes a few features that would greatly make our work easier.
The first is the availability of arrays of complex types.  This means
it would be possible to pass in all required SQL query parameters as
data structures, allowing the stored procedure to have all the
information required in a way which is verifiably meaningful in input.

The second is that it is easy to query values from arrays in 8.4 using
the unnest() function, making some of the headaches we faced early on
in 8.1 go away.

Finally, we have the availability of windowing functions.  These are
functions which are calculated across data windows on the output set,
such as running totals.  In prior versions of PostgreSQL, there has
been no graceful method to add something like a running total.
Starting with 8.4, that has changed, and now these can be added.

Portability:  Supported on all major platforms.

Proposal 3:  Require Plack and go to PSGI
-----------------------------------------------------------------
PSGI is an interface for writing web applications which is designed to
be portable between frameworks (for example fcgi, mod_perl, cgi).  I'd
recommend going this direction for new code as it will allow us to
serve more environments than we can get currently.

I'd recommend we go with Plack as an application framework for web
interfaces from here on out.  It's becoming fairly common, and I think
it would offer us benefits here in terms of portability and
consistency we can't get by sticking with CGI::Simple modules.

Portability:  Supported on RHEL 5+, Debian Stable, etc.

Proposal 4:  New Namespaces
----------------------------------------------
I propose the new namespaces:

LedgerSMB::SODA would provide query mapping and error
processing/exception raising services

LedgerSMB::WebApp would provide http request and response handling
services, as well as global storage for a request, locale, and user
object.

LedgerSMB::PGDate would provide input/output formatting for dates and times

LedgerSMB::PGNumeric would provide input/output formatting for numbers.

Proposal 5:  Exception Handling
-----------------------------------------------
I'd like to use Try::Tiny for all new exception handling, and use
actual exceptions for errors instead of the functional approach we
have today.

Proposal 6:  Stored procedure interface
---------------------------------------------------------
Procedure arguments would be named with prefixes, either arg_ or prop_
(for new functions) or in_ for old functions.  Args starting with
prop_ would be assumed to be object properties, while those prefixed
by arg_would be assumed to be extrinsic to any existing objects.  in_
would be handled as like arg_ for interop with 1.3-style functions.

Additionally the new interface would have the ability to add window
specs, order_by lists, and aggregates across data windows to stored
procedure calls.  This would mean among other things you could add
running totals to stored procedure calls easily.


Proposal 7:  Allow access to LedgerSMB/Scripts directory
-----------------------------------------------------------------------------------
The idea here is to allow easier inheritance and specification of
inheritance in perl workflow scripts.  Long-run we could have a URL
handler that could address this automatically, but for now, this is a
baby step in the right direction.

Any feedback?