[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Initial steps for 1.6, and some longer-term questions

On Sun, Nov 1, 2015 at 12:06 AM, Erik Huelsmann <..hidden..> wrote:
As folks know here I have made several major attempts to rewrite the financial logic but it is a very big task and I never get as far as I would like.  While 1.6 is already expected to have great improvements in multicurrency support, I am not expecting to do a lot on financial code at present.

While this may seem like a weird move given the mantra of the development team that the "old code" needs to be removed and the recent confirmation that the old code is still very fragile, the existing code has shown to be "mostly working" (for most people). So, I agree that we need to do some maintenance on other aspects of the code which may have degenerated since the project started, due to the focus of replacing code: in the areas you point out below, there's a lot of room for improvement. With the desire to replace old code, it seems like the replacement code wasn't always required to either implement replacements for *all* aspects of the original functionality. While that may be acceptable short term, I don't think that's a good idea longer term. With the release of LedgerSMB 1.3.0 now more than 5 years ago (see: http://ledgersmb.org/release/ledgersmb-130-released -- dated 2011-10-10!), I concur with Chris that time for consolidation has come, even for newer code. Time to iron out some long standing inconveniences and problems which may have been caused by a heavy thrust forward over the past 10 years of LedgerSMB and the past 5 years of 1.3.x and 1.4.x. 
There are a few things architecturally however that I would hope to accomplish with 1.6:

1.  From 1.2 through 1.4, ease of installation has not been a priority and has degraded  We need to reverse that.  Ease of installation and management needs to be a top priority.  I think we have the back-end stuff down, so the question is front-end and administrator tooling.

I would say this is particularly true: we don't have a web tool which points out which libraries or other dependencies are missing from an installation. Tools like Zabbix have that, allowing an admin to quickly address the issue(s) and assess the level of support for any of the optional features/dependencies. This could seriously improve our "first user experience". 
2.  Contact and user management has become more feature-rich but harder to use.  The code here has also become  bit of a mess even though it is rewritten code.  I would like to fix these problems too.

Right. Recent consulting support shows that our contact management -- especially the difference between the "entity" and the "eca" -- and the relation between contacts ("entities" and "ecas") and orders/invoices is hard to understand at first.

Personally, I actually completely understand this for multiple reasons (listing them here, because I think these should be on the list to be addressed as part of addressing any issues in "contact management"):

* The system hinges on two thoughts regarding entities and ecas: when selecting a customer an invoice, one has to enter the name of the *entity*
* The system clearly hinges on two thoughts regarding entities and ecas: when there are two ecas (both customers) within 1 entity, every "update" leads to the counterparty selection screen
* The fact that there's a "contact type" (customer/vendor/...) on the entity definition (company/person) screen, leads people to believe that the customer already has been created when they save the company/person (being presented with the ECA screen which lets them define and save the customer for real)
* The way there currently is no extremely obvious indication that addresses/bank accounts/notes/etc are being added to either the entity or the selected ECA, make the use of these items very hard to use

First, I agree with these points.

Secondly I think there are two major architectural problems also which need to be sorted out.

On top of the UI issues we have the fact that right now we are generating a large form in HTML which then gets dojoized on the client.  This leads to a number of other problems.  So in addition to the points you make above regarding customer confusion, there are some other problems that come out of this.

1.  I start entering a note, switch to the address tab, enter an address, and come back relatively gracefully. The whole page is submitted and recreated and that is less than ideal.
2.  Because of this, the Perl code that generates the page, the controller logic, is ugly, complicated, and difficult to maintain.

If the UI was in dojo, with a clear front/back-end separation, these problems would go away.  This by itself would provide better usability, but it also gives an ability to, as you point out, to clear up the user confusion issues.
3  Workflow and usability issues need to be the top priority.

Agreed. I think this starts with defining what the various workflows should look like and how we can create screens which support those workflows. I'm thinking of presenting un-editable screens for documents (transactions, orders, ...) which have been saved and/or posted; an "Edit"(for saved documents) or "Copy to new"(for posted documents) button at the bottom should move the document into a state where it can be edited, presenting the user with a fully editable document. This will also help users understand the idea of "Save Info", because the only editable field in an invoice *after posting* will be the "internal notes" field. Completely obvious to what the "Save Info" button will apply, right?

Another one of my gripes is with usability and scheduling of transactions. I think we should present a listing of all scheduled transaction postings when scheduling a transaction. That way the user gets to validate his/her expectations regarding the planned postings. I for one *always* get this wrong...

A good scheduling system would be a plus here.
On a technical side, I would like to switch to external tooling for administration and management.  This means one gets the admin tools, and can install and manage LedgerSMB instances from there.  It also means these tools need to play really well together.

I'm affraid this sounds pretty vague from just this sentence. I'm affraid people will have a hard time judging how this will impact them. Will we still deliver these tools in such a way that installation will be easy? Will these be part of the source distro so packagers for distributions automatically get the right sources to package and distribute? Could you elaborate a bit more on your plans in that respect?
Other than that, if the tools come with the same source tarball and are a huge improvement over what we have now, I think we only can cheer to that :-)

I was actually thinking of a couple different approaches we could take here.

1.  Install the admin and installation tools.  Run them to check your system, download, and install latest versions of whatever major branch you want (starting with 1.6).  Such could also update a system and could be run with different permissions than the main web server.  In other words, you:
   a. download a tarball. 
   b.  Run a program.  It checks your system.  If it can start a web server on a higher port it does so.  Otherewise it gives clear feedback about what you are missing.
   c.  Log into the web interface for more system diagnostics, installation and management help.

2.  We could bundle all our tools together.  In this case you still run the same program which checks your system, sees whether it can run a web server and if not gives information as to why not.  Such a program would then typically not update a system but could change file permissions after running and setting things up.

In both cases, the installer checks external prerequisites, CPAN dependencies, and can either solve or advise on how to solve missing dependencies.  I would like LedgerSMB 1.6 to be easier to install than SQL-Ledger 2.6 was. Mostly this is hand-holding, providing clear feedback on how to resolve problems, and a nice interface for showing the information.

I also want to make sure we have both command line and web tools.  Command line tools give experienced administrators a productivity edge here (because it is easier to quickly send complex information to the program), but web tools give new users an ability to get up and running with less immediate knowledge.

Could you spend a few lines detailing which problems we'll be solving with these new tools?

I would also like to have the contact and user interfaces primarily client-side _javascript_ driven by Dancer-based web services.  This would make things much easier to manage the code.

Both that as well as the fact that having the services makes it much easier for tech savvy people to migrate to LedgerSMB or to interface LedgerSMB to other systems.
As we get into that process, I am hoping that we will be better equipped to discuss options for removing legacy code in an orderly manner.

It's my conception that the problem with the legacy code is that it's too interwoven in order to be disentangled without breaking everything else. My idea is that heavy browser-side widgets powered by webservices, offer the opportunity to modularize the code base, allowing decommissioning of the old codebase in a step-by-step approach. Modules to be factored out first, would be (imo):

 - order/invoice/quotation customer/vendor lookup -- we should be able to implement this in a widget, eliminating the need for the intermediate "select customer" screen
 - order/invoice/quotation part/service lookup -- we should be able to implement this in a widget in similar ways we do for customers/vendors
 - rewrite the entire GL entry screen into a series of widgets and webservices -- although not a big part of the legacy code base, it's definitely a step to decommissioning code
 - tax calculation widget (service based automatic calculation) -- eliminates the need for the "update" button, at least as far as tax calculation is required
 - ship-to lookup/entry (popup) widget -- eliminates the complexity relating to the existing "ship to" intermediate screen
 - e-mail (popup) widget -- for posted documents; eliminates the complexity in dealing with the current intermediate code paths jumping across modules
 - widget to hold all invoice lines (and which knows how to add more) -- eliminates the need for the update button while composing the invoice
 - widget to hold all the payment lines (and which knows how to add more) -- eliminanets the need for the update button while composing the invoice payments

While I do agree that this doesn't eliminate any of the problems in the database schema yet, it's definitely a way to start decommissioning the Perl code in steps. As of 1.5, we can add any regular or self-developed widget to our interface simply by adding a DIV tag and a data-dojo-type attribute. I don't expect that adding tags and attributes like that, replacing the *use* of other (generated) HTML tags and attributes, will be unmeasurably destabilizing. Note that I'm not saying that we immediately should eliminate the associated code as well, because *that* could be *very* destabilizing (as I found in the 1.4-mc work).

First, I hope I am not misunderstood here.  I am not saying we should top working on removing legacy code even on the financial side but that we really cannot refactor with a chain saw here, and that at some point, we wll need to have some discussions about the best way to create a top-notch accounting system.

For example the following appear to me to be immediately useful:

1.  The GL entry could probably be moved to new code as is.  No db rewrite needed there.  That's not too bad.
2.  We could move the screens to templates (I am not sure that services are doable in a first run to be sure and then move them to new code.  That would allow us to remove a lot of legacy code there.
3.  New features (I expect manufacturing to be a focus) can be done in a way that minimizes problems for a db rewrite.  I.e. everything should go against the gl table.
4.  Things like parts and warehousing need to be redesigned on new code anyway.
5.  There is a lot of other legacy code we can remove.

As an open source project which receives contributions in various forms (from code to sponsored development), one can expect us to work over all productive paths simultaneously in proportion to how accessible they are at the moment, and how much interest and support (in whatever form) we have.

If the external tooling approach works well for administrative tools, we could opt to do this for financial logic.  If it doesn't work so well, we will need to find another approach.

We have gone over this idea before and while I do see why you would want to try it, I don't really see how it reduces the size of the problem. Aren't you simply writing the code from scratch either way? [In your earlier attempts *and* in the attempt at externalizing the financial logic?] My thoughts are that in all cases it's a major effort that - in order to produce stable software and a manageable upgrade path for existing users - should be broken into smaller, well-understood steps that we can develop tests for in order to assess stability of the software.

I think we are both going for the same target here.  What I am hoping though is that with the financial improvements of 1.6 and continued improvement in this area in 1.7 and beyond, and with the improvements to user workflows and architecture, that we will have more options.

Also I want to specify clearly I don't think we are in a position yet to discuss the fundamentally ambitious issues like database rewrites.  For example, at some point we need to make payments first class transactions and that is going to have implications everywhere.  Moving AR and AP tables so that they join GL or similar is also going to be a major change.  We could probably rename the gl and acc_trans tables to journal_entry and journal_line, but even a change like that wouldn't happen in a day.

But I also think the more improvements we make on the current codebase, the more options we have.  So I am not saying we should stall work on it.  But I am saying i don't think this area can be refactored with a chain saw, and I think as we go forward we will want some discussions about the best way forward to create a top-notch system here.

But I don't think we can refactor the financial logic the same way we have refactored the rest.  It is going to be a lot slower and a lot more gradual.

Thanks for writing your mail! I hope for feedback from others too, so we can find a joint way forward with everybody involved in the project, hopefully even finding more contributors as we go!



http://efficito.com -- Hosted accounting and ERP.
Robust and Flexible. No vendor lock-in.


Ledger-smb-devel mailing list

Best Wishes,
Chris Travers

Efficito:  Hosted Accounting and ERP.  Robust and Flexible.  No vendor lock-in.
Ledger-smb-devel mailing list