Re: Beyond 1.5

Heh, accidently replied off-list, which is probably good since a real keyboard ensures I don't skip over some related things.

On Tue, Dec 13, 2016 at 7:23 AM, Erik Huelsmann <..hidden..> wrote:

John and I were exchanging mails with some of the other developers on the project when one thing led to another and suddenly we were discussing how to proceed beyond 1.5, how to get the much desired old code eliminated and how to get the financial rewrite done. I'm bringing it to this list because I think the subject concerns the community at large - not just a few of the developers.

Quoted text below from John Locke's mail.

[ .. snip .. ] we've been dancing around old code for a long, long time -- are we ever going to clean it up incrementally?

While I've been hesitant to answer that question in the past (but always believed in it), I can answer with a firm "Yes!" these days. The plan is as follows, as far as I'm concerned:

* Isolate old code so it's completely clear to everybody which code we want to remain untouched
* Create Dojo widgets to gradually replace functionality in old code with a well defined web API and a client UI
* Create web services to support the Dojo widgets (and other remote access, of course)

As a matter of proof that this is going to work to phase out old code: we already have a functionality in 1.5 which follows exactly this approach and has led to deprecation of a significant complexity in old code: the parts selector in the invoice, order, quotation, customer price matrix and parts (bom listing) screens has been replaced by a single Dojo widget.

However as a note, this does not touch the really hard/interdependent parts. Also I think it would be a serious mistake to build an API around assumptions we intend to change, because otherwise we haven't decoupled anything.

I'm thinking we're better off at this point with a rewrite -- or at least for the time being, a solid enough plan in place so that the new stuff we're creating will port straight into a 2.0 when the time comes.

Well, given that we have found a way forward now to get rid of the old code, I'm thinking we should proceed on that route for a while -- it might just be the key we needed.

Right, and in areas we can do so, I think we should keep with that approach for a number of reasons (easier maintenance, less old code, and more). Maybe it will succeed everywhere and I will be proven wrong. But having worked with areas where I think we will run into real problems, I don't think it's the only approach we should take.

It's also the main driver behind my efforts to start talking about the document state diagram and the requirements for a web service framework: it's *the* driver to be able to continue on this route and a major enabler of a lot of other efforts.

That's extremely valuable work regardless of how things go btw. I think we need to have all document states discussed and documented in all cases.

If the core issue keeping us from tackling the financial rewrite and heading for 2.0 is the database schema, I think we should drill into that and make that solid, as soon as we possibly can. [ .. snip .. ]

Agreed. My definition of "as soon as we possibly can" would be: once we have phased out the old code. Seriously, there's a lot of dependency in old code on the current schema, without the sorely needed separation of concerns. Rewriting the database schema now would require editing a lot of old code -- something we really don't want to do. As soon as we have removed our dependency on old code, it'll become a lot easier to rewrite the stored procedures. It'll also become possible to rewrite without breaking it *all*, because we'll have many, many more tests in place to validate consistency and correct operation.

But then we either have an API that is built on assumptions in the current db (in which case still have a lot of the same coupling issues *and* we have to maintain that)...

I can definitely see that it may take us years to get there, and we may have several more 1.x releases -- but clearly there's a need here, and the longer we wait to start on it, the longer it will take to get it working...

While in general I would agree, I'm also seeing a major uptake in development activity over the past 12 months. We have put in place a number of enablers to help people join the project in the same period. To me, it seems the strategy is working and while in the past development may have been slow and painful, it's now "just" painful, but no longer slow.

Here's my recommendation: we do both at once. Nip out low hanging areas in the old code. Start a branch for 2.0. There are a couple major reasons I think we should do this:

1. I think we need to accept that 2.0 will be a db *import* rather than a db *upgrade* process. Given the depth of the changes I think we will need to do, customizations are not going to just work no matter how we do it. We might as well plan on copy-and-import rather than migrate in place.

2. I think we need to understand that this *will* break customizations in big ways and I don't think we want to guarantee any backwards compatibility.

3. We *really* do not want to be stuck trying to maintain 1.x compatibility in an API.

When we have more testing in place (BDD or otherwise), development won't even be painful anymore.

So... what's wrong with the financial schema

A *very* relevant question which I think should be answered in the short term so we can make sure that all changes move in the right direction. I however, don't have the answer.

@Chris, do you have mails, notes, <whatever document> which lists all or part of the problem(s).

(Ok, in all fairness, I know about:
- Journals should not run across dates
- payments should be journals, not lines on an AP item
- journals should be referenced from 'transactions' or 'gl', not from ar/ap
)

Well, no specific documentation and notes. However they have been spread across a large number of emails over the years. Some problems we have been able to fix but a lot we can't.

Here are the top issues in order of the damage they cause.

1. Payments are not first class. This causes problems in payment management, invoice management, banking reconciliation, and much more. It also makes it very hard to come up with a solid way of managing invoices over an API.

2. AP/AP/GL tables splitting up unnecessary duplication causes a lot of headaches in reporting. Also it means when we fix this, we have to rewrite *all* the report queries.

3. The way things are split up gives us no real hard enforcement of referential integrity. We have partial enforcement but not what I think we need.

--
Bye,

Erik.

http://efficito.com -- Hosted accounting and ERP.
Robust and Flexible. No vendor lock-in.

Best Wishes,

Chris Travers

Efficito: Hosted Accounting and ERP. Robust and Flexible. No vendor lock-in.

http://www.efficito.com/learn_more