Re: Proposal regarding 1.6

On Mon, Oct 12, 2015 at 4:12 PM, Erik Huelsmann <..hidden..> wrote:

Hi Chris,

I have been thinking a bit regarding ways forward. In particular the financial rewrite never seems to go the way I would like it to.

We spoke a bit about this over PM last week. Maybe we can use this (or a new) thread to define what it means to you? I'm not really clear on what the term "financial rewrite" really means to you. Although I do have ideas about breaking down the scope of rewriting the remaining SQL Ledger code into a larger number of smaller rewrites. Before we can do that, though, I think we have a number of areas where we should put requirements/design/workflow in place. I'm thinking about things like these:

Long run for the financial rewrite we need a few things:

1. Payments need to be first-class transactions.

I *think* I know why you say this: it's because of the cash-based reporting that it's required, isn't it? Are there any other reasons? I'm asking, because I thought the same thing, but now I'm torn (although I thoroughly dislike "transactions" running across multiple days): If payments are first-class transactions, then don't we put the burden on AR/AP transactions to link to payments [which is simply the reverse from what we do now?] and if that's the case, then won't it simply be true that *other* queries become problematic (but the problem of complexity by itself hasn't been solved?).

Yeah. It makes it hard to identify unique payments for things like reversal.

A payment as a transaction cannot shouldn't run across multiple days though. At any rate for reversal purposes, a payment is considered a unique source for a unique customer or vendor on a unique date.

Having first-class payments as transactions would make things much easier when looking at automatically integrating ach payments and the like or in cases where we don't yet have a source identifier.

Don't get me wrong: I *do* want things to improve, however, you've spent a fair amount of time on trying to improve things without much having come from it -- by bringing forward all these questions, I hope we can get to the bottom of what you learned so far and use that to our advantage.

These are good questions.

There was some discussion between the two of us a few months back about cash reporting and the Cashflow Statement though that "simply" marking accounts as "cash related" didn't work.

The problem with the cash flow statement is different. It is because despite the name it is actually an accrual report, but the sectioning is different than what we support. So supporting that report requires additional categorization on accounts.

What I remember is that it *mostly* worked, but the problem came when assets were sold off at their end-of-life. What I remember is that the cash flow of the end-of-life sale is to be reported as gains/losses from investment activities, which was a problem given that the regular movements on the investment section (depreciation) is a loss from operational activities. To understand the true problem again, I/we'll have to go through the entire cycle of creating a mapping of accounts to a cashflow statement again.

Yeah, I think you are thinking of the cash-basis income statement? I suspect things may have different names in different places?

2. We would have much better performance and clearer code if the journal tables were unified or split off in better ways.

Agreed. Although this is a change with widespread consequences in the Perl code, it's a pretty minor change in the sql code, I think (yes, the three tables are union-queried all over the place, but if they are mostly replaced by a single table listing all transactions, many uses of the ar and ap tables can simply be dropped.)

Well, in the end, not addressing with this proposal how and when we get there. I think there are a lot of things to think about such as whether we need to move code anyway onto Dancer, and if we need to do that, what the best way forward on that is. So after our previous conversation I am not sure we are in a place yet to map out exactly what needs to be done to rewrite the financial code. I do however think important steps can be taken in 1.x in this direction.

So that means getting off the current SQL-Ledger perl code and db design.
I can see why rewriting the payments is dependent on rewriting AR/AP. However, if we take into account that at some point we want to rewrite payments, while rewriting AR/AP, I think it should be doable (but not a task to take lightly).

Since, like the multi-currency effort, there's considerable effort to be spent before benefits can be seen, I'd like to start by drawing a picture of what it should look like past the horizon: drawing up database diagrams and workflows, so we can evaluate the schema in some of the more contrived use-cases such as we did when we evaluated the changes required for the cashflow statement. [My questions above regarding payments basically goes to establish the same.]

Agreed. I think we will need some good discussion pieces. But before we get too far there, I would like to put time in first on addressing immediate shortcomings.

But I now think the first steps involve taking easier tasks and using them to figure out how to get some of the complexities in the current application simplified. Hence making the contact management code nice, clean, service-oriented may be the single biggest step we can take to get positioned to do this.

- Document workflow: Create -> Save -> [ Edit -> Return to Save ] -> Post -> Reproduce [ -> Copy to new ] [ -> Schedule ]
- Rounding
- Subcent prices on anything that's not in the ledger (parts prices?)

stuff like that.

Well, parts and manufacturing can be done in place I think.

Ok. And by switching to services with dojo widgets for vendor&customer selection, we can cause AR/AP(invoicing/transactions) to be much more loosely tied to ECAs than they are now.

Yeah, that's kind of where my current thoughts are going.

Thinking we should do the same for the e-mailing functionality (which is currently hopelessly tied to the invoices, orders, etc) and for the shipping addresses (same line of reasoning). That should greatly simplify "unexpected" code flows in the old code: each of these steps will be the conversion of replacing an existing "code+html" mix from the old SL code with a few targetted specifically services and a dojo widget. My vision is that all that's required for instantiating a dojo widget to e-mail an invoice will be to provide it an eca id, a purpose (billing, sales, etc) and a document-service-url. The Dojo widget+dojo services would do the rest.

Agreed that could be a part of it.

We have at least one important feature slated for 1.6, and that is the multi-currency improvements Erik has been working on. I would like to suggest we do one other thing as well, namely finish App::LedgerSMB::Admin and App::LedgerSMB::Admin::Web, and use these to replace the current setup.pl.

It's my understanding that these will serve a much broader range of functionalities than setup.pl, am I correct? More like a toolset for managing companies within a PostgreSQL cluster (i.e. multiple databases)? Is there anything more to say about it than that?

Agreed generally. My view is though: removing Setup.pl and making sure the new admin tool is well integrated gives us some experience integrating Dancer apps with the main application. This is something we will need experience with going forward anyway, and I am also helping that a good, purpose built admin tool will help to some extent with our setup and installation difficulties.

Sounds good. Having a better -specifically targetted- installation tool will help a lot with our first user experience. I'm not sure I understand the issue about the Dancer apps integration though: We have everything for both old and new code running on Plack, which would be a great place to do URL-remapping to have one consistent URL space presented to "the outside world" while working with different paradigms internally. Or at least, that's what I'd like the world to look like :-)

It's just a matter of setting up plack wrappers and routers for the most part. But there may be design stuff that needs to be figured out there.

It also occurs to me that if the admin tool is sufficiently service-oriented, then the installation wizard could be a _javascript_ app bundled with the main program.

There is one limitation that has occurred to me and that is that the current approach returns html in most cases rather than using services and dojo. Although I think we support services for xhr calls, that doesn't do much for things like lwp. If we do a clean service/dojo API though then that goes away. I think getting this right would be a good step towards getting started on other service areas.

Work that is left to do here is to add a setup wizard and add user management functions to the web interface. A major advantage would be an ability to manage multiple LedgerSMB instances from one console, better command line management, and restful interfaces for core administrative functions.

Also, version 1.x of the admin interface uses jquery and templates in a fairly old-fashioned way. We may want to move to dojo there and start a 2.x branch of this.

Sounds good, but I think that creating services early on allows easier dojoification, because Dojo applications typically benefit from a well structured service infrastructure on the server. Having a full-scale web app based on templates before going Dojo will be a lot of effort wasted on the 'traditional' webapp.

Service are largely supported already in the admin tool, iirc, especially for things like backup and restore. And moving the rest to services and dojo here is extremely easy (even trivial).

Great! :-) (We probably want to do that then...)

If we want to go with Dancer as an http and web services framework, I think this would be the first step.

Agreed. Would the second step be the above external/internal URL mapping and hooking up existing functionalities?

Yeah.

That would allow us to remove

LedgerSMB::Darabase
LdgerSMB::Scripts::setup
and maybe a little more.

Removal is generally good. There's one important question though: should we move these into our tree or not? As we factorize many components, it might become harder for people to install the factored-out dependencies. While this may or may not be the case, I think it's something to think about long and hard before we throw up road blocks to installing LedgerSMB.

Agreed generally. Again part of what I am hoping is that if we do this well, it will help with those roadblocks and at least give us a path forward on solving them.

Yea. If this helps resolve with (some) of our installation issues, that'd be a huge step forward, really.

I think it would give us a better platform to address installation problems.

I would also like to suggest we look closely at one of the following areas (would be interested in hearing where folks largest frustrations are here) in terms of UX, web services, and the like:

chart of accounts, or contact management.

Any thoughts on prioritiing these?

If I can have a vote: Please start with services for contact management. That'll allow easier Dojoification of the contact management screens, which hopefully will help with the clarification of them.

So maybe we should prioritize for 1.6:

1) Integration of dancer apps (probably the new Admin tool)
2) Installation and loading.
3) Contact management (if the other two go well, maybe another dancer app with a dojo front-end?)

Since you bring up installation above, I am wondering if it would make sense, since Dancer apps can run entirely locally on a non-privileged port, to have a web interface that installs all Perl dependencies to a local directory?

That could be a good idea. If I look at the Zabbix experience, it's at least able to tell me what's missing *and* *how* to install it (generic instructions, not distro-specific). That'll help a *lot*.

That's a good point too.

For 1.6, I'd like to push forward with the QA work that's ongoing now. I'm thinking that with all the changes we have behind us and so much move even ahead, we really want to make sure the various parts in the system work correctly. Even though QA can be a big investment, even the little bit that I did so far has helped me out in a number of occasions already.
In this area, I'm thinking of pushing for SauceLabs integration and get something up and running on Travis CI (and preferably for local testing too) for testing webserver requests.

Agreed and I think with 1.5 we are now in a point where we can more readily test these things.

QA isn't a priority by itself though; it's more of a parallel priority, which is important regardless of the work being delivered. Any step in that area is a major step forward.

Automated tests would help quite a bit as we go through beta periods though.

--
Bye,

Erik.

http://efficito.com -- Hosted accounting and ERP.
Robust and Flexible. No vendor lock-in.

------------------------------------------------------------------------------

_______________________________________________
Ledger-smb-devel mailing list
..hidden..
https://lists.sourceforge.net/lists/listinfo/ledger-smb-devel

Best Wishes,

Chris Travers

Efficito: Hosted Accounting and ERP. Robust and Flexible. No vendor lock-in.

http://www.efficito.com/learn_more