Re: Initial steps for 1.6, and some longer-term questions

On 10/31/2015 11:50 PM, Chris Travers wrote:

There are a few things architecturally however that I would hope to accomplish with 1.6:

1. From 1.2 through 1.4, ease of installation has not been a priority and has degraded We need to reverse that. Ease of installation and management needs to be a top priority. I think we have the back-end stuff down, so the question is front-end and administrator tooling.

I would say this is particularly true: we don't have a web tool which points out which libraries or other dependencies are missing from an installation. Tools like Zabbix have that, allowing an admin to quickly address the issue(s) and assess the level of support for any of the optional features/dependencies. This could seriously improve our "first user experience".

Well, I do think that using Docker can alleviate a lot of this pain, in 2 ways:

1. Provide a "ready-to-run" installation provisioned in two steps (run a Postgres container, run an LSMB container).

2. Provide a supported installation recipe for anyone who wants to run it themselves (the Dockerfile). We could support multiple flavors of this: install from git (what the current dockerfile does), install from tarball, install from package. And of course, some steps for creating a development environment (e.g. use Docker to provide all dependencies but mount a git tree into the container that you can develop on directly).

There are some small blockers/disadvantages here, but this is close to being ready for use, and I've been using a Dockerized install in production for many tasks. The current downsides/blockers:

- Size. Current images are > 1.5GB, largely due to LaTeX and other dependencies.
- Issues #854, #917 -- templates cannot be easily managed in the filesystem in Docker, so until all templates can be loaded into the database, this is impractical for casual users.
- Server-side printing -- this might be something relatively easy to solve by setting up some print variables and mounting a print device inside the container, have not addressed this yet.

So we currently use the regular CGI for printing checks and sending invoices, but have migrated most other activities into Docker, with the much faster Starman configuration (2 instances sharing the same database). And for new users, this is a really fast way to get a working install up and running.

On a technical side, I would like to switch to external tooling for administration and management. This means one gets the admin tools, and can install and manage LedgerSMB instances from there. It also means these tools need to play really well together.

I'm affraid this sounds pretty vague from just this sentence. I'm affraid people will have a hard time judging how this will impact them. Will we still deliver these tools in such a way that installation will be easy? Will these be part of the source distro so packagers for distributions automatically get the right sources to package and distribute? Could you elaborate a bit more on your plans in that respect?

Other than that, if the tools come with the same source tarball and are a huge improvement over what we have now, I think we only can cheer to that :-)

I was actually thinking of a couple different approaches we could take here.

1. Install the admin and installation tools. Run them to check your system, download, and install latest versions of whatever major branch you want (starting with 1.6). Such could also update a system and could be run with different permissions than the main web server. In other words, you:
   a. download a tarball.
   b. Run a program. It checks your system. If it can start a web server on a higher port it does so. Otherewise it gives clear feedback about what you are missing.

   c. Log into the web interface for more system diagnostics, installation and management help.

Cool idea. But is this the best use of our time?

With the rise of Docker, I think a lot of this is becoming largely unnecessary. We can create a single supported environment that can run anywhere, even on Windows, and not have to support all the possible variations of environments -- we should be focusing our efforts on the financial system rewrite, not admin tools to fix a bunch of different environments.

If somebody really wants a bare-bones install in their specific environment and not run in a container, that sounds like a perfect opportunity for paid consultants to step in -- for the core team, this seems like a huge distraction.

2. We could bundle all our tools together. In this case you still run the same program which checks your system, sees whether it can run a web server and if not gives information as to why not. Such a program would then typically not update a system but could change file permissions after running and setting things up.

In both cases, the installer checks external prerequisites, CPAN dependencies, and can either solve or advise on how to solve missing dependencies. I would like LedgerSMB 1.6 to be easier to install than SQL-Ledger 2.6 was. Mostly this is hand-holding, providing clear feedback on how to resolve problems, and a nice interface for showing the information.

I would argue that we should skip all the dependency-checking, etc, and point people either to packages for Debian/RHEL/etc, or the Docker images.

Where we do need to spend time hand-holding is once the system is up and running -- setting up the Chart of Accounts, providing guidance on user permissions, how to set up bulk imports, etc. Nice interfaces for showing best accounting practices, templating guidance, etc.

> I also want to make sure we have both command line and web tools. Command line tools give experienced administrators a productivity edge here (because it is easier to quickly send complex information to the program), but web tools give new users an ability to get up and running with less immediate knowledge.

By "command line" tools, do you mean something more than what's already there -- the ability to run any of the .pl scripts directly and pass in info using a Bash script, for example?

I'm not sure command line/shell tools are the real need -- I think the critical thing is to get an API in place, bit by bit. We can certainly wrap those with a shell tool, but I think the real need is for a solid API.

The rest of the mail goes into specifics, which all seem appropriate to me. I just wanted to make the point that an API can help us clean up the legacy code. If we start developing a solid API, module by module, and using that to interact with the existing financial code, then it becomes far easier to swap out the implementation when we get there. I know this is easier said than done, given the spaghetti logic in there ;-)

Perhaps the API can be done as a 2-step approach? I'm thinking we already discussed tagging the API with a version. What if we start a v2 right now -- v1 being essentially a wrapper around the existing POST variables that all the forms currently expect, and v2 being what we think it should look like, where this varies? Then for each module, we could go through the current code, and replace each variable in the module that comes from the browser/form post with something that we set in the API handler for that module -- and can then create both v1 and v2 at the same time. Once we have a stable v2 API across the board, then we have a great framework for unit testing in place and can far more reasonably rip out and rewrite the core logic.

I'm seeing the API as functional scaffolding to help with the financial rewrite, as well as being an end in itself...

Cheers,
John Locke
http://www.freelock.com