[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Cleaning up the repository. How does the community feel about it?

Hi everyone,

Erik (ehuelsmann), John (freelock), and myself (dcg) have been discussing the possibility of stripping at least dojox (unused) and preferably  all of the dojo source tree from the LedgerSMB repository primarily to reduce the download size, and disk footprint.

It is worth pointing out that it is undesirable to use an OS package to provide dojo as there are known to be issues from minor release to minor release that would break LedgerSMB. This forces us to use and update our own local version of dojo which has historically been the complete source held within the UI/lib/dojo/ directory

If we do remove the entire dojo src it would be replaced with a "built" version of dojo that only includes the parts we are using.
This has a number of benefits,
    * reduced download size at install or update,
    * during runtime
        * reduced number of server requests,
        * reduced number of bytes downloaded from the server.

In testing it was found that the following size improvements were obtained.


 lsmb-test-full
 lsmb-test-shrunk lsmb-test-shrunk-2

includes
    dojo
    dojox
 removed
    dojox
 removed
    dojo
    dojox
download size **
 43.64 MiB
 27.94 MiB  (-15.60 MiB)
 21.17 MiB (-22.47 MiB)
total size on disk ***
 151 MiB
 89 MiB (-62 MiB) 63 MiB (-88 MiB)
 




*    To obtain these gains dojox and or dojo were removed from the entire history of the repository using a git tool called "bfg"
     Once the built version of dojo is added, there will be a small size increase that is yet to be determined but probably about 1-2M (download)
**   taken from git pull progress after completion
***  obtained with "du -sh . " from the top level of the repo


To allow development changes to the dojo source we use, it would be moved to a github submodule that would only need to be used by developers working directly on the dojo source.

The benefits are significant, even if we only remove dojox.
Consider the fact that an active developer like Erik may do a fresh clone of the repository 10 to 20 times in a day,
a saving of 15M to 20M per clone (400M) can save a lot of time spent waiting.

There are however some downsides that will only affect some users and developers.
If a you have copy of the repository with local changes there will be a one off requirement to either rebase, or merge the local changes with the updated repository.
If we only remove dojox, this should be a simpler task, however if we remove the entire dojo source it could be more work.

We are still doing tests to verify how much work this would be for both cases,
but decided it was time to ask for comments from the community.

The three sample repositories were created as a snapshot in time from the normal ledgersmb repository using the methods outlined below.

lsmb-test-full          will work to run LedgerSMB, but this is NOT RECOMMENDED
lsmb-test-shrunk    will in theory work to run LedgerSMB, but this has not been tested at the time of writing
lsmb-test-shrunk-2 will not be usable to run LedgerSMB from as dojo source has been removed and there is no built version included.

All three lsmb-test-* repositories contain the exact same LedgerSMB code.
The only differences are the removal of dojo/dojox from the shrunk repos

These lsmb-test* repositories are for testing what will happen if/when we strip dojo/dojox from the repository.
Please do not use them for any other purpose.
They will be kept in exactly the same state until a decision to make the change or not is made, at which time they will be removed.

Keep an eye out for a follow up (reply to this thread) email with some things we would like people to test.


Regards
David G


Create A full Mirror

http://blog.plataformatec.com.br/2013/05/how-to-properly-mirror-a-git-repository/

  1. Create an empty repository on github (in this case lsmb-test-full on my account sbts)
  2. git clone --mirror https://github.com/ledgersmb/LedgerSMB
  3. cd LedgerSMB.git
  4. git push --mirror https://github.com/sbts/lsmb-test-full.git
    You can expect to see some messages ending in"(deny updating a hidden ref)"
    They don't seem to be a problem and are most likely pull requests that git prevents you from readding to a repo
  5. cd ..
  6. [optionally]  rm -rf LedgerSMB.git

clone the original repo

  1. git clone https://github.com/ledgersmb/LedgerSMB.git

clone the new repo

  1. git clone https://github.com/sbts/lsmb-test-full.git

you can now do some checks to make sure the 2 repo's are the same.


Lets try shrinking it by deleting dojox from all of history.

https://rtyley.github.io/bfg-repo-cleaner/

The steps are very similar to creating a mirror, we just need to run BFG before we push to the new repo (which can also be the original repo if you want).

we will use a sample repository in my userspace (sbts) that was created using the above Mirror technique from lsmb-test-full
https://github.com/sbts/lsmb-test-shrunk.git

  1. download bfg .jar using wget http://repo1.maven.org/maven2/com/madgag/bfg/1.12.8/bfg-1.12.8.jar
    this link is from near the bottom of https://rtyley.github.io/bfg-repo-cleaner/
  2. git clone --mirror https://github.com/sbts/lsmb-test-shrunk.git
  3. java -jar bfg-1.12.8.jar --delete-folders dojox --no-blob-protection lsmb-test-shrunk.git
  4. cd lsmb-test-shrunk.git
  5. git reflog expire --expire=now --all && git gc --prune=now --aggressive
  6. git push
  7. cd ..
  8. [optionally]  rm -rf lsmb-test-shrunk.git

The same shrinking process was used to create lsmb-test-shrunk-2 but removing dojo instead of dojox

------------------------------------------------------------------------------

_______________________________________________
Ledger-smb-users mailing list
..hidden..
https://lists.sourceforge.net/lists/listinfo/ledger-smb-users