[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Cleaning up the repository. How does the community feel about it?



Hi,

John asked if those users that have modified repositories could do some tests for us.

<quote>

I would say that to give us feedback on whether this will work, and what kinds of trouble users may face, ask people to do the following:

1. Make sure all your changes/local values are committed
2. Make an entire copy of an existing git clone of LSMB they use in production, especially any containing modified code
3. In the copy, edit the .git/config file, and under [remote "origin"] (or whatever remote they have set up for our main git repo) change:

    url = "" class="moz-txt-link-abbreviated" href="mailto:..hidden..:ledgersmb/LedgerSMB.git">..hidden..:ledgersmb/LedgerSMB.git
-- or --
    url = "" class="moz-txt-link-freetext" href="https://github.com/ledgersmb/LedgerSMB.git">https://github.com/ledgersmb/LedgerSMB.git

... to ...
    url ="" moz-do-not-send="true" class="moz-txt-link-freetext" href="https://github.com/sbts/lsmb-test-shrunk.git"> https://github.com/sbts/lsmb-test-shrunk.git

4. ... and then attempt to update using their normal process (git fetch origin; git merge origin/1.4 is what I typically do)

5. Let us know if this works, or if it all blows up!


For cleanup of failed merges, here are some things to try:

* git merge --abort
* git rebase --abort
* git reset --hard HEAD
* git clean -df (try with -nd first to make sure you don't destroy unexpected files)


Merge strategies to test:

* git merge origin/1.4 (or master) - for me, a bunch of merge conflicts
* git rebase origin/1.4 - for me, a bunch of merge conflicts
* git merge -s theirs origin/1.4 (not available in older git versions, but should do the right thing if you have no patches to core files, only new additions)

... if none of those work, two other alternatives to try:

a. Merge with strategy "ours." If you have local modifications, find the last commit that you merged in from the LedgerSMB master, and use "git log origin/1.4" (or master) to identify the new commit id. For example, I see the last commit before I merged is in the new branch as 23dcc024d. You should then be able to:

git merge -s ours 23dcc024d # this puts all the commits up to this point into your history, without changing anything in the working copy
git merge origin/1.4  # this gets you the rest of the way up to date


b. Manually resolve merge conflicts. For each unmerged path, unless you know you have a local change, you should be able to grab the new version from the new upstream and add it to the index. For my copy, this looks like it will do it:

git checkout origin/1.4 LedgerSMB UI Locale utils
git commit


Cheers,

John Locke
Principal, Freelock
..hidden..
206-577-0540 x20

Sign up for News and Drupal updates from Freelock!

</quote>

On 31/12/15 22:50, David G wrote:
Hi everyone,

Erik (ehuelsmann), John (freelock), and myself (dcg) have been discussing the possibility of stripping at least dojox (unused) and preferably  all of the dojo source tree from the LedgerSMB repository primarily to reduce the download size, and disk footprint.

It is worth pointing out that it is undesirable to use an OS package to provide dojo as there are known to be issues from minor release to minor release that would break LedgerSMB. This forces us to use and update our own local version of dojo which has historically been the complete source held within the UI/lib/dojo/ directory

If we do remove the entire dojo src it would be replaced with a "built" version of dojo that only includes the parts we are using.
This has a number of benefits,
    * reduced download size at install or update,
    * during runtime
        * reduced number of server requests,
        * reduced number of bytes downloaded from the server.

In testing it was found that the following size improvements were obtained.


lsmb-test-full
lsmb-test-shrunk lsmb-test-shrunk-2

includes
    dojo
    dojox
removed
    dojox
removed
    dojo
    dojox
download size **
43.64 MiB
27.94 MiB  (-15.60 MiB)
21.17 MiB (-22.47 MiB)
total size on disk ***
151 MiB
89 MiB (-62 MiB) 63 MiB (-88 MiB)




*    To obtain these gains dojox and or dojo were removed from the entire history of the repository using a git tool called "bfg"
     Once the built version of dojo is added, there will be a small size increase that is yet to be determined but probably about 1-2M (download)
**   taken from git pull progress after completion
***  obtained with "du -sh . " from the top level of the repo


To allow development changes to the dojo source we use, it would be moved to a github submodule that would only need to be used by developers working directly on the dojo source.

The benefits are significant, even if we only remove dojox.
Consider the fact that an active developer like Erik may do a fresh clone of the repository 10 to 20 times in a day,
a saving of 15M to 20M per clone (400M) can save a lot of time spent waiting.

There are however some downsides that will only affect some users and developers.
If a you have copy of the repository with local changes there will be a one off requirement to either rebase, or merge the local changes with the updated repository.
If we only remove dojox, this should be a simpler task, however if we remove the entire dojo source it could be more work.

We are still doing tests to verify how much work this would be for both cases,
but decided it was time to ask for comments from the community.

The three sample repositories were created as a snapshot in time from the normal ledgersmb repository using the methods outlined below.

lsmb-test-full          will work to run LedgerSMB, but this is NOT RECOMMENDED
lsmb-test-shrunk    will in theory work to run LedgerSMB, but this has not been tested at the time of writing
lsmb-test-shrunk-2 will not be usable to run LedgerSMB from as dojo source has been removed and there is no built version included.

All three lsmb-test-* repositories contain the exact same LedgerSMB code.
The only differences are the removal of dojo/dojox from the shrunk repos

These lsmb-test* repositories are for testing what will happen if/when we strip dojo/dojox from the repository.
Please do not use them for any other purpose.
They will be kept in exactly the same state until a decision to make the change or not is made, at which time they will be removed.

Keep an eye out for a follow up (reply to this thread) email with some things we would like people to test.


Regards
David G


Create A full Mirror

http://blog.plataformatec.com.br/2013/05/how-to-properly-mirror-a-git-repository/

  1. Create an empty repository on github (in this case lsmb-test-full on my account sbts)
  2. git clone --mirror https://github.com/ledgersmb/LedgerSMB
  3. cd LedgerSMB.git
  4. git push --mirror https://github.com/sbts/lsmb-test-full.git
    You can expect to see some messages ending in"(deny updating a hidden ref)"
    They don't seem to be a problem and are most likely pull requests that git prevents you from readding to a repo
  5. cd ..
  6. [optionally]  rm -rf LedgerSMB.git

clone the original repo

  1. git clone https://github.com/ledgersmb/LedgerSMB.git

clone the new repo

  1. git clone https://github.com/sbts/lsmb-test-full.git

you can now do some checks to make sure the 2 repo's are the same.


Lets try shrinking it by deleting dojox from all of history.

https://rtyley.github.io/bfg-repo-cleaner/

The steps are very similar to creating a mirror, we just need to run BFG before we push to the new repo (which can also be the original repo if you want).

we will use a sample repository in my userspace (sbts) that was created using the above Mirror technique from lsmb-test-full
https://github.com/sbts/lsmb-test-shrunk.git

  1. download bfg .jar using wget http://repo1.maven.org/maven2/com/madgag/bfg/1.12.8/bfg-1.12.8.jar
    this link is from near the bottom of https://rtyley.github.io/bfg-repo-cleaner/
  2. git clone --mirror https://github.com/sbts/lsmb-test-shrunk.git
  3. java -jar bfg-1.12.8.jar --delete-folders dojox --no-blob-protection lsmb-test-shrunk.git
  4. cd lsmb-test-shrunk.git
  5. git reflog expire --expire=now --all && git gc --prune=now --aggressive
  6. git push
  7. cd ..
  8. [optionally]  rm -rf lsmb-test-shrunk.git

The same shrinking process was used to create lsmb-test-shrunk-2 but removing dojo instead of dojox



------------------------------------------------------------------------------


_______________________________________________
Ledger-smb-devel mailing list
..hidden..
https://lists.sourceforge.net/lists/listinfo/ledger-smb-devel

------------------------------------------------------------------------------
_______________________________________________
Ledger-smb-devel mailing list
..hidden..
https://lists.sourceforge.net/lists/listinfo/ledger-smb-devel