Two-repo Netlify technique for module CI
Preface
I recently discovered that my Hugo module for responsive images was causing a great deal of bandwidth usage. This is because I had the demo/test site (including images) as a subdirectory in the module itself. To resolve this, I now have the test/demo site in a separate git repository that is only used for CI and deploying the site. The the module repository is now much smaller and only pulls the demo repository during CI.
Getting there wasn’t as easy as I would have liked, so I describe the process, and the resulting configuration, in this article.
Prerequisites
This guide assumes the use of Netlify for CI site deployment and GitHub as the hosting service for the module and demo/test site repos. If you are using other providers you will need to adjust accordingly.
Moving the large files out of the module
Make a copy of the test site, elsewhere
This is the easy part; simply use your normal OS copy operation to make sure you have a copy of all the files for your test site in a safe location (outside the repo you will be shrinking).
This is important because later in the process we will remove the files not only from the working tree, but from the history of the repository as well. This means that the files cannot be restored using Git.
Make a list of any LFS files you wish to remove
Before you remove the LFS files you will need to make list of objects using a command such as:
git lfs ls-files --all --long
Remove references to unwanted files from CI
If your CI is GitHub workflows, this easiest way to achieve this is to
move your .github/workflows folder to .github-disabled/disabled-workflows.
Once you commit and push that change, the workflows will not run for this repository. You will want to edit the workflows to adjust them to the new way of doing things, once you have removed the problem files.
Remove unwanted files and references to them
- Now you need to
git rm -fand/orgit rm -rfany unwanted files. - Then you need to edit any files that reference the removed files (as they will no longer be accessible) and commit the cleaned files.
Obliterate the large files
In order to prevent git clone or Hugo modules (which use Go modules which in
turn uses git clone) from pulling large amounts of data we need to completely
remove the files, which means we have two choices:
- Start over fresh with only the data we want.
- This results in the loss of history
- This is easier to do.
- It may also be less confusing for users of your repository.
- It may be hard to get your users to switch to the new repository.
- Rewrite history to remove any reference to the large files
- Preserves history for the files you keep.
- Some consider rewriting history an ‘Ultimate Evil’TM
- Others with branches of your repository will need to rebase not merge.
- It takes more work.
I chose option two because I wanted to preserve as much history as possible, while still achieving my goal.
Use git-filter-repo or similar tool
https://github.com/newren/git-filter-repo
For LFS, have the remote admin remove the LFS files
Note: you need to remove the files from you local repository and push that
first (per git-filter-repo or similar tool).
On GitHub, one would send the list of objects one generated to GitHub Support and ask them to remove the listed objects.
Create a new Hugo module repo with the site
Remember the copy of the test/demo site you made? Now you need to create a Hugo module repository that contains the site (in the root of the repo, not a subdirectory).
While this is out of scope for this article, it is just a standard Hugo site creation exercise.
Note that while we will use Netlify for the original module’s CI, one could deploy the demo/test site elsewhere and using other tools. The important thing is that the site is able to be pulled as Hugo a module by another Hugo module. (See Hugo’s module documentation).
Make sure the main module repository still works
Using your demo/test site which relies on the module from which your removed the site, update the site’s config to pull the ‘module under test’, and verify the site builds.
Once the site is building and the module is working correctly, you can implement CI for that module, using Netlify.
Use a subdirectory of the module as a ‘site’
The only file you need to add to the subdirectory is the
site’s config.toml or hugo.toml (for very recent releases
of Hugo, only). The files go.mod and go.sum will be added by
the procedure below.
I use the name exampleSite for the subdirectory, and will use that in this
documentation.
Add a site config in a subdirectory of the module
The ‘magic’ section of an example exampleSite/config.toml is:
[module]
[[module.mounts]]
source = "../static"
target = "static"
[[module.mounts]]
source = "../layouts"
target = "layouts"
[[module.mounts]]
source = "../README.md"
target = "assets/README-dfd-image-handling.md"
[[module.mounts]]
source = "../LICENSE"
target = "static/LICENSE"
[[module.mounts]]
source = "../ACKNOWLEDGEMENTS.md"
target = "content/post/ACKNOWLEDGEMENTS.md"
[[module.mounts]]
source = "../README-NOTES.md"
target = "content/post/README-NOTES.md"
[[module.mounts]]
source = "../i18n"
target = "i18n"
[[module.imports]]
path = "github.com/danielfdickinson/image-handling-mod-demo"
ignoreConfig = true
ignoreImports = true
[[module.imports.mounts]]
source = "static"
target = "static"
[[module.imports.mounts]]
source = "layouts"
target = "layouts"
[[module.imports.mounts]]
source = "data"
target = "data"
[[module.imports.mounts]]
source = "assets"
target = "assets"
[[module.imports.mounts]]
source = "i18n"
target = "i18n"
[[module.imports.mounts]]
source = "archetypes"
target = "archetypes"
[[module.imports.mounts]]
source = "content"
target = "content"
[[module.imports.mounts]]
source = "README-assets"
target = "assets/README-assets"
[[module.imports]]
path = "github.com/danielfdickinson/minimal-test-theme-hugo-dfd"
ignoreConfig = true
ignoreImports = true
[[module.imports.mounts]]
source = "static"
target = "static"
[[module.imports.mounts]]
source = "layouts"
target = "layouts"
[[module.imports.mounts]]
source = "data"
target = "data"
[[module.imports.mounts]]
source = "assets"
target = "assets"
[[module.imports.mounts]]
source = "i18n"
target = "i18n"
[[module.imports.mounts]]
source = "archetypes"
target = "archetypes"
[[module.imports]]
path = "github.com/danielfdickinson/link-handling-mod-hugo-dfd"
Note the use of relative paths such as ../layouts. This is how
we make the module itself available to the ‘site’ that exists as a
subdirectory of the module.
Also note the lines
[[module.imports]]
path = "github.com/danielfdickinson/image-handling-mod-demo"
ignoreConfig = true
ignoreImports = true
[[module.imports.mounts]]
source = "static"
target = "static"
[[module.imports.mounts]]
source = "layouts"
target = "layouts"
[[module.imports.mounts]]
source = "data"
target = "data"
[[module.imports.mounts]]
source = "assets"
target = "assets"
[[module.imports.mounts]]
source = "i18n"
target = "i18n"
[[module.imports.mounts]]
source = "archetypes"
target = "archetypes"
[[module.imports.mounts]]
source = "content"
target = "content"
That pulls in the demo/test site (which was created as a module) and makes it available to the build.
Initialize exampleSite as a Hugo module
For example
hugo mod init github.com/danielfdickinson/image-handling-mod-hugo-dfd/exampleSite
Activate any modules from the config.toml
You will need to be pulling in at least the demo/test site module.
hugo mod tidy
Test with the Hugo local server
Make sure you are in your project root (e.g. the directory containing the
exampleSite subdirectory), then issue the following command:
hugo serve --source exampleSite --config config.toml
You should now be able to browse to https://localhost:1313/ on same
host as where you ran that command and see the demo/test site against
the most current version of your module.
Create a build script for use with Netlify
For example save the following as scripts/netlify_build.sh in your repo:
#!/bin/bash
set -e
set -o pipefail
## Commented out lines are additional validation
## you may wish to add (out of scope for article)
# pip install pre-commit
# pre-commit install --install-hooks
# pre-commit run --all-files
# bash ./tests/scripts/hugo-audit.sh
# rm -rf public exampleSite/public
## The following is only relevant if you use `site.Params.deployedbaseurl`
## for some purpose, in a template.
# if [ "$CONTEXT" = "production" ]; then
# export HUGO_PARAMS_DEPLOYEDBASEURL="$URL"
# else
# export HUGO_PARAMS_DEPLOYEDBASEURL="$DEPLOY_PRIME_URL"
# fi
HUGO_RESOURCEDIR="$(pwd)/resources" hugo --gc --minify -b $URL --source exampleSite --destination $(pwd)/public --config config.toml
Deploy on Netlify
Follow one of Netlify’s guides, such as https://docs.netlify.com/welcome/add-new-site/#import-from-an-existing-repository to create the initial CI version of the demo/test site. Note that the repo to add is the module from which you removed demo site, not the demo site repo.
Make sure you use the above script in your build command.
Build command:
bash scripts/netlify_build.sh
Also set HUGO_VERSION environment variable on Netlify to the same version of
Hugo as you use for your local testing.
The initial deploy will likely fail because on initial import HUGO_VERSION
is not honoured.
All you need to do is trigger the deploy again. The resulting build will use
the HUGO_VERSION you specified.
Set your GitHub branch protection rules to require successful deploy
By requiring the Netlify status checks that will now be available in your branch protection settings, you can require that the Netlify build be successful before merging is allowed.
Remember: git rebase vs git merge/git pull for existing
If you (or those who have forked your repository) have existing branches with
the old version of your repo (including main), you will need to use
git checkout <branch>
git fetch <remote> <branch>
git rebase <remote>/branch
and not
git checkout <branch>
git pull <remote> <branch>
nor this
git checkout <branch>
git fetch <remote> <branch>
git merge <remote>/<branch>
Once the branch is using the rewritten version, however, one can use the usual commands.
Conclusion
As you can see there is a ‘bit’ of a process to removing a large demo/test site from a module and replacing that with a separate CI deployment for the module versus the demo/test site main deployment, but it is achievable.
Finally, if you want to see a pair of public repositories where I have implemented this, you may view:
image-handling-mod-hugo-dfd
and
image-handling-mod-demo
which are deployed as
https://image-handling-mod-ci.demo.wildtechgarden.com
andhttps://image-handling-mod.demo.wildtechgarden.com
respectively.
Good luck!