Extra modules cleanup into a script and document it

**What**
- Extract the vendor and modules cleanup into a separate shell script.
  This provides more space for documenting each step cleanly.
- This scripting also removes the need for the `GO_REPLACE.txt` hack,
  it will automatically modify any local replacements in the original
  go.mod file, so that they work as expected.
- Update the README to describe how dependencies work in the three
  supported modes: modules with vendoring, modules without vendoring,
  and traditional vendoring via dep.

Signed-off-by: Lucas Roesler <[email protected]>

Author: LucasRoesler

README.md

@@ -28,6 +28,7 @@ The two templates are equivalent with `golang-http` using a structured request/r
 You can manage dependencies in one of the following ways:
 
 * To use Go modules without vendoring, add `--build-arg GO111MODULE=on` to `faas-cli up`, you can also use `--build-arg GOPROXY=https://` if you want to use your own mirror for the modules
+* You can also Go modules with vendoring, run `go mod vendor` in your function folder and add `--build-arg GO111MODULE=on` to `faas-cli up`
 * For traditional vendoring with `dep` give no argument, or add `--build-arg GO111MODULE=off` to `faas-cli up`
 
 ## 1.0 golang-http
@@ -361,7 +362,37 @@ func Handle(w http.ResponseWriter, r *http.Request) {
 }
 ```
 
-#### Advanced usage - Go sub-modules via `GO_REPLACE.txt`
+#### Advanced usage
+
+##### Sub-packages
+
+It is often natural to organize your code into sub-packages, for example you may have a function with the following structure
+
+```
+./echo
+├── go.mod
+├── go.sum
+├── handler.go
+└── pkg
+    └── version
+        └── version.go
+```
+
+First update your go.mod file to replace `handler/function` with your local folder
+
+```go
+go mod edit -replace=handler/function=./
+```
+
+Now if you want to reference the`version` sub-package, import it as
+
+```go
+import "handler/function/pkg/version"
+```
+
+This replacement is handled gracefully by the template at build time and your local development environment will now recognize the sub-package.
+
+##### Go sub-modules
 
 For this example you will need to be using Go 1.13 or newer and Go modules, enable this via `faas-cli build --build-arg GO111MODULE=on`.
 
@@ -384,16 +415,13 @@ func Echo(w http.ResponseWriter, r *http.Request) {
 }
 ```
 
-To include a relative module such as this new `handlers` package, you should create a `GO_REPLACE.txt` file as follows:
+To include a relative module such as this new `handlers` package, you should update your `go.mod` file as follows:
 
 ```
-replace github.com/alexellis/vault/purchase/handlers => ./function/handlers
+go mod edit -replace=github.com/alexellis/vault/purchase/handlers=./handlers
 ```
 
-How did we get to that? Let's say your GOPATH for your GitHub repo is: `github.com/alexellis/vault/` and your OpenFaaS function is `purchase` generated by `faas-cli new purchase --lang golang-middleware`.
-
-Your relative GOPATH is: `github.com/alexellis/vault/purchase`, so add a redirect as per below to redirect the "handlers" package to where it exists in the build container.
-
+At build time, this relative path will be handled correctly inside the template.
 
 Now if you want to reference the handlers package from within your `handler.go` write the following:
 

template/golang-http/Dockerfile

@@ -20,49 +20,10 @@ COPY . .
 ARG GO111MODULE="off"
 ARG GOPROXY=""
 ARG GOFLAGS=""
+ARG DEBUG=0
 
-# Add user overrides to the root go.mod, which is the only place "replace" can be used
-RUN cat function/GO_REPLACE.txt >> ./go.mod || exit 0
-RUN if [ -d ./function/vendor ]; then \
-    echo "moving handler vendor" && \
-    mv -f ./function/vendor .;\
-    else \
-    echo "vendor not found "; \
-    fi
-
-RUN if [ "${GO111MODULE}" == 'on' ]; then \
-        # copy the user's go.mod
-        mv -f ./function/go.mod . && \
-        mv -f ./function/go.sum . && \
-        # clean up the go.mod,
-        # remove any local require for the handler/function, this can exist _if_ the
-        # developer has subpackages in their handler. It is ok to just remove it
-        # because we will replace it later
-        grep -v "handler/function" go.mod > gomod2; mv gomod2 go.mod && \
-        # now update the go.mod
-        # first, replace handler/function to point at the local code
-        # second, we need to rename the module to handler to match our main
-        go mod edit \
-            -replace=handler/function=./function \
-            -module handler && \
-        cat go.mod && \
-        if [ -d ./vendor ]; then \
-            # when vendored, we need to do similar edits to the vendor/modules.txt
-            # first we need to replace any possible copy of the handler code
-            rm -rf vendor/handler && \
-            # in modules.txt, we remove existing references to the handler/function
-            # these are replaced later with the new structure
-            grep -v "handler/function" ./vendor/modules.txt> modulestext; mv modulestext ./vendor/modules.txt && \
-            # add the mising replace to the vendor/modules.txt
-            echo "## explicit" >> ./vendor/modules.txt && \
-            echo "# handler/function => ./function" >> ./vendor/modules.txt && \
-            cat ./vendor/modules.txt; \
-        else \
-            echo "skip adding replace to ' ./vendor/modules.txt'"; \
-        fi \
-    else \
-        echo "skip go.mod handling"; \
-    fi
+# Lift the vendor and go.mod to the main package, cleanup any relative references
+RUN sh modules-cleanup.sh
 
 # Run a gofmt and exclude all vendored code.
 RUN test -z "$(gofmt -l $(find . -type f -name '*.go' -not -path "./vendor/*" -not -path "./function/vendor/*"))" || { echo "Run \"gofmt -s -w\" on your Golang code"; exit 1; }

template/golang-http/modules-cleanup.sh

@@ -0,0 +1,144 @@
+#!/usr/bin/env sh
+
+set -e
+
+GO111MODULE=$(go env GO111MODULE)
+
+# move_vendor will copy the function's vendor folder,
+# if it exists.
+move_vendor() {
+    if [ ! -d ./function/vendor ]; then
+        echo "vendor not found"
+        return
+    fi
+
+    echo "moving function vendor"
+    mv -f ./function/vendor .
+}
+
+
+# cleanup_gomod will move the function's go module
+cleanup_gomod() {
+
+    # Nothing to do when modules is explicitly off
+    # the z prefix protects against any SH wonkiness
+    # see https://stackoverflow.com/a/18264223
+    if [ "z$GO111MODULE" = "zoff" ]; then
+        echo "modules disabled, skipping go.mod cleanup"
+        return;
+    fi
+
+    echo "cleaning up go.mod"
+
+    # Copy the user's go.mod
+    mv -f ./function/go.mod .
+    mv -f ./function/go.sum .
+
+    # Clean up the go.mod
+
+    # Cleanup any sub-module replacements.
+    # This requires modifying any replace that points to "./*",
+    # the user has will use this to reference sub-modules instead
+    # of sub-packages, which we cleanup below.
+    echo "cleanup local replace statements"
+    # 1. Replace references to the local folder with `./function`
+    sed -i 's/=> \.\//=> \.\/function\//' go.mod
+
+
+    # Remove any references to the handler/function module.
+    # It is ok to just remove it because we will replace it later.
+    #
+    # Note that these references may or may not exist. We expect the
+    # go.mod to have a replace statement _if_  developer has subpackages
+    # in their handler. In this case they will need a this replace statement
+    #
+    #    replace handler/function => ./
+    #
+    # `go mod` will then add a line that looks like
+    #
+    #    handler/function v0.0.0-00010101000000-000000000000
+    #
+    # both of these lines need to be replaced, this grep selects everything
+    # _except_ those offending lines.
+    grep -v "\shandler/function" go.mod > gomod2; mv gomod2 go.mod
+
+    # Now update the go.mod
+    #
+    # 1. use replace so that imports of handler/function use the local code
+    # 2. we need to rename the module to handler because our main.go assumes
+    #    this is the package name
+    go mod edit \
+        -replace=handler/function=./function \
+        -module handler
+
+
+
+    if [ "$DEBUG" -eq 1 ]; then
+        cat go.mod
+        echo ""
+    fi
+}
+
+
+# cleanup_vendor_modulestxt will cleanup the modules.txt file in the vendor folder
+# this file is needed when modules are enabled and it must be in sync with the
+# go.mod.  To function correctly we need to modify the references to handler/function,
+# if they exist.
+cleanup_vendor_modulestxt() {
+    if [ ! -d ./vendor ]; then
+        echo "no vendor found, skipping modules.txt cleanup"
+        return
+    fi
+
+    # Nothing to do when modules is explicitly off
+    # the z prefix protects against any SH wonkiness
+    # see https://stackoverflow.com/a/18264223
+    if [ "z$GO111MODULE" = "zoff" ]; then
+        echo "modules disabled, skipping modules.txt cleanup"
+        return;
+    fi
+
+    echo "cleanup vendor/modules.txt"
+
+    # just in case
+    touch "./vendor/modules.txt"
+
+    # when vendored, we need to do similar edits to the vendor/modules.txt
+    # as we did to the go.mod
+
+    # 1. we need to replace any possible copy of the handler code
+    rm -rf vendor/handler && \
+
+    # 2. in modules.txt, we remove existing references to the handler/function
+    #    we reconstruct these in the last step
+    grep -v "\shandler/function" ./vendor/modules.txt> modulestext; mv modulestext ./vendor/modules.txt
+
+    # 3. Handle any other local replacements.
+    # any replace that points to `./**` needs to be udpat    echo "cleanup local replace statements"
+    sed -i 's/=> \.\//=> \.\/function\//' ./vendor/modules.txt
+
+    # 4. To make the modules.txt consistent with the new go.mod,
+    #    we add the mising replace to the vendor/modules.txt
+    echo "## explicit" >> ./vendor/modules.txt
+    echo "# handler/function => ./function" >> ./vendor/modules.txt
+
+    if [ "$DEBUG" -eq 1 ]; then
+        cat ./vendor/modules.txt;
+        echo ""
+    fi
+}
+
+# has_local_replacement checks if the file contains local go module replacement
+has_local_replacement() {
+    return "$(grep -E -c '=> \./\S+' "$1")"
+}
+
+
+################
+#   main
+################
+move_vendor
+
+cleanup_gomod
+
+cleanup_vendor_modulestxt

template/golang-middleware/Dockerfile

@@ -20,15 +20,10 @@ COPY . .
 ARG GO111MODULE="off"
 ARG GOPROXY=""
 ARG GOFLAGS=""
+ARG DEBUG=0
 
-# Add user overrides to the root go.mod, which is the only place "replace" can be used
-RUN cat function/GO_REPLACE.txt >> ./go.mod || exit 0
-RUN if [ -d ./function/vendor ]; then \
-    echo "moving handler vendor" && \
-    mv -f ./function/vendor .; \
-    else \
-    echo "using modules or vendor not found "; \
-    fi
+# Lift the vendor and go.mod to the main package, cleanup any relative references
+RUN sh modules-cleanup.sh
 
 # Run a gofmt and exclude all vendored code.
 RUN test -z "$(gofmt -l $(find . -type f -name '*.go' -not -path "./vendor/*" -not -path "./function/vendor/*"))" || { echo "Run \"gofmt -s -w\" on your Golang code"; exit 1; }