Rework hack/ and build/ directories.

* Rewrite a bunch of the hack/ directory with modular reusable bash libraries.
* Have 'build/*' build on 'hack/*'.  The stuff in build now just runs hack/* in a docker container.
* Use a docker data container to enable faster incremental builds.
* Standardize output to _output/{local,dockerized}/bin/OS/ARCH/*.  This regularized placement makes cross compilation work.
* Move travis specific scripts under hack/travis

With new dockerized incremental builds, I can do a no-op `make quick-release` in ~30s.  This is a significant improvement.

Author: jbeda

.gitignore

@@ -37,4 +37,4 @@ network_closure.sh
 /third_party/pkg
 
 # also ignore etcd installed by hack/install-etcd.sh
-/third_party/etcd
+/third_party/etcd*

.travis.yml

@@ -6,17 +6,17 @@ go:
 
 install:
   - go get code.google.com/p/go.tools/cmd/cover
-  - ./hack/install-etcd.sh
+  - ./hack/travis/install-etcd.sh
   - ./hack/verify-gofmt.sh
   - ./hack/verify-boilerplate.sh
-  - ./hack/install-std-race.sh
+  - ./hack/travis/install-std-race.sh
   - ./hack/build-go.sh
   - go get ./contrib/podex
 
 script:
   - KUBE_TIMEOUT='-timeout 60s' ./hack/test-go.sh
-  - PATH=$HOME/gopath/bin:./third_party/etcd/bin:$PATH ./hack/test-cmd.sh
-  - PATH=$HOME/gopath/bin:./third_party/etcd/bin:$PATH ./hack/test-integration.sh
+  - PATH=$HOME/gopath/bin:./third_party/etcd:$PATH ./hack/test-cmd.sh
+  - PATH=$HOME/gopath/bin:./third_party/etcd:$PATH ./hack/test-integration.sh
   - ./hack/benchmark-go.sh
 
 notifications:

Makefile

@@ -9,7 +9,8 @@
 OUT_DIR = _output
 GODEPS_PKG_DIR = Godeps/_workspace/pkg
 
-export GOFLAGS
+KUBE_GOFLAGS = $(GOFLAGS)
+export KUBE_GOFLAGS
 
 # Build code.
 #
@@ -49,7 +50,7 @@ check test:
 #   make test_integration
 test_integration test_integ:
 	hack/test-integration.sh
-.PHONY: integration
+.PHONY: test_integration test_integ
 
 # Build and run end-to-end tests.
 #
@@ -94,10 +95,10 @@ release:
 .PHONY: release
 
 # Build a release, but skip tests
-# 
+#
 # Example:
 #   make release-skip-tests
-release-skip-tests:
+release-skip-tests quick-release:
 	KUBE_RELEASE_RUN_TESTS=n build/release.sh
-.PHONY: release-skip-tests
+.PHONY: release-skip-tests quick-release
 

build/README.md

@@ -18,12 +18,13 @@ There is also early support for building Docker "run" containers
 
 ## Key scripts
 
-* `make-server.sh`: This will compile all of the Kubernetes server binaries for linux/amd64
-* `make-client.sh`: This will make all cross-compiled client binaries
-* `run-tests.sh`: This will run the Kubernetes unit tests
-* `run-integration.sh`: This will build and run the integration test
-* `copy-output.sh`: This will copy the contents of `_output/build` from any remote Docker container to the local `_output/build`.  Right now this is only necessary on Mac OS X with `boot2docker`.
-* `make-clean.sh`: Clean out the contents of `_output/build` and remove any local built container images.
+* `run.sh`: Run a command in a build docker container.  Common invocations:
+  *  `run.sh hack/build-go.sh`: Build just linux binaries in the container.  Pass options and packages as necessary.
+  *  `run.sh hack/build-cross.sh`: Build all binaries for all platforms
+  *  `run.sh hack/test-go.sh`: Run all unit tests
+  *  `run.sh hack/test-integration.sh`: Run integration test
+* `copy-output.sh`: This will copy the contents of `_output/dockerized/bin` from any remote Docker container to the local `_output/dockerized/bin`.  Right now this is only necessary on Mac OS X with `boot2docker` when your git repo isn't under `/Users`.
+* `make-clean.sh`: Clean out the contents of `_output/dockerized` and remove any local built container images.
 * `shell.sh`: Drop into a `bash` shell in a build container with a snapshot of the current repo code.
 * `release.sh`: Build everything, test it, and (optionally) upload the results to a GCS bucket.
 
@@ -81,10 +82,10 @@ If the release script is set to upload to GCS, it'll do the following:
 
 These are in no particular order
 
-* [ ] Harmonize with scripts in `hack/`.  How much do we support building outside of Docker and these scripts?
+* [X] Harmonize with scripts in `hack/`.  How much do we support building outside of Docker and these scripts?
 * [ ] Get a cluster up and running with the Docker images.  Perhaps start with a local cluster and move up to a GCE cluster.
 * [ ] Implement (#186)[https://github.com/GoogleCloudPlatform/kubernetes/issues/186].  This will make it easier to develop Kubernetes.
-* [ ] Deprecate/replace most of the stuff in the hack/
+* [X] Deprecate/replace most of the stuff in the hack/
 * [ ] Create an install script that'll let us do a `curl https://[URL] | bash` to get that tarball down and ensure that other dependencies (cloud SDK?) are installed and configured correctly.
 * [ ] Support/test Windows as a client.
 * [ ] Finish support for the Dockerized runtime. Issue (#19)[https://github.com/GoogleCloudPlatform/kubernetes/issues/19].  A key issue here is to make this fast/light enough that we can use it for development workflows.

build/build-image/Dockerfile

@@ -25,6 +25,10 @@ ENV GOARCH  amd64
 # Get the code coverage tool and godep
 RUN go get code.google.com/p/go.tools/cmd/cover github.com/tools/godep
 
+# We use rsync to copy some binaries around.  It is faster (0.3s vs. 1.1s) on my
+# machine vs. `install`
+RUN apt-get update && apt-get install -y rsync
+
 # Download and symlink etcd.  We need this for our integration tests.
 RUN mkdir -p /usr/local/src/etcd &&\
     cd /usr/local/src/etcd &&\
@@ -39,6 +43,10 @@ WORKDIR /go/src/github.com/GoogleCloudPlatform/kubernetes
 
 # Propagate the git tree version into the build image
 ADD kube-version-defs /kube-version-defs
+ENV KUBE_GIT_VERSION_FILE /kube-version-defs
+
+# Make output from the dockerized build go someplace else
+ENV KUBE_OUTPUT_SUBPATH _output/dockerized
 
 # Upload Kubernetes source
 ADD kube-source.tar.gz /go/src/github.com/GoogleCloudPlatform/kubernetes