ci: fail build if binary files committed

Binary files can be whitelisted with the special line 'Adds-Binary:
<path/to/binary/file>' included in the commit message body. Deliberately
checked-in binary files should be very rare.

With minimal effort, this could probably be repurposed as a local
postcommit hook.

.circleci/config.yml

@@ -161,6 +161,7 @@ jobs:
     working_directory: /go/src/github.com/influxdata/influxdb
     steps:
       - checkout
+      - run: make checkcommit
 
       # Speed up `make build` by restoring caches from previous runs.
       - restore_cache:

Makefile

@@ -102,6 +102,9 @@ checktidy:
 checkgenerate:
 	./etc/checkgenerate.sh
 
+checkcommit:
+	./etc/circle-detect-committed-binaries.sh
+
 generate: subdirs
 
 test-js: node_modules

etc/circle-detect-committed-binaries.sh

@@ -0,0 +1,33 @@
+#!/bin/bash
+
+# This script wraps detect-committed-binaries.sh in a way that is usable from the Makefile.
+
+# Why not put these changes into detect-committed-binaries.sh?
+# Because we aren't using CircleCI for the enterprise code base,
+# and I don't want the OSS copy to diverge from the enterprise copy.
+
+# Why not just do the conditional logic in the Makefile?
+# Because I don't know Makefile syntax and practices well enough to do it in a reasonable amount of time.
+# If you know how to do it, please refactor the logic into the Makefile.
+
+if [ -n "$CIRCLE_PR_NUMBER" ] || [ -n "$CIRCLE_PULL_REQUEST" ]; then
+	# We want the PR number, but sometimes it isn't set (bug on CircleCI's side).
+	# https://discuss.circleci.com/t/circle-pr-number-missing-from-environment-variables/3745
+	CIRCLE_PR_NUMBER="${CIRCLE_PR_NUMBER:-${CIRCLE_PULL_REQUEST##*/}}"
+	# Looks like we're running on CircleCI.
+	# You might think we could use CIRCLE_COMPARE_URL, but that compares commits with the previous push,
+	# not with the base branch.
+	# This is roughly how you're supposed to determine the base branch/sha according to Circle:
+	# https://circleci.com/blog/enforce-build-standards/
+	PR_URL="https://api.github.com/repos/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME/pulls/$CIRCLE_PR_NUMBER?access_token=$GITHUB_READONLY_TOKEN"
+	COMMIT_RANGE="$(curl -s "$PR_URL" | jq -r '.head.sha + "..." + .base.sha')"
+	echo "Calculated commit range: $COMMIT_RANGE"
+else
+	# We're not running on circle.
+	# There's no reliable way to figure out the appropriate base commit,
+	# so just take a reasonable guess that we're comparing to master.
+	COMMIT_RANGE="HEAD...master"
+fi
+
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+"$DIR/detect-committed-binaries.sh" "$COMMIT_RANGE"

etc/detect-committed-binaries.sh

@@ -0,0 +1,53 @@
+#!/bin/bash
+
+if [ $# -ne 1 ]; then
+	>&2 echo "Usage: $0 REVISION_RANGE
+
+$0 iterates through each commit in REVISION_RANGE and exits 1 if all of these conditions are met:
+
+1. The commit introduces a file that git considers to be a binary file.
+2. The commit body does NOT contain a line of matching the pattern 'Adds-Binary: path/to/binary/file'.
+   The path must be relative to the repository root.
+
+REVISION_RANGE is typically given as mybranch...master. Note: 3 dots, not 2."
+	exit 1
+fi
+
+ORIG_IFS="$IFS"
+IFS=$'\n' # Set internal field separator to newlines only, so the "for binFile" loop can handle filenames containing spaces.
+BINARY_FILES_INTRODUCED=0
+for rev in $(git rev-list "$1"); do
+	# This loop is a bit complicated, so here is the explanation:
+	# First, use git log on the single revision.
+	# We can't use --numstat because that doesn't differentiate between binary files added and removed.
+	# Grep for lines indicating a Binary file went from zero to non-zero bytes.
+	# Then cut down to just the entire field before the pipe. This will break if we ever have a binary file whose name contains the pipe character.
+	# Finally, print just the first field without leading or trailing spaces. (https://unix.stackexchange.com/a/205854)
+	for binFile in $(git log -1 --format='' --stat=255 "$rev" | grep ' Bin 0 ->' | cut -d '|' -f 1 | awk '{$1=$1;print}'); do
+		# We have found a new binary file in $rev.
+		# Was it in the commit's whitelist?
+		if git log -1 --format=%b "$rev" | grep -q -F -x "Adds-Binary: $binFile"; then
+			# Yes it was. Skip this file.
+			echo "Revision $rev $(git log -1 --format='[%s]' "$rev") added whitelisted binary file: $binFile"
+			continue
+		fi
+
+		echo "Revision $rev $(git log -1 --format='[%s]' "$rev") introduced binary file: $binFile"
+		BINARY_FILES_INTRODUCED=1
+	done
+done
+IFS="$ORIG_IFS"
+
+if [ $BINARY_FILES_INTRODUCED -eq 1 ]; then
+	echo
+	echo '--------------------------------------------------'
+	echo "This changeset introduced unexpected binary files.
+
+If you meant to include them, amend the commit(s) that introduced the file(s),
+to include a line that matches exactly 'Adds-Binary: path/to/binary/file'.
+
+If you did not mean to include the file, please add an appropriate line to .gitignore
+so that other developers do not mistakenly commit that file, and please amend your commit
+to remove the file."
+	exit 1
+fi