Race condition in API server tests

[bodom0015]

First run of the API server tests will create an account/namespace called test.
Second run of the tests will attempt to delete this account before recreating it to reset the test state.

With PVCs and running pods in the namespace, this can create a race condition - since we don't wait for the namespace to be fully deleted before continuing the tests, the account creation can fail because the namespace is still being cleaned up:

24/Sep/2018:10:26:07 -0500 200 8874μs "GET /api/accounts HTTP/1.1" admin "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"
E0924 10:26:08.162690   16237 server.go:1070] Operation cannot be fulfilled on namespaces "test": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.
24/Sep/2018:10:26:08 -0500 500 17416μs "DELETE /api/accounts/test HTTP/1.1" admin "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"
24/Sep/2018:10:26:09 -0500 409 7774μs "POST /api/register HTTP/1.1" - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"
24/Sep/2018:10:26:10 -0500 404 1478μs "PUT /api/register/verify HTTP/1.1" - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"
24/Sep/2018:10:26:11 -0500 200 4244μs "GET /api/accounts HTTP/1.1" admin "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"
24/Sep/2018:10:26:12 -0500 404 1574μs "GET /api/register/approve?u=test&t=vKq8XWqIB5iOn6RXO8ucgcKS8yM HTTP/1.1" - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"
24/Sep/2018:10:26:13 -0500 200 4472μs "GET /api/accounts HTTP/1.1" admin "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"
runtime error: index out of range
goroutine 247 [running]:
runtime/debug.Stack(0xc4204454b8, 0x1e42b80, 0x2803b50)
	/usr/local/Cellar/go/1.8/libexec/src/runtime/debug/stack.go:24 +0x79
github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*RecoverMiddleware).MiddlewareFunc.func1.1(0x2803750, 0x27c1540, 0xc4203a2690)
	/Users/lambert8/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/recover.go:41 +0x6e
panic(0x1e42b80, 0x2803b50)
	/usr/local/Cellar/go/1.8/libexec/src/runtime/panic.go:489 +0x2cf
main.(*Server).GetAccount(0xc420098fd0, 0x27c1500, 0xc42016f180, 0xc420148e80)
	/Users/lambert8/go/src/github.com/ndslabs/apiserver/cmd/server/server.go:628 +0xc79
main.(*Server).GetAccount-fm(0x27c1500, 0xc42016f180, 0xc420148e80)
	/Users/lambert8/go/src/github.com/ndslabs/apiserver/cmd/server/server.go:331 +0x48
github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*router).AppFunc.func1(0x27c1500, 0xc42016f180, 0xc420148e80)
	/Users/lambert8/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/router.go:56 +0xa3
github.com/ndslabs/apiserver/vendor/github.com/StephanDollberg/go-json-rest-middleware-jwt.(*JWTMiddleware).middlewareImpl(0xc420018480, 0x27c1500, 0xc42016f180, 0xc420148e80, 0xc42020ca90)
	/Users/lambert8/go/src/github.com/ndslabs/apiserver/vendor/github.com/StephanDollberg/go-json-rest-middleware-jwt/auth_jwt.go:103 +0x226
github.com/ndslabs/apiserver/vendor/github.com/StephanDollberg/go-json-rest-middleware-jwt.(*JWTMiddleware).MiddlewareFunc.func2(0x27c1500, 0xc42016f180, 0xc420148e80)
	/Users/lambert8/go/src/github.com/ndslabs/apiserver/vendor/github.com/StephanDollberg/go-json-rest-middleware-jwt/auth_jwt.go:82 +0x51
github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*IfMiddleware).MiddlewareFunc.func1(0x27c1500, 0xc42016f180, 0xc420148e80)
	/Users/lambert8/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/if.go:47 +0x78
github.com/ndslabs/apiserver/pkg/middleware.(*NoCacheMiddleware).MiddlewareFunc.func1(0x27c1500, 0xc42016f180, 0xc420148e80)
	/Users/lambert8/go/src/github.com/ndslabs/apiserver/pkg/middleware/no_cache.go:17 +0x135
github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*ContentTypeCheckerMiddleware).MiddlewareFunc.func1(0x27c1500, 0xc42016f180, 0xc420148e80)
	/Users/lambert8/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/content_type_checker.go:38 +0x200
github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*JsonIndentMiddleware).MiddlewareFunc.func1(0x27c1540, 0xc4203a2690, 0xc420148e80)
	/Users/lambert8/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/json_indent.go:35 +0xd8
github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*RecoverMiddleware).MiddlewareFunc.func1(0x27c1540, 0xc4203a2690, 0xc420148e80)
	/Users/lambert8/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/recover.go:57 +0x8a
github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*PoweredByMiddleware).MiddlewareFunc.func1(0x27c1540, 0xc4203a2690, 0xc420148e80)
	/Users/lambert8/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/powered_by.go:26 +0xae
github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*RecorderMiddleware).MiddlewareFunc.func1(0x27c1580, 0xc420148ea0, 0xc420148e80)
	/Users/lambert8/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/recorder.go:22 +0xa2
github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*TimerMiddleware).MiddlewareFunc.func1(0x27c1580, 0xc420148ea0, 0xc420148e80)
	/Users/lambert8/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/timer.go:20 +0x122
github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.(*AccessLogApacheMiddleware).MiddlewareFunc.func1(0x27c1580, 0xc420148ea0, 0xc420148e80)
	/Users/lambert8/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/access_log_apache.go:87 +0x54
github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest.adapterFunc.func1(0x27c03c0, 0xc420456700, 0xc4204a4000)
	/Users/lambert8/go/src/github.com/ndslabs/apiserver/vendor/github.com/ant0ine/go-json-rest/rest/middleware.go:70 +0x122
net/http.HandlerFunc.ServeHTTP(0xc42020cde0, 0x27c03c0, 0xc420456700, 0xc4204a4000)
	/usr/local/Cellar/go/1.8/libexec/src/net/http/server.go:1942 +0x44
net/http.serverHandler.ServeHTTP(0xc420098e70, 0x27c03c0, 0xc420456700, 0xc4204a4000)
	/usr/local/Cellar/go/1.8/libexec/src/net/http/server.go:2568 +0x92
net/http.(*conn).serve(0xc4201368c0, 0x27c1280, 0xc420277980)
	/usr/local/Cellar/go/1.8/libexec/src/net/http/server.go:1825 +0x612
created by net/http.(*Server).Serve
	/usr/local/Cellar/go/1.8/libexec/src/net/http/server.go:2668 +0x2ce
24/Sep/2018:10:26:14 -0500 500 19321μs "GET /api/accounts/test HTTP/1.1" admin "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"

If it is simple enough, we should look into modifying the "Delete Account" test case to wait until the namespace has been fully deleted before moving onto the next test.

Another pattern might be to see if newman has a Ruby-esque "afterAll" hook (such as the one offered by protractor and karma) that will let us specify cleanup code that should run once all test cases have completed.