No ability to retry failed attempts to update Consul #38
Description
Currently, redishappy can fail to update Consul because it's not running:
INFO: 2015/08/18 00:42:46 consul_flipperclient.go:38: InitialiseRunningState called : [{"externalPort":6379,"name":"redis-default","ip":"1.2.3.4","port":6379}]
ERROR: 2015/08/18 00:42:46 consul_flipperclient.go:82: Error updating consul : Put http://127.0.0.1:8500/v1/catalog/register: dial tcp 127.0.0.1:8500: connection refused
OK, I thought, I'll just add some ordering in the upstart scripts so that we start redishappy just after Consul has started. However, Consul takes a few seconds to get going, and in the meantime:
INFO: 2015/08/18 00:45:03 consul_flipperclient.go:38: InitialiseRunningState called : [{"externalPort":6379,"name":"redis-default","ip":"10.10.10.5","port":6379}]
ERROR: 2015/08/18 00:45:03 consul_flipperclient.go:82: Error updating consul : Unexpected response code: 500 (No cluster leader)
In both cases, unless the Redis topography actually changes after this point, redishappy won't make any further attempt to update Consul. I guess I'm looking for some sort of retry mechanism. Or, even better, if redishappy were to "ensure" the value in Consul by reattempting updates whenever the service-state-in-Consul was different from expected, or the state couldn't be checked.