Testing is one of the most important parts of the development process. However, it is difficult to reproduce all real-world scenarios by simulating user behaviour in a sandbox environment. Even when software is thoroughly tested in the traditional way, there may be specific problems that appear only when real users use the software in the production environment. To improve our confidence in the system beyond what testing can provide, at Plumbee we have implemented a mechanism that allows us to push software changes to a small fraction of the user base in a controlled way – commonly known as Canary Testing or Canary Deployment. The name “canary” is used with reference to the canaries that were often sent to coal mines to alert miners about toxic gases reaching dangerous levels.
Our requirements for canary deployment are: