Avoiding Performance Testing in Production by mistake

Ever been in the situation that the tests you were running or supposed to be running in the Lab environment were actually running in production?

Why this could have happened?

• The tester was careless
• We are all prone to make mistakes.

All these happened because we were using a parameter file which were pointing to the production environment. In our case, we use the same scenario and modify the parameter file entries. One set of entries/URLs are of the Lab environment and the other set are of the Production environment. The file is supposed to contain the set for the right environment. 

I have a suggestion on how this can be prevented.

• Have a check in your code to see if the URLs you are using from the parameter file are pointing to production
• If so see if the current system date & time generated from the scripts matches an entry from another parameter file. I go for a time based counter with a two hour window. 
• If the time and date falls with in the 2 hour window in file, the scenario can continue execution. Else provide error messages and exit the vUsers. 
• So each time we need to run a test in Prod, we need to make sure that the parameter file has the right time and date for the execution to continue.

Also I would suggest setting up some alerting mechanism so that your monitoring team is aware of some tests  happening in production. We all want to inform them before starting our testing but do forget from time to time. I use Splunk regularly and have set up an alert in case the events per minute for the user-agent I am using is greater than a predetermined value. 

"I don't always test my code, but when I do, I prefer to do it in Production." :)