Writing Submission Tests #

This guideline contains some information that can help creators to create correct codes for their challenges. The set of rules was made to ensure that each challenge or contest is done in such a way that it is fair and qualitative. Therefore, the experience of each user should be good and of high quality, with no technical problems.

General Guidelines #

• Read ahead! As mentioned in the General Coding Guidelines, solutions are very important and maintain the quality of the challenge.

• Verify the outcome! The tests should verify the outcome of solutions, not their approach or the implementation of details. When a solution is correct and satisfies all requirements, it must be accepted regardless of the way it was written or the implementation details.

• Framework matters! Some languages ​​use a specific type of framework. Be sure that you understand and know what types of framework the platform uses when you write the challenge. You can find more information in the Language part of the Guide.

• Be organized! Organize and group the tests into different groups or subgroups. You can name these groups with distinctive names to help the user when there are several tests.

• Know your language! It is very important to master the information before providing it further, so it is preferable to know what can be tested and what cannot.

• Clear and to the point! Compose clear and short error messages and assertion messages. Thus, users can understand what they did wrong or good and not get lost in the details.

• Keep it low! Keep the number of entries in test reports reasonably small.

• Stay classy! Tests should handle unexpected answers gracefully.

• Fast & Furious! Tests should not take long to run, preferably not more than 3 seconds.

• Independent! Tests should be independent.

• Take into consideration! Tests should not print excessively to standard output

Fixed Tests #

Fixed tests are tests with predetermined inputs and outputs, and do not change between test runs.

• Each request must come with a fixed test that is consistent with the described request. Each element of the challenge requirements must be explicitly tested with at least one dedicated and properly labeled assertion.

• Fixed tests are usually easier to debug, so for an ideal scenario, a fixed test should fail before the challenge is tested with random tests.

• The purpose of the tests is to verify the solution both with edge cases and with cases that need different handling in the context of the task. To understand better, if a challenge has been mentioned in the description that such inputs do not need to be taken into consideration:

  • empty arrays must be tested for problems involving arrays with lengths 0.
  • problems seeking pairs of values ​​in arrays and empty strings must be tested for problems involving arrays with length 1.

Random Tests #

Even if random tests are not something so common, it’s good to use them. Their purpose is to reject invalid approaches based on input probing, hard coding and other workarounds. The most important thing for random tests is to make the expected return values ​​and their order unpredictable so that only the solution that passes the test may pass.

• Random tests generate test cases for all scenarios that can’t be completely tested with fixed tests. In some cases, you may be needed to build different kinds of random input generators. If a specific kind of input has a very low chance of occurring purely at random it's better to build a specific random generator that can enforce this kind of input rather than rely on 1000 random tests and just pray for the specific case to come up. You can take into consideration the idea of keeping one fully random generator, because it may generate new and random cases.

• Random tests should be able to manage to see that it’s infeasible to pass tests by counting test cases. Cases shouldn’t be grouped by output type or behavior especially if the expected output is a boolean variable or when it comes to error checking.

• Keep the number of random tests reasonable if you don't use them to verify the performance of the user's solutions.

• You will run the random test after the fixed ones. Not all testing frameworks permit for easy ordering of tests, but fixed tests should run, and eventually fail, before random tests.

• Use appropriate random utilities available in your language. Know how to use random number generators, and how to randomly generate types of inputs you need, be it numbers in a range, large numbers, strings, big integers, floating-point values, collections, etc. Know how to do random sampling of collections, how to shuffle them, and how to avoid duplicates.

• Between runs, performance tests should be consistent.

• Try to make debugging random tests easy if you need to rely on them to extensively check the correctness of the user's solutions. There will be situations where you're not gonna be able to build fixed tests in a way that allows users to be able to catch most of the holes. Sometimes when random tests fail, it can be very difficult to debug because the input is very large and cannot be easily read. You can spit the random test in two batches:

• One with small debuggable inputs.
• The other one with proper, large input values. Note that both parts should still contain all application scenarios.