Skip to content

Assignment 1 Speed Bonus

This document outlines the requirements for the Assignment 1 Speed Bonus. The information here supersedes any details in the assignment PDF, as it is updated more frequently.

Warning

The information here may be updated often. Check back regularly to ensure you have the latest information. We will try to make announcements if the changes are major.

Rules

  1. Bonus Marks:

    • Each team can earn a maximum of 2 bonus marks per team for this speed bonus.
    • These marks are awarded based on how fast your program runs (wall-clock time) relative to others in CS3210.
    • This will be evaluated via a set of live leaderboards during the assignment period
    • You will submit your executables to a folder so that we can evaluate them, more details in the following sections.

  2. How Bonus Marks Are Awarded:

    • Your marks will be determined by your team's position in the Overall Leaderboard.
    • This position is determined via your team's average leaderboard position across all testcases.
      • Any leaderboards where you do not have a valid time will count as you having leaderboard position 300 as a default.
    • We will award 2 bonus marks for the top 10 teams on the Overall Leaderboard
    • We will award 1 bonus mark for the 11th to 15th teams on the Overall Leaderboard.
    • If there are any ties, both teams will be considered to be within the same leaderboard position.
    • Only one person from each team should submit to the leaderboard! Inform us immediately if you have accidentally submitted something, and we will help you remove it.

  3. Submission Requirements by Deadline:

    • At the end of the assignment, you must submit your bonus code as well as your main assignment code within the same GitHub Classroom repo.
    • You must include a Makefile recipe called bonus, that produces two executables: called bonus and bonus.perf.
      • That is, make bonus should create your two bonus executables bonus and bonus.perf.
      • As in the main submission, bonus should run the validator and bonus.perf should not.
    • The code you submit for the bonus marks must fulfill all the same requirements as the main assignment.
    • We will test your bonus submission again by compiling your source code after the deadline. If the timing deviates significantly from the leaderboard timing, or correctness checks fail, or the .perf and non .perf versions are different in any significant way besides validator calls, we will invalidate your bonus marks.

  4. IMPORTANT: Additional report requirements:

    • if you want to be considered for the bonus, add a (maximum one-page, not counted in the report page limit) section to your final report, explaining the techniques you used for your bonus, and how they contribute to performance.
    • If you do not do this, you will not be awarded the bonus marks!

Note

We reserve the right to disqualify submissions that do not follow the rules. Furthermore, we will enforce the spirit of the rules—we aim to reward genuine attempts at fast and correct code, not exploits of the system itself.

Configuration and Leaderboards

Configuration

  • Hardware: We will test your code on a single i7-13700 machine. We will run your executable with 8 threads, with the cpu-bind/map settings set to the default current behavior with srun.
  • Correctness: We will perform partial correctness checks and reject any submissions that are incorrect. Incorrect submissions may be rejected after the deadline if we find more test cases that your program fails.
  • Timing: We will take the average of the times over several runs and publish that time to the leaderboard.
  • Timeout: If your program takes too long, it will timeout, and not be published to the leaderboard.

Leaderboards

  • Per-testcase Leaderboards: Each leaderboard will be for a specific private test case. They should be titled <testcase_name>.in.
  • Overall Leaderboard: This will be the average of your team's position across all testcases, and this is the one that determines your bonus marks.

Submission Procedure

Warning

To be considered for the bonus, you must successfully submit to the Leaderboard by the end of the assignment period (i.e., your time must be recorded!)!

We will not consider your submission for the bonus if you do not successfully submit to the Leaderboard. That is, if our system has not finished evaluating your submission by the deadline, that time will not be counted.

We suggest submitting often to get up-to-date information on your speed on our private test cases.

Submission Steps

  1. Prepare Your Executables:

    • You must submit two executables each time for a bonus evaluation: <your_nusnet_id_lowercase> and <your_nusnet_id_lowercase>.perf. As before, the executable without .perf should call the validator, and the one with .perf should not call the validator.

    • For example, if your NUSNET ID is e0002863, your two executables should be e0002863 and e0002863.perf

      • Only one person from the group needs to submit these executables each time, and it should be the same person from the group each time to avoid multiple leaderboard entries per team (only one will count).

      • The person who compiled the executable should be the one who submits. If in doubt, run:

        stat -c "%U" e1234567

        Replace e1234567 with your executable name. The output should match the executable name.

  2. Copy the Executables:

    • Copy your executable (only the executables!) to the folder /nfs/bonus/:

      cp e0002863 /nfs/bonus/
cp e0002863.perf /nfs/bonus/

  3. Wait for Evaluation:

    • We will automatically evaluate leaderboard submissions approximately every 30 minutes, though we might shorten this time if submissions are very frequent and the cluster has capacity, or lengthen the time if the cluster is overloaded with non-bonus runs.
    • Your executables will be removed from the folder after evaluation.

  4. Check the Leaderboards:

    • View the leaderboard (updated approx every 5 minutes) at /nfs/leaderboard/board.txt.
    • You can submit as many times as you wish. We will take the fastest time so far (unless we wipe the leaderboard for some reason and require everyone to resubmit).

FAQ

My submission is missing from the Leaderboard

There are three possible reasons for this:

  1. Incorrect Executable Names:

    • The executables were named incorrectly. Please refer to the submission instructions above.

  2. Did Not Pass Correctness Checking:

    • The non-.perf executable did not pass our correctness requirements. This could be due to an incorrect implementation, the executable crashing, or other similar reasons.

  3. Exceeded Time Limit:

    • The executable took too long to run and was killed. Currently, the time limit is 60 seconds, and any executable exceeding this time limit will not be shown on the leaderboard.

If you are unable to determine why your submission is failing, please email us, and we will check the logs for you.