Skip to content

Assignment 1 Speed Bonus

This document outlines the requirements for the Assignment 1 Speed Bonus. The information here supersedes any details in the assignment PDF, as it is updated more frequently.

Warning

The information here may be updated often. Check back regularly to ensure you have the latest information. We will try to make announcements if the changes are major.

Rules

  1. Bonus Marks:

    • Each team can earn a maximum of 1 bonus mark per team for this speed bonus.
    • These marks are awarded based on how fast your program runs (wall-clock time) relative to others in CS3210.
    • This will be evaluated via a set of live leaderboards during the assignment period
    • You will place your executable to a folder within your NFS directory so that we can evaluate them, more details in the following sections.

  2. How Bonus Marks Are Awarded:

    • Your marks will be determined by your team's position in the Overall Leaderboard.
    • This position is determined via your team's average leaderboard position across all testcases.
      • Any leaderboards where you do not have a valid time will count as you having leaderboard position 300 as a default.
    • We will award 1 bonus mark for the top 10 teams on the Overall Leaderboard
    • We will award 0.5 bonus marks for the 11th to 15th teams on the Overall Leaderboard.
    • If there are any ties, both teams will be considered to be within the same leaderboard position.
    • Only one person from each team should submit to the leaderboard! Inform us immediately if you have accidentally submitted something, and we will help you remove it.

  3. Submission Requirements by Deadline:

    • At the end of the assignment, you must submit your bonus code as well as your main assignment code within the same GitHub Classroom repo.
    • You must include two new Makefile recipes called bonus.debug and bonus.perf, which generate the bonus executables for us to run. This is to allow students to have different main and bonus submissions.
      • As in the main submission, bonus.debug should add the -DDEBUG flag and bonus.perf should not.
    • The code you submit for the bonus marks must fulfill all the same requirements as the main assignment.
    • We will test your bonus submission again by compiling your source code after the deadline. If the timing deviates significantly from the leaderboard timing, or correctness checks fail, we will invalidate your bonus marks.

  4. IMPORTANT: Additional report requirements:

    • if you want to be considered for the bonus, add a (maximum one-page, not counted in the report page limit) section to your final report, explaining the techniques you used for your bonus, and how they contribute to performance.
    • If you do not do this, you will not be awarded the bonus marks!

Note

We reserve the right to disqualify submissions that do not follow the rules. Furthermore, we will enforce the spirit of the rules—we aim to reward genuine attempts at fast and correct code, not exploits of the system itself.

Configuration and Leaderboards

Configuration

  • Hardware: We will test your code on a single i7-13700 machine. We will run your executable with 8 threads, with the cpu-bind/map settings set to the default current behavior with srun.
  • Correctness: We will perform partial correctness checks and reject any submissions that are incorrect. Incorrect submissions may be rejected after the deadline if we find more test cases that your program fails.
  • Timing: We will take the average of the times over several runs and publish that time to the leaderboard.
  • Timeout: If your program takes too long, it will timeout, and not be published to the leaderboard.

Leaderboards

  • Per-testcase Leaderboards: Each leaderboard will be for a specific private test case. They should be titled <testcase_name>.in.
  • Overall Leaderboard: This will be the average of your team's position across all testcases, and this is the one that determines your bonus marks.

Submission Procedure

Warning

To be considered for the bonus, you must successfully submit to the Leaderboard by the end of the assignment period (i.e., your time must be recorded!)!

We will not consider your submission for the bonus if you do not successfully submit to the Leaderboard. That is, if our system has not finished evaluating your submission by the deadline, that time will not be counted.

We suggest submitting often to get up-to-date information on your speed on our private test cases.

Submission Steps

  1. Prepare Your Executables:

    • Build your executable (the .perf version!) - any name will do for the live leaderboard submission

  2. Copy the Executables:

    • Copy your executable (only the executable!) to any folder within your NFS directory e.g., a full path of /nfs/home/e0002863/a1_bonus/bonus.perf (assuming your NUSNET ID is e0002863 and your executable is called bonus.perf):

  3. Submit your folder name to our system:

    • Access our bonus system at http://soctf-pdc-017.comp.nus.edu.sg:8000/bonus/ (note: view our FAQ for updated instructions to access this) and submit the full path to your bonus executable. Use realpath <executable> to find the full path that contains the executable!

  4. Wait for Evaluation:

    • We will automatically evaluate leaderboard submissions approximately every 30 minutes, though we might shorten this time if submissions are very frequent and the cluster has capacity, or lengthen the time if the cluster is overloaded with non-bonus runs.

  5. Check the Leaderboards:

    • View the leaderboard in Grafana ("CS3210 Assignment 1 Bonus Leaderboard" dashboard)
    • You can submit as many times as you wish. We will take the fastest time so far (unless we wipe the leaderboard for some reason and require everyone to resubmit).

FAQ

My submission is missing from the Leaderboard

There are three possible reasons for this:

  1. Incorrect Executable Folder:

    • You might have submitted the wrong folder name to us. Type pwd in the folder to get the full folder name / use realpath on the executable to get the exact path.

  2. Did Not Pass Correctness Checking:

    • The executable did not pass our correctness requirements. This could be due to an incorrect implementation, the executable crashing, or other similar reasons.

  3. Exceeded Time Limit:

    • The executable took too long to run and was killed. Currently, the time limit is 60 seconds, and any executable exceeding this time limit will not be shown on the leaderboard.

If you are unable to determine why your submission is failing, please email us, and we will check the logs for you.