Assignment 1 - Testing

1. Goal

The goal of this assignment is to experiment with coverage measurement and improvement. You will select an open-source GitHub project, measure its test coverage, and enhance it by creating new test cases or improving existing ones.

2. Deliverables and Presentation

The group will fork the source code repository of the GitHub project of its choice and perform modifications there. The URL of the forked repository must be submitted on Canvas. The README.md file of the forked project should follow the report template for this assignment, available here.

Deadline: 27/06/2024.

In summary:

1. Forked Repository URL: Submit on Canvas.

2. Final Report (README.md): Follow the provided template.

3. Presentation: Demonstrate your work and answer questions.

Presentation Requirements

The group must present the work. This means that one of the group members should share their screen during the presentation, and the person evaluating the group will ask questions about the part done by each group member. The group should be ready to execute parts of the code as requested.

Prior to the presentation day, the group must sign in for a presentation slot here.

Presentation day: 28/06/2024.

3. Tasks

3.1 Project Choice

Each group can choose a project written in any programming language. The project must:

• Be hosted on GitHub;

• Have an open-source initiative-approved license (see Open Source Licenses);

• Use automated unit testing on the part written in the programming language chosen by the group;

• Have existing branch coverage (by the existing tests) less than 100%;

• Have at least two contributors outside the course group;

• Have at least 10,000 lines of code (10 KLOC) in the programming language the group is using.

After the group finds a project, register it here. Each group should choose a unique project, i.e., a project that is not registered in the list.

3.2 Coverage Measurement

Measure the coverage of the tests with an existing tool (see TOOLS for suggestions). These will give you an output that you can use to then analysis your selected project's code base.

From there identify regions of code that are in need of testing. This can be in the form of no tests are written for those areas of code or not all conditional branches are covered in the currently existing tests.

3.3 Function Selection and Instrumentation

With these untested regions found, your team will then pick two functions (methods) per person to further analyze. Don't make these too difficult for yourself and pick functions that are not too long or overly complex; however, these functions should have at least two different conditional paths through the function. Ideally, this should be done in a separate development git branch per person, but it is fine if the group decides to not do so.

Your own function-based coverage tool should be implemented by adding code to your two selected functions. An example instrumentation is as follows (see also a concrete example here):

• Identify all conditional branches in the chosen functions and assign a group wide unique number (ID) to each one (might be ideal to set a wide range per person to use as id);

• Before the program starts, create data structures that hold coverage information about the conditional branches;

• Add a line before the first statement of each conditional branch outcome that sets a flag if the conditional branch is reached;

• At the end of the program, write all information about the conditional branches taken to a file or console.

3.4 Coverage Improvement

Improve branch coverage by creating new test cases or enhancing existing ones. Each group member must create/enhance one test case to cover each function they have instrumented in the previous task. The suggested steps are:

• Consider the existing tests and check how they cover the conditional branches;

• Identify the requirements that are tested or untested by the existing tests;

• Use the requirements for creating inputs and assertions;

• Follow naming conventions for unit test names based on the repository or create one as a group (for example {Function Name}_{Condition or Parameters}_{Expected Result}).

Look at getting to at least 80% branch coverage of each function you have identified and added code to.

3.5 Merge Results

In the end, all new/enhanced tests should be merged into the main git branch for your fork (not to the main development git branch). Then, measure the coverage using an existing tool accounting for all the modifications in the test suite by the group.

3.6 Report (README.md)

The group's work should be reported as the README.md file in the root of the forked repository (the original README.md file should be renamed to README_original.md). The README.md file should follow the report template for this assignment (see Section 2).

3.7 Summary: Step-by-Step Instructions

Step 1: Choose a project

Choose a GitHub project that:

- Is hosted on GitHub.

- Has an open-source license.

- Uses automated unit testing.

- Has existing branch coverage less than 100%.

- Has at least 10,000 lines of code (10 KLOC).

- Has at least two contributors outside the course group.

Checklist:

- [ ] Meets all criteria.

- [ ] Register project here

Step 2: Coverage Measurement

1. Use a coverage measurement tool (see tools).

2. Identify untested regions of the code.

Step 3: Function Selection and Instrumentation

1. Each group member selects two functions with conditional paths.

2. Instrument these functions to measure coverage.

Checklist:

- [ ] Choose 2 functions per person.

- [ ] Add code to track coverage.

Step 4: Coverage Improvement

1. Enhance or create test cases to cover the selected functions.

2. Ensure at least 80% branch coverage.

Checklist:

- [ ] Develop/enhance test cases.

- [ ] Achieve desired coverage.

Step 5: Merging Results

1. Open pull requests for each member’s changes (if different git branches were used).

2. Review and merge into the main branch.

Checklist:

- [ ] Submit pull requests (if different git branches were used).

- [ ] Merge after review by another student.

Step 6: Final Report and Presentation

1. Compile your findings and work into the README.md.

2. Prepare for the presentation.

3. Register a slot for your presentation.

4. Evaluation and Grading

All group members will have the same grade unless it is noted that some members contributed less. Grades for such students will be lower than the others in the group.

4.1 Grading Rubrics

Criteria	Ratings	Points
1) Execution of existing coverage measurement tool.	2 Pts: There is evidence (screenshot of the results) that a tool was executed on the project. 0 Pts: Otherwise.	2 pts
2) Instrumentation of functions for coverage measurement.	2 Pts: There is evidence (code diff + screenshot of the results) that each student measured the coverage of 2 functions. 1 Pt: There is evidence (code diff + screenshot of the results) that the group measured coverage, but not for 2 functions per member. 0 Pts: Otherwise.	2 pts
3) Functionality of the instrumentation for coverage measurement.	2 Pts: There is evidence (execution during the presentation) that the instrumentation of each group member works. 1 Pt: There is evidence (execution during the presentation) that the instrumentation of some group members works. 0 Pts: Otherwise.	2 pts
4) Creation/enhancement of new tests.	2 Pts: There is evidence (code) that each student created/enhanced tests to reach 80% conditional branch coverage. 1 Pt: There is evidence (code) that the group created/enhanced tests 2 tests per member. 0 Pts: Otherwise.	2 pts
5) Functionality of the new/enhanced tests.	2 Pts: There is evidence (execution during the presentation) that the tests created/enhanced by each member work. 1 Pt: There is evidence (execution during the presentation) that some tests work. 0 Pts: Otherwise.	2 pts
6) Effectiveness of each new/enhanced test in improving coverage.	2 Pts: There is evidence (screenshot of the old and new results) that the tests created/enhanced by each member improve coverage. 1 Pt: There is evidence (screenshot of the old and new results) that some tests improve coverage, but not all. 0 Pts: Otherwise.	2 pts
7) Overall coverage improvement.	2 Pts: There is evidence (screenshot of the results) of the overall coverage improvement as measured with an existing tool. 0 Pts: Otherwise.	2 pts
8) Organization and clarity of the report (README.md).	2 Pts: Good. 1 Pt: Sufficient. 0 Pts: Insufficient.	2 pts

Total: 16 pts

4.2 Eligibility

To have a grade > 0, the following criteria must be met:

• The group is composed of more than one student;

• The chosen project meets the requirements (Section 3.1) and is not used by another group;

• The forked repository URL is submitted on Canvas by the deadline;

• There are no modifications in the forked repository after the deadline;

• The report (README.md) contains a statement of individual contributions;

• All group members are present during the presentation (in case a member cannot attend, a compelling reason should be provided).

5. Suggestions

5.1 GitHub Projects

Some suggestions on how to find a project to work on are:

• Choose a project you are using yourself (e.g., a program or library);

• Navigate the list of beginner-friendly projects here;

• Consider Apache Projects;

• Consider Eclipse Projects.

Note that the approach the group uses to find a project does not matter. What matters is that all requirements described in Section 3.1 are fulfilled.

5.2 Tool for Counting Lines of Code

We recommend that you use lizard for counting the lines of code since it supports many programming languages.

5.3 Tools for Measuring Test Coverage

You can choose any coverage measurement tool for your project.

• For Java, we recommend Cobertura, OpenClover, and JaCoCo.

• For Python, we recommend Coverage.py.

6. Ethics

This is teamwork. Therefore, be responsible and responsive to your teammates. Do not leave the tasks to be done at the last minute as this might cause stress for your teammates or impact the whole group's grade.

Moreover, information exchange and collaboration are fully allowed within your own groups. Cases of plagiarism or inter-group collaboration and assignment contents exchange will be reported to and managed by the official fraud committee. In case of fraud, the consequences of those acts may potentially lead to formal warning, inclusion of the formal warning in your VU student file, suspension from taking exams for a given period, expulsion from the VU.