In Outcomes Assessment, evidence is the collected sample of student assignments. This evidence is assessed for the evaluation of programs, courses, research projects, faculty and student achievements, and more.
Assessment managers and assessment administrators create evidence collections, sample the collections, and assign evaluators.
Create Evidence Collections
You create evidence collections to gather evidence from courses that is relevant for a specific project or assessment activity. For example, create an evidence collection to gather all assignment submissions from courses within a particular program or business unit that are aligned to a specific goal.
You can set up an evidence collection by defining the criteria used to collect artifacts from Blackboard Learn courses. You define an evidence collection by:
- Access Goals and Assessments on the Outcomes tab or the link on the My Institution tab. In addition, administrators can access it on the administrator panel under Tools and Utilities.
- Click View Collected Evidence to see a list of existing collections.
- Click +Collect Evidence to create a new collection.
- On the Collect Evidence page, click Find Goals.
- On the Discover Goals pop-up window, select the goals the evidence must be aligned to for inclusion in the collection. Click Submit.
- In the Define Collection Options section, select where evidence will be collected from: the entire system, a limited scope, or from specific places. Within each of those options, you can further narrow the criteria by restricting dates.
- Give the collection a name. You can specify start and end dates which can be used to help you manage the timelines of various collection projects and understand when projects have been completed.
- Click Submit if you are finished or Collect More Evidence if you need to create another evidence collection.
In order to collect evidence, content within courses must be aligned to the relevant goals. See Goals for details about creating and aligning content to goals.
Sample a Collection to Create an Evidence Set
An evidence set is the subset of artifacts, which are student assignment submissions that meet the collection criteria, that were randomly drawn from the evidence collection. You create an evidence set by sampling the evidence collection to reduce the number of artifacts that will be evaluated for institutional assessment activities. After sampling, evaluation can occur.
- On the View Evidence Collections page, select a collection name.
- On the collection's page, you can access the:
- Status of the collection at the top of the page
- Buttons for your next available action
- Collect tab to display the items that match the evidence collection criteria
- Sample tab to display the randomly selected subset of artifacts from the collection. This tab appears after sampling has occurred.
- Click Sample Artifacts. If the collection was already sampled, you can click Sample Again.
- In the pop-up window, choose to Include All Artifacts or Randomly Select Artifacts. If you choose to randomly select, type the total number of artifacts needed and, optionally, the minimum number of artifacts needed from each assignment.
- Click Submit. The sampling process runs immediately and only takes a few seconds to complete.
Evaluate Artifacts in an Evidence Set
You select one or more evaluators to review artifacts in an evidence set. They use a rubric to evaluate the artifacts that have been assigned to them. Any user in the system can be added to an evaluation session. They do not require any special permission or access. Evaluators are automatically sent invitations through email and the Blackboard notifications system to review an assigned selection of artifacts within an evidence set.
Create an Evaluation Session
- On the View Collected Evidence page, select a collection name. The collection must be sampled before you can create an evaluation session.
- Click Create Evaluation Session.
- In the pop-up window, under Select Evaluators, browse for one or more evaluators.
- For Evaluation Setting, select the number of evaluators that need to evaluate each artifact.
- For Grading Rubrics, browse for a rubric for evaluators to use.
- Select a Due Date.
- Check the Privacy box to enable anonymous evaluation.
- Click Start Session. Users are automatically notified by email that you have selected them as evaluators for this session. A notification is also posted in the Updates section of My Blackboard. Both the email and the notification contain a link to the Evaluator View.
About Evaluation Sessions
Evaluators see only the artifacts assigned to them. This enables the assessment manager to more effectively compare the scores of different evaluators of the same artifact and better understand inter-rater reliability.
The assessment manager can return to the evaluation session page to monitor the progress of various evaluators. They can opt to check the artifact evaluations themselves, drop or add evaluators, and close the session. If evaluators are added or removed, the system automatically redistributes the remaining non-evaluated artifacts across the updated list of evaluators.
How Users Access the Evaluator View
After you create an evaluation session, evaluators are automatically sent invitations through email and are notified on their Updates page in My Blackboard.
- Evaluators click the link to their assigned evaluations in the email message or on the Updates page. If accessing from the email message, evaluators must log in to Blackboard Learn before clicking the link.
- Only the artifacts within their assigned evidence sets appear. Evaluators access the contextual menu for an artifact and select Evaluate.
- The student's submission appears. The evaluator clicks either Evaluate or Evaluate in New Window to view the rubric and select point values.
- The evaluator clicks Submit.
Analyze the Results of the Evaluation
After evaluation of an evidence set is complete, assessment managers can run a report to understand the results. If they do not choose to perform a separate outcomes evaluation on collected evidence, they can instead report on the rubric scores collected from assignments within the course.
If the collected assignment submissions were graded in the course with a rubric, those rubric evaluations are also collected. The course rubrics display in the list of rubrics available when analyzing results.
- On the View Collected Evidence page, select a collection name. The collection must be sampled before you can analyze results.
- Click Analyze Results.
- On the Analyze Evidence Collection Results page, you can view the results now or download the results.
- Select a rubric. The list of rubrics includes Outcomes Assessment rubrics and rubrics associated with the assignment within a course.
- If you chose to view the results now, you have additional options: select the date range for the results report and how to compare the results.
- Click View Results. If you chose to view the results now, they appear on the screen. If you chose to download, you are prompted to save the file.