Community Review - Moderation Process (LV2)

All Community Reviewers' reviews will be subject to an automated review process using Catalyst Flags. Catalyst Flags are pieces of code known as scripts that are run to analyze all reviews by the Community Reviewers. Catalyst scripts will raise flags in case of:

• Length ~ blank reviews and reviews containing less than 150 characters in any impact, feasibility, and value-for-money criteria.

• Similarity ~ reviews which are identical or almost identical to a previously submitted review

• Profanity ~ reviews that contain profanities or abusive content or language

• Use of AI ~ review that do not reflect the opinion of the Community Reviewer or which are provided by others or an AI-agent acting on their behalf.

All reviews that have been flagged will go through a moderation process involving Community Moderators (LV2) except for the length check, as those flags will be automatically processed prior.

• Community Moderators will be randomly allocated flagged reviews for further assessment. In addition, between 1-5% percent of reviews not flagged by scripts will be allocated to LV2 Community Reviewers for a spot check (depending on the total reviews received).

• Community Moderators will be provided a secure login for their moderation function in the Moderation Module.

• Community Moderators can see which flags scripts have raised about the review/reviewer.

• After reading the review and flags, Community Moderators will submit their moderations, marking reviews they believe were legitimately flagged and providing a reason for removing the review. E.g. Where the flag is raised because of potential ChatGPT AI involvement or plagiarism, the Moderator must judge whether they believe the review to not be a reviewer's own opinion. If the Moderator agrees with the flag, they mark it as REMOVE and give a rationale.

• The rationale should be a short written statement; a selection must be made from a menu of reasons. For example, the moderator would select from options such as

  • I believe this is not the person’s own opinion

  • There is profanity

  • There is too much similarity with another review

  • Rationale doesn’t correspond with the score given

  • Rationale contains a false statement

• If Moderators disagree with the flag and believe the review should be included for voters to consider, they mark it as VALID and give a rationale.

• Moderators should only discard reviews if there is a valid reason why voters shouldn’t see that review. Unless the review has an apparent reason, such as the below flags or another obvious reason, for instance, spamming the system, then the review should be included.