Four main scoring rubrics are defined, and will be continuously refined by request of authors and admins. Prospective authors may request access to these rubrics to test their papers before new submissions.
Total paper score = Total of all individual scores.
All papers receive an initial scoring done by AI:
1. Structure: AI scoring gives an idea of the quality of the paper, giving an objective scoring of set criteria. All papers are scored using the same AI and the same set of criteria in a formal rubric (structure prompt).
2,3. Content: All papers are scored for content, using two different AIs and the same formal rubric (content prompt).
4. '50 point score' score (content and structure).
4. Once papers are listed and sorted by AI score, prospective authors are invited (required if they want to author or post new papers) to provide more subjective scoring, based on paper structure and content. Authors are also requested to validate the AI scores so we can strengthen our scoring rubrics (prompts). This last score is not applied to the paper total but listed separate. High score here is a validation of AI scores. Low score here means AI did poorly at review and awarded too high or too low a score for the paper structure and/or content.
The rubric below is the standard by which AI scores all papers for listing on this site. Papers are listed and ranked by total score. The main purpose of AI is to check for paper quality, as in structure and scientific method.
Note: It's early days for us, and we're still working on AI to get consistent and valid scores. Current scores are likely to change.
An AI score of content is provided as per rubric below. Since AI is not so adept at subjective scoring, further scoring by humans are required.
Note: It's early days for us, and we're still working on AI to get consistent and valid scores. Current scores are likely to change.
AI will give a points-based review on the paper for structure and content.
In order to close the loop, humans will review the initial AI reviews for structure and content, and validate the score given by AI. This score is used to validate and improve AI rubrics. Humans will review the paper for structure and content. This complements the initial scoring by AI and the paper can now be considered as formally reviewed. Multiple reviews by separate reviewers are envisaged for each paper.