scholar-evaluation

Review scholarly works with evidence-traceable rubrics and local JSON quality-control scripts.

Updated Aug 12, 2026
One-click install
npx skills add https://github.com/Mzane0803/latent-minds-skills-marketplace --skill scholar-evaluation-mzane0803
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: scholar-evaluation
Source: https://github.com/Mzane0803/latent-minds-skills-marketplace/tree/main/plugins/research-writing/skills/scholar-evaluation
Command: npx skills add https://github.com/Mzane0803/latent-minds-skills-marketplace --skill scholar-evaluation-mzane0803

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve? Research feedback and low-stakes assessment processes often lack traceable evidence, documented constructs, and governance controls, and risk being misused for consequential personnel decisions. This Skill provides a qualitative-first developmental review workflow for scholarly works with strict safety boundaries and auditable local tooling. ## Core Features & Use Cases - Developmental Review Workflow: Guides construct definition, rubric adaptation, independent rating, and qualitative synthesis for papers, drafts, protocols, and research ideas. - Local Quality-Control CLIs: Bundled standard-library Python scripts validate rubrics, compute bounded descriptive scores, check evidence traceability, summarize inter-rater agreement, test weight sensitivity, and audit process governance. - Hard Safety Boundaries: Prohibits ranking people, proxy metrics like impact factor or h-index, and any use in hiring, tenure, funding, admissions, or award decisions. - Use Case: A journal club committee wants structured, evidence-cited feedback on a manuscript draft. They adapt the rubric template, record ratings with evidence references, run the traceability and agreement checks, and release a developmental report after human committee review. ## Quick Start Ask the assistant to developmentally review a scholarly work using the scholar-evaluation rubric template and validate it with the bundled local scripts.

Frequently Asked Questions about scholar-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I review a research paper with a structured rubric?▼

Adapt the rubric template in assets/rubric_template.json, record criterion ratings in an evaluation JSON with evidence references, then run calculate_scores.py and check_traceability.py locally. Synthesis leads with criterion-level evidence, not a composite score.

How to check inter-rater agreement for scholarly review ratings?▼

Prepare a CSV with pseudonymous rater IDs and criterion scores, then run summarize_agreement.py with your rubric. It reports exact agreement, within-one-step agreement, and mean absolute difference per criterion and overall.

Can this skill be used for hiring, tenure, or grant decisions?▼

No. The skill has a hard safety boundary prohibiting hiring, promotion, tenure, admissions, funding, awards, discipline, and any person ranking. It only supports developmental feedback on scholarly works and process-only audits.

Does the rubric scoring require internet access or external packages?▼

No. All bundled scripts use only the Python 3.11+ standard library and process bounded local JSON/CSV files. There is no network access, credential use, model calls, or subprocess execution.

Why does the process checklist template fail validation?▼

The process checklist template is intentionally unconfirmed and fails closed. You must complete every governance, fairness, accessibility, and data-protection control from documented local records before check_process.py reports a complete status.

What are the limitations of the ScholarEval-based rubric scores?▼

Scores are ordinal rubric summaries, not validated psychometric measurements. The referenced ScholarEval preprint is experimental, uncertainty ranges are not confidence intervals, and agreement or weight stability does not establish validity or fairness.