Proposal Evaluation Criteria
The following criteria will guide the PIs and the Board in evaluating proposals made through the Online Commons. We strongly encourage anyone who is considering making a proposal to read the following carefully.
1. Problem-Relevant. Are the theoretical motivations, proposed concepts and survey items relevant to ongoing controversies among researchers? How will the data that the proposers expect to observe advance the debate? What specific analyses of the data will be performed? What might these analyses reveal? How would these findings be relevant to specific questions or controversies?
2. Suitability to ANES. The primary mission of the ANES is to advance our understanding of voter choice and electoral participation. Ceteris paribus, concepts and instrumentation that are relevant to our understanding of these phenomena will be considered more favorably than items tapping other facets of politics, public opinion, American culture or society.
3. Building on Solid Theoretical Footing. Does the proposed instrumentation follow from a plausible theory of political behavior?
4. Demonstrated Validity and Reliability of Proposed Items. Proposed items should be accompanied by evidence demonstrating their validity and reliability. Validity has various facets: e.g., construct validity, concurrent validity, discriminant validity and predictive validity. Any assessment of predictive validity should keep in mind criterion 2, above. Reliability can be demonstrated in various ways; one example is test-retest reliability. We understand that proposals for novel concepts and/or instrumentation will almost always lack empirical evidence demonstrating validity and/or reliability. Proposals for truly “novel” instrumentation might be best suited for the series of smaller, cross-sectional studies ANES will field in the period 2010 through the summer of 2012; as a general matter, we are highly unlikely to field untested instrumentation on the Fall 2012 pre-election and post-election surveys.
5. Breadth of Relevance and Generalizability. Will the research that results from the proposed instrumentation be useful to many scholars? Given the broad usage of ANES data, we may be unable to accommodate requests to include items that are relevant for one – or only a few – hypothesis tests. Ceteris paribus, items that are potentially relevant for a wide range of analyses will be considered more favorably than items that would seem to have less applicability.