Many studies run into a familiar paradox: the questionnaire is distributed, a reasonable number of responses is collected, the scales can be analysed, yet when results are written up the researcher still struggles to answer the original question. The cause is usually not the software or the sample size. It begins earlier, when questionnaire items are selected because they sound plausible rather than because they are guided by the research question and the theoretical mechanism that the study needs to test.
A questionnaire is not a simplified copy of a proposal. It is an instrument for turning an abstract problem into observations that can be collected, compared, and interpreted. Before asking whether respondents are satisfied or intend to continue using something, a researcher therefore needs to know exactly what that answer will clarify. Without that connection, adding ten more questions normally lengthens the survey without improving the quality of evidence.
A questionnaire does not begin with wording
A common habit is to open an old questionnaire, replace the setting, and add items that seem relevant. It saves time at the beginning but can create three forms of misalignment. First, measures are selected because they are popular rather than because the research question requires them. Second, an item mixes cause, outcome, and overall evaluation in one statement. Third, respondents are asked to recall or judge matters for which they do not have sufficiently direct information.
A better starting point is to write a short description for every content area to be measured: which respondent is being asked, what phenomenon must be observed, over what period, and which argument the response will help test. If a study examines how an online learning experience affects learner engagement, for example, it needs to distinguish the design of the session, instructor support, the learner’s ability to participate, and engagement itself. An item such as “This course is very good” may capture a general impression, but it cannot show what should be improved.
Maintain an unbroken chain of reasoning
Measure design can be checked through a chain of four links. The first is the research question: what does the study genuinely seek to describe, compare, or explain? The second is the construct: which theoretical phenomenon can explain that question? The third is the mechanism: why should this construct lead to that outcome in this specific setting? The final link is the indicator: what evidence can respondents provide that allows the researcher to observe the construct and mechanism?
This chain exposes variables that are interesting but have no clear role. In a study of intention to use an internal application, for instance, the novelty of the technology may be an appealing topic. Yet if the model explains how perceived usefulness and organizational support affect intention, novelty should be added only when there is an argument that it changes a particular relationship or is needed to understand the outcome. Otherwise, it merely expands the questionnaire and makes the model harder to read.
| Check point | Question to answer | Signal to revise |
|---|---|---|
| Research question | Which part of the problem will this measure help answer? | No analytic decision can be identified that the measure will serve. |
| Construct | Does the measure capture a distinct content area or restate another variable? | Items repeat one another, or the construct is too broad or vague. |
| Mechanism | Why might this variable relate to the outcome in this setting? | The relationship rests only on intuition or an old result from a different context. |
| Indicator | Do respondents have sufficiently direct experience to answer accurately? | The item asks them to guess policy, technology, or someone else’s behaviour. |
Write items so respondents understand the same thing
A strong indicator is not an item that sounds academic. It must be specific enough for different respondents to understand it in the same direction, without steering them toward a positive or negative answer. Absolute terms such as “always,” “perfect,” and “never” should therefore be avoided unless the setting truly requires them. Double-barrelled items should also be avoided. “The system is easy to use and helps me work faster,” for example, combines two ideas. A respondent may agree with the first and disagree with the second, leaving the data unable to show what is being evaluated.
Time context also needs a clear anchor. “I often use the system” is not the same as “During the past four weeks, I have used the system for assigned tasks.” The second item lets respondents draw on recent experience and gives the researcher a firmer basis for cautious interpretation. Where a new change is being studied, asking about too long a period can also mix experiences before and after the change.
Example: surveying an internal training programme
A company wants to know whether a data-training programme helps employees apply learning at work. If it asks only “How satisfied are you with the course?”, the result mostly reflects a general evaluation. To test the mechanism, the questionnaire can separate: how closely exercises match actual tasks; learners’ confidence in performing a specified data task; manager support after the course; and frequency of application in the past four weeks. The company can then distinguish whether learning design, post-training support, or conditions at the unit require improvement.
Five review passes before fielding a questionnaire
The first pass reviews content. Create a map linking each item to its measure, working definition, scale source, and relevant hypothesis or research question. If an item has no place on that map, it should be removed or its role should be explained again. The second pass reviews language. Read every item as a respondent unfamiliar with specialist terminology would, then replace terms that can be misunderstood with descriptions closer to real experience.
The third pass reviews order. Questions that are easy and less sensitive are often best placed early so participants understand the setting; items requiring recall or evaluation should be grouped by topic. The fourth pass is a cognitive try-out with a small number of people close to the target sample. Do not ask only whether they “understand” the question. Ask them to explain how they understood it and how they selected an answer. This quickly reveals terms with multiple meanings or an unclear time frame.
The final pass is a pilot with a small, suitable group. A pilot is not only for checking Cronbach’s alpha or survey length. It helps test completion time, item nonresponse, answer distributions, and items that cause respondents to pause. If many people choose the same scale point not because they share an experience but because the response options are hard to distinguish, revising wording matters more than retaining every indicator.
Conclusion
Data can answer only what an instrument collects. When researchers start from the research question, specify constructs and mechanisms, and then select appropriate indicators, the questionnaire becomes part of the research argument rather than an administrative step before analysis. Care at the design stage prevents many forced explanations later and makes findings more useful to decision-makers.
References
DeVellis, R. F., & Thorpe, C. T. (2021). Scale development: Theory and applications (5th ed.). SAGE.
Hinkin, T. R. (1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1(1), 104–121. https://doi.org/10.1177/109442819800100106
MacKenzie, S. B., Podsakoff, P. M., & Podsakoff, N. P. (2011). Construct measurement and validation procedures in MIS and behavioral research. Journal of Business Research, 64(10), 1139–1146. https://doi.org/10.1016/j.jbusres.2010.06.006
