john latham

measurement plan

How will you measure the independent, dependent, and moderating variables? For some studies, it is important to identify a way to measure the change in the variables. For quantitative deductive studies non-parametric measurement instruments lead to non-parametric statistical analyses which are acceptable but not as powerful as the parametric statistical analyses. For qualitative studies the measurement might be qualitative descriptions based on the responses to the questions in an interview guide. For mixed method quasi-deductive studies the measurement plan might include both qualitative descriptions and quantitative measures (e.g., survey questions with scales, performance measures such as financial performance). The measurement plan should be consistent with the overall approach identified in the previous step and the conceptual framework and research questions. Remember - we “measure” variables and “analyze” relationships.

data collection plan

How will you collect the data using the instruments and measurement approaches identified in the measurement plan? The data collection plan includes a sampling plan. Be explicit about the nature of your population and sample, the kinds of data gathering instruments you will use, and the circumstances under which the data are likely to be gathered. Include any specific activities that you would propose to reduce bias and increase validity. The type and level of data that is collected will determine the analysis options that are available. Triangulation of sources and methods is a common technique to mitigate bias and enhance validity.

choosing a sampling approach

There are two main types of samples - probability and non-probability. The main sampling difference for each methodology (qualitative and quantitative) is based primarily on the purpose of the research. If the purpose is to deductively “test” a specific hypothesis then a random sample that is sufficiently large to represent the population is the desired sampling approach for a quantitative study. That way the findings can be generalized to that larger population. On the other end of the spectrum is an exploratory qualitative study with a purpose of “building” a theory. Qualitative inductive theory building studies worry less about representative samples and more about getting the right people to provide a rich data set - often called a “purposive” sample. Of course there are many variations to these approaches including sampling for mixed methods studies. In the end, when practical you want to work toward a representative sample but unless you are expanding the theory to increase generalizability to other populations or testing a theory a purposive sample might be more appropriate.

pilot test?

The need for a pilot study depends on several factors including: (a) the amount of previous experience with the survey (what steps others have taken to validate the instrument); (b) where (what participant groups) the survey have been used with; and (c) your particular situation. Read more...

example

Chad McAllister’s[Prospectus Version] - The population to be explored includes users who are involved in specifying requirements for IS and developers who create information systems. An opportunity sample will be used consisting of three to four companies that meet the following criteria:

• Sufficient size to create NGT groups of users and developers.
• Publicly traded company performing in the top 49% of their industry group (a measure of success determined by the stock market).
• Each company will be from a different industry to obtain a broader perspective.

NGT will be used to generate the factors from both user and developer groups. NGT is a consensus generating technique often used in small groups of six to 12 participants. NGT ensures that each participant has a voice in the group and that all participants are equal in their contributions. The size of the sample is based on the work of Griffin and Hauser (1993) and related research, which determined that 20-30 participants can express 90% or more of the factors. Consequently, three to four NGT small groups of users are expected to capture at least 90% of the factors users believe contribute to misunderstanding requirements. The same is true for developers, resulting in three to four user-developer pairs of NGT small groups. The NGT is expected to involve two hours with each small group. NGT itself is a bias-limiting approach that helps to reduce the influence of the facilitator. As part of the approach, a standard introduction will be used for each group that describes the ground rules and the approach. Also, selecting pairs of small groups from the same company improves validity by implicitly controlling variables that may impact differences in factors between users and developers, such as different software development methodologies, definitions of quality, and the like. After collecting the factors from users and developers via NGT, two aggregated lists will be created—one for users and the other for developers. Two web-based survey instruments will be used to weight the importance of the factors. One will contain the user factors and users will be asked to complete the survey. The other will contain developer factors and developers will be asked to complete the survey. The survey participants will be the same individuals who participated in the NGT small groups.

to do

• Develop a measurement plan for the variables in the research questions and hypotheses.
• Develop a data collection plan including instruments and sampling strategy.
• Support your discussion with solid peer-reviewed references and research methods texts.