Conducting Educational Research
Step 9: Develop Instruments


Adapted from Singleton & Straits (2010)

As can be seen from the figure above, once the research design has been determined, then the researcher has two separate responsibilities: identifying the sample and determining how the key variables will be measured. The key variables should have been identified in Step 2: Identify Key Variables and Research Design. Now that the sample has been identified, the researcher must now transition to thinking about what the variables mean both theoretically and practically to determine how to measure the key variables in the study. Measurement consists of assigning numbers or labels to the units of analysis that accurately represent their position on the variables under study. For example, if a research study is examining Maths achievement of JS1 students, each unit of analysis (JS1 student) must be assigned a number that represents their position in Maths achievement. This can simply be done by assigning them a score on a Maths exam.

Except for qualitative research designs in which the key sources of data include words, pictures, or objects, the key variables need to be translated into numbers so the researcher can analyze the data using statistics. The process of measurement consists of moving from the theoretical definition of a variable (typically called the construct definition) to the concrete mode of measuring that variable in the research study. Measurement is typically done by developing an instrument, which can be a questionnaire, an examination, an interview, an observation schedule, etc.

When designing an instrument, keep in mind the following:

  • The conclusions drawn in a research study are only as good as the data that is collected.
  • The data that is collected is only as good as the instrument that collects the data.
  • A poorly designed instrument will lead to bad data, which will lead to bad conclusions.
  • Therefore, developing a good instrument is the most important part of conducting a high quality research study.

Developing an instrument takes a lot of work and advanced preparation. It is very important that the instrument has been thoroughly critiqued, evaluated, and pilot tested by the student, supervisor, and others before it is administered for the actual study. Once the instrument has been administered, it cannot be changed. Any problems with the instrument after it has been administered will require the student to completely redo the data collection process, wasting considerable time and money. Therefore, researchers must take great thought and care when developing their instrument.

There are many steps that must be done in developing an instrument, which will be described below.

Step 1: Identify Other Research Studies that Study the Key Variable

As just mentioned in the previous Step 8: Search Literature, the first step in measurement is to find other research studies that have examined the same variables. This step is helpful in developing the instrument for two reasons.

  • Develop a construct definition of the variable. Use other research studies to develop the construct definition of each variable (Step 2 of developing an instrument below).
  • Provide ideas on how each variable should be measured. As just stated in Step 8: Search Literature, the way a variable was measured in other research studies provides direction on how you should measure the variable. If the instrument that was used in other research studies cannot be adopted exactly, it will at least provide a general idea of how you should attempt to measure the variable.

Step 2: Develop a Construct Definition

The second step in measurement is to develop a construct definition of each of the key variables. The construct definition is the theoretical definition, and this definition is best obtained by referencing other research studies that have measured the same variable. For example, a well-established definition for socio-economic status is a person's economic standing based on lifestyle, prestige, power, and control of resources (Liu, Ali, Soleck, Hopps, Dunston, Pickett, 2004).

A researcher must develop a construct definition for each key variable identified from Step 2: Identify Key Variables and Research Design. The variables that are associated with a treatment such as a new teaching method or counseling procedure, are not measured but manipulated. These variables will not pass through the measurement process, but instead will be manipulated by treatment and control groups. For more information on manipulating variables, go to Step 11: Manipulate the Treatment.

Step 3: Operationalize the Construct Definition

Once the construct definition has been developed, the researcher must translate that theoretical definition into a concrete way of measuring the variable so numbers can be assigned. The abstract construct definition of socioeconomic status provides no information about how to measure socioeconomic status. Therefore, the researcher needs to think of how to make that definition more concrete, or how it will be practically measured, a process called operationalize. The operational definition of a variable consists of a statement of specifically how the construct will be measured or implemented in the study. In most studies, socio-economic status is typically measured by a composite of an individual's level of income, occupation, and level of education.

The operational definition must directly relate to the construct definition. For example, the operational definition of socio-economic status must be closely related to the definition "a person's economic standing based on lifestyle, prestige, power, and control of resources." It would be inaccurate for a person to give an operational definition of socioeconomic status as "amount of time spent at work" because this operational definition does not closely relate to the construct definition. The best operational definition is one that has the clearest relationship to the construct definition.

In causal-comparative studies, it is also important that the operational definition states how participants will be assigned to the groups. If gender is the key independent variable in a causal-comparative study, it is clear that a participant's self-report of their gender will assign them into the appropriate group. However, other studies might not be quite as clear. For example, a causal-comparative study might classify participants into groups based on socio-economic status. What criteria will be used to determine whether a person is low or medium status? This criteria must be clearly specified.

When developing the operational definition, a researcher must consider practical limitations. Imagine you want to measure a primary 1 child's level of socio-economic status. Can a small child report their family's income, occupation, and level of education? Even measuring income by adults might be difficult as non-civil servants oftentimes do not have a steady income, or may be hesitant to report how much they earn. How can these limitations be overcome? This requires perseverance and creativity on behalf of the researcher: perseverance by reviewing other research studies to find out how other researchers overcame this difficulty and creativity by thinking up new methods when necessary. Perhaps socio-economic status can be measured in small children by the type of school they attend (public for low socio-economic status and private for middle socio-economic status) or by mode of transportation (taxi for low, own vehicle for middle, and multiple vehicles for high) for adults.

Other research studies and types of variables have other practical considerations. Can small children complete a questionnaire? How then will a researcher examine the emotional development of a small child? Perhaps through an observational checklist by a parent or teacher. How will a researcher measure engagement in examination malpractice? Can people be honest in reporting their dishonesty? Maybe a questionnaire will work, but participants need to be continually assured of their anonymity in responding to the questionnaire so they feel comfortable being honest. Developing an operational definition of a variable requires considerable thought.

Another practical consideration is the knowledge of the participant. Sometimes, I have seen questionnaires that ask participants to indicate socio-economic status by circling either low, middle, or high. How would a participant know whether they are low, middle, or high status? How is the participant supposed to know what "low" means? Participants need objective criteria so they can give an unbiased account of their socioeconomic status. The process of measurement requires finding an objective, indisputable way of assigning numbers to low, middle, and high. A participant must have enough knowledge to be able to objectively state their response.

Types of Measurement

There are multiple types of measurement to consider when thinking about how to operationally define a variable.

  • Self-Report. Participants can be asked to report on their own demographic characteristics (also called biodata), attitudes, beliefs, knowledge, feelings, and behavior. As will be discussed in Writing Questionnaire Items, participants cannot accurately report on the attitudes, beliefs, knowledge, and feelings of others except in very rare instances. For example, teachers cannot report on how well their students enjoy a particular topic. Instead, students must self-report their own enjoyment. Self-reports can be either a questionnaire or an interview.
  • Observation. Researchers can also observe the behavior of others. For example, a researcher wants to study how often teachers use open-ended questions in their lessons. The best way to measure the frequency of open-ended questions is for a researcher to go into a classroom, observe the instruction, and tick the number of times the teacher uses an open-ended question. Typically, observation requires a researcher to complete a check-list of the presence or absence of particular behaviors. In rare cases, participants can also complete observational checklists on others. For example, a parent could complete an observational checklist on the behavior of their children. However, researchers must use this strategy thoughtfully, ensuring that the participant has a close association with the person they are reporting on. Will a teacher who has 50 students in the classroom be able to accurately complete a behavioral checklist on each student?
  • Archival Records. Archival records refers to existing information. For example, educational researchers are oftentimes interested in students' academic achievement. Depending on the particulars of the research study, academic achievement can sometimes be obtained by going into students' school records and collecting their grades from previous terms.
  • Examinations. Particularly when a researcher is interested in participants' knowledge, it is best to give an examination over the topic. For example, a researcher might be interested in teachers' knowledge of literacy development. Teachers likely will not accurately self-report on their knowledge, either because they do not know their level of knowledge, or they might not want to appear ignorant and therefore will bias their responses. Instead of asking teachers to self-report on their knowledge (e.g., I understand how children learn to read), giving an examination will be more accurate.

Step 4: Choose an Instrument

Each variable in a research study needs to be measured separately. A research study examining the effect of intrinsic motivation on academic achievement may be able to use the Intrinsic Motivation Inventory to measure intrinsic motivation. Another instrument will be necessary to measure academic achievement. Because each variable must be measured separately, a researcher may use five or six different "instruments" may be used for one research study. For information on how to combine the instruments, go to Developing the Questionnaire Format.

"Development of new tests is a complex and difficult process that requires considerable training in educational and psychological measurement. Therefore, we recommend that you make certain no suitable test is available before developing your own" (Gall, Gall, & Borg, 2003, p. 216).

Using a pre-developed instrument to measure the key variables in your study has a number of advantages.

  1. An instrument published in the literature has already been developed, so it saves the time and energy necessary to develop a new instrument.
  2. The researcher typically has conducted an entire research study for the sole purpose of validating the instrument, so the instrument will likely be more valid. Furthermore, since the instrument has already been validated, you do not have to spend the time and energy necessary to validate a new instrument. It is often said that you can either develop an instrument for your entire thesis or use an instrument that has been developed elsewhere to test your research hypotheses. In other words, the development and validation of a new instrument could be an entire thesis in and of itself.
  3. Using a pre-developed instrument connects your research study to the entire body of research that has used the same instrument. When using a newly developed instrument, it is often unclear about how your research findings relate to those found by other researchers. However, if you use an instrument developed by other researchers, then you know that your study fits well into the worldwide knowledge base on your variables of interest.

Finding an already-existing instrument is best done through a literature review search. When reading over the instruments section of a research study, look to see if a table or an appendix provides the actual items on the questionnaire. Other times, the researcher will cite the original source of the instrument, so check the citation. For example, a line might read, "Subtests from the Intrinsic Motivation Inventory (Ryan, 1982) were used to assess relatedness, intrinsic motivation, autonomy, and competence." The source of the instrument is Ryan (1982), so find this reference in the References section. If you can get a copy of the Ryan article, then you likely will get a copy of that instrument in either the appendix or a table.

A number of resources are also available on the internet to assist in finding an instrument that will measure the key variables in your research study.

As Gall, Gall, and Borg (2003) stated, a researcher should make a thorough search of the literature and internet to try to find an instrument to use that has already been developed. A researcher should make their own questionnaire only if an appropriate instrument is not found.

Step 5: Write the Operational Definition

There is currently debate in the Faculty of Education at UniJos about whether there should be an Operational Definitions section to Chapter 1. However, a researcher absolutely must develop an operational definition for each variable as part of the planning process regardless of whether this section is required or not. It is best if a researcher presents each operational definition to the supervisor when getting approval for the instrument even if it is not required as part of the thesis itself.

The operational definition should include the following components:

  • The construct definition
  • The type of measurement (i.e., self-report questionnaire or archival records)
  • Specifics of the measurement:
    • For Self-Report, Checklist, and Examinations: How many items measure the variable? What types of items are used? For example: open-ended, likert scale, multiple choice, etc?
    • For Archival Data: What type of data was used? For example: exam scores, WAEC scores, attendance records, etc?
  • For Causal-Comparative Studies: Criteria for assigning to groups if the variable is an independent variable. For example, what was the cut-off point in yearly income to determine whether the participant was low or medium socioeconomic status?

For example, a good operational definition of socio-economic status might read:

Socio-economic status is typically defined as a person's economic standing based on lifestyle, prestige, power, and control of resources (Liu, Ali, Soleck, Hopps, Dunston, Pickett, 2004). In this study, socio-economic status was defined as participants' self-report of their average monthly income.

Another operational definition might read:

Intrinsic motivation is defined as the motivation for a behavior is based on the inherent enjoyment in the behavior itself (Ryan and Deci, 2002). In this study, intrinsic motivation was measured by responses to seven Likert-scale items on a self-report questionnaire.

For a causal-comparative study, an operational definition might read:

Cheating is typically defined as dishonest practices by students that misrepresent their true understanding of course content. Cheating behavior was operationally defined as whether students were caught blatantly cheating on the assignment or final exam in their 100-level educational psychology course.

NEXT

Return to Educational Research Steps

Copyright 2012, Katrina A. Korb, All Rights Reserved