Principles of Experimental Design

1 Introduction

Experimental design is a structured approach used in scientific research to plan, conduct, and analyze experiments in a way that ensures valid, reliable, and efficient results. The primary goal is to establish cause-and-effect relationships by systematically manipulating variables while controlling for extraneous influences.

The role and prominence of experimental design may seem diminished in some fields today due to shifts in technology, data availability, and the overall landscape of analytical fields. It reflects a combination of shifting academic priorities, an overcrowded curriculum, and changes in the statistical landscape.

• Big Data and Observational Analysis: With massive datasets available, some researchers lean on sophisticated statistical or machine learning models to infer relationships without running controlled experiments.

• A/B Testing Is Ubiquitous but Simple: In the business world, many practical applications require using basic A/B testing (various comparisons among groups), which doesn’t require deep design knowledge. This leads people to think that experimental design is not a core technique anymore.

• Simulation and Modeling: In engineering or systems biology, simulations can sometimes replace physical experiments, though they’re often based on data originally gathered from well-designed experiments.

• Data-Driven Culture: There’s sometimes a bias toward exploratory data analysis over hypothesis-driven work, which can downplay the importance of prior design.

However, fields like agriculture, biology, medicine, marketing, UX (user experience), and manufacturing still depend on it heavily.

In this short and non-technical note, we will introduce the basics of the principles of the classic experimental designs.

2 The Logical Process of Experimental Design

Experimental design follows a systematic, step-by-step approach to ensure that research is valid, reliable, and interpretable. It is like building a roadmap—it ensures you reach valid conclusions without getting lost in biases or errors. By following this logical process, researchers can confidently test hypotheses and contribute meaningful findings to science.

Below is the logical sequence researchers follow when designing an experiment:

1. Define the Research Problem & Objectives

• What is the research question? We start with a clearly defined research question. For example, does a new fertilizer increase crop yield compared to the standard one?

• What are the hypotheses? The well-formulated research question needs to be translated into an analytic question in the form of a statistical hypothesis. For example,

\[ H_0: \text{No difference between fertilizers} \ \ \ \text{vs} \ \ \ H_a: \text{The new fertilizer increases yield} \]

2. Identify Variables

• Independent Variable (IV) - the cause: The factor variable that is manipulated by the experimenter. For example, fertilizer type, drug doses in clinical trials, etc.

• Dependent Variable (DV) - the effect: The outcome that is measured. For example, crop yield, recovery rate after receiving the drug, etc.

• Control Variables (Constants): Factors kept unchanged across groups to ensure that only the IV affects the DV. For example, soil type, water amount, age group, environmental condition, etc.

• Extraneous Variables (Confounding Factors): Unwanted variables that could influence the DV if not controlled. For example, sunlight, pests, diet, stress levels, or measurement errors.

3 Choose the Experimental Design

Select a design that best fits the research question. There are many different designs. The following is a short list of commonly used designs. We will discuss some of them with more detail in the next Section.

• Completely Randomized Design (CRD): A Completely Randomized Design (CRD) assigns treatments to experimental units purely at random to guarantee that each unit has an equal probability of receiving any treatment. CRD is excellent for isolating causal effects when units are homogeneous

• Randomized Block Design (RBD): In a Randomized Block Design, experimental units are grouped (blocked) to share a known characteristic that is not primarily studied, but it could affect the outcome. Within each block, all treatments are randomly assigned. Controlling for known blocking variables improves the precision of treatment estimates. For example, dividing plants by soil quality or patients by age/gender.

• Factorial Design: A factorial design tests two or more independent variables (factors) simultaneously, using all possible combinations of their levels. This allows assessment of both main effects (individual factor impact) and interaction effects (when the effect of one factor depends on another): For example, in a recent Lancet study, older adults were randomized across two vaccine types (Pfizer‑BioNTech vs. Oxford‑AstraZeneca) and two dosing intervals (3 weeks vs. 12 weeks), creating four groups. The researchers measured antibody levels to assess main effects (vaccine type, dose interval) and whether there was an interaction—i.e., whether the dosing interval effect differed by vaccine type. They found higher antibodies overall with Pfizer, and longer intervals boosted response in both vaccine groups.

• Repeated Measures: This design involves measuring the same participants (or units) multiple times under different conditions or time points. It controls for individual differences and boosts statistical power because each participant acts as their control. Some examples include:

  • Cognitive tests at ages 30, 40, and 50 to track decline or improvement;
  • Drug A vs. Drug B - each participant takes both in different orders;
  • Testing three different drugs for smoking cravings - the same participants receive each drug on different days.

5. Randomization, Blinding & Assignment

These three techniques are critical for minimizing bias, ensuring validity, and strengthening causal inferences in experiments. Below is a detailed breakdown of each concept, its importance, and how they are implemented.

• Randomize participants or units to treatments to avoid selection bias.
  • Simple Randomization - For example, in a drug trial, 100 patients are randomly assigned to either the new drug or placebo (pill with no drug ingredient) group using a computer-generated random sequence.
  • Block Randomization - Ensures equal group sizes by randomizing within small blocks.
  • Stratified Randomization - Randomizes within subgroups (e.g., by age or gender) to maintain balance.
• Blind participants and/or researchers when possible, to reduce expectation effects.
  • Single-Blind: to participants only. It prevents placebo effects.
  • Double-Blind: to both participants and researchers. It eliminates experimenter bias & participant bias.
  • Triple-Blind to participants, researchers, and data analysts. It reduces bias in interpretation.
• Assignment refers to how participants or experimental units are allocated to different treatment groups. Proper assignment is crucial for valid comparisons and minimizing bias.
  • Between-Subjects (Independent Measures): Different participants are assigned to each treatment group.
  • Within-Subjects (Repeated Measures): The same participants experience all conditions
  • Matched-Pairs Design: Participants are paired based on key traits (e.g., age, IQ), then split into groups.

6. Plan Sampling & Data Collection

• Sampling refers to how you select experimental units (e.g., plots, animals, people) from the larger population—ideally through random sampling to ensure representativeness.

• Record measurements accurately (e.g., crop weight, plant height).

7. Analyze Data and Interpretation

This will be discussed in detail in separate notes. The key is to choose appropriate statistical procedures to perform analysis and make a relevant diagnosis. This is an iterative process. Make sure that all assumptions are satisfied before interpreting the results.

8. Reporting and Validating

This is the final step. We will also discuss this in the subsequent note. The key idea is to follow the best practices:

3 CRD and RBD

This section briefly introduces two most commonly used experimental designs: the completely randomized design and the randomized block design, in more detail. We will also organize the resulting data in a format suitable for data analysis using either a software program or manually.

3.1 Completely Randomized Design

The Completely Randomized Design (CRD) is one of the most basic and widely used experimental designs in research. It is particularly suitable when experimental units are homogeneous, meaning there are no significant differences among them before applying treatments. CRD follows the principle of randomization, where subjects (or experimental units) are randomly assigned to treatments. That means each subject has an equal chance of receiving any treatment, and the assignment is done independently of any grouping.

Depending on the number of replications per treatment (i.e., the number of experimental units that receive the same treatment independently), a CRD can be classified as either balanced or imbalanced. Understanding the differences between these two variations is crucial for proper experimental design and statistical analysis.

In the above figure, the number of subjects in each treatment is equal, \(n_1 = n_2 = n_3\), and the hypothetical CRD is a balanced design.

This design is commonly used in agricultural experiments, clinical trials, industrial studies, and laboratory research due to its simplicity and flexibility.

The key characteristics of CRD include:

• Random Assignment: Treatments are allocated randomly to experimental units to minimize bias.

• Homogeneous Experimental Units: All subjects or plots are assumed to be similar before treatment application.

• No Blocking: Unlike Randomized Block Design (RBD), CRD does not group units into blocks to control variability.

• Flexibility: It can accommodate any number of treatments and replications.

• Simple Analysis: Data from CRD can be easily analyzed using one-way ANOVA (Analysis of Variance).

A Non-numerical Example

Suppose a pharmaceutical company wants to test three different drug formulations (A, B, C) on blood pressure. A balanced, completely randomized design will follow the next few steps.

Step 1: Select 30 similar patients.

Step 2: Randomly assign 10 patients to each drug group.

Step 3: Administer treatments and measure blood pressure after a set period.

The next figure illustrates how to read the CRD data table cell.

The formats required for analysis using different software programs will be discussed in subsequent notes focusing on analyzing different experimental designs.

The Completely Randomized Design is a fundamental experimental approach due to its simplicity and ease of application. While it is highly effective in homogeneous settings, its limitations become apparent when dealing with variability. Researchers must carefully assess whether CRD is appropriate or if more advanced designs (e.g., Randomized Block Design) are needed. Despite its constraints, CRD remains a cornerstone of experimental research, providing a solid foundation for statistical analysis and scientific discovery.

The following YouTube video explains CRD (first 15 Minutes, https://www.youtube.com/watch?v=KJ5G2KjcXcA). The analysis part will be discussed in the next note on One-way ANOVA.

3.2 Randomized Block Design

The Randomized Block Design (RBD) is a widely used experimental design that improves precision by controlling known sources of variability. Unlike the Completely Randomized Design (CRD), which assumes homogeneous experimental units, RBD groups similar units into blocks and then randomizes treatments within each block. This approach reduces experimental error and enhances the accuracy of treatment comparisons. RBD is commonly applied in agriculture, clinical trials, industrial experiments, and ecological studies where external factors (e.g., soil fertility, patient age, machine differences) may influence results.

Two related (but sometimes confusing) concepts

• Extraneous Variable - Any variable other than the independent variable that might affect the outcome (dependent variable). It may or may not be a problem. If properly controlled or randomized, it doesn’t bias the results. For example, in a plant growth experiment, sunlight and water are extraneous variables if you’re testing the effect of fertilizer.

• Confounding Variable - A type of extraneous variable that is not controlled and systematically varies with the treatment. It confounds (i.e., mixes up) the effect of the treatment, making it hard to know whether the observed effect is due to the treatment or the confounder.

To summarize, we create the following table to do back-to-back comparisons.

Term	Is it a problem?	Systematically related to treatment?	Affects outcome?
Extraneous Variable	Not always	Not necessarily	Possibly
Confounding Variable	Yes (if uncontrolled)	Yes	Yes

3.2.1 Major Components in RBD

As mentioned earlier, an RBD is an experimental design where subjects or experimental units are grouped into homogeneous groups /blocks based on some similarity (like age, weight, location) to control for variability that isn’t of primary interest. Within each block, treatments are randomly assigned to the units.

Experimental Unit

This is the smallest division of the experimental material to which a treatment can be assigned. For example, in agricultural experiments, a plot of land can be an experimental unit where different fertilizers (treatments) are applied.

Treatment

This refers to the different conditions or interventions whose effects are being studied. For instance, different dosages of a drug are used in a medical trial.

Block

A block is a group of experimental units that are similar in some way that is expected to affect the response to the treatments. The idea is to make comparisons within blocks to reduce the impact of variability between blocks.

In CRD, multiple experimental units are assigned to each treatment. In RBD, introducing a blocking variable makes the design and data layout more complex than the CRD since we need to consider the combinations of blocks and treatments when assigning experimental units.

3.2.2 RBD without Replicates (Basics RBD)

In this scenario, for each block, there is exactly one experimental unit assigned to each treatment. If there are \(t\) treatments, then each block will have \(t\) experimental units, one for each treatment.

The resulting data table has the following structure.

	treat 1	treat 2	\(\cdots\)	treat t
block 1	\(y_{11}\)	\(y_{21}\)	\(\cdots\)	\(y_{t1}\)
block 2	\(y_{12}\)	\(y_{22}\)	\(\cdots\)	\(y_{t2}\)
\(\cdots\)	\(\cdots\)	\(\cdots\)	\(\cdots\)	\(\cdots\)
block b	\(y_{1b}\)	\(y_{2b}\)	\(\cdots\)	\(y_{tb}\)

Example: Let’s consider an agricultural example to make this concrete. Assume that there are three treatments: 3 different types of fertilizers (F1, F2, F3) and four blocks: 4 different fields (Block1, Block2, Block3, Block4) which may vary in soil quality, sunlight, etc. In each block (field), we have 3 plots (experimental units), each assigned to one of the 3 fertilizers randomly, and then measure the yields (\(y_{ij}\)).

The assignment table has the following layout:

	Plot 1	Plot 2	Plot 3
Block 2	F2	F1	F3
Block 3	F1	F3	F1
Block 4	F3	F2	F2

Here, each treatment appears exactly once in each block, and within each block, the assignment is random. The corresponding data table is

	Plot 1	Plot 2	Plot 3
Block 2	\(y_{11}\)	\(y_{21}\)	\(y_{31}\)
Block 3	\(y_{12}\)	\(y_{22}\)	\(y_{32}\)
Block 4	\(y_{13}\)	\(y_{23}\)	\(y_{33}\)

This design guarantees that:

• Control of Variability: By grouping similar experimental units into blocks, we control for the variability between blocks. This increases the precision of the treatment effect estimates.

• Balance: Each treatment appears equally often in each block, making the design balanced, which simplifies analysis.

• Randomization: Random assignment within blocks helps in avoiding systematic bias.

3.2.3 RBD with Replicates (Replicated RBD)

The basic RBD is simple and easy to understand. Some practical information is not available in this simple design, for example,

• We cannot estimate interaction between treatments and blocks or have a pure error estimate within blocks since only one experimental unit is assigned in the combination of treatment and block.

• In the modeling phase, we cannot assess the potential interactive effect. That is, the effect of a treatment depends on the block (or vice versa). For example, if F1 performs best in Block 1 but worst in Block 2, there is likely a treatment × block interaction.

• The Randomized Block Design (RBD) requires homogeneity (i.e., equal variance) within blocks and treatments in order to be effective. However, the basic RBD does not provide enough information to assess this homogeneity because there is only a single experimental unit for each combination of block and treatment.

There are different ways to overcome the above limitations. One way is to modify the basic RBD to allow assigning multiple experimental units to every combination of blocks and treatments - Replicated RBD.

A Replicated RBD is an extension of the standard Randomized Block Design where each treatment is applied to multiple experimental units within each block. The basic structure of replicated RBD is characterized in the following.

• Treatments (a): Different levels of the factor being tested (e.g., fertilizers, drugs).

• Blocks (b): Groups of homogeneous experimental units (e.g., fields, patients with similar characteristics).

• **Replicates (p)*: Number of times each treatment is repeated per block.

The general data table of a replicated RBD design is depicted by

The data table cell has three indices to capture information about blocking, treatments, and measurements of replicated response, as shown below.

The next YouTube video explains the RBD (with no replicates) using a nice example. You can skip the analysis part now. The video is at (https://www.youtube.com/watch?v=zqZ9iuk5Ngk&t=193s)

3.2.4 Concluding Remarks on RBD

The Randomized Block Design (RBD) is a fundamental experimental approach that enhances precision by grouping similar experimental units into blocks to control for known sources of variability. The choice between RBD without replication (one experimental unit per treatment per block) and RBD with replication (multiple units per treatment per block) depends on the research objectives, resource availability, and underlying assumptions about treatment-block interactions.

RBD without replication is a simple yet powerful design when interactions between treatments and blocks are negligible. It efficiently reduces error variance by accounting for block effects while requiring fewer experimental units. However, its inability to test for interactions means that any hidden treatment-block dependencies may inflate the error term, potentially masking true treatment effects. This design is ideal if we have prior knowledge suggesting consistent treatment effects across blocks.

In contrast, RBD with replication offers greater flexibility and precision by allowing researchers to detect and quantify treatment-block interactions. This design also has limitations. For example, this design cannot handle the situation where a fertilizer performs differently in distinct soil types. The analysis of replicated RBD is also more complex than the basic RBD.

Ultimately, the decision between these two approaches hinges on balancing practical constraints with scientific rigor. RBD without replication provides an efficient solution for straightforward comparisons, while RBD with replication delivers deeper insights when interactions are plausible or higher precision is required.