2. Defining objectives

An essential step in any scientific study is defining your objectives. After all, if you don't define your objectives carefully, how can you expect to fulfill them!

The purpose of this section of the course is to convince you of the importance of having explicit objectives, to give you the tools to identify the objectives of published studies, and to give you the tools to define and evaluate the ecological and statistical objectives for new studies in vegetation science.

What kind of study is it?

Ecological objectives

Statistical objectives

Hints

• Frame the ecological question so it can be answered by the study results

• Make the objectives consistent

• Recognize what the statistical population is

• Set meaningful and useful levels of confidence and reliability

Putting your knowledge to the test

What kind of study is it?

The first step is to determine what kind of study you are undertaking. Then you need to set your ecological and statistical objectives.

Consider as an example the growing industry in harvesting non-timber plants from forests for the florist trade. For example, sword fern (Polystichum munitum) is becoming an important crop. There are at least three types of questions a vegetation scientist might ask about sword fern in the woods. A quantity to be estimated (with confidence interval). For the sword-fern example, a question of this type would be "how much sword fern is there in this forest?" You might use this figure as part of a business plan to supply sword fern to the florist market. A decision to be made, like "should my company thin this forest to promote sword fern?" A hypothesis to be tested, like "forest thinning promotes sword fern growth."

Ecological objectives

It is important to make your objectives clear and explicit, because different objectives might require a different set of field methods and analysis. For virtually any vegetation science project, start by asking yourself six questions. (Examples for the questions about sword fern are in italics.)

1. What ecological question am I trying to answer? (How much sword fern is there in this forest?)
2. What attribute am I trying to measure? (Sword fern biomass.)
3. For what patch of nature do I want an answer? That is, what is the statistical population? (I want an answer for the 80 ha forest in the next county owned by my uncle, who is thinking about supplying sword fern to local florists.)
4. What level of confidence do I want in my answer? (I want to estimate sword fern biomass with 90% confidence.)
5. How reliable must my statements be? (I want to estimate sword fern biomass within 40 kg per ha.)

Before trying to obtain the study's objectives, it is important to identify any important constraints in getting an answer. (My uncle has given me $800 to pay for a field assistant. I need to get an answer by the end of next month. My uncle has given me permission to harvest sword fern from field plots in deriving my estimate of overall biomass.)

Statistical objectives

After setting your ecological objectives, it is time to express some of these objectives in statistical terms. It is important to think first in ecological terms, to stay true to your objectives. But it is important also to think in statistical terms, to help you collect field data that will meet your ecological objectives in a statistically valid way. When going through this list you might want to refer to a table showing ecological and statistical objectives side by side.

1. The first step is determining the question you are trying to answer. This is a purely ecological question.
2. Understanding what you are trying to describe: Will the measurements be on individuals or on areas? This determines how samples are selected.
3. Understanding what you are trying to describe: Defining the sampling universe. This determines how widely you can make valid inferences.
4. What is the required statistical confidence level?
5. What is the required confidence interval? The narrower the confidence interval, the more precise is your estimate.

The constraints you identified under ecological objectives also set boundaries on the amount and quality of the statistical information you can collect.

Hints

Frame the ecological question so it can be answered by the study results

The ecological question needs to be one that can be answered. Answering a question like "how can the sword fern harvest be made sustainable?" is beyond the scope of a single study. This goal of sustainability is served by well-defined studies that ask questions like "what is the current biomass of sword fern in forest tract 3B?" and "what is the growth rate of sword fern after harvest?"

Make the objectives consistent

Success requires that the objectives be consistent with each other and that the statistical objectives support the ecological objectives. For example, what would you think of a study of sword-fern harvest in the Cascade Mountains, actually sampled in the forest behind the author's home? Or a study that wants to use its data to convince dubious forest managers, but has a statistical objective of a 75% confidence level?

Recognize what the statistical population is

Many vegetation scientists not only conduct studies, but evaluate the studies of others. Sometimes the evaluation is of reports or published articles, sometimes of study proposals. An important step in evaluating a study is recognizing the statistical population, also called the sampling universe. As it says in the Background chapter, "identifying the statistical population helps determine the proper sampling scheme and how widely you can apply your conclusions." Mismatches between statistical population, sampling scheme, and conclusions can be fatal flaws.

There are several detective techniques you can use to recognize a study's statistical population.

• If you're lucky, its stated explicitly, like "the statistical population was the 412 trees larger than 120 cm DBH in McDonald Forest."
• A defined study area can be a good hint, like "the study area was the 20-m wide riparian zone along the Sandy River from the 5-km mark to the 20-km mark." But recognize that often such "study areas" are larger that the actual statistical population. For example, bridges or tributaries or campgrounds within the study area might have been excluded.  Such exclusions should be part of the objectives, but rarely are.
• The sampling design can provide clues. When the methods state something like "quadrats were sampled every 10 m along a transect bisecting the field," it usually means that the authors consider the field to be the sampling universe. But as you'll learn in the Sampling Designs chapter, the true statistical population for this type of sampling is much smaller than the entire field.
• The breadth of conclusions is another hint. A statement like "sword fern biomass within section 2 of the BLM district averaged 52 g/m² suggests that section 2 was the sampling universe and all possible sampling units within section 2 made up the statistical population. You will also encounter statements like "sword fern biomass in these western hemlock forests is higher than reported for Douglas-fir forests." This implies that the sampling universe was all western hemlock forests, which is almost certainly not the case.

When these clues give inconsistent indications of the statistical population, it is time to consider carefully whether the study's conclusions are valid. Happy sleuthing!

Set meaningful and useful values of confidence and reliability

You might be wondering how you know what level of reliability and confidence you need. Setting the required reliability is a biological question. For example, in a study of grazing effects, you might want to estimate plant cover only to within 15% cover because you have reason to believe that a decline of 15% cover is the threshold for a meaningful biological effect. In other words, the required confidence interval describes how small a signal you want to be able to detect. Unfortunately, there are no hard and fast rules in vegetation science for setting the required width of the confidence interval.

• One approach is the psychological one. Imagine yourself reporting results to a group of coworkers and supervisors (heck, throw in a professor or two). Imagine yourself standing up and saying: "I estimate that the biomass of Aster curtus is 8.5 kg/ha ± 5.0 kg/ha." Would you feel comfortable that you were conveying useful information? Perhaps the confidence interval is a bit too wide for you. What if you estimated biomass as 8.5 kg/ha ± 3.0 kg/ha? How about 8.5 kg/ha ± 1.0 g/ha? You can go through this psycho-drama to figure out your intuitive sense of adequate precision.  Once your intuitive sense is out in the open, you can examine it biologically.  In the example, you can ask "why is ± 5.0 kg/ha not good enough, but ± 3.0 kg/ha is adequate reliability?"
• Another approach is to use previous studies. Did someone show that a difference in a species's biomass of X was a meaningful difference, but a difference of only Y was not? This kind of information can help set your target level of reliability. (But reviewing the literature is not a requirement for BOT 440/540.)
• Sometimes it is useful to know the range of natural variability.  Any future change beyond the range of natural variability would trigger concern.  If this situation fits your objectives, you can set your required confidence interval to the range of natural variability.  That is, you are specifying that you want to detect any variability that exceeds the range of natural variaiblity.
• A hybrid approach is to think of your study as serving as baseline information for future monitoring and management. Consider how different the values would have to be in the future to warrant action. For example, if shrub cover is now 20% but 5 years from now it was 21%, would that trigger shrub control? How about if it grew to 30%? 50%? At whatever point action is triggered is the point you want to be able to detect. And that determines how precise your current measurements need to be. Let's say it would take a change from 20% to 30% to trigger action. If your current measurements had a precision of 20% ±18%, they wouldn't be precise enough to confidently reveal the biologically important change from 20% to 30%. That is, you need to design a better study.

Confidence level is a social and statistical question because it determines how well you can convince others.

• For example, if you are collecting data to influence management actions, you might want to set your confidence level high if it will take a high level of confidence to convince the management agency.  Your reasoning might go like this: "The outcome of this study could likely influence a land management decision, so managers would need strong evidence to be convinced of the results. Therefore I will require a high confidence level, 95%."
• Maybe you and your audience are exploring patterns. In this case a high level of confidence is unimportant, because promising results will be testing by follow-up studies.
• Consider a third example, where you might want to estimate weed abundance reliably and with high confidence because, if your estimates exceed a certain threshold, expensive control measures kick in.
• Or if you plan to publish your study in a particular journal, the field of that journal might have an "industry standard" level of confidence, which is often 95%.

Warnings

• How hard your study is has nothing to do with how reliable your statements must be. If the work is so hard that you cannot obtain adequate reliability, then the project is not worth doing. Likewise, how much the study costs has nothing to do with the level of confidence you need. Your audience will not be more convinced by your results if you tell them you had to pay overtime wages to your crew. If the work is so costly that you cannot obtain a sufficient confidence level to convince your intended audience, then the project is not worth doing because the results would not be useful. You can incorporate costs, by designing the study to be big enough to reach the required objectives, but no bigger.
• The required confidence interval and confidence level should be determined independently. They should not be bargained against each other. If you can't accomplish the study to meet both of these objectives, the study is not worth doing. Either they are requirements, or they are not.

Implications of the objectives you define

Decisions about sampling universe, confidence interval, and confidence level directly determine how long the field work will take (which might push against a constraint). You must take more measurements in the following circumstances:

• In order to relate your results to a larger sampling universe (for example, the whole county vs. your uncle's forest).
• In order to increase the precision of your estimate to shorten the confidence interval (for example, to estimate sword fern biomass to ±20 kg per ha vs. ±40 kg per ha)
• In order to increase the confidence you have in your estimate (for example, wanting to be right 95% of the time vs. 90% of the time)

An example from a real-world project

A past student designed his field study to answer a current question at his agency.  I have paraphrased his study (and my comments to his proposal) to serve as an example of how to set confidence levels.

Picture this scenario. In 2005 wildlife biologists determined that the deer herd would suffer if bitterbrush biomass dropped by more than 15% from its level at that time, which was 1000 kg/ha . If that happened, it would trigger management actions like controlling hunting and planting bitterbrush. You are designing a sampling survey for to monitor bitterbrush biomass. What level of reliability do you require?

Management actions would be triggered if bitterbrush biomass dropped below 850 kg/ha, 15% less than 2005 levels. If you designed your study to have a confidence interval of ±500 kg/ha, you would have little idea of the actual bitterbrush biomass. For example, your estimate might be 790 kg/ha ± 500 kg/ha. It is true that 790 kg/ha is less than 850 kg/ha, but you should hesitate to trigger management action. After all, your confidence interval includes the trigger point of 850 kg/ha. Your confidence interval even includes the 2005 bitterbrush estimate of 1000! That is, you’re not sure that bitterbrush has declined at all, let alone declined by more than 15%!

Now look at the situation when you have higher reliability. If your confidence interval was ±50 kg/ha, the outcome is much different. This high level of reliability permits confident decisions. Management actions are triggered when they should be, and false triggers are kept low.

The distinction between bad and good occurs when the confidence interval overlaps with the threshold for management action. (Remember, setting the required confidence interval is a biological decision.) If your confidence interval is ±150 kg/ha, or 15% of the mean, it is just good enough to be useful for future decision making. For example, if you estimate that current bitterbrush biomass is 849 kg/ha ±150 kg/ha, at least you are confidence that bitterbrush has declined, and the chances are greater than 50% that bitterbrush has declined past the trigger point. Therefore an objective for your sampling design should be to have this level of reliability or better.

Where next?

With all the pressures to increase the number of measurements, and with constraints in time and money, vegetation scientists are tempted to use shortcuts in their sampling. Before we can explore types of field study designs suitable for vegetation science, we need to explore further some important questions: What attribute am I trying to measure? How do I measure it? From what patch of nature do I want an answer? These topics are covered in the sections How to Measure and Sampling Designs.

Putting your knowledge to the test

Now it is time to put your understanding of the importance of setting ecological and statistical objectives into practice. Complete the exercise Defining objectives.

© 2007 Mark V. Wilson and Oregon State University