Skip to contents

1. Dealing with serial correlation

Source: vignettes/b-prewhitening.Rmd
b-prewhitening.Rmd

Why this matters

Serial correlation occurs when successive observations in a time series are not statistically independent. Positive autocorrelation can increase false trend detection (Type I errors; von Storch and Navarra, 1995), whereas negative autocorrelation can reduce the ability to detect genuine trends (Type II errors; O’Brien et al., 2021). Ignoring serial dependence can compromise statistical inference. However, unnecessary or inappropriate prewhitening may reduce statistical power or distort trend estimates. Serial correlation should therefore be diagnosed and treated only when justified by the data (Yue and Wang, 2002).

What prewhiten() does

prewhiten() receives a raster time series and returns a transformed series together with cell-level diagnostics. Its default method is selective: only cells whose diagnostic indicates relevant serial autocorrelation are modified. All prewhitening procedures implemented in sptrends are designed to preserve the underlying trend signal. Classical prewhitening is deliberately excluded because filtering the raw series directly may attenuate the trend being analysed (Yue et al., 2002).

Basic workflow 

pw <- prewhiten(r, report = FALSE, verbose = FALSE)
pw
#> <Wang & Swail (2001) prewhitening result>
#> Prewhitened: 5987 of 15675 valid cells (38.2%)
#> Use summary() for diagnostic detail, or inspect $diagnostics directly.
summary(pw)
#> Valid cells: 15675
#> Prewhitened: 5987 (38.2%)
#> Mean rho among prewhitened cells: 0.4493
#> Median Durbin-Watson (all valid cells): 1.5306

The initial Durbin-Watson statistic (Durbin and Watson, 1950) provides the evidence used by the default selective procedure. Values near 2 indicate little first-order serial correlation. With the default diagnostic thresholds, values below 1.4 indicate relevant positive autocorrelation and values above 2.6 indicate relevant negative autocorrelation.

plot(pw)

Spatial diagnostics from selective trend-free prewhiteningSpatial diagnostics from selective trend-free prewhiteningSpatial diagnostics from selective trend-free prewhitening

The diagnostic maps show the initial Durbin-Watson statistic, the estimated lag-1 autocorrelation and the consequence of the selective decision. Only cells crossing the diagnostic criterion and completing the transformation successfully are marked as prewhitened.

Understanding the results

pw$series contains the transformed raster time series to be used in subsequent analyses when prewhitening is considered necessary. pw$diagnostics records the initial Durbin–Watson statistic, the estimated lag-1 autocorrelation coefficient, whether each cell was modified and any numerical-stability warning. Cells that are not modified retain their original observations.

Choosing the main options

sptrends provides four trend-preserving prewhitening procedures. They differ mainly in how autocorrelation is estimated and in whether the transformation is applied selectively or to every valid cell.

Method Main idea Typical use
TFPW_WS (Wang & Swail, 2001) Selective, Durbin-Watson-gated (1950) Recommended starting point
TFPW_Y (Yue et al., 2002) Trend-free treatment of every valid cell Uniform treatment
TFPW_Z (Zhang et al., 2000) Iterative treatment of every valid cell Ungated alternative
VCTFPW (Wang et al., 2015) Selective, variance-corrected treatment Published alternative

Classical prewhitening is deliberately excluded because filtering the raw series can attenuate the trend that the analysis seeks to detect (Yue et al., 2002).

Common mistakes

  • Do not assume that every time series requires prewhitening; examine the diagnostics first and apply treatment only when serial correlation and the analytical context justify it.
  • Do not interpret the default Durbin-Watson thresholds as a formal significance test; use dw_method = "test" when formal critical-value-based inference is required.
  • Do not treat prewhitening procedures as interchangeable; inspect the original and transformed series, compare their diagnostic maps and, when several methods are scientifically plausible, evaluate them as a sensitivity analysis before selecting one.
  • Do not combine prewhitening with modified Mann-Kendall (MMK) without a clear methodological justification; both approaches address temporal autocorrelation (see trend-test vignette).
  • Do not assume TFPW_Y preserves the input series length; it returns one fewer temporal observation, so verify the output dimensions before proceeding to later analytical stages.

Next steps

Continue to vignette("c-trend-test") and pass either the transformed series or the original series, according to the analytical decision.

Further details

See ?prewhiten for equations, statistical assumptions, diagnostics, method comparisons, limitations, external validation and references.

References