---
title: "2. Spatially-clustered STEM models"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{2. Spatially-clustered STEM models}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.width = 6.5,
  fig.height = 4.5
)
```

```{r load}
library(Stem)
```

## Why cluster the regression

A STEM model imposes a single vector of regression coefficients on the whole
monitoring network. Over a large and heterogeneous domain this is often not
credible: the way concentrations respond to altitude or to a co-pollutant may
differ between the center of a plain and the areas nearer the reliefs.

The spatially-clustered STEM (SC-STEM) model relaxes the assumption. Each
location $i$ belongs to one of $K$ latent regimes, and conditionally on regime
$k$

$$
z_{it} = x_{it}' \beta_k + A_i y^{(k)}_t + e_{it}, \qquad
y^{(k)}_t = G_k y^{(k)}_{t-1} + \eta^{(k)}_t,
$$

so that coefficients, variance components *and* latent dynamics are all
regime-specific. Setting $K = 1$ returns the pooled fit.

Labels and parameters are estimated jointly by maximizing a Potts-penalized
log-likelihood

$$
Q = \sum_{i=1}^{d} \ell_i(k_i)
  + \phi\, c \sum_{(i,j) \in E} I(k_i = k_j),
$$

where $E$ is the edge set of the symmetrized $k$-nearest-neighbor graph, each
unordered pair counted once, and $\phi \ge 0$ controls how strongly neighboring
locations are pushed into the same regime.
This is the penalty of Sugasawa and Murakami (2021), in the form used for
spatially-clustered Fay--Herriot and spatial autoregressive models.

## The data

```{r data}
data(povalley)

Tn <- 150L                       # a five-month slice, to keep the vignette quick
Tfull <- nrow(povalley$z)
d <- ncol(povalley$z)
keep <- as.vector(outer(seq_len(Tn), (seq_len(d) - 1L) * Tfull, "+"))

z <- povalley$z[seq_len(Tn), ]
X <- povalley$covariates[keep, ]

dim(z)
head(round(povalley$coords, 3), 3)
```

```{r network, fig.alt = "Location of the 36 monitoring stations"}
plot(povalley$coords, pch = 19, cex = 0.9,
     xlab = "Longitude", ylab = "Latitude",
     main = "Po Valley background stations")
```

```{r model}
phi <- list(
  beta        = matrix(c(1.25, -0.00003, 0.64), 3, 1),
  sigma2eps   = 18.66,
  sigma2omega = 1e-06,
  theta       = 2e-06,
  G           = matrix(0.59, 1, 1),
  Sigmaeta    = matrix(4.25, 1, 1),
  m0          = as.matrix(0),
  C0          = as.matrix(1)
)

mod <- STEM_Model(z = z, covariates = X, coordinates = povalley$coords,
                  phi = phi, A = matrix(1, d, 1))
```

Coordinates are longitude and latitude, so every call below uses
`distance = "geo"`.

## Fitting an SC-STEM model

```{r fit}
fit <- SCSTEM_Estimation(mod, K = 3, phi_penalty = 0.05, distance = "geo",
                    precision = 0.05)
fit
```

The algorithm alternates two steps. Given the labels, a STEM model is estimated
in each regime. Given the parameters, the labels are updated by a sequential
**ICM** sweep (Besag, 1986): locations are visited in turn and each label
maximizes its own penalized contribution given the current labels of all the
others. Because the penalty is recomputed on the fly, the sweep cannot decrease
$Q$ *at fixed parameters* and cannot cycle - unlike the simultaneous update,
still available through `label_update = "simultaneous"`.

Where the alternation starts depends on the penalty. An unpenalized fit starts
from the k-means of how each location departs from the pooled model - the mean
of its residual, the slopes of the residual on the covariates, and the
autocorrelation and variance of what is left (`init_method = "departures"`, the
default). A penalized fit such as the one above fits $\phi = 0$ first and
starts from its solution.

The alternation as a whole, however, is not guaranteed to be monotone, and the
trace below may well show a dip. The reason is structural: $\ell_{ik}$ is a
conditional *pseudo*-likelihood, because the spatial covariance couples the
locations and the exact marginal likelihood does not factorize across them,
while the parameter step maximizes the exact within-cluster likelihood through
the EM algorithm. A parameter update can therefore lower $Q$ while raising the
exact likelihood. This is a property of the spatio-temporal specification, not
a numerical artifact. To keep the answer well defined, `SCSTEM_Estimation()` always
returns the best partition visited, and reports both objectives
(`best_objective`, `last_objective`).

```{r trace}
fit$obj_trace
```

```{r map, fig.alt = "Estimated spatial regimes"}
plot(povalley$coords, col = fit$group, pch = 19, cex = 1.2,
     xlab = "Longitude", ylab = "Latitude", main = "Estimated regimes")
legend("topright", legend = paste("regime", seq_len(3)),
       col = seq_len(3), pch = 19, bty = "n")
```

### Two safeguards worth knowing about

Within-cluster homogeneity is exactly what the assignment step pursues, so the
regime-specific variances shrink as locations are reallocated, and the regime
with the smallest residual variance tends to attract *everything*. This is the
clusterwise counterpart of the degenerate-likelihood problem of Gaussian
mixtures. Two defaults keep it under control:

* `enforce_min_size = TRUE` forbids a move that would take a regime below
  `min_cluster_size`;
* `swap_pass = TRUE` then restores mobility by exchanging the labels of two
  locations whenever that increases $Q$, which leaves the regime sizes - and
  hence feasibility - untouched.

### The scale of the penalty

Each location contributes $T$ observations to the likelihood, so a value of
$\phi$ borrowed from a cross-sectional clustered model would be far too weak.
The default `phi_scale = "auto"` rescales the penalty by the typical spread of
the location-wise log-likelihood contributions, so that $\phi \approx 1$ is the
point where agreeing with one's neighbors is worth about as much as picking
the best-fitting regime.

A fit with $\phi > 0$ starts from the partition of the unpenalized fit at the
same $K$, which `SCSTEM_Estimation()` computes first, and the spread is measured
there, on fitted regimes. The scale is then the same for every $\phi$ at that
$K$ and does not depend on the initialization, and it is large: it is of the
order of the whole gain of the right regime over the wrong ones, so the useful
values of $\phi$ are a few hundredths.

```{r scale}
c(multiplier = fit$phi_multiplier, effective_phi = fit$phi_effective)
```

## Choosing $K$ and $\phi$

`SCSTEM_Infocrit()` fits the whole grid and reports the criteria together with
an admissibility flag.

```{r grid}
ic <- SCSTEM_Infocrit(mod, K_grid = 1:3, phi_grid = c(0, 0.025, 0.05, 0.1, 0.2),
                      distance = "geo", precision = 0.05)
ic$table[, c("K", "phi", "K_eff", "admissible", "loglik", "BIC", "min_size")]
```

Two caveats make a naive "minimize the BIC" rule unsafe here. The partition is
itself optimized on the data, so the maximized likelihood keeps an optimism bias
that the parameter count does not fully correct; and the shrinking variance
components inflate the likelihood of the least regularized configurations. Both
effects are strongest at $\phi \approx 0$, where in-sample criteria overselect
$K$ - and where fits often collapse a regime altogether, which is what the
`admissible` column is for.

`SCSTEM_Select()` therefore applies a two-step rule: pick $K$ by BIC *inside a
moderate band* of penalties, then pick the $\phi$ with the smallest BIC at that
$K$ over the whole grid. Every penalized fit starts from the unpenalized one, so
a penalty is chosen only when it leads to a partition with a higher likelihood.
The default grid of `SCSTEM_Infocrit()` adds the strong values $0.5$ and $1$,
which compete in the second step only; this example uses the moderate values
to keep it fast.

```{r select}
sel <- SCSTEM_Select(ic, band = c(0.025, 0.2))
sel
```

The pooled $K = 1$ model competes in the first step: when no partition improves
on it, `SCSTEM_Select()` returns it and the second step does not apply.

## Uncertainty: the refit-with-clustering bootstrap

Conditioning on the estimated partition would understate the uncertainty,
because the partition is itself an estimate. `SCSTEM_Bootstrap()` therefore
generates data regime by regime from the fitted model and re-runs the *entire*
procedure - clustering included - on every draw.

```{r boot}
best <- sel$fit
boot <- SCSTEM_Bootstrap(best, B = 12, seed = 1)
boot
```

`SCSTEM_BootInference()` relocates each refit onto the original regimes by the
majority rule and summarizes the draws.

```{r inference}
inf <- SCSTEM_BootInference(boot)
inf
```

Percentile intervals for the regression coefficients:

```{r ci}
s <- inf$summary
s[grepl("beta", s$parameter),
  c("cluster", "parameter", "estimate", "se",
    grep("^perc_", names(s), value = TRUE))]
```

And the question that matters substantively - do the regimes really differ?

```{r tests}
inf$tests[grepl("beta", inf$tests$parameter), ]
```

The co-clustering matrix records how often two locations end up in the same
regime across the refits, and is a direct measure of how reproducible the
partition is:

```{r coclust}
round(inf$coclustering[1:6, 1:6], 2)
```

## References

Besag, J. (1986). On the statistical analysis of dirty pictures. *Journal of the
Royal Statistical Society, Series B*, 48, 259--302.

Cerqueti, R., Maranzano, P. and Mattera, R. (2025). Spatially-clustered spatial
autoregressive models with application to agricultural market concentration in
Europe. *Journal of Agricultural, Biological and Environmental Statistics*.

Sugasawa, S. and Murakami, D. (2021). Spatially clustered regression. *Spatial
Statistics*, 44, 100525.
