
Spatio-temporal expectation-maximization models, and their spatially-clustered extension.
Stem estimates hierarchical spatio-temporal models by
maximum likelihood, using Kalman filtering and smoothing inside an EM
algorithm. Version 2.0.0 adds the spatially-clustered STEM
(SC-STEM) family, in which regression coefficients, variance
components and latent dynamics are allowed to differ across spatial
regimes that are estimated from the data rather than imposed, and an
optional ridge, lasso or elastic-net penalty on the
regression coefficients for the collinear designs environmental data
usually produce.
STEM separates what is measured from what is real, and splits the real part into a regression surface, a dynamic component shared by the whole network, and small-scale spatial noise. With \(z_t\) the \(d\)-vector of observations at time \(t\):
z_t = X_t \beta + A y_t + e_t, \qquad y_t = G y_{t-1} + \eta_t, \qquad
\Sigma_e = \sigma^2_\varepsilon I + \sigma^2_\omega \exp(-\theta h),
with \(e_t \sim N(0, \Sigma_e)\), \(\eta_t \sim N(0, \Sigma_\eta)\) and \(y_0 \sim N(m_0, C_0)\): a linear Gaussian state-space model.
SC-STEM assigns each location to one of \(K\) spatial regimes, estimated from the data, and gives every regime its own STEM model and its own full parameter set \(\Psi_k = (\beta_k, \sigma^2_{\varepsilon k}, \sigma^2_{\omega k}, \theta_k, G_k, \Sigma_{\eta k}, m_{0k})\). Labels and parameters are estimated together by maximizing a Potts-penalized log-likelihood,
Q = \sum_{i=1}^{d} \ell_i(k_i) + \phi \, c \sum_{(i,j) \in E} I(k_i = k_j),
whose penalty rewards neighboring locations sharing a regime; \(K = 1\) is the pooled model. An optional ridge, lasso or elastic net acts on the regression coefficients of every regime.
Details: the models, the penalty on the coefficients.
Every model, the pooled one and each regime of SC-STEM, is estimated
by an EM-type algorithm on the state-space form: the Kalman
filter gives the log-likelihood, the Kalman
smoother the conditional moments of the latent states (the
E-step), and the M-step updates \(\beta\), \(\sigma^2_\omega\), \(G\), \(\Sigma_\eta\) and \(m_0\) in closed form and the spatial
parameters by Newton-Raphson. Four algorithms are available through
STEM_control(algorithm = ); they reach the same maximum and
differ in speed and in how close to it a given stopping rule leaves
them.
algorithm |
what it does | time | distance from the maximum at the stop |
|---|---|---|---|
"EM" |
the EM algorithm | 1 | 0.020 |
"ECME" |
as EM, with \(\beta\) and \(m_0\) updated on the observed likelihood (a GLS step computed by the Kalman filter) | 1.17 | 0.017 |
"SQUAREM" (default) |
EM iterations extrapolated along the direction of slowest convergence, with a monotonicity safeguard | 0.53 | 0.0027 |
"SQUAREM-ECME" |
the same on the ECME iterations | 0.62 | 0.0023 |
Time relative to EM on whole SC-STEM grids of the simulation study; distance: median shortfall of the log-likelihood on single fits, at the default tolerances. On the same data SQUAREM selected the same models as EM, and its estimates matched the exact maximum against the true values.
The iterations stop when the largest relative change of a parameter
or the absolute change of the log-likelihood falls below
1e-3; with SQUAREM they stop about 0.003 log-likelihood
units short of the maximum. The criterion on the log-likelihood is
absolute, because the differences that matter, between fits and in the
information criteria, are absolute. No jitter is added to the matrices
the iterations invert (regularization = 0): with one, they
converged to a point that is not the maximum.
Details, with formulas and the numerical experiments, including the limits within which the range of the spatial correlation is estimated: computational aspects and the Computational Supplement that ships with the package (see Documentation).
# install.packages("remotes")
remotes::install_github("PaoloMaranzano/Stem")Two functions are all a first session needs:
STEM_Model() to build the object, STEM_Fit()
to estimate it. Everything else is either a layer above (selection,
uncertainty, prediction) or an engine below.
%%{init: {"flowchart": {"wrappingWidth": 400}}}%%
flowchart TD
D["Data: series z, covariates,<br/>coordinates, starting values"] --> M["STEM_Model()"]
M --> F["STEM_Fit()"]
F -->|"K = 1"| E1["STEM_Estimation():<br/>pooled STEM"]
F -->|"K > 1"| E2["SCSTEM_Estimation():<br/>SC-STEM"]
M --> I["SCSTEM_Infocrit():<br/>fits on a grid of (K, φ)"]
I --> S["SCSTEM_Select():<br/>the two-step rule"]
S -->|"selected fit"| B["SCSTEM_Bootstrap():<br/>the partition re-estimated at every draw"]
B --> BI["SCSTEM_BootInference():<br/>standard errors, intervals, tests"]
E1 --> P1["STEM_Signal(), STEM_Complete(),<br/>STEM_Kriging(), STEM_Simulation(), STEM_Bootstrap()"]
E2 --> P2["SCSTEM_Signal(), SCSTEM_Complete(),<br/>SCSTEM_CV()"]
S --> P2
Build and fit
| Function | Purpose |
|---|---|
STEM_Model() |
build the model object from data, coordinates and starting values |
STEM_Fit() |
the entry point. Data and hyperparameters in, one
fitted model out. K chooses pooled or clustered,
(alpha, lambda) unpenalized or penalized |
Choosing the hyperparameters
| Function | Purpose |
|---|---|
SCSTEM_Infocrit() |
fit a grid of (K, phi) and return the criteria, with
the effective degrees of freedom when a penalty is in force |
SCSTEM_Select() |
the two-step rule: K by the modal criterion in a band,
phi by the smallest criterion at that K |
After the fit
| Function | Purpose |
|---|---|
STEM_Signal(), SCSTEM_Signal() |
fitted values: the conditional mean the model estimates, regression plus latent process, without the measurement error |
STEM_Complete(), SCSTEM_Complete() |
gap-filling: E[z given observed], which returns the
data wherever the data exist |
STEM_Kriging() |
spatial prediction at unobserved locations |
STEM_Simulation() |
simulate from a fitted or specified model |
STEM_Bootstrap() |
parametric bootstrap for the pooled model |
SCSTEM_Bootstrap() |
refit-with-clustering bootstrap: the partition is re-estimated on every draw |
SCSTEM_BootInference() |
standard errors, intervals and between-regime tests |
SCSTEM_CV() |
optional validation: blocked cross-validation (locations, times,
both, random cells) of one or more fits on the same folds, ranked by
predictive error; it does not choose (K, phi) |
The two engines, which STEM_Fit()
dispatches to. Call them directly only if you want to bypass the
dispatch; the arguments and the return values are the same either way,
and the historical API is preserved.
| Function | Purpose |
|---|---|
STEM_Estimation() |
the pooled fit: Kalman filter and smoother inside an EM-type algorithm (SQUAREM by default) |
SCSTEM_Estimation() |
the clustered fit: STEM_Estimation() per regime,
alternated with an ICM sweep on the labels |
Two datasets ship with the package: pm10 (22 stations,
366 days, the original example) and povalley (36 background
stations of the Po Valley, daily PM2.5 over 2019–2023, with altitude and
PM10 as covariates).
library(Stem)
data(povalley)
Tn <- 365L
Tfull <- nrow(povalley[["z"]]); d <- ncol(povalley[["z"]])
keep <- as.vector(outer(seq_len(Tn), (seq_len(d) - 1L) * Tfull, "+"))
phi <- list(beta = matrix(c(1.25, -0.00003, 0.64), 3, 1),
sigma2eps = 18.66, sigma2omega = 1e-06, theta = 2e-06,
G = matrix(0.59, 1, 1), Sigmaeta = matrix(4.25, 1, 1),
m0 = as.matrix(0), C0 = as.matrix(1))
mod <- STEM_Model(z = povalley[["z"]][seq_len(Tn), ],
covariates = povalley[["covariates"]][keep, ],
coordinates = povalley[["coords"]],
phi = phi, A = matrix(1, d, 1))
# one entry point for every estimator: K chooses pooled or clustered,
# lambda chooses penalized or not
fit <- STEM_Fit(mod, distance = "geo") # classical STEM
# explore the grid and let the two-step rule choose K and phi
ic <- SCSTEM_Infocrit(mod, K_grid = 1:4, distance = "geo")
sel <- SCSTEM_Select(ic)
sel
# the same clustered model with a ridge on the coefficients of every regime
fitr <- STEM_Fit(mod, K = sel[["K_selected"]], phi_penalty = sel[["phi_selected"]],
alpha = 0, lambda = 2, distance = "geo")
# uncertainty, with the partition re-estimated at every draw
boot <- SCSTEM_Bootstrap(sel[["fit"]], B = 200, seed = 1)
SCSTEM_BootInference(boot)Topic pages, in docs/:
| page | what it covers |
|---|---|
| The models | STEM and SC-STEM, their parameters, and what the model class covers and does not |
| Computational aspects | how STEM and SC-STEM are estimated, following Supplement A of the paper section by section: the Kalman filter and smoother, the EM, ECME and SQUAREM algorithms, the stopping rule, the limits of the range, missing responses, the start of a fit and the scale of the Potts penalty, degeneracy and the swap pass, the label update, the information criteria, the bootstrap and its tests, the Potts penalty on point-referenced data, the numerical experiments |
| Ridge, lasso and elastic net | the penalty on the regression coefficients |
| How the functions fit together | the map of the package |
| Theoretical references | the works behind the package |
Shipped with the package:
vignette("getting-started", package = "Stem") # the classical STEM workflow
vignette("SCSTEM", package = "Stem") # spatially-clustered STEM
vignette("function-map", package = "Stem") # map of the package
vignette("computational-notes", package = "Stem") # how the filter and the M-step are computed
vignette("estimation-experiments", package = "Stem") # the numerical experiments behind the defaults
# Supplement A of the paper: the estimation algorithms and their derivations
browseURL(system.file("extdata", "Stem-computational-supplement.pdf", package = "Stem"))
# a reading guide from each equation of the papers to the function that implements it
browseURL(system.file("extdata", "STEM_model_verification.html", package = "Stem"))The development record - the defects fixed in this release and the
reasoning behind the algorithmic changes - is kept in the repository,
outside the built package, at dev/code-changes-report.html.
The user-facing summary is NEWS.md,
and the dated development log is CHANGELOG.md.
The theory of the regularized estimator is at dev/regularization/stem-elastic-net.tex.
| Author | Role | ORCID | GitHub |
|---|---|---|---|
| Michela Cameletti | aut, cph | 0000-0002-6502-7779 | @michelacameletti |
| Paolo Maranzano | aut, cre, cph | 0000-0002-9228-2759 | @PaoloMaranzano |
Contributor: Francesco Caccia (ctb, cph), for earlier contributions
to the code and the preparation of the povalley data.
Licensed under GPL (>= 2).