This vignette is the conceptual map of Stem: what each
function is for, and how they call one another. It is meant to be read
once, before diving into the manual pages.
The same diagram is installed with the package and can be opened from R:
Everything starts from a STEM_Model object, built by
STEM_Model() from four ingredients: the observation matrix
z (\(T \times d\)), the
stacked covariate matrix ((T \cdot d) \times r, blocks
ordered by location), the coordinates (\(d
\times 2\)) and a list phi of starting values. Two
internal constructors do the validation: STEM_Skeleton()
checks the parameter list and the loading matrix A,
STEM_Data() checks the data and derives the dimensions.
Two datasets ship with the package: pm10, the small
22-station example, and povalley, a real network of 36
background stations observed daily over 2019–2023.
STEM_Estimation() is the entry point. It hands the fit
to stem_em_fit() (R/em-fit.R), which runs the
algorithm chosen in STEM_control(): the iterations of the
EM or of the ECME algorithm, plain or accelerated by SQUAREM
(R/squarem.R). One iteration is an E-step,
stem_estep() (R/estep.R), followed by an
M-step, stem_mstep_em() (R/mstep-em.R) or
stem_mstep_ecme() (R/mstep-ecme.R), both built
from the conditional updates of R/mstep-updates.R. The
E-step rests on kalman() (R/kalman.R), the
state-space core of the model: the Kalman filter, the fixed-interval
smoother and the lag-one covariance smoother at the current parameters.
Underneath:
filtering() / filterstep() - the forward
Kalman recursions, which also accumulate the log-likelihood. They never
form the \(d \times d\) predictive
covariance \(Q_t\): the Woodbury
identity turns each step into \(p \times
p\) algebra against a Cholesky factor of \(\Sigma_\varepsilon\) that is computed once
per pass, and the log-density is evaluated directly rather than as the
logarithm of a density that would underflow on a large network;stem_obs_index(), stem_blocks_cache(),
stem_missing_blocks() - the missing-data bookkeeping: which
locations are observed at each time point, and the Gaussian conditioning
blocks that complete the sufficient statistics;smoothing() / smootherstep() /
smootherstep.uni() - the backward recursions producing the
smoothed latent states;stem_gls_mean() - the augmented Kalman filter of the
ECME step, which updates \(\beta\) and
\(m_0\) on the observed
likelihood;Q_function_addendo1() - the part of the expected
complete-data log-likelihood that depends on the spatial
parameters;d1_Q(), d2_Q(), d12_Q() and
the d1/d2_Sigmastar_*() derivatives - the gradient and
Hessian of the Newton-Raphson step that updates \(\theta\) and \(\log b = \log(\sigma^2_\varepsilon /
\sigma^2_\omega)\);Sigmastar.exp() and
changedimension_covariates() - matrix utilities.Closed-form M-steps update \(\beta\), \(\sigma^2_\omega\), \(G\), \(\Sigma_\eta\) and \(m_0\); the Newton-Raphson step handles \(\theta\) and \(\sigma^2_\varepsilon\).
How the iterations are organized. Four algorithms
are available, chosen by STEM_control(algorithm = ):
"SQUAREM" (the default), "EM",
"ECME" and "SQUAREM-ECME", which extrapolates
the ECME iterations. One iteration is a map, the parameters in
and the parameters out together with the log-likelihood at the
parameters in, and the map is run either plainly or by SQUAREM until the
stopping rule. STEM_Estimation() builds the data of the fit
once (distances, covariates, the index of the observed rows), runs the
algorithm, and calls kalman() once more at the estimates
for the log-likelihood and the smoothed states it reports, so that
estimates, log-likelihood and states always refer to the same
parameters.
The settings are those of STEM_control():
em_tol_par, em_tol_loglik,
em_maxit and em_stop for the pooled fit and
the final refits; alt_em_tol_par,
alt_em_tol_loglik and alt_em_maxit inside the
alternation of SC-STEM; nr_maxit for the Newton-Raphson
step. Each fit reports in convergence.par the number of
iterations, the Newton-Raphson iterations of each, the algorithm,
whether the stopping rule was met, the limits of \(\theta\) and whether the estimate sits at
one of them (convergence.par$theta.bound in
STEM_Estimation(), theta_bound in
SCSTEM_Estimation()). The algorithms, the stopping rule and
the limits of \(\theta\) are described
in Section A1 of the Computational and Algorithmic Supplement of the
SC-STEM paper; the experiments behind the defaults are in the vignette
“5. Numerical experiments on the estimation algorithms”.
STEM_Simulation() draws a new dataset from a
STEM_Model at the parameter values held in its skeleton. It
is the generator behind both bootstraps.STEM_Kriging() predicts at unobserved locations through
spatial.pred().STEM_Complete() returns \(E[z
\mid z_{obs}]\) at the observed locations, so a gap in the
response is filled by the model rather than left as NA; it
is the object a cross-validation exercise scores.
SCSTEM_Complete() does the same regime by regime and
reassembles the network.STEM_Bootstrap() repeats simulate, then
re-estimate through STEM_Bootstrap.fn(), giving the
sampling distribution of the parameters of a single pooled model.This layer sits on top of the previous ones: every cluster-wise fit
is an ordinary STEM_Model estimated by
STEM_Estimation().
SCSTEM_Estimation() alternates between estimating one
STEM model per regime and reallocating locations by an ICM sweep on a
Potts-penalized objective. Its helpers are
scstem_neighbors() (symmetrized k-nearest-neighbor graph),
scstem_init() and scstem_repair_partition()
(initialization), scstem_loglike_i() (the per-location
contribution), scstem_potts_pairs() and
scstem_pen_local() (the penalty),
scstem_swap_pass() (the size-preserving swap step) and
scstem_npar() (the parameter count behind the information
criteria).SCSTEM_Infocrit() loops
SCSTEM_Estimation() over a grid of \((K, \phi)\) and collects the criteria and
the admissibility flags.SCSTEM_Select() applies the two-step tuning rule to
that grid, and uses scstem_ari() to compare every partition
on it with the selected one.SCSTEM_Bootstrap() regenerates the data regime by
regime with STEM_Simulation() and re-runs the whole of
SCSTEM_Estimation() on every draw, so the uncertainty of
the partition is propagated.SCSTEM_BootInference() aligns the refits with
scstem_align_labels() and turns the draws into standard
errors, confidence intervals and between-regime tests.SCSTEM_CV() is an optional validation, separate from
the choice of \((K, \phi)\): it refits
one or more models on blocked folds (locations, time blocks, both, or
random cells), predicts the removed cells with
SCSTEM_Complete() and, at removed locations, with
scstem_predict_new(), which kriges within the regime of the
nearest retained location, and ranks the models by their predictive
error on the same folds.library(Stem)
data(povalley)
# 1. build the model object
mod <- STEM_Model(z = ..., covariates = ..., coordinates = ...,
phi = ..., A = ...)
# 2. the pooled benchmark
pooled <- STEM_Estimation(mod, distance = "geo")
# 3. explore the (K, phi) grid and select
ic <- SCSTEM_Infocrit(mod, K_grid = 1:4, distance = "geo")
sel <- SCSTEM_Select(ic)
# 4. quantify the uncertainty, partition included
boot <- SCSTEM_Bootstrap(sel$fit, B = 200, seed = 1)
inf <- SCSTEM_BootInference(boot)The edge list underlying the diagram is obtained by parsing
R/ and recording which package functions each function
calls; it is stored, in text form, in
The estimation code has been checked equation by equation against the three companion works of Fasso and Cameletti that define the STEM model - the heterogeneous-network paper of 2007, the technical report that introduced this package, and the 2010 synthesis whose Equations (12)-(21) the code implements. The result, including the one defect the check turned up, is documented in: