3. Map of the package

This vignette is the conceptual map of Stem: what each function is for, and how they call one another. It is meant to be read once, before diving into the manual pages.

Call graph of the Stem package, organized in four layers: data and model object, estimation engine, simulation and prediction, spatially-clustered STEM

The same diagram is installed with the package and can be opened from R:

system.file("extdata", "STEM_function_map.svg", package = "Stem")

The four layers

1. Data and the model object

Everything starts from a STEM_Model object, built by STEM_Model() from four ingredients: the observation matrix z (\(T \times d\)), the stacked covariate matrix ((T \cdot d) \times r, blocks ordered by location), the coordinates (\(d \times 2\)) and a list phi of starting values. Two internal constructors do the validation: STEM_Skeleton() checks the parameter list and the loading matrix A, STEM_Data() checks the data and derives the dimensions.

Two datasets ship with the package: pm10, the small 22-station example, and povalley, a real network of 36 background stations observed daily over 2019–2023.

2. The estimation engine

STEM_Estimation() is the entry point. It hands the fit to stem_em_fit() (R/em-fit.R), which runs the algorithm chosen in STEM_control(): the iterations of the EM or of the ECME algorithm, plain or accelerated by SQUAREM (R/squarem.R). One iteration is an E-step, stem_estep() (R/estep.R), followed by an M-step, stem_mstep_em() (R/mstep-em.R) or stem_mstep_ecme() (R/mstep-ecme.R), both built from the conditional updates of R/mstep-updates.R. The E-step rests on kalman() (R/kalman.R), the state-space core of the model: the Kalman filter, the fixed-interval smoother and the lag-one covariance smoother at the current parameters. Underneath:

Closed-form M-steps update \(\beta\), \(\sigma^2_\omega\), \(G\), \(\Sigma_\eta\) and \(m_0\); the Newton-Raphson step handles \(\theta\) and \(\sigma^2_\varepsilon\).

How the iterations are organized. Four algorithms are available, chosen by STEM_control(algorithm = ): "SQUAREM" (the default), "EM", "ECME" and "SQUAREM-ECME", which extrapolates the ECME iterations. One iteration is a map, the parameters in and the parameters out together with the log-likelihood at the parameters in, and the map is run either plainly or by SQUAREM until the stopping rule. STEM_Estimation() builds the data of the fit once (distances, covariates, the index of the observed rows), runs the algorithm, and calls kalman() once more at the estimates for the log-likelihood and the smoothed states it reports, so that estimates, log-likelihood and states always refer to the same parameters.

The settings are those of STEM_control(): em_tol_par, em_tol_loglik, em_maxit and em_stop for the pooled fit and the final refits; alt_em_tol_par, alt_em_tol_loglik and alt_em_maxit inside the alternation of SC-STEM; nr_maxit for the Newton-Raphson step. Each fit reports in convergence.par the number of iterations, the Newton-Raphson iterations of each, the algorithm, whether the stopping rule was met, the limits of \(\theta\) and whether the estimate sits at one of them (convergence.par$theta.bound in STEM_Estimation(), theta_bound in SCSTEM_Estimation()). The algorithms, the stopping rule and the limits of \(\theta\) are described in Section A1 of the Computational and Algorithmic Supplement of the SC-STEM paper; the experiments behind the defaults are in the vignette “5. Numerical experiments on the estimation algorithms”.

3. Simulation, prediction and uncertainty

4. Spatially-clustered STEM

This layer sits on top of the previous ones: every cluster-wise fit is an ordinary STEM_Model estimated by STEM_Estimation().

A typical session

library(Stem)
data(povalley)

# 1. build the model object
mod <- STEM_Model(z = ..., covariates = ..., coordinates = ...,
                  phi = ..., A = ...)

# 2. the pooled benchmark
pooled <- STEM_Estimation(mod, distance = "geo")

# 3. explore the (K, phi) grid and select
ic  <- SCSTEM_Infocrit(mod, K_grid = 1:4, distance = "geo")
sel <- SCSTEM_Select(ic)

# 4. quantify the uncertainty, partition included
boot <- SCSTEM_Bootstrap(sel$fit, B = 200, seed = 1)
inf  <- SCSTEM_BootInference(boot)

Regenerating the map

The edge list underlying the diagram is obtained by parsing R/ and recording which package functions each function calls; it is stored, in text form, in

system.file("extdata", "STEM_function_map.md", package = "Stem")

Is the implementation faithful to the papers?

The estimation code has been checked equation by equation against the three companion works of Fasso and Cameletti that define the STEM model - the heterogeneous-network paper of 2007, the technical report that introduced this package, and the 2010 synthesis whose Equations (12)-(21) the code implements. The result, including the one defect the check turned up, is documented in:

browseURL(system.file("extdata", "STEM_model_verification.html", package = "Stem"))