---
title: "3. Map of the package"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{3. Map of the package}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
```

This vignette is the conceptual map of `Stem`: what each function is for, and
how they call one another. It is meant to be read once, before diving into the
manual pages.

```{r diagram, echo = FALSE, out.width = "100%", fig.alt = "Call graph of the Stem package, organized in four layers: data and model object, estimation engine, simulation and prediction, spatially-clustered STEM"}
knitr::include_graphics("function-map.svg")
```

The same diagram is installed with the package and can be opened from R:

```{r file, eval = FALSE}
system.file("extdata", "STEM_function_map.svg", package = "Stem")
```

## The four layers

### 1. Data and the model object

Everything starts from a `STEM_Model` object, built by `STEM_Model()` from four
ingredients: the observation matrix `z` ($T \times d$), the stacked covariate
matrix (`(T \cdot d) \times r`, blocks ordered by location), the coordinates
($d \times 2$) and a list `phi` of starting values. Two internal constructors do
the validation: `STEM_Skeleton()` checks the parameter list and the loading
matrix `A`, `STEM_Data()` checks the data and derives the dimensions.

Two datasets ship with the package: `pm10`, the small 22-station example, and
`povalley`, a real network of 36 background stations observed daily over
2019--2023.

### 2. The estimation engine

`STEM_Estimation()` is the entry point. It hands the fit to `stem_em_fit()`
(`R/em-fit.R`), which runs the algorithm chosen in `STEM_control()`: the
iterations of the EM or of the ECME algorithm, plain or accelerated by SQUAREM
(`R/squarem.R`). One iteration is an E-step, `stem_estep()` (`R/estep.R`),
followed by an M-step, `stem_mstep_em()` (`R/mstep-em.R`) or
`stem_mstep_ecme()` (`R/mstep-ecme.R`), both built from the conditional updates
of `R/mstep-updates.R`. The E-step rests on `kalman()` (`R/kalman.R`), the
state-space core of the model: the Kalman filter, the fixed-interval smoother
and the lag-one covariance smoother at the current parameters. Underneath:

* `filtering()` / `filterstep()` - the forward Kalman recursions, which also
  accumulate the log-likelihood. They never form the $d \times d$ predictive
  covariance $Q_t$: the Woodbury identity turns each step into $p \times p$
  algebra against a Cholesky factor of $\Sigma_\varepsilon$ that is computed
  once per pass, and the log-density is evaluated directly rather than as the
  logarithm of a density that would underflow on a large network;
* `stem_obs_index()`, `stem_blocks_cache()`, `stem_missing_blocks()` - the
  missing-data bookkeeping: which locations are observed at each time point,
  and the Gaussian conditioning blocks that complete the sufficient statistics;
* `smoothing()` / `smootherstep()` / `smootherstep.uni()` - the backward
  recursions producing the smoothed latent states;
* `stem_gls_mean()` - the augmented Kalman filter of the ECME step, which
  updates $\beta$ and $m_0$ on the observed likelihood;
* `Q_function_addendo1()` - the part of the expected complete-data
  log-likelihood that depends on the spatial parameters;
* `d1_Q()`, `d2_Q()`, `d12_Q()` and the `d1/d2_Sigmastar_*()` derivatives - the
  gradient and Hessian of the Newton-Raphson step that updates $\theta$ and
  $\log b = \log(\sigma^2_\varepsilon / \sigma^2_\omega)$;
* `Sigmastar.exp()` and `changedimension_covariates()` - matrix utilities.

Closed-form M-steps update $\beta$, $\sigma^2_\omega$, $G$, $\Sigma_\eta$ and
$m_0$; the Newton-Raphson step handles $\theta$ and $\sigma^2_\varepsilon$.

**How the iterations are organized.** Four algorithms are available, chosen by
`STEM_control(algorithm = )`: `"SQUAREM"` (the default), `"EM"`, `"ECME"` and
`"SQUAREM-ECME"`, which extrapolates the ECME iterations. One iteration is a
*map*, the parameters in and the parameters out together with the
log-likelihood at the parameters in, and the map is run either plainly or by
SQUAREM until the stopping rule. `STEM_Estimation()` builds the data of the fit
once (distances, covariates, the index of the observed rows), runs the
algorithm, and calls `kalman()` once more at the estimates for the
log-likelihood and the smoothed states it reports, so that estimates,
log-likelihood and states always refer to the same parameters.

The settings are those of `STEM_control()`: `em_tol_par`, `em_tol_loglik`,
`em_maxit` and `em_stop` for the pooled fit and the final refits;
`alt_em_tol_par`, `alt_em_tol_loglik` and `alt_em_maxit` inside the
alternation of SC-STEM; `nr_maxit` for the Newton-Raphson step. Each fit
reports in `convergence.par` the number of iterations, the Newton-Raphson
iterations of each, the algorithm, whether the stopping rule was met, the
limits of $\theta$ and whether the estimate sits at one of them
(`convergence.par$theta.bound` in `STEM_Estimation()`, `theta_bound` in
`SCSTEM_Estimation()`). The algorithms, the stopping rule and the limits of
$\theta$ are described in Section A1 of the Computational and Algorithmic
Supplement of the SC-STEM paper; the experiments behind the defaults are in the
vignette "5. Numerical experiments on the estimation algorithms".

### 3. Simulation, prediction and uncertainty

* `STEM_Simulation()` draws a new dataset from a `STEM_Model` at the parameter
  values held in its skeleton. It is the generator behind both bootstraps.
* `STEM_Kriging()` predicts at unobserved locations through `spatial.pred()`.
* `STEM_Complete()` returns $E[z \mid z_{obs}]$ at the observed locations, so a
  gap in the response is filled by the model rather than left as `NA`; it is
  the object a cross-validation exercise scores. `SCSTEM_Complete()` does the
  same regime by regime and reassembles the network.
* `STEM_Bootstrap()` repeats *simulate, then re-estimate* through
  `STEM_Bootstrap.fn()`, giving the sampling distribution of the parameters of a
  single pooled model.

### 4. Spatially-clustered STEM

This layer sits on top of the previous ones: every cluster-wise fit is an
ordinary `STEM_Model` estimated by `STEM_Estimation()`.

* `SCSTEM_Estimation()` alternates between estimating one STEM model per regime and
  reallocating locations by an ICM sweep on a Potts-penalized objective. Its
  helpers are `scstem_neighbors()` (symmetrized k-nearest-neighbor graph),
  `scstem_init()` and `scstem_repair_partition()` (initialization),
  `scstem_loglike_i()` (the per-location contribution),
  `scstem_potts_pairs()` and `scstem_pen_local()` (the penalty),
  `scstem_swap_pass()` (the size-preserving swap step) and `scstem_npar()`
  (the parameter count behind the information criteria).
* `SCSTEM_Infocrit()` loops `SCSTEM_Estimation()` over a grid of $(K, \phi)$ and
  collects the criteria and the admissibility flags.
* `SCSTEM_Select()` applies the two-step tuning rule to that grid, and uses
  `scstem_ari()` to compare every partition on it with the selected one.
* `SCSTEM_Bootstrap()` regenerates the data regime by regime with
  `STEM_Simulation()` and re-runs the whole of `SCSTEM_Estimation()` on every draw,
  so the uncertainty of the partition is propagated.
* `SCSTEM_BootInference()` aligns the refits with `scstem_align_labels()` and
  turns the draws into standard errors, confidence intervals and between-regime
  tests.
* `SCSTEM_CV()` is an optional validation, separate from the choice of
  $(K, \phi)$: it refits one or more models on blocked folds (locations, time
  blocks, both, or random cells), predicts the removed cells with
  `SCSTEM_Complete()` and, at removed locations, with `scstem_predict_new()`,
  which kriges within the regime of the nearest retained location, and ranks
  the models by their predictive error on the same folds.

## A typical session

```{r session, eval = FALSE}
library(Stem)
data(povalley)

# 1. build the model object
mod <- STEM_Model(z = ..., covariates = ..., coordinates = ...,
                  phi = ..., A = ...)

# 2. the pooled benchmark
pooled <- STEM_Estimation(mod, distance = "geo")

# 3. explore the (K, phi) grid and select
ic  <- SCSTEM_Infocrit(mod, K_grid = 1:4, distance = "geo")
sel <- SCSTEM_Select(ic)

# 4. quantify the uncertainty, partition included
boot <- SCSTEM_Bootstrap(sel$fit, B = 200, seed = 1)
inf  <- SCSTEM_BootInference(boot)
```

## Regenerating the map

The edge list underlying the diagram is obtained by parsing `R/` and recording
which package functions each function calls; it is stored, in text form, in

```{r edges, eval = FALSE}
system.file("extdata", "STEM_function_map.md", package = "Stem")
```

## Is the implementation faithful to the papers?

The estimation code has been checked equation by equation against the three
companion works of Fasso and Cameletti that define the STEM model - the
heterogeneous-network paper of 2007, the technical report that introduced this
package, and the 2010 synthesis whose Equations (12)-(21) the code implements.
The result, including the one defect the check turned up, is documented in:

```{r verification, eval = FALSE}
browseURL(system.file("extdata", "STEM_model_verification.html", package = "Stem"))
```
