Package {TextAnalysisR}

Title:	A Text Mining Workflow Tool
Version:	0.1.4
Description:	Provides a text mining and natural language processing workflow for documents. Includes preprocessing via 'quanteda', lexical analysis (term frequency-inverse document frequency, log-odds ratios, lexical diversity) via 'tidytext', topic modeling via 'stm' and the 'BERTopic' approach, semantic similarity and document clustering on transformer representations, an interactive 'Shiny' interface with 'ggplot2' visualization, optional 'spaCy' preprocessing, and local 'sentence-transformers' or web-based ('OpenAI', 'Gemini') model providers for retrieval-augmented generation, as described in Shin et al. (2026) <doi:10.1177/07319487251412879>.
License:	GPL (≥ 3)
Language:	en-US
Encoding:	UTF-8
LazyData:	true
LazyDataCompression:	xz
Imports:	dplyr, DT, ggplot2, htmltools, htmlwidgets, igraph, magrittr, Matrix, methods, quanteda, quanteda.textstats, RColorBrewer, rlang, scales, shiny, tibble, tidyr, tidytext, widyr, withr
Suggests:	aricode, tidylo, broom, patchwork, cluster, clusterCrit, colourpicker, covr, dbscan, digest, httr, jsonlite, knitr, markdown, MASS, moments, numform, officer, openxlsx, pdftools, progress, proxy, pscl, purrr, readxl, reticulate (≥ 1.28), rmarkdown, roxygen2, Rtsne, shinyBS, shinybusy, shinyjs, spelling, stm, stopwords, stringr, syuzhet, testthat (≥ 3.0.0), textdata, umap, plotly
Depends:	R (≥ 4.0)
SystemRequirements:	Python (>= 3.8) with spaCy (>= 3.5) for lemmatization and sentence-transformers (>= 2.2) for embeddings (optional)
URL:	https://mshin77.github.io/TextAnalysisR/, https://github.com/mshin77/TextAnalysisR
BugReports:	https://github.com/mshin77/TextAnalysisR/issues
VignetteBuilder:	knitr
Config/testthat/edition:	3
Config/roxygen2/version:	8.0.0
NeedsCompilation:	no
Packaged:	2026-07-17 17:44:05 UTC; shinm
Author:	Mikyung Shin [aut, cre]
Maintainer:	Mikyung Shin <shin.mikyung@gmail.com>
Repository:	CRAN
Date/Publication:	2026-07-27 14:40:13 UTC

TextAnalysisR: A Text Mining Workflow Tool

Description

Provides a text mining and natural language processing workflow for documents. Includes preprocessing via 'quanteda', lexical analysis (term frequency-inverse document frequency, log-odds ratios, lexical diversity) via 'tidytext', topic modeling via 'stm' and the 'BERTopic' approach, semantic similarity and document clustering on transformer representations, an interactive 'Shiny' interface with 'ggplot2' visualization, optional 'spaCy' preprocessing, and local 'sentence-transformers' or web-based ('OpenAI', 'Gemini') model providers for retrieval-augmented generation, as described in Shin et al. (2026) doi:10.1177/07319487251412879.

Author(s)

Maintainer: Mikyung Shin shin.mikyung@gmail.com (ORCID)

Authors:

• Mikyung Shin shin.mikyung@gmail.com (ORCID)

See Also

Useful links:

• https://mshin77.github.io/TextAnalysisR/

• https://github.com/mshin77/TextAnalysisR

• Report bugs at https://github.com/mshin77/TextAnalysisR/issues

Pipe operator

Description

See magrittr::%>% for details.

Usage

lhs %>% rhs

Arguments

Value

The result of calling rhs(lhs).

Special education technology bibliographic data

Description

Contains bibliographic data of journal articles and dissertations on the use of technology in teaching mathematics to students with disabilities.

Usage

SpecialEduTech

Format

An object of class tbl_df (inherits from tbl, data.frame) with 490 rows and 6 columns.

Source

Bibliographic records compiled and coded by the author for Shin, M., Ok, M. W., Choo, S., Hossain, G., Bryant, D. P., & Kang, E. (2023). A content analysis of research on technology use for teaching mathematics to students with disabilities: Word networks and topic modeling. International Journal of STEM Education, 10, Article 23. doi:10.1186/s40594-023-00414-x. Shared as open data at https://github.com/mshin77/math-tech-sped.

Examples

SpecialEduTech

Acronym List

Description

A dataset containing common acronyms used in text processing

Format

A character vector of acronyms

Analyze Document Clustering

Description

Complete document clustering analysis with dimensionality reduction and optional clustering

Usage

analyze_document_clustering(
  feature_matrix,
  method = "UMAP",
  clustering_method = "none",
  ...
)

Arguments

Value

List containing coordinates, clusters, method info, and quality metrics

Analyze Semantic Evolution

Description

Analyzes semantic evolution patterns in temporal results

Usage

analyze_semantic_evolution(temporal_results, verbose = FALSE, ...)

Arguments

Value

List containing evolution analysis

Analyze Text Sentiment

Description

Performs sentiment analysis on text data using the syuzhet package. Classification uses the per-token average score with a +/- 0.05 neutral band, so document length does not drive polarity. Lexicon methods score words in isolation; negation, irony, and idioms are not handled. For negation-sensitive text, see analyze_sentiment_llm().

Usage

analyze_sentiment(texts, method = "syuzhet", doc_ids = NULL)

Arguments

Value

A data frame with columns:

document

Document identifier

text

Original text

sentiment_score

Raw summed sentiment score (length-dependent)

n_tokens

Number of whitespace-delimited tokens

avg_sentiment

sentiment_score / n_tokens

sentiment

Classification: "positive", "negative", or "neutral"

Examples

abstracts <- TextAnalysisR::SpecialEduTech$abstract[1:10]
sentiment_results <- analyze_sentiment(abstracts)
print(sentiment_results)

LLM-based Sentiment Analysis

Description

Analyzes sentiment using Large Language Models (OpenAI or Gemini). Provides nuanced sentiment understanding including sarcasm detection, mixed emotions, and contextual interpretation that lexicon-based methods miss.

Requests use temperature 0, which makes outputs near-deterministic but not fully reproducible across runs or model versions; record the returned model_used field when reporting results.

Usage

analyze_sentiment_llm(
  texts,
  doc_names = NULL,
  provider = c("openai", "gemini"),
  model = NULL,
  api_key = NULL,
  batch_size = 5,
  include_explanation = FALSE,
  verbose = TRUE
)

Arguments

Details

LLM-based sentiment analysis offers several advantages over lexicon methods:

• Understands context and nuance

• Detects sarcasm and irony

• Handles mixed emotions

• Works across domains without retraining

Value

A list containing:

document_sentiment

Data frame with document-level sentiment scores

summary_stats

Summary statistics of the analysis

model_used

Model name used for analysis

provider

AI provider used

Examples

if (interactive()) {
  abstracts <- TextAnalysisR::SpecialEduTech$abstract[1:5]

  sentiment_openai <- analyze_sentiment_llm(abstracts, provider = "openai")

  sentiment_gemini <- analyze_sentiment_llm(abstracts, provider = "gemini",
                                             include_explanation = TRUE)
}

Analyze Similarity Gaps Between Categories

Description

Identifies unique items, missing content, and cross-category learning opportunities based on similarity thresholds. Useful for gap analysis in policy documents, topic comparisons, or any cross-category similarity study.

Usage

analyze_similarity_gaps(
  similarity_data,
  ref_var = "ref_id",
  other_var = "other_id",
  similarity_var = "similarity",
  category_var = "other_category",
  ref_label_var = NULL,
  other_label_var = NULL,
  unique_threshold = 0.6,
  cross_policy_min = 0.6,
  cross_policy_max = 0.8
)

Arguments

Value

A list containing:

Examples

articles <- TextAnalysisR::SpecialEduTech[1:6, ]
articles$display_name <- paste0("d", seq_len(nrow(articles)))
term_matrix <- as.matrix(quanteda::dfm(quanteda::tokens(articles$abstract)))
normalized_matrix <- term_matrix / sqrt(rowSums(term_matrix ^ 2))
similarity_matrix <- normalized_matrix %*% t(normalized_matrix)
dimnames(similarity_matrix) <- list(articles$display_name, articles$display_name)
cross_similarities <- extract_cross_category_similarities(
  similarity_matrix  = similarity_matrix,
  docs_data          = articles,
  reference_category = "thesis",
  compare_categories = "journal_article",
  category_var       = "reference_type",
  id_var             = "display_name"
)
gap_analysis <- analyze_similarity_gaps(
  similarity_data = cross_similarities,
  ref_var = "ref_id",
  other_var = "other_id",
  similarity_var = "similarity",
  category_var = "other_category",
  unique_threshold = 0.6
)
print(gap_analysis$summary_stats)

Apply Standard Plotly Layout

Description

Applies consistent layout styling to plotly plots following TextAnalysisR design standards. This ensures all plots have uniform fonts, colors, margins, and interactive features.

Usage

apply_standard_plotly_layout(
  plot,
  title = NULL,
  xaxis_title = NULL,
  yaxis_title = NULL,
  margin = list(t = 60, b = 80, l = 80, r = 40),
  show_legend = FALSE
)

Arguments

Details

Design standards applied:

• Title: 14px Roboto, #0c1f4a

• Axis titles: 13px Roboto, #0c1f4a

• Axis tick labels: 12px Roboto, #3B3B3B

Value

A plotly plot object with standardized layout

Assess Embedding Topic Model Stability

Description

Evaluates the stability of embedding-based topic modeling by running multiple models with different random seeds and comparing their results. High stability (high ARI, consistent keywords) indicates stable topic structure in the data.

Usage

assess_embedding_stability(
  texts,
  n_runs = 5,
  embedding_model = "all-MiniLM-L6-v2",
  select_best = TRUE,
  base_seed = 123,
  verbose = TRUE,
  ...
)

Arguments

Details

Stability is assessed via:

• Adjusted Rand Index (ARI): Measures agreement in topic assignments across runs

• Keyword Jaccard similarity: Measures overlap in top keywords per topic

• Quality variance: Variance in silhouette scores across runs

Value

A list containing:

• stability_metrics: List with mean_ari, sd_ari, mean_jaccard, quality_variance

• best_model: Best model by silhouette score (if select_best = TRUE)

• all_models: List of all fitted models

• is_stable: Logical, TRUE if mean ARI >= 0.6 (considered stable)

• recommendation: Text recommendation based on stability

Examples

if (interactive()) {
  texts <- c("Machine learning for image recognition",
             "Deep learning neural networks",
             "Natural language processing models")

  stability <- assess_embedding_stability(
    texts = texts,
    n_runs = 3,
    verbose = TRUE
  )

  # Check if results are stable
  stability$is_stable
  stability$stability_metrics$mean_ari

  # Use the best model
  best_model <- stability$best_model
}

Auto-tune BERTopic Hyperparameters

Description

Automatically searches for optimal hyperparameters for embedding-based topic modeling. Evaluates multiple configurations of UMAP and HDBSCAN parameters and returns the best model based on the specified metric. Embeddings are generated once and reused across all configurations for efficiency.

Usage

auto_tune_embedding_topics(
  texts,
  embeddings = NULL,
  embedding_model = "all-MiniLM-L6-v2",
  n_trials = 12,
  metric = "silhouette",
  seed = 123,
  verbose = TRUE
)

Arguments

Details

The function searches over these parameters:

• n_neighbors: UMAP neighborhood size (5, 10, 15, 25)

• min_cluster_size: HDBSCAN minimum cluster size (3, 5, 10)

• cluster_selection_method: "eom" (broader) or "leaf" (finer-grained)

Value

A list containing:

• best_config: Data frame with the optimal hyperparameter configuration

• best_model: The topic model fitted with optimal parameters

• all_results: List of all evaluated configurations with metrics

• n_trials_completed: Number of configurations successfully evaluated

Examples

if (interactive()) {
  texts <- c("Machine learning for image recognition",
             "Deep learning neural networks",
             "Natural language processing models",
             "Computer vision applications")

  tuning_result <- auto_tune_embedding_topics(
    texts = texts,
    n_trials = 6,
    metric = "silhouette",
    verbose = TRUE
  )

  # View best configuration
  tuning_result$best_config

  # Use the best model
  best_model <- tuning_result$best_model
}

Calculate Assignment Consistency

Description

Calculates consistency between two sets of assignments

Usage

calculate_assignment_consistency(assignments1, assignments2, ...)

Arguments

Value

List containing consistency metrics

Calculate Clustering Quality Metrics

Description

Calculates common clustering evaluation metrics including Silhouette Score, Davies-Bouldin Index, and Calinski-Harabasz Index.

Usage

calculate_clustering_metrics(
  clusters,
  data_matrix,
  dist_matrix = NULL,
  metrics = "all"
)

Arguments

Details

• Silhouette Score: Measures how similar an object is to its own cluster compared to other clusters. Range: -1 to 1, higher is better.

• Davies-Bouldin Index: Average similarity between each cluster and its most similar cluster. Lower values indicate better clustering.

• Calinski-Harabasz Index: Ratio of between-cluster to within-cluster variance. Higher values indicate better-defined clusters.

Value

A named list containing:

silhouette

Silhouette score (-1 to 1, higher is better)

davies_bouldin

Davies-Bouldin index (lower is better)

calinski_harabasz

Calinski-Harabasz index (higher is better)

n_clusters

Number of clusters

cluster_sizes

Table of cluster sizes

Examples

abstracts <- TextAnalysisR::SpecialEduTech$abstract[1:20]
term_matrix <- as.matrix(quanteda::dfm(quanteda::tokens(abstracts)))
kmeans_result <- stats::kmeans(term_matrix, centers = 2)
metrics <- calculate_clustering_metrics(kmeans_result$cluster, term_matrix)
print(metrics)

Validate Semantic Coherence

Description

Validates semantic coherence of topic assignments.

Usage

calculate_coherence(embeddings, topic_assignments)

Arguments

Value

Coherence metrics.

Web Accessibility Utility Functions

Description

Functions for ensuring WCAG 2.1 Level AA compliance in the Shiny application

Calculates the contrast ratio between two colors according to WCAG 2.1 standards using the relative luminance formula from W3C guidelines. Used to verify text/background color combinations meet accessibility requirements.

Usage

calculate_contrast_ratio(foreground, background)

Arguments

Value

Numeric contrast ratio (1-21)

WCAG 2.1 Level AA Compliance

This package follows Web Content Accessibility Guidelines (WCAG) 2.1 Level AA:

• 1.1.1 Non-text Content (Level A): Alt text for images and visualizations

• 1.4.3 Contrast Minimum (Level AA): 4.5:1 ratio for normal text, 3:1 for large text/UI

• 2.1.1 Keyboard (Level A): Full keyboard navigation support

• 2.4.1 Bypass Blocks (Level A): Skip navigation links

• 3.1.1 Language of Page (Level A): Page language identification

• 4.1.2 Name, Role, Value (Level A): ARIA labels and roles Calculate Color Contrast Ratio

WCAG Requirements

• Normal text: Minimum 4.5:1 (Level AA)

• Large text (18pt+ or 14pt+ bold): Minimum 3:1 (Level AA)

• UI components and graphics: Minimum 3:1 (Level AA)

Calculate Cosine Similarity Matrix

Description

Calculates the cosine similarity between all pairs of rows in a matrix.

Usage

calculate_cosine_similarity(matrix_data)

Arguments

Value

A square similarity matrix with values between -1 and 1

Calculate Cross-Matrix Cosine Similarity

Description

Calculates cosine similarity between two different embedding matrices, useful for comparing documents/topics across different categories or groups.

Usage

calculate_cross_similarity(
  embeddings1,
  embeddings2,
  labels1 = NULL,
  labels2 = NULL,
  normalize = TRUE
)

Arguments

Value

A list containing:

Examples

abstracts <- TextAnalysisR::SpecialEduTech$abstract[1:6]
term_matrix <- as.matrix(quanteda::dfm(quanteda::tokens(abstracts)))
similarity_result <- calculate_cross_similarity(
  term_matrix[1:3, ], term_matrix[4:6, ],
  labels1 = paste("Doc", 1:3),
  labels2 = paste("Doc", 4:6)
)
similarity_result$similarity_matrix

Calculate Dispersion Metrics

Description

Computes quantitative dispersion metrics for terms, measuring how evenly distributed they are across the corpus.

Usage

calculate_dispersion_metrics(tokens_object, terms)

Arguments

Value

Data frame with columns:

• term: The search term

• frequency: Total occurrences

• doc_count: Number of documents containing term

• doc_ratio: Proportion of documents containing term

• juilland_d: Juilland's D dispersion (0-1, higher = more even), computed on within-document proportions

• rosengren_s: Rosengren's S dispersion adjusted for document sizes (1/n_docs to 1, higher = more even)

Calculate Document Similarity

Description

Calculates similarity between documents using traditional NLP methods or modern embedding-based approaches. Metrics are automatically computed unless disabled.

Usage

calculate_document_similarity(
  texts,
  document_feature_type = "words",
  semantic_ngram_range = 2,
  similarity_method = "cosine",
  use_embeddings = FALSE,
  embedding_model = "all-MiniLM-L6-v2",
  calculate_metrics = TRUE,
  verbose = TRUE
)

Arguments

Value

A list containing:

similarity_matrix

N x N similarity matrix

feature_matrix

Document feature matrix used for calculation

method_info

Information about the method used

metrics

Similarity metrics (if calculate_metrics = TRUE)

execution_time

Time taken for analysis

Examples

  data(SpecialEduTech)
  texts <- SpecialEduTech$abstract[1:5]

  result <- calculate_document_similarity(
    texts = texts,
    document_feature_type = "words",
    similarity_method = "cosine"
  )

  print(result$similarity_matrix)
  print(result$metrics)

Calculate Keyword Stability

Description

Calculates stability between the topic keyword sets of two independently fitted models. Topic numbering is arbitrary across fits, so topics are matched one-to-one by greedy assignment on the pairwise keyword Jaccard matrix before averaging matched-pair similarity.

Usage

calculate_keyword_stability(keywords1, keywords2)

Arguments

Value

Mean Jaccard similarity of matched topic pairs (0-1).

Calculate Lexical Dispersion

Description

Computes lexical dispersion data for specified terms across a corpus. Shows where terms appear within each document, useful for understanding term distribution patterns.

Usage

calculate_lexical_dispersion(
  tokens_object,
  terms,
  scale = c("relative", "absolute")
)

Arguments

Value

Data frame with columns:

• doc_id: Document identifier

• term: The search term

• position: Position in document (relative or absolute)

• doc_length: Total tokens in document

Examples

tokens <- quanteda::tokens(TextAnalysisR::SpecialEduTech$abstract[1:5])
dispersion <- calculate_lexical_dispersion(tokens, c("learning", "instruction"))

Calculate Log Odds Ratio Between Categories

Description

Compares word usage between categories using the log-odds ratio with a uniform Dirichlet prior, ranked by z-score. Identifies words distinctively used in one category. For the informative-prior weighted log-odds, see calculate_weighted_log_odds().

Usage

calculate_log_odds_ratio(
  dfm_object,
  group_var,
  comparison_mode = c("binary", "one_vs_rest", "pairwise"),
  reference_level = NULL,
  top_n = 10,
  min_count = 5
)

Arguments

Value

Data frame with columns:

• term: The word/feature

• category1: First category in comparison

• category2: Second category in comparison

• count1: Count in category 1

• count2: Count in category 2

• odds1: Smoothed odds in category 1, (count + 1) / (total + V - count - 1)

• odds2: Smoothed odds in category 2

• odds_ratio: Ratio of odds

• log_odds_ratio: Log of odds ratio (positive = more in compared category)

• variance: Approximate variance of the log odds ratio

• z_score: Log odds ratio divided by its standard error

• significant: TRUE when |z| >= 1.96

Terms are ranked by absolute z-score.

Examples

articles <- TextAnalysisR::SpecialEduTech[1:20, ]
corpus <- quanteda::corpus(
  articles$abstract,
  docvars = data.frame(reference_type = articles$reference_type)
)
dfm_object <- quanteda::dfm(quanteda::tokens(corpus))
log_odds <- calculate_log_odds_ratio(dfm_object, "reference_type")

Calculate Similarity Metrics

Description

Calculates similarity metrics including statistical measures and network properties. Internal function used by document_similarity_analysis.

Usage

calculate_metrics(similarity_matrix, labels = NULL, method_info = NULL)

Arguments

Value

A list of metrics.

Calculate NPMI for Topic Top-Terms

Description

NPMI (Normalized Pointwise Mutual Information) measures coherence of top-K terms per topic using internal document co-occurrence from the supplied DFM (boolean presence).

Usage

calculate_npmi(top_terms_list, dfm, top_k = 10, epsilon = 1e-12)

Arguments

Value

list with mean_npmi (scalar) and per_topic_npmi (numeric vector).

Calculate Semantic Drift

Description

Calculates semantic drift across time periods

Usage

calculate_semantic_drift(temporal_results, ...)

Arguments

Value

List containing drift metrics

Calculate Document Similarity with Fallbacks

Description

Calculates document similarity with fallback methods and diagnostics. Attempts embeddings first, falls back to Jaccard similarity if needed.

Usage

calculate_similarity_robust(
  texts,
  method = "embeddings",
  embedding_model = "all-MiniLM-L6-v2",
  cache_embeddings = TRUE,
  min_word_length = 3,
  doc_names = NULL
)

Arguments

Value

List containing similarity matrix, method used, embeddings, and diagnostics

Examples

if (interactive()) {
  abstracts <- TextAnalysisR::SpecialEduTech$abstract[1:5]
  similarity_result <- calculate_similarity_robust(abstracts)
  print(similarity_result$similarity_matrix)
  print(similarity_result$diagnostics)
}

Calculate Text Readability

Description

Calculates multiple readability metrics for texts including Flesch Reading Ease, Flesch-Kincaid Grade Level, Gunning FOG index, and others. Optionally includes lexical diversity metrics and sentence statistics.

Usage

calculate_text_readability(
  texts,
  metrics = c("flesch", "flesch_kincaid", "gunning_fog"),
  include_lexical_diversity = TRUE,
  include_sentence_stats = TRUE,
  doc_names = NULL
)

Arguments

Value

A data frame with document names and readability scores

Examples

data(SpecialEduTech, package = "TextAnalysisR")
texts <- SpecialEduTech$abstract[1:10]
readability <- calculate_text_readability(texts)
print(readability)

Calculate Topic-Cluster Correspondence

Description

Computes Jaccard-based correspondence between topic keywords and cluster keywords.

Usage

calculate_topic_cluster_correspondence(topic_keywords, cluster_keywords, ...)

Arguments

Value

List with correspondence metrics.

Calculate Topic Diversity (Unique-Word Proportion)

Description

Diversity = unique top-K terms across all topics / (n_topics * top_k). Range 0 (all topics identical) to 1 (fully disjoint). Complementary to NPMI coherence: a good model scores high on both.

Usage

calculate_topic_diversity(top_terms_list, top_k = 25)

Arguments

Value

Numeric between 0 and 1, or NA if input empty.

Calculate Topic Probabilities

Description

Extracts and summarizes topic probabilities (gamma values) from an STM model, returning a formatted data table of mean topic prevalence.

Usage

calculate_topic_probability(stm_model, top_n = 10, verbose = TRUE, ...)

Arguments

Value

A DT::datatable showing topics and their mean gamma (prevalence) values, rounded to 3 decimal places.

Examples

if (interactive()) {
  data <- TextAnalysisR::SpecialEduTech
  united <- unite_cols(data, c("title", "keyword", "abstract"))
  tokens <- prep_texts(united, text_field = "united_texts")
  dfm_obj <- quanteda::dfm(tokens)
  stm_data <- quanteda::convert(dfm_obj, to = "stm")

  topic_model <- stm::stm(
    documents = stm_data$documents,
    vocab = stm_data$vocab,
    K = 10,
    verbose = FALSE
  )

  prob_table <- calculate_topic_probability(topic_model, top_n = 10)
  print(prob_table)
}

Calculate Semantic Topic Quality Metrics

Description

Internal function to calculate quality metrics for semantic topic modeling results.

Usage

calculate_topic_quality(
  embeddings,
  topic_assignments,
  similarity_matrix = NULL
)

Arguments

Value

A list of quality metrics.

Calculate Topic Stability

Description

Calculates stability of topics across consecutive time periods. Each period is fitted independently, so topics are matched one-to-one on keyword Jaccard similarity before scoring (see calculate_keyword_stability()).

Usage

calculate_topic_stability(temporal_results)

Arguments

Value

Stability metrics: matched-pair stability per consecutive-period transition and their mean.

Calculate Weighted Log Odds Ratio

Description

Computes weighted log odds ratios using the method from Monroe, Colaresi, and Quinn (2008) "Fightin' Words" via the tidylo package. This method weights log odds by variance (z-score) to identify words that reliably distinguish between groups, accounting for sampling variability.

Usage

calculate_weighted_log_odds(dfm_object, group_var, top_n = 10, min_count = 5)

Arguments

Value

A data frame with the grouping variable, feature, n, log_odds_weighted (from tidylo::bind_log_odds), and significant (TRUE when |log_odds_weighted| >= 1.96)

References

Monroe, B. L., Colaresi, M. P., & Quinn, K. M. (2008). Fightin' words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16(4), 372-403.

Silge, J., & Robinson, D. (2017). Text mining with R: A tidy approach. O'Reilly Media. https://www.tidytextmining.com/

Examples

if (requireNamespace("tidylo", quietly = TRUE)) {
  articles <- TextAnalysisR::SpecialEduTech[1:20, ]
  dfm_object <- quanteda::dfm(quanteda::tokens(articles$abstract))
  quanteda::docvars(dfm_object, "reference_type") <- articles$reference_type
  weighted_odds <- calculate_weighted_log_odds(dfm_object, "reference_type",
                                                top_n = 5)
}

Analyze and Visualize Word Frequencies Across a Continuous Variable

Description

This function analyzes and visualizes word frequencies across a continuous variable.

Usage

calculate_word_frequency(
  dfm_object,
  continuous_variable,
  selected_terms,
  height = 500,
  width = 900
)

Arguments

Details

This function requires a fitted STM model object and a quanteda dfm object. The continuous variable should be a column in the metadata of the dfm object. The selected terms should be a vector of terms to analyze trends for. The required packages are 'htmltools', 'splines', and 'broom' (plus additional ones loaded internally).

Value

A list containing Plotly objects and tables with the results.

See Also

extract_keywords_tfidf() and extract_keywords_keyness() for ranking terms by importance; plot_word_frequency() for the standard frequency-by-doc plot

Examples

  mydata <- TextAnalysisR::SpecialEduTech

  united_tbl <- TextAnalysisR::unite_cols(
    mydata,
    listed_vars = c("title", "keyword", "abstract")
  )

  tokens <- TextAnalysisR::prep_texts(united_tbl, text_field = "united_texts")

  dfm_object <- quanteda::dfm(tokens)

  word_freq_results <- TextAnalysisR::calculate_word_frequency(
    dfm_object,
    continuous_variable = "year",
    selected_terms = c("calculator", "computer"),
    height = 500,
    width = 900
  )
  print(word_freq_results$plot)
  print(word_freq_results$table)

Call Gemini Chat API

Description

Makes a chat completion request to Google's Gemini API.

Usage

call_gemini_chat(
  system_prompt,
  user_prompt,
  model = "gemini-2.5-flash",
  temperature = 0,
  max_tokens = 8192,
  api_key
)

Arguments

Value

Character string with the model's response

Call LLM API (Unified Wrapper)

Description

Unified wrapper for calling different LLM providers (OpenAI, Gemini). Automatically routes to the appropriate provider-specific function.

Usage

call_llm_api(
  provider = c("openai", "gemini"),
  system_prompt,
  user_prompt,
  model = NULL,
  temperature = 0,
  max_tokens = 150,
  api_key = NULL
)

Arguments

Value

Character string with the model's response

See Also

sanitize_llm_input() to clean text before prompting; get_best_embeddings() for vector embeddings; run_rag_search() for RAG search (retrieval + generation)

Examples

if (interactive()) {
# Using OpenAI
response <- call_llm_api(
  provider = "openai",
  system_prompt = "You are a helpful assistant.",
  user_prompt = "Generate a topic label",
  api_key = Sys.getenv("OPENAI_API_KEY")
)

# Using Gemini
response <- call_llm_api(
  provider = "gemini",
  system_prompt = "You are a helpful assistant.",
  user_prompt = "Generate a topic label",
  api_key = Sys.getenv("GEMINI_API_KEY")
)
}

Call OpenAI Chat Completion API

Description

Makes a chat completion request to OpenAI's API.

Usage

call_openai_chat(
  system_prompt,
  user_prompt,
  model = "gpt-4.1-mini",
  temperature = 0,
  max_tokens = 150,
  api_key
)

Arguments

Value

Character string with the model's response

Check Alt Text Presence

Description

Validates that images and visualizations have alternative text descriptions. Required for WCAG 1.1.1 (Non-text Content).

Note: Decorative images should use empty alt text (alt="") to indicate they should be ignored by assistive technology.

Usage

check_alt_text(alt_text, element_type = "image", decorative = FALSE)

Arguments

Value

Logical TRUE if valid, FALSE with warning if missing/inadequate

Check Docker Deployment

Description

Detects whether the app is running in a Docker container. Docker deployments have full Python/spaCy capability.

Usage

check_docker_deployment()

Value

Logical TRUE if running in Docker

Check Feature Status

Description

Checks if a specific optional feature is available in the current environment.

Usage

check_feature(feature)

Arguments

Value

Logical TRUE if feature is available

Check Multimodal Prerequisites

Description

Checks all prerequisites for multimodal PDF extraction. Uses R-native pdftools for rendering (no Python required).

Usage

check_multimodal_prerequisites(
  vision_provider = "gemini",
  vision_model = NULL,
  api_key = NULL,
  envname = "textanalysisr-env"
)

Arguments

Value

List with:

• ready: Logical - TRUE if all prerequisites met

• missing: Character vector of missing components

• instructions: Character - Detailed setup instructions

• details: List with component-specific status

Check Python Environment Status

Description

Checks if Python environment is available and properly configured.

Usage

check_python_env(envname = "textanalysisr-env")

Arguments

Value

List with status information:

• available: Logical, TRUE if environment exists

• active: Logical, TRUE if environment is currently active

• packages: List of installed package versions

Examples

if (interactive()) {
status <- check_python_env()
print(status)
}

Check Rate Limit

Description

Check Rate Limit

Usage

check_rate_limit(
  session_token,
  user_requests,
  max_requests = 100,
  window_seconds = 3600
)

Arguments

Value

TRUE if within limit, stops with error if exceeded

Check Vision Model Availability

Description

Check if required vision models are available for multimodal processing.

Usage

check_vision_models(provider = "gemini", api_key = NULL)

Arguments

Value

List with availability status and recommendations

Examples

status <- check_vision_models("openai", api_key = Sys.getenv("OPENAI_API_KEY"))
status <- check_vision_models("gemini", api_key = Sys.getenv("GEMINI_API_KEY"))

Check WCAG Contrast Compliance

Description

Validates if color combination meets WCAG 2.1 Level AA contrast requirements.

Usage

check_wcag_contrast(foreground, background, large_text = FALSE)

Arguments

Value

Logical TRUE if compliant, FALSE if not

Check Deployment Environment

Description

Detects whether the app is running on a web server (shinyapps.io, Posit Connect) versus locally via run_app().

Usage

check_web_deployment()

Value

Logical TRUE if running on web server, FALSE if local

Clean Similarity Matrix

Description

Cleans a similarity matrix by handling NA/Inf values, ensuring symmetry, and setting diagonal to 1.

Usage

clean_similarity_matrix(similarity_matrix)

Arguments

Value

Cleaned similarity matrix.

Clear Lexical Diversity Cache

Description

Clears the internal cache used for lexical diversity calculations. Call this function to free memory or force fresh calculations.

Usage

clear_lexdiv_cache()

Value

Invisible NULL

Embedding-based Document Clustering

Description

This function performs clustering analysis using various methods, ordered from simple to detailed: k-means (simplest), hierarchical (intermediate), and UMAP+DBSCAN (most detailed).

Usage

cluster_embeddings(
  data_matrix,
  method = "kmeans",
  n_clusters = 0,
  umap_neighbors = 15,
  umap_min_dist = 0.1,
  umap_n_components = 10,
  umap_metric = "cosine",
  dbscan_eps = 0,
  dbscan_min_samples = NULL,
  reduce_outliers = TRUE,
  outlier_strategy = "centroid",
  outlier_threshold = 0,
  seed = 123,
  verbose = TRUE
)

Arguments

Value

A list containing cluster assignments, method used, and quality metrics. For method = "umap_dbscan" with reduce_outliers = TRUE, the list also reports how noise points were reassigned: outlier_strategy ("embeddings" or "centroid"), outlier_metric ("cosine" or "euclidean"), outlier_space ("original embedding space" or "UMAP space"), and outliers_reassigned (count).

Examples

if (interactive()) {
  mydata <- TextAnalysisR::SpecialEduTech

  united_tbl <- TextAnalysisR::unite_cols(
    mydata,
    listed_vars = c("title", "keyword", "abstract")
  )

  tokens <- TextAnalysisR::prep_texts(united_tbl, text_field = "united_texts")

  dfm_object <- quanteda::dfm(tokens)

  data_matrix <- as.matrix(dfm_object)

  kmeans_result <- TextAnalysisR::cluster_embeddings(
    data_matrix,
    method = "kmeans",
    n_clusters = 5
  )
  print(kmeans_result)

  hierarchical_result <- TextAnalysisR::cluster_embeddings(
    data_matrix,
    method = "hierarchical",
    n_clusters = 5
  )
  print(hierarchical_result)

  umap_dbscan_result <- TextAnalysisR::cluster_embeddings(
    data_matrix,
    method = "umap_dbscan",
    umap_neighbors = 15,
    umap_min_dist = 0.1
  )
  print(umap_dbscan_result)
}

Create Formatted Analysis Data Table

Description

Creates a consistently formatted DT::datatable for analysis results with export buttons and optional numeric formatting.

Usage

create_analysis_datatable(
  data,
  colnames = NULL,
  numeric_cols = NULL,
  digits = 3,
  export_formats = c("copy", "csv", "excel", "pdf", "print"),
  page_length = 25,
  font_size = "16px"
)

Arguments

Value

A DT::datatable object

Generate ARIA Label

Description

Creates accessible ARIA label for UI elements.

Usage

create_aria_label(element_type, action, context = NULL)

Arguments

Value

Character string with ARIA label

Empty plotly placeholder

Description

Empty plotly placeholder

Usage

create_empty_plot_message(message, color = "#6B7280", font_size = 16)

Arguments

Value

A plotly htmlwidget.

Single-cell message DT table

Description

Single-cell message DT table

Usage

create_message_table(message, font_size = "16px", color = "#6c757d")

Arguments

Value

A DT::datatable htmlwidget.

Create Screen Reader Text

Description

Generates visually hidden text for screen readers (WCAG 4.1.2).

Usage

create_sr_text(text)

Arguments

Value

HTML span with sr-only class

Create Standard ggplot2 Theme

Description

Returns a standardized ggplot2 theme matching TextAnalysisR design standards.

Usage

create_standard_ggplot_theme(base_size = 11)

Arguments

Value

A ggplot2 theme object

Describe Image Using Vision LLM

Description

Unified dispatcher for image description using vision LLMs. Routes to the appropriate provider (OpenAI or Gemini).

Usage

describe_image(
  image_base64,
  provider = "gemini",
  model = NULL,
  api_key = NULL,
  prompt =
    "Describe this image: charts, diagrams, tables, and text. Extract visible text.",
  timeout = 120
)

Arguments

Value

Character string description, or NULL on failure

Describe Image with Gemini Vision API

Description

Describe Image with Gemini Vision API

Usage

describe_image_gemini(
  image_base64,
  prompt =
    "Describe this image: charts, diagrams, tables, and text. Extract visible text.",
  model = "gemini-2.5-flash",
  max_tokens = 500,
  api_key
)

Arguments

Value

Character string description, or NULL on failure

Describe Image with OpenAI Vision API

Description

Describe Image with OpenAI Vision API

Usage

describe_image_openai(
  image_base64,
  prompt =
    "Describe this image: charts, diagrams, tables, and text. Extract visible text.",
  model = "gpt-4.1",
  max_tokens = 500,
  api_key
)

Arguments

Value

Character string description, or NULL on failure

Detect Multi-Word Expressions

Description

This function detects multi-word expressions (collocations) of specified sizes that appear at least a specified number of times in the provided tokens.

Usage

detect_multi_words(tokens, size = 2:5, min_count = 2)

Arguments

Value

A character vector of detected collocations.

Examples

  mydata <- TextAnalysisR::SpecialEduTech[seq_len(50), ]

  united_tbl <- TextAnalysisR::unite_cols(
    mydata,
    listed_vars = c("title", "keyword", "abstract")
  )

  tokens <- TextAnalysisR::prep_texts(united_tbl, text_field = "united_texts")

  collocations <- TextAnalysisR::detect_multi_words(tokens, size = 2:3, min_count = 2)
  print(collocations)

Detect PDF Content Type

Description

Analyzes PDF to determine if it contains readable text.

Usage

detect_pdf_content_type(file_path)

Arguments

Details

Attempts text extraction using pdftools. Returns "text" if successful, or "unknown" if extraction fails or PDF is empty.

For table extraction from PDFs, use extract_tables_from_pdf_py.

Value

Character string: "text" or "unknown"

Examples

pdf_path <- "path/to/document.pdf"
content_type <- detect_pdf_content_type(pdf_path)
print(content_type)

Detect PDF Content Type using Python

Description

Analyzes PDF to determine if it contains primarily tabular data or text.

Usage

detect_pdf_content_type_py(file_path, envname = NULL)

Arguments

Value

Character string: "tabular", "text", or "unknown"

Examples

if (interactive()) {
setup_python_env()

pdf_path <- "path/to/document.pdf"
content_type <- detect_pdf_content_type_py(pdf_path)
print(content_type)
}

Export Document Clustering Analysis

Description

Export document clustering analysis results to CSV

Usage

export_document_clustering(
  coordinates,
  clusters = NULL,
  labels = NULL,
  doc_ids = NULL,
  file_path
)

Arguments

Value

Invisible data frame of the exported data

Extract Cross-Category Similarities from Full Similarity Matrix

Description

Given a full similarity matrix and category information, extracts pairwise similarities between a reference category and other categories into a long-format data frame suitable for visualization and analysis.

Usage

extract_cross_category_similarities(
  similarity_matrix,
  docs_data,
  reference_category,
  compare_categories = NULL,
  category_var = "category",
  id_var = "display_name",
  name_var = NULL
)

Arguments

Value

A data frame with columns:

Examples

articles <- TextAnalysisR::SpecialEduTech[1:6, ]
articles$display_name <- paste0("d", seq_len(nrow(articles)))
term_matrix <- as.matrix(quanteda::dfm(quanteda::tokens(articles$abstract)))
normalized_matrix <- term_matrix / sqrt(rowSums(term_matrix ^ 2))
similarity_matrix <- normalized_matrix %*% t(normalized_matrix)
dimnames(similarity_matrix) <- list(articles$display_name, articles$display_name)
cross_similarities <- extract_cross_category_similarities(
  similarity_matrix  = similarity_matrix,
  docs_data          = articles,
  reference_category = "thesis",
  compare_categories = "journal_article",
  category_var       = "reference_type",
  id_var             = "display_name",
  name_var           = "title"
)

Extract Keywords Using Statistical Keyness

Description

Extracts distinctive keywords by comparing document groups using log-likelihood ratio (G-squared).

Usage

extract_keywords_keyness(
  dfm,
  target,
  top_n = 20,
  measure = "lr",
  min_count = 0
)

Arguments

Value

Data frame with columns: Keyword, Keyness_Score

Examples

abstracts <- TextAnalysisR::SpecialEduTech$abstract[1:10]
dfm_object <- quanteda::dfm(quanteda::tokens(quanteda::corpus(abstracts)))
keywords <- extract_keywords_keyness(dfm_object, target = 1)
print(keywords)

Extract Keywords Using TF-IDF

Description

Extracts top keywords from a document-feature matrix using TF-IDF weighting. Uses quanteda's default scheme (raw term counts, base-10 idf), which differs from tidytext's proportion-based tf; rankings favor terms in longer documents.

Usage

extract_keywords_tfidf(dfm, top_n = 20, normalize = FALSE)

Arguments

Value

Data frame with columns: Keyword, TF_IDF_Score, Frequency

Examples

  library(quanteda)
  corp <- corpus(c("text analysis", "data mining", "text mining"))
  dfm_obj <- dfm(tokens(corp))
  keywords <- extract_keywords_tfidf(dfm_obj, top_n = 5)
  print(keywords)

Extract Morphological Features

Description

Uses spaCy to extract morphological features from text. Returns data with Number, Tense, VerbForm, Person, Case, Mood, Aspect, etc.

Usage

extract_morphology(
  tokens,
  features = c("Number", "Tense", "VerbForm", "Person", "Case", "Mood", "Aspect"),
  include_pos = TRUE,
  include_lemma = TRUE,
  model = "en_core_web_sm"
)

Arguments

Details

Morphological features follow Universal Dependencies annotation. Common features include:

• Number: Sing (singular), Plur (plural)

• Tense: Past, Pres (present), Fut (future)

• VerbForm: Fin (finite), Inf (infinitive), Part (participle), Ger (gerund)

• Person: 1, 2, 3 (first, second, third person)

• Case: Nom (nominative), Acc (accusative), Gen (genitive), Dat (dative)

• Mood: Ind (indicative), Imp (imperative), Sub (subjunctive)

• Aspect: Perf (perfective), Imp (imperfective), Prog (progressive)

Value

A data frame with token-level morphological annotations including morph_* columns for each requested feature.

Examples

if (interactive()) {
  tokens <- quanteda::tokens(TextAnalysisR::SpecialEduTech$abstract[1])
  morphology_data <- extract_morphology(tokens)
  print(morphology_data)
}

Extract Named Entities from Tokens

Description

Uses spaCy to extract named entities (NER) from tokenized text. Returns a data frame with token-level entity annotations.

Usage

extract_named_entities(
  tokens,
  include_pos = TRUE,
  include_lemma = TRUE,
  model = "en_core_web_sm"
)

Arguments

Details

This function requires the Python with spaCy installed. If spaCy is not initialized, this function will attempt to initialize it with the specified model.

Value

A data frame with columns:

• doc_id: Document identifier

• token: Original token

• entity: Named entity type (e.g., PERSON, ORG, GPE)

• pos: Universal POS tag (if include_pos = TRUE)

• lemma: Lemmatized form (if include_lemma = TRUE)

Examples

if (interactive()) {
  tokens <- quanteda::tokens(TextAnalysisR::SpecialEduTech$abstract[1])
  entity_data <- extract_named_entities(tokens)
  print(entity_data)
}

Extract Noun Chunks

Description

Extract noun chunks (base noun phrases) from texts. Useful for keyphrase extraction.

Usage

extract_noun_chunks(x, model = "en_core_web_sm")

Arguments

Value

A data frame with noun chunk information.

Extract PDF with Multimodal Analysis

Description

Extract both text and visual content from PDFs using R-native pdftools and vision LLM APIs. No Python required.

Usage

extract_pdf_multimodal(
  file_path,
  vision_provider = "gemini",
  vision_model = NULL,
  api_key = NULL,
  describe_images = TRUE,
  envname = "textanalysisr-env"
)

Arguments

Details

Workflow:

1. Extracts text using pdftools (R-native)

2. Renders each page as an image

3. Sends sparse-text pages to vision LLM for description

4. Merges text + descriptions into a single text corpus

Value

List with:

• success: Logical

• combined_text: Character string with all content for text analysis

• text_content: List of text chunks

• image_descriptions: List of image descriptions

• num_images: Integer count of described pages

• vision_provider: Character indicating provider used

• message: Character status message

Smart PDF Extraction with Auto-Detection

Description

Extracts text and visual content from PDFs using R-native pdftools and vision LLM APIs. Routes directly to multimodal extraction.

Usage

extract_pdf_smart(
  file_path,
  doc_type = "auto",
  vision_provider = "gemini",
  vision_model = NULL,
  api_key = NULL,
  envname = "textanalysisr-env"
)

Arguments

Value

List with extracted content ready for text analysis

Extract Part-of-Speech Tags from Tokens

Description

Uses spaCy to extract part-of-speech (POS) tags from tokenized text. Returns a data frame with token-level POS annotations.

Usage

extract_pos_tags(
  tokens,
  include_lemma = TRUE,
  include_entity = FALSE,
  include_dependency = FALSE,
  model = "en_core_web_sm"
)

Arguments

Details

This function requires the Python with spaCy installed. If spaCy is not initialized, this function will attempt to initialize it with the specified model.

Value

A data frame with columns:

• doc_id: Document identifier

• sentence_id: Sentence number within document

• token_id: Token position within sentence

• token: Original token

• pos: Universal POS tag (e.g., NOUN, VERB, ADJ)

• tag: Detailed POS tag (e.g., NN, VBD, JJ)

• lemma: Lemmatized form (if include_lemma = TRUE)

• entity: Named entity type (if include_entity = TRUE)

• head_token_id: Head token in dependency tree (if include_dependency = TRUE)

• dep_rel: Dependency relation type, e.g., nsubj, dobj (if include_dependency = TRUE)

Examples

if (interactive()) {
  tokens <- quanteda::tokens(TextAnalysisR::SpecialEduTech$abstract[1])
  pos_data <- extract_pos_tags(tokens)
  print(pos_data)
}

Extract Subjects and Objects

Description

Extract subject-verb-object (SVO) triples from texts using dependency parsing.

Usage

extract_subjects_objects(x, model = "en_core_web_sm")

Arguments

Value

A data frame with SVO information.

Extract Tables from PDF using Python

Description

Extracts tabular data from PDF using pdfplumber (Python). No Java required - pure Python solution.

Usage

extract_tables_from_pdf_py(file_path, pages = NULL, envname = NULL)

Arguments

Details

Uses pdfplumber Python library through reticulate. Works with complex table layouts without Java dependency.

Value

Data frame with extracted table data Returns NULL if no tables found or extraction fails

Examples

if (interactive()) {
setup_python_env()

pdf_path <- "path/to/table_document.pdf"
table_data <- extract_tables_from_pdf_py(pdf_path)
head(table_data)
}

Extract Text from PDF

Description

Extracts text content from a PDF file using pdftools package.

Usage

extract_text_from_pdf(file_path)

Arguments

Details

Uses pdftools::pdf_text() to extract text from each page. Preserves page structure and cleans whitespace. Works best with text-based PDFs (not scanned images).

Value

Data frame with columns: page (integer), text (character) Returns NULL if extraction fails or PDF is empty

Extract Text from PDF using Python

Description

Extracts text content from a PDF file using pdfplumber (Python). No Java required - uses Python environment.

Usage

extract_text_from_pdf_py(file_path, envname = NULL)

Arguments

Details

Uses pdfplumber Python library through reticulate. Requires Python environment setup. See setup_python_env().

Value

Data frame with columns: page (integer), text (character) Returns NULL if extraction fails or PDF is empty

Build a topic-term data frame from any supported topic model

Description

Unified helper that produces the long-format data.frame(topic, term, beta) expected by generate_topic_labels() from an STM model or an embedding result. Dispatches on the object's structure:

• STM model (has ⁠$beta$logbeta⁠ and ⁠$vocab⁠) -> top terms via stm::labelTopics() FREX

• Embedding result (has ⁠$topic_keywords⁠) -> c-TF-IDF keywords with rank-derived pseudo-beta

Usage

extract_topic_terms_df(model, n = 7)

Arguments

Value

data.frame(topic, term, beta) in long format.

Find Optimal Number of Topics

Description

Searches for the optimal number of topics (K) using stm::searchK. Produces diagnostic plots to help select the best K value. The default spectral initialization is deterministic; with LDA or random initialization, fit several seeds per K before selecting a model.

Usage

find_optimal_k(
  dfm_object,
  topic_range,
  max.em.its = 75,
  emtol = 1e-04,
  cores = 1,
  categorical_var = NULL,
  continuous_var = NULL,
  height = 600,
  width = 800,
  verbose = TRUE,
  ...
)

Arguments

Value

A list containing search results and diagnostic plots.

See Also

plot_quality_metrics() to visualize topic-count diagnostics; stm::stm() to fit the chosen model; fit_embedding_model() for an embedding-based alternative to STM

Find Similar Words

Description

Find words most similar to a given word using word vectors. Requires a spaCy model with word vectors (en_core_web_md or en_core_web_lg).

Usage

find_similar_words(word, top_n = 10L, model = "en_core_web_md")

Arguments

Value

A data frame with similar words and similarity scores.

Find Similar Topics

Description

This function finds the most similar topics to a given query using semantic similarity analysis. It works with both semantic topic models and traditional STM models by creating topic representations using transformer embeddings and calculating cosine similarity scores.

Usage

find_topic_matches(
  topic_model,
  query,
  top_n = 10,
  method = "cosine",
  embedding_model = "all-MiniLM-L6-v2",
  include_terms = TRUE
)

Arguments

Value

A list containing similar topics and their similarity scores.

Examples

if (interactive()) {
  mydata <- TextAnalysisR::SpecialEduTech
  united_tbl <- TextAnalysisR::unite_cols(
    mydata,
    listed_vars = c("title", "keyword", "abstract")
  )
  texts <- united_tbl$united_texts

  topic_model <- TextAnalysisR::fit_embedding_model(
    texts = texts,
    method = "umap_hdbscan",
    n_topics = 8
  )

  similar_topics <- TextAnalysisR::find_topic_matches(
    topic_model = topic_model,
    query = "mathematical learning",
    top_n = 5
  )

  print(similar_topics)
}

Fit Embedding-based Topic Model

Description

This function performs embedding-based topic modeling using transformer embeddings and specialized clustering techniques. Supports two backends:

• Python backend (default): Uses BERTopic library which combines transformer embeddings with UMAP dimensionality reduction and HDBSCAN clustering for optimal topic discovery.

• R backend: Uses R-native packages (umap, dbscan, Rtsne) for users without Python/BERTopic installed. Provides similar functionality with c-TF-IDF keyword extraction.

Usage

fit_embedding_model(
  texts,
  method = "umap_hdbscan",
  n_topics = 10,
  embedding_model = "all-MiniLM-L6-v2",
  backend = "auto",
  clustering_method = "kmeans",
  similarity_threshold = 0.7,
  min_topic_size = 10,
  min_cluster_size = NULL,
  cluster_selection_method = "eom",
  umap_neighbors = 15,
  umap_min_dist = 0,
  umap_n_components = 5,
  umap_metric = "cosine",
  tsne_perplexity = 30,
  pca_dims = 50,
  dbscan_eps = 0.5,
  dbscan_minpts = 5,
  representation_method = "c-tfidf",
  diversity = 0.5,
  reduce_outliers = TRUE,
  outlier_strategy = "probabilities",
  outlier_threshold = 0,
  seed = 123,
  verbose = TRUE,
  precomputed_embeddings = NULL
)

Arguments

Value

A list containing topic assignments, topic keywords, and quality metrics.

See Also

get_best_embeddings() to supply precomputed embeddings; generate_topic_labels() for AI-suggested topic names; find_optimal_k() for an STM-based alternative

Examples

if (interactive()) {
  mydata <- TextAnalysisR::SpecialEduTech
  united_tbl <- TextAnalysisR::unite_cols(
    mydata,
    listed_vars = c("title", "keyword", "abstract")
  )
  texts <- united_tbl$united_texts

  # Embedding-based topic modeling (powered by BERTopic)
  result <- TextAnalysisR::fit_embedding_model(
    texts = texts,
    method = "umap_hdbscan",
    n_topics = 8,
    min_topic_size = 3
  )

  print(result$topic_assignments)
  print(result$topic_keywords)
}

Embedding-based Topic Modeling (Deprecated)

Description

This function is deprecated. Please use fit_embedding_model() instead.

Usage

fit_embedding_topics(
  texts,
  method = "umap_hdbscan",
  n_topics = 10,
  embedding_model = "all-MiniLM-L6-v2",
  clustering_method = "kmeans",
  similarity_threshold = 0.7,
  min_topic_size = 10,
  cluster_selection_method = "eom",
  umap_neighbors = 15,
  umap_min_dist = 0,
  umap_n_components = 5,
  representation_method = "c-tfidf",
  diversity = 0.5,
  reduce_outliers = TRUE,
  outlier_strategy = "probabilities",
  outlier_threshold = 0,
  seed = 123,
  verbose = TRUE,
  precomputed_embeddings = NULL
)

Arguments

Value

A list containing topic assignments, topic keywords, and quality metrics.

Fit Semantic Model

Description

Performs semantic analysis including similarity, dimensionality reduction, and clustering. This is a high-level wrapper function.

Usage

fit_semantic_model(
  texts,
  analysis_types = c("similarity", "dimensionality_reduction", "clustering"),
  document_feature_type = "embeddings",
  similarity_method = "cosine",
  use_embeddings = TRUE,
  embedding_model = "all-MiniLM-L6-v2",
  dimred_method = "UMAP",
  clustering_method = "umap_dbscan",
  n_components = 2,
  n_clusters = 5,
  seed = 123,
  verbose = TRUE
)

Arguments

Value

A list containing results from requested analyses.

Examples

if (interactive()) {
  data(SpecialEduTech)
  texts <- SpecialEduTech$abstract[1:10]

  results <- fit_semantic_model(
    texts = texts,
    analysis_types = c("similarity", "clustering")
  )

  print(results)
}

Fit Temporal Topic Model

Description

Analyzes how topics evolve over time by fitting topic models to different time periods and tracking semantic changes.

Usage

fit_temporal_model(
  texts,
  dates,
  time_windows = "yearly",
  embeddings = NULL,
  verbose = TRUE
)

Arguments

Value

A list containing temporal analysis results with topic evolution patterns.

Generate Cluster Label Suggestions (Human-in-the-Loop)

Description

Suggests descriptive labels for clusters using AI. Labels are suggestions for human review - users should edit and approve before using. Supports OpenAI or Gemini for AI generation.

Usage

generate_cluster_labels(
  cluster_keywords,
  provider = "auto",
  model = NULL,
  temperature = 0.3,
  max_tokens = 50,
  api_key = NULL,
  verbose = TRUE
)

Arguments

Value

A list of generated labels.

Examples

if (interactive()) {
  cluster_keywords <- list(
    "1" = c("calculator", "arithmetic", "elementary", "remedial"),
    "2" = c("computer", "instruction", "multiplication", "drill")
  )
  labels_openai <- generate_cluster_labels(cluster_keywords, provider = "openai")
  labels_gemini <- generate_cluster_labels(cluster_keywords, provider = "gemini")
}

Generate Cluster Labels

Description

Generate descriptive labels for document clusters

Usage

generate_cluster_labels_auto(
  feature_matrix,
  clusters,
  method = "tfidf",
  n_terms = 3
)

Arguments

Value

Named list of cluster labels

Generate Embeddings

Description

Generates L2-normalized embeddings for texts using sentence transformers.

Usage

generate_embeddings(texts, model = "all-MiniLM-L6-v2", verbose = TRUE)

Arguments

Value

A matrix of L2-normalized embeddings.

Generate Semantic Topic Keywords (c-TF-IDF)

Description

Generate keywords for topics using c-TF-IDF (class-based TF-IDF), similar to BERTopic. This method treats all documents in a topic as a single document and calculates TF-IDF scores relative to other topics.

Usage

generate_semantic_topic_keywords(
  texts,
  topic_assignments,
  n_keywords = 10,
  method = "c-tfidf",
  diversity = 0.5
)

Arguments

Value

A list of keywords for each topic.

Generate Content from Topic Terms

Description

Uses Large Language Models (LLMs) to generate various types of content based on topic model terms. Supports multiple content types with optimized default prompts, or fully custom prompts.

Usage

generate_topic_content(
  topic_terms_df,
  content_type = c("survey_item", "research_question", "theme_description",
    "policy_recommendation", "interview_question", "custom"),
  topic_var = "topic",
  term_var = "term",
  weight_var = "beta",
  provider = c("openai", "gemini"),
  model = "gpt-4.1-mini",
  temperature = 0,
  system_prompt = NULL,
  user_prompt_template = NULL,
  max_tokens = 150,
  api_key = NULL,
  output_var = NULL,
  verbose = TRUE
)

Arguments

Details

The function generates one piece of content per unique topic. Each content type has optimized default prompts, but these can be overridden with custom prompts.

Requires an API key set via the api_key parameter or the relevant environment variable (OPENAI_API_KEY or GEMINI_API_KEY).

Value

A data frame with generated content joined to original topic terms.

See Also

generate_topic_labels() for the step that creates topic labels; get_content_type_prompt() and get_content_type_user_template() to inspect or override default prompts

Examples

if (interactive()) {
# Generate survey items
survey_items <- generate_topic_content(
  topic_terms_df = top_terms,
  content_type = "survey_item",
  provider = "openai",
  model = "gpt-4.1-mini"
)

# Generate research questions
research_qs <- generate_topic_content(
  topic_terms_df = top_terms,
  content_type = "research_question",
  provider = "gemini",
  model = "gemini-2.5-flash-lite"
)

# Generate with custom prompt
custom_content <- generate_topic_content(
  topic_terms_df = top_terms,
  content_type = "custom",
  system_prompt = "You are an expert in educational policy...",
  user_prompt_template = "Based on {terms}, generate a learning objective:"
)
}

Generate Topic Keywords

Description

Internal function to generate keywords for topics using TF-IDF analysis.

Usage

generate_topic_keywords(texts, topic_assignments, n_keywords = 10)

Arguments

Value

A list of keywords for each topic.

Generate Topic Labels Using AI

Description

This function generates descriptive labels for each topic based on their top terms using AI providers (OpenAI or Gemini).

Usage

generate_topic_labels(
  top_topic_terms,
  provider = "auto",
  model = NULL,
  system = NULL,
  user = NULL,
  temperature = 0.5,
  api_key = NULL,
  openai_api_key = NULL,
  verbose = TRUE
)

Arguments

Value

A data frame containing the top terms for each topic along with their generated labels.

See Also

get_topic_terms() to extract top terms first; generate_topic_content() for survey items / RQs / themes grounded in the same terms; call_llm_api() for the direct AI provider call

Examples

if (interactive()) {
top_topic_terms <- get_topic_terms(stm_model, top_term_n = 10)

# Auto-detect provider (tries OpenAI -> Gemini)
labels <- generate_topic_labels(top_topic_terms)

# Use specific provider
labels_openai <- generate_topic_labels(top_topic_terms, provider = "openai")
labels_gemini <- generate_topic_labels(top_topic_terms, provider = "gemini")
}

Get Embeddings from API

Description

Generates text embeddings using OpenAI or Gemini embedding APIs.

Usage

get_api_embeddings(
  texts,
  provider = c("openai", "gemini"),
  model = NULL,
  api_key = NULL,
  batch_size = 100
)

Arguments

Value

Matrix with embeddings (rows = texts, columns = dimensions)

Get Available Document-Feature Matrix with Fallback

Description

Returns the first non-NULL DFM from a priority fallback chain. Useful when multiple DFM processing stages exist and the most processed available version is needed.

Usage

get_available_dfm(
  dfm_lemma = NULL,
  dfm_outcome = NULL,
  dfm_final = NULL,
  dfm_init = NULL
)

Arguments

Details

Priority order (highest to lowest):

1. dfm_lemma - Lemmatized tokens (most processed)

2. dfm_outcome - Preprocessed tokens

3. dfm_final - Final processed version

4. dfm_init - Initial unprocessed tokens

Value

The first non-NULL DFM from the priority chain, or NULL if all are NULL

Examples

  dfm1 <- quanteda::dfm(quanteda::tokens("assistive technology supports learning"))
  result <- get_available_dfm(dfm_init = dfm1)

Get Available Tokens with Fallback

Description

Returns the first non-NULL tokens object from a priority fallback chain. Useful when multiple token processing stages exist and the most processed available version is needed.

Usage

get_available_tokens(
  final_tokens = NULL,
  processed_tokens = NULL,
  preprocessed_tokens = NULL,
  united_tbl = NULL
)

Arguments

Details

Priority order (highest to lowest):

1. final_tokens - Fully processed tokens

2. processed_tokens - Partially processed tokens

3. preprocessed_tokens - Early stage preprocessed tokens

4. united_tbl - Raw text (will be tokenized if used)

Value

The first non-NULL tokens from the priority chain, or NULL if all are NULL

Examples

  toks <- quanteda::tokens("assistive technology supports learning")
  result <- get_available_tokens(processed_tokens = toks)

Get Best Available Embeddings

Description

Auto-detects and uses the best available embedding provider with the following priority:

1. sentence-transformers (local Python) - if Python environment is set up

2. OpenAI API - if OPENAI_API_KEY is set

3. Gemini API - if GEMINI_API_KEY is set

Usage

get_best_embeddings(
  texts,
  provider = "auto",
  model = NULL,
  api_key = NULL,
  verbose = TRUE
)

Arguments

Value

Matrix with embeddings (rows = texts, columns = dimensions)

Examples

if (interactive()) {
data(SpecialEduTech)
texts <- SpecialEduTech$abstract[1:5]

# Auto-detect best available provider
embeddings <- get_best_embeddings(texts)

# Force specific provider
embeddings <- get_best_embeddings(texts, provider = "openai")

dim(embeddings)
}

Get Default System Prompt for Content Type

Description

Returns the default system prompt for a given content type.

Usage

get_content_type_prompt(content_type)

Arguments

Value

Character string with the system prompt.

Get Default User Prompt Template for Content Type

Description

Returns the default user prompt template for a given content type.

Usage

get_content_type_user_template(content_type)

Arguments

Value

Character string with the user prompt template containing {terms} placeholder.

Generate DFM Setup Instructions Text

Description

Generates standardized text instructions for creating a DFM. Used in console output or verbatim text displays.

Usage

get_dfm_setup_instructions(feature_name = "this feature")

Arguments

Value

Character vector of instruction lines

Default DT datatable options

Description

Default DT datatable options

Usage

get_dt_options(scroll_y = "400px", page_length = 25, show_buttons = TRUE)

Arguments

Value

A list for DT::datatable(options = ...).

Get Feature Status

Description

Returns availability status for all optional features.

Usage

get_feature_status()

Value

Named list with feature availability

Get Gemini Embeddings (Internal)

Description

Get Gemini Embeddings (Internal)

Usage

get_gemini_embeddings(texts, model, api_key)

Get OpenAI Embeddings (Internal)

Description

Get OpenAI Embeddings (Internal)

Usage

get_openai_embeddings(texts, model, api_key)

Plotly hover tooltip config

Description

Plotly hover tooltip config

Usage

get_plotly_hover_config(bgcolor = "#ffffff", fontcolor = "#0c1f4a")

Arguments

Value

A list for plotly::layout(hoverlabel = ...).

Get Sentences

Description

Segment texts into sentences using spaCy's sentence boundary detection.

Usage

get_sentences(x, model = "en_core_web_sm")

Arguments

Value

A data frame with sentence information.

Generate Sentiment Color Gradient

Description

Generates a color based on sentiment score using a gradient from red (negative) through gray (neutral) to green (positive).

Usage

get_sentiment_color(score)

Arguments

Value

Hex color string

Examples

get_sentiment_color(-0.8)  # Red
get_sentiment_color(0)     # Gray
get_sentiment_color(0.8)   # Green

Get Sentiment Color Palette

Description

Returns standardized color mapping for sentiment analysis.

Usage

get_sentiment_colors()

Value

Named vector of colors

Get spaCy Word Embeddings

Description

Get word vector embeddings for words or texts using spaCy. Requires a spaCy model with word vectors.

Usage

get_spacy_embeddings(texts, model = "en_core_web_md")

Arguments

Value

A matrix of word embeddings (rows = texts, cols = dimensions).

Get spaCy Model Information

Description

Get information about the currently loaded spaCy model.

Usage

get_spacy_model_info()

Value

A list with model information including name, language, pipeline components, and vector availability.

Get Topic Prevalence (Gamma) from STM Model

Description

Extracts topic prevalence values (gamma/theta) from a fitted STM model, returning mean prevalence for each topic as a data frame.

Usage

get_topic_prevalence(stm_model, category = NULL, include_theta = FALSE)

Arguments

Value

A data frame with columns:

Examples

if (interactive() && requireNamespace("stm", quietly = TRUE)) {
  # Requires fitting an STM model first; uses 'stm::gadarian' for demo
  data("gadarian", package = "stm")
  proc <- stm::textProcessor(gadarian$open.ended.response, metadata = gadarian)
  prep <- stm::prepDocuments(proc$documents, proc$vocab, proc$meta)
  topic_model <- stm::stm(prep$documents, prep$vocab, K = 3,
                           data = prep$meta, max.em.its = 5,
                           verbose = FALSE)
  prevalence <- get_topic_prevalence(topic_model)
  prevalence_label <- get_topic_prevalence(topic_model, category = "demo")
}

Select Top Terms for Each Topic

Description

This function selects the top terms for each topic based on their word probability distribution (beta).

Usage

get_topic_terms(stm_model, top_term_n = 10, verbose = TRUE, ...)

Arguments

Value

A data frame containing the top terms for each topic.

Examples

if (interactive() && requireNamespace("stm", quietly = TRUE)) {
  mydata <- TextAnalysisR::SpecialEduTech

  united_tbl <- TextAnalysisR::unite_cols(
    mydata,
    listed_vars = c("title", "keyword", "abstract")
  )

  tokens <- TextAnalysisR::prep_texts(united_tbl, text_field = "united_texts")

  dfm_object <- quanteda::dfm(tokens)

  out <- quanteda::convert(dfm_object, to = "stm")

  stm_15 <- stm::stm(
    data = out$meta,
    documents = out$documents,
    vocab = out$vocab,
    max.em.its = 75,
    init.type = "Spectral",
    K = 15,
    prevalence = ~ reference_type + s(year),
    verbose = TRUE
  )

  top_topic_terms <- TextAnalysisR::get_topic_terms(
    stm_model = stm_15,
    top_term_n = 10,
    verbose = TRUE
  )
  print(top_topic_terms)
}

Convert Topic Terms to Text Strings

Description

Concatenates top terms for each topic into text strings suitable for embedding generation. Useful for creating topic representations for semantic similarity analysis.

Usage

get_topic_texts(
  top_terms_df,
  topic_var = "topic",
  term_var = "term",
  weight_var = NULL,
  sep = " ",
  top_n = NULL
)

Arguments

Value

A character vector of topic text strings, one per topic, ordered by topic number.

Examples

# Topic-term frame as produced by get_topic_terms()
top_terms <- data.frame(
  topic = c(1, 1, 1, 2, 2, 2),
  term  = c("calculator", "arithmetic", "elementary",
            "computer", "instruction", "multiplication"),
  prob  = c(0.10, 0.08, 0.07, 0.12, 0.09, 0.06)
)
get_topic_texts(top_terms)
get_topic_texts(top_terms, weight_var = "prob", top_n = 2)

Calculate Word Similarity

Description

Calculate semantic similarity between two words using word vectors. Requires a spaCy model with word vectors (en_core_web_md or en_core_web_lg).

Usage

get_word_similarity(word1, word2, model = "en_core_web_md")

Arguments

Value

A list with similarity score and metadata.

Identify Topic Trends

Description

Identifies trending topics in temporal results

Usage

identify_topic_trends(temporal_results, ...)

Arguments

Value

List containing identified trends

Process Files

Description

This function processes different types of files and text input based on the dataset choice.

Usage

import_files(dataset_choice, file_info = NULL, text_input = NULL)

Arguments

Value

A data frame containing the processed data.

Examples

if (interactive()) {
  mydata <- TextAnalysisR::SpecialEduTech
  mydata <- TextAnalysisR::import_files(dataset_choice = "Upload an Example Dataset")
  head(mydata)

  xlsx_path <- system.file("extdata", "SpecialEduTech.xlsx",
                           package = "TextAnalysisR")
  file_info <- data.frame(filepath = xlsx_path)
  mydata <- TextAnalysisR::import_files(dataset_choice = "Upload Your File",
                                          file_info = file_info)
  head(mydata)


  text_input <- paste("Virtual manipulatives for algebra instruction",
                      "manipulatives mathematics learning disability",
                      "This study examined virtual manipulatives effects on",
                      "students with learning disabilities")
  mydata <- TextAnalysisR::import_files(dataset_choice = "Copy and Paste Text",
                                          text_input = text_input)
  head(mydata)
}

Initialize spaCy NLP

Description

Initialize the spaCy NLP pipeline with the specified model. Uses a cached instance for efficiency.

Usage

init_spacy_nlp(model = "en_core_web_sm", force = FALSE)

Arguments

Details

Available models:

• en_core_web_sm: Small English model (fast, no word vectors)

• en_core_web_md: Medium English model (word vectors)

• en_core_web_lg: Large English model (best accuracy)

Value

Invisibly returns the SpacyNLP Python object.

Examples

if (interactive()) {
init_spacy_nlp("en_core_web_sm")
}

Lemmatize Tokens with Batch Processing

Description

Converts tokens to their lemmatized forms using spaCy, with batch processing to handle large document collections without timeout issues.

Usage

lemmatize_tokens(
  tokens,
  batch_size = 50,
  model = "en_core_web_sm",
  verbose = TRUE
)

Arguments

Details

Uses spaCy for linguistic lemmatization producing proper dictionary forms (e.g., "studies" -> "study", "better" -> "good"). Batch processing prevents timeout errors with large document collections.

Value

A quanteda tokens object containing lemmatized tokens.

Examples

if (interactive()) {
tokens <- quanteda::tokens(c("The studies showed better results"))
lemmatized <- lemmatize_tokens(tokens, batch_size = 50)
}

Lexical Analysis Functions

Description

Functions for lexical analysis including:

• Linguistic Annotation (POS tagging, NER)

• Frequency Analysis (word frequency, n-grams, MWEs)

• Keywords (TF-IDF, keyness)

• Lexical Diversity (TTR, MTLD, MATTR)

• Readability (Flesch, Gunning Fog, etc.)

Lexical Diversity Analysis

Description

Calculates multiple lexical diversity metrics for a document-feature matrix (DFM) or tokens object. Supports all quanteda.textstats measures plus MTLD (Measure of Textual Lexical Diversity), which is the most recommended measure according to McCarthy & Jarvis (2010) for being independent of text length.

Usage

lexical_diversity_analysis(x, measures = "all", texts = NULL, cache_key = NULL)

Arguments

Details

MTLD (Measure of Textual Lexical Diversity) is calculated using the algorithm from McCarthy & Jarvis (2010). It counts the number of "factors" needed to reduce TTR below 0.72, then divides the number of tokens by the number of factors. This provides a length-independent measure of lexical diversity.

Important notes:

• For MTLD accuracy, pass a tokens object (not DFM) as input

• If using DFM, provide the 'texts' parameter for MTLD calculation

• MATTR and MSTTR window sizes are automatically adjusted for short documents

• Raw TTR falls mechanically as documents lengthen; compare TTR only across documents of similar length

• MTLD and MATTR are most reliable at 100+ tokens per document

• Results are cached when cache_key is provided for repeated analysis

Value

A list containing:

• lexical_diversity: Data frame with per-document lexical diversity scores

• summary_stats: List of summary statistics (mean, median, sd) for each measure

References

McCarthy, P. M., & Jarvis, S. (2010). MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods, 42(2), 381-392.

See Also

calculate_text_readability() for grade-level / Flesch metrics on the same input; calculate_lexical_dispersion() for term spread across documents; plot_lexical_diversity_distribution() to visualize

Examples

data(SpecialEduTech)
texts <- SpecialEduTech$abstract[1:10]
corp <- quanteda::corpus(texts)
toks <- quanteda::tokens(corp)
# Preferred: pass tokens object for accurate MTLD
lex_div <- lexical_diversity_analysis(toks, texts = texts)
# With caching for repeated analysis
cache_key <- digest::digest(texts)
lex_div <- lexical_diversity_analysis(toks, texts = texts, cache_key = cache_key)
# Alternative: pass DFM with texts for MTLD accuracy
dfm_obj <- quanteda::dfm(toks)
lex_div <- lexical_diversity_analysis(dfm_obj, texts = texts)
print(lex_div)

Lexical Frequency Analysis

Description

Wrapper function for plot_word_frequency for lexical analysis.

Usage

lexical_frequency_analysis(...)

Arguments

Value

A plotly bar chart of word frequencies

Log Security Event

Description

Log Security Event

Usage

log_security_event(event_type, details, session_info, level = "info")

Arguments

Parse Morphology String

Description

Parse spaCy's morphology string format into individual columns. Used internally by morphology analysis functions. Always extracts all common morphology features (Number, Tense, VerbForm, Person, Case, Mood, Aspect) regardless of the features parameter.

Usage

parse_morphology_string(data, features = NULL)

Arguments

Value

Data frame with additional morph_* columns for each feature.

Plot Cluster Top Terms

Description

Creates a horizontal bar plot showing the top terms in a cluster or document group.

Usage

plot_cluster_terms(
  terms,
  cluster_id = NULL,
  title = NULL,
  n_terms = 10,
  color = "#337ab7",
  height = 500,
  width = NULL
)

Arguments

Value

A plotly object

Plot Cross-Category Similarity Comparison

Description

Creates a faceted ggplot heatmap for cross-category document similarity comparison. Accepts either a pre-built long-format data frame or extracts from a similarity matrix.

Usage

plot_cross_category_heatmap(
  similarity_data,
  docs_data = NULL,
  row_var = "ld_doc_name",
  col_var = "other_doc_name",
  value_var = "cosine_similarity",
  category_var = "other_category",
  row_category = NULL,
  col_categories = NULL,
  row_display_var = NULL,
  col_display_var = NULL,
  method_name = "Cosine",
  title = NULL,
  show_values = TRUE,
  row_label = "Documents",
  label_max_chars = 25,
  order_by_numeric = TRUE,
  height = 600,
  width = NULL
)

Arguments

Value

A ggplot object

Examples

articles <- TextAnalysisR::SpecialEduTech[1:7, ]
term_matrix <- as.matrix(quanteda::dfm(quanteda::tokens(articles$abstract)))
normalized_matrix <- term_matrix / sqrt(rowSums(term_matrix ^ 2))
similarity_matrix <- normalized_matrix %*% t(normalized_matrix)
thesis_rows <- which(articles$reference_type == "thesis")[1:3]
article_cols <- which(articles$reference_type == "journal_article")[1:4]
similarity_data <- expand.grid(
  ld_doc_name    = paste("Thesis", seq_along(thesis_rows)),
  other_doc_name = paste("Article", seq_along(article_cols)),
  stringsAsFactors = FALSE
)
similarity_data$cosine_similarity <- as.vector(
  similarity_matrix[thesis_rows, article_cols]
)
similarity_data$other_category <- articles$reference_type[article_cols]
plot_cross_category_heatmap(
  similarity_data = similarity_data,
  row_var = "ld_doc_name",
  col_var = "other_doc_name",
  value_var = "cosine_similarity",
  category_var = "other_category",
  row_label = "Theses"
)

Plot Document Sentiment Trajectory

Description

Creates a line chart showing sentiment scores across documents with color gradient.

Usage

plot_document_sentiment_trajectory(
  sentiment_data,
  top_n = NULL,
  doc_ids = NULL,
  text_preview = NULL,
  title = "Document Sentiment Scores"
)

Arguments

Value

A ggplot2 line chart with color gradient

Plot Emotion Radar Chart

Description

Creates a polar/radar chart for NRC emotion analysis with optional grouping.

Usage

plot_emotion_radar(
  emotion_data,
  group_var = NULL,
  normalize = FALSE,
  title = "Emotion Analysis"
)

Arguments

Value

A ggplot2 polar chart

Plot Named Entity Frequencies

Description

Creates a bar plot showing the frequency distribution of named entity types.

Usage

plot_entity_frequencies(
  entity_data,
  top_n = 20,
  title = "Named Entity Type Frequency",
  color = NULL,
  height = 500,
  width = NULL,
  custom_colors = NULL
)

Arguments

Value

A plotly object

Examples

if (interactive()) {
  entity_df <- data.frame(
    entity = c("PERSON", "ORG", "GPE", "DATE", "MONEY"),
    n = c(300, 250, 200, 150, 100)
  )
  plot_entity_frequencies(entity_df)

  # With custom colors
  plot_entity_frequencies(entity_df, custom_colors = c(PERSON = "#ff0000"))
}

Create Error Plot for Plotly

Description

Creates a plotly error/status plot with a message

Usage

plot_error(message, color = "#ef4444")

Arguments

Value

A plotly plot object displaying the message

Plot Statistical Keyness

Description

Creates a horizontal bar plot of distinctive keywords by keyness score.

Usage

plot_keyness_keywords(keyness_data, title = NULL, group_label = NULL)

Arguments

Value

A plotly bar chart

Plot Keyword Comparison (TF-IDF vs Frequency)

Description

Creates a grouped bar plot comparing TF-IDF scores with term frequencies.

Usage

plot_keyword_comparison(
  tfidf_data,
  top_n = 10,
  title = NULL,
  normalized = FALSE
)

Arguments

Value

A plotly grouped bar chart

Plot Lexical Dispersion

Description

Creates an X-ray plot showing where terms appear across documents. Each row represents a term, and marks indicate occurrences.

Usage

plot_lexical_dispersion(
  dispersion_data,
  scale = "relative",
  title = "Lexical Dispersion",
  colors = NULL,
  height = 400,
  width = NULL,
  marker_size = 8
)

Arguments

Value

A plotly object

Examples

tokens <- quanteda::tokens(TextAnalysisR::SpecialEduTech$abstract[1:5])
dispersion <- calculate_lexical_dispersion(tokens, c("learning", "instruction"))
plot_lexical_dispersion(dispersion)

Plot Lexical Diversity Distribution

Description

Creates a boxplot showing the distribution of a lexical diversity metric.

Usage

plot_lexical_diversity_distribution(lexdiv_data, metric, title = NULL)

Arguments

Value

A plotly boxplot

Examples

abstracts <- TextAnalysisR::SpecialEduTech$abstract[1:10]
tokens <- quanteda::tokens(quanteda::corpus(abstracts))
diversity_result <- lexical_diversity_analysis(tokens, texts = abstracts)
diversity_plot <- plot_lexical_diversity_distribution(
  diversity_result$lexical_diversity, "MTLD"
)
print(diversity_plot)

Plot Log Odds Ratio

Description

Creates a horizontal bar plot showing log odds ratios for comparing word usage between categories. Positive values indicate higher usage in the first category, negative in the second.

Usage

plot_log_odds_ratio(
  log_odds_data,
  top_n = 10,
  facet_by = NULL,
  color_positive = "#10B981",
  color_negative = "#EF4444",
  height = 600,
  width = NULL,
  title = "Log Odds Ratio Comparison"
)

Arguments

Value

A plotly object

Examples

articles <- TextAnalysisR::SpecialEduTech[1:20, ]
corpus <- quanteda::corpus(
  articles$abstract,
  docvars = data.frame(reference_type = articles$reference_type)
)
dfm_object <- quanteda::dfm(quanteda::tokens(corpus))
log_odds <- calculate_log_odds_ratio(dfm_object, "reference_type",
                                      comparison_mode = "binary")
plot_log_odds_ratio(log_odds, top_n = 5)

Plot Topic Model Comparison Scatter

Description

Creates a scatter plot comparing topic model metrics across K values. Automatically selects the best available metric combination.

Usage

plot_model_comparison(search_results, title = NULL, height = 600, width = 800)

Arguments

Value

A plotly scatter plot

Plot Morphology Feature Distribution

Description

Creates a bar chart showing the distribution of a morphological feature using consistent package styling.

Usage

plot_morphology_feature(data, feature, title = NULL, colors = NULL)

Arguments

Value

A plotly object.

Examples

if (interactive()) {
  tokens <- quanteda::tokens(TextAnalysisR::SpecialEduTech$abstract[1])
  morphology_data <- extract_morphology(tokens)
  plot_morphology_feature(morphology_data, "Tense")
}

Plot Multi-Word Expression Frequency

Description

Creates a bar plot showing multi-word expression frequencies with optional source-based coloring to distinguish between detected and manually added expressions.

Usage

plot_mwe_frequency(
  mwe_data,
  title = "Multi-Word Expression Frequency",
  color_by_source = TRUE,
  primary_color = "#10B981",
  secondary_color = "#A855F7",
  height = 500,
  width = NULL
)

Arguments

Value

A plotly object

Plot N-gram Frequency

Description

Creates a bar plot showing n-gram frequencies with optional highlighting of selected n-grams. Supports both detected n-grams and selected multi-word expressions.

Usage

plot_ngram_frequency(
  ngram_data,
  top_n = 30,
  selected = NULL,
  title = "N-gram Frequency",
  highlight_color = "#10B981",
  default_color = "#6B7280",
  height = 500,
  width = NULL,
  show_stats = TRUE
)

Arguments

Value

A plotly object

Examples

  ngram_df <- data.frame(
    collocation = c("machine learning", "deep learning", "neural network"),
    count = c(150, 120, 90),
    lambda = c(5.2, 4.8, 4.1),
    z = c(12.3, 10.5, 9.2)
  )
  plot_ngram_frequency(ngram_df, selected = c("machine learning"))

Plot Part-of-Speech Tag Frequencies

Description

Creates a bar plot showing the frequency distribution of part-of-speech tags.

Usage

plot_pos_frequencies(
  pos_data,
  top_n = 20,
  title = "Part-of-Speech Tag Frequency",
  color = "#337ab7",
  height = 500,
  width = NULL
)

Arguments

Value

A plotly object

Examples

if (interactive()) {
  pos_df <- data.frame(
    pos = c("NOUN", "VERB", "ADJ", "ADV", "PRON"),
    n = c(500, 400, 250, 150, 100)
  )
  plot_pos_frequencies(pos_df)
}

Plot Topic Model Quality Metrics

Description

Creates individual diagnostic metric plots across different K values from stm::searchK results.

Usage

plot_quality_metrics(search_results)

Arguments

Value

A named list of ggplot objects, one per available metric (possible keys: semcoh, residual, heldout, lbound).

Plot Readability by Group

Description

Creates grouped boxplots comparing readability across categories.

Usage

plot_readability_by_group(readability_data, metric, group_var, title = NULL)

Arguments

Value

A plotly boxplot

Plot Readability Distribution

Description

Creates a boxplot showing the overall distribution of a readability metric.

Usage

plot_readability_distribution(readability_data, metric, title = NULL)

Arguments

Value

A plotly boxplot

Examples

data(SpecialEduTech, package = "TextAnalysisR")
texts <- SpecialEduTech$abstract[1:20]
readability <- calculate_text_readability(texts)
plot <- plot_readability_distribution(readability, "flesch")
print(plot)

Plot Semantic Analysis Visualization

Description

Creates interactive visualizations for semantic analysis results including similarity heatmaps, dimensionality reduction plots, and clustering visualizations.

Usage

plot_semantic_viz(
  analysis_result = NULL,
  plot_type = "similarity",
  data_labels = NULL,
  color_by = NULL,
  height = 600,
  width = 800,
  title = NULL,
  coords = NULL,
  clusters = NULL,
  hover_text = NULL,
  hover_config = NULL,
  cluster_colors = NULL
)

Arguments

Value

A ggplot2 object showing the specified visualization.

Examples

  data(SpecialEduTech)
  texts <- SpecialEduTech$abstract[1:5]
  result <- semantic_similarity_analysis(texts)
  plot <- plot_semantic_viz(result, plot_type = "similarity")
  print(plot)

Plot Sentiment Box Plot by Category

Description

Creates a box plot showing sentiment score distribution by category.

Usage

plot_sentiment_boxplot(
  sentiment_data,
  category_var = "category_var",
  title = "Sentiment Score Distribution"
)

Arguments

Value

A ggplot2 box plot

Plot Sentiment by Category

Description

Creates a grouped or stacked bar plot showing sentiment distribution across categories.

Usage

plot_sentiment_by_category(
  sentiment_data,
  category_var,
  plot_type = "bar",
  title = NULL
)

Arguments

Value

A ggplot2 grouped/stacked bar chart

Examples

articles <- TextAnalysisR::SpecialEduTech[1:20, ]
sentiment_results <- analyze_sentiment(articles$abstract)
sentiment_data <- cbind(
  reference_type = articles$reference_type,
  sentiment_results
)
sentiment_plot <- plot_sentiment_by_category(sentiment_data, "reference_type")
print(sentiment_plot)

Plot Sentiment Distribution

Description

Creates a bar plot showing the distribution of sentiment classifications.

Usage

plot_sentiment_distribution(sentiment_data, title = "Sentiment Distribution")

Arguments

Value

A ggplot2 bar chart

Examples

abstracts <- TextAnalysisR::SpecialEduTech$abstract[1:10]
sentiment_data <- analyze_sentiment(abstracts)
sentiment_plot <- plot_sentiment_distribution(sentiment_data)
print(sentiment_plot)

Plot Sentiment Violin Plot by Category

Description

Creates a violin plot showing sentiment score distribution by category.

Usage

plot_sentiment_violin(
  sentiment_data,
  category_var = "category_var",
  title = "Sentiment Score Distribution"
)

Arguments

Value

A ggplot2 violin plot

Plot Document Similarity Heatmap

Description

Creates an interactive heatmap visualization of document similarity matrices with support for document metadata, feature-specific colorscales, and rich tooltips. Supports both symmetric (all-vs-all) and cross-category comparison modes.

Usage

plot_similarity_heatmap(
  similarity_matrix,
  docs_data = NULL,
  feature_type = "words",
  method_name = "Cosine",
  title = NULL,
  category_filter = NULL,
  doc_id_var = NULL,
  colorscale = NULL,
  height = 600,
  width = NULL,
  row_category = NULL,
  col_categories = NULL,
  category_var = "category_display",
  show_values = FALSE,
  facet = NULL,
  row_label = NULL,
  output_type = "plotly"
)

Arguments

Value

A ggplot2 heatmap object

Examples

articles <- TextAnalysisR::SpecialEduTech[1:5, ]
term_matrix <- as.matrix(quanteda::dfm(quanteda::tokens(articles$abstract)))
normalized_matrix <- term_matrix / sqrt(rowSums(term_matrix ^ 2))
similarity_matrix <- normalized_matrix %*% t(normalized_matrix)
plot_similarity_heatmap(similarity_matrix)

document_metadata <- data.frame(
  document_number     = paste("Doc", 1:5),
  document_id_display = articles$title,
  category_display    = articles$reference_type
)
plot_similarity_heatmap(similarity_matrix, docs_data = document_metadata,
                        feature_type = "embeddings")

plot_similarity_heatmap(
  similarity_matrix,
  docs_data      = document_metadata,
  row_category   = "thesis",
  col_categories = "journal_article",
  show_values    = TRUE,
  facet          = TRUE
)

Plot Term Frequency Trends by Continuous Variable

Description

Creates a faceted line plot showing how term frequencies vary across a continuous variable (e.g., year, time period).

Usage

plot_term_trends_continuous(
  term_data,
  continuous_var,
  terms = NULL,
  title = NULL,
  height = 600,
  width = NULL
)

Arguments

Value

A plotly object with faceted line plots

Examples

term_df <- data.frame(
  year = rep(2010:2020, each = 3),
  term = rep(c("learning", "education", "technology"), 11),
  word_frequency = sample(10:100, 33, replace = TRUE)
)
plot_term_trends_continuous(term_df, "year", c("learning", "education"))

Plot TF-IDF Keywords

Description

Creates a horizontal bar plot of top keywords by TF-IDF score.

Usage

plot_tfidf_keywords(tfidf_data, title = NULL, normalized = FALSE)

Arguments

Value

A plotly bar chart

Plot Top Documents by Readability

Description

Creates a bar plot of documents ranked by readability metric.

Usage

plot_top_readability_documents(
  readability_data,
  metric,
  top_n = 15,
  order = "highest",
  title = NULL
)

Arguments

Value

A plotly bar chart

Plot Topic Effects for Categorical Variables

Description

Creates a faceted plot showing how categorical variables affect topic proportions.

Usage

plot_topic_effects_categorical(
  effects_data,
  ncol = 2,
  height = 800,
  width = 1000,
  title = "Category Effects",
  base_font_size = 11
)

Arguments

Value

A plotly object

Plot Topic Effects for Continuous Variables

Description

Creates a faceted plot showing how continuous variables affect topic proportions.

Usage

plot_topic_effects_continuous(
  effects_data,
  ncol = 2,
  height = 800,
  width = 1000,
  title = "Continuous Variable Effects",
  base_font_size = 11
)

Arguments

Value

A plotly object

Plot Per-Document Per-Topic Probabilities

Description

Generates a bar plot showing the prevalence of each topic across all documents.

Usage

plot_topic_probability(
  gamma_data,
  top_n = 10,
  use_topic_labels = FALSE,
  colors = NULL,
  ylab = "Topic Proportion",
  base_font_size = 11
)

Arguments

Value

A ggplot2 object showing a bar plot of topic prevalence.

Plot Weighted Log Odds

Description

Creates a faceted horizontal bar plot showing weighted log odds for comparing word usage across categories using the Fightin' Words method (Monroe et al. 2008). Each group is displayed in a separate facet showing its most distinctive terms.

Usage

plot_weighted_log_odds(
  weighted_data,
  top_n = 10,
  color_positive = "#10B981",
  color_negative = "#EF4444",
  height = 600,
  width = NULL,
  title = "Weighted Log Odds by Group"
)

Arguments

Value

A plotly object

Examples

if (requireNamespace("tidylo", quietly = TRUE)) {
  articles <- TextAnalysisR::SpecialEduTech[1:20, ]
  dfm_object <- quanteda::dfm(quanteda::tokens(articles$abstract))
  quanteda::docvars(dfm_object, "reference_type") <- articles$reference_type
  weighted_odds <- calculate_weighted_log_odds(dfm_object, "reference_type",
                                                top_n = 5)
  plot_weighted_log_odds(weighted_odds)
}

Plot Word Frequency

Description

Creates a bar plot showing the most frequent words in a document-feature matrix (dfm).

Usage

plot_word_frequency(dfm_object, n = 20, height = NULL, width = NULL, ...)

Arguments

Value

A ggplot object showing word frequency.

Examples

  data(SpecialEduTech, package = "TextAnalysisR")
  texts <- SpecialEduTech$abstract[1:10]
  dfm <- quanteda::dfm(quanteda::tokens(texts))
  plot <- plot_word_frequency(dfm, n = 10)
  print(plot)

Plot Word Probabilities by Topic

Description

Creates a faceted bar plot showing the top terms and their probabilities (beta values) for each topic in a topic model.

Usage

plot_word_probability(
  top_topic_terms,
  topic_label = NULL,
  ncol = 3,
  height = 1200,
  width = 800,
  ylab = "Word probability",
  title = NULL,
  colors = NULL,
  measure_label = "Beta",
  base_font_size = 11,
  ...
)

Arguments

Value

A ggplot2 object showing word probabilities faceted by topic.

Preprocess Text Data

Description

Preprocesses text data following the complete workflow implemented in the Shiny application:

• Constructing a corpus from united texts

• Tokenizing text into words with configurable options

• Converting to lowercase with acronym preservation option

• Applying character length filtering

• Optional multi-word expression detection and compound term creation

• Stopword removal and lemmatization capabilities

This function serves as the foundation for all subsequent text analysis workflows.

Usage

prep_texts(
  united_tbl,
  text_field = "united_texts",
  min_char = 2,
  lowercase = TRUE,
  remove_punct = TRUE,
  remove_symbols = TRUE,
  remove_numbers = TRUE,
  remove_url = TRUE,
  remove_separators = TRUE,
  split_hyphens = TRUE,
  split_tags = TRUE,
  include_docvars = TRUE,
  keep_acronyms = FALSE,
  padding = FALSE,
  remove_stopwords = FALSE,
  stopwords_source = "snowball",
  stopwords_language = "en",
  custom_stopwords = NULL,
  custom_valuetype = "glob",
  math_mode = FALSE,
  verbose = FALSE,
  ...
)

Arguments

Value

A tokens object that contains the preprocessed text data.

See Also

unite_cols() to combine text columns first; lemmatize_tokens() to reduce words to base form (e.g., running -> run); quanteda::dfm() to build a document-feature matrix from the result

Examples

mydata <- TextAnalysisR::SpecialEduTech

united_tbl <- TextAnalysisR::unite_cols(
  mydata,
  listed_vars = c("title", "keyword", "abstract")
)

tokens <- TextAnalysisR::prep_texts(united_tbl,
                                         text_field = "united_texts",
                                         min_char = 2,
                                         lowercase = TRUE,
                                         remove_punct = TRUE,
                                         remove_symbols = TRUE,
                                         remove_numbers = TRUE,
                                         remove_url = TRUE,
                                         remove_separators = TRUE,
                                         split_hyphens = TRUE,
                                         split_tags = TRUE,
                                         include_docvars = TRUE,
                                         keep_acronyms = FALSE,
                                         padding = FALSE,
                                         verbose = FALSE)
print(tokens)

Process PDF File

Description

Main function to process PDF files - extracts text content using pdftools. For table extraction, use process_pdf_file_py.

Usage

process_pdf_file(file_path, content_type = "auto")

Arguments

Details

This function extracts text content from PDFs using pdftools package. Works best with text-based PDFs (not scanned images).

For PDFs containing tables or complex layouts, use the Python-based process_pdf_file_py which provides better table extraction.

Value

List with:

• data: Data frame with extracted content

• type: Character string indicating content type ("text" or "error")

• success: Logical indicating success

• message: Character string with status message

Process PDF File using Python

Description

Main function to process PDF files using pdfplumber (Python). Automatically detects content type and extracts data accordingly. No Java required.

Usage

process_pdf_file_py(file_path, content_type = "auto", envname = NULL)

Arguments

Details

This function uses Python's pdfplumber library which:

• Handles both text and tables

• No Java dependency

• Better accuracy than tabulizer for complex tables

• Uses TextAnalysisR Python environment

Value

List with:

• data: Data frame with extracted content

• type: Character string indicating content type

• success: Logical indicating success

• message: Character string with status message

Process PDF File (Unified Entry Point)

Description

Unified PDF processing:

1. Multimodal (R-native pdftools + Vision LLM) if enabled

2. R pdftools text extraction as fallback

Usage

process_pdf_unified(
  file_path,
  use_multimodal = FALSE,
  vision_provider = "gemini",
  vision_model = NULL,
  api_key = NULL,
  describe_images = TRUE
)

Arguments

Value

List: success, data, type, method, message

See Also

import_files() for the generic file-loading entry point; describe_image() for standalone image captioning

Dimensionality Reduction Analysis

Description

This function performs dimensionality reduction using various methods including PCA, t-SNE, and UMAP. UMAP runs directly on inputs with 1000 or fewer features (e.g. dense embeddings); PCA pre-reduction applies to wider inputs and to the PCA and t-SNE methods.

Usage

reduce_dimensions(
  data_matrix,
  method = "PCA",
  n_components = 2,
  pca_dims = 50,
  tsne_perplexity = 30,
  tsne_max_iter = 1000,
  umap_neighbors = 15,
  umap_min_dist = 0.1,
  umap_metric = "cosine",
  seed = 123,
  verbose = TRUE
)

Arguments

Value

A list containing the reduced dimensions, method used, and additional metadata.

Examples

if (interactive()) {
  mydata <- TextAnalysisR::SpecialEduTech

  united_tbl <- TextAnalysisR::unite_cols(
    mydata,
    listed_vars = c("title", "keyword", "abstract")
  )

  tokens <- TextAnalysisR::prep_texts(united_tbl, text_field = "united_texts")

  dfm_object <- quanteda::dfm(tokens)

  data_matrix <- as.matrix(dfm_object)

  pca_result <- TextAnalysisR::reduce_dimensions(
    data_matrix,
    method = "PCA"
  )
  print(pca_result)

  tsne_result <- TextAnalysisR::reduce_dimensions(
    data_matrix,
    method = "t-SNE"
  )
  print(tsne_result)

  umap_result <- TextAnalysisR::reduce_dimensions(
    data_matrix,
    method = "UMAP"
  )
  print(umap_result)
}

Remove Shiny notification by id

Description

Remove Shiny notification by id

Usage

remove_notification_by_id(id, session = NULL)

Arguments

Value

Invisibly NULL.

Render displaCy Dependency Visualization

Description

Renders spaCy's displaCy dependency visualization as SVG. Shows syntactic structure with arrows between words.

Usage

render_displacy_dep(text, compact = TRUE, model = "en_core_web_sm")

Arguments

Value

SVG string with dependency tree.

Render displaCy Entity Visualization

Description

Renders spaCy's displaCy entity visualization as HTML. Highlights named entities with colored labels.

Usage

render_displacy_ent(text, model = "en_core_web_sm", colors = NULL)

Arguments

Value

HTML string with entity highlighting.

Render PDF Pages to Base64 PNG

Description

Renders each page of a PDF as a PNG image and returns base64-encoded strings. Uses pdftools (R-native, no Python required).

Usage

render_pdf_pages_to_base64(file_path, dpi = 150)

Arguments

Value

List of base64-encoded PNG strings, one per page

Renumber Clusters Sequentially

Description

Renumbers cluster assignments to sequential integers starting from 1.

Usage

renumber_clusters_sequentially(clusters)

Arguments

Value

Vector with clusters renumbered sequentially (1, 2, 3, ...).

Require Feature

Description

Checks feature availability and shows notification if unavailable.

Usage

require_feature(feature, session = NULL)

Arguments

Value

Logical TRUE if available, FALSE if not

Launch the TextAnalysisR app

Description

Launch the TextAnalysisR Shiny application.

Usage

run_app(launch.browser = interactive())

Arguments

Value

No return value, called for side effects (launching Shiny app).

Examples

if (interactive()) {
  run_app()
}

Neural Topic Modeling

Description

Implements neural topic modeling using deep learning architectures for improved topic discovery and representation learning.

Usage

run_neural_topics_internal(
  texts,
  n_topics = 10,
  hidden_layers = 2,
  hidden_units = 100,
  dropout_rate = 0.2,
  embedding_model = "all-MiniLM-L6-v2",
  seed = 123
)

Arguments

Value

List containing neural topic model and diagnostics

RAG Semantic Search

Description

Simple in-memory RAG (Retrieval Augmented Generation) for question-answering over document corpus with source attribution. Uses OpenAI or Gemini embeddings for semantic search and LLM for answer generation.

Usage

run_rag_search(
  query,
  documents,
  provider = c("openai", "gemini"),
  api_key = NULL,
  embedding_model = NULL,
  chat_model = NULL,
  top_k = 5
)

Arguments

Details

Simple RAG workflow:

1. Generate embeddings for documents and query

2. Find top-k similar documents via cosine similarity

3. Generate answer using LLM with retrieved context

Value

List with:

• success: Logical

• answer: Generated answer

• confidence: Confidence score (0-1)

• sources: Vector of source document indices

• retrieved_docs: Retrieved document chunks

• scores: Similarity scores

See Also

get_best_embeddings() for the retrieval step alone; call_llm_api() for the answer-generation step alone; sanitize_llm_input() for an input safety check before calling

Examples

if (interactive()) {
documents <- c(
  "Assistive technology helps students with disabilities access curriculum.",
  "Universal Design for Learning provides multiple means of engagement.",
  "Response to Intervention uses tiered support systems."
)

# Using OpenAI (requires API key)
result <- run_rag_search(
  query = "How does assistive technology support learning?",
  documents = documents,
  provider = "openai"
)

if (result$success) {
  cat("Answer:", result$answer, "\n")
  cat("Sources:", paste(result$sources, collapse = ", "), "\n")
}
}

Complete Text Mining Workflow

Description

This function provides a complete text mining workflow that follows the same sequence as the Shiny application: file processing -> text uniting -> preprocessing -> DFM creation -> analysis. It serves as a convenience function for users who want to execute the entire pipeline programmatically.

Usage

run_text_workflow(
  dataset_choice,
  file_info = NULL,
  text_input = NULL,
  listed_vars,
  min_char = 2,
  remove_punct = TRUE,
  remove_symbols = TRUE,
  remove_numbers = TRUE,
  remove_url = TRUE,
  detect_compounds = FALSE,
  compound_size = 2:3,
  compound_min_count = 2,
  verbose = TRUE
)

Arguments

Value

A list containing processed data, tokens, DFM, and metadata.

Examples

  workflow_result <- TextAnalysisR::run_text_workflow(
    dataset_choice = "Upload an Example Dataset",
    listed_vars = c("title", "keyword", "abstract")
  )

if (interactive()) {
  file_info <- data.frame(filepath = "path/to/file.xlsx")
  workflow_result <- TextAnalysisR::run_text_workflow(
    dataset_choice = "Upload Your File",
    file_info = file_info,
    listed_vars = c("column1", "column2")
  )

  workflow_result <- TextAnalysisR::run_text_workflow(
    dataset_choice = "Copy and Paste Text",
    text_input = "Your text content here",
    listed_vars = "text"
  )
}

Sanitize LLM Input

Description

Sanitizes user input before inclusion in LLM prompts to mitigate prompt injection attacks. Filters common injection patterns such as instruction overrides, system prompt markers, and role-switching attempts. Distinct from sanitize_text_input() which targets XSS.

Usage

sanitize_llm_input(text, max_length = 2000)

Arguments

Value

Sanitized character string

NIST Compliance

Implements NIST SI-10 (Information Input Validation) for AI/LLM contexts.

Sanitize Text Input

Description

Sanitize Text Input

Usage

sanitize_text_input(text)

Arguments

Value

Sanitized text

Semantic Document Clustering

Description

Creates a unified visualization of document clustering with optional clustering

Usage

semantic_document_clustering(embeddings, method = "UMAP", clusters = NULL, ...)

Arguments

Value

A ggplot2 visualization of document clusters

Semantic Similarity Analysis

Description

Wrapper for calculate_document_similarity

Usage

semantic_similarity_analysis(...)

Arguments

Value

Similarity analysis results

See Also

calculate_document_similarity() for the core computation; plot_similarity_heatmap() to render the resulting matrix

Embedding-based Sentiment Analysis

Description

Performs sentiment analysis using transformer-based embeddings and neural models. This approach uses pre-trained language models for contextual sentiment detection without requiring sentiment lexicons. Particularly effective for handling:

• Complex contextual sentiment

• Implicit sentiment and sarcasm

• Domain-specific sentiment

• Negation and intensifiers (automatically handled by the model)

Usage

sentiment_embedding_analysis(
  texts,
  embeddings = NULL,
  model_name = "distilbert-base-uncased-finetuned-sst-2-english",
  doc_names = NULL,
  use_gpu = FALSE
)

Arguments

Value

A list containing:

document_sentiment

Data frame with document-level sentiment scores and classifications

emotion_scores

NULL (emotion detection not currently supported for embeddings)

summary_stats

Summary statistics including document counts and average scores

model_used

Name of the transformer model used

feature_type

"embeddings"

Examples

if (interactive()) {
  abstracts <- TextAnalysisR::SpecialEduTech$abstract[1:10]
  embedding_sentiment <- sentiment_embedding_analysis(abstracts)
  print(embedding_sentiment$document_sentiment)
  print(embedding_sentiment$summary_stats)
}

Analyze Sentiment Using Tidytext Lexicons

Description

Performs lexicon-based sentiment analysis on a DFM object using tidytext lexicons. Supports AFINN, Bing, and NRC lexicons with scoring and emotion analysis.

Tokens absent from the lexicon are ignored (not treated as neutral); summary_stats$token_match_rate reports the share of corpus tokens the lexicon covered. AFINN classification uses a +/- 0.5 band on the per-sentiment-word average (avg_sentiment); its n_sentiment_words column counts matched sentiment tokens, not document length.

Usage

sentiment_lexicon_analysis(
  dfm_object,
  lexicon = "bing",
  texts_df = NULL,
  feature_type = "words",
  ngram_range = 2,
  texts = NULL
)

Arguments

Value

A list containing:

document_sentiment

Data frame with sentiment scores per document

emotion_scores

Data frame with emotion scores (NRC only)

summary_stats

List of summary statistics

feature_type

Feature type used for analysis

Examples

abstracts <- TextAnalysisR::SpecialEduTech$abstract[1:10]
corpus <- quanteda::corpus(abstracts)
dfm_object <- quanteda::dfm(quanteda::tokens(corpus))
lexicon_results <- sentiment_lexicon_analysis(dfm_object, lexicon = "bing")
print(lexicon_results$document_sentiment)

Setup Python Environment

Description

Sets up a tiered Python virtual environment.

Usage

setup_python_env(envname = "textanalysisr-env", tier = "core", force = FALSE)

Arguments

Details

Default tier = "core" keeps the install light – spaCy NLP and PDF text extraction only. Add "embeddings" for sentence-transformers-based similarity/sentiment, and "topics" for BERTopic.

The virtual environment is isolated; system Python is not modified.

Value

Invisible TRUE if successful, stops with error message if failed

Examples

if (interactive()) {
  setup_python_env()                              # core only (~200 MB)
  setup_python_env(tier = c("core", "embeddings"))  # +1 GB
  setup_python_env(tier = c("core", "embeddings", "topics"))  # full stack
}

Show completion notification

Description

Show completion notification

Usage

show_completion_notification(message, duration = 5, session = NULL)

Arguments

Value

Invisibly the notification id.

Show DFM instructions modal

Description

Show DFM instructions modal

Usage

show_dfm_instructions_modal(
  output_id,
  feature_name = "this feature",
  session = NULL
)

Arguments

Value

Invisibly NULL.

Show DFM-required modal

Description

Show DFM-required modal

Usage

show_dfm_required_modal(
  feature_name = "this feature",
  additional_message = NULL
)

Arguments

Value

Invisibly NULL.

Show error notification

Description

Show error notification

Usage

show_error_notification(message, duration = 7, session = NULL)

Arguments

Value

Invisibly the notification id.

Show guide modal

Description

Show guide modal

Usage

show_guide_modal(guide_name, title, size = "l")

Arguments

Value

Invisibly NULL.

Show persistent loading notification

Description

Show persistent loading notification

Usage

show_loading_notification(message, id = NULL, session = NULL)

Arguments

Value

Invisibly the notification id.

Show no-DFM notification

Description

Show no-DFM notification

Usage

show_no_dfm_notification(feature_name = "this feature", duration = 7)

Arguments

Value

Invisibly the notification id.

Show no-feature-matrix notification

Description

Show no-feature-matrix notification

Usage

show_no_feature_matrix_notification(duration = 7)

Arguments

Value

Invisibly the notification id.

Show preprocessing-required modal

Description

Show preprocessing-required modal

Usage

show_preprocessing_required_modal(
  message = "Please complete preprocessing steps first.",
  title = "Preprocessing Required"
)

Arguments

Value

Invisibly NULL.

Show preprocessing-steps modal

Description

Show preprocessing-steps modal

Usage

show_preprocessing_steps_modal(
  title = "Preprocessing Required",
  message,
  required_steps,
  optional_steps = NULL,
  additional_note = NULL
)

Arguments

Value

Invisibly NULL.

Show united-texts-required notification

Description

Show united-texts-required notification

Usage

show_unite_texts_required_notification(duration = 5)

Arguments

Value

Invisibly the notification id.

Show warning notification

Description

Show warning notification

Usage

show_warning_notification(message, duration = 5, session = NULL)

Arguments

Value

Invisibly the notification id.

Show Web Deployment Banner

Description

Show Web Deployment Banner

Usage

show_web_banner(disabled = NULL)

Arguments

Value

A shiny.tag object containing the banner HTML for inclusion in a Shiny UI, or NULL when not running in a web deployment context.

Extract Named Entities with spaCy

Description

Extract named entities from texts using spaCy NER.

Usage

spacy_extract_entities(x, model = "en_core_web_sm")

Arguments

Value

A data frame with entity information:

• doc_id: Document identifier

• text: Entity text

• label: Entity type (PERSON, ORG, GPE, etc.)

• start_char: Start character position

• end_char: End character position

Check if Model Has Word Vectors

Description

Check if the loaded spaCy model has word vectors for similarity calculations.

Usage

spacy_has_vectors()

Value

Logical; TRUE if model has vectors, FALSE otherwise.

Check if spaCy is Initialized

Description

Check whether spaCy has been initialized.

Usage

spacy_initialized()