---
title: "Validation of self-stigma questionnaires"
author:
  - name: Josef Mana
    orcid: 0000-0002-7817-3978
    affiliations:
      - name: Institute of Psychology, Czech Academy of Sciences
        department: Department of Cognitive Psychology
        address: Pod Vodárenskou věží 4
        city: Prague
        country: Czech Republic
        postal-code: 18200
format:
  apaquarto-html:
    css: muni-style.css
self-contained: true
warning: false
bibliography: references.bib
---

# Methods

## Statistical analyses

Data were analyzed using R (version `r with(version, paste(major, minor, sep = '.'))`) [@R]. The analysis pipeline was managed via the _targets_ package [@targets] to ensure reproducibility. Full analysis code is available at <https://github.com/josefmana/stighelp.git>

All demographic and outcome variables were described using contingency tables for nominal and ordinal (gender, education level, town size, past therapy) data and sample mean ± standard deviation for continuous variables (age and questionnaires' sum scores).

Descriptive correlation analyses were conducted using polychoric correlations for ordinal variables (education level, town size, single item scores), biserial correlations for dichotomous variables (gender, past therapy), Pearson correlations for continuous variables (Age) and Spearman correlation for questionnaires' sum scores, as implemented in `mixedCor()` function from the R package *psych* [@psych].

We evaluated internal consistency using Cronbach’s $\alpha$ and Guttman’s $\lambda_6$ coefficients computed via the _psych_ R package [@psych]. Values between 0.70 and 0.80 were considered adequate for early research stages, values above 0.80 to show overall adequate internal consistency, and values above 0.90 to indicate redundancy between items [@nunnally1994; @streiner2003]. To complement $\alpha$, which is affected by scale lengths, we inspected average inter-item correlations. Values between 0.15–0.20 were interpreted as indicating adequate internal consistency for broad constructs (BHSS total score, ATSPPH-SF) and 0.40–0.50 for narrow constructs (BHSS subscales, PSOSH, SSOSH) [@streiner2003].

To evaluate evidence of construct validity, we fitted a series of Confirmatory Factor Analyses (CFA) using the _lavaan_ R package [@lavaan]. Given the ordinal nature of the items, we employed the diagonally weighted least squares (WLSMV) estimator with theta parametrization, as this approach is more robust for categorical data than traditional Maximum Likelihood [@li2015]. We tested unidimensional models for the PSOSH, SSOSH and ATSPPH-SF. For the BHSS, we compared a single-factor model, a five-factor multidimensional model with covaried latent factors, and a second order where the five subscales loaded onto a global "Barriers" higher-level factor. Model fit was evaluated using robust versions of the Tucker-Leviws Index (TLI), the Comparative Fit Index (CFI) and Root Mean Square Error of Approximation (RMSEA) [@savalei2018; @zhang2023]. The fit was considered adequate if RMSEA was below 0.08 and CFI and TLI above 0.90, and good if these metrics were below 0.05 and above 0.95 respectively [@browne1992].

Finally, measurement invariance with respect to gender and past therapy experience was evaluated via multi-group CFA (MGCFA) using the `measEq.syntax()` function from the *semTools* R package [@semtools]. We followed a hierarchical procedure of increasingly restrictive constraints to assess:

1. Configural invariance by fitting a CFA in each subgroup (men/women and with/without past experience with therapy),
2. Metric invariance by constraining factor loadings to be equal across groups (to test whether the meaning of the construct remains the same), and
3. Scalar invariance by constraining item thresholds to be equal across groups (to allow valid group comparisons of latent means). 

Following @chen2007, we defined measurement _non-invariance_ as a change in CFI (cutoff: $|\Delta CFI| \geq 0.005$), supplemented by a change in RMSEA (cutoff: $|\Delta RMSEA| \geq 0.010$). In cases where model estimation failed due to empty response categories in specific subgroups (e.g., zero-frequency cells), those specific paths were excluded from the invariance hierarchy.

Since the aim of the article is to provide validation of the evaluated scales for the Czech population, only participants that claimed Czech nationality were included in the sample. Furthermore, participants who did not identify as either a man or a woman were excluded from the sample to ensure adequate statistical power for group analyses. Finally, complete cases analysis was used for all CFA and pairwise complete analysis was used for correlations.

```{r}
#| label: import
#| echo: false

library(targets)
library(tidyverse)
library(gt)
library(corrplot)

tar_source()
tar_source("../../R")
new_folder("tables")
new_folder("figures")
```

# Results

## Data description

```{r}
#| label: desc

tar_load(trimmed_data)
```

Of the `r trimmed_data$n_original` initial responses, `r trimmed_data$n_nonczech` participants of non-Czech nationality and further `r trimmed_data$n_nonbinary` participants who identified as non-binary or did not disclose their gender were excluded. This resulted in a final sample of `r trimmed_data$n_final` respondents. Age and demographic distributions are detailed in @fig-age and @tbl-desc, respectively. The sample was characterized by a right-skewed age distribution, comprising primarily persons aged 20-40 years, and a significant over-representation of women.

## Internal consistency

Cronbach's $\alpha$ and Guttman's $\lambda_6$ estimates alongside the average inter-item correlations are presented in @tbl-reli. Following the criteria established above, PSOSH, SSOSH, BHSS total score, BHSS resignation and BHSS emotional control showed evidence of good internal consistency; ATSPPH-SF, BHSS self-control and BHSS privacy showed evidence of internal consistency adequate for early research; and BHSS distrust showed evidence of inadequate internal consistency. Inter-item correlations generally align with these findings, falling within or above the ranges of adequate internal consistency. However, several scales representing narrow characteristics fell below the 0.40-0.50 target range: SSOSH (r = 0.31), BHSS self-control (r = 0.25), BHSS distrust (r = 0.19) and BHSS privacy (r = 0.32).

```{r}
#| label: tbl-reli
#| tbl-cap: Summary of results regarding internal consistency and factor structure of included scales.

tar_read(reliability_indexes_table)
```

## Confirmatory factor analysis

CFA fit indices are presented in @tbl-reli. SSOSH demonstrated good fit across all indices, whereas PSOSH achieved good fit for CFI and TLI but did not reach the RMSEA threshold. The ATSPPH-SF model failed to reach adequate fit criteria on any metric, suggesting a potential non-unidimensional structure or significant measurement error in this sample.

For the BHSS, the unidimensional model performed poorly as expected. While the multidimensional and second-order models improved fit substantially, both remained below the thresholds for adequate fit. Consequently, while our data provide strong support for the unidimensional structure of SSOSH and moderate support for PSOSH, evidence regarding the factor structures of ATSPPH-SF and BHSS remains relatively weak.

## Measurement invariance

The results of the invariance testing across gender and therapy experience are summarized in @tbl-invariance. For the majority of the scales, full scalar invariance was supported by the $\Delta CFI$ criteria, suggesting that the underlying constructs are measured consistently across these groups.

Regarding gender, most models demonstrated stable fit indices through the scalar level. However, metric measurement _non-invariance_ was detected in the BHSS subscales. While $\Delta CFI$ remained within acceptable bounds for metric invariance, $\Delta RMSEA$ exceeded the 0.010 threshold for all BHSS subscales. Furthermore, for the PSOSH scale, evidence of scalar _non-invariance_ was detected ($\Delta RMSEA = 0.015$), suggesting that the item thresholds may vary by gender. Invariance testing for the BHSS Self-Control subscale could not be completed for gender due to empty response categories in the men subgroup, which prohibited stable WLSMV estimation.

Regarding therapy experience, the ATSPPH-SF, BHSS Total, and BHSS (Distrust) scales failed to reach metric invariance based on the $\Delta CFI$ criterion. This suggests that the relationship between specific items and their latent factors differs depending on whether a respondent has prior experience with psychological therapy. Similar to the gender results, the PSOSH scale showed scalar non-invariance according to RMSEA. On the other hand, the SSOSH scale demonstrated robust invariance across all levels and groups, suggesting its suitability for mean-level comparisons in this population.

```{r}
#| label: tbl-invariance
#| tbl-cap: Measurement invariance across gender and prior therapy experience.

tar_read(cfa_invariance_table)
```

## Correlation analysis

Spearman correlations between demographic variables and scale sum scores are presented in @fig-sumscores. Inter-scale correlations were consistently stronger than associations with demographic or clinical variables. While age correlated significantly with all scales, the coefficients were generally low ($|\rho| \leq 0.19$). Similarly, gender, education level, and town size demonstrated only weak associations with the evaluated scales ($|\rho| \leq 0.13$). On the other hand, past therapy experience showed small but significant correlations with SSOSH ($\rho = -0.22$), BHSS Resignation ($\rho = -0.26$), and ATSPPH-SF ($\rho = 0.25$). Notably, correlation between therapy experience and PSOSH was near zero ($\rho = 0.01$). These overarching patterns of associations can be seen more granularly in the item-level correlations presented in @fig-s through @fig-polycor.


```{r}
#| label: corrs

corrs1 <- tar_read(sum_scores_correlations)
corrs2 <- tar_read(item_descriptors_correlations)
corrs3 <- tar_read(polychoric_correlations)

nms <- tar_read(names_map)
desc <- tar_read(descriptors)

p1 <- plot_sum_scores_correlations(corrs1)
p2 <- plot_item_descriptor_correlations(corrs2, nms, desc)
p3 <- plot_polychoric_correlations(corrs3, save = TRUE)
```

![Spearman's correlation matrix measuring strength of associations between questionnaires' sum scores and demographic and therapy-related variables.](../projects/valireli/figures/sum_scores_spearman_corrs.jpg){#fig-sumscores}


# Appendix

## Data description

```{r}
#| label: tbl-desc
#| tbl-cap: Descriptive statistics of demography and therapy-related variables.

tar_read(counts_table) |>
  filter(rowSums(across(c("PSOSH", "SSOSH", "BHSS", "ATSPPH-SF"))) > 0) |>
  gt(groupname_col = "var", rowname_col = "Var1") |>
  fmt_number(decimals = 0) |>
  cols_align(columns = 1, align = "left") |>
  cols_align(columns = -1, align = "center") |>
  cols_width(Var1 ~ px(150), everything() ~ px(60))
```

```{r}
#| label: age-plt

age_plt <- tar_read(age_plot)

ggsave(
  plot = age_plt,
  file = "figures/age_distributions.jpg",
  dpi = 300,
  width = 16,
  height = 10.4
)
```

![Age distribution of participants with no missing dat for each questionnaire.](../projects/valireli/figures/age_distributions.jpg){#fig-age}

## Correlation matrixes

### Perceived stigmatisation by others

![Mixed correlation matrix measuring strength of associations between responses to stigmatisation-by-others scale, demographic and therapy-related variables.](../projects/valireli/figures/corrplot_psosh.jpg){#fig-s}

### Self-stigmatisation

![Mixed correlation matrix measuring strength of associations between responses to self-stigmatisation scale, demographic and therapy-related variables.](../projects/valireli/figures/corrplot_ssosh.jpg){#fig-ss}

### Barriers

#### Self-control

![Mixed correlation matrix measuring strength of associations between responses to barriers self-control subscale, demographic and therapy-related variables.](../projects/valireli/figures/corrplot_bhss_self-control.jpg){#fig-f1}

#### Resignation

![Mixed correlation matrix measuring strength of associations between responses to barriers resignation subscale, demographic and therapy-related variables.](../projects/valireli/figures/corrplot_bhss_resignation.jpg){#fig-f2}

#### Distrust

![Mixed correlation matrix measuring strength of associations between responses to barriers distrust subscale, demographic and therapy-related variables.](../projects/valireli/figures/corrplot_bhss_distrust.jpg){#fig-f3}

#### Privacy

![Mixed correlation matrix measuring strength of associations between responses to barriers privacy subscale, demographic and therapy-related variables.](../projects/valireli/figures/corrplot_bhss_privacy.jpg){#fig-f4}

#### Emotional control

![Mixed correlation matrix measuring strength of associations between responses to barriers emotional control subscale, demographic and therapy-related variables.](../projects/valireli/figures/corrplot_bhss_emotional_control.jpg){#fig-f5}

### Attitudes

![Mixed correlation matrix measuring strength of associations between responses to attitudes scale, demographic and therapy-related variables.](../projects/valireli/figures/corrplot_atspph-sf.jpg){#fig-p}

### Item scores

![Polychoric correlation matrix measuring strength of inter-item associations.](../projects/valireli/figures/item_cormat.jpg){#fig-polycor}

# References