Validation of self-stigma questionnaires


Josef Mana

Department of Cognitive Psychology, Institute of Psychology, Czech Academy of Sciences



Author Note

Josef Mana Orcid ID Logo: A green circle with white letters ID https://orcid.org/0000-0002-7817-3978


Validation of self-stigma questionnaires

Methods

Statistical analyses

Data were analyzed using R (version 4.6.1) (R Core Team, 2024). The analysis pipeline was managed via the targets package (Landau, 2021) to ensure reproducibility. Full analysis code is available at https://github.com/josefmana/stighelp.git

All demographic and outcome variables were described using contingency tables for nominal and ordinal (gender, education level, town size, past therapy) data and sample mean ± standard deviation for continuous variables (age and questionnaires’ sum scores).

Descriptive correlation analyses were conducted using polychoric correlations for ordinal variables (education level, town size, single item scores), biserial correlations for dichotomous variables (gender, past therapy), Pearson correlations for continuous variables (Age) and Spearman correlation for questionnaires’ sum scores, as implemented in mixedCor() function from the R package psych (Revelle, 2022).

We evaluated internal consistency using Cronbach’s \alpha and Guttman’s \lambda_6 coefficients computed via the psych R package (Revelle, 2022). Values between 0.70 and 0.80 were considered adequate for early research stages, values above 0.80 to show overall adequate internal consistency, and values above 0.90 to indicate redundancy between items (Nunnally & Bernstein, 1994; Streiner, 2003). To complement \alpha, which is affected by scale lengths, we inspected average inter-item correlations. Values between 0.15–0.20 were interpreted as indicating adequate internal consistency for broad constructs (BHSS total score, ATSPPH-SF) and 0.40–0.50 for narrow constructs (BHSS subscales, PSOSH, SSOSH) (Streiner, 2003).

To evaluate evidence of construct validity, we fitted a series of Confirmatory Factor Analyses (CFA) using the lavaan R package (Rosseel, 2012). Given the ordinal nature of the items, we employed the diagonally weighted least squares (WLSMV) estimator with theta parametrization, as this approach is more robust for categorical data than traditional Maximum Likelihood (Li, 2015). We tested unidimensional models for the PSOSH, SSOSH and ATSPPH-SF. For the BHSS, we compared a single-factor model, a five-factor multidimensional model with covaried latent factors, and a second order where the five subscales loaded onto a global “Barriers” higher-level factor. Model fit was evaluated using robust versions of the Tucker-Leviws Index (TLI), the Comparative Fit Index (CFI) and Root Mean Square Error of Approximation (RMSEA) (Savalei, 2018; Zhang & Savalei, 2023). The fit was considered adequate if RMSEA was below 0.08 and CFI and TLI above 0.90, and good if these metrics were below 0.05 and above 0.95 respectively (Browne & Cudeck, 1992).

Finally, measurement invariance with respect to gender and past therapy experience was evaluated via multi-group CFA (MGCFA) using the measEq.syntax() function from the semTools R package (Jorgensen et al., 2026). We followed a hierarchical procedure of increasingly restrictive constraints to assess:

  1. Configural invariance by fitting a CFA in each subgroup (men/women and with/without past experience with therapy),
  2. Metric invariance by constraining factor loadings to be equal across groups (to test whether the meaning of the construct remains the same), and
  3. Scalar invariance by constraining item thresholds to be equal across groups (to allow valid group comparisons of latent means).

Following Chen (2007), we defined measurement non-invariance as a change in CFI (cutoff: |\Delta CFI| \geq 0.005), supplemented by a change in RMSEA (cutoff: |\Delta RMSEA| \geq 0.010). In cases where model estimation failed due to empty response categories in specific subgroups (e.g., zero-frequency cells), those specific paths were excluded from the invariance hierarchy.

Since the aim of the article is to provide validation of the evaluated scales for the Czech population, only participants that claimed Czech nationality were included in the sample. Furthermore, participants who did not identify as either a man or a woman were excluded from the sample to ensure adequate statistical power for group analyses. Finally, complete cases analysis was used for all CFA and pairwise complete analysis was used for correlations.

Results

Data description

Of the 4806 initial responses, 315 participants of non-Czech nationality and further 22 participants who identified as non-binary or did not disclose their gender were excluded. This resulted in a final sample of 4311 respondents. Age and demographic distributions are detailed in and , respectively. The sample was characterized by a right-skewed age distribution, comprising primarily persons aged 20-40 years, and a significant over-representation of women.

Internal consistency

Cronbach’s \alpha and Guttman’s \lambda_6 estimates alongside the average inter-item correlations are presented in . Following the criteria established above, PSOSH, SSOSH, BHSS total score, BHSS resignation and BHSS emotional control showed evidence of good internal consistency; ATSPPH-SF, BHSS self-control and BHSS privacy showed evidence of internal consistency adequate for early research; and BHSS distrust showed evidence of inadequate internal consistency. Inter-item correlations generally align with these findings, falling within or above the ranges of adequate internal consistency. However, several scales representing narrow characteristics fell below the 0.40-0.50 target range: SSOSH (r = 0.31), BHSS self-control (r = 0.25), BHSS distrust (r = 0.19) and BHSS privacy (r = 0.32).

Confirmatory factor analysis

CFA fit indices are presented in . SSOSH demonstrated good fit across all indices, whereas PSOSH achieved good fit for CFI and TLI but did not reach the RMSEA threshold. The ATSPPH-SF model failed to reach adequate fit criteria on any metric, suggesting a potential non-unidimensional structure or significant measurement error in this sample.

For the BHSS, the unidimensional model performed poorly as expected. While the multidimensional and second-order models improved fit substantially, both remained below the thresholds for adequate fit. Consequently, while our data provide strong support for the unidimensional structure of SSOSH and moderate support for PSOSH, evidence regarding the factor structures of ATSPPH-SF and BHSS remains relatively weak.

Measurement invariance

The results of the invariance testing across gender and therapy experience are summarized in . For the majority of the scales, full scalar invariance was supported by the \Delta CFI criteria, suggesting that the underlying constructs are measured consistently across these groups.

Regarding gender, most models demonstrated stable fit indices through the scalar level. However, metric measurement non-invariance was detected in the BHSS subscales. While \Delta CFI remained within acceptable bounds for metric invariance, \Delta RMSEA exceeded the 0.010 threshold for all BHSS subscales. Furthermore, for the PSOSH scale, evidence of scalar non-invariance was detected (\Delta RMSEA = 0.015), suggesting that the item thresholds may vary by gender. Invariance testing for the BHSS Self-Control subscale could not be completed for gender due to empty response categories in the men subgroup, which prohibited stable WLSMV estimation.

Regarding therapy experience, the ATSPPH-SF, BHSS Total, and BHSS (Distrust) scales failed to reach metric invariance based on the \Delta CFI criterion. This suggests that the relationship between specific items and their latent factors differs depending on whether a respondent has prior experience with psychological therapy. Similar to the gender results, the PSOSH scale showed scalar non-invariance according to RMSEA. On the other hand, the SSOSH scale demonstrated robust invariance across all levels and groups, suggesting its suitability for mean-level comparisons in this population.

Correlation analysis

Spearman correlations between demographic variables and scale sum scores are presented in . Inter-scale correlations were consistently stronger than associations with demographic or clinical variables. While age correlated significantly with all scales, the coefficients were generally low (|\rho| \leq 0.19). Similarly, gender, education level, and town size demonstrated only weak associations with the evaluated scales (|\rho| \leq 0.13). On the other hand, past therapy experience showed small but significant correlations with SSOSH (\rho = -0.22), BHSS Resignation (\rho = -0.26), and ATSPPH-SF (\rho = 0.25). Notably, correlation between therapy experience and PSOSH was near zero (\rho = 0.01). These overarching patterns of associations can be seen more granularly in the item-level correlations presented in through .

Table 1

Summary of results regarding internal consistency and factor structure of included scales.

Confirmatory factor analysis
Internal consistency
Scale N K
RMSEA
CFI TLI
Cronbach's α
G6 r
estimate 90% CI raw stand.
PSOSH 3,741 25 0.126 [0.110, 0.144] 0.98 0.96 0.88 0.88 0.86 0.59
SSOSH 3,501 50 0.069 [0.064, 0.074] 0.96 0.95 0.81 0.82 0.82 0.31
BHSS 2,958 155 0.144 [0.142, 0.146] 0.44 0.40 0.89 0.89 0.92 0.20
ATSPPH-SF 2,776 40 0.129 [0.121, 0.136] 0.80 0.74 0.73 0.74 0.74 0.22
BHSS (privacy)a 2,978 25 0.114 [0.099, 0.130] 0.94 0.89 0.70 0.70 0.67 0.32
BHSS (emotional control)b 2,986 20 0.196 [0.171, 0.221] 0.97 0.91 0.83 0.83 0.82 0.55
BHSS (multidimensional) 2,966 111 0.104 [0.101, 0.107] 0.81 0.78 - - - -
BHSS (second order) 2,966 109 0.104 [0.101, 0.107] 0.81 0.78 - - - -
BHSS (self-control) 2,976 50 0.256 [0.249, 0.263] 0.46 0.31 0.76 0.77 0.80 0.25
BHSS (resignation) 2,977 30 0.177 [0.165, 0.189] 0.93 0.88 0.88 0.88 0.87 0.55
BHSS (distrust) 2,976 30 0.143 [0.131, 0.156] 0.78 0.63 0.58 0.58 0.56 0.19
a The CFA model included five independent but covaried latent factors of Barriers scale.
b The CFA model included five independent latent factors that further loaded on a common Barriers higher-level factor.
N: number of participants; K: number of parameters of the confirmatory factor analysis model; RMSEA: root-mean-square error approximation; CI: confidence interval; CFI: comparative fit index, TLI: Tucker-Lewis Index; raw: Cronbach's α based upon the covariances; stand.: Cronbach's α based upon the correlations, G6: Guttman's λ 6 reliability; r: the average inter-item correlation; for RMSEA, values less than 0.08 indicating an adequate model fit (especially values of upper 90% CI bound lower than 0.08), for CFI and TLI, values exceeding 0.95 are considered to indicate a good model fit.

Table 2

Measurement invariance across gender and prior therapy experience.

Model
Across gender
Across therapy experience
CFI ΔCFI sig. RMSEA ΔRMSEA sig. CFI ΔCFI sig. RMSEA ΔRMSEA sig.
PSOSH
Configural 0.999
0.058
0.999
0.062
Metric 0.999 0.000 0.049 0.009 0.999 0.000 0.052 0.010
Scalar 0.999 0.000 0.034 0.015 * 0.999 0.000 0.037 0.015 *
SSOSH
Configural 0.993
0.054
0.993
0.054
Metric 0.992 0.001 0.054 0.000 0.992 0.001 0.054 0.000
Scalar 0.993 0.001 0.045 0.008 0.992 0.000 0.049 0.005
ATSPPH-SF
Configural 0.951
0.077
0.944
0.078
Metric 0.950 0.001 0.073 0.004 0.934 0.010 * 0.080 0.002
Scalar 0.951 0.001 0.069 0.005 0.936 0.002 0.074 0.006
BHSS
Configural —

—

0.835
0.159
Metric — — — — — — 0.830 0.005 * 0.158 0.000
Scalar — — — — — — 0.832 0.002 0.152 0.006
BHSS (multidimensional)
Configural 0.969
0.091
0.967
0.091
Metric 0.969 0.000 0.090 0.002 0.966 0.001 0.090 0.001
Scalar 0.969 0.000 0.085 0.005 0.966 0.000 0.085 0.005
BHSS (self-control)
Configural —

—

0.799
0.236
Metric — — — — — — 0.798 0.001 0.223 0.013 *
Scalar — — — — — — 0.799 0.001 0.198 0.024 *
BHSS (resignation)
Configural 0.994
0.110
0.993
0.111
Metric 0.994 0.000 0.099 0.011 * 0.993 0.000 0.100 0.012 *
Scalar 0.994 0.000 0.078 0.022 * 0.993 0.000 0.082 0.017 *
BHSS (distrust)
Configural 0.909
0.101
0.920
0.095
Metric 0.907 0.002 0.091 0.011 * 0.906 0.013 * 0.090 0.004
Scalar 0.909 0.002 0.073 0.018 * 0.906 0.000 0.073 0.017 *
BHSS (privacy)
Configural 0.989
0.068
0.987
0.073
Metric 0.989 0.000 0.059 0.010 0.985 0.002 0.067 0.006
Scalar 0.989 0.001 0.043 0.016 * 0.985 0.000 0.050 0.017 *
BHSS (emotional control)
Configural 0.998
0.109
0.998
0.106
Metric 0.998 0.000 0.083 0.026 * 0.998 0.000 0.081 0.025 *
Scalar 0.998 0.000 0.054 0.029 * 0.998 0.000 0.056 0.025 *
PSOSH: Perceptions of Stigmatization by Others for Seeking Help; SSOSH: Self-stigma of Seeking Help Scale; ATSPPH-SF: Attitudes Toward Seeking Professional Psychological Help Scale - Short Form; BHSS: Barriers to Help Seeking Scale; RMSEA: root-mean-square error approximation; CFI: comparative fit index, ΔCFI: change in CFI, ΔRMSEA: change in RMSEA; sig. indicator of measurement noninvariance following the cutoffs stated in the text.

Figure 1

Spearman’s correlation matrix measuring strength of associations between questionnaires’ sum scores and demographic and therapy-related variables.

Appendix

Appendix

Data description

Table A1

Descriptive statistics of demography and therapy-related variables.

PSOSH SSOSH BHSS ATSPPH-SF
Gender
Woman 3,557 3,333 2,828 2,631
Man 195 182 158 148
Education
Elementary school 289 265 212 186
High school 1,571 1,477 1,265 1,174
Bachelors 848 792 674 635
Masters 1,008 952 809 760
PhD 36 29 26 24
Town size
< 5,000 inhabitants 1,065 996 823 754
5,000 - 50,000 inhabitants 888 834 728 677
50,000 - 100,000 inhabitants 328 311 256 233
> 100,000 inhabitants 1,471 1,374 1,179 1,115
Past therapy
Yes (as a child & adult) 562 534 469 446
Yes (as a child only) 327 308 257 240
Yes (as an adult only) 1,434 1,339 1,134 1,055
No 1,422 1,327 1,121 1,033
Undisclosed 7 7 5 5

Figure A1

Age distribution of participants with no missing dat for each questionnaire.

Correlation matrixes

Perceived stigmatisation by others

Figure A2

Mixed correlation matrix measuring strength of associations between responses to stigmatisation-by-others scale, demographic and therapy-related variables.

Self-stigmatisation

Figure A3

Mixed correlation matrix measuring strength of associations between responses to self-stigmatisation scale, demographic and therapy-related variables.

Barriers

Self-control.

Figure A4

Mixed correlation matrix measuring strength of associations between responses to barriers self-control subscale, demographic and therapy-related variables.

Resignation.

Figure A5

Mixed correlation matrix measuring strength of associations between responses to barriers resignation subscale, demographic and therapy-related variables.

Distrust.

Figure A6

Mixed correlation matrix measuring strength of associations between responses to barriers distrust subscale, demographic and therapy-related variables.

Privacy.

Figure A7

Mixed correlation matrix measuring strength of associations between responses to barriers privacy subscale, demographic and therapy-related variables.

Emotional control.

Figure A8

Mixed correlation matrix measuring strength of associations between responses to barriers emotional control subscale, demographic and therapy-related variables.

Attitudes

Figure A9

Mixed correlation matrix measuring strength of associations between responses to attitudes scale, demographic and therapy-related variables.

Item scores

Figure A10

Polychoric correlation matrix measuring strength of inter-item associations.

References

Browne, M. W., & Cudeck, R. (1992). Alternative ways of assessing model fit. Sociological Methods & Research, 21(2), 230–258. https://doi.org/10.1177/0049124192021002005
Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 464–504. https://doi.org/10.1080/10705510701301834
Jorgensen, T. D., Pornprasertmanit, S., Schoemann, A. M., & Rosseel, Y. (2026). semTools: Useful tools for structural equation modeling. https://CRAN.R-project.org/package=semTools
Landau, W. M. (2021). The targets r package: A dynamic make-like function-oriented pipeline toolkit for reproducibility and high-performance computing. Journal of Open Source Software, 6(57), 2959. https://doi.org/10.21105/joss.02959
Li, C.-H. (2015). Confirmatory factor analysis with ordinal data: Comparing robust maximum likelihood and diagonally weighted least squares. Behavior Research Methods, 48(3), 936–949. https://doi.org/10.3758/s13428-015-0619-7
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.
R Core Team. (2024). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/
Revelle, W. (2022). Psych: Procedures for psychological, psychometric, and personality research. https://CRAN.R-project.org/package=psych
Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02
Savalei, V. (2018). On the computation of the RMSEA and CFI from the mean-and-variance corrected test statistic with nonnormal data in SEM. Multivariate Behavioral Research, 53(3), 419–429. https://doi.org/10.1080/00273171.2018.1455142
Streiner, D. L. (2003). Starting at the beginning: An introduction to coefficient alpha and internal consistency. Journal of Personality Assessment, 80(1), 99–103. https://doi.org/10.1207/s15327752jpa8001_18
Zhang, X., & Savalei, V. (2023). New computations for RMSEA and CFI following FIML and TS estimation with missing data. Psychological Methods, 28(2), 263–283. https://doi.org/10.1037/met0000445