Validation of self-stigma questionnaires

Josef Mana

Department of Cognitive Psychology, Institute of Psychology, Czech Academy of Sciences

Author Note

Josef Mana https://orcid.org/0000-0002-7817-3978

Validation of self-stigma questionnaires

Methods

Statistical analyses

Data were analyzed using R (version 4.6.1) (R Core Team, 2024). The analysis pipeline was managed via the targets package (Landau, 2021) to ensure reproducibility. Full analysis code is available at https://github.com/josefmana/stighelp.git

All demographic and outcome variables were described using contingency tables for nominal and ordinal (gender, education level, town size, past therapy) data and sample mean ± standard deviation for continuous variables (age and questionnaires’ sum scores).

Descriptive correlation analyses were conducted using polychoric correlations for ordinal variables (education level, town size, single item scores), biserial correlations for dichotomous variables (gender, past therapy), Pearson correlations for continuous variables (Age) and Spearman correlation for questionnaires’ sum scores, as implemented in mixedCor() function from the R package psych (Revelle, 2022).

We evaluated internal consistency using Cronbach’s \alpha and Guttman’s \lambda_6 coefficients computed via the psych R package (Revelle, 2022). Values between 0.70 and 0.80 were considered adequate for early research stages, values above 0.80 to show overall adequate internal consistency, and values above 0.90 to indicate redundancy between items (Nunnally & Bernstein, 1994; Streiner, 2003). To complement \alpha, which is affected by scale lengths, we inspected average inter-item correlations. Values between 0.15–0.20 were interpreted as indicating adequate internal consistency for broad constructs (BHSS total score, ATSPPH-SF) and 0.40–0.50 for narrow constructs (BHSS subscales, PSOSH, SSOSH) (Streiner, 2003).

To evaluate evidence of construct validity, we fitted a series of Confirmatory Factor Analyses (CFA) using the lavaan R package (Rosseel, 2012). Given the ordinal nature of the items, we employed the diagonally weighted least squares (WLSMV) estimator with theta parametrization, as this approach is more robust for categorical data than traditional Maximum Likelihood (Li, 2015). We tested unidimensional models for the PSOSH, SSOSH and ATSPPH-SF. For the BHSS, we compared a single-factor model, a five-factor multidimensional model with covaried latent factors, and a second order where the five subscales loaded onto a global “Barriers” higher-level factor. Model fit was evaluated using robust versions of the Tucker-Leviws Index (TLI), the Comparative Fit Index (CFI) and Root Mean Square Error of Approximation (RMSEA) (Savalei, 2018; Zhang & Savalei, 2023). The fit was considered adequate if RMSEA was below 0.08 and CFI and TLI above 0.90, and good if these metrics were below 0.05 and above 0.95 respectively (Browne & Cudeck, 1992).

Finally, measurement invariance with respect to gender and past therapy experience was evaluated via multi-group CFA (MGCFA) using the measEq.syntax() function from the semTools R package (Jorgensen et al., 2026). We followed a hierarchical procedure of increasingly restrictive constraints to assess:

1. Configural invariance by fitting a CFA in each subgroup (men/women and with/without past experience with therapy),
2. Metric invariance by constraining factor loadings to be equal across groups (to test whether the meaning of the construct remains the same), and
3. Scalar invariance by constraining item thresholds to be equal across groups (to allow valid group comparisons of latent means).

Following Chen (2007), we defined measurement non-invariance as a change in CFI (cutoff: |\Delta CFI| \geq 0.005), supplemented by a change in RMSEA (cutoff: |\Delta RMSEA| \geq 0.010). In cases where model estimation failed due to empty response categories in specific subgroups (e.g., zero-frequency cells), those specific paths were excluded from the invariance hierarchy.

Since the aim of the article is to provide validation of the evaluated scales for the Czech population, only participants that claimed Czech nationality were included in the sample. Furthermore, participants who did not identify as either a man or a woman were excluded from the sample to ensure adequate statistical power for group analyses. Finally, complete cases analysis was used for all CFA and pairwise complete analysis was used for correlations.

Results

Data description

Of the 4806 initial responses, 315 participants of non-Czech nationality and further 22 participants who identified as non-binary or did not disclose their gender were excluded. This resulted in a final sample of 4333 respondents. Age and demographic distributions are detailed in Figure A1 and Table A1, respectively. The sample was characterized by a right-skewed age distribution, comprising primarily persons aged 20-40 years, and a significant over-representation of women.

Internal consistency

Cronbach’s \alpha and Guttman’s \lambda_6 estimates alongside the average inter-item correlations are presented in Table 1. Following the criteria established above, PSOSH, SSOSH, BHSS total score, BHSS resignation and BHSS emotional control showed evidence of good internal consistency; ATSPPH-SF, BHSS self-control and BHSS privacy showed evidence of internal consistency adequate for early research; and BHSS distrust showed evidence of inadequate internal consistency. Inter-item correlations generally align with these findings, falling within or above the ranges of adequate internal consistency. However, several scales representing narrow characteristics fell below the 0.40-0.50 target range: SSOSH (r = 0.31), BHSS self-control (r = 0.25), BHSS distrust (r = 0.19) and BHSS privacy (r = 0.32).

Confirmatory factor analysis

CFA fit indices are presented in Table 1. SSOSH demonstrated good fit across all indices, whereas PSOSH achieved good fit for CFI and TLI but did not reach the RMSEA threshold. The ATSPPH-SF model failed to reach adequate fit criteria on any metric, suggesting a potential non-unidimensional structure or significant measurement error in this sample.

For the BHSS, the unidimensional model performed poorly as expected. While the multidimensional and second-order models improved fit substantially, both remained below the thresholds for adequate fit. Consequently, while our data provide strong support for the unidimensional structure of SSOSH and moderate support for PSOSH, evidence regarding the factor structures of ATSPPH-SF and BHSS remains relatively weak.

Measurement invariance

The results of the invariance testing across gender and therapy experience are summarized in Table 2. For the majority of the scales, full scalar invariance was supported by the \Delta CFI criteria, suggesting that the underlying constructs are measured consistently across these groups.

Regarding gender, most models demonstrated stable fit indices through the scalar level. However, metric measurement non-invariance was detected in the BHSS subscales. While \Delta CFI remained within acceptable bounds for metric invariance, \Delta RMSEA exceeded the 0.010 threshold for all BHSS subscales. Furthermore, for the PSOSH scale, evidence of scalar non-invariance was detected (\Delta RMSEA = 0.015), suggesting that the item thresholds may vary by gender. Invariance testing for the BHSS Self-Control subscale could not be completed for gender due to empty response categories in the men subgroup, which prohibited stable WLSMV estimation.

Regarding therapy experience, the ATSPPH-SF, BHSS Total, and BHSS (Distrust) scales failed to reach metric invariance based on the \Delta CFI criterion. This suggests that the relationship between specific items and their latent factors differs depending on whether a respondent has prior experience with psychological therapy. Similar to the gender results, the PSOSH scale showed scalar non-invariance according to RMSEA. On the other hand, the SSOSH scale demonstrated robust invariance across all levels and groups, suggesting its suitability for mean-level comparisons in this population.

Correlation analysis

Spearman correlations between demographic variables and scale sum scores are presented in Figure 1. Inter-scale correlations were consistently stronger than associations with demographic or clinical variables. While age correlated significantly with all scales, the coefficients were generally low (|\rho| \leq 0.19). Similarly, gender, education level, and town size demonstrated only weak associations with the evaluated scales (|\rho| \leq 0.13). On the other hand, past therapy experience showed small but significant correlations with SSOSH (\rho = -0.22), BHSS Resignation (\rho = -0.26), and ATSPPH-SF (\rho = 0.25). Notably, correlation between therapy experience and PSOSH was near zero (\rho = 0.01). These overarching patterns of associations can be seen more granularly in the item-level correlations presented in Figure A2 through Figure A10.

Table 1

Summary of results regarding internal consistency and factor structure of included scales.

	Confirmatory factor analysis								Internal consistency
Scale	N	df	χ2	CFI	TLI	RMSEA		SRMR	Cronbach's α		λ6	ω	r
						est.	90% CI		raw	std.
Unidimensional scales
PSOSH	3,759	5	192.4	0.98	0.96	0.126	[0.110, 0.143]	0.020	0.88	0.88	0.86	0.92	0.59
SSOSH	3,518	35	718.8	0.96	0.95	0.068	[0.063, 0.073]	0.036	0.81	0.82	0.82	0.87	0.31
ATSPPH-SF	2,789	35	894.0	0.80	0.74	0.128	[0.121, 0.136]	0.070	0.73	0.74	0.74	0.82	0.22
BHSS: competing whole-scale models
BHSS	2,972	434	28,579.1	0.44	0.40	0.144	[0.142, 0.146]	0.130	0.89	0.89	0.92	0.92	0.20
BHSS (multidimensional)a	2,972	424	15,736.6	0.71	0.68	0.105	[0.103, 0.107]	0.094	—	—	—	0.63 to 0.91	—
BHSS (second order)b	2,972	429	15,138.1	0.71	0.68	0.105	[0.103, 0.106]	0.095	—	—	—	0.63 to 0.91	—
BHSS: subscale models
BHSS (self-control)	2,990	35	5,892.2	0.46	0.31	0.256	[0.249, 0.263]	0.190	0.76	0.77	0.80	0.86	0.25
BHSS (resignation)	2,991	9	715.9	0.93	0.88	0.176	[0.165, 0.188]	0.048	0.88	0.88	0.87	0.91	0.55
BHSS (distrust)	2,990	9	364.9	0.78	0.63	0.144	[0.132, 0.157]	0.070	0.58	0.58	0.56	0.67	0.19
BHSS (privacy)	2,992	5	141.5	0.94	0.89	0.115	[0.100, 0.131]	0.041	0.70	0.70	0.67	0.76	0.32
BHSS (emotional control)	3,000	2	211.4	0.97	0.91	0.197	[0.173, 0.222]	0.038	0.83	0.83	0.82	0.88	0.55
a Five covaried first-order latent factors over all 31 items.
b Five first-order latent factors loading on a common higher-order Barriers factor.
N: participants entering the model (complete cases); df: model degrees of freedom; χ2: robust (scaled) chi-square; CFI and TLI: comparative fit and Tucker-Lewis indices, robust; RMSEA: root-mean-square error of approximation, robust, with its 90% confidence interval; SRMR: standardised root mean square residual. Cronbach's α is Pearson-based as computed by psych::alpha: raw is covariance-based, std. is correlation-based, and neither is ordinal α. λ6: Guttman's lambda 6. ω: model-implied ordinal omega on the latent response scale, computed from the standardized solution of the WLSMV confirmatory model, given as a range where a model has more than one factor. r: average inter-item correlation. Fit was judged adequate at RMSEA below 0.08 with CFI and TLI above 0.90, and good at RMSEA below 0.05 with CFI and TLI above 0.95. RMSEA is not interpreted for models with few degrees of freedom, where it is inflated and unstable; SRMR and the residual matrix carry the interpretation there. A dash denotes a quantity that is not defined for the model in question.

Table 2

Measurement invariance across gender and prior therapy experience.

Model	Across gender						Across therapy experience
	CFI	ΔCFI	sig.	RMSEA	ΔRMSEA	sig.	CFI	ΔCFI	sig.	RMSEA	ΔRMSEA	sig.
PSOSH
Configural	0.999			0.058			0.999			0.062
Metric	0.999	0.000		0.049	0.009		0.999	0.000		0.052	0.010
Scalar	0.999	0.000		0.034	0.015	*	0.999	0.000		0.037	0.015	*
SSOSH
Configural	0.993			0.054			0.993			0.054
Metric	0.992	0.001		0.054	0.000		0.992	0.001		0.054	0.000
Scalar	0.993	0.001		0.045	0.008		0.992	0.000		0.049	0.005
ATSPPH-SF
Configural	0.951			0.077			0.944			0.078
Metric	0.950	0.001		0.073	0.004		0.934	0.010	*	0.080	0.002
Scalar	0.951	0.001		0.069	0.005		0.936	0.002		0.074	0.006
BHSS
Configural	—			—			0.835			0.159
Metric	—	—	—	—	—	—	0.830	0.005	*	0.158	0.000
Scalar	—	—	—	—	—	—	0.832	0.002		0.152	0.006
BHSS (multidimensional)
Configural	0.969			0.091			0.967			0.091
Metric	0.969	0.000		0.090	0.002		0.966	0.001		0.090	0.001
Scalar	0.969	0.000		0.085	0.005		0.966	0.000		0.085	0.005
BHSS (self-control)
Configural	—			—			0.799			0.236
Metric	—	—	—	—	—	—	0.798	0.001		0.223	0.013	*
Scalar	—	—	—	—	—	—	0.799	0.001		0.198	0.024	*
BHSS (resignation)
Configural	0.994			0.110			0.993			0.111
Metric	0.994	0.000		0.099	0.011	*	0.993	0.000		0.100	0.012	*
Scalar	0.994	0.000		0.078	0.022	*	0.993	0.000		0.082	0.017	*
BHSS (distrust)
Configural	0.909			0.101			0.920			0.095
Metric	0.907	0.002		0.091	0.011	*	0.906	0.013	*	0.090	0.004
Scalar	0.909	0.002		0.073	0.018	*	0.906	0.000		0.073	0.017	*
BHSS (privacy)
Configural	0.989			0.068			0.987			0.073
Metric	0.989	0.000		0.059	0.010		0.985	0.002		0.067	0.006
Scalar	0.989	0.001		0.043	0.016	*	0.985	0.000		0.050	0.017	*
BHSS (emotional control)
Configural	0.998			0.109			0.998			0.106
Metric	0.998	0.000		0.083	0.026	*	0.998	0.000		0.081	0.025	*
Scalar	0.998	0.000		0.054	0.029	*	0.998	0.000		0.056	0.025	*
PSOSH: Perceptions of Stigmatization by Others for Seeking Help; SSOSH: Self-stigma of Seeking Help Scale; ATSPPH-SF: Attitudes Toward Seeking Professional Psychological Help Scale - Short Form; BHSS: Barriers to Help Seeking Scale; RMSEA: root-mean-square error approximation; CFI: comparative fit index, ΔCFI: change in CFI, ΔRMSEA: change in RMSEA; sig. indicator of measurement noninvariance following the cutoffs stated in the text.

Figure 1

Spearman’s correlation matrix measuring strength of associations between questionnaires’ sum scores and demographic and therapy-related variables.

Appendix

Appendix

Data description

Table A1

Descriptive statistics of demography and therapy-related variables.

	PSOSH	SSOSH	BHSS	ATSPPH-SF
Gender
Woman	3,557	3,333	2,828	2,631
Man	195	182	158	148
Undisclosed	14	14	10	10
Other	4	4	4	3
Education
Elementary school	294	270	217	191
High school	1,580	1,486	1,272	1,180
Bachelors	850	794	675	636
Masters	1,010	954	810	761
PhD	36	29	26	24
Town size
< 5,000 inhabitants	1,069	1,000	827	758
5,000 - 50,000 inhabitants	892	838	732	680
50,000 - 100,000 inhabitants	330	313	258	235
> 100,000 inhabitants	1,479	1,382	1,183	1,119
Past therapy
Yes (as a child & adult)	567	539	472	448
Yes (as a child only)	330	311	260	243
Yes (as an adult only)	1,440	1,345	1,139	1,060
No	1,426	1,331	1,124	1,036
Undisclosed	7	7	5	5

Figure A1

Age distribution of participants with no missing dat for each questionnaire.

Correlation matrixes

Perceived stigmatisation by others

Figure A2

Mixed correlation matrix measuring strength of associations between responses to stigmatisation-by-others scale, demographic and therapy-related variables.

Self-stigmatisation

Figure A3

Mixed correlation matrix measuring strength of associations between responses to self-stigmatisation scale, demographic and therapy-related variables.

Barriers

Self-control.

Figure A4

Mixed correlation matrix measuring strength of associations between responses to barriers self-control subscale, demographic and therapy-related variables.

Resignation.

Figure A5

Mixed correlation matrix measuring strength of associations between responses to barriers resignation subscale, demographic and therapy-related variables.

Distrust.

Figure A6

Mixed correlation matrix measuring strength of associations between responses to barriers distrust subscale, demographic and therapy-related variables.

Privacy.

Figure A7

Mixed correlation matrix measuring strength of associations between responses to barriers privacy subscale, demographic and therapy-related variables.

Emotional control.

Figure A8

Mixed correlation matrix measuring strength of associations between responses to barriers emotional control subscale, demographic and therapy-related variables.

Attitudes

Figure A9

Mixed correlation matrix measuring strength of associations between responses to attitudes scale, demographic and therapy-related variables.

Item scores

Figure A10

Polychoric correlation matrix measuring strength of inter-item associations.

References

Browne, M. W., & Cudeck, R. (1992). Alternative ways of assessing model fit. Sociological Methods & Research, 21(2), 230–258. https://doi.org/10.1177/0049124192021002005

Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 464–504. https://doi.org/10.1080/10705510701301834

Jorgensen, T. D., Pornprasertmanit, S., Schoemann, A. M., & Rosseel, Y. (2026). semTools: Useful tools for structural equation modeling. https://CRAN.R-project.org/package=semTools

Landau, W. M. (2021). The targets r package: A dynamic make-like function-oriented pipeline toolkit for reproducibility and high-performance computing. Journal of Open Source Software, 6(57), 2959. https://doi.org/10.21105/joss.02959

Li, C.-H. (2015). Confirmatory factor analysis with ordinal data: Comparing robust maximum likelihood and diagonally weighted least squares. Behavior Research Methods, 48(3), 936–949. https://doi.org/10.3758/s13428-015-0619-7

Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.

R Core Team. (2024). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/

Revelle, W. (2022). Psych: Procedures for psychological, psychometric, and personality research. https://CRAN.R-project.org/package=psych

Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02

Savalei, V. (2018). On the computation of the RMSEA and CFI from the mean-and-variance corrected test statistic with nonnormal data in SEM. Multivariate Behavioral Research, 53(3), 419–429. https://doi.org/10.1080/00273171.2018.1455142

Streiner, D. L. (2003). Starting at the beginning: An introduction to coefficient alpha and internal consistency. Journal of Personality Assessment, 80(1), 99–103. https://doi.org/10.1207/s15327752jpa8001_18

Zhang, X., & Savalei, V. (2023). New computations for RMSEA and CFI following FIML and TS estimation with missing data. Psychological Methods, 28(2), 263–283. https://doi.org/10.1037/met0000445