Beyond Statistical Fairness: A Three-Layer Framework for Benchmark-Independent Evaluation of Semantic Debiasing

Abstract

Yair Oppenheim

The rapid deployment of Large Language Models (LLMs) has intensified concerns regarding demographic bias and semantic unfairness in generated text. Existing evaluation methodologies primarily assess fairness through benchmark- specific performance or statistical significance, providing limited evidence regarding whether semantic debiasing methods remain effective across heterogeneous benchmark ecosystems. Consequently, current evaluation approaches often fail to distinguish benchmark optimization from genuine benchmark-independent semantic fairness.

This paper introduces a comprehensive three-layer evaluation framework for semantic debiasing and validates it through the Dynamic Bias Semantic Debiasing (DBSD) framework. The proposed methodology integrates three complementary evaluation layers: (1) Semantic Effectiveness, which quantifies semantic bias reduction within each benchmark; (2) Statistical Generalizability, which evaluates whether semantic improvements generalize across heterogeneous benchmark families through inferential statistical analysis; and (3) Semantic Robustness, a novel engineering-oriented evaluation layer that introduces the Benchmark Robustness Score (BRS) and the derived Semantic Performance (SP) metric to quantify the stability of semantic bias reduction across benchmark ecosystems.

The framework is empirically evaluated using five widely adopted fairness benchmarks—CrowS-Pairs, StereoSet, BBQ, WinoBias, and HolisticBias—covering multiple demographic dimensions. Experimental results demonstrate statistically significant semantic bias reduction while preserving high semantic utility. One-way ANOVA confirms that the observed improvements generalize across benchmark families, whereas the proposed robustness analysis demonstrates near- complete benchmark-independent robustness among independently selected benchmark ecosystems.

The principal scientific contribution of this work is not merely the validation of DBSD, but the introduction of a reproducible multi-layer methodology for evaluating semantic fairness beyond conventional benchmark-specific metrics. The proposed framework establishes a systematic connection between semantic effectiveness, statistical generalization, and robust engineering, thereby providing a more comprehensive foundation for evaluating semantic debiasing algorithms in future LLM research.

This paper proposes a new evaluation methodology for semantic fairness. DBSD serves as the validation case study rather than the primary contribution.

PDF