Extracting a reliable signal from heterogeneous data

Abstract

The article is devoted to the study of extracting a common reliable signal from data divided into heterogeneous groups. A soft maximum estimate of the maximum value is proposed as a computationally attractive alternative aimed at achieving a balance between a combined estimate and a (hard) estimate of the maximum value. The problem of extracting a common signal from heterogeneous data is considered. Since heterogeneity prevails in large-scale systems, the goal is a computationally efficient estimator (solution) with good statistical proper-ties with varying degrees of data heterogeneity. Using this estimate can lead to more reliable estimates for heterogeneous data compared to an estimate that does not take into account grouping, that is, a combined estimate. In large-scale data processing systems, where data heterogeneity is usually found, the computational aspect of evaluation is crucial. In substantiation of this thesis, the article provides an analysis of the effectiveness of soft maximum estimation for approval of large-scale data processing systems, confirming the effectiveness of the applied method. In summary soft maximin estimation will be practically useful in a number of different contexts, as a way of aggregating explained variances across groups.