The Challenge of Non-Linear Regression on Large Datasets with Asymmetric Heavy Tails
Regression becomes unstable under a heavy-tail error distribution due to dominant effects ofutliers. Traditional robust estimators are helpful under symmetricerror, by reducing the effect of outliers equally from both sides of the distribution. Under asymmetricerror, however, those estimators are biasedbecause the outliers appear only one side of the distribution. Motivated by datamining problems for the insurance industry, we propose in this paper a new approach to robust regression that is tailored to deal with asymmetric error. The main idea is to estimate a majority of the parameters using quantile regression(which may be biased but robust), and to estimate the few remaining using least squares estimator to correct the biases. Theoretical analysis shows conditions when the conditional expectations can be recovered from the conditional quantiles and what can be gained asymptotically with the proposed algorithm. Experiments confirm the clear advantages of the approach. Results are on synthetic data as well as real insurance data, using both linear and neural-network predictors.