Regression and Statistical Inference
Definition
The applied-statistics toolkit from a Columbia quantitative-techniques course: how to get data (sampling), look at it (EDA), describe its spread (variance), trust the instrument (reliability vs validity), and model relationships (regression).
Course text: Introductory Statistics with Randomization and Simulation (ISRS), with Red State, Blue State, Rich State, Poor State (Gelman) as the running case study.
Core Ideas
Sampling and adjustment
Populations vs samples, sampling from a population, the normal distribution model and the 68-95-99.7 rule, when a sample proportion is nearly normal, and confidence intervals for a proportion.
The course pairs this with political-polling commentary (FiveThirtyEight on over-reacting to single polls; NYT on what’s wrong with polling) — the point being that sampling error is the default explanation for a surprising number, before any story about a real shift.
Exploratory data analysis
EDA analyzes datasets to summarize their main characteristics, often with statistical graphics and visualization. A model may or may not be used, but EDA is primarily for seeing what the data can tell beyond formal modeling.
The contrast that defines it: traditional hypothesis testing selects the model before seeing the data. EDA — promoted by John Tukey since 1970 — explores first and may formulate hypotheses that lead to new data collection.
EDA encompasses initial data analysis (IDA), which is narrower: checking the assumptions needed for model fitting, handling missing values, transforming variables.
Practical foundations: observations / variables / data matrices, scatterplots for paired data, variance and standard deviation, mapping data.
Variance and standard deviation
- Mean 平均值
- Variance 方差 — σ² for a population; roughly the average squared distance from the mean
- Standard deviation 标准差 — σ; the square root of the variance, describing how close the data sit to the mean
Measurement: reliability vs validity
Two different failures, easily conflated:
- Reliability — consistency (does it give the same answer again?)
- Validity — accuracy (is it measuring the intended thing?)
A measure can be perfectly reliable and entirely invalid.
Linear regression
- Fitting a line by least squares; reading regression output from software.
- Correlation R — always between −1 and 1; describes the strength of the linear relationship between two variables. Note it says nothing about the form beyond linearity.
- Regression is a partitioning of variance in Y into a part explained by the model and a residual-error part — which is what ANOVA is.
ANOVA (方差分析) — compares whether the means of two or more samples differ significantly, by comparing between-group variance against within-group variance, to judge whether a categorical predictor significantly affects a continuous outcome.
Multiple regression: the two effects worth knowing
Multicollinearity (多重共线性) — predictors are strongly linearly related, so one can be linearly predicted from the others. Canonical example: predicting house price from floor area and bedroom count — bigger houses have more bedrooms, so the two carry overlapping information.
Suppression (抑制效应) — a variable that has little or no direct correlation with the outcome, yet increases another predictor’s explanatory power when added to the model.
The lesson both effects teach:
When adding multiple predictors, remember that the meaning of a variable can change depending on which other variables are included.
The Gelman case study is the memorable version of the same trap: richer people are more likely to vote conservative while richer states are more likely to vote liberal — the level of aggregation changes the relationship.
Relationships
- Data Science Methodology — where these techniques sit in the CRISP-DM lifecycle
- Machine Learning — linear regression as the entry point, and R² intuition
- Data Leakage — the validation counterpart to getting the sample right
- Factor Model (Fama-French) — multiple regression applied to returns, where multicollinearity between factors is a live problem
- Quantitative Trading — the course is paired with Ernest Chan’s Quantitative Trading in the source note
- Data Quality — reliability and validity are its statistical formulation
- AI & Machine Learning
References
- Quantitative Techniques - Columbia Uni
- Introductory Statistics with Randomization and Simulation (ISRS) · Red State, Blue State, Rich State, Poor State — Gelman