SoC Co-Is Paul Clarke and Spyros Samothrakis, together with British Academy Post-doctoral Fellow Dr Annalivia Polselli and MiSoC post-doc Dr Binzhi Chen conduct research on the use of machine learning (ML) in the social sciences. The team also included former MiSoC PhD student Dr Damian Machlanski who is now a Research Associate in Causal AI at the University of Edinburgh.
ML is an exciting field at the interface of artificial intelligence and computer science, concerned with developing algorithms for solving complex high-dimensional (e.g. regression or classification) prediction problems.
These algorithms work best whenever Big Data are available, but ML also has great potential for quantitative social science. In particular, the powerful algorithms of supervised ML allow researchers to fit, or learn, algorithms that accurately predict an outcome variable when it is a high-dimensional or non-linear function of predictor variables, but without the analyst having to specify a functional form for this relationship or to choose which variables are important to the analysis and which are not. ML algorithms are flexible and accurate but typically produce only predictions and output that is hard to interpret scientifically.
Social scientists typically rely on standard statistical and econometric parametric models for the relationship between predictors and outcomes. They choose which variables to include in their analyses based on a combination of the existing literature, their own intuition, and statistical tests. These traditional models are much simpler and less flexible than ML algorithms but, crucially, have the advantage that the parameters have a clear scientific meaning which aids interpretation and enables us to test scientific hypotheses. However, predictions based these models could potentially be much less accurate than those based on ML algorithms if the true relationship between the variables is incorrectly specified or some relevant variable omitted from the analysis.
The focus of this strand is on ML for the causal analysis of observational data. This combines the benefits of standard statistics/econometrics in that we target a clearly defined and interpretable parameter like the average treatment effect (ATE) or the conditional average treatment effect (CATE) for different groups of people, but often have to correctly estimate relationships (referred to as nuisance parameters) about which we have no substantive interest or theoretical guidance but which are, nonetheless, important to the analysis.
Hence, using learners from ML can potentially allow us to draw more reliable conclusions from observational data. It also makes previously infeasible tasks, like causal discovery, possible
Our programme of research is currently investigating the use of various types of learners (e.g., Lasso, random forests, boosted trees, neural networks) to see how effective these are at reducing bias compared with conventional approaches.
Our current research is led by Dr Annalivia Polselli, Dr Binzhi Chen and Paul Clarke.
Double Machine Learning with Panel Data: From Individual Effects to Common Effects
1. Causal Inference and Panel Data
The fundamental goal in most empirical work is to estimate causal effects. The difficulty is not calculating just a correlation (or any other type of association) but determining whether an observed association reflects a genuine causal mechanism or is instead driven by coincidence or unobserved factors pulling on both the treatment and the outcome.
Researchers in economics and other social sciences usually have access to longitudinal or panel datasets, where the same units (individuals, firms, regions, countries) are observed across multiple time periods or waves. Panel data provide a rich source of information about the units of interest as they allow researchers to track how outcomes evolve over time and to compare units with different histories. On the other hand, the same features that make panel data so rich also introduce estimation challenges that a simple cross-sectional estimation technique cannot handle properly.
Three challenges complicate this task, the first common to any empirical study and the latter two specific to panel data.
Confounding factors. The researcher usually has access to a set of observed covariates (e.g., demographic characteristics, economic indicators, regional attributes, etc.) that influence both the policy intervention (or treatment) and the outcome of interest. To remove their confounding role, one must adjust for them. But how? In most realistic settings, the true functional form (i.e., how these observed factors enter the relationships among the other variables) is unknown. A simple linear regression imposes a specific shape in this relationship (i.e., linear). However, if that shape is wrong, the estimate of the treatment effect is misleading as it inherits the misspecification error. This problem is present in both cross-sectional and panel data.
Fixed effects. Every unit has characteristics that are stable over time and unobserved by the researcher, for example, intrinsic motivation, managerial quality, geographic endowments, cultural norms. The time-invariant unobserved heterogeneity (or fixed effects) becomes a problem whenever they are correlated with the included observed factors and treatment assignment. The standard solution is to transform the data so that the fixed effects cancel for example, by taking first differences across time or by demeaning each unit’s observations (the ‘within’ transformation), or to model the fixed effects in a way to make them uncorrelated with the included variables (correlated random effects). These transformations are well understood and widely applied in empirical research.
Interactive fixed effects. Fixed-effect transformations deal only with omitted variables specific to each individual that do not vary over time. But panel data also expose researchers to factors which may be both unobserved and time-varying. These can be captured using interactive fixed effects (IFE). This is a more subtle problem not captured using conventional fixed effects, where the unobserved factor is not one shock affecting all units equally, but rather a mixture of common factors (e.g., a global recession, a regulatory shift, a technological wave) that affect all individuals at the same wave, but which are allowed to affect affect different individuals in different ways. Ignoring IFE therefore leads to biased estimates even in long panels with many time periods.
2. Addressing These Challenges
Each of the three challenges described above has attracted dedicated solutions in the statistical and econometrics literature. The contribution of the ongoing research is to combine and generalise them.
2.1 ‘Double’ Machine learning (DML) for flexible covariate adjustment
The first problem (unknown functional form of the observed factors) is the domain where machine learning excels. Algorithms such as Lasso, random forests, gradient boosting, and neural networks are designed to predict (or, in jargon, learn) complex, nonlinear relationships from data without requiring the researcher to specify a parametric form in advance. However, plugging machine learning predictions directly into a causal model introduces bias in the final estimates. Specifically, the in-sample fit of a machine learning model is biased (regularisation deliberately reduces the variance but increases the bias of the point estimate), contaminating the final causal estimate.
The Double Machine Learning (DML) framework, introduced by Chernozhukov et al. (2018), resolves this for cross-sectional data. The core idea is to run the machine learning step on held-out data (cross-fitting), estimate residuals from the treatment and outcome equations separately, and then regress the outcome residual on the treatment residual. This procedure removes the bias introduced by machine learning regularisation and restores valid inference, unlike naive combinations of machine learning.
Our research programme uses DML.
2.2 DML with fixed effects
The second problem (time-invariant heterogeneity) combined with the first one is handled by transforming or modelling the data before estimation. In a former article of the group, Clarke and Polselli (2026) adapt panel transformations within the DML framework, proposing a method that handles both unknown functional form and fixed effects simultaneously. The DML method for panel data with fixed effects is implemented in the R package `xtdml` freely available on CRAN (Polselli, 2025).
A recent extension of our work (Baiardi et al., 2026) focuses on the case in which a time-varying unobserved confounder shifts both the treatment and the outcome. This problem requires a different solution. The standard approach is to find an instrumental variable (IV) which is correlated with the treatment but affects the outcome only through that channel, with no direct path of its own. When a valid instrument exists, the IV estimator recovers the causal effect even when the treatment is endogenous due to the time-varying confounder. The challenge is that IV strategies typically require the researcher to control for many observed covariates before the instrument can be considered valid, and misspecifying those adjustments generates inconsistent estimates. We address this challenge by developing a DML method for panel data with instrumental variables. The three elements of the method are: (a) it works on panel data, (b) it uses an instrumental variable (IV) strategy to handle the unobserved factors, and (b) it leans on the double machine learning (DML) framework to flexibly adjust for observed characteristics. The R package `xtivdml` implements the proposed method.
2.3 DML with interactive fixed effects
The third problem (unobserved and time-varying effects) has been addressed by a separate strand of the literature. Estimators such as that of Bai (2009) treat the factor structure as unknown parameters and use iterative principal-component methods to remove them. Rücker et a. (2025) extended this approach to high-dimensional cases with many covariates. However, these estimators are only valid under linear treatment and outcome equations, and break down when the true relationships are nonlinear: precisely the realistic case where machine learning is valuable. Combining IFE removal with flexible functional form adjustment was, until recently, an open methodological challenge.
We propose a panel DML-IFE method for settings where the unobserved confounder takes the more general interactive fixed-effects form (Chen et al. 2026). Here, a simple first-difference or time-demeaning transformation does not remove the confounder, because the factor loadings vary across units. The method combines the DML framework with an iterative removal of the factor structure. The result is a causal estimate of the effect of the treatment on the outcome that is robust to: (a) unknown functional form of the observed factors, (b) any latent common factors, and (c) nonlinear and discontinuous covariate relationships. The R package xtifedml implements the proposed method by extending an earlier package for linear models xtife.
3. Applications and Future Directions
Despite targeting different sources of unobserved confounding, the methods in our research programme share a common architecture and motivation: flexible machine learning adjustment, valid inference, and a panel structure that reflects how economic data are actually collected.
The practical reach of this toolkit is wide. Panel DML addresses fixed effects, panel IV DML handles time-varying endogeneity, and panel DML-IFE tackles interactive fixed effects, covering the estimation challenges that arise most frequently in applied work. Our survey (Baiardi et al., 2026) shows that around 68% of empirical articles published in American Economic Review between 2011 and 2018 used panel data methods, and roughly 40% of these combined panel data with instrumental variables. Interactive fixed effects arise with similar frequency in macroeconomics, finance, and cross-country comparisons. High-dimensional observational panel datasets where the treatment mechanism is not fully understood are the norm in applied economics, which is precisely the setting these methods are designed for.
Future work will integrate the DML framework with dynamic panel models, where lagged outcomes enter the treatment equation, and heterogeneous treatment effects.
Our previous research was led by Dr Damian Machlanski, Spyros Samothrakis and Paul Clarke. Based at the interface of computer science and social science methodology, it focussed on developing new learning algorithms for causal estimation
We also understanding the role that ‘hyperparameter selection’ plays in ensuring accurate and reliable learning takes place. Machine learning algorithms are highly complex compared with those used for traditional statistical estimation. Each learner has its own set of hyperparameters. The analyst needs to select the values of a learner’s hyperparameters to ensure it works well. In practice, researchers tend to overlook the importance of choosing ‘good’ hyperparameters and simply use the default values supplied with the software. We showed that more systematic and data-driven hyperparameter selection is needed to avoid poor performance.