Authors: Roberto Morales
Modern machine learning often begins with an implicit assumption: somewhere, somehow, the data can be brought together. But in many of the settings where artificial intelligence is most needed (hospitals, banks, public agencies, mobile devices, and sensor networks) the data cannot simply be pooled. They are private, fragmented, legally constrained, and statistically heterogeneous.
Consider a group of hospitals that would like to build a shared predictive model (see Figure 1). Each hospital has its own patients, its own data distribution, and its own priorities. One may care most about early detection; another may wish to reduce false alarms; a third may need to control disparities across patient groups. At the same time, a central health authority may impose broader requirements such as safety, robustness, transparency, and regulation.
In what follows, we use “decentralized” in a broad sense: data and computation are distributed across agents, while a coordinating mechanism may still exist. More precisely, the architecture is close to federated learning with an explicit coordination layer, rather than a fully peer-to-peer system.
while a coordinator represents system-level requirements such as regulation, fairness, and robustness.
The tempting shortcut is to let every hospital train the model on its own data and then average the resulting updates. This is the basic intuition behind many federated learning methods. It is attractive because sensitive records never need to leave the institutions that own them.
Yet this simple idea hides a mathematical difficulty. Averaging local updates is most natural when all agents are essentially solving the same problem. Real systems rarely behave so cleanly. Hospitals may serve different populations, follow different protocols, and optimize different objectives. What improves the model for one institution may have little value, or even a negative effect, for another.
Averaging is not a neutral operation
A better way to understand the problem is to stop treating decentralized learning as a single-objective task. Each agent solves a local learning problem, shaped by its own data and priorities. The system, however, must also satisfy global requirements: fairness, stability, robustness, interpretability, or compliance with external rules.
Mathematically, the key change is to stop pretending that all goals can be compressed into a single number without loss of meaning. Each hospital has its own loss, and the coordinator may add system-level criteria.
Suppose that M agents train a shared model with parameters \theta. Agent i has a local loss C_i(Θ), while the coordinator evaluates global criteria S_1(\theta),\ldots, S_N(\theta), which may represent fairness, robustness, stability, or regulatory constraints. A fairness term might penalize large discrepancies in error rates across patient groups. A robustness term might penalize sensitivity of the model to small perturbations. A stability term might discourage successive local models from drifting too far apart.
A coordinated formulation is
Since one cannot generally minimize all objectives in (1) at once, one may select a compromise through scalarization:
The problem (2) is not meant to eliminate the conflict among objectives. It is a way of choosing one point among many possible compromises. Scalarization is only the simplest way to select compromises; more general multi-objective methods may be needed when the trade-off geometry is nonconvex.
The set of meaningful compromises is often called the Pareto frontier (see Figure 2). Each point on this frontier represents a model for which no objective can be improved without sacrificing at least one other objective. The relevant question is therefore not whether local autonomy or global coordination should dominate, but where the right compromise lies.
frontier changes the balance between local autonomy and global coordination.
What does the coordinator actually do?
The coordinator does not need to see the raw data. Instead, it can send global penalties, weights, constraints, or correction signals that modify the local training objectives. Thus, each agent still trains locally, but not in isolation. A simple implementation is to replace the local loss Ci by a coordinated loss of the form
A coordinated formulation is
Local training directions are therefore no longer determined only by local data. They are tilted by the global objectives supplied by the coordinator before the local models are aggregated.
This is the conceptual difference between classical averaging and coordinated learning (see Figure 3). In classical federated averaging, coordination appears mainly after local training, through the aggregation step. In coordinated learning, global objectives already shape the local updates themselves.
training, while coordinated learning lets global requirements influence the local update step.
This is close to how real institutions operate. A hospital does not make decisions only from internal statistics. It also follows clinical guidelines, legal rules, ethical principles, and safety standards. These global rules do not replace local knowledge; they shape how local knowledge is used.
When no one sees the full picture
To illustrate this idea, consider a deliberately difficult version of handwritten-digit classification on MNIST. Five agents are asked to train a shared model, but each agent sees only two digit classes:
No single agent sees the full ten-class problem. This mimics the hospital example: each institution observes only a partial view of the global population.
Each agent trains the same ten-class network, but its local batches contain examples from only two classes. Thus, the global model must be assembled from partial and biased views of the classification task.
The model is a convolutional neural network trained for 50 communication rounds. Each agent performs one local epoch per round, which helps reduce the risk of agent drift. The global model is then evaluated on the full ten-class MNIST test set. The coordinator uses a simple regularization objective; its role is not to create missing data, but to prevent local models from moving too far toward narrow, agent-specific solutions. The results of the experiments are summarized in Table 1.
| Setting | Coordination \lambda | Test accuracy | Test F1 score |
|---|---|---|---|
| IID data | 0.00 | 0.9837 | 0.9836 |
| IID data | 0.87 | 0.9409 | 0.9404 |
| Non-IID data | 0.00 | 0.6600 | 0.6389 |
| Non-IID data | 0.87 | 0.8056 | 0.7999 |
When the data are IID, the agents already see representative samples of the global problem; too much coordination can therefore slow or distort local learning. But when the data are highly non-IID, coordination becomes useful. It limits the tendency of each agent to specialize excessively in its own two digits and helps the global model generalize beyond any single local distribution.
The goal is not to outperform a fully centralized model that sees all the data at once. Centralized learning has full access. Decentralized learning must work under privacy, fragmentation, and communication constraints. The goal is different: to build a useful compromise that respects the structure of the real system.
Why this matters
The future of artificial intelligence will not be built only in centralized data centers. Many important applications involve distributed and sensitive data: hospitals, mobile devices, banks, public agencies, research institutions, industrial systems, and sensor networks.
In these settings, decentralized learning is not only a matter of efficiency. It is also a matter of governance. Who decides what the shared model should optimize? How should local accuracy be balanced against fairness, robustness, and regulation? How do we select a model that is acceptable both locally and globally?
These questions cannot be answered by averaging alone. A multi-objective viewpoint makes the trade-offs visible and gives the coordinator a precise mathematical role: to make competing objectives explicit, to navigate the Pareto frontier, and to select compromises that reflect both local data and system-level requirements.
Decentralized learning is not only about training many local models; it is about helping many different learners build one model together. In this sense, its mathematics is not only the mathematics of optimization. It is also the mathematics of compromise.
References
[1] R. Morales & U. Biccari (2025). A multi-objective optimization framework for decentralized learning with coordination constraints. arXiv preprint[2] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, and others (2021) Advances and open problems in Federated Learning. Foundations and Trends in Machine Learning, 14(1-2), 1-210
[3] B. McMahan, E. Moore, D. Ramage, S. Hampson & B. Aguera y Arcas (2017) Communication efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, 54, 1273-1282
[4] K. Miettinen (1999) Nonlinear Multiobjective Optimization. Springer Science & Business Media, Vol. 12
|| Go to the CoDeFeL blog main page