Federated Average Algorithm

The Federated Average Algorithm is a key component of Federated Learning, facilitating the aggregation of locally trained model parameters from multiple devices or workers into a global model. Here's a more detailed explanation of how the Federated Average Algorithm works:

1. Initialization:

• Initially, a global model with its parameters is defined. This model is typically a neural network architecture tailored for the specific task at hand (e.g., image classification, natural language processing).
• Each participating device or worker initializes its local model with the same parameters as the global model.

2. Local Model Training:

• Each device or worker trains its local model using its own local dataset. This training process is typically performed using standard optimization techniques such as stochastic gradient descent (SGD) or its variants.
• During training, the local model parameters are updated based on the gradients computed from the local dataset.

3. Model Parameter Aggregation:

• Once local training is complete, the updated parameters of each local model are communicated back to the central server or aggregator (often referred to as the federated server).
• The federated server collects the parameters from all participating devices.

4. Federated Averaging:

• The federated server performs aggregation, usually through simple averaging, to compute a new set of global model parameters.
• This aggregation process combines the parameters from all participating devices to generate a more robust and generalized global model.

5. Distribution of Global Model:

• The updated global model parameters are then distributed back to all participating devices.
• This updated global model serves as the basis for the next round of local model training.

Iterative Process:

• The entire process repeats iteratively over multiple rounds.
• With each round, the global model tends to improve as it incorporates insights from diverse data sources and learns from different device-specific patterns.

Advantages of Federated Average Algorithm:

• Privacy Preservation: Since raw data remains on the local devices and only model parameters are exchanged, federated learning preserves user privacy and data security.
• Decentralization: Federated learning enables distributed model training across devices, reducing the need for centralized data storage and processing.
• Scalability: It can scale to a large number of devices, making it suitable for applications with massive user bases.

Challenges and Considerations:

• Communication Overhead: Communication between devices and the central server introduces latency and bandwidth constraints.
• Heterogeneity: Devices may have varying computational capabilities, network conditions, and data distributions, necessitating techniques to handle heterogeneity.
• Model Drift: As devices update the global model based on their local data, there is a risk of model drift, where the global model may diverge from the optimal solution due to variations in local datasets.
• Security Concerns: Federated learning introduces new security risks, such as model poisoning attacks and privacy breaches, which need to be addressed through robust security measures. Overall, the Federated Average Algorithm forms the backbone of Federated Learning, enabling collaborative model training across distributed devices while preserving privacy and scalability.

The Experiment -

In this program I have implement 6 workers (virtual devices) that take the MNIST data and train on 10000 data points one each. The Global model is made using the Fed-Avg Algo that is used for the aggregation of the parameters.

Algo description -

1. Since the parameters of the main model and parameters of all local models in the nodes are randomly initialized, all these parameters will be different from each other. For this reason, the main model sends its parameters to the nodes before the training of local models in the nodes begins.
2. Nodes start to train their local models over their own data by using these parameters.
3. Each node updates its parameters while training its own model. After the training process is completed, each node sends its parameters to the main model.
4. The main model takes the average of these parameters and sets them as its new weight parameters and passes them back to the nodes for the next iteration.

Results

• Graph to showing accuracy of the 6 worker models in test set.