How does Facebook configure millions of servers every day?

Facebook, the world's largest social network, currently has more than 2 billion users on millions of servers. So to edit, change the configuration for each server, how did Facebook have to do?

Previously, to solve this problem, Facebook used the open source configuration system Zoo Keeper. In one run, Zoo Keeper only distributed configurations up to 5Mb up to 2500 servers. This speed does not meet the requirements of Facebook when they want to change the configuration for the entire server as quickly as possible when launching new features, all users (can be divided by zone geographic area) can be provided in the fastest way and must be true in real time. This means that all Facebook users in Canada and Facebook users in the US must use the same latest functionality.

How does Facebook configure millions of servers every day? Picture 1

To give users the best experience, the team of Facebook engineers want to create a new tool with a speed of configuration distribution at 5s and a configuration capacity 10 times that of Zoo Keeper. And they have developed a system that can automatically distribute the configuration on the entire server called Location Aware Delivery (LAD).

LAD consists of two parts:

1. The first part is a proxy located on Facebook's main system. Its mission is to provide the latest configurations to any server that needs it.
2. The second part is a distribution system, which provides configuration information to all of Facebook's server systems. This system continuously checks the latest configuration tables from the proxy system, when a new configuration table is discovered, a server diagram tree is created and will automatically perform updates on all that server. This tree structure will ensure that updates are sent to proxies, servers, and child servers. Such a process will help solve the overload when distributing changes at the maximum level.

LAD of Facebook can increase the update file size to 100MB, distribute up to 40,000 servers once, increase 16 times compared to Zoo Keeper.

See more:

1. Google and 2 billion lines of code, how does the search giant work?
2. Want to know what surprise Facebook holds about you, visit here