Implementing Federated Learning to enhance Agentic AI

Original Media · 43m · English · Podcast · Practical AI

TLDR

Lorenzo Sani, a research scientist at Flower Labs and PhD candidate at Cambridge, discusses his journey from physics to federated learning, explaining how federated learning enables training on private, distributed data without centralizing it. He explores applications in LLM training and agentic systems, shares personal projects like a Halloween costume-judging agent platform, and envisions a future where individuals run powerful AI pipelines locally on their own infrastructure.

For you

PRO members can sign in and request more information — or their own version of this digest, tuned to what they care about.

Sign in to unlock

Surprising Insights

•

Lorenzo and his flatmates built a Halloween costume-judging platform using local vision-language agents that guests tried to 'prompt inject' with signs asking for high scores.

•

The same federated learning infrastructure developed for privacy-preserving medical applications can be repurposed for party games, showing the versatility of local AI tooling.

•

Lorenzo's household uses a kitchen whiteboard for evening research discussions and plans to deploy an agent that records, summarizes, and connects those conversations to relevant papers automatically.

Key takeaways

•

Federated learning moves computation to where data resides rather than centralizing data, enabling training on private or siloed data like medical records while keeping raw data local.

•

Privacy in federated learning relies on data never leaving its source, but models can still memorize private information; techniques like differential privacy provide statistical guarantees but are context-dependent and sometimes impractical.

•

Open-source models offer more control and auditability for privacy-sensitive applications compared to closed-source models, since you can inspect, probe, and fix issues yourself.

•

Agentic systems generate complex trajectories containing prompts, tool calls, and environmental interactions — all highly private — making federated learning a natural fit for improving these systems without exposing user data.

•

Enterprises are increasingly deploying smaller, capable models locally and need control over training trajectories to debug failures, improve performance, and protect intellectual property without relying on external providers.

•

Running open-weight models on personal infrastructure enables creative, privacy-preserving applications like home automation, research briefings, and wildlife monitoring, turning AI into a personal utility rather than just a cloud service.

•

Collaborative learning techniques like federated learning can pool diverse real-world episodes (e.g., surgical procedures across hospitals) to improve models in high-stakes domains where simulation isn't yet viable.

Notable quotes

“The idea behind Federated Learning is the following. So data is collected in many different places at once. Right? We collect data from our edge devices like phones, or laptops, and so on and so forth. And all this data is usually then centralized into the data centers that are doing the machine learning pipelines to train models and to learn from them.”

▸ Notable quotes · 6:18

“Federated Learning tries to move the training pipeline, so all the computations that you need to train your machine learning model to where the data is collected.”

▸ Notable quotes · 7:12

“The first ingredient for protecting your privacy is the fact that your data never moves from where it's collected. So no one can intercept traffic or actually look at your data.”

▸ Notable quotes · 8:51

“I would say that if I had to develop a privacy heavy application, I would prefer to use an open source model as a starting point, especially because I have more control over it.”

▸ Notable quotes · 12:01

“Now these models, especially the most capable ones, those that can do things, are getting smaller and smaller. Easier and easier to deploy, right? The amount of usage that this model have, these models have, is is even bigger and bigger, right? So the scale of usage is increasing, the scale of the models is decreasing.”

▸ Notable quotes · 31:34

“My strong hope is that the work I am doing right now and the work I did in the past few years is gonna serve as a massive enabler for people. Like, as models are reduced in size, as time goes, as compute gets better, I hope that everybody will be able to run the same pipelines I've developed and I'm developing on their own laptop, on their own infrastructure, in their own houses, and to enable them to do whatever they want.”

▸ Notable quotes · 40:56

Chapters

•

1:50 — Background: Physics to Federated Learning: Lorenzo describes his physics background in Bologna, how he discovered federated learning during his master's while working on privacy-preserving medical applications, and his path to a PhD at Cambridge with the Flower Labs team.

•

6:06 — What Is Federated Learning: Lorenzo explains federated learning as moving training computation to where data is collected (edge devices) rather than centralizing data, enabling learning from private data like medical records while complying with privacy regulations.

•

8:51 — Privacy Guarantees and Limitations: Discussion of how federated learning protects privacy by keeping data local, but models can still memorize private information; differential privacy and other techniques provide statistical guarantees but are context-dependent and sometimes impractical for industry.

•

11:05 — Open vs Closed Models for Privacy: Lorenzo argues open-source models are preferable for privacy-heavy applications because they offer more control, auditability, and the ability to fix issues, whereas closed models are opaque and harder to remediate.

•

14:52 — Federated Learning for LLM and Agentic Systems: Application of federated learning to train language models on distributed private data, and extension to agentic systems where trajectories (prompts, tool calls, environmental interactions) contain highly sensitive user information that can improve agents without centralization.

•

22:12 — Personal Projects: Halloween Agents and Home AI: Lorenzo shares a Halloween party project where local vision-language agents judged costumes, guests attempted prompt injection, and describes other home automation ideas like research paper briefings and conversation recording.

•

31:05 — Enterprise Adoption and Future Vision: Enterprises are deploying smaller models locally and need trajectory control for debugging, IP protection, and interpretability in high-stakes domains; Lorenzo envisions a future where everyone runs powerful AI pipelines on personal infrastructure for both work and life enrichment.

Speakers

•

Chris Benson — Host

•

Lorenzo Sani — Guest

⋮