HomeLM: A foundation model for ambient AI

The vision of Ambient AI in smart homes promises an intelligent system that understands and adapts to its occupants. Despite an abundance of smart devices offering varying capabilities, smart home ecosystems remain disjoint and fragmented. Each device has been carefully designed to serve a primary function in isolation, yet they rarely coalesce into a truly intelligent system. This article discusses a proposal for a novel foundation model, HomeLM, which can be designed to bridge this gap. HomeLM’s vision is to transform disparate sensor signals from heterogeneous devices into a cohesive and human-interpretable narrative of activity within the home.

From ambient sensing to ambient AI

The ambition of smart homes extends beyond simple automation. Homes should instinctively gather context, adapt to their occupants’ needs and interact naturally. A key enabler is ambient sensing, which involves the continuous and unobtrusive collection of environmental characteristics without explicit user input. Examples include Wi-Fi reflections that indicate movement, mmWave radar patterns that reveal micro-motions and gestures, changes in Bluetooth Low Energy (BLE) RSSI as wearables enter a room and fluctuations in signal attributes of PIR sensors, ultrasound radars, microphones or even cameras.

Although task-specific algorithms exist for presence detection, distance estimation and gesture recognition, a critical missing piece is a unified model capable of interpreting the invisible signals emanating from sensors placed in different parts of the home. The core challenge lies in converting these raw sensor streams into meaningful and actionable information. To enable embedded intelligence, simply sensing doesn’t suffice. Raw sensor data, while abundant, is often noisy, incomplete, and inherently ambiguous. For example, a motion sensor can detect presence, but not who is present, what they are doing or their emotional state. Wi-Fi reflections can indicate movement, but distinguishing between a person walking, a pet or even a robot vacuum requires more sophisticated interpretation.

To achieve Ambient AI, these isolated and low-level signals must be fused, contextualized and interpreted to infer higher-level human activities, intentions and even well-being. HomeLM proposes a leap from data collection and sensing to intelligent understanding, which can enable the home to preemptively interpret user needs, rather than reacting to explicit commands.

Lessons from cross-domain foundation models

In prior work, cross-domain foundation models have been used to achieve a similar fusion of multivariate signals.

In 2021, OpenAI published CLIP (Contrastive Language–Image Pretraining), introducing a new paradigm of vision-language models (VLMs). Instead of being trained on fixed categories, CLIP was trained on a vast dataset of images with natural language captions from the internet. It used contrastive learning, a machine learning technique where it embedded text and images into a shared semantic space and learned their relationships by differentiating between similar and dissimilar data points. It demonstrated learned relationships between visual concepts and textual descriptions, enabling zero-shot recognition, allowing the model to classify objects it had never explicitly encountered during training. This ability to understand general concepts rather than just memorizing specific labels marked a leap forward for vision applications.

Google Research recently published SensorLM, extending the paradigm of CLIP to time-series sensor signals. SensorLM was trained by aligning physiological and behavioral data (e.g., heart rate, step count, skin temperature) with textual captions over 60 million hours of data from Fitbit and Pixel watches. It learned to generate descriptive summaries such as “Sleep occurred between 11:00 pm and 6:00 am” or “Elliptical exercise between 7:30 and 8:00 pm” and showcased robust zero-shot activity recognition, few-shot health prediction and sensor-to-text retrieval capabilities.

The fundamental lesson from these models is clear – aligning raw signals with language (referred to as pretraining), a model can learn implicit relationships and showcase strong generalization across tasks, efficiently adapt with minimal supervision and even offer interpretability of its understanding in human terms.

Unifying sensor streams with language

Smart home environments are often rich in connected devices, equipped with a diverse set of sensing modalities capable of continuously generating data. Some examples of sensing modalities include:

• BLE and ultra-wideband (UWB) on devices and wearables capable of detecting proximity and localizing the user.
• Wi-Fi channel state information (CSI), capable of tracking user motion and detecting presence.
• Ultrasound and acoustics offering high-confidence proximity-based interactions.
• mmWave radar for capturing precise posture, vital signs, gestures and enabling micro-motion tracking with limited range and field-of-view.
• Environmental sensors (temperature, carbon dioxide, humidity, AQI) offering contextual awareness.
• Microphone arrays for acoustic information and directionality, derived using Angle-of-Arrival (AoA) from speech interactions.
• Cameras for images, motion detection outside and around the house and object identification.

Each of these modalities produces high-dimensional data that is inherently difficult to interpret directly. However, when considered collectively, they paint a continuous narrative of human presence and activity within a space. HomeLM would serve as the foundational model, learning to align these heterogeneous ambient sensor streams with natural language.

The concept of Ambient Intelligence spans far more than simple automation scripts; it requires:

• Explainability: To build human trust and understanding of the system’s decisions.
• Interoperability: To allow diverse sensors to contribute to a unified understanding of the home.
• Scalability: To ensure the model can adapt to new sensors, tasks and evolving home environments and requirements.
• Privacy-aware: Intelligence in the home should not come at the cost of intrusion.

While LLMs excel at processing textual streams of data, they inherently struggle with the high-dimensional, continuous and temporally extensive nature of signal data. HomeLM would transform raw Radio-Frequency (RF) captures, acoustic signals and environmental data into the rich contextual understanding that a large language model (LLM) or an AI agent can readily reason with.

Training HomeLM: A hierarchical captioning approach

Given the scarcity of extensive home sensor data annotated with human activity text, HomeLM can be trained by adopting the hierarchical captioning strategy introduced by SensorLM.

This approach involves three distinct layers:

1. Statistical layer

This layer would summarize raw sensor metrics, providing insights, for instance,

• “Wi-Fi CSI variance increased sharply at 10:05 pm.”
• “BLE RSSI dropped by 15 dBm.”
• “Ultrasound sensor recorded user proximity of 2 feet at 10:06 pm.”

2. Structural layer

This layer would identify temporal patterns and trends in sensor data, translating them into descriptions of movement or interaction, for instance,

• “Ultrasound device detected movement from hallway to kitchen.”
• “mmWave radar captured a user sitting on the couch in front of the TV.”
• “A smart device perceived an increase in BLE RSSI from a user’s phone.”

3. Semantic layer

This high-level layer would provide contextual understanding, articulating human activities and presence, such as,

• “A person entered the living room and TV was switched on at 10:22 pm.”
• “Two occupants entered the apartment at 5:09 pm and cooked in the kitchen.”

Deriving inspiration from SensorLM, HomeLM could be trained on these meticulously crafted captions to construct shared embeddings for both text and sensor streams. Utilizing a combination of contrastive loss (similar to CLIP) and generative loss (similar to CoCa), HomeLM would be capable of aligning and generating explanations for ambient signals.

Capabilities of a HomeLM

What makes a foundation model like HomeLM powerful is its ability to learn generalizable representations of sensor streams, allowing them to be reused, recombined and adapted across diverse tasks. This fundamentally differs from traditional signal processing and machine learning pipelines in RF sensing, which are typically confined to single tasks and modalities.

Traditional ML models for smart home sensing are often narrow in scope, for example:

• A BLE RSSI model for room-level localization or distance estimation.
• A Wi-Fi CSI model for user motion tracking, presence and fall detection.
• A mmWave radar model for micro-motion tracking, gesture recognition, tracking vitals and sleep quality.
• An inertial (IMU) model for gesture recognition, activity detection or user trajectories.

Each of these models excels in its specific domain but fails to generalize beyond it. Introducing a new task necessitates new data collection, labeling and an entirely new training pipeline, impacting scalability and flexibility. In contrast, HomeLM is designed to be task-agnostic and multimodal. Once trained on vast datasets of sensor–language pairs, it would gain powerful capabilities:

• Zero-shot recognition: HomeLM can recognize novel activities it has never explicitly been trained on. For instance, if it understands “someone cooking,” it can infer “someone baking” or “someone washing dishes” without requiring further retraining.
• Few-shot adaptation: For rare or critical events, such as detecting specific appliance misuse or a fall, HomeLM can adapt rapidly and effectively with only a handful of labeled examples, significantly reducing the data overhead typical of traditional ML.
• Natural-language interaction: Users can query their home’s sensor data in natural language through AI assistants like Alexa, Gemini or Siri. Imagine asking: “Were there any unusual movements in the kitchen last night?” or “Did the front door open while I was away?” HomeLM would provide direct, textual answers, eliminating the need to interpret raw sensor logs and seamlessly integrate with AI assistants.
• Sensor fusion: HomeLM would offer the ability to fuse data from heterogeneous sensors. Each sensor modality offers only a partial view of the home environment; BLE provides coarse distance estimation from devices, Wi-Fi CSI captures motion patterns, ultrasound sensor detects proximity with high confidence and an mmWave radar precisely captures posture, breathing and gestures. While these signals can be noisy and ambiguous individually, when integrated, they provide complementary perspectives that create a richer and complete understanding.
• Advanced reasoning: HomeLM’s multimodal encoders and cross-attention layers can be designed to align these diverse streams within a shared representation space, enabling the model to learn not only the distinct features of each sensor but also their intricate relationships. This fusion capability allows for complex reasoning that no single sensor could achieve.

An example of HomeLM in practice

Consider a typical evening scenario — you enter your apartment at 6 pm. Since your phone advertises BLE beacons periodically, your arrival is registered by your smart home devices. As you cross the living room, Wi-Fi CSI patterns shift, confirming your movement. You settle onto the couch, and mmWave radar in the TV detects a seated posture with regular breathing. You use your voice to turn on the TV, and the smart speakers triangulate your position in the living room. After you finish watching the TV, you go into your bedroom, and your ultrasound-enabled smart speaker detects your presence. Wi-Fi CSI shows minor changes once you’re in bed.

While these are merely data points in a time series to all these devices, HomeLM could interpret and summarize them as: “The primary owner returned home at 6:02 pm, sat in the living room, and switched on the TV. They watched TV for 1 hour and 32 minutes and then went into the bedroom. The device detected that the user motion decreased and inferred that the user had gone to sleep.”

While traditional ML models often output useful but disjointed probabilities or classifications, HomeLM, by contrast, can produce a coherent narrative. This shift from raw scores to contextual explanations is crucial for user experience. These narratives not only improve usability but also enhance system transparency, making the AI’s behavior more interpretable and trustworthy.

Research challenges and the path forward

Building HomeLM presents significant research challenges that must be addressed:

• Data availability: Unlike the vast internet-scale image-text datasets used to train CLIP, there are no equivalent large corpora of annotated home sensor data. The development of robust data collection and automatic captioning pipelines will be crucial.
• Heterogeneity: The diverse nature of sensor modalities (BLE, Wi-Fi CSI, ultrasound, RF radar) with their varying sampling rates, data formats and noise characteristics requires a flexible and adaptive encoder architecture.
• Generalization across homes: Homes vary considerably in layout, materials and device placement. HomeLM must be designed to be robust to these variations while generalizing effectively across different environments and deployments.
• Privacy-first: Even without cameras, ambient sensing raises legitimate concerns regarding consent and data storage. HomeLM must be developed with privacy as a foundational principle.
• Efficiency: Deploying large models on edge devices within the home is challenging. Efficient model architectures, edge deployment strategies and model distillation methods will be vital.

Imagine if you will…

The promise of intelligent homes with ambient sensing, wearable devices, AI for IoT (AIoT) and edge computing has yet to reach its full potential. A unified intelligence layer is essential to enable ambient AI experiences. CLIP revolutionized vision–language understanding by aligning images with text. SensorLM extended this paradigm to wearables, fusing physiological and behavioral data with language. The next step is the home itself. A foundation model that unifies diverse in-home sensors and environmental signals with natural language provides a path forward. Such a model would enable zero-shot recognition, intuitive natural-language queries, rapid few-shot adaptation and narrative explanations, fundamentally transforming homes from a collection of smart devices into truly intelligent environments. While challenges remain, the immense payoff of a future where our homes not only connect devices but also understand the patterns of daily life makes this endeavor worthwhile.

This article is published as part of the Foundry Expert Contributor Network.
Want to join?