Multimodal interaction design

IoT-based Thermoelectric Air Management For Smart Home

This design investigates the primary multimodal interaction of the thermoelectric air management smartly controlled by the Internet of Things (IoT)-based configuration for smart homes. Air cooling management was done through thermoelectric coolers and a wireless internet hub with various sensors to control the indoor climatic conditions based on outdoor conditions. This thermoelectric air management utilises the guideline of Peltier modules, thereby working as an air-cooler in summer and a warmer during the winter.​

Home_Interior_Design-Hero-Slider1.jpeg
Home_Interior_Design-Hero-Slider2.jpeg
Home_Interior_Design-Hero-Slider3.jpeg

Table of Contents

Overview

This design aims to build up an IoT thermoelectric air management and leverage smart homeowners’ experience in terms of efficiency and eco benevolence. The design process will utilise recent studies relating to these topics: IoT, Peltier, thermoelectric air conditioning, and energy consumption. Therefore, some multimodal interactions of this design are like some existing systems or can be found in science fiction. Other than that, this design further improves interaction design by simulating human-human interaction.

The system

Literature Review

Air conditioning units are being used for space cooling, and their popularity in regions with hot and humid climatic conditions results in high energy consumption [1, 2]. A significant amount of energy consumption is wasted due to inadequate planning, building architecture, design, installation, and energy management processes [3, 4].

Space cooling using a thermoelectric module, an electronic unit that converts electrical energy into a temperature gradient [5], is feasible to improve energy efficiency by the year 2030 [6, 7]. The smart home concept (Figure 1) enhances residents’ efficiency in terms of energy use, surveillance, and integration of life-related facilities via utilising a range of techniques, including computers, automated control, and multimedia [8]. Thanks to the smart home concept development, IoT has become a viable choice for improving process productivity while lowering operational costs and resources consumption [9]. IoT can detect and break down distinctive ecological limits, such as temperature, warm and humid weather, by gathering information from its sensors [10].

Figure 1. The basis for all these technological advances in a smart home is the IoT. Adapted from How It Works

Some studies [11-15] suggested that detecting the association between temperature variations of the space and the resident (i.e., body position, skin temperature, human psychological response) via infrared sensors or modern signal transfer methods ensures more energy efficiency. However, they are not cost-effective.

Taking advantage of adjusting a room air conditioner based on human temperature and comfort level [16], thermal systems or Peltier elements are optimal because they have solid-state construction for high reliability, precise temperature control, and low operating and maintenance costs [17].

Figure 2. Thermoelectric module activity for cooling and warming. Adapted from [10].

Interaction Modalities

Several modalities will be employed in this design, and they are divided into two groups: input and output. The information has touch-based gestures, voice commands, Peltier element, and IoT-based sensors to detect environmental parameters, such as humidity, temperature, light, and smoke. The output has haptic feedback (vibration and thermal), visual, sound, and automated activities, e.g., smarter slumber, sunshades, and entertainment. Compared to the study of the six senses and modalities [18], this design only covers tactile/haptic, auditive, and visual modalities.

Touch-based gestures: To start with, multi touch and a significantly richer gesture vocabulary will be used on the design’s central control unit and internet wireless hub. Since these devices have touch screens, the user is instructed clearly how to perform the gestures at the moment a finger lands on screen. Touch-based gesture interactions require continuous feedback, but it does not ensure these feedbacks are precise and the user’s finger or hand is recognised promptly. In this design, devices with a touch screen allow the smart homeowner to:

  • Double tap: Wake the device up. The device automatically turns off the screen after an amount of time to save energy.
  • Long press: Input fingerprint so the device can identify if the user is the registered resident and recognise if the user is a child or adult based on the manual data input during the first-time registration.
  • Tap: Perform actions on the touch screen.
  • Scroll: Scroll the menu of options.
  • Pinch open/close/Swipe: This gesture applies for surveillance live videos because the screen size may be small, and the user can navigate between areas by swiping on the touch screen.

As noted, the program receives event whenever a new touch point is detected, touch point moves or disappears as the user lifts a finger (or stylus) and when touch event is cancelled. The cancellation happens when the finger moves out of the device’s touch screen. For each touch point, position on screen is available.

Peltier element: This thermal system has two sides relative temperatures and can be controlled by using DC current. Both sides are connected and when one side gets cooler, the other gets hotter. In the design, thermoelectric module is employed in the internet wireless hub (or smart thermostat) and some automated home electronics, such as bath’s water, intelligent fridge, smarter slumber. Some devices which are in direct contact with human must have the temperatures vary around human neutral zone, between 28-40 Celsius degree (◦C). In this design, thermal systems are used as the highlighted feature in parallel with vibration, which works as secondary feedback. In a smart home, where the residents are surrounded with many electronic devices, Peltier elements offer noise-free solution for the user. However, the environment temperature may affect the user’s sensation which requires future improvements on this. Additionally, temperature variations must be controlled properly by the IoT in order to avoid uncomfortable stimulation.

The IoT-based thermoelectric air management for smart home has two modes: auto (including ‘default’ and ‘save energy’ settings) and manual.

  • Default: The temperature difference between indoor and outdoor climatic conditions maximally reducing or increasing 5◦C.
  • Save energy: Maximum temperature reduction or increase between inside and outside is 2-3.5◦C due to back heat transfer and inefficiency between inside and outside climatic conditions of the heat sink attached to the fan to exhaust heat.
  • Manual:
    • Cold: Room temperature, oven, sauna, bath water, and more is cooling down.
    • Normal: It means that the appliance is in the normal state.
    • Warm: It means that something is warming or exposing more heat.

Vibrational feedback: Vibrating motors are embedded in both central control unit and internet wireless hub. It provides relatively small amplitude directly to the skin through the surface of the device. The frequency of the vibration depends on the feedback.

  • One vibration pulse: One temperature level change.
  • Two vibration pulses: Reached the limit setting (maximum or minimum).
  • Continuous: Notification.

These patterns are applied for temperature, light, humidity, and other adjustment performed manually.

Voice commands and sound: The speech-based interaction works as a feedback medium like vibration. They are added due to the concern of accessibility. Nowadays, voice and sound commands are popular and preferred among people because there will be always hands-busy or eyes-busy situations and many people with permanent impairment must not be neglected. This design applies the conversational assistant feature on both central control unit and internet wireless hub, which are embedded an array of microphones for speech recognition and listen to 360◦ direction. All the input will be private, and the user must remember the commands and does not need to use other modalities to activate voice input. At this point, the design has not had a specific name, but the user is presumed to use key phrase based to activate the voice input. The persona’s gender of this voice interface can be modified according to the user preference. Regarding its characteristics, the system must behave friendly, and the voice should be calm. The scope of this design does not cover advanced smart home central control system, but only focuses on the thermal aspect of the holistic system. Therefore, the voice interface is designed as a fixed speech interface. The user must train the thermoelectric air management at the first use, and it will consistently follow this terminology. Accordingly, the data throughput is from 100-150 words per minutes for speech input (~120 bits/second) on average.

  • Confirm the voice and/or touch-based commands inputs.
  • Format: [action][object][purpose]. For example, “closing the sunshade for cooling down”.

This design does not employ eye tracker for the privacy concern and reducing energy consumption. All the main tasks will be conducted on the touch screen of both central control unit and internet wireless hub unless the user use voice and sound as the request. With vision, it is possible to focus attention very precisely, especially when it relates to important data like temperatures and energy consumption. Consisting with the goal, all devices with touchscreen will show meaningful data relating to energy consumption (e.g., how much electricity and CO2 is in use in specific room) as the user can know how to become a data-driven energy saver. For the user interface design, it is crucial to check the colour palette which can be used for colour-blind people. At this point, red and green colour are not in use.

Multimodal Interaction

This IoT-based thermoelectric air management for smart homes incorporates thermal feedback based on Peltier elements as the hardware and home automation solution as the domain. Furthermore, a comprehensive IoT framework includes four distinctive segments: sensors/gadgets, network, information preparation, and a user interface (UI) [10].

In the input group, tactile and haptic modalities are synchronised in this order: monitoring, control, and user interfaces (Figure 3). The mounted sensors or gadgets gather information from their current circumstance, which can be as essential as a temperature perusing or a perplexing as a full video feed. Some sensors can be packed together on one gadget. For example, the smarter slumber is used in the baby room. It has numerous sensors such as a temperature sensor, camera, sound receiver, and so on to ensure that when the parents leave the baby in the room, they can even take their eye on the baby and be present promptly in any case.

Figure 3. Input modalities are from the sensors (monitoring), wireless network (control), visual, touch-based, and speech-based (user interfaces)

Like other multimodal system, this IoT-based thermoelectric air management for smart home must strive for meaning. To interpret these inputs, the design must ensure that information is transmitted from the cloud and the central control unit to other devices and sensors within the smart home. These sensors and gadgets must be associated with the cloud through an assortment of strategies including cell, Wi-Fi, Bluetooth, low-power wide region organisations (LPWAN). However, the trade-offs between power utilisation, reach, and transfer speed are the challenges of this design. As soon as the information gets to the cloud, it will be handled immediately.

Thanks to the employment of IoT and smart home concept in this design, the data can reach the end-users in many ways and ubiquitously, like an alarm to the smart homeowner via email, text, notice, or the internet wireless hub. The text-based or speech-based notification with vibration when the temperature is excessively high in the baby room is a vibrant example. In the auto mode, the system will change the temperature consequently by means of predefined rules without needing the resident’s presence. In the concern of data security, the information must go through the encryption procedure for encoding.

Figure 4. Design space for multimodal systems. Adapted from [19].

According to Figure 4 and the mentioned IoT and smart home concept, this design utilises the combined sequential and parallel uses of modalities with the meaning in fusion. Regarding the alternate class of multimodal systems, the use of the design’s modalities is sequential, and they have linked together in the same task thanks to the cloud. For example, the smart homeowner can perform a tactile gesture by touching the internet wireless hub to feel the heat of the fevered baby, who is sleeping in the baby room, and giving a speech command to lower the temperature of the smart slumber. The synergistic class of multimodal systems means that modalities are parallel and they are linked together. For example, the user can select the room with the touch-based and adjust its temperature with a spoken command at the same time.

It is cost-effective to execute and perform an IoT framework in more unpredictable temperatures, warm and humid weather, “cooling informational collections having a serious level of non-linearity” [10].

Hardware

The artificial neural networks (ANNs) and IoT-based intelligent systems control energy consumption for improving energy efficiency. Thermoelectric module (TEM), e.g., Peltier module, which is an electronic device that changes electrical energy into temperature gradient. A study [20] suggested that the cooling combination of a IoT-based TE-AC system and the ANN technique will be simpler to implement and perform better on forecasting the desired results. Therefore, for further improvement, the design must take this finding into account.

IoT-based systems analyse various environmental parameters, such as humidity, temperature, and more, at regular intervals and then control the system. The data generated by IoT-based sensors are used to control the air-conditioner and automate activity, such as automatically switching the air-conditioner on/off, dimming the lights, closing/opening the curtain, when the room temperature is high/low.

Recently, there are many touch-screen central control units on the market that fits different user preferences. It can be mounted on the wall or placed permanently at a specific area. Whereas the internet wireless hub varies from personal devices such as desktop, laptop, mobile phone, and tablet, to a hand-size thermostat like the Nest Learning Thermostat in Figure 5a. Ultimately, this design proposes the idea of allowing the user to feel the heat via the wireless hub (e.g., Nest design) but still limits it within the human neutral zone. This feature only applies for caring and well-being purpose, excluding other tasks such as cooking, sauna, bath water. The goal of this finding is to simulate the human-human interaction via a device. Figure 5b demonstrates how tactile/haptic and other modalities work as a part of multimodal output to reduce the cost.
My FindingsNgoc Nguyen (Rachel)

Users and use Context

This design is for smart homeowners who are busy, single/living with family, and love automatic/hand-free management. The design is not relevant to people with severe impairments relating to tactile/haptic, auditory, and vision. The use context allows users to directly use the central control system via touching, rotating, adjusting the device interface and using their voice commands to control the display remotely. Consisting with the introduction, the IoT-based thermoelectric air management is designed for smart homes in regions with hot and humid climatic conditions. It is expected to apply for cold and arid climatic conditions in the future. In detail, air conditioning systems are used for space warming, which results in a large amount of energy consumption. Overall, this design is mainly for the private situation.

Example Interaction 1​

- User A activates the auto-saving energy mode for the second floor by voice command: “[name of the system], turn on auto-saving energy.”

- System replies: “Turning on auto mode for saving energy”.

Sensors on floor 2 start working.
User A is playing the game in this figure, and the sunshade is open. Both are exposing heat to the room. The heat from the light also affects the temperature and helps user A focus on the TV screen. The light should be dimmer.

- User A can see the light is dimmer, the sunshade is closing, and the air conditioner decreases the temperature by 2 degrees.

The TV’s brightness is gradually lower after all those feedback, which can save more energy.

Example Interaction 2

- User B uses a wireless hub to control the baby room while sitting in the living room by using touch-based gestures.

The baby has a slight fever, so user B wants to keep an eye on the baby while resting in the living room.

- System shows the temperature of the baby via the wireless hub thanks to the sensor embedded in the smart slumber. User B can also feel the hub is warming according to the temperature.

- If the temperature of the baby reaches the limit that user B set before (i.e., 37.5), the device will vibrate 2 pulses and the hub will speak: “checking the baby for cooling down.”

- User B is resting in the living room watching TV shows and assuming that user B set the autosave energy mode.

- System adjusts the light dimmer. The TV's brightness is gradually lower after all those feedback, which can save more energy. The backlight of the room will change into a more comfortable yellow shade.

Evaluation

Laboratory testing:

  • Background: A controlled user test is held in a usability laboratory with 10-15 test participants.
  • Participants: Adults (above 18) who are familiar with one of these topics: IoT, smart home, multimodal interaction, save energy. They can have temporary or permanent impairments but not too serious which affects how they perform the tasks with existing devices.
  • Setup: The system will be available for testing for a week with participants who have pre-registered and are subject to strict COVID-19 Feedback will be gathered with a simple questionnaire on a separate computer placed next to the system and followed by 10 minutes interview.
  • Evaluation: Conducting usability test to gather data. Conducting user satisfaction questionnaire (Likert scale, long or short answer). Standard SUXES questionnaires.

Before the evaluation: Participants are recruited and their background information is gathered anonymously, the circumstances of the controlled environment are examined.

During the evaluation: Participants would be using the mobile application without the guidance, some tooltips are offered for first-time users opening the application.

After the evaluation: Data analysis collected by moderator and observers along with data from questionnaires and interview.

Data Collection:

  • Qualitative:
    • Haptics signal recording and tactile temperature of the prototype combined with user think-aloud audio recording.
    • Video recording and notes from the interviews. One observer in charge of observing technology, system and another observe in charge of observing participant’s facial expression, action, thought.
  • Quantitative:
    • Data from the filled-in System Usability Scale (SUS) questionnaire and SUXES questionnaires.
    • Data on the central control unit, internet wireless hub, and cloud (log data).
    • Interview answer will be coded into themes.
  • The technical aspects of the mobile application and its associated devices will be evaluated.
  • The usability and user experience of the whole system will be evaluated.
  • The tactile/haptic, voice commands and sound, visual modalities will be examined.
  • Usability of the product will explore more new findings in terms of specific user in a specific context using a specific product/feature. For example, it should answer at least these questions:
    • Does the IoT-based thermoelectric air management help your smart home save more energy?
    • How comfortable are you when the temperature is adjusting?
    • (For users with disabilities) What can be improved to meet your preferences?
  • UX of the product
  • Expectation levels of the IoT-based thermoelectric air management in terms of context of use, modalities reception
    • Can you feel the temperature change? Which levels?
  • Improvement notes and TTS evaluation and acceptance.
  • Safety levels when using the system, especially about the temperature and data aspects.
  • Inclusiveness levels
  • Evaluation metrics: Recognition accuracy for all three modality interactions, ASR confusion matrix to analyse speech recognition accuracy, gestures recognition accuracy, and users’ haptic pattern recognition accuracy.
  • Real-life testing with set-up at smart home environment.

Self-heating handling: The Peltier module itself generates heat in addition to the heat coming from the object to be cooled. Therefore, the heat sink must be able to dissipate this self-generated heat in addition to the heat transferred across the module from the cooled object.

Missing vibration: User can only check the setting manually in the central control unit by using touch-based gesture. If the vibration is set for notification, voice command cannot be used to turn it off. User must use touch-based gesture to turn off the notification.

Maximum reach: The vibration pulses can help the user to acknowledge that s/he has reached the limit of the choice.

Two-step confirmation: The system will always ask the user if s/he would like to it to perform the task in the manual mode.

Identification: The system needs the users input their fingerprint, face, and voice at the very first use. The system will learn the voice (e.g., tone, volume, gender) of the homeowners to recognize and customize according to their preferences.

Heat transfer feature: The system can only transfer the heat as a type of feedback within the acceptable temperature that human can handle (below 42.3 Celsius degree).

References

  1. Irshad, K.; Habib, K.; Saidur, R.; Kareem, M.W.; Saha, B.B. (2019). Study of thermoelectric and photovoltaic facade system for energy efficient building development: A review. Clean. Prod. 209, 1376–1395. [CrossRef]
  2. Mohammed Ali, Anwar & Abdul Shukor, Shazmin & Abdul Rahim, Norasmadi & Mohamad Razlan, Zuradzman & Zahid Jamal, Zul Azhar & Kohlhof, Karl. (2019). IoT- Based Smart Air Conditioning Control for Thermal Comfort. 289-294. 10.1109/I2CACIS.2019.8825079.
  3. Kamal, A.; Al-Ghamdi, S.G.; Koç, M. (2019). Role of energy efficiency policies on energy consumption and CO2 emissions for building stock in Qatar. Clean. Prod. 235, 1409–1424. [CrossRef]
  4. Sáez, P.C.; Astorqui, J.S.C.; Merino, M.D.R.; Moyano, M.D.P.M.; Sánchez, A.R. (2018). Estimation of construction and demolition waste in building energy efficiency retrofitting works of the vertical envelope. Clean. Prod. 172, 2978–2985. [CrossRef]
  5. Cheng, Chin-Chi & Lee, Dasheng. (2014). Smart Sensors Enable Smart Air Conditioning Control. Sensors (Basel, Switzerland). 14. 11179-203. 10.3390/s140611179.
  6. He, Y.; Liao, N.; Bi, J.; Guo, L. (2019). Investment decision-making optimization of energy efficiency retrofit measures in multiple buildings under financing budgetary restraint. Clean. Prod. 215, 1078–1094. [CrossRef]
  7. Piderit, M.B.; Agurto, S.; Marín-Restrepo, L. (2019). Reconciling energy and heritage: Retrofit of heritage buildings in contexts of energy vulnerability. Sustainability, 11, 823. [CrossRef]
  8. Li, W., Yigitcanlar, T., Erol, I., & Liu, A. (2021). Motivations, barriers and risks of smart home adoption: From systematic literature review to conceptual framework. Energy Research & Social Science, 80, 102211.
  9. Song, Wei & Feng, Ning & Yifei, Tian & Fong, Simon. (2017). An IoT-Based Smart Controlling System of Air Conditioner for High Energy Efficiency. 442-449. 10.1109/iThings-GreenCom-CPSCom-SmartData.2017.72.
  10. Aarsh, A., & Kumar, V. (2021). An IoT-Based Thermoelectric Smart Airconditionr Using Peltier(No. 5077). EasyChair.
  11. Jang,Hyeonwoo&Kang,Byeongkwan&Keonhee,Cho& Jang, Kyu & Park, Sehyun. (2019). Design and Implementation of IoT-based HVAC and Lighting System for Energy Saving. MATEC Web of Conferences. 260. 02012. 10.1051/matecconf/201926002012.
  12. Paniagua,Estephany&Macazana,Jhonatan&Lopez,Joshi & Tarrillo, Jimmy. (2019). IoT-based Temperature Monitoring for Buildings Thermal Comfort Analysis. 1-4. 10.1109/INTERCON.2019.8853608.
  13. Ruano, Antonio & Silva, Sergio & Duarte, Helder & Ferreira, Pedro. (2018). Wireless Sensors and IoT Platform for Intelligent HVAC Control. Applied Sciences. 8. 370. 10.3390/app8030370.
  14. Cheng, Chin-Chi & Lee, Dasheng. (2016). Enabling Smart Air Conditioning by Sensor Development: A Review. Sensors. 16. 2028. 10.3390/s16122028.
  15. Zang, Miao & Xing, Zhiqiang & Tan, (2019). IOT- Based Thermal Comfort Control for Livable Environment. 10.1007/978-3-030-22968-9_32.
  16. Azman, Noor & Ismail, Mohd & Ramli, Nor Azuana. (2020). Control and Monitoring Air Conditioner: Perspective of Internet of Things. Test Engineering and Management. 83. 8215-8220.
  17. Thermoelectric Coolers. Retrieved from https://www.lairdthermal.com/products/thermoelectric-cooler-modules
  18. Silbernagl, S., & Despopoulos, A. (2007). Taschenatlas Physiologie. Georg Thieme Verlag.
  19. Laurence Nigay and Joëlle Coutaz. (1993). A design space for multimodal systems: concurrent processing and data fusion. Human Factors in Computing Systems, INTERCHI ’93 Conference Proceedings, ACM Press, 172–178.
  20. Irshad, K., Almalawi, A., Khan, A. I., Alam, M. M., Zahir, M., & Ali, A. (2020). An IoT-based thermoelectric air management framework for Smart building applications: A case study for tropical Sustainability, 12(4), 1564.
Ngoc Nguyen
©2025

You cannot copy the content of this page.