Can Your Smart Assistants Work With Other Devices? A Test Analysis from Allion’s Interoperability Validation

As we mentioned in last article: Smart Enough? The Three Most Frequent Issues in Smart Assistant Applications, smart assistant can be vocally activated to provide entertainment support, help search information, and control other smart devices in houses, offices, or hotel rooms. Generally, a smart assistant, to complete these tasks, has to work with other devices, such as going online with wireless access points (AP) or synchronizing with smart phones via Bluetooth. However, we found several interoperability issues after paring up six smart assistants with 20 APs and 20 smart phones. In this article, we argue that, even for the products that have been certified by Wi-Fi and Bluetooth, interoperability issues can occur during the complex matches among the various products in the market. We will share the test results from Allion’s Wi-Fi and Bluetooth interoperability validation and explain the issues, including irrational latencies, connection failures, inconsistent user experience, and functionality shortages, caused by poor interoperability of products.

 

Smart Assistant’s Most Mature Application

Among the three applications mentioned previously, entertainment support (e.g. listen to music or radio) tends to be the more mature technology and thus be the most widely used service for smart assistant owners. According to a survey from Voicebot Ai., in January, 2018, over 75% of the users, on monthly basis, listen to music via smart assistant. A report from Kim Bayley (i.e. CEO of Entertainment Retailers Association) and Geoff Taylor (i.e. CEO of BPI and BRIT Awards) also states that “over 80% of the use case around these devices [i.e. smart assistant] is listening music.” (Everybody’s Talkin’ Smart Speakers & their impact on music consumption, p. 25).

Figure 1: The Data Cites from Voicebot AI.

 

Wi-Fi Interoperability Issues

One way that allows the smart assistants to play music is to connect them to digital music providers, such as Amazon Music, YouTube, etc., via the 802.11 protocol (a.k.a. Wi-Fi). Nevertheless, Wi-Fi interoperability issues could occur due to the complex combinations between the various smart assistants and routers in the market. By simply pairing up the six most popular brands’ smart assistants (i.e. models from Amazon, Lenovo, Xiaomi, Alibaba, etc.) with 20 access points (i.e. the top-selling models from Apple, Google, Belkin, Netgear, etc), Allion’s experts have found irrational latencies and connection failures on some of the matches. These issues can seriously decrease user satisfaction.

Figure 2: Test Bed (Wireless Access Points) shows in Allion’s Device Library which catalogs over 10 thousands devices.

 

Irrational Latency 

When receiving a voice command from user, smart assistant has to connect to its database via internet to provide the appropriate reply. Therefore, we asked the 6 voice assistants (i.e. we named them as “P, M, I, A, D, and L”) to play songs from assigned artists via the 20 routers (i.e. we named them from “A” to “T”) in 10 times and averaged their response periods (sec.). We also used Ping to measure the throughputs between all the APs and smart assistants.

Figure 3: Test “P, M, I, A, D, and L” smart assistants with 20 wireless routers (A-T)

 

Unfortunately, we found the average throughputs of “I,” “A,” and “D” smart assistants are varied while connecting with different routers. For example, the average throughput range of “I smart assistant” were from 17.52 (Mbps) to 149.66 (Mbps). When we extended the distance between the DUT (Device under Test; smart assistant) and the test beds longer, the average throughput range of “I smart assistant” would become 4.25 (Mbps) to 115.23 (Mbps). “D smart assistant” had even larger gap under the longer distance test condition: 5.67 (Mbps) to 235.06 (Mbps). The varied throughput values had negative correlations with the DUTs response time, causing different levels of latencies to DUTs.

 

Connection Failure 

Besides latencies, the poor interoperability of smart assistants and routers can trigger connection failures. In order to provide a better entertainment experience, “Brand P,” for example, has developed an App to allow users to stream their media contents from different devices, such as PC or mobile phones, to smart assistants under a same Wi-Fi network. However, during the initial setup process, we found that, after receiving a connection request from the App on mobile devices, a connection failure occurred between “B router” and “P smart assistant.” The issue could finally cause the whole system crash.

 

Bluetooth Interoperability Issues

Some users are used to connecting smart assistants to smart phones via Bluetooth, allowing them to synchronize personal media contents (i.e. playlist for music, video, etc.). However, we found functionality shortages and inconsistent user experience after paring up the 6 DUTs (i.e. smart assistant) with 20 mobile phones (i.e. models from Apple, Google, SONY, OPPO, etc.).

Figure 4: Test Bed (Mobile Phone) shows in Allion’s Device Library which catalogs over 10 thousands devices.

 

Inconsistent User Experience 

During the Bluetooth interoperability validations, we found that some Bluetooth connections between smart assistants (“P, M, I, A, D, and L”) and mobile phones (i.e. we named them from “A” to “T”) are able to be rebuilt once turning on the Bluetooth function of mobile phones. Yet, same phenomenon was not observed on some of the matches. For example, smart assistants and phones from the U.S., compared to those from China, were less likely to establish the connection automatically during our validation cases. The issue becomes critical because, if the connection can be automatically re-established after the switches, it means that one of the devices was keeping sending connection requests to another. This could not only make the device consume more power to create inconsistent user experiences but also cause security concerns on some of the cases.
Figure 5: Test “P, M, I, A, D, and L” smart assistants with 20 mobile phones (A-T)

 

Functionality Shortage

Besides issues discovered from establishing Bluetooth connections, we found functionality shortages during the synchronizations of smart assistants and phones. For instance, the volume bar on “S mobile phone” was not able to adjust to the volume bar on “L,” “I,” “A,” and “D” smart assistants. The “next” and “previous” function on “I smart assistant” could neither work properly. The “next” function changed to “Fast Forward” and the “previous” function changed to “Rewind” when paring up with “G smart phone.”

 

Potential Issues Revealed by Complex Matches

Allion’s experts have listed some root-causes that could cause the issues, including irrational latencies (Wi-Fi), connection failures (Wi-Fi), inconsistent user experience (Bluetooth), and functionality shortages (Bluetooth), mentioned in the early paragraphs. For Wi-Fi interoperability issues, one of the evidences shows that the incompatible designs of broadcast technologies might make the two devices (i.e. P smart assistant & B router) fail to exchange information, causing connection failures. They also suggest that, since chipsets of our test beds cover different manufacturers such as Qualcomm, Broadcom, MediaTek, etc., the uninteroperable chipsets could decelerate the connection speed of internet. For the inconsistent user experience and functionality shortages during the Bluetooth connections between smart assistants and mobiles phones, evidence shows that it might be caused by the incompatible setting of devices’ Audio-Video Remote Control Profile (AVRCP) and Generic Access Protocol (GAP). Nevertheless, these issues could not be discovered during the compliance tests of Wi-Fi and Bluetooth.

 

Conclusion

In conclusion, we have shared the test results from Allion’s Wi-Fi and Bluetooth interoperability validation and provide some implications by analyzing the issues, including irrational latencies, connection failures, inconsistent user experience, and functionality shortages, caused by poor interoperability of products. We argue that, even for the products that have been certified by Wi-Fi and Bluetooth, interoperability issues can occur during the complex matches among the various products in the market.

Dedicating to make electronic devices seamlessly integrate into users’ lives, Allion’s experts specialized in revealing interoperability issues by designing complex combinations from the variety of products. We are the not only the Authorized Test Lab of Wi-Fi and Bluetooth, but our Device Library also features thousands of products, including wireless routers, mobile phones, IoT devices. We further provide technical consultations and debugging support. The one-stop solution will thus allow vendors to reduce delays and uncertainties during the manufacturing process.

Let us know if you are interested in any of our service: service@allion.com