Managing QA is one of the most difficult to do things for new IoT product development and startups that enter this space. The biggest challenge in QA for IoT hardware is that there’s absolutely zero information about it that’s available to everyone.
Here’s how QA works for most of the teams developing IoT hardware products
Firmware is written on an IoT prototype, and idea is validated
Firmware, BOM and hardware design specs are generated and sent to a manufacturer
Sample hardware is built and tested to see if it is working
Revised specs are sent for mass production
If you find these steps to be good enough, you’re missing on the competitive advantage that technology brings to the table. And, you are increasing the efforts and risks that are even needed to compete in today’s market or bring a new product to life.
This is not QA, this is called Drunk Testing in IoT! The reason why it is called so is pretty simple, you do everything mindlessly, you simply check a hardware “whether it works or not” that too not extensively. That’s a disaster.
Moving away from Drunk Testing to efficient QA for hardware products
I will walk you through a very simple, yet efficient QA process that you can follow for your own IoT product. We will take QA for a BLE beacon as an example for the entire blog.
I personally used this QA process two years back for a really large scale BLE beacon deployment project that was deployed in a Fortune 500 facility and had zero defects – yes, that’s how efficient this process was!
The BLE beacon had the following sensory and hardware modules:
- BLE antenna
- Temperature sensor
- Humidity sensor
- Voltage regulator
We will take a look at where to start QA, how to approach QA for a BLE beacon and what you should or shouldn’t do. Trust me, you can avoid a lot of IoT hardware defects and a lot of unnecessary costs if you follow the right QA process.
Let’s say you are done with building your IoT prototype. And, you are just about to get some builds from a manufacturer to do a little beta testing of your real product, there are a few steps that you must know:
- Step #1: Order 20-30 pieces using the final beacon specs
- Step #2: Go for Firmware testing of the BLE beacon
- Step #3: Perform RF testing for the BLE beacon
- Step #4: Testing electrical characteristics
- Step #5: Mechanical testing of your BLE hardware
Should these 20-30 hardware prototypes have separate debuggers attached to them to be made flexible for a variety of testing use cases?
These steps when finalized transforms to a test plan. This is what a test plan looks like:
#1 Test plan governance document
- Create purpose of the plan
- Define testing procedures
- Build guidelines for analysis, results and sign off of the plan
#2 Creation of Reference documents
- Detail on your choices and references that were taken while building this reference document
#3 Building test plan overview
- Build operational description documents
- Define internal terminologies for the operational description documents
#4 Do Pre-test preparation
- Setup test equipments and environments
- Test setup and calibration
#5 Execute system tests
The execution of system level tests are described below. But, you saw that how things changed from those simple 5 tests to a more comprehensive, evidence based QA process
In very broad terms, a complete QA plan for hardware testing should have:
- Firmware level testing – You can expect to weed out 80-90% of all product related bugs right if you can do this right. Includes functional/unit tests and integration testing
- Hardware quality testing – Focuses more on the hardware test builds, includes low level tests such as expected voltages, current levels etc
- Mechanical design testing – Focuses on how reliable your built is, and does it really stands up to the environment related design considerations
So, as per this flow you are going to initially order 20-30 test hardware product samples from the production shop.
We just don’t straightforward order these BLE test beacons from the production floor.
We will introduce the foundation firmware and RF testing when we order these test hardwares. I first went for firmware testing with the project. And here’s what follows.
Caution: Do not downplay on the importance of recording results, witnessing and verification authority. They are extremely critical to hardware development, especially into IoT. Even experts that have been working in this field since long might not be able to fix or do everything properly, chances of this happening when you are technology manager or a high level decision maker are even less. Don’t buy sweet talks, but real reports and verified results instead when it comes to hardware.
We are a software development company that gradually got into IoT 7 years ago. So, our software testing practices are a bit different than what most other IoT companies would follow. Read here about what is TDD – We often use this approach when we write large scale firmwares.
Let’s say we wrote a firmware for BLE beacon. For firmware testing purposes we will then split firmware in different versions like V1, V2, V3, etc
These are the main units of the boards, they are the firmware critical parts of your BLE beacon that would absolutely require independent and isolated testing to deliver the quality that you would need.
These are very specific firmware versions and are specific to validate test results for the BLE beacon.
The test firmwares (v1, v2, etc) are firmwares focused on specific modules. For this BLE hardware, the test firmwares was split based on the hardware modules and the sensory modules. So, breaking down our firmware test cases, we can categorize them with:
- Single hardware module focused firmware
- Test firmwares that has two hardware modules in it
- Test firmwares that has three hardware modules in it
- Firmwares that have all modules in it, but tweaked for performance and optimization
If you are from a software background, you would wonder how big of a waste would it be to create so many different firmware versions. But, It is important to note that this is hardware, the approach varies as with different combinations you can come across different level of bugs and possibly product threatening scenarios.
The entire goal of creating this firmware testing suite is to make sure that you test your product for isolated and cumulative failures.
Running the first set of firmwares
The bring in how well each hardware module/sensor is working in isolation. This is a necessary step for you to find out if there’s any issue with the firmware when it comes to the individual components.
If you find any issue here, optimize the firmware and repeat the process again.
My first test firmware was dedicated specifically towards the accelerometer and was tested in isolation to make sure that this was good enough from both firmware and hardware design perspectives.
If you want to see the real results from my test plan report, here’s what they looked like
(Confidential information has been redacted)
Running the second and third set of Firmwares
Here’s what we are going to do next:
- Run tests on Firmwares with two modules paired together
- Create the third set of Firmwares that clubs all of these firmware modules together
When we run 2nd set of firmwares in testing setup, we are trying to find if two modules are going to work together efficiently or not, we try to explore if they could somehow create an issue when they are working together. Things like a module interfering with functionality of another module is actually an issue that I’ve seen surface from time to time.
This approach highlights incompatibility between any two modules in very early stages of hardware production.
With the 3rd set of firmware tests, we can can either go with 3 modules clubbed together, but, I would recommend from my experience that you can skip going for 3 module incompatibility and run firmware level testing on all of them placed together. As our hardware isn’t very complicated, we don’t need to spend more time running additional tests. As a final firmware test, we can test all of these modules together with our testing parameters and KPIs in place.
Why this is extremely important?
When you run the complete suite of firmware tests, you can easily weed out 80-90% of the hardware related bugs before hand. In my own project, I was able to remove 95% of the problems right when I was doing the firmware testing.
Debugging hardware is extremely complicated, on top of that if you are building a simple IoT hardware like a beacon, it would become extremely expensive to carry out RF testing using automated tools.
For my BLE project, we had a Fortune 500 working on it, so we had access to myriad of automated testing facilities that someone with less than $200k in a budget won’t be able to get access to.
Even if you have such a huge budget, I wouldn’t recommend you going for this type of testing unless you are working on an IoT product that has more than 22 layers in it. It simply isn’t justifiable enough when you can carry out simple yet extremely effective tests avoid unnecessary expenses.
With context to this BLE beacon, our testing goals are inclined towards:
- Field testing
- RF and Firmware testing
RF testing tools that are available in the market would cost you around $5k-10k, if you are fine with that then you can easily move forward with that path. It will be pretty straightforward if you go with RF testing tools.
But if that’s not something that follow these simple steps:
- Step 1: Get a hardware PCB with a firmware version specifically dedicated to your RF modules. We talked about getting one such module at the start of Firmware testing, so if you follow the firmware testing process properly you should be able to get this very easily
- Step 2: Implement a field testing plan – Sounds more complicated, but is pretty easy and straightforward
In Step 1, our focus would be more on testing RF module firmware in isolation with another sensor or hardware module.
If you are thinking “Why do we need to run so weird tests?”
So when your RF module is placed on a chip, RF performance is impacted by a wide variety of factors like:
- Power and ground lines nearby cause impedance
- Additional hardware and sensory modules interfere with your RF signals
Once you run this firmware and test modules + RF compatibility you should be able to generate a report like this
Once you pass all expected results with the firmware, it is now time to perform field testing. In order to do this testing you should go to different “workplaces” and see how your RF modules/transmission/reception quality is.
This process of field testing for a simpler BLE beacon type product usually takes around 2 weeks.
To make the field testing extremely cost effective you can use a BLE sniffer. To make sure that we are able to quantify RF’s field testing, we would use two parameters to draw conclusive results:
- RSSI signal strength with distance
- BLE packet loss information
The threshold that I used for the project:
- BLE packet loss
My own thresholds were very conservative as we were looking for exceptionally high performance. You can be a bit more aggressive with your thresholds and use the following values as a quality indicator as well:
- RSSI: -26dBM for a few inches, -100 dBM till 40-50 meters
- BLE packet loss: Should be less than 90% for connection intervals less than 100ms
Apart from those, if you are interested in a much more deeper testing and analysis. I prefer running tests to keep an extensive record of payload bits received and it’s description. The table below shows some of the test bits and description from a real life project.
I’ve spoken about this before and non technical founders often fall for this, don’t ever assume a range just because the specs say. If you don’t have the packet loss information, you won’t know what type of range you can actually expect in your hardware.
That’s everything you need to do for RF and Firmware testing for your IoT product. If you follow this process, expect to get extremely high quality well polished Beacon hardware that can match any expectations that you want.
Angular RF testing: If your antenna isn’t specifically designed to capture incoming information at all angles, chances are that your product isn’t going to work efficiently at all. You need to take your hardware and test it at various angles and distances to discover potential limitations before you define your product capabilities.
Testing electrical characteristics
BLE beacons are usually compact and can vary from 2 to 4 layer PCBs. It is extremely important to visually inspect the inner layers of the multilayer PCB before it is laminated. This is because it is impossible to visually test the inner layers of PCB once it is completed. Therefore, if you are working with a remote manufacturer request for the images before lamination or inspect the site.
These beacons are usually compact and can be from 2 layer to 4 layer PCBs. When you are building multi-level PCBs for your IoT project It gets extremely important to visually inspect your inner layers before the board has been completed. So, if you are working with a remote manufacturer request images or do inspection on site.
Mechanical testing for your IoT product
Before getting into vibration testing, I would like to mention that you should take very extreme caution on how you bind your hardware board to your product enclosure. If you go for binding them with screws you would end up increasing the width of your PCB.
There are different ways to make sure that you can bind your PCB with your enclosure without using screws. But that’s not what we are talking about, it was just a side thought that you should know about.
In my case the beacon was expected to suffer vibration into 3 directions. That gives us 3 degrees of freedom in which the hardware product was supposed to suffer vibrations. Our vibration tests were a 3 step procedure:
- Step 1: Isolated 1 dimensional vibrations – These were spring based tests that heavily pushed the beacon into a single dimension
- Step 2: Isolated 2 dimensional vibrations – An electrodynamic shaker with two degrees of freedom was used here
- Step 3: Combined 3 dimensional vibrations – The same electrodynamic shaker but now set with 3 degrees of freedom
The PCB prototypes that pass your firmware and circuit testing now has proven its worth in terms of electronic engineering design. The BLE reference project that I am referencing here had some beacons that were supposed to be placed outdoors. Which increased the range of temperatures to which the IoT Beacons would be exposed to.
Difference of freezing nights to scorching heat gave some unexpected (yet easy to handle) surprises. The actual fluctuation of temperature was around 62.6F to 97F in a day.
The initial trace lines were set at 0.05’. The design considerations were only for the max fluctuation of 15’F. These things happen, requirements change all the time. By this stage we’ve dealt with 99% of the most common challenges, and you will see how proper QA doesn’t only make you deliver right product, but also allow you to gain some form of agility.
We could’ve deployed 1,000 beacons and then would eventually have faced this problem, and if that would’ve happened it would have been catastrophic for the entire IoT project.
Right QA to the rescue
Remember when we spoke about Drunk Testing? Not subjecting to thermal tests would have been just that!
So, the moment me and my team saw the temperature fluctuations causing thermal breakdowns, cracks and trace line disruptions. We increased the trace width from 0.05’ to 0.07’. That required absolutely no modifications in the hardware design. We just communicated this information to manufacturing lab, new 20 prototypes came out and just like that everything was fixed.
In my case, beacons were fixed to a surface that vibrates but there wasn’t really any need for an extensive drop test. But we did some drop testing from our end to make sure the enclosure and electronic design are good enough to bring reliability to the table.
Water proofing can mean a lot of things for a lot of people. Some of the hardware startups I met applied DIY epoxies to their hardwares and thought it w
The same goes with a DIY silicone as well. Be it epoxy or silicone, if you do a DIY implementation, it would simply end up cracking or wearing off in less than 2 hours of applying.
Make sure that the hardware that passes your QA is actually waterproof. The first step for you would be to hire a professional to cover your hardware with an epoxy that is waterproof and actually bonds to your PCB. Once that is done the hardware should be dried and cured at the right temperature for a reasonable amount of time. A hardware that goes through the waterproofing process like this is actually resistive to water.
To add a check on your waterproofing QA, make sure to test the hardware by yourself and get reports on how the waterproofing was done (temperature that was used for drying, time duration of curing, etc).
I covered how to effectively tackle trace quality related problem in my previous blog on IoT hardware product development guide. If your IoT hardware is even mildly complicated, consider adding a design pattern that lays a foundation for placing trace lines.
As a high level checklist for the sake of QA, I would consider the following:
- No two trace lines should cross each other
- There should be sufficient metal on your trace
- There should be a considerable distance between trace lines to avoid short circuit
If you using automated machines to produce hardware, you won’t find these errors to be common.
Log file generation in hardware devices
Last but not the least, keep logs of firmware related operations that happen in your hardware. These logs are often extremely helpful in debugging and would make it extremely easy for the customer to collaboratively diagnose issues, especially if you are planning to provide less on-site and more remote support services.
Once all of these tests are done, ideally your or someone from your QA team should look into creating a test clearance documents that has every detail of what we’ve looked into so far. This documentation not only lists QA measures were taken, but also brings in tons of insights into why certain actions were taken and what outcome was expected.
In some of the large IoT projects that I’ve been a part of, the original built modules and firmwares that existing leadership had no idea of. QA documentation in my opinion is one of the most collaborative front of reducing these things.
Ideally a QA document for large scale project should be extremely data driven, lists out stakeholder concerns, business concerns, customer concerns and should paint an history QA evolution of the product.
That’s it for now, what we looked into a pretty solid QA process that doesn’t costs you $100,000+ to implement and is extremely easy to implement whether you have a tech or even a non tech background.
If you have any questions, feel free to post a comment. If in case you need immediate feedback on something skip the long route and mail me at email@example.com