In-car user testing — useful tips for testing your mobile apps with drivers
All the people that had ever tried to test their applications or products with users are aware that a good test is not only a matter of a well-designed research scenario but also a lot of logistics and practical organisation for an in-car user testing.
This is even more true when you need to go out of your cosy lab or conference room and face the unpredictable circumstances of the user’s car and road traffic. This is where the fun begins ;-)
Use your imagination when preparing for the in-car user test
Of course, as always when you prepare research, start planning the test of your application for drivers with the questions you want to study and with a decent testing scenario.
Try to imagine how would the test look like in details:
- Will you conduct the full test in the car or you can start it in the lab, office, café or at user’s home before you hit the road? Maybe you should invite the user for a cup of hot coffee after the drive to sum up the test?
- What route would you like the user to take? Do you want her to drive her usual way somewhere or go to some specific location? Will the user drive you back? If not, how will you get back home/ office from the test?
- Will the whole test be recorded? What will you record? Will it take place at day or after dusk? What is the weather, so also the light, going to be like on the day of the test?
- What recording tools will you need? Where will you place them? How will you make sure they will not run out of power or storage space during the test? How will you carry them before and after the test?
- Will you need internet/GPS access during the whole test or only at some stages? How will you make sure it is available in different places? What will you do when the access is lost.
- Will you use your phone or the user’s smartphone? Will it be paired with the in-car screen? Do you need any backup? What happens if the smartphone battery goes low?
- Where will be the smartphone located? Do you want to record the way the user uses it on everyday basis, so sometimes against the law, or would you rather concentrate on the safety requirements?
- Where will you sit in the car? When to ask which questions? When it will be best not to ask questions?
Pick the right place to start and finish the test
There are different logistic, but also technological factors that will determine the best place to start and finish the usability test with a driver. I recommend considering at least 4 of them:
- In what circumstances would the user usually use the app?
Would it be an everyday use on the route from home to work, or from work to the child’s school, etc? Or is it rather connected with going to some new place, maybe city, the user does not know? Or maybe there is a specific place your app is dedicated for?
Try to recreate the natural circumstances and route for your scenario as much as possible, e.g. if the user should use the app at the way to work, try to accompany her some day in the morning, when she actually drives to work. Then it would be natural to start the test at the user’s home – maybe it would be even possible to have a cup of tea together before you go? If yes, it would be great if you bring along a croissant or two ;-) Insights about the context of the service you are providing may turn out invaluable.
- How would the user normally start using the app?
Would it be rather the last moment before she turns the engine on, when standing in the traffic jam or at the gas station when she wants to find the closest place to park cashlessly with your app? Or maybe she would use it for some planning and will need to find some information or set up something at home an evening before she goes anywhere?
If you want to test something that would naturally be done out of the car, you can do it in the lab or any calm place where you can sit down with the driver. If the app is usually searched for or turned on in the car – start by sitting in and let the driver show you how she does it her way.
Remember that the place your test ends also influences what information you get. Maybe if it started in the car, you can now talk for several minutes in some quiet place and sum up? If she is in hurry, ask the user if you can call her in the afternoon or tomorrow to ask what she thinks after several hours.
- What is the stage of development of the app for drivers you want to test?
If you have the functional app, do not hesitate to adjust 100% to user’s habits when possible. If you are checking some functionalities with existing users – ask them where and how they use the app when you recruit them and try to shadow them.
If your application is by now only a bunch of sketches or mockups, it may be wiser to start in some calm environment to teach the user a little bit about these artifacts and instruct her how to think about and interact with them. When the user is ready, it is high time to move to the car.
- How much time and effort will it take for you to adjust to the user’s natural use scenario?
Think about the user’s comfort, but do not forget about your time and budget. If it is not a problem to get to the user’s home or office to start a test from there – do it. But since she drives a car and you are going to go to some place she does not know either way, maybe she can pick you up from the office or drop you off there at the end? Or maybe she can imagine, for the sake of the test, that she does not work in another city, but in the office centre just several streets away from your place? One thing is for sure – at the end of the day you do not want to land in the middle of nowhere in the night with no taxi or bus home ;-)
You are always trying to balance pros and cons of some research decisions – just make sure you pick the right combination of factors that make it possible to validate the product and get a valuable feedback from the driver. For me – it usually means getting as close to the natural use of the app as possible, on the routes that the drivers would pick themselves.
Pick the right tools for in-car user testing
Think if you really need a video recording of the whole test. If the test is short and you will check with the users only one or two functionalities, it may be sufficient to record the interview with the voice recorder on your smartphone and use your smartphone as a video recording device in the crucial moments. Just make sure your battery is high, and you have quite a lot of storage space for saving the videos. You do not want your smartphone to black out just in the middle of the test ;-)
The most important con of this approach is that when recording the video with a smartphone, you set some kind of a device-barrier between you and the user. You also start concentrating on recording with your phone – it consumes part of your attention, which makes you a less concentrated and less attentive researcher.
From my experience it is better to record the user tests with drivers (and user tests in general ;-)) with a small portable camera used together with several types of grips.
At Skyrise we have used a Go Pro HERO 4 Silver for the last 18 months – it is just sufficient for our needs. It is light and can be used in every circumstances – that is especially important if you conduct the test outside the lab and as a single researcher.
The quality of videos is decent and easy to control (from 720p to 4K, we usually use 1440p and medium or ultra wide field of view), one charge of the battery lets us conduct up to 3 1-hour tests without recharging, and the camera is small and quite easy to use, which is extremely important when you travel or need to use it in unusual circumstances.
Extra tip: Pick the lowest quality possible which looks OK for you for the recordings– you will save storage space and energy of the camera. If you need, you can use the mode which includes both recording video and taking pictures every several seconds. Sometimes the set of pictures may be easier and faster to use in the report than cutting the parts out of the video.
Remember about the grips
Especially in the car, the camera is nothing without the grips. You do not need to spend a fortune on them. Use the grips that can be easily matched and reshaped. Most of them is small and will fit in most of the laptop backpacks.
When the table is available, the tripod works just great. When I am in the car with a driver, I usually use the multi-part grip with the sucker to place the camera near the rear-view mirror, viewing a full windscreen with the road in front of the car and the top of a dashboard – this is usually the area where the drivers keep their smartphones. If I want the camera to capture also the driver or other parts of the car, I can easily change the angle by manipulating parts of the grip or put the camera in the top right corner of the windscreen.
Extra tip: with Go Pro you can use the upside-down mode when recording if you need to. The option becomes handy if you do not want to waste time to rotate the video after downloading or uploading it to your PC.
Remember about the thorough preparation
At least one day before you take off for the test with a driver:
- check what kind of frames you get when placing the camera in different places in your own car. Experiment also with the placement and position of the smartphone. That will help you make a decision and set up the camera fast when you enter any car.
- try recording in-car in the different times of the day and different lighting conditions. The settings you need for the night or cloudy January afternoon will be different than for a sunny July noon. In case of a lot of sunlight, you may need some sun shield for a smartphone’s screen. Otherwise nothing will be visible on the video.
- check how much time will the battery in your device last and how much video recordings will your micro-SD card (or any other storage device) hold.
- at the end of the day check if the camera is charged with the storage card inside ;-)
As a practitioner who always tries to counteract the worst scenarios before they become reality, I would also recommend that you take your laptop to the field with you. Thanks to that you will be able to download the files immediately after recording and recharge the camera a bit before another test. If you also have a webcam camera and a roll of office tape with you, as well as some screen recording software on the computer, then you have an emergency kit of recording devices ready to use if needed ;-)
I recommend also that your smartphone is fully charged before you start the user testing – you may need it for communication, but also as a hotspot if there is a problem with the internet connection on the user’s smartphone.
Extra tip: Check the video settings of the camera once again just before you start recording. Then start recording as soon as possible and screen the camera from time to time to check if the light that means recording is on. If you have a problem of forgetting about the technicalities when conducting the tests, put the big question “Am I recording?” at the very top of every page of the list of research questions you have with you.
Learn as much as possible about the driver in advance
Use recruiting the drivers and setting to get as much information as possible.
Ask about driver’s habits, how do they drive, where they put their smartphone in the car, do they pair it with any devices in the car and how much they use it during the drive and during the day. Ask whether they have a smartphone car charger – if not, bring one with you.
One of the most important things you have to learn is whether the driver uses any kind of car phone holder. It may occur that the user has it, but does not use it or does not have one at all. This is the place where the research gets really tricky: you have to ask yourself if you want to test the application with the minimal intervention and maximum realism or do you put safety first.
Either way, I recommend you always have some car holder with you when you go on tests with drivers.
For many reasons, I very rarely make the decision that the need for full realism of the research context beats the need for responsible behaviour behind the wheel. Here is why:
When you work on applications for drivers and have a chance to test your products with them, you fast become more aware that what you develop may influence the lives and health of many people, even those outside your target group.
You see drivers distracted by talking to you. You see drivers so concentrated on the app that they stop looking at the street. You see people holding their smartphones in their hands or on their laps, looking down on them, losing their concentration and awareness of the surroundings of the car. You see people behaving irresponsibly and irrationally, forgetting about the pedestrians, other drivers, cyclists. And you definitely do not want your product to be the cause of it, neither a part of it.
Distracted driving, especially the one caused by using mobile devices, has been a hot topic for several years and using mobile devices while driving is banned to some extent by law in many European countries and United States. At the same time it is still a huge problem – e.g. the U.S. Center For Disease Control and Prevention estimates that 9 people are killed every day in the U.S. as a result of crashes involving a distracted driver. It is though very important to reduce the distractions to minimum – design the application so it requires minimum attention and interaction (would be great if it had a voice interface!), but also make sure the cell phone is located in the spot where having a look at it does not require the driver to take her eyes off the road.
If the driver you are meeting with is still unaware of the possible effects of being distracted by her phone, it is your role to educate her. I do it by shortly describing that the safety is most important (not only) during the research, that is why I ask her to concentrate mainly on the road and the traffic, and only secondary on the app. I underline also that looking down the phone when driving is dangerous – this social ad may be useful for building the awareness:
Avoid distracting the driver
But not only the electronic devices may distract the driver during the test. It can be also you and your questions that cause tension, pull the driver’s attention off the road and make it difficult to stay concentrated on driving. That is why you should:
- ask most of your questions before you start a ride or after you finish it
- ask the driver to talk-aloud what is she doing when driving, but try not to ask further questions or make her concentrate on the app. If you see it requires driver’s effort, tell her to stop. You will see how much attention does the app draw anyways by observing what is happening.
- ask some questions carefully when you stand in the traffic jam or stop at a red traffic light, but make sure answering them does not require more than 1-2 clicks in the app or much thinking.
Remember – a small talk between a passenger and a driver is not a crime, but you must be sensitive to the capabilities of the person you are interviewing in this specific circumstances.
Extra tip: If you can see that some functionality of the app causes distraction and may be a source of a dangerous behaviour on the road, make sure your team redesigns or removes it from the app.
Be an agreeable guest in somebody’s private zone
People treat their cars as extensions of themselves or their homes – bubbles of privacy in the public space. They decorate it, they keep their things inside, they play their favourite music or audiobooks on the car audio, they let it be maximally messy or keep it as tidy as they like. Each car is not only a utility, but also a real vehicle of memory for its owner (the fact well caught in the amateur ad that went viral earlier this year).
This is great news for you, because by visiting the car you can learn a lot about the driver – your customer, and use what you see as a starting point for the meaningful conversation. But beware not to cross the line – the driver may also feel very uncomfortable with the unfamiliar nosy person on the board.
As Michael Margolis remarks in one of his latest posts on Medium – when you are conducting a test with a user, you should behave like a good host – do your best to make her comfortable and build a rapport by being open, attentive and encouraging, but not overwhelming.
The “Gracious Host” rule applies equally well when you, the researcher, are the guest. A gracious guest arrives on time (the fashionably late rule does not apply here!) and is flexible, willing, appreciative and courteous.
So if I can give you just the last extra tip: when conducting the mobile app test with a driver, try to do nothing more and nothing less than be a gracious guest.