How Augmented Reality-Based Indoor Navigation Works

How Augmented Reality-Based Indoor Navigation Works
By
Andrew Makarov,
Augmented Reality Solution Architect at MobiDev

The proliferation of AR-based indoor navigation applications in different consumer sectors is estimated to experience tremendous growth in the coming years as technologies continue to evolve. As users increasingly become more digital savvy and ready to apply new technologies in their everyday life.

There are many like Alex, a senior manager at a tech company. He was looking for an Apple Store, so he went to the Westfield Valley Fair, a midsize mall in the Silicon Valley. Which is easy to find via Google maps even if you’re not local. It took him 5 minutes to buy a new iPhone. But before that, he spent like 20 minutes walking around to find the store inside the mall. Despite using public maps placed on some type of city lights around the mall, it still could be a challenging task. As facilities are growing, the way people navigate is the same as decades ago.

As Alex shared the story, I was already digging into the indoor navigation topic. And, obviously, thought of beacons to save the World. After a year and a bunch of experiments, I don’t think that way anymore.

We believe there is a clear road map to success in the world of Augmented Reality technology especially when you consider that organizations are now moving from standard solutions such as face masks and AR games, and instead are going for custom solutions (think AR Indoor Navigation applications).

However, to achieve that success, business owners, project managers, CTOs, and CIOs must first gain a 360-degree view of this nascent technology and how it can seamlessly suit their particular needs.

Through this primer, we want to give you a detailed look at Augmented Reality indoor navigation technology and how our software company can help you easily deploy it.

Note: We’ll focus on mobile AR technologies using ARKit for iOS and ARCore for Android – these are software development kits designed by Apple and Google respectively. These SDK’s allow software engineers to add augmented reality functionalities to native apps (iOS and Android apps). These kits are not focused specifically on navigation, but on AR itself.

Introducing Augmented Reality Indoor Navigation

Indoor navigation is quite different in terms of complexity compared to outdoor navigation. With outdoor navigation, millions of people currently use the technology as it doesn’t require much performance; modern smartphones and even smart watches have built-in GPS and maps.

AR indoor navigation technology, on the other hand, is quite complex as it consists of 3 modules which have to be factored in; and these include: Positioning, Mapping, and Rendering.

Currently, Mapping is the only module that is straightforward. With a map and coordinates, it’s easy to make a route. This module is easy to upgrade and customize based on a given business use case/need.

Rendering module manages the design of AR content and its work directly depends on the precision of positioning. We can easily draw a route in 3D but face some challenges while matching the virtual objects with the real world. The quality and precision of rendering will grow along with the evolution of the ARKit.

But when it comes to Positioning, the situation becomes a little bit daunting. There’s no accurate way of determining the precise location of users indoors, including the exact floor. Determining the proper accuracy levels is also a challenge. For example, is 10 meters enough? What about 5 meters? Or 1 meter?

The good thing is we have managed to solve this conundrum. Our research shows the minimum accuracy for human perception is no more than 10% of the minimum distance between the points of destination. Therefore, if the minimum distance between gates at an airport is 40 meters, the required accuracy for indoor navigation is 40*0.1 = 4 meters. In an office building, where the doors are 5 meters apart, it’s half a meter.

Available Technologies for Indoor User Positioning

Even though there exist a number of technologies for user positioning, most of them have a few

GPS is one of the existing disadvantages. Here’s a look at some of them.

GPS For Indoor Navigation

technologies in the navigation space; however, it cannot provide accurate positioning inside a building. It’s relatively accurate in large and low-rise buildings, such as airports; but it can’t determine finer details such as the floor number, and the only way to achieve this is to do it manually, like the way it’s done with Apple & Google Maps. The takeaway here is GPS works but it’s not a viable solution.

Visual Positioning System (VPS)

Visual Positioning System (VPS) – is advanced compared to the two and holds so much promise. Google, for example, uses Street View data to clarify a user’s position in AR-based Outdoor Navigation, using surrounding buildings as reference points.

Furthermore, ARKit 2 introduced the ARWorldMap class, which can also serve a similar purpose. The ARWorldMap is essentially a set of feature points around a user, like “the world’s fingerprint”, which can be recognized.

However, we can’t entirely rely only on ARWorldMap to determine the exact location for indoor navigation because:

Beacon Based Indoor Navigation

Beacon technology is another buzzword when it comes to indoor navigation.The most common example is the 2,000 battery-powered Bluetooth Low Energy beacons installed at Gatwick airport in the UK which gives, as they claim, +/-3m accuracy. But according to Apple documentation, beacons give only an approximate distance value hence we can’t rely on the signal strength while trying to calculate this data manually.

And when you consider other factors such as cost ($10-20 per item), battery replacement (once every 1-2 years) and the working range (10-100 metres), it’s clear that the use of beacons for indoor navigation is effective only under certain conditions.

But all in all, beacons can still do a good job as most of the current indoor navigation technologies on the market are beacon-based and use AR for route visualization (only). However it’s possible to leverage ARKit and ARCore SDKs’ features to solve the problem of user positioning.

Visual Markers – AR-Based Indoor Navigation Solution

Visual Markers, also known as AR Markers, or ARReferenceImage are set to rewrite all these. A visual marker is an image recognized by Apple’s ARKit, Google’s ARCore and other AR SDKs. Visual markers are used to tell the app where to place AR content, for instance, “Hey, place it there”. Consider this video demo that demonstrates how to place content on a business card.

If we place a visual marker somewhere in space, on the floor surface or onto the wall, and store its exact location in the cloud, then, when scanning, we’ll receive not just “there”, but rather its exact coordinates in the real world.

Watch the video below as an illustration of how visual markers can be used for very accurate positioning with ARCore; up to several millimeters. In fact, we refer to this technology as SVPS – Simplified Visual Positioning System.

What about the accuracy levels? Clients usually want to know what accuracy levels to expect for their apps. We’ve discovered that the farther it’s from the starting point, the more inaccurate the result is. To solve this problem, new visual markers have to be placed on average every 50 meters.

How To Develop An AR-Based Indoor Navigation APP

Are you planning to deploy an AR-based indoor navigation application for your company? If your answer is yes, well that’s a good strategy. However, you need to first understand how to choose the right software company/team, how the development process looks like, what are the timelines, among other factors.

To start this process, ensure the augmented reality company you choose has the following team members: iOS/Android developers with ARKit/ARCore experience, UI/UX designers, a project manager and a QA engineer. Let’s take a closer look at each stage.

The pre-contract stage is where the software company gets to clarify the project requirements and Client’s business’ needs, brainstorm on efficient options, and thereafter, comes up with a technical strategy.

Business analysts and project managers work with product owners and stakeholders to create a consistent overall vision of the AR app, considering all the constraints of the project (think deadlines, budget, technology, corporate environment). Technical analysts provide an overview of the available development tools to narrow down the technology stack.

The Design stage consists of usual sketching, wireframing, mock-ups creation, 3D objects creation plus mapping. We’ll discuss mapping in detail shortly.

The Development stage involves creating a PoC (Proof of Concept) to test the idea and the result in minimum time; and if needed, a mocked (hard-coded) route and custom features (think voice recognition) are also created. For AR app development, it will mean go with Apple’s ARKit or Google’s ARCore software development kits depending on the project requirements. Based on our experience, ARKit may give more accurate results and has more features.

Mapping for AR Indoor Navigation

Without maps, indoor navigation is impossible. And again, it is not always common to find a map of a particular building with the required picture quality and scalability. That’s why you need a custom map made.

Our process normally looks like this – we create a map with Cartesian coordinate system, which is used in 3D graphics and ARKit, then align the map with Geographic coordinates and azimuth using a satellite image from Google Maps. Here, there is no need for much accuracy, so no special equipment is required.

Essentially, this allows us to know the plan’s coordinates in space and adjust the compass; it also facilitates scalability.

Note: Using bitmap images is not the best choice as each map is a vector image for high-quality zooming and minimum transmitted data to ensure high performance. Additionally, at the design stage, AR Ruler is not recommended because of its margin of error. It’s good for sketching, but by using traditional measuring tools absolute accuracy of the map is ensured.

The next step involves adding metadata for navigation: corridors and rooms to create a graph. Then analyze the surroundings and add visual markers appropriately, that is, at the entrance, in the hall, near the elevators, etc.

Caution needs to be taken to ensure the map matches the real world. To achieve this, physical visual markers have to be placed at specific points marked on the plan. Physical markers location has to perfectly match their placement on a digital map to correct and adjust positioning.

The idea is to place the marker on the plan, precisely transfer it to the real world, scan the marker and then fine tune the process by shifting and turning the map until it fully matches the real world. The final step is to save the updated and precise marker position to the cloud.

Drawing Indoor Navigation Routes

Then we build the route using the graph theory and drawing polyline or arrows (based on the design requirements) according to the coordinates. However, there are a few challenges when drawing a route in the real world using Augmented Reality. The thing is, AR content is always placed on top of the image from the camera, which creates an illusion of placing digital content in the real world. If the entire route is drawn, it will look strange, drawn right through the walls.

And since the aim is to make the route look more natural, the visible part of the route has to be occluded somehow. This means the route has to be drawn around the user with fade-out along the radius.

The next challenge that needs solving is how to draw the final destination pin of the same size regardless of the distance, so as to make it easy to read the labels and handle the touches. Due to the perspective, the farther the objects are from the camera, the smaller they look. So, the far pin has to be made larger, but both of them will look the same on the screen.

Here’s a look at the requirements for visual markers that need to be taken into consideration:

Challenges of AR-Based Positioning

Even though the use of AR for positioning offers so much promise compared to beacon technology, there are still some challenges that need addressing before AR-powered indoor navigation can gain mainstream acceptance. First of all, each visual marker or ARReferenceImage has a unique ID which is used to get information from the cloud about the position of the marker in space.

Admittedly, the growing number of markers can decrease performance, because all markers have to be checked to find a match. If the application is designed for a small building where 10-20 visual markers are required, that’s okay. But if navigation needs to be set up in a chain of supermarkets in a number of cities, for instance, with thousands of visual markers – now that can be a challenge.

However, by using GPS, this challenge can be solved as it will be easier to determine the building where the user is located; and work only with local visual markers. Essentially, that’s how it’s possible to add as many maps as necessary. There can be millions of visual markers in the cloud, but only a limited set nearby is needed.

Interesting to note is having a dynamically updated map brings more flexibility to the app. For example, we have created an indoor navigation map for a supermarket where the Fruits section is located at the far-left corner. If the section is moved to the far-right corner instead of the Greenery section, we must then update its current position in the database and build a new route to it.

Wi-Fi RTT To Solve Problems of Indoor Positioning

The second challenge is the technical need for an uninterrupted session. To get the right accuracy levels, users should keep the camera active after scanning the initial marker until they reach the final point. This limitation can be addressed by leveraging new ways of accurate positioning such as the Wi-Fi RTT (Round-Trip-Time), an emerging technology that will enable positioning of a mobile device inside buildings with an accuracy level of up to 1 meter.

Note: Wi-Fi positioning system (WPS) currently used for indoor navigation measures signal strength from access points and creates so called Wi-Fi fingerprints of the building. The method gives an accuracy between 5 and 15 meters, which depends on many factors. Wi-Fi RTT technology is based on measurement of Round-Trip-Time necessary for a signal to reach a user’s device and be received back. Having three and more Wi-Fi routers in the game, multilateration algorithm allows to calculate the user’s distance from the routers and position him with an accuracy of 1 to 2 meters.

Lastly, some business owners do not like the idea of placing visual markers on the walls or floor because of their incompatibility with certain interior design standards. However, a creative approach to the markers design can solve this problem.

Another issue to take into consideration is making sure that visual markers remain placed where they were planned initially and are in perfect conditions for proper scanning.

Timeframe for AR Indoor Navigation App Development

The time taken to complete the development of an AR-based indoor navigation app largely depends on mapping and its complexity. Based on our process, we’ve noticed that if we build a map editor, the process takes more time. On average, it can take up to 2-3 months for Proof of Concept development and a further 6 months for MVP development. Here’s an example of PoC, we designed with ARKit with minimum of time put.

The Future of AR Indoor Navigation

So, what does the future of AR indoor navigation look like? Recently, Apple announced iOS 13 with a new version of ARKit and most likely, it will continue evolving following current trends. Presented in September 2019, the iPhone 11 Pro triple-camera system may improve the quality of tracking in future ARKit releases. The performance of mobile devices will also advance, which will allow for more complex algorithms.

Wi-Fi RTT technology is likely to be adopted by major communication companies and could become a game-changer for indoor navigation. Our plans for the near future include testing a hybrid solution, which combines visual markers and the Visual Positioning System to clarify the exact position at intermediate points to compensate for the margin of error of ARKit.

Final Thoughts on AR Indoor Navigation Adoption

No doubt about it – Augmented Reality indoor navigation technology is set to experience significant growth and, as a result, will be firmly on the radar as an investment theme in most companies regardless of the industry type.

For business owners, project managers, CTOs and CIOs, there will be challenges; however, it will be imperative to define how to approach this technology, looking holistically at the potential use cases and business models in the Augmented Reality indoor navigation fast-growing market and then finding ways in which to integrate it with other technologies to achieve optimal results.

By doing this, they will be able to provide customers the tailored experiences they want in a cost-effective way. Buyers will be able to search, compare and purchase products and services on their terms across multiple channels.

Additionally, organizations will also be able to reduce workloads during operations thanks to capabilities such as accurate localization, advanced user experiences, among others; and, thus will improve the overall performance which is good for the bottom line.

Posted with permission from Mobidev