Augmented Reality and Mobile Design

Augmented reality (AR) is the technology that integrates virtual elements into physical environments. It layers images, sound, and text on real-world surfaces. AR alters users' perception of the real-life environment — or augments it. The most common examples are video games like Pokemon Go or Snapchat filters.

To qualify as augmented reality, the technology must be interactive — i.e., it should respond to user input and account for changes to the environment. For example, a filter might change when a user opens their mouth and have a butterfly emerge out of it on the screen.

There are two main ways to deliver augmented reality:

• Mobile augmented reality (MAR) is experienced on mobile devices like smartphones and tablets.
• AR glasses are a wearable smart display with a see-through viewing experience.

Virtual reality (VR) and augmented reality (AR) are the two primary forms of immersive technology available today. Although both use 3D graphics, they are quite different. VR creates an entirely new reality for users. It takes users from their actual reality and places them in a new, digitally created world. To experience VR, users need to wear special headsets. VR headsets are designed with computer screens to fully immerse users in the virtual world. In virtual reality, you can't physically see any real-world elements.

Augmented reality does not create an entirely new reality but works with the existing environment. It adds digital layers to real-world elements. While AR alters users' perception of the environment, they can still easily distinguish virtual items from real ones. Also, users can easily experience AR on their phones or get AR glasses if they wish to.

The choice between AR and VR depends on the intended user experience. AR works best for applications that need to reference real-world context, like navigation or product visualization. VR excels in scenarios requiring full immersion, such as training simulations in virtual environments.

AR technology is great for products that users need to try on or visualize before buying. For example:

• Home décor, furniture, or vehicle customization. Preview placement gives customers a glimpse of what a product will look like when placed in their home or car.
• Clothes and eyeglasses are available in AR as virtual try-ons. This allows customers to see how the product looks on them and if it matches their style.
• AR filters allow users to see how makeup would look on them and helps them match it to their skin tone and personal style.
• 3D scanning in apps like Google Lens to capture an image and instantly search it. This can help users easily find products around them online.

AR photo filters use facial recognition and tracking to overlay digital effects on user images in real-time. These filters can detect facial features, movements, and expressions to accurately position virtual elements like masks, makeup, or decorative effects. Common applications include social media platforms like Instagram and Snapchat and camera apps.

Face tracking technology maps key points on users' faces to maintain proper alignment of virtual elements. This allows filters to stay in position even as users move or change expressions. Advanced filters can also respond to specific gestures or facial movements, creating interactive experiences.

Released in 2016, Pokémon Go was the game that popularized AR gaming. By placing virtual creatures into real-world environments, Pokémon Go brought the fantasy of Pokémon living among us to life. The game uses the player's location to display Pokémon on their phone screens, encouraging exploration of the real world. Whether catching a Pokémon on the way to work or during a stroll in the park, the game blends digital and physical spaces seamlessly.

Other games, like Ingress Prime and Jurassic World Alive, use similar mechanics, blending the virtual and real worlds. Games like Kings of Pool offer a different experience, turning traditional board games into AR environments by superimposing digital elements onto flat surfaces like tables.

Location-based AR determines where users are and shows the information relevant to their location. Its application ranges from navigating public places to assisting emergency teams during accidents. For example, the Live View feature of Google Maps uses AR to help users orient themselves on the map. Arrows and directions are placed in the real world to guide users.

Mercedez-Benz offers a similar navigation feature for cars. Along with turning instructions, drivers can see house numbers and street names popping out as the vehicle approaches them. These apps use GPS, digital compasses, motion tracking, and location recognition to determine users' location.

For indoor navigation, systems like Bluetooth or Wi-Fi markers are used to track users’ locations within large buildings. These systems use a network of specialized devices installed in the building to determine users' position with high accuracy. Indoor navigation is especially useful in large or complex spaces like airports, malls, or hospitals. For example, if you’re at an airport and need to find your gate, an IPS (indoor positioning system) app can show you a map overlaid on your phone’s screen with directions to get there quickly.

Here are some other applications of AR technology:

• AR is increasingly being used in education, providing interactive learning experiences. Apps allow users to visualize complex concepts, perform simulations, and engage with 3D models for better understanding.
• Medical professionals use AR for visualizing patient data and anatomy in 3D. Surgeons can overlay digital images onto a patient’s body during surgery, which helps in planning and precision.
• Some brands are using AR for ads that users can interact with in real time, often on their phone screens. For example, by scanning a poster, a user might unlock a virtual character or promotional offer.

The potential dangers of excessive AR use include:

• Physical danger: When users get too carried away with AR and can't distinguish what's real, it can lead to accidents.
• Mental side effects: Smartphones have already changed how we interact with others, and AR can further aggravate the already existing problems of depersonalization, depression, and other mental health issues.
• Social side effects: As AR glasses become more advanced, there is a risk of increasing social inequality. Those who can afford expensive AR technology may gain advantages like instant access to information and enhanced educational opportunities. This could widen the gap between the wealthy and those who cannot access such technology, potentially reinforcing existing social divides.

AR content placement raises important ethical considerations for digital experience design. Virtual elements in AR can affect physical spaces in ways similar to physical alterations, requiring thoughtful implementation of location-aware restrictions. In 2016, Pokémon Go was caught in a controversy when the algorithm made a Pokémon available for capture in Auschwitz.^[1] This is why such sensitive locations, memorials, and religious sites, need special protection through geofencing and content moderation.

AR can also endanger private property rights. Placing AR works outside people's houses can draw unwanted visitors to specific neighborhoods. Responsible AR design includes creating clear boundaries between public and private virtual spaces. Design systems should incorporate opt-out mechanisms for property owners and location-based content filters. As AR platforms evolve, designers must consider how their virtual content impacts physical spaces and communities.

AR applications require careful privacy protection measures when handling user data and environmental scanning. Face recognition and 3D scanning features, while powerful, can expose sensitive personal information without user consent. Modern AR devices with recording capabilities raise additional privacy concerns about unauthorized data collection.

Real-time data streaming from AR devices creates extensive digital footprints of user behavior and environment. Companies collecting this data gain unprecedented insights into users' daily lives, locations, and interactions. This data collection enables highly targeted advertising but raises serious privacy concerns.

Ethical AR design requires implementing strong privacy controls and transparent data practices. Users need clear information about what data is collected, how it's used, and who has access to it. Applications should also include options to limit data collection and control recording features.

AR technology can potentially misrepresent reality in ways that mislead users. In real estate, AR could make properties appear more attractive by adding virtual renovations, landscaping, or removing unsightly features. This digital enhancement might create false expectations and impact purchasing decisions.

Age-altering AR filters pose serious ethical and legal concerns, especially in age-restricted contexts. For example, AR could make underage individuals appear older or vice versa, creating risks in situations where age verification is crucial. These manipulations could enable deceptive practices in dating apps, age-restricted venues, or identity verification.

Vulnerable users, particularly children and individuals with cognitive disabilities, face higher risks from reality distortion. They may struggle to differentiate between AR enhancements and physical reality. Safety features should include parental controls and accessibility settings. These might involve reality verification prompts ("Is this real or virtual?"), simplified toggle controls for AR effects, and clear documentation of which elements are augmented. For commercial applications, mandatory disclosure labels can highlight when AR enhancements modify the appearance of products or spaces.

The rapid growth of AR technology has raised concerns about its potential for misuse, especially since it remains largely unregulated. Designers need to consider not only the needs of their users but also the broader public implications. For example, AR could be used for false advertising, tricking consumers into purchasing products or services based on misleading virtual enhancements.

Additionally, facial recognition technology, often integrated with AR, can pose serious risks, such as falsely implicating individuals due to its potential inaccuracies, which could lead to wrongful accusations or even incarceration. As AR continues to evolve, there is also the risk of unwanted exposure to inappropriate content. Imagine an adult-themed AR ad appearing near a daycare, for instance.

Like all technologies, AR must be designed responsibly. Users should have control over the AR content they are exposed to and the data they choose to share. The ability to decide what is seen and how personal information is handled should lie with the consumer, ensuring that their privacy and interests are protected.