Experiments in Augmented Reality

The growth of augmented reality in many ways feels inevitable. It could take some time, but it is likely that the next generation will become so infatuated by the potential, power and accessibility of the technology that it eventually becomes widely adopted amongst the masses. I could be wrong about this, part of me hopes I am, but for the sake of this rant we're going to assume I'm right.

As this growth continues, it will likely go in one of two directions:

Hypothetical 1 (monopoly): AR becomes monopolized and Snapchat and Instagram become the gatekeepers of our digital realities and spaces like the way Twitter and Facebook are for our political rhetoric. In this reality, the "gatekeepers", whomever they may be, have probably developed a marketplace, similar to the app store, where users can freely purchase and download a diverse selection of AR digital realities. Also in this reality, anti-trust lawsuits are a plenty and Mark Zuckerberg has participated in over 100 senate hearings.

Hypothetical 2 (open-source): Alternatively, as AR grows it will become increasingly more democratized and accessible. We will ultimately live in a world where open-source AR is a reality where users can choose what digital realities they want to see and live in by simply downloading a plugin or an application from a variety of different options. People can make their own AR applications using a variety of easy to use open-source tools like Spark AR or WebXR and easily share them with their friends, families and communities without any restrictions. This outcome seems more plausible.

Open-source means less regulations and less barriers to entry, meaning that anyone and everyone can participate. Though on the surface level this seems benign, a different point of view though might suggest though that maybe a reality where someone can hypothetically see whatever you want is not necessarily a good thing, especially when a lack of restrictions and regulations likely means a lack of accountability.There's really a multitude of problems that could emerge - ranging from people exposing themselves in virtual spaces created by a cohort of students to white supremacist groups developing an application that allows its users to see people of color as racial caricatures. In these scenarios, who is held accountable? Is it the developers of the applications? The hardware that hosted the application? Or the user who downloaded and used the application? Given the lack of legal and ethical frameworks around AR, and the ambiguous nature of the first amendment, it can be hard to say. Furthermore, there are many surface level issues that emerge such as distraction and immersion liability. Consider the distracting tendency of AR. It is no surprise that people find cell phones distracting. Traffic deaths are up after years of decline, likely because people are texting and driving. Phones are attractive nuisances, and we are generally less good than we think we are at splitting our attention between them and the real world (insert UPENN white paper link). Liability and immersion would also be a considerable issue in hypothetical 1 (monopoly), but in this open-source reality hypothetical, accessibility means people can put whatever they want on a screen, without any consequence for not considering the potential harm it could cause.

An exciting opportunity area for open-source AR is how it could be used for civil disobedience and corporate accountability. For example, AR has the opportunity to make police misconduct records more accessible and potentially help in the fight for criminal justice reform as well as expose corrupt corporations, ranging from pharmaceuticals to consumer electronic products.

The city of New York in the summer of 2020 made all NYPD officer's misconduct records open and available to the public through the following database: https://www.nyclu.org/en/campaigns/nypd-misconduct-database. In the context of making this information as accessible as possible, I've prototyped an AR application that would show the misconduct records of a police officer upon scanning their badge:

This prototype was built on After Effects and is modeled after the WebXR interaction framework, an emerging AR library for the web built on top of three.js. With the accessibility of machine learning and optical character recognition (OCR) software, applications like this are becoming increasingly easier to build.

For many, civil disobedient applications like this present an opportunity to tackle issues like criminal justice reform. However, laws around filming police officers are ambiguous and vary from state to state and country to country. According to a white paper by the NYU Innovation Law Center on the public's right to record police in public, yes: "the public's right to record the police is a vital check against police misconduct". Generally, they are of the belief that filming the police falls under first amendment protections and in most instances, across most states, this precedent stands.

However, in France a security bill passed recently passed (in December of 2020) prohibits the publication of images of policemen "with the intent to cause harm" . Laws like this in France are in many ways intertwined with their views on identity politics. France in Germany, the two biggest economies in the European Union, do not collect any demographic data on ethnicity or race. Therefore, citizens in Germany and France literally have no way of knowing how much more likely people of color are to be stopped, discriminated or harassed by the police. It actually goes even further than that - there's also no way to know if they're being discriminated against in the workplace or housing market or to die of coronavirus. This bears the question - if an application like this were to reach market, would it be prohibited in countries like France and Germany? Is that a problem? If so, it's important for technologists and regulators to be thinking about these issues before AR reaches main stream audiences.

Returning to the conversation on the first amendment - do those same principles for image capture apply to AR? Something that is important to note about AR, is that when we look at the tech stack enabling AR, there is a new type of data being captured, unique to AR. It's the computer vision-generated, machine-readable 3D map of the world, enabled by sensors like LIDAR. This type of data has never been captured at scale before.

I asked Michael Wineberg about this, who is the Executive Director of the Engelberg Center on Innovation Law & Policy at NYU. He says that despite this being an emerging technology, that the first amendment principle would still probably apply for this situation because "it's still capturing the video that is in front of the camera... there are a lot of videos that capture 3D spaces... it's an incremental, not a state change".

Civil disobedience in augmented reality also has the potential to critically expose corporate corruption in a multitude of settings. The example I developed involves exposing pharmaceutical brands in a brick and mortar retail setting.

Similar to the police misconduct example, this example leverages an image tracker to render information from a database: https://violationtracker.goodjobsfirst.org/parent/johnson-and-johnson. It also was built in After Effects and uses the WebXR interaction framework.

It's no secret that corporate America has become unhinged. Ranging from airlines to financial services, it is often those who are the most affluent receiving the most bailouts and face the fewest restrictions. Recently, we've seen this with Senate Majority Leader Mitch Mcconell refusing to pass a COVID-19 relief bill without corporate protections. Though some organizations have taken it upon themselves to become more transparent in their business operations, it mostly falls short. I believe that journalists, technologists & global civil society will continue to expose corporate greed and immorality, and that it's just a matter of time before they use AR as a platform to do so.

Corporations, facing daunting drops in their stocks, inevitably will try to hinder and muzzle all attempts at exposing their corporate greed. The question then becomes - what restrictions will the creators of these types of applications face when trying to bring said application to market?

Phone Story was a satirical mobile video game conceived by Yes Lab activist Michael Pineschi with the stated aim of demonstrating what the developers refer to as "the darkside of your favorite smart phone". The game consisted of minigames which required the player to compete in activities such as forcing children to mine Coltan and preventing suicides at a Foxconn factory. The game however was eventually blocked by the app store.

NO AD is another civil disobedient AR project that attempts to replace subway ads with art. In its current format, it's similarly relatively benign but its not hard to picture a reality where major companies try to take legal action if enough people are blocking the advertisements that they're paying for.

These examples re-raise the question: what does this mean for the future of AR development if we have certain gatekeepers who get to decide what products do and do not reach market? Michael Wineberg had a pretty optimistic outlook, claiming that (users) "don't have any obligation to experience it (advertisements or products) in the way companies demand." He created an interesting analogy to sports games, where certain broadcasters will digitally render advertisements over other advertisements. Hopefully, Michael's right about this.

To quote Garnet Hertz: "Critical and Speculative Design (Dunne & Raby) are worthwhile approaches within industrial design, but perhaps not adversarial enough to reply to contemporary populist right-wing movements (Brexit, Trump & Le Pen). Questions like 'Is it moral to punch Nazis in the face?' should be answered with smart alternatives to violence that are provocative pieces of direct action." Its important to remember the importance of civil disobedience, and its only a matter of time before AR becomes a main stream platform for action.

The city of New York recently made all NYPD officers' misconduct records open and available to the public through the following database: https://www.nyclu.org/en/campaigns/nypd-misconduct-database. In the context of making this information as accessible as possible, I've prototyped an AR application that would show the misconduct records of a police officer upon scanning their badge.

An integral part of this project is image recognition. Specifically, the image recognition of police badges to render the visuals containing the misconduct and background information of that police officer. To prototype this interaction and simulation, I created an AR app in Spark AR that would allow me to convey this idea.

Another integral part of this project is machine learning. The AR application I create will have to be able to recognize police badges in order to create the visualization. To test this, I created a model with Teachable Machine and exported it to p5.js.

I created a video prototype on After Effects that would hopefully allow users to further their understanding of the interaction and visual aspect of my project.

I received a lot of great feedback from various people.

1. Safety Concerns/Liability: Probably the biggest concern about this project is that the laws regarding filming police officers are very ambiguous and they vary from state to state. The whole point of this project is to make people feel safer, but it simultaneously has the potential to put things at hard. A potential solution for this is putting a disclaimer (in the form of a modal when you open the app or website) with the appropriate legal information so users know their rights when using the application.
‍
2. Lighting: Like most AR applications, lighting would be a concern. Especially for the machine learning/OCR component.
‍
3. Distance: Realistically, how far would the user need to be for the application to recognize the characters on the badge? Would probably need a large sample size of badges for the machine learning/OCR model to pick it up correctly at adequate distances so the application can be practical and useful.
‍
4. Information Storage: Users should be able to find an efficient way to store the data/information that is rendered so they can have it for future encounters, etc. Also calls into question whether the current visual application is the best way to display the information.