The promise, and pitfalls, of virtual reality

In January 2015, virtual reality pioneer Jaron Lanier issued a global call for doctoral students to spend a summer working with him on a new project called Comradre.

Microsoft had just announced HoloLens, a holographic computing platform built into a headset that integrates real-world data with holograms to create a “mixed reality” experience for single users. But Lanier – a famed computer scientist, author, and musician who is an Interdisciplinary Scientist at Microsoft Research – wanted to go even further, and create multi-person mixed-reality.

Eight young researchers joined him in Redmond, Washington, for the Comradre project. One of them was Perimeter Institute doctoral student Andrzej Banburski.

Recently, Lanier visited Perimeter to catch up with his young collaborator and to give a talk. He sat down for an interview with Perimeter Senior Writer Tenille Bonoguore first.

PI: Welcome! What brings you back to Perimeter?

JL: I had a Perimeter doctoral candidate, Andrzej, working with me as a research intern over the summer. We explored a new way to do mathematics in mixed reality. I’m interested in continuing that research. I have a long friendship and collaboration with [Perimeter Faculty member] Lee Smolin, and I like to find any opportunity to connect. 

PI: Do you come here with an open mind for other avenues to pursue?

JL: Oh, I come here with a completely closed mind and nothing but pre-decisions. I’m highly judgmental. [Breaks into a laugh.]

PI: But you’ve got so much on the go. Don’t you have enough to deal with?

JL: This is a wonderful moment when physics is becoming more computation, and computation is becoming more physical. I am enjoying it a great deal. In many ways, both computer science and physics are more open-ended and harder to predict than they have been since the middle of the 20th century.

PI: From a layman’s perspective, science seems to feed into technology, but we’re at this point now where technology is very much feeding into science.

JL: From my perspective, that’s always been true. We have to remember that improved data coming from instruments has generally been necessary for improvement on the theoretical front. Freeman Dyson has made this argument, I think passionately, that until somebody learned how to make a decent prism, you couldn’t have had Newton looking at light and colour.

For something like half a century, theoretical physics has been working from a set of observations and a set of observational technologies that had been well established. Just recently, we’re starting to see many new frontiers open up because of improved technologies. Some of these are just continuing on a traditional agenda, like upping the power of the LHC or improving cosmological observations. But there are others that really are entirely new frontiers, like building quantum computers in particular. In that case, there’s a whole new experimental frontier. 

If the past can help us predict anything, we should expect there to be really interesting new theories emerging in the coming years because of that. It’s an incredibly exciting time for the collaboration between computer scientists and physicists.

PI: When you were developing virtual reality, and then Web 2.0, it was all potential and possibility. Now you’re working with students who grew up with this stuff. Is there a distance between you?

JL: I don’t want to fall into the clichés of the older researcher with young researchers. To some degree, it’s probably true that I wanted to rediscover a little bit of that feeling from my own early 20s, which was so exciting.

One of the things I did over the summer that I think worked was, I put young researchers in an environment where they had to do something so fundamentally different from what had been done before that they couldn’t tie into any pre-existing resources. They had to build everything from scratch.

At one point I had Andrzej cutting, grinding, and coating lenses himself in order to make a device we needed, which is not something one would normally subject a theoretical physicist to. But when you’re doing fundamentally new work, you really have no choice. By working in areas that are fundamentally new, where you really have to do everything yourself, you separate yourself from these huge thrusts of super-well-funded technology areas.

Right now, virtual reality is fashionable again. Of course, these things go in and out of fashion. One of the differences between being a very young researcher and not being so young anymore is that you get to see this process of things coming in and out of fashion.

PI: Does that make you a more resilient researcher?

JL: It has to. You have to learn to be able to be skeptical of fashion. Virtual reality was quite fashionable in the ‘80s, and it’s gone through periods of varying degrees of fashionability. Now it’s so fashionable that there’s almost, I hate to say this, but there might be too much money in it. There might be too much adulation. That can actually be distracting. It can be a problem.

By breaking out into something that original [with the Comradre project], what you do is you step outside of that stream of giant funding and you have to reconsider from scratch how to do something. It forces the students to really think from fundamentals, which is a crucial experience. If you need a new lens, you grind the lens. If you need a new algorithm, you have to make the algorithm. You’re not just part of this giant structure of legacy and establishment.

PI: That sounds like a lot more fun.

JL: It’s more fun, but it also teaches you to think from fundamentals. I think it’s joyous. To me, I’d almost call it ecstatic. There’s something about hitting something really fresh like that, it’s an incomparable feeling. You might ask, “Why is there a Perimeter student participating in this?” The answer is this interesting convergence between computation and theory and theoretical physics that we’re seeing.

PI: Andrzej was working to bring math to life. Does that work get us closer to what you always imagined for virtual reality?

JL: The beautiful thing about writing down mathematical equations on a blackboard or a whiteboard is that you’re forced to be economical and precise. The bad thing about it is that you’re forced to adjust a kind of thinking that’s often very broad and experiential and visual and body-based into this more constrained world of notation.

When I was a teenager, I had the good fortune to be around Richard Feynman a fair amount. The first thing he ever showed me was, he said “Think with your body,” and he showed me how to make a tetrahedron with his fingers, which I can still do. He said “Think about this as a tetrahedron.”

I’ve always thought that was important. As you’re thinking, you’re not just thinking about the content but you’re also trying to manipulate your own cognitive structure to be able to be matched to the content. I also imagined that someday, with virtual reality, there’d be a way to express and collaborate and explore mathematical ideas that transcended this. If we could find a way to do something with the flexibility of virtual reality that actually gets us to a different working method, and a different way of applying our minds to math, would that be important? Might it be important? That was a starting point for me, way, way, way back.

PI: Is the stuff you were doing over the summer getting closer to what you always envisioned technology could do?

JL: We don’t know yet. This is a research project. I think what Andrzej achieved over the summer is fantastic. I enjoy it, I enjoy playing with it. I don’t think we’ve gotten far enough to have either a confirmation or not of this idea that we could really change the way we connect to math.

PI: Have you kept your daughter away from your virtual reality work?

JL: Oh no, no. She plays with it all the time. She’s overexposed. She’s very critical. I’ll give her a headset, and she’ll say, “Well, papa, the latency on this one isn’t really as good as it could be.” She’s the worst critic in the world, she’s the toughest customer for VR there is. She’s completely jaded. Her class had been reading A Wrinkle in Time [by Madeleine L’Engle], and she said “Make me a tesseract.” [A tesseract  is a four-dimensional cube, or a “hypercube”.] Of course, I’m the lazy bum and I get to just shift work to graduate students, in the unfair way that the world works. So Andrzej kindly made her a tesseract over the summer, so she’s been able to play with a four-dimensional cube.

PI: There are many sides to being human. One could help create virtual reality, and yet also be a multi-instrumentalist who performs and records. Is technology failing to capture all the sides of what it means to be human?

JL: I think the problem with technology right now is too much money. There are so many temptations to build empires that everybody’s grabbing power and trying to use data to manipulate other people to get money and power. It’s become a little bit of an insane quest that’s lost touch with both the practical benefits and the joy of technology. If we can remember why we like the stuff in the first place, we can sort of overcome this corrupting influence.

This interview has been edited for length and clarity.

Jaron Lanier (right) shows PI researchers – including Comradre collaborator Andrzej Banburski (front left) – how to make a tetrahedron with their fingers.