The Weekly Anthropocene Interviews: Rex Hansen, Digital Twin Cartographer
Rex Hansen is a Product Manager for the ArcGIS Maps and SDKs for Game Engines at Esri, the world’s leading geospatial software company. He is currently working on using game engines to create “digital twins” to help with sea level rise preparation, industrial planning, wildfire control, and more!
In the interview below, this writer’s questions and comments are in bold, Mr. Hansen’s words are in regular text, and extra clarification (links, etc) added after the interview are in bold italics or footnotes.
Thank you for joining us today!
Just a brief introduction, I'm a product manager for our game engine work based in Redlands, Southern California, at Esri HQ. I've been working with Esri for about 30 years in a variety of different capacities, but lately, my passion project is really about bringing GIS and game engines together to be able to provide high-powered visualization, 3D real-time display and interaction, as well as simulation workflows. Trying to bring all those together for these more immersive experiences across industries, across the board.
It's been an exciting time. We've got a team of about 150 people in our development group using a number of developer tools, including the most popular game engines like Unity and Unreal Engine. A number of our customers and partners are really taking advantage of that. We're also working closely with Epic Games and Unity to be able to bring some of these real world and digital twin scenarios to life.
I personally haven't worked with digital twins that much, but I do have an ArcGIS Online account, I’ve done a lot of GIS work. But for someone who hasn’t, could you just summarize in your own words what GIS is and what a digital twin is?
So GIS means geographic information systems. Basically, it's an authoritative system of record for being able to store, manage, distribute, visualize, and analyze geospatial information and geospatial records.
What’s key there across industries is that space and time are important. Managing objects in the real world involves managing objects in space through time. GIS is built to handle that in a digital fashion, in a manageable fashion for extensive amounts of geospatial information. To be able to not just record, manage, share, analyze and visualize, but to be able to synthesize that information in a better way to ultimately make better decisions.
Whether you're trying to educate a community about a new development, you're trying to plan for different climactic changes. or you're looking at building a new structure, you want to be able to plan that out to be effective and efficient. Generally, managing that geospatial information in a GIS is critical to being able to make effective decisions, to be as efficient and as responsible as possible.
So, digital twins. Humans have always been trying to manage space through time. As we transition into the digital age, we move from a paper-based workflow to a digital workflow. But you still have a level of abstraction there, right? Trying to take what's in the real world, four dimensions, and put that in a 2D experience. There’s a significant level of abstraction required to go from 2D to the real world, whether you're going from paper or digital. It's a challenge for users to be able to abstract between this 2d representation of reality and physical reality, right?
It was important to build out a GIS going from 2D to 3D, going from flat point symbols and polygons to something that had a third dimension. Because then we could reduce the abstraction between our digital representation of reality and reality itself. As Esri advanced our technology, what we saw was that customers were investing time and effort into transitioning their 2D GIS to 3D. That started, I would say, probably in earnest about six or seven years ago. And added, of course, an element of time, temporal data.
In some cases, folks, you might have heard the term “metaverse,” and maybe it's been overused. But if you think about the metaverse in the context of Esri, think of that as a digital twin of the universe, starting with the Earth. A digital twin that doesn't replace reality but makes reality a better place. That's really our concept of the metaverse.
Very eloquent! What are some more early digital twin use cases that you're excited to talk about?
We have a couple of examples specifically around environmental awareness, climate science, and really enhancing our understanding of the ecosystem. On Kauai in Hawaii, they built out their 3D GIS so they could develop a plan for the future.
They're looking at shoreline change, sea level rise, so that constituents that live on the island can plan for that. They can develop policy, they can organize and prepare for the future as they begin to simulate with this real-world representation of reality today. They can use different climate models and different ocean models to be able to determine as the sea levels change, what's going to be impacted first and how is it going to be impacted and what should they expect.
So policymakers are involved here at the very beginning to be able to understand what those changes are and where they need to be able to focus their efforts. They want to handle that. They also want to be able to communicate this with residents, property owners that are in the area, and businesses. One way to do that is to provide a medium that looks and acts as near or as close to reality as possible.
That's one of the benefits of working with a 3D GIS, a digital twin of reality, a game engine which provides this immersive experience that can work on a desktop or a mobile device or a headset. It can provide you with that level of immersive capabilities that really resonates with not just people that are making policies and planners that are mitigating those potential issues, but also residents and community members that need to understand the impacts of some of these changes.
Also, simulation models. This is where things like machine learning and AI come into play, so that we can showcase what something might look like 10, 20, or 50 years down the road. That enables us to be more knowledgeable about the decisions that are made.
Kauai is in that place right now. They're leveraging a full 3D GIS. They're working with game engine technology. They're bringing both of those technologies to bear to provide this high-powered visualization capability for policymakers, planners and residents. They're using that to provide real time updates when participants are in the field. They're using simulation workflows to be able to determine what to plan for in the future.
Could you walk me through a workflow example in the Kauai process? Are they simulating sea level rise, sea level rise plus extreme weather, sea level rise plus extreme weather plus king tide? How does that work?
Yeah, so it'd be different aspects of that modeling workflow. In most cases, they're concerned about sea level rise, they're looking at the impacts of sea level rise and the wave action against the shoreline. How do they foresee the shoreline changing? Where do they see there being vulnerable areas along the shoreline that may be impacted, may collapse, may change? They need to reinforce or mitigate some of those challenges with assets that are either on land or just offshore to be able to handle the sea level rise in that aspect that might allow for them to retain some of the territory.
If there’s any infrastructure that's there, roads and other businesses that might be near the shoreline, what needs to happen? Do they need to be moved because of the sea level rise? Would it be more financially viable to actually move that property to a different location, then clean up that region so there's no pollution there when the sea level does rise? Looking at atmospheric changes and climate changes could have a longer term impact, the cycle of storms or the direction or the impact of storms over a longer period of time. That's something you can factor in to determine if there's an additional significant impact that needs to be evaluated. It usually starts with a primary approach and then you add secondary variables.
So how does this work? Kauai started with a 2D approach. Like most folks that are working in GIS, they probably had something that was on paper or vellum or some historical media. They transitioned to a 2D GIS, working with Esri's system to bring in digital points, lines, polygons, imagery, and rasters to represent the real world. That's a great first step.
The next step is building out a 3D experience. Representations of buildings and structures, integrated mesh as a 3D representation of the environment on the island, terrain information, imagery that accentuates terrain details, and then combining all of that information together in a real-world context.
Then bringing that 3D geospatial information, which now looks more like the real world, into working with simulation capabilities, with climate models and sea level rise models that might be coming from another source. Those models indicate what might change with respect to wave action, coastal action, and how that might increase, say, erosion in those areas.
Also, this needs to include not just an image of the island or things that are on the island, but actual properties, right? What type of rock are we talking about? What type of land masses are we talking about? They're going to be impacted by water. Looser, fine-grained materials may erode more easily than more solid materials over time. So how are those going to be impacted by the differences in tidal action as well as the sea level rise?
It's really about adding additional variables to this digital twin that would allow you to provide better insights, more accurate insights into how these spatial phenomena would react to a scenario. That's the challenge. It's really not just about visualization. It is about the representation of reality within the underlying 3D GIS.
Because again, what we're trying to do is forecast the future so that we can plan for the future. It becomes important to make sure we have as accurate a representation of reality as possible. Living in 3D or living in 4D means that we need to have an accurate digital representation to be able to reduce that abstraction between what we experience today and what we plan for in the future.
That’s a really good way of putting it. What are some other examples of what this looks like on the ground?
There's another example in Norfolk, Virginia, with Old Dominion University. They got their foundational 2D GIS in place, then advanced that to better represent reality in 3D. They started building out structures, representations of the city of Norfolk and the surrounding region.
They were concerned about sea level rise, also atmospheric effects, and they were looking at trying to figure out if there was a flood event, how would that impact the city, the infrastructure, the population? There could be data coming from the city itself, third parties that basically capture data that could be ingested within a 3D GIS. Ultimately the goal would be to create the most accurate digital twin of the region on which then they could employ different models or analytics.
At that point, they did something very similar to Kauai. Once they had a 3D digital twin, they were looking at, “How can we mitigate seasonal flooding or incremental flooding?” and “How can we mitigate this long-term impact of sea level rise?” Most of that had to deal with just water levels within the city. It's one thing to provide like a spreadsheet or a flat map or maybe a perspective in 3D, but it's another thing to provide an interactive experience.
So they leveraged game engines, bringing the best visualization possible with the ability to build out focused or bespoke applications that were targeted specifically for Norfolk and that region. For the use of policymakers in the region, but also residents and community members, to be able to see the impact of seasonal flooding as well as long-term sea level rise. They could see the areas within the city, they could zoom into those areas. It also supports cross platform capabilities, being able to deliver that experience for field use if we want to take a mobile device into the field.
Being able to put that power in the hands of constituents or residents in the community to be able to see or visualize those changes was critical. Game engines, again, enabled the ability to provide a realistic visualization of those impacts for folks to make decisions, or just to be aware of what's happening.
Have you done any digital twin type work for wildfire risk? That seems like a potential use case.
It is, totally. At this stage, any climate-related event could factor into use with a 3D digital twin. With respect to wildfires, right now, we are seeing this effort with organizations that are responsible for emergency response or public safety. They're using their investment in the 3D digital twin to be able to practice and plan for environmental events like wildfires or floods. Look at the fire service agencies and the work that they need to do to plan for this multi-pronged approach to be able to attack and then mitigate or control or contain a wildfire. A 3D digital twin can be used organize the effort from commanders to staff that are in the field so that they can understand from a tactical perspective, almost a command-and-control perspective, the situation on the ground and what their plans are for being able to control that event.
In the past, fire crews, fire service agencies, a variety of other emergency response teams, might have something like a sand table where they would sculpt out an area that their fire team was expected to go to and handle or mitigate or control an event. Now, a lot of those organizations have this digital representation of the area in which they're operating, so instead of this physical tabletop that someone has to manually manipulate, they've got a virtual tabletop based on prior investment in a 3D GIS.
At that point, you can work within an extended reality workflow. You can deliver this extended reality capability. We have XR headsets, some of which are mobile, that provide a fairly high optical quality digital representation of reality, but also being able to see what's around them. Like a Quest 3 or Vision Pro. It's basically mixed reality. We've got a digital representation of the real world on a virtual tabletop, and you can see the physical world around you.
Where you've got individuals that are responsible for the emergency response to an environmental event, they can gather around their 3D digital twin, put on a couple of headsets with purpose and be able to see what's happening on the ground, be able to change that landscape quickly instead of having to manually wipe things out and reconstruct it, be able to see things as they're happening in near real-time on the ground. They’ll be able to see the same 3D geographic space on a virtual tabletop and then interact with one another about how they plan to mitigate or contain that event. It provides a level of interactivity that has not been achieved to date at this point.
We see organizations doing that today, looking to replace their physical tabletops or their physical sand tables that go into the field with a fairly lightweight mobile headset, gathering members of their team around a virtual tabletop. A mission planning approach to being able to handle what's happening on the ground.
What's also nice is, not everybody has to be in the same place. We might have commanders on the ground and commanders in the back office and HQ and they can conceivably gather around the same virtual tabletop, and work together around that.
We see a variety of efforts happening in that space, powered by game engines.
Interesting! This is clearly widely applicable.
I love writing, but words can be endlessly debated with no real objective resolution available. With a real-world map, there's more of an objective reality to it, the places are either like that or not, which is pleasant in an age of AI-generated slop and misinformation and a bunch of crazy stuff happening. It's strangely ironic that you're bringing a video game engine to do this, enriching that sense of objective reality and helping people respond rapidly with a shared 3D geographic visualization.
Exactly, exactly. In many cases, our customers will bring their 3D data to life because they want a common operating picture. They want to have a situational awareness of what's happening on the ground. If they can provide that with as realistic a visualization as possible, it reduces the abstraction for folks using the system.
We see a lot of that happening now with respect to just keeping an eye on or being aware of events that are happening in real time. Also, customers are trying to simulate, not just mitigate those workflows, but simulate to determine how they best organize their assets if a situation occurs.
It also is safer. Instead of in the field training on how to do something for the first time, they can start them with simulation workflows in as realistic an environment as possible so that when they do go into the field, they have kind of a leg up. We see that a number of organizations found a significant reduction in on-site injuries. And it's relatively inexpensive to do, to get new employees trained.
What specific work sites were those?
The one that I'm thinking of is that we work with a company called Voestalpine. They're a steel manufacturer. One of their largest sites is in Linz, Austria, I think it's about 26 square kilometers. They've invested in Esri's technology, they have a 3D digital twin that they've built out since 2017. They use it for a variety of things. They're looking at simulation workflows, because they have a lot of vehicles moving around the site. They have engineers that are on site and work with the digital twin to make sure that new construction or revised construction is represented properly.
They're doing measurements digitally, but they're also going to the field to do measurements, evaluating and maintaining the accuracy of their digital twin. Then there’s the training aspect that I mentioned. If they're driving a forklift or an earth mover or something like that around the site, if they get trained first using the digital twin, it enables them to be safer when they do go into the field eventually.
Awesome!
I've been asking for individual examples, and I love the examples I'm getting. Are there more examples you’d like to share, or something more general, or something else?
I think one of the major benefits of working with the game engine has been the ability to participate in these XR workflows, extended reality workflows. One thing that's interesting there is just the advancements that we're making in being able to accurately position and orient devices in the field and then capture information, right?
I would encourage really a crawl, walk, run approach. Start simple and progressively increase the level of engagement that you get through XR workflows. I think would be very important for folks to really take advantage of.
But understand your audience, right? Don't overstep. Sometimes that happens, where folks really enjoy the XR workflows, their staff get headsets that they take in the field, and it's too bulky or it doesn't work well or it's not really focused or it's not accurate enough.
Today, we're in a situation where we're on the cusp of really being able to provide more valuable industrial XR solutions to the market, usable by staff or residents or just the general public. I'm excited for that. I think that would be useful. It would be incredibly valuable to really bring this sort of 3D GIS, this digital twin to the masses.
That's really thought-provoking.
I've been reading a lot about sort of automated and GIS-linked management of gigawatt-scale, kilometer-scale solar farms, especially in China but also all around the world. They’re managed sometimes with inspection drones or robots to brush dust off, there’s some really heavy degrees of automation that we're seeing.
Have you done any digital twin collaborations with solar farms so far?
Nothing that I'm aware that we can talk about publicly, but we do have a number of partners that are working with organizations and agencies to build out and enhance our support for green energy, whether it's solar, geothermal, or wind energy. With respect to solar energy, part of that is being able to orient to optimize the distribution of solar panels to be able to acquire as much energy as possible. That's basically a geospatial problem. The angle of the orientation of the sun, the terrain in which you plan to introduce solar panels, and then being able to track and maintain those panels over time. Also thinking about how they impact the environment around them. A 3D GIS can bring in simulation workflows to be able to determine and optimize different placement options for solar panels to be able to capture and maximize your solar energy potential.
I think the other part is being able to visualize that, making something that is compelling and immersive for individuals that will be investing their time or money in this. Often that's where we see game engines enter the equation, to build out these lifelike experiences with a time series of the construction plan, how much energy is being generated, where is that energy stored, how does it flow through the system, and being able to provide that in an interactive fashion that's as realistic as possible to reduce the abstraction.
It's one thing to see a 2D map that doesn't have any interactivity, it's another thing to see an immersive 3D map or 3D scene. You can interact with that. That has been, I think, critical in seeing some of these projects continue to be invested in going forward. That's great for outreach with the policymakers and investors that want to be able to see these projects completed. You can also create a video of some of these sites so that it can be easily communicated to constituents that will be impacted positively by the investment in that site.
What’s nice is that game engines are designed to do that. They're designed to create a high-fidelity media output that can be displayed and utilized in a variety of capacities. That immersive visual capability can be memorialized in a video. We've worked with the island of Oahu to provide a kind of immersive experience where you could fly through and see the current state of Oahu. The plan was there to be able to see as things change, you know, what's the experience like. That was really designed for constituents that live on the island to be able to see what the impacts of climate change might be on that island going forward.
A similar thing happens, although I can't talk about it specifically, for investors that are looking at different green energy opportunities, notably solar farms. There's a value there in being able to distribute high fidelity, visual quality videos for community members that are impacted. So that's great to see.
What's interesting too is that in some of these sites, even after they've been constructed, there is an effort there to provide a common operating picture to keep that situational awareness, that overview of what's happening on the site. This is where folks, the staff that are actually maintaining or managing the site, need a common operating picture that works and looks as realistic as possible.
That's also where the game engine comes into play. You've got a very focused effort there for supervisors in the back office and for staff on the field to be able to provide a common operating picture. Most of them aren't necessarily a technical enthusiast, they want to get their job done right, and so if you can provide something that's simple, reduces abstraction, and allows them to do their job more efficiently, that's key.
Fascinating. What potential use cases have you looked at for digital twins in the biological sciences? I'd imagine that's much harder just because there's generally records of human stuff and there's not always records of, say, every tree in a forest.
We've got a number of customers in that space. I'd say a Digital Twin is both natural and human, right? Together in most cases, especially if we talk about the work that's going on with Kauai, where they’re looking at sort of the impact of sea level rise on their natural environment. They're using their 3D GIS and the Digital Twin that's present there for terrain, rock type, and landscapes to be able to determine the impact.
We've had organizations that are looking at representing forested areas, for policymakers or different forestry industries looking to be able to curate forests effectively. Being able to evaluate what it would look like in different scenarios as the forestry industry looks to harvest and then replant different forests is important. We've seen organizations use game engines and their representation of a digital twin to be able to provide that aspect.
Now, in that scenario, it's not really important that they map each individual tree. It's more important that they know the areas in which different forests are present and maybe the density of the forest. They would use realistic landscape imagery and then assets that represent trees based upon, let's say, forest type or density, models that are accessible through a game engine. Then you can use those to procedurally populate these regions to showcase what a forest would look like if you’re, say, on a highway moving through a valley, or on a mountaintop looking down over a region. I think the policy makers need to be able to put themselves in a visitor's shoes to determine what the impact would be on the ground or in different locations, to see what they would see before they made a decision.
In order to do that, game engines were pretty critical because they provided that high fidelity visualization snd the ability to render assets that looked and functioned realistically as they respond to different wind and particle systems.
Also, I don't know if you've heard about this, but the European Union is looking at building out a digital twin of the ocean. A 3D GIS that represents the shoreline as well as the bathymetry of the ocean around the European Union, then enhance it going forward with different data. Not just atmospheric, but ocean currents, different aspects of the ecosystem below water and on the shorelines, and then impacts and changes. They want to be able to track not just the current state, but changes that happen over time, different impacts of shipping lanes or other environmental situations that happen like an oil spill. If they need to lay down new piping or electrical wire or something like that, they need to be able to see what kind of impacts that's going to have.
And if they have a digital twin of the ocean around the European Union, around areas in which they control, they feel they'd have a better handle on what the current state is and what the impacts of any changes in the future will be. They're going to bring other simulations and relation workflows to bear on that too. Of course, to identify things, they're probably going to be working with machine learning and AI workflows. All this is going to be recorded within a geospatial system of record, a 3D GIS.
Now, being able to visualize it is important, for whatever the capacity might be. It could be policymakers for the European Union. It could be planners that have been tasked with building something or at least mitigating some issue that might come up if you need to add or enhance infrastructure. It could be residents that make their life or livelihood off the ocean. It could be things like fishermen being able to evaluate the locations in which events have occurred, maybe where there are permitted places to fish and places that aren't, or where there might be maybe challenging aspects or assets that might impact their ability to make a living.
So there's certainly an ecological perspective. There's a business perspective. There's a human element to that. But I admire the effort within the European Union to build out that digital twin of the ocean and the shoreline in areas that they control. Having that foundation and using Esri as the foundation is, I think, critical to getting a better understanding of the different aspects and decisions and being able to communicate the current state and maybe future state going forward. That's where game engines have become very valuable in some of their capabilities.
Of course, for a Digital Twin of the Ocean, the European Union's not alone. I think they've got a good plan, though, because they have a coherent set of participants in that space that they're bringing to bear. I imagine that's going to be a model, actually, for other organizations and entities around the world. It’s great to see the European Union get that off the ground.
This is fascinating. I’ve actually been looking for some of these examples of Esri digital twin work online, like the European Union oceans modeling, and I’m not finding much. Could you send me some more information on that?
I'll say this too, Sam. Some of this is pretty cutting edge. As these relationships start to crystallize, we begin to see some things really land, and organizations get more comfortable talking about the partnerships they have around the technology that they're using. I think we'll begin to see more of this, but we're kind of on the cutting edge of real-world 3D digital twins actually, basically maturing to a point where they're being used in production.
So it is still a bit challenging. Part of the cost of being on the cutting edge of this becoming real is trying to make sure that everyone's on board going forward. But that just takes time. I think over time, we'll begin to see more of these. But I can certainly share some of the examples that I mentioned today that are public.
Absolutely. Don't share anything confidential. But yeah, this is fascinating! I'm honored to be hearing about the cutting edge geospatial work you’re doing.
Thank you so much.
Thank you.
Absolutely fascinating interview! I was completely unfamiliar with the concept of digital twins. Technology is moving faster than we can keep up.