Transcript
Nicholas Herring: Today I'd like to go through how we adopted observability into a system that's 20 years old and also has newer mechanisms evolving underneath it as well, and go through how this differs from the video game architectural space versus traditional application development. Compare a little bit of that with FinTech as well because there are some similarities there. Ultimately, we'll talk about how our system is 100% head base sampling, which makes a lot of people clutch their pearls and gasp, because we don't use tail base sampling which means we're buying in a lot of those tradeoffs. We'll talk about what those are. How we go about doing that by blending dynamic and deterministic sampling techniques, the blessing and curse that is exponential moving average sampling for dynamic sampling, we'll dig into that a bit more. That's the crux of the tricks that we're using. How we ultimately use all of this to find a balance between good enough for fidelity and cost.
Traditional vs. Video Game Architectural Patterns
Before we get into that let's first dig into a bit of the differences between traditional and video game architectures. Most traditional software is creating a unique experience for a single user. Very rarely does traditional software build a shared experience with many users at the same time. A lot of that is obvious in how we build and scale out these systems where there's a single, everyone has their own web page that they experience, it's not the same web page that everyone else experiences. In contrast, multiplayer video games, they have to create shared experiences. To create these shared experiences it means we have a lot of the problems that FinTech has in weird different ways in the sense that we have to synchronize a lot of data, it has to be transactional, and we have to be able to replicate that data so that everybody can experience that shared experience at the same time.
On top of that, in contrast to FinTech is the concept that video games do a lot of stuff that the system doesn't really need to care about. Simple waving at somebody in a video game, that's just an expression of that player, of what they're doing in that virtual world. For EVE Online we'll go over some of those bits as well. The video game industry has always been the fun uncle that has a hot rod where it's cool to build those and it's fun to take to the grocery store but most people look at and go, why would you do it that way? The main reason for a lot of that is building this ability to have an expression in a virtual world and communicate with others in various different ways that you might not get to even in real life.
What is EVE Online?
Before we hop into more details about how we actually monitor our systems, can I get a sense of how many gamers do we have here? How many of you have actually heard of EVE Online? I'm going to go over some details of EVE Online, what it is, and where it's at right now. Internet spaceships are serious business. Ultimately, it's known for its epic espionage stories. If you don't know about EVE Online you've probably heard about it through, I think the biggest press that we've gotten before is like a CNN article on somebody stealing hundreds of thousands of dollars' worth of assets in-game. It's also important to know that it's a single sharded MMO, which means that we do not have a North American server, we do not have an APAC server, we don't have a Europe server. It is all a single server where everybody experiences the same universe at the same time, which is the again uncle with the hot rod grocery store thing.
Because why would you do that on purpose, that creates all sorts of technical challenges. I have a separate talk on that all together. The other thing to keep in mind is that it's a player-driven ecosystem. There's probably about half a million transactions on our market per day, and that represents about 12 billion items that move around in that ecosystem. Keep in mind this also counts literally every single bullet that's in the universe, and that's also something to keep in the back of your head while we're talking about this, because also we have to talk about tracing and all of the bullets and lasers and missiles flying around in the ecosystem and how we deal with that, and we'll get to those. Just to give you a sense of the size of EVE Online and how it's been working, we hold a few Guinness World Records with the most costly battle that happened in the game costing almost $400,000, the largest PvP battle over a span of time with almost 9,000 players in it.
Then the most concurrent players in a multiplayer video game player versus player battle at 6,500 players. That is 6,500 players all in the same place trying to punch each other in the face constantly. That gives you a sense of scale of EVE Online and connect that back to the systems that you guys are interacting with and building in different contexts, traditional software. Whereas you can focus on the session of a user, we have to focus on the state of a solar system for everyone that's experiencing that solar system.
20 Years of EVE Development
The other piece of EVE Online is this is a very old project. The development started before 2000, so late '90s is when the actual development of it started, the first versions of EVE Online started. It has a little over 2 million changelists that are real. I'm trying to factor out robots. It has about 1.7 million lines of C, C++. I also have this unit of Unreal Engines, comparing the line count to that gives you a good idea of the size there. EVE Online was built with a bucket of Cat 5 and a firehose of SQL procs. We also have 2.6 million lines of SQL that is mostly a majority of all the actual legacy systems running inside of an MSSQL server. Then the wonderful folder that we have that's called script that is 3.4 million lines of Python code. The other joyful part about this is it is stackless 2.7 Python.
We have a longer project to get to 3. What stackless 2.7 Python means is that we run on one logical processor. Not one core, one logical processor. We do one thing at a time. That comes into play when we talk about sampling. If that didn't bother you, the universe is defined in roughly 53 million lines of YAML. This is one of my active personal crusades in the organization to make that number significantly lower. Then we're going to talk a little bit about the evolution of this. I have separate talks that I've done at GDC as well, where we talk about the evolution of where EVE Online's backend tech is going. It's moving away from this monolithic structure of these things that we've listed out here into an umbrella term that we call Quasar. All Quasar is, is a term that we use to talk about Kube, Golang, gRPC, protobuf, RabbitMQ, NATs, all of those fun things that everybody's been working with in the past decade or so.
These two systems combined, the legacy system, and the newer Quasar ecosystem, they do roughly about 30,000 messages a second. By messages, those could be anything from events to requests to responses to also what we call notices, which is information that we push to the clients themselves. We've built up this message bus ecosystem that started our journey of building domain services, because I think microservices is a bit of a dirty word. Because it's been around for a while, and no one can still tell me exactly what they are. We work in domain services, where we have the concept of these are users. These are ships. This is the solar system. Those are the services that we have that are operating in this. We also use those in the sense of features as well. How we build up things, like we have a system in EVE Online that's relatively novel in the sense that it allows players to create quests and missions for other players. That is its own service and ecosystem in and of itself.
The In-Game Experience
Let's take a look at what all of that on paper looks like. This is a battle that happened about a month ago. I just grabbed something off of Twitch from one of our wonderful community managers. In here, we can see that there's around 300 people in a battle. There's two different fleets, they've organized the UI, they have red versus blue, as is tradition. This gives you an idea of how we have to deal with the aspects of the expression of the players. Now, you can't really see the expression of the players, because at this scale, things blur together, and you can't really get into the fine details. If you were to look closer at some of these, you get to see what our players start expressing on their own. We have a system in the game called Skinner, yes, very creative, where you create ship skins with a whole design system.
Players have been using this to express different uniforms or different feelings in some cases, like a mood ring for a very expensive spaceship. These can be expressed in a way where we've tested this system, where we had 500 people in a local test system, and we probably had, we're going to say, three Raspberry Pi potatoes running somewhere. Then trying to have them all switch their ship skin as quickly as they possibly could. We turned off all the rate limiters. Then that in and of itself generated the same amount of messages as the entire universe, so about 50,000 a second, when we had 500 people switching those. I'm highlighting this because it also gets back to the importance of being able to express yourself in a virtual environment, and the problems that we need to solve in order to do that. This became a way that players could then communicate that was different than chat or shooting at one another. This became yet another venue where they could express themselves in a certain way. There are also different variations of this that we have. You can use, we have space fireworks, of course, and various other things. I think we even have snowballs in some winter festivities.
When we start thinking about this a little bit more, that's all the fun and games part. That's all the fun bit. We start talking about where's the motto that our organization has, which is making virtual games more meaningful in real life. Take from that what you will. That is the idea of where EVE came from. The general original statement for EVE Online was that death is a serious matter. To illustrate that, part of what we have in EVE, one of the biggest ships we have in EVE are called Titans. Those are in-game probably around 16 kilometers long and roughly around €1,000 per ship for all the bill of materials and fitting them and munitions, and everything that goes into it. If you translate all of that from in-game currency and translate it to real world currency, it hovers around about €1,000 per ship. There's gravity in that because if that ship blows up, it's gone.
They don't get it back. There's no way to get those back. There is if you complain to a GM maybe and get lucky, but highly unlikely, they've been highly trained to ignore those requests. Then on-screen here, we have an entire bubble of them, probably around 40, which represents $40,000 in real world currency that's on screen. Now you can understand why our biggest battle cost roughly $387,000. Because on the field of that battle, were roughly 257 of these enormous ships. That's what we can understand is at stake. At the same time, understand how we allow players to express themselves in the ecosystem. Like this one guy who brought out his $1,000 ship at the top here and decided to be fabulous in all pink. I don't know if you noticed it yet, but he's right there up top. Ultimately, as we come back to what we're here to talk about, is how do we then differentiate between these things that really matter, like this $1,000 worth of imaginary internet spaceship versus a pink color that pops up on the screen? Where do we track those and how do we manage that?
Head vs. Tail Base Sampling
The first part is talking about head versus tail base sampling. Real briefly, the decision that we made coming into that ecosystem from a 20-year-old project is around the fact that we didn't want to take on any other extra maintenance burden. Another big part of that is what I talked about earlier, where it has to do with the capabilities that we had. Again, Python on one logical processor. I can't even get the spans out fast enough to do tail base sampling without killing the rest of the simulation. There are ways that we could invest and optimize that, but at that time, that just wasn't on the table. We leaned in more into head base sampling versus tail base sampling. It also led us down an interesting path of simplicity over complexity. When we did this, we also thought about the centralized concept. There's a central element to tail base sampling where you have to send all of your spans to one place.
My bias against that is that I don't like to go blind in big chunks of my ecosystem. We just didn't do that. We don't even have a relay for emitting the spans that we have now. What that does allow for is when the system starts to degrade, we can infer which part of the systems are degrading, because only those things can't get to, for instance, Honeycomb, for that example.
Deterministic vs. Dynamic Sampling
Then we leaned and looked at more of the deterministic versus dynamic sampling. Again, because of performance, actually doing dynamic sampling in an ecosystem where you already have this 20-year-old fine-tuned machine that is simulating a whole universe, we can't really go in and start working on things that are more dynamically sampled. We were forced to look at, we need to have that deterministic on that side of the equation. Then the newer systems will be inside of Quasar. We'll have more dynamic sampling in that case. This also became super handy for us when we started thinking about, what if we had our desktop clients actually emit spans as well? There's a lot of performance data that we could understand from that and understand there as well. Again, same problem. The desktop client also operates as this monolith that is well-tuned for what it needs to do. We didn't want to affect that either. This is yet another thing that pushed us more towards head base sampling.
Exponential Moving Average (EMA) Sampling
Then we get into the exponential moving average, or EMA sampling. This is a huge part of what we did with how we trace a majority of the newer systems in Quasar. Briefly, what EMA sampling is, it basically is an algorithm where you select a key for what you want to track. For us, it became the type URL for the protobuf messages that we were slinging around everywhere. That became our key. The interesting part about that is it becomes your key space. The key space will be important in a bit as well. As your system evolves and expands and you put more and more things into that ecosystem, that key space starts to grow. That is one element that you have to keep in mind when you're tracking how much information you're actually going to emit as a span. The other thing that EMA does is it will dynamically look at that key space and track how many times it's seen each one of those keys.
It does this really cool organic thing right out of the box of the algorithm of the more I see something, the less I sample. The less I see of something, the more I sample it. Which is great because that solves a majority of the original problem that we had, which was, I don't need to care about 500 people switching their ship skin constantly. If that's constantly happening, or bullets flying around the ecosystem constantly, I don't need to worry about that. EMA will turn that down. The problem then becomes the bullets aren't fired from the new system, they're fired from the old system, so now we have to deal with deterministic sampling in that regard. The other aspect of EMA sampling is that none of it is coordinated, which means that unlike tail base sampling, you're funneling it all to one place. Whereas with EMA sampling, specifically head base EMA sampling, you are looking at distinct instances of ingress into your ecosystem.
Then you have these three key components, which is the frequency of messages, the key space, and how many instances of that ecosystem are running. When you put all those together, there's this interesting thing that happens where you get this striation of the data, where you get this high watermark that slowly works its way up into your ecosystem. What I mean by high watermark is once we scale out, so this graph, I threw a bunch of numbers in to illustrate this. If we talk about the Quasar aspect of EVE Online, that's roughly 30,000 messages. We have a key space size of probably around 2,000 messages. We have 35 instances for ingress into that ecosystem. When you throw all those numbers in, there's this interesting thing that happens. For us, we end up averaging out around 60 million events a day, which is great. That's a nice, cheap number in the grand scheme of things.
It allows us to stay in that wheelhouse and understand what's going on. It also lets us know that if we need to scale up in those instances, we have to be wary of where the next high watermark will be, because that will be the part that we can't go back below. Back below, what I mean by that is it means that no matter how my frequency changes or how I control the sample goal. EMA also has a sample goal. The sample goal is basically, what is it trying to target for that singular instance? If its goal is 1 in 3,000, it will try to do that for its key space, for its frequency, and in its own instance. If I try to make the sample goal universal across the system of 1 in 3,000, it actually can't go below our system because of having 30, 35 instances, and having the frequency and having the key space that we have, having the goal of 1 in 3,000 per instance prevents it from dropping below around 60 million a day.
Because it will always sample something, and it will never leave anything out. You will always see at least one of everything, which is also a good thing, and we'll talk about it a little bit later. I also try to slice and dice this from a different view when we're talking about key spaces. When your key space grows, a lot of weird things happen with EMA in a distributed mechanism. In that sense, it gets more interesting from the perspective of this is very unlikely, because this would require your teams to be generating hundreds of new types of messages very quickly in the ecosystem, which at the human rate, that doesn't necessarily happen. The AI rate, that might be a problem later. We'll find out.
All of that coupled with the hybrid configuration that we talked about, EMA is very useful for the dynamic sampling that we have, but also using the deterministic sampling. One of the most important parts about using deterministic sampling, and also with dynamic sampling as well, is having tooling that can show you what the sample rate is and also extrapolate what those numbers might mean. This is a screenshot of the samples that we have from our simulation frames. Our simulation engine is a deterministic lockstep simulation engine that we call Destiny. Hilariously, there are other parts of it that are named after various other Destiny Child members. That's a different talk as well. With these, because so much is happening in that simulation frame and so much is happening in that system, we have to sample these deterministically. One, because of performance. That's not where we want a performance hit.
Two, there's just so many of them that we can't really care about all of them. At the bottom of this graph, what you can see is the wild ranging sampling rates that we have set up to deterministically sample these mechanisms that are happening in the simulation state. The more important part is what is happening in the top right, where Honeycomb is taking those sampling rates, and the frequency of samples that it has, and extrapolating those out. We use this quite often. I would argue that it's even with rates like this, and sparse data like this, we've seen it usually be within ±300 units of whatever the thing is. It's actually pretty reliable when you're dealing with statistically significant information.
Feature Flags
Then another trick that we have is abusing feature flags. We, again, hot rod grocery store, we use LaunchDarkly to jam a bunch of JSON everywhere. What we do here is we have a prefix mechanism where it allows us to express two different concepts. One is that we can say, I want to know more about this message. Or, I want to make this domain much more quiet, because it's less important to what we're doing. If you look at the last line there, that is one that we definitely don't care too much about. It is ultimately what window is in focused, open, closed, focused, unfocused, when they moved it around, drag and drop, all sorts of stuff. We care that those are happening. We don't care about all of them. Again, those are statistically significant. From the perspective of a client, 1 in 1,000 is pretty sparse.
From the perspective of 30,000 clients all doing 1 in 1,000, it's much less sparse. Then we also have other pieces of the ecosystem that are just more frequent, where harvesting is any concept of mining, whether that be mining lasers, drones, gas harvesting. There's a whole ecosystem of minerals and industry inside of EVE Online. We use this quite a bit to focus on what we want when we want to. I think that's another important part of what we do. With these tools, what we also do is we have a mechanism where we can look at the activity of a specific ship or a specific character. Then when we turn these things on, our quota graph looks like this, and you'll see the two spikes in the far right. When I read this graph, it looks like this, because it means somebody was looking very intently at something specific going on in the universe. These are a collection of all the tools that we have of what we've been working towards and what we've been pushed into from the sense of performance, legacy, the evolution of the newer ecosystems, and how all those pieces work together.
Takeaways
Then, ultimately, I want to go through some takeaways here and reiterate some of the things that we've been talking about. The high watermark is very important, because it allows you to manage cost when you understand what the output of that high watermark is. This is a screenshot from a website that has been around for quite some time called EVE Offline. It has a great chart for how many players are playing EVE Online in a given moment. As you can see, with EVE Online, we have, some call it a luxury, depends on who you talk to, but we have a very stable player base. We have spikes where we go up from 30K to 40K, depending on if we release an expansion and people get really excited about content, these types of things. Ultimately, we don't have to really worry about a huge burst. The importance of the high watermark has a lot to do with the behavior of our product.
That allows us to do a lot of planning on how much we're going to spend, what we can use to target, and how we can instrument more detailed information. Another technique we have is targeting 90% quota utilization. If you look at this as well, you can see that we're roughly coming in at 50 million a day. That allows us to, again, turn on the Eye of Sauron when we need to and not worry about doing big overages. There's also a lot of cool toys that we have with Honeycomb for like burst protection and those types of things. Another big part and purpose of why we smashed all of these techniques together is that we wanted to make sure that we have at least one of everything. Again, we want to know that a bullet was fired somewhere in the universe. We definitely want to know that happened at least once, because then we can extrapolate that based from sample rates or whatever the case might be and we can infer the health of the system. We can also then figure out what's contributing to actual debugging versus what's just noise. Again, the focus on identifying what's statistically significant versus what are facts of the universe that you really need to keep track of.
This is more just a detailed note on scheduling flags. Sometimes we forget to turn off the Eye of Sauron and then we get a lovely little automated message saying, you've just spent a lot of money. Just making sure that everyone has that on their checklist if they're looking at tackling this problem space. Then we talk about the implications of the public application instrumentation. We talked a little bit about the actual desktop client. We've also looked at this in our launcher as well. We're probably just going to stop doing that all together from a pure technical perspective. Because, ultimately, what ends up happening is because of those systems, we then make sampling decisions much earlier in the trace that affect what we can and can't see. A lot about a video game has to do with what the interactions of the player are. If that player is then doing something that's very important like anchoring a space station, we potentially won't see as many of those because they are 1 in 1,000 sampling because they're coming from the client, which means none of our tooling that we have allow us to crank that up because we don't want to affect the actual performance of those systems.
We want to move that decision-making process from the actual clients themselves to a trusted network so when it hits our ecosystem in our networks, and then start the decision-making and sampling there. This is something that we've learned over time. It gave us some cool insights as to latency, but that's something that you can get a different way. You don't need spans to get that from 30,000 different clients. You need more averaging of metrics in that sense. There's another interesting side effect of this technique, which gets into optimizing SLO fidelity. When you start working with building SLOs, and for the brief SLO, basically you want to measure your error budget, which means you need to know successful things versus not successful things. That works really well in head base sampling because you don't get a choice of how many successful and unsuccessful events there are and what the ratio of those are compared to tail base sampling, where you get to toy with that ratio and understand it a bit more.
We can't do that because we're doing head base sampling. Then we also have to be careful about how sensitive those SLOs become with things like statistically significant events and whether or not we should even put an SLO on the statistically significant events. For example, if we're doing head base deterministic sampling for the UI interactions, and it's 1 in 1000, and we have an SLO on that, that is going to be problematic because we're going to see one of those every couple of hours. If we get an error in that, it's going to burn the budget immediately because of how that SLO is built because we don't control those ratios like you would with tail base sampling.
Tying it All Together
Ultimately, to tie this all back together, we use all of these tricks with deterministic sampling, EMA, head base sampling, and targeted temporal sampling, so the Eye of Sauron that we turn on and off, depending on what we actually need to focus on, to get into the at least one of everything state that we'd like to be in. This has been mostly working very well. There's been some cases where we've run into, that'd be really cool with tail base sampling, but it's just not enough to get us to the point where we set up all that infrastructure to connect all those pieces, to solve all the challenges of performance, to really get the benefit of tail base sampling, at least from our return on investment if we're looking at those systems. That covers the suite of tips and tricks that we have for how we connect all those pieces together.
Questions and Answers
Participant 1: I worked on Sea of Thieves and we tried to set up something like this a while ago and we ran into the cost-benefit problem in a very real way, because the Unreal Engine is hot when it comes to logs. Have you found managing like what the cost-benefit at a higher level is for a system like this?
Nicholas Herring: At a higher level, it's one of these things where we start introducing Quasar and all the modern technologies. The biggest problem I saw immediately, so when we were originally designing that ecosystem, we were looking at the difference between a message bus ecosystem and a service mesh ecosystem. Ultimately, it was a judgment call on the maturity of our organization where we just weren't at a place where a service mesh would actually make sense for what we needed to learn, what we needed to do. Keep in mind, we adopted Kube pre-1.0. This was a while ago. The reason I'm talking about that is ultimately, we then realized that no matter what we picked, we needed something to be able to keep track of all these pieces and to ensure the success of this newer ecosystem to start shaving the yak, as it were, to move that legacy system over into migrating.
That became so real when we did an initial test where we set up 500 pirates in a solar system and put a ship right in the middle and we set off a smart bomb. Smart bombs blow up like this in a radial pattern and don't affect you. Then we watched what happens as it hit 500 other ships and we got a trace with 35,000 spans in it. Also, this is very much pre-AI. Then I looked at this, and go, we're learning stuff about our system that we simply just did not know. I was able to demonstrate this and show people, we have to have this. This is so important because it also helps us go in and look at the older systems and deconstruct and decompose those and understand how they're really working when we turn on those mechanisms there as well. Then it became a game of cat and mouse of figuring out, what are the important things and how do we really work with that?
Because we've also done fun things like where we've emitted spans per frame of the client, which is not a great thing to do when you have 30,000 GPUs screaming at you. We've definitely broken a few things here and there. It's more the mechanism of just observability in general of saying like, this is the cost that we have that this would need to get there, but this is what we get from it. Every engineer in our org at this point in time can't imagine a world before that. They're like, how did we get anything done? I think it's a big piece of championing the concept of it and illustrating what it provides. I think it was the first time in a long time that we could see into the past of this archaic system. The best way I can explain it is there's this always a sci-fi trope of, this old technology that no one understands. I was like, I hate this sci-fi trope so much. Then I started to work for CCP. I'm like, that makes so much more sense now.
Participant 2: How do you decide the granularity of your traces? Because when you were talking about the simulation and so on, it feels like one tick of your universe could cover a lot. Is there a consistent way of representing that? Because traces, when they get very big, can be very difficult to navigate. If they contain so much information and you're sampling them out, you lose a whole lot of information versus smaller traces. What are your thoughts on that?
Nicholas Herring: One of the things that we first learned, like when we were talking about rediscovering how this older system works, there's a bounty system in EVE Online, it's not quite what you think. Ultimately when you destroy an NPC ship, they have a bounty on them that gets paid out. That doesn't get paid out instantly. We have a coroutine somewhere going, every five minutes that I'll go pay out all of those. We discovered that thing quick when we turned on observability mechanisms. Because suddenly it was like, cool, the accountant for the universe showed up and started paying out everyone. We had to then target specific things. We're like, ok, we have this known pattern in our system where it's a worker mechanism. We short circuit those so the traces never start as a root there. We always then break those up so then we can start sampling them a different way.
There are also other tricks that we employed as well, because you just talking about that triggered some other things in my mind. When we're talking about sending notices to the client, we don't actually trace the delivery of those to the client because they are best effort. If they get there, they get there. If they don't, we'll send another one. It's fine. We don't actually trace those. That removed a lot of the quota from us. Another trick that we did was we integrated request-responses into a single span. We actually bisect the attribute namespace based on the request and response mechanisms. When we look at a request in our system in Honeycomb, one span represents the whole round trip. That cut our cost by half. It's great. There's a lot of little things that we do like that. Again, it's a judgment call on what's important for us. What do we actually care about here? Then tweaking how we instrument the system that way.
Liz Fong-Jones: How often do your developers go, I can't find that one specific trace that I'm looking for? Because that is the tradeoff of when you're sampling away data.
Nicholas Herring: I think the only time that happens is when they forget to turn on tracing. By default, in our ecosystem, when we have development, everyone has their own local solar system that they're working with, their local universe that they're working with. When they work with that, we'd by default turn those off because we have probably 50 people that would be running their own tenant of the universe. We don't want to trace all of those. We don't care about all of those. On some of our nameservers, our public test servers, we have that turned on. Some people will turn that on and they'll probably forget to actually enable the sampling because we turn that off by default for that ecosystem. That's usually when people are like, where is this? I don't see it. Then we say, turn this on. Then they usually get to see everything that happens. The key space in EMA usually guarantees that we at least get one of those somewhere. The frequency on local development versions of that are so sparse that it usually catches everything that they need.
Participant 3: I'm personally a little bit more familiar with a little game called Star Citizen. They make a very big deal about their persistence and data storage. I'm really curious what the sense of scale is of like an EVE Online or Star Citizen compared to a more conventional enterprise in sense of like data storage and messages, just amounts.
Nicholas Herring: For our ecosystem, I would estimate that we probably do roughly 60,000 messages a second, like real time. That's what we're doing right now as I speak. For storage, it really depends. The history of EVE is very old. Ultimately, I think our production database got up to, I'm going to say around 17-18 terabytes. Not that big. Also, it's very optimized for what's happening in a gaming universe. Now we have more and more happening where we're tracking even more things. For example, I brought up the system where people could create missions for other players. That has a running ledger of who's contributed to those missions and how much, because some of those are group projects. Then we have to pay out rewards for those types of things. The amount of data that we're actually persisting is growing quite a lot. A lot of what we're doing with the designs of the newer system, is to put in constraints around the sizing and the TTLs of that data so that we don't create infinite tables and then I have a dynamo bill show up, and that's the whole thing.
More importantly, a lot of what we do with the systems of the ingress point, so I mentioned like the 35 instances that we have, we call those our service gateways. They also fork every fact of the universe into an Azure Event Hub. Then, whether or not our data engineering team wants it or not, I'm sending it. They have a data lake that they're just filling up with all the details of that. Then as they build up the hub, they then process select information. For example, when we actually deployed the ship skin ecosystem, I immediately got a data engineer at my desk going, what are you sending us? Because in order to make that work, we had to make Quasar aware of where every ship in the universe was, at every tick of the simulation frame, which is an event in the ecosystem. Then he showed up going, I can't process this.
I'm like, you don't have to process it. I just need you to put it somewhere. I think it's a modest size of data and what we're doing. It's more about the interaction and the scale of the interaction of how many people can interact with the same part of space at the same time. I think that's what we're really focusing on for EVE.
See more presentations with transcripts