14 Oct, 2011, teqneek wrote in the 41st comment:
Votes: 0
Runter said:
teqneek said:
People are still going all over on the DB speed thing, that wasnt even what this topic was about lol. Thanks for all the previous posts and suggestions, im slowly moving fwd on a blend of the ideas ive read in this topic about a a ID/Tag system.


Sure it is. If you proselytize based on falsehood then it's certainly more than on topic for people to dispute said false information. Without this, threads could do more harm to readers when taken for granted that those false statements had undisputed merit. Furthermore, any type of claims that are questionable, even those not related to the topic at hand, has to be disputed for the same reason.
teqneek said:
Regardless… using an MMO period is a bad example.. Great EVE uses technology most of us dont have access to.. and SSD bridges the huge gap in performance between RAM and Disk, they have their game spread over 100 SOL Blade servers, 100 Proxy servers, and over 5 database clusters. All using stacklesse python, and SSD access. :P

Oh and they are noted as having the largest super computer in the gaming industry :P

I generally agree with you that eve online is a bad example, because it gives you too many outs like the ones you are using here. Why do you think using Python gives you speed unachieved in C or C++ or any other language for that matter? Yes, it's a great technology, but that's not the reason they're using python. You make it sound like the reason they're using all of this expensive technology is because they foolishly decided use a database. Madness, indeed. You haven't really responded to my previous post, and I'm the only person here who has actually posted benchmarks and actual factual data. I wasn't running on a solid state, server cluster or on a super computer. I ran it on my laptop with a normal hard drive.


In no way was I saying they chose this technology because of a database, I was saying how foolish it is to use a large commercial project as an example of those who use can achieve performance with a real time access database when they are using some pretty massive hardware to obtain the performance they want, at one point they did a massive upgrade to their clusters and SQL server just to keep up with the demand. Id like to see what Blizzard did with their system, as they house a substantially larger concurrent user base. While MUDs will never reach these requirements, alot of us do this as a hobby (Not sure how many actually make a living off of it :) ) so therefore trying to achieve results with a particular method no matter the size of the project is always a fun en-devour.

As far as your post on benchmarks, Yeah, using the command line i can see those same results. INSERTS and UPDATES I never saw as an issue, unless you request the inserted ID or updated ID you dont require a response from the DB Server. As far as select, I would love to see (Im sure i can mock up something for myself) the same select examples, using a more reasonable table return (more than 2 fields of INTs, as the more data retrieved is going to require more time) using joins, nested selects amd various other techniques. It would seem like an extreme waste of a SQL DB capabilities to simply pull from one table at a time and two fields. Do you not agree these factors would indeed increase query execution and return times?
14 Oct, 2011, Runter wrote in the 42nd comment:
Votes: 0
Blizzard shards their servers. When their "substantiall larger concurrent" user base gets too big, they add more servers. I think you're vastly mistaken about the size of their concurrent userbase per server. It's about 5000 users on taxed servers and half that on average. Games like runescape do the same thing with their worlds. I think 5000 is their magic number too. Furthermore, you're saying you want to engineer a solution for more users than you'll ever get just for fun. That's fine, but let's not confuse the reason one should do this. It's for fun, not because it's required.

Quote
Do you not agree these factors would indeed increase query execution and return times?


Yes, getting more fields would increase the query execution, but getting 3 fields individually is slower than grabbing 3 fields at once… so I'm not sure what your point is. There's actually a lot of synergy in database query. It's not more expensive and slower as the dataset increases. It's more efficient assuming you needed the data in the first place, and if you didn't…why are you including it in the query?
15 Oct, 2011, quixadhal wrote in the 43rd comment:
Votes: 0
That's why I used EVE as an example. They don't shard… they have a single cluster of machines working as a single universe. Last time I played (about 6 months ago?) the average count of users online was about 50,000, with it dipping down to 30,000 in off hours and peaking at almost 60K a few times.

They can only break down their universe into star systems, meaning a single system can't span multiple servers. So, busy systems like Jita have 500-1200 players in them at a time, all fighting or trading or moving around. ALL activities which require significant access to the data in their SQL database.

So, you have a few thousand system nodes, each of which has to keep a database connection open and perform some regular queries, as well as player activity which can be significant if in combat (EVE has limited ammo, so firing a weapon will cause a DB transaction to remove the ammo from the player and adjust the shield/armor/hull value of the target). Some hundreds of those nodes will have dozens or hundreds of players, and a handful of "hub" nodes will have hundreds of players.

FWIW, if you like you can grab a snapshot of .... The devs make one available every so often, as it's very handy for fan sites to be able to directly access the SQL data for equpiment stats and such. If your MUD schema becomes even 1/10th as complex, it'd be pretty cool.

You seem to be working under the assumption that large commercial projects have magic wands that mere mortals can't dream of using. Generally, not true. They use the same technology we do, but they have MUCH higher demands on them, and therefore HAVE to spend lots of money to try and squeeze every drop of performance they can out of what's available.

Is Oracle or MS SQL Server better than PostgreSQL or MySQL? Well… maybe. Both have more ways to control where the data lives and how it gets internally prioritized – and you have to be a good DBA to perform the in-depth analysis needed to actually make that work FOR you, rather than against you. They also both have features that the typical hobbyist won't usually need, like nested transactions and first-class stored procedures. But given a "normal" schema with 20 million rows of data, will they be significantly faster on your average insert/update/select? I doubt it.

Why did EVE use an SSD array? Because hard drives are too slow for SQL? No. Because they have such a large quantity of data, accessed in a large number of ways, by a large number of simultaneous clients, that there's no way to build a machine with enough RAM to cache enough of it. Your MUD, even if it becomes a super-popular one, will never come anywhere NEAR that level of performance requirements.

Heck, how's this for an example? WileyMUD (a popular DikuMUD from 1993) has a database of about 4300 rooms, 710 objects, 723 NPC's, and about 500 players. All of that, PLUS compressed log files, takes up a whopping 20M of disk space. If I were to load that into an SQL schema (with relationships and all), I would suspect it would expand to about 50M. Guess what? My 6 year old server has 768M of RAM. There's no doubt the entire database would be cached by PostgreSQL or MySQL. That makes it faster than an SSD drive, and faster than my clumsy attempts to cache disk I/O to files myself as well.

For simplicity, I suggest looking into one of the many "ORM" API's for SQL databases. Rather than trying to hand-craft a schema, these simply allow you to load and store structures directly in your database. It's not quite as elegant or efficient as doing it by hand, but for the level of performance you need you'd never notice and it's much simpler to work with in code.
15 Oct, 2011, Runter wrote in the 44th comment:
Votes: 0
Quote
Heck, how's this for an example? WileyMUD (a popular DikuMUD from 1993) has a database of about 4300 rooms, 710 objects, 723 NPC's, and about 500 players. All of that, PLUS compressed log files, takes up a whopping 20M of disk space. If I were to load that into an SQL schema (with relationships and all), I would suspect it would expand to about 50M. Guess what? My 6 year old server has 768M of RAM. There's no doubt the entire database would be cached by PostgreSQL or MySQL. That makes it faster than an SSD drive, and faster than my clumsy attempts to cache disk I/O to files myself as well.


I think that's a good point, and it's one of the reasons I showed the numbers for in memory databases.
15 Oct, 2011, Rarva.Riendf wrote in the 45th comment:
Votes: 0
for info
Affects: 5915
Areas: 158
ExDes: 2484
Exits: 33459
Helps: 938
Mobs -
Total: 3641
In Use: 8970
Objects: 5042/ 11176
Resets: 14079
Rooms: 14041
Shops: 317
Socials: 357

ROM/DIKU/MERC based whatever, it takes around 30meg of RAM… (basicaly a mob is nearly everything a player is)
And Plamzi use a DB without any problems with his mud Bedlam.
I do not just cause I think I have more important stuff to code than that, but if another coder would join, it definitely is on the todo list.
If you gonna change the system, just use a db all the way.
16 Oct, 2011, teqneek wrote in the 46th comment:
Votes: 0
Rarva.Riendf said:
for info
Affects: 5915
Areas: 158
ExDes: 2484
Exits: 33459
Helps: 938
Mobs -
Total: 3641
In Use: 8970
Objects: 5042/ 11176
Resets: 14079
Rooms: 14041
Shops: 317
Socials: 357

ROM/DIKU/MERC based whatever, it takes around 30meg of RAM… (basicaly a mob is nearly everything a player is)
And Plamzi use a DB without any problems with his mud Bedlam.
I do not just cause I think I have more important stuff to code than that, but if another coder would join, it definitely is on the todo list.
If you gonna change the system, just use a db all the way.


And ive converted several muds for fun to full SQL DBs, that wasnt my issue with one of the ideas being presented… the discussion in the previous posts werent about using it as a storage method soley, but more along the lines of using it as a full blown solution where in many cases you arent loading that data into memory, but using it real time as SQL allows you to access it.
16 Oct, 2011, Runter wrote in the 47th comment:
Votes: 0
teqneek said:
Rarva.Riendf said:
for info
Affects: 5915
Areas: 158
ExDes: 2484
Exits: 33459
Helps: 938
Mobs -
Total: 3641
In Use: 8970
Objects: 5042/ 11176
Resets: 14079
Rooms: 14041
Shops: 317
Socials: 357

ROM/DIKU/MERC based whatever, it takes around 30meg of RAM… (basicaly a mob is nearly everything a player is)
And Plamzi use a DB without any problems with his mud Bedlam.
I do not just cause I think I have more important stuff to code than that, but if another coder would join, it definitely is on the todo list.
If you gonna change the system, just use a db all the way.


And ive converted several muds for fun to full SQL DBs, that wasnt my issue with one of the ideas being presented… the discussion in the previous posts werent about using it as a storage method soley, but more along the lines of using it as a full blown solution where in many cases you arent loading that data into memory, but using it real time as SQL allows you to access it.


Why do you think a SQL store can't be held in memory?
16 Oct, 2011, Idealiad wrote in the 48th comment:
Votes: 0
Why do you think that's what he said?
16 Oct, 2011, Rarva.Riendf wrote in the 49th comment:
Votes: 0
Yeah that was my whole point, with everydata loaded I have it is only 30 meg, a DB has no reason to not cache it. Unless you force every transaction to be written to hard drive every single request, most of the load will only be done once in a while. Even with using files, the hard drive wont write it right away.
So the whole point is really…why create an id system at all if you use a db, the db will take care of it.
16 Oct, 2011, Runter wrote in the 50th comment:
Votes: 0
Idealiad said:
Why do you think that's what he said?


This would be why.
" but using it real time as SQL allows you to access it. "

You can access it straight from the store with SQL regardless of where the database is. Even if it's in memory.

In other words, I think he is correct in what he believes those previous stated positions were, but I think he's came to an erroneous conclusion that direct database access can't be in memory.

In fact, I gave an example of a sqlite3 database completely in memory with no disk writes/reads, a lot of times it's going to cache it without you needing to do anything. So I don't know why we're thinking anyone would make these assumptions about the position that you can use direct access to a database on a mud without having performance problems. It sounds like we're moving the goal post after losing the argument, to me.
17 Oct, 2011, David Haley wrote in the 51st comment:
Votes: 0
I think a lot of people don't realize that database doesn't have to mean a fully separate server process. :shrug:
40.0/51