One Hour One Life Forums

a multiplayer game of parenting and civilization building

You are not logged in.

#101 News » Update: Jungle Flora » 2019-12-20 01:49:11

Kaos
Replies: 0

HcSI9No.gif

Wow, what a week!  Steam, of course, had the expected effect:  an absolute flood of new players, and the resulting chaos.

OmlQ4Tg.png

This is the first weekly update post-Steam, and as you can tell, it's a bit later than usual.  And the release of the update, for any of you playing on Sunday evening when it came out, was quite a bit bumpier than usual.  I'll explain more about that in a bit, but first, the update itself.

As you can see from the GIF, there's a brand new biome, the first new one in a while.  If you spend some time there, you'll find that it has a pretty good balance between benefits and drawbacks, though that balance will be tweaked over time.  It also contains raw ingredients that will help you push transportation tech into even more efficient and durable realms.

Beyond the new content, the biggest change client-side is that the crafting hint filter system has been overhauled to align better with people's expectations.  In general, the crafting hint system shows you hints for what you can do with what you are holding, or what you just touched.  It does not show you specifically how to make an end product, but steps along the way that you might want to explore.  The filter system (/hatchet) is meant to cull down the list for what you are holding to show you only steps along thew way that lead to a hatchet.

First, an easy fix.  The old filter system would show you any steps along the way AND anything you could do with the hatchet once you got it.  This isn't very helpful---you're only filtering by /hatchet because you don't have one yet.  Who cares what you can do with it?  So those are hidden now, when filtering.

One point of confusion:  What happens when a hatchet itself can be used to make some of the end-products for a hatchet?  This may not be true for a hatchet, but it's true for an axe, which requires kindling to make, but can also make kindling.  Still, it's best not to show this when filtering for /axe.  There have to be other ways to make kindling, after all, because you don't have the axe yet.

Finally, what happens when none of the steps for what you're holding are relevant to /hatchet?  Let's say you grab a piece of flint.  In the old system, the full, unfiltered list for flint would be shown, and "NONE RELEVANT" would be shown next to the filter.  But a lot of people didn't see this, and it was confusing.  Why is it showing that I can skin a wolf with flint?  I'm trying to make a hatchet!  Now it shows no steps at all, and spells it out for you.  "MAKING HATCHET (FLINT NOT RELEVANT)"

Also, I found and fixed some mistakes in the filtering logic that led to the display of extra, irrelevant steps.  To summarize, the whole filtering thing works much better than it used to.  Try it.  You'll like it.


Okay, so what happened during the update process?

First, an old version mismatch issue which had bitten me before reared its head.  The client makes sure that it is at or ahead of the server's version, and if it is ahead, it makes sure that it's data version at least matches the server's version.  This works fine most of the time.  For the weekly update, I can decide to either update the client binary first, or update the data an server first.

But during weeks where the client and server receive several updates, while the data version remains untouched (like this week, when I was fixing all kinds of steam issues), we can get into a dangerous situation.  The server and data version MUST be updated together, first, before the client is updated.

But this week, there were protocol changes requiring that the client be updated first.  The server was going to send newly-formatted messages that the client wouldn't be able to parse.  So, I blindly updated the client first, without realizing that we were already several versions ahead of the data version.

On next startup, the client checks if its version matches the server.  Nope.  Server is v167.  Client is v168.  Okay, well, is the client data version at least matching the server's version?  Nope.  Data is v164.  Client displays a version mismatch error.

This was put in place to make sure that people connecting to private servers wouldn't experience content-mismatch crashes if their private server lagged behind the official update schedule.  But actually, the above logic is too strict.  If the client is really at v168, we are ahead of the server, but we can see that our data version number is behind the server, so we're actually okay.  What we really need to worry about, in the case of private servers, is if both the data version and binary version number are ahead of the server's version number.  So, the client logic has been fixed to prevent this kind of version mismatch in future updates.

So that was fire #1.  It caused about 34 minutes of downtime for people who were downloading the update and trying to reconnect.  You can see the first huge downward spike in the above graph.


After that got sorted, and everyone had a working client again, it was time to push out the content update.  That all went according to plan, and the Steam clients got the update correctly.  Everything went fine, until server2 tried to start up.  It got through its init process, and started accepting connections, but then got bogged down into a huge processing load.  99% cpu, even with 0 players connected.  And still accepting connections.

The reflector, the bit of server code that tells you which server you should go to, had a 3-second timeout in place for a server.  So, when checking on server2, it would wait 3 seconds, and then timeout.  But with hundreds of client requests coming in, 3 seconds of waiting for each one was too much.  And server2 was at the top of the list, meaning that the reflector would consider it first, for everyone, before considering other servers.  100s of stalled web requests means that nginx starts running out of processes, eventually triggering the dreaded 504 gateway timeout.  This results in no one getting successfully assigned to a server, and error messages getting displayed in the client.

Fortunately, I was able to track the problem to server2 very quickly, and take it off the list.  I think there was something like 15 minutes of downtime from this, fire #2.

But I also realized that server1, which was in the queue to update next, would likely have a similar problem.  Other servers didn't seem to be affected.

Initial debugging showed that server2 was bogged down with 200,000 moving animals.  Hmm.... maybe I made mosquitoes too common in this update.  That must be it.... the Mosquito Meltdown.

But further investigation showed that it wasn't mosquitoes at all, but wild sheep and boars.  200,000 of them, all trying to move at the same time, right after server startup.

Well, thank you Steam, for exploring more of the map on server2 than had ever been explored, because you unearthed a horrible, lurking issue.  Animals aren't supposed to move unless they are seen by a player.  So, even if there are 200,000 of them known to be in the map (seen by some player at some time in the past), at server startup, this should be no different than a map containing 200,000 trees or rocks.

However, a particular bit of server map processing code was inadvertently "looking" at these moving animals, causing them all to get put in the queue of moving things that need active updating.

This has apparently been a lurking problem for a while, but historically small enough that the server could simply brute-force its way through it at start up, and get back to processing requests.  Not this week.  200,000 moving animals.

The first step was to fix the reflector so that it would timeout more quickly.  These servers are all connected by high-speed networks.  3 seconds is overkill.  I was actually able to bring the timeout down to 1/8 second (125 ms).  If a sever is too bogged to respond in that time, better think of it as offline anyway, for now, right?

And with that fix in place, when server1 restarted and had exactly the same issue as server2, there was no widespread outage triggered.

With everyone back online and happy, I spent the rest of a very long night refining the server startup code to avoid "looking at" any of the map objects that are being processed.  I got server1 down to the point where it could startup, process the entire map, including all 200,000 animals, but not result in a single animal waking up and moving, at least not until a player spawned in to see them.

This also dramatically sped up the server startup and shutdown process.  The other servers will get this improved code during the next update, later this week.

#102 News » Update: Games of Chance » 2019-12-20 01:46:14

Kaos
Replies: 0

IQvXfwo.gif

This week I developed an extension of the category system which allows crafting transitions to have more than one possible outcome.  This is obviously needed for things like fishing, which is included in a minor way in this update.  The old "do this and get that, guaranteed" just doesn't feel right when dipping a hook in the water.  You'd be hauling 'em out one after another.

All transitions in this game are A + B = C + D.  In the past, A and B could be categories, which might mean that an axe can chop anything in the "kindling source" category into kindling.

The new probability system now allows C and D to be categories as well, but a new kind of category with weights for each member.  Whenever the transition is invoked, one of the members is picked at random, according to the weights.  The weights can be any probability distribution.  Some outcomes can then be very rare, when that's appropriate (like catching a special fish, or finding a diamond in a mine).

But I didn't stop at fishing.  To fully put this new system through its paces, it made sense to explore the human activity that led to the development of probability theory in the first place:  games of chance.  Also known as gambling.  Thus, I didn't just stick in a probability set with a few outcomes, like fishing.  I included one set with 21 outcomes (for a pair of rolled dice), and another with 52 outcomes (for a deck of cards).  Good thing I did this, because including categories this big unearthed a serious, long-term crash lurking in the category system.

And come to think of it, fishing and gambling really do go hand in hand.

The tutorial was also updated with some improved wording and a "hint hallway" for people who are really stuck (age-gated at 35, so you have to have been trying for 20 minutes before you get any hints).

Those of you who saw moving ice holes related to penguins, that bug has finally been found and fixed.  And a new protocol is in place (PING/PONG) to help the client detect true cases of the bouncing-forever bug, as opposed to just genuine network outages.  This should result in far fewer false reports.  And one new cause of bouncing-forever, this time caused by the new reconnect system, has been fixed.

And stacks, and partially-used objects, no longer revert to their full state on server restart.  Their true states are correctly remembered.  This also means no more weird universal locks after server restart.

All that said, I still have quite a long list of mini-bugs on GitHub to fix this week before the Steam release on Thursday.  That will be my focus, and there will be no new content this week (it would be weird to ship a content update on the very day that the game goes live---the new players will need a week to settle in).  I hope to push out the final bug fixes (including content fixes) on Tuesday, to allow time for some last-minute fixes on Wednesday as well.

The plan is to put out a nice-sized content update the following week, and then continue with weekly content updates thereafter.

#103 News » Update: Plaster and Paint » 2019-12-20 01:45:18

Kaos
Replies: 0

tVzPd9K.gif

Plaster is cheap, but pigments are a bit harder to come by.  Except for yellow.  That one is easy, but oh so cruel.  Outlawed in 1908, in fact.  Have you ever noticed that mangoes make your mouth itch a little bit?  Now imagine being force-fed the leaves!

And yes, I suspect that I'm one of the only game designers in the world who, upon deciding to add wall paints to my game, realized that I would also need to add mango trees.  That leads me to ask:  where would this game be without Wikipedia?  There's an information rabbit hole lurking around every corner.

Steam integration seems to be up and running smoothly.  When this update went out, the Steam client fetched and installed it (with a bit of automated goading on my part).  The plan is to get one more content update out the door next week before Steam launch week.  I also have a nice fat queue of GitHub bug reports to go through.  Thank you, and keep 'em coming, folks.

#104 News » Update: Steam integration complete » 2019-12-20 01:44:26

Kaos
Replies: 0

All the pieces are finally in place to make the game work on Steam.  The automated update scripts (both for client binaries and platform-independent content updates) have been augmented to include a Steam build step, which posts the changes to Steam automatically.  Steam actually has a pretty powerful command-line interface (steamcmd), which makes scripting possible.

Account creation, server-side, based on Steam ownerership, is also working.  And the final pieces are in place to enable all of you, the existing owners of the game, to unlock the game on Steam and automatically access your account details on the Steam side (you won't even need to type in your key again after installing the game through Steam).  However, I'm still waiting for Valve to flip the switch on the GrantPackage feature for OHOL.  Hopefully, they will do that early next week, and then you can all start testing the game on Steam.

So close!

One thing I should warn you all about is that the Steam release will be Windows-only for the time being.  The build process is complicated enough, and the Mac platform is finicky enough, that I've decided not to tackle it for now.  Come to think of it, the old Mac that I build the game on probably can't even run Steam, so I'd have no way to test it.  And sadly, Linux on Steam is even less of a priority.  In my experience, Mac and Linux users make up only a tiny sliver of the Steam audience anyway.  The Mac and Linux builds will continue being updated off-Steam as usual, however (and anyone who buys the game on Steam will be able to access them).  Hey, I'm typing this update message from Linux right now, so I'm a true believer, but I don't even have Steam installed on here.

And the bug reports keep on coming.  Thank you everyone for your diligent work reporting all this little stuff to me.  When it's all organized on GitHub, I can just go down the list and knock out fix after fix.  And this week, there are a ton of fixes.  You can now use your baby's head to open doors and launch rail carts.  Donkey Town residents see a count of their excess curse score.  Some 10-year-old causes of spurious network disconnects have been fixed.

As I type this, there are 15 more new bug reports that just came in.  More, please.   Content issues go here:
https://github.com/jasonrohrer/OneLifeData7/issues

Programming issues go here:
https://github.com/jasonrohrer/OneLife/issues

Now that the Steam stuff is mostly dealt with, I will spend the next few weeks returning to content updates, as well as fixing the remaining bugs.

#105 News » Update: More bug fixes » 2019-12-20 01:43:29

Kaos
Replies: 0

First things first.  The Steam page for One Hour One Life is live here:

https://store.steampowered.com/app/5956 … _One_Life/

Steam integration has been going well, including support for automating the weekly update process.  Next week, I'll be working on Steam account integration.  Hopefully, those of you who already bought the game will be able to unlock it on Steam soon and help with Steam testing.

Beyond that, this has been another week of pretty deep bug fixes.  It's astounding to me how long some of these took to track down.  The infamous "bouncing forever" bug has had many causes.  Two more big ones have just been found and fixed thanks to your reports.  The fixes involved some general robustness changes to the protocol that will improve and normalize the behavior of the game in all kinds of situations.  Using a randomized network latency simulator, I found that the game was degenerating pretty horribly when messages arrive with inconsistent delays and bursts.

For example, some messages sent by the server are meant to go together.  When you pick up a stone the game, the server sends two messages:  one about your changed state (a PLAYER_UPDATE message, you now have a stone in your hand), and one about the map's changed state (a MAP_CHANGE message, there's now an empty tile where the stone used to be).  The server sends these to you at the same time, and under normal networking conditions, those messages will arrive at the client at the same time, and end up changing the display (of you and of the map) during the same rendering frame.  However, with a flaky network, these messages can arrive with a big time gap between them.  This would mean that you would get updated to show you holding the stone, while the map would still show the stone on the ground, giving the false appearance of temporarily duplicating stones.  Things get even worse when the messages are about parents and their held babies.

This has been corrected with a new message FRAME protocol.  Messages that happen during the same server step are grouped into a frame, and the client waits until all messages in that frame have arrived before processing any of them.  Thus, the updates to you and the map, when you pick up the stone, are always shown at the same time on the client, because the server sends them in the same frame.

There were also some lingering inconsistencies with actions that held babies attempt to take right before being picked up, depending on the order in which messages arrive at the server.  This has been fixed by the server ignoring further messages from held babies after it has decided that they are indeed held---until they send a JUMP message to leave their parent's arms.  Before, even something as simple as a baby removing a hat right as they were picked up would cause a weird behavior---a seemignly disappearing hat.  Now the server enforces a strict ordering of events for held babies, and the behavior is much more consistent.

Most of these issues have resulted in occasional glitchy behavior that has been with us since the beginning.  Good to finally have these things fixed now.

There were also a bunch of other little content and bug fixes.  Fence gates are a thing now, for example, and other non-gate fences cannot be easily opened with a bare hand.  Teleporters have been fixed.

Many of these fixes were the direct result of people logging issues on GitHub, which is the main way that I keep track of things that need to be fixed.  You can log content bugs here:

https://github.com/jasonrohrer/OneLifeData7/issues

If you think you've found a programming bug, you can log that here:

https://github.com/jasonrohrer/OneLife/issues

Special thanks to Stylingirl and IEatDaSpaghetti for all their helpful bug reports this week.

#106 News » Update: Temperature system fixed » 2019-12-20 01:42:22

Kaos
Replies: 0

This week's update focuses on a bunch of important fixes. First of all, the temperature system, which was always very finicky, has been smoothed out. The "just keep running" exploit to lock your temperature no longer works, and your temperature will no longer jump around suddenly and unexpectedly when you step next to a fire. Instead, your body temperature gradually moves toward the environment temperature over time. Furthermore, temperature updates happen whether or not you are moving (every two seconds). This also means that if the fire goes out next to you, but you don't move, your temperature will eventually go down to reflect the new, cooler environment around you.

The code that determines where you click when you click on yourself has been fixed to prevent mistaken clicks on your clothing that extends behind your face when you click on your face. This was particularly troublesome for children wearing backpacks or aprons. Clicking their face to eat food used to mistakenly put the food into the backpack or apron pocket (because for a child, the backpack strap and apron top are behind their face). Now the head takes precedence, given that it's actually in front of the clothing anyway.

Some of you may be aware of the game recording feature. For bug-catching purposes, every game you play is recorded in a compact, text-only form. These are saved in your recordedGames folder. They contain all mouse, keyboard, and network events, allowing the game you played to be simulated at a later time. Dropping one of these files into your playbackGame folder, and running the game, will allow you to watch a "ghost" version of yourself playing. As you can imagine, this is very useful for reproducing bugs.

However, these recording files can also take up quite a bit if disk space over time. The game now defaults to keeping only the most-recent 20 recordings (though this behavior can be tweaked in the settings folder). Turning recording off completely has always been possible, and it will increase performance a little but, but I encourage people to keep it on for the time being, because you never know when you will get a recording of a rare bug.

That hilarious clothing blushing bug has been fixed, along with a few lurking server bugs.

Next week, I'll be focusing on the Steam integration process. Hopefully, there will be a way to "unlock" the game on Steam for all of you to start testing next week. I have discovered a way to do this without handing out Steam keys, which is even easier for the end-user, and there's always the danger that unwanted Steam keys will get sold on key marketplaces at a discount.

#107 News » Update: Emotion » 2019-12-20 01:41:46

Kaos
Replies: 0

PyKaKYE.gif

This week's update contains a few little fixes, but the main focus of the update is a new emotion system.  The available emotions are currently:

/HAPPY
/SAD
/ANGRY
/MAD
/JOY
/DEVIOUS
/BLUSH

More will be added in the future.

I'm also busily preparing to release One Hour One Life on Steam.  The game is in a good place, and I think it's ready for a larger audience.  Preparing for Steam is quite a bit of work, especially for a multiplayer game, and I expect this process to take a few weeks.  In the mean time, I'll still be putting out weekly updates, but they will be on the smaller side.

After the game launches on Steam, all players will continue playing together on the same servers.  The game will continue to be sold off-Steam, and your non-Steam game accounts will keep working as usual.  Everyone who bought the game off-Steam will get a free Steam key to unlock the game on Steam---but this is optional.

The only down-side to a Steam release is the division of the game's community.  There will be Steam forums, and in my experience, Steam users don't like to leave Steam and join external forums.  So discussion about the game will be split into two places.

#108 News » Update: Tree Farm » 2019-12-20 01:40:52

Kaos
Replies: 0

SIKvjqr.gif

People have been asking about this for a while:  you can replant trees now.  But trees are slow-growing things, so it's really a multi-generational project.  And Bonsai---those are really slow growers.

Before this update, each server tracked curse scores separately, meaning that you could grief and get cursed into Donkey Town on one server, but then switch servers and be uncursed again.  Now there's a global, centralized curse server that all the game servers share, so each player has one global curse score, regardless of which server they are playing on.  If they're in Donkey Town, they're in Donkey Town on all servers.

There are a bunch of other, small content fixes.  A bunch of other things, like flat rocks and paper, are now stackable.

#109 News » Update: Dogs and Donkeys » 2019-12-20 01:38:57

Kaos
Replies: 0

izr5NGC.gif

Are you a pie gobbler?  Tool hider?  Seed thrower?  Forest chopper?  Congratulations!  You just won an all-expense paid trip to Donkey Town.  There, you will find a paradise where you and your fellow pie gobblers can gobble each other's pies for hours of endless, giddy enjoyment.

There will also be dogs.  Too many dogs, from early reports.  Not just in Donkey Town, but everywhere.  Did you ever read the book "Go Dog. Go!" by P. D. Eastman?  We're talking more dogs than that, even.  Perhaps we will add "dog-overbreeder" to the list, and you can hop on that train to Donkey Town with the rest of them.

The clicking user interface for the backpack, when you're holding food in your hand, has been improved.  Left-clicking on the pack puts food into the pack, right-click swaps the food with the bottom item in the pack, and left-clicking on yourself causes you to eat the food.

#110 News » Update: Friends and Strangers » 2019-12-20 01:35:06

Kaos
Replies: 0

f3Adouq.gif

I have returned from my family vacation, and I've brought you home an update.

First, two new characters, from two new skin tones.

Then loads of bug fixes and little improvements.  You can now right-click to swap what you're holding with items in your backpack.  Some glitches in baby naming have been fixed.  At least one cause of the bouncing-forever bug has been found and fixed.  Flying objects, when you pick something up and then immediately walk away, have been fixed to snap right to you as soon as you start moving.  You can no longer steal backpacks from old people by swapping their pack with another pack.

Finally, last week, you may have experienced a few deadly "lag waves," where everyone on the server freezes for 30+ seconds, and then suddenly the connection comes back.  Depending on hunger status at the time of the temporary outage, a whole village could potentially be wiped out.  I traced this issue to some kind of networking problem with my hosting provider.  They investigated, and decided to perform an emergency migration of Server1 to a different hardware node.  Hopefully, the problem should be fixed.

#111 News » Update: Written Word » 2019-12-20 01:34:05

Kaos
Replies: 0

RZvo98n.gif

You can write now.

You need to make paper first, and something to write with, of course.  And like little Cupid, you can now send someone a Valentine message by air mail.

There are also a bunch of things that are now stackable that weren't before, like kindling, ore, and dishes.

After 24 weeks of updates, I'm off on vacation.  There will be no update next week.

#112 News » Update: Roads and Rails » 2019-12-20 01:32:59

Kaos
Replies: 0

XlcGsNW.gif

New transportation systems.  Roads and rails.

I've also reduced the distance in the Eve spreading algorithm by a factor of two, so it should be more realistic to travel between towns and connect them with roads.

One more update, next week, and then I'm taking a much needed family vacation for a few weeks.  There have been weekly updates for 22 weeks in a row at this point.

#113 News » Update: Bison and Butter » 2019-12-20 01:32:04

Kaos
Replies: 0

This is the first substantial content update in a while.

The past few weeks have had me focused on rooting out the causes of server lag issues. After testing various solutions on Server1, the issues were finally resolved. Earlier this week, the final replacement database engine was rolled out to all fifteen servers, allowing us to push the player population cap up to 200 per server.

After that, I returned my focus to content. I've been wanting to add more animal husbandry for a while, and this update gives you a new one to domesticate. And what good is butter if you don't have anything to spread it on?

#114 News » Update: All servers are now running the new, low-latency database » 2019-12-20 01:31:31

Kaos
Replies: 0

After testing for a bit on Server 1, the new database engine has been rolled out to all fifteen servers, and the player caps on each server has been pushed up to 200. CPU and disk usage on the servers is looking good.

Server lag should be a thing of the past.

I'm finally returning to working on new content for the game. There will be some kind of content update at the end of this week.

#115 News » Update: Even newer, even-lower-latency database engine is live » 2019-12-20 01:31:04

Kaos
Replies: 0

You probably feel like you've heard about this issue too much already.

But making a lag-free server is a top priority, and getting the database engine right is the most important part.

Last week, I introduced a custom-coded database engine that made null-lookups (when you find out that nothing is there on the map) extremely fast. These are the most common database actions---imagine someone walking around in the wilderness, and we need to find out that nothing human-made is on each and every map cell that they're exploring. That's a lot of null-lookups.

However, the architecture of this new database engine also made inserts---a much less common operation---quite a bit slower. The hash table is spread across the data file, and that means that newly inserted data is spread randomly across the data file. KISSDB and my previous replacement STACKDB did not distribute the data across the disk in hash table order, but I never thought about why.

It turns out that writing in a bunch of random locations in a file is really slow, because it causes cache misses constantly. So it's best to write all new data at the end of the file, in an arbitrary order, instead of treating the file like one big hash table.

Of course, this doesn't matter most of the time.

Except when loading the tutorial map into the world for a player. At that moment, we insert thousands of new records into the database. This was taking something like 8 seconds on server1 with the new database engine. That's an 8-second lag for EVERYONE every time any player loads the tutorial map.

The latest database engine does things quite a bit differently, keeping the entire hash table in RAM and keeping data on the disk in the order that they are inserted. Thus, when a big sequence of inserts happens, like when the tutorial is loaded into the world for someone, all of those inserts happen in order at the end of the file, without a single file seek along the way.

And, given that the entire table is kept in RAM, null-lookups, and all other operations, are substantially faster than in any previous database engine.

The result of all this is that Server1 is back online and hopefully more lag-free than ever. 43 players are currently on it, and it's only using 7% CPU for that load.

There's still a small bit of optimization work to be done. When the map database is huge, as it is on Sever1, tutorial map inserts are still a bit slower than I would like them to be (a little over 1 second), but given that it's 9pm on Friday night, and my kids are waiting to play League Of Legends with me, that sounds like something I will tackle on Monday morning.

And yes, the bison is coming soon... promise!

#116 News » Update: New low-latency database engine is live » 2019-12-20 01:29:54

Kaos
Replies: 0

With weary coding fingertips I type to let you know that a very long week has paid off.

linearDBCPUGraph.png

More profiling of the bedraggled Server1, which has the largest map data set of any server, revealed that file IO inefficiencies in the custom-coded StackDB were to blame.  StackDB was designed to quickly answer questions about recently-accessed map cells---assuming that people in cities are often looking at the same stuff, so that stuff should be kept near the top of the stack.  The old off-the-shelf KissDB did not do that, meaning that the newest stuff was the slowest to access as the data set grew.

However, none of these optimizations addressed what is actually the most common case:  asking about a map cell that isn't in the database at all.  When you wander around in the wilderness and look at the empty ground, we have to ask the database to confirm that that patch of ground is indeed empty.  Maybe someone visited that spot earlier and dropped something there that you should see.

It turns out that in both KissDB and StackDB, this is the slowest operation of all.  A non-existent cell can never be at the top of the stack, because it doesn't exist, which means that we need to walk to the bottom of the stack to find out for sure that it doesn't exist.

Finally, KissDB and StackDB are both hash table systems, but both of them use fixed size hash tables.  In the case of Server1, there were 15 million data records crammed into an 80,000 slot hash table.  This means lots of pages to look through in each slot (KissDB) or deep stacks to wade to the bottom of (StackDB) to find out that a given map cell really isn't there, and therefore is empty.

Even worse, the architecture of both engines requires loads of random-access disk seeks to move through the pages or the stack.  And disk seeks are extremely slow, relatively speaking, especially when they are jumping around a huge file and missing the cache over and over.

LinearDB, my latest custom-coded database engine, has an ever-expanding hash table based on a very clever algorithm---which I did not invent---called Linear Hashing.  The hash table grows gradually along with the data, essentially never letting it get too many layers deep.  In addition, a kind of "mini map" of data fingerprints is kept in RAM, allowing us to ask questions about map cells that don't exist without touching the disk at all.

The performance gains here are pretty astounding.

On a simple benchmark where a single player walks in a straight line through the wilderness for a minute, the old database engine performed 1.9 million disk seeks and 3.8 million disk reads.

During the same single-player journey, the new database engine performs less than 4700 seeks and 1600 reads.

Yes, that's a 427x and 2400x in disk seeks and reads, respectively.

According to a system call time profiler, this results in approximately 180x less time spent waiting for the disk.  In other words, this part of the server is now one hundred and eighty times faster than it used to be.

Server1 is the only server that has this new engine installed so far.  It has the biggest data sets and was seeing the most lag with the old database, so it's the best server to stress test with.  It's now back in circulation at the top of the list and seems to be lag-free.  I'll be incrementally increasing the player cap over the next few days and seeing how it handles the load.

Assuming that all goes well, I will be rolling the new database engine out to the other servers next week.  The end of server lag is almost in sight.

A big thanks goes out to sc0rp, who spent many hours discussing the intricacies of these systems with me in the forums, and filled my head with all sorts of great ideas.  I had never even heard of linear hashing until sc0rp told me about it.

#117 News » Update: Server lag optimzation and client improvements » 2019-12-20 01:27:30

Kaos
Replies: 0

With the recent influx of new players to the game, the servers have been struggling to keep up.

Several months ago, I spent a lot of time on server database optimization, which made the servers around seven times more efficient than they were originally.  But as maps fill up with player content, there's more and more information to process, and the load generally grows over time.  In the past week, Server1, which has the most extensively settled map, had gotten very laggy.  It was time to take another look.

This round of profiling revealed a bunch of hot spots that weren't database-related.  The optimization process involves running a server with a profiler (I use the amazing Callgrind profiler from the Valgrind project), finding the most obvious hotspot, figuring out if there's a way to speed it up or---even better---skip that operation entirely, and then repeating with a new build to find the next biggest hotspot.  I ended up going through this process nine times, fixing nine hotspots along the way.  Some of these changes resulted in 7x speedups to certain parts of the server code.

However, even after several days of intensive work that showed huge performance gains in the profiler, when I finally brought Server1 back online for public use and a load of 38 players, the lag returned.  There are more issue afoot with Server1 than just slow code.  CPU usage jumped up back to 70%, while Server3 sits happily at around 20% with exactly the same player load.

So, Server1 and Server2 will remain "on ice" over the weekend, at the end of the server list where no one will use them by default, until Monday when I will resume diagnosing Server1's lag issues.

You may have also noticed that the connection management features of the client have been greatly improved.  You can now specify a custom server from the SETTINGS screen, and copy/paste server addresses to share with friends.  Bugs in the twin matching have also been resolved, so joining twin games is reliable.  Some issues that caused logins to fail have also been fixed.

#118 News » Update: Twins » 2019-12-20 01:26:27

Kaos
Replies: 0

WlIpZlF.gif

You want to play with friends.

But how can you do that without subverting the entire premise of the game?  This is a game about being born as the helpless baby of a total stranger, and depending on that stranger for your survival.  It's also about playing a small part in a family of strangers.  You didn't know your mother in real life before you were born, and that's what makes her so special.  Imagine if your mother was really your college roommate in disguise, and you knew it.  Imagine if your mother talked to you over voice chat, and she sounded just like your college roommate.

But you want to play with friends.  There's only one solution here, and I never said it was going to be pretty.

How can you play with friends and yet still be born as a helpless baby of a total stranger?  Only if you and your and your friends are ALL born as helpless babies to the same total stranger.  Twins.  Triplets.  Quadruplets.

If you act like you've known each other since before birth, you have---you shared a womb together, after all.  To others in the game, you almost seem to read each other's minds.  And you look identical.  And your poor mother---one baby is hard enough.  It's not easy being a twin.

Also, a bunch of little bugs have been fixed along the way.

#119 News » Update: Curses » 2019-12-20 01:25:40

Kaos
Replies: 0

griefers.... Griefers!  GRIEFERS!!!

Yes, you can kill them, just like in real life, but there's one little wrinkle:  this game has reincarnation.  Thus, you may be dogged by one wayward soul life after life after life.  You just can't seem to get rid of them.  Even if everyone in your village agrees, you're helpless, long-term, to deal with this handful of trouble-makers.

A metaphysical problem needs a metaphysical solution:  The brand new curse system.

Everyone starts out with a curse score of zero and a single curse token to spend.  They spend their token by saying CURSE JOHN SMITH.  Mr. Smith's curse score goes up by one.

If you have no curse token, you earn another one after playing/living for two hours.  You have a max of one token.

If you have a curse score, it goes down by one point after playing/living for one hour.

If Mr. Smith's curse score ever reaches 10, he will be born into his next life marked with inverse-color speech bubbles.

And yes, cursing is done by name, so unique baby names are more important than ever.  If two people share a name, the one that received the name first gets the curse point.  And nameless people cannot be referred to by name.  Name your babies, people.

Also in this update, along with few fixes, is a new home arrow distance estimate if the home arrow (or bell tower) is more than 1000 tiles away. And the tutorial has been updated with a few new things.... even veteran players might want to take another look at it, especially past the end area.

#120 News » Update: Tutorial » 2019-12-20 01:25:08

Kaos
Replies: 0

If you bounced off the game in the past because it was too hard to figure out, this is a great time to give it another shot.

This week's update brings a brand new tutorial.

This has always been a game that just throws you right in there, and that will remain the case, so don't worry.  The tutorial mostly explains the nuances of the controls and systems.  There are a lot of little things that just aren't explained anywhere else.  Some of these things are lost on veteran players.

Also added, and explained in the tutorial, is the new crafting-hint filter feature.  If you want to make a hatchet, for example, you can cull down the hint list to show only steps that are relevant along the way to the hatchet.  This greatly reduces the "sharp stone does 46 things" problem, and should allow pretty much everyone to be able to figure out the final crafting challenge at the tail end of the tutorial.

You have to get through the tutorial, and solve that last crafting challenge, to actually play the game, so I'm really betting the farm on this one.  At least you'll know that if a baby is born to you in-game, that player is not totally clueless.

This new filter system also just might make wikis and other external crafting guides unnecessary, but we'll see how it goes.

A bunch of little content issues have been fixed.  Committing murder now "counts" as living a full time block in a family line, just like being a victim of murder does.  Griefers can no longer kill and then suicide before 30 minutes elapse to come back at the same family again.  The lineage ban has been upped to 1.5 hours of life lived in other lines before returning to a given family line.

#121 News » Update: Emergency Medicine » 2019-12-20 01:24:23

Kaos
Replies: 0

ptILRXv.gif

This week's update ushers in a major change to the core of the game:  wounds are no longer guaranteed to be fatal.

But hold on a second, don't equip that health pack just yet.  Performing emergency surgery on yourself is harder than it sounds.  Better done with the help of a friend.

And this is why wound recovery does not immediately aid a lone griefer.  There is safety in numbers.

If you're dreaming of armor and a backpack full of medical syringes as a means of solo self defense, you might want to consider what happens when a griefer gets their hands on the same load-out.

On a completely unrelated note:  there are now pigs.

Next week, I'll be working on a much-needed tutorial to explain some of the more subtle features to new players.

And before I hear any grumbling about war crimes and my use of a red cross, this is an artistic work of fiction.  The Geneva Conventions were never intended to apply.

#122 News » Update: Good Eating » 2019-12-20 01:22:43

Kaos
Replies: 0

s27QIcH.gif

There's now a good reason to eat all sorts of delicious foods, instead of just mono-gorging the most efficient and convenient food.  Omelettes and tacos and kraut and stew.  And even more reason to share food and meals than there ever was before, and hopefully a good reason to have a village chef that might coordinate all of this for you.  Tuesday is taco night.

This update adds a YUM mechanic, which gives you a bonus for feasting with a long chain of food variety.  The more variety you eat, the better you feel.  So let's eat beans for every meal!   Er... yeah, so plain bean tacos do get a little boring.  Future updates will add some toppings to spice things up, and maybe even some slow-cooked carnitas.  Get your limed corn ready, people.

Some of the more fiddly and plentiful seeds can now be stored in bowls, and a new domesticated animal sneaked in there too.

#123 News » Update: Big Farm » 2019-12-20 01:21:40

Kaos
Replies: 0

eCzGkEi.gif

With most of the post-launch growing pains behind us, I was freed up this week to spend all most all of my time on content.  There's a lot of new stuff in there, mostly concerning expanded gardening and things you can make with the products of that gardening.

A few of the end-products are personally meaningful to me---things that my family has been making every year for many years, with all most all of the real-life steps in place.  In fact, I just ate some of our red-cabbage sauerkraut during my lunch break today.

Oh, and if you've ever tried to cut up a hubbard squash, you know that a kitchen knife isn't going to do the job.  The first time I saw one of these monsters in person, many years ago, I felt like I was looking at something straight out of Star Wars.  One of the strangest looking edible fruits around.

If you're paying close attention, you will also notice that the days of berry farm utopia are coming to an end, with berries now requiring soil, just like every other crop.  However, soil has become both more plentiful and easier to move around.  I'm still trying to design new foods in the game so that everything has its niche, but that approach will only take us so far.  I envision 100s of unique foods in this game eventually, and I'm currently tinkering with new mechanics that will motivate a varied diet in a simple, straight-forward way (no, it's not a tedious nutrition system, don't worry).

#124 News » Update: Small Farm » 2019-12-20 01:20:49

Kaos
Replies: 0

This week's update lays the groundwork for the Big Farm update that is coming next week.

The biggest content additions this week are corn farming, a straw hat, and a high-capacity source of sewing thread from wool.

But the underlying resource systems have been given a complete overhaul.

Water sources now have a fixed, low refill rate, and emptying any source to the bottom is inconsequential, but higher and higher tech water sources have larger and larger capacity.  With a deep well and a bucket, water can now be moved 10 units at a time.  Moving large quantities of water around is way more efficient, if you reach that level of technology.  Wells must be built on top of natural water sources, making cisterns much more useful for on-site water storage far away from a well site.  Wells now have a visual indication of how full they are, so you know when you need to start building a new one.

Soil is now a necessary, consumed ingredient for all farming activities, which means it's easier to balance the value of various foods now and in the future (carrots no longer have a huge leg up, because they consume soil too).  This has been balanced by making natural soil much more plentiful, and compost much more productive.  Tilling is now needed to work soil into the ground instead of to re-work a hardened carrot row.

These changes pave the way for even more advanced water sources and soil fertility technology in the distant future, but also for farming of other, balanced food types in the immediate future.

As promised, blue roses are in there too.

#125 News » Update: Families and Funerals » 2019-12-20 01:20:17

Kaos
Replies: 0

tqn7yIW.gif

Four new characters, for a total of six in each skin tone now.  The family code has been fixed so that your kids and grand kids are more likely to look like you.

Graves have had mouse-over names for a while now, but this update pushes further with labeled grave stones that last forever.  And of course, you can't have grave stones without roses.  But it turns out that roses are a bit of a pain to grow from seed.  Welcome to the world of seed stratification, folks.

If you see a white rose in-game, you're looking at something pretty special.  Blue roses are coming in the next update.

Board footer

Powered by FluxBB