Keeping up with “Can't keep up!” – An Overloaded Guide to Java Vanilla Server Tick-Per-Second Lag-Busting -or- How I Learned To Stop Worrying and Love /debug
Keeping up with “Can't keep up!” – An Overloaded Guide to Java Vanilla Server Tick-Per-Second Lag-Busting
We've all been there. The game lags, and despite our best efforts the reason seems to remain an enigma. Well, let's try to drill to the bottom of it.
Full disclaimer: this guide is not 100% comprehensive; this is just a compilation of everything I have learned while trying to lag-bust the server I run. I personally run a vanilla Minecraft 1.12.2 server on a Windows 7×64 machine with Java 8.161. I'm going to start right off the bat by saying: this guide ignores Spigot. This is a vanilla guide. Spigot can hugely help performance, but it also fundamentally changes the way some things (like hoppers) work, so for people who do not want to go modded at all, this is the guide for them. All of this information theoretically can also apply to Spigot servers, but it was tested with vanilla in mind and I have no idea if it actually applies to Spigot servers too.
There are three different main sources that “lag” can come from.
If the world is a single-player world, there is no network lag, but Minecraft does run single-player worlds as an internal server on the machine you are playing on, so the server/client distinction does remain even in single-player worlds.
To dramatically oversimplify network and client lag:
Network lag looks like frequently timing out, a red ping meter if you press tab, and in-game rubberbanding. Online speedtests can sometimes help diagnose connection issues. Weak wifi signal, NAT acceleration, and high load on routers can cause network lag or disconnection issues.
Client lag looks like frames-per-second issues. FPS can be seen in-game if you press F3 – left side, second row from the top. Client lag can be helped by changing video settings, installing Optifine, making sure video card drivers are up-to-date, making sure the version of java you are running matches the server's java version, and making sure there are sufficient resources available on the client machine to run the game (for example, close other programs and set antiviruses to gaming mode if they have it). Shift+F3 in-game brings up a pie chart that can help you identify in-game sources of client lag. The rendering of lots of regular entities, tile entities, or lighting updates can cause client lag (those things existing/happening also contribute to server lag).
But this guide is focused on the third kind of lag: server lag. Server lag manifests as ticks-per-second lag.
The game wants to run at 20 game ticks per second. If it is unable to complete all the things it wants to do in that 1/20 of a second, the tickrate begins to fall. Once the server runs more than 2000 ms behind, a message appears in the server console: “CONSOLE: thread/WARN>: Can't keep up! Did the system time change, or is the server overloaded? Running <####>ms behind, skipping <##> tick(s)”. If a single tick ever takes a full minute, the game will crash with a server watchdog fatal error. It is not a good idea to ignore server lag once the warning signs appear. It is fairly easy to end up with runaway tick loss, partially due to bug: https://bugs.mojang.com/browse/MC-121196
In-game, your players might notice that consuming food/potions takes a bit longer than it should, the sun and moon jerk in the sky, and sometimes after breaking a block it seems to flash for a moment and then finally drop as an entity. Full day/night cycles take longer than IRL 20 minutes. And every player on the server is effected the same at the same time when this is happening, regardless of how good their connection is, or how good the client machines are.
There are several factors to look at here:
For server system spec recommendations, please look here: https://minecraft.gamepedia.com/Server/Requirements/Dedicated
System spec recommendations from Mojang for Minecraft CLIENTS can be found here: https://help.mojang.com/customer/en/portal/articles/325948-minecraft-system-requirements
There are three different hardware bottlenecks that can potentially cause server lag.
Hard drive speed (ROM)
I recommend running a server backend that allows you to monitor system resources while the server is running. Personally, I use MCMyAdmin; it is free and wonderful.
Minecraft will only ever use one CPU core, even if your machine has multiple cores. There is only a need for graphics processing if the machine in question is also running a client (i.e. single-player).
For maximum server performance, you will want to make sure that there is not too much stress on the CPU, which can be helped somewhat by dealing with other processes running on the machine. If you have an antivirus with a gaming-mode, enable that mode. If you are running Windows 7, you might want to disable Aero or GUI animations (Control Panel > System > Advanced system settings > Advanced > Performance > Settings > choose 'best performance'). Processes like Apple Software Update and Windows Update can cause a huge performance hit, especially if they've installed updates are are waiting for the system to reboot to complete installation.
It is a very common instinct to just allocate more memory to a server that is struggling to keep up, but you should be aware that by allocating more memory, you are also making the heap larger, meaning the server has to work harder (with its CPU) to do its garbage collection process. It's a bit of a trade-off. If you have an excellent processor, and you implement G1GC (see 'server configuration' section below), you can probably safely allocate 6gb or more of memory if your machine has it available. Do be aware that if you are running a 32-bit system, the maximum amount of memory you will be able to allocate to Minecraft is 1GB.
The game auto-saves once every 45 seconds, which can cause a hard drive IO bottleneck if the drive is under a lot of load. You will want to make sure that there are not any processes running on the machine that scan or index the Minecraft server files in real time (like a backup service, antivirus, or Windows search indexing), so you'll want to add exceptions to those programs for the folders that contain the Minecraft world. I would recommend running a Minecraft server on a separate drive from other stuff on the machine. It is even better if you can make that drive be an SSD or RAMDISK. If (and ONLY if) you are running on a HDD (NOT an SSD), you may want to periodically defragment the drive to help improve disk IO performance. Do not defragment SSDs.
Before you even launch your server, you will want the Server JRE, which does not come standard with a normal installation of java or the server jar. Go here: http://www.oracle.com/technetwork/java/javase/downloads/server-jre8-downloads-2133154.html and then make sure your server knows to use this java.exe to run the minecraft server. (With MCMyAdmin, this is easy: the MCMyAdmin.conf file has an option that tells the server backend the exact filepath of the java version to use).
Next you will want to think about startup options and Java arguments. There is a lot of confusing information out there, with old deprecated arguments from Java 6 and 7. I recommend Googling any argument you wish to use before implementing it, just to make sure it still exists in the version of Java you are using. For Java 8, I recommend the following arguments:
-serverThis makes sure the server is using the server virtual machine. It helps performance.
-Xms512M -Xmx2048MWhat those do is they allocate the minumum amount of memory the server can use (Xms) to half a GB, and the maximum amount of memory the server can use (Xmx) to 2GB. I would NOT recommend setting those two numbers to be equal to each other. A higher Xms can help reduce start-up lag, if that is a problem, but if startup lag is not a big problem, you can probably leave Xms fairly low. Feel free to adjust those numbers to suit your server's needs and capabilities.
-XX:+UseG1GCThis sets the garbage collector to be garbage-first collection, which is designed to have minimal delays even with large heap sizes. This is the garbage-collector most recommended for Minecraft servers, especially ones with a lot of RAM allocated.
noguiThe server GUI console that appears when you start a server normally causes a LOT of strain on the server's resources. Adding “nogui” (without the quotes) to the end of your startup arguments will make the server start without that laggy console gui. (If you're using MCMyAdmin, nogui is configured slightly differently, and you will still be able to access the server console through the server backend web-browser-interface).
Another configuration option that may come up as an idea to reduce lag is reducing the server render distance. I do NOT recommend lowering the render distance below 10 chunks, due to this bug: https://bugs.mojang.com/browse/MC-2536. Mob spawning will grind to a halt if you turn the render distance lower than 10. If you do not care about mob spawning, then the lowest I would recommend is 6, due to potential issues in sky rendering if you go lower than that.
IN-GAME CAUSES OF TPS LAG
In-game, there are a bunch of different things that can contribute to server load. If you're going on a lag-busting binge, here are some things to keep in mind:
Repeating command blocks that are set to store their most recent output packets cause more lag than ones that are not set to store their most recent output.
If you have more than 64 block updates in a single chunk in a single tick, the game sends the whole chunk to be updated. Don't put a lot of fill-clocks in the same chunk.
Lighting updates are incredibly laggy. Try to make sure your redstone circuits with pistons, rs torches, repeaters, and comparators are all well lit to reduce these lighting updates from happening in your circuits. Don't have a lot of unecessary flashing lamps, either.
Large numbers of fluid updates can be laggy while they are happening.
Entities, especially ones with AI, are probably the single laggiest thing in the game, and the single-most-common cause of massive server lag that most people experience. To see how many entities there are in an area, press F3 in-game and look at the fourth line, left-hand side: “E: #/#”. The first number is the number of entities within your field of view, and that includes seeing through walls. The second number is the total number of entities loaded around you. Entities include: mobs, minecarts, boats, item frames, players, items, xp orbs, etc. – basically anything that is not a block or tile entity. Large numbers of entities WILL cause the game to lag.
Perhaps at-first counterintuitively, being in a well-conditioned area causes more resources per tick to be spent on the mob spawning algorithm than would happen in a less-well-conditioned area. If the mob cap is not reached, then the mob spawning algorithm keeps making attempts to spawn mobs in each chunk, as it keeps failing over and over again. However, do keep in mind that the resources used by this process is significantly less than the resources the mob entities themselves would use if you were at the mob cap.
The act of generating new terrain is incredibly stressful on the server, especially if the player is moving quickly (like flying with elytra). Some administrators like to pre-generate a large amount of terrain before opening the server to members, or while all members are offline, in order to reduce the stress on the server while people are trying to play.
ARGH THERE IS STILL SERVER LAG WHAT DO
Finally, we've made sure everything is running as smoothly as it can, but we're still experiencing enigmatic in-game TPS lag. Let me now formally introduce you to your new best friend: the /debug command. You don't need carpetmod for /tick health, as there is seriously powerful functionality available in vanilla! /debug is this game's best-kept secret IMHO.
You can start a debug session by running “/debug start” (without the quotes) and then stop a debug session by running “/debug stop” (without the quotes).
If you are looking at the server console when it is running, you will be able to see messages if something takes too long: “
I recommend running debug sessions for at least one minute (to make sure it gets at least one autosave). Running a debug session can itself cause some lag, so don't leave it running all the time. You can run around, loading different dimensions or different people's bases, to get an idea of the TPS performance in each area, as there may be something in a particular location causing server lag.
The true power of /debug comes in the profiler log it makes. Each time /debug is run it makes a new profiler log file. In order to view the log you will need access to the server files after running a debug session. Debug profiler logs are saved in a folder called “debug” in the root of the folder containing your Minecraft save (the “debug” folder is found in the same place as the crash reports folder, the whitelist file, the “world” folder, the mods folder, the plugins folder, etc.). The debug profiler logs are formatted carefully, so open them in a program like Notepad++ that maintains the formatting (Notepad won't cut it). Each log beautifully details out how much percentage of each tick is taken up by various parts of the game.
From /u/mynameisperl's comment: https://wtbblue.com/r/Minecraft/comments/3xea29/need_help_with_debug_command/cy3y7zl/?utm_content=permalink&utm_medium=front&utm_source=wtbblue.com&utm_name=Minecraft
Each row shows the proportion of the total time spent on a particular game activity. The first number in square brackets is the depth of the tree displayed at that point, starting with <00> at the top level. All rows with <00> are at the same depth and the percentage of their activities will sum to 100% – that's the first percentage in the row after the '-'. Under each <00> subtree, the rows beginning <01> are also at the same level as each other, and their percentages sum to 100% of their parent's time. The second percentage, after the '/', is the time taken in the activity as a proportion of the whole profile.
I can't understand what everything is completely in this log, and I haven't been able to find a complete guide to what everything is, but some things are super obvious, like regular entities with AI, ticking tile entites (things like hoppers, furnaces, etc – called blockEntities in the log, broken down by type to show performance impact), commandFunction (which includes gameLoopFunction impact), etc. You can use this to help pinpoint the things that eat up server resources the most.
Read more: wow towers of certain doom
I hope this guide helps people. I'm sorry, I can't be available to help people troubleshoot their individual problems, so please don't beg me for help in the comments with your individual issues. This guide is meant as a place for you to start to help yourself.
If I have missed anything or gotten anything wrong, please let me know and/or correct me in the comments!