First Find the System.Properties file. Mine is located at /usr/lib/unifi/data/

The majority of this data is from: https://help.ui.com/hc/en-us/articles/115005159588-UniFi-How-to-Tune-the-Controller-for-High-Number-of-UniFi-Devices

UniFi - How to Tune the Network Application for High Number of UniFi Devices

Overview

This article will help users diagnose a UniFi Network application running under a high load, as well as describe modifications that can be done for optimal performance.

NOTES & REQUIREMENTS:

  • Applicable to the latest UniFi Network release versions.
  • This article describes advanced configuration options and should only be attempted by advanced users.
  • See the UniFi - Explaining the system.properties File article for more information on the system.properties file.
  • Create a backup using the UniFi - How to Create and Restore a Backup help center article and export/save your backup configuration to a separate device before following this article.

Introduction

There are a few things to take into consideration when managing large UniFi installs with hundreds of devices and several sites connected to a single UniFi Network application. A Network application running under such a high load might encounter some issues if configured the same way as an application managing a much lighter load. There are some symptoms that can be recognized, diagnosed and worked on to improve the application’s performance.

The sections below show how to modify the UniFi Network system settings by altering or adding values to the system.properties file. See the UniFi - Explaining the system.properties File article for more information on this file.

WARNING: Before proceeding with the configurations outlined below, make sure to create a backup of your machine and application configurations. A typo error could break the system. See the UniFi - How to Create and Restore a Backup article for more information.

Symptom: High CPU Usage

One of the most important metrics to monitor, is CPU usage on your UniFi OS Console (or other host if self-hosting the UniFi Network application). High CPU usage is the first indication that there is an issue. Unfortunately, there is no silver bullet for this and just increasing the CPU is not necessarily the answer.

Allocating Additional Memory

Before increasing the RAM allocation on your machine, first try increasing the `XMX` and `XMS` options. By default, the UniFi Network application has these set to 1GB. The following lines set the xmx and xms to values to 2GB (2048MB):

unifi.xmx=2048
unifi.xms=2048

The changes above would increase the memory the UniFi Network application is allowed to consume from 1 to 2GB. The reason why not having enough memory could cause high CPU usage, is because the Java Virtual Machine would be spending too many CPU cycles collecting garbage to stay within the 1 GB of memory allocated to the application. So before moving to a machine with more CPU resources, it is recommended to max out the available memory on that machine with the above settings. Then observe if CPU usage decreases. If 2GB is not sufficient, administrators may want to raise the limit to 4-8GB depending on the scale. In this case it would simply be increasing the value above in increments of 1024, i.e. 4 GB= 4096.

NOTE: The jstat -gcutil Java command can be used to check whether the memory allocation is sufficient on your machine. See the Oracle documentation for more information.

Increase Mongo WiredTiger engine cache

If you already increased the UniFi Network application memory settings to at least 4GB (xmx), you may need to also change the default Mongo WiredTiger engine cache. By default the UniFi Network Application uses:

db.mongo.wt.cache_size=256

On UniFi Network version 6.5.13 and above you can change this setting or let Mongo pick the default value by using:

db.mongo.wt.cache_size_default=true

For more information please refer to Mongo documentation.

Enabling High Performance Java Garbage Collector

If increasing memory does not solve the problem, administrators may consider also adding this line to the system.properties file (note this only applies to applications not hosted on Cloud Keys):

unifi.G1GC.enabled=true

This enables a new Java Garbage Collector that can help optimize performance. However, if after these changes high CPU usage does continue after memory increase, a larger machine with more CPU cores and more memory to handle the workload may be necessary.

Changing Mongo Version/Engine
In persistent cases, administrators may consider updating MongoDB version to 3.2+ with WiredTiger as storage engine, as a means of better scaling their UniFi Network deployment. See the following links for instructions:

Symptom: Heartbeat Missed or Slow to Provision

No matter the number, all devices will try to inform back to the Network application. By default, the application can handle 200 simultaneous connections from devices, so devices with heartbeats missed shouldn’t be an issue unless a single application is managing thousands of devices. If it is only managing a few hundred devices the adjustment described in this section can be tried out, but it may not have the desired results. The number of simultaneous inform messages that can be processed can be set in the system.properties file by adding the following lines and adjusting the values to best suit your needs. It does not matter where these lines are added in the file.

inform.num_thread=200
inform.max_keep_alive_requests=100

The default value is 200 and the max_keep_alive_requests value should always be lower than than num_thread. Try adjusting up from there. An increase in device stability should be seen, and by pushing the configuration out to other devices, become even more stable.

Database Connection Tuning

When running a large UniFi installation, it may be desired to run an external Mongo cluster to be able to scale the database independently from the UniFi Network application. Discussion on that can be found here on our Community Beta forum (need access?). If high CPU usage is seen on the Mongo process, it can indicate the need of a bigger box or the need to separate the mongodb process as mentioned above. Once that is done, the following can be tuned to see if it results in better application performance:

db.mongo.connections_per_host=100db.mongo.threads_multiplier=5

This results in 500 threads that can be waiting for a Mongo connection. So keep in mind that more threads can mean higher CPU usage because the CPU has to context-switch between threads. It may allow for higher DB throughput, but only if the Mongo process is able to consume more CPU to serve requests faster.

Conclusions

The best way to increase performance and provide stability for large installations is to ensure you have monitoring on your system resources. Beyond that, practices such as offloading the database workload, and increasing memory can allow UniFi Network to serve more clients and devices. This should be reflected in the resource usage and the performance of the UniFi Network application.

UniFi - How to Create and Restore a Backup

UniFi - Explaining the system.properties File

UniFi - Configuring Multiple Sites