This document helps you troubleshoot a Countly instance, especially when you have a problem with Linux instance, MongoDB, processes, or Countly application service itself.
Countly Memory Management in General
- As a general rule of thumb, you should do your planning based on 5M data points per core, and 4 GB of RAM for each core for standalone (Countly + MongoDB) servers. Contact us if you want to use HA and replica sets and we'll be happy to provide you with a server topology.
- Countly forks a child process for each available core in the system so a 6 core system will run 6 Countly API nodes. Each will use around 2 GB RAM max.
- MongoDB engine uses a mix of in RAM storage and filesystem. RAM is preferable to MongoDB, and to achieve that, MongoDB will try to store the "working set" (documents that were accessed recently) in memory. If you don't see Countly server using all the RAM, this is why it won't always use the entire available RAM in the system, it'll use what it needs at that time.
- If you have page fault monitoring in your instance, you'll see an increased number of page faults if RAM is not enough for MongoDB to store the working set in memory - which means MongoDB is doing file operations which will be drastically slow.
Checking Disk Space
df -kh
Checking Resource Usage
#overall vmstat 1 #memory free -m #by process top
Checking if Process Was Killed by System
Sometimes you can't find, in the logs, why process was killed, as it just exited. You can check here if process was killed because, for example, server ran out of memory.
dmesg | tail #or tail /var/log/kern.log -n 1000 #or tail /var/log/syslog -n 1000
Getting Countly Version
countly version
Or if command line does not work
#run inside countly dir cat package.json
Getting Countly Installation Directory
countly dir
or switching to Countly directory
cd `countly dir`
if Countly command line does not work
#if running on systemd cat /etc/systemd/system/countly.service #if running on upstart cat /etc/init/countly-supervisor.conf
Enabling Countly Command Line if it's Not Available
#run inside countly dir bash bin/scripts/detect.init.sh
Checking if Countly is Running
countly status
Checking Countly Logs
#run inside countly dir #get latest 500 lines of api log tail log/countly-api.log -n 500 #get latest 500 lines of dashboard log tail log/countly-dashboard.log -n 500
Restarting Countly
#simply restart process countly restart #regenerate production files, install mising dependencies and restart process countly upgrade
Checking and removing ghost processes
In rare cases, if a Countly process is stuck and you may want to kill and restart Countly.
# check current processes (should contain node or nodejs processes for countly) top #stop countly countly stop #recheck if still has node or nodejs processes top #if has them, kill them pkill node #start countly countly start
Checking Countly HTTP Access Without nginx
#checking if dashboard process runs (should return webpage) wget -O- 127.0.0.1:6001 #checking if api process runs (should return 400 Bad request) wget -O- 127.0.0.1:3001/i
Checking Port Connections
#checking amount of connections to mongodb (assuming default port 27017) sudo netstat -nputw | grep 27017 | wc -l
Checking Connection to Database
#connects to db, gets document with configurations and outputs them countly config list values
Increasing Countly Limits
Limits for the process are managed by init system, so you need to change values in their config files which are
#if running on systemd cat /etc/systemd/system/countly.service #if running on upstart cat /etc/init/countly-supervisor.conf
Adding a User to Dashboard
#adds global admin from command line countly add_user <username> <password> #and then when not needed anymore to remove countly remove_user <username> <password>
Clearing Failed Login Time
#connect to mongo
mongo
#or use required parameters based on your mongodb setup
use countly
db.failed_logins.remove({_id:"<username>"}Checking MongoDB Version
mongod --version
Checking if mongod Process is Running
It should also reveal path to config file.
ps -ax | grep mongo
Starting mongod Process
#starting MongoDB #systemd systemctl start mongod #upstart start mongod #centos/rhel 6 service mongod start
Restarting MongoDB Process
#restarting MongoDB on #systemd systemctl restart mongod #upstart restart mongod #centos/rhel 6 service mongod restart
Checking MongoDB Logs
#default log path is /var/log/mongodb/mongod.log or check in mongod.conf file for systemLog.path #config file is usually at/etc/mongod.conf or check if process running to get path tail /var/log/mongodb/mongod.log -n 500
Connecting to MongoDB
#In simple case if mongodb is on the same server its just mongo `countly mongo` #if port is changed to for example 111222 mongo --port 111222 #if it is on another server, for example 192.168.3.100 mongo --host 192.168.3.100 #if it is behind authentication mongo --username <username> --password <password> --authenticationDatabase <dbname> #if it is through ssl mongo --ssl --sslAllowInvalidCertificates #if it is replica set mongo --host <replSetName>/<hostname1><:port>,<hostname2><:port>,<...>
Checking How Countly Connects to MongoDB
#run on countly server inside countly directory cat api/config.js #or establish mongo shell connection (requires mongodb tools on Countly server) #using Countly connection data mongo `countly mongo`
Checking Current/Allowed Connections
#connect to mongodb mongo #or some other connect approach depending on your mongodb configuration #output connections info db.serverStatus().connections
Check how many connections are listening to a specific port
ss -a | grep ESTAB | grep 65017 | sed 's/ / /g' | sed 's/ / /g' | cut -d' ' -f 11 | cut -d':' -f1 | sort | uniq -c
Checking Current Reads and Writes
mongotop #or use correct params (from connecting to mongodb) based on your mongodb configuration #press Ctrl+C to exit
If reads and writes are quite long (longer than 10 seconds), your MongoDB instance might be over-utilized and need to be scaled.
To check which reads or writes exactly are the longest (adjust running seconds based on the longest time in mongotop)
mongo
db.currentOp({"secs_running": {$gte: 10}})Checking Currently Used Resources
mongostat #or use correct params (from connecting to mongodb) based on your mongodb configuration #press Ctrl+C to exit
If, for example, there are lots of memory faults, it would mean the memory is being swapped and MongoDB would require more RAM to speed up queries.
Checking Replica Set Status
#connect to mongodb mongo #or use some correct parameters depending on your mongodb configuration #output replica set status rs.status() #check how much secondary is behind the primary rs.printSlaveReplicationInfo() #check oplog window (in which secondary must catch up with primary) rs.printReplicationInfo()
Checking Process Limits
First, you need to create script that checks the limits
#!/bin/bash
if [ "$#" -ne "1" ]; then
echo ""
echo -e "\033[01;32mLimit checker\033[00m"
echo -e "\033[01;37mUsage:\033[01;33m $0 process_name\033[00m"
echo ""
exit 0
fi
return-limits(){
for process in $@; do
process_pids=`ps -C $process -o pid --no-headers | cut -d " " -f 2`
if [ -z $@ ]; then
echo "[no $process running]"
else
for pid in $process_pids; do
echo "[$process #$pid -- limits]"
cat /proc/$pid/limits
done
fi
done
}
return-limits $1Then you can use it as
mongoserver041 ~ $ ./limits.sh mongod [mongod #43447 -- limits] Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 10485760 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 32000 32000 processes Max open files 1000000 1000000 files Max locked memory 65536 unlimited bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 125029 125029 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us
Increasing MongoDB Limits
To maximize system limits for MongoDB, we suggest two-step configuration.
1. For system-wide limits, /etc/sysctl.conf
fs.file-max = 392000 kernel.pid_max = 256000 kernel.threads-max = 256000 vm.max_map_count = 512000
2. For the user of MongoDB service limits, /etc/security/limits.conf
{MONGODB_USER} soft nproc 256000
{MONGODB_USER} hard nproc 256000
{MONGODB_USER} soft nofile 392000
{MONGODB_USER} hard nofile 392000To get MongoDB user's name:
grep mongod /etc/passwd | awk -F':' '{print $1}'Re-indexing Data
In some cases (due to data transfers or concurrency) some index may be missing. If you think this is the case, you can run reindexing scripts, which will add all missing indexes. It is completely safe to run and rerun it multiple times
#go to countly directory cd `countly dir` #run script to add missing core indexes node bin/scripts/add_indexes.js #if you are on enterprise edition, you should also reindex granualr data node plugins/drill/update_index.js
This script runs in serial with least effect on performance, but note that on large data, this process may take multiple hours. To check the progress of the index that is currently added, you can run this command:
mongo
db.currentOp(true).inprog.forEach(function(op){ if(op.msg) print(op.msg) }) Debugging and Troubleshooting Incoming SDK Requests
If you would like to understand why data is not arriving to Countly server, or why you see partial data, check this FAQ item to troubleshoot further.
Checking if 3rd Party Email Service Functions Correctly
- Got to Management > Email reports and create a new report.
- Press Preview button to make sure there is data to report and all is showing correctly.
- Optionally, clear the logs in Countly, specifically the countly/log/countly-api.log file.
- Press send now to send report to your provided email address.
- If no error in the log, it usually means e-mail has left the server. It is not possible for us to track it further.
- Check if the e-mail has arrived. There may be cues in your 3rd party service about this particular transaction.