This document helps you troubleshoot a Countly instance, especially when you have a problem with Linux instance, MongoDB, processes, or Countly application service itself.

Countly Memory Management in General

• As a general rule of thumb, you should do your planning based on 5M data points per core, and 4 GB of RAM for each core for standalone (Countly + MongoDB) servers. Contact us if you want to use HA and replica sets and we'll be happy to provide you with a server topology.

• Countly forks a child process for each available core in the system so a 6 core system will run 6 Countly API nodes. Each will use around 2 GB RAM max.

• MongoDB engine uses a mix of in RAM storage and filesystem. RAM is preferable to MongoDB, and to achieve that, MongoDB will try to store the "working set" (documents that were accessed recently) in memory. If you don't see Countly server using all the RAM, this is why it won't always use the entire available RAM in the system, it'll use what it needs at that time.

• If you have page fault monitoring in your instance, you'll see an increased number of page faults if RAM is not enough for MongoDB to store the working set in memory - which means MongoDB is doing file operations which will be drastically slow.

Checking Disk Space

df -kh

Checking Resource Usage

#overall
vmstat 1

#memory
free -m

#by process
top

Checking if Process Was Killed by System

Sometimes you can't find, in the logs, why process was killed, as it just exited. You can check here if process was killed because, for example, server ran out of memory.

dmesg | tail

#or

tail /var/log/kern.log -n 1000

#or 

tail /var/log/syslog -n 1000

Getting Countly Version

countly version

Or if command line does not work

#run inside countly dir
cat package.json

Getting Countly Installation Directory

countly dir

or switching to Countly directory

cd `countly dir`

if Countly command line does not work

#if running on systemd
cat /etc/systemd/system/countly.service

#if running on upstart
cat /etc/init/countly-supervisor.conf

Enabling Countly Command Line if it's Not Available

#run inside countly dir
bash bin/scripts/detect.init.sh

Checking if Countly is Running

countly status

Checking Countly Logs

#run inside countly dir

#get latest 500 lines of api log
tail log/countly-api.log -n 500

#get latest 500 lines of dashboard log
tail log/countly-dashboard.log -n 500

Restarting Countly

#simply restart process
countly restart

#regenerate production files, install mising dependencies and restart process
countly upgrade

Checking and removing ghost processes

In rare cases, if a Countly process is stuck and you may want to kill and restart Countly.

# check current processes (should contain node or nodejs processes for countly)
top

#stop countly
countly stop

#recheck if still has node or nodejs processes
top

#if has them, kill them
pkill node

#start countly
countly start

Checking Countly HTTP Access Without nginx

#checking if dashboard process runs (should return webpage)
wget -O- 127.0.0.1:6001

#checking if api process runs (should return 400 Bad request)
wget -O- 127.0.0.1:3001/i

Checking Port Connections

#checking amount of connections to mongodb (assuming default port 27017)
sudo netstat -nputw | grep 27017 | wc -l

Checking Connection to Database

#connects to db, gets document with configurations and outputs them
countly config list values

Increasing Countly Limits

Limits for the process are managed by init system, so you need to change values in their config files which are

#if running on systemd
cat /etc/systemd/system/countly.service

#if running on upstart
cat /etc/init/countly-supervisor.conf

Adding a User to Dashboard

#adds global admin from command line
countly add_user <username> <password>

#and then when not needed anymore to remove
countly remove_user <username> <password>

Clearing Failed Login Time

#connect to mongo
mongo

#or use required parameters based on your mongodb setup

use countly
db.failed_logins.remove({_id:"<username>"})

Checking MongoDB Version

mongod --version

Checking if mongod Process is Running

It should also reveal path to config file.

ps -ax | grep mongo

Starting mongod Process

#starting MongoDB

#systemd
systemctl start mongod

#upstart
start mongod

#centos/rhel 6
service mongod start

Restarting MongoDB Process

#restarting MongoDB on

#systemd
systemctl restart mongod

#upstart
restart mongod

#centos/rhel 6
service mongod restart

Checking MongoDB Logs

#default log path is /var/log/mongodb/mongod.log or check in mongod.conf file for systemLog.path
#config file is usually at/etc/mongod.conf or check if process running to get path
tail /var/log/mongodb/mongod.log -n 500

Connecting to MongoDB

#In simple case if mongodb is on the same server its just
mongo `countly mongo`

#if port is changed to for example 111222
mongo --port 111222

#if it is on another server, for example 192.168.3.100
mongo --host 192.168.3.100

#if it is behind authentication
mongo --username <username> --password <password> --authenticationDatabase <dbname>

#if it is through ssl
mongo --ssl --sslAllowInvalidCertificates

#if it is replica set
mongo --host <replSetName>/<hostname1><:port>,<hostname2><:port>,<...>

Checking How Countly Connects to MongoDB

#run on countly server inside countly directory
cat api/config.js

#or establish mongo shell connection (requires mongodb tools on Countly server) 
#using Countly connection data

mongo `countly mongo` 

Checking Current/Allowed Connections

#connect to mongodb
mongo

#or some other connect approach depending on your mongodb configuration

#output connections info
db.serverStatus().connections

 Check how many connections are listening to a specific port

ss -a | grep ESTAB | grep 65017 | sed 's/   / /g' | sed 's/   / /g' | cut -d' ' -f 11 | cut -d':' -f1 | sort | uniq -c

Checking Current Reads and Writes

mongotop

#or use correct params (from connecting to mongodb) based on your mongodb configuration
#press Ctrl+C to exit

If reads and writes are quite long (longer than 10 seconds), your MongoDB instance might be over-utilized and need to be scaled.

To check which reads or writes exactly are the longest (adjust running seconds based on the longest time in mongotop)

mongo
db.currentOp({"secs_running": {$gte: 10}})

Checking Currently Used Resources

mongostat

#or use correct params (from connecting to mongodb) based on your mongodb configuration
#press Ctrl+C to exit

If, for example, there are lots of memory faults, it would mean the memory is being swapped and MongoDB would require more RAM to speed up queries.

Checking Replica Set Status

#connect to mongodb
mongo

#or use some correct parameters depending on your mongodb configuration

#output replica set status
rs.status()

#check how much secondary is behind the primary
rs.printSlaveReplicationInfo()

#check oplog window (in which secondary must catch up with primary)
rs.printReplicationInfo()

Checking Process Limits

First, you need to create script that checks the limits

#!/bin/bash
if [ "$#" -ne "1" ]; then
   echo ""
   echo -e "\033[01;32mLimit checker\033[00m"
   echo -e "\033[01;37mUsage:\033[01;33m $0 process_name\033[00m"
   echo ""
   exit 0
fi

return-limits(){
   for process in $@; do
      process_pids=`ps -C $process -o pid --no-headers | cut -d " " -f 2`

      if [ -z $@ ]; then
         echo "[no $process running]"
      else
         for pid in $process_pids; do
            echo "[$process #$pid -- limits]"
            cat /proc/$pid/limits
      done
      fi
   done
}

return-limits $1

Then you can use it as

mongoserver041 ~ $ ./limits.sh mongod
[mongod #43447 -- limits]
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            10485760             unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             32000                32000                processes
Max open files            1000000              1000000              files
Max locked memory         65536                unlimited            bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       125029               125029               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

Increasing MongoDB Limits

To maximize system limits for MongoDB, we suggest two-step configuration.

1. For system-wide limits, /etc/sysctl.conf

fs.file-max = 392000
kernel.pid_max = 256000
kernel.threads-max = 256000
vm.max_map_count = 512000

2. For the user of MongoDB service limits, /etc/security/limits.conf

{MONGODB_USER} soft nproc 256000
{MONGODB_USER} hard nproc 256000
{MONGODB_USER} soft nofile 392000
{MONGODB_USER} hard nofile 392000

To get MongoDB user's name:

grep mongod /etc/passwd | awk -F':' '{print $1}'

Re-indexing Data

In some cases (due to data transfers or concurrency) some index may be missing. If you think this is the case, you can run reindexing scripts, which will add all missing indexes. It is completely safe to run and rerun it multiple times

#go to countly directory
cd `countly dir`

#run script to add missing core indexes
node bin/scripts/add_indexes.js

#if you are on enterprise edition, you should also reindex granualr data
node plugins/drill/update_index.js 

This script runs in serial with least effect on performance, but note that on large data, this process may take multiple hours. To check the progress of the index that is currently added, you can run this command:

mongo 
db.currentOp(true).inprog.forEach(function(op){ if(op.msg) print(op.msg) }) 

Debugging and Troubleshooting Incoming SDK Requests

If you would like to understand why data is not arriving to Countly server, or why you see partial data, check this FAQ item to troubleshoot further.

Checking if 3rd Party Email Service Functions Correctly

1. Got to Management > Email reports and create a new report.
2. Press Preview button to make sure there is data to report and all is showing correctly.
3. Optionally, clear the logs in Countly, specifically the countly/log/countly-api.log file.
4. Press send now to send report to your provided email address.
5. If no error in the log, it usually means e-mail has left the server. It is not possible for us to track it further.
6. Check if the e-mail has arrived. There may be cues in your 3rd party service about this particular transaction.