This article serves as a comprehensive guide for customers experiencing issues with Countly. It provides a detailed breakdown of common problems and offers probable solutions to help you resolve them efficiently.

Types of issues 

Below is the list of issues that can occur:

1. Countly Dashboard issues.
2. Issues related to SDK integration.
3. Infrastructure issues.

Countly Dashboard Issues

These are the types of issues that can occur on the Countly dashboard you are accessing. These issues may arise from multiple layers of the system, including the user interface and the foundational architecture supporting its operations. It’s important to be aware of these potential challenges so that you can navigate through the dashboard effectively and report any issues for resolution. These issues can be further classified into two main categories:

• Frontend Issues.
• Backend Issues.

Frontend Issues

Frontend issues refer to problems or challenges that arise with Countly’s user interface (UI) and user experience (UX). Frontend issues can include but are not limited to:

1. Cross-browser compatibility issues: The Countly dashboard is tested on latest Google chrome versions and it should work properly on chrome. However, it may work perfectly in one browser but, may have issues in another, which can be referred to as cross-browser compatibility issues.
2. Responsive design issues: The software may not be optimized for different screen sizes, making UI elements appear off-center or difficult to read.

Seek Solutions: Frontend Issues

When there are issues in the Frontend, you are requested to provide the support team with the following:

1. Screenshots/Screen Recording of the issue: It is important to see the issue itself, but please be careful and do not expose any sensitive information.
2. URL address: It is important to share the URL address as it contains information about which section view is used and information about the query being made

Distinguishing Frontend from Browser-side Issues

To differentiate between frontend and browser-side issues, you can follow these steps:

1. Reproduce the issue: Try to replicate the problem by performing the same actions that led to the issue.
2. Check the browser console: Open the browser console and check for any error messages that might indicate the issue. Look for errors related to JavaScript, CSS, or HTML.
3. Check the network tab: Check the network tab to see if any resources are failing to load, such as CSS or JavaScript files.
4. Check cross-browser compatibility: Test the website or application in different browsers to see if the issue is specific to a particular browser.
5. Clear the cache: After upgrades, you can try to clear the cache or use incognito mode as some old version files may still be remain in the cache. 

If the issue is related to the browser, it will likely manifest in only one browser, while frontend issues are more general and affect the UI and UX of the application across all browsers.

For example, a frontend issue might be that a button on a web page is not responding when clicked, while a browser-side issue might be that the button is not visible in a particular browser or is not clickable due to a browser-specific issue.

By following these steps, you can differentiate between frontend and browser-side issues and work towards resolving them appropriately.

Browser-side Issues

Browser-side issues refer to problems or challenges arising from the specific browser you are using to access a software application. This issue occurs on the user side, or it can occur on the user’s machine.

These issues, in general, can include but are not limited to:

1. Rendering issues: The browser may not render the web page or application correctly, leading to elements being misaligned, overlapping, or not displayed at all.
2. Compatibility issues: The web page or application may not be compatible with the browser being used, leading to errors or unexpected behavior.
3. Performance issues: The browser may become slow or unresponsive due to the web page or application consuming too much memory or processing power.
4. Plugin issues: The browser may have issues with installed plugins or extensions, leading to conflicts or unexpected behavior.
5. Network issues: The browser may have trouble connecting to the internet or communicating with the server, leading to slow loading times or incomplete data retrieval.
6. Configuration issues: The browser may have settings or preferences that conflict with the web page or application, leading to unexpected behavior or errors.

These are just a few examples of browser-side issues that can arise when accessing a software application. Identifying and addressing these issues is essential to ensure that users have a positive experience with the application and can access its functionality without any hindrance.

User Interface issues

If the Countly Platform showcases any error, such as missing User Interface elements, User Interface sections, Menu items, etc., or if a particular view of the User Interface is not loading, then it can be considered as a User Interface issue.  

Operational issues from the User Interface side:

Operational issues related to User Interface can significantly impact the usability and functionality of the software. If there are functional issues, e.g., 

• It takes longer than expected to load a certain event or related data.
• The data is not being displayed correctly.
• Any input button is not working correctly.

Seek Solutions: Browser side Issues

When encountering these types of issues, kindly provide the following information to the support team:

Logs

Logs are essential for troubleshooting, monitoring, and gaining insights into software behavior. In the case of browser or frontend issues, it is recommended to collect the following logs for effective diagnosis.

Clearing cache

This is the first thing you should do when encountering a User Interface issue is to clear your browser cache or try using incognito mode.

To clear your browser’s cache, please follow the steps below:

1. Press Ctrl + F5  for Windows.
2. Press Cmd + Shift + R for Mac OS.

Console Logs (Chrome)

After clearing the browser cache, if the issue persists, clear the console logs and refresh the page. Check for the problem and copy all the texts from the console logs or the texts highlighted in red.

To get the Console logs, follow these steps:

1. Press Ctrl + Shift + J for Windows.
2. Press Opt + Cmd + J Mac OS.
3. You can also go to More tools > Developer tools.
4. Click on the Console tab and make sure all log levels are enabled.

Network Logs (Chrome)

When Countly fails to load some resource or perform any action with API/DB, you must collect the Network logs.

We will need the following to debug the issue:

1.  The Headers tab with the Status Code.
2.  The Payload tab.
3.  The Response tab

To get all of the above, click on each failed red request.

To access Network logs, follow these steps:

1. Press Ctrl + Shift + I for Windows.
2. Press Opt + Cmd + I for Mac OS.
3. You can also go to More tools > Developer tools.
4. Click on the Network tab.

Console logs (Safari)

In Safari Browser, console, and network logs are available as a part of "Develop Menu". By default, these options are disabled. To enable "Develop Menu",

1. Launch "Safari".
2. Go to Safari >> Preferences >> Advanced  (From the header).
3. Enable "Show Develop Menu in Menu Bar".

Post enabling "Develop Menu,” go to the desired webpage and click Develop >> Show Javascript Console.

Network logs(Safari)

Opening the Network Log

1. Open Safari JavaScript Console.
2. Switch to "Network Tab".
3. Refresh the webpage.
4. Errors are displayed in Red font. Clicking on the error would open the details area. You can find critical error information in the header and Preview tabs.

Console logs(Mozilla Firefox)

In Mozilla FireFox, the Console Logs are available as a part of FireFox developer tools.

To open the Web Console:

1. either select "Web Console" from the Web Developer submenu in the Firefox Menu (or Tools menu if you display the menu bar or are on Mac OS X)
2. or press the Ctrl+Shift+K (Command+Option+K on OS X) keyboard shortcut.

Network logs(Mozilla Firefox)

Opening the Network Monitor:

1. Press  Ctrl + Shift + E ( Command + Option + E on a Mac).

2. Select "Network" from the Web Developer menu, (which is a submenu of the Tools menu on OS X and Linux).

3. Click the wrench icon, which is in the main toolbar or under the Hamburger menu, then select "Network".

You can also try using incognito mode to ensure cache is cleared.

Countly Backend Issues - Node.js processes

Backend issues related to Node.js processes can include problems or challenges that arise in the backend or server-side of a Node.js application. Some examples of backend issues related to Node.js processes include:

1. Memory leaks: A memory leak can occur when a Node.js process allocates memory but fails to release it, leading to a gradual depletion of system resources and eventual application failure.
2. Blocking code: Node.js is built to be non-blocking, but blocking code can cause performance issues, leading to slow response times or even application crashes.
3. Database connection issues: If a Node.js process is unable to connect to the database, it will not be able to retrieve or store data, leading to errors or unexpected behavior.
4. File system issues: Node.js applications often interact with the file system to read or write data. Issues with the file system can cause data corruption or application failure.
5. Error handling issues: Node.js processes can encounter errors that may cause the application to crash or behave unexpectedly. Proper error handling is necessary to ensure that the application can recover from errors and continue to function as expected.
6. Scalability issues: Node.js processes are designed to be scalable, but issues can arise when attempting to scale an application, such as increased response times or server crashes.

This issue usually arises within the client's server environment. Understanding the nature of these issues is crucial for finding the right solutions. The following are the two specific issues: 

Functional issues

When some of the functionality on the Countly server is not working.

Performance issues

When a particular event is taking longer than expected, i.e., if a response to a query or any event is not loading any data and also taking longer time than expected.

Seek Solutions: Countly Backend Issues

For effective resolution of backend issues, comprehensive data is required, ranging from server type to server logs.

Deployment Scenario

In order to effectively address backend issues related to Countly, it’s crucial for our support-team to know the server instance that is being used by your application. Please share the details with us about the type of your instance, which can be from the following:

1. Standalone instance.
2. Replica set.
3. Sharded cluster.
4. Auto scale group.

Deployment type

Logs play an important role in debugging backend issues. To effectively utilize the logs for troubleshooting, we need to know about the Deployment type of Countly. It can be any of them from the following three options:

1. VM
2. Docker
3. Kubernetes

Web server type

Server-related problems can also lead to backend issues. To better understand and address the issue, our Support team needs details about your server and any associated load balancers. Although Nginx is the default load balancer for the Countly server, in cases where Docker or Kubernetes setup is employed, alternative web servers may be utilized.

Countly Version

Now, lastly you need to inform the support team about the version of Countly Server being used. This helps us assist you better by understanding your specific setup.

Server Logs

To ensure that bugs are resolved properly, the logs play a very crucial role. You can follow on, to learn more about how to collect logs in each scenario.

You can collect the API and Dashboard logs from the Countly server. To do so, go to Management > Logs. And there, you can select the logs that you want to collect.

Another way to collect logs is mentioned below.

API Logs

To collect the API logs, you can use the following commands.

1. countly/log/countly-api.log
2. Docker logs: docker logs <api pod name>
3. Kubernetes logs: kubectl logs <api pod name>

Dashboard Logs

If there are login and logout issues or dashboard loading, you can use the following commands to get the logs.

1. countly/log/countly-dashboard.log
2. Docker logs: docker logs <frontend pod name> 
3. Kubernetes logs: kubectl logs <frontend pod name>

Issues related to SDK integration

This issue occurs when the SDKs are not correctly integrated into your application. Sometimes this can happen due to platform changes or SDK updates in case of breaking changes. This can lead to various potential issues, such as:

• Data collection issues - When the data collected by the SDK does not appear in the Countly dashboard, or when it appears in a wrong/unexpected way. 
• Features not working as expected.

Validate the integration of SDKs

To ensure the seamless functioning of integrated SDKs, it is crucial to conduct thorough validation. The following steps verify that the SDK has been properly incorporated into your application.

Seek Solutions: SDK Integration

When you come across the previously described issues or have observed that the integration is failing the validation check, please share integration specific information with the support team.

Every SDK has its own configuration. Mis-configuration could lead to some subtle bugs or certain features not working. For effective debugging on the SDK side, the SDK team needs to know the full context of the integration and the circumstances under which the problem occurred. That includes the following information:

1. SDK Name (Android, Web, Flutter, Java, etc)
2. SDK Version (22.02.5 etc)
3. On what target platform was the problem observed? Some SDKs can be run on multiple platforms, therefore the differentiation is important.
4. In what kind of project was the SDK integrated? Some SDKs can be used in mobile apps, servers, and frontend applications. For cases like .NET apps, there can even be multiple ways to create apps in all app types. What specific framework/application project type are you using?
5. Is it possible to reproduce the issue, or does it appear rarely, or intermittently? If it is reproducible, could you describe the steps or example code needed to reproduce the problem?
6. What is the expected behavior, and what is the observer behavior?
7. SDK logs from the beginning of the SDK initialization till the observed event happens. For more information on where to find the logs, check here.

Depending on the bug or issue, some or all of these can be required:

1. The SDK initialization code.
2. Information on where the SDK was initialized.
3. Information on how the device ID is managed.
4. Information on what happens from your side after the user “logs in” and “logs out”.
5. If you have a custom class for the Countly integration, the code for that class will also be required.

Infrastructure issues

Infrastructure issues encompass a range of challenges in managing servers and databases.

Common concerns include performance constraints, security vulnerabilities, and resource management. Addressing these concerns requires a combination of proactive maintenance, monitoring, and adherence to best practices. This can be further categorized into two main categories:

• MongoDB Issue
• Countly Server Issue

MongoDB - Database Issues

MongoDB database issues can include problems or challenges arising in a MongoDB-powered application’s database layer. Some examples of MongoDB database issues include:

1. Connection issues: If the application is unable to connect to the MongoDB database, it will not be able to read or write data, leading to errors or unexpected behavior.
2. Performance issues: If the MongoDB database is not optimized, it can lead to slow query response times or other performance issues, such as high memory usage or slow disk I/O.
3. Data consistency issues: MongoDB databases can encounter data consistency issues, such as duplicate data or missing data, which can lead to unexpected behavior or application errors.
4. Security issues: MongoDB databases can have security vulnerabilities that can be exploited by attackers, such as unsecured databases or weak authentication and authorization mechanisms.
5. Indexing issues: If the MongoDB database is not indexed correctly, it can lead to slow query response times or even crashes.
6. Replication issues: If the MongoDB database is configured for replication, issues such as data inconsistencies or replication lag can arise when replicating data across multiple servers.
7. Sharding issues: If the MongoDB database is sharded, issues can arise when distributing data across multiple shards, such as imbalanced shards or slow query response times.

This issue occurs in the database of Countly, and hence, you will have to collect the logs and contact the Countly support team. The issue can be of two types:

Performance issues

When data collection/operations take longer than expected.

Data issues

When the event runs on your application, the data is sent from SDK. But there is no update in the Countly platform as it is not getting recorded on Countly server/database.

Seek Solutions: MongoDB Issues

Fixing MongoDB problems is a team effort. Besides technical details, effective communication and timely updates are vital. When encountering such issues, kindly provide the following details:

Deployment Scenario:

For MongoDB-related concerns as well, it is crucial that our support team is aware of the server instance being used by your application.Please share the details with us about the type of your instance, which can be from the following:

1. Standalone instance.
2. Replica set.
3. Sharded cluster.

Deployment type:

Logs play an important role in debugging an issue related to MongoDB as well. To effectively utilize the logs for troubleshooting, we need to know about the Deployment type of Countly. It can be any of them from the following three options:

1. VM
2. Docker
3. Kubernetes

Web server type:

For troubleshooting issues specific to MongoDB, our Support team requires information about the server and any associated load balancers. Nginx is the default load balancer of the Countly server, however, if you are using Docker or Kubernetes setup, you can use other web servers.

Countly Version

Lastly, please make sure to notify the support team about the the version of Countly Server being used. 

Logs

You can use the following methods to collect the logs and submit it.

MongoDB - Logs

To get MongoDB Logs for any environment, you can use the following commands:

1. This command is used to get the path of MongoDB logs: /var/logs/mongodb/mongod.log
   1. countly root/log/countly-mongod.log
   2. Docker logs: docker logs <mongodb pod name> 
   3. Kubernetes logs: kubectl logs <mongodb pod name>

2. If the Log path does not exist, the correct path can be found in /etc/mongod.conf under systemLog path.

3. If /etc/mongod.conf does not exist, the path to the config file can be found using the “systemctl status mongod” command –config.

MongoDB - mongostat

The mongostat utility provides a quick overview of the status of a currently running mongod or mongos instance. mongostat is functionally similar to the UNIX/Linux file system utility vmstat, but provides data regarding mongod and mongos instances. Using ‘mongostat’ for mongos instance also means that, there is a sharded cluster deployment. Hence, mongostat will run with an additional flag: “mongostat --discover”.

Here is some basic information about mongostat:

1. It needs authentication parameters if authentication is enabled
2. Dirty over 5% means it does more writes than it can handle, which can cause overloading.
3. Used over 80% means insufficient RAM to fit the working set.

MongoDB - mongotop

mongotop provides a method to track how much time a MongoDB instance spends reading and writing data. mongotop provides statistics on a per-collection level. By default, mongotop returns a value every second.

Here is some basic information about mongotop:

1. It needs authentication parameters if authentication is enabled
2. It shows the slowest collections of the clusters.
3. It is also recommended to wait for a certain time, to collect more information as the loads can peak periodically, and thus, there might be an inaccuracy. collecting accurate information can help us resolve the problem quickly.

MongoDB - current operations

It provides information about all the operations running currently.

You can use the following commands to get more information:

1. mongo //with authentication parameters 
2. db.currentOp() - to get a full list of running operations,

MongoDB - piping to file

You can use the following commands to attach it to the file:

1. mongostat -n 20 > mongostat.txt
2. mongotop -n 20 > mongotop.txt
3. mongo --eval "db.currentOp()" > currentops.txt

Please note that mongostat & mongotop may take a while to finish execution.

MongoDB - diagnostic folder

1. This helps you know about MongoDB stats.
2. It is Located in the MongoDB data folder named diagnostic.data.
3. It does not contain customer data.
4. The path to the data folder can be found in /etc/mongod.conf under dbPath.
5. If /etc/mongod.conf does not exist, the path to the config file can be found using the “systemctl status mongod” command under –config.

MongoDB replica set

MongoDB achieves replication by the use of a replica set. A replica set is a group of mongod instances that host the same data set. In a replica, one node is a primary node that receives all write operations. All other instances, such as secondaries, apply operations from the primary so that they have the same data set. A replica set can have only one primary node.

You are requested to use the following commands:

1. mongo //with authentication parameters. 
2. rs.printReplicationInfo()

To check the oplog size.

3. rs.printSecondaryReplicationInfo() 

To check the lag of secondary servers, there may be an issue if it is higher than 30 seconds and does not decrease with time.

4. rs.status().members

To check the status from the viewpoint of a current member, if others are reachable, etc.

5. rs.conf()

To get the Replica set configuration, number of nodes, etc.

MongoDB sharded cluster

Sharding is the process of storing data records across multiple machines, and it is MongoDB's approach to meeting the demands of data growth. As the data size increases, more than a single machine may be required to store the data or provide an acceptable read-and-write throughput. Sharding solves the problem with horizontal scaling.

When you run your instance in a sharded cluster, you are requested to use the following commands:

1. mongo //with authentication parameters
2. All replica set commands <hyperlink to replica set>
3. sh.status()
4. sh.getBalancerState()
5. sh.isBalancerRunning()

Countly Server Issue

When there is any issue within the load balancer, the data might arrive late, which can cause disruption to the real-time analytics provided by Countly.

Seek Solutions: Countly Server Issues

If you encounter any issues with the Countly server, please provide the following details:

Countly Version

You need to inform the support team about the version of Countly Server being used. This ensures more accurate assistance for your specific configuration.

Deployment Scenario

To resolve server-related concerns, we also need information about the server instance your application is using. Please share the details about your instance type, which can be selected from the options below:

1. Standalone instance.
2. Auto scale group.

Deployment type

Effective troubleshooting for Countly server issues relies on utilizing logs. To effectively utilize the logs for troubleshooting, we need to know about the Deployment type of Countly. It can be any of them from the following three options:

1. VM
2. Docker
3. Kubernetes

Web server type

To solve server-related issues, our Support team requires information about both the server and the load balancers used by your server. Nginx is the default load balancer of the Countly server, however, if you are using Docker or Kubernetes setup, you can use other web servers.

Logs

To get the path of the logs related to your server, use the below command

/var/logs/nginx/error.log or docker/kubernetes equivalents

If there is a bug related to the Load balancer, the following information will be needed: 

1. Amount of requests
2. Amount of failed requests
3. Breakdown of failed request HTTP code

       To get the path of the logs related to your server, use the below command

/var/logs/nginx/error.log or docker/kubernetes equivalents

Third-party solutions such as Dynatrace, Datadog can be used as well

Countly Command Line

Below are a few commands that can help you collect data about your Countly instance.

1. countly dir - outputs the path where countly is installed in your machine.
2. countly mongo - outputs mongo connection parameters.
3. countly status - check if the process is running correctly.
4. countly version - check your current countly version.
5. countly server-stats dp - to get datapoint usage by your application.
6. countly config list values - lists all settings that are being used in the Countly platform.

For more commands, please visit:
https://support.count.ly/hc/en-us/articles/360037444912-Countly-Command-Line

Troubleshooting Drill

If you encounter any difficulties with Drill, please share the corresponding URL with our support team. The Drill URL has been enhanced, and now it includes selected period, metric, visualization, time bucket, and executed flag. This will enable our support team to see exactly what the customer sees on their end, and better assistance can be provided.