Crash Symbolication

Last modified: September 23, 2024 16:43

Crash Symbolication lets you symbolicate/de-obfuscate crash reports and convert them into human readable format, helping you pinpoint where in your code a crash is originating from.

Availability

The Symbolication plugin is available in Countly Enterprise and as an add-on in Flex.

Understanding Crash Symbolication

What is "Symbolication"?

When releasing your application, you might be skipping crucial steps such as debugging information from the final binary format, obfuscating your source code, or minifying it by removing unnecessary characters. Doing all this can help make your application harder to reverse engineer, but you may then receive error stack traces that have obfuscated information in them, making it impossible to track down the source of your crash.

Depending on the platform, your stack trace may be full of memory addresses or transformed function names. Symbolication is the process of converting them into human readable, class/method names, file names, and line numbers.

For information on how to setup your app for symbolication, check here.

Symbolicating iOS, Android, and JavaScript Crashes

Save your symbol or mapping files!

For the obfuscation and minification processes to be reversible, a symbol or mapping file is also produced while building your application. It is imperative for you to keep and archive those files as the symbolication process is only possible with them. You will also need to archive symbol files for every version you would like to symbolicate.

Currently, Countly supports symbolication for Apple-provided tools in iOS mobile applications, DexGuard and ProGuard in Android, and source maps in JavaScript web applications.

Android Symbolication Samples

java.lang.IllegalStateException: Could not execute method for android:onClick
at android.view.View$DeclaredOnClickListener.onClick(View.java:4725)
at android.view.View.performClick(View.java:5637)
at android.view.View$PerformClick.run(View.java:22429)
at android.os.Handler.handleCallback(Handler.java:751)
at android.os.Handler.dispatchMessage(Handler.java:95)
at android.os.Looper.loop(Looper.java:154)
at android.app.ActivityThread.main(ActivityThread.java:6121)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779)
Caused by: java.lang.reflect.InvocationTargetException
at java.lang.reflect.Method.invoke(Native Method)
at android.view.View$DeclaredOnClickListener.onClick(View.java:4720)
... 9 more
Caused by: java.lang.Exception: Exception at the end of the call
at ly.count.android.demo.a.b(SourceFile:29)
at ly.count.android.demo.a.a(SourceFile:21)
at ly.count.android.demo.ActivityExampleCrashReporting.c(SourceFile:98)
at ly.count.android.demo.ActivityExampleCrashReporting.b(SourceFile:94)
at ly.count.android.demo.ActivityExampleCrashReporting.a(SourceFile:90)
at ly.count.android.demo.ActivityExampleCrashReporting.onClickCrashReporting10(SourceFile:82)
... 11 more

java.lang.IllegalStateException: Could not execute method for android:onClick
at android.view.View$DeclaredOnClickListener.onClick(View.java:4725)
at android.view.View.performClick(View.java:5637)
at android.view.View$PerformClick.run(View.java:22429)
at android.os.Handler.handleCallback(Handler.java:751)
at android.os.Handler.dispatchMessage(Handler.java:95)
at android.os.Looper.loop(Looper.java:154)
at android.app.ActivityThread.main(ActivityThread.java:6121)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779)
Caused by: java.lang.reflect.InvocationTargetException
at java.lang.reflect.Method.invoke(Native Method)
at android.view.View$DeclaredOnClickListener.onClick(View.java:4720)
... 9 more
Caused by: java.lang.Exception: Exception at the end of the call
at ly.count.android.demo.Utility.void DeepCall_b()(SourceFile:29)
at ly.count.android.demo.Utility.void DeepCall_a()(SourceFile:21)
at ly.count.android.demo.ActivityExampleCrashReporting.void deepFunctionCall_3()(SourceFile:98)
at ly.count.android.demo.ActivityExampleCrashReporting.void deepFunctionCall_2()(SourceFile:94)
at ly.count.android.demo.ActivityExampleCrashReporting.void deepFunctionCall_1()(SourceFile:90)
at ly.count.android.demo.ActivityExampleCrashReporting.void onClickCrashReporting10(android.view.View)(SourceFile:82)
... 11 more

iOS Symbolication Samples

CoreFoundation                      0x00000001e5669ebc    252
libobjc.A.dylib                     0x00000001e4839a50 objc_exception_throw   56
CoreFoundation                      0x00000001e55e1384 _CFArgv   0
CoreFoundation                      0x00000001e557157c    0
CountlyTestApp-iOS                  0x0000000104c8910c CountlyTestApp-iOS   37132
CountlyTestApp-iOS                  0x0000000104c9a8c0 CountlyTestApp-iOS   108736
UIKitCore                           0x0000000212b62458    1348
UIKitCore                           0x0000000212b626bc    268
UIKitCore                           0x000000021296087c    296
UIKitCore                           0x000000021294e878    384
UIKitCore                           0x000000021297d880    132
CoreFoundation                      0x00000001e55f96bc    32
CoreFoundation                      0x00000001e55f4350    412
CoreFoundation                      0x00000001e55f48f0    1264
CoreFoundation                      0x00000001e55f40e0 CFRunLoopRunSpecific   436
GraphicsServices                    0x00000001e786d584 GSEventRunModal   100
UIKitCore                           0x0000000212954c00 UIApplicationMain   212
CountlyTestApp-iOS                  0x0000000104ca2c3c CountlyTestApp-iOS   142396
libdyld.dylib                       0x00000001e50b2bb4    4

__exceptionPreprocess (in CoreFoundation) + 252
objc_exception_throw (in libobjc.A.dylib) + 56
_CFArgv (in CoreFoundation) + 0
-[NSDate dateByAddingTimeInterval:] (in CoreFoundation) + 0
+[EYCrashTesting crashTest1] (in CountlyTestApp-iOS) (EYCrashTesting.m:0)
-[MainViewController tableView:didSelectRowAtIndexPath:] (in CountlyTestApp-iOS) (MainViewController.m:0)
-[UITableView _selectRowAtIndexPath:animated:scrollPosition:notifyDelegate:] (in UIKitCore) + 1348
-[UITableView _userSelectRowAtPendingSelectionIndexPath:] (in UIKitCore) + 268
_runAfterCACommitDeferredBlocks (in UIKitCore) + 296
_cleanUpAfterCAFlushAndRunDeferredBlocks (in UIKitCore) + 384
_afterCACommitHandler (in UIKitCore) + 132
__CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ (in CoreFoundation) + 32
__CFRunLoopDoObservers (in CoreFoundation) + 412
__CFRunLoopRun (in CoreFoundation) + 1264
CFRunLoopRunSpecific (in CoreFoundation) + 436
GSEventRunModal (in GraphicsServices) + 100
UIApplicationMain (in UIKitCore) + 212
main (in CountlyTestApp-iOS) (main.m:19)
libdyld.dylib                       0x00000001e50b2bb4    4

JavaScript Symbolication Samples

Error: Error at depth 3
    at cause_error (http://127.0.0.1:5501/examples/symbolication/dist/main.js:5657:13)
    at multiply (http://127.0.0.1:5501/examples/symbolication/dist/main.js:5651:3)
    at saveRecords (http://127.0.0.1:5501/examples/symbolication/dist/main.js:5648:3)
    at initializeCoreFunctions (http://127.0.0.1:5501/examples/symbolication/dist/main.js:5645:3)
    at save (http://127.0.0.1:5501/examples/symbolication/dist/main.js:5642:3)
    at removeHashes (http://127.0.0.1:5501/examples/symbolication/dist/main.js:5639:3)
    at calculateParams (http://127.0.0.1:5501/examples/symbolication/dist/main.js:5636:3)
    at addRecords (http://127.0.0.1:5501/examples/symbolication/dist/main.js:5633:3)
    at HTMLButtonElement.unhandled_error (http://127.0.0.1:5501/examples/symbolication/dist/main.js:5675:5)

Error: Error at depth 3
    at src/index.js:66:12
    at cause_error (src/index.js:59:2)
    at multiply (src/index.js:56:2)
    at saveRecords (src/index.js:53:2)
    at initializeCoreFunctions (src/index.js:50:2)
    at save (src/index.js:47:2)
    at removeHashes (src/index.js:44:2)
    at calculateParams (src/index.js:41:2)
    at addRecords (src/index.js:87:6)

Configuring the Server for Symbolication

The first step for using symbolication is installing and enabling the Symbolication plugin.

To do so, in the main Countly Dashboard, go to Management > Plugins and enable the Crash Symbolication toggle.

After enabling the Symbolication plugin, you might notice an additional entry added to your Crashes submenu items:

Overview takes you to the a crash general overview.
Manage Symbols takes you to a screen where you can manage all the symbol files uploaded for this application.

Another entry would be added to thje Utilities section:

Symbolication logs is similar to the request logs, where a short history of your symbolication requests is kept and therefore, this view is only populated if there are problems with the symbolication processes. More information here.

For iOS and Android applications, the next step is creating the connection to the Countly symbolication server. To do so, you will need a symbolication API key provided by Countly. The Countly symbolication server address is https://symbolication.count.ly. When you get your API key, you will need to set it in the server configuration screen in Management > Settings, under the Crashes section.

After you have made sure that those fields contain valid information, click the Test Connection button to make sure everything is working correctly. If your provided server URL or API key is wrong or your Countly server cannot reach the symbolication server, you may diagnose it with this button.

If everything is working correctly, you will see the alert messages in the image below.

You are now done setting up your Countly instance to use iOS and Android Symbolication.

Uploading the Symbol File

Now that the connection to the symbolication server has been established and the necessary symbol files are procured, we can get to the symbolication of specific stack traces.

Only one symbol file upload required

You should only upload one symbol file per application version (Android) / UUID (iOS). Multiple symbol file uploads are not supported.

To upload a symbol file, open the Manage Symbols section. At first it will look empty, as shown below:

There, you will have to click on the Add Symbolication File button, which will open a drawer with the fields described below. Note that you will need to do this for every version of your app you would like to symbolicate.

Select Type: In the Type selection, you need to choose the platform for the symbol file you are uploading.

Build UUID / Android app version: For Android, the Build ID field is the version name. For iOS, the Build UUID field is your app's Build UUIDs. For JavaScript, this must match the app_version you use when initializing the Countly SDK in your web application.

You can get Build UUIDs using the dwarfdump --uuid command on the dSYM file, and you may specify more than one comma-separated Build UUID for each architecture.

Note that the Android Build ID and iOS Build UUID specified here must match your app's corresponding Build ID or Build UUID.

Example:

dwarfdump --uuid CountlyTestApp-iOS.app.dSYM
UUID: AD4F5647-B2A0-3D85-8DF4-78D4DFD83296 (armv7) CountlyTestApp-iOS.app.dSYM/Contents/Resources/DWARF/CountlyTestApp-iOS
UUID: 70B0DE66-D337-3952-914D-A1E542667E20 (arm64) CountlyTestApp-iOS.app.dSYM/Contents/Resources/DWARF/CountlyTestApp-iOS

Symbol File: While choosing a Symbolication/Mapping File upon uploading the page for iOS, do not forget to zip the dSYM file before uploading. Otherwise, you will not be able to upload a dSYM file in a folder structure.

Notes: Free text optional field.

When everything is filled out, simply click Upload file.

Again, keep in mind you should do this process for every version or build that you would like to symbolicate.

Symbolicating a Crash

After some time has passed, you will start to see new exceptions in your Crashes Overview dashboard. If you have uploaded the correct symbols for those versions, then you are ready to move forward with the symbolication process.

To symbolicate a crash, you first have to go to its detailed view by clicking on the title of the crash.

As shown below, the detailed crash view will be divided in two parts: the top contains the crash group stack trace, and the bottom contains entries for each specific crash and its circumstances.

When clicking on a specific crash entry, it opens up additional information which contains a Symbolicate toggle button, assuming it's possible to symbolicate the crash (e.g. symbol file is uploaded for corresponding version). At the top of the crash group section, you may also see the Symbolicate button.

When you click on the Symbolicate button, you should see the Symbolication in progress message. When that process is complete, you should be able to see the symbolicated stack trace.

The current symbolication implementation has some technical limitations, and the top crash group will show a Symbolicate button only after a crash from a newer app version is received. Therefore, if you would like to symbolicate crashes from a version you had before you enabled Crash Symbolication, you will have to scroll down to a specific crash entry at the bottom of the detailed crash view.

Looking for help?

Countly Enterprise Support

Create a ticket for any support request you might have as a Countly Enterprise Customer

Countly Community

Join us on Discord to engage in discussions, share your feature ideas, and learn from fellow Countly users