Crashpad

Installing depot_tools (ref)

MacOS

• git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
• sudo echo "export PATH=/path/to/depot_tools:$PATH" >> ~/.zshrc

Getting the Crashpad source (ref)

Initial checkout

• mkdir ~/crashpad
• cd ~/crashpad
• fetch crashpad

Subsequent checkouts

• cd ~/crashpad/crashpad
• git pull -r
• gclient sync

Building

• cd ~/crashpad/crashpad
• gn gen out/Default
• ninja -C out/Default

Building Shared Libraries (ref)

Windows

After running gn gen out\Default, you can edit the out\Default\toolchain.ninja file to add extra compiler flags to the command for the cc and cxx rules.

rule cc
  command = ninja -t msvc -e environment.amd64 -- cl.exe ... ${cflags} ${cflags_c} /c ...
                                 add the /MD compiler flag after the others, here ^

Integrating Crashpad with Qt

Linking (ref)

MacOS

Building Crashpad outputs several .a files which need to be added to your project along with the corresponding include directories. In addition to the .a files Crashpad generates a collection of .o files that need to be included. You will also need to link with libbsm, Security.Framework and AppKit.Framework. Finally, you'll need to add config parameters force_debug_info and separate_debug_info. The following is a snippet from the myQtCrasher.pro file:

# Create a dSYM file for dump_syms
CONFIG += force_debug_info
CONFIG += separate_debug_info

# Include directories for Crashpad libraries
INCLUDEPATH += $$PWD/Crashpad/Include/crashpad
INCLUDEPATH += $$PWD/Crashpad/Include/crashpad/third_party/mini_chromium/mini_chromium

# Crashpad rules for MacOS
macx {
    # Crashpad libraries
    LIBS += -L$$PWD/Crashpad/Libraries/MacOS/ -lbase
    LIBS += -L$$PWD/Crashpad/Libraries/MacOS/ -lutil
    LIBS += -L$$PWD/Crashpad/Libraries/MacOS/ -lclient
    LIBS += "$$PWD/Crashpad/Libraries/MacOS/util/mach/*.o"

    # System libraries
    LIBS += -L/usr/lib/ -lbsm
    LIBS += -framework AppKit
    LIBS += -framework Security

    # Copy crashpad_handler to build directory
    crashpad.commands = mkdir -p $$OUT_PWD/crashpad && cp $$PWD/Crashpad/Bin/MacOS/crashpad_handler $$OUT_PWD/crashpad
    first.depends = $(first) crashpad
    export(first.depends)
    export(copydata.commands)
    QMAKE_EXTRA_TARGETS += first crashpad
}

Configuring Crashpad (ref)

Building Crashpad outputs the crashpad_handler executable. Copy this file into your project and note its path. Ensure that this file is distributed with your application.

Add the following snippet to the entry point of your application:

bool initializeCrashpad(char *dbName, char *appName, char *appVersion )
{
    // Ensure that handler is shipped with your application
    base::FilePath handler("../../../crashpad/crashpad_handler");

    // Directory where reports will be saved. Important! Must be writable or crashpad_handler will crash.
    base::FilePath reportsDir("../../../crashpad");

    // Directory where metrics will be saved. Important! Must be writable or crashpad_handler will crash.
    base::FilePath metricsDir("../../../crashpad");

    // Configure url with your BugSplat database
    std::string url;
    url = "https://";
    url += dbName;
    url += ".bugsplat.com/post/bp/crash/crashpad.php";

    // Metadata that will be posted to BugSplat
    std::map<std::string, std::string> annotations;
    annotations["format"] = "minidump";                 // Required: Crashpad setting to save crash as a minidump
    annotations["product"].assign(appName);             // Required: BugSplat appName
    annotations["version"].assign(appVersion);      	// Required: BugSplat appVersion
    annotations["key"] = "Sample key";                  // Optional: BugSplat key field
    annotations["user"] = "fred@bugsplat.com";          // Optional: BugSplat user email
    annotations["list_annotations"] = "Sample comment";	// Optional: BugSplat crash description

    // Disable crashpad rate limiting so that all crashes have dmp files
    std::vector<std::string> arguments;
    arguments.push_back("--no-rate-limit");

    // Initialize crashpad database
    std::unique_ptr<CrashReportDatabase> database = crashpad::CrashReportDatabase::Initialize(reportsDir);
    if (database == NULL) return false;

    // Enable automated crash uploads
    Settings *settings = database->GetSettings();
    if (settings == NULL) return false;
    settings->SetUploadsEnabled(true);

    // Start crash handler
    CrashpadClient *client = new CrashpadClient();
    bool status = client->StartHandler(handler, reportsDir, metricsDir, url, annotations, arguments, true, true);
    return status;
}

Call initializeCrashpad at the entry point of your application passing it the name of your BugSplat database, the name of your application and the current version.

char *dbName = (char *)"Fred";
char *appName = (char *)"myQtCrasher";
char *appVersion = (char *)"1.0";

initializeCrashpad(dbName, appName, appVersion);

Add code that will crash your application and ensure it will run after Crashpad has been initialized.

void crash() {
    *(volatile int *)0 = 0;
}

Ensure your project builds before moving on to the next step.

Generating Symbols

The Crashpad repository is missing a few tools that are available in the Breakpad repository. In order to generate sym files from your compiled executable you'll need to clone the Breakpad repository and build the dump_syms utility.

Getting the Breakpad source (ref)

• mkdir ~/breakpad
• cd ~/breakpad
• fetch breakpad

Building dump_syms

MacOS

Open breakpad/src/src/tools/mac/dump_syms/dump_syms.xcodeproj. Switch the configuration to dump_syms and build the project. The report navigator tab (icon looks like a chat bubble in Xcode 11) will show you the file system location with the compiled executable. Copy the dump_syms executable into your project.

Running dump_syms (ref)

• ./dump_syms -g path/to/myApp.dSYM path/to/myApp > myApp.sym

Uploading Symbols

Building symupload

MacOS

Open breakpad/src/src/tools/mac/sym_upload/symupload.xcodeproj and build the project. The report navigator tab (icon looks like a chat bubble in Xcode 11) will show you the file system location with the compiled executable. Copy the sym_upload executable into your project.

If your application generates large sym file sizes, or your network bandwidth is limited you may run into upload timeouts. If this happens, increase the value of timeoutInterval in breakpad/src/src/common/mac/HTTPMultipartUpload.m and rebuild the symupload project.

Running symupload (ref)

• ./symupload "path/to/myApp.sym" "https://{database}.bugsplat.com/post/bp/symbol/breakpadsymbols.php?appName={appName}&appVer={appVersion}"
• Be sure to replace {database} with your BugSplat database
• Be sure that {appName} and {appVersion} are the same values used to initialize the Crashpad handler

Generate a Crash

Run your application to generate a crash report. The crash report should show up on BugSplat's Crashes page. Click the number in the ID column to see more information about the crash report. If everything is set up correctly you should see something like this:

Troubleshooting

The Debugger Output tab on the Crash page will show you the output from minidump_stackwalk. If file names and line numbers are not being displayed in the stack trace, the most common cause is mismatched sym files. Look for the following line in the Debugger Output tab to determine the identifier of the module that was loaded at crash time.

2020-04-23 09:42:03: simple_symbol_supplier.cc:196: INFO: No symbol file at /www/src/RemoteStackAnalyzer/build/symbols/breakpad/Fred/myQtCrasher-1.0/myQtCrasher/63DE7DDC72203043B92951A0582ADE1B0/myQtCrasher.sym

Verify that the correct sym file has been uploaded to BugSplat on the Symbols page. The application name and version of the symbol store must match the application name and version configured in initializeCrashpad.

Click symbol store link to be taken to the Symbol Details page. Click the link in the download column and open the sym file in a text editor. If the identifier at top of the sym file does not match the identifier from the Debugger Output it is not the correct sym file.

MODULE mac x86_64 63DE7DDC72203043B92951A0582ADE1B0 myQtCrasher

Finally, if you are using copy protection or anti-cheat software, ensure that dump_syms has been run after your executable has been modified by your copy protection or anti-cheat solution.