Debug Information Files

Debug information files allow Sentry to extract stack traces and provide more information about crash reports for most compiled platforms. Information stored in debug files includes original function names, paths to source files and line numbers, source code context, or the placement of variables in memory. Sentry can use some of this information and display it on the issue details page.

Sentry requires access to debug information files of your application as well as system libraries to provide fully symbolicated crash reports. You can either upload your files to Sentry or put them on a compatible Symbol Server to be downloaded by Sentry when needed.

Debug information files can be managed on the Debug Files section in Project Settings. This page lists all uploaded files and allows to configure symbol servers for automatic downloads.

Debug Information

Sentry differentiates in four kinds of debug information:

  • Debug Information: Provides function names, paths to source files, line numbers and inline frames. The process of resolving this information from instruction addresses is called "symbolication". This information is relatively large compared to the executable and usually put into a separate file. In Sentry, these files are designated as debug companions and show the debug tag.

  • Symbol Tables: If debug information is not available for a certain library, Sentry can use symbol tables as a fallback to retrieve function names. Symbol tables are usually included in both the executable and debug companion files. However, they do not contain sufficient information to resolve inline functions or file names and line numbers. The symtab tag indicates symbol tables.

  • Source Code: Conventionally, source code is not part of regular debug information files. Sentry CLI can bundle source code of your application and upload it to display source context in stack traces in Sentry. These bundles show up with the sources tag.

  • Unwind Information: Enables Sentry to extract stack traces from Minidumps and other binary crash formats of optimized builds. This process is referred to as "stack unwinding" or "stack walking". Since this is also required when throwing exceptions in C++, this information is often included in the executable or library. If an uploaded file contains this information, it shows the unwind tag. Note that on some platforms no actual unwinding takes place. For instance, WebAssembly currently does not have the equivalent of minidumps which means we do not require that type of information in such cases.

Compilers place the above debug information in different files passed on the target platform, architecture, build flags or optimization level. Consequently, Sentry might not need all of the above information to process crash reports. Still, it is always a good idea to provide all available debug information.

sentry-cli can be used to list properties of supported debug files and validate their contents. See Debug Information Files in sentry-cli for more information.

For WebAssembly, we support DWARF in WASM containers. Note that we do not support source maps, which are also a format used for WASM debugging, but have shortcomings that make them impractical for a crash reporting tool like Sentry.

Since WASM does not specify debug/build IDs yet, we provide a separate tool to add build IDs and split files called wasm-split to help you create a debug companion file ready for uploading to Sentry while removing all debug information from the release binary.

Debug Identifiers

Each debug information file specifies a unique identifier. Crash reports declare these identifiers to allow debuggers and crash reporting systems to resolve the correct files. Sentry distinguishes two kinds of identifiers:

  • Code Identifier: The unique identifier of the executable or dynamic library -- the code file. The contents of this identifier are platform-dependent: MachO files use a UUID, ELF files a SHA hash, PE files use a concatenation of certain header attributes. For WebAssembly we use an embedded UUID in the build_id section of the file.

  • Debug Identifier: The unique identifier of the debug companion file. In contrast to the code identifier, Sentry enforces the same structure on all platforms. On Windows, this is the actual unique id of the PDB file; on all other platforms this is a lossy transformation of the code identifier.

When uploading debug information files to Sentry, the CLI and server will always compute a Debug Identifier for each uploaded file. This identifier is associated with executables and libraries as well as debug companions to ensure that they can be uniquely located via one common mechanism.

For native events, the issue details page displays a list of Loaded Images. This list contains the executable and all loaded dynamic libraries including their debug identifiers. You can copy this identifier and search for the exact files that match it in the Debug Files settings screen.

sentry-cli can help to print properties of debug information files like their debug identifier. See Checking Debug Information Files for more information.

WASM Build IDs

WebAssembly does not yet support build IDs. The option proposed to implement build IDs for WebAssembly (Build ID Section for WASM) has not yet found widespread adoption. Instead, we use a custom extension to WebAssembly.

Our recommendation is to embed a UUID in the build_id custom section as raw binary. Our wasm-split tool can do this for you automatically.

Uploading Files

The most straightforward way to provide Sentry with debug information file is to upload them using sentry-cli. Depending on your workflow, you may want to upload as part of your build pipeline or when deploying and publishing your application:

debug files workflow

Files can be uploaded using the upload-dif command. This command will scan a given folder recursively for files and upload them to Sentry:

Copied
wasm-split /path/to/myfile.wasm -d /path/to/myfile.debug.wasm --strip
sentry-cli upload-dif -o <org> -p <project> /path/to/files

> Found 1 debug information files
> Prepared debug information files for upload
> Uploaded 1 missing debug information files
> File processing complete:

  PENDING 1ddb3423-950a-3646-b17b-d4360e6acfc9 (mylib; wasm debug companion)

For all available options and more information refer to Uploading Debug Information.

Always ensure that debug files are uploaded before deploying or releasing your application so that crash reports can be processed. For manual testing, use the sentry-cli upload-dif --wait before sending the first native crash or error event.

If you upload a file that was previously reported as missing by Sentry in a crash report, it may take up to one hour until Sentry starts using this file for new crash reports. Existing events and issues are not processed again.

In Sentry, your uploaded files are associated with projects. You can view and manage uploads at Project Settings > Debug Files. This screen displays the most important properties of debug files:

  1. The debug identifier. There may be multiple entries sharing the same identifier, if debug information is split across multiple files.
  2. The name of the debug file. Sentry uses the name on the file system when uploading the file using sentry-cli.
  3. The architecture, and type of the file. This can be used to distinguish files into executables and debug companions if their debug identifiers and names match.
  4. The debug information available in these files. This can include unwind, debug, symtab and sources.
  5. Metadata like the size and time of upload.

If multiple projects require the same file, it needs to be re-uploaded. Sentry does not access debug files stored in other projects to symbolicate crash reports.

Reprocessing

Sentry can suspend incoming crash reports until all required debug information files have been uploaded. This feature is called Reprocessing. It can be configured in Project Settings > Processing Issues. By default, this feature is disabled.

If enabled, crash reports with missing debug files will not be displayed in the issues stream. Instead, you will receive a warning that events cannot be processed until all debug files have been uploaded.

Once an issue is shown in the issues stream, it is no longer processed. Even with enabled reprocessing, new file uploads will not effect such events.

At the moment, this feature only applies to iOS crashes sent with the Cocoa SDK and is not compatible with Symbol Servers.

Symbol Servers

Sentry can download debug information files from external repositories. This allows you to stop uploading debug files and instead configure a public symbol server or run your own. It is also possible to configure external repositories and upload debug files at the same time.

To configure external repositories, go to Project Settings > Debug Files. Above the list of uploaded files, there are two settings to configure external repositories:

  1. Custom Repositories: Configures custom repositories containing debug files. You can choose from configuring an HTTP symbol server, Amazon S3 bucket or Google Cloud Storage bucket. This requires a Business or Enterprise plan.

  2. Built-In Repositories: Allows to select from a list of pre-configured symbol servers. By default, iOS and Microsoft are enabled.

Sentry queries external repositories for debug information files in the order they are configured. If custom repositories are configured, those are probed first. Only debug information files that are not found on one of the custom repositories are queried from the built-in ones.

Built-In Repositories

To enable a built-in repository, select it from the dropdown list. This immediately adds the repository and uses its debug information files to symbolicate new crash reports. Likewise, any built-in repository can be disabled by clicking on the X next to the name.

Adding or removing external repositories applies immediately. As a result, events may group differently with the new information and create new issues. Beware that these cause notifications to your team members.

Custom Repositories

Independent of the internal format, Sentry supports three kinds of custom repositories:

  • HTTP Symbol Server: An HTTP server that serves debug files at a configurable path. Lookups in the server should generally be case-insensitive, although an explicit casing can be configured in the settings.

  • Amazon S3 Bucket: Either an entire S3 bucket or a subdirectory. This requires s3:GetObject, and optionally s3:ListBucket permissions for the configured Access Key. Lookups in the bucket are case sensitive, which is why we recommend storing all files lower-cased.

  • Google Cloud Storage Bucket: Either an entire GCS bucket or a subdirectory. This requires storage.objects.get and storage.objects.list permissions for the configured service account. Lookups in the bucket are case sensitive, which is why we recommend storing all files lower-cased.

Apart from authentication configuration, all types have common config parameters:

  1. Name: A name to identify the repository.

  2. Path Casing: Overrides which casing Sentry uses to query for debug information files. The default is a mixed case, which will use the case described in the next section. When overridden, all access is either lowercased or uppercased. Defaults to "mixed case".

  3. Directory Layout: The internal structure of the bucket, or the protocol of the symbol server. There are three layouts to choose from which are discussed in the next section. Defaults to "Platform Specific".

Directory Layouts

Sentry supports multiple layouts for external repositories. Based on the selected layout and the file type, we try to download files at specific paths.

The following table contains a mapping from the supported layouts to file path schemas applied for specific files:

LayoutMachOELFPEPDBBreakpad
Platform-SpecificLLDBBuildIDSymStoreSymStoreBreakpad
Microsoft SymStore--SymStoreSymStore-
Microsoft SymStore (index2.txt)--Index2Index2-
Microsoft SSQPSSQPSSQPSSQPSSQP-

The path schemas in the table above are defined as follows:

Breakpad
Path: <DebugName>/<BREAKPADid>/<SymName>

Breakpad always uses a Breakpad ID to store symbols. These identifiers can be computed from Debug Identifiers by removing dashes and applying the following casing rules:

  • The signature part of the id (first 32 characters) are uppercase.
  • The age part of the id (remaining characters) are lowercase.

The name of the symbol file is platform dependent. On Windows, the file extension (Either .exe, .dll or .pdb) is replaced with .sym. On all other platforms, the .sym extension is appended to the full file name including potential extensions.

Examples:

  • wkernel32.pdb/FF9F9F7841DB88F0CDEDA9E1E9BFF3B51/wkernel32.sym
  • MyFramework.dylib/5E012A646CC536F19B4DA0564049169B/MyFramework.dylib.sym

LLDB
Path: XXXX/XXXX/XXXX/XXXX/XXXX/XXXXXXXXXXXX[.app]

The LLDB debugger on macOS can read debug symbols from File Mapped UUID Directories. The UUID is broken up by splitting the first 20 hex digits into 4 character chunks, and a directory is created for each chunk. In the final directory, LLDB usually expects a symlink named by the last 12 hex digits, which it follows to the actual dSYM file.

This is not actually an LLVM feature. This is in fact a feature of CoreFoundation and exclusively implemented on macOS on top of spotlight. Spotlight indexes these paths and the private DBGCopyFullDSYMURLForUUID API is used by lldb to locate the symbols. macOS uses the symlinks of those locations.

Since the executable or library shares the same UUID as the dSYM file, the former are distinguished with a .app suffix.

The hex digits are uppercase, the app suffix is lowercase.

Examples:

  • 5E01/2A64/6CC5/36F1/9B4D/A0564049169B (debug companion)
  • 5E01/2A64/6CC5/36F1/9B4D/A0564049169B.app (executable or library)

BuildID
Path: nn/nnnnnnnnnnnnnnnn...[.debug]

GDB supports multiple lookup methods, depending on the way the debug info file is specified. Sentry uses the Build ID Method: Assuming that a GNU build ID note or section has been written to the ELF file, this specifies a unique identifier for the executable which is also retained in the debug file.

The GNU build ID is a variable-length binary string, usually consisting of a 20-byte SHA1 hash of the code section (.text). The lookup path is pp/nnnnnnnn.debug, where pp are the first 2 hex characters of the build ID bit string, and nnnnnnnn are the rest of the hex string. To look up executables, the .debug suffix is omitted.

Examples:

  • b5/381a457906d279073822a5ceb24c4bfef94ddb (executable or library)
  • b5/381a457906d279073822a5ceb24c4bfef94ddb.debug (stripped debug file)

SSQP
Path: <file_name>/<prefix>-<identifier>/<file_name>

SSQP Key Conventions are an extension to the original Microsoft Symbol Server protocol for .NET. It specifies lookup paths for PE, PDB, MachO and ELF files. The case of all lookup paths is generally lowercase except for the age field of PDB identifiers which should be uppercase.

For MachO files and ELF files, SSQP specifies to use the same identifiers as used in the LLDB and GNU build id method, respectively. See the sections above for more information. This results in the following paths for all possible file types:

  • <code_name>/<timestamp><size_of_image>/<code_name> (PE file)
  • <debug_name>/<signature><AGE>/<debug_name> (PDB file)
  • <code_name>/elf-buildid-<buildid>/<code_name> (ELF binary)
  • _.debug/elf-buildid-sym-<buildid>/_.debug (ELF debug file)
  • <code_name>/mach-uuid-<uuid>/<code_name> (MachO binary)
  • _.dwarf/mach-uuid-sym-<uuid>/_.dwarf (MachO binary)

SSQP specifies an additional lookup method by SHA1 checksum over the file contents, commonly used for source file lookups. Sentry does not support this lookup method.

Examples:

  • wkernel32.pdb/ff9f9f7841db88f0cdeda9e1e9bff3b5A/wkernel32.pdb
  • kernel32.dll/590285e9e0000/kernel32.dll
  • libc-2.23.so/elf-buildid-b5381a457906d279073822a5ceb24c4bfef94ddb/libc-2.23.so
  • _.debug/elf-buildid-sym-b5381a457906d279073822a5ceb24c4bfef94ddb/_.debug
  • CoreFoundation/mach-uuid-36385a3a60d332dbbf55c6d8931a7aa6/CoreFoundation
  • _.dwarf/mach-uuid-sym-36385a3a60d332dbbf55c6d8931a7aa6/_.dwarf

SymStore
Path: <FileName>/<SIGNATURE><AGE>/<FileName>

The public symbol server provided by Microsoft used to only host PDBs for the Windows platform. These use a signature-age debug identifier in addition to the file name to locate symbols. File paths are identical to SSQP, except for the default casing rules:

  • Filenames are as given
  • The signature and age of a PDB identifier are uppercase.
  • The timestamp of a PE identifier is uppercase, but the size is lowercase.

Since the original Microsoft Symbol Server did not serve ELF or MachO files, we do not recommend using this convention for these types. However, Sentry will support the SSQP conventions with adapted casing rules when this layout is selected.

Examples:

  • wkernel32.pdb/FF9F9F7841DB88F0CDEDA9E1E9BFF3B5A/wkernel32.pdb
  • KERNEL32.dll/590285E9e0000/KERNEL32.dll

Index2
Path: <Fi>/<FileName>/<SIGNATURE><AGE>/<FileName>

This layout is identical to SymStore, except that the first two characters of the file name are prepended to the path as an additional folder.

Examples:

  • wk/wkernel32.pdb/FF9F9F7841DB88F0CDEDA9E1E9BFF3B5A/wkernel32.pdb
  • KE/KERNEL32.dll/590285E9e0000/KERNEL32.dll

Compression of Debug Files

Sentry supports the following compression methods when downloading debug information files from external sources: Gzip, zlib (both with and without header), Zstandard, and Cabinet (CAB).

The convention on Microsoft's Symbol Server protocol is to store such files with the last character of the file extension replaced with _. A full example would be: KERNEL32.dll/590285E9e0000/KERNEL32.dl_. This is not required on your own repositories, as Sentry detects compression on all paths.

Source Context

If Sentry has access to application source code, it can show snippets of code around the location of stack frames. Certain SDKs can resolve this source context automatically, such as the Python SDK, because they have access to unobfuscated source code at runtime.

To get source context for native applications, source code needs to be uploaded alongside the debug information files. The recommended way to do this is by using sentry-cli. See Creating Source Bundles for more information.

Source bundles show up as regular debug files on the Debug Files settings page. They are designated as "source bundle" and feature a sources tag. To match them with crash reports, they carry the same debug file as the respective debug information file they have been created from.

You can edit this page on GitHub.