What is a Kernel Panic ?
In this blog, we will be talking about analyzing iOS Kernel panic logs. A kernel panic occurs when the operating system kernel encounters a fatal error. This error is so severe that the kernel cannot continue operating or safely recover the system to a stable state. By default, the iOS kernel reboots in the event of a panic.
Common causes of kernel panics include:
- Faulty or incompatible hardware.
- Software bugs in the kernel or system-level components.
- Malfunctioning kernel extensions.
- Memory corruption or overflows.
Oftentimes, security researchers look at the Kernel panic logs to identify whether the Panic happened due to a vulnerability.
Extracting Kernel Panic Logs
In an iOS device. Panics can be identified by going to (Settings -> Privacy – > Analytics), or in the filesystem by going to (/private/var/mobile/Library/Logs/CrashReporter) if you have a jailbroken iOS device. In some cases, there might be files inside the CrashReporter folder so it’s always a nice idea to do a recursive search for the panic log.
Kernel panic logs follow a specific naming convention panic-full-yyyy-mm-dd-HHMMSS.000.ips. Since the same folder also contains userland crash logs, this naming convention helps identify a Kernel panic log from a userland crash log.
In the image below, you can see one of the Kernel panic logs present on a Jailbroken iOS device.
And this is how a Kernel Panic log looks like. In this blog we will try and make some sense out of it.
It is also possible to simply extract the Kernel panic log file using tools like idevicecrashreport from https://libimobiledevice.org/.
Additionally, one can also extract the Panic logs from the device using Xcode by connecting your device to your computer and ensuring that it’s recognized by Xcode, then going to Window -> Devices and Simulators -> Open Recent Logs
This will take you to Finder where it has extracted all the Logs and you can identify the Kernel Panic Logs using the same naming convention panic-full-yyyy-mm-dd-HHMMSS.000.ips
Now that we have looked at how we can extract a Kernel Panic log from the device, lets try and make sense out of the Panic file.
The Panic String
Within a kernel panic file, a panic string refers to a specific line of text that describes the reason for the panic. It is typically a concise error message or a code identifier that provides a clue about what went wrong in the kernel.
When examining a kernel panic file, locating and analyzing the panic string is often one of the first steps in diagnosing the problem. It can indicate issues such as memory corruption, hardware failures, software bugs, or incompatible drivers. By understanding the panic string and its associated context, researchers can narrow down the potential causes and work towards resolving the underlying problem. As you can see even from the XNU source code, the panic() function takes a string as an argument which is the Panic string.
For e.g in the image below from the Panic file you will see the Panic string present inside the Panic file.
If you are lucky enough, you might find the panic string present within the XNU kernel code itself that is shipped on opensource.apple.com, in that case, you can find the exact code which is causing the problem and this makes it easier to perform the analysis.
In case they are not present, simply opening the kernelcache and cross referencing the panic string can give you an idea of what went wrong, since the strings are not obfuscated inside the iOS kernelcache.
Simplifying the Panic file
The following script named panicparser.py https://gist.github.com/PsychoTea/d9ca14d2687890f15900d901f600bf6a can be used to convert the Kernel Panic file into a more readable format.
The panic string is visible in the very first line after the command.
Some of the notable things are
Kernel version: Darwin Kernel Version 22.0.0: Tue Aug 16 20:52:01 PDT 2022; root:xnu-8792.2.11.0.1~1/RELEASE_ARM64_T8101
Kernel UUID: 72C7327A-D999-3613-8F7B-C86E542122D3
KernelCache slide: 0x00000000266bc000
KernelCache base: 0xfffffff02d6c0000
Kernel slide: 0x0000000027314000
Kernel text base: 0xfffffff02e318000
Kernel text exec slide: 0x00000000273cc000
The inclusion of the kernel UUID enables Apple to locate the kernel and automatically symbolicate it. ٌ
You can also see the task responsible for the panic and the backtrace of the thread in the output.
Panicked Task:
- Task: 0xffffffecfe770c50 refers to the memory address of the task structure.
- Threads: 1 indicates that the task had only one active thread.
- Process: pid 395: xpcproxy points to the process ID (395) and the name of the task, xpcproxy, which is commonly used in iOS for managing interprocess communication.
Panicked Thread:
- Thread Address: 0xffffffede1884340 indicates the memory address of the panicked thread.
- Backtrace: 0xffffffe882d83320 is the starting memory address of the stack trace for the panicked thread.
- Thread ID: 7480 refers to the unique identifier for this thread.
Backtrace Details
The backtrace provides a list of function calls (return addresses) leading up to the kernel panic. Each line includes:
- lr (Link Register):
- Refers to the return address of a function call (the instruction pointer after the function call).
- Example: lr: 0xfffffff007107ef8.
- fp (Frame Pointer):
- Refers to the base address of the current stack frame.
- Example: fp: 0xffffffe882d83360.
- Sequence:
- Each step in the backtrace corresponds to a specific function or kernel routine that was executed before the panic occurred.
The addresses in the backtrace are all slid. To convert the backtrace to actual addresses, we need to subtract the kernel slide value from the provided addresses. The Kernel Slide value is 0x0000000027314000 from the panic log.
For each backtrace address (denoted as lr):
Actual Address = Address in Backtrace – Kernel Slide
For e.g Using the provided kernel slide (0x0000000027314000), here is the converted backtrace:
- lr: 0xfffffff007107ef8
0xfffffff007107ef8 – 0x0000000027314000 = 0xfffffff0043fc4f8
It is also worth noting that the addresses starting with 0xffffff7f… typically belong to kernel extensions (kexts), which are third-party or system-provided modules. If these appear in the backtrace, the issue might relate to the corresponding kext. However. In this case, this is not the case so it seems like the issue is most likely related to the main Kernel image.
Symbolicating a Kernel Panic File
Even though we have the addresses it would be good to put symbols to those addresses. One of the ways to do that is to symbolicate the Kernel Panic file. There are several tools that can be used to symbolicate the Panic file however the one that we are going to use is ipsw (https://github.com/blacktop/ipsw).
The first thing though would be to download the corresponding ipsw file for that particular Device and iOS version. The details of the iOS version and Product Version for a device can be found by using ideviceinfo.
One can then go to ipsw.me and download the corresponding ipsw file from ipsw.me
Once the ipsw is downloaded, we are now ready to use ipsw to symbolicate the Kernel panic file. As we can see from the help options, we need the ipsw file and the corresponding Panic file for it.
So lets go ahead and run the Kernelcache symbolication command
And we can now see the symbols in the userland mode. Another option would be to run the same command as above with the –unslide option to ensure addresses are converted to actual static addresses that can be identified in the Kernelcache.
Identifying the Root Cause
When trying to identify the root cause of the issue, let’s start with the Panic string.
pmap_mark_page_as_ppl_page_internal: page still has mappings, pa=0x8bf880000 @pmap_data.c:1101
Looking in the XNU source code and inside the function pmap_mark_page_as_ppl_page_internal we find the panic string on Line #1092 in an XNU version that is closer to the one in iOS (Note that iOS XNU versions are not open source) that matches with the Panic string that we see in the Log file. As mentioned from the comments and googling a bit and learning from https://github.com/felix-pb/kfd/blob/main/writeups/exploiting-puafs.md , we can conclude that this is most likely due to an exploit failure where the given page that is supposed to be used is mapped outside of the physical aperture of PPL, meaning that a freed page supposed to be allocated for PPL has mapping outside of the PPL physical aperture. This is verified by the function and in case of failure a panic is triggered. This issue occurs during an exploitation of an exploit that uses a PUAF primitive to ensure a virtual address page from userland has no mappings to physical address even though the physical address has now been marked as ready for use by the Kernel, and in this case, the issue occurs due to an additional check by PPL based code that checks whether a physical page mapped for PPL has mappings outside of the physical page aperture for it.
If you are interested in learning more about this PUAF primitive, check out our “Offensive iOS Internals” course at 8ksec Academy.
Ofcourse in this case we were lucky that the panic string was identified in the XNU code base, however this might not be the case always and one must try and identify the string in the Kernelcache and cross-reference it to find the place where the crash occured.
Conclusion
Analyzing an iOS kernel panic log is an essential skill for diagnosing and resolving system-level issues, especially for security researchers and developers. Kernel panics can stem from a wide range of causes, including faulty hardware, software bugs, memory corruption, or malfunctioning kernel extensions. By carefully extracting and analyzing the logs, you can gain valuable insights into the root cause of the Kernel panic.
This blog walked you through the process of:
- Extracting kernel panic logs from an iOS device using tools like Xcode, idevicecrashreport, or directly from the filesystem.
- Identifying the panic string, which provides the initial clue about the nature of the crash.
- Simplifying and symbolizing the panic log using scripts like panicparser.py and tools like ipsw, which help make the logs more readable and actionable.
- Interpreting key details, such as the panicked task, thread backtrace, and slid addresses, to locate the issue in the kernel or kernel extensions.
- Symbolicating kernel addresses to map them to specific functions or routines, enabling deeper analysis.
- Identifying the Root Cause of the Panic using further analysis.
For security researchers, kernel panic logs are particularly useful for identifying potential vulnerabilities. By analyzing patterns in crashes and backtraces, you may uncover subtle issues that could lead to security exploits or performance bottlenecks.
Kernel panic analysis can seem daunting at first, but with the right tools and methodology, it becomes a powerful way to understand and resolve low-level system issues. We hope this blog has provided you with a solid foundation to start investigating kernel panics effectively.
Looking to elevate your expertise in iOS Security?
Offensive iOS Internals Training
365 Days of Access | Hands-On Learning | Self-Paced Training
Explore Our On-Demand Courses
If you’re interested in diving deeper into topics like kernel panic analysis, vulnerability research, and low-level system debugging, 8ksec Academy offers a wide range of on-demand courses tailored for security professionals and enthusiasts.
Visit academy.8ksec.io to explore our full catalog of courses. Whether you’re a beginner or an experienced professional, you’ll find resources to enhance your skills and stay ahead in the fast-evolving field of Mobile Security.
Feel free to reach out to us at support@ to ask any questions related to our blogs or any future blogs that you would like to see.
Have a great day !
