Remote debugging MacOS from Windows

[moomoohk]

Hi there,
I'm attempting to debug a process running a MacOS machine from my main Windows machine (where Ghidra is running).

On the Mac I've attached a debugger to the process using /Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/debugserver <windows ip>:<port> --attach=<mac process name>. This command seems to run fine and the debugger waits for incoming connections.

In the Ghidra debugger I open the new debugger target dialog, select the "LLVM lldb local agent via GADP/TCP" option, fill in the address+port details and hit "Connect".

At this point I receive the following error:

java.lang.RuntimeException: Agent terminated with error: java.library.path => C:\Program 
Files\OpenJDK\jdk-17\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files\Eclipse 
Adoptium\jdk-11.0.18.10-hotspot\bin;C:\Program Files (x86)\VMware\VMware Workstation\bin\;C:\Program Files\NVIDIA GPU 
Computing Toolkit\CUDA\v11.7\bin;C:\Program Files\NVIDIA GPU Computing 
Toolkit\CUDA\v11.7\libnvvp;C:\Python311\Scripts\;C:\Python311\;C:\Scripts;C:\Python310\Scripts\;C:\Python310\;C:\Py... 
agent.lldb.manager.impl.LldbManagerImpl.<init>(LldbManagerImpl.java:96)
	at agent.lldb.manager.LldbManager.newInstance(LldbManager.java:58)
	at agent.lldb.model.impl.LldbModelImpl.<init>(LldbModelImpl.java:70)
	at agent.lldb.gadp.impl.LldbGadpServerImpl.<init>(LldbGadpServerImpl.java:40)
	at agent.lldb.gadp.LldbGadpServer.newInstance(LldbGadpServer.java:55)
	at agent.lldb.gadp.LldbGadpServer$LldbRunner.run(LldbGadpServer.java:74)
	at agent.lldb.gadp.LldbGadpServer.main(LldbGadpServer.java:43)

---------------------------------------------------
Build Date: 2022-Nov-15 1249 EST
Ghidra Version: 10.2.2
Java Home: C:\Program Files\OpenJDK\jdk-17
JVM Version: Oracle Corporation 17
OS: Windows 10 10.0 amd64
Workstation: DESKTOP

The message is quite uninformative and even seems truncated such that I can't understand what the real issue is.

Help would be appreciated.

Thanks

[d-millar]

@moomoohk OK, so a couple of (hopefully) clarifying details up front: the host and port in "LLVM lldb local agent via GADP/TCP" are the host and port for connecting to the Ghidra agent wrapping the lldb interface (via JNI). They are NOT the host and port exported by debugserver. The use case requiring these entries is typically one in which you either (a) are worried about native methods crashing your Ghidra instance, or (b) debugging in an environment where lldb is not available locally. Have you built or imported lldb for Windows? If not, you'll need to either do so or run the agent a platform that supports lldb. Yell if this isn't clear. The host and port for debugserver should be specified using the standard way, i.e. "gdb-remote host:port" from the Interpreter window or use the options in "Connect to server..." (under Objects->Connectors) after launching either in IN-VM or GADP agents from Targets.

[d-millar]

Yes, you've got it. So, to elaborate:

1. Ghidra - currently on windows, probably good to go.
2. SWIG/JNI interface gets built as liblldb-java.so or .dylib typically. Theoretically, this could be built as a .dll, but I'm not sure I've ever actually tried that. (Maybe I will for fun!) This library gets loaded by the agent:
3. Ghidra agent is the portion of Ghidra that wraps native interfaces, either through JNA, JNI, or, in the case of gdb, the MI2 interface. It can run either in the Java VM hosting Ghidra (the IN-VM variants) or in a separate Java VM, running either locally or remotely, i.e. could be running on Windows or in the WSL or on some other box, such as the target Mac.
4. debugserver - runs remotely on the Mac
   Note that if you're running the agent on the Mac, you don't necessarily need the debugserver. Also, the debugserver config is relatively new, and a good many bugs are fixed in the source (and will be in the next release) but may not behave correctly in 10.2.2.

[d-millar]

OK, just verified I could do this all on Windows, i.e. build llvm-project, build lldb/liblldb.dll, build the SWIG interface, build lldb-java.dll, run the IN-VM agent from Windows (load the dlls), and debug notepad.exe. One tricky bit, the java.library.path is a little squirrelly. Needs to point to build/bin in the llvm-project and has to use semicolons as separators, e.g -Djava.library.path=C:\Users\me\git\llvm-project\build\bin;C:\Users\me\git\ghidra\Ghidra\Debug\Debugger-swig-lldb\build\os\win_x86_64

[moomoohk]

Hey D,
I think I'm really close to getting everything to work... I've borked my WSL so bad I had to reinstall it completely twice, numerous restarts, built LLVM on Windows 3 times (ran Ninja with infinite threads which wrecked my machine - oops), built Gradle, I don't even know if my computer will ever recover from this...

BUT -
I have lldb.exe compiled and running on Windows and in terms of the guide I'm stuck right at the final step of stage (3) ("gradle build"ing the JNI interface).

Here is the output from Gradle, I'm hoping you can help me make some sense of it:

> Configure project :
OS: Windows 10
Arch: amd64 (gradle: x86-64)
Using platform: win_x86_64
Searching for latest Visual Studio and required components...
  -> Installation Directory: C:\Program Files (x86)\Microsoft Visual Studio\2019\Community
  -> VCTools Version (default): 14.29.30133
  -> SDK Directory (default): C:\Program Files (x86)\Windows Kits\10\
  -> SDK Version (default): 10.0.19041.0
  -> SDK Version (override): N/A

> Task :Debugger-swig-lldb:linkMainWin_x86_64SharedLibrary
LINK : fatal error LNK1181: cannot open input file 'liblldb.lib'


> Task :Debugger-swig-lldb:compileJava FAILED
C:\workspace\java\ghidra\Ghidra\Debug\Debugger-swig-lldb\src\main\java\SWIG\TraceInstructionControlFlowType.java:15: error: cannot find symbol
  public final static TraceInstructionControlFlowType eTraceInstructionControlFlowTypeInstruction = new TraceInstructionControlFlowType("eTraceInstructionControlFlowTypeInstruction", lldbJNI.eTraceInstructionControlFlowTypeInstruction_get());
                                                                                                                                                                                              ^
  symbol:   method eTraceInstructionControlFlowTypeInstruction_get()
  location: class lldbJNI
C:\workspace\java\ghidra\Ghidra\Debug\Debugger-swig-lldb\src\main\java\SWIG\TraceInstructionControlFlowType.java:16: error: cannot find symbol
  public final static TraceInstructionControlFlowType eTraceInstructionControlFlowTypeBranch = new TraceInstructionControlFlowType("eTraceInstructionControlFlowTypeBranch", lldbJNI.eTraceInstructionControlFlowTypeBranch_get());
                                                                                                                                                                                    ^
  symbol:   method eTraceInstructionControlFlowTypeBranch_get()
  location: class lldbJNI
C:\workspace\java\ghidra\Ghidra\Debug\Debugger-swig-lldb\src\main\java\SWIG\TraceInstructionControlFlowType.java:17: error: cannot find symbol
  public final static TraceInstructionControlFlowType eTraceInstructionControlFlowTypeTakenBranch = new TraceInstructionControlFlowType("eTraceInstructionControlFlowTypeTakenBranch", lldbJNI.eTraceInstructionControlFlowTypeTakenBranch_get());
                                                                                                                                                                                              ^
  symbol:   method eTraceInstructionControlFlowTypeTakenBranch_get()
  location: class lldbJNI
C:\workspace\java\ghidra\Ghidra\Debug\Debugger-swig-lldb\src\main\java\SWIG\TraceInstructionControlFlowType.java:18: error: cannot find symbol
  public final static TraceInstructionControlFlowType eTraceInstructionControlFlowTypeCall = new TraceInstructionControlFlowType("eTraceInstructionControlFlowTypeCall", lldbJNI.eTraceInstructionControlFlowTypeCall_get());
                                                                                                                                                                                ^
  symbol:   method eTraceInstructionControlFlowTypeCall_get()
  location: class lldbJNI
C:\workspace\java\ghidra\Ghidra\Debug\Debugger-swig-lldb\src\main\java\SWIG\TraceInstructionControlFlowType.java:19: error: cannot find symbol
  public final static TraceInstructionControlFlowType eTraceInstructionControlFlowTypeReturn = new TraceInstructionControlFlowType("eTraceInstructionControlFlowTypeReturn", lldbJNI.eTraceInstructionControlFlowTypeReturn_get());
                                                                                                                                                                                    ^
  symbol:   method eTraceInstructionControlFlowTypeReturn_get()
  location: class lldbJNI
5 errors

> Task :Debugger-swig-lldb:linkMainWin_x86_64SharedLibrary FAILED

FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':Debugger-swig-lldb:compileJava'.
> Compilation failed; see the compiler error output for details.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':Debugger-swig-lldb:linkMainWin_x86_64SharedLibrary'.
> A build operation failed.
      Linker failed while linking lldb-java.dll.
  See the complete log at: file:///C:/workspace/java/ghidra/Ghidra/Debug/Debugger-swig-lldb/build/tmp/linkMainWin_x86_64SharedLibrary/output.txt

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.6/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 12s
23 actionable tasks: 3 executed, 20 up-to-date

Envars:

LLVM_BUILD=C:\workspace\c\lldb-build\bin
LLVM_HOME=C:\workspace\c\llvm-project
JAVA_HOME=C:\Program Files\OpenJDK\jdk-17

LLVM_BUILD contains lldb.exe and liblldb.dll.
LLVM_HOME is the root of the LLVM source repo.

LLDB version:

lldb.exe -v
lldb version 17.0.0git (https://github.com/llvm/llvm-project.git revision 9bd58f6f051c9b5546bbaa891fa11d3624b8d9a4)
  clang revision 9bd58f6f051c9b5546bbaa891fa11d3624b8d9a4
  llvm revision 9bd58f6f051c9b5546bbaa891fa11d3624b8d9a4

---

(I'm trying to resuscitate my WSL in parallel, I'll try running through the steps on there as well. But I feel so close on pure Windows...)

[moomoohk]

The issue was with step (2b): I first deleted src\main then copied in the data from build/generated/src/main (instead of copying it in straight away overwriting some files and leaving extra junk).

After that gradle build still errors after a few seconds because of:

> Task :Debugger-swig-lldb:linkMainWin_x86_64SharedLibrary
LINK : fatal error LNK1181: cannot open input file 'liblldb.lib'

None of the envars actually seem to point to liblldb.lib which is in C:\workspace\c\lldb-build\lib (not to be confused with C:\workspace\c\lldb-build\bin which contains lldb.dll).

So I tried updated the LIBPATH envar to point at the lib directory:

$env:LIBPATH="$env:LIBPATH;C:\workspace\c\lldb-build\lib"

No dice, Gradle is still outputting the same message (LNK1181). However, it seems to keep running and apparently goes through all tasks before reporting the build failed after about 4 minutes:

> Task :Debugger-swig-lldb:linkMainWin_x86_64SharedLibrary FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':Debugger-swig-lldb:linkMainWin_x86_64SharedLibrary'.
> A build operation failed.
      Linker failed while linking lldb-java.dll.
  See the complete log at: file:///C:/workspace/java/ghidra/Ghidra/Debug/Debugger-swig-lldb/build/tmp/linkMainWin_x86_64SharedLibrary/output.txt
   > Linker failed while linking lldb-java.dll.

So that's where I'm at currently...

[d-millar]

I'm not positive this is the issue, but it might shed light on the actual issue. We normally build against llvm/lldb version 14. I have not tried building against 17. I will do so, but, in the meantime, try:
(1) from llvm-project, "git checkout release/14.x" then "ninja lldb"
(2) from Ghidra/Debug/Debugger-swig-lldb, "gradle generateSwig", "cp -r build/generated/src/main src", "gradle build"

[moomoohk]

Well I can't believe it...
Copying liblldb.lib into my current directory solved the linking issue and Gradle actually succeeded. It's not elegant but it works :)
Added the relevant java.library.path line to the launch file and it looks like I can add an IN-VM LLDB target on my Windows machine!

I guess the final tip I need is how to connect to the debugserver running on my Mac.
Thanks

[d-millar]

Awesome - that's very exciting, and, yes, am already figuring out how to upgrade both our SWIG versions and documentation to reflect what you've worked through. Greatly appreciated!

[moomoohk]

Glad to hear!

Now that I have my session set up I think it's worth mentioning that nearly every tree operation results in a timeout exception (expanding threads node, etc.)

Is there anything I can do improve the stability of the connection between the agent and the remote machine?

[d-millar]

The timeouts usually happen if the debuggee is failing to respond. Any chance the target is still running?

[d-millar]

Also, was "Sync" on or off when you connected? "Sync" on should pause until the debugger gets a notification from the debuggee that the state has changed, typically stopped. With "Sync" off, the debugger assumes the target is stopped and tries to update immediately.

[moomoohk]

The timeouts usually happen if the debuggee is failing to respond. Any chance the target is still running?

Yes I've verified and debugserver is still listening.

You can also up the timeout value in Objects.

I assume under Maintenance > Set Node Timeout: I've upped it to 120 for these next tests.

Also, was "Sync" on or off when you connected? "Sync" on should pause until the debugger gets a notification from the debuggee that the state has changed, typically stopped. With "Sync" off, the debugger assumes the target is stopped and tries to update immediately.

I only see "Async" not "Sync" so I'll assume you meant that.

Connection attempt with Auto checked and Async checked:

1. debugserver: Acknowledges connection - "Waiting for debugger instructions for process 0"
2. Ghidra: Nothing seems to happen initially. No node loading progress bar, no output in debug console. I purposefully didn't expand the Sessions node to prevent interference with connection process.
3. Expanding the Sessions node finally reveals an "In Progress..." node and hourglass progress bar appears.
4. Progress stays at 0% and after 120 seconds I get a Timeout Exception.
5. Refreshing the Sessions node reveals the progress bar again which stays on 100% for 120 seconds then Timeout Exception.
6. Few seconds later the debugee process node magically appears. Modules, threads, etc. seem to load as well.
7. Debug console also outputs some messages which seem in order.

So I think the issue is that things are updating in the background, taking ages and not being properly reflected in the GUI. Also they time out a lot...

I'm biting the bullet and trying to build the agent on the Mac in case debugserver is the issue.

Any other insights?

[d-millar]

You can also up the timeout value in Objects.

[d-millar]

@moomoohk sigh really sorry this has been such a struggle....

Am guessing (and definitely guessing here) that there's a significant mismatch between the debugserver version and the liblldb version. I would:
(1) Run "debugserver --version" to get that info.
(2) Make sure the IP you're using makes sense. For example, I've debugged some devices using the local IP port forwarded to the external IP. In some cases, you cannot use 127.0.0.1 or localhost, and have to use 0.0.0.0.
(3) Run Ghidra using "support/ghidraDebug" instead of "ghidraRun" (or run it from Eclipse) and make note of anything that shows up in either the GUI Debug Console or in the shell where you invoked ghidraDebug when you connect.
(4) Unselect Async (yes, not Sync - sorry about that) and Auto, connect, and run "gdb-remote ip:port". Certain things may not work so well using this approach, but you may get more explicit info in the Interpreter window.

Not sure what your target is. I have not seen "Waiting for debugger instructions for process 0". That sounds like the initial connect succeeded, but hard to tell whether the target process has actually broken. Unfortunately, for some targets, the debugger may not send any messages to the debugger on connection, so the debugger doesn't know whether the target is stopped or running. As mentioned, if the process is running (for most targets), any requests made of the debuggee will timeout. Normally, sending an interrupt request will resolve the situation, but interrupt requests are often sent out-of-band and are notoriously difficult to implement consistently. In Ghidra, there are three ways to send the interrupt (from the Objects toolbar, the Interpreter toolbar, or Ctrl-I), but, if one fails, it's likely all will fail.

One other thing I would do (although am guessing you already have) is make sure lldb.exe connects correctly without involving Ghidra in the mix.

[moomoohk]

Hey no worries. I accept that Ghidra is still relatively young and some edges are gonna be rough. I just hope this process will be easier for people in the future :)

1. The debugserver I was using came packaged with an ancient version of XCode which was indeed old. I spent the day compiling the latest version of lldb (no errors, just took hours to run!!! at least 3...) on the Mac so I went ahead and compiled the lldb-server as well. Now both lldb components on the Windows and on the Mac are up to date (~v17).
2. Both machines are on the same LAN, no NAT separating them or anything like that.
3. Tried it and got the same errors and warnings that I did in the GUI debug console (there are some other debug messages that don't mean much to me).
4. I tried running without async however things still don't seem to update properly and timeout errors keep popping up. Additionally, the Interpreter pane isn't available for me until a connection is actually successfully made so I don't have anywhere to input "gdb-remote".

• The "Waiting for debugger instructions for process 0" message indeed indicates a successful connection on the old version of debugserver. It's different on the new version of lldb-server. Anyways, I get your tip about the ambiguous state. The toolbar automatically assumes the target is broken so pause is disabled and clicking play does nothing. The Interpreter pane has "No executable module" so clicking pause there does nothing. Same for ctrl+I.

Other things I've tried:

• GDB over SSH: Doesn't seem to be fully implemented on MacOS.
• Compiled a new version of debugserver (doesn't appear to be the same as lldb-server): Same behavior as the old version. Manages to connect but keeps timing out and "lags".
• Compiling the agent on Mac: The final gradle build command doesn't create the expected file despite succeeding. The buid/os folder is missing... Checked the envars and fiddled with them to no avail.

It's starting to look like my only option is building Ghidra on the Mac and just working from there...

[moomoohk]

I didn't follow up on your suggestion to connect lldb directly without Ghidra.

Ran debugserver same as always (with the --attach flag) on the Mac and gdb-remote <host>:<port> through lldb on the Windows. Everything seemed to work just fine straight away. It's worth mentioning that a symbol load was immediately initiated and took a very long time (way over, say, 120 seconds). The interpreter wasn't responsive during this stage although I could see the work being done.

This could explain some of the behavior I was experiencing in Ghidra. If symbols do indeed start loading immediately, there's nothing to reflect it in the GUI... I'll try upping the timeout much much more and see if that does anything.

[d-millar]

OK, maybe re symbol load. Some notes on your notes:

Re symbol delays, might be worth running debugserver with an actual but small executable, e.g. /usr/bin/xclock or nano. Also, have you tried debugging something locally on the Windows box (am guessing so) but....like using lldb to run notepad.exe?

Re 1, am still a little worried about some of the errors you got in the generateSwig/buildGhidra portion early on and the fact that I haven't tested the current version of lldb. It's a pain but might be still worth trying release/14.x.

Re 3, probably a lot but feel free to dump a chunk of error messages in the stream here - might be something less than obvious that we would pick up on.

Re 4, to get the interpreter w/o loading any objects in the tree, you'll want BOTH Auto & Async off. Auto basically executes the LLDB API equivalent of "gdb-remote". That might allow you to try the Ctrl-I or break from the Interpreter.

Re GDB over SSH, lots of folks misinterpret what this means (again docs may need work). Basically, this is going to launch an ssh session with a cmd option equal the path to gdb you specify. So, if you specify 127.0.0.1, the local box must have an sshd daemon running and most have gdb at the location specified. Theoretically, you can also connect gdb to debugserver/lldb-server (they share the same protocol) but the results are less than satisfying in my experience.

Re compiling the agent on Mac, I'd be interested in any error messages there as well. Is the Mac an Intel or Mx processor?

Re building on the Mac, the Ghidra part is straighforward but the lldb part less so. Checkout the Discussion #4937 so you know what you're up against. Signing and dealing with the the SIP are always a bit of a drag.

[moomoohk]

Hey

I probably should have initially tried with a smaller binary... But indeed after setting the timeout to 20 minutes I did some other things while letting Ghidra (presumably) load symbols. When I came back after some time everything seemed to be working perfectly... Threads, modules, tree nodes, etc. were all loaded and responsive.

I think my main takeaway and suggestion for improvement is that there should be better indication in the GUI that something is happening in the background.

Re Re 1: I don't think the problem had to do with the newer version evidenced by everything working as intended. I think it's worth noting that debugserver was what ended up working for me, lldb-server was giving me some issues. I'm not sure what the differences are between them but there you go.

Re Re 3: I closed that window ages ago, sorry... I don't recall seeing anything indicative about a process running in the background gumming up the works, or anything about symbols.

Re Re 4: Great tip. Thanks.

Re GDB over SSH: I actually got SSH to work fairly easily (enabling sshd on Macs is apparently super easy) but I'm working on a system process which requires super perms to attach to. This was probably the issue here. Since the SSH session starts in a user context it just wasn't possible to use this channel. Maybe consider piping in a super password to sudo somehow? I guess it's not that critical.

Re agent on Mac: Intel processor. The guide does include a troubleshooting tip in case the final binary doesn't show up however it didn't help (like I wrote above, played around with envars and nothing changed). Otherwise got no errors.

Re Ghidra on Mac: LLDB was actually a breeze to compile (especially after the ordeal of compiling for Windows...). Ghidra was giving me some errors but since I seem to have gotten remote debugging to work, I don't have a reason to keep investigating.

All in all, this was one hell of a nut to crack but we got there in the end :)
I'll be sure to report anything else I see relevant.
Thanks for being so responsive and helpful!
Have a great week.

[d-millar]

Wow, OK - glad to hear things panned out (eventually). I will take the "better indication in the GUI" as an action item. I think we also have an issue on the books regarding limiting what gets loaded by default, which, if nothing else would have helped in trying to isolate the problem.

Really appreciate your hanging in there, trying stuff out, and giving us feedback. Will definitely help folks down the line!

[moomoohk]

Hey,
Popping back in to point out this page which mentions a symbol caching feature in LLDB: https://reviews.llvm.org/D115324?id=394656

These settings seem extremely relevant to my issues with long symbol-loading times:

(lldb) settings set symbols.enable-lldb-index-cache true
(lldb) settings set symbols.load-on-demand true
(lldb)settings set symbols.enable-lldb-index-cache 0

Caches go here:

(lldb) settings show symbols.lldb-index-cache-path

[d-millar]

Didn't have a chance to play around with these today, but very interesting. Thanks!