Strange Behavior (including Segmentation Faults) when interacting over RPC Unix Socket

[ProofOfKeags]

Issue and Steps to Reproduce

1. Compiled commit hash d9b2482415a888d90e8d2a0f486bf96788e84b6f (master) for MacOS
2. Start lightningd with ./lightningd/lightningd
3. In separate terminal, attempt to send rpc request over the domain socket ~/.lightning/bitcoin/lightning-rpc in one of 5 ways

Situation 1 (Unevaluated newline literals appended to json rpc message)

echo '{ "jsonrpc": "2.0", "method": "getinfo", "params": [], "id": 0 }\n\n' | nc -v -U ~/.lightning/bitcoin/lightning-rpc

• This yields two json rpc responses to the single request:

{"jsonrpc":"2.0","id":null,"error":{"code":-32600,"message":"Invalid token in json input"} }

{"jsonrpc":"2.0","id":0,"result":{"id":"039410872e426eba0284e7b90b997701e767a0b2f8da9692efd874ebaad4b0007d","alias":"PEEVEDSPAWN","color":"039410","num_peers":0,"num_pending_channels":0,"num_active_channels":0,"num_inactive_channels":0,"address":[],"binding":[{"type":"ipv6","address":"::","port":9735}],"version":"v0.8.1-50-gd9b2482","blockheight":344820,"network":"bitcoin","msatoshi_fees_collected":0,"fees_collected_msat":"0msat","lightning-dir":"/Users/keagan/.lightning/bitcoin","warning_bitcoind_sync":"Bitcoind is not up-to-date with network."} }

• It also yields in the lightningd stdout log:

2020-03-04T21:42:49.021Z UNUSUAL jsonrpc#29: Invalid token in json input: '\n\n?'

Situation 2 (Evaluated newlines appended to json rpc message)

echo -e '{ "jsonrpc": "2.0", "method": "getinfo", "params": [], "id": 0 }\n\n' | nc -v -U ~/.lightning/bitcoin/lightning-rpc

• This yields a segmentation fault

Situation 3 (Unevaluated newline literals appended to json rpc message, No echo newline)

echo -n '{ "jsonrpc": "2.0", "method": "getinfo", "params": [], "id": 0 }\n\n' | nc -v -U ~/.lightning/bitcoin/lightning-rpc

• This yields identical behavior to Situation 1

Situation 4 (Evaluated newlines appended to json rpc message, No echo newline)

echo -ne '{ "jsonrpc": "2.0", "method": "getinfo", "params": [], "id": 0 }\n\n' | nc -v -U ~/.lightning/bitcoin/lightning-rpc

• This yields a segmentation fault

Situation 5 (No newlines at the end of json rpc message, No echo newline)

echo -n '{ "jsonrpc": "2.0", "method": "getinfo", "params": [], "id": 0 }' | nc -v -U ~/.lightning/bitcoin/lightning-rpc

• This yields segmentation fault

getinfo output

N/A, bug report is related to getting "getinfo"

Commentary

The closest I was able to get to behavior that was sane is with unevaluated newlines at the end of the message. This presumably pleases whatever system is reading messages off the socket, but is not pleasing the json parser, yet it still responds in a partially reasonable way: it gives me the response I was looking for, albeit with an erroneous extra complaint about the existence of newlines. The newline inserted by the echo command itself does not seem to make a difference in the behavior here. In every case, I would expect that misuse of the API would result in a json rpc error as opposed to segfault and termination of lightningd. Similarly, getting two responses for a single request seems like unexpected behavior as well.

[cdecker]

I'm wondering if it is something with the newlines and nc on Mac. I tried all 5 scenarios against the commit you mention, but all of them return correct results, no crashes and no extraneous newlines.

You can run lightningd using gdb as follows:

gdb lightningd -ex 'r your usual options'

It'll immediately start running, and interrupt as soon as it hits the segfault. Once it interrupts you can use bt to get a stack trace, up to move up in the stack, and p to print the contents of variables.

[ProofOfKeags]

So I don't typically use gdb, so I'm not super sure what "your usual options" are. I tried just 'r', and it basically just hangs, doesn't do anything and I cannot recover it without doing a Ctrl+Z. Any advice here?

[ProofOfKeags]

OK so I managed to get SOMETHING.

Here's the backtrace:

Thread 2 received signal SIGSEGV, Segmentation fault.
#0  0x000000010001057a in ?? ()
#1  0x0000000000000000 in ?? ()

Doesn't look promising though, except maybe a nullptr deref

[ProofOfKeags]

K it looks like I need to recompile with -g

[ProofOfKeags]

The Makefile is a bit over my head here, since it appears that you are using ncc as opposed to gcc. I need to recompile so that the symbols are not stripped from the final product. If you have any advice on how to do this, I'd appreciate the help. Thanks!

[Fonta1n3]

I am having the same issue, I expose the unix socket domain to a port to try and mimic Bitcoin Core's rpcport functionality to make http json-rpc post request, so that I can get the same sort of functionality with my c-lightning node over a hidden service as I do in Fully Noded with Bitcoin Core.

In swift I create a url request that looks something like:

request.httpMethod = "POST"
request.setValue("text/plain", forHTTPHeaderField: "Content-Type")
request.httpBody = "{\"jsonrpc\":\"2.0\",\"id\":\"1\",\"method\":\"getinfo",\"params\":[]}".data(using: .utf8)

The command that gets issued:
{"jsonrpc":"2.0","id":"1","method":"getinfo","params":[]}

The result in the lightningd log:

Invalid token in json input: 'POST / HTTP/1.1??Host: 7luqavfpkcdarq2c7wl4umlespbpgefgqmkpel6n7'

Where 7luqavfpkcdarq2c7wl4umlespbpgefgqmkpel6n7 is an abbreviated portion of the v3 hidden service's hostname I have set up to talk to the port controlling the domain socket.

I just want to talk to lightning-cli directly, remotely, over a hidden service with nothing else in between.

[ZmnSCPxj]

@Fonta1n3 We use unwrapped plaintext JSONRPC. It looks like your interface is using HTTP. You will need to remove the HTTP wrapping and issue JSONRPC requests directly. Note as well that JSONRPC allows multiple parallel requests.

Your choices are either to remove the HTTP wrapping at the client side, or add yet another server-side program that implements an HTTP server and unwraps the HTTP and issues the requests locally on the Unix socket.

[ZmnSCPxj]

Exposing the interface over the network without any kind of password or challenge is also very dangerous, as anyone who pushes JSONRPC requests at the port can abscond with all your Lightning Bitcoins. You are better off creating a proper server that requires some password or challenge, that wraps the unix socket.

Our assumption, and the reason we use a unix socket, is that you are in sole control of your hardware, and thus can open a unix socket, and thus we just trust whatever command is issued over the Unix RPC port. Exposing the socket over the network breaks this assumption.

[Fonta1n3]

@ZmnSCPxj thanks for the explanation. Completely understand the concern/risk of exposing the socket to network connections. In this use case though you can only expose it to localhost and use Tor V3 authentication which is very secure and easy for newbs like myself to configure, it feels like if Bitcoin Core allows that then so should lightning.

I have found a plugin thanks to @ProofOfKeags which should accomplish what I am after anyways, thank you for your time.