Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix TLS proxy scenario (#802) #804

Closed
wants to merge 15 commits into from
Closed

Conversation

wolfkor
Copy link

@wolfkor wolfkor commented Apr 2, 2025

Fix for #802

{
}
}
internal sealed class SocketClosedException(Exception? innerException) : Exception("Socket has been closed.", innerException);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please rollback all the formatting changes. makes the review quite difficult. thanks.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hope it is easier now

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you

#else
await _socket.ConnectAsync(host, port, cts.Token).ConfigureAwait(false);
await _socket.ConnectAsync(proxyOpts.Host, proxyOpts.Port, cts.Token).ConfigureAwait(false);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if this is giving the ability to swap out the host:port on top of the callback, can you also use OnConnectingAsync to take care of this swap instead?

public Func<(string Host, int Port), ValueTask<(string Host, int Port)>>? OnConnectingAsync { get; set; }

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't OnConnectingAsync intended for swaping the bus address? Why would it be needed for the proxy address as well?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just trying to understand if we can use OnConnectingAsync + OnSocketThing instead of this change.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, because OnConnecting would require the socket for the proxy, which is created afterward. Furthermore, you would also have to consider the WebSocket case.
I even considered deleting OnSocketAvailable again. But I wasn't sure if there were other scenarios besides proxy where it would make sense.

{
// Create the CONNECT request with proxy authentication
var auth = Convert.ToBase64String(Encoding.UTF8.GetBytes(proxyOpts.Auth));
connectAuth = $"Proxy-Authorization: Basic {auth}\r\n";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

other auth methods? Would there be other headers we need to think of?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've thought about that, too. In my experience with our customers so far, I haven't encountered any other options, but there are certainly other proxy authentication options. Do you have a complete list of the methods to be supported here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't. my concern is how we'd support this going forward. maybe we can have a callback instead?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be possible. But I think every methodology has its own fixed format. For additional methods, I would extend properties in NatsProxyOpts. Do you want to leave that entirely up to the user?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure. what's your take? I think if we can support it with reasonable confidence just setting up the proxy host and port would be easy for the application. then the edge cases would require a mew release. (btw we must think of the security implications as well)

// Validate proxy response
var receiveBuffer = new byte[4096];
var read = await ReceiveAsync(receiveBuffer).ConfigureAwait(false);
var response = Encoding.UTF8.GetString(receiveBuffer, 0, read);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this needs to read the response until whatever. can't rely on a single read which might not return the whole response.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really understand what you're getting at? Isn't the code almost identical to your final example for OnSocketAvailableAsync from #647?

Copy link
Member

@mtmk mtmk Apr 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that was just an example. you need to check the read count and read more if you need more etc. e.g. you might only get part of the message 200 Connec first time, then the next.

wolfkor added a commit to wolfkor/nats.net that referenced this pull request Apr 3, 2025
@caleblloyd
Copy link
Collaborator

What protocols are you trying to proxy:

  • NATS Protocol - Unencrypted
  • NATS Protocol - default Opportunistic TLS - TlsMode.Require
  • NATS Protocol - Implicit TLS (sometimes called TLS Handshake First) - TlsMode.Implicit
  • WebSocket - Unencrypted
  • WebSocket - TLS

And what kind of proxy are you trying to use?

  • SOCKS Proxy
  • HTTP Proxy
  • Something else?

}

var serverWithPort = $"{host}:{port}";
var connectBuffer = Encoding.UTF8.GetBytes($"CONNECT {serverWithPort} HTTP/1.1\r\nHost: {serverWithPort}\r\n{connectAuth}Proxy-Connection: Keep-Alive\r\n\r\n");
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am confused here, this appears to be an HTTP Proxy? Aren't HTTP Proxies only for proxying HTTP(S) connections? But this is in the TcpConnection class which is used for the NATS protocol, which is not a HTTP-based protocol.

For WebSockets, there is NatsWebSocketOpts.ConfigureClientWebSocketOptions which could be used to set the ClientWebSocketOptions.Proxy property to a WebProxy

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see that HTTP CONNECT can work with protocols other than HTTP - MDN docs

Aside from enabling secure access to websites behind proxies, a HTTP tunnel provides a way to allow traffic that would otherwise be restricted (SSH or FTP) over the HTTP(S) protocol.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our current problem to solve for the customer environment is indeed NATS protocol Unencrypted or default Opportunistic TLS with HTTP proxy. The current code solves it

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm concerned we won't be able to support the proxy ourselves giving it the attention it deserves with all the different scenarios. Can you instead implement a IScocketConnection factory like suggested below? we can add that to NatsConnection class switching between the blow two blocks of code:

if (uri.IsWebSocket)
{
var conn = new WebSocketConnection();
await conn.ConnectAsync(uri, Opts).ConfigureAwait(false);
_socket = conn;
}
else
{
var conn = new TcpConnection(_logger);
await conn.ConnectAsync(target.Host, target.Port, Opts).ConfigureAwait(false);
_socket = conn;
if (Opts.TlsOpts.EffectiveMode(uri) == TlsMode.Implicit)
{
// upgrade TcpConnection to SslConnection
var sslConnection = conn.UpgradeToSslStreamConnection(Opts.TlsOpts);
await sslConnection.AuthenticateAsClientAsync(uri, Opts.ConnectTimeout).ConfigureAwait(false);
_socket = sslConnection;
}
}

if (url.IsWebSocket)
{
_logger.LogDebug(NatsLogEvents.Connection, "Trying to reconnect using WebSocket {Url} [{ReconnectCount}]", url, reconnectCount);
var conn = new WebSocketConnection();
await conn.ConnectAsync(url, Opts).ConfigureAwait(false);
_socket = conn;
}
else
{
_logger.LogDebug(NatsLogEvents.Connection, "Trying to reconnect using TCP {Url} [{ReconnectCount}]", url, reconnectCount);
var conn = new TcpConnection(_logger);
await conn.ConnectAsync(url.Host, url.Port, Opts).ConfigureAwait(false);
_socket = conn;
if (Opts.TlsOpts.EffectiveMode(url) == TlsMode.Implicit)
{
// upgrade TcpConnection to SslConnection
_logger.LogDebug(NatsLogEvents.Connection, "Trying to reconnect and upgrading to TLS {Url} [{ReconnectCount}]", url, reconnectCount);
var sslConnection = conn.UpgradeToSslStreamConnection(Opts.TlsOpts);
await sslConnection.AuthenticateAsClientAsync(FixTlsHost(url), Opts.ConnectTimeout).ConfigureAwait(false);
_socket = sslConnection;
}
}

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm very busy at the moment - but hope to do it by the end of the week.

@caleblloyd
Copy link
Collaborator

I think a more flexible API may be to allow for specifying a SocketConnectionFactory in NatsOpts that has a signature with something like:

Func<NatsUri, NatsOpts, CancellationToken, ValueTask<ISocketConnection>>? SocketConnectionFactory

And that would be responsible for returning an opened ISocketConnection

@mtmk
Copy link
Member

mtmk commented Apr 10, 2025

cc @johannespfeiffer

wolfkor added a commit to wolfkor/nats.net that referenced this pull request Apr 11, 2025
@wolfkor
Copy link
Author

wolfkor commented Apr 11, 2025

For our scenario i had to add

public class ProxyTcpConnection(ILogger<NatsConnection> logger) : TcpConnection(logger)
{
    public new async ValueTask ConnectAsync(string host, int port, TimeSpan timeout)
    {
        SocketProxy proxy = SystemUtilsService.SocketProxy;

        using CancellationTokenSource cts = new CancellationTokenSource(timeout);
        try
        {
            if (proxy.Proxy != null)
            {
                await Socket.ConnectAsync(proxy.ProxyHost, proxy.ProxyPort, cts.Token).ConfigureAwait(false);

                string connectAuth = null;
                if (proxy.ProxyAuth != null)
                {
                    // Create the CONNECT request with proxy authentication
                    string auth = Convert.ToBase64String(Encoding.UTF8.GetBytes(proxy.ProxyAuth));
                    connectAuth = $"Proxy-Authorization: Basic {auth}\r\n";
                }

                string serverWithPort = $"{host}:{port}";
                byte[] connectBuffer = Encoding.UTF8.GetBytes(
                    $"CONNECT {serverWithPort} HTTP/1.1\r\nHost: {serverWithPort}\r\n{connectAuth}Proxy-Connection: Keep-Alive\r\n\r\n");

                // Send CONNECT request to proxy
                await SendAsync(connectBuffer).ConfigureAwait(false);

                // Validate proxy response
                byte[]        receiveBuffer   = new byte[4096];
                StringBuilder responseBuilder = new StringBuilder();
                int           read;
                do
                {
                    read = await ReceiveAsync(receiveBuffer).ConfigureAwait(false);
                    responseBuilder.Append(Encoding.UTF8.GetString(receiveBuffer, 0, read));
                } while (read > 0 && !responseBuilder.ToString().Contains("\r\n\r\n"));

                string response = responseBuilder.ToString();
                if (!response.Contains("200 Connection established"))
                    throw new Exception($"Proxy connection failed. Response: {response}");
            }
            else
                await Socket.ConnectAsync(host, port, cts.Token).ConfigureAwait(false);
        }
        catch (Exception ex)
        {
            await DisposeAsync().ConfigureAwait(false);
            if (ex is OperationCanceledException)
                throw new SocketException(10060); // 10060 = connection timeout.

            throw;
        }
    }
}

and

public class BusSocketConnectionFactory : DefaultSocketConnectionFactory, ISocketConnectionFactory
{
    public new static       BusSocketConnectionFactory       Instance => LazyInstance.Value;
    private static readonly Lazy<BusSocketConnectionFactory> LazyInstance = new(() => new BusSocketConnectionFactory());

    public new async Task<ISocketConnection> OnConnectionAsync(NatsUri uri, NatsConnection natsConnection, ILogger<NatsConnection> logger)
    {
        (string Host, int Port) target = await BeforeConnectingAsync(natsConnection, logger, (uri.Host, uri.Port));

        logger.LogInformation(NatsLogEvents.Connection, $"Try to connect NATS '{uri}' with {nameof(BusSocketConnectionFactory)}");

        ProxyTcpConnection connection = new ProxyTcpConnection(logger);
        await connection.ConnectAsync(target.Host, target.Port, natsConnection.Opts.ConnectTimeout).ConfigureAwait(false);
        return connection;
    }

    public new async Task<ISocketConnection> OnReconnectionAsync(NatsUri uri, NatsConnection natsConnection, ILogger<NatsConnection> logger, int reconnectCount)
    {
        if (natsConnection.OnConnectingAsync != null)
        {
            (string Host, int Port) target = (uri.Host, uri.Port);
            logger.LogInformation(
                NatsLogEvents.Connection,
                $"Try to invoke OnConnectingAsync before connect to NATS [{reconnectCount}] with {nameof(BusSocketConnectionFactory)}",
                reconnectCount);
            (string Host, int Port) newTarget = await natsConnection.OnConnectingAsync(target).ConfigureAwait(false);

            if (newTarget.Host != target.Host || newTarget.Port != target.Port) uri = uri.CloneWith(newTarget.Host, newTarget.Port);
        }

        logger.LogInformation(NatsLogEvents.Connection, $"Tried to connect NATS {uri} [{reconnectCount}] with {nameof(BusSocketConnectionFactory)}");

        NatsOpts opts = natsConnection.Opts;
        logger.LogDebug(NatsLogEvents.Connection, $"Trying to reconnect using TCP {uri} [{reconnectCount}] with {nameof(BusSocketConnectionFactory)}");
        ProxyTcpConnection tcpConnection = new ProxyTcpConnection(logger);
        await tcpConnection.ConnectAsync(uri.Host, uri.Port, opts.ConnectTimeout).ConfigureAwait(false);
        ISocketConnection socket = await AfterReconnectingAsync(natsConnection, uri, opts, logger, reconnectCount, tcpConnection);

        return socket;
    }
}

and in NatsOpts instance
SocketConnectionFactory = BusSocketConnectionFactory.Instance

Copy link
Member

@mtmk mtmk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @wolfkor appreciate the work but I think we can simplify this a little. I want to make sue the changes are minimal to avoid disturbing existing applications in case of unintended behavior changes we might introduce while not opening up internal APIs as much as possible. hope it makes sense.


namespace NATS.Client.Core;

public class DefaultSocketConnectionFactory : ISocketConnectionFactory
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this all the functionality moved from connection class? Can't we just use the callback Func<NatsUri, NatsOpts, CancellationToken, ValueTask<ISocketConnection>>? SocketConnectionFactory?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no i don't see that. We need a change inside TcpConnection or WebSocketConnection for sending to the proxy. How to hide and give the customer the chance to change it.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably the only way i see is a callback which returns the pure Socket given by the proxy and put it in the constructor of TcpConnection. But the current OnSocketAvailable imlementations will break with that

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean we don't need this class. see below.

/// <summary>
/// Hook when socket is available.
/// </summary>
Func<ISocketConnection, ValueTask<ISocketConnection>>? OnSocketAvailableAsync { get; set; }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can't really change the public API

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OnSocketAvailableAsync was implemented for proxy scenario but is simply not working for Proxy+TLS. It must be replaced bei the new functionality

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can leave it but down call it - not a smart way i think

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer not to break existing applications


namespace NATS.Client.Core;

public interface ISocketConnectionFactory
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same. shouldn't need this.

@@ -258,6 +259,30 @@ public virtual async ValueTask DisposeAsync()
}
}

public NatsUri FixTlsHost(NatsUri uri)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we have to open this up I think we should documented as an internal api.

}

_logger.LogInformation(NatsLogEvents.Connection, "Try to connect NATS {0}", uri);
if (uri.IsWebSocket)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can switch here. if we have the callback in the options use that to assign to _socket otherwise let the rest of the code stay the same.

Copy link
Member

@mtmk mtmk Apr 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here and in reconnect loop add these:

                    _logger.LogInformation(NatsLogEvents.Connection, "Tried to connect NATS {Url} [{ReconnectCount}]", url, reconnectCount);
+                    if (Opts.SocketConnectionFactory != null)
+                    {
+                        _logger.LogDebug(NatsLogEvents.Connection, "Trying to reconnect using SocketFactory {Url} [{ReconnectCount}]", url, reconnectCount);
+                        var conn = await Opts.SocketConnectionFactory(url, Opts, _disposedCancellationTokenSource.Token);
+                        await conn.ConnectAsync(url, Opts).ConfigureAwait(false);
+                        _socket = conn;
+                    }
+                    else if (url.IsWebSocket)
-                    if (url.IsWebSocket)
                    {

these and with the option addition that should be it.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ISocketConnection has no ConnectAsync

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's even better. we can just return a connected socket and avoid the extra connect call.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and the caller has to implement for the return value of SocketConnectionFactory ISocketConnection on his own? That was one point for unsealing TcpConnection

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, we can start with that. there is very little code in TcpConnection with almost no logic. it's not worth trying to reuse it. creates more unnecessary coupling.

}
}

if (OnSocketAvailableAsync != null)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same. can't break api

}

_logger.LogInformation(NatsLogEvents.Connection, "Tried to connect NATS {Url} [{ReconnectCount}]", url, reconnectCount);
if (url.IsWebSocket)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same. switch if we have factory defined.

@@ -97,6 +97,8 @@ public sealed record NatsOpts

public Encoding SubjectEncoding { get; init; } = Encoding.ASCII;

public ISocketConnectionFactory SocketConnectionFactory { get; init; } = DefaultSocketConnectionFactory.Instance;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change to Func<NatsUri, NatsOpts, CancellationToken, ValueTask<ISocketConnection>>? defaulting to null

{
public SocketClosedException(Exception? innerException)
: base("Socket has been closed.", innerException)
{
}
}

internal sealed class TcpConnection : ISocketConnection
public class TcpConnection : ISocketConnection
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if we should open these up. there isn't a lot of code here to be reused. implementations can copy paste the bits they need.


internal sealed class WebSocketConnection : ISocketConnection
public class WebSocketConnection : ISocketConnection
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as tcp connection. no a lot of code to reuse. I'd rather not open too much. if we have to we must document as internal API.

wolfkor added a commit to wolfkor/nats.net that referenced this pull request Apr 11, 2025
@mtmk
Copy link
Member

mtmk commented Apr 12, 2025

thanks for the changes @wolfkor!

btw you need to sign your commits. you will have to force push with a new commit. otherwise repo is set to block unsigned commits. all commits in a pr must be signed.

image

Copy link
Member

@mtmk mtmk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks @wolfkor

(after commits signed)

will give @caleblloyd a chance to review.

@mtmk mtmk requested a review from caleblloyd April 12, 2025 04:21
wolfkor added a commit to wolfkor/nats.net that referenced this pull request Apr 12, 2025
wolfkor and others added 12 commits April 12, 2025 20:50
* Update docs and example links

* Hide docs folder

[nats:update-docs]
[nats:update-docs]
* Fix docs workflow

* Disable conditional check for docfx workflow trigger

Make sure this runs every time so that if it breaks, we know straightaway.
* Fix example code paths and reorg

* Revert readme nats by example link
Copy link
Collaborator

@caleblloyd caleblloyd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think something happened with the diff, it appears a folder was moved and now there are a bunch of changes in here due to that. Can you revert that?

@caleblloyd
Copy link
Collaborator

Oh maybe it was caused by merging main into this PR, I think this may need to be squashed and rebased

@mtmk
Copy link
Member

mtmk commented Apr 13, 2025

Oh maybe it was caused by merging main into this PR, I think this may need to be squashed and rebased

yes looks like we need a rebase @wolfkor. changes for this PR are only in NatsConnection, NatsOpts and NatsUri classes.

@wolfkor
Copy link
Author

wolfkor commented Apr 13, 2025

I had to sign all my commits including the main merge by amend command. Give me a hint what to do. Making changes again in new branch?

@mtmk
Copy link
Member

mtmk commented Apr 13, 2025

I had to sign all my commits including the main merge by amend command. Give me a hint what to do. Making changes again in new branch?

yes, opening a new pr and reference this one in comments. that might be easier.

@mtmk
Copy link
Member

mtmk commented Apr 13, 2025

@mtmk mtmk closed this Apr 13, 2025
@wolfkor wolfkor deleted the proxy-tls branch April 14, 2025 06:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants