Python 3.13: an unexpected Authorization header because of NETRC file

[yermulnik]

When sending POST request using Python 3.13 with requests v2.32.3 there appear an unexpected Authorization header.
This is not the case with Python 3.10 with the same version of requests lib.

Expected Result

POST request does not get updated with an unexpected Authorization header.

Actual Result

Python 3.13

> ~/tmp/zzz.py 2>&1 | egrep -m 1 "^send: "
send: b'POST /oauth2/v1/device/authorize HTTP/1.1\r\nHost: okta.oktapreview.com\r\nUser-Agent: gimme-aws-creds 2.8.2;linux;3.13.2\r\nAccept-Encoding: gzip, deflate\r\nAccept: application/json\r\nConnection: keep-alive\r\nContent-Length: 52\r\nContent-Type: application/x-www-form-urlencoded\r\nAuthorization: Basic Og==\r\n\r\n'

Python 3.10

> ~/tmp/zzz.py 2>&1 | egrep -m 1 "^send: "
send: b'POST /oauth2/v1/device/authorize HTTP/1.1\r\nHost: okta.oktapreview.com\r\nUser-Agent: gimme-aws-creds 2.8.2;linux;3.13.2\r\nAccept-Encoding: gzip, deflate, br\r\nAccept: application/json\r\nConnection: keep-alive\r\nContent-Length: 52\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\n'

Reproduction Steps

import requests
import logging

try:
    import http.client as http_client
except ImportError:
    # Python 2
    import httplib as http_client
http_client.HTTPConnection.debuglevel = 1

logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

data = {
    "client_id": "<client_id>",
    "scope": "openid okta.apps.sso"
}
url = "https://okta.oktapreview.com/oauth2/v1/device/authorize"
headers = {'User-Agent': 'gimme-aws-creds 2.8.2;linux;3.13.2', 'Accept': 'application/json'}

response = requests.post(url, data=data, headers=headers, verify=True)

print(response.text)

System Information

$ python -m requests.help

{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "3.4.1"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.10"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.13.2"
  },
  "platform": {
    "release": "6.8.0-52-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.32.3"
  },
  "system_ssl": {
    "version": "30400010"
  },
  "urllib3": {
    "version": "2.3.0"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}

Ref: Nike-Inc/gimme-aws-creds#485

[sigmavirus24]

This is strange as Og== is base64 encoded from :.

My best guess is that adding trust_env=False would fix this on Python 3.13. Why it fixes it though, is likely do to something in the netrc standard library.

If you check ~/.netrc do you have any configuration for the particular host you care about?

[sigmavirus24]

Are the versions of requests the same between the two environments as well?

[yermulnik]

My best guess is that adding trust_env=False would fix this on Python 3.13. Why it fixes it though, is likely do to something in the netrc standard library.

If you check ~/.netrc do you have any configuration for the particular host you care about?

Can confirm the ~/.netrc is the issue and the only matching host entry for the host is the default entry:

default
        macdef init
                set prompt "[%n@%M][%/]> "

Seems to have nothing to do with auth though, but removing this section does fix the issue (similarly setting trust_env=False for requests session fixes the issue too).

Since my issue is with another tool that utilizes requests, do you by any chance have a hint of whether trust_env value can be modified (set False) via an external approach like an Env var?

Are the versions of requests the same between the two environments as well?

Yes.

> for i in 3.10 3.13; do echo -n "$i: "; python$i -m requests.help | jq -r '.requests.version'; done
3.10: 2.32.3
3.13: 2.32.3

[sigmavirus24]

It would be ironic to have an environment variable telling the library to not trust the environment.

[yermulnik]

Haha, agree =)) I think I'm just too focused on .netrc part of this.

Looks like my best (and sort of the only) option at the moment is to try and PR to the tool, that I rely upon, a flag to switch trust_env for requests (and accept the fact that this will also break use cases where someone relies upon proxy env vars) 🤔

Am I right assuming that the Authorization header in my conditions (where there are no auth parameters set for the host via .netrc) is added by Python 3.13 rather than by requests lib or is it something that requests can potentially rectify or remediate?

Thanks.

[sigmavirus24]

Can you reproduce it just with the standard library? If not, can you write a failing test for https://github.com/psf/requests/blob/main/src%2Frequests%2Futils.py#L207

[yermulnik]

Can you reproduce it just with the standard library?

Sorry, I can't seem to figure out what you mean 😕 Isn't the «Reproduction Steps» from the issue description what you're asking about? 🤔

#!/usr/bin/env python3.13
import requests

data = {
    "client_id": "0[…]7",
    "scope": "openid okta.apps.sso"
}
url = "https://[…].oktapreview.com/oauth2/v1/device/authorize"
headers = {'Accept': 'application/json'}

response = requests.post(url, data=data, headers=headers, verify=True)

print(response.text)

> ~/tmp/zzz.py
{"errorCode":"invalid_client","errorSummary":"Invalid value for 'client_id' parameter.","errorLink":"invalid_client","errorId":"oaeohowcwaASwevPhC5oeoeHA","errorCauses":[]}

> NETRC="" ~/tmp/zzz.py
{"device_code":"c[…]f","user_code":"L[…]B","verification_uri":"https://[…].oktapreview.com/activate","verification_uri_complete":"https://[…].oktapreview.com/activate?user_code=L[…]B","expires_in":600,"interval":5}

If not, can you write a failing test for main/src%2Frequests%2Futils.py#L207

I could try... though I'm not a Python dev and it can take me a challenge. If you could provide some guidance probably (or point out to an existing test example): I probably need to put a .netrc file somewhere under /tests/ dir (create new dir like files/ and add file to it?); the known to me endpoint that returns failure in this case is my employer's Okta tenant (incl Okta App Client ID), which I don't think is a good endpoint to run test against from within pfs/requests repo; so on 🤔

[yermulnik]

@sigmavirus24 Any hints?
I do understand that my case is pretty much rare and uncommon, though it's probably better to rectify this glitch sooner or later and I'll be happy to assist testing the fix. Thank you.