False-positive type errors when passing **kwargs to a function that has other keyword args

[aaron-siegel]

Bug Report

If a function signature contains one or more explicit keyword parameters followed by a **kwargs parameter, and the function is called on kwargs only, then mypy generates false positive type errors.

The number of errors generated is unbounded and appears to be equal to the number of distinct types found among the explicit keyword parameters. In addition, the errors are mutually inconsistent with one another (see example below).

To Reproduce

from typing import Any, Optional

def example(
	*,
	a: Optional[int] = None,
	b: Optional[float] = None,
	c: Optional[bool] = None,
	d: Optional[list] = None,
	**kwargs
) -> None:
	print(kwargs)

kwargs = {'hello': 'world'}
example(**kwargs)

Expected Behavior

I'd expect this to pass type-checking: it looks perfectly correct to me. All I'm doing is passing kwargs into kwargs.

Actual Behavior

example.py:14: error: Argument 1 to "example" has incompatible type "**dict[str, str]"; expected "Optional[int]"  [arg-type]
example.py:14: error: Argument 1 to "example" has incompatible type "**dict[str, str]"; expected "Optional[float]"  [arg-type]
example.py:14: error: Argument 1 to "example" has incompatible type "**dict[str, str]"; expected "Optional[bool]"  [arg-type]
example.py:14: error: Argument 1 to "example" has incompatible type "**dict[str, str]"; expected "Optional[list[Any]]"  [arg-type]
Found 4 errors in 1 file (checked 1 source file)

In addition to being false positives, these errors are mutually inconsistent: mypy is claiming that an Optional[int] is expected in Argument 1, and also that an Optional[float] is expected in Argument 1, and also that an Optional[bool] is expected in Argument 1, and so on!

If I change the last line to this:

example(a=None, b=None, c=None, d=None, **kwargs)

then the errors go away. Is this really what mypy wants me to do??

Your Environment

• Mypy version used: 1.14.1
• Mypy command-line flags: None
• Mypy configuration options from pyproject.toml:

allow_redefinition = true
disable_error_code = "no-any-unimported, no-any-return"
disallow_any_unimported = true
check_untyped_defs = true
follow_imports = "silent"
plugins = "pydantic.mypy"
show_error_codes = true
strict_optional = false
warn_return_any = true
warn_unused_ignores = true

• Python version used: 3.9.20

[erictraut]

This is a behavior that is not well defined in the typing spec currently. An argument can be made that mypy's behavior is correct here (since it prevents false negatives), but a counterargument can also be made in favor of reducing false positives.

Pyright currently adopts the same behavior as mypy in this case (for consistency). This behavior also affects overload call evaluation, which we are attempting to standardize, so it would be good to clarify the correct behavior in the typing spec. That way, all conformant Python type checkers would behave the same here.

If you'd like to spearhead an effort to standardize this behavior, here is the process by which the typing spec can be updated and amended. A good starting point is to create a new discussion thread in the Python typing forum.

[aaron-siegel]

Independent of one's interpretation of the type spec, there's still a mypy bug, in that the 2nd, 3nd, and 4th error messages are false (it is simply not true that Argument 1 expects an Optional[float], for example).

The pattern used by this method (a few named kwargs followed by a variable **kwargs) is common in Python, e.g., it's used by Pandas stylers. It's rather annoying to have to either restate the defaults for every named parameter - I'm pretty sure this wasn't the Pandas team's intention when they created that API - or to put in a type-ignore qualifier, every time one of them is invoked. I'd argue this scenario is far more common than the corresponding "false-negative" one, which involves someone putting into an entry into a **kwargs dict that conflicts with a named parameter. Mypy is rife with false-negatives that are far more likely.

I don't have bandwidth to drive some proposal through a process like this but I did want to surface this. Thanks for your quick response!

[tyralla]

Independent of one's interpretation of the type spec, there's still a mypy bug, in that the 2nd, 3nd, and 4th error messages are false (it is simply not true that Argument 1 expects an Optional[float], for example).

I think you are mistaking argument with parameter. (I also do it all the time, so I just checked on Wikipedia...)

[joooeey]

I see the same thing when trying to pass a default namespace as kwarg dictionary to xml.etree.ElementTree.Element.findtext:

MCVE:

import xml.etree.ElementTree as ET

NS = {"namespaces": {"": "urn:skogforsk:stanford2010"}}

def gettext() -> str | None:
    return ET.XML("<a>123</a>").findtext("a", **NS)

MyPy:

my.py:15: error: Argument 2 to "findtext" of "Element" has incompatible type "**dict[str, dict[str, str]]"; expected "str | None"  [arg-type]

typeshed:

    @overload
    def findtext(self, path: str, default: None = None, namespaces: dict[str, str] | None = None) -> str | None: ...
    @overload
    def findtext(self, path: str, default: _T, namespaces: dict[str, str] | None = None) -> _T | str: ...

This is indeed a bug: MyPy compares the type of argument 2 (the keyword argument "namespaces" of type dict[str, str]) to parameter 2 ("default" of type str | None).

For my particular use case, I can please the type checker by declaring a TypedDict at a single point in the codebase:

class NameSpaces(TypedDict):
    namespaces: dict[str, str]

NS: NameSpaces = {"namespaces": {"": "urn:skogforsk:stanford2010"}}

[sandra-selfdecode]

I also think there's a bug. I have a class where each method is manipulating some input and then passing it to _run. There are four kwargs in _run that can be set with any method, but it seemed messy to type them out every time, so instead I did:

def runTaskA(self, 'input': str, sex: str, regions: List[str] = [], **kwargs):
    ...do some stuff here to construct command...
   return self._run(input, cmd, message, **kwargs)

Until the latest update there were no errors, but now it raises an error if kwargs isn't typed and when I type kwargs in the definition the typing isn't passed to _run correctly.

**kwargs: Union[str, Path, bool] gets me:

1229: error: Argument 4 to "_run" of "MyClass" has incompatible type "**dict[str, str | Path | bool]"; expected "str"  [arg-type]
1229: error: Argument 4 to "_run" of "MyClass" has incompatible type "**dict[str, str | Path | bool]"; expected "Path | str | None"  [arg-type]
1229: error: Argument 4 to "_run" of "MyClass" has incompatible type "**dict[str, str | Path | bool]"; expected "bool"  [arg-type]

Constructing a TypedDict and using **kwargs: RunKwargs gets me:
1421: error: Argument 3 to "_run" of "MyClass" has incompatible type "**dict[str, RunKwargs]"; expected "str | Path | bool" [arg-type]

I sort of got it to work if I stopped using ** in the function definition, but that's not correct usage, and I then got an error in some places where it was telling me that one of my kwargs was being used as both a positional arg and a kwarg. I believe it was because it sorted alphabetically before a kwarg that was explicitly defined in the function and separated *args and **kwargs.

[tyralla]

@sandra-selfdecode

I am still not sure there is a real bug. Maybe you just forgot to use Unpack?

I tried to concretise your problem description, and this is what Mypy 1.15 reports:

def runTaskA(input: str, sex: str, regions: list[str] = [], **kwargs): pass

d = {"k": "v"}
runTaskA(input="x", sex="y", **d)  # error: Argument 3 to "runTaskA" has incompatible type "**dict[str, str]"; expected "list[str]"  [arg-type]
runTaskA(input="x", sex="y", regions=["z"], **d)
 
####################################################

from typing import TypedDict, Unpack

class RunKwargs(TypedDict):
    k: str
    
def runTaskB(input: str, sex: str, regions: list[str] = [], **kwargs: Unpack[RunKwargs]): pass

td = RunKwargs(k="v")
runTaskB(input="x", sex="y", **td)
runTaskB(input="x", sex="y", regions=["z"], **td)

With the additional TypedDict power (runTaskB), Mypy is smart enough not to report the same error as it reports for the untyped case (runTaskA). And, as discussed above, the error report for the untyped case might be inconvenient, but it can help to increase type-safety.

[domdfcoding]

There has certainly been a change over time, as upgrading from 1.8.0 to 1.16 I've started seeing these errors that I wasn't seeing before.