Skip to content

Fix the User-Agent header name sent by mp_ua_header() - #936

Open
Jonas Lejon (jonaslejon) wants to merge 1 commit into
microsoft:mainfrom
jonaslejon:fix/user-agent-header-name
Open

Fix the User-Agent header name sent by mp_ua_header()#936
Jonas Lejon (jonaslejon) wants to merge 1 commit into
microsoft:mainfrom
jonaslejon:fix/user-agent-header-name

Conversation

@jonaslejon

Copy link
Copy Markdown

mp_ua_header() returns the key UserAgent, which is not the HTTP User-Agent
header. The result is that the MSTICPy identifier is sent under a header name
nothing reads, and httpx fills in the real User-Agent with its own default, so
requests are seen by upstream services as python-httpx/<version>.

Reproducing

>>> from msticpy.common.utility import mp_ua_header
>>> mp_ua_header()
{'UserAgent': 'MSTICPy3.0.2-Python3.12.7-linux'}

Pointing any HttpTIProvider at a local HTTP server and dumping the received
headers shows both, with only the wrong one carrying our identifier:

User-Agent: python-httpx/0.28.1
UserAgent:  MSTICPy3.0.2-Python3.12.7-linux

Where it came from

The header was set correctly until #454. Before that change,
http_provider.py read:

if "User-Agent" not in req_dict["headers"]:
    req_dict["headers"]["User-Agent"] = _MSTICPY_USER_AGENT

and it became:

if "User-Agent" not in req_dict["headers"]:
    req_dict["headers"].update(mp_ua_header())

The guard still tests User-Agent, but the key written is now UserAgent, so
the guard never observes its own write. This means the intent of #319, to
identify MSTICPy in outbound HTTP requests, has not taken effect since then.

The change

Two lines:

  1. common/utility/package.py returns {"User-Agent": ...}.
  2. data/drivers/azure_monitor_driver.py used mp_ua_header()["UserAgent"] to
    get the string for UserAgentPolicy. It now uses the already-exported
    MSTICPY_USER_AGENT constant, which is what it actually wants and is
    independent of the header name.

mp_ua_header() has 25 call sites across 14 modules, all of which pass it to
httpx as headers=, so they are all corrected by (1).

Providers that already declare their own User-Agent (crowdsec,
virustotal, prismacloud_driver) are unaffected: the guard in
http_provider.py continues to leave them alone, and that guard now works as
written.

Verification

  • tests/context/test_tiproviders.py and
    tests/data/drivers/test_azure_monitor_driver.py pass. Changing only (1)
    fails test_ti_config_and_load with KeyError: 'UserAgent', which is what
    identified the azure_monitor_driver usage.
  • Re-running the local-server check above shows a single
    User-Agent: MSTICPy... and no UserAgent header.

Note

This changes what every MSTICPy deployment sends on outbound requests, from
python-httpx/<version> to the MSTICPy identifier. That is the intended
behaviour, but it is a visible change to upstream services rather than a no-op,
so it may be worth a release-note line.

`mp_ua_header()` returned the key `UserAgent`, which is not the HTTP
`User-Agent` header, so the MSTICPy identifier was sent under a header
name nothing reads while httpx filled in the real `User-Agent` with its
own default. Outbound requests were seen by services as
`python-httpx/<version>`.

`azure_monitor_driver` read `mp_ua_header()["UserAgent"]` to obtain the
string for `UserAgentPolicy`. It now uses the exported
`MSTICPY_USER_AGENT` constant, which is what it wants and is independent
of the header name.

Providers that declare their own `User-Agent` are unaffected: the guard
in `http_provider.py` still leaves them alone, and now works as written.
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@jonaslejon

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant