Fetching a URL

Every other resource is addressed at something an administrator connected: a bucket, a collection, one configured host. core.web_fetch is the one where your agent names the destination — and the reason it can is that the fetch happens in a network-isolated container under a policy your organization configured, not in your agent's own process, which has no outbound network at all.

from im_sdk import ImCore, im_agent
 
 
@im_agent("feed-reader")
async def main(core: ImCore, data: dict) -> str:
    # 1 — is it still there? A HEAD, not a download.
    check = await core.web_fetch.probe("web", data["feed"])
    if not check.output.reachable:
        return f"{check.output.final_url} is gone (status {check.output.status})"
 
    # 2 — the feed, decoded
    feed = await core.web_fetch.fetch("web", data["feed"])
    urls = [link for link in parse_feed(feed.text)]
 
    # 3 — one linked PDF, as bytes, handed to document_parse
    pdf = await core.web_fetch.fetch("web", urls[0], as_text=False)
    elements = (
        await core.document_parse.partition(
            "unstructured", file=pdf.output.content_base64
        )
    ).output.elements
 
    return "\n".join(e.text for e in elements if e.text)

Four things to know first

  • The service name is the policy, not the destination. "web" above names a connected ciu-fetch service whose configuration decides which content types and ports are permissible, whether internal hosts are reachable at all, whether robots.txt is honoured, and the byte and time ceilings a per-call value may only tighten. The URL is yours; what may come back is not. Two services can allow different things and be granted to different agents, and nothing you pass widens either one.
  • Internal addresses are unreachable, and no argument changes that. Loopback, private, CGNAT and link-local ranges — the cloud metadata endpoint 169.254.169.254 included — are refused after resolution and checked on the resolved address, so a public hostname that points inward does not get through either. An administrator can re-enable named hosts per service; an agent cannot.
  • A body is text or bytes, never both. as_text=True is the default and is refused for a media type that is not textual; as_text=False gives you Base64, which is what a PDF on its way to document_parse wants. result.text reads either.
  • Nothing you send carries a credential. Authorization and Cookie cannot be set, there is no cookie jar, and Set-Cookie is stripped from the response. Each fetch is stateless. That is what makes an agent-chosen destination safe to allow at all — so it is a property of the resource, not a default you can turn off.

Fetching — core.web_fetch.fetch

result = await core.web_fetch.fetch(
    "web",                                  # the connected service = the policy
    "https://example.org/feed.xml",
    as_text=True,                           # default; False → Base64
    max_bytes=200_000,                      # may only tighten the service ceiling
    timeout=10,                             # likewise
    accept="application/rss+xml, text/xml",
    headers={"if-none-match": etag},        # conditional requests only
)
 
result.text                     # the body, either representation
result.output.status            # 200
result.output.content_type      # "application/rss+xml" — parameters stripped
result.output.size_bytes        # what actually arrived
result.output.final_url         # not necessarily what you asked for
result.output.redirect_chain    # [RedirectHop(url=…, status=301), …]

Exactly one of output.content_text / output.content_base64 is set — and neither when output.not_modified says a conditional request won. result.text gives you the decoded body in the first case, the decoded bytes in the second when they are UTF-8, and "" when they are not: a binary body yields nothing rather than mojibake, so a caller that meant to handle bytes finds out by getting nothing instead of by getting nonsense.

max_bytes truncates your budget and refuses the organization's

The two ceilings behave in opposite ways, on purpose:

Past the capWhat happens
the max_bytes you passedthe body is cut and output.truncated is True. You asked for a budget
the service's ceilingthe fetch is refused — ImForbiddenError, kind="response_too_large"

So passing max_bytes is what makes an oversized body arrive truncated rather than refused. Handing half a PDF to document_parse because an administrator's ceiling was silently applied is the failure the split avoids.

Conditional requests are supported and cheap

Keep output.etag (or output.last_modified) and send it back:

result = await core.web_fetch.fetch("web", url, headers={"if-none-match": etag})
if result.output.not_modified:
    ...                                     # nothing changed, no body transferred

headers takes only if-none-match, if-modified-since, accept and accept-language. Anything else is refused in your process, before the call leaves it — a ImValidationError rather than a round trip. A silently dropped header is how a caller comes to believe a control applies when it does not.

accept is a separate argument for a reason worth keeping straight: it is a preference expressed to the origin, while what the service permits back is independent of it. Asking for application/pdf does not make a PDF allowed.

Setting both accept and headers={"accept": ...} is not an error — the header wins, and exactly one Accept goes on the wire. That is the same last-write rule every other allowlisted header follows.

Probing — core.web_fetch.probe

check = await core.web_fetch.probe("web", url, follow_redirects=True, timeout=5)
 
check.output.reachable          # the answer
check.output.status             # 200 · 404 · None when nothing answered at all
check.output.content_type       # what a fetch would return…
check.output.content_length     # …and roughly how big, if the origin said
check.output.elapsed_ms

A HEAD (falling back to a GET abandoned at the headers), so "is this URL still there" costs a round trip instead of a body. It answers rather than raising:

SituationAnswer
the origin said 404reachable=False, status=404
the name does not resolvereachable=False, status=None
nothing accepted the connectionreachable=False, status=None
the URL is forbidden by policyraises ImForbiddenError
the budget ran outraises ImTimeoutError

The last two are deliberate. "Not permitted" must not read like "not there" — an agent that saw reachable=False for a target its organization forbids would go looking for the wrong problem. And a timeout is a verdict about our budget rather than about the URL, which also makes it the one of the three worth retrying.

robots.txt is never consulted for a probe: it requests no content. content_type is reported, which is how to find out whether a fetch would be allowed before spending one.

Output variants

Callresult.outputFields
fetchFetchOutput.status, .final_url, .redirect_chain, .content_type, .charset, .size_bytes, .truncated, .not_modified, .etag, .last_modified, and exactly one of .content_text / .content_base64
probeProbeOutput.reachable, .status, .final_url, .redirect_chain, .content_type, .content_length, .elapsed_ms

Both, plus RedirectHop, FetchRequest, ProbeRequest and ALLOWED_FETCH_HEADERS, are importable from the package root: from im_sdk import FetchOutput.

Errors worth handling

The specific refusal travels as exc.context["kind"], because that is the distinction worth acting on: "this organization does not permit PDFs" is a configuration conversation and "that URL is unreachable" is a retry.

ErrorkindMeans
ImForbiddenErrorcontent_type_not_allowedthe service does not permit that media type. Not retryable — the remedy is a different URL or a word with an administrator
content_type_missingthe origin sent no parseable Content-Type. Unclassifiable means unallowed
magic_bytes_mismatchit claims to be a PDF and does not begin %PDF-
address_not_allowedthe URL resolved to a loopback / private / CGNAT / link-local address
dns_resolution_failedthe name does not resolve (a probe answers reachable=False instead)
port_not_allowed · scheme_not_allowedthe service's port list; http/https only
url_credentials_not_allowed · header_not_alloweduser:pass@host; a header outside the allowlist
robots_disallowedthe site's robots.txt says no
response_too_large · decompression_ratio_exceededpast the service's ceiling — your own max_bytes truncates instead
too_many_redirectsthe hop cap
—fetch or probe is not granted to this agent
ImQuotaError—this session has spent its web_fetch calls. A runaway stop, not a bill — and off unless an administrator turned it on, so most deployments never raise it
ImValidationErrorinvalid_url · as_text_not_allowedthe request cannot be served as written. Also raised locally for an empty URL, a non-http(s) scheme, and a disallowed header
ImResourceErrorupstream_unreachable · upstream_http_error · redirect_invalidthe origin. Worth a retry
ImTimeoutErrorupstream_timeoutthe wall clock ran out

None of them carries the fetched body, and a refusal names only a host or a port — never a path or a query.

Traces keep the shape, never the content

Fetched bodies are kept out of trace exports with no opt-in, and the URL is reduced to its host. What travels is status, content_type, charset, size_bytes, truncated, not_modified, etag, last_modified, reachable, content_length, elapsed_ms and the redirect hop count — plus the structural call parameters (service, as_text, max_bytes, timeout, accept, follow_redirects). Request-header names survive; their values do not.

The organization-level log_data_payloads switch, which lets data export database rows verbatim, does not extend to this resource: a fetched body is arbitrary third-party content your agent chose to pull, and no consent about an organization's own rows says anything about it. The path and query go for a second reason on top — they name somebody else's resource and routinely carry an identifier or a token.

Your own log() lines are your own output. If you log the bytes you fetched, that is in the trace because you put it there.

The step renders as crawler on the execution timeline, alongside web_search and web_crawler.

What is deliberately not here

  • No post, and no other verb. POST to a URL an agent chose, carrying a service credential, is a credential-exfiltration path. For a fixed endpoint behind a token, use core.http_capability.
  • No article extraction. This returns bytes or text; a parsed article is core.web_crawler's job, and it drives a real browser to do it.
  • No cookie jar, no session, no redirect-following you control. Redirects are walked one hop at a time with the address rules re-run on each, because a 302 to an internal address is the whole attack.
  • No per-agent throttle you can rely on. The platform bounds how many fetches are in flight at once, but that bound is shared by every agent on the deployment: firing a hundred in parallel does not fail, it queues — behind and in front of everyone else's. Pace your own fan-out rather than treating the platform's cap as yours.

See also

  • Capabilities — the signatures alongside every other resource
  • Files & storage — the durable place for what you fetched, and where document_parse usually reads from instead
  • What an administrator configures — docs/web-fetch.md in the platform repository: the eight policy fields, the thirteen security rules, and the honest limits of each