Fetching a URL
Every other resource is addressed at something an administrator connected: a
bucket, a collection, one configured host. core.web_fetch is the one where
your agent names the destination — and the reason it can is that the fetch
happens in a network-isolated container under a policy your organization
configured, not in your agent's own process, which has no outbound network at
all.
from im_sdk import ImCore, im_agent
@im_agent("feed-reader")
async def main(core: ImCore, data: dict) -> str:
# 1 — is it still there? A HEAD, not a download.
check = await core.web_fetch.probe("web", data["feed"])
if not check.output.reachable:
return f"{check.output.final_url} is gone (status {check.output.status})"
# 2 — the feed, decoded
feed = await core.web_fetch.fetch("web", data["feed"])
urls = [link for link in parse_feed(feed.text)]
# 3 — one linked PDF, as bytes, handed to document_parse
pdf = await core.web_fetch.fetch("web", urls[0], as_text=False)
elements = (
await core.document_parse.partition(
"unstructured", file=pdf.output.content_base64
)
).output.elements
return "\n".join(e.text for e in elements if e.text)Four things to know first
- The service name is the policy, not the destination.
"web"above names a connectedciu-fetchservice whose configuration decides which content types and ports are permissible, whether internal hosts are reachable at all, whetherrobots.txtis honoured, and the byte and time ceilings a per-call value may only tighten. The URL is yours; what may come back is not. Two services can allow different things and be granted to different agents, and nothing you pass widens either one. - Internal addresses are unreachable, and no argument changes that.
Loopback, private, CGNAT and link-local ranges — the cloud metadata endpoint
169.254.169.254included — are refused after resolution and checked on the resolved address, so a public hostname that points inward does not get through either. An administrator can re-enable named hosts per service; an agent cannot. - A body is text or bytes, never both.
as_text=Trueis the default and is refused for a media type that is not textual;as_text=Falsegives you Base64, which is what a PDF on its way todocument_parsewants.result.textreads either. - Nothing you send carries a credential.
AuthorizationandCookiecannot be set, there is no cookie jar, andSet-Cookieis stripped from the response. Each fetch is stateless. That is what makes an agent-chosen destination safe to allow at all — so it is a property of the resource, not a default you can turn off.
Fetching — core.web_fetch.fetch
result = await core.web_fetch.fetch(
"web", # the connected service = the policy
"https://example.org/feed.xml",
as_text=True, # default; False → Base64
max_bytes=200_000, # may only tighten the service ceiling
timeout=10, # likewise
accept="application/rss+xml, text/xml",
headers={"if-none-match": etag}, # conditional requests only
)
result.text # the body, either representation
result.output.status # 200
result.output.content_type # "application/rss+xml" — parameters stripped
result.output.size_bytes # what actually arrived
result.output.final_url # not necessarily what you asked for
result.output.redirect_chain # [RedirectHop(url=…, status=301), …]Exactly one of output.content_text / output.content_base64 is set — and
neither when output.not_modified says a conditional request won. result.text
gives you the decoded body in the first case, the decoded bytes in the second
when they are UTF-8, and "" when they are not: a binary body yields nothing
rather than mojibake, so a caller that meant to handle bytes finds out by getting
nothing instead of by getting nonsense.
max_bytes truncates your budget and refuses the organization's
The two ceilings behave in opposite ways, on purpose:
| Past the cap | What happens |
|---|---|
the max_bytes you passed | the body is cut and output.truncated is True. You asked for a budget |
| the service's ceiling | the fetch is refused — ImForbiddenError, kind="response_too_large" |
So passing max_bytes is what makes an oversized body arrive truncated rather
than refused. Handing half a PDF to document_parse because an administrator's
ceiling was silently applied is the failure the split avoids.
Conditional requests are supported and cheap
Keep output.etag (or output.last_modified) and send it back:
result = await core.web_fetch.fetch("web", url, headers={"if-none-match": etag})
if result.output.not_modified:
... # nothing changed, no body transferredheaders takes only if-none-match, if-modified-since, accept and
accept-language. Anything else is refused in your process, before the call
leaves it — a ImValidationError rather than a round trip. A silently dropped
header is how a caller comes to believe a control applies when it does not.
accept is a separate argument for a reason worth keeping straight: it is a
preference expressed to the origin, while what the service permits back is
independent of it. Asking for application/pdf does not make a PDF allowed.
Setting both accept and headers={"accept": ...} is not an error — the header
wins, and exactly one Accept goes on the wire. That is the same last-write rule
every other allowlisted header follows.
Probing — core.web_fetch.probe
check = await core.web_fetch.probe("web", url, follow_redirects=True, timeout=5)
check.output.reachable # the answer
check.output.status # 200 · 404 · None when nothing answered at all
check.output.content_type # what a fetch would return…
check.output.content_length # …and roughly how big, if the origin said
check.output.elapsed_msA HEAD (falling back to a GET abandoned at the headers), so "is this URL still there" costs a round trip instead of a body. It answers rather than raising:
| Situation | Answer |
|---|---|
| the origin said 404 | reachable=False, status=404 |
| the name does not resolve | reachable=False, status=None |
| nothing accepted the connection | reachable=False, status=None |
| the URL is forbidden by policy | raises ImForbiddenError |
| the budget ran out | raises ImTimeoutError |
The last two are deliberate. "Not permitted" must not read like "not there" — an
agent that saw reachable=False for a target its organization forbids would go
looking for the wrong problem. And a timeout is a verdict about our budget
rather than about the URL, which also makes it the one of the three worth
retrying.
robots.txt is never consulted for a probe: it requests no content.
content_type is reported, which is how to find out whether a fetch would
be allowed before spending one.
Output variants
| Call | result.output | Fields |
|---|---|---|
fetch | FetchOutput | .status, .final_url, .redirect_chain, .content_type, .charset, .size_bytes, .truncated, .not_modified, .etag, .last_modified, and exactly one of .content_text / .content_base64 |
probe | ProbeOutput | .reachable, .status, .final_url, .redirect_chain, .content_type, .content_length, .elapsed_ms |
Both, plus RedirectHop, FetchRequest, ProbeRequest and
ALLOWED_FETCH_HEADERS, are importable from the package root:
from im_sdk import FetchOutput.
Errors worth handling
The specific refusal travels as exc.context["kind"], because that is the
distinction worth acting on: "this organization does not permit PDFs" is a
configuration conversation and "that URL is unreachable" is a retry.
| Error | kind | Means |
|---|---|---|
ImForbiddenError | content_type_not_allowed | the service does not permit that media type. Not retryable — the remedy is a different URL or a word with an administrator |
content_type_missing | the origin sent no parseable Content-Type. Unclassifiable means unallowed | |
magic_bytes_mismatch | it claims to be a PDF and does not begin %PDF- | |
address_not_allowed | the URL resolved to a loopback / private / CGNAT / link-local address | |
dns_resolution_failed | the name does not resolve (a probe answers reachable=False instead) | |
port_not_allowed · scheme_not_allowed | the service's port list; http/https only | |
url_credentials_not_allowed · header_not_allowed | user:pass@host; a header outside the allowlist | |
robots_disallowed | the site's robots.txt says no | |
response_too_large · decompression_ratio_exceeded | past the service's ceiling — your own max_bytes truncates instead | |
too_many_redirects | the hop cap | |
| — | fetch or probe is not granted to this agent | |
ImQuotaError | — | this session has spent its web_fetch calls. A runaway stop, not a bill — and off unless an administrator turned it on, so most deployments never raise it |
ImValidationError | invalid_url · as_text_not_allowed | the request cannot be served as written. Also raised locally for an empty URL, a non-http(s) scheme, and a disallowed header |
ImResourceError | upstream_unreachable · upstream_http_error · redirect_invalid | the origin. Worth a retry |
ImTimeoutError | upstream_timeout | the wall clock ran out |
None of them carries the fetched body, and a refusal names only a host or a port — never a path or a query.
Traces keep the shape, never the content
Fetched bodies are kept out of trace exports with no opt-in, and the URL is
reduced to its host. What travels is status, content_type, charset,
size_bytes, truncated, not_modified, etag, last_modified, reachable,
content_length, elapsed_ms and the redirect hop count — plus the
structural call parameters (service, as_text, max_bytes, timeout,
accept, follow_redirects). Request-header names survive; their values do
not.
The organization-level log_data_payloads switch, which lets data export
database rows verbatim, does not extend to this resource: a fetched body is
arbitrary third-party content your agent chose to pull, and no consent about an
organization's own rows says anything about it. The path and query go for a
second reason on top — they name somebody else's resource and routinely carry an
identifier or a token.
Your own log() lines are your own output. If you log the bytes you fetched,
that is in the trace because you put it there.
The step renders as crawler on the execution timeline, alongside
web_search and web_crawler.
What is deliberately not here
- No
post, and no other verb. POST to a URL an agent chose, carrying a service credential, is a credential-exfiltration path. For a fixed endpoint behind a token, usecore.http_capability. - No article extraction. This returns bytes or text; a parsed article is
core.web_crawler's job, and it drives a real browser to do it. - No cookie jar, no session, no redirect-following you control. Redirects are walked one hop at a time with the address rules re-run on each, because a 302 to an internal address is the whole attack.
- No per-agent throttle you can rely on. The platform bounds how many fetches are in flight at once, but that bound is shared by every agent on the deployment: firing a hundred in parallel does not fail, it queues — behind and in front of everyone else's. Pace your own fan-out rather than treating the platform's cap as yours.
See also
- Capabilities — the signatures alongside every other resource
- Files & storage — the durable place for what you
fetched, and where
document_parseusually reads from instead - What an administrator configures —
docs/web-fetch.mdin the platform repository: the eight policy fields, the thirteen security rules, and the honest limits of each