Surf
Contents
Surf is a headless browser, useful for web scraping, testing, and automating interactions with websites. Rye wraps it so you can navigate pages, click elements, fill forms, and manage cookies from a Rye program.
This package needs a build with -tags "b_contrib,b_surf".
Navigate a page
; Create a browser
browser: surf
; Open a page
browser .Open "https://example.com"
; Read the title and current URL
print browser .Title
print browser .Url
; Move through history
browser .Back
browser .Forward
browser .Reload
Find and inspect elements
Find returns a selection of elements matching a CSS selector. From a selection you can read its text, count elements, or iterate over them.
browser .Open "https://example.com"
; Find all links
links: browser .Find "a"
; How many did we get?
print links .Length?
; Get the combined text
print links .Text?
; Iterate over each element
links .Each [ el |
print el .Text?
]
Fill and submit forms
browser .Open "https://example.com/login"
; Get the form and fill its fields
form: browser .Form "form#login"
form .Input "username" "myuser"
form .Input "password" "mypass"
; Submit it
form .Submit
Bookmarks, cookies, and downloads
; Bookmark the current page and return to it later
browser .Bookmark "home"
browser .Open-bookmark "home"
; Set a cookie
browser .Set-cookie dict { "name" "session" "value" "abc123" "domain" ".example.com" "path" "/" }
; Download a file
browser .Download "https://example.com/file.pdf"
Impersonating a browser
Some sites behave differently based on the user agent. Surf has a builder that lets you impersonate a specific browser, or you can set a user agent directly.
; Set a user agent directly on the browser
browser: surf
browser .Set-user-agent "Mozilla/5.0 (compatible; MyBot/1.0)"
browser .Open "https://example.com"
; Or build a browser that impersonates Chrome
browser2: surf-client .Builder .Impersonate .Chrome .Build
browser2 .Open "https://example.com"
Rye also ships ready-made user agent strings such as surf-agent-chrome, surf-agent-firefox, and surf-agent-safari, plus surf-agent-create-version to build a custom one.