Go colly
last modified October 30, 2026
In this article we show how to scrape web pages with Colly, a scraping and crawling framework for Go.
Colly downloads pages, parses the HTML and calls event handlers for the
elements that match a CSS selector. The same collector can follow links,
restrict the crawl to one domain and depth, limit the request rate, run
asynchronously and handle cookies. The module is
github.com/gocolly/colly/v2; the examples use version 2.3.0 with
Go 1.27.0.
To keep the examples reproducible, they scrape a small local site. Save the two
files below in a site directory and serve it with
python3 -m http.server 8080. A third page,
fox.html, is identical to falcon.html apart from the
title and the text, so it is not shown again.
Reading the page title
A collector is created with colly.NewCollector, handlers are
registered on it, and Visit starts the request. The
OnHTML handler runs for every element that matches the selector.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Falcon shop</title>
</head>
<body>
<h1>Falcon shop</h1>
<ul>
<li class="product" data-sku="F-1">
<a href="/falcon.html">Peregrine falcon</a>
<span class="price">120</span>
</li>
<li class="product" data-sku="F-2">
<a href="/fox.html">Arctic fox</a>
<span class="price">90</span>
</li>
</ul>
</body>
</html>
The home page contains a heading, two product items and links to the detail pages.
package main
import (
"fmt"
"log"
"github.com/gocolly/colly/v2"
)
func main() {
c := colly.NewCollector()
c.OnHTML("title", func(e *colly.HTMLElement) {
fmt.Println("title:", e.Text)
})
if err := c.Visit("http://127.0.0.1:8080/"); err != nil {
log.Fatal(err)
}
}
The text of the title element is available as
e.Text. It is already trimmed of surrounding whitespace, so the
handler prints Falcon shop.
$ python3 -m http.server 8080 # in site/ $ go run main.go title: Falcon shop
Links and absolute URLs
Attribute values are read with Attr. Relative links must be
resolved against the URL of the current page, which
e.Request.AbsoluteURL does.
package main
import (
"fmt"
"log"
"github.com/gocolly/colly/v2"
)
func main() {
c := colly.NewCollector()
c.OnHTML("a[href]", func(e *colly.HTMLElement) {
link := e.Request.AbsoluteURL(e.Attr("href"))
fmt.Printf("%s -> %s\n", e.Text, link)
})
if err := c.Visit("http://127.0.0.1:8080/"); err != nil {
log.Fatal(err)
}
}
The a[href] selector matches only links that actually have an
href attribute. The handler prints the link text together with the
absolute address that the crawler would request next.
$ go run main.go Peregrine falcon -> http://127.0.0.1:8080/falcon.html Arctic fox -> http://127.0.0.1:8080/fox.html
Extracting structured data
ChildText returns the text of the first child that matches a
selector, and ChildAttr returns an attribute of such a child. The
data-sku attribute is read from the product element itself.
package main
import (
"fmt"
"log"
"github.com/gocolly/colly/v2"
)
func main() {
c := colly.NewCollector()
c.OnHTML("li.product", func(e *colly.HTMLElement) {
fmt.Printf("%s | %s | %s\n",
e.Attr("data-sku"),
e.ChildText("a"),
e.ChildText(".price"))
})
if err := c.Visit("http://127.0.0.1:8080/"); err != nil {
log.Fatal(err)
}
}
The selector li.product isolates the two products, and the fields
are read from inside each one. Colly uses the same CSS selectors as goquery,
so class, attribute and descendant selectors all work.
$ go run main.go F-1 | Peregrine falcon | 120 F-2 | Arctic fox | 90
The order of the callbacks
Handlers run in a fixed order for every request: OnRequest before
the request is sent, OnResponse after the response arrives,
OnHTML while the document is parsed, and OnScraped
after all HTML handlers have run.
package main
import (
"fmt"
"log"
"github.com/gocolly/colly/v2"
)
func main() {
c := colly.NewCollector()
c.OnRequest(func(r *colly.Request) {
fmt.Println("request:", r.URL)
})
c.OnResponse(func(r *colly.Response) {
fmt.Println("response:", r.StatusCode)
})
c.OnHTML("h1", func(e *colly.HTMLElement) {
fmt.Println("heading:", e.Text)
})
c.OnScraped(func(r *colly.Response) {
fmt.Println("scraped:", r.Request.URL)
})
if err := c.Visit("http://127.0.0.1:8080/"); err != nil {
log.Fatal(err)
}
}
The four lines follow that order. OnError is called when the
request fails; the next section shows it.
$ go run main.go request: http://127.0.0.1:8080/ response: 200 heading: Falcon shop scraped: http://127.0.0.1:8080/
Following links
A crawler follows links by calling Visit with the resolved address
inside an OnHTML handler. colly.MaxDepth stops the
crawl after the given number of levels.
<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <title>Peregrine falcon</title> </head> <body> <h1>Peregrine falcon</h1> <p class="description">The fastest bird in the world.</p> </body> </html>
The detail page that the crawler reaches in the second level.
package main
import (
"fmt"
"log"
"github.com/gocolly/colly/v2"
)
func main() {
c := colly.NewCollector(colly.MaxDepth(2))
c.OnHTML("a[href]", func(e *colly.HTMLElement) {
link := e.Request.AbsoluteURL(e.Attr("href"))
fmt.Println("found:", link)
e.Request.Visit(link)
})
c.OnHTML("h1", func(e *colly.HTMLElement) {
fmt.Println("heading:", e.Text)
})
if err := c.Visit("http://127.0.0.1:8080/"); err != nil {
log.Fatal(err)
}
}
The home page is depth 1, and the two detail pages are depth 2. The heading of the home page is printed last because the synchronous visits of the children happen inside the link handler, before the remaining handlers of the parent page.
$ go run main.go found: http://127.0.0.1:8080/falcon.html heading: Peregrine falcon found: http://127.0.0.1:8080/fox.html heading: Arctic fox heading: Falcon shop
Handling errors
A failed request calls OnError with the response and the error.
Visit also returns the error to its caller, so both styles can be
combined.
package main
import (
"fmt"
"github.com/gocolly/colly/v2"
)
func main() {
c := colly.NewCollector()
c.OnError(func(r *colly.Response, err error) {
fmt.Println("error:", r.StatusCode, err)
})
if err := c.Visit("http://127.0.0.1:8080/missing.html"); err != nil {
fmt.Println("visit returned:", err)
}
}
The missing page returns 404, which Colly reports as an error. The handler
prints the status, and the returned error is printed as well; a caller that
ignores the return value would still see the OnError output.
$ go run main.go error: 404 Not Found visit returned: Not Found
Setting the user agent
Request headers can be read and changed in OnRequest. The
UserAgent field of the collector sets the agent for every request.
package main
import (
"fmt"
"log"
"github.com/gocolly/colly/v2"
)
func main() {
c := colly.NewCollector()
c.UserAgent = "zetcode-demo/1.0"
c.OnRequest(func(r *colly.Request) {
fmt.Println("user-agent:", r.Headers.Get("User-Agent"))
})
if err := c.Visit("http://127.0.0.1:8080/"); err != nil {
log.Fatal(err)
}
}
The handler sees the value assigned before Visit. Sites that block
the default agent can be asked politely with an identifying string.
$ go run main.go user-agent: zetcode-demo/1.0
Collecting results into JSON
The handlers can accumulate values in ordinary Go variables. This example parses the prices, builds a slice of structs and encodes it as indented JSON.
package main
import (
"encoding/json"
"fmt"
"log"
"os"
"strconv"
"github.com/gocolly/colly/v2"
)
type Product struct {
SKU string `json:"sku"`
Name string `json:"name"`
Price int `json:"price"`
}
func main() {
c := colly.NewCollector()
var products []Product
c.OnHTML("li.product", func(e *colly.HTMLElement) {
price, err := strconv.Atoi(e.ChildText(".price"))
if err != nil {
return
}
products = append(products, Product{
SKU: e.Attr("data-sku"),
Name: e.ChildText("a"),
Price: price,
})
})
if err := c.Visit("http://127.0.0.1:8080/"); err != nil {
log.Fatal(err)
}
enc := json.NewEncoder(os.Stdout)
enc.SetIndent("", " ")
if err := enc.Encode(products); err != nil {
log.Fatal(err)
}
fmt.Println("count:", len(products))
}
The struct tags control the JSON field names, and the encoder writes the slice directly to standard output. The fields keep the order in which Colly visits the products.
$ go run main.go
[
{
"sku": "F-1",
"name": "Peregrine falcon",
"price": 120
},
{
"sku": "F-2",
"name": "Arctic fox",
"price": 90
}
]
count: 2
Asynchronous scraping
With Async enabled, Visit returns immediately and the
requests run in parallel. Wait blocks until all of them finish.
Shared results need a mutex, and the output needs sorting to be
reproducible.
package main
import (
"fmt"
"log"
"sort"
"sync"
"github.com/gocolly/colly/v2"
)
func main() {
c := colly.NewCollector()
c.Async = true
var mu sync.Mutex
var titles []string
c.OnHTML("title", func(e *colly.HTMLElement) {
mu.Lock()
titles = append(titles, e.Text)
mu.Unlock()
})
urls := []string{
"http://127.0.0.1:8080/",
"http://127.0.0.1:8080/falcon.html",
"http://127.0.0.1:8080/fox.html",
}
for _, u := range urls {
if err := c.Visit(u); err != nil {
log.Fatal(err)
}
}
c.Wait()
sort.Strings(titles)
fmt.Println(titles)
}
Three pages are fetched concurrently, their titles are collected under a mutex, and the slice is sorted before printing. Without sorting, the order would change between runs.
$ go run main.go [Arctic fox Falcon shop Peregrine falcon]
Source
In this article we have shown how to scrape pages with Colly: reading the title, resolving links, extracting structured data, the callback order, following links with a depth limit, handling errors, setting the user agent, encoding results as JSON, and scraping asynchronously.
Author
List all Go tutorials.