Goroutines and the Scheduler, and What They Cost

The first concurrent Go program everyone writes kicks off some work with go, prints something, and then exits without the work ever happening. Nothing errors, and the exit status is zero. It is the most common way to lose an afternoon to Go's concurrency model.

Go's answer to threads is the goroutine, and the design goal was to make starting one cheap enough that you stop asking whether you can afford it. That goal was met, which is why the interesting failure is not the goroutine that crashes but the one that blocks forever, holds whatever it holds, and never appears in a log line.

When main returns, the program is over

Here is the program that does nothing:

package main

import (
	"fmt"
	"time"
)

func main() {
	go func() {
		time.Sleep(50 * time.Millisecond)
		fmt.Println("worker: finished")
	}()
	fmt.Println("main: returning now")
}

main: returning now
exit code: 0

The worker's line never appears. Go does not wait for goroutines at exit: when main returns, the runtime terminates the process and every other goroutine dies wherever it happened to be. There is no shutdown sequence and no warning. Liveness is something you arrange, with a channel or with sync.WaitGroup.

A goroutine costs about two kilobytes to start

A goroutine is not a thread. It is a function invocation with its own stack, scheduled by the Go runtime onto an operating system thread. The stack starts at about two kilobytes and grows by copying when the function needs more. An OS thread's stack is reserved up front in megabytes, which is why a Java or C service struggles past a few thousand threads while a Go service shrugs at a hundred thousand goroutines.

Measured on the machine this lesson was written on, a program that starts 100,000 goroutines which each sleep for two seconds adds about 59 MB of live heap, roughly 605 bytes per goroutine, and starting all of them takes 117 milliseconds.

goroutines at start: 1
spawned 100000 goroutines in 117ms
goroutines now: 100001
heap: 160 KiB before, 59244 KiB with 100000 live goroutines, 605 bytes each
goroutines after wait: 1

Those numbers come from one machine and one Go release, so treat the ratio as the point rather than the digits. A goroutine per request, per connection or per item is rarely the wrong decision. The runtime also keeps the thread count far below the goroutine count: a program holding 5,001 goroutines, all asleep, reported 10 OS threads.

GOMAXPROCS: 8 NumCPU: 8
goroutines: 5001 OS threads: 10

The scheduler is the thing in between, and knowing its shape explains most of the surprising behaviour you will meet.

The scheduler juggles three things: G, M and P

The runtime's scheduler is described in terms of three types, named after goroutine, machine and processor:

The rule that makes the model click is that a G only runs while it holds a P. There are exactly GOMAXPROCS of them, so at most that many goroutines execute Go code at any instant. When a goroutine blocks on a channel, a mutex or a network read, the runtime parks it and hands the P to another runnable goroutine. No thread is wasted waiting.

A system call is different, because an M is stuck inside the kernel. The runtime detaches the P from that thread and gives it to a free or newly created thread. This is why a program doing hundreds of concurrent blocking reads grows a large number of threads, and why that growth is a symptom worth noticing rather than a bug in the scheduler.

A goroutine that never blocks is eventually preempted, so a tight loop cannot starve the rest of the program. Each P keeps a local run queue, and a P that runs dry steals work from another's queue.

go starts a goroutine and tells you nothing else

The syntax is a keyword and a call. That is the whole language feature.

go doWork(item)

Two consequences follow from how little it does. The call returns immediately, so the function it started has not necessarily run by the next line, and there is no value to check: a go statement has no result, not even an error. A panic inside the new goroutine is not delivered to the parent. It unwinds that goroutine's own stack and crashes the process unless something in that goroutine recovers it.

The other consequence is the loop variable, which used to be a trap. Before Go 1.22, the loop variable was a single variable reused on every iteration, so goroutines started in a loop all observed the final value. Current Go gives each iteration a fresh variable, so this works as written:

sums := make([]int, 5)
var wg sync.WaitGroup
for i := 0; i < 5; i++ {
	wg.Add(1)
	go func() {
		defer wg.Done()
		sums[i] = i * 10
	}()
}
wg.Wait()

[0 10 20 30 40]

Run under -race it is clean, because each goroutine writes a different element. The change is tied to the language version in go.mod, so a module declaring an older Go version keeps the old behaviour. If you are maintaining code that predates it, or you want the intent to be obvious to a reader who does not know which rule applies, pass the value in:

for i := 0; i < 5; i++ {
	wg.Add(1)
	go func(i int) {
		defer wg.Done()
		sums[i] = i * 10
	}(i)
}

An argument is evaluated at the call, so each goroutine owns its own copy regardless of the language version. Passing it in removes the question for a reader who does not know which rule applies to your module.

sync.WaitGroup is a counter with a wait

A WaitGroup is a counter of outstanding work. Add increments it, Done decrements it, and Wait blocks until it reaches zero. The zero value is usable, so a package-level WaitGroup needs no construction.

func slow(id int) int {
	time.Sleep(time.Second)
	return id * id
}

func main() {
	start := time.Now()
	serial := 0
	for id := 1; id <= 4; id++ {
		serial += slow(id)
	}
	fmt.Printf("serial:   %d in %v\n", serial, time.Since(start).Round(10*time.Millisecond))

	start = time.Now()
	totals := make([]int, 4)
	var wg sync.WaitGroup
	for id := 1; id <= 4; id++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			totals[id-1] = slow(id)
		}()
	}
	wg.Wait()
	concurrent := 0
	for _, v := range totals {
		concurrent += v
	}
	fmt.Printf("concurrent: %d in %v\n", concurrent, time.Since(start).Round(10*time.Millisecond))
}

serial:   30 in 4s
concurrent: 30 in 1s

Four seconds of sleeping becomes one, and the totals agree. The measured speedup depends entirely on the work: sleeping is perfectly concurrent, arithmetic on one CPU is not, and a function that holds a shared lock serialises no matter how many goroutines call it.

Three mistakes around WaitGroup account for nearly all of its misuse. Calling Add inside the goroutine is a race, because Wait may run before the goroutine has started, see a zero counter and return immediately; the Add belongs before the go statement. A goroutine that calls Add after Wait has already observed zero panics with "WaitGroup is reused before previous Wait has returned". And passing a WaitGroup by value to a helper copies the counter, so the helper's Done decrements the copy. Pass a pointer, or capture it in the closure.

A goroutine blocked forever is a leak

Go has no runtime that cancels a goroutine for you. A goroutine waiting on a channel nobody will send to is never detected and never killed, and the memory it holds stays held. A service that leaks one goroutine per request grows until the process is killed, and the symptom arrives hours after the mistake.

You can see it in the goroutine count:

for i := 0; i < 50; i++ {
	go func() {
		<-make(chan struct{}) // waits forever; nobody can ever send
	}()
	go func() {
		time.Sleep(10 * time.Millisecond)
	}()
}

goroutines: 1 before, 101 after 100 goroutines, 51 still running

The fifty sleeping goroutines finished. The fifty waiting on a channel they created themselves never can, because the channel is unreachable from anywhere else.

The goroutine dump tells you where the stuck ones are. Send SIGQUIT to a Go process, or set GOTRACEBACK=all before it crashes, and the runtime prints every goroutine with its state and stack. Here are two headers from a program with one goroutine waiting to receive and another waiting to send:

goroutine 19 gp=0x1d250cb023c0 m=nil [chan receive]:
runtime.gopark(0x70010100000000?, 0x102f859a8?, 0x0?, 0x66?, 0x70?)
	/opt/homebrew/Cellar/go/1.26.3/libexec/src/runtime/proc.go:462 +0xbc
runtime.chanrecv(0x1d250ca90000, 0x0, 0x1)

goroutine 20 gp=0x1d250cb025a0 m=nil [chan send]:
runtime.gopark(0x102cb36d8?, 0x70?, 0x40?, 0x78?, 0x1d250cb06070?)
	/opt/homebrew/Cellar/go/1.26.3/libexec/src/runtime/proc.go:462 +0xbc
runtime.chansend(0x1d250cb06070, 0x102b464f8, 0x1, 0x0?)

Read the bracketed state first: [chan receive] and [chan send] say what each goroutine is waiting for. Then read down to the first line of your own code, which in a real leak names the function and line to change. In a dump with five thousand entries the shape matters: a hundred goroutines parked in the same function is one leak, not a hundred.

Something has to be able to end the wait: a buffered channel with room for the result, a select with a case that fires, or a context the caller cancels. The last of those is the third lesson in this module.

GOMAXPROCS caps concurrency, not goroutines

GOMAXPROCS is the number of Ps, so it caps how many goroutines can execute Go code simultaneously. It does not limit how many you may create. The default comes from the logical CPUs the process can use, and on Linux the runtime also reads the cgroup CPU quota, so a container limited to two CPUs runs with GOMAXPROCS at two even on a sixty-four core host. Ask for the current value with runtime.GOMAXPROCS(0), which returns the setting without changing it.

GOMAXPROCS: 8 NumCPU: 8

Raising it is almost always the wrong move. More Ps than CPUs means more contention for the same cores and more time in the scheduler, not more throughput. One case justifies it: goroutines making blocking system calls the runtime cannot detach fast enough. Confirm that with a profile before touching the setting.

Lowering it is the more defensible change, for a service that must not consume the whole machine or a test that needs single-core behaviour. An explicit value also switches off the periodic recomputation of the default, so if you set it for a benchmark, set it back.

It has nothing to do with how much work can be in flight. Ten thousand goroutines waiting on network responses all make progress with GOMAXPROCS at eight, because a parked goroutine is not competing for a P. The setting caps how many can compute at the same instant, a far smaller number than how many can be in progress.

package main

import (
	"fmt"
	"math/rand"
	"os"
	"os/signal"
	"runtime"
	"sync"
	"syscall"
	"time"
)

func fetch(id int) (string, error) {
	time.Sleep(time.Duration(50+rand.Intn(151)) * time.Millisecond)
	return fmt.Sprintf("item-%d", id), nil
}

func main() {
	start := time.Now()
	for id := 1; id <= 4; id++ {
		if _, err := fetch(id); err != nil {
			fmt.Println("error:", err)
		}
	}
	fmt.Println("serial:", time.Since(start).Round(10*time.Millisecond))

	start = time.Now()
	results := make([]string, 4)
	var wg sync.WaitGroup
	for id := 1; id <= 4; id++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			item, err := fetch(id)
			if err != nil {
				results[id-1] = "error: " + err.Error()
				return
			}
			results[id-1] = item
		}()
	}
	wg.Wait()
	fmt.Println("concurrent:", time.Since(start).Round(10*time.Millisecond), results)
}