Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug(engine): goroutine leaks #170

Open
9 tasks
steebchen opened this issue Feb 16, 2024 · 0 comments
Open
9 tasks

bug(engine): goroutine leaks #170

steebchen opened this issue Feb 16, 2024 · 0 comments

Comments

@steebchen
Copy link
Member

steebchen commented Feb 16, 2024 •
edited

List of goroutine leaks which are currently ignored, but should be investigate if they could become a problem.

Related code:

hatchet/examples/loadtest/cli/cli_e2e_test.go

Lines 54 to 57 in 00111d8

goleak.IgnoreTopFunction("go.opencensus.io/stats/view.(*worker).start"),
goleak.IgnoreTopFunction("google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run"),
goleak.IgnoreTopFunction("internal/poll.runtime_pollWait"),
goleak.IgnoreTopFunction("google.golang.org/grpc/internal/transport.(*controlBuffer).get"),

Goroutine leaks:

  • go.opencensus.io/stats/view.(*worker).start

    [Goroutine 6 in state select, with go.opencensus.io/stats/view.(*worker).start on top of the stack:
go.opencensus.io/stats/view.(*worker).start(0xc0002feb80)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/[email protected]/stats/view/worker.go:292  0x128
created by go.opencensus.io/stats/view.init.0 in goroutine 1
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/[email protected]/stats/view/worker.go:34  0xf4

  • google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run

     Goroutine 23 in state select, with google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run on top of the stack:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000373500, {0x102fde440, 0xc00004aaf0})
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76  0x150
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 54
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52  0x1f8

  • google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run

     Goroutine 24 in state select, with google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run on top of the stack:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000373530, {0x102fde440, 0xc00004ab40})
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76  0x150
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 54
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52  0x1f8

  • google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run

     Goroutine 25 in state select, with google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run on top of the stack:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000373560, {0x102fde440, 0xc00004ab90})
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76  0x150
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 54
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52  0x1f8
 Goroutine 40 in state IO wait, with internal/poll.runtime_pollWait on top of the stack:
internal/poll.runtime_pollWait(0x14d67ce38, 0x72)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/runtime/netpoll.go:343  0xa0
internal/poll.(*pollDesc).wait(0xc00052c0a0, 0xc0000dd200?, 0x0)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/internal/poll/fd_poll_runtime.go:84  0xb8
internal/poll.(*pollDesc).waitRead(...)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00052c080, {0xc0000dd200, 0x900, 0x900})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/internal/poll/fd_unix.go:164  0x2e0
net.(*netFD).Read(0xc00052c080, {0xc0000dd200, 0x900, 0x900})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/net/fd_posix.go:55  0x48
net.(*conn).Read(0xc00051e018, {0xc0000dd200, 0x900, 0x900})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/net/net.go:179  0x8c
crypto/tls.(*atLeastReader).Read(0xc00069a870, {0xc0000dd200, 0x900, 0x900})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:805  0x7c
bytes.(*Buffer).ReadFrom(0xc00060a2a8, {0x102fd69b8, 0xc00069a870})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/bytes/buffer.go:211  0xf4
crypto/tls.(*Conn).readFromUntil(0xc00060a000, {0x14d67cf30?, 0xc00051e018}, 0x5)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:827  0x18c
crypto/tls.(*Conn).readRecordOrCCS(0xc00060a000, 0x0)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:625  0x480
crypto/tls.(*Conn).readRecord(...)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:587
crypto/tls.(*Conn).Read(0xc00060a000, {0xc0005e2000, 0x8000, 0x50000c0003bfb38?})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:1369  0x1c4
bufio.(*Reader).Read(0xc000863d80, {0xc000596200, 0x9, 0x9})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/bufio/bufio.go:244  0x390
io.ReadAtLeast({0x102fd5e58, 0xc000863d80}, {0xc000596200, 0x9, 0x9}, 0x9)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/io/io.go:335  0xcc
io.ReadFull(...)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc000596200, 0x9, 0x9}, {0x102fd5e58, 0xc000863d80})
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237  0x68
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0005961c0)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498  0xbc
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00057c240, 0xc00060a000?)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1587  0x1e8
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 26
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:398  0x2234

  • google.golang.org/grpc/internal/transport.(*controlBuffer).get

     Goroutine 41 in state select, with google.golang.org/grpc/internal/transport.(*controlBuffer).get on top of the stack:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0004e4460, 0x1)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:418  0x120
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0000e77a0)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:552  0xf0
google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:452  0x114
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 26
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:450  0x288c

  • google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run

     Goroutine 55 in state select, with google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run on top of the stack:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00012b920, {0x102fde440, 0xc00004ac30})
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76  0x150
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 10
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52  0x1f8

  • google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run

     Goroutine 56 in state select, with google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run on top of the stack:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00012b950, {0x102fde440, 0xc00004ad70})
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76  0x150
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 10
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52  0x1f8

  • google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run

     Goroutine 57 in state select, with google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run on top of the stack:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00012b980, {0x102fde440, 0xc00004adc0})
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76  0x150
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 10
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52  0x1f8
 Goroutine 61 in state IO wait, with internal/poll.runtime_pollWait on top of the stack:
internal/poll.runtime_pollWait(0x14d67cd40, 0x72)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/runtime/netpoll.go:343  0xa0
internal/poll.(*pollDesc).wait(0xc00021dda0, 0xc0000ddb00?, 0x0)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/internal/poll/fd_poll_runtime.go:84  0xb8
internal/poll.(*pollDesc).waitRead(...)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00021dd80, {0xc0000ddb00, 0x900, 0x900})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/internal/poll/fd_unix.go:164  0x2e0
net.(*netFD).Read(0xc00021dd80, {0xc0000ddb00, 0x900, 0x900})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/net/fd_posix.go:55  0x48
net.(*conn).Read(0xc00051e0d0, {0xc0000ddb00, 0x900, 0x900})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/net/net.go:179  0x8c
crypto/tls.(*atLeastReader).Read(0xc000155d10, {0xc0000ddb00, 0x900, 0x900})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:805  0x7c
bytes.(*Buffer).ReadFrom(0xc00060b0a8, {0x102fd69b8, 0xc000155d10})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/bytes/buffer.go:211  0xf4
crypto/tls.(*Conn).readFromUntil(0xc00060ae00, {0x14d67cf30?, 0xc00051e0d0}, 0x5)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:827  0x18c
crypto/tls.(*Conn).readRecordOrCCS(0xc00060ae00, 0x0)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:625  0x480
crypto/tls.(*Conn).readRecord(...)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:587
crypto/tls.(*Conn).Read(0xc00060ae00, {0xc000440000, 0x8000, 0x50000c0005258b8?})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:1369  0x1c4
bufio.(*Reader).Read(0xc00018f140, {0xc0000d63c0, 0x9, 0x9})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/bufio/bufio.go:244  0x390
io.ReadAtLeast({0x102fd5e58, 0xc00018f140}, {0xc0000d63c0, 0x9, 0x9}, 0x9)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/io/io.go:335  0xcc
io.ReadFull(...)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc0000d63c0, 0x9, 0x9}, {0x102fd5e58, 0xc00018f140})
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237  0x68
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0000d6380)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498  0xbc
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00057cd80, 0xc00060ae00?)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1587  0x1e8
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 14
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:398  0x2234

  • google.golang.org/grpc/internal/transport.(*controlBuffer).get

     Goroutine 62 in state select, with google.golang.org/grpc/internal/transport.(*controlBuffer).get on top of the stack:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00004b540, 0x1)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:418  0x120
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000520d20)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:552  0xf0
google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:452  0x114
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 14
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:450  0x288c
]
@steebchen steebchen changed the title Goroutine leaks bug(engine): goroutine leaks Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@steebchen