Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

backupDSLoop exception: mark end of backup operation: driver: bad connection; driver: bad connection #4535

Open
kalavt opened this issue Apr 2, 2024 · 2 comments
Open

backupDSLoop exception: mark end of backup operation: driver: bad connection; driver: bad connection #4535

kalavt opened this issue Apr 2, 2024 · 2 comments
Labels
Could not reproduce

Comments

@kalavt
Copy link

kalavt commented Apr 2, 2024 •
edited
Loading

Describe the bug
mark end of backup operation: driver: bad connection; driver: bad connection

Expected behavior
version: 1.22.1

Screenshots
`
2024-04-02T13:32:11.775Z ERROR utils.misc misc/misc.go:926 Panic detected. Application will crash. {"stack": "goroutine 430 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 0x5e\ngithub.com/rudderlabs/rudder-server/utils/misc.BugsnagNotify.func1.1()\n\t/rudder-server/utils/misc/misc.go:927 0x37e\nsync.(*Once).doSlow(0x60f1280?, 0xc002b31e18?)\n\t/usr/local/go/src/sync/once.go:74 0xbf\nsync.(*Once).Do(...)\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/rudderlabs/rudder-server/utils/misc.BugsnagNotify.func1()\n\t/rudder-server/utils/misc/misc.go:916 0xbb\npanic({0x32a8f60?, 0xc08b668dc0?})\n\t/usr/local/go/src/runtime/panic.go:914 0x21f\ngithub.com/rudderlabs/rudder-server/jobsdb.(*Handle).backupDSLoop(0xc00147f880, {0x40ae8d0, 0xc00086d860})\n\t/rudder-server/jobsdb/backup.go:105 0x345\ngithub.com/rudderlabs/rudder-server/jobsdb.(*Handle).startBackupDSLoop.func1()\n\t/rudder-server/jobsdb/jobsdb.go:1016 0x1f\ngithub.com/rudderlabs/rudder-server/jobsdb.(*Handle).startBackupDSLoop.WithBugsnag.func2()\n\t/rudder-server/utils/misc/misc.go:945 0x5c\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 573\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 0x96\n", "panic": "mark end of backup operation: driver: bad connection; driver: bad connection", "team": "Core", "goRoutines": 1169, "version": "1.22.1", "releaseStage": "development"}
2024-04-02T13:32:11.776Z ERROR utils.misc misc/misc.go:926 goroutine 430 [running]:
github.com/rudderlabs/rudder-go-kit/logger.(*logger).Fatalw(0xc001490a50, {0x38fbe93?, 0x4bf?}, {0xc089ae1500?, 0x1?, 0x4ffbcc?})
/go/pkg/mod/github.com/rudderlabs/[email protected]/logger/logger.go:332 0xa5
github.com/rudderlabs/rudder-server/utils/misc.BugsnagNotify.func1.1()
/rudder-server/utils/misc/misc.go:926 0x5f6
sync.(*Once).doSlow(0x60f1280?, 0xc002b31e18?)
/usr/local/go/src/sync/once.go:74 0xbf
sync.(*Once).Do(...)
/usr/local/go/src/sync/once.go:65
github.com/rudderlabs/rudder-server/utils/misc.BugsnagNotify.func1()
/rudder-server/utils/misc/misc.go:916 0xbb
panic({0x32a8f60?, 0xc08b668dc0?})
/usr/local/go/src/runtime/panic.go:914 0x21f
github.com/rudderlabs/rudder-server/jobsdb.(*Handle).backupDSLoop(0xc00147f880, {0x40ae8d0, 0xc00086d860})
/rudder-server/jobsdb/backup.go:105 0x345
github.com/rudderlabs/rudder-server/jobsdb.(*Handle).startBackupDSLoop.func1()
/rudder-server/jobsdb/jobsdb.go:1016 0x1f
github.com/rudderlabs/rudder-server/jobsdb.(*Handle).startBackupDSLoop.WithBugsnag.func2()
/rudder-server/utils/misc/misc.go:945 0x5c
golang.org/x/sync/errgroup.(*Group).Go.func1()
/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 573
/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 0x96

2024/04/02 13:32:11 notifying bugsnag: mark end of backup operation: driver: bad connection; driver: bad connection
2024/04/02 13:32:11 bugsnag.Notify: bugsnag/payload.deliver: invalid api key: ''
2024/04/02 13:32:11 bugsnag/sessions/publisher.publish invalid API key: ''
panic: mark end of backup operation: driver: bad connection; driver: bad connection [recovered]
panic: mark end of backup operation: driver: bad connection; driver: bad connection [recovered]
panic: mark end of backup operation: driver: bad connection; driver: bad connection

goroutine 430 [running]:
github.com/bugsnag/bugsnag-go/v2.AutoNotify({0xc06e6099f0, 0x3, 0x7f09e72068b8?})
/go/pkg/mod/github.com/bugsnag/bugsnag-go/[email protected]/bugsnag.go:114 0x3e8
panic({0x32a8f60?, 0xc08b668dc0?})
/usr/local/go/src/runtime/panic.go:914 0x21f
github.com/rudderlabs/rudder-server/utils/misc.BugsnagNotify.func1.1()
/rudder-server/utils/misc/misc.go:935 0x614
sync.(*Once).doSlow(0x60f1280?, 0xc002b31e18?)
/usr/local/go/src/sync/once.go:74 0xbf
sync.(*Once).Do(...)
/usr/local/go/src/sync/once.go:65
github.com/rudderlabs/rudder-server/utils/misc.BugsnagNotify.func1()
/rudder-server/utils/misc/misc.go:916 0xbb
panic({0x32a8f60?, 0xc08b668dc0?})
/usr/local/go/src/runtime/panic.go:914 0x21f
github.com/rudderlabs/rudder-server/jobsdb.(*Handle).backupDSLoop(0xc00147f880, {0x40ae8d0, 0xc00086d860})
/rudder-server/jobsdb/backup.go:105 0x345
github.com/rudderlabs/rudder-server/jobsdb.(*Handle).startBackupDSLoop.func1()
/rudder-server/jobsdb/jobsdb.go:1016 0x1f
github.com/rudderlabs/rudder-server/jobsdb.(*Handle).startBackupDSLoop.WithBugsnag.func2()
/rudder-server/utils/misc/misc.go:945 0x5c
golang.org/x/sync/errgroup.(*Group).Go.func1()
/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 573
/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 0x96
`
@gitcommitshow
Copy link
Contributor

gitcommitshow commented Apr 3, 2024 •
edited
Loading

@kalavt can you please share more details? How to reproduce this issue?

@Digital365Staking
Copy link

The error message you're seeing indicates that there's a problem with the database connection during the backup operation in the rudder-server application. The error message "driver: bad connection" typically means that the application is unable to establish a connection to the database⁴.

Here are a few steps you can take to troubleshoot this issue:

  1. Check the Database Connection: Ensure that the database is running and that the application has the correct connection details. If the database is running on a different server, check that the network connection between the two servers is working correctly.

  2. Check the Database Driver: The error message suggests there might be an issue with the database driver. Make sure you're using the correct driver for your database and that the driver is up-to-date.

  3. Inspect the Application Logs: The logs can provide more details about what the application was doing when the error occurred. Look for any error messages or warnings that occurred around the same time as the "driver: bad connection" error.

  4. Update Rudder Server: The error logs indicate that you're using Rudder Server version 1.22.1. If there's a newer version available, consider updating to see if the issue has been resolved in a later release.

Regarding the "invalid api key" error for Bugsnag, it seems like the API key provided to Bugsnag is not valid. You might want to check if the API key is correctly set in your configuration¹. If you're using different environments (like development and production), make sure to use the correct API key for each environment¹.

If none of these steps resolve the issue, you might want to consider reaching out to the Rudder Server community or support for more specific guidance. They might be able to provide more insight based on the specifics of your setup and the exact error messages you're seeing²⁵.

(1) Troubleshooting :: Rudder Documentation. https://docs.rudder.io/reference/8.1/administration/troubleshooting.html.
(2) javascript - Bugsnag: Invalid API key 'undefined' - Stack Overflow. https://stackoverflow.com/questions/42693851/bugsnag-invalid-api-key-undefined.
(3) Docker setup is broken · Issue #448 · rudderlabs/rudder-server. #448.
(4) backupDSLoop exception: mark end of backup operation: driver: bad .... #4535.
(5) Bugsnag | RudderStack Docs. https://www.rudderstack.com/docs/destinations/streaming-destinations/bugsnag/.
(6) undefined. https://docs.rudderstack.com/get-started/installing-and-setting-up-rudderstack/docker.
(7) undefined. segment-boneyard/analytics.js-integrations#61.
(8) undefined. https://d[2](https://stackoverflow.com/questions/42693851/bugsnag-invalid-api-key-undefined)wy8f7a9ursnm.cloudfront.net/bugsnag-3.min.js.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Could not reproduce
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kalavt @gitcommitshow @Digital365Staking