Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Migration of Rundeck (Server Server DB) from Windows to Linux caused Quartz Scheduler misbehaving #9497

Open
wasimb-atica opened this issue Dec 12, 2024 · 8 comments
Open

Migration of Rundeck (Server Server DB) from Windows to Linux caused Quartz Scheduler misbehaving #9497

wasimb-atica opened this issue Dec 12, 2024 · 8 comments

Comments

@wasimb-atica
Copy link

wasimb-atica commented Dec 12, 2024 •
edited
Loading

After migrating the Rundeck server and database from Windows to Linux, the Quartz Scheduler is not triggering jobs at the appropriate time as defined in the cron schedules. This behavior was not present in the original Rundeck server on Windows.

1. Old Rundeck Server:

  • Version: 5.2
  • Installation type: JAR file
  • OS: Windows
  • Database: PostgreSQL (DB Name: rundeck)

2. New Rundeck Server:

  • Version: 5.7
  • Installation type: System Service
  • OS: Linux
  • Database: PostgreSQL (DB Name: rundeck2)

AWS Region: Both servers are running in the same AWS region: us-east-1

Steps to Reproduce

1.Migrate the Rundeck database from the old server (Windows) to the new server (Linux) using AWS DMS to avoid impacting the original database.
2. Confirm that the data, including job definitions, has been successfully migrated.
3. Start the new Rundeck server and enable job schedules and executions.
4. Observe that jobs do not get triggered as per their defined schedules.

Expected behavior

Jobs should trigger at the correct times based on the defined cron schedules. Most jobs are scheduled to the Asia/Kolkata timezone.

What I tried

  1. Disabling and re-enabling the job schedules and executions on the new Rundeck server.
  2. Increasing the Quartz Scheduler Max Thread Count to 20 (previously set to the default value of 10).

Additional context

The migration steps were executed as follows:

1. Used AWS DMS to migrate the PostgreSQL database from rundeck (Windows) to rundeck2 (Linux).

2. Verified the data and job definitions were migrated successfully.

3. Despite these efforts, the Quartz Scheduler does not behave as expected, causing job execution failures to trigger.

4. Please let me know if additional configurations are required to troubleshoot this.
@wasimb-atica
Copy link
Author

wasimb-atica commented Dec 14, 2024 •
edited
Loading

I think I should try this
#1305
Not sure, this will work since this issue and resolution suggested is almost 9 years ago.
But one thing is that, upon checking the workflow_worflow_step table, it is not having any indexes.

@wasimb-atica
Copy link
Author

wasimb-atica commented Dec 15, 2024 •
edited
Loading

I have created the required indexes mentioned #1305 here for workflow_workflow_step and increased the JVM HEAP memory (Max 4GB and Min 1GB) and thread pool count (100)
But still jobs are not getting triggered at scheduled time.

@w00dst0ck
Copy link

Have you tried to export and import your projects?

@wasimb-atica
Copy link
Author

No, I haven't tried it. I have terminated my older (windows) rundeck server and db belonging to this as well.

@w00dst0ck
Copy link

Would give it try on the new instance. It's possible that missing db records would recreated while importing a project.

@wasimb-atica
Copy link
Author

wasimb-atica commented Dec 16, 2024 •
edited
Loading

Yes, it would recreate while using export/import feature of rundeck but currently I don't have previous server's database.
I kept new server db in sync with old rundeck database (using AWS DMS) as long as I tweak few settings on new server, and later deleted the older db.

@wasimb-atica
Copy link
Author

wasimb-atica commented Dec 16, 2024 •
edited
Loading

Upon checking the logs for the service.log
grep -i "quartz" /var/log/rundeck/service.log | tail -n 50

org.quartz.SchedulerException: Job threw an unhandled exception. at org.quartz.core.JobRunShell.run(JobRunShell.java:213) ~[quartz-2.3.2.jar!/:?] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) ~[quartz-2.3.2.jar!/:?] at rundeck.quartzjobs.ExecutionJob.initialize(ExecutionJob.groovy:343) ~[classes!/:?] at rundeck.quartzjobs.ExecutionJob$initialize.callCurrent(Unknown Source) ~[?:?] at rundeck.quartzjobs.ExecutionJob.execute_internal(ExecutionJob.groovy:122) ~[classes!/:?] at rundeck.quartzjobs.ExecutionJob$_execute_closure1.doCall(ExecutionJob.groovy:101) ~[classes!/:?] at rundeck.quartzjobs.ExecutionJob$_execute_closure1.doCall(ExecutionJob.groovy) ~[classes!/:?] at rundeck.quartzjobs.ExecutionJob.execute(ExecutionJob.groovy:100) ~[classes!/:?] at org.quartz.core.JobRunShell.run(JobRunShell.java:202) ~[quartz-2.3.2.jar!/:?] at rundeck.quartzjobs.ExecutionJob.initialize(ExecutionJob.groovy:343) ~[classes!/:?] at rundeck.quartzjobs.ExecutionJob$initialize.callCurrent(Unknown Source) ~[?:?] at rundeck.quartzjobs.ExecutionJob.execute_internal(ExecutionJob.groovy:122) ~[classes!/:?] at rundeck.quartzjobs.ExecutionJob$_execute_closure1.doCall(ExecutionJob.groovy:101) ~[classes!/:?] at rundeck.quartzjobs.ExecutionJob$_execute_closure1.doCall(ExecutionJob.groovy) ~[classes!/:?] at rundeck.quartzjobs.ExecutionJob.execute(ExecutionJob.groovy:100) ~[classes!/:?] at org.quartz.core.JobRunShell.run(JobRunShell.java:202) ~[quartz-2.3.2.jar!/:?] at rundeck.quartzjobs.ExecutionJob.initialize(ExecutionJob.groovy:343) ~[classes!/:?] at rundeck.quartzjobs.ExecutionJob$initialize.callCurrent(Unknown Source) ~[?:?] at rundeck.quartzjobs.ExecutionJob.execute_internal(ExecutionJob.groovy:122) ~[classes!/:?] at rundeck.quartzjobs.ExecutionJob$_execute_closure1.doCall(ExecutionJob.groovy:101) ~[classes!/:?] at rundeck.quartzjobs.ExecutionJob$_execute_closure1.doCall(ExecutionJob.groovy) ~[classes!/:?] at rundeck.quartzjobs.ExecutionJob.execute(ExecutionJob.groovy:100) ~[classes!/:?] at org.quartz.core.JobRunShell.run(JobRunShell.java:202) ~[quartz-2.3.2.jar!/:?] org.quartz.JobExecutionException: org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: org.springframework.dao.DataIntegrityViolationException: could not execute statement; SQL [n/a]; constraint [workflow_step_pkey]; nested exception is org.hibernate.exception.ConstraintViolationException: could not execute statement] at org.quartz.core.JobRunShell.run(JobRunShell.java:218) ~[quartz-2.3.2.jar!/:?] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) ~[quartz-2.3.2.jar!/:?] Caused by: org.quartz.SchedulerException: Job threw an unhandled exception. at org.quartz.core.JobRunShell.run(JobRunShell.java:213) ~[quartz-2.3.2.jar!/:?] at rundeck.quartzjobs.ExecutionJob.initialize(ExecutionJob.groovy:343) ~[classes!/:?] at rundeck.quartzjobs.ExecutionJob$initialize.callCurrent(Unknown Source) ~[?:?] at rundeck.quartzjobs.ExecutionJob.execute_internal(ExecutionJob.groovy:122) ~[classes!/:?] at rundeck.quartzjobs.ExecutionJob$_execute_closure1.doCall(ExecutionJob.groovy:101) ~[classes!/:?] at rundeck.quartzjobs.ExecutionJob$_execute_closure1.doCall(ExecutionJob.groovy) ~[classes!/:?] at rundeck.quartzjobs.ExecutionJob.execute(ExecutionJob.groovy:100) ~[classes!/:?] at org.quartz.core.JobRunShell.run(JobRunShell.java:202) ~[quartz-2.3.2.jar!/:?] at rundeck.quartzjobs.ExecutionJob.initialize(ExecutionJob.groovy:343) ~[classes!/:?] at rundeck.quartzjobs.ExecutionJob$initialize.callCurrent(Unknown Source) ~[?:?] at rundeck.quartzjobs.ExecutionJob.execute_internal(ExecutionJob.groovy:122) ~[classes!/:?] at rundeck.quartzjobs.ExecutionJob$_execute_closure1.doCall(ExecutionJob.groovy:101) ~[classes!/:?] at rundeck.quartzjobs.ExecutionJob$_execute_closure1.doCall(ExecutionJob.groovy) ~[classes!/:?] at rundeck.quartzjobs.ExecutionJob.execute(ExecutionJob.groovy:100) ~[classes!/:?] at org.quartz.core.JobRunShell.run(JobRunShell.java:202) ~[quartz-2.3.2.jar!/:?] at rundeck.quartzjobs.ExecutionJob.initialize(ExecutionJob.groovy:343) ~[classes!/:?] at rundeck.quartzjobs.ExecutionJob$initialize.callCurrent(Unknown Source) ~[?:?] at rundeck.quartzjobs.ExecutionJob.execute_internal(ExecutionJob.groovy:122) ~[classes!/:?] at rundeck.quartzjobs.ExecutionJob$_execute_closure1.doCall(ExecutionJob.groovy:101) ~[classes!/:?] at rundeck.quartzjobs.ExecutionJob$_execute_closure1.doCall(ExecutionJob.groovy) ~[classes!/:?] at rundeck.quartzjobs.ExecutionJob.execute(ExecutionJob.groovy:100) ~[classes!/:?] at org.quartz.core.JobRunShell.run(JobRunShell.java:202) ~[quartz-2.3.2.jar!/:?]

@w00dst0ck
Copy link

Yes, it would recreate while using export/import feature of rundeck but currently I don't have previous server's database. I kept new server db in sync with old rundeck database (using AWS DMS) as long as I tweak few settings on new server, and later deleted the older db.

I understood that. But why not try exporting the project in the new instance and importing it into a new project?
I would then temporarily deactivate the base project to prevent jobs from being executed twice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@w00dst0ck @wasimb-atica