Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v1 Scheduled Jobs / Third Party Sync Migration #171

Open
wants to merge 49 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 48 commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
2afc27f
Added a scheduled job manager to handle the creation of cron jobs loc…
mattwilshire Sep 25, 2024
6712e00
Added main logic, storing jobs to database and parsing them on retrie…
mattwilshire Oct 1, 2024
b99c788
Added ScheduledJobDTO for API endpoints and handled a possible row in…
mattwilshire Oct 1, 2024
815e30d
Fix failing maven tests.
mattwilshire Oct 1, 2024
09b28bf
Added SQL migration. Removed isAllowedExecutionOverlap, as the Disall…
mattwilshire Oct 2, 2024
97b0dfb
Changed conditional property to match current quartz condition.
mattwilshire Oct 2, 2024
542c530
Attached the job properties to the Job Type in the enum definition
mattwilshire Oct 2, 2024
596970b
Added Slack Notification for failed third party sync
mattwilshire Oct 2, 2024
08812c4
Fix failing maven tests
mattwilshire Oct 2, 2024
4233988
Trigger mvn tests again
mattwilshire Oct 2, 2024
023b65d
Fix maven tests ?
mattwilshire Oct 2, 2024
0c27798
Fix exception thrown if @Value is missing
mattwilshire Oct 2, 2024
44b00ec
Changed condition on property value
mattwilshire Oct 2, 2024
a8b4ad4
Send pollable task ID in notification and other small changes.
mattwilshire Oct 4, 2024
41a6899
Denormalized job type and job status. Added version to base scheduled…
mattwilshire Oct 11, 2024
581169f
Cleanup & Set end / start date to null depending on current job status
mattwilshire Oct 16, 2024
e1cc0d1
Added more ScheduledJob endpoints and veto the job execution if its d…
mattwilshire Oct 17, 2024
9c3b156
Added ScheduledJobResponse for structured API responses
mattwilshire Oct 17, 2024
e1b586b
Use UUID for job id
mattwilshire Oct 17, 2024
11d0841
Fix failing tests
mattwilshire Oct 18, 2024
1753761
Fix deadlock that would occur in audit table when two syncs started a…
mattwilshire Oct 21, 2024
9988f38
Clean up map
mattwilshire Oct 21, 2024
38ed481
Refactoring & Added PagerDuty incident creation on job failure
mattwilshire Oct 22, 2024
e925acd
Add flyway schema & Add PD incident title to configuration
mattwilshire Oct 22, 2024
cb6c841
Added configuration examples, Fix deadlock occurring at the method le…
mattwilshire Oct 22, 2024
d551add
Comment changes
mattwilshire Oct 23, 2024
c91f2f9
Add starting tests, ensuring they work in HSQL
mattwilshire Oct 23, 2024
026b63e
Tests check
mattwilshire Oct 23, 2024
23d5112
Fix tests by not using new application context
mattwilshire Oct 25, 2024
73739ac
Fix tests by manually adding rows flyway would handle that HSQL doesn…
mattwilshire Oct 25, 2024
01684cd
Review comments to fix up SQL schema
mattwilshire Oct 25, 2024
685f707
Review comments, ZonedDateTime & Foreign Key, BIT changes
mattwilshire Oct 25, 2024
abd3493
Capture PathVariable using UUID class
mattwilshire Oct 25, 2024
a7f7b1e
Review comment changes and refactoring
mattwilshire Oct 25, 2024
09c5460
Revert scheduler change
mattwilshire Oct 29, 2024
217dd67
Change endpoints back to POST methods
mattwilshire Oct 29, 2024
40f9568
Rename Job Status and Type, extra logging
mattwilshire Oct 30, 2024
26d3241
Use HashSet instead of list, throw run time error if duplicate uuids …
mattwilshire Oct 30, 2024
f43cf29
Custom timeout for third party sync in application.properties
mattwilshire Oct 30, 2024
6e4f247
Add metric for third party syncs that were scheduled and pagerduty in…
mattwilshire Oct 30, 2024
88aa990
Renamed Status enum for WS response, disallow triggering disabled jobs
mattwilshire Oct 30, 2024
97eaa0c
Added auto incremented primary key
mattwilshire Oct 30, 2024
9e0d560
Fix tests
mattwilshire Oct 30, 2024
a9a1336
Small refactor
mattwilshire Oct 31, 2024
88fa2ea
Remove unique key, added insert statements for hsqldb
mattwilshire Nov 1, 2024
069b660
Log pollable task and scheduled job url on PD failure
mattwilshire Nov 4, 2024
78167c4
Added Integration Tests, Unique key on UUID column
mattwilshire Nov 4, 2024
fd4ba88
Constructor Autowire for WS
mattwilshire Nov 4, 2024
91912ac
Update tests to use ID watcher and removed redundant tests
mattwilshire Nov 6, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
175 changes: 175 additions & 0 deletions webapp/src/main/java/com/box/l10n/mojito/entity/ScheduledJob.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
package com.box.l10n.mojito.entity;

import com.box.l10n.mojito.json.ObjectMapper;
import com.box.l10n.mojito.service.scheduledjob.ScheduledJobProperties;
import jakarta.persistence.Basic;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.ForeignKey;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import jakarta.persistence.JoinColumn;
import jakarta.persistence.ManyToOne;
import jakarta.persistence.PostLoad;
import jakarta.persistence.Table;
import jakarta.persistence.Transient;
import java.time.ZonedDateTime;
import org.hibernate.envers.Audited;

@Audited
@Entity
@Table(name = "scheduled_job")
public class ScheduledJob {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we extend BaseEntity here, which will then provide the id field handling in the parent class? Might have missed a previous conversation about this

@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;

@Basic(optional = false)
@Column(name = "uuid")
private String uuid;

@ManyToOne
@JoinColumn(
name = "repository_id",
foreignKey = @ForeignKey(name = "FK__SCHEDULED_JOB__IMPORT_REPOSITORY__ID"))
private Repository repository;
garionpin marked this conversation as resolved.
Show resolved Hide resolved

@ManyToOne
@JoinColumn(name = "job_type", foreignKey = @ForeignKey(name = "FK__JOB_TYPE__JOB_TYPE_ID"))
private ScheduledJobType jobType;

@Column(name = "cron")
private String cron;

@Transient private ScheduledJobProperties properties;
garionpin marked this conversation as resolved.
Show resolved Hide resolved

@Basic(optional = false)
@Column(name = "properties")
private String propertiesString;

@ManyToOne
@JoinColumn(name = "job_status", foreignKey = @ForeignKey(name = "FK__JOB_STATUS__JOB_STATUS_ID"))
private ScheduledJobStatus jobStatus;

@Column(name = "start_date")
private ZonedDateTime startDate;

@Column(name = "end_date")
private ZonedDateTime endDate;

@Basic(optional = false)
@Column(name = "enabled")
private Boolean enabled = true;

@PostLoad
public void deserializeProperties() {
ObjectMapper objectMapper = new ObjectMapper();
try {
this.properties =
objectMapper.readValue(propertiesString, jobType.getEnum().getPropertiesClass());
} catch (Exception e) {
throw new RuntimeException(
"Failed to deserialize properties '"
+ propertiesString
+ "' for class: "
+ jobType.getEnum().getPropertiesClass().getName(),
e);
}
}

public Long getId() {
return id;
}

public void setId(Long id) {
this.id = id;
}

public String getUuid() {
return uuid;
}

public void setUuid(String uuid) {
this.uuid = uuid;
}

public Repository getRepository() {
return repository;
}

public void setRepository(Repository repository) {
this.repository = repository;
}

public ScheduledJobType getJobType() {
return jobType;
}

public void setJobType(ScheduledJobType jobType) {
this.jobType = jobType;
}

public String getCron() {
return cron;
}

public void setCron(String cron) {
this.cron = cron;
}

public ScheduledJobProperties getProperties() {
return properties;
}

public void setProperties(ScheduledJobProperties properties) {
this.properties = properties;

ObjectMapper objectMapper = new ObjectMapper();
try {
this.propertiesString = objectMapper.writeValueAsString(this.properties);
} catch (Exception e) {
throw new RuntimeException("Failed to serialize properties", e);
}
}

public String getPropertiesString() {
return propertiesString;
}

public void setPropertiesString(String properties) {
this.propertiesString = properties;
}

public ScheduledJobStatus getJobStatus() {
return jobStatus;
}

public void setJobStatus(ScheduledJobStatus jobStatus) {
this.jobStatus = jobStatus;
}

public ZonedDateTime getStartDate() {
return startDate;
}

public void setStartDate(ZonedDateTime startDate) {
this.startDate = startDate;
}

public ZonedDateTime getEndDate() {
return endDate;
}

public void setEndDate(ZonedDateTime endDate) {
this.endDate = endDate;
}

public Boolean getEnabled() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be named isEnabled()?

return enabled;
}

public void setEnabled(Boolean enabled) {
this.enabled = enabled;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
package com.box.l10n.mojito.entity;

import static org.hibernate.envers.RelationTargetAuditMode.NOT_AUDITED;

import com.box.l10n.mojito.rest.View;
import com.fasterxml.jackson.annotation.JsonView;
import jakarta.persistence.Basic;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.EnumType;
import jakarta.persistence.Enumerated;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
import org.hibernate.envers.Audited;

@Entity
@Table(name = "scheduled_job_status_type")
@Audited(targetAuditMode = NOT_AUDITED)
public class ScheduledJobStatus extends BaseEntity {
@Id private Long id;

@Basic(optional = false)
@Column(name = "name")
@Enumerated(EnumType.STRING)
@JsonView(View.Repository.class)
private com.box.l10n.mojito.service.scheduledjob.ScheduledJobStatus jobStatus;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use an import here? This looks funky to me

Copy link
Member Author

@mattwilshire mattwilshire Nov 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No we can't, the name of the imported class matches the name of the current class, the definition needs to be this specific.


public com.box.l10n.mojito.service.scheduledjob.ScheduledJobStatus getEnum() {
return jobStatus;
}

public void setJobStatus(com.box.l10n.mojito.service.scheduledjob.ScheduledJobStatus jobStatus) {
this.jobStatus = jobStatus;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
package com.box.l10n.mojito.entity;

import static org.hibernate.envers.RelationTargetAuditMode.NOT_AUDITED;

import com.box.l10n.mojito.rest.View;
import com.fasterxml.jackson.annotation.JsonView;
import jakarta.persistence.Basic;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.EnumType;
import jakarta.persistence.Enumerated;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
import org.hibernate.envers.Audited;

@Entity
@Table(name = "scheduled_job_type")
@Audited(targetAuditMode = NOT_AUDITED)
public class ScheduledJobType extends BaseEntity {
@Id private Long id;

@Basic(optional = false)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hold on. Why is this needed? The column is not not nullable, so should the config not reflect that. This Basic property to my understanding enforces that the property is not null locally but not at a db level (I might be wrong)

Copy link
Member Author

@mattwilshire mattwilshire Nov 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have updated the other columns that are not nullable to have this, I noticed it was on other entity objects so I decided to use it. It looks like you can generate SQL schemas off of entity mappings, not that we would need to use this but best practice to have annotations like this!

@Column(name = "name")
@Enumerated(EnumType.STRING)
@JsonView(View.Repository.class)
private com.box.l10n.mojito.service.scheduledjob.ScheduledJobType jobType;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar comment as before

Copy link
Member Author

@mattwilshire mattwilshire Nov 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check previous comment.


@Override
public Long getId() {
return id;
}

@Override
public void setId(Long id) {
this.id = id;
}

public com.box.l10n.mojito.service.scheduledjob.ScheduledJobType getEnum() {
return jobType;
}

public void setEnum(com.box.l10n.mojito.service.scheduledjob.ScheduledJobType jobType) {
this.jobType = jobType;
}
}
Loading