Skip to content

Asychronous import and updates when applying Fair Data Station rdf #2210

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 50 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
a8b99af
create new model FairDataStationUpload #2197
stuzart May 14, 2025
bd1bf04
include the purpose as an enum #2197
stuzart May 14, 2025
931cc22
fix silly error #2197
stuzart May 14, 2025
3de4a3e
add Policy to FairDataStationUpload #2197
stuzart May 14, 2025
fa84fdd
handle creating the FairDataStationUpload record for the submit_faird…
stuzart May 14, 2025
a66c637
queue the job #2197
stuzart May 14, 2025
dd44b80
perform the job #2197
stuzart May 15, 2025
051d335
removed forced throwing forward of exception #2197
stuzart May 15, 2025
11d7cc0
hacky first version of tracking and showing the import status #2197
stuzart May 16, 2025
d9607e1
test coverage for showing the import status, and preventing improper …
stuzart May 21, 2025
2d57a0d
show the external id alongside the upload being imported #2197
stuzart May 21, 2025
e142c35
check the errors are recorded in the task, including add the short er…
stuzart May 22, 2025
857636d
refactored shorter names for the tasks and some cody tidying with rub…
stuzart May 22, 2025
1dd41c8
add a flag whether the status should be shown, to give the ability to…
stuzart May 23, 2025
d44b5f8
action to change the show_status flag for fairdatastationupload, with…
stuzart May 27, 2025
a40e175
javascript to close the import status #2197
stuzart May 27, 2025
488206d
bootstrap based close alert icon #2197
stuzart May 29, 2025
c7f403d
cleaned up the import page, and show the import statuses in a panel #…
stuzart May 29, 2025
9929d07
submitting the TTL for update creates the record and queues job #2197
stuzart Jun 2, 2025
a2e0ad5
refactored matching_updates_in_progress to be based in investigation …
stuzart Jun 2, 2025
fc9a181
perform the update job #2197
stuzart Jun 2, 2025
32a9994
test for when the update job fails #2197
stuzart Jun 2, 2025
172876f
fix/remove failing tests #2197
stuzart Jun 2, 2025
142679b
moved obselete functional test for ignoring disabled EMT's over to th…
stuzart Jun 2, 2025
2240c2a
print some info about RDF::Query::Solution when test randomly fails #…
stuzart Jun 2, 2025
48d925f
show and update the update job status #2197
stuzart Jun 3, 2025
1b96866
better controls over preventing a new update whilst one is currently …
stuzart Jun 3, 2025
6930cb2
handle the action to close the status item #2197
stuzart Jun 3, 2025
88f5e87
a dedicated fair data station helper #2197
stuzart Jun 3, 2025
9f7b16a
updated tests #2197
stuzart Jun 4, 2025
7a0c9d5
better fair data station helper test coverage #2197
stuzart Jun 4, 2025
19a837f
rationalised the javascript for the ajax status polling #2197
stuzart Jun 4, 2025
572c761
FairDataStationUpload doesn't have to have a project, but the investi…
stuzart Jun 6, 2025
7c43a35
validate according to purpose #2197
stuzart Jun 6, 2025
30a027e
clarify validation error message #2197
stuzart Jun 6, 2025
fbb86b4
rubocop #2197
stuzart Jun 6, 2025
6190a36
use a join in the queries to filter by task status #2197
stuzart Jun 9, 2025
0c5b841
use kebab-case without underscores for data attributes #2197
stuzart Jun 9, 2025
990b190
minor tweaks after checking over PR #2197
stuzart Jun 11, 2025
05dcd27
add some indexes on the foreign keys #2197
stuzart Jun 11, 2025
90f014c
fix schema following hasty commit #2197
stuzart Jun 11, 2025
8cda595
merge later migration into original create table #2197
stuzart Jun 11, 2025
2ce2db6
show the statuses in reverse order of id, most recent at the top #2212
stuzart Jun 11, 2025
8755560
added bootstrap 4.0 margin and padding definitions and applied to com…
stuzart Jun 12, 2025
3557150
minor nitpic about job_status being unnecessarily passed to partial #…
stuzart Jun 13, 2025
04aceed
more robust error checking and reporting when submitting the fds file…
stuzart Jun 23, 2025
e869523
mismatching_external_id is boolean and also some code reformatting #2197
stuzart Jun 23, 2025
ccfec42
improve error message #2197
stuzart Jun 23, 2025
c5e6352
detect and record when no sample type is found during import #2197
stuzart Jun 23, 2025
bb8f1a6
test for the update job, for if the sample type has become hidden sin…
stuzart Jun 23, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions app/assets/javascripts/application.js
Original file line number Diff line number Diff line change
Expand Up @@ -84,3 +84,4 @@
//= require plotly-2.27.0.min
//= require extended_metadata_type
//= require institution-ror-typeahead
//= require fair_data_station
59 changes: 59 additions & 0 deletions app/assets/javascripts/fair_data_station.js.erb
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
<% environment.context_class.instance_eval { include Seek::Util.routes } %>

const FairDataStation = {
registerStatusCloseButtons: function(ancestor) {
$j(ancestor + ' .close-status-button').on('click', function() {
const btn = $j(this);
btn.attr('disabled', true);
const uploadId = btn.data('upload-id');
const projectId = btn.data('project-id');
const investigationId = btn.data('investigation-id');
const purpose = btn.data('purpose');
let url = '';
if (purpose == 'import') {
url = '<%= hide_fair_data_station_import_status_project_path('~project_id~') %>';
url = url.replace('~project_id~', projectId);
}
else {
url = '<%= hide_fair_data_station_update_status_investigation_path('~investigation_id~') %>';
url = url.replace('~investigation_id~', investigationId);
}

$j.ajax(url, {
data: { 'upload_id': uploadId },
method: 'POST',
success: function (response) {
$j(btn).parents('div.fair-data-station-status').fadeOut();
},
error: function(response, textStatus, errorThrown) {
alert('There was an error when trying to close: ' + errorThrown);
btn.attr('disabled', false);
}
}
);
});
},
reenableNewUpdateButtons: function() {
$j('input.disabled[type=submit]').removeAttr('data-tooltip');
$j('input.disabled[type=submit]').popover('destroy');
$j('input.disabled[type=submit]').removeAttr('onclick');
$j('input.disabled[type=submit]').removeClass('disabled');
$j('input.disabled#datastation_data').removeAttr('data-tooltip');
$j('input.disabled#datastation_data').popover('destroy');
$j('input.disabled#datastation_data').removeAttr('onclick');
$j('input.disabled#datastation_data').removeClass('disabled');
},
updateStatusPolling: function(endpointUrl, elementId, jobStatus) {
elementSelector = '#'+elementId;
setTimeout(function () {
$j.ajax(endpointUrl, {
data: {'previous_status': jobStatus},
success: function (html) {
$j(elementSelector).replaceWith(html);
FairDataStation.registerStatusCloseButtons(elementSelector);
}
}
);
}, 5000);
}
};
108 changes: 107 additions & 1 deletion app/assets/stylesheets/bootstrap_include.scss
Original file line number Diff line number Diff line change
Expand Up @@ -128,4 +128,110 @@ textarea.form-control {
/* is missing from the documented options - https://getbootstrap.com/docs/4.0/utilities/colors/ - so added to match */
.text-secondary {
color: $text-secondary !important
}
}

/* Bootstrap 4 margins and padding - https://getbootstrap.com/docs/4.1/utilities/spacing/ */
.m-0 { margin:0!important; }
.m-1 { margin:.25rem!important; }
.m-2 { margin:.5rem!important; }
.m-3 { margin:1rem!important; }
.m-4 { margin:1.5rem!important; }
.m-5 { margin:3rem!important; }

.mt-0 { margin-top:0!important; }
.mr-0 { margin-right:0!important; }
.mb-0 { margin-bottom:0!important; }
.ml-0 { margin-left:0!important; }
.mx-0 { margin-left:0 !important;margin-right:0 !important; }
.my-0 { margin-top:0!important;margin-bottom:0!important; }

.mt-1 { margin-top:.25rem!important; }
.mr-1 { margin-right:.25rem!important; }
.mb-1 { margin-bottom:.25rem!important; }
.ml-1 { margin-left:.25rem!important; }
.mx-1 { margin-left:.25rem!important;margin-right:.25rem!important; }
.my-1 { margin-top:.25rem!important;margin-bottom:.25rem!important; }

.mt-2 { margin-top:.5rem!important; }
.mr-2 { margin-right:.5rem!important; }
.mb-2 { margin-bottom:.5rem!important; }
.ml-2 { margin-left:.5rem!important; }
.mx-2 { margin-right:.5rem!important;margin-left:.5rem!important; }
.my-2 { margin-top:.5rem!important;margin-bottom:.5rem!important; }

.mt-3 { margin-top:1rem!important; }
.mr-3 { margin-right:1rem!important; }
.mb-3 { margin-bottom:1rem!important; }
.ml-3 { margin-left:1rem!important; }
.mx-3 { margin-right:1rem!important;margin-left:1rem!important; }
.my-3 { margin-bottom:1rem!important;margin-top:1rem!important; }

.mt-4 { margin-top:1.5rem!important; }
.mr-4 { margin-right:1.5rem!important; }
.mb-4 { margin-bottom:1.5rem!important; }
.ml-4 { margin-left:1.5rem!important; }
.mx-4 { margin-right:1.5rem!important;margin-left:1.5rem!important; }
.my-4 { margin-top:1.5rem!important;margin-bottom:1.5rem!important; }

.mt-5 { margin-top:3rem!important; }
.mr-5 { margin-right:3rem!important; }
.mb-5 { margin-bottom:3rem!important; }
.ml-5 { margin-left:3rem!important; }
.mx-5 { margin-right:3rem!important;margin-left:3rem!important; }
.my-5 { margin-top:3rem!important;margin-bottom:3rem!important; }

.mt-auto { margin-top:auto!important; }
.mr-auto { margin-right:auto!important; }
.mb-auto { margin-bottom:auto!important; }
.ml-auto { margin-left:auto!important; }
.mx-auto { margin-right:auto!important;margin-left:auto!important; }
.my-auto { margin-bottom:auto!important;margin-top:auto!important; }

.p-0 { padding:0!important; }
.p-1 { padding:.25rem!important; }
.p-2 { padding:.5rem!important; }
.p-3 { padding:1rem!important; }
.p-4 { padding:1.5rem!important; }
.p-5 { padding:3rem!important; }

.pt-0 { padding-top:0!important; }
.pr-0 { padding-right:0!important; }
.pb-0 { padding-bottom:0!important; }
.pl-0 { padding-left:0!important; }
.px-0 { padding-left:0!important;padding-right:0!important; }
.py-0 { padding-top:0!important;padding-bottom:0!important; }

.pt-1 { padding-top:.25rem!important; }
.pr-1 { padding-right:.25rem!important; }
.pb-1 { padding-bottom:.25rem!important; }
.pl-1 { padding-left:.25rem!important; }
.px-1 { padding-left:.25rem!important;padding-right:.25rem!important; }
.py-1 { padding-top:.25rem!important;padding-bottom:.25rem!important; }

.pt-2 { padding-top:.5rem!important; }
.pr-2 { padding-right:.5rem!important; }
.pb-2 { padding-bottom:.5rem!important; }
.pl-2 { padding-left:.5rem!important; }
.px-2 { padding-right:.5rem!important;padding-left:.5rem!important; }
.py-2 { padding-top:.5rem!important;padding-bottom:.5rem!important; }

.pt-3 { padding-top:1rem!important; }
.pr-3 { padding-right:1rem!important; }
.pb-3 { padding-bottom:1rem!important; }
.pl-3 { padding-left:1rem!important; }
.py-3 { padding-bottom:1rem!important;padding-top:1rem!important; }
.px-3 { padding-right:1rem!important;padding-left:1rem!important; }

.pt-4 { padding-top:1.5rem!important; }
.pr-4 { padding-right:1.5rem!important; }
.pb-4 { padding-bottom:1.5rem!important; }
.pl-4 { padding-left:1.5rem!important; }
.px-4 { padding-right:1.5rem!important;padding-left:1.5rem!important; }
.py-4 { padding-top:1.5rem!important;padding-bottom:1.5rem!important; }

.pt-5 { padding-top:3rem!important; }
.pr-5 { padding-right:3rem!important; }
.pb-5 { padding-bottom:3rem!important; }
.pl-5 { padding-left:3rem!important; }
.px-5 { padding-right:3rem!important;padding-left:3rem!important; }
.py-5 { padding-top:3rem!important;padding-bottom:3rem!important; }
76 changes: 65 additions & 11 deletions app/controllers/investigations_controller.rb
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,9 @@ class InvestigationsController < ApplicationController
before_action :investigations_enabled?
before_action :fair_data_station_enabled?, only: %i[update_from_fairdata_station submit_fairdata_station]
before_action :find_assets, only: [:index]
before_action :find_and_authorize_requested_item, only: [:edit, :manage, :update, :manage_update, :destroy, :show, :update_from_fairdata_station, :submit_fairdata_station, :new_object_based_on_existing_one]
before_action :find_and_authorize_requested_item, only: [:edit, :manage, :update, :manage_update, :destroy, :show,
:update_from_fairdata_station, :submit_fairdata_station, :fair_data_station_update_status,
:hide_fair_data_station_update_status, :new_object_based_on_existing_one]

#project_membership_required_appended is an alias to project_membership_required, but is necesary to include the actions
#defined in the application controller
Expand Down Expand Up @@ -36,25 +38,77 @@ def new_object_based_on_existing_one
end

def submit_fairdata_station
path = params[:datastation_data].path
data_station_inv = Seek::FairDataStation::Reader.new.parse_graph(path).first
error = nil
in_progress = []
mismatching_external_id = false
if params[:datastation_data].present?
path = params[:datastation_data].path
fair_data_station_inv = Seek::FairDataStation::Reader.new.parse_graph(path).first

if fair_data_station_inv.present?
in_progress = FairDataStationUpload.matching_updates_in_progress(@investigation, fair_data_station_inv.external_id)
mismatching_external_id = fair_data_station_inv.external_id != @investigation.external_identifier
else
error = "Unable to find an #{t('investigation')} within the file"
end
else
error = 'No file was submitted'
end

if mismatching_external_id
error = "#{t('investigation')} external identifiers do not match"
elsif in_progress.any?
error = "An existing update of this #{t('investigation')} is currently already in progress."
end

begin
Investigation.transaction do
@investigation = Seek::FairDataStation::Writer.new.update_isa(@investigation, data_station_inv, current_person, @investigation.projects, @investigation.policy)
@investigation.save!
if error.nil?
content_blob = ContentBlob.new(tmp_io_object: params[:datastation_data],
original_filename: params[:datastation_data].original_filename)
fair_data_station_upload = FairDataStationUpload.new(contributor: current_person,
investigation: @investigation,
investigation_external_identifier: fair_data_station_inv.external_id,
purpose: :update, content_blob: content_blob
)
if fair_data_station_upload.save
FairDataStationUpdateJob.new(fair_data_station_upload).queue_job
redirect_to update_from_fairdata_station_investigation_path(@investigation)
else
error = 'Unable to save the record'
end
rescue ActiveRecord::RecordInvalid, Seek::FairDataStation::ExternalIdMismatchException => e
flash.now[:error] = e.message
end

if flash[:error].present?
if error.present?
flash[:error] = error
respond_to do |format|
format.html { render action: :update_from_fairdata_station, status: :unprocessable_entity }
end
end

end

def fair_data_station_update_status
upload = FairDataStationUpload.for_investigation_and_contributor(@investigation, current_person).update_purpose.where(id: params[:upload_id]).first
if upload
respond_to do |format|
format.html { render partial: 'fair_data_station_update_status', locals: { upload: upload } }
end
else
respond_to do |format|
format.html { redirect_to(@investigation) }
format.html { render plain:'', status: :forbidden }
end
end
end

def hide_fair_data_station_update_status
upload = FairDataStationUpload.for_investigation_and_contributor(@investigation, current_person).update_purpose.where(id: params[:upload_id]).first
if upload && (upload.update_task.completed? || upload.update_task.cancelled?)
upload.update_attribute(:show_status, false)
respond_to do |format|
format.html { render plain:'' }
end
else
respond_to do |format|
format.html { render plain:'', status: :forbidden }
end
end
end
Expand Down
Loading