WeBWorK 2.20 Release Candidate #2721
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Use the course_admin_id instead of 'admin' when talking about the
admin course name on this page and in its help.
Mention setting $permissionLevels{login}='admin' in course.conf
in the help as a way to control access to the admin course.
Use `overflow-auto` and `max-height` in the select which admin users
to add to the new course div.
When adding a new course, and someone unchecks all admin users to be copied to the new course, ensure they stay unchecked if an error occurs and the page is reloaded.
When adding a new course, there is now an 'Add Another Instructor' button which can be used to add more than one new user to the newly created course. Each time this button is hit, the form expands allowing input for a new user. It is possible to set the permission level of each of these users and to keep the form balanced, the student ID can now also be included. This also updates assigning the initial user to sets and achievements. The intention was to assign new instructors to these, so any user (either copied from the admin course or added initially) whose permission is less than admin will be assigned to the sets and achievements.
When copying over old sets and achievements only assign the initial users to sets or achievements once. This makes it so any initial user is always assigned to everything, and will ignore (skip over) whatever they were assigned to in the old course. Also assign all initial users (including admins) to sets/achievements.
Add an "Enter information for additional instructor number X." statement for each additional instructor to clarify what inputs go with what instructor, and to break the inputs apart a little better.
* Remove statements about Course ID and User ID restrictions and turn them into tooltips. * Limit Course ID to 40 characters. * Change language from "instructor" to "user" with other language tweaks. * Make "Add Additional User" button secondary. * Update add course help.
Make password fields for new users text, so they will be saved after hitting add another user or errors appear. Since they are passwords for other users no need to hide them. Remove confirm password and student ID input to make each new user take up a little less space. Also make it so new student users are current (enrolled) in the course, while all other users are observers.
Adds a page in the admin course that can be used to copy OTP secrets from one user to another or to reset OTP secrets. This allows copying an OTP secret from a single user in one course to a different user in the same or different course. Copying multiple secrets from users in a single course to one or more other courses, provided the user names are the same. Or to reset one or more OTP secrets for users in a single course.
Large courses can cause a lot of CPU load when the javascript generates the menus to select users of the course. Also large courses will be hard to navigate with using the menus. By default courses with 200 or more users will not be listed in the drop down menus. If any course is not included, a warning will state which courses have been ignored and a button to show all courses will appear. The ignored courses are still listed in the list of destination courses when copying multiple secrets.
Use compute_unreduced_grade to determine if a problem is correct or not on the student's grade table to highlight correct problems.
Use reduced credit when checking if problem is correct in Grades table.
This has two components, a regular Webwork Authen module and a Mojolicious plugin for SAML2. The plugin implements a SAML2 SP using the Net::SAML2 library. Net::SAML2 claims to be compatible with many SAML2 implementations, including Shibboleth. The intent for this plugin is to replace the old Shibboleth auth that depended on the shibd service which requires Apache's mod_shib. The WeBWorK::Authen::Saml2 authen module's main purpose is to call the Saml2 plugin helper sendLoginRequest() that properly sends the user to the IdP to authenticate. There is a bypass option that allows skipping the Saml2 authen in favour of the authen module after it. So you can, for example, put Basic_TheLastOption after it allow access to the internal username/password auth. It seems to be standard for Mojolicous plugins to be located in lib/Mojolicious/Plugin, so I've put the Saml2 plugin there. Additional detail and configuration instructions can be found in lib/Mojolicious/Plugin/Saml2/README.md The Saml2 plugin will only be loaded if the corresponding conf file exists. Copy conf/authen_saml2.dist.yml to conf/authen_saml2.yml to enable it. 3 settings in the conf are crucial, the idp.metadata_url must be set to the IdP's xml metadata endpoint and a unique sp.cert & sp.signing_key pair should be generated. The example cert and signing_key must not be used for prod under any circumstances. I initially put the config in conf/mojolicious.webwork.yml under a saml2 section but seeing as how other authen modules have their own conf files, I figure it was better to follow that convention. And as there seems to be more work around the yaml config, I've made the saml2 authen conf also a yaml file. The NotYAMLConfig plugin for reading yaml conf files gave me some trouble though, as I didn't want to step on the main conf and had to read the code to figure out how to use just the load conf file functionality. The Saml2 plugin will generate its own xml metadata that can be used by the IdP for configuration. This is available at the /saml2/metadata URL under the default config. Note that endpoints are configurable as I wanted to be able to change them to match shibd endpoints. The Saml2 plugin has its own set of controllers and a router for them. The AcsPostController is the most important one as it handles the SAMLResponse that sends the user back to Webwork from the IdP. The errors aren't the most friendly unfortunately, should probably add a proper 401 error template so users don't see the more scary stacktrace one. Note that unlike shibd, attribute maps are not supported. So you probably have to replace user friendly attribute names like "studentNumber" with URN style names like "urn:mace:dir:attribute-def:studentNumber". You can check your IdP's attribute map xml for the official URN names. Some discussion about alternatives that I tried before settling on the Mojolicious plugin approach: The Saml2 as a plugin idea started after trying out Mojolicious::Plugin::SAML. Mojolicious::Plugin::SAML didn't work out in the end due to two downsides. The major one being a lack of RelayState support. RelayState is the defacto standard way for SPs to store where the user wants to go after returning from a successful auth from the IdP. This is a necessary feature for Webwork as auth to each course is handled separately and we need to know exactly what course the user wants to get into. The minor issue is that it doesn't parse the SAMLResponse attributes for you and I really didn't want to have to muck with xml parsing. Apache mod_proxy using mod_shib and shibd was another possibility. This requires passing shib params through HTTP headers via the use of the ShibUseHeaders setting. Shibboleth documentation says the ShibUseHeaders option should be avoided as they're not confident in the protections against spoofed HTTP header attacks. Running Webwork under mod_perl again so we don't need to use mod_proxy. This resulted in hard to debug issues with a few sections of async code. While I was able to 'unasync' those sections to get the thing running, it feels unmaintainable in the long run, especially if Webwork increases usage of async features.
WEBWORK_ROOT_URL definition in Dockerfiles used two colons before the port when it's supposed to be only 1. Also deleted version line from docker-compose.dist.yml, as I get a warning message that `version` is obsolete.
Mojo::Base's -strict option removed where -signature is present as it makes strict redundant. Use Mojo::JSON's functions instead of JSON. When using WeBWorK::Debug, specify import of debug() function. Remove import of WeBWorK::Debug and Data::Dumper where they're not actually being used. Fix README.md to pass markdownlint inspection.
Reformatted to follow .editorconfig rules.
This fixes some issues with #2511. This pull request is built on that one. That pull request is a nice start, but has some issues. Thanks to @ionparticle for starting this work. One of the most important things introduced in that pull request is the development identity provider via SimpleSAMLphp. That gave me a way to work with this. Although I liked the idea of using a Mojolicious plugin, that approach does not fit into the webwork2 course environment system. So, unfortunately, that had to be changed. So this rewrites the SAML2 authentication module to use the usual approach that all of the other authentication modules use. There is a `authen_saml2.conf.dist` file that should be copied to `authen_saml2.conf` to use SAML2 authentication. All settings for the authentication module are in that file. The primary limitation of the plugin approach was that it is not possible for different courses to use different identity providers. Or for one course to use SAML2 authentication and another only use the basic password authentication (without the need to add a URL parameter). This is something that is expected of the authentication modules, and is desirable for multi-institutional servers. Another problem with the implementation is that is does not work with `$session_management_via = "key"` set. The reason that doesn't work is because the implementation broke the rule of creating and accessing a session before authentication is completed. So this reimplementation uses the database via the "nonce" key hack much like LTI 1.1 authentication does. The previous implentation also used the `WEBWORK_ROOT_URL` environment variable in the code. That is not an environment variable that is defined for webwork2. It is only defined for the docker build, and can not be used in the code. The previous implementation also does not work with two factor authentication. If an identity provider does not provide two factor authentication, then webwork2's two factor authentication should be used. Of course, if the identity provider does provide two factor authentication, then the two factor authentication can be disabled in `localOverrides.conf`. An option to enable service provider initiated logout has also been added. It is off by default, but if enabled, when the user clicks the "Log Out" button, a request is sent to the identity provider to also end its session. The `README.md` file that was with the plugin code has been moved to the `docker-config/idp` directory, and is now purely for instructions on how to use the simpleSAMLphp development instance. The readme also includes instructions on how to set up a simpleSAMLphp development instance without docker. The documentation on how to configure and use the SAML2 authentication module is now all in the `authen_saml2.conf.dist` file. Note that the webwork2 service provider certificate files are no longer directly in the configuration file. Instead file locations are to be provided. It didn't really make sense to have the actual certificates in the configuration file, then write them to a file, which the Net::SAML2 module would read. Furthermore, it would be very easy to add the certificates incorrectly to the configuration file. With the YAML file approach if indentation were not correct, then the configuration file would fail to load.
Any route can now specify the methods that are allowed by adding a `methods` key to the route parameters. The value of the key should be a reference to an array containing the allowed methods. The ACS route is the only route that uses this at this point to restrict to the POST method only.
This option is for the case that the identity provider offers multi
factor authentication, and yet the $saml2{bypass_query} is also
allowed. In this case you would not want webwork2's two factor
authentication to be used when signing in via the identity provider.
However, two factor authentication should be used if the bypass query is
used. Setting $saml2{twoFAOnlyWithBypass} to 1 makes it so that
webwork2's two factor authentication is skipped for users signing in via
the identity provider, but still required for users signing in with a
username/password. If this is set to 0, then webwork2's two factor
authentication will always be required.
Really, the fix has been published in an update pg-codemirror-editor package (actually the bug was in the codemirror-lang-pg package). This just updates the version in the package.json file.
Add course admin tool to manage OTP secrets.
Updates to adding users to newly created courses.
Fix PGML Emphasis in the PG CodeMirror editor.
Hide the "Start New Test" button when acting as another user unless the user has permissions to do so. Fix some logic where the warning about creating new test version wouldn't show with the record_answers_when_acting_as_student permission, and instead the version would be created without warning. Translate the error messages that are created for an invalidSet. Remove the test for invalidProblem, which isn't used in tests. Update error messages to state 'test' instead of 'set'.
When acting as another user, disable the "Start New Test" button instead of hiding it if user doesn't have permission to start a test as a different user. Include a tooltip that states why the button is disabled.
There is no need for the license to be in all files. That creates an unnecessary maintenance hassle with keeping the copyright years up to date, and it is sufficient to have the license (with the copyright years) in the `LICENSE` file alone. Furthermore, this is done quite inconsistently. Sometimes files are added without the license, and potentially worse is a file being added with the license but not in the exact format needed for the `bin/dev_scripts/update-copyright` script to detect and properly update the copyright years. The `bin/dev_scripts/update-copyright` script is thus no longer needed and has been removed. Also, the fallback copyright years for the footer now is just "unknown" instead of being the correct copyright years. If the version and copyright years are not correctly loaded in the course environment from the `VERSION` file then that is what you will see. That should never happen. So from now on the only files that need to be updated are the `LICENSE`, `README.md`, and `VERSION` files. In addition the `docker-config/docker-compose.dist.yml` and `DockerfileStage2` files need to have `forWW...` tag name updated to the current version (for the two stage build).
Change the version to 2.20 and remove the license from files.
changes to email template for feedback
…`Readme.md` file. This was missed in #2722 and was also missed for the 2.19 release obviously. So by deleting it from the file it won't be missed again!
This takes advantage of things available with Mojolicious.
This does not use `Data::UUID` anymore. A comment stated that it was
overkill. It is and it was never a good idea to use that for this
purpose. This just uses the `Mojo::Asset::Memory::to_file` method which
converts to a `Mojo::Asset::File` object and saves the file to a
temporary file using `Mojo::File::tempfile` (which uses `File::Temp`
under the hood). That is guaranteed to give a safe filename that does
not conflict with any existing files. Setting the environment variable
`MOJO_TMPDIR` locally to be the passed in directory (which is
`$ce->{webworkDirs}{uploadCache}`) ensures that the file is saved in the
usual webwork2 upload location (although there is no need for that, and
we could drop that configuration variable and just use the system
temporary directory for this).
The info file that is written is now UTF-8 encoded and decoded. This
fixes issue #2690.
The `dir` option which was previously required and the only "option" is
now just a required argument. It doesn't make sense to use an "option"
hash things that are required. Particularly if there is only one option
and it is required.
Other than what is mentioned above the module behaves much the same.
Remove the current version in "Information for Downloading" from the `Readme.md` file.
Rewrite the WeBWorK::Upload module.
If you have payed attention (or tested #2709 and #2710), then you know that this takes a long time if there are a large number of users. Way to long to be at all reasonable. This uses techniques like those used for assigning multiple sets to users, i.e., the `WeBWorK::DB::Schema::NewSQL::Std::insert_records` and `WeBWorK::DB::Schema::NewSQL::Std::update_records` methods, to speed up assignment of achievements to users. This brings the assignment time for 5000 users down from more than 8 minutes, to less than 30 seconds.
The main thing that this does is implement the `WeBWorK::DB::Ex` exceptions as the comments in `lib/WeBWorK/DB.pm` (that have been there since 2008) say they were intended to be used. For `WeBWorK::DB::Ex::DependencyNotFound` and `WeBWorK::DB::Ex::RecordNotFound` the error message for the exception is the same as before. So for those the only difference is that they are now Exception::Class objects (which contain more information than just the message). For `WeBWorK::DB::Ex::RecordExists` and `WeBWorK::DB::Ex::TableMissing` the error message for the exception is the error message that DBI gives. The previous messages for `WeBWorK::DB::Ex::RecordExists` like "addUserSet: user set exists (perhaps you meant to use putUserSet?)" no longer replace the DBI exception messages. So as the comments from 2008 stated don't check for the string "user set exists" in the exception message anymore. Instead check if `WeBWorK::DB::Ex::RecordExists->caught` is true. In addition the code for auto creation of password records in `getPasswords` and the auto creation of permission records in `getPermissionLevels` has been deleted simply because that code doesn't even work. It hasn't worked for a long time if it ever did. In both methods the `gets` method is called to retrieve the requested records. The `gets` method does not even include non-existent records in the list it returns. So the auto creation loops were only looping over records that were guaranteed to exist. So that code was purely an inneficient loop that did nothing. Next, the `addMultipleUserSets` and `addUserMultipleProblems` methods of `lib/WeBWorK/DB.pm` were removed. Instead what is needed from those methods is used directly in the `lib/WeBWorK/Utils/Instructor.pm` module where those methods were called. Those methods added a high level of inneficiency to the process of assigning sets and problems to multiple users. For example the methods in `lib/WeBWorK/Utils/Instructor.pm` check that the sets don't already exist and if so it skips them. Then the `addMultipleUserSets` method again checks if the sets exist. Note that the previous `lib/WeBWorK/DB.pm` methods did check for the existence of the users, and this no longer does. That check slows things down considerably, and in all cases those users come from the database to begin with. So it is not really a necessary check. This rewrite speeds up the assignment of multiple sets and problems considerably. Finally, the methods in `lib/WeBWorK/Utils/Instructor.pm` no longer return unused lists of failures. In all of the uses of those methods in the webwork code, there was only one place where those return values were ever used, and even in that case it was not in an essential way. Furthermore, that one usage was in `lib/WeBWorK/SetActions.pm` (for the WebworkWebservice) which is not actually used by webwork2 anyway (or probably even by anyone). Instead of catching and rethrowing exceptions, the exceptions are just let through. The exceptions that were ignored before (but put into a return value that was later ignored) are still ignored. The POD in this file is cleaned up as well.
The database layout is now defined in the Perl `WeBWorK::DB::Layout`
package. An instance of the `WeBWorK::DB` package is no longer
instantiated by passing the layout. Instead it is passed a course
environment object. An instance of the `WeBWorK::DB::Layout` package is
constructed from that.
In addition, the `WeBWorK::DB::Driver`, `WeBWorK::DB::Driver::Null`, and
`WeBWorK::DB::Driver::SQL.pm` modules have been deleted. Instead there
is the `WeBWorK::DB::Database` module that does everything those modules
used to do. It is now the only "driver" for all tables. Although, that
never should have been called a driver. The driver is still optional
and is `DBD::MariaDB` or `DBD::mysql`. That is what is usually referred
to as the driver, not the DBI connection handle which is what the
`WeBWorK::DB::Database` module deals with, and what the previous
"driver" modules did as well. Furthermore, only one instance of this
module is constructed when a `WeBWorK::DB` object is constructed, and
that is passed to each table. That is essentially the same as before in
terms of DBI connection handles except that before there were multiple
`WeBWorK::DB::Driver::SQL` instances that all shared the same DBI
conneciton handle via the `connect_cached` call.
So the layout is simplified considerably. The "driver", "source",
"engine", and "character_set" are no longer options for a table, and
there are no "%sqlParams" passed to each table. The only thing that is
variable in the layout is the "$courseName" used for the tableOverride.
The "source", "engine", and "character_set" are parameters of the
`WeBWorK::DB::Database` object and retrieved from there as needed.
Since there is no choice of database layout there is no point in having
that be part of the authentication module selection in the configuration
files. So the `$authen{user_module}` in the configuration files should
be either a string naming a single authentication module or an array of
strings. For example,
```perl
$authen{user_module} = [
'WeBWorK::Authen::LTIAdvantage',
'WeBWorK::Authen::LTIAdvanced',
'WeBWorK::Authen::Basic_TheLastOption'
];
```
The old format of `{ '*' => 'WeBWorK::Authen::LTIAdvantage' }` (where
`*` meant use this authentication for all database layouts) is still
allowed for now, but eventually support for that will be dropped.
Several related unused files were deleted. Those are
* `bin/check_database_charsets.pl`
(This should have never been in the repository.)
* `bin/wwdb`
(This has been broken for a long time.)
* `lib/WeBWorK/DB/Driver/Null.pm`
(This was mentioned above, but I don't think this ever was used or
even made sense to be used.)
* `lib/WeBWorK/Utils/CourseManagement/sql_moodle.pm`
(This is not used, I doubt this has worked for a long time, and is no
longer supported in any case.)
* `lib/WeBWorK/Utils/CourseManagement/sql_single.pm`
(Also not used and hasn't been working for a while now.)
* `lib/WeBWorK/Utils/DBImportExport.pm`
(This hasn't worked for a while now and code that called this in
`lib/WeBWorK/DB.pm` was removed.)
* `lib/WeBWorK/DB/Schema/NewSQL/VersionedMerge.pm`
(This is unused. At some point the `lib/WeBWorK/DB/Schema/NewSQL/Merge.pm`
module was updated to handle its functionality directly.)
A non beta version 0.0.1 of the `@openwebwork/pg-codemirror-editor` has now been published. So make that the version for the 2.20 release. If there are any other changes the version can still be bumped if needed, but I doubt that there will be at this point. The `@openwebwork/codemirror-lang-pg` package that the `@openwebwork/pg-codemirror-editor` depends on has a non beta version published (version 0.0.1). The `codemirror-lang-perl` and `codemirror-lang-mt` packages are now also published in non beta status. Those are under my name still at this point since they are not specific to the openwebwork project.
Convert the conf/database.conf.dist file to a Perl module.
Database code clean up.
…r-release Finalize the PG CodeMirror Editor for the 2.20 release.
The `MOJO_TMPDIR` attempt was not working. The problem is that the `Mojo::Upload` object is already created before the `WeBWorK::Upload::store` method is called, and so the temporary file location is already determined and the temporrary file saved there. So when the upload is later retrieved it is not where it is expected to be. This does not happen with rather small files because a Mojo::Upload::Memory object is used instead of a Mojo::Upload::File object, and in that case the file is not saved to a temporary file to begin with.
Also fix extraction of zip or tar files that have files inside a directory, but not the directory itself.
… Secrets" page. This page loads quickly regardless of if these courses are filtered out. Furthermore the menus are generally responsive even when these large courses are included. The only case where something is a bit slow is in the case that on the "Copy Single Secret" tab the selected "Source Course ID" is one of these large courses. In that case if you click on the "Source User ID" it takes some time for the dropdown menu to appear and even then it has to be an astronomically large course for that to be slow (more than 20,000 users). However, even if the large courses are included as long as a large course is not selected the menus are quick. It is interesting that the multiple select elements on the "Copy Multiple Secrets" tab are still fast even with astronomically large courses. It is just the single select elements on the "Copy Single Secret" tab that experience a noticeable slowdown. In any case, the point is that there is no reason to filter these courses. Doing so just adds unnecessary steps for the user.
…r own OTP secret.
Fix file uploads.
…eed-tweak Speed up assignments of achievements.
First get all of the necessary achievement data from the database, then process it and print it to the file. Comparing scoring all course achievements for all users in a course with 5000 users shows a considerable speed improvement. It takes more than three minutes for this with the develop branch, and less than 3 seconds with this pull request. Obviously the generated scoring files are identical.
The usual thing (by now with this series of pull request) is done. That is removing database access from loops and reducing to single queries as much as possible. In addition instead of using the Mojolicious tag helpers to render the "earned" checkbox and "counter" text field, direct html is used. I discovered that this renders much faster. With the usual 5000 user test the rendering of the template alone takes 5 seconds after changing a checkbox or counter value and saving, while with the direct html it takes something like 0.2 seconds. I believe that it has something to do with their code to set the values of the inputs that is causing the slow down. I am not sure on that, but I know that the direct html is much faster. That database changes are still the biggest part of the speed improvement here in any case. This page was rendering quite slowly even on initial load before, and that part was fast with the tag helpers. After making the database changes things sped up considerably, but then timing parts of the code when saving revealed the tag helper issue.
Speed up achievement scoring.
…ements Speed up the achievement users page.
…-restrictions Remove the filtering of large courses on the course admin "Manage OTP Secrets" page and fix security vulnerability.
Importing the `default_achievements.axp` file into a course with no achievements and 5000 users and assigning to all users took about 8 minutes with the WeBWorK-2.20 branch. With this pull request the time decreased to about 20 seconds.
…-speed-improvement Speed up the assignment of achievments when importing achievments.
Update the pg modules to include Plots modules.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It is time for this. We have talked about just using the develop branch, but I think it is nice to have a separate branch for the release candidate. It just keeps things separate in a nice way.
So as usual, re-target pull requests for this branch that you want to get into the release, and I will handle synchronizing those to develop as they are merged.