URL imports

[jpolitz]

URL Imports

It's useful to have imports that come from raw HTTPS urls for a number of reasons.

One particular use case that's come up recently is hosting files on Github: https://raw.githubusercontent.com/ urls are a useful place to host and fetch static code from a repository, which is already a good place to store code. This can be used to cut out publishing steps of going through Google Drive.

This proposal has a few steps aimed at different use cases and ergonomics

Raw url imports

Add new syntax:

import url(https-url) as I

This expects:

• The URL is publicly available
• The URL replies to cross-origin CORS requests (githubusercontent satisfies this)

This will be implemented with a basic fetch call from the browser/from the command line that does a simple HTTP CORS GET request to the URL with no special headers or preprocessing. The response is expected to by Pyret code (no importing raw JS via this mechanism). It is compiled and cached in memory (for CPO environments) and to disk in compiled/ with a name based on the hash of its URL (for CLI environments). The identity of the module is the URL: if you import two modules with the same text with non-equal URLs, they are both instantiated separately. This does require a little care in not accidentally grabbing extra ? or # fragments of URLs when copy/pasting, which could have spurious inequalities.

url-file imports

One possible (and I hope to become common) workflow will be to author starter code and student libraries in VScode or in a Github Codespace, which has nice file management and version control built in along with the full-featured editor for reactors, etc.

One issue with authoring student-facing code meant to be used on code.pyret.org is that when publishing, it's useful to have a stable link in the import line. But while authoring, it's often useful to be able to make and test quick edits. my-gdrive and shared-gdrive are clumsy for this because they require edits and refreshes. The analog for development with url(...) in a codespace or vscode would be either:

• Frequently commit and push a library, wait for the push before re-running the student starter code that imports it with url. This requires pushing lots of little edits that are likely to have errors, and is cumbersome. Further, pushing to the canonical place of import could make users see these bugs.
• Switching to import file(...) while developing, and then switching back to import url("https://raw.githubuerscontent...") for a push that publishes. If multiple files are involved and all of them need to be changed, it can be error-prone to push it out. Also it adds friction to making and testing a small edit.

What's needed is an import form that can be stable in source, but do “the right thing” depending on its context. We propose adding

import url-file(base-url, rel-path) as I

For example:

import url-file('https://raw.githubusercontent.com/bootstrapworld/curriculum/HEAD/courses/data-science/', 'lib/data-science-2025.arr')`

The overall idea is that the URL is split into a base URL and a relative path fragment. The concatenation of them is used to fetch the URL when remote access is needed, while the relative path can be used in some development cases when the file is available locally on disk.

Along with this we introduce a new compiler option --url-file-mode with three options: all-local, all-remote, and local-if-present.

The semantics are:

• A url-file import always has an identity equal to the concatenation of base-url and rel-path. This is important! It would be really weird if you could have the same URL present at two different “paths” on disk because it could lead to cases where either the module graph is different on CPO than on the command line, or there are conflicting sources of truth for the content of a file.
• When resolving a url-file module:
  • In all-local mode, all url-file imports are treated as file imports of rel-path, except with module identity of the URL rather than the path on disk
  • In all-remote mode, all url-file imports are treated as url imports with the URL made from concatenating base-url and rel-path
  • In local-if-present mode, the compiler first checks if a file is present on disk at the rel-path, in which case it loads it like file import, otherwise it fetches it remotely with the constructed URL

The default is always all-remote. In CPO hosted at code.pyret.org there is no way to change the mode from all-remote. At the CLI (e.g. pyret-npm) the behavior can be controlled with --url-file-mode. In the VScode extension there will be an extension and project setting to pick the mode (with the default all-remote).

Here's what this enables:

• Rachel is a Bootstrap curriculum author. She writes complex, multi-file support code for students using a Github Codespace in her fork of a curriculum repository called rachel/curriculum. She toggles the mode to local-if-present when she does her work. The import statements all look like:

  import url-file("https://raw.githubusercontent.com/bootstrapworld/curriculum/HEAD/...", "path/to/file.arr")

  As she edits files in her Codespace and re-runs, she sees all the changes immediately for testing and debugging. When she's happy with her code, she pushes it to her fork and submits a pull request to bootstrapworld/curriculum.

  When the code merges, the import statements point to URLs hosted on Github that contain the current up-to-date copy of the code; the merge is a “deployment”.

  Rachel's students do their work from code.pyret.org in single-file Pyret projects. They start their work from links like https://code.pyret.org/editor#urlshare=https://raw.githubusercontent.com/bootstrapworld/curriculum/HEAD/, save copies of those files to their Google Drive, and always see the most up to date version of the curriculum.

• Kathi is an undergraduate instructor. All of her students work in single-file Pyret projects, and her TA team is used to making a single repository per assignment.

  Her students use a mix of code.pyret.org and VScode based on their preferences and experience. Each repository has student starter code and some support files. Her TAs also write code using url-file(...) imports.

  Some students do their work by forking the starter repository, cloning it, and opening it in VScode. If they want, the can see the library code provided in the project, but editing it doesn't change the behavior they see (because of the all-remote setting).

  Other students do their work by copy-pasting the single starter file into a new CPO window. The imports work because the refer to the Github static file URLs, and they don't need copies of all the files (just the one single-file project starter).

  Other students do their work by copy-pasting the single starter file into a new .arr file in VScode. The imports work because the refer to the Github static file URLs, and they don't need copies of all the files (just the one single-file project starter).

• Daniel is an undergraduate instructor. His students work on multi-file projects where they are expected to edit multiple files in a project directory, and may create new files. They also rely on some background code that they should not edit.

  His TAs use url-file for the provided libary code that students shouldn't be able to edit. For the initial structure of the test and code files students see, which are intended to be editable by students, they use plain file imports. If students add more student-authored files, they use file for these as well.

  Students are expected to use VScode or a Github Codespace to edit multi-file projects.

Some notes:

• VScode-using students always use the default mode; they basically never need local-if-present, that is for TAs/curriculum authors
• CPO-using students have no change to their user experience except for the initial share URL looking slightly different
• The same starter code infrastructure can be shared across VScode users, CPO users, and command-line users.
• There are many build options for how you “deploy” (e.g. we could make all the url-file base-urls be bootstrapworld.org URLS for BS curricula). The key thing is that the final deploy URL is in the source of all the relevant files, so updates are immediately seen, while there is a mechanism for developing without deploying.
• This should make it so starter code can all be maintained on Github, as it should be.
• You can “hide” library code by using url or url-file. The main goal of this is to make it un-editable; students can easily curl the contents.
• For something like hidden bug/butterfly (wheat/chaff) implementations, would need to look into more serious obfuscation. Probably here we'd precompile to JS, force students into a VScode workspace, and use js-file imports (which is orthogonal to this infrastructure).