while that pr is merged but still there is no way to uninstall swiftly itself, how can we uninstall it completely?

I think the best way right now would be the following:

$ swiftly uninstall all
$ rm -r ~/.local/share/swiftly
$ rm ~/.local/bin/swiftly

This is assuming you've installed to the default location.
You may also want to delete any swiftly-specific lines from your .profile, .bash_profile, .bash_login, or .zprofile, if there are any in any of those files.

Note, when going from 0.2.0 -> 0.3.0 one may get into trouble if one hasn't uninstalled all / removed all the symlinks

find ~/.local/bin/ -maxdepth 1 -type l 
find ~/.local/bin/ -maxdepth 1 -type l -delete