I struck a small blow for better security today.
It started last night on an IRC channel with A&D regular Susan Sons admonishing the regulars to rotate their ssh keys regularly – that is, generate and export new key pairs so that is someone cracks the crypto on one out of your sight it won’t be replayable forever.
This is one of those security tasks that doesn’t get done often enough because it’s a fiddly pain in the ass. But (I thought to myself) I have a tool that reduces the pain. Maybe I should try to eliminate it? And started hacking.
The tool was, until yesterday, named ssh-installkeys. It’s a script wrapper written in Python that uses a Python expect engine to login into remote sites and install (or remove) ssh public keys. What makes it useful is that it remembers a lot of annoying details like fixing file and directory permissions so your ssh server won’t see a potential vulnerability and complain. Also, unlike some competing tools, it only requires you to enter your password once per update.
Some time ago I taught this code to log its installations in a config file so you have a record of where you have remote-installed keys. I realized that with a little work this meant I could support a rotate option – mass-install new keys on every site you have recorded. And did that.
I’ve been meaning for some time to change the tool’s name; ssh-installkeys is too long and clumsy. So it’s now sshexport. I also updated it to know about, and generate, ed25519 keys (that being the new hotness in ssh crypto).
In order to reduce the pain, sshexport can now now store your passwords in its list of recorded sites, so you only have to enter the password the first time you install keys and all later rotations are no-hands operations. This doesn’t actually pose much additional security risk because by hypothesis anyone who can read this file has read access to your current private ssh keys already. The correct security measure is whatever you already do to protect other sensitive data in your dot directories, like GPG directories and web passwords stored by your browser. I use drive encryption.
The result is pretty good. Not perfect; the big missing feature is that it doesn’t know how to update your keys on sites like GitLab. That would take a custom method for each such site, probably implemented with curl. Perhaps in a future release.
You could do something similar with ssh-copy-id per http://www.thegeekstuff.com/2008/11/3-steps-to-perform-ssh-login-without-password-using-ssh-keygen-ssh-copy-id/
I wanted to point to http://martin.kleppmann.com/2013/05/24/improving-security-of-ssh-private-keys.html but it turned out that it’s complicated openssl-using code can now be shortened to `ssh-keygen -o`
@esr –
Don’t forget to mention this on your Patreon page.
I didn’t realize I needed this in my life, but I do. Thanks so much! This saves much pain!
@Justin Andrusk – ssh-copy-id only copies to one site (rather than maintaining a list of them), and doesn’t appear to have any provision for removing older versions of the same key. I assume that sshexport does both.
> In order to reduce the pain, sshexport can now now store your passwords in its list of recorded sites, so you only have to enter the password the first time you install keys and all later rotations are no-hands operations.
Wait a damn minute… Putting aside the security implications, you shouldn’t need passwords after the first time – it should be able to use the old private key to authenticate itself to install the new private key. What if you’ve changed the password since then, or disabled password authentication?
> I use drive encryption.
And what of people who use a passphrase?
>it should be able to use the old private key to authenticate itself to install the new private key.
That possibility occurred to me, but looking at the manual pages I couldn’t figure out how to do that. Possibly I just missed it; if so, a pointer or quote would be useful.
What might be useful is a method for having SSH warn you when a key you log in with is “too old”. Is there a way to get information on the ssh authentication method from .profile?
>What might be useful is a method for having SSH warn you when a key you log in with is “too old”. Is there a way to get information on the ssh authentication method from .profile?
Not that I know of.
@esr
> I use drive encryption.
Drive encryption is only useful to mitigate the evil maid problem. Your desktop or laptop
is much more likely to be hacked when you are at the keyboard (PEBKAC). You might be hit with a remote exploit, a trojan advertisement, trojan Javascript, a trojan email, etc.. Then your disk encryption is useless as your drive is in the unlocked state. You need to do something like gpg encrypt any file with a password an only open it as needed.
@esr
>In order to reduce the pain, sshexport can now now store your passwords in its list of recorded > sites, so you only have to enter the password the first time you install keys and all later > rotations are no-hands operations.
Why store the remote password? You already have the private key!
> This doesn’t actually pose much additional security risk because by hypothesis anyone who > can read this file has read access to your current private ssh keys already.
Uh, no. Or rather, only if you foolishly neglected to protect your private keys with a local pass phrase. The only time a private key should not have a pass phrase is on an unattended server so it can reboot to full operation.
I am a firm believer that once you have lost root you are pretty much hosed, locally. But if you have a pass phrase on your private keys then you need not have also lost all the hosts you connect to as well.
Use ssh-add once at the start of the day, then your cleartext private keys are stored in RAM until you log out. No need to reenter the passphrase when using scp, ssh, etc. Whan I am AFK, no one getting root on my host can get my private keys and escalate to hosts that accept my private key. And stealing my private key files is useless without the pass phrase.
Of course you still want the disk encryption so the evil maid can not trojan your ssh and grab your keys.
Thank you.
I never much got into the habit of using key logins with ssh because my previous job specifically outlawed it. All logins had to be interactive with a password entered, not automated.
But this is very useful.
I could never get ssh-copy-id to work well for me, so I’ll try this out.
But I already have a feature request. Given the recent SSH key leakage bug, I feel the need to rotate my ssh keys pretty soon. I use the same ed25519 and rsa keys from my desktop, laptop, and netbook. From those three to many servers. Looks like sshexport may handle pushing my public keys to the servers, but I’d also like to push the (pass phrase encrypted) private keys from my desktop to my laptop and netbook.
>I’d also like to push the (pass phrase encrypted) private keys from my desktop to my laptop and netbook.
That should be doable. If it doesn’t work for you, maybe that indicates the need for more features or a rethink.
a bug:
sshexport # make install
cp sshexport /usr/bin
gzip /usr/share/man/man1/sshexport.1.gz
Per the FHS, /usr/share is only for distro installed man pages. This should go by default into /usr/local/share/, or similar.
http://www.pathname.com/fhs/pub/fhs-2.3.html#USRSHAREARCHITECTUREINDEPENDENTDATA
“Any program or package which contains or requires data that doesn’t need to be modified should store that data in /usr/share (or /usr/local/share, if installed locally).”
Hm. Thanks for informing me, Gary, but I’m not actually sure I’m doing the wrong thing, here. I usually design my builds on the assumption that they’re going to be picked up by a distributor. Does the FHS give any guidance that I shouldn’t do this?
As I understand it, both SSH best practice and the documentation for sshexport describe a tool for copying SSH *public* keys. You generate a new private key, then everywhere you use that key, add the new public key, and remove the old one. When you’re done, you delete the old private key (which you were presumably using to authorize the key updates because you disabled password auth decades ago).
This article (and comments so far) seem to describe tools for copying SSH *private* keys, or at least the wording is ambiguous enough to raise enough doubt about what each tool does that I have to look at the code to find out. Private key copies are usually (always?) wrong. If you need private keys in multiple different places (e.g. they’re on segregated networks so you can’t use ssh-agent, or you want a backup copy in case your primary key storage fails), you create separate private keys and authorize each corresponding public key individually.
@Zygo – I don’t know what you’re seeing that implies copying the private keys – naturally when you generate a new private key you want to push copies of the public key associated with that private key.
I think the thing with the password saving is because keeping the old private key active until the copying process is done, as you described, hadn’t occurred to ESR.
@Random832: the article mentions “keys”, “key pairs”, and “private keys” when it should be discussing “public keys” and nothing else. There’s a similar problem around the ssh-copy-id page in the first comment.
Granted, it’s possible that all the confusion here is my own. I’ve had some bad experiences with users who didn’t understand the difference between id_rsa and id_rsa.pub.
> previous job specifically outlawed [key authentication]. All logins had to be interactive with a password entered, not automated.
That seems counterproductive. Arguably, one could comply with the letter of that rule by using a passphrase-encrypted private key and no SSH agent. The server would not be able to distinguish this from an automated key agent, but the server would also not be able to distinguish an interactively entered password from one supplied by a password agent (or pasting the password in via the clipboard). The server could periodically blacklist public keys to enforce a private key rotation policy too.
Most of the attack surface is similar (secrets in RAM vulnerable to trojans and whatnot), but with agent-free keyboard-interactive password auth you’re using a weak secret much more often, and that secret has to get down the input stack, in the clear, from a human to the CPU core, radiating information all the way.
Do the security people from your previous job still work there? ;)
I have this use case too; I disable password auth everywhere I can get away with it. If that can be supported I’d find this tool extremely useful, though.
This feels like a stupid question, but: If I’m reading this right it copies the keys across using ssh_session.scp, which in turn calls out to the external scp command. Shouldn’t this be as simple as supplying it with the `-i $keyfile` option?
(unrelated: at some point A&D got a preview pane for posts. For some reason it covers up the post-comment button and I have to delete the preview node to get at it. I’m not sure if that’s a problem with my browser or the page but I thought I’d mention it)
I’m confused — what’s the difficulty here? If you can use your old key then sshexport should be able to use it too.
Someone else suggested the -i option, but that has the problem that if you select a specific key you’re not guaranteed that your remote will be able to accept a key of that type. For example, if I select an ed25519 the server at ibiblio will hork. Some of what ssh is giving us is protocol discovery.
Maybe my mental model of how ssh works is wrong. How does it choose which key to try with any given remote? I thought the key file names were magic, but if it actually tries anything under .ssh that looks like a private key the you’re right, we can just leave the old one in place.
It’s kind of ugly but there’s a way. Look at the documentation for the authorized_keys file contained in the sshd(8) manpage, and note that you can prefix key lines with an ‘environment=’ option. You can use this to set an environment variable containing the key’s fingerprint or some other identifier.
Any way of automating the configuration and deployment of 2-factor authentication?
It just tries every key it knows about until the server accepts one :-(. It “knows about” a key if it’s loaded into the agent, if it has one of the magic pathnames, or if it’s provided with -i.
Two suggestions:
1) If SSH_AUTH_SOCK is set, perhaps along with sshexport option -A, llist available keys with ssh-add -L.
My keys are on an OpenPGP smartcard, so (“id_dsa.pub”, “id_rsa.pub”, “id_ed25519.pub”) isn’t going to find them.
2) Allow removal of a specific public key (if “-r”, from all hosts).
Use case: 3 sysadmins all had access to root@somewhere. Sysadmin #2 leaves. How can sysadmin #1 efficiently remove their key from N hosts without also removing sysadmin #3? This could also provide an alternative to saving passwords; instead of -d (blow away authorized_keys entirely) followed by adding the new key, first add one key then remove the old.
I did a test, and while passing -i on the command line does disable the magic names, you can pass -i multiple times.
>I did a test, and while passing -i on the command line does disable the magic names, you can pass -i multiple times.
Aha! I think that may be the thread across this abyss. That is, the way to get around the magic-filenames problem without using passwords.
Sketch of implementation when rotating keys:
1. Rename your magic filenames by suffixing with (“-old”).
2. Generate new key pairs.
3. Use multiple -i options to enable remote installation of the new public keys.
4. Use the new public keys to install a new authorized_keys.
5. Remove the old keys.
> It’s kind of ugly but there’s a way. Look at the documentation for the authorized_keys file contained in the sshd(8) manpage, and note that you can prefix key lines with an ‘environment=’ option. You can use this to set an environment variable containing the key’s fingerprint or some other identifier.
Well, that provides enough information for a warning (you could include a SSH_KEY_EXPIRY variable or some such, and compare a timestamp in your .profile), but not enough to actually prevent login if it’s too old.
Why is it that SSH public keys don’t have an embedded expiration date, anyway? PKI certificates have them.
It’s rotating existing keys, right? Presumably you can assume the remote will accept the previous key or it wouldn’t be the previous key. I’m pretty sure SSH falls back to password auth if a key for whatever reason doesn’t work, if that helps. I’m not sure about scp.
Now that I think about it, what’s the order of operations when it replaces a key? Unless you’re doing something wonky with the scp/ssh commands, they should try to automatically use id_rsa or whatever’s in .ssh/config even without -i. As long as the last per-host op is to update authorized_keys, and the last whole-procedure op is to set the new private key in place, do you even need to do anything special to make things work?
(I can think of at least a couple different ways that could go wrong so it’s entirely possible I’m talking out my ass here…)
Possible bug, by the way:
Grep doesn’t find any references in the source to -a other than the message suggesting it ought to be available.
>Grep doesn’t find any references in the source to -a other than the message suggesting it ought to be available.
Docstring bug, fixed in masters. The -r option used to be named -a.
First thought is that they were aimed at end users, and it’s hard enough to get end users to deal with password expirations…
> Why is it that SSH public keys don’t have an embedded expiration date, anyway? PKI certificates have them.
Because the very idea that keys used to authenticate a user to a server should expire is relatively new. I personally don’t think the threat model justifies it. After all, I can set the permissions on ~/.ssh/authorized_keys so that only my account and root can even read the public key. So in order to know what you’re trying to crack, you already have the keys to the kingdom anyway.
That having been said, there’s a good argument for having a site-specific key pair, but then you’d need a key management system to keep track of them all. Unless you use a deterministic method to generate the site-specific key pair from a master secret like SQRL does.
It’s been possible to set expiration dates on keypair certifications since openssh 5, and proprietary ssh.com well before that. ssh-keygen -s -V.
@The Monster – since public keys aren’t secret, I’m not sure I would trust the protocol to protect them (either from a server which I am trying to log into which does not have it installed, or from someone trying to log in as me to a server which does have it installed) without some specific documentation showing that it is not exchanged.
@Random832 According to RFC 4252 the protocol does not provide for the server to send the public key to the client, so if you have a site-specific keypair, the client can only disclose to the server the very public key that server needs to have already associated with the account in the first place.
And the whole idea of someone brute-forcing a private key based on having possession of the public key is suspect. The reason we use expiring certificates in SSL/TLS is because of the risk of the private key becoming compromised, say because an unscrupulous person with admin access exfiltrates the private key before leaving the organization.
Monster, public keys are public and SSH leaks them all over the place. Examples:
1. Supposing you have multiple keys loaded into an agent, the client will send each one of them to the server. So now you’ve leaked your site-specific public key to every other server you log into.
2. Given two RSA signatures from the same key, you can solve for the public key: http://crypto.stackexchange.com/questions/26188/rsa-public-key-recovery-from-signatures
3. Github publishes every public key you upload. https://changelog.com/github-exposes-public-ssh-keys-for-its-users/
See https://blog.filippo.io/ssh-whoami-filippo-io/ for a fun PoC that exploits the combination of #1 and #3 above.
> > Why is it that SSH public keys don’t have an embedded expiration date, anyway? PKI certificates have them.
> Because the very idea that keys used to authenticate a user to a server should expire is relatively new.
“Relatively new”? I think you’ve forgotten some history there.
21 years ago, when I was migrating from Kerberos to SSH, one of the huge selling features of ssh over ksh was that ssh keys didn’t expire. Even the possibility of not expiring was a big deal. Typical Kerberos deployments at the time had ticket lifetimes measured in hours (go ahead, try to get any work done that involves long-running distributed batch processing with a hard 12-hour ticket limit, I dare you).
I put the date (and software version in case there’s another Debian RNG bug) when a key was created in the ssh private key comment field, and every few years (or every few scary security bugs, whichever comes first) I create new ones. I also keep .ssh/authorized_keys in git, so a quick ‘git blame’ will tell me which keys are too old from the server’s point of view.
This is beautiful. So beautiful that I’ve just doubled my monthly Patreon contribution (sorry it’s still not much, but I have kids in college…).
>That seems counterproductive. Arguably, one could comply with the letter of that rule by using a passphrase-encrypted private key and no SSH agent.
Er, no that wouldn’t. ‘No ssh key logins’ was non-negotiable.
>The server would not be able to distinguish this from an automated key agent, but the server would also not be able to distinguish an interactively entered password from one supplied by a password agent (or pasting the password in via the clipboard). The server could periodically blacklist public keys to enforce a private key rotation policy too.
Let’s put it this way- there was a daily audit process that looked for ssh keys on hosts. There was a whitelist (certain functions were automated with password and passphrase-less key logins, but the exception process was arduous) but in general if ssh keys were found on a server, especially in ~/.ssh, someone was getting a talking to.
>Do the security people from your previous job still work there? ;)
You’ve not done much government-related work, I guess.
Why is it that SSH public keys don’t have an embedded expiration date, anyway? PKI certificates have them.
Because as soon as you start adding all sorts of metadata to a key, then everyone will start adding all sorts of metadata to keys, with all sorts of obscure rules about how metadata interact with the environment and various implementations whether a key works or not. And then the lawyers will show up and insist that you imbed 30 page PDFs of Word docs of someone’s T&Cs and their contracts of adhesion and their “dont hold anyone with money responsible for anything” disclaimers into metadata (you think I joke, I do not at all, this literally regularly happens with “standards based” PKI certs. And then your keys are going to be huge weirdly encoded binary blobs of shit that you don’t have good tools to manipulate. And you will need to keep special indexes of them, and “bundles” of them, in multiple conflicting filesystem paths and “key stores”.
Part of why SSH took off at all in the first is because it doesn’t have this complex garbage wankery . An SSH public key is a SINGLE LINE, of printable ASCII7. You can edit and clean up your ~/.ssh/authorized_keys file with a textmode text editor.
The lack of metadata in SSH is a feature, not a problem.
>Because as soon as you start adding all sorts of metadata to a key, then everyone will start adding all sorts of metadata to keys […] And you will need to keep special indexes of them, and “bundles” of them, in multiple conflicting filesystem paths and “key stores”.
/me bows in acknowledgement of a truly epic rant.
Why is it that SSH public keys don’t have an embedded expiration date, anyway?
And also…
You do that, and if your clock goes wrong on a machine, or someone compromises your NTP data, and suddenly your SSH infrastructure quits working, which is something you need to work to find and fix the problem.
Don’t make SSH more brittle.
> > Do the security people from your previous job still work there? ;)
> You’ve not done much government-related work, I guess.
Say no more. That sort of nonsense pops up everywhere when people are isolated from the negative outcomes of their decisions.
I just shipped sshexport 2.1.
Stored credentials are now role-oriented; that is, it tracks root@thyrsus.com separately from esr@thyrsus.com. This means you can use it to install your keys on both everyday and root accounts,
I also lengthened the operation timeout in response to a bug report by Jason Azze.
In what time I can spare from NTPsec I’m going to try to eliminate storage of passwords by doing a bit of shell game with the old and new keys. Alas, my first two attempts at this have failed; ssh’s ability to take multiple -i options should make this possible, but it’s trickier than it looks.
The other possible way to solve that is to generate the new keys under non-default (non-magic) names, leaving the old ones in place under the magic names until the process is complete. Or was that one of your failed attempts?
>The other possible way to solve that is to generate the new keys under non-default (non-magic) names, leaving the old ones in place under the magic names until the process is complete. Or was that one of your failed attempts?
It was. I’m not sure whether I tripped over an ssh limitation or just misimplemented. Either way, it is not as simple as it sounds.
> I’m going to try to eliminate storage of passwords by doing a bit of shell game with the old and new keys.
Isn’t that what ssh-agent is for?
Having multiple active keys should be trivial. At one point I routinely had dozens.
@zygo
> Isn’t that what ssh-agent is for?
As zygo implied, this line will load up all the keys with magic names:
ssh-add
Then this will load up any other keys you also want available:
ssh-add [key-file] [key-file] …
ssh-add then passes all keys on to ssh-agent, ssh-agent will pass all the keys to ssh for ssh to try on each host. Then no need to mess with ‘ssh -i’ or need to guess which keys may work on any given remote host.
Why in the hell would I do that? My agent (PuTTY) knows what key it is to use for each server it connects to. It only offers that one key to that server, because it’s the only key that I ever want it to use there.
ssh_config can match identities to hosts (set IdentityFile on each Host section) and there’s an IdentitiesOnly option to prevent use of any keys not explicitly listed. You don’t get the agent-enabled password-less login, but in some cases that’s a feature.
This feature does have its uses, e.g. if you have an egregiously high latency connection, it can take a long time to go through many round-trips trying one identity after another with ssh-agent. It’s also handy when you have multiple keys that are locked to different commands (using the command=”” option in authorized_keys) at the server end.
> First thought is that they were aimed at end users, and it’s hard enough to get end
> users to deal with password expirations…
My first thought was the server way back there in the corner of the machine room that hasn’t been logged into in 7 years but oddly runs some critical infrastructure.
SSH is one of those things that has to work when nothing else does.
Which is also why root’s default shell should be something like /sbin/sh and should, like everything else in that directory, be *statically* linked. Because you never know, and it’s a really cheap insurance policy. But I lost that battle years ago because dynamic linking was The Right Way To Do It.
I wonder, If you wrote a restricted shell, or a standard for one, that only allowed for authorized keys to be updated, if sites like Github would adopt it.
In principle there’s no reason they couldn’t be updated over SFTP (does/can this script do that?) – the problem with Github etc, though, is that they associate more metadata with keys than can be stored in the authorized_keys file. Specifically, the date they were added and last used.
In principle, this could still be managed, with a specialized SFTP daemon that intercepts such read/writes (whenever “.ssh/authorized_keys” is read, generate it from their own-format database. Whenever it is written, scan the new file for removals and additions). Probably this would be a better path than by trying to simulate a shell session.
Incidentally, @esr, I looked at the sshexport file and it doesn’t seem to have an embedded SPDX license tag. It seems that by embedding pexpect you’ve gone to some effort to make it a self-contained single file.
>Incidentally, @esr, I looked at the sshexport file and it doesn’t seem to have an embedded SPDX license tag. It seems that by embedding pexpect you’ve gone to some effort to make it a self-contained single file.
Yes, I hadn’t converted everything in my software directory to use SPDX yet. I just fixed this in the sshexport repo.
Interesting concept. Though I think I would implement it differently.
I’d rather log in with the password than the old key to be sure that I /could/ login and fix things if the new key failed in some way. (See my PWChanger utility for more details, http://dotfiles.tnetconsulting.net/tools/pwchanger/pwchanger.html)
That being said, I would be MUCH more interested in using SSH certificates, which include expiration date and a lot more restriction abilities.
SSH certificates are even nicer in that you don’t have to rotate things on the target servers. – The target server is configured to trust the signing CA and then will accept any SSH cert it signs. The SSH cert is where the time limit is imposed. So, clients simply get a new SSH (client) cert every … oh, I don’t know … 24 hours. That way the client can’t log in for more than 24 hours after it last got an SSH certificate. No need to actually rotate anything on the remote servers, or record any additional information about them that could potentially be mis-used.
Note: It’s trivial to configure the SSH server to trust the signing CA and disallow ~/.ssh/authorized_keys, so that people can’t add their own supplementary keys.
Sshexport sounds sexual to me.
I’ll use it in a sentence:
“My new fembot comes with 2 sshexports.”
No doubt enabled with full teledildonics.
Toward standard automated password and key changes
Automating password change and the rotation of SSH keys is a need that uber-programmer Eric S. Raymond is addressing (re the latter case) with “a script wrapper… using a Python expect engine to login into remote sites and install (or remove) ssh public keys.” I’d like to see a more general solution, one that might require less programming and (or fussing around when users want to change things password manually.) Pardon, if it’s all been thought of before and sheer laziness or some other factor has prevented implementation.
I’ll state the case for passwords.
The simple way is to move to a standard method of password change – just enter your present password, a space, and then your desired new password instead of only entering just your present password. (Of course, this means passwords can’t include spaces.) Only if your password is correct is the change implemented, of course (and extra verification steps such as text messages still happen as they do now.) A pop up window can then ask users to enter the password again, if desired, to prevent an unnoticed typo from locking you out; but password managers can ignore that window, and the change will go through.
Just for completeness, I’ll add a second implementation (all sorts of variations are possible):
Passwords aren’t allowed to start with “0” or “00.” Your password manager has the option of entering “16brandnewpasswordcrustyoldpassword” which logs you in with your previous pw (“crustyoldpassword”) but it does more than just that: it also and also changes your password for to “brandnewpassword” which happens to be 16 characters long. You or your password manager can enter your password as usual if you don’t want to change it.
When a user wants to manually make a change, this string concatenation can be hidden from the user entirely, if desired: the website or service can create this string from the usual user inputs in the usual places, and send it along to the server.
Note that passwords can’t be shorter than 10 characters long or longer than 100 characters for this method – however websites and services are free to insist on more than 10 digits for security purposes, or insist on some limit on size that’s less than 100, for convenience. (A length-standard for all sites and services might be nice, however!) Also the service must accept strings in the password field twice-as-long-plus-2 as the longest password they will accept (which might well be less than 100.)
Note that this implementation, too, is fail-safe not fail-unsafe. If you mistakenly enter the wrong number of digits for the new password, then the string you just entered will be rejected because of a password mismatch on the (lengthened or truncated) old password, and nothing happens except an error message.
Both methods automate the process of changing passwords in a way that is uniform across all websites and services. Your password management software, if it wishes, can change the password every single time it logs in anywhere, which is an obvious advantage. But, particularly with the first method any user can do so, too, without much thought or fuss, in a standard way.