End of SSH-RSA support for Azure Repos

Bohdan Janousek

February 15th, 202445 3

Azure Repos provides two methods for users to access a git repository in Azure Repos – HTTPS and SSH. To use SSH, you need to create a key pair using one of the supported encryption methods. In the past we’ve been supporting only SSH-RSA and we’ve asked users to enable the SSH-RSA here. This is not required to be done anymore as in 2022 we’ve added support for the RSA-SHA2-256 and RSA-SHA2-512 to Azure DevOps Service. Later that year, the same support was also added to Azure DevOps Server 2022 and in August 2023 to Azure DevOps Server 2020 and 2019. The relevant release notes are linked here:

We are now announcing the deprecation of SSH-RSA as a supported encryption method for connecting to Azure Repos using SSH.

The SSH-RSA is a weak encryption method. It is also already deprecated by OpenSSH and cannot be used unless enabled explicitly.

This change impacts you immediately if you are using Azure DevOps Service and are using SSH-RSA keys to connect to repos through SSH.

If you use Azure DevOps Server, then this change does not currently impact you. However, this change will impact you when you upgrade to the next version of Azure DevOps Server 2022.3 (expected to be released towards the end of 2024), since that version will not support SSH-RSA keys. If you use any of the following versions of Azure DevOps Server, you are strongly encouraged to move from SSH-RSA keys to more secure RSA-SHA2-256 or RSA-SHA2-512 keys:

  • Azure DevOps Server 2019 Update 1.2 Patch 4 and later
  • Azure DevOps Server 2020 Update 1.2 Patch 7 and later
  • Azure DevOps Server 2022

Migrating to more secure ciphers as supported by current versions of Azure DevOps Server will prevent issues in the future when you upgrade to newer versions of the server.

The deprecation of SSH-RSA ciphers in Azure DevOps Service will be done in four phases as explained below.

Phase I – User opt-in

Any user of Azure DevOps Service can migrate from SSH-RSA to more secure ciphers supported by Azure Repos. These are RSA-SHA2-256 or RSA-SHA2-512. To do so, users can follow these steps:

  1. Generate new public private key either by running ssh-keygen -t rsa-sha2-256 or ssh-keygen -t rsa-sha2-512.
  2. Add the key generated in point 1 to the SSH agent. This can be done by running command ssh-add <PathToYourPrivateKey>.
  3. Change the local SSH configuration so the key generated in point 1 is listed before the SSH-RSA key. This is to ensure more secure algorithms will be used instead of SSH-RSA.
  4. Upload the public part of the key generated in point 1 to Azure DevOps. See this how to do so.

Phase II – Throttling/delaying

Starting early March 2024, we will start to delay any SSH operation where the SSH-RSA was used to secure the SSH channel. There will be a warning shown in the command line output stating:

“ssh-rsa is about to be deprecated and your request has been throttled. Please use rsa-sha2-256 or rsa-sha2-512 instead. Your session will continue automatically. For more details see https://devblogs.microsoft.com/devops/ssh-rsa-deprecation.”

Phase III – Brown out

In April 2024, we will start to fail the executions of any operation where the SSH-RSA was used to secure the channel. We will execute the failures in a couple of stages each with different number of failure intervals and different length of the individual interval. The intervals will start at random times during the day. Each stage will last for about a week.

Stage Interval length Count of intervals in a day Total failure time in a day
1 30 minutes 1 30 minutes
2 1 hour 3 3 hours
3 2 hours 4 8 hours
4 1 hour 12 12 hours

To ensure it is clear to user why the SSH operation failed we will add an error message to command line output. The error message will be:

“You’re using ssh-rsa that is about to be deprecated and your request has been blocked intentionally. Any SSH session using SSH-RSA is subject to brown out (failure during random time periods). Please use rsa-sha2-256 or rsa-sha2-512 instead. For more details see https://devblogs.microsoft.com/devops/ssh-rsa-deprecation.”

Phase IV – SSH-RSA removal

Late in Q2 2024, we will start failing the execution of any operation where the SSH-RSA was used to secure the SSH channel. To ensure it is clear to user why the SSH operation failed we will add to command line output the following error message:

“You’re using ssh-rsa that is unsupported. Please use rsa-sha2-256 or rsa-sha2-512 instead. For more details see https://devblogs.microsoft.com/devops/ssh-rsa-deprecation.”

FAQ

Q: I am running the Azure DevOps Server. Can I use SSH-RSA going forward?
A: Yes, you can for now. Nonetheless, when you are on any version of Azure DevOps Server supporting new ciphers, then you should consider moving to these more secure ones. See the Phase I – User opt-in chapter for more details how to do so.

Q: I am running Azure DevOps Server version without support for the new ciphers, but I do want to use a more secure cipher than SSH-RSA. Can I do that?
A: Yes, but you need to upgrade to any version of Azure DevOps Server supporting these more secure ciphers.

Q: Are there plans to remove support for the SSH-RSA also from the Azure DevOps Server?
A: Yes. In a future version we will remove the support for SSH-RSA from Azure DevOps Server. So, please consider moving to a more secure cipher. See the Phase I – User opt-in chapter for more details how to do so.

Q: Will I be able to use the SSH-RSA to connect to Azure DevOps Server after you remove support for SSH-RSA?
A: Yes, but this will require your organization administrators or IT specialists to take special action. Without that, you will be able to use only the supported ciphers.

Bohdan Janousek Senior Program Manager, Azure DevOps

Read next

45 comments

Discussion is closed. Login to edit/delete existing comments.

Newest
Newest
Popular
Oldest
  • Dávid Rózsa · Edited 2

    In my opinion the brownout process causes unnecessary errors. During the brownout the server response still contains ssh-rsa:

    server proposal: host key algorithms: ssh-rsa,rsa-sha2-256,rsa-sha2-512

    This means that even if after the removal everything would work(as the response would only contain rsa-sha2-256,rsa-sha2-512) it will still result in an error during brownout unless ssh-rsa is explicitly disabled on client side, as a lot of clients will default to that if its available.

    It would make sense for me that during the brownout ssh-rsa is not returned as a valid option by the azure server.

    • Tomasz Bondarewicz · Edited 0

      I fully agree.
      I suppose they suppose, that if you support ssh-rsa then you don’t support anything more secure and should be warned with denying your access to the repo. Which assumption is completely wrong.

      But, taking the reported (and experienced) brownouts randomness – how can you be sure, that cited server proposal have reached the “brownout’ed” endpoint? 😉

      The implementation of this change is just messed and our complaints seem to be misunderstood (or ignored) 🙁

      • Eric Newton 0

        Another example of how botched this rollout is. I simply dont think they actually understand what they are even talking about, given the lack of information and downright incorrect information on how to generate a proper new key.

  • Snyder, George D. 0

    Honest Question: What’s the best way to hear about these sorts of things before they happen? To us, we didn’t know anything about it until our systems/processes were down due to not being able to reach ADO. This effectively got the message to us, but at the cost of down-time on our end.

    Should I be subscribing for notifications somewhere so that I can know about these brownouts?

    • Dahl, Tim 0

      It’s 5:30am, May 1st, 2024 in Atlanta, GA USA. Moments after the ArgoCD patch was delivered yesterday the brownouts stopped. The prophecy of total SSH-RSA blackout on May 1st is, at the moment, not true.

      We applied the ArgoCD patch across all of our [Dev, QA, Prod] environments and then reverted Dev back. We wanted to see Dev fail and all other environments work today.

      Everything is working…

      Reminds me of 2012 when the Mayan Calendar ended/reset…

      Some feared Armageddon… Some hoped for enlightenment… There was a movie…

      Some were engaged with Microsoft SEV A 24×7 support where they tirelessly and pro-actively looked into the concern… Some also scheduled on-call staff to be ready in case a change was noticed or a call came in…

      But it was just another day…

      So far…

    • Eric Newton 0

      Good luck with that. Every one of Microsofts teams wants control over their own messaging so it seems like every single one does their own thing.

      The warning messaging shouldve been going out since at least Jan-2024, I only started seeing it in March 2024. Removing ciphers is a big deal. Problem i’m facing is the system Jenkins is running on cant be upgraded without a lot of pain. And we were tring to get off of Jenkins, now Devops forces us to invest time into a system we’re trying to deprecate ourselves.

      The rollout shouldve been to create a new endpoint that only supported the new stuff. Send out warnings for the old endpoint about old ciphers getting decommissioned, and that gives people the ability to start testing using the new endpoints and finding out what will work.

      Instead we get this garbage policy of “selective, random brownouts” that give you no information of when it ends, so you start testing something, the window closes and your “fix” starts working but oh wait, its not in the brownout window… Absolutely terrible way to manage a rollout.

      • Snyder, George D. 0

        Thanks for sharing your experience with this. I have also been having trouble figuring out how to test that we’ve resolved it on our end and agree it is difficult without knowing when the brownouts will happen, or having a test endpoint to go after.

        But also, I don’t know what I don’t know. Maybe they do have a test endpoint that we could go after to confirm we have it working, and maybe there is some centralized channel for messaging we just don’t know about?

        I figure I won’t know unless I ask since other times with other things I’ve assumed a service was just bad all around and then came to realize they had provided for me, but that I had not looked in the right places or done what they were assuming I would do (which of course is partially their fault, but I definitely feel like it gets harder with larger products/companies – for me the main takeaway was that they were actively trying and heard my feedback).

        Keeping my fingers crossed 🤞

      • Tomasz Bondarewicz 0

        Maybe there’s a way to open a support ticket in Azure DevOps itself? This should probably give you more control over the information exchange process and a better chance of getting your questions answered. At the same time, if individual user reports were handled, information about the problems, their scale and solutions would not be available globally, which I consider a negative aspect. It seems that our complaints in this “just a blog post” 🙂 did not attract much attention from Microsoft.
        Sure, I understand and share your need for some sort of official channel to inform users about such changes. I wish we could live to see such a channel (or learn about it’s existence 🙂 )

      • Eric Newton 0

        Why publish a blog at all then, if the author of the article doesnt monitor comments on his own blog post.

        Making core connection changes is a big deal, and this style of seemingly random brownouts does nothing to help diagnose and setup for the future.

        The proper way wouldve been to setup a new endpoint that 100% prevents the old rsa-ssh (SHA1). While the old endpoint would be generating warnings for any connections using rsa-ssh (SHA1) for the forseeable future, along with a HARD DATE (not “late Q2 2024”) as to when it enforces.

        Absolute piss poor project management IMO.

  • Eric Newton 0

    Now again at 8:00 pm i’m not able to continue working for a f**king hour. THANKS GUYS.

  • Eric Newton · Edited 1

    Really having a devil of a time trying to get jenkins on ubuntu 14 to work. It doesnt work and then it works and then it doesnt work again.

    Thanks for the nondeterministic testing ability. Couldn’t even give a number of seconds the window would be denying requests??? Had you reported the number of seconds the window would be denying requests, then AT LEAST i’d have something tangible to work with.

    I do things to try to fix, it starts “working”, and so I think my last change makes it work. And then a f**king hour later its back to not working again.

    This is an absolutely atrocious way to conduct a phase out. The brownout period shouldnt last more than 10-30 minutes, and should BE DECLARED IN THE ERROR MESSAGE WHEN IT ENDS.

    Yet, its 6:00pm eastern time 25-Apr-2024 and now I cant work for one entire hour. Thanks y’all. REALLY APPRECIATED.

  • Mcneal, Evan B 6

    For ArgoCD users that are experiencing pain during the recent random “brownouts”, it seems that the ArgoCD Team is aware of an issue as outlined here: https://github.com/argoproj/argo-cd/issues/17634

    Using ecdsa or ED25519 would be ideal but Azure DevOps only supports rsa2-512 and 256, so we are waiting on either a fix from ArgoCD to explicitly support those or Microsoft to actually support new standard ssh tokens.

    @microsoft please consider supporting more than rsa in the future as this is a clearly huge problem for many people. The brownouts were NOT a good method to allow devs to test changes to this and this whole migration experience has been terrible.

  • Lucas Gonçalves 1

    Can you please prorogate a little bit these Brown out? We discover right now this “issue” and we need a bunch of ArgoCD to updates and there is a issue open for that:

    https://github.com/argoproj/argo-cd/issues/17634

  • Panos Sdonas 1

    Anyone knows when is the next brown out period? We can’t tell if our changes worked or not at the moment.

  • Tomasz Bondarewicz 2

    Hi,
    The way this change is being introduced unfortunately, in my opinion, complicates error analysis. Because when my software, which authenticates to the Azure DevOps repository, encounters errors, I don’t know how to pinpoint the exact cause of the problem or how to test fixes.
    Errors occur randomly. Time windows can start and end at any moment, and during their duration, not all connection attempts result in an error.
    If it starts to work – I cannot be sure if it’s because of my fixes or just because the “error window” has ended.
    I think, there should be some separate “endpoint” available, perhaps under a specific DNS address, that already has permanently disabled support for ssh-rsa. It could be used for verification to ensure that the systems I’m using are correctly configured.
    The advice in the “Phase I – User opt-in” section is, in my knowledge, misleading.
    There is no such thing as “generating rsa-sha2-256 or rsa-sha2-512 keys with ssh-keygen -t parameter.” What matters is whether the system from which we connect supports a given cipher and whether rsa-sha2-256 / rsa-sha2-512 are among the supported ciphers. The key itself does not imply a specific cipher but it’s negotiated during the handshaking between the client and the server.
    If my system (specifically, I use Flux on Azure Kubernetes Services with Ubuntu 22.04 worker nodes) started receiving errors when attempting to connect to repositories on Azure DevOps, it seems to me that I don’t need to generate new keys. Additionally, I am unable to disable the ssh-rsa cipher from the list supported by AKS worker nodes because they are provisioned by Azure.
    Please let me know if I’m wrong in the above interpretation.
    E.g. I have regenerated my keys yesterday’s morning and the eerors stopped. But then they reappeared after ~8 hours for ~1,5h. It’s been “quiet” since then through the night. But I have no idea if it’s OK now just because there is no “failure interval” for me right now.

    • Bohdan JanousekMicrosoft employee Author 0

      ssh-keygen support (of course) to generate the desired signature type when signing certificates using an RSA CA key. The available RSA signature variants are “ssh-rsa” SHA1 signatures, not recommended), “rsa-sha2-256”, and “rsa-sha2-512” (the default). See https://www.man7.org/linux/man-pages/man1/ssh-keygen.1.html for more details.

      Step 3 of the opt-in phase explicitly asked to “Change the local SSH configuration so the key generated in point 1 is listed before the SSH-RSA key. This is to ensure more secure algorithms will be used instead of SSH-RSA.”

      Guidance how to deal with common errors is provided here https://learn.microsoft.com/en-us/azure/devops/repos/git/use-ssh-keys-to-authenticate?view=azure-devops#q-ssh-cannot-establish-a-connection-what-should-i-do (that is linked in the blog post too).

      • Tomasz Bondarewicz 0

        What about environments where there is restricted (or none) access to local SSH configuration, like AKS worker nodes ?
        Someone can also assume, that the “local SSH configuration” regards the OS-level ssh config, while it can be as well the ssh implementation in installed software – like (as I suppose) ssh go library in FluxCD.

        Random, undocumented brown-out windows make it nightmare even more.

      • Rouke Broersma 1

        @bohdan The link you sent specifically says this about the -t option:

           -t dsa | ecdsa | ecdsa-sk | ed25519 | ed25519-sk | rsa
           Specifies the type of key to create.  The possible values
           are “dsa”, “ecdsa”, “ecdsa-sk”, “ed25519”, “ed25519-sk”,
           or “rsa”.

                   This flag may also be used to specify the desired
           signature type when signing certificates using an RSA CA
           key.  The available RSA signature variants are “ssh-rsa”
           (SHA1 signatures, not recommended), “rsa-sha2-256”, <strong>and
           “rsa-sha2-512” (the default).</strong>

        This indicates to me that this is not the most relevant step in the guide. The more secure option is after all already the default so most people will have keys using the secure signature algorithm.

        To me this makes Step 3 the most important step in the guide, yet this is the step with the least information. It would be great if you could modify the blogpost and write down exactly what should be changed in ~/.ssh/config in order to work with azure devops. Right now you’re leaving this as an exercise to the reader, instead of explicitly listing the required configuration. This is unhelpful when coupled with random brownouts. We have no way of confirming if configuration is correct or not, until it is too late.

      • Pablo Patruno 0

        Hi, did you get any luck ? I was struggling with this all the day . Even though I generate the key with the 256 or 512 signature I am still getting the same error .

    • Peter Ernst 3

      We have a similar thing happening with ArgoCD. Debugging this is extremely difficult since there is no way to check if our changes really fixed the issue, or if the Brown-out-Window has closed. Additionally we have been unable to restrict the used Algorithms through standard Linux methods inside of the container.
      I really like the idea of a endpoint, where the ssh-rsa HostKeyAlgorithm is disabled, or some other way such that you can reproduce this issue.

      • Alex Movergan · Edited 2

        Same nightmare with FluxCD. I’ve regenerated keys like 100 times. Same story. The error won’t go away.

        UPD: I tried to set the flux source controller to use --ssh-kex-algos=rsa-sha2-512, and now I can’t checkout with a new error:

        no common algorithm for key exchange; client offered: [ext-info-c kex-strict-c-v00@openssh.com], server offered: [diffie-hellman-group1-sha1 diffie-hellman-group14-sha1 diffie-hellman-group-exchange-sha256]

        UPD2: fixed it by setting

          set {
    name  = "sourceController.container.additionalArgs[0]"
    value = "--ssh-hostkey-algos=rsa-sha2-512"
  }
      • Panos Sdonas 2

        Hopefully this will be useful for anyone using FluxCD https://github.com/fluxcd/flux2/issues/4726. Unfortunately we won’t know until it crashes again.

      • Tomasz Bondarewicz 0

        Thanks @Alex Movergan !

        @Bohdan Janousek I still think that you are not doing this deprecation right :-/ Info about key regeneration is misleading in the first place. Second are the random brown-out windows.

Feedback usabilla icon