Discussion

Disruption with some GitHub services

ahstilde: github is at one nine, basically: https://news.ycombinator.com/item?id=47428035

Imustaskforhelp: 9% ? /s (To be honest I genuinely wouldn't be surprised if things go down so bad too at this point either)

steeleduncan: What has changed at GitHub to cause this?

qudat: Their primary goal in the last year was to move to Azure. Any massive infra migration is going to cause issues.

sc__: Microslop

inaros: Every day more Microsofty...they should rename to "Your Repository Needs To Restart To Apply Updates"

amarant: Lol, someone should make a pre -commit hook that reboots your computer with a message like this!

corvad: Just wait until github comes up with an outage tuesday.

nasretdinov: Must be Tuesday then

dsm4ck: Microslop at it again

yoyohello13: Vibe coding features.

rvz: GitHub goes down at least once a week as I said before. [0] thanks to Copilot, Tay.ai and Zoe chatbots wrecking the platform instead of humans maintaining it.If there was a prediction market for when GitHub experiences an outage every week, then you would make a lot of money.[0] https://news.ycombinator.com/item?id=47487881

pak9rabid: Vegas should start taking bets

bartread: Fuck sake. Again?Sorry, I realise this comment isn't up to HN's usual standards for thoughtfulness but... look, I'd bet the majority of us on this site rely on GitHub and I can't be the only one becoming incredibly frustrated with its recent unreliability?(And, yes, I did enough basic data analysis to confirm that it IS indeed getting worse versus a year, two years, and three years ago, and is particularly bad since the start of this year.)

guywithabike: The worst part of all this is that GitHub's CTO and VP of Engineering sent out the usual "here's what we'll do to fix things" letter to their larger customers and, without exaggeration, it boiled down to: 1) "Here's a bunch of stuff we already did!" which... clearly isn't working, and 2) "We're continuing our Azure migration." also clearly not working.So needless to say, if you depend on GitHub for critical business operations, you need to start thinking about what a world without GitHub looks like for your business and start working your way toward that. I know my confidence in GitHub's engineering leadership is at rock bottom.

Eji1700: I could sorta see a situation where the reality is "we're in the middle of a miserable transition and it'll clean up when we're done" but I don't think anyone has confidence that's all it is at this point.

pbkompasz: Vibe check?

paxys: Took a full 8 years for a Microsoft acquisition to go to shit, which is probably a record. Kudos to the Github team for holding out this long.

xeonmc: How fast was Skype?

msandford: I once worked at a place with more micro services than engineers. We joked about "we have as many 8s of uptime as you need!"

the_real_cher: seven nines? That's nothing , bro we got twelve eights!

packetlost: I second this. I'm done.

duped: Does github not do any kind of blue/green rollouts or what

esafak: Microsoft products are so human, they stop working weekly like it's the sabbath ...

rileymichael: looking forward to the `addressing-githubs-recent-availability-issues-3` news post

cmckn: While we’re complaining about GitHub, I’ve recently seen nonsensical behavior when creating a PR using the CLI. Sometimes, I get this error back: GraphQL: Resource not accessible by integration (createPullRequest.pullRequest) But the pull request is, in fact, created successfully. Just broken bullshit.

staticassertion: I assume this is all of the pains of going from "GHA is sorta kinda on Azure", which was a bad state, to "GHA is going full Azure", which is a painful state to get to but presumably simplifies things.

dec0dedab0de: You never go full Azure

merlindru: GitHub has always been incredibly outage riddled no? This is not a MSFT thing

0x3f: > I once worked at a place with more micro services than engineers.Currently consulting somwhere with 30 services per engineer. I cannot convince them this is hell. Maybe that makes it my personal hell.

htrp: To be fair a bunch of this is because the CEO after Nat Friedman (Thomas Dohmke) was pushed out in August 25.

workfromspace: Who was also the last CEO, right? Is this a coincidence?

ekropotin: Remember when GitHub was cool? Pepperidge Farm remembers.

whalesalad: I remember. My GitHub user ID is #5907, account created 2008-04-08T20:27:36Z. I think it is inevitable that all good things come to an end, but it's still a bummer to see.

Freedom2: As do I. Mine is even earlier as well!

stevepotter: I'm just going to stand by until Microsoft is back in everyone's good graces again by releasing some oss software that we all swoon over

wenbin: I guess vibe coding can't solve such problem for now...

nine_k: I don't remember that happening so much (if ever) in, say, 2016. But the frequency of noticeable incidents seemingly has been rising steadily since around 2023. The Azure migration apparently only exacerbated it.

hirako2000: I'm glad I moved over to forgejo. Being selfhosted, the UI loads faster. Most importantly, the thing is always responsive.

mfenniak: As a developer working on Forgejo -- glad you like it!

AndroTux: I’m still baffled that Minecraft is doing so well, despite the whole Bedrock thing. At this point I think Microsoft just forgot that they bought Mojang.

pylua: Anyone else notice other Microsoft cloud services ( for instance inside azure ) with bad performance also?I can’t be specific but we are constantly complaining.

sysworld: The Azure management UI, yes, so slowww. But the services (VMs etc) have been good.

rdtsc: From five nines to nine fives

gbear605: Circa 2019, my office had a bell that we would ring whenever GitHub had an outage, and it was rung several times per week.

ryukoposting: Is "migrating to Azure" the new "migrating to SAP?"

proc0: It's starting to really look like the AI effect. It might be coincidence but I've noticed a lot more downtime and bad software lately. The last Nvidia drivers gave me a blue screen (last week or so), and speaking about Windows, I froze updates last year because it was clear they were introducing a bunch of issues with every update (not to mention unwanted features).I like AI but actually not for coding because code quality is correlated to how well you understand the underlying systems you're building on, and AI is not really reasoning on this level at all. It's clearly synthesizing training data and it's useful in limited ways.

newbish: I think maybe it's not that GitHub is using AI, but that the amount of AI slop going into GitHub may be more than they expected.

hirako2000: It hosts all the repositories backing applycreatures, we ran dozens of git projects on the same instance, have teams, you guys did a phenomenal work. I would say it's even easy to customise.https://foja.applycreatures.comEdit: it has a wonderful API so I posted the link it may tempt some to ditch MS/Azure hub.

jrm4: Do your part; remind people that Github is not git. Git is decentralizable and people should know this.

sysworld: ooooh, they're migrating to Azure, now everything makes sense.

cyanydeez: they're not just migrating to Azure, they're vibrating to Azure!

newbish: So am I the only one thinking that maybe GitHub is succumbing to the weight of AI slop that's coming in from all the vibecoding, clawbots, and other AI workflows?

zelphirkalt: Man, a while ago I thought: "It happens often, alright, but every 2 weeks? Sounds like a slight exaggeration." But it really is every 2 weeks, isn't it? If I imagine in a previous job anything production being down every 2 weeks ... phew, would have had to have a few hard talks and course corrections.

keithnz: No? Azures been rock solid for us.

pylua: Front door did have a major outage last year.

pm90: i heard that they asked LinkedIn to do this too and they either refused or their systems were too complex so they refused to. Maybe that explains why LI availability seems ok

kleene_op: Azure, the color of BSOD

smartmic: AIpocalypse. Eaten too much Copilot dog food.

bartread: Perhaps even AIslopalypse.

KaiserPro: Looking at the status, its not one long outage, but lots of little ones, microslops if you will.

zahlman: > I realise this comment isn't up to HN's usual standards for thoughtfulness> And, yes, I did enough basic data analysis to confirmPerhaps you'd consider showing us that analysis? That sounds like it would make a pretty substantive, thoughtful comment.

KaiserPro: > consider showing usGaze upon the tapestry in which github paints it's failure with a thin copper red thread:https://www.githubstatus.com/

kace91: Is that a cyclical thing? I went through the VS code+typescript good graces era but I didn't know there were previous cycles.

bartread: @KaiserPro has pasted the link to someone else's heatmap, which is really good. Mine was just an Excel spreadsheet with a graph that I'd intended to write a blog about but then got demotivated on because I was too busy with other things and I saw that heatmap as well. Maybe I will do a proper write up next time GitHub has an outage and I'm blocked by it.

philipallstar: You've been here before!

zahlman: I've been using "slopocalypse". People already know AI is responsible, but slop existed before — e.g. conventionally generated SEO spam. It's just... so much worse now.

bartread: "Slopocalypse": yeah, I like that. Easier to pronounce too.At any rate, it seems like GitHub is back up now, so we'll see how long that lasts.

spauldo: They'd lose a whole lot of users if they killed Java edition, since the modded community is so large. They'd quickly find one of the Minecraft clones reaching feature parity. And there's no good reason for it - it's not like Java is a threat anymore.

AndroTux: Exactly. So why isn't Microsoft doing just that? Isn't that how Microsoft usually handles things? Just look at Xbox. They essentially screwed up everything they could and then some.

7777332215: Minecraft is a trick up their sleeve yet to be used. Manipulate and indoctrinate the youth.

Biganon: Indoctrinated by cubic cows

adzm: Weird Al needs to capitalize on this whole AI/Al thing

georgel: I remember it going down semi-regularly in the 2013+ era, and seeing HN posts about it. Especially if you were using a package manager reliant on GitHub like Cocoapods. It seems to me it is more "impactful" on the dev community now that they have gone past just being a centralized Git server for the team, to being the thing that does deploys and all sorts of other things.

RevEng: Not that I remember. VS Code was a surprising turnabout for a company that was both adamantly closed and threw FUD around like monkeys in a zoo.

someperson: GitHub has been unreliable since before AI. Though it's definitely gotten far worse.Seemingly the decline started with the Microsoft acquisition in 2018, and subsequent "unlimited private repository" change in 2019 (to match Gitlab's popular offer)

hparadiz: One example is the search being broken for CI logs. It takes over your browser's search hotkey too. What happens is every stage of the log is collapsed so the search doesn't work until you trigger the expansion but if you attempt to search before expanding the search will never work after it's been initialized. It's pretty infuriating when you're trying to find something in a giant build log.

justinholmes: Moved to self hosted crowci instead of actions.

jbmilgrom: h8 github so much. ahhhhhh

mememememememo: Down? No sir we are not down. There are elevated error rates and degraded performance.

xtracto: An isolated group of customers are experiencing elevated error rates and degraded performance.FTFY. (I've read AWS word it like that)

mememememememo: Technucally a set is a subset of itself.

messe: For fucks sake.I've been considering it for a while, but I'm definitely now pitching a move away from GitHub at our organization.

everforward: Even that doesn’t really make sense to me, unless they’ve done it in a way where everything has to move at once.Everywhere I’ve worked, if a migration is causing this much downtime then you kill the migration or slow it down. If every change has a 10% chance of bringing the site down, you only do a change every week or two until you can work out the kinks.

shrikant: ...or you keep fighting forward with the migration, because if it's seen as a failure then some pretty big heads will have to roll...

gchamonlive: [delayed]

butterlesstoast: That Go rewrite continues to rear its head, eh?

anotherjesse: We pride ourself on 9 5s!