Tag Archives: hosting

c-ares

bye bye hosting c-ares web

3 Comments

At some point during 2003, my friend Bjørn Reese (from Dancer) and I were discussing back and forth and planning to maybe create our own asynchronous DNS/name resolver library. We felt that the synchronous APIs provided by gethostname() and getaddrinfo() were too limiting in for example curl. We could really use something that would not block the caller.

While thinking about this and researching what was already out there, I found the ares library written by Greg Hudson. It was an effort that was almost exactly what we had been looking for. I decided I would not make a new library but rather join the ares project and help polish that further to perfect it – for curl and for whoever else who wants such functionality.

It was soon made clear to me that the original author of this library did not want the patches I deemed were necessary, including changes to make it more portable to Windows and beyond. I felt I had no choice but to fork the project and instead I created c-ares. It would show its roots but not be the same. The c could be for curl, but it also made it into an English word like “cares” which was enough for me.

The first c-ares release I did was called version 1.0.0, published in February 2004.

The ares project did not have a website, but I am of the opinion that a proper open source project needs one, to provide downloads and not the least its documentation etc. A home. I created a basic c-ares website and since then I have hosted it on my server on the behalf of the c-ares project.

The was available as c-ares.haxx.se for many years but was recently moved over to c-ares.org. It has never been a traffic magnet so quite easy to manage.

In the backseat

In recent years, I have not participated much in the c-ares development. I have had my hands full with curl while the c-ares project has been in a pretty good shape and has been cared for in an excellent manner by Brad House and others.

I have mostly just done the occasional website admin stuff and releases.

Transition

Starting now, the c-ares website is no longer hosted by me. A twenty years streak is over and the website is now instead hosted on GitHub. I own the domain name and I run the DNS for it, but that is all.

The plan is that Brad is also going to take over the release duty. Brad is awesome.

cURL and libcurl

What if GitHub is the devil?

11 Comments

Some critics think the curl project shouldn’t use GitHub. The reasons for being against GitHub hosting tend to be one or more of:

  1. it is an evil proprietary platform
  2. it is run by Microsoft and they are evil
  3. GitHub is American thus evil

Some have insisted on craziness like “we let GitHub hold our source code hostage”.

Why GitHub?

The curl project switched to GitHub (from Sourceforge) almost eleven years ago and it’s been a smooth ride ever since.

We’re on GitHub not only because it provides a myriad of practical features and is a stable and snappy service for hosting and managing source code. GitHub is also a developer hub for millions of developers who already have accounts and are familiar with the GitHub style of developing, the terms and the tools. By being on GitHub, we reduce friction from the contribution process and we maximize the ability for others to join in and help. We lower the bar. This is good for us.

I like GitHub.

Self-hosting is not better

Providing even close to the same uptime and snappy response times with a self-hosted service is a challenge, and it would take someone time and energy to volunteer that work – time and energy we now instead can spend of developing the project instead. As a small independent open source project, we don’t have any “infrastructure department” that would do it for us. And trust me: we already have enough infrastructure management to deal with without having to add to that pile.

… and by running our own hosted version, we would lose the “network effect” and convenience for people that already are on and know the platform. We would also lose the easy integration with cool services like the many different CI and code analyzer jobs we run.

Proprietary is not the issue

While git is open source, GitHub is a proprietary system. But the thing is that even if we would go with a competitor and get our code hosting done elsewhere, our code would still be stored on a machine somewhere in a remote server park we cannot physically access – ever. It doesn’t matter if that hosting company uses open source or proprietary code. If they decide to switch off the servers one day, or even just selectively block our project, there’s nothing we can do to get our stuff back out from there.

We have to work so that we minimize the risk for it and the effects from it if it still happens.

A proprietary software platform holds our code just as much hostage as any free or open source software platform would, simply by the fact that we let someone else host it. They run the servers our code is stored on.

If GitHub takes the ball and goes home

No matter which service we use, there’s always a risk that they will turn off the light one day and not come back – or just change the rules or licensing terms that would prevent us from staying there. We cannot avoid that risk. But we can make sure that we’re smart about it, have a contingency plan or at least an idea of what to do when that day comes.

If GitHub shuts down immediately and we get zero warning to rescue anything at all from them, what would be the result for the curl project?

Code. We would still have the entire git repository with all code, all source history and all existing branches up until that point. We’re hundreds of developers who pull that repository frequently, and many automatically, so there’s a very distributed backup all over the world.

CI. Most of our CI setup is done with yaml config files in the source repo. If we transition to another hosting platform, we could reuse them.

Issues. Bug reports and pull requests are stored on GitHub and a sudden exit would definitely make us lose some of them. We do daily “extractions” of all issues and pull-requests so a lot of meta-data could still be saved and preserved. I don’t think this would be a terribly hard blow either: we move long-standing bugs and ideas over to documents in the repository, so the currently open ones are likely possible to get resubmitted again within the nearest future.

There’s no doubt that it would be a significant speed bump for the project, but it would not be worse than that. We could bounce back on a new platform and development would go on within days.

Low risk

It’s a rare thing, that a service just suddenly with no warning and no heads up would just go black and leave projects completely stranded. In most cases, we get alerts, notifications and get a chance to transition cleanly and orderly.

There are alternatives

Sure there are alternatives. Both pure GitHub alternatives that look similar and provide similar services, and projects that would allow us to run similar things ourselves and host locally. There are many options.

I’m not looking for alternatives. I’m not planning to switch hosting anytime soon! As mentioned above, I think GitHub is a net positive for the curl project.

Nothing lasts forever

We’ve switched services several times before and I’m expecting that we will change again in the future, for all sorts of hosting and related project offerings that we provide to the work and to the developers and participators within the project. Nothing lasts forever.

When a service we use goes down or just turns sour, we will figure out the best possible replacement and take the jump. Then we patch up all the cracks the jump may have caused and continue the race into the future. Onward and upward. The way we know and the way we’ve done for over twenty years already.

Credits

Image by Elias Sch. from Pixabay

Updates

After this blog post went live, some users remarked than I’m “disingenuous” in the list of reasons at the top, that people have presented to me. This, because I don’t mention the moral issues staying on GitHub present – like for example previously reported workplace conflicts and their association with hideous American immigration authorities.

This is rather the opposite of disingenuous. This is the truth. Not a single person have ever asked me to leave GitHub for those reasons. Not me personally, and nobody has asked it out to the wider project either.

These are good reasons to discuss and consider if a service should be used. Have there been violations of “decency” significant enough that should make us leave? Have we crossed that line in the sand? I’m leaning to “no” now, but I’m always listening to what curl users and developers say. Where do you think the line is drawn?

cURL and libcurl, Haxx, Web

A server transition

The main physical server (we call it giant) we’ve been using at Haxx for a very long time to host sites and services for 20+ domains and even more mailing lists. The machine – a physical one – has been colocated in an ISP server room for over a decade and has served us very well. It has started to show its age.

Some of the more known sites and services it hosts are perhaps curl, c-ares, libssh2 and this blog (my entire daniel.haxx.se site). Some of these services are however primarily accessed via fronting CDN servers.

giant is a physical Dell PowerEdge 1850 server from 2005, which has undergone upgrades of CPU, disks and memory through the years.

giant featured an Intel X3440 Xeon CPU at 2.53GHz with 8GB of ram when decommissioned.

New host

The new host is of course entirely virtual and we’ve finally taken the step into the modern world of VPSes. The new machine is hosted by the same provider as before but as an entirely new instance.

We’ve upgraded the OS, all packages and we’ve remodeled how we run the web services and all our jobs and services from before have been moved into this new fresh server in an attempt to leave some of the worst legacies behind.

The former server will not be used anymore and will be powered down and sent for recycling.

Glitches in this new world

We’ve tried really hard to make this transition transparent and ideally not many users will notice anything or have a reason to bother about this, but of course we also realize that we probably have not managed this to 100% perfection. If you detect something on any of the services we run that used to work or exist but isn’t anymore, do let us know so that become aware of it and can work on a fix!

This site (daniel.haxx.se) already moved weeks ago and nobody noticed. The curl site changed on October 23 and are much more likely to get glitches because of all the many more scripts and automatic things setup for it. Both sites are served via Fastly so ordinary users will not detect or spot that there’s a new host in the back end.

Rockbox

Rockbox services transition

1 Comment

Remember Rockbox? It is a free software firmware replacement for mp3 players. I co-founded the project back in 2001 together with Björn and Linus. I officially left the project back in 2014.

The project is still alive today, even of course many of us can’t hardly remember the concept of a separate portable music player and can’t figure out why that’s a good idea when we carry around a powerful phone all days anyway that can do the job – better.

Already when the project took off, we at Haxx hosted the web site and related services. Heck, if you don’t run your own server to add fun toy projects to, then what kind of lame hacker are you?

None of us in Haxx no longer participates in the project and we haven’t done so for several years. We host the web site, we run the mailing lists, we take care of the DNS, etc.

Most of the time it’s no biggie. The server hosts a bunch of other things anyway for other project so what is a few extra services after all?

Then there are times when things stop working or when we get a refreshed bot attack or web crawler abuse against the site and we get reminded that here we are more than eighteen years later hosting things and doing work for a project we don’t care much for anymore.

It doesn’t seem right anymore. We’re pulling the plug on all services for Rockbox that occasionally gives us work and annoyances. We’re offering to keep hosting DNS and the mailing lists – but if active project members rather do those too, feel free. It never was a life-time offer and the time has come for us.

If people still care for the project, it is much better if those people will also care for these things for the project’s sake. And today there are more options than ever for an open source project to get hosting, bug tracking, CI systems etc setup for free with quality. There’s no need for us ex-Rockboxers to keep doing this job that we don’t want to do.

I created a wiki page to detail The Transition. We will close down the specified services on January 1st 2021 but I strongly urge existing Rockboxers to get the transition going as soon as possible.

I’ve also announced this on the rockbox-dev mailing list, and I’ve mentioned it in the Rockbox IRC.

Network, Technology

Bye bye Crystone

or, why we should give up on service providers that don’t treat us well enough.

We co-locate

We (Haxx) have a server (technically speaking we have more than one but this is about our main one that hosts most of our public stuff). This server is ours. We bought it, installed it, configured it and then we handed it over to a company that “co-locates” it for us. It means they put our hardware in their big server room and we pay them for it and for the bandwidth our server consumes.

It also means that we have less control over it and we need to call the company to get access to our machine and so on. Ok, so we’ve used Crystone for this for a long time. They’ve been cheap enough and they haven’t complained when we’ve greatly overrun our bandwidth “allowance” for many months in a row.

A bad track record

We did have concerns a while ago (August 2009 and then again in March 2010) when they had power problems in their facility and we suffered from outages and server down-times. Crystone was then really bad at communicating with what happened, what they do and we started to look around for alternative providers since it started to get annoying and they didn’t seem to care for us properly. But we didn’t really get around to actually moving and time passed.

Maybe they had fixed their flaws and things were now fine?

A Saturday in May

Suddenly, on the early morning Saturday May 22nd 2010 our machine didn’t respond to network traffic anymore. We didn’t find out until we woke up and tried to use our services and after having tried a few things. we contacted Crystone to hear if the problem was theirs or if the problem was ours – we’ve had some troubles lately with the network interface card and we feared that perhaps the network might had stopped working due to this flaky hardware.

The customer service at Crystone immediately said that they were experiencing problems due to their move of the server park to the new facilities (they moved from Liljeholmen to Hammarby, both different locations within the general Stockholm area). They said they had network problems and that they were working on it. They did not give any estimation of when our machine would be back online.

They also said that they had mailed their customers about this move, and yeah we felt a bit bad about not having noticed such a mail so that we had been prepared.

The entire day passed. No network. Their web site mentioned problems due to this particular server move. We waited, we got no further info. We were unhappy.

Saturday become Sunday

How big problems can you have when the down-time for your customers exceeds 24 hours and you still haven’t fixed it nor told us what the problems actually are? The Sunday passed and they updated their web site a few times. The last update mentioned the time 16:03 and it said “most customers” are now back online and that if there’s any remaining problem we should contact their customer service. I spotted that message a couple of hours later, when our machine still wasn’t available. And what did customer service have to say to us about it? Nothing, they were closed. Our server remained dead and inaccessible.

Monday, now beyond 50 hours

In the wee hours of the Monday we passed 50 hours offline time and when the customer service “desk” opened in the morning and answered our phone call, they could get our machine back online. By rebooting it. No explanation from their part why our machine was like the only one that suffered this long.

A search in the mail logs also proved that Crystone never mailed us to tell that our server would move. Isn’t that odd? (not really, as we would find out later)

We won’t stand it

Already during the weekend we had decided we are fed up with this complete ignorance and crappy treatment. Down-times and problems happen, but the complete lack of information and care about us – their customers – is what made it clear we are not suitable to be their customers. We had to go elsewhere.

Crystone offered us a month fee worth of deduction on the hosting charges as a compensation for the troubles we had. That was nice of them, but really this service isn’t expensive so it’s not the cost of this that is burdensome. We just can’t stand having a service this unreliable and working with a company that is this uncommunicative.

This big server move was Crystone moving a lot of equipment over to the facility that is owned and run by Phonera, another ISP, and the one that we happened to have an offer from since before when we were looking for alternatives. Handy – we thought – perhaps we could just go there and carry our server over from one shelf to another and we’ll be fine. Phonera is slightly more expensive but hey, perhaps we’d get peace of mind!

“We don’t steal customers”

Phonera was first glad to accept us as customers, but surprised us greatly when they turned around and declined getting us as new customers, since they claimed they don’t want to “steal” customers from Crystone (that are now themselves customers of Phonera). Baffled, we simply sent off another request to Portlane instead and within minutes we had a decision made and a contract signed.

Later that afternoon, a Phonera guy got back to us and had changed position again and said that perhaps we could become customers anyway. They had figured out that none of them would gain by us going to a third company, but in any case it was now too late for them and we had already made up our minds about going Portlane.

“Sir, your server is not here”

On Tuesday 13:00, Björn (as co-admin of the server) had an appointment with Crystone to extract our server from their care to take it over to its new home. When he appeared in Hammarby at the new facility to get the server he was up for (another) surprise. It wasn’t there. Now Crystone could inform us that our server is still left in the old facility in Liljeholmen. It was never moved!

Glad our business with these guys would soon be over, Björn  handed over our 1U of server to Portlane and within a short while it had found a new home, with a new IP address and a new caretaker.

We could once again take a deep breath of relief and carry on with whatever we were doing before again.