Project 510 final report

This is the final report for project 510, Move all websites from HTTP to HTTPS: mp-unit.

This project covered the Managed Platform Unit's share of the conversion of our user-facing web sites to HTTPS. The overall project to convert all websites to HTTPS - including those run by the MPU and by the other units - is project 454, Move all web sites from HTTP to HTTPS. More details on the reasons for this work can be found in that project.

Stages of work

October 2020

(Effort: 4 hours.)

A survey of our websites revealed that only www.lcfg.org and computing.help.inf.ed.ac.uk offered an HTTP interface to users.

Stephen altered www.lcfg.org to make HTTP redirect to HTTPS. (This site was already being served via both HTTP and HTTPS, with no authentication complications, so the redirect was all that needed to be added.)

November 2020

(Effort: 16 hours.)

Neil Brown came up with a proof-of-concept configuration which separated the authentication interface of a site from its HTTPS user access interface. Neil's solution was what we went with. (This took him 9 hours, which will be attributed to project 511, Move all websites from HTTP to HTTPS: services-unit.) I spent some time trying out variations of it to help my understanding.

January 2021

(Effort: 37 hours.) I adapted Neil's configuration plan to suit the computing help servers' configuration. I got the new authentication configuration working on a development server, then another development server, and then on the backup server. I then reconfigured the backup server to use a DNS alias for its service address, rather than just its fully qualified hostname - and this exposed the fact that the new configuration only worked with the fully qualified hostname. Disaster.

March 2021

(Effort: 55 hours.) I went back and simplified and tidied the computing help servers' configuration, then that of the x509 certificates, the cosign configuration, and the Apache virtual hosts. This produced a setup which worked when the site was served over whatever arbitrary DNS alias had been assigned to it. The servers were successfully converted to this configuration. Finally, new 403 and 404 error pages were introduced, and links to restricted pages were tweaked to point to the authenticated site.

Lessons

Technical

I learned about SIXKTS and Let's Encrypt certificates, what each is used for and how to correctly specify them, and how and where to tie this in with Cosign and Apache configurations to construct a web site which uses our weblogin service to authenticate users.

Getting Cosign authentication working requires an exact match between various bits of configuration using the cosign component, the x509 component, the apacheconf component, and multiple bits of DNS. It's easy to get this wrong. To find out how to construct a site that'll use Cosign authentication, start with Toby's HTTPSCosignTutorial.

Mental

On my first attempt, my approach was to take a new piece of configuration which I didn't entirely understand, and to do my best to bolt it on to a larger mass of configuration which I didn't entirely understand. Amazingly, I got this to work, but it eventually became clear that it only worked in limited circumstances. I had to throw it away and start again.

My second approach was far more successful. This time I started from scratch. I worked out exactly what certificates the service would need, and exactly how to cleanly specify these; how many web addresses the site would have to respond to, and the desired behaviour of each; how to tie it successfully in to our cosign infrastructure; and so on. At each stage I made sure that I understood every detail, and how all the parts fitted together. This new understanding often helped me to spot chunks of duplicate or nonsensical configuration which could be tidied or thrown out. Ultimately, it enabled me to construct a solid machine which did exactly what it needed to, and to be confident that it would work properly.

I was able to do this thanks to one trick: I started to write everything down. Absolutely everything. All the thoughts I had, all the theories, all the discoveries - they all went into my journal. For me, writing down my thoughts enables a different way of thinking. My mind no longer darts about, making disorganised leaps, but rather I start to be able to work rationally through the problem in the correct order, learning and tidying as I go. I'd recommend this approach to anyone who finds that they have a panicky, baffled reaction to a complex problem area.

Effort

The project took 112 hours in total, a little over 3 weeks. Had I known then what I know now about working effectively, I could have done it in significantly less time. Nevertheless, I choose to believe that the extra hours were worth it for the valuable lessons I learned in the course of this project.

Thanks

To Toby, for patiently going through my broken configurations several times, so that he could tell me where the problems probably were and how to fix them, and for his helpful HTTPSCosignTutorial. To Neil, for devising the configuration which I adapted, and helpfully explaining it. To Stephen, each of whose recommendations on how to work with LCFG proved to be bang on. (Roughly, these were: be very clear which certificate is your external Let's Encrypt certificate, and which the internal SIXKTS certificate, and what each of them is for; and that you have a far better chance of being able to understand your configuration properly if you use LCFG resources all the way, rather than using the shortcut macros which insert several LCFG resources for you behind the scenes.) And to Alastair, who suggested encouraging ways forward when I got stuck.

-- ChrisCooke - March 2021

Topic revision: r6 - 25 Mar 2021 - 10:46:37 - ChrisCooke
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies