Hi all, I’m Sebastian Alvis. I work on the Pangeo Project at UW and recently put out the blog post Deploying JupyterHub-Ready Infrastructure with Terraform on AWS. I’m looking to contribute a guide on how to set up a minimal EKS cluster for Zero-to-JupyterHub-k8s, but would like some guidance on what this minimal guide would look like.
In the blog post’s attached repo, I supplied the Terraform configuration files necessary to set up the infrastructure, but this infrastructure had a lot of opinionated design choices that were several steps up from the bare minimum. I think @choldgraf has maintained that Zero-to-JupyterHub-k8s should be minimal but include links to further resources. I plan to make a smaller set of resources for this guide. However, since my blog post used a set of Terraform modules (terraform-aws-eks and terraform-aws-vpc) to configure many of the pieces without a lot of input from me, it may deploy more than the minimal set of resources needed. Is that a fine place to start, or do we need a minimum deployment resource set?
As I work through creating the deployment and guide, I’ll use this thread as a place to drop more questions and I would appreciate feedback. I also plan to make a guide for deploying the JupyterHub via Terraform, but that is a bit further into the future.
I am now actively working on this Terraform Guide.
@manics thanks for letting me know it’s a premium post. I’m thinking of putting a markdown / rST file for the blog post into the repo so that people can read it for free.
@betatim I will definitely add a link in that section. The repo has multiple ways to use Terraform to deploy infrastructure that people will hopefully find helpful.
I’ve read through some of the Jupyter Contribution Guide (mostly the Documentation Guide sections) and the open issues related to Terraform (there are only two that mention it). It seems like the main desire for Z2JH guides are that they are easy to understand and general enough that they can be adapted to more advanced use-cases. I feel like Terraform, though a new tool, does simplify the workflow for deploying infrastructure that could be beneficial for a lot of new users (this is absolutely biased because I have had success with Terraform). It has broad support across cloud providers, though functionality is somewhat behind each cloud providers’ own CLI, and makes it significantly easier to tear down resources you are done with.
As I am building this guide, I’m walking through the Z2JH guide for setting up Kubernetes on AWS w/ EKS to try and match that section’s level of detail. I noticed that the VPC section really just links to the Getting Started with Amazon EKS guide. The three options (private subnets, public subnets, and both) all are launched via CloudFormation templates, which aren’t really transparent to users.
For Terraform, I can similarly reference the terraform-aws-vpc module, which will simplify the instructions greatly, but isn’t exactly easy for a user to figure out. I figure this is a good compromise linking to good existing resources and also doesn’t require me to manually figure out VPC networking right now. Does that seem fine to be an option in Zero-to-JupyterHub-K8s? Or would it still be recommended as a configuration to be linked in the community section? If so, would the lowest-level Terraform configuration (ie manual networking / no modules) be what Z2JH wants or is Terraform separate enough from the other tools such that it should be kept in the community section?
This question can extend to the EKS cluster itself, which the terraform-aws-eks module would handle. Whatever seems like the best option will be followed for both of these main pieces.
I can also open an issue on the Z2JH to solicit more input.
Having a set of Terraform instructions on a separate repository is very valuable, and since it’s in a separate repo you can use your judgement as to how complicated or configurable you want to make it.
I think it’s best in a separate repo because unlike the existing guides it’s not just a set of commands to run, it involves Terraform files. Having the docs and example of files in their own repo makes it easier to test and update.
Fair enough. I’ll set up the minimal repo with the two modules and open an issue saying I will contribute some links to these repos in the Community section of Z2JH!