Harbor supports OIDC authentication with a few providers out of the box. In this example, we’re configuring GitHub as the auth provider which means we need Dex.

Dex is an OAuth provider that you’re app can integrate with but in this case we just want to deploy it as a server on kubernetes.

Configuring Dex for GitHub Auth

Treat the dex configuration file as a secret because it has GitHub OAuth settings.

You must replace:

• - Example: https://dex.example.com
• - The GitHub client ID you get when creating an OAuth application in GitHub
• - The GitHub client secret you get when creating an OAuth application in GitHub
• - The GitHub organization users need to belong to in order to log in
• - Your made-up clientID for harbor
• - Your made-up clientSecret for harbor
• - Example: https://core.harbor.example.com

issuer: "<issuerUrl>"
storage:
  type: memory
connectors:
- type: github
  id: github
  name: GitHub
  config:
    clientID: <ghClientID>
    clientSecret: <ghClientSecret>
    redirectURI: "<issuerUrl>/callback"
    orgs:
    - name: <ghOrg>
staticClients:
- id: harbor
  name: <harborClientID>
  secret: <harborClientSecret>
  redirectURIs:
    - "<harborUrl>/c/oidc/callback"

Deploying Dex on K8s

This yaml example assumes that you created a kubernetes secret named dex-config with the key config.yaml containing the config file from above.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dex
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dex
  template:
    metadata:
      name: dex
      labels:
        app: dex
    spec:
      containers:
        - name: dex
          image: ghcr.io/dexidp/dex:v2.37.0
          imagePullPolicy: IfNotPresent
          args: 
            - dex
            - serve
            - --web-http-addr
            - 0.0.0.0:5556
            - --telemetry-addr
            - 0.0.0.0:5558
            - /etc/dex/config.yaml
          ports:
            - name: http
              containerPort: 5556
              protocol: TCP
            - name: telemetry
              containerPort: 5558
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /healthz/live
              port: telemetry
          readinessProbe:
            httpGet:
              path: /healthz/ready
              port: telemetry
          volumeMounts:
            - name: dex-config
              mountPath: /etc/dex
      volumes:
        - name: dex-config
          secret:
            secretName: dex-config

Configuring Harbor to use Dex OIDC

Finally, we need to tell harbor to use OIDC auth and point it to our Dex server endpoint.

Log in as an admin user and then click on “Configuration”.

Set the following:

• OIDC Provider Name: github
• OIDC Endpoint: <issuerUrl> from above
• OIDC Client ID: <harborClientID> from above
• OIDC Client Secret: <harborClientSecret> from above
• OIDC Scope: openid,profile,email

Sometimes we want to scale up and scale down the desired capacity of an autoscaling group based on a schedule.

For example:

• Work hours - evenings and weekends are not busy
• Business reasons - tax season, gift-giving season, etc.

To do this, Amazon offers scheduled scaling

Terraform supports creating scaling schedules using the aws_autoscaling_schedule resource.

Example

Here are some things learned from using aws_autoscaling_schedule:

• If you don’t set a start_time, the schedule will run immediately and then follow the schedule after that. Its a good idea to set the start time to the recurrence.
• You’ll want to ignore the changes to start_time otherwise terraform will want to update the resource with each terraform apply.
• Setting min_size, max_size or desired_capacity to -1 leaves the setting unchanged.

Scheduled Scale Down

resource "aws_autoscaling_schedule" "scheduled_scale_down" {
  scheduled_action_name  = "my-cluster-scale-down"
  min_size               = 1
  max_size               = -1
  desired_capacity       = 1
  start_time             = "${local.upcoming_friday}T23:59:00Z"
  recurrence             = "59 23 * * FRI"
  time_zone              = "US/Pacific"
  autoscaling_group_name = aws_autoscaling_group.my_cluster.name
  lifecycle {
    ignore_changes = [start_time]
  }
}

Scheduled Scale Up

resource "aws_autoscaling_schedule" "scheduled_scale_up" {
  scheduled_action_name  = "my-cluster-scale-down"
  min_size               = local.my_cluster_member.min_size
  max_size               = local.my_cluster_member.max_size
  desired_capacity       = local.my_cluster_member.desired_capacity
  start_time             = "${local.upcoming_monday}T03:00:00Z"
  recurrence             = "00 03 * * MON"
  time_zone              = "US/Pacific"
  autoscaling_group_name = aws_autoscaling_group.my_cluster.name
  lifecycle {
    ignore_changes = [start_time]
  }
}

When automating a GitHub Enterprise Server deployment, the set-settings API can be used to pass a settings.json object to configure the server. The settings.json object can be retreived from the get-settings API of your enterprise server.

If however you want to enable actions, the administrator usually needs to do it from the administration console of the GitHub Enterprise server.

It turns out you can automate this too by passing some “hidden” settings. First you will need to make sure the enterprise.feature_toggles.actions.enabled setting is true and then add the enterprise.actions_storage settings.

S3 example with only the settings for enabling Actions shown:

{
  "enterprise": {
    "feature_toggles": {
      "actions": {
        "enabled": true
      }
    },
    "actions_storage": {
      "blob_provider": "s3",
      "s3": {
        "bucket_name": "<my-bucket-name>",
        "service_url": "https://s3.<my-aws-region>.amazonaws.com",
        "access_key_id": "<redacted>",
        "access_secret": "<redacted>"
      }
    }
  },
  "run_list": [
    "recipe[enterprise-configure]"
  ]
}

Azure example with only the settings for enabling Actions shown:

{
  "enterprise": {
    "feature_toggles": {
      "actions": {
        "enabled": true
      }
    },
    "actions_storage": {
      "blob_provider": "azure",
      "azure": {
        "connection_string": "<my-connection-string>"
      }
    }
  },
  "run_list": [
    "recipe[enterprise-configure]"
  ]
}

Note: If you access the get-settings API after enabling actions, the storage secrets will not be returned.

The habitat supervisor is a component of Chef Habitat that will manage the lifecycle of habitat services but what manages the habitat supervisor?

There are a few good tutorials out there on how to do this via systemd but let’s look at doing it using runit.

What is runit?

Runit is a Unix init scheme that will start services on boot and keep them running. It will also constantly monitor the state of the service and restart (with a 1s cooldown) it if it fails. It’s included in many popular linux distros and is considered a light-weight alternative to systemd. If you’ve had any experience with chef-server or other chef enterprise software in the past, you might recognize it.

Getting started

For this tutorial we’re going to be using vagrant so make sure that’s installed first.

Create a directory to work out of and a Vagrantfile.

$ mkdir ~/hab-runit
$ cd ~/hab-runit
$ touch Vagrantfile

The contents of your Vagrantfile should be the following.

Vagrant.configure("2") do |config|
  config.vm.box = "gscho/centos-7-4-habitat"
  config.vm.box_version = "0.88.0"
end

This vagrant box is based on centos 7.4 and has habitat 0.88.0 pre-installed. After the file has been updated you can bring up a vagrant guest and login. And switch to root.

$ vagrant up
$ vagrant ssh
$ sudo su

The last step is installing runit since centos 7.4 does not come with it by default. You can skip this step on distros that come with runit pre-installed or included in their package repo.

$ curl -s https://packagecloud.io/install/repositories/imeyer/runit/script.rpm.sh | sudo bash
$ yum install -y runit

Creating the hab-sup runit template

To set up a runit template for the hab-sup service we create the required directory structure inside of /etc/. To do this we first create the /etc/sv/hab-sup directory and the run files that will instruct runit how to run the service and collect its logs.

$ mkdir /etc/sv
$ mkdir /etc/sv/hab-sup
$ mkdir /etc/sv/hab-sup/log
$ mkdir /etc/sv/hab-sup/env
$ touch /etc/sv/hab-sup/run
$ touch /etc/sv/hab-sup/log/run
$ chmod +x /etc/sv/hab-sup/run
$ chmod +x /etc/sv/hab-sup/log/run

We also need to create a directory that we can log our services output to. Let’s use /var/log/hab-sup.

$ mkdir -p /var/log/hab-sup

The run file in /etc/sv/hab-sup/log

The run file inside of the service’s log directory will tell runit how and where to log the output of your service. For this runit comes with a configurable loggin tool called svlogd which handles logging output and rotating the log file once it gets too large or reaches a certain age. It is expected that the monitored service logs to stdout.

Our /etc/sv/hab-sup/log/run file will simply be this:

#!/bin/sh
exec svlogd -tt /var/log/hab-sup

We execute the svlogd command and pass it the directory we want it to log to. The -tt flag means it will append each log entry with a human readable timestamp.

The env directory in /etc/sv/hab-sup

The env directory is an optional service directory which will contain all of the environment variables you wish to set for your service. It can be anywhere on the filesystem but having it inside the /etc/sv/hab-sup directory might make the most sense.

Runit expects that the name of each file in the env directory to be the name (key) of the environment variable and the contents of the files to be the value.

Let’s set the HAB_BLDR_URL environment variable for our service.

$ echo "https://bldr.habitat.sh" >> /etc/sv/hab-sup/env/HAB_BLDR_URL

The run file in /etc/sv/hab-sup

Next we need to tell runit how to run the habitat supervisor process. There is another tool called chpst (change process state) that can be used for modifying thing like which user the service should run as or setting environment variables.

Our /etc/sv/hab-sup/run file should look like the following:

#!/bin/sh
exec chpst -e /etc/sv/hab-sup/env /bin/hab sup run 2>&1

Here we execute chpst and pass the -e flag to tell it the path to our environment directory. We also redirect stderr to stdout since syslogd collects stdout only.

Running the service

The final step now that our service template has been created is to add a symbolic link for /etc/sv/hab-sup to /etc/service. Most distros use this directory as the service directory but you can double check using ps -ef | grep runsvdir to see what runit is watching. This step makes runit aware that the hab-sup service should be started and kept alive.

$ ln -s /etc/sv/hab-sup /etc/service/hab-sup

Within 5 seconds runit will recognize the change and start the service!

Controlling the hab-sup process

Once the service is up and running, you can use the sv command to control the state of the process.

Here are some basic commands to get your started:

• sv u hab-sup - starts the service
• sv d hab-sup - stops the service
• sv t hab-sup - kills the service (which causes it to start up again, in essence a restart)

Logging

Logs will be output to the file /var/log/hab-sup/current. This file will be automatically rotated when it reaches the default size (1000000 bytes).

Credits

Thanks to Mike Perham and the Rubyists.

This is a question I get a lot from clients, meetups and conferences (even ChefConf). If you visit the habitat website you’ll probably see some marketing mumbo jumbo like “Application Automation Framework” or “Automation that travels with your app”. The real answer is habitat consists of a few different components so it’s hard to categorize in a single sentence.

Habitat is a cross-platform package manager

The first component of habitat is a cross-platform package manager that supports linux and windows (not darwin). This part of habitat is most closely related to the nix package manager or chocolatey and allows you to create a package using bash or powershell. All packages are built and uploaded to an artifact repository called Builder where you can tag versions with a channel like unstable, stable or dev/qa/prod.

This alone is a pretty cool feature since it’s the only package manager that works accross both windows and linux that I have come across. It also allows you to create packages, manage them and install them the same way in all of your VMs and containers.

For example, installing and adding jq to your $PATH on linux, windows or inside a docker container is all the same command.

$ hab pkg install core/jq-static --binlink

Habitat is a service manager

The next component of habitat is the habitat supervisor. This component is a service manager, not unlike systemd, runit, init.d, etc. that can manage habitat services. Habitat services are also packages but they include an application that you’d like to run as a service. This could be a web server like nginx, a ruby on rails application, or just about anything else you could imagine.

The habitat supervisor is responsible for running these services and making sure they stay up. It also exposes an API on 0.0.0.0:9631 that lets you query for services and their health.

$ curl localhost:9631/services/redis/prod/health
> {"status":"OK","stdout":"","stderr":""}

Habitat is a service discovery and configuration tool

The habitat supervisor has the ability to peer with other habitat supervisors to create a GOSSIP ring. Through peering supervisors, you’re able to create service groups which can share configuration with each other. For example if you have peered N supervisors each running a redis service, you can apply a configuration to all of the redis servers in your ring at once with a single command.

$ hab config apply redis.prod $(date +%s) redis.toml

Habitat rings also open up the ability to bind with other service groups so you can do service discovery. For example, if you had defined a rails application that required redis, you can use the habitat bind information to discover its IP and port and automatically configure the rails app with that information.

$ hab svc load gscho/my-rails-pkg --bind redis.prod

Summary

The nice thing about chef habitat is that you can pick and choose which components you’d like to use. For example you could just use it as a package manager or you could use it as a services manager but use another tool like consul for service discovery.

What is fog?

Fog is a ruby cloud service library that provides a common abstraction over cloud provider SDKs. It supports a large number of cloud providers like AWS, Azure, GCP, DigialOcean, etc. albeit with varying degrees of maturity. You can find the whole list here. The documentation is lack-luster but most fog cloud provider libraries have extensive test suites that serve as a how-to for that particular provider.

Why fog?

So you might be asking yourself, “Doesn’t terraform do this for me?”.

Terraform at its heart a large diffing algorithm that maintains a snapshot of your infrastructure’s state and tries to make changes based on the current vs. desired state. It uses a configuration language developed by HashiCorp called hcl to create, update and manage that state.

Fog on the other hand is a ruby library that can be used like any other rubygem. It’s just ruby so you get all of the goodness of a dynamic, object-oriented programming language when creating cloud resources. Where fog really shines is as a cross-platform SDK that gives a consitent interface to doing common tasks like creating servers, volumes, etc. Another great feature is Fog.mock that lets you unit test your logic without creating any cloud resources and incurring costs.

Use Cases

I recently choose fog for a project where I needed to create a rails app that would allow users to CRUD compute resources and security groups for AWS, Azure and GCP.

Fog is a good choice if:

• your team is already using ruby and you’re tasked with creating and managing your own compute and storage infrastructure.

• you want loops, branching logic (if-else), OOP design patterns, mock testing using rspec or minitest

• you need to use more than one cloud-provider sdk and want a common interface across compute and storage resources.

Example

In the example script below I’m using fog to create one ec2 instance. Once the instance is created, the instance’s ID is saved to a yaml file to make things idempotent.

# frozen_string_literal: true

require 'fog/aws'
require 'yaml/store'

STORE = YAML::Store.new 'fog-aws.yml'
instance_id = STORE.transaction { STORE['instance_id'] }
ec2 = Fog::Compute.new(provider: 'AWS', region: 'us-east-1')

if instance_id.nil?
  key_name = 'fog-test'
  ec2.key_pairs.create(name: key_name) if ec2.key_pairs.get(key_name).nil?
  
  instance = ec2.servers.create(
    tags: { Name: 'fog-ec2' },
    key_name: key_name,
    flavor_id: 't3.small',
    image_id: 'ami-087c17d1fe0178315'
  )
  puts 'Instance is creating'
  STORE.transaction { STORE['instance_id'] = instance.id }
  instance.wait_for { ready? }
  puts 'Instance is ready!'
else
  puts 'Instance already exists'
  instance = ec2.servers.get(instance_id)
end

puts <<-EOF
public_dns_name = http://#{instance.dns_name}
public_ip_address = http://#{instance.public_ip_address}
private_dns_name = http://#{instance.private_dns_name}
private_ip_address = http://#{instance.private_ip_address}
EOF