Dito Legislative Insights - Serverless Deployment

Dito Legislative Insights can be installed as a serverless deployment using Google Cloud Run and Memorystore Redis as shown in the following diagram.

Serverless Deployment of Legislative Insights

Users connect to the application via a load balancer that routes its requests through an Identity-Aware Proxy (IAP). IAP establishes a central authorization layer for applications accessed by HTTPS. Legislative Insights uses it to implement access control. Users are added to a Cloud IAM group which is granted permission to access the application. IAM verifies these permissions for each request to the application and grants or denys access as appropriate.

Authorized users are granted access to a single page application. It makes client side requests to the OpenStates API to identify URLs for State bills. URLs are submitted to the Insights Engine for processing. The Insights Engine proxies requests for these URLs to State Legislatures. It downloads PDF copies of legislative bills, extracts their text and facilitates analysis using an LLM. A similar process is followed for Federal legislation. A Cloud NAT Gateway is required for the Insights Engine to communicate with external resources.

Before you begin

  1. Create a dedicated Google Cloud Project for Legislative Insights
  2. Create a VPC and Subnet in the project
  3. Register a Google AI Studio API Key
  4. Identify the fully qualified domain name (FQDN) that you want to associate with the application. You will need to create an A-Record associating it with the IP address of the load balancer that this process creates.
  5. Download the Legislative Insights install scripts from Dito

Installation Procedure

Installation is controlled by shell scripts which use gcloud commands to setup the environment:

  • config.sh: sets environment variables for the installation
  • base-deployment.sh: creates the base install for Legislative Insights
  • load-balancer-deployment.sh: configures a load balancer to serve the application
  • iap-configuration.sh: configures an Identity-Aware Proxy to restrict access to the application
  • cleanup.sh: delete resources created by the other scripts

Configure the environment

Environment variables to control the installation are defined in a single file. Edit and save the config.sh script. Supply values for the variables shown in CAPITALS below:

#!/bin/bash
## GCP project configuration
project_id="GCP_PROJECT_ID"
location_id="GCP_REGION"
network_id="VPC_NETWORK_ID"
subnet_id="VPC_SUBNET_ID"

## Legislative Insights configuration
redis_instance_name="li-redis"
redis_instance_size=1 # Size in GB
service_account_id="insights-sa"
service_account_email="$service_account_id@$project_id.iam.gserviceaccount.com"
service_name="legislative-insights"
google_api_key="GOOGLE_API_KEY"

## SSL certificate configuration
domain_name="FQDN"
static_ip_name="li-static-ip"
certificate_name="li-ssl-cert"

## Load Balancer configuration
backend_service_name="li-backend-service"
url_map_name="li-url-map"
https_proxy_name="li-https-proxy"
forwarding_rule_name="li-forwarding-rule"
neg_name="${service_name}-neg"

## IAP OAuth configuration
application_title="Legislative Insights"
oauth_client_display_name="Legislative Insights Client"
support_email="EMAIL-ADDRESS"

Create the base installation

  1. Run the base-deployment.sh script (~10 minutes)

     ./base-deployment.sh
 WARNING: Your active project does not match the quota project in your local Application Default Credentials file. This might result in unexpected quota issues.

 To update your Application Default Credentials quota project, use the `gcloud auth application-default set-quota-project` command.
 Updated property [core/project].
 Creating Cloud Router 'nat-li-router'...
 Creating router [nat-li-router]...done.

 ...

 Cloud Run service deployed successfully.
 Next steps:

 1. Test the deployed service.
 2. Use the load-balancer-deployment.sh script to place the Cloud Run service behind a Global External Application Load Balancer.
 3. Use the iap-configuration.sh script to configure Identity-Aware Proxy (IAP) for the Cloud Run service.

    Ignore the quota project warning

  2. Find the Service URL in the output. Example:

     Service [legislative-insights] revision [legislative-insights-00001-dsx] has been deployed and is serving 100 percent of traffic.
 Service URL: https://legislative-insights-109863735454.us-central1.run.app

  3. Navigate to the Service URL in a browser and test the application by following the instructions in the Quick Start guide.

Configure the load balancer

  1. Run the load-balancer-deployment.sh script (~5 minutes)

     ./load-balancer-deployment.sh
 WARNING: Your active project does not match the quota project in your local Application Default Credentials file. This might result in unexpected quota issues.

 To update your Application Default Credentials quota project, use the `gcloud auth application-default set-quota-project` command.
 Updated property [core/project].
 Reserving static IP address 'li-static-ip'...
 Created [https://www.googleapis.com/compute/v1/projects/my-project-219/global/addresses/li-static-ip].

 -------------------------------------------------------------
 Static IP address has been reserved.
 Please update your DNS 'A' record for 'insights.mydomain.com' to point to the following IP address:
 130.211.12.75
 This is required for the SSL certificate validation to succeed.
 -------------------------------------------------------------

 Have you updated your DNS 'A' record? (y/n):

  2. Create or update a DNS A record for your FQDN to resolve to the supplied IP address.

  3. Answer y to the confirmation prompt to continue with the installation or n to exit.

     Have you updated your DNS 'A' record? (y/n): y
 Creating Google-managed SSL certificate 'li-ssl-cert'...
 Created [https://www.googleapis.com/compute/v1/projects/my-project-219/global/sslCertificates/li-ssl-cert].
 NAME         TYPE     CREATION_TIMESTAMP             EXPIRE_TIME  REGION  MANAGED_STATUS
 li-ssl-cert  MANAGED  2025-03-07T10:54:00.528-08:00                       PROVISIONING
     insights.mydomain.com: PROVISIONING
 Creating Serverless NEG 'legislative-insights-neg'...

 NAME                BACKENDS  PROTOCOL
 li-backend-service            HTTP
 Checking if Serverless NEG 'legislative-insights-neg' is already a backend in backend service 'li-backend-service'...
 Adding Serverless NEG 'legislative-insights-neg' to backend service 'li-backend-service'...
 Updated [https://www.googleapis.com/compute/v1/projects/my-project-219/global/backendServices/li-backend-service].
 Creating URL map 'li-url-map'...
 Created [https://www.googleapis.com/compute/v1/projects/my-project-219/global/urlMaps/li-url-map].
 NAME        DEFAULT_SERVICE
 li-url-map  backendServices/li-backend-service
 Creating HTTPS target proxy 'li-https-proxy'...
 Created [https://www.googleapis.com/compute/v1/projects/my-project-219/global/targetHttpsProxies/li-https-proxy].
 NAME            SSL_CERTIFICATES  URL_MAP     REGION  CERTIFICATE_MAP
 li-https-proxy  li-ssl-cert       li-url-map
 Creating forwarding rule 'li-forwarding-rule'...
 Created [https://www.googleapis.com/compute/v1/projects/my-project-219/global/forwardingRules/li-forwarding-rule].
 Load balancer setup complete.

 -------------------------------------------------------------
 You should now be able to access your Cloud Run service at https://insights.mydomain.com
 Note it may take some time for the SSL certificate to be provisioned and the DNS record to propagate.
 The service will not be accessible until this is complete.
 -------------------------------------------------------------

 You can check the status of the SSL certificate by running:
 gcloud compute ssl-certificates describe li-ssl-cert --global

 You can check the status of the forwarding rule by running:
 gcloud compute forwarding-rules describe li-forwarding-rule --global

 You can check the status of the DNS record by running:
 nslookup insights.mydomain.com

 Next steps:
 1. Monitor the status of the SSL certificate:
 gcloud compute ssl-certificates describe li-ssl-cert --global
 Wait for the certificate status to change from ' PROVISIONING' to 'ACTIVE'.

 2. Test the service by visiting https://insights.mydomain.com

 3. Use the iap-configuration.sh script to configure Identity-Aware Proxy (IAP) for the Cloud Run service.
 ./iap-configuration.sh

  4. Wait for the SSL certificate to change from ‘PROVISIONING’ to ‘ACTIVE’.

    PROVISIONING example

     cloud-run % gcloud compute ssl-certificates describe li-ssl-cert --global
 creationTimestamp: '2025-03-07T10:54:00.528-08:00'
 id: '2621575326792612115'
 kind: compute#sslCertificate
 managed:
 domainStatus:
     insights.mydomain.com: PROVISIONING
 domains:
 - insights.mydomain.com
 status: PROVISIONING
 name: li-ssl-cert
 selfLink: https://www.googleapis.com/compute/v1/projects/my-project-219/global/sslCertificates/li-ssl-cert
 type: MANAGED

    ACTIVE example

     gcloud compute ssl-certificates describe li-ssl-cert --global
 certificate: |
 -----BEGIN CERTIFICATE-----
 MIIFRjCCBC6gAwIBAgIQGqQaI0RZnWgQ40KLPAV9/TANBgkqhkiG9w0BAQsFADA7
 MQswCQYDVQQGEwJVUzEeMBwGA1UEChMVR29vZ2xlIFRydXN0IFNlcnZpY2VzMQww

 ...

 +qduBmpvvYuR7hZL6Dupszfnw0Skfths18dG9ZKb59UhvmaSGZRVbNQpsg3BZlvi
 d0lIKO2d1xozclOzgjXPYovJJIultzkMu34qQb9Sz/yilrbCgj8=
 -----END CERTIFICATE-----
 creationTimestamp: '2025-03-07T10:54:00.528-08:00'
 expireTime: '2025-06-05T11:54:02.000-07:00'
 id: '2621575326792612115'
 kind: compute#sslCertificate
 managed:
 domainStatus:
     insights.mydomain.com: ACTIVE
 domains:
 - insights.mydomain.com
 status: ACTIVE
 name: li-ssl-cert
 selfLink: https://www.googleapis.com/compute/v1/projects/my-project-219/global/sslCertificates/li-ssl-cert
 subjectAlternativeNames:
 - insights.mydomain.com
 type: MANAGED

  5. Test the application from the load balanced URL.

IAP Configuration

  1. Run the iap-configuration.sh script (~3 minutes)

     ./iap-configuration.sh
 Updated property [core/project].
 WARNING: IAP only protects requests that go through the Cloud Load Balancer. See the IAP documentation for important security best practices: https://cloud.google.com/iap/
 Updated [https://www.googleapis.com/compute/v1/projects/my-project-219/global/backendServices/li-backend-service].
 ✓ Updating... Done.
 Done.
 Service [legislative-insights] has been updated.
 ✓ Updating... Done.
 Done.
 Service [legislative-insights] has been updated.
 Enabling IAP API...
 OAuth brand for 'peter.goldthorp@ditoweb.com' exists.
 Checking if OAuth client exists...
 Creating OAuth client 'Legislative Insights Client'...
 Deleting firewall rule 'allow-cloudrun-ingress'...
 Deleted [https://www.googleapis.com/compute/v1/projects/my-project-219/global/firewalls/allow-cloudrun-ingress].
 IAP configuration complete.
 The Cloud Run service 'legislative-insights' is now protected by IAP.

 Next steps:

 1. Navigate to https://console.cloud.google.com/auth/audience
 Set the User type to 'External' if you need to grant access to users outside of your organization.

 2. Navigate to https://console.cloud.google.com/security/iap
 Grant access to the application by adding members to the 'IAP-secured Web App User' role.

 3. Test the IAP-secured Cloud Run service to verify that it requires authentication.

  2. After running the iap-configuration.sh script neither of the URLs tested earlier should allow access. The Cloud Run base URL should no longer exist and the load balanced one should display a screen that says “You don’t have access”. The following steps will grant access for a list of authorized individuals

  3. Navigate to https://console.cloud.google.com/auth/audience and set the User type to ‘External’ if you need to grant access to users outside of your organization. Click on the User Type button to toggle between external and internal

    Toggle User Type

  4. Navigate to https://console.cloud.google.com/security/iap. Check the IAP status. Toggle the service off and on if the current status is Error

    Toggle IAP

  5. Acknowledge the configuration requirements and hit the TURN ON button

    Enable IAP

  6. Click on the checkbox at the start of the row to open an info panel for the service

    Open the info panel

  7. Click the ADD PRINCIPAL button to grant user access to the application

    Grant user access

  8. Enter the email addresses for the users who need access into the New principals field. Click on the Select a role field and select Cloud IAP in Quick access followed by IAP-secured Web App User in Roles

    Grant user access

  9. Test the application to verify:

    • Authorized users are able to access the load balanced URL.
    • Unauthorized users are not able to access the load balanced URL.
    • Unauthenticated users (e.g. from an incognito session) redirected to a login screen and that after authentication they are granted or denied access as appropriate.

Cleanup

  1. Run the cleanup.sh script (~ 5 minutes) or delete the GCP project to remove resources and prevent unnecessary costs if the environment is no longer required.

     ./cleanup.sh
 Updated property [core/project].
 Starting cleanup process for project 'my-project-219' in region 'us-central1'...
 Deleting forwarding rule 'li-forwarding-rule'...
 Deleted [https://www.googleapis.com/compute/v1/projects/my-project-219/global/forwardingRules/li-forwarding-rule].
 Deleting HTTPS target proxy 'li-https-proxy'...
 Deleted [https://www.googleapis.com/compute/v1/projects/my-project-219/global/targetHttpsProxies/li-https-proxy].
 HTTP target proxy '' does not exist.
 Deleting URL map 'li-url-map'...
 Deleted [https://www.googleapis.com/compute/v1/projects/my-project-219/global/urlMaps/li-url-map].
 Deleting backend service 'li-backend-service'...
 Deleted [https://www.googleapis.com/compute/v1/projects/my-project-219/global/backendServices/li-backend-service].
 Deleting Serverless NEG 'legislative-insights-neg'...
 Deleted [https://www.googleapis.com/compute/v1/projects/my-project-219/regions/us-central1/networkEndpointGroups/legislative-insights-neg].
 Deleted network endpoint group [legislative-insights-neg].
 Releasing static IP address 'li-static-ip'...
 Deleted [https://www.googleapis.com/compute/v1/projects/my-project-219/global/addresses/li-static-ip].
 Deleting SSL certificate 'li-ssl-cert'...
 Deleted [https://www.googleapis.com/compute/v1/projects/my-project-219/global/sslCertificates/li-ssl-cert].
 Deleting Cloud Run service 'legislative-insights'...
 Deleting [legislative-insights]...done.
 Deleted service [legislative-insights].
 Deleting Redis instance 'li-redis'...
 Delete request issued for: [li-redis]
 Waiting for operation [projects/my-project-219/locations/us-central1/operations/operation-1741378658451-62fc6505b414b-847612d4-ec0ff685] to complete...done.
 Deleted instance [li-redis].
 Deleting Cloud Router 'nat-li-router'...
 Deleted [https://www.googleapis.com/compute/v1/projects/my-project-219/regions/us-central1/routers/nat-li-router].
 Deleting service account 'insights-sa@my-project-219.iam.gserviceaccount.com'...
 deleted service account [insights-sa@my-project-219.iam.gserviceaccount.com]
 Firewall rule 'allow-cloudrun-ingress' does not exist.
 Deleting Oauth client
 Deleted proxy client [109163735555-8e9a3v3h67m11snemheut69fp3qc2gcg.apps.googleusercontent.com].
 Cleanup process complete.

Copyright © Dito LLC, 2022, 2025