Seamless Data Management with AWS SAM: Integrating DynamoDB 📊(Part 3/5)

Building upon the foundation set in our first article, we now venture into the realm of data with DynamoDB. Discover how AWS SAM makes easier the integration of DynamoDB into your serverless applications. Whether you’re storing user profiles handling game scores, or managing IoT device data, DynamoDB combined with SAM is the winning combination you need.

If you're following this Serverless adventure since the first article, you already know what's a DynamoDB, and you also have built a AWS Lambda.

This week we'll learn how to use AWS SAM to build a DynamoDB table, we'll also use the lastweek project to implement a method to write and read data from our NOSQL table on DynamoDB.

If you missed the last part of the series, I recommend you taking a break, and reviewing the previous episode here.

Let's start by setting up our project and creating a table

Last week we've made a folder structure and implemented a hello lambda function in it. Now, we'll need to change our project and create a new lambda to work with DynamoDB.

Before we begin, check out our previous project and adjust to match the following structure:

.
└── aws-opa-lambda-authorizer/
    ├── template.yaml
    └── database/
        └── data/
            ├── read.go
            └── update.go
        └── models/
            └── user.go
        └── main.go
    └── hello/
        └── main.go

Now we need to adjust the template.yaml file, to include our new lambda function, and also set up our new DynamoDB table. To do this, open your template file and change it to match the following code:

// template.yaml

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: AWS Serverless Lambda Template

Globals:
  Function:
    Architectures:
      - x86_64
    MemorySize: 128
    Runtime: go1.x
    Timeout: 25

Parameters:
  DynamoEndpointUrl:
    Type: String
    Description: The DynamoDB local URL
    Default: 'http://dynamodb:8000'

  TableName:
    Type: String
    Description: The DynamoDB table for storing user information.
    Default: 'UserTable'

  RegionName:
    Type: String
    Description: Default region for deployment.
    Default: 'sa-east-1'

  AwsEnvironmentName:
    Type: String
    Description: AWS Environment where code is being executed (AWS_SAM_LOCAL or AWS).
    Default: 'AWS_SAM_LOCAL'
    
Resources:
  UserTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: !Ref TableName
      AttributeDefinitions:
        - AttributeName: userId
          AttributeType: S
      KeySchema: 
        - AttributeName: userId
          KeyType: HASH
      ProvisionedThroughput: 
        ReadCapacityUnits: 2
        WriteCapacityUnits: 2

  HelloFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: ./hello/
      Handler: main
      Environment:
        Variables:
          message: "Hello, World!"

  UserFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: ./database/
      Handler: main
      Policies: AmazonDynamoDBFullAccess
      Environment:
        Variables:
          AWSENV: !Ref AwsEnvironmentName
          ENDPOINT: !Ref DynamoEndpointUrl
          REGION: !Ref RegionName
          TABLE: !Ref TableName

Let's understand the changes that were made.

Firt, we've inserted a new lambda function declaration named UserFunction. We've removed some attributes from the previous function (Architectures, MemorySize, Runtime and Timeout) because they're the same for HelloFunction and UserFunction, and declared a new Globals block at the top of the file to apply these attributes to all functions in the template file.

Attention! Be carefull when you refer to any policy in your functions. Here, I've used the AmazonDynamoDBFullAccess policy to simplify the code and demonstrate the construction of the template.yaml file, but I strongly recommend that you create your own role and policy with only the necessary permitions that your function will use.

Next, we've created a new Parameters block in the file to declare a parameter with the name of your table. This parameter will be referenced by the DynamoDB and UserFunction blocks, and can be really useful when you need to change something in your template. Parameters are like variables.

Finnaly, we've declared a DynamoDB block to create a UserTable. It'll be a simple table with userId as the Partition Key, and 2 units of read and write capacity. To learn more about the attributes you can use to set up your table, I recommend reading the oficial documentation here.

It's time to code our lambda

This is the best part our hands-on: it's time to code our lambda! As we said in the first episode, we'll use Golang to build all our projects.

Let's start creating our user model, inserting the following code in user.go file.

// database/models/user.yaml

package models

type User struct {
  ID       string `json:"userId"`
  Name     string `json:"name"`
  Lastname string `json:"lastname"`
  UserType string `json:"userType"`
}

Here we're creating our User struct with the four attributes. We're also using the json:"" tag at the end of each attribute to specify the equivalent name of the attribute if we import or export a JSON file.

Now, we'll need at least two methods to demostrate how to interact with DynamoDB from our lambda function. The first one will allow us to read data from our table. Place the following code in your read.go file:

// database/data/read.yaml

package data

import (
  "fmt"
  "os"

  "opa-lambda-database/models"

  "github.com/aws/aws-sdk-go/aws"
  "github.com/aws/aws-sdk-go/service/dynamodb"
  "github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute"
)

func Read(client *dynamodb.DynamoDB, userId string) (models.User, error) {
  input := &dynamodb.GetItemInput{
    TableName: aws.String(os.Getenv("TABLE")),
    Key: map[string]*dynamodb.AttributeValue {
      "userId": {
        S: aws.String(userId),
       },
    },
 }

 result, err := client.GetItem(input)
 if err != nil {
   return models.User{}, err
 }
 if result.Item == nil {
   return models.User{}, fmt.Errorf("Cold not find '%s'", userId)
 }

 user := models.User{}

 err = dynamodbattribute.UnmarshalMap(result.Item, &user)
 if err != nil {
   return models.User{}, fmt.Errorf("Failed to unmarshal Record, %v", err)
 }

  return user, nil
}

We're almost there! The second method will allow us to update and also insert a record in our table. Place the following code in your update.go file:

// database/data/update

package data

import (
  "os"

  "opa-lambda-database/models"

  "github.com/aws/aws-sdk-go/aws"
  "github.com/aws/aws-sdk-go/service/dynamodb"
  "github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute"
)

func Update(client *dynamodb.DynamoDB, user models.User) error {
  item, err := dynamodbattribute.MarshalMap(user)
  if err != nil {
    return err
  }

  input := &dynamodb.PutItemInput{
    TableName: aws.String(os.Getenv("TABLE")),
    Item:      item,
  }

  _, err = client.PutItem(input)
  return err
}

Note that in both methods, we've used the environment variable TABLE to reference the table name declared in our template.yaml file. That's a game-changer when we need to modify a table name, because all of our components are using the same reference. If the keys are the same, we won't need to modify our function codes, only the parameter content.

Finally, we need to write our main.go file. It will be a simple code to demonstrate the integration, but we'll modify it a little bit next week when we start building our API Gateway. Right now we'll just insert, read and update a record on our table. When you're ready, write the following code in your file:

// database/main.go

package main

import (
  "context"
  "encoding/json"
  "fmt"
  "os"
  "strings"
  
  "opa-lambda-database/data"
  "opa-lambda-database/models"
  
  "github.com/aws/aws-lambda-go/events"
  "github.com/aws/aws-lambda-go/lambda"
  "github.com/aws/aws-sdk-go/aws"
  "github.com/aws/aws-sdk-go/aws/session"
  "github.com/aws/aws-sdk-go/service/dynamodb"
)

var client *dynamodb.DynamoDB

func init() {
  sess := session.Must(session.NewSession())
  config := aws.NewConfig().WithRegion(os.Getenv("AWS_REGION"))
  
  if os.Getenv("AWSENV") == "AWS_SAM_LOCAL" {
    config = config.WithEndpoint(os.Getenv("ENDPOINT"))
  }
  
  client = dynamodb.New(sess, config)
}

func handler(ctx context.Context, request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
  var (
    result []string
    user   models.User
    err    error
  )
  
  // inserting a new record
  temp := models.User{
    ID:       "babayaga",
    Name:     "John",
    Lastname: "Wick",
    UserType: "junior_killer",
  }
  
  err = data.Update(client, temp)
  if err != nil {
    panic(fmt.Sprintln("Failed to insert record:", err))
  }
  
  // reading our record from dynamodb table
  user, err = data.Read(client, temp.ID)
  if err != nil {
    panic(fmt.Sprintln("Failed to read record:", err))
  }
  
  result = append(
    result,
    fmt.Sprintf("Great news!! We've successfully inserted %s %s record in our DynamoDB table.", user.Name, user.Lastname),
  )
  
  // updating our record in dynamodb table
  user.UserType = "senior_killer"
  
  err = data.Update(client, user)
  if err != nil {
    panic(fmt.Sprintln("Failed to update record:", err))
  }
  
  // reading our updated record from dynamodb table
  user, err = data.Read(client, user.ID)
  if err != nil {
    panic(fmt.Sprintln("Failed to read record:", err))
  }
  
  result = append(
    result,
    fmt.Sprintf("Congratulations!! %s %s has just promoted as a %s user.", user.Name, user.Lastname, user.UserType),
  )
  
  out, err := json.Marshal(result)
  if err != nil {
    panic(fmt.Sprintln("Failed to convert to JSON object:", err))
  }

  return events.APIGatewayProxyResponse{
      StatusCode: 200,
      Body:       string(out),
    }, nil
}

func main() {
  lambda.Start(handler)
}

You can notice that we have used another three additional environment variables here to create a connection with our DynamoDB table. The first variable specifies the region that we are using, the second variable indicates whether we are running in a local environment, and the last variable contains the address of the endpoint defined to our DynamoDB container.

Attention!! To run a local DynamoDB container you need to put all of containers in the same virtual network. See the Docker documentation to learn about it. Just run this project after the DynamoDB container is running and properly configured in your docker desktop.

To accelerate the process, see the following script for how to create your virtual network and start your DynamoDB container:

# here we're creating our virtual network named local-api-network
docker network create local-api-network

# Now we can run our container using the previous virtual network
# we're also defining a hostname for our container, the same used in the
# endpoint url of the template.yaml and we're also using the latest dynamodb
# image available
docker run -d -p 8000:8000 --network local-api-network --hostname dynamodb --name dynamodb amazon/dynamodb-local:latest

We're ready to finish coding by setting up our Golang project, as we did last week with HelloFunction. Just in case you don't remember how to do this, open the terminal on your computer, go to your lambda function folder /aws-opa-lambda-authorizer/database, then execute the following commands:

go mod init opa-lambda-database
go mod tidy

As we discussed last week, those commands will create the go.mod and go.sum files which specify our project namespace and dependencies.

Building the project

We also discussed about how easy it is to build a project with AWS SAM CLI. But it's important to reiterate this. Simply run the following command on your terminal:

sam build -t template.yaml

If everything is all right, you'll probably see a "Build Succeeded" message, and after that we'll be ready to performy our Lambda function tests.

demo of sam build command

It's time to performing a local test

Before we test the lambda function, start it. Make sure you have a Docker Desktop installed and running on your computer. Also check if your DynamoDB container is up and running too. If both aren't ready, you cannot proceed.

Start the function as the project build command we executed before. Simply run the following command on your terminal:

sam local start-lambda --docker-network local-api-network

Attention!! Don't forget use the parameter --docker-network to define that your lambda image will use your virtual network.

Once the Lambda is running, it will provide the execution URL, in our case: http://127.0.0.1:3001. You will use this URL as a local endpoint to run your Lambda with the help of the AWS CLI, using the following command:

aws lambda invoke --function-name"UserFunction" --endpoint-url="http://localhost:3001" out.txt

This command will invoke your Lambda function and generate an out.txt file with the response to your request. When you open the file, you will see that your Lambda function has responded with the logs we've defined on our insert, read and update routine.

demo of sam local start-lambda command

// out.txt

{
    "statusCode": 200,
    "headers": null,
    "multiValueHeaders": null,
    "body": [
        "Great news!! We've successfully inserted John Wick record in our DynamoDB table.",
        "Congratulations!! John Wick has just promoted as a senior_killer user."
    ]
}

Those are some links that you can read and learn more about Docker and AWS DynamoDB configuration. Also you can take a look at the repository with the complete project on my GitHub profile:

GitHub - raywall/aws-opa-lambda-authorizer

Home

AWS::DynamoDB::Table

This basic example shows you how powerfully the AWS Serverless Application Model is, and how it opens up a wide range of possibilites for creating applications and microservices with NOSQL database table integration.

Next wee we will contunie our journey on serverless world, buinding one more peace of our infrastructure. We will add our functions and DynamoDB in a API Gateway as a Rest API.

Before you go, check out our schedule for the next episodes:

1. Sept. 29th — Unleash the Power of AWS SAM: A Developer’s Guide to Building Serverless Magic with Go
2. Oct. 6th — Supercharge Your Apps with AWS Lambda: Getting Started with SAM
3. Oct. 13th — Seamless Data Management with AWS SAM: Integrating DynamoDB
4. Oct. 20th — API Nirvana: Building Serverless APIs with AWS SAM and API Gateway
5. Oct. 27th — Locking Down Your Serverless World: Implementing Lambda Authorizers

I hope you liked, and I see you in the next week, bye!