封面

aws dynamoDB使用总结

最近有一个新需求:需要将一部分敏感用户信息存入aws dynamoDB,作为一个长年被高墙圈养的开发者,自然对aws这种高端的东西一无所知,至到如今身处东京才能有幸接触。所以各种查看官网的各种文档和github的demo示例,跟踪源码调查报错等各种手段。从最开始的无从下手,到摸清整个流程花了大概1天半时间。大致感觉是和其他db整合没什么太大区别,尤其是和mongodb这种noSQL的DB用法很像。

概念

Amazon DynamoDB 是一种完全托管的 NoSQL 数据库服务,提供快速且可预测的性能,同时还能够实现无缝扩展。使用 DynamoDB,您可以免除操作和扩展分布式数据库的管理工作负担,因而无需担心硬件预置、设置和配置、复制、软件修补或集群扩展等问题。DynamoDB 还提供静态加密,这消除了在保护敏感数据时涉及的操作负担和复杂性。

使用 DynamoDB,您可以创建数据库表来存储和检索任意量级的数据,并提供任意级别的请求流量。您可以扩展或缩减您的表的吞吐容量,而不会导致停机或性能下降。此外,您还可以使用 AWS 管理控制台来监控资源使用情况和各种性能指标。

DynamoDB 提供了按需备份功能。它允许您创建表的完整备份以进行长期保留和存档,从而满足监管合规性需求。

您可以为 Amazon DynamoDB 表创建按需备份以及启用时间点恢复。时间点恢复有助于保护表免遭意外写入或删除操作。使用时间点恢复,您可以使该表还原到最近 35 天中的任何时间点。

DynamoDB 可以从表中自动删除过期的项,从而帮助您降低存储用量,减少用于存储不相关数据的成本。

Cli安装

  1. 通过python环境安装

pip install awscli

  1. pkg安装

    https://docs.aws.amazon.com/zh_cn/cli/latest/userguide/install-cliv2-mac.html

aws cli操作dynamoDB

  1. 配置权限

    aws configure 参考aws command line用法

  2. 查看数据库表

    aws dynamodb describe-table --table-name staging-g123-platform-user-profile

{
"Table": {
"AttributeDefinitions": [
{
"AttributeName": "provider",
"AttributeType": "S"
},
{
"AttributeName": "user_id",
"AttributeType": "S"
}
],
"TableName": "staging-g123-platform-user-profile",
"TableStatus": "ACTIVE",
"CreationDateTime": "2020-11-13T12:26:51.565000+09:00",
"ProvisionedThroughput": {
"NumberOfDecreasesToday": 0,
"ReadCapacityUnits": 1,
"WriteCapacityUnits": 1
},
"TableSizeBytes": 0,
"ItemCount": 0,
"TableArn": "arn:aws:dynamodb:ap-northeast-1:174631816133:table/staging-g123-platform-user-profile",
"TableId": "a04d71da-4903-421f-baca-2896013509ba"
}
}
  1. 帮助命令

    aws dynamodb help 查询出来的内容和官网给出的文档一致

aws dynamoDB官方客户端

下载地址:https://docs.aws.amazon.com/zh_cn/amazondynamodb/latest/developerguide/workbench.settingup.html

NoSql Workbench会读取本地~/.aws/config下的认证信息

image-20201117152708701

对应配置文件中的三个连接,当然前提得在~/.aws/credentials有配置对应的aws_access_key_idaws_secret_access_key, 配置命令: aws configure --profile prod,然后输入对应的key和id和secret,参考aws command line用法

image-20201117152756173

可以自动生成python/java/js的示例代码

image-20201116160046238

也可以查看表结构的Metadata,注意KeySchema的KeyType。Hash对应@DynamoDBHashKey(attributeName = “user_id”),RANGE对应@DynamoDBRangeKey(attributeName = “provider”)

image-20201117151507038

查询结果和mysql类似

image-20201117151645612

spring boot整合aws java sdk dynamodb

gradle

implementation 'com.amazonaws:aws-java-sdk-dynamodb:1.11.901'

maven

<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-dynamodb</artifactId>
<version>1.11.901</version>
</dependency>

application-dev.yml

dynamodb:
tableNames:
userInfoTable: stg-g123-platform-user-profile
orderTable: stg-g123-platform-order-profile

application-prod.yml

dynamodb:
tableNames:
userInfoTable: production-g123-platform-user-profile
orderTable: production-g123-platform-order-profile

yaml配置中对应的加载配置类

package jp.g123.g123_auth.property;

import java.util.Map;
import javax.validation.constraints.NotNull;
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.validation.annotation.Validated;

@ConfigurationProperties(prefix = "dynamodb")
@Validated
@Data
public class DynamoDBProperties {
@NotNull private Map<String, String> tableNames;
}

dynamoDB配置类(这里没有写权限的配置项,所以它会默认读取本地aws的配置项【~/.aws文件夹下的权限配置】)

TableNameResolver会将注入的dbProperties中的表中配置进去,可以做到不同环境下使用不同的数据库表

package jp.g123.g123_auth.configuration;

import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper;
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapperConfig;
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBTable;
import java.util.Map;
import javax.validation.constraints.NotNull;
import jp.g123.g123_auth.property.DynamoDBProperties;
import lombok.RequiredArgsConstructor;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
@EnableConfigurationProperties(DynamoDBProperties.class)
@RequiredArgsConstructor
public class DynamoDBConfiguration {

@Bean
public AmazonDynamoDB getDynamoDBClient() {
// use default dynamic db config
return AmazonDynamoDBClientBuilder.standard().build();
}

@Bean
public DynamoDBMapper getDynamoDBMapper(
AmazonDynamoDB client, DynamoDBProperties dynamoDBProperties) {
@NotNull Map<String, String> tableNames = dynamoDBProperties.getTableNames();
DynamoDBMapperConfig mapperConfig =
DynamoDBMapperConfig.builder()
.withTableNameResolver(
(clazz, config) -> {
String tableName =
clazz.getAnnotation(DynamoDBTable.class).tableName();
return tableNames.get(tableName);
})
.build();
return new DynamoDBMapper(client, mapperConfig);
}
}

实体类的基类

// BasicItem
package jp.g123.g123_auth.dynamicdb.base;

public abstract class BasicItem {}

数据库表映射(表名注意不要写错,需要一个常量字符串)

// UserInfoItem
package jp.g123.g123_auth.dynamicdb.domain;

import com.amazonaws.services.dynamodbv2.datamodeling.*;
import java.util.Date;
import jp.g123.g123_auth.dynamicdb.base.BasicItem;
import lombok.*;

@Data
@Builder
@AllArgsConstructor
@NoArgsConstructor
@EqualsAndHashCode(callSuper = false)
@DynamoDBTable(tableName = "staging-g123-platform-user-profile")
public class UserInfoItem extends BasicItem {
private String userId;
private String provider;
private String profile;

@DynamoDBHashKey(attributeName = "user_id")
public String getUserId() {
return userId;
}

@DynamoDBRangeKey(attributeName = "provider")
public String getProvider() {
return provider;
}

@DynamoDBTypeConvertedTimestamp(pattern = "yyyy-MM-dd HH:mm:ss", timeZone = "Japan")
@DynamoDBAttribute(attributeName = "created_at")
private Date createdAt;

@DynamoDBTypeConvertedTimestamp(pattern = "yyyy-MM-dd HH:mm:ss", timeZone = "Japan")
@DynamoDBAttribute(attributeName = "updated_at")
private Date updatedAt;
}

dao基类(提供了基本的CURD功能)

package jp.g123.g123_auth.dynamicdb.base;

import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper;
import java.util.List;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Repository;
import org.springframework.util.CollectionUtils;

@Repository
public class BasicOperationDao<T extends BasicItem> {
@Autowired private DynamoDBMapper mapper;

public void save(T item) {
if (item == null) {
return;
}
this.mapper.save(item);
}

public void batchSave(List<T> items) {
if (CollectionUtils.isEmpty(items)) {
return;
}
this.mapper.batchSave(items);
}

public void delete(T keyItem) {
if (keyItem == null) {
return;
}
this.mapper.delete(keyItem);
}

public void batchDelete(List<T> keyItems) {
if (CollectionUtils.isEmpty(keyItems)) {
return;
}
this.mapper.batchDelete(keyItems);
}

public void update(T item) {
if (item == null) {
return;
}
this.mapper.save(item);
}

public void batchUpdate(List<T> items) {

if (CollectionUtils.isEmpty(items)) {
return;
}
this.mapper.batchSave(items);
}

public T load(T keyItem) {
return this.mapper.load(keyItem);
}

public List<T> batchLoad(List<T> keyItems) {
return this.mapper.load(keyItems);
}
}

UserInfoDao,有了基类dao,没有特殊业务需要就不需要额外写别的内容

package jp.g123.g123_auth.dynamicdb.dao;

import jp.g123.g123_auth.dynamicdb.base.BasicOperationDao;
import jp.g123.g123_auth.dynamicdb.domain.UserInfoItem;
import org.springframework.stereotype.Repository;

@Repository
public class UserInfoDao extends BasicOperationDao<UserInfoItem> {}

业务对应类 UserInfoBo

package jp.g123.g123_auth.dynamicdb.bo;

import java.util.Date;
import javax.servlet.http.HttpServletRequest;
import jp.g123.g123_auth.dynamicdb.domain.UserInfoItem;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;

@Builder
@Data
@AllArgsConstructor
@NoArgsConstructor
public class UserInfoBo {
private String userId;

private String provider;

private String profile;

private Date createdAt;

private Date updatedAt;

public UserInfoItem toItem() {
return UserInfoItem.builder()
.userId(this.userId)
.provider(this.provider)
.profile(profile)
.createdAt(createdAt)
.updatedAt(updatedAt)
.build();
}

public static UserInfoBo fromItem(UserInfoItem userItem) {
if (userItem == null) {
return null;
}

return UserInfoBo.builder()
.userId(userItem.getUserId())
.provider(userItem.getProvider())
.profile(userItem.getProfile())
.createdAt(userItem.getCreatedAt())
.updatedAt(userItem.getUpdatedAt())
.build();
}
}

UserInfoService 有了这个类,剩下的就是和其他spring处理业务没有任何区别。

package jp.g123.g123_auth.dynamicdb.service;

import java.util.Date;
import jp.g123.g123_auth.dynamicdb.bo.UserInfoBo;
import jp.g123.g123_auth.dynamicdb.dao.UserInfoDao;
import jp.g123.g123_auth.dynamicdb.domain.UserInfoItem;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class UserInfoService {
@Autowired private UserInfoDao userInfoDao;

public void insertOrUpdateUserInfo(UserInfoBo userBo) {
if (userBo == null) {
return;
}

UserInfoBo existUser = getUserInfoById(userBo);
if (existUser != null) {
existUser.setUpdatedAt(new Date());
updateUserInfo(existUser);
} else {
userBo.setCreatedAt(new Date());
userBo.setUpdatedAt(new Date());
insertUserInfo(userBo);
}
}

public void deleteUserInfoById(UserInfoBo userInfoBo) {
if (userInfoBo == null) {
return;
}
UserInfoItem userInfoItem = userInfoBo.toItem();
userInfoDao.delete(userInfoItem);
}

public void insertUserInfo(UserInfoBo userBo) {
if (userBo == null) {
return;
}
UserInfoItem item = userBo.toItem();
userInfoDao.save(item);
}

public void updateUserInfo(UserInfoBo userBo) {
if (userBo == null) {
return;
}
UserInfoItem item = userBo.toItem();
userInfoDao.save(item);
}

public UserInfoBo getUserInfoById(UserInfoBo userInfoBo) {
if (userInfoBo == null) {
return null;
}
UserInfoItem userInfoItem = userInfoBo.toItem();
UserInfoItem userItem = userInfoDao.load(userInfoItem);
return UserInfoBo.fromItem(userItem);
}
}

查看插入的数据

image-20201117151645612

参考资料

  1. aws官方开发文档

  2. java sdk 示例

  3. aws demo github(非官方)

文章目录
  1. 1. 概念
  2. 2. Cli安装
  3. 3. aws cli操作dynamoDB
  4. 4. aws dynamoDB官方客户端
  5. 5. spring boot整合aws java sdk dynamodb
  6. 6. 查看插入的数据
  7. 7. 参考资料


twitter分享


如果想及时收到回复,可在 订阅中心Participating中勾选Email

Fork me on GitHub