作者 Administrator

合并分支 'test' 到 'master'

Test



查看合并请求 !1
正在显示 100 个修改的文件 包含 3253 行增加132 行删除

要显示太多修改。

为保证性能只显示 100 of 100+ 个文件。

... ... @@ -23,7 +23,11 @@ _testmain.go
*.test
.log
.idea
.vscode
app.log
go.sum
lastupdate.tmp
*.log
public/*
\ No newline at end of file
... ...
... ... @@ -3,13 +3,12 @@ ENV APP_DIR $GOPATH/src/character-library-metadata-bastion
RUN mkdir -p $APP_DIR
WORKDIR $APP_DIR/
COPY ./pkg pkg
COPY ./conf conf
COPY ./config config
COPY ./go.mod go.mod
COPY ./go.sum go.sum
COPY ./main.go main.go
RUN ["ln","-sf","/usr/share/zoneinfo/Asia/Shanghai","/etc/localtime"]
ENV GO111MODULE on
ENV GOPROXY https://goproxy.io
ENV GOPROXY https://goproxy.cn
RUN ["go","mod","tidy"]
RUN ["ls"]
RUN ["go","build"]
... ...
... ... @@ -3,9 +3,450 @@
## 源文件管理
- 文件列表 - list
- 上传 - 上传oss / 创建file
- 上传 - 上传 oss / 创建 file
- 加载 - loadDataTable
- 编辑 - editDataTable
- 持久化 - flushDataTable
- 导出 - url
- 删除 - delete
- 操作日志 - log
- editDataTable params 列表
### 加载表格数据 loadDataTable - 查询
```json
{
"fileId": 1,
"where": [
{
"field": {
"index": 1,
"name": "产品名称"
},
"in": ["a", "b"],
"ex": ["c", "d"],
"sort": ["a", "asc"]
}
]
}
```
### 编辑表格 editDataTable
```json
{
"field": {
"index": 1,
"name": "产品名称"
},
"operation": {
"desc": ["拆分", "按字符数"],
"code": "split_by_char_number"
},
"params": []
}
```
精简
```json
{
"field": "产品名称",
"desc": ["拆分", "按字符数"],
"operationCode": "split_by_char_number",
"params": []
}
```
`params 列表`
### 数据展示
```json
{
"code": 0,
"data": {
"dataFields": [
{
"index": 1,
"name": "产品名称",
"type": "string"
},
{
"index": 2,
"name": "产品数量",
"type": "int"
}
],
"dataRows": [
["素面", 200],
["冻豆腐", 400],
["冻豆腐1", 300],
["冻豆2", "A"]
],
"total": 100,
"pageNumber": 1,
"inValidCells": [
{
"x": 1,
"y": 3,
"error": "不是一个有效的数值"
}
]
},
"msg": "ok"
}
```
## 表关联关系
- [x] 可追加数据的表列表 /tables/search-appended-list
- [x] 校验文件列表 /files/search-verified-file
- [x] 匹配方案列表 /mapping-rule-config/search
- [x] 匹配方案主表 /mapping-rule-config/prepare //主表 校验表 主表字段 校验文件表字段
- [x] 匹配方案添加 /mapping-rule-config/
- [x] 匹配方案删除 /mapping-rule-config/:id
- [x] 追加数据到表格 /append-data-to-table // 验证是否追加过
- [ ] 取消校验中的文件 /cancel-verifying-file //
- [x] 表结构更新 /tables/update-table-struct
- [x] 表结构添加 /tables/add-table-struct
- [x] 分表列表 /tables/search
- [x] 表复制 /tables/copy-data-table
- [x] 表删除 /tables/:id // 若是删除主表,需级联删除关联的分表,删除内容包括表数据及表结构;? 分表的副表是否要删除
- [x] 表详情 /tables/:id // 表结构
- [x] 表更新 /tables/:id // 表结构、分表才可以编辑
- [x] 日志搜索 /log/search
- [x] 校验步骤日志 /log/verified-step-Log
## 数据预览
- [x] 表数据预览(格式) /table/preview
- [x] 表数据自定义查询 /table/preview where conditions 升序、降序 包含、不包含
- [x] 表数据字段可选值搜索 /table/field-optional 文本匹配
- [x] 表数据更新、添加、删除 /table/row-data-mutation
- [x] 表数据导出 /table/export-table
## 数据验证
- [x] 文件验证 /data/edit-data-table
## 底层字库接口
- [x] 数据预览 1
- [x] 表格编辑 1
- [x] 保存校验文件 (文件地址) 1
- [x] 生成主表 1
- [x] 表复制 (副表)1
- [x] 追加数据 (主表、副表)
- [ ] 表删除 (主表、副表)~~、分表~~
- [x] 表拆分 1
- [x] 更新表结构(分表)1
- [x] 编辑、添加、删除表数据(副表) 1
- [ ] 取消校验
## 定时作业
- [x] 隔天清理校验中的文件
- [x] 隔天清理 public 临时文件
## 表数据导出
- [ ] 加锁,只允许当前用户同时只能发起一次导出命令 ,3min 过期
- [ ] 单次拉取数量 MR
- [ ] 100W ..
- [ ] 50W 120s 读取数据库:30s 保存文件:10s 下载:30M/500K=60S;RAR 压缩 24M/500k=50S
- [ ] 20W ..
- [ ] 10W ..
- [ ] 保存单个文件、压缩 | 保存多个文件、压缩
## 讨论事项
- [ ] 校验动作,参数模型讨论
- [ ] 校验日志错误(标红)
- [ ] 校验完毕应答实体,类型修改即使错误,也要返回修改完毕的表
## 参数说明
### 通用格式
```json
{
"objectId": 1,
"processFields": [],
"action": "xx",
"params": {}
}
```
processFields:操作字段
### 常规
1. 删除列
```json
{
"action": "remove-column",
"params": {}
}
```
2. 复制列
```json
{
"action": "copy-column",
"params": {}
}
```
3. 重命名
```json
{
"action": "rename-column",
"params": {
"newColumnName": "新的列名称"
}
}
```
4. 替换值
```json
{
"action": "replace-column",
"params": {
"replaceMethod": "replace",
"searchValue": "搜索值",
"replaceValue": "替换值"
}
}
```
参数说明
```
replaceMethod: 替换方法(1.replace:替换值 2.add-prefix:添加前缀 3.add-postfix:添加后缀 4.remove-prefix:去除前缀 5.remove-postfix:去除后缀 6.remove-chars:去除固定字符 7.clean:清除)
searchValue: 搜索值-replace,remove-prefix,remove-postfix,remove-chars参数
replaceValue: 替换值-replace,add-prefix,add-postfix参数
```
### 格式 formatMethod
1. 大写
```json
{
"action": "format-column",
"params": {
"formatMethod": "upper"
}
}
```
参数说明
```
formatMethod: 格式化方法(1.upper:大写2.lower:小写3.capitalize:首字母大写4.strip:修整)
```
2. 小写
```json
{
"action": "format-column",
"params": {
"formatMethod": "lower"
}
}
```
3. 首字母大写
```json
{
"action": "format-column",
"params": {
"formatMethod": "capitalize"
}
}
```
4. 清除
```json
{
"action": "replace-column",
"params": {
"replaceMethod": "clean"
}
}
```
5. 修整
```json
{
"action": "format-column",
"params": {
"formatMethod": "strip"
}
}
```
6. 添加前缀
```json
{
"action": "replace-column",
"params": {
"replaceMethod": "add-prefix",
"replaceValue": "前缀值"
}
}
```
7. 添加后缀
```json
{
"action": "replace-column",
"params": {
"replaceMethod": "add-postfix",
"replaceValue": "后缀值"
}
}
```
8. 去除前缀
```json
{
"action": "replace-column",
"params": {
"replaceMethod": "remove-prefix",
"searchValue": "前缀值"
}
}
```
9. 去除后最
```json
{
"action": "replace-column",
"params": {
"replaceMethod": "remove-postfix",
"searchValue": "后缀值"
}
}
```
10. 去除固定字符
```json
{
"action": "replace-column",
"params": {
"replaceMethod": "remove-chars",
"searchValue": "字符"
}
}
```
### 拆分 split-column
1. 按分隔符
```json
{
"action": "split-column",
"params": {
"splitMethod": "separator",
"separator": "|",
"splitDirection": "left",
"splitCount": "1"
}
}
```
参数说明
```
separator: 分割符号 ‘|’
splitDirection: 拆分方向(1.left:从左边 2.right:从右边)
splitCount: 拆分次数
```
2. 按字符数
```json
{
"action": "split-column",
"params": {
"splitMethod": "char-length",
"splitDirection": "left",
"charLength": "10",
"splitCount": "1"
}
}
```
参数说明
```
charLength: 字符长度-char-length专属参数
splitDirection: 拆分方向(1.left:从左边 2.right:从右边)
splitCount: 拆分次数 (拆分策略是重复时:值0或者非1整数)
```
### 提取 extract-column
1. 按日期
```json
{
"action": "extract-column",
"params": {
"extractMethod": "by-date"
}
}
```
参数说明
```
extractMethod: 提取方法(1.by-date:按日期 2.by-number:按数值)
```
2. 按数值 action
```json
{
"action": "extract-column",
"params": {
"extractMethod": "by-number"
}
}
```
### 修改字段类型
```json
{
"action": "convert-column-type",
"params": {
"convertType": "STRING"
}
}
```
参数说明
```
convertType:转换类型 STRING 数值: INT 小数: FLOAT 日期: DATE 时间: DATETIME
```
... ...
POSTGRESQL_DB_NAME = allied_creation_dev
POSTGRESQL_DB_NAME = allied_creation_test
POSTGRESQL_HOST = 114.55.200.59
POSTGRESQL_PORT = 31543
POSTGRESQL_USER = postgres
... ... @@ -9,3 +9,12 @@ HTTP_PORT = 8081
ENABLE_KAFKA_LOG11 = true
HTTPS_PORT = 8143
ALLIED_CREATION_USER_HOST = http://allied-creation-user-dev.fjmaimaimai.com
# AUTH_SERVER_HOST = http://127.0.0.1:8081
BYTE_CORE_HOST = http://47.97.5.102:8303
METADATA_BASTION_HOST = http://106.75.231.90:9999
STARROCKS_DB_NAME = character_library
STARROCKS_USER = root
STARROCKS_PASSWORD = eagle1010
STARROCKS_HOST = 220.250.41.79
STARROCKS_PORT = 9030
... ...
/*1.初始化索引*/
/*files*/
CREATE INDEX IF NOT EXISTS idx_files_company_id_file_type ON metadata.files USING btree((context->>'companyId'),file_type);
CREATE INDEX IF NOT EXISTS idx_files_source_file_id ON metadata.files USING btree(source_file_id);
/*tables*/
CREATE INDEX IF NOT EXISTS idx_tables_company_id_table_type ON metadata.tables USING btree((context->>'companyId'),table_type);
CREATE INDEX IF NOT EXISTS idx_tables_parent_id ON metadata.tables USING btree(parent_id);
/*logs*/
CREATE INDEX IF NOT EXISTS idx_logs_company_id_log_type_source_id ON metadata.logs USING btree((context->>'companyId'),log_type,source_id);
CREATE INDEX IF NOT EXISTS idx_logs_company_id_object_name ON metadata.logs USING btree((context->>'companyId'),object_name);
CREATE INDEX IF NOT EXISTS idx_logs_company_id_object_type ON metadata.logs USING btree((context->>'companyId'),object_type);
CREATE INDEX IF NOT EXISTS idx_logs_company_id_operation_type ON metadata.logs USING btree((context->>'companyId'),operation_type);
CREATE INDEX IF NOT EXISTS idx_logs_company_id_content ON metadata.logs USING btree((context->>'companyId'),content);
CREATE INDEX IF NOT EXISTS idx_logs_company_id_operator_name ON metadata.logs USING btree((context->>'companyId'),operator_name);
CREATE INDEX IF NOT EXISTS idx_logs_company_id_created_at ON metadata.logs USING btree((context->>'companyId'),created_at);
/*mapping_rules*/
CREATE INDEX IF NOT EXISTS idx_mapping_rules_company_id_table_id_file_id ON metadata.mapping_rules USING btree((context->>'companyId'),table_id,file_id);
\ No newline at end of file
... ...
... ... @@ -2,7 +2,7 @@ apiVersion: v1
kind: Service
metadata:
name: character-library-metadata-bastion
namespace: <replace-your-k8s-namespace>
namespace: mmm-suplus-dev
labels:
k8s-app: character-library-metadata-bastion
spec:
... ... @@ -17,7 +17,7 @@ apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: character-library-metadata-bastion
namespace: <replace-your-k8s-namespace>
namespace: mmm-suplus-dev
labels:
k8s-app: character-library-metadata-bastion
spec:
... ... @@ -52,12 +52,69 @@ spec:
- mountPath: /opt/logs
name: accesslogs
env:
- name: POSTGRESQL_DB_NAME
valueFrom:
configMapKeyRef:
name: suplus-config
key: postgresqlalliedcreation.dbname
- name: POSTGRESQL_USER
valueFrom:
configMapKeyRef:
name: suplus-config
key: postgresql.user
- name: POSTGRESQL_PASSWORD
valueFrom:
configMapKeyRef:
name: suplus-config
key: postgresql.password
- name: POSTGRESQL_HOST
valueFrom:
configMapKeyRef:
name: suplus-config
key: postgresql.host
- name: POSTGRESQL_PORT
valueFrom:
configMapKeyRef:
name: suplus-config
key: postgresql.port
- name: REDIS_HOST
valueFrom:
configMapKeyRef:
name: suplus-config
key: redis.ip
- name: REDIS_PORT
valueFrom:
configMapKeyRef:
name: suplus-config
key: redis.port
- name: REDIS_AUTH
value: ""
- name: LOG_LEVEL
value: "debug"
- name: ERROR_BASE_CODE
value: "1"
- name: ERROR_BASE_CODE_MULTIPLE
value: "1000"
value: "2000"
- name: ENABLE_KAFKA_LOG
value: "true"
- name: HTTP_PORT
value: "8082"
- name: SERVICE_ENV
value: "dev"
- name: METADATA_BASTION_HOST
value: "http://character-library-metadata-bastion-dev.fjmaimaimai.com"
- name: BYTE_CORE_HOST
value: "http://47.97.5.102:8303"
- name: STARROCKS_HOST
value: "220.250.41.79"
- name: STARROCKS_PORT
value: "9030"
- name: STARROCKS_DB_NAME
value: "character_library_standard"
- name: STARROCKS_USER
value: "root"
- name: STARROCKS_PASSWORD
value: "eagle1010"
volumes:
- name: accesslogs
emptyDir: {}
\ No newline at end of file
... ...
#!/bin/bash
export PATH=/root/local/bin:$PATH
kubectl -n <replace-your-k8s-namespace> get pods | grep -q character-library-metadata-bastion
kubectl -n mmm-suplus-dev get pods | grep -q character-library-metadata-bastion
if [ "$?" == "1" ];then
kubectl create -f /tmp/dev/character-library-metadata-bastion/character-library-metadata-bastion.yaml --record
kubectl -n <replace-your-k8s-namespace> get svc | grep -q character-library-metadata-bastion
kubectl -n mmm-suplus-dev get svc | grep -q character-library-metadata-bastion
if [ "$?" == "0" ];then
echo "character-library-metadata-bastion service install success!"
else
echo "character-library-metadata-bastion service install fail!"
fi
kubectl -n <replace-your-k8s-namespace> get pods | grep -q character-library-metadata-bastion
kubectl -n mmm-suplus-dev get pods | grep -q character-library-metadata-bastion
if [ "$?" == "0" ];then
echo "character-library-metadata-bastion deployment install success!"
else
... ... @@ -17,24 +17,24 @@ if [ "$?" == "1" ];then
fi
else
kubectl delete -f /tmp/dev/character-library-metadata-bastion/character-library-metadata-bastion.yaml
kubectl -n <replace-your-k8s-namespace> get svc | grep -q character-library-metadata-bastion
kubectl -n mmm-suplus-dev get svc | grep -q character-library-metadata-bastion
while [ "$?" == "0" ]
do
kubectl -n <replace-your-k8s-namespace> get svc | grep -q character-library-metadata-bastion
kubectl -n mmm-suplus-dev get svc | grep -q character-library-metadata-bastion
done
kubectl -n <replace-your-k8s-namespace> get pods | grep -q character-library-metadata-bastion
kubectl -n mmm-suplus-dev get pods | grep -q character-library-metadata-bastion
while [ "$?" == "0" ]
do
kubectl -n <replace-your-k8s-namespace> get pods | grep -q character-library-metadata-bastion
kubectl -n mmm-suplus-dev get pods | grep -q character-library-metadata-bastion
done
kubectl create -f /tmp/dev/character-library-metadata-bastion/character-library-metadata-bastion.yaml --record
kubectl -n <replace-your-k8s-namespace> get svc | grep -q character-library-metadata-bastion
kubectl -n mmm-suplus-dev get svc | grep -q character-library-metadata-bastion
if [ "$?" == "0" ];then
echo "character-library-metadata-bastion service update success!"
else
echo "character-library-metadata-bastion service update fail!"
fi
kubectl -n <replace-your-k8s-namespace> get pods | grep -q character-library-metadata-bastion
kubectl -n mmm-suplus-dev get pods | grep -q character-library-metadata-bastion
if [ "$?" == "0" ];then
echo "character-library-metadata-bastion deployment update success!"
else
... ...
apiVersion: v1
kind: Service
metadata:
name: character-library-metadata-bastion
namespace: mmm-suplus-test
labels:
k8s-app: character-library-metadata-bastion
spec:
ports:
- name: "http"
port: 80
targetPort: 8082
selector:
k8s-app: character-library-metadata-bastion
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: character-library-metadata-bastion
namespace: mmm-suplus-test
labels:
k8s-app: character-library-metadata-bastion
spec:
replicas: 1
template:
metadata:
labels:
k8s-app: character-library-metadata-bastion
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- preference: {}
weight: 100
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- cn-hangzhou.i-bp1djh1xn7taumbue1ze
- cn-hangzhou.i-bp1djh1xn7taumbue1zd
- cn-hangzhou.i-bp1euf5u1ph9kbhtndhb
- cn-hangzhou.i-bp1hyp5oips9cdwxxgxy
containers:
- name: character-library-metadata-bastion
image: 192.168.0.243:5000/mmm/character-library-metadata-bastion:dev
imagePullPolicy: Always
ports:
- containerPort: 8082
volumeMounts:
- mountPath: /opt/logs
name: accesslogs
env:
- name: POSTGRESQL_DB_NAME
valueFrom:
configMapKeyRef:
name: suplus-config
key: postgresqlalliedcreation.dbname
- name: POSTGRESQL_USER
valueFrom:
configMapKeyRef:
name: suplus-config
key: postgresql.user
- name: POSTGRESQL_PASSWORD
valueFrom:
configMapKeyRef:
name: suplus-config
key: postgresql.password
- name: POSTGRESQL_HOST
valueFrom:
configMapKeyRef:
name: suplus-config
key: postgresql.host
- name: POSTGRESQL_PORT
valueFrom:
configMapKeyRef:
name: suplus-config
key: postgresql.port
- name: REDIS_HOST
valueFrom:
configMapKeyRef:
name: suplus-config
key: redis.ip
- name: REDIS_PORT
valueFrom:
configMapKeyRef:
name: suplus-config
key: redis.port
- name: REDIS_AUTH
value: ""
- name: LOG_LEVEL
value: "debug"
- name: ERROR_BASE_CODE
value: "1"
- name: ERROR_BASE_CODE_MULTIPLE
value: "2000"
- name: ENABLE_KAFKA_LOG
value: "false"
- name: HTTP_PORT
value: "8082"
- name: SERVICE_ENV
value: "test"
- name: METADATA_BASTION_HOST
value: "http://character-library-metadata-bastion-test.fjmaimaimai.com"
- name: BYTE_CORE_HOST
value: "http://47.97.5.102:8303"
- name: STARROCKS_HOST
value: "220.250.41.79"
- name: STARROCKS_PORT
value: "9030"
- name: STARROCKS_DB_NAME
value: "character_library"
- name: STARROCKS_USER
value: "root"
- name: STARROCKS_PASSWORD
value: "eagle1010"
volumes:
- name: accesslogs
emptyDir: {}
\ No newline at end of file
... ...
#!/bin/bash
export PATH=/root/local/bin:$PATH
kubectl -n mmm-suplus-test get pods | grep -q character-library-metadata-bastion
if [ "$?" == "1" ];then
kubectl create -f /tmp/test/character-library-metadata-bastion/character-library-metadata-bastion.yaml --record
kubectl -n mmm-suplus-test get svc | grep -q character-library-metadata-bastion
if [ "$?" == "0" ];then
echo "character-library-metadata-bastion service install success!"
else
echo "character-library-metadata-bastion service install fail!"
fi
kubectl -n mmm-suplus-test get pods | grep -q character-library-metadata-bastion
if [ "$?" == "0" ];then
echo "character-library-metadata-bastion deployment install success!"
else
echo "character-library-metadata-bastion deployment install fail!"
fi
else
kubectl delete -f /tmp/test/character-library-metadata-bastion/character-library-metadata-bastion.yaml
kubectl -n mmm-suplus-test get svc | grep -q character-library-metadata-bastion
while [ "$?" == "0" ]
do
kubectl -n mmm-suplus-test get svc | grep -q character-library-metadata-bastion
done
kubectl -n mmm-suplus-test get pods | grep -q character-library-metadata-bastion
while [ "$?" == "0" ]
do
kubectl -n mmm-suplus-test get pods | grep -q character-library-metadata-bastion
done
kubectl create -f /tmp/test/character-library-metadata-bastion/character-library-metadata-bastion.yaml --record
kubectl -n mmm-suplus-test get svc | grep -q character-library-metadata-bastion
if [ "$?" == "0" ];then
echo "character-library-metadata-bastion service update success!"
else
echo "character-library-metadata-bastion service update fail!"
fi
kubectl -n mmm-suplus-test get pods | grep -q character-library-metadata-bastion
if [ "$?" == "0" ];then
echo "character-library-metadata-bastion deployment update success!"
else
echo "character-library-metadata-bastion deployment update fail!"
fi
fi
\ No newline at end of file
... ...
version: v1
kind: HttpApi
metadata:
service: log
path: /logs
endpoints:
- method: createLog
route:
post: /
- method: updateLog
route:
put: /{logId}
- method: getLog
route:
get: /{logId}
- method: removeLog
route:
delete: /{logId}
- method: listLog
route:
get: /
params:
- name: offset
- name: limit
- method: searchLog
route:
post: /search
... ...
version: v1
kind: HttpApi
metadata:
service: mappingRule
path: /mapping-rules
endpoints:
- method: createMappingRule
route:
post: /
- method: updateMappingRule
route:
put: /{mappingRuleId}
- method: getMappingRule
route:
get: /{mappingRuleId}
- method: removeMappingRule
route:
delete: /{mappingRuleId}
- method: listMappingRule
route:
get: /
params:
- name: offset
- name: limit
- method: search
route:
post: /search
- method: prepare
route:
post: /prepare
\ No newline at end of file
... ...
version: v1
kind: HttpApi
metadata:
service: querySet
path: /query-sets
endpoints:
- method: createQuerySet
route:
post: /
- method: updateQuerySet
route:
put: /{Id}
- method: getQuerySet
route:
get: /{Id}
- method: removeQuerySet
route:
delete: /{Id}
- method: listQuerySet
route:
get: /
params:
- name: offset
- name: limit
- method: changeStatus
route:
post: /change-status
- method: copy
route:
post: /copy
- method: dependencyGraph
route:
post: /dependency-ggraph
- method: move
route:
post: /move
- method: rename
route:
post: /rename
- method: searchQuerySet
route:
post: /search
... ...
version: v1
kind: HttpApi
metadata:
service: table
path: /tables
endpoints:
- method: createTable
route:
post: /
- method: updateTable
route:
put: /{tableId}
- method: getTable
route:
get: /{tableId}
- method: removeTable
route:
delete: /{tableId}
- method: listTable
route:
get: /
params:
- name: offset
- name: limit
- method: splitDataTable
route:
post: /split-data-table
- method: batchEditSubTable
route:
post: /batch-edit-sub-table
- method: copyDataTable
route:
post: /copy-data-table
- method: search
route:
post: /search
\ No newline at end of file
... ...
version: v1
kind: Attribute
metadata:
name: context
description: 扩展
type:
array: string
... ...
version: v1
kind: Attribute
metadata:
name: mainTableFields
description: 主表列
type:
primitive: string
... ...
version: v1
kind: Attribute
metadata:
name: mappingFields
description: 校验文件列
type:
primitive: string
... ...
version: v1
kind: Attribute
metadata:
name: mappingRuleId
description: 匹配规则ID
type:
primitive: int
... ...
version: v1
kind: Attribute
metadata:
name: verifiedFileFields
description: 校验文件列
type:
primitive: string
... ...
version: v1
kind: Attribute
metadata:
name: flag
description: 标识 1.group 2.query-set
type:
primitive: string
... ...
version: v1
kind: Attribute
metadata:
name: name
description: 名称
type:
primitive: string
... ...
version: v1
kind: Attribute
metadata:
name: pinName
description: 拼音(排序使用)
type:
primitive: string
... ...
version: v1
kind: Attribute
metadata:
name: queryComponents
description: 查询组件
type:
array: string
... ...
version: v1
kind: Attribute
metadata:
name: querySetId
description: 查询集合ID
type:
primitive: int
... ...
version: v1
kind: Attribute
metadata:
name: querySetInfo
description: 查询集合信息
type:
primitive: string
... ...
version: v1
kind: Attribute
metadata:
name: status
description: 状态 1:启用 2:关闭 (子过程默认启用)
type:
primitive: int
... ...
version: v1
kind: Attribute
metadata:
name: type
description: schema:方案 sub-process:子过程
type:
primitive: string
... ...
version: v1
kind: Schema
metadata:
name: mappingRule
description: 匹配规则配置
attributes:
- ref: mappingRuleId
required: true
- ref: name
required: true
- ref: tableId
required: true
- ref: fileId
required: true
- ref: mainTableFields
required: true
- ref: verifiedFileFields
required: true
- ref: mappingFields
required: true
- ref: createdAt
required: true
- ref: updatedAt
required: true
- ref: deletedAt
required: true
... ...
version: v1
kind: Schema
metadata:
name: querySet
description: 查询集合
attributes:
- ref: querySetId
required: true
- ref: type
required: true
- ref: flag
required: true
- ref: name
required: true
- ref: pinName
required: true
- ref: parentId
required: true
- ref: status
required: true
- ref: querySetInfo
required: true
- ref: queryComponents
required: true
- ref: sort
required: true
- ref: createdAt
required: true
- ref: updatedAt
required: true
- ref: deletedAt
required: true
- ref: context
required: true
... ...
version: v1
kind: Method
metadata:
name: createLog
type: command
description: 创建日志服务
payload:
- ref: objectName
required: true
- ref: objectType
required: true
- ref: operationType
required: true
- ref: content
required: true
- ref: operatorName
required: true
result:
- name: log
type:
schema: log
required: true
... ...
version: v1
kind: Method
metadata:
name: getLog
type: query
description: 返回日志服务
payload:
- ref: logId
required: true
result:
- name: log
type:
schema: log
required: true
... ...
version: v1
kind: Method
metadata:
name: listLog
type: query
description: 返回日志服务列表
payload:
- ref: offset
required: true
- ref: limit
required: true
result:
- ref: count
required: true
- name: logs
type:
array: log
required: true
... ...
version: v1
kind: Method
metadata:
name: removeLog
type: command
description: 移除日志服务
payload:
- ref: logId
required: true
result:
- name: log
type:
schema: log
required: true
... ...
version: v1
kind: Method
metadata:
name: searchLog
type: command
description: 搜索日志
payload:
- ref: content
required: false
result:
- name: log
type:
schema: log
required: true
... ...
version: v1
kind: Method
metadata:
name: updateLog
type: command
description: 更新日志服务
payload:
- ref: logId
required: true
result:
- name: log
type:
schema: log
required: true
... ...
version: v1
kind: Service
metadata:
name: log
description: 日志服务
... ...
version: v1
kind: Method
metadata:
name: createMappingRule
type: command
description: 创建匹配规则服务
payload:
- ref: name
required: true
result:
- name: mappingRule
type:
schema: mappingRule
required: true
... ...
version: v1
kind: Method
metadata:
name: getMappingRule
type: query
description: 返回匹配规则服务
payload:
- ref: mappingRuleId
required: true
result:
- name: mappingRule
type:
schema: mappingRule
required: true
... ...
version: v1
kind: Method
metadata:
name: listMappingRule
type: query
description: 返回匹配规则服务列表
payload:
- ref: offset
required: true
- ref: limit
required: true
result:
- ref: count
required: true
- name: mappingRules
type:
array: mappingRule
required: true
... ...
version: v1
kind: Method
metadata:
name: prepare
type: command
description: 匹配规则预准备(新建规则)
payload:
- ref: tableId
required: true
- ref: fileId
required: true
result:
- name: mappingRule
type:
schema: mappingRule
required: true
... ...
version: v1
kind: Method
metadata:
name: removeMappingRule
type: command
description: 移除匹配规则服务
payload:
- ref: mappingRuleId
required: true
result:
- name: mappingRule
type:
schema: mappingRule
required: true
... ...
version: v1
kind: Method
metadata:
name: search
type: command
description: 搜索规则
payload:
- ref: name
required: false
result:
- name: mappingRule
type:
schema: mappingRule
required: true
... ...
version: v1
kind: Method
metadata:
name: updateMappingRule
type: command
description: 更新匹配规则服务
payload:
- ref: mappingRuleId
required: true
result:
- name: mappingRule
type:
schema: mappingRule
required: true
... ...
version: v1
kind: Service
metadata:
name: mappingRule
description: 匹配规则服务
... ...
version: v1
kind: Method
metadata:
name: changeStatus
type: command
description: 修改状态
payload:
- ref: querySetId
required: true
- ref: status
required: true
result:
- name: querySet
type:
schema: querySet
required: true
... ...
version: v1
kind: Method
metadata:
name: copy
type: command
description: 移动
payload:
- ref: flag
required: true
- ref: parentId
required: true
- ref: name
required: true
- ref: querySetId
required: true
result:
- name: querySet
type:
schema: querySet
required: true
... ...
version: v1
kind: Method
metadata:
name: createQuerySet
type: command
description: 创建查询集合服务
payload:
- ref: type
required: true
- ref: flag
required: true
- ref: name
required: true
- ref: parentId
required: false
result:
- name: querySet
type:
schema: querySet
required: true
... ...
version: v1
kind: Method
metadata:
name: dependencyGraph
type: query
description: 依赖关系图
payload:
- ref: querySetId
required: true
result:
- name: querySet
type:
schema: querySet
required: true
... ...
version: v1
kind: Method
metadata:
name: getQuerySet
type: query
description: 返回查询集合服务
payload:
- ref: querySetId
required: true
result:
- name: querySet
type:
schema: querySet
required: true
... ...
version: v1
kind: Method
metadata:
name: listQuerySet
type: query
description: 返回查询集合服务列表
payload:
- ref: offset
required: true
- ref: limit
required: true
result:
- ref: count
required: true
- name: querySets
type:
array: querySet
required: true
... ...
version: v1
kind: Method
metadata:
name: move
type: command
description: 移动
payload:
- ref: querySetId
required: true
- ref: parentId
required: true
result:
- name: querySet
type:
schema: querySet
required: true
... ...
version: v1
kind: Method
metadata:
name: removeQuerySet
type: command
description: 移除查询集合服务
payload:
- ref: querySetId
required: true
result:
- name: querySet
type:
schema: querySet
required: true
... ...
version: v1
kind: Method
metadata:
name: rename
type: command
description: 重命名
payload:
- ref: querySetId
required: true
- ref: name
required: true
result:
- name: querySet
type:
schema: querySet
required: true
... ...
version: v1
kind: Method
metadata:
name: searchQuerySet
type: query
description: 返回查询集合服务列表
payload:
- ref: offset
required: true
- ref: limit
required: true
result:
- ref: count
required: true
- name: querySets
type:
array: querySet
required: true
... ...
version: v1
kind: Method
metadata:
name: updateQuerySet
type: command
description: 更新查询集合服务
payload:
- ref: querySetId
required: true
result:
- name: querySet
type:
schema: querySet
required: true
... ...
version: v1
kind: Service
metadata:
name: querySet
description: 查询集合服务
... ...
version: v1
kind: Method
metadata:
name: batchEditSubTable
type: command
description: 批量编辑分表
payload:
- ref: tableId
required: true
result:
- name: table
type:
schema: table
required: true
... ...
version: v1
kind: Method
metadata:
name: copyDataTable
type: command
description: 表复制
payload:
- ref: tableId
required: true
result:
- name: table
type:
schema: table
required: true
... ...
version: v1
kind: Method
metadata:
name: createTable
type: command
description: 创建表服务
payload:
- ref: name
required: true
result:
- name: table
type:
schema: table
required: true
... ...
version: v1
kind: Method
metadata:
name: getTable
type: query
description: 返回表服务
payload:
- ref: tableId
required: true
result:
- name: table
type:
schema: table
required: true
... ...
version: v1
kind: Method
metadata:
name: listTable
type: query
description: 返回表服务列表
payload:
- ref: offset
required: true
- ref: limit
required: true
result:
- ref: count
required: true
- name: tables
type:
array: table
required: true
... ...
version: v1
kind: Method
metadata:
name: removeTable
type: command
description: 移除表服务
payload:
- ref: tableId
required: true
result:
- name: table
type:
schema: table
required: true
... ...
version: v1
kind: Method
metadata:
name: search
type: query
description: 表搜索
payload:
- ref: tableType
required: true
result:
- name: table
type:
schema: table
required: true
... ...
version: v1
kind: Method
metadata:
name: splitDataTable
type: command
description: 表拆分
payload:
- ref: tableId
required: true
result:
- name: table
type:
schema: table
required: true
... ...
version: v1
kind: Method
metadata:
name: updateTable
type: command
description: 更新表服务
payload:
- ref: tableId
required: true
result:
- name: table
type:
schema: table
required: true
... ...
version: v1
kind: Service
metadata:
name: table
description: 表服务
... ...
... ... @@ -3,32 +3,45 @@ module gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion
go 1.16
require (
github.com/Shopify/sarama v1.25.0
github.com/ajg/form v1.5.1 // indirect
github.com/beego/beego/v2 v2.0.1
github.com/bwmarrin/snowflake v0.3.0
github.com/dgrijalva/jwt-go v3.2.0+incompatible
github.com/extrame/xls v0.0.1
github.com/fasthttp-contrib/websocket v0.0.0-20160511215533-1f3b11f56072 // indirect
github.com/fatih/structs v1.1.0 // indirect
github.com/gavv/httpexpect v2.0.0+incompatible
github.com/go-pg/pg/v10 v10.9.0
github.com/go-redis/redis v6.15.9+incompatible // indirect
github.com/go-redis/redis/v7 v7.4.1 // indirect
github.com/golang/snappy v0.0.3 // indirect
github.com/google/go-cmp v0.5.6 // indirect
github.com/go-pg/pg/v10 v10.10.6
github.com/go-redis/redis v6.15.9+incompatible
github.com/google/go-querystring v1.1.0 // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/google/gofuzz v1.2.0
github.com/google/uuid v1.3.0
github.com/gookit/event v1.0.6
github.com/imkira/go-interpol v1.1.0 // indirect
github.com/linmadan/egglib-go v0.0.0-20210313060205-8b5e456b11f7
github.com/moul/http2curl v1.0.0 // indirect
github.com/onsi/ginkgo v1.15.2
github.com/onsi/gomega v1.11.0
github.com/onsi/ginkgo v1.16.5
github.com/onsi/gomega v1.18.1
github.com/patrickmn/go-cache v2.1.0+incompatible
github.com/sergi/go-diff v1.2.0 // indirect
github.com/shopspring/decimal v1.3.1
github.com/smartystreets/goconvey v1.7.2 // indirect
github.com/stretchr/testify v1.7.1
github.com/valyala/fasthttp v1.38.0 // indirect
github.com/xeipuuv/gojsonschema v1.2.0 // indirect
github.com/xuri/excelize/v2 v2.6.0
github.com/yalp/jsonpath v0.0.0-20180802001716-5cc68e5049a0 // indirect
github.com/yudai/gojsondiff v1.0.0 // indirect
github.com/yudai/golcs v0.0.0-20170316035057-ecda9a501e82 // indirect
github.com/yudai/pp v2.0.1+incompatible // indirect
github.com/zeromicro/go-zero v1.3.4
golang.org/x/text v0.3.7
gorm.io/driver/mysql v1.3.6
gorm.io/driver/postgres v1.3.9
gorm.io/gorm v1.23.8
)
replace github.com/linmadan/egglib-go v0.0.0-20210313060205-8b5e456b11f7 => github.com/tiptok/egglib-go v0.0.0-20220421085958-9682d0ac42c1
replace (
github.com/extrame/xls v0.0.1 => github.com/tiptok/xls v1.0.1
github.com/linmadan/egglib-go v0.0.0-20210313060205-8b5e456b11f7 => github.com/tiptok/egglib-go v0.0.0-20220421085958-9682d0ac42c1
)
... ...
... ... @@ -3,18 +3,21 @@ package main
import (
"fmt"
"github.com/beego/beego/v2/server/web"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/crontab"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/constant"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/redis"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/log"
"time"
_ "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/constant"
_ "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/pg"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/pg"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/redis"
_ "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/redis"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/starrocks"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/log"
_ "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/log"
_ "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/port/beego"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/port/event"
)
const Version = "v1.2.0"
func main() {
defer func() {
if r := recover(); r != nil {
... ... @@ -24,8 +27,17 @@ func main() {
log.InitLogHook(constant.ENABLE_KAFKA_LOG, true)
redis.InitRedis()
time.Sleep(time.Second)
redis.InitZeroCoreRedis()
pg.Init()
if err := starrocks.Init(); err != nil {
log.Logger.Error(err.Error())
}
cron := crontab.NewCrontabService(nil)
cron.StartCrontabTask()
defer cron.StopCrontabTask()
event.Start()
log.Logger.Info("Service:" + constant.SERVICE_NAME)
log.Logger.Info("Version:" + Version)
log.Logger.Info("server start!")
web.Run()
log.Logger.Info("server stop!")
... ...
package crontab
import (
"context"
"fmt"
"github.com/beego/beego/v2/task"
pgTransaction "github.com/linmadan/egglib-go/transaction/pg"
"github.com/linmadan/egglib-go/utils/xtime"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/factory"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/dao"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/starrocks"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/log"
"io/fs"
"os"
"path/filepath"
"time"
)
type CrontabService struct {
}
func NewCrontabService(options map[string]interface{}) *CrontabService {
newCrontabService := &CrontabService{}
return newCrontabService
}
func (crontabService *CrontabService) initTask() {
autoRemoveExpiredTemporaryFile := task.NewTask("定时清理过期临时文件记录", "0 10 */1 * * *", AutoRemoveExpiredTemporaryFile)
task.AddTask("autoRemoveExpiredTemporaryFile", autoRemoveExpiredTemporaryFile)
autoRemovePublicDownloadFile := task.NewTask("定时清理缓存文件", "0 20 */1 * * *", AutoRemovePublicDownloadFile)
task.AddTask("autoRemovePublicDownloadFile", autoRemovePublicDownloadFile)
autoRemoveTemporaryTable := task.NewTask("定时清理临时表", "0 57 */1 * * *", AutoRemoveTemporaryTable)
task.AddTask("autoRemoveTemporaryTable", autoRemoveTemporaryTable)
}
func (crontabService *CrontabService) StartCrontabTask() {
crontabService.initTask()
task.StartTask()
log.Logger.Info("crontab start!")
}
func (crontabService *CrontabService) StopCrontabTask() {
task.StopTask()
log.Logger.Info("crontab stop!")
}
func AutoRemoveExpiredTemporaryFile(ctx context.Context) error {
defer func() {
if r := recover(); r != nil {
log.Logger.Error(fmt.Sprintf("%v", r), map[string]interface{}{"task": "定时清理过期临时文件记录"})
}
}()
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return err
}
if err := transactionContext.StartTransaction(); err != nil {
return err
}
defer func() {
if err != nil {
log.Logger.Error("【定时清理过期临时文件记录】 失败:" + err.Error())
}
transactionContext.RollbackTransaction()
}()
log.Logger.Debug("【定时清理过期临时文件记录】 启动")
end := xtime.New(time.Now()).BeginningOfDay().Add(-time.Hour * 6)
begin := end.AddDate(0, 0, -7)
fileRepository, _, _ := factory.FastPgFile(transactionContext, 0)
_, files, err := fileRepository.Find(map[string]interface{}{"limit": 100, "fileType": domain.TemporaryFile.ToString(), "updatedAtBegin": begin, "updatedAtEnd": end})
if err != nil {
return err
}
deleteFileService, _ := factory.CreateDeleteFileService(transactionContext)
if err = deleteFileService.DeleteFiles(&domain.Context{
OperatorName: "系统",
}, files...); err != nil {
return err
}
if err = transactionContext.CommitTransaction(); err != nil {
return err
}
return nil
}
func AutoRemovePublicDownloadFile(ctx context.Context) error {
var err error
defer func() {
if r := recover(); r != nil {
log.Logger.Error(fmt.Sprintf("%v", r), map[string]interface{}{"task": "定时清理缓存文件"})
}
}()
defer func() {
if err != nil {
log.Logger.Error("【定时清理缓存文件】 失败:" + err.Error())
}
}()
log.Logger.Debug("【定时清理缓存文件】 启动")
root := "public"
end := xtime.New(time.Now()).BeginningOfDay().AddDate(0, 0, -1)
err = filepath.Walk(root, func(path string, info fs.FileInfo, err error) error {
t := info.ModTime()
if !info.IsDir() && xtime.Before(t, end) {
err = os.Remove(path)
if err != nil {
return err
}
log.Logger.Info(fmt.Sprintf("[%v] 删除文件:%v 大小:%v 路径:%v 文件创建时间:%v", "系统", info.Name(), info.Size(), path, info.ModTime()))
return nil
}
return nil
})
if err != nil {
return err
}
return nil
}
func AutoRemoveTemporaryTable(ctx context.Context) error {
defer func() {
if r := recover(); r != nil {
log.Logger.Error(fmt.Sprintf("%v", r), map[string]interface{}{"task": "定时清理过期临时文件记录"})
}
}()
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return err
}
if err := transactionContext.StartTransaction(); err != nil {
return err
}
defer func() {
if err != nil {
log.Logger.Error("【定时清理临时表】 失败:" + err.Error())
}
transactionContext.RollbackTransaction()
}()
log.Logger.Debug("【定时清理临时表】 启动")
end := xtime.New(time.Now()).BeginningOfDay().Add(-time.Hour * 12)
begin := end.AddDate(0, 0, -7)
tableRepository, _, _ := factory.FastPgTable(transactionContext, 0)
_, tables, err := tableRepository.Find(map[string]interface{}{"beginTime": begin, "endTime": end, "tableTypes": []string{domain.TemporaryTable.ToString()}})
if err != nil {
return nil
}
for i, t := range tables {
if err = dao.TableDelete(transactionContext.(*pgTransaction.TransactionContext), t.TableId, domain.TemporaryTable); err != nil {
log.Logger.Error(err.Error())
return nil
}
log.Logger.Info(fmt.Sprintf("序号:%d 清理临时表 %v", i, t.SQLName))
if err = starrocks.DropView(starrocks.DB, t.SQLName); err != nil {
log.Logger.Error(err.Error())
return nil
}
}
//if err = dao.TableDeleteByTime(transactionContext.(*pgTransaction.TransactionContext), domain.TemporaryTable, begin, end); err != nil {
// return err
//}
if err = transactionContext.CommitTransaction(); err != nil {
return err
}
return nil
}
... ...
package command
import "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
type TableEventCommand struct {
EventTable *domain.EventTable
}
... ...
package service
import (
"github.com/linmadan/egglib-go/core/application"
pgTransaction "github.com/linmadan/egglib-go/transaction/pg"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/event/command"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/factory"
tablecommand "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/table/command"
tableservice "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/table/service"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/cache"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/domainService"
)
type TableEventService struct {
}
func (tableEventService *TableEventService) Handler(ctx *domain.Context, cmd *command.TableEventCommand) (interface{}, error) {
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
data := cmd.EventTable
tableId := 0
switch data.Type {
case domain.TableDataImportEvent, domain.TableDataEditEvent, domain.TableDeleteEvent:
tableId = data.Table.TableId
case domain.QuerySetUpdateEvent:
tableId = data.QuerySet.QuerySetInfo.BindTableId
}
if tableId == 0 {
return nil, nil
}
// tableId 相关联的
tableRepository, _, _ := factory.FastPgTable(transactionContext, 0)
_, tables, err := tableRepository.Find(map[string]interface{}{"context": data.Context, "tableTypesNotIn": []string{domain.TemporaryTable.ToString(), domain.ExcelTable.ToString()}})
if err != nil {
return nil, err
}
tableDependencyService, _ := domainService.NewTableDependencyService(transactionContext.(*pgTransaction.TransactionContext))
tableDependTree := tableDependencyService.TableDependTree(tables, tableId)
tree := tableDependTree.Tree
tableService := tableservice.NewTableService(nil)
for i := range tree {
cache.DefaultDataTableCacheService.DeleteDataTable(tree[i])
// fresh cache
tableService.TablePreview(data.Context, &tablecommand.TablePreviewCommand{
TableId: tree[i],
ObjectType: domain.ObjectMetaTable,
PageSize: 10000,
PageNumber: 0,
UseCache: true,
})
}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return nil, nil
}
func NewTableEventService(options map[string]interface{}) *TableEventService {
svr := &TableEventService{}
return svr
}
... ...
package factory
import (
"github.com/linmadan/egglib-go/core/application"
"github.com/linmadan/egglib-go/transaction/pg"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/domainService"
)
func FastLog(transactionContext application.TransactionContext, logType domain.LogType, sourceId int, logEntry domainService.Log) error {
logService, _ := domainService.NewPGLogService(transactionContext.(*pg.TransactionContext))
return logService.Log(logType, sourceId, logEntry)
}
func CreateLoadDataTableService(transactionContext application.TransactionContext) (domain.PreviewDataTableService, error) {
return domainService.NewLoadDataTableService(transactionContext.(*pg.TransactionContext))
}
func CreateEditDataTableService(transactionContext application.TransactionContext) (domain.EditDataTableService, error) {
return domainService.NewEditDataTableService(transactionContext.(*pg.TransactionContext))
}
func CreateFlushDataTableService(transactionContext application.TransactionContext) (domain.FlushDataTableService, error) {
return domainService.NewFlushDataTableService(transactionContext.(*pg.TransactionContext))
}
func CreateDeleteFileService(transactionContext application.TransactionContext) (domain.DeleteFileService, error) {
return domainService.NewDeleteFileService(transactionContext.(*pg.TransactionContext))
}
func CreateGenerateMainTableService(transactionContext application.TransactionContext) (domain.GenerateMainTableService, error) {
return domainService.NewGenerateMainTableService(transactionContext.(*pg.TransactionContext))
}
func CreateCopyDataTableService(transactionContext application.TransactionContext) (domain.CopyDataTableService, error) {
return domainService.NewCopyDataTableService(transactionContext.(*pg.TransactionContext))
}
func CreateDeleteDataTableService(transactionContext application.TransactionContext) (domain.DeleteDataTableService, error) {
return domainService.NewDeleteDataTableService(transactionContext.(*pg.TransactionContext))
}
func CreateUpdateTableStructService(transactionContext application.TransactionContext) (domain.UpdateTableStructService, error) {
return domainService.NewUpdateTableStructService(transactionContext.(*pg.TransactionContext))
}
func CreateAddTableStructService(transactionContext application.TransactionContext) (domain.AddTableStructService, error) {
return domainService.NewAddTableStructService(transactionContext.(*pg.TransactionContext))
}
func CreateAppendDataToTableService(transactionContext application.TransactionContext) (domain.AppendDataToTableService, error) {
return domainService.NewAppendDataToTableService(transactionContext.(*pg.TransactionContext))
}
func CreateTableEditDataService(transactionContext application.TransactionContext) (domain.TableEditDataService, error) {
return domainService.NewTableEditDataService(transactionContext.(*pg.TransactionContext))
}
// 字库核心
func CreateByteCoreService(transactionContext application.TransactionContext) (domain.ByteLibService, error) {
return domainService.ByteCore, nil
}
... ...
package factory
import (
"github.com/linmadan/egglib-go/core/application"
pgTransaction "github.com/linmadan/egglib-go/transaction/pg"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/domainService"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/starrocks"
)
func FastError(err error) error {
return application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
func FastDataTable(options starrocks.QueryOptions) (*domain.DataTable, error) {
var err error
// 待优化分批下载,压缩
var dataTable *domain.DataTable
dataTable, err = starrocks.Query(options, starrocks.WrapQueryFuncWithDB(starrocks.DB))
if err != nil {
return nil, err
}
dataTable.Total, err = starrocks.WrapQueryCountWithDB(options, starrocks.DB)()
if err != nil {
return nil, err
}
return dataTable, nil
}
func FastQuerySetServices(transactionContext application.TransactionContext) (*domainService.QuerySetService, error) {
return domainService.NewQuerySetService(transactionContext.(*pgTransaction.TransactionContext))
}
... ...
package factory
import (
"github.com/linmadan/egglib-go/core/application"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
)
// FastPgFile 快速返回文件对象
//
// transactionContext 事务
// id 对象唯一标识
func FastPgFile(transactionContext application.TransactionContext, id int) (domain.FileRepository, *domain.File, error) {
var rep domain.FileRepository
var mod *domain.File
var err error
if value, err := CreateFileRepository(map[string]interface{}{
"transactionContext": transactionContext,
}); err != nil {
return nil, nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
} else {
rep = value
}
if id > 0 {
if mod, err = rep.FindOne(map[string]interface{}{"fileId": id}); err != nil {
if err == domain.ErrorNotFound {
return nil, nil, application.ThrowError(application.RES_NO_FIND_ERROR, "该文件不存在")
}
return nil, nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
}
//if err = fastPgDataAuth(transactionContext, mod, options...); err != nil {
// return nil, nil, err
//}
return rep, mod, err
}
// FastPgTable 快速返回表格对象
//
// transactionContext 事务
// id 对象唯一标识
func FastPgTable(transactionContext application.TransactionContext, id int) (domain.TableRepository, *domain.Table, error) {
var rep domain.TableRepository
var mod *domain.Table
var err error
if value, err := CreateTableRepository(map[string]interface{}{
"transactionContext": transactionContext,
}); err != nil {
return nil, nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
} else {
rep = value
}
if id > 0 {
if mod, err = rep.FindOne(map[string]interface{}{"tableId": id}); err != nil {
if err == domain.ErrorNotFound {
return nil, nil, application.ThrowError(application.RES_NO_FIND_ERROR, "该表格不存在")
}
return nil, nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
}
return rep, mod, err
}
// FastPgLog 快速返回日志对象
//
// transactionContext 事务
// id 对象唯一标识
func FastPgLog(transactionContext application.TransactionContext, id int) (domain.LogRepository, *domain.Log, error) {
var rep domain.LogRepository
var mod *domain.Log
var err error
if value, err := CreateLogRepository(map[string]interface{}{
"transactionContext": transactionContext,
}); err != nil {
return nil, nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
} else {
rep = value
}
if id > 0 {
if mod, err = rep.FindOne(map[string]interface{}{"logId": id}); err != nil {
if err == domain.ErrorNotFound {
return nil, nil, application.ThrowError(application.RES_NO_FIND_ERROR, "该日志不存在")
}
return nil, nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
}
return rep, mod, err
}
// FastPgMappingRule 快速返回匹配规则对象
//
// transactionContext 事务
// id 对象唯一标识
func FastPgMappingRule(transactionContext application.TransactionContext, id int) (domain.MappingRuleRepository, *domain.MappingRule, error) {
var rep domain.MappingRuleRepository
var mod *domain.MappingRule
var err error
if value, err := CreateMappingRuleRepository(map[string]interface{}{
"transactionContext": transactionContext,
}); err != nil {
return nil, nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
} else {
rep = value
}
if id > 0 {
if mod, err = rep.FindOne(map[string]interface{}{"mappingRuleId": id}); err != nil {
if err == domain.ErrorNotFound {
return nil, nil, application.ThrowError(application.RES_NO_FIND_ERROR, "该匹配规则不存在")
}
return nil, nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
}
return rep, mod, err
}
// FastPgQuerySet 快速返回查询集合
//
// transactionContext 事务
// id 对象唯一标识
func FastPgQuerySet(transactionContext application.TransactionContext, id int) (domain.QuerySetRepository, *domain.QuerySet, error) {
var rep domain.QuerySetRepository
var mod *domain.QuerySet
var err error
if value, err := CreateQuerySetRepository(map[string]interface{}{
"transactionContext": transactionContext,
}); err != nil {
return nil, nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
} else {
rep = value
}
if id > 0 {
if mod, err = rep.FindOne(map[string]interface{}{"querySetId": id}); err != nil {
if err == domain.ErrorNotFound {
return nil, nil, application.ThrowError(application.RES_NO_FIND_ERROR, "该查询集合不存在")
}
return nil, nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
}
return rep, mod, err
}
... ...
... ... @@ -29,3 +29,19 @@ func CreateFileRepository(options map[string]interface{}) (domain.FileRepository
}
return repository.NewFileRepository(transactionContext)
}
func CreateMappingRuleRepository(options map[string]interface{}) (domain.MappingRuleRepository, error) {
var transactionContext *pg.TransactionContext
if value, ok := options["transactionContext"]; ok {
transactionContext = value.(*pg.TransactionContext)
}
return repository.NewMappingRuleRepository(transactionContext)
}
func CreateQuerySetRepository(options map[string]interface{}) (domain.QuerySetRepository, error) {
var transactionContext *pg.TransactionContext
if value, ok := options["transactionContext"]; ok {
transactionContext = value.(*pg.TransactionContext)
}
return repository.NewQuerySetRepository(transactionContext)
}
... ...
package command
import (
"fmt"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type AppendDataToTableCommand struct {
// 文件ID
FileId int `cname:"文件ID" json:"fileId" valid:"Required"`
// 文件ID
TableId int `cname:"表ID" json:"tableId" valid:"Required"`
// 校验文件列
MappingFields []*domain.MappingField `json:"mappingFields"`
}
func (cmd *AppendDataToTableCommand) Valid(validation *validation.Validation) {
}
func (cmd *AppendDataToTableCommand) ValidateCommand() error {
valid := validation.Validation{}
b, err := valid.Valid(cmd)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(cmd).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
package command
import (
"fmt"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type CancelVerifyingFileCommand struct {
// 文件ID
FileId int `cname:"文件ID" json:"fileId" valid:"Required"`
}
func (cmd *CancelVerifyingFileCommand) Valid(validation *validation.Validation) {
}
func (cmd *CancelVerifyingFileCommand) ValidateCommand() error {
valid := validation.Validation{}
b, err := valid.Valid(cmd)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(cmd).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
package command
import (
"fmt"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type CheckFileVerifyStatusCommand struct {
// 文件ID
FileId int `cname:"文件ID" json:"objectId" valid:"Required"`
}
func (cmd *CheckFileVerifyStatusCommand) Valid(validation *validation.Validation) {
}
func (cmd *CheckFileVerifyStatusCommand) ValidateCommand() error {
valid := validation.Validation{}
b, err := valid.Valid(cmd)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(cmd).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
... ... @@ -2,6 +2,8 @@ package command
import (
"fmt"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"path/filepath"
"reflect"
"strings"
... ... @@ -13,12 +15,16 @@ type CreateFileCommand struct {
Name string `cname:"名称" json:"name" valid:"Required"`
// 文件地址
Url string `cname:"文件地址" json:"url" valid:"Required"`
// 文件大小
// 文件大小 单位KB
FileSize int `cname:"文件大小" json:"fileSize" valid:"Required"`
}
func (createFileCommand *CreateFileCommand) Valid(validation *validation.Validation) {
validation.SetError("CustomValid", "未实现的自定义认证")
ext := filepath.Ext(createFileCommand.Name)
if !(ext == domain.XLS || ext == domain.XLSX) {
validation.Error(fmt.Sprintf("仅支持文件格式 xls 、 xlsx"))
return
}
}
func (createFileCommand *CreateFileCommand) ValidateCommand() error {
... ...
... ... @@ -2,6 +2,7 @@ package command
import (
"fmt"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"reflect"
"strings"
... ... @@ -9,12 +10,26 @@ import (
)
type EditDataTableCommand struct {
// 文件ID
FileId int `cname:"文件ID" json:"fileId" valid:"Required"`
//// 文件ID
//FileId int `cname:"文件ID" json:"fileId" valid:"Required"`
//
//Fields []*domain.Field
domain.EditTableRequest
}
func (editDataTableCommand *EditDataTableCommand) Valid(validation *validation.Validation) {
validation.SetError("CustomValid", "未实现的自定义认证")
if len(editDataTableCommand.ProcessFieldNames) == 0 {
validation.Error("未选择操作列")
return
}
if len(editDataTableCommand.Action) == 0 {
validation.Error("未知操作类型:" + editDataTableCommand.Action)
return
}
if editDataTableCommand.FileId == 0 {
validation.Error("文件ID不能为空")
return
}
}
func (editDataTableCommand *EditDataTableCommand) ValidateCommand() error {
... ...
package command
import (
"fmt"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type ExportFileCommand struct {
// 文件ID
FileId int `cname:"文件ID" json:"fileId" valid:"Required"`
}
func (cmd *ExportFileCommand) Valid(validation *validation.Validation) {
}
func (cmd *ExportFileCommand) ValidateCommand() error {
valid := validation.Validation{}
b, err := valid.Valid(cmd)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(cmd).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
... ... @@ -10,11 +10,15 @@ import (
type FlushDataTableCommand struct {
// 文件ID
FileId int `cname:"文件ID" json:"fileId" valid:"Required"`
ObjectId int `cname:"文件ID" json:"objectId" valid:"Required"`
// 记录数
//RowCount int `cname:"记录数" json:"rowCount" valid:"Required"`
// 数据列
//DataFields []*domain.Field `cname:"数据列" json:"fields" valid:"Required"`
}
func (flushDataTableCommand *FlushDataTableCommand) Valid(validation *validation.Validation) {
validation.SetError("CustomValid", "未实现的自定义认证")
}
func (flushDataTableCommand *FlushDataTableCommand) ValidateCommand() error {
... ...
... ... @@ -11,10 +11,12 @@ import (
type GenerateMainTableCommand struct {
// 文件ID
FileId int `cname:"文件ID" json:"fileId" valid:"Required"`
// 表名
TableName string `cname:"表名" json:"tableName" valid:"Required"`
}
func (generateMainTableCommand *GenerateMainTableCommand) Valid(validation *validation.Validation) {
validation.SetError("CustomValid", "未实现的自定义认证")
}
func (generateMainTableCommand *GenerateMainTableCommand) ValidateCommand() error {
... ...
... ... @@ -2,6 +2,7 @@ package command
import (
"fmt"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"reflect"
"strings"
... ... @@ -10,11 +11,18 @@ import (
type LoadDataTableCommand struct {
// 文件ID
FileId int `cname:"文件ID" json:"fileId" valid:"Required"`
FileId int `cname:"文件ID" json:"objectId" valid:"Required"`
Fields []*domain.Field `json:"fields"`
domain.Where
}
func (loadDataTableCommand *LoadDataTableCommand) Valid(validation *validation.Validation) {
validation.SetError("CustomValid", "未实现的自定义认证")
if loadDataTableCommand.PageNumber == 0 {
loadDataTableCommand.PageNumber = 1
}
if loadDataTableCommand.PageSize == 0 {
loadDataTableCommand.PageSize = 20
}
}
func (loadDataTableCommand *LoadDataTableCommand) ValidateCommand() error {
... ...
package command
import (
"fmt"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type PrepareTemporaryFileCommand struct {
// 文件ID
FileId int `cname:"文件ID" json:"fileId" valid:"Required"`
}
func (cmd *PrepareTemporaryFileCommand) Valid(validation *validation.Validation) {
}
func (cmd *PrepareTemporaryFileCommand) ValidateCommand() error {
valid := validation.Validation{}
b, err := valid.Valid(cmd)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(cmd).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
... ... @@ -14,7 +14,7 @@ type RemoveFileCommand struct {
}
func (removeFileCommand *RemoveFileCommand) Valid(validation *validation.Validation) {
validation.SetError("CustomValid", "未实现的自定义认证")
}
func (removeFileCommand *RemoveFileCommand) ValidateCommand() error {
... ...
package dto
import "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
type DataTableDto struct {
FileId int `json:"fileId"`
DataFields []*Field `json:"dataFields"`
DataRows [][]interface{} `json:"dataRows"`
Total int `json:"total"`
PageNumber int `json:"pageNumber"`
InValidCells []InValidCell `json:"inValidCells"`
}
type Field struct {
// 索引序号
// Index int `json:"index"`
// 名称
Name string `json:"name"`
// 对应数据库类型
Type string `json:"type"`
}
type InValidCell struct {
X int `json:"x"`
Y int `json:"y"`
Error string `json:"error"`
}
func NewDataTableDtoDemo(fileId int) DataTableDto {
return DataTableDto{
DataFields: []*Field{
{
//Index: 1,
Name: "产品名称",
Type: domain.String.ToString(),
},
{
//Index: 2,
Name: "产品数量",
Type: domain.Int.ToString(),
},
},
DataRows: [][]interface{}{
{"素面", 200},
{"冻豆腐", 400},
{"冻豆腐1", 300},
{"冻豆2", "A"},
},
InValidCells: []InValidCell{
{
X: 1,
Y: 3,
Error: "不是一个有效的数值",
},
},
PageNumber: 1,
Total: 100,
FileId: fileId,
}
}
... ...
package dto
import "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
type EditDataTableDto struct {
Code int `json:"errNo,omitempty"`
Error string `json:"errMsg,omitempty"`
InValidCells []domain.InValidCell `json:"inValidCells,omitempty"`
}
func (d *EditDataTableDto) Load(m *domain.DataEditDataTable) *EditDataTableDto {
d.Error = ""
if len(m.InValidCells) > 0 {
d.Error = "类型转换错误"
d.Code = 1001 // 1001:类型转换错误
}
//d.InValidCells = m.InValidCells
return d
}
... ...
package dto
import (
"github.com/linmadan/egglib-go/utils/xtime"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
)
type FileDto struct {
// 文件ID
FileId int `json:"fileId"`
// 名称
Name string `json:"name"`
// 文件地址
Url string `json:"url"`
// 文件类型
FileType string `json:"fileType"`
// 后缀扩展
Ext string `json:"ext"`
// 创建时间
Time string `json:"time"`
}
func (d *FileDto) Load(f *domain.File) {
d.FileId = f.FileId
d.Name = f.FileInfo.Name
d.Url = f.FileInfo.Url
d.FileType = f.FileType
d.Ext = f.FileInfo.Ext
d.Time = xtime.New(f.UpdatedAt).Local().Format("2006-01-02 15:04:05")
}
... ...
package query
import (
"fmt"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type SearchFileQuery struct {
// 查询偏离量
// Offset int `cname:"查询偏离量" json:"offset"`
// 查询限制
Limit int `cname:"查询限制" json:"limit"`
// 页码
// PageNumber int `cname:"页码" json:"pageNumber,omitempty"`
// 页数
FileName string `cname:"文件名称" json:"fileName,omitempty"`
PageSize int `cname:"页数" json:"pageSize,omitempty"`
LastId int `cname:"最后一条记录ID" json:"lastId"`
FileType domain.FileType `cname:"文件类型" json:"fileType" valid:"Required"`
Context *domain.Context
}
func (cmd *SearchFileQuery) Valid(validation *validation.Validation) {
cmd.Limit = cmd.PageSize
}
func (cmd *SearchFileQuery) ValidateQuery() error {
valid := validation.Validation{}
b, err := valid.Valid(cmd)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(cmd).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
package service
import (
"bytes"
"fmt"
"github.com/beego/beego/v2/client/httplib"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/excel"
"time"
"github.com/linmadan/egglib-go/core/application"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/factory"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/file/command"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/file/dto"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/redis"
)
// FilePreview 加载表格数据
func (fileService *FileService) FilePreview(ctx *domain.Context, loadDataTableCommand *command.LoadDataTableCommand) (interface{}, error) {
if err := loadDataTableCommand.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
loadDataTableService, _ := factory.CreateLoadDataTableService(transactionContext)
data, err := loadDataTableService.Preview(ctx, loadDataTableCommand.FileId, loadDataTableCommand.Fields, loadDataTableCommand.Where)
if err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return data, nil
}
// PrepareTemporaryFile 准备临时文件
func (fileService *FileService) PrepareTemporaryFile(ctx *domain.Context, cmd *command.PrepareTemporaryFileCommand) (interface{}, error) {
if err := cmd.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
loadDataTableService, _ := factory.CreateLoadDataTableService(transactionContext)
data, err := loadDataTableService.CreateTemporaryFile(ctx, cmd.FileId)
if err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
fileDto := &dto.FileDto{}
fileDto.Load(data)
return fileDto, nil
}
// EditDataTable 编辑表格数据
func (fileService *FileService) EditDataTable(ctx *domain.Context, editDataTableCommand *command.EditDataTableCommand) (interface{}, error) {
if err := editDataTableCommand.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
cache := redis.NewFileCacheService()
temporaryFile, err := cache.Get(redis.KeyTemporaryFileInfo(editDataTableCommand.FileId))
if err != nil {
return nil, factory.FastError(err)
}
editDataTableCommand.Fields = temporaryFile.Fields
editDataTableCommand.ProcessFields = temporaryFile.MatchFields(editDataTableCommand.ProcessFieldNames)
if len(editDataTableCommand.ProcessFields) == 0 {
return nil, factory.FastError(fmt.Errorf("请至少选择一个数据列"))
}
// allowAction := func(fields []*domain.Field, action string) error {
// for _, f := range fields {
// if f.SQLType != string(domain.String) &&
// !(action == domain.RemoveColumn || action == domain.CopyColumn || action == domain.RenameColumn || action == domain.ConvertColumnType) {
// return fmt.Errorf("列【%v】必须先转字符串类型",f.Name)
// }
// }
// return nil
// }
// if err = allowAction(editDataTableCommand.ProcessFields, editDataTableCommand.Action); err != nil {
// return nil, factory.FastError(err)
// }
editDataTableService, _ := factory.CreateEditDataTableService(transactionContext)
response, err := editDataTableService.Edit(ctx, editDataTableCommand.EditTableRequest)
if err != nil {
return nil, factory.FastError(err)
}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return (&dto.EditDataTableDto{}).Load(response), nil
}
// FlushDataTable 持久化表格数据
func (fileService *FileService) FlushDataTable(ctx *domain.Context, flushDataTableCommand *command.FlushDataTableCommand) (interface{}, error) {
if err := flushDataTableCommand.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
flushDataTableService, _ := factory.CreateFlushDataTableService(transactionContext)
cache := redis.NewFileCacheService()
temporaryFile, err := cache.Get(redis.KeyTemporaryFileInfo(flushDataTableCommand.ObjectId))
if err != nil {
return nil, factory.FastError(err)
}
if _, err := flushDataTableService.Flush(ctx, flushDataTableCommand.ObjectId, &domain.Table{
DataFields: temporaryFile.Fields,
RowCount: temporaryFile.Total,
}); err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return struct{}{}, nil
}
// GenerateMainTable 生成主表
func (fileService *FileService) GenerateMainTable(ctx *domain.Context, generateMainTableCommand *command.GenerateMainTableCommand) (interface{}, error) {
if err := generateMainTableCommand.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
generateMainTableService, _ := factory.CreateGenerateMainTableService(transactionContext)
_, err = generateMainTableService.GenerateTable(ctx, generateMainTableCommand.FileId, generateMainTableCommand.TableName)
if err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return struct{}{}, nil
}
// AppendDataToTable 追加数据
func (fileService *FileService) AppendDataToTable(ctx *domain.Context, cmd *command.AppendDataToTableCommand) (interface{}, error) {
if err := cmd.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
generateMainTableService, _ := factory.CreateAppendDataToTableService(transactionContext)
result, err := generateMainTableService.AppendData(ctx, cmd.FileId, cmd.TableId, cmd.MappingFields)
if err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return result, nil
}
// ExportFile 文件下载
func (fileService *FileService) ExportFile(ctx *domain.Context, cmd *command.ExportFileCommand) (interface{}, error) {
if err := cmd.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
_, file, err := factory.FastPgFile(transactionContext, cmd.FileId)
if err != nil {
return nil, factory.FastError(err)
}
var response = struct {
Url string `json:"url"`
Ext string `json:"ext"`
FileName string `json:"fileName"`
}{}
if file.FileType == domain.SourceFile.ToString() {
response.Url = file.FileInfo.Url
response.Ext = domain.XLSX
response.FileName = file.FileInfo.Name
return response, nil
}
_, table, err := factory.FastPgTable(transactionContext, file.FileInfo.TableId)
if err != nil {
return nil, factory.FastError(err)
}
f, err := httplib.Get(file.FileInfo.Url).Bytes()
if err != nil {
return nil, factory.FastError(err)
}
reader := bytes.NewReader(f)
var importer *excel.Importer = excel.NewExcelImportByFile(file.FileInfo.Ext)
data, err := importer.OpenExcelFromIoReader(reader)
if err != nil {
return nil, factory.FastError(err)
}
filename := fmt.Sprintf("%v_%v.xlsx", file.FileInfo.Name, time.Now().Format("060102150405"))
path := fmt.Sprintf("public/%v", filename)
writerTo := excel.NewXLXSWriterTo(importer.Reader().Header().Columns, data)
writerTo.ToInterfaces = domain.MakeToInterfaces(table.DataFields)
if err := writerTo.Save(path); err != nil {
return nil, factory.FastError(err)
}
response.Url = domain.DownloadUrl(filename)
response.FileName = file.FileInfo.Name
response.Ext = domain.XLSX
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return response, nil
}
... ...
... ... @@ -2,12 +2,18 @@ package service
import (
"fmt"
"path/filepath"
"time"
"github.com/linmadan/egglib-go/core/application"
"github.com/linmadan/egglib-go/utils/tool_funs"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/factory"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/file/command"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/file/dto"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/file/query"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/domainService"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/utils"
)
// 文件服务
... ... @@ -15,7 +21,7 @@ type FileService struct {
}
// 创建文件服务
func (fileService *FileService) CreateFile(createFileCommand *command.CreateFileCommand) (interface{}, error) {
func (fileService *FileService) CreateFile(ctx *domain.Context, createFileCommand *command.CreateFileCommand) (interface{}, error) {
if err := createFileCommand.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
... ... @@ -29,90 +35,47 @@ func (fileService *FileService) CreateFile(createFileCommand *command.CreateFile
defer func() {
transactionContext.RollbackTransaction()
}()
newFile := &domain.File{
//Name: createFileCommand.Name,
//Url: createFileCommand.Url,
//FileSize: createFileCommand.FileSize,
}
var fileRepository domain.FileRepository
if value, err := factory.CreateFileRepository(map[string]interface{}{
"transactionContext": transactionContext,
}); err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
} else {
fileRepository = value
fileInfo := &domain.FileInfo{
Name: domain.FileName(createFileCommand.Name),
Url: createFileCommand.Url,
FileSize: createFileCommand.FileSize,
Ext: filepath.Ext(createFileCommand.Name),
}
if file, err := fileRepository.Save(newFile); err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
} else {
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return file, nil
}
}
// 编辑表格数据
func (fileService *FileService) EditDataTable(editDataTableCommand *command.EditDataTableCommand) (interface{}, error) {
if err := editDataTableCommand.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return nil, nil
}
newFile := &domain.File{
FileType: domain.SourceFile.ToString(),
FileInfo: fileInfo,
SourceFileId: 0,
//Operator: "",
CreatedAt: time.Now(),
UpdatedAt: time.Now(),
Context: ctx,
}
fileRepository, _, _ := factory.FastPgFile(transactionContext, 0)
// 持久化表格数据
func (fileService *FileService) FlushDataTable(flushDataTableCommand *command.FlushDataTableCommand) (interface{}, error) {
if err := flushDataTableCommand.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
// 文件名相同进行替换
if oldFile, findOldFileErr := fileRepository.FindOne(map[string]interface{}{
"context": ctx,
"fileName": fileInfo.Name,
"fileType": domain.SourceFile.ToString(),
}); oldFile != nil && findOldFileErr == nil {
oldFile.FileInfo = fileInfo
oldFile.UpdatedAt = time.Now()
newFile = oldFile
}
defer func() {
transactionContext.RollbackTransaction()
}()
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return nil, nil
}
// 生成主表
func (fileService *FileService) GenerateMainTable(generateMainTableCommand *command.GenerateMainTableCommand) (interface{}, error) {
if err := generateMainTableCommand.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
transactionContext, err := factory.CreateTransactionContext(nil)
file, err := fileRepository.Save(newFile)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
if err = factory.FastLog(transactionContext, domain.CommonLog, file.FileId, &domainService.FileUploadSuccessLog{
LogEntry: domain.NewLogEntry(file.FileInfo.Name, domain.SourceFile.ToString(), domain.FileUpload, ctx),
}); err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return nil, nil
return struct{}{}, nil
}
// 返回文件服务
... ... @@ -143,7 +106,7 @@ func (fileService *FileService) GetFile(getFileQuery *query.GetFileQuery) (inter
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
if file == nil {
return nil, application.ThrowError(application.RES_NO_FIND_ERROR, fmt.Sprintf("%s", string(getFileQuery.FileId)))
return nil, application.ThrowError(application.RES_NO_FIND_ERROR, fmt.Sprintf("%d", getFileQuery.FileId))
} else {
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
... ... @@ -188,9 +151,9 @@ func (fileService *FileService) ListFile(listFileQuery *query.ListFileQuery) (in
}
}
// 加载表格数据
func (fileService *FileService) LoadDataTable(loadDataTableCommand *command.LoadDataTableCommand) (interface{}, error) {
if err := loadDataTableCommand.ValidateCommand(); err != nil {
// 返回文件服务列表
func (fileService *FileService) SearchFile(listFileQuery *query.SearchFileQuery) (interface{}, error) {
if err := listFileQuery.ValidateQuery(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
transactionContext, err := factory.CreateTransactionContext(nil)
... ... @@ -203,14 +166,35 @@ func (fileService *FileService) LoadDataTable(loadDataTableCommand *command.Load
defer func() {
transactionContext.RollbackTransaction()
}()
var fileRepository domain.FileRepository
if value, err := factory.CreateFileRepository(map[string]interface{}{
"transactionContext": transactionContext,
}); err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
} else {
fileRepository = value
}
count, files, err := fileRepository.Find(utils.ObjectToMap(listFileQuery))
if err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
var fileDtos = make([]*dto.FileDto, 0)
for _, file := range files {
var item = &dto.FileDto{}
item.Load(file)
fileDtos = append(fileDtos, item)
}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return nil, nil
return map[string]interface{}{
"count": count,
"files": fileDtos,
}, nil
}
// 移除文件服务
func (fileService *FileService) RemoveFile(removeFileCommand *command.RemoveFileCommand) (interface{}, error) {
func (fileService *FileService) RemoveFile(ctx *domain.Context, removeFileCommand *command.RemoveFileCommand) (interface{}, error) {
if err := removeFileCommand.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
... ... @@ -237,16 +221,17 @@ func (fileService *FileService) RemoveFile(removeFileCommand *command.RemoveFile
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
if file == nil {
return nil, application.ThrowError(application.RES_NO_FIND_ERROR, fmt.Sprintf("%s", string(removeFileCommand.FileId)))
return nil, application.ThrowError(application.RES_NO_FIND_ERROR, fmt.Sprintf("%d", removeFileCommand.FileId))
}
if file, err := fileRepository.Remove(file); err != nil {
deleteFileService, _ := factory.CreateDeleteFileService(transactionContext)
err = deleteFileService.DeleteFiles(ctx, file)
if err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
} else {
}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return file, nil
}
return struct{}{}, nil
}
// 更新文件服务
... ... @@ -277,7 +262,7 @@ func (fileService *FileService) UpdateFile(updateFileCommand *command.UpdateFile
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
if file == nil {
return nil, application.ThrowError(application.RES_NO_FIND_ERROR, fmt.Sprintf("%s", string(updateFileCommand.FileId)))
return nil, application.ThrowError(application.RES_NO_FIND_ERROR, fmt.Sprintf("%d", updateFileCommand.FileId))
}
if err := file.Update(tool_funs.SimpleStructToMap(updateFileCommand)); err != nil {
return nil, application.ThrowError(application.BUSINESS_ERROR, err.Error())
... ... @@ -292,6 +277,82 @@ func (fileService *FileService) UpdateFile(updateFileCommand *command.UpdateFile
}
}
// 取消校验中的文件
func (fileService *FileService) CancelVerifyingFile(ctx *domain.Context, cmd *command.CancelVerifyingFileCommand) (interface{}, error) {
if err := cmd.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
_, file, err := factory.FastPgFile(transactionContext, cmd.FileId)
if err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
if file.FileType != domain.TemporaryFile.ToString() {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, "校验中的文件才允许取消")
}
deleteFileService, _ := factory.CreateDeleteFileService(transactionContext)
err = deleteFileService.DeleteFiles(nil, file)
if err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
byteCore, _ := factory.CreateByteCoreService(transactionContext)
_, err = byteCore.CancelFile(domain.ReqCancelFile{})
if err != nil {
return nil, factory.FastError(err)
}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return struct{}{}, nil
}
// CheckFileVerifyStatus 检查文件校验状态
func (fileService *FileService) CheckFileVerifyStatus(ctx *domain.Context, cmd *command.CheckFileVerifyStatusCommand) (interface{}, error) {
if err := cmd.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
fileRepository, file, err := factory.FastPgFile(transactionContext, cmd.FileId)
if err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
var res = struct {
ExistVerifyFile bool `json:"existVerifyFile"`
}{}
_, verifyFiles, findErr := fileRepository.Find(map[string]interface{}{"context": ctx, "fileType": domain.VerifiedFile.ToString(), "sourceFileId": file.SourceFileId})
if findErr == nil && len(verifyFiles) > 0 {
res.ExistVerifyFile = true
}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return res, nil
}
func NewFileService(options map[string]interface{}) *FileService {
newFileService := &FileService{}
return newFileService
... ...
package command
import (
"fmt"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type CreateLogCommand struct {
// 对象名称 数据表名 / 文件名
ObjectName string `cname:"对象名称 数据表名 / 文件名" json:"objectName" valid:"Required"`
// 对象类型 1.主表 2.分表 3.副表 4.源文件 5.校验文件
ObjectType string `cname:"对象类型 1.主表 2.分表 3.副表 4.源文件 5.校验文件" json:"objectType" valid:"Required"`
// 操作类型 1.主表生成 2.主表拆分 3.数据导入 4.分表生成 5.表复制 6.编辑记录 7.文件上传 8.文件校验
OperationType string `cname:"操作类型 1.主表生成 2.主表拆分 3.数据导入 4.分表生成 5.表复制 6.编辑记录 7.文件上传 8.文件校验" json:"operationType" valid:"Required"`
// 日志内容
Content string `cname:"日志内容" json:"content" valid:"Required"`
// 操作人名称
OperatorName string `cname:"操作人名称" json:"operatorName" valid:"Required"`
}
func (createLogCommand *CreateLogCommand) Valid(validation *validation.Validation) {
validation.SetError("CustomValid", "未实现的自定义认证")
}
func (createLogCommand *CreateLogCommand) ValidateCommand() error {
valid := validation.Validation{}
b, err := valid.Valid(createLogCommand)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(createLogCommand).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
package command
import (
"fmt"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type RemoveLogCommand struct {
// 日志ID
LogId int `cname:"日志ID" json:"logId" valid:"Required"`
}
func (removeLogCommand *RemoveLogCommand) Valid(validation *validation.Validation) {
validation.SetError("CustomValid", "未实现的自定义认证")
}
func (removeLogCommand *RemoveLogCommand) ValidateCommand() error {
valid := validation.Validation{}
b, err := valid.Valid(removeLogCommand)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(removeLogCommand).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
package command
import (
"fmt"
"reflect"
"strings"
"time"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"github.com/beego/beego/v2/core/validation"
)
type SearchLogCommand struct {
// 日志类型 VerifiedStepLog:校验步骤 CommonLog:常规日志
LogType string `json:"logType"`
// 日志内容
MatchContent string `cname:"匹配日志内容" json:"matchContent,omitempty"`
// 源数据ID
SourceId int `cname:"源数据ID" json:"sourceId"`
// 多个源数据ID
InSourceId []int `cname:"多个源数据ID" json:"inSourceId"`
// 页码
PageNumber int `cname:"页码" json:"pageNumber,omitempty"`
// 页数
PageSize int `cname:"页数" json:"pageSize,omitempty"`
// 对象名称
ObjectName string `cname:"页数" json:"objectName,omitempty"`
Year int `cname:"年" json:"year,omitempty"`
Month int `cname:"月" json:"month,omitempty"`
Day int `cname:"日" json:"day,omitempty"`
// 开始时间
BeginTime time.Time `cname:"开始时间" json:"beginTime"`
// 结束时间
EndTime time.Time `cname:"结束时间" json:"endTime"`
// 按log_id 排序
SortByLogId string `json:"sortByLogId"`
Context *domain.Context
}
func (cmd *SearchLogCommand) Valid(validation *validation.Validation) {
if cmd.Year > 0 && cmd.Month == 0 && cmd.Day == 0 {
cmd.BeginTime = time.Date(cmd.Year, 1, 1, 0, 0, 0, 0, time.Local)
cmd.EndTime = cmd.BeginTime.AddDate(1, 0, 0)
}
if cmd.Year > 0 && cmd.Month > 0 && cmd.Day == 0 {
cmd.BeginTime = time.Date(cmd.Year, time.Month(cmd.Month), 1, 0, 0, 0, 0, time.Local)
cmd.EndTime = cmd.BeginTime.AddDate(0, 1, 0)
}
if cmd.Year > 0 && cmd.Month > 0 && cmd.Day > 0 {
cmd.BeginTime = time.Date(cmd.Year, time.Month(cmd.Month), cmd.Day, 0, 0, 0, 0, time.Local)
cmd.EndTime = cmd.BeginTime.AddDate(0, 0, 1)
}
if cmd.PageNumber == 0 {
cmd.PageNumber = 1
}
}
func (cmd *SearchLogCommand) ValidateCommand() error {
valid := validation.Validation{}
b, err := valid.Valid(cmd)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(cmd).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
package command
import (
"fmt"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type UpdateLogCommand struct {
// 日志ID
LogId int `cname:"日志ID" json:"logId" valid:"Required"`
}
func (updateLogCommand *UpdateLogCommand) Valid(validation *validation.Validation) {
validation.SetError("CustomValid", "未实现的自定义认证")
}
func (updateLogCommand *UpdateLogCommand) ValidateCommand() error {
valid := validation.Validation{}
b, err := valid.Valid(updateLogCommand)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(updateLogCommand).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
package dto
import "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
type LogDto struct {
// 日志ID
LogId int `json:"logId"`
// 日志类型 1.校验步骤 2.常规日志
//LogType string `json:"logType"`
// 源数据ID
SourceId int `json:"sourceId"`
// 对象名称 数据表名 / 文件名
ObjectName string `json:"objectName"`
// 对象类型 1.主表 2.分表 3.副表 4.源文件 5.校验文件
ObjectType string `json:"objectType"`
// 操作类型 1.主表生成 2.主表拆分 3.数据导入 4.分表生成 5.表复制 6.编辑记录 7.文件上传 8.文件校验
OperationType string `json:"operationType"`
// 日志内容
Content string `json:"content"`
// 操作人名称
OperatorName string `json:"operatorName"`
// 创建时间
CreatedAt string `json:"createdAt"`
}
func (d *LogDto) Load(m *domain.Log) {
d.LogId = m.LogId
d.SourceId = m.SourceId
d.ObjectName = m.ObjectName
d.ObjectType = m.ObjectType
d.OperationType = m.OperationType
d.Content = m.Content
d.OperatorName = m.OperatorName
d.CreatedAt = m.CreatedAt.Local().Format("2006-01-02 15:04:05")
}
... ...
package dto
import "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
type VerifiedStepLogDto struct {
// 日志ID
LogId int `json:"logId"`
// 日志类型 1.校验步骤 2.常规日志
//LogType string `json:"logType"`
// 源数据ID
//SourceId int `json:"sourceId"`
// 对象名称 数据表名 / 文件名
//ObjectName string `json:"objectName"`
// 对象类型 1.主表 2.分表 3.副表 4.源文件 5.校验文件
//ObjectType string `json:"objectType"`
// 操作类型 1.主表生成 2.主表拆分 3.数据导入 4.分表生成 5.表复制 6.编辑记录 7.文件上传 8.文件校验
//OperationType string `json:"operationType"`
// 日志内容
Content string `json:"content"`
// 操作人名称
//OperatorName string `json:"operatorName"`
// 创建时间
CreatedAt string `json:"createdAt"`
// 错误级别
Level string `json:"level"`
// 错误信息
Error string `json:"error"`
}
func (d *VerifiedStepLogDto) Load(m *domain.Log) {
d.LogId = m.LogId
//d.SourceId = m.SourceId
//d.ObjectName = m.ObjectName
//d.ObjectType = m.ObjectType
//d.OperationType = m.OperationType
d.Content = m.Content
//d.OperatorName = m.OperatorName
d.CreatedAt = m.CreatedAt.Local().Format("2006-01-02 15:04:05")
d.Level = m.Entry.Level
d.Error = m.Entry.Error
if len(d.Level) == 0 {
d.Level = domain.LevelInfo.ToString()
}
}
... ...
package query
import (
"fmt"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type GetLogQuery struct {
// 日志ID
LogId int `cname:"日志ID" json:"logId" valid:"Required"`
}
func (getLogQuery *GetLogQuery) Valid(validation *validation.Validation) {
validation.SetError("CustomValid", "未实现的自定义认证")
}
func (getLogQuery *GetLogQuery) ValidateQuery() error {
valid := validation.Validation{}
b, err := valid.Valid(getLogQuery)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(getLogQuery).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
package query
import (
"fmt"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type ListLogQuery struct {
// 查询偏离量
Offset int `cname:"查询偏离量" json:"offset" valid:"Required"`
// 查询限制
Limit int `cname:"查询限制" json:"limit" valid:"Required"`
}
func (listLogQuery *ListLogQuery) Valid(validation *validation.Validation) {
validation.SetError("CustomValid", "未实现的自定义认证")
}
func (listLogQuery *ListLogQuery) ValidateQuery() error {
valid := validation.Validation{}
b, err := valid.Valid(listLogQuery)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(listLogQuery).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...