作者 yangfu

Merge branch 'test' into dev

正在显示 88 个修改的文件 包含 3824 行增加248 行删除

要显示太多修改。

为保证性能只显示 88 of 88+ 个文件。

... ... @@ -23,6 +23,7 @@ _testmain.go
*.test
.log
.idea
.vscode
app.log
go.sum
... ...
FROM golang:latest
#FROM 192.168.0.243:5000/mmm/character-library-metadata-bastion:base
ENV APP_DIR $GOPATH/src/character-library-metadata-bastion
RUN mkdir -p $APP_DIR
WORKDIR $APP_DIR/
... ... @@ -8,7 +9,7 @@ COPY ./go.mod go.mod
COPY ./main.go main.go
RUN ["ln","-sf","/usr/share/zoneinfo/Asia/Shanghai","/etc/localtime"]
ENV GO111MODULE on
ENV GOPROXY https://goproxy.io
ENV GOPROXY https://goproxy.cn
RUN ["go","mod","tidy"]
RUN ["ls"]
RUN ["go","build"]
... ...
... ... @@ -3,7 +3,7 @@
## 源文件管理
- 文件列表 - list
- 上传 - 上传oss / 创建file
- 上传 - 上传 oss / 创建 file
- 加载 - loadDataTable
- 编辑 - editDataTable
- 持久化 - flushDataTable
... ... @@ -11,7 +11,6 @@
- 删除 - delete
- 操作日志 - log
- editDataTable params 列表
### 加载表格数据 loadDataTable - 查询
... ... @@ -20,15 +19,15 @@
{
"fileId": 1,
"where": [
{
"field": {
"index": 1,
"name": "产品名称"
},
"in": ["a","b"],
"ex": ["c","d"],
"sort": ["a","asc"]
}
{
"field": {
"index": 1,
"name": "产品名称"
},
"in": ["a", "b"],
"ex": ["c", "d"],
"sort": ["a", "asc"]
}
]
}
```
... ... @@ -37,137 +36,523 @@
```json
{
"field": {
"index": 1,
"name": "产品名称"
},
"operation": {
"desc": ["拆分","按字符数"],
"code": "split_by_char_number"
},
"params": []
"field": {
"index": 1,
"name": "产品名称"
},
"operation": {
"desc": ["拆分", "按字符数"],
"code": "split_by_char_number"
},
"params": []
}
```
精简
```json
{
"field": "产品名称",
"desc": ["拆分","按字符数"],
"operationCode": "split_by_char_number",
"params": []
"field": "产品名称",
"desc": ["拆分", "按字符数"],
"operationCode": "split_by_char_number",
"params": []
}
```
`params 列表`
### 数据展示
```json
{
"code": 0,
"data": {
"dataFields": [
{
"index": 1,
"name": "产品名称",
"type": "string"
},
{
"index": 2,
"name": "产品数量",
"type": "int"
}
],
"dataRows": [
[
"素面",
200
],
[
"冻豆腐",
400
],
[
"冻豆腐1",
300
],
[
"冻豆2",
"A"
]
],
"total": 100,
"pageNumber": 1,
"inValidCells": [
{
"x": 1,
"y": 3,
"error": "不是一个有效的数值"
}
]
},
"msg": "ok"
"code": 0,
"data": {
"dataFields": [
{
"index": 1,
"name": "产品名称",
"type": "string"
},
{
"index": 2,
"name": "产品数量",
"type": "int"
}
],
"dataRows": [
["素面", 200],
["冻豆腐", 400],
["冻豆腐1", 300],
["冻豆2", "A"]
],
"total": 100,
"pageNumber": 1,
"inValidCells": [
{
"x": 1,
"y": 3,
"error": "不是一个有效的数值"
}
]
},
"msg": "ok"
}
```
## 表关联关系
- [x] 可追加数据的表列表 /tables/search-appended-list
- [x] 校验文件列表 /files/search-verified-file
- [x] 匹配方案列表 /mapping-rule-config/search
- [x] 匹配方案主表 /mapping-rule-config/prepare //主表 校验表 主表字段 校验文件表字段
- [x] 匹配方案添加 /mapping-rule-config/
- [x] 匹配方案删除 /mapping-rule-config/:id
- [x] 追加数据到表格 /append-data-to-table // 验证是否追加过
- [ ] 取消校验中的文件 /cancel-verifying-file //
- [x] 表结构更新 /tables/update-table-struct
- [x] 表结构添加 /tables/add-table-struct
- [x] 分表列表 /tables/search
- [x] 表复制 /tables/copy-data-table
- [x] 表删除 /tables/:id // 若是删除主表,需级联删除关联的分表,删除内容包括表数据及表结构;? 分表的副表是否要删除
- [x] 表详情 /tables/:id // 表结构
- [x] 表更新 /tables/:id // 表结构、分表才可以编辑
- [x] 日志搜索 /log/search
- [x] 校验步骤日志 /log/verified-step-Log
- [x] 可追加数据的表列表 /tables/search-appended-list
- [x] 校验文件列表 /files/search-verified-file
- [x] 匹配方案列表 /mapping-rule-config/search
- [x] 匹配方案主表 /mapping-rule-config/prepare //主表 校验表 主表字段 校验文件表字段
- [x] 匹配方案添加 /mapping-rule-config/
- [x] 匹配方案删除 /mapping-rule-config/:id
- [x] 追加数据到表格 /append-data-to-table // 验证是否追加过
- [ ] 取消校验中的文件 /cancel-verifying-file //
- [x] 表结构更新 /tables/update-table-struct
- [x] 表结构添加 /tables/add-table-struct
- [x] 分表列表 /tables/search
- [x] 表复制 /tables/copy-data-table
- [x] 表删除 /tables/:id // 若是删除主表,需级联删除关联的分表,删除内容包括表数据及表结构;? 分表的副表是否要删除
- [x] 表详情 /tables/:id // 表结构
- [x] 表更新 /tables/:id // 表结构、分表才可以编辑
- [x] 日志搜索 /log/search
- [x] 校验步骤日志 /log/verified-step-Log
## 数据预览
- [x] 表数据预览(格式) /table/preview
- [x] 表数据自定义查询 /table/preview where conditions 升序、降序 包含、不包含
- [x] 表数据字段可选值搜索 /table/field-optional 文本匹配
- [x] 表数据预览(格式) /table/preview
- [x] 表数据自定义查询 /table/preview where conditions 升序、降序 包含、不包含
- [x] 表数据字段可选值搜索 /table/field-optional 文本匹配
- [x] 表数据更新、添加、删除 /table/row-data-mutation
- [x] 表数据导出 /table/export-table
- [x] 表数据导出 /table/export-table
## 数据验证
- [x] 文件验证 /data/edit-data-table
- [x] 文件验证 /data/edit-data-table
## 底层字库接口
```json
{
"file": {},
"fields": [],
"action":"filed_rename",
"params": ["产品名2"]
}
```
- [ ] 数据预览 1
- [ ] 表格编辑 1
- [ ] 保存校验文件 (文件地址) 1
- [ ] 生成主表 1
- [ ] 表复制 (副表)
- [ ] 追加数据 (主表、副表)
- [x] 数据预览 1
- [x] 表格编辑 1
- [x] 保存校验文件 (文件地址) 1
- [x] 生成主表 1
- [x] 表复制 (副表)1
- [x] 追加数据 (主表、副表)
- [ ] 表删除 (主表、副表)~~、分表~~
- [x] ~~表拆分~~
- [x] ~~更新表结构(分表)~~
- [ ] 编辑、添加、删除表数据(副表) 1
- [x] 表拆分 1
- [x] 更新表结构(分表)1
- [x] 编辑、添加、删除表数据(副表) 1
- [ ] 取消校验
## 定时作业
- [x] 隔天清理校验中的文件
- [x] 隔天清理public临时文件
\ No newline at end of file
- [x] 隔天清理 public 临时文件
## 表数据导出
- [ ] 加锁,只允许当前用户同时只能发起一次导出命令 ,3min 过期
- [ ] 单次拉取数量 MR
- [ ] 100W ..
- [ ] 50W 120s 读取数据库:30s 保存文件:10s 下载:30M/500K=60S;RAR 压缩 24M/500k=50S
- [ ] 20W ..
- [ ] 10W ..
- [ ] 保存单个文件、压缩 | 保存多个文件、压缩
## 表达式解析
### 表达式类型说明
- ValueExprAST: 值表达式(数值,字符串) e.g. 1 、 2011-1-1 、字符串
```json
{
"exprType":"ValueExprAST",
"val":"",
"str":"业绩2"
}
```
- FieldExprAST: 字段表达式(处理的字段) e.g. 生产明细.业绩
```json
{
"exprType":"FieldExprAST",
"str":"业绩1",
"field":{
"tableId":1,
"tableName":"测试ABC",
"tableSqlName":"table_abc_test",
"fieldName":"业绩",
"fieldSqlName":"ye_ji_1",
"fieldSqlType":"Float"
}
}
```
- BinaryExprAST: 二元表达式 e.g. 1 + 2 、 100 * 生产明细.业绩
```json
{
"exprType":"BinaryExprAST",
"op":"/",
"lhs":{},
"rhs":{}
}
```
- FunCallerExprAST:函数表达式 e.g. sum(arg1,arg2,arg3)
```json
{
"arrayFlag": false,
"exprType": "FunCallerExprAST",
"name": "sum",
"args": []
}
```
### 用例
- 输入表达式 `SUM(1/COUNTIFS(【业绩】,【业绩】))`
```json
{
"arrayFlag":false,
"exprType":"FunCallerExprAST",
"name":"sum",
"args":[
{
"exprType":"BinaryExprAST",
"op":"/",
"lhs":{
"exprType":"ValueExprAST",
"val":"",
"str":"1"
},
"rhs":{
"arrayFlag":false,
"exprType":"FunCallerExprAST",
"name":"countifs",
"args":[
{
"exprType":"FieldExprAST",
"str":"业绩1",
"field":{
"tableId":1,
"tableName":"测试ABC",
"tableSqlName":"table_abc_test",
"fieldName":"业绩",
"fieldSqlName":"ye_ji_1",
"fieldSqlType":"Float"
}
},
{
"exprType":"ValueExprAST",
"val":"",
"str":"业绩2"
}
]
}
}
]
}
```
## 讨论事项
- [ ] 校验动作,参数模型讨论
- [ ] 校验日志错误(标红)
- [ ] 校验完毕应答实体,类型修改即使错误,也要返回修改完毕的表
## 参数说明
### 通用格式
```json
{
"objectId": 1,
"processFields": [],
"action": "xx",
"params": {}
}
```
processFields:操作字段
### 常规
1. 删除列
```json
{
"action": "remove-column",
"params": {}
}
```
2. 复制列
```json
{
"action": "copy-column",
"params": {}
}
```
3. 重命名
```json
{
"action": "rename-column",
"params": {
"newColumnName": "新的列名称"
}
}
```
4. 替换值
```json
{
"action": "replace-column",
"params": {
"replaceMethod": "replace",
"searchValue": "搜索值",
"replaceValue": "替换值"
}
}
```
参数说明
```
replaceMethod: 替换方法(1.replace:替换值 2.add-prefix:添加前缀 3.add-postfix:添加后缀 4.remove-prefix:去除前缀 5.remove-postfix:去除后缀 6.remove-chars:去除固定字符 7.clean:清除)
searchValue: 搜索值-replace,remove-prefix,remove-postfix,remove-chars参数
replaceValue: 替换值-replace,add-prefix,add-postfix参数
```
### 格式 formatMethod
1. 大写
```json
{
"action": "format-column",
"params": {
"formatMethod": "upper"
}
}
```
参数说明
```
formatMethod: 格式化方法(1.upper:大写2.lower:小写3.capitalize:首字母大写4.strip:修整)
```
2. 小写
```json
{
"action": "format-column",
"params": {
"formatMethod": "lower"
}
}
```
3. 首字母大写
```json
{
"action": "format-column",
"params": {
"formatMethod": "capitalize"
}
}
```
4. 清除
```json
{
"action": "replace-column",
"params": {
"replaceMethod": "clean"
}
}
```
5. 修整
```json
{
"action": "format-column",
"params": {
"formatMethod": "strip"
}
}
```
6. 添加前缀
```json
{
"action": "replace-column",
"params": {
"replaceMethod": "add-prefix",
"replaceValue": "前缀值"
}
}
```
7. 添加后缀
```json
{
"action": "replace-column",
"params": {
"replaceMethod": "add-postfix",
"replaceValue": "后缀值"
}
}
```
8. 去除前缀
```json
{
"action": "replace-column",
"params": {
"replaceMethod": "remove-prefix",
"searchValue": "前缀值"
}
}
```
9. 去除后最
```json
{
"action": "replace-column",
"params": {
"replaceMethod": "remove-postfix",
"searchValue": "后缀值"
}
}
```
10. 去除固定字符
```json
{
"action": "replace-column",
"params": {
"replaceMethod": "remove-chars",
"searchValue": "字符"
}
}
```
### 拆分 split-column
1. 按分隔符
```json
{
"action": "split-column",
"params": {
"splitMethod": "separator",
"separator": "|",
"splitDirection": "left",
"splitCount": "1"
}
}
```
参数说明
```
separator: 分割符号 ‘|’
splitDirection: 拆分方向(1.left:从左边 2.right:从右边)
splitCount: 拆分次数
```
2. 按字符数
```json
{
"action": "split-column",
"params": {
"splitMethod": "char-length",
"splitDirection": "left",
"charLength": "10",
"splitCount": "1"
}
}
```
参数说明
```
charLength: 字符长度-char-length专属参数
splitDirection: 拆分方向(1.left:从左边 2.right:从右边)
splitCount: 拆分次数 (拆分策略是重复时:值0或者非1整数)
```
### 提取 extract-column
1. 按日期
```json
{
"action": "extract-column",
"params": {
"extractMethod": "by-date"
}
}
```
参数说明
```
extractMethod: 提取方法(1.by-date:按日期 2.by-number:按数值)
```
2. 按数值 action
```json
{
"action": "extract-column",
"params": {
"extractMethod": "by-number"
}
}
```
### 修改字段类型
```json
{
"action": "convert-column-type",
"params": {
"convertType": "STRING"
}
}
```
参数说明
```
convertType:转换类型 STRING 数值: INT 小数: FLOAT 日期: DATE 时间: DATETIME
```
## 优化点
- [] 0.测试服务、数据库的上限(QPS,TPS)查询瓶颈的接口
- [] 1.模型详情缓存(tables、query_set)
- [] 2.列表搜索缓存 (tables、query_set),减轻数据库压力
```
更新、删除、重命名
get list:queryset:cxxx:*
删除所有匹配的缓存
消息队列更新事件、确保缓存一定移除成功、缓存时间控制
```
... ...
POSTGRESQL_DB_NAME = allied_creation_dev
POSTGRESQL_DB_NAME = allied_creation_test
POSTGRESQL_HOST = 114.55.200.59
POSTGRESQL_PORT = 31543
POSTGRESQL_USER = postgres
... ... @@ -9,3 +9,19 @@ HTTP_PORT = 8081
ENABLE_KAFKA_LOG11 = true
HTTPS_PORT = 8143
ALLIED_CREATION_USER_HOST = http://allied-creation-user-dev.fjmaimaimai.com
# AUTH_SERVER_HOST = http://127.0.0.1:8081
BYTE_CORE_HOST = http://47.97.5.102:8303
METADATA_BASTION_HOST = http://106.75.231.90:9999
KAFKA_HOST =47.97.5.102:9092
#192.168.100.35:9092
STARROCKS_DB_NAME = character_library
STARROCKS_USER = root
STARROCKS_PASSWORD = eagle1010
STARROCKS_HOST = 220.250.41.79
STARROCKS_PORT = 9030
BLACK_LIST_USER = 0
BLACK_LIST_COMPANY = 1612991734952759296
WHITE_LIST_USERS = 22,23
\ No newline at end of file
... ...
/*
1. 查询依赖的查询集
*/
select * from metadata.query_sets where query_set_info->>'BindTableId' in (
select distinct table_id::text from ( /*element_id,table_info,table_type*/
select json_array_elements(to_json(table_info->'dependencyTables'))::text::int element_id,table_info,table_id,table_type
from metadata.tables where table_info->'dependencyTables'::text <>'null' and table_type in ('SubProcess','CalculateTable')
) a where a.element_id in (select table_id from metadata.tables where table_type in ('MainTable','SubTable','SideTable'))
order by table_id asc
) and context->>'operatorId' = '22'
... ...
... ... @@ -18,4 +18,7 @@ CREATE INDEX IF NOT EXISTS idx_logs_company_id_operator_name ON metadata.logs US
CREATE INDEX IF NOT EXISTS idx_logs_company_id_created_at ON metadata.logs USING btree((context->>'companyId'),created_at);
/*mapping_rules*/
CREATE INDEX IF NOT EXISTS idx_mapping_rules_company_id_table_id_file_id ON metadata.mapping_rules USING btree((context->>'companyId'),table_id,file_id);
\ No newline at end of file
CREATE INDEX IF NOT EXISTS idx_mapping_rules_company_id_table_id_file_id ON metadata.mapping_rules USING btree((context->>'companyId'),table_id,file_id);
/*query_sets*/
CREATE INDEX IF NOT EXISTS idx_query_sets_company_id_type_deleted_at ON metadata.query_sets USING btree((context->>'companyId'),type,deleted_at);
\ No newline at end of file
... ...
ALTER TABLE files ADD file_from TEXT;
ALTER TABLE files ADD app_key TEXT;
Update files set file_from = 'ByteBankWebClient';
CREATE INDEX IF NOT EXISTS idx_files_app_key ON metadata.files USING btree(app_key);
alter table metadata.tables add column apply_at timestamptz;
update metadata.tables set apply_at = null where table_type in ('MainTable','SubTable','SideTable') and apply_at is null;
\ No newline at end of file
... ...
... ... @@ -102,9 +102,9 @@ spec:
- name: METADATA_BASTION_HOST
value: "http://character-library-metadata-bastion-dev.fjmaimaimai.com"
- name: BYTE_CORE_HOST
value: "http://192.168.100.34:8303"
value: "http://47.97.5.102:8303"
- name: STARROCKS_HOST
value: "118.178.239.45"
value: "220.250.41.79"
- name: STARROCKS_PORT
value: "9030"
- name: STARROCKS_DB_NAME
... ...
apiVersion: v1
kind: Service
metadata:
name: character-library-metadata-bastion
namespace: mmm-suplus-test
labels:
k8s-app: character-library-metadata-bastion
spec:
ports:
- name: "http"
port: 80
targetPort: 8082
selector:
k8s-app: character-library-metadata-bastion
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: character-library-metadata-bastion
namespace: mmm-suplus-test
labels:
k8s-app: character-library-metadata-bastion
spec:
replicas: 1
template:
metadata:
labels:
k8s-app: character-library-metadata-bastion
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- preference: {}
weight: 100
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- cn-hangzhou.i-bp1djh1xn7taumbue1ze
containers:
- name: character-library-metadata-bastion
image: 192.168.0.243:5000/mmm/character-library-metadata-bastion:dev
imagePullPolicy: Always
ports:
- containerPort: 8082
volumeMounts:
- mountPath: /opt/logs
name: accesslogs
env:
- name: POSTGRESQL_DB_NAME
valueFrom:
configMapKeyRef:
name: suplus-config
key: postgresqlalliedcreation.dbname
- name: POSTGRESQL_USER
valueFrom:
configMapKeyRef:
name: suplus-config
key: postgresql.user
- name: POSTGRESQL_PASSWORD
valueFrom:
configMapKeyRef:
name: suplus-config
key: postgresql.password
- name: POSTGRESQL_HOST
valueFrom:
configMapKeyRef:
name: suplus-config
key: postgresql.host
- name: POSTGRESQL_PORT
valueFrom:
configMapKeyRef:
name: suplus-config
key: postgresql.port
- name: REDIS_HOST
valueFrom:
configMapKeyRef:
name: suplus-config
key: redis.ip
- name: REDIS_PORT
valueFrom:
configMapKeyRef:
name: suplus-config
key: redis.port
- name: REDIS_AUTH
value: ""
- name: LOG_LEVEL
value: "debug"
- name: ERROR_BASE_CODE
value: "1"
- name: ERROR_BASE_CODE_MULTIPLE
value: "2000"
- name: KAFKA_HOST
value: "47.97.5.102:9092"
- name: HTTP_PORT
value: "8082"
- name: SERVICE_ENV
value: "test"
- name: METADATA_BASTION_HOST
value: "https://character-library-metadata-bastion-test.fjmaimaimai.com"
- name: BYTE_CORE_HOST
value: "http://47.97.5.102:8303"
- name: STARROCKS_HOST
value: "220.250.41.79"
- name: STARROCKS_PORT
value: "9030"
- name: STARROCKS_DB_NAME
value: "character_library"
- name: STARROCKS_USER
value: "root"
- name: STARROCKS_PASSWORD
value: "eagle1010"
- name: BLACK_LIST_USER
value: "1"
- name: BLACK_LIST_COMPANY
value: "1646025721363042304"
- name: WHITE_LIST_USERS
value: "0"
volumes:
- name: accesslogs
emptyDir: {}
\ No newline at end of file
... ...
#!/bin/bash
export PATH=/root/local/bin:$PATH
kubectl -n mmm-suplus-test get pods | grep -q character-library-metadata-bastion
if [ "$?" == "1" ];then
kubectl create -f /tmp/test/character-library-metadata-bastion/character-library-metadata-bastion.yaml --record
kubectl -n mmm-suplus-test get svc | grep -q character-library-metadata-bastion
if [ "$?" == "0" ];then
echo "character-library-metadata-bastion service install success!"
else
echo "character-library-metadata-bastion service install fail!"
fi
kubectl -n mmm-suplus-test get pods | grep -q character-library-metadata-bastion
if [ "$?" == "0" ];then
echo "character-library-metadata-bastion deployment install success!"
else
echo "character-library-metadata-bastion deployment install fail!"
fi
else
kubectl delete -f /tmp/test/character-library-metadata-bastion/character-library-metadata-bastion.yaml
kubectl -n mmm-suplus-test get svc | grep -q character-library-metadata-bastion
while [ "$?" == "0" ]
do
kubectl -n mmm-suplus-test get svc | grep -q character-library-metadata-bastion
done
kubectl -n mmm-suplus-test get pods | grep -q character-library-metadata-bastion
while [ "$?" == "0" ]
do
kubectl -n mmm-suplus-test get pods | grep -q character-library-metadata-bastion
done
kubectl create -f /tmp/test/character-library-metadata-bastion/character-library-metadata-bastion.yaml --record
kubectl -n mmm-suplus-test get svc | grep -q character-library-metadata-bastion
if [ "$?" == "0" ];then
echo "character-library-metadata-bastion service update success!"
else
echo "character-library-metadata-bastion service update fail!"
fi
kubectl -n mmm-suplus-test get pods | grep -q character-library-metadata-bastion
if [ "$?" == "0" ];then
echo "character-library-metadata-bastion deployment update success!"
else
echo "character-library-metadata-bastion deployment update fail!"
fi
fi
\ No newline at end of file
... ...
version: v1
kind: HttpApi
metadata:
service: querySet
path: /query-sets
endpoints:
- method: createQuerySet
route:
post: /
- method: updateQuerySet
route:
put: /{Id}
- method: getQuerySet
route:
get: /{Id}
- method: removeQuerySet
route:
delete: /{Id}
- method: listQuerySet
route:
get: /
params:
- name: offset
- name: limit
- method: changeStatus
route:
post: /change-status
- method: copy
route:
post: /copy
- method: dependencyGraph
route:
post: /dependency-ggraph
- method: move
route:
post: /move
- method: rename
route:
post: /rename
- method: searchQuerySet
route:
post: /search
... ...
version: v1
kind: Attribute
metadata:
name: context
description: 扩展
type:
array: string
... ...
version: v1
kind: Attribute
metadata:
name: flag
description: 标识 1.group 2.query-set
type:
primitive: string
... ...
version: v1
kind: Attribute
metadata:
name: name
description: 名称
type:
primitive: string
... ...
version: v1
kind: Attribute
metadata:
name: pinName
description: 拼音(排序使用)
type:
primitive: string
... ...
version: v1
kind: Attribute
metadata:
name: queryComponents
description: 查询组件
type:
array: string
... ...
version: v1
kind: Attribute
metadata:
name: querySetId
description: 查询集合ID
type:
primitive: int
... ...
version: v1
kind: Attribute
metadata:
name: querySetInfo
description: 查询集合信息
type:
primitive: string
... ...
version: v1
kind: Attribute
metadata:
name: status
description: 状态 1:启用 2:关闭 (子过程默认启用)
type:
primitive: int
... ...
version: v1
kind: Attribute
metadata:
name: type
description: schema:方案 sub-process:子过程
type:
primitive: string
... ...
version: v1
kind: Schema
metadata:
name: querySet
description: 查询集合
attributes:
- ref: querySetId
required: true
- ref: type
required: true
- ref: flag
required: true
- ref: name
required: true
- ref: pinName
required: true
- ref: parentId
required: true
- ref: status
required: true
- ref: querySetInfo
required: true
- ref: queryComponents
required: true
- ref: sort
required: true
- ref: createdAt
required: true
- ref: updatedAt
required: true
- ref: deletedAt
required: true
- ref: context
required: true
... ...
version: v1
kind: Method
metadata:
name: changeStatus
type: command
description: 修改状态
payload:
- ref: querySetId
required: true
- ref: status
required: true
result:
- name: querySet
type:
schema: querySet
required: true
... ...
version: v1
kind: Method
metadata:
name: copy
type: command
description: 移动
payload:
- ref: flag
required: true
- ref: parentId
required: true
- ref: name
required: true
- ref: querySetId
required: true
result:
- name: querySet
type:
schema: querySet
required: true
... ...
version: v1
kind: Method
metadata:
name: createQuerySet
type: command
description: 创建查询集合服务
payload:
- ref: type
required: true
- ref: flag
required: true
- ref: name
required: true
- ref: parentId
required: false
result:
- name: querySet
type:
schema: querySet
required: true
... ...
version: v1
kind: Method
metadata:
name: dependencyGraph
type: query
description: 依赖关系图
payload:
- ref: querySetId
required: true
result:
- name: querySet
type:
schema: querySet
required: true
... ...
version: v1
kind: Method
metadata:
name: getQuerySet
type: query
description: 返回查询集合服务
payload:
- ref: querySetId
required: true
result:
- name: querySet
type:
schema: querySet
required: true
... ...
version: v1
kind: Method
metadata:
name: listQuerySet
type: query
description: 返回查询集合服务列表
payload:
- ref: offset
required: true
- ref: limit
required: true
result:
- ref: count
required: true
- name: querySets
type:
array: querySet
required: true
... ...
version: v1
kind: Method
metadata:
name: move
type: command
description: 移动
payload:
- ref: querySetId
required: true
- ref: parentId
required: true
result:
- name: querySet
type:
schema: querySet
required: true
... ...
version: v1
kind: Method
metadata:
name: removeQuerySet
type: command
description: 移除查询集合服务
payload:
- ref: querySetId
required: true
result:
- name: querySet
type:
schema: querySet
required: true
... ...
version: v1
kind: Method
metadata:
name: rename
type: command
description: 重命名
payload:
- ref: querySetId
required: true
- ref: name
required: true
result:
- name: querySet
type:
schema: querySet
required: true
... ...
version: v1
kind: Method
metadata:
name: searchQuerySet
type: query
description: 返回查询集合服务列表
payload:
- ref: offset
required: true
- ref: limit
required: true
result:
- ref: count
required: true
- name: querySets
type:
array: querySet
required: true
... ...
version: v1
kind: Method
metadata:
name: updateQuerySet
type: command
description: 更新查询集合服务
payload:
- ref: querySetId
required: true
result:
- name: querySet
type:
schema: querySet
required: true
... ...
version: v1
kind: Service
metadata:
name: querySet
description: 查询集合服务
... ...
... ... @@ -3,51 +3,57 @@ module gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion
go 1.16
require (
github.com/Shopify/sarama v1.30.0 // indirect
github.com/ajg/form v1.5.1 // indirect
github.com/aliyun/aliyun-oss-go-sdk v2.2.7+incompatible
github.com/beego/beego/v2 v2.0.1
github.com/bwmarrin/snowflake v0.3.0
github.com/dgrijalva/jwt-go v3.2.0+incompatible
github.com/extrame/xls v0.0.1
github.com/fasthttp-contrib/websocket v0.0.0-20160511215533-1f3b11f56072 // indirect
github.com/fatih/structs v1.1.0 // indirect
github.com/gavv/httpexpect v2.0.0+incompatible
github.com/go-gota/gota v0.12.0
github.com/go-pg/pg/v10 v10.10.6
github.com/go-redis/redis v6.15.9+incompatible
github.com/google/go-cmp v0.5.7 // indirect
github.com/google/go-querystring v1.1.0 // indirect
github.com/google/gofuzz v1.2.0
github.com/google/uuid v1.3.0
github.com/imkira/go-interpol v1.1.0 // indirect
github.com/gookit/event v1.0.6
github.com/linmadan/egglib-go v0.0.0-20210313060205-8b5e456b11f7
github.com/mattn/go-colorable v0.1.9 // indirect
github.com/mattn/go-isatty v0.0.14 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 // indirect
github.com/moul/http2curl v1.0.0 // indirect
github.com/onsi/ginkgo v1.16.5
github.com/onsi/gomega v1.18.1
github.com/prometheus/client_golang v1.12.2 // indirect
github.com/sergi/go-diff v1.2.0 // indirect
github.com/patrickmn/go-cache v2.1.0+incompatible
github.com/shopspring/decimal v1.3.1
github.com/smartystreets/goconvey v1.7.2 // indirect
github.com/stretchr/testify v1.7.1
github.com/xuri/excelize/v2 v2.6.0
github.com/zeromicro/go-zero v1.3.4
golang.org/x/text v0.3.7
gorm.io/driver/mysql v1.3.6
gorm.io/driver/postgres v1.3.9
gorm.io/gorm v1.23.8
)
require (
github.com/ajg/form v1.5.1 // indirect
github.com/fasthttp-contrib/websocket v0.0.0-20160511215533-1f3b11f56072 // indirect
github.com/fatih/structs v1.1.0 // indirect
github.com/go-sql-driver/mysql v1.7.0 // indirect
github.com/google/go-querystring v1.1.0 // indirect
github.com/imkira/go-interpol v1.1.0 // indirect
github.com/moul/http2curl v1.0.0 // indirect
github.com/sergi/go-diff v1.2.0 // indirect
github.com/sirupsen/logrus v1.9.0 // indirect
github.com/smartystreets/goconvey v1.7.2 // indirect
github.com/valyala/fasthttp v1.38.0 // indirect
github.com/xeipuuv/gojsonschema v1.2.0 // indirect
github.com/xuri/excelize/v2 v2.6.0
github.com/xuri/efp v0.0.0-20220603152613-6918739fd470 // indirect
github.com/yalp/jsonpath v0.0.0-20180802001716-5cc68e5049a0 // indirect
github.com/yudai/gojsondiff v1.0.0 // indirect
github.com/yudai/golcs v0.0.0-20170316035057-ecda9a501e82 // indirect
github.com/yudai/pp v2.0.1+incompatible // indirect
go.uber.org/automaxprocs v1.5.1 // indirect
golang.org/x/net v0.0.0-20220421235706-1d1ef9303861 // indirect
golang.org/x/sys v0.0.0-20220429233432-b5fbb4746d32 // indirect
golang.org/x/text v0.3.7
golang.org/x/tools v0.1.5 // indirect
google.golang.org/protobuf v1.28.0 // indirect
gorm.io/driver/mysql v1.3.6
gorm.io/driver/postgres v1.3.9
gorm.io/gorm v1.23.8
github.com/zeromicro/go-queue v1.1.6
golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d // indirect
)
replace (
github.com/extrame/xls v0.0.1 => github.com/tiptok/xls v1.0.1
github.com/linmadan/egglib-go v0.0.0-20210313060205-8b5e456b11f7 => github.com/tiptok/egglib-go v0.0.0-20220421085958-9682d0ac42c1
github.com/go-sql-driver/mysql v1.7.0 => github.com/StarRocks/go-mysql-driver v1.7.0
//github.com/linmadan/egglib-go v0.0.0-20210313060205-8b5e456b11f7 => github.com/tiptok/egglib-go v0.0.0-20220421085958-9682d0ac42c1
github.com/linmadan/egglib-go v0.0.0-20210313060205-8b5e456b11f7 => github.com/tiptok/egglib-go v1.0.2
)
... ...
... ... @@ -5,19 +5,18 @@ import (
"github.com/beego/beego/v2/server/web"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/crontab"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/constant"
_ "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/constant"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/pg"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/redis"
_ "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/redis"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/starrocks"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/log"
"time"
_ "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/constant"
_ "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/redis"
_ "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/log"
_ "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/port/beego"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/port/event"
)
const Version = "v1.0.1"
const Version = "v1.4.0"
func main() {
defer func() {
... ... @@ -28,6 +27,7 @@ func main() {
log.InitLogHook(constant.ENABLE_KAFKA_LOG, true)
redis.InitRedis()
redis.InitZeroCoreRedis()
pg.Init()
if err := starrocks.Init(); err != nil {
log.Logger.Error(err.Error())
... ... @@ -35,8 +35,7 @@ func main() {
cron := crontab.NewCrontabService(nil)
cron.StartCrontabTask()
defer cron.StopCrontabTask()
time.Sleep(time.Second)
event.Start()
log.Logger.Info("Service:" + constant.SERVICE_NAME)
log.Logger.Info("Version:" + Version)
log.Logger.Info("server start!")
... ...
... ... @@ -4,9 +4,12 @@ import (
"context"
"fmt"
"github.com/beego/beego/v2/task"
pgTransaction "github.com/linmadan/egglib-go/transaction/pg"
"github.com/linmadan/egglib-go/utils/xtime"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/factory"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/dao"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/starrocks"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/log"
"io/fs"
"os"
... ... @@ -28,6 +31,9 @@ func (crontabService *CrontabService) initTask() {
autoRemovePublicDownloadFile := task.NewTask("定时清理缓存文件", "0 20 */1 * * *", AutoRemovePublicDownloadFile)
task.AddTask("autoRemovePublicDownloadFile", autoRemovePublicDownloadFile)
autoRemoveTemporaryTable := task.NewTask("定时清理临时表", "0 57 */1 * * *", AutoRemoveTemporaryTable)
task.AddTask("autoRemoveTemporaryTable", autoRemoveTemporaryTable)
}
func (crontabService *CrontabService) StartCrontabTask() {
... ... @@ -115,3 +121,49 @@ func AutoRemovePublicDownloadFile(ctx context.Context) error {
}
return nil
}
func AutoRemoveTemporaryTable(ctx context.Context) error {
defer func() {
if r := recover(); r != nil {
log.Logger.Error(fmt.Sprintf("%v", r), map[string]interface{}{"task": "定时清理过期临时文件记录"})
}
}()
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return err
}
if err := transactionContext.StartTransaction(); err != nil {
return err
}
defer func() {
if err != nil {
log.Logger.Error("【定时清理临时表】 失败:" + err.Error())
}
transactionContext.RollbackTransaction()
}()
log.Logger.Debug("【定时清理临时表】 启动")
end := xtime.New(time.Now()).BeginningOfDay().Add(-time.Hour * 12)
begin := end.AddDate(0, 0, -7)
tableRepository, _, _ := factory.FastPgTable(transactionContext, 0)
_, tables, err := tableRepository.Find(map[string]interface{}{"beginTime": begin, "endTime": end, "tableTypes": []string{domain.TemporaryTable.ToString()}})
if err != nil {
return nil
}
for i, t := range tables {
if err = dao.TableDelete(transactionContext.(*pgTransaction.TransactionContext), t.TableId, domain.TemporaryTable); err != nil {
log.Logger.Error(err.Error())
return nil
}
log.Logger.Info(fmt.Sprintf("序号:%d 清理临时表 %v", i, t.SQLName))
if err = starrocks.DropView(starrocks.DB, t.SQLName); err != nil {
log.Logger.Error(err.Error())
return nil
}
}
if err = transactionContext.CommitTransaction(); err != nil {
return err
}
return nil
}
... ...
package command
import "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
type TableEventCommand struct {
EventTable *domain.EventTable
}
... ...
package service
import (
"fmt"
"github.com/linmadan/egglib-go/core/application"
pgTransaction "github.com/linmadan/egglib-go/transaction/pg"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/event/command"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/factory"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/constant"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/api/digitalLib"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/broker"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/domainService"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/log"
"math"
"time"
)
func (tableEventService *TableEventService) DigitalPlatformEventSubscribe(ctx *domain.Context, cmd *command.TableEventCommand) (interface{}, error) {
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
var (
dataChanged = true
structChanged = true
ok bool
tableId int
)
event := cmd.EventTable
if tableId = resolveTableId(event); tableId == 0 {
return nil, nil
}
if event.Type == domain.TableApplyOnEvent {
dataChanged = false
}
var notifyData = &NotifyData{
DataChanged: dataChanged,
StructChanged: structChanged,
TableId: tableId,
Event: event.Type.ToString(),
Metadata: event.Metadata,
}
// 表类型
tableRepository, table, _ := factory.FastPgTable(transactionContext, tableId)
if table == nil && event.Table != nil {
table = event.Table
notifyData.CompanyId = table.Context.CompanyId
}
notifyData.SetType(event, table)
// 依赖的表 \ 依赖的查询集合
_, tables, err := tableRepository.Find(map[string]interface{}{"context": event.Context, "tableTypesNotIn": []string{domain.TemporaryTable.ToString(), domain.ExcelTable.ToString()}})
if err != nil {
return nil, err
}
tableDependencyService, _ := domainService.NewTableDependencyService(transactionContext.(*pgTransaction.TransactionContext))
tableDependTree := tableDependencyService.TableDependTree(tables, tableId)
tree := tableDependTree.Tree
querySetRepository, _, _ := factory.FastPgQuerySet(transactionContext, 0)
var mapTableQuerySet = make(map[int]*domain.QuerySet)
if len(tree) > 0 {
_, querySets, _ := querySetRepository.Find(map[string]interface{}{
"types": []string{domain.SchemaTable.ToString(), domain.CalculateItem.ToString(), domain.CalculateSet.ToString()},
"bindTableIds": tree,
"status": domain.StatusOn,
})
for _, q := range querySets {
mapTableQuerySet[q.QuerySetInfo.BindTableId] = q
}
}
// 过滤出需要推送的表
for i := range tree {
table, ok = tableDependencyService.TableMap[tree[i]]
if !ok {
continue
}
if notifyData.CompanyId == 0 {
notifyData.CompanyId = table.Context.CompanyId
}
switch table.TableType {
case domain.MainTable.ToString(), domain.SubTable.ToString(), domain.SideTable.ToString():
if table.TableInfo != nil {
applyOn := domain.ModuleDigitalCenter | domain.ModuleChartTemplate
if table.TableInfo.ApplyOnModule&applyOn == 0 {
continue
}
}
break
case domain.SubProcessTable.ToString(), domain.CalculateTable.ToString():
continue
case domain.SchemaTable.ToString(), domain.CalculateSet.ToString(), domain.CalculateItem.ToString():
var querySet *domain.QuerySet
if querySet, ok = mapTableQuerySet[tree[i]]; !ok {
continue
}
// 不是当前的查询集。且状态为关闭的都补推送
if querySet.Status != domain.StatusOn && querySet.QuerySetInfo.BindTableId != 0 && querySet.QuerySetInfo.BindTableId != tableId {
continue
}
}
notifyData.TableAffectedList = append(notifyData.TableAffectedList, tree[i])
}
// 包含自己
if !exist(notifyData.TableAffectedList, tableId) {
notifyData.TableAffectedList = append(notifyData.TableAffectedList, tableId)
}
// 通过消息队列发送
if err = tableEventService.send(notifyData, tableEventService.sendBroker); err != nil {
return nil, err
}
if err = transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return nil, nil
}
func (tableEventService *TableEventService) send(notifyData *NotifyData, sendFunc func(*NotifyData) error) error {
var err error
if err = sendFunc(notifyData); err != nil {
log.Logger.Error(fmt.Sprintf("通知数控失败:%s", err.Error()))
if t, ok := notifyData.Retry(); ok {
tableEventService.TimingWheel.SetTimer(notifyData.Key(), &notifyData, t)
log.Logger.Debug(fmt.Sprintf("通知数控重试 key:%s wait:%vs", notifyData.Key(), t.Seconds()))
}
}
return err
}
func (tableEventService *TableEventService) sendHttp(notifyData *NotifyData) error {
var err error
lib := digitalLib.NewDigitalLib(constant.DIGITAL_SERVER_HOST)
if _, err = lib.SyncNotice(digitalLib.RequestSyncNotice{Body: notifyData}); err != nil {
return err
}
return nil
}
func (tableEventService *TableEventService) sendBroker(notifyData *NotifyData) error {
var err error
if err = broker.Push(constant.KAFKA_HOST, constant.TOPIC_TABLE_DATA_SYNC, notifyData); err != nil {
return err
}
return nil
}
func resolveTableId(event *domain.EventTable) (tableId int) {
switch event.Type {
case domain.TableDataImportEvent, domain.TableDataEditEvent, domain.TableDeleteEvent, domain.TableStructEditEvent:
tableId = event.Table.TableId
case domain.QuerySetUpdateEvent, domain.QuerySetUpdateRenameEvent:
if event.QuerySet.Status != domain.StatusOn {
return
}
if !domain.AssertTableType(event.QuerySet.Type, domain.SchemaTable, domain.CalculateItem, domain.CalculateSet) {
return
}
tableId = event.QuerySet.QuerySetInfo.BindTableId
case domain.QuerySetUpdateStatusEvent:
if !domain.AssertTableType(event.QuerySet.Type, domain.SchemaTable, domain.CalculateItem, domain.CalculateSet) {
return
}
tableId = event.QuerySet.QuerySetInfo.BindTableId
case domain.TableApplyOnEvent:
tableId = event.Table.TableId
case domain.QuerySetDeleteEvent:
tableId = event.Table.TableId
}
return tableId
}
func exist(idList []int, target int) bool {
found := false
for _, id := range idList {
if id == target {
found = true
}
}
return found
}
type NotifyData struct {
DataChanged bool `json:"dataChanged"` // 数据有变化
StructChanged bool `json:"structChanged"` // 结构有变化
TableId int `json:"tableId"` // 表ID
TableType string `json:"tableType"` // 表类型:导入模块(主表,副表,分表) 拆解模块(方案、子过程、计算表) 计算模块(计算项,计算集)
ObjectType string `json:"objectType"` // 导入模块、拆解模块、计算模块
CompanyId int `json:"companyId"` // 公司
Event string `json:"event"` // 事件名称
TableAffectedList []int `json:"tableAffectedList"` // 级联影响到的表
Metadata map[string]interface{} `json:"metadata"` // 元数据
sendRetry int
}
func (n *NotifyData) SetType(event *domain.EventTable, table *domain.Table) {
var tableType string
// 表类型
if tableType == "" && table != nil {
tableType = table.TableType
}
if tableType == "" && event.QuerySet != nil {
tableType = event.QuerySet.Type
}
n.TableType = domain.EnumsDescription(domain.ObjectTypeMap, tableType)
if table != nil {
switch domain.TableType(tableType) {
case domain.MainTable, domain.SubTable, domain.SideTable:
n.ObjectType = "导入模块"
case domain.SchemaTable, domain.SubProcessTable, domain.CalculateTable:
n.ObjectType = "拆解模块"
case domain.CalculateItem, domain.CalculateSet:
n.ObjectType = "计算模块"
}
}
}
func (n *NotifyData) Key() string {
return fmt.Sprintf("delay:notify:table:%d", n.TableId)
}
func (n *NotifyData) Retry() (time.Duration, bool) {
n.sendRetry++
if n.sendRetry > 3 {
return n.Delay(), false
}
if n.sendRetry == 1 {
return n.Delay(), true
}
return n.Delay() * time.Duration(int(math.Pow(float64(2), float64(n.sendRetry)))), true
}
func (n *NotifyData) Delay() time.Duration {
return time.Second * 10
}
func (n *NotifyData) RetryTime() int {
return n.sendRetry
}
... ...
package service
import (
"fmt"
"github.com/linmadan/egglib-go/core/application"
pgTransaction "github.com/linmadan/egglib-go/transaction/pg"
"github.com/zeromicro/go-zero/core/collection"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/event/command"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/factory"
tablecommand "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/table/command"
tableservice "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/table/service"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/api/digitalLib"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/cache"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/domainService"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/log"
"time"
)
type TableEventService struct {
TimingWheel *collection.TimingWheel
}
func (tableEventService *TableEventService) Handler(ctx *domain.Context, cmd *command.TableEventCommand) (interface{}, error) {
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
data := cmd.EventTable
tableId := 0
switch data.Type {
case domain.TableDataImportEvent, domain.TableDataEditEvent, domain.TableDeleteEvent, domain.TableStructEditEvent:
tableId = data.Table.TableId
case domain.QuerySetUpdateEvent:
tableId = data.QuerySet.QuerySetInfo.BindTableId
default:
return nil, err
}
if tableId == 0 {
return nil, nil
}
// tableId 相关联的
tableRepository, _, _ := factory.FastPgTable(transactionContext, 0)
_, tables, err := tableRepository.Find(map[string]interface{}{"context": data.Context, "tableTypesNotIn": []string{domain.TemporaryTable.ToString(), domain.ExcelTable.ToString()}})
if err != nil {
return nil, err
}
tableDependencyService, _ := domainService.NewTableDependencyService(transactionContext.(*pgTransaction.TransactionContext))
tableDependTree := tableDependencyService.TableDependTree(tables, tableId)
tree := tableDependTree.Tree
tableService := tableservice.NewTableService(nil)
for i := range tree {
cache.DefaultDataTableCacheService.DeleteDataTable(tree[i])
// fresh cache
tableService.TablePreview(data.Context, &tablecommand.TablePreviewCommand{
TableId: tree[i],
ObjectType: domain.ObjectMetaTable,
PageSize: 10000,
PageNumber: 0,
UseCache: true,
})
}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return nil, nil
}
func NewTableEventService(options map[string]interface{}) *TableEventService {
svr := &TableEventService{}
delayNotifyTimingWheel, _ := collection.NewTimingWheel(time.Second, 10, svr.TimingWheelFunc)
svr.TimingWheel = delayNotifyTimingWheel
return svr
}
func (tableEventService *TableEventService) TimingWheelFunc(key, value interface{}) {
v, ok := value.(*NotifyData)
if !ok {
return
}
lib := digitalLib.NewDigitalLib("")
if _, err := lib.SyncNotice(digitalLib.RequestSyncNotice{Body: v}); err != nil {
log.Logger.Error(fmt.Sprintf("通知数控失败:%s", err.Error()))
if t, ok := v.Retry(); ok {
if err = tableEventService.TimingWheel.SetTimer(v.Key(), v, t); err == nil {
log.Logger.Debug(fmt.Sprintf("通知数控重试(%d) key:%s wait:%vs", v.RetryTime(), v.Key(), t.Seconds()))
return
}
}
}
tableEventService.TimingWheel.RemoveTimer(v.Key())
}
... ...
package service
import (
"errors"
"fmt"
"github.com/linmadan/egglib-go/core/application"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/event/command"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/factory"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/log"
"reflect"
"sort"
)
func (tableEventService *TableEventService) HandlerTableAffectedMarkToConflictStatus(ctx *domain.Context, cmd *command.TableEventCommand) (interface{}, error) {
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
data := cmd.EventTable
tableId := 0
switch data.Type {
case domain.TableStructEditEvent, domain.TableDeleteEvent:
tableId = data.Table.TableId
case domain.QuerySetUpdateEvent:
// 结构变更才报冲突
if !checkStructChange(cmd) {
return nil, err
}
tableId = data.QuerySet.QuerySetInfo.BindTableId
case domain.QuerySetUpdateRenameEvent:
tableId = data.QuerySet.QuerySetInfo.BindTableId
default:
return nil, err
}
if tableId == 0 {
return nil, nil
}
// tableId 相关联的
tableRepository, _, _ := factory.FastPgTable(transactionContext, 0)
_, tables, err := tableRepository.Find(map[string]interface{}{"context": ctx, "dependencyTable": tableId, "tableTypesNotIn": []string{domain.TemporaryTable.ToString()}})
if errors.Is(err, domain.ErrorNotFound) {
return nil, nil
}
tableIds := make([]int, 0)
for _, table := range tables {
tableIds = append(tableIds, table.TableId)
}
if len(tableIds) == 0 {
return nil, nil
}
querySetRepository, _, _ := factory.FastPgQuerySet(transactionContext, 0)
_, querySets, _ := querySetRepository.Find(map[string]interface{}{"context": ctx, "bindTableIds": tableIds})
for _, querySet := range querySets {
log.Logger.Debug(fmt.Sprintf("【集合状态更新】 id:%v name:%v ReadyStatus:1", querySet.QuerySetId, querySet.Name))
querySet.QuerySetInfo.WithConflictStatus()
_, err = querySetRepository.Save(querySet)
if err != nil {
return nil, err
}
}
//tableDependencyService, _ := domainService.NewTableDependencyService(transactionContext.(*pgTransaction.TransactionContext))
//tableDependTree := tableDependencyService.TableDependTree(tables, tableId)
//tree := tableDependTree.Tree
//
//querySetRepository, _, _ := factory.FastPgQuerySet(transactionContext, 0)
//if len(tree) > 0 {
// _, querySets, _ := querySetRepository.Find(map[string]interface{}{
// "types": []string{domain.SchemaTable.ToString(), domain.SubProcessTable.ToString(), domain.CalculateTable.ToString()},
// "bindTableIds": tree,
// })
// for _, querySet := range querySets {
// log.Logger.Debug(fmt.Sprintf("【集合状态更新】 id:%v name:%v 标记冲突", querySet.QuerySetId, querySet.Name))
// querySet.QuerySetInfo.WithConflictStatus()
// _, err = querySetRepository.Save(querySet)
// if err != nil {
// return nil, err
// }
// }
//}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return nil, nil
}
func checkStructChange(cmd *command.TableEventCommand) bool {
var (
newSet = cmd.EventTable.QuerySet
oldSet = cmd.EventTable.OldQuerySet
newTable = cmd.EventTable.Table
oldTable = cmd.EventTable.OldTable
)
if newSet == nil || oldSet == nil {
return false
}
var (
t string = newSet.Type
)
switch t {
case domain.SchemaTable.ToString(), domain.SubProcessTable.ToString():
// 第一步表有变更
newSetDepTables := newSet.GetDependencyTables(newSet.QueryComponents)
oldSetDepTables := newSet.GetDependencyTables(oldSet.QueryComponents)
sort.SliceStable(newSetDepTables, func(i, j int) bool {
return newSetDepTables[i] < newSetDepTables[j]
})
sort.SliceStable(oldSetDepTables, func(i, j int) bool {
return oldSetDepTables[i] < oldSetDepTables[j]
})
if !reflect.DeepEqual(newSetDepTables, oldSetDepTables) {
log.Logger.Debug(fmt.Sprintf("方案/过程:%v 依赖变更 %v -> %v", t, oldSetDepTables, newSetDepTables))
return true
}
case domain.CalculateTable.ToString():
case domain.CalculateItem.ToString(), domain.CalculateSet.ToString():
return false
}
if newTable == nil || oldTable == nil {
return false
}
// 第二步判断字段是否有变更
newTableFields := tableFields(newTable)
oldTableFields := tableFields(oldTable)
sort.Strings(newTableFields)
sort.Strings(oldTableFields)
if !reflect.DeepEqual(newTableFields, oldTableFields) {
log.Logger.Debug(fmt.Sprintf("计算表:%v 结构变更 %v -> %v", t, oldTableFields, newTableFields))
return true
}
return false
}
func tableFields(t *domain.Table) []string {
var result = make([]string, 0)
for _, f := range t.Fields(false) {
result = append(result, f.Name)
}
return result
}
... ...
... ... @@ -4,7 +4,6 @@ import (
"github.com/linmadan/egglib-go/core/application"
"github.com/linmadan/egglib-go/transaction/pg"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain/bytecore"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/domainService"
)
... ... @@ -58,6 +57,6 @@ func CreateTableEditDataService(transactionContext application.TransactionContex
}
// 字库核心
func CreateByteCoreService(transactionContext application.TransactionContext) (bytecore.ByteLibService, error) {
func CreateByteCoreService(transactionContext application.TransactionContext) (domain.ByteLibService, error) {
return domainService.ByteCore, nil
}
... ...
... ... @@ -2,7 +2,9 @@ package factory
import (
"github.com/linmadan/egglib-go/core/application"
pgTransaction "github.com/linmadan/egglib-go/transaction/pg"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/domainService"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/starrocks"
)
... ... @@ -10,6 +12,18 @@ func FastError(err error) error {
return application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
func FastErrorResponse(err error, args ...interface{}) interface{} {
var response = make(map[string]interface{})
response["internalErr"] = err.Error()
for i := 0; i < len(args); i += 2 {
if i+1 >= len(args) {
break
}
response[args[i].(string)] = args[i+1]
}
return response
}
func FastDataTable(options starrocks.QueryOptions) (*domain.DataTable, error) {
var err error
// 待优化分批下载,压缩
... ... @@ -25,3 +39,7 @@ func FastDataTable(options starrocks.QueryOptions) (*domain.DataTable, error) {
}
return dataTable, nil
}
func FastQuerySetServices(transactionContext application.TransactionContext) (*domainService.QuerySetService, error) {
return domainService.NewQuerySetService(transactionContext.(*pgTransaction.TransactionContext))
}
... ...
... ... @@ -23,9 +23,9 @@ func FastPgFile(transactionContext application.TransactionContext, id int) (doma
if id > 0 {
if mod, err = rep.FindOne(map[string]interface{}{"fileId": id}); err != nil {
if err == domain.ErrorNotFound {
return nil, nil, application.ThrowError(application.RES_NO_FIND_ERROR, "该文件不存在")
return rep, nil, application.ThrowError(application.RES_NO_FIND_ERROR, "该文件不存在")
}
return nil, nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
return rep, nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
}
//if err = fastPgDataAuth(transactionContext, mod, options...); err != nil {
... ... @@ -52,9 +52,9 @@ func FastPgTable(transactionContext application.TransactionContext, id int) (dom
if id > 0 {
if mod, err = rep.FindOne(map[string]interface{}{"tableId": id}); err != nil {
if err == domain.ErrorNotFound {
return nil, nil, application.ThrowError(application.RES_NO_FIND_ERROR, "该表格不存在")
return rep, nil, application.ThrowError(application.RES_NO_FIND_ERROR, "该表格不存在")
}
return nil, nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
return rep, nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
}
return rep, mod, err
... ... @@ -78,9 +78,9 @@ func FastPgLog(transactionContext application.TransactionContext, id int) (domai
if id > 0 {
if mod, err = rep.FindOne(map[string]interface{}{"logId": id}); err != nil {
if err == domain.ErrorNotFound {
return nil, nil, application.ThrowError(application.RES_NO_FIND_ERROR, "该日志不存在")
return rep, nil, application.ThrowError(application.RES_NO_FIND_ERROR, "该日志不存在")
}
return nil, nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
return rep, nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
}
return rep, mod, err
... ... @@ -104,9 +104,35 @@ func FastPgMappingRule(transactionContext application.TransactionContext, id int
if id > 0 {
if mod, err = rep.FindOne(map[string]interface{}{"mappingRuleId": id}); err != nil {
if err == domain.ErrorNotFound {
return nil, nil, application.ThrowError(application.RES_NO_FIND_ERROR, "该匹配规则不存在")
return rep, nil, application.ThrowError(application.RES_NO_FIND_ERROR, "该匹配规则不存在")
}
return nil, nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
return rep, nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
}
return rep, mod, err
}
// FastPgQuerySet 快速返回查询集合
//
// transactionContext 事务
// id 对象唯一标识
func FastPgQuerySet(transactionContext application.TransactionContext, id int) (domain.QuerySetRepository, *domain.QuerySet, error) {
var rep domain.QuerySetRepository
var mod *domain.QuerySet
var err error
if value, err := CreateQuerySetRepository(map[string]interface{}{
"transactionContext": transactionContext,
}); err != nil {
return nil, nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
} else {
rep = value
}
if id > 0 {
if mod, err = rep.FindOne(map[string]interface{}{"querySetId": id}); err != nil {
if err == domain.ErrorNotFound {
return rep, nil, application.ThrowError(application.RES_NO_FIND_ERROR, "该查询集合不存在")
}
return rep, nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
}
return rep, mod, err
... ...
... ... @@ -37,3 +37,11 @@ func CreateMappingRuleRepository(options map[string]interface{}) (domain.Mapping
}
return repository.NewMappingRuleRepository(transactionContext)
}
func CreateQuerySetRepository(options map[string]interface{}) (domain.QuerySetRepository, error) {
var transactionContext *pg.TransactionContext
if value, ok := options["transactionContext"]; ok {
transactionContext = value.(*pg.TransactionContext)
}
return repository.NewQuerySetRepository(transactionContext)
}
... ...
package command
import "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
type AppTableFileAppendDataCommand struct {
Name string `json:"name"`
// name 字段中文名
Fields []*domain.Field `json:"fields"`
// 数据列表 key:name(字段中文名) value:值(字符串类型)
Data []map[string]string `json:"data"`
AppKey string `json:"appKey"`
// 追加表数据标识 true:往应用表里面追加数据 false:跳过
AppendTableDataFlag bool `json:"appendTableDataFlag"`
}
... ...
package command
import "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
type CreateAppTableFileCommand struct {
Name string `json:"name"`
// name 字段中文名
Fields []*domain.Field `json:"fields"`
// 数据列表 key:name(字段中文名) value:值(字符串类型)
Data []map[string]string `json:"data"`
// 生成表标识 true:实例化一个表 false:跳过
GenerateTableFlag bool `json:"generateTableFlag"`
}
... ...
package command
type DeleteAppTableFileCommand struct {
Name string `json:"name"`
AppKey string `json:"appKey"`
}
... ...
package command
import "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
type UpdateAppTableFileCommand struct {
Name string `json:"name"`
AppKey string `json:"appKey"`
AddFields []*domain.Field `json:"addFields"`
}
... ...
... ... @@ -19,7 +19,14 @@ type AppendDataToTableCommand struct {
}
func (cmd *AppendDataToTableCommand) Valid(validation *validation.Validation) {
newMappingFields := make([]*domain.MappingField, 0)
for i := range cmd.MappingFields {
if len(cmd.MappingFields[i].VerifiedFileFieldName) == 0 {
continue
}
newMappingFields = append(newMappingFields, cmd.MappingFields[i])
}
cmd.MappingFields = newMappingFields
}
func (cmd *AppendDataToTableCommand) ValidateCommand() error {
... ...
package command
import (
"fmt"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type CheckFileVerifyStatusCommand struct {
// 文件ID
FileId int `cname:"文件ID" json:"objectId" valid:"Required"`
}
func (cmd *CheckFileVerifyStatusCommand) Valid(validation *validation.Validation) {
}
func (cmd *CheckFileVerifyStatusCommand) ValidateCommand() error {
valid := validation.Validation{}
b, err := valid.Valid(cmd)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(cmd).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
... ... @@ -17,14 +17,37 @@ type CreateFileCommand struct {
Url string `cname:"文件地址" json:"url" valid:"Required"`
// 文件大小 单位KB
FileSize int `cname:"文件大小" json:"fileSize" valid:"Required"`
// 文件来源
FileFrom string `json:"-"`
// AppKey
AppKey string `json:"-"`
// 生成表标识 true:实例化一个表 false:跳过
GenerateTableFlag bool `json:"-"`
// name 字段中文名
Fields []*domain.Field `json:"-"`
}
var MaxFileSize = 50 * 1024 * 1024
func (createFileCommand *CreateFileCommand) Valid(validation *validation.Validation) {
ext := filepath.Ext(createFileCommand.Name)
if !(ext == domain.XLS || ext == domain.XLSX) {
validation.Error(fmt.Sprintf("仅支持文件格式 xls 、 xlsx"))
validation.Error("仅支持文件格式 xls 、 xlsx")
return
}
if createFileCommand.FileSize > 0 && createFileCommand.FileSize > MaxFileSize {
validation.Error("文件大小超过50M")
return
}
if createFileCommand.GenerateTableFlag {
for _, f := range createFileCommand.Fields {
if err := f.Valid(); err != nil {
validation.Error(err.Error())
return
}
}
}
}
func (createFileCommand *CreateFileCommand) ValidateCommand() error {
... ...
... ... @@ -15,10 +15,11 @@ type EditDataTableCommand struct {
//
//Fields []*domain.Field
domain.EditTableRequest
HeaderRow int `json:"headerRow"` // 行号 默认:0
}
func (editDataTableCommand *EditDataTableCommand) Valid(validation *validation.Validation) {
if len(editDataTableCommand.ProcessFields) == 0 {
if len(editDataTableCommand.ProcessFieldNames) == 0 {
validation.Error("未选择操作列")
return
}
... ... @@ -30,6 +31,7 @@ func (editDataTableCommand *EditDataTableCommand) Valid(validation *validation.V
validation.Error("文件ID不能为空")
return
}
editDataTableCommand.Where.HeaderRow = domain.SetHeaderRow(editDataTableCommand.HeaderRow)
}
func (editDataTableCommand *EditDataTableCommand) ValidateCommand() error {
... ...
package command
import (
"fmt"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type ExportFileCommand struct {
// 文件ID
FileId int `cname:"文件ID" json:"fileId" valid:"Required"`
}
func (cmd *ExportFileCommand) Valid(validation *validation.Validation) {
}
func (cmd *ExportFileCommand) ValidateCommand() error {
valid := validation.Validation{}
b, err := valid.Valid(cmd)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(cmd).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
... ... @@ -2,7 +2,6 @@ package command
import (
"fmt"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"reflect"
"strings"
... ... @@ -11,11 +10,11 @@ import (
type FlushDataTableCommand struct {
// 文件ID
FileId int `cname:"文件ID" json:"objectId" valid:"Required"`
ObjectId int `cname:"文件ID" json:"objectId" valid:"Required"`
// 记录数
RowCount int `cname:"记录数" json:"rowCount" valid:"Required"`
//RowCount int `cname:"记录数" json:"rowCount" valid:"Required"`
// 数据列
DataFields []*domain.Field `cname:"数据列" json:"fields" valid:"Required"`
//DataFields []*domain.Field `cname:"数据列" json:"fields" valid:"Required"`
}
func (flushDataTableCommand *FlushDataTableCommand) Valid(validation *validation.Validation) {
... ...
... ... @@ -23,6 +23,7 @@ func (loadDataTableCommand *LoadDataTableCommand) Valid(validation *validation.V
if loadDataTableCommand.PageSize == 0 {
loadDataTableCommand.PageSize = 20
}
loadDataTableCommand.HeaderRow = domain.SetHeaderRow(loadDataTableCommand.HeaderRow)
}
func (loadDataTableCommand *LoadDataTableCommand) ValidateCommand() error {
... ...
package command
import (
"fmt"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type PrepareTemporaryFileCommand struct {
// 文件ID
FileId int `cname:"文件ID" json:"fileId" valid:"Required"`
}
func (cmd *PrepareTemporaryFileCommand) Valid(validation *validation.Validation) {
}
func (cmd *PrepareTemporaryFileCommand) ValidateCommand() error {
valid := validation.Validation{}
b, err := valid.Valid(cmd)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(cmd).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
package command
import (
"fmt"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type ResetTableHeaderCommand struct {
// 文件ID
FileId int `cname:"文件ID" json:"objectId" valid:"Required"`
domain.Where
}
func (loadDataTableCommand *ResetTableHeaderCommand) Valid(validation *validation.Validation) {
loadDataTableCommand.HeaderRow = domain.SetHeaderRow(loadDataTableCommand.HeaderRow)
}
func (loadDataTableCommand *ResetTableHeaderCommand) ValidateCommand() error {
valid := validation.Validation{}
b, err := valid.Valid(loadDataTableCommand)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(loadDataTableCommand).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
package dto
import "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
type EditDataTableDto struct {
Code int `json:"errNo,omitempty"`
Error string `json:"errMsg,omitempty"`
InValidCells []domain.InValidCell `json:"inValidCells,omitempty"`
}
func (d *EditDataTableDto) Load(m *domain.DataEditDataTable) *EditDataTableDto {
d.Error = ""
if len(m.InValidCells) > 0 {
d.Error = "类型转换错误"
d.Code = 1001 // 1001:类型转换错误
}
//d.InValidCells = m.InValidCells
return d
}
... ...
... ... @@ -14,14 +14,77 @@ type FileDto struct {
Url string `json:"url"`
// 文件类型
FileType string `json:"fileType"`
// 后缀扩展
Ext string `json:"ext"`
// 创建时间
Time string `json:"time"`
// 行号
HeaderRow int `json:"headerRow"`
// 所属应用
AppKey string `json:"appKey"`
// 表ID
TableId int `json:"tableId"`
// 允许表生成标识 1:允许生成分表 0:不允许
AllowTableGenerateFlag int `json:"-"`
// 允许表处理标识 1:允许 0:不允许
AllowTableProcessFlag int `json:"-"`
// 允许表导出标识 1:允许 0:不允许
AllowTableExportFlag int `json:"-"`
// 标志位 1:可校验 2:可分表生成 4:导出
Flag int `json:"-"`
// 标志位 1:可校验 2:可分表生成 4:导出
Flags []int `json:"flags"`
}
func (d *FileDto) Load(f *domain.File) {
func (d *FileDto) Load(f *domain.File) *FileDto {
if f == nil {
return nil
}
d.FileId = f.FileId
d.Name = f.FileInfo.Name
d.Url = f.FileInfo.Url
d.FileType = f.FileType
d.Time = xtime.New(f.CreatedAt).Local().Format("2006-01-02 15:04:05")
d.Ext = f.FileInfo.Ext
d.Time = xtime.New(f.UpdatedAt).Local().Format("2006-01-02 15:04:05")
d.HeaderRow = domain.GetHeaderRow(f.FileInfo.HeaderRow)
d.AppKey = f.AppKey
//d.AllowTableProcessFlag = 1
//d.AllowTableExportFlag = 1
d.AddFlag(5) // 默认可以 校验、导出
if len(f.AppKey) > 0 && f.FileInfo.TableId > 0 {
d.TableId = f.FileInfo.TableId
//d.AllowTableGenerateFlag = 1
d.AddFlag(2) // 可分表生成
d.RemoveFlag(1) // 不可校验
//d.RemoveFlag(4) // 不可校验
//d.AllowTableProcessFlag = 0
//d.AllowTableExportFlag = 0
}
for i := 1; i <= 4; i = i << 1 {
if i&d.Flag == i {
d.Flags = append(d.Flags, i)
}
}
return d
}
func (d *FileDto) AddFlag(flag int) {
if d.Flag&flag == flag {
return
}
d.Flag |= flag
}
func (d *FileDto) RemoveFlag(flag int) {
if d.Flag&flag != flag {
return
}
d.Flag ^= flag
}
type AppDto struct {
AppId int64 `json:"appId"`
AppKey string `json:"appKey"`
AppName string `json:"appName"`
Files []*FileDto `json:"files"`
}
... ...
... ... @@ -10,11 +10,15 @@ import (
type GetFileQuery struct {
// 文件ID
FileId int `cname:"文件ID" json:"fileId" valid:"Required"`
FileId int `cname:"文件ID" json:"fileId"`
FileName string `json:"name"`
FileType string `json:"type"`
}
func (getFileQuery *GetFileQuery) Valid(validation *validation.Validation) {
validation.SetError("CustomValid", "未实现的自定义认证")
}
func (getFileQuery *GetFileQuery) ValidateQuery() error {
... ...
package query
type ListAppTableFileCommand struct {
Name string `json:"name"`
AppKey string `json:"appKey"`
}
... ...
... ... @@ -17,11 +17,12 @@ type SearchFileQuery struct {
// 页码
// PageNumber int `cname:"页码" json:"pageNumber,omitempty"`
// 页数
FileName string `cname:"文件名称" json:"fileName,omitempty"`
PageSize int `cname:"页数" json:"pageSize,omitempty"`
LastId int `cname:"最后一条记录ID" json:"lastId"`
FileType domain.FileType `cname:"文件类型" json:"fileType" valid:"Required"`
Context *domain.Context
FileName string `cname:"文件名称" json:"fileName,omitempty"`
PageSize int `cname:"页数" json:"pageSize,omitempty"`
LastId int `cname:"最后一条记录ID" json:"lastId"`
FileType domain.FileType `cname:"文件类型" json:"fileType" valid:"Required"`
InAppKeys []string `json:"inAppKeys"`
Context *domain.Context
}
func (cmd *SearchFileQuery) Valid(validation *validation.Validation) {
... ...
package service
import (
"bytes"
"errors"
"fmt"
"github.com/beego/beego/v2/client/httplib"
"github.com/linmadan/egglib-go/core/application"
pgTransaction "github.com/linmadan/egglib-go/transaction/pg"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/factory"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/file/command"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/file/query"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/constant"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/api/apilib"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/domainService"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/excel"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/starrocks"
"os"
"strings"
"time"
)
func (fileService *FileService) CreateAppTableFile(ctx *domain.Context, cmd *command.CreateAppTableFileCommand) (*command.CreateFileCommand, error) {
response := &command.CreateFileCommand{}
var (
titles = make([]string, 0)
dataList = make([][]string, 0)
)
for _, filed := range cmd.Fields {
titles = append(titles, filed.Name)
}
for i := range cmd.Data {
row := make([]string, 0)
for _, filed := range titles {
if v, ok := cmd.Data[i][filed]; ok {
row = append(row, v)
} else {
row = append(row, "")
}
}
dataList = append(dataList, row)
}
fileUpload, err := saveFile(cmd.Name, titles, dataList, nil)
if err != nil {
return nil, factory.FastError(err)
}
response.Name = cmd.Name
if !strings.HasSuffix(response.Name, domain.XLSX) {
response.Name = response.Name + domain.XLSX
}
response.Url = fileUpload.Url
response.FileSize = int(fileUpload.FileSize)
response.FileFrom = domain.FileFromDigitalAppClient
return response, nil
}
func saveFile(name string, title []string, dataList [][]string, toInterfaces func([]string) []interface{}) (FileUpload, error) {
var (
response = FileUpload{}
err error
)
var writerTo = excel.NewXLXSWriterTo(title, dataList)
if toInterfaces != nil {
writerTo.ToInterfaces = toInterfaces
}
filename := fmt.Sprintf("%v_%v.xlsx", name, time.Now().Format("060102150405"))
path := fmt.Sprintf("public/%v", filename)
if err = writerTo.Save(path); err != nil {
return response, factory.FastError(err)
}
api := apilib.NewApiAuthLib(constant.OPEN_API_HOST)
uploadResponse, err := api.Upload(apilib.RequestUpload{
UploadFileMap: map[string]string{"file": path},
})
if err != nil {
return response, err
}
if stat, err := os.Stat(path); err == nil {
response.FileSize = stat.Size()
}
response.Url = domain.ConvertInternalFileUrlToPublic(uploadResponse.Path)
response.FileName = name
response.Ext = domain.XLSX
return response, nil
}
func saveCsvFile(name string, title []string, dataList [][]string, toInterfaces func([]string) []interface{}) (FileUpload, error) {
var (
response = FileUpload{}
err error
)
var writerTo = excel.NewCSVWriterTo(title, dataList)
filename := fmt.Sprintf("%v_%v.csv", name, time.Now().Format("060102150405"))
path := fmt.Sprintf("public/%v", filename)
if err = writerTo.Save(path); err != nil {
return response, factory.FastError(err)
}
api := apilib.NewApiAuthLib(constant.OPEN_API_HOST)
uploadResponse, err := api.Upload(apilib.RequestUpload{
UploadFileMap: map[string]string{"file": path},
})
if err != nil {
return response, err
}
if stat, err := os.Stat(path); err == nil {
response.FileSize = stat.Size()
}
response.Url = domain.ConvertInternalFileUrlToPublic(uploadResponse.Path)
response.FileName = name
response.Ext = domain.CSV
return response, nil
}
type FileUpload struct {
Url string `json:"url"`
Ext string `json:"ext"`
FileName string `json:"fileName"`
FileSize int64 `json:"fileSize"`
}
func (fileService *FileService) DeleteAppTableFile(ctx *domain.Context, cmd *command.DeleteAppTableFileCommand) (interface{}, error) {
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
fileRepository, file, _ := factory.FastPgFile(transactionContext, 0)
file, err = fileRepository.FindOne(map[string]interface{}{"appKey": cmd.AppKey, "fileName": cmd.Name, "fileType": domain.SourceFile})
if err == domain.ErrorNotFound {
return nil, factory.FastError(errors.New("文件不存在"))
}
if err != nil {
return nil, factory.FastError(err)
}
if _, err := fileRepository.Remove(file); err != nil {
return nil, factory.FastError(err)
}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return struct{}{}, nil
}
func (fileService *FileService) AppTableFileAppendData(ctx *domain.Context, cmd *command.AppTableFileAppendDataCommand) (interface{}, error) {
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
fileRepository, file, _ := factory.FastPgFile(transactionContext, 0)
file, err = fileRepository.FindOne(map[string]interface{}{"appKey": cmd.AppKey, "fileName": cmd.Name, "fileType": domain.SourceFile})
if err == domain.ErrorNotFound {
return nil, factory.FastError(errors.New("文件不存在"))
}
if err != nil {
return nil, factory.FastError(err)
}
// 下载文件
f, err := httplib.Get(domain.ConvertFileUrlToInternal(file.FileInfo.Url)).Bytes()
if err != nil {
return nil, factory.FastError(err)
}
reader := bytes.NewReader(f)
var (
importer *excel.Importer = excel.NewExcelImportByFile(file.FileInfo.Ext)
appendTableDataList = make([][]string, 0)
)
data, err := importer.OpenExcelFromIoReader(reader)
if err != nil {
return nil, factory.FastError(err)
}
titles := importer.Reader().Header().Columns
for _, f := range cmd.Fields {
found := false
for _, column := range titles {
if column == f.Name {
found = true
break
}
}
if !found {
titles = append(titles, f.Name)
}
}
// 填充旧数据
// 追加文件
for i := range data {
if len(data[i]) < len(titles) {
for j := 0; j < (len(titles) - len(data[i])); j++ {
data[i] = append(data[i], "")
}
}
}
for i := range cmd.Data {
row := make([]string, 0)
for _, filed := range titles {
if v, ok := cmd.Data[i][filed]; ok {
row = append(row, v)
} else {
row = append(row, "")
}
}
data = append(data, row)
if cmd.AppendTableDataFlag {
appendTableDataList = append(appendTableDataList, row)
}
}
//if !cmd.AppendTableDataFlag {
// 上传文件
fileUpload, err := saveFile(cmd.Name, titles, data, nil)
if err != nil {
return nil, factory.FastError(err)
}
// 更新文件
file.FileInfo.Url = fileUpload.Url
file.FileInfo.FileSize = int(fileUpload.FileSize)
file.FileInfo.RowCount = len(data)
_, err = fileRepository.Save(file)
if err != nil {
return nil, factory.FastError(err)
}
//}
//else if cmd.AppendTableDataFlag && file.FileInfo.TableId != 0 { // 追加数据到应用表
//var table *domain.Table
//if _, table, err = factory.FastPgTable(transactionContext, file.FileInfo.TableId); err != nil {
// return nil, factory.FastError(err)
//}
//// 上传文件
//fileUpload, err := saveCsvFile(cmd.Name, titles, appendTableDataList, nil)
//if err != nil {
// return nil, factory.FastError(err)
//}
//appendDataToTableService, _ := domainService.NewAppendDataToTableService(transactionContext.(*pgTransaction.TransactionContext))
//var mappingFields = make([]*domain.MappingField, 0)
//for _, f := range cmd.Fields {
// mappingFields = append(mappingFields, &domain.MappingField{
// MainTableField: &domain.Field{Name: f.Name},
// VerifiedFileFieldName: f.Name,
// })
//}
//if _, err = appendDataToTableService.AppendDataDirectly(ctx, fileUpload.Url, table, mappingFields); err != nil {
// return nil, factory.FastError(err)
//}
//}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return struct{}{}, nil
}
func (fileService *FileService) AppTableAppendData(ctx *domain.Context, cmd *command.AppTableFileAppendDataCommand) (interface{}, error) {
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
fileRepository, file, _ := factory.FastPgFile(transactionContext, 0)
file, err = fileRepository.FindOne(map[string]interface{}{"appKey": cmd.AppKey, "fileName": cmd.Name, "fileType": domain.SourceFile})
if err == domain.ErrorNotFound {
return nil, factory.FastError(errors.New("文件不存在"))
}
if err != nil {
return nil, factory.FastError(err)
}
if file.FileInfo.TableId == 0 {
return nil, factory.FastError(errors.New("表不存在"))
}
var (
appendTableDataList = make([][]string, 0)
titles = make([]string, 0)
table *domain.Table
)
_, table, err = factory.FastPgTable(transactionContext, file.FileInfo.TableId)
if err != nil {
return nil, factory.FastError(err)
}
for _, f := range table.Fields(false) {
titles = append(titles, f.Name)
}
for _, f := range cmd.Fields {
found := false
for _, column := range titles {
if column == f.Name {
found = true
break
}
}
if !found {
titles = append(titles, f.Name)
}
}
for i := range cmd.Data {
row := make([]string, 0)
for _, filed := range titles {
if v, ok := cmd.Data[i][filed]; ok {
row = append(row, v)
} else {
row = append(row, "")
}
}
appendTableDataList = append(appendTableDataList, row)
}
// 上传文件
fileUpload, err := saveCsvFile(cmd.Name, titles, appendTableDataList, nil)
if err != nil {
return nil, factory.FastError(err)
}
appendDataToTableService, _ := domainService.NewAppendDataToTableService(transactionContext.(*pgTransaction.TransactionContext))
var mappingFields = make([]*domain.MappingField, 0)
for _, f := range cmd.Fields {
mappingFields = append(mappingFields, &domain.MappingField{
MainTableField: &domain.Field{Name: f.Name},
VerifiedFileFieldName: f.Name,
})
}
if _, err = appendDataToTableService.AppendDataDirectly(ctx, fileUpload.Url, table, mappingFields); err != nil {
return nil, factory.FastError(err)
}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return struct{}{}, nil
}
func (fileService *FileService) AppTableAppendDataDirect(ctx *domain.Context, cmd *command.AppTableFileAppendDataCommand) (interface{}, error) {
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
fileRepository, file, _ := factory.FastPgFile(transactionContext, 0)
file, err = fileRepository.FindOne(map[string]interface{}{"appKey": cmd.AppKey, "fileName": cmd.Name, "fileType": domain.SourceFile})
if err == domain.ErrorNotFound {
return nil, factory.FastError(errors.New("文件不存在"))
}
if err != nil {
return nil, factory.FastError(err)
}
if file.FileInfo.TableId == 0 {
return nil, factory.FastError(errors.New("表不存在"))
}
var (
titles = make([]string, 0)
table *domain.Table
)
_, table, err = factory.FastPgTable(transactionContext, file.FileInfo.TableId)
if err != nil {
return nil, factory.FastError(err)
}
for _, f := range table.Fields(false) {
titles = append(titles, f.Name)
}
mapNameField := domain.Fields(table.Fields(false)).ToMap()
for _, f := range cmd.Fields {
found := false
for _, column := range titles {
if column == f.Name {
found = true
break
}
}
if !found {
titles = append(titles, f.Name)
}
}
var mapData = make([]map[string]string, 0)
for i := range cmd.Data {
mapItem := make(map[string]string)
for k, v := range cmd.Data[i] {
if f, ok := mapNameField[k]; ok {
mapItem[f.SQLName] = v
}
}
mapData = append(mapData, mapItem)
}
editDataService, _ := factory.CreateTableEditDataService(transactionContext)
_, err = editDataService.BatchAdd(ctx, domain.EditDataRequest{
TableId: table.TableId,
Table: table,
Where: domain.Where{},
UpdateList: nil,
AddList: domainService.MapArrayToFieldValues(mapData, table, nil, false),
RemoveList: nil,
IgnoreTableType: true,
})
if err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return struct{}{}, nil
}
func (fileService *FileService) AppTableFileList(ctx *domain.Context, cmd *query.ListAppTableFileCommand) (interface{}, error) {
return fileService.GetAppFile(ctx, cmd.AppKey, cmd.Name)
}
func (fileService *FileService) UpdateAppTableFile(ctx *domain.Context, cmd *command.UpdateAppTableFileCommand) (interface{}, error) {
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
fileRepository, file, _ := factory.FastPgFile(transactionContext, 0)
file, err = fileRepository.FindOne(map[string]interface{}{"appKey": cmd.AppKey, "fileName": cmd.Name, "fileType": domain.SourceFile})
if err == domain.ErrorNotFound {
return nil, factory.FastError(errors.New("文件不存在"))
}
if err != nil {
return nil, factory.FastError(err)
}
if len(cmd.AddFields) == 0 {
return nil, nil
}
tableRepository, table, _ := factory.FastPgTable(transactionContext, file.FileInfo.TableId)
if err == domain.ErrorNotFound {
return nil, factory.FastError(errors.New("文件表不存在"))
}
builder := domainService.NewDataFieldsBuilder()
for i, _ := range cmd.AddFields {
if _, ok := table.MatchField(cmd.AddFields[i]); ok {
return nil, factory.FastError(errors.New("字段已存在"))
}
}
for _, f := range cmd.AddFields {
dataField := builder.NewDataField(f.Name, f.SQLType, domain.MainTableField)
table.DataFields = append(table.DataFields, dataField)
if err = starrocks.AddTableColumn(starrocks.DB, table.SQLName, dataField); err != nil {
return nil, factory.FastError(err)
}
}
if table, err = tableRepository.Save(table); err != nil {
return nil, factory.FastError(err)
}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return struct{}{}, nil
}
... ...
package service
import (
"bytes"
"fmt"
"github.com/beego/beego/v2/client/httplib"
"github.com/linmadan/egglib-go/core/application"
"github.com/linmadan/egglib-go/transaction/pg"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/factory"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/file/command"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/file/dto"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/domainService"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/excel"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/redis"
)
// FilePreview 加载表格数据
... ... @@ -33,10 +41,88 @@ func (fileService *FileService) FilePreview(ctx *domain.Context, loadDataTableCo
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
//return dto.NewDataTableDtoDemo(loadDataTableService.GetFileId()), nil
return data, nil
}
func (fileService *FileService) ResetHeaderRow(ctx *domain.Context, loadDataTableCommand *command.ResetTableHeaderCommand) (interface{}, error) {
if err := loadDataTableCommand.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
cache := redis.NewFileCacheService()
temporaryFile, err := cache.Get(redis.KeyTemporaryFileInfo(loadDataTableCommand.FileId))
if err != nil {
return nil, factory.FastError(err)
}
loadDataTableService, _ := factory.CreateLoadDataTableService(transactionContext)
data, err := loadDataTableService.RePreview(ctx, loadDataTableCommand.FileId, temporaryFile.Fields, loadDataTableCommand.Where)
// 处理错误
level := domain.LevelInfo
errMsg := ""
if err != nil {
level = domain.LevelError
errMsg = err.Error()
}
if logErr := domainService.FastLog(transactionContext.(*pg.TransactionContext),
domain.VerifiedStepLog, temporaryFile.FileId, &domainService.ExcelTableResetHeaderLog{
LogEntry: domain.NewLogEntry(temporaryFile.FileName, domain.VerifiedFile.ToString(), domain.FileVerify,
ctx.WithValue(domain.ContextWithLogLevel, level).
WithValue(domain.ContextWithLogMsg, errMsg)),
HeaderRow: domain.GetHeaderRow(loadDataTableCommand.HeaderRow),
}); logErr != nil {
return nil, logErr
}
if err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return data, nil
}
// PrepareTemporaryFile 准备临时文件
func (fileService *FileService) PrepareTemporaryFile(ctx *domain.Context, cmd *command.PrepareTemporaryFileCommand) (interface{}, error) {
if err := cmd.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
loadDataTableService, _ := factory.CreateLoadDataTableService(transactionContext)
data, err := loadDataTableService.CreateTemporaryFile(ctx, cmd.FileId)
if err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
fileDto := &dto.FileDto{}
fileDto.Load(data)
return fileDto, nil
}
// EditDataTable 编辑表格数据
func (fileService *FileService) EditDataTable(ctx *domain.Context, editDataTableCommand *command.EditDataTableCommand) (interface{}, error) {
if err := editDataTableCommand.ValidateCommand(); err != nil {
... ... @@ -53,15 +139,47 @@ func (fileService *FileService) EditDataTable(ctx *domain.Context, editDataTable
transactionContext.RollbackTransaction()
}()
cache := redis.NewFileCacheService()
temporaryFile, err := cache.Get(redis.KeyTemporaryFileInfo(editDataTableCommand.FileId))
if err != nil {
return nil, factory.FastError(err)
}
editDataTableCommand.Fields = temporaryFile.Fields
editDataTableCommand.ProcessFields = temporaryFile.MatchFields(editDataTableCommand.ProcessFieldNames)
if len(editDataTableCommand.ProcessFields) == 0 {
return nil, factory.FastError(fmt.Errorf("请至少选择一个数据列"))
}
if editDataTableCommand.Action == "remove-column" && len(temporaryFile.Fields) == len(editDataTableCommand.ProcessFields) {
return nil, factory.FastError(fmt.Errorf("请至少保留一个数据列"))
}
if editDataTableCommand.Action == "rename-column" {
targetColumn := editDataTableCommand.ProcessFieldNames[0]
newColumnName := editDataTableCommand.Params["newColumnName"].(string)
if len(temporaryFile.MatchFields([]string{newColumnName})) > 0 && newColumnName != targetColumn {
return nil, factory.FastError(fmt.Errorf("已存在相同名称,修改无效"))
}
}
// allowAction := func(fields []*domain.Field, action string) error {
// for _, f := range fields {
// if f.SQLType != string(domain.String) &&
// !(action == domain.RemoveColumn || action == domain.CopyColumn || action == domain.RenameColumn || action == domain.ConvertColumnType) {
// return fmt.Errorf("列【%v】必须先转字符串类型",f.Name)
// }
// }
// return nil
// }
// if err = allowAction(editDataTableCommand.ProcessFields, editDataTableCommand.Action); err != nil {
// return nil, factory.FastError(err)
// }
editDataTableService, _ := factory.CreateEditDataTableService(transactionContext)
_, err = editDataTableService.Edit(ctx, editDataTableCommand.EditTableRequest)
response, err := editDataTableService.Edit(ctx, editDataTableCommand.EditTableRequest)
if err != nil {
return nil, factory.FastError(err)
}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return struct{}{}, nil
return (&dto.EditDataTableDto{}).Load(response), nil
}
// FlushDataTable 持久化表格数据
... ... @@ -80,16 +198,19 @@ func (fileService *FileService) FlushDataTable(ctx *domain.Context, flushDataTab
transactionContext.RollbackTransaction()
}()
flushDataTableService, _ := factory.CreateFlushDataTableService(transactionContext)
fields := make([]*domain.Field, 0)
for _, f := range flushDataTableCommand.DataFields {
fields = append(fields, &domain.Field{
Name: f.Name,
SQLType: f.SQLType,
})
}
if _, err := flushDataTableService.Flush(ctx, flushDataTableCommand.FileId, &domain.Table{
DataFields: fields,
RowCount: flushDataTableCommand.RowCount,
cache := redis.NewFileCacheService()
temporaryFile, err := cache.Get(redis.KeyTemporaryFileInfo(flushDataTableCommand.ObjectId))
if err != nil {
return nil, factory.FastError(err)
}
if err = temporaryFile.Valid(); err != nil {
return nil, factory.FastError(err)
}
if _, err := flushDataTableService.Flush(ctx, flushDataTableCommand.ObjectId, &domain.Table{
DataFields: temporaryFile.Fields,
RowCount: temporaryFile.Total,
HeaderRow: temporaryFile.HeaderRow,
}); err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
... ... @@ -131,6 +252,9 @@ func (fileService *FileService) AppendDataToTable(ctx *domain.Context, cmd *comm
if err := cmd.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
if len(cmd.MappingFields) == 0 {
return nil, factory.FastError(fmt.Errorf("请选择对应字段"))
}
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
... ... @@ -152,3 +276,85 @@ func (fileService *FileService) AppendDataToTable(ctx *domain.Context, cmd *comm
}
return result, nil
}
// AppendDataToTablePreflightCheck 追加数据预查
func (fileService *FileService) AppendDataToTablePreflightCheck(ctx *domain.Context, cmd *command.AppendDataToTableCommand) (interface{}, error) {
if err := cmd.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
generateMainTableService, _ := factory.CreateAppendDataToTableService(transactionContext)
result, err := generateMainTableService.PreflightCheck(ctx, cmd.FileId, cmd.TableId, cmd.MappingFields)
if err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return result, nil
}
// ExportFile 文件下载
func (fileService *FileService) ExportFile(ctx *domain.Context, cmd *command.ExportFileCommand) (interface{}, error) {
if err := cmd.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
_, file, err := factory.FastPgFile(transactionContext, cmd.FileId)
if err != nil {
return nil, factory.FastError(err)
}
var response = FileUpload{}
if file.FileType == domain.SourceFile.ToString() {
response.Url = file.FileInfo.Url
response.Ext = domain.XLSX
response.FileName = file.FileInfo.Name
return response, nil
}
_, table, err := factory.FastPgTable(transactionContext, file.FileInfo.TableId)
if err != nil {
return nil, factory.FastError(err)
}
f, err := httplib.Get(domain.ConvertFileUrlToInternal(file.FileInfo.Url)).Bytes()
if err != nil {
return nil, factory.FastError(err)
}
reader := bytes.NewReader(f)
var importer *excel.Importer = excel.NewExcelImportByFile(file.FileInfo.Ext)
data, err := importer.OpenExcelFromIoReader(reader)
if err != nil {
return nil, factory.FastError(err)
}
response, err = saveFile(file.FileInfo.Name, importer.Reader().Header().Columns, data, domain.MakeToInterfaces(table.DataFields))
if err != nil {
return nil, factory.FastError(err)
}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return response, nil
}
... ...
... ... @@ -2,6 +2,14 @@ package service
import (
"fmt"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/constant"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/api/authlib"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/api/bytelib"
"path/filepath"
"sort"
"strings"
"time"
"github.com/linmadan/egglib-go/core/application"
"github.com/linmadan/egglib-go/utils/tool_funs"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/factory"
... ... @@ -9,19 +17,16 @@ import (
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/file/dto"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/file/query"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain/bytecore"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/domainService"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/utils"
"path/filepath"
"time"
)
// 文件服务
// FileService 文件服务
type FileService struct {
}
// 创建文件服务
func (fileService *FileService) CreateFile(ctx *domain.Context, createFileCommand *command.CreateFileCommand) (interface{}, error) {
// CreateFile 创建文件服务
func (fileService *FileService) CreateFile(ctx *domain.Context, createFileCommand *command.CreateFileCommand) (*domain.File, error) {
if err := createFileCommand.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
... ... @@ -35,25 +40,81 @@ func (fileService *FileService) CreateFile(ctx *domain.Context, createFileComman
defer func() {
transactionContext.RollbackTransaction()
}()
fileInfo := &domain.FileInfo{
Name: domain.FileName(createFileCommand.Name),
Url: createFileCommand.Url,
FileSize: createFileCommand.FileSize,
Ext: filepath.Ext(createFileCommand.Name),
}
newFile := &domain.File{
FileType: domain.SourceFile.ToString(),
FileInfo: &domain.FileInfo{
Name: domain.FileName(createFileCommand.Name),
Url: createFileCommand.Url,
FileSize: createFileCommand.FileSize,
Ext: filepath.Ext(createFileCommand.Name),
},
FileType: domain.SourceFile.ToString(),
FileInfo: fileInfo,
SourceFileId: 0,
//Operator: "",
CreatedAt: time.Now(),
UpdatedAt: time.Now(),
Context: ctx,
FileFrom: createFileCommand.FileFrom,
AppKey: createFileCommand.AppKey,
}
fileRepository, _, _ := factory.FastPgFile(transactionContext, 0)
// 文件名相同,进行替换
if oldFile, findOldFileErr := fileRepository.FindOne(map[string]interface{}{
"context": ctx,
"fileName": fileInfo.Name,
"fileType": domain.SourceFile.ToString(),
}); oldFile != nil && findOldFileErr == nil {
oldFile.FileInfo = fileInfo
oldFile.UpdatedAt = time.Now()
newFile = oldFile
if err = factory.FastLog(transactionContext, domain.CommonLog, oldFile.FileId, &domainService.FileReplaceLog{
LogEntry: domain.NewLogEntry(oldFile.FileInfo.Name, domain.SourceFile.ToString(), domain.FileUpload, ctx),
}); err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
}
file, err := fileRepository.Save(newFile)
if err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
// 同时生成主表
// 前置需要进行预览、保持文件,才能生成主表
if createFileCommand.GenerateTableFlag {
table := domainService.NewTable(domain.MainTable, fileInfo.Name, createFileCommand.Fields, 0).
WithPrefix(domain.MainTable.ToString()).WithContext(&domain.Context{})
tableRepository, _, _ := factory.FastPgTable(transactionContext, 0)
if table, err = tableRepository.Save(table); err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
domainService.ByteCore.LoadDataTable(domain.ReqLoadDataTable{
FileId: file.FileId,
FileName: file.FileInfo.Name,
Url: file.FileInfo.Url,
Ext: file.FileInfo.Ext,
//Where: where,
OriginalTableId: fmt.Sprintf("%v", file.FileId),
IsFromOriginalTable: true,
TableFileUrl: file.FileInfo.Url,
ColumnSchemas: bytelib.DomainFieldsToColumnSchemas(table.Fields(false)),
SortParameters: make(map[string]interface{}),
})
response, err := domainService.ByteCore.SaveTable(domain.ReqSaveTable{FileId: file.FileId, Table: table})
if err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
if _, err = domainService.ByteCore.GenerateTable(ctx, domain.ReqGenerateTable{
file.FileId, response.Url, table,
}); err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
file.FileInfo.TableId = table.TableId
if file, err = fileRepository.Save(file); err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
}
if err = factory.FastLog(transactionContext, domain.CommonLog, file.FileId, &domainService.FileUploadSuccessLog{
LogEntry: domain.NewLogEntry(file.FileInfo.Name, domain.SourceFile.ToString(), domain.FileUpload, ctx),
}); err != nil {
... ... @@ -62,11 +123,11 @@ func (fileService *FileService) CreateFile(ctx *domain.Context, createFileComman
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return struct{}{}, nil
return file, nil
}
// 返回文件服务
func (fileService *FileService) GetFile(getFileQuery *query.GetFileQuery) (interface{}, error) {
// GetFile 返回文件服务
func (fileService *FileService) GetFile(ctx *domain.Context, getFileQuery *query.GetFileQuery) (interface{}, error) {
if err := getFileQuery.ValidateQuery(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
... ... @@ -88,21 +149,35 @@ func (fileService *FileService) GetFile(getFileQuery *query.GetFileQuery) (inter
} else {
fileRepository = value
}
file, err := fileRepository.FindOne(map[string]interface{}{"fileId": getFileQuery.FileId})
if err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
response := map[string]interface{}{
"file": nil,
}
if file == nil {
return nil, application.ThrowError(application.RES_NO_FIND_ERROR, fmt.Sprintf("%s", string(getFileQuery.FileId)))
} else {
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return file, nil
options := map[string]interface{}{"context": ctx}
if getFileQuery.FileId > 0 {
options["fileId"] = getFileQuery.FileId
}
if len(getFileQuery.FileName) > 0 {
options["fileName"] = domain.FileName(getFileQuery.FileName)
}
if len(getFileQuery.FileType) > 0 {
options["fileType"] = getFileQuery.FileType
}
// 未传递参数
if len(options) == 1 {
return response, nil
}
file, _ := fileRepository.FindOne(options)
//if err != nil {
// return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
//}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
response["file"] = (&dto.FileDto{}).Load(file)
return response, nil
}
// 返回文件服务列表
// ListFile 返回文件服务列表
func (fileService *FileService) ListFile(listFileQuery *query.ListFileQuery) (interface{}, error) {
if err := listFileQuery.ValidateQuery(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
... ... @@ -180,8 +255,131 @@ func (fileService *FileService) SearchFile(listFileQuery *query.SearchFileQuery)
}, nil
}
// 移除文件服务
func (fileService *FileService) RemoveFile(removeFileCommand *command.RemoveFileCommand) (interface{}, error) {
// SearchAppFile 返回文件服务列表
func (fileService *FileService) SearchAppFile(ctx *domain.Context, listFileQuery *query.SearchFileQuery) (interface{}, error) {
if err := listFileQuery.ValidateQuery(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
var fileRepository, _, _ = factory.FastPgFile(transactionContext, 0)
apiAuthLib := authlib.NewApiAuthLib(constant.AUTH_SERVER_HOST).WithToken(ctx.AccessToken)
response, err := apiAuthLib.MeAppInfo(authlib.RequestUserMeQuery{UserId: ctx.TenantId})
if err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
inAppKeys := make([]string, 0)
for _, app := range response.Apps {
inAppKeys = append(inAppKeys, app.AppKey)
}
var (
fileDtos = make([]*dto.FileDto, 0)
total int64
)
if len(inAppKeys) > 0 {
queryOptions := utils.ObjectToMap(listFileQuery)
queryOptions["inAppKeys"] = inAppKeys
queryOptions["limit"] = 1000
queryOptions["fileName"] = listFileQuery.FileName
count, files, err := fileRepository.Find(queryOptions)
if err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
for _, file := range files {
var item = &dto.FileDto{}
item.Load(file)
fileDtos = append(fileDtos, item)
}
total = count
}
var apps = make([]*dto.AppDto, 0)
for _, app := range response.Apps {
if len(listFileQuery.FileName) > 0 && !strings.Contains(app.AppName, listFileQuery.FileName) {
continue
}
files := make([]*dto.FileDto, 0)
for _, file := range fileDtos {
if file.AppKey == app.AppKey {
files = append(files, file)
}
}
sort.SliceStable(files, func(i, j int) bool {
return files[i].Name < files[j].Name
})
apps = append(apps, &dto.AppDto{
AppId: app.AppId,
AppKey: app.AppKey,
AppName: app.AppName,
Files: files,
})
}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return map[string]interface{}{
"apps": apps,
"count": total,
}, nil
}
// GetAppFile 返回应用对应的文件服务列表
func (fileService *FileService) GetAppFile(ctx *domain.Context, appKey string, fileName string) (interface{}, error) {
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
var fileRepository, _, _ = factory.FastPgFile(transactionContext, 0)
var (
fileDtos = make([]*dto.FileDto, 0)
total int64
)
queryOptions := make(map[string]interface{})
queryOptions["fileType"] = domain.SourceFile
queryOptions["inAppKeys"] = []string{appKey}
queryOptions["limit"] = 100
count, files, err := fileRepository.Find(queryOptions)
if err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
for _, file := range files {
var item = &dto.FileDto{}
if fileName != "" && file.FileInfo.Name != fileName {
continue
}
item.Load(file)
fileDtos = append(fileDtos, item)
}
total = count
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return map[string]interface{}{
"count": total,
"files": fileDtos,
}, nil
}
// RemoveFile 移除文件服务
func (fileService *FileService) RemoveFile(ctx *domain.Context, removeFileCommand *command.RemoveFileCommand) (interface{}, error) {
if err := removeFileCommand.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
... ... @@ -208,10 +406,10 @@ func (fileService *FileService) RemoveFile(removeFileCommand *command.RemoveFile
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
if file == nil {
return nil, application.ThrowError(application.RES_NO_FIND_ERROR, fmt.Sprintf("%s", string(removeFileCommand.FileId)))
return nil, application.ThrowError(application.RES_NO_FIND_ERROR, fmt.Sprintf("%d", removeFileCommand.FileId))
}
deleteFileService, _ := factory.CreateDeleteFileService(transactionContext)
err = deleteFileService.DeleteFiles(nil, file)
err = deleteFileService.DeleteFiles(ctx, file)
if err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
... ... @@ -221,7 +419,7 @@ func (fileService *FileService) RemoveFile(removeFileCommand *command.RemoveFile
return struct{}{}, nil
}
// 更新文件服务
// UpdateFile 更新文件服务
func (fileService *FileService) UpdateFile(updateFileCommand *command.UpdateFileCommand) (interface{}, error) {
if err := updateFileCommand.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
... ... @@ -249,7 +447,7 @@ func (fileService *FileService) UpdateFile(updateFileCommand *command.UpdateFile
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
if file == nil {
return nil, application.ThrowError(application.RES_NO_FIND_ERROR, fmt.Sprintf("%s", string(updateFileCommand.FileId)))
return nil, application.ThrowError(application.RES_NO_FIND_ERROR, fmt.Sprintf("%d", updateFileCommand.FileId))
}
if err := file.Update(tool_funs.SimpleStructToMap(updateFileCommand)); err != nil {
return nil, application.ThrowError(application.BUSINESS_ERROR, err.Error())
... ... @@ -264,7 +462,7 @@ func (fileService *FileService) UpdateFile(updateFileCommand *command.UpdateFile
}
}
// 取消校验中的文件
// CancelVerifyingFile 取消校验中的文件
func (fileService *FileService) CancelVerifyingFile(ctx *domain.Context, cmd *command.CancelVerifyingFileCommand) (interface{}, error) {
if err := cmd.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
... ... @@ -295,7 +493,7 @@ func (fileService *FileService) CancelVerifyingFile(ctx *domain.Context, cmd *co
}
byteCore, _ := factory.CreateByteCoreService(transactionContext)
_, err = byteCore.CancelFile(bytecore.ReqCancelFile{})
_, err = byteCore.CancelFile(domain.ReqCancelFile{})
if err != nil {
return nil, factory.FastError(err)
}
... ... @@ -306,6 +504,40 @@ func (fileService *FileService) CancelVerifyingFile(ctx *domain.Context, cmd *co
return struct{}{}, nil
}
// CheckFileVerifyStatus 检查文件校验状态
func (fileService *FileService) CheckFileVerifyStatus(ctx *domain.Context, cmd *command.CheckFileVerifyStatusCommand) (interface{}, error) {
if err := cmd.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
fileRepository, file, err := factory.FastPgFile(transactionContext, cmd.FileId)
if err != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
var res = struct {
ExistVerifyFile bool `json:"existVerifyFile"`
}{}
_, verifyFiles, findErr := fileRepository.Find(map[string]interface{}{"context": ctx, "fileType": domain.VerifiedFile.ToString(), "sourceFileId": file.SourceFileId})
if findErr == nil && len(verifyFiles) > 0 {
res.ExistVerifyFile = true
}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return res, nil
}
func NewFileService(options map[string]interface{}) *FileService {
newFileService := &FileService{}
return newFileService
... ...
... ... @@ -2,11 +2,12 @@ package command
import (
"fmt"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"reflect"
"strings"
"time"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"github.com/beego/beego/v2/core/validation"
)
... ... @@ -23,6 +24,8 @@ type SearchLogCommand struct {
PageNumber int `cname:"页码" json:"pageNumber,omitempty"`
// 页数
PageSize int `cname:"页数" json:"pageSize,omitempty"`
// 对象名称
ObjectName string `cname:"页数" json:"objectName,omitempty"`
Year int `cname:"年" json:"year,omitempty"`
Month int `cname:"月" json:"month,omitempty"`
... ... @@ -31,8 +34,9 @@ type SearchLogCommand struct {
BeginTime time.Time `cname:"开始时间" json:"beginTime"`
// 结束时间
EndTime time.Time `cname:"结束时间" json:"endTime"`
Context *domain.Context
// 按log_id 排序
SortByLogId string `json:"sortByLogId"`
Context *domain.Context
}
func (cmd *SearchLogCommand) Valid(validation *validation.Validation) {
... ... @@ -46,7 +50,10 @@ func (cmd *SearchLogCommand) Valid(validation *validation.Validation) {
}
if cmd.Year > 0 && cmd.Month > 0 && cmd.Day > 0 {
cmd.BeginTime = time.Date(cmd.Year, time.Month(cmd.Month), cmd.Day, 0, 0, 0, 0, time.Local)
cmd.EndTime = cmd.BeginTime.AddDate(0, 0, cmd.Day)
cmd.EndTime = cmd.BeginTime.AddDate(0, 0, 1)
}
if cmd.PageNumber == 0 {
cmd.PageNumber = 1
}
}
... ...
... ... @@ -21,6 +21,10 @@ type VerifiedStepLogDto struct {
//OperatorName string `json:"operatorName"`
// 创建时间
CreatedAt string `json:"createdAt"`
// 错误级别
Level string `json:"level"`
// 错误信息
Error string `json:"error"`
}
func (d *VerifiedStepLogDto) Load(m *domain.Log) {
... ... @@ -32,4 +36,9 @@ func (d *VerifiedStepLogDto) Load(m *domain.Log) {
d.Content = m.Content
//d.OperatorName = m.OperatorName
d.CreatedAt = m.CreatedAt.Local().Format("2006-01-02 15:04:05")
d.Level = m.Entry.Level
d.Error = m.Entry.Error
if len(d.Level) == 0 {
d.Level = domain.LevelInfo.ToString()
}
}
... ...
... ... @@ -12,11 +12,9 @@ import (
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/utils"
)
// 日志服务
type LogService struct {
}
// 创建日志服务
func (logService *LogService) CreateLog(createLogCommand *command.CreateLogCommand) (interface{}, error) {
if err := createLogCommand.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
... ... @@ -56,7 +54,6 @@ func (logService *LogService) CreateLog(createLogCommand *command.CreateLogComma
}
}
// 返回日志服务
func (logService *LogService) GetLog(getLogQuery *query.GetLogQuery) (interface{}, error) {
if err := getLogQuery.ValidateQuery(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
... ... @@ -93,7 +90,6 @@ func (logService *LogService) GetLog(getLogQuery *query.GetLogQuery) (interface{
}
}
// 返回日志服务列表
func (logService *LogService) ListLog(listLogQuery *query.ListLogQuery) (interface{}, error) {
if err := listLogQuery.ValidateQuery(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
... ... @@ -129,7 +125,6 @@ func (logService *LogService) ListLog(listLogQuery *query.ListLogQuery) (interfa
}
}
// 移除日志服务
func (logService *LogService) RemoveLog(removeLogCommand *command.RemoveLogCommand) (interface{}, error) {
if err := removeLogCommand.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
... ... @@ -169,7 +164,6 @@ func (logService *LogService) RemoveLog(removeLogCommand *command.RemoveLogComma
}
}
// 搜索日志
func (logService *LogService) SearchLog(searchLogCommand *command.SearchLogCommand) (int64, interface{}, error) {
if err := searchLogCommand.ValidateCommand(); err != nil {
return 0, nil, application.ThrowError(application.ARG_ERROR, err.Error())
... ... @@ -204,7 +198,6 @@ func (logService *LogService) SearchLog(searchLogCommand *command.SearchLogComma
return count, result, nil
}
// 搜索日志
func (logService *LogService) VerifiedStepLog(searchLogCommand *command.SearchLogCommand) (int64, interface{}, error) {
if err := searchLogCommand.ValidateCommand(); err != nil {
return 0, nil, application.ThrowError(application.ARG_ERROR, err.Error())
... ... @@ -252,7 +245,6 @@ func (logService *LogService) VerifiedStepLog(searchLogCommand *command.SearchLo
}, nil
}
// 更新日志服务
func (logService *LogService) UpdateLog(updateLogCommand *command.UpdateLogCommand) (interface{}, error) {
if err := updateLogCommand.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
... ...
... ... @@ -13,7 +13,7 @@ type UpdateMappingRuleCommand struct {
// 匹配规则ID
MappingRuleId int `cname:"匹配规则ID" json:"mappingRuleId" valid:"Required"`
// 名称
Name string `cname:"名称" json:"name" valid:"Required"`
Name string `cname:"名称" json:"name"`
// 校验文件列
MappingFields []*domain.MappingField `cname:"匹配规则列表" json:"mappingFields" valid:"Required"`
}
... ...
... ... @@ -13,11 +13,11 @@ import (
"time"
)
// 匹配规则服务
// MappingRuleService 匹配规则服务
type MappingRuleService struct {
}
// 创建匹配规则服务
// CreateMappingRule 创建匹配规则服务
func (mappingRuleService *MappingRuleService) CreateMappingRule(ctx *domain.Context, cmd *command.CreateMappingRuleCommand) (interface{}, error) {
if err := cmd.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
... ... @@ -47,7 +47,7 @@ func (mappingRuleService *MappingRuleService) CreateMappingRule(ctx *domain.Cont
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
if duplicateRule, e := mappingRuleRepository.FindOne(map[string]interface{}{"context": ctx, "name": cmd.Name}); e == nil && duplicateRule != nil {
if duplicateRule, e := mappingRuleRepository.FindOne(map[string]interface{}{"context": ctx, "name": cmd.Name, "tableId": cmd.TableId}); e == nil && duplicateRule != nil {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, "方案名称重复")
}
... ... @@ -81,7 +81,7 @@ func (mappingRuleService *MappingRuleService) CreateMappingRule(ctx *domain.Cont
return result, nil
}
// 返回匹配规则服务
// GetMappingRule 返回匹配规则服务
func (mappingRuleService *MappingRuleService) GetMappingRule(getMappingRuleQuery *query.GetMappingRuleQuery) (interface{}, error) {
if err := getMappingRuleQuery.ValidateQuery(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
... ... @@ -109,7 +109,7 @@ func (mappingRuleService *MappingRuleService) GetMappingRule(getMappingRuleQuery
return result, nil
}
// 返回匹配规则服务列表
// ListMappingRule 返回匹配规则服务列表
func (mappingRuleService *MappingRuleService) ListMappingRule(listMappingRuleQuery *query.ListMappingRuleQuery) (interface{}, error) {
if err := listMappingRuleQuery.ValidateQuery(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
... ... @@ -145,7 +145,7 @@ func (mappingRuleService *MappingRuleService) ListMappingRule(listMappingRuleQue
}
}
// 匹配规则预准备(新建规则)
// Prepare 匹配规则预准备(新建规则)
func (mappingRuleService *MappingRuleService) Prepare(prepareCommand *command.PrepareCommand) (interface{}, error) {
if err := prepareCommand.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
... ... @@ -182,7 +182,7 @@ func (mappingRuleService *MappingRuleService) Prepare(prepareCommand *command.Pr
return ruleDto, nil
}
// 移除匹配规则服务
// RemoveMappingRule 移除匹配规则服务
func (mappingRuleService *MappingRuleService) RemoveMappingRule(removeMappingRuleCommand *command.RemoveMappingRuleCommand) (interface{}, error) {
if err := removeMappingRuleCommand.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
... ... @@ -222,7 +222,7 @@ func (mappingRuleService *MappingRuleService) RemoveMappingRule(removeMappingRul
}{}, nil
}
// 搜索规则
// Search 搜索规则
func (mappingRuleService *MappingRuleService) Search(searchCommand *command.SearchCommand) (interface{}, error) {
if err := searchCommand.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
... ... @@ -259,7 +259,7 @@ func (mappingRuleService *MappingRuleService) Search(searchCommand *command.Sear
}, nil
}
// 更新匹配规则服务
// UpdateMappingRule 更新匹配规则服务
func (mappingRuleService *MappingRuleService) UpdateMappingRule(ctx *domain.Context, cmd *command.UpdateMappingRuleCommand) (interface{}, error) {
if err := cmd.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
... ... @@ -288,11 +288,14 @@ func (mappingRuleService *MappingRuleService) UpdateMappingRule(ctx *domain.Cont
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, err.Error())
}
if duplicateRule, e := mappingRuleRepository.FindOne(map[string]interface{}{"context": ctx, "name": cmd.Name}); e == nil && duplicateRule != nil && duplicateRule.MappingRuleId != cmd.MappingRuleId {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, "方案名称重复")
if len(cmd.Name) != 0 {
if duplicateRule, e := mappingRuleRepository.FindOne(map[string]interface{}{"context": ctx, "name": cmd.Name}); e == nil && duplicateRule != nil && duplicateRule.MappingRuleId != cmd.MappingRuleId {
return nil, application.ThrowError(application.INTERNAL_SERVER_ERROR, "方案名称重复")
}
}
if len(cmd.Name) > 0 {
mappingRule.Name = cmd.Name
}
mappingRule.Name = cmd.Name
mappingRule.MappingFields = cmd.MappingFields
mappingRule.VerifiedFileFields = fileTable.Fields(false)
mappingRule.UpdatedAt = time.Now()
... ...
package command
import (
"fmt"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type ChangeStatusCommand struct {
// 查询集合ID
QuerySetId int `cname:"查询集合ID" json:"querySetId" valid:"Required"`
// 状态 1:启用 2:关闭 (子过程默认启用)
Status int `cname:"状态 1:启用 2:关闭 (子过程默认启用)" json:"status" valid:"Required"`
}
func (changeStatusCommand *ChangeStatusCommand) Valid(validation *validation.Validation) {
if !(changeStatusCommand.Status == domain.StatusOn || changeStatusCommand.Status == domain.StatusOff) {
validation.Error("状态值有误")
}
}
func (changeStatusCommand *ChangeStatusCommand) ValidateCommand() error {
valid := validation.Validation{}
b, err := valid.Valid(changeStatusCommand)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(changeStatusCommand).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
package command
import (
"fmt"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type CopyCommand struct {
// Schema:方案 SubProcess:子过程
Type string `cname:"类型" json:"type" valid:"Required"`
// 父级ID
ParentId int `cname:"父级ID" json:"parentId" valid:"Required"`
// 名称
Name string `cname:"名称" json:"name" valid:"Required"`
// 查询集合ID
QuerySetId int `cname:"查询集合ID" json:"querySetId" valid:"Required"`
}
func (cmd *CopyCommand) Valid(validation *validation.Validation) {
if err := domain.ValidQuerySetType(cmd.Type); err != nil {
validation.Error(err.Error())
return
}
}
func (cmd *CopyCommand) ValidateCommand() error {
valid := validation.Validation{}
b, err := valid.Valid(cmd)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(cmd).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
package command
import (
"fmt"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type CreateQuerySetCommand struct {
// Schema:方案 SubProcess:子过程
Type string `cname:"类型" json:"type" valid:"Required"`
// 标识
Flag string `cname:"标识" json:"flag" valid:"Required"`
// 名称
Name string `cname:"名称" json:"name"`
// 父级ID
ParentId int `cname:"父级ID" json:"parentId,omitempty"`
}
func (cmd *CreateQuerySetCommand) Valid(validation *validation.Validation) {
if err := domain.ValidQuerySetType(cmd.Type); err != nil {
validation.Error(err.Error())
return
}
if err := domain.ValidQuerySetFlag(cmd.Flag); err != nil {
validation.Error(err.Error())
return
}
}
func (cmd *CreateQuerySetCommand) ValidateCommand() error {
valid := validation.Validation{}
b, err := valid.Valid(cmd)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(cmd).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
package command
import (
"fmt"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type MoveCommand struct {
// 查询集合ID
QuerySetId int `cname:"查询集合ID" json:"querySetId" valid:"Required"`
// 父级ID
ParentId int `cname:"父级ID" json:"parentId" valid:"Required"`
// 排序
Index int `cname:"序号" json:"index" valid:"Required"`
}
func (moveCommand *MoveCommand) Valid(validation *validation.Validation) {
//validation.SetError("CustomValid", "未实现的自定义认证")
}
func (moveCommand *MoveCommand) ValidateCommand() error {
valid := validation.Validation{}
b, err := valid.Valid(moveCommand)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(moveCommand).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
package command
import (
"fmt"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type RefreshQuerySetCommand struct {
// 查询集合ID
QuerySetId int `cname:"查询集合ID" json:"querySetId" valid:"Required"`
}
func (updateQuerySetCommand *RefreshQuerySetCommand) Valid(validation *validation.Validation) {
//validation.SetError("CustomValid", "未实现的自定义认证")
}
func (updateQuerySetCommand *RefreshQuerySetCommand) ValidateCommand() error {
valid := validation.Validation{}
b, err := valid.Valid(updateQuerySetCommand)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(updateQuerySetCommand).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
package command
import (
"fmt"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type RemoveQuerySetCommand struct {
// 查询集合ID
QuerySetId int `cname:"查询集合ID" json:"querySetId" valid:"Required"`
}
func (removeQuerySetCommand *RemoveQuerySetCommand) Valid(validation *validation.Validation) {
//validation.SetError("CustomValid", "未实现的自定义认证")
}
func (removeQuerySetCommand *RemoveQuerySetCommand) ValidateCommand() error {
valid := validation.Validation{}
b, err := valid.Valid(removeQuerySetCommand)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(removeQuerySetCommand).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
package command
import (
"fmt"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type RenameCommand struct {
// 查询集合ID
QuerySetId int `cname:"查询集合ID" json:"querySetId" valid:"Required"`
// 名称
Name string `cname:"名称" json:"name" valid:"Required"`
}
func (renameCommand *RenameCommand) Valid(validation *validation.Validation) {
//validation.SetError("CustomValid", "未实现的自定义认证")
}
func (renameCommand *RenameCommand) ValidateCommand() error {
valid := validation.Validation{}
b, err := valid.Valid(renameCommand)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(renameCommand).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
package command
import (
"fmt"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type UpdateQuerySetCommand struct {
// 查询集合ID
QuerySetId int `cname:"查询集合ID" json:"querySetId" valid:"Required"`
QueryComponents []*domain.QueryComponent `cname:"查询组件" json:"queryComponents" valid:"Required"`
}
func (updateQuerySetCommand *UpdateQuerySetCommand) Valid(validation *validation.Validation) {
//validation.SetError("CustomValid", "未实现的自定义认证")
}
func (updateQuerySetCommand *UpdateQuerySetCommand) ValidateCommand() error {
valid := validation.Validation{}
b, err := valid.Valid(updateQuerySetCommand)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(updateQuerySetCommand).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
package dto
import "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
type QuerySetDetailDto struct {
// 查询集合ID
QuerySetId int `json:"querySetId"`
// Schema:方案 SubProcess:子过程
Type string `json:"type"`
// 标识 分组:Group 子过程/方案:Set
Flag string `json:"flag"`
// 名称
Name string `json:"name"`
// 查询组件
QueryComponents []*domain.QueryComponent `json:"queryComponents"`
// 查询集绑定的表
TableId int `json:"tableId"`
// 在冲突状态 true:冲突异常 false:正常
InConflict bool `json:"inConflict"`
}
func (d *QuerySetDetailDto) Load(m *domain.QuerySet, mapTables map[int]*domain.Table) *QuerySetDetailDto {
d.QuerySetId = m.QuerySetId
d.Type = m.Type
d.Flag = m.Flag
d.Name = m.Name
d.QueryComponents = m.QueryComponents
if m.QuerySetInfo != nil {
d.TableId = m.QuerySetInfo.BindTableId
}
hasUpdateTable := false
for i, q := range d.QueryComponents {
if q.MasterTable != nil && q.MasterTable.TableId != 0 {
if t, ok := mapTables[q.MasterTable.TableId]; ok {
d.QueryComponents[i].MasterTable = domain.NewQueryComponentTable(t)
d.QueryComponents[i].UpdateTables(t)
hasUpdateTable = true
}
}
if d.QueryComponents[i].Aggregation != nil {
d.QueryComponents[i].Aggregation.Aggregation.AllFields = d.QueryComponents[i].Aggregation.AggregationFields()
}
if !hasUpdateTable && len(mapTables) == 1 {
for _, t := range mapTables {
d.QueryComponents[i].UpdateTables(t)
hasUpdateTable = true
break
}
}
}
d.InConflict = false
if m.QuerySetInfo != nil {
if m.QuerySetInfo.ReadyStatus == 1 {
d.InConflict = true
}
}
return d
}
... ...
package dto
import "gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
type QuerySetDto struct {
// 查询集合ID
QuerySetId int `json:"querySetId"`
// Schema:方案 SubProcess:子过程
Type string `json:"type"`
// 标识 分组:Group 子过程/方案:Set
Flag string `json:"flag"`
// 名称
Name string `json:"name"`
// 父级ID
ParentId int `json:"parentId"`
// 状态 1:启用 2:关闭 (子过程默认启用)
Status int `json:"status"`
// 排序
Sort int `json:"sort"`
// 时间
Time string `json:"time"`
// 绑定的表ID
BindTableId int `json:"tableId"`
// 在冲突状态 true:冲突异常 false:正常
InConflict bool `json:"inConflict"`
}
func (d *QuerySetDto) Load(m *domain.QuerySet) *QuerySetDto {
d.QuerySetId = m.QuerySetId
d.Type = m.Type
d.Flag = m.Flag
d.Name = m.Name
d.ParentId = m.ParentId
d.Status = m.Status
d.Sort = m.Sort
d.Time = m.CreatedAt.Local().Format("2006-01-02 15:04:05")
d.BindTableId = m.QuerySetInfo.BindTableId
d.InConflict = m.QuerySetInfo.ReadyStatus == 1
return d
}
func NewQuerySetDtoList(querySets []*domain.QuerySet) []*QuerySetDto {
var result = make([]*QuerySetDto, 0)
for _, set := range querySets {
var item = &QuerySetDto{}
item.Load(set)
result = append(result, item)
}
return result
}
... ...
package query
import (
"fmt"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type CalculateItemPreviewQuery struct {
// 查询集合ID
QuerySetId int `cname:"查询集合ID" json:"querySetId" valid:"Required"`
// 公式
Formula *domain.FieldExpr `json:"formula" valid:"Required"`
}
func (dependencyGraphQuery *CalculateItemPreviewQuery) Valid(validation *validation.Validation) {
//validation.SetError("CustomValid", "未实现的自定义认证")
}
func (dependencyGraphQuery *CalculateItemPreviewQuery) ValidateQuery() error {
valid := validation.Validation{}
b, err := valid.Valid(dependencyGraphQuery)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(dependencyGraphQuery).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
package query
import (
"fmt"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type DependencyGraphQuery struct {
// 查询集合ID
QuerySetId int `cname:"查询集合ID" json:"querySetId" valid:"Required"`
}
func (dependencyGraphQuery *DependencyGraphQuery) Valid(validation *validation.Validation) {
//validation.SetError("CustomValid", "未实现的自定义认证")
}
func (dependencyGraphQuery *DependencyGraphQuery) ValidateQuery() error {
valid := validation.Validation{}
b, err := valid.Valid(dependencyGraphQuery)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(dependencyGraphQuery).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
package query
import (
"fmt"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type GetQuerySetQuery struct {
// 查询集合ID
QuerySetId int `cname:"查询集合ID" json:"querySetId" valid:"Required"`
}
func (getQuerySetQuery *GetQuerySetQuery) Valid(validation *validation.Validation) {
//validation.SetError("CustomValid", "未实现的自定义认证")
}
func (getQuerySetQuery *GetQuerySetQuery) ValidateQuery() error {
valid := validation.Validation{}
b, err := valid.Valid(getQuerySetQuery)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(getQuerySetQuery).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
package query
import (
"fmt"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type ListQuerySetQuery struct {
// 查询偏离量
Offset int `cname:"查询偏离量" json:"offset" valid:"Required"`
// 查询限制
Limit int `cname:"查询限制" json:"limit" valid:"Required"`
}
func (listQuerySetQuery *ListQuerySetQuery) Valid(validation *validation.Validation) {
//validation.SetError("CustomValid", "未实现的自定义认证")
}
func (listQuerySetQuery *ListQuerySetQuery) ValidateQuery() error {
valid := validation.Validation{}
b, err := valid.Valid(listQuerySetQuery)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(listQuerySetQuery).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
package query
import (
"fmt"
"github.com/beego/beego/v2/core/validation"
"reflect"
"strings"
)
type StatisticsQuery struct {
}
func (q *StatisticsQuery) Valid(validation *validation.Validation) {
//validation.SetError("CustomValid", "未实现的自定义认证")
}
func (q *StatisticsQuery) ValidateQuery() error {
valid := validation.Validation{}
b, err := valid.Valid(q)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(q).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
package query
import (
"fmt"
"github.com/beego/beego/v2/adapter/utils"
"reflect"
"strings"
"github.com/beego/beego/v2/core/validation"
)
type SearchQuerySetQuery struct {
Type string `cname:"类型" json:"type"`
Types []string `cname:"类型" json:"types"`
Flag string `cname:"标识" json:"flag"`
Status int `cname:"状态 1:启用 2:关闭" json:"status"`
MatchName string `cname:"匹配名称" json:"matchName"`
SortByName string `json:"sortByName"`
SortByTime string `json:"sortByTime"`
}
func (searchQuerySetQuery *SearchQuerySetQuery) Valid(validation *validation.Validation) {
optionsValues := []string{"asc", "desc"}
if searchQuerySetQuery.SortByName != "" && !utils.InSlice(strings.ToLower(searchQuerySetQuery.SortByName), optionsValues) {
validation.Error("排序值有误 ASC|DESC")
}
if searchQuerySetQuery.SortByTime != "" && !utils.InSlice(strings.ToLower(searchQuerySetQuery.SortByTime), optionsValues) {
validation.Error("排序值有误 ASC|DESC")
}
}
func (searchQuerySetQuery *SearchQuerySetQuery) ValidateQuery() error {
valid := validation.Validation{}
b, err := valid.Valid(searchQuerySetQuery)
if err != nil {
return err
}
if !b {
elem := reflect.TypeOf(searchQuerySetQuery).Elem()
for _, validErr := range valid.Errors {
field, isExist := elem.FieldByName(validErr.Field)
if isExist {
return fmt.Errorf(strings.Replace(validErr.Message, validErr.Field, field.Tag.Get("cname"), -1))
} else {
return fmt.Errorf(validErr.Message)
}
}
}
return nil
}
... ...
package service
import (
"fmt"
"github.com/linmadan/egglib-go/core/application"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/factory"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/querySet/command"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/application/table/dto"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/domain"
"gitlab.fjmaimaimai.com/allied-creation/character-library-metadata-bastion/pkg/infrastructure/excel"
"time"
)
// CalculateSetPreview 计算集预览
func (querySetService *QuerySetService) CalculateSetPreview(ctx *domain.Context, updateQuerySetCommand *command.UpdateQuerySetCommand) (interface{}, error) {
if err := updateQuerySetCommand.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
var querySet *domain.QuerySet
_, querySet, err = factory.FastPgQuerySet(transactionContext, updateQuerySetCommand.QuerySetId)
if err != nil {
return nil, factory.FastError(err)
}
svr, _ := factory.FastQuerySetServices(transactionContext)
var dataTable *domain.DataTable
dataTable, err = svr.LoadCalculateSetData(ctx, querySet, updateQuerySetCommand.QueryComponents)
if err != nil {
return nil, factory.FastError(err)
}
response := (&dto.TablePreviewDto{})
response.ObjectType = querySet.Type
response.Name = querySet.Name
response.Fields = dataTable.Fields
response.Data = domain.GripData(domain.ToFieldData(dataTable.Fields, dataTable.Data, false, true), int64(len(dataTable.Data)))
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return response, nil
}
// CalculateSetExport 计算集导出
func (querySetService *QuerySetService) CalculateSetExport(ctx *domain.Context, updateQuerySetCommand *command.UpdateQuerySetCommand) (interface{}, error) {
if err := updateQuerySetCommand.ValidateCommand(); err != nil {
return nil, application.ThrowError(application.ARG_ERROR, err.Error())
}
transactionContext, err := factory.CreateTransactionContext(nil)
if err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
if err := transactionContext.StartTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
defer func() {
transactionContext.RollbackTransaction()
}()
var querySet *domain.QuerySet
_, querySet, err = factory.FastPgQuerySet(transactionContext, updateQuerySetCommand.QuerySetId)
if err != nil {
return nil, factory.FastError(err)
}
svr, _ := factory.FastQuerySetServices(transactionContext)
var dataTable *domain.DataTable
dataTable, err = svr.LoadCalculateSetData(ctx, querySet, updateQuerySetCommand.QueryComponents)
if err != nil {
return nil, factory.FastError(err)
}
var fields []string
filename := fmt.Sprintf("%v_%v.xlsx", querySet.Name, time.Now().Format("060102150405"))
path := fmt.Sprintf("public/%v", filename)
for _, f := range dataTable.Fields {
fields = append(fields, f.Name)
}
excelWriter := excel.NewXLXSWriterTo(fields, dataTable.Data)
excelWriter.IgnoreTitle = false
if err = excelWriter.Save(path); err != nil {
return nil, factory.FastError(err)
}
if err := transactionContext.CommitTransaction(); err != nil {
return nil, application.ThrowError(application.TRANSACTION_ERROR, err.Error())
}
return map[string]interface{}{
"url": domain.DownloadUrl(filename),
}, err
}
... ...