Compare commits

...

10 Commits

@ -0,0 +1,210 @@
# Cursor项目规范
## 基本设置
- 使用中文回答所有问题
- 按照Sequential Thinking方法进行代码设计和实现
## 依赖约束
- 使用Maven作为依赖管理工具
- 使用Maven项目结构
- 使用SpringBoot2.1.4.RELEASE作为框架
- 使用JDK1.8作为开发语言
- 使用Lombok作为代码简化工具
## 项目结构和命名规范
- 组件命名PascalCase
- 函数命名camelCase
- 常量命名UPPER_CASE
- 文件夹结构遵循Maven项目规范
- 接口命名使用I前缀如IUserService
- 类型定义使用T前缀如TUserData
- 文档类统一输出至docs文件夹并使用Markdown格式
## 代码风格
- 使用TypeScript类型定义
- 统一代码缩进和格式
- 使用ESLint和Prettier规则
## 开发流程规则
1. 问题分析:明确任务目标和约束条件
2. 设计方案:设计数据结构和算法
3. 实现步骤:自顶向下实现功能
4. 测试验证:编写单元测试和集成测试
## 代码审查标准
- 代码必须符合既定规范
- 关键函数和组件必须有注释
- 复杂算法需要解释思路
## 提交规范
- 遵循语义化提交信息格式feat:, fix:, docs:等
- 主分支只接受经过测试的代码
## 文档要求
- API接口必须有清晰文档
- 更新README文件反映最新项目状态
## 代码稳定性保障
- 修改已完成功能前必须先理解其完整设计意图
- 所有API更改必须向下兼容
- 使用单元测试保护核心功能逻辑
- 重构代码时必须保持功能等价性
- 在修改前创建功能快照或临时分支
- 每次更改后必须验证不破坏现有功能
- 使用TODO或FIXME标签清晰标记未完成修改
## 版本控制规则
- 使用语义化版本管理
- 关键功能更改必须通过代码审查
- 维护变更日志记录所有修改
- 为已稳定功能模块添加"锁定注释"标记:/* @stable - 请勿修改 */
## UI设计规范
- 遵循现代设计趋势和最佳实践
- 使用设计系统确保一致性如Material Design、Ant Design或自定义设计系统
- 实现响应式设计,确保在不同设备上显示良好
- 使用CSS变量统一管理颜色、字体、间距等设计标记
- 优先采用Flexbox或Grid布局系统
- 确保适当的留白和视觉层次
- 实现无障碍访问标准WCAG 2.1
## UI组件标准
- 使用组件库作为基础如MUI、Chakra UI、Tailwind UI等
- 自定义组件需符合现代设计审美
- 设计组件应包含默认、悬停、聚焦、禁用等状态
- 使用适当的动画和过渡效果增强用户体验
- 确保设计一致性:同类元素使用相同样式
- 使用主题系统支持亮色/暗色模式切换
- 根据用户操作提供视觉反馈
## 设计资源
- 维护设计风格指南和UI组件库
- 使用标准化图标库如Heroicons、Material Icons等
- 图片和插图需保持一致的风格
- 颜色选择须符合品牌标识并确保足够对比度
## 依赖管理规范
- 优先使用国内镜像站点安装依赖
- npm包使用淘宝镜像https://registry.npmmirror.com/
- yarn设置yarn config set registry https://registry.npmmirror.com
- pnpm设置pnpm config set registry https://registry.npmmirror.com
- pip包使用清华镜像https://pypi.tuna.tsinghua.edu.cn/simple
- Docker镜像使用阿里云https://cr.console.aliyun.com/
- Maven依赖使用阿里云https://maven.aliyun.com/repository/public
- Gradle依赖使用阿里云https://developer.aliyun.com/mvn/guide
- 安装新依赖前先验证其在国内是否可访问
- 如确实需使用国外资源,应提供备选方案或离线安装包
- package.json中添加镜像设置脚本便于团队统一配置
- 记录所有依赖的具体版本和来源以便追踪
## 测试自动化规范
- 编写单元测试覆盖所有关键功能
- 实现端到端测试验证用户流程
- 使用测试驱动开发(TDD)方法
- 每次提交前运行自动化测试
- 维护测试数据与生产环境隔离
## 调试与日志规范
- 统一使用日志级别error, warn, info, debug
- 日志信息需包含上下文和时间戳
- 生产环境禁用调试代码和console语句
- 关键流程增加日志记录点便于问题追踪
- 使用结构化日志格式便于分析
## 代码复杂度控制
- 函数不超过50行单个文件不超过300行
- 每个函数只做一件事情,保持单一职责
- 嵌套不超过3层避免过深条件嵌套
- 控制圈复杂度不超过10
- 复杂逻辑应拆分为多个小函数
## 状态管理规范
- 明确状态管理方案如Redux、MobX或Context API
- 区分本地状态和全局状态
- 避免状态冗余和重复存储
- 实现不可变状态更新模式
- 为复杂状态提供初始值和验证机制
## 异步操作规范
- 统一使用async/await或Promise
- 实现请求超时和重试机制
- 处理并发请求限制
- 取消不必要的请求以节省资源
- 使用Loading状态指示异步操作进行中
## 代码重用策略
- 提取通用逻辑为Hooks或工具函数
- 使用组合而非继承实现代码复用
- 避免复制粘贴代码,而应重构为共享组件
- 通用功能应考虑发布为内部npm包
- 明确区分业务逻辑和技术实现
## 项目文档体系
- 维护项目架构图和关键流程图
- 记录技术选型理由和限制条件
- 为API和数据模型提供详细文档
- 记录已知问题和解决方案
- 提供新开发者入门指南
## 构建与发布流程
- 使用CI/CD实现自动化构建和部署
- 区分开发、测试、预发布和生产环境
- 实现回滚机制应对紧急问题
- 使用版本化静态资源避免缓存问题
- 配置文件应随环境变化而不需修改代码
## 可访问性和兼容性规范
- 确保键盘可访问性
- 使用ARIA标签增强屏幕阅读器支持
- 确保适当的颜色对比度
- 响应式设计支持从移动到桌面的所有设备
- 针对低带宽和弱网络场景优化
## 问题追踪与修复流程
- 使用标准化的问题报告模板
- 问题修复前先编写复现步骤
- 每个bug修复应包含相应测试避免回归
- 重大问题需进行根本原因分析
- 定期审查常见错误类型并改进开发流程
## 代码注释最佳实践
- 写清楚为什么这样做,而不仅是做了什么
- 用注释标记未来需要优化的地方
- 所有公共API必须有注释文档
- 复杂业务逻辑需要解释业务规则
- 避免废弃或过时的注释
## 数据处理与安全
- 敏感数据传输必须加密
- 用户输入必须验证和清洗
- 实现数据备份和恢复策略
- 遵循数据最小化原则,只收集必要信息
- 实现适当的数据过期和销毁机制
## 性能优化指南
- 实现资源懒加载和按需加载
- 优化关键渲染路径提高首屏加载速度
- 使用适当的缓存策略减少网络请求
- 批量处理DOM操作避免重排和重绘
- 定期进行性能审计和优化
## 可维护性原则
- 优先考虑代码可理解性而非简洁性
- 避免过早优化和不必要的抽象
- 关键决策需在代码中记录理由
- 避免使用黑魔法和难以理解的技巧
- 团队培训和知识共享机制
## 技术债务管理
- 使用TODO和FIXME标记需要改进的地方
- 定期安排时间清理技术债务
- 新功能实现前评估现有代码质量
- 记录所有已知问题和临时解决方案
- 使用代码质量工具定期评估项目健康度
## 用户体验设计规范
- 所有操作必须提供用户反馈
- 错误信息应该清晰并提供解决方案
- 降低操作复杂度,减少用户认知负担
- 界面风格保持一致性
- 考虑边缘情况和用户出错恢复路径

6
.gitignore vendored

@ -1,6 +1,6 @@
/GFSData/
/tablestoreConf.json
/log/
/log*/
/*/target/
/tablestore-grid-master/tableConf.json
/tablestore-grid-master/tablestoreConf.json
@ -21,3 +21,7 @@
*.iws
*.iml
*.ipr
*.json
*.cursor

@ -0,0 +1,69 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<!--父pom.xml-->
<groupId>com.htfp</groupId>
<artifactId>weather</artifactId>
<version>0.0.1</version>
<packaging>pom</packaging>
<name>weather app</name>
<description>weather service prepublish</description>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.6.13</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>1.8</java.version>
</properties>
<!--声明子模块-->
<modules>
<module>tablestore-grid-master</module>
<module>weather-service</module>
</modules>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<skipTests>false</skipTests> <!--默认关掉单元测试 -->
</configuration>
</plugin>
</plugins>
</build>
</project>

@ -7,6 +7,20 @@
<groupId>com.aliyun.tablestore</groupId>
<artifactId>tablestore-grid</artifactId>
<version>1.1-SNAPSHOT</version>
<packaging>jar</packaging>
<parent>
<groupId>com.htfp</groupId>
<artifactId>weather</artifactId>
<version>0.0.1</version>
<relativePath>../pom.xml</relativePath>
</parent>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>1.8</java.version>
</properties>
<repositories>
<repository>
@ -113,10 +127,18 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>8</source>
<target>8</target>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
</plugins>
</build>
</project>

@ -50,25 +50,28 @@ public class TableStoreGrid implements GridStore {
}
@Override
public void createStore(TableOptions tableOptions) throws Exception {
// create meta table
this.tableOptions = tableOptions;
// create buffer table
try {
this.tableOptions = tableOptions;
this.asyncClient.createTable(RequestBuilder.buildCreateMetaTableRequest(config.getMetaTableName(), tableOptions), null).get();
tableOptions.setAllowUpdate(true); // 数据表必须允许更新,否则无法导入数据
this.asyncClient.createTable(RequestBuilder.buildCreateDataTableRequest(config.getDataTableName(), tableOptions), null).get();
} catch (TableStoreException ex) {
if (!ex.getErrorCode().equals(ErrorCode.OBJECT_ALREADY_EXIST)) {
throw ex;
}
}
// create buffer table
// create meta table
try {
tableOptions.setAllowUpdate(false); // 禁止update操作
this.asyncClient.createTable(RequestBuilder.buildCreateDataTableRequest(config.getDataTableName(), tableOptions), null).get();
tableOptions.setAllowUpdate(false); // 属性表为了设置过期时间,禁止update操作只允许put操作
this.asyncClient.createTable(RequestBuilder.buildCreateMetaTableRequest(config.getMetaTableName(), tableOptions), null).get();
} catch (TableStoreException ex) {
if (!ex.getErrorCode().equals(ErrorCode.OBJECT_ALREADY_EXIST)) {
throw ex;
}
}
}
/**

@ -1,5 +1,9 @@
package com.aliyun.tablestore.grid.consts;
import com.alicloud.openservices.tablestore.model.search.FieldType;
import java.util.function.Consumer;
/**
* @Author : shiyi
* @Date : 2024/5/8 15:27
@ -7,14 +11,14 @@ package com.aliyun.tablestore.grid.consts;
*/
public enum AttributionEnum {
//
STATUS("status", "导入状态"),
CREATE_TIME("create_time", "创建时间"),
REFERENCE_TIME("reference_time", "起报时间"),
STATUS("status", "导入状态", FieldType.KEYWORD),
CREATE_TIME("create_time", "创建时间", FieldType.LONG),
REFERENCE_TIME("reference_time", "起报时间", FieldType.LONG),
;
public String name;
public String info;
public FieldType type;
public String getName() {
return name;
}
@ -22,8 +26,13 @@ public enum AttributionEnum {
public String getInfo() {
return info;
}
AttributionEnum(String name, String info) {
public FieldType getType() {
return type;
}
AttributionEnum(String name, String info, FieldType type) {
this.name = name;
this.info = info;
this.type = type;
}
}

@ -91,6 +91,7 @@ public class TableStoreDataWriter implements GridDataWriter {
if (meta.getStoreOptions().getStoreType().equals(StoreOptions.StoreType.SLICE)) {
List<Column> columns = splitDataToColumns(grid2D);
writeToTableStore(variable, t, z, columns);
columns = null;
} else {
throw new IllegalArgumentException("unsupported store type");
}

@ -52,7 +52,6 @@ public abstract class Grid {
}
public Array toArray() {
Array array = Array.factory(dataType, shape, buffer.duplicate());
return array;
return Array.factory(dataType, shape, buffer.duplicate());
}
}

@ -1,4 +1,7 @@
import com.aliyun.tablestore.grid.model.grid.Grid;
import com.aliyun.tablestore.grid.model.grid.Grid2D;
import ucar.ma2.Array;
import ucar.ma2.DataType;
import ucar.ma2.Index;
import ucar.ma2.InvalidRangeException;
import ucar.nc2.NetcdfFile;
@ -6,6 +9,7 @@ import ucar.nc2.NetcdfFiles;
import ucar.nc2.Variable;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
@ -13,7 +17,15 @@ import java.time.format.DateTimeFormatter;
* Unit test for simple App.
*/
public class TestReadGrib2ByNC {
public static void main(String[] args){
public static void main(String[] args) throws InterruptedException {
// while (true) {
// ByteBuffer allocate = ByteBuffer.allocate(500 * 500 * 4);
// Grid2D grid = new Grid2D(allocate, DataType.FLOAT, new int[]{0, 0}, new int[]{500,500});
// allocate = null; grid=null;
// Thread.sleep(300);
// }
}
public static void main1(String[] args){
// String ncFile = "surface.nc";
String file = "C:\\Users\\shi_y\\Desktop\\java_learn\\data_download\\GFSData\\UTC-20230910\\BJT-20230911-0200.grib2";
try (NetcdfFile ncFile = NetcdfFiles.open(file)) {

@ -42,7 +42,7 @@ public class CreateStoreExample extends TableStoreGridExample {
this.tableStoreGrid.createMetaIndex(ExampleConfig.GRID_META_INDEX_NAME, indexSchema);
}
public static void main(String[] args) throws Exception {
public static void test(String[] args) throws Exception {
CreateStoreExample example = new CreateStoreExample();
try {
example.createStore();

@ -6,7 +6,6 @@ import com.aliyun.tablestore.grid.GridDataFetcher;
import com.aliyun.tablestore.grid.model.grid.Grid4D;
import ucar.ma2.Array;
import java.util.Arrays;
import java.util.Collections;
public class DataFetchExample extends TableStoreGridExample {
@ -71,7 +70,7 @@ public class DataFetchExample extends TableStoreGridExample {
fetch3();
}
public static void main(String[] args) throws Exception {
public static void test(String[] args) throws Exception {
DataFetchExample example = new DataFetchExample();
try {
example.run();

@ -12,7 +12,6 @@ import ucar.ma2.DataType;
import ucar.nc2.NetcdfFile;
import ucar.nc2.Variable;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
@ -97,7 +96,7 @@ public class DataImportExample extends TableStoreGridExample {
updateMeta(meta);
}
public static void main(String[] args) throws Exception {
public static void test(String[] args) throws Exception {
DataImportExample example = new DataImportExample();
try {
log.info("导入数据");

@ -11,7 +11,6 @@ import com.aliyun.tablestore.grid.model.QueryGridDataSetResult;
import com.aliyun.tablestore.grid.model.QueryParams;
import lombok.extern.slf4j.Slf4j;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;

@ -0,0 +1,210 @@
# Cursor项目规范
## 基本设置
- 使用中文回答所有问题
- 按照Sequential Thinking方法进行代码设计和实现
## 依赖约束
- 使用Maven作为依赖管理工具
- 使用Maven项目结构
- 使用SpringBoot2.1.4.RELEASE作为框架
- 使用JDK1.8作为开发语言
- 使用Lombok作为代码简化工具
## 项目结构和命名规范
- 组件命名PascalCase
- 函数命名camelCase
- 常量命名UPPER_CASE
- 文件夹结构遵循Maven项目规范
- 接口命名使用I前缀如IUserService
- 类型定义使用T前缀如TUserData
- 文档类统一输出至docs文件夹并使用Markdown格式
## 代码风格
- 使用TypeScript类型定义
- 统一代码缩进和格式
- 使用ESLint和Prettier规则
## 开发流程规则
1. 问题分析:明确任务目标和约束条件
2. 设计方案:设计数据结构和算法
3. 实现步骤:自顶向下实现功能
4. 测试验证:编写单元测试和集成测试
## 代码审查标准
- 代码必须符合既定规范
- 关键函数和组件必须有注释
- 复杂算法需要解释思路
## 提交规范
- 遵循语义化提交信息格式feat:, fix:, docs:等
- 主分支只接受经过测试的代码
## 文档要求
- API接口必须有清晰文档
- 更新README文件反映最新项目状态
## 代码稳定性保障
- 修改已完成功能前必须先理解其完整设计意图
- 所有API更改必须向下兼容
- 使用单元测试保护核心功能逻辑
- 重构代码时必须保持功能等价性
- 在修改前创建功能快照或临时分支
- 每次更改后必须验证不破坏现有功能
- 使用TODO或FIXME标签清晰标记未完成修改
## 版本控制规则
- 使用语义化版本管理
- 关键功能更改必须通过代码审查
- 维护变更日志记录所有修改
- 为已稳定功能模块添加"锁定注释"标记:/* @stable - 请勿修改 */
## UI设计规范
- 遵循现代设计趋势和最佳实践
- 使用设计系统确保一致性如Material Design、Ant Design或自定义设计系统
- 实现响应式设计,确保在不同设备上显示良好
- 使用CSS变量统一管理颜色、字体、间距等设计标记
- 优先采用Flexbox或Grid布局系统
- 确保适当的留白和视觉层次
- 实现无障碍访问标准WCAG 2.1
## UI组件标准
- 使用组件库作为基础如MUI、Chakra UI、Tailwind UI等
- 自定义组件需符合现代设计审美
- 设计组件应包含默认、悬停、聚焦、禁用等状态
- 使用适当的动画和过渡效果增强用户体验
- 确保设计一致性:同类元素使用相同样式
- 使用主题系统支持亮色/暗色模式切换
- 根据用户操作提供视觉反馈
## 设计资源
- 维护设计风格指南和UI组件库
- 使用标准化图标库如Heroicons、Material Icons等
- 图片和插图需保持一致的风格
- 颜色选择须符合品牌标识并确保足够对比度
## 依赖管理规范
- 优先使用国内镜像站点安装依赖
- npm包使用淘宝镜像https://registry.npmmirror.com/
- yarn设置yarn config set registry https://registry.npmmirror.com
- pnpm设置pnpm config set registry https://registry.npmmirror.com
- pip包使用清华镜像https://pypi.tuna.tsinghua.edu.cn/simple
- Docker镜像使用阿里云https://cr.console.aliyun.com/
- Maven依赖使用阿里云https://maven.aliyun.com/repository/public
- Gradle依赖使用阿里云https://developer.aliyun.com/mvn/guide
- 安装新依赖前先验证其在国内是否可访问
- 如确实需使用国外资源,应提供备选方案或离线安装包
- package.json中添加镜像设置脚本便于团队统一配置
- 记录所有依赖的具体版本和来源以便追踪
## 测试自动化规范
- 编写单元测试覆盖所有关键功能
- 实现端到端测试验证用户流程
- 使用测试驱动开发(TDD)方法
- 每次提交前运行自动化测试
- 维护测试数据与生产环境隔离
## 调试与日志规范
- 统一使用日志级别error, warn, info, debug
- 日志信息需包含上下文和时间戳
- 生产环境禁用调试代码和console语句
- 关键流程增加日志记录点便于问题追踪
- 使用结构化日志格式便于分析
## 代码复杂度控制
- 函数不超过50行单个文件不超过300行
- 每个函数只做一件事情,保持单一职责
- 嵌套不超过3层避免过深条件嵌套
- 控制圈复杂度不超过10
- 复杂逻辑应拆分为多个小函数
## 状态管理规范
- 明确状态管理方案如Redux、MobX或Context API
- 区分本地状态和全局状态
- 避免状态冗余和重复存储
- 实现不可变状态更新模式
- 为复杂状态提供初始值和验证机制
## 异步操作规范
- 统一使用async/await或Promise
- 实现请求超时和重试机制
- 处理并发请求限制
- 取消不必要的请求以节省资源
- 使用Loading状态指示异步操作进行中
## 代码重用策略
- 提取通用逻辑为Hooks或工具函数
- 使用组合而非继承实现代码复用
- 避免复制粘贴代码,而应重构为共享组件
- 通用功能应考虑发布为内部npm包
- 明确区分业务逻辑和技术实现
## 项目文档体系
- 维护项目架构图和关键流程图
- 记录技术选型理由和限制条件
- 为API和数据模型提供详细文档
- 记录已知问题和解决方案
- 提供新开发者入门指南
## 构建与发布流程
- 使用CI/CD实现自动化构建和部署
- 区分开发、测试、预发布和生产环境
- 实现回滚机制应对紧急问题
- 使用版本化静态资源避免缓存问题
- 配置文件应随环境变化而不需修改代码
## 可访问性和兼容性规范
- 确保键盘可访问性
- 使用ARIA标签增强屏幕阅读器支持
- 确保适当的颜色对比度
- 响应式设计支持从移动到桌面的所有设备
- 针对低带宽和弱网络场景优化
## 问题追踪与修复流程
- 使用标准化的问题报告模板
- 问题修复前先编写复现步骤
- 每个bug修复应包含相应测试避免回归
- 重大问题需进行根本原因分析
- 定期审查常见错误类型并改进开发流程
## 代码注释最佳实践
- 写清楚为什么这样做,而不仅是做了什么
- 用注释标记未来需要优化的地方
- 所有公共API必须有注释文档
- 复杂业务逻辑需要解释业务规则
- 避免废弃或过时的注释
## 数据处理与安全
- 敏感数据传输必须加密
- 用户输入必须验证和清洗
- 实现数据备份和恢复策略
- 遵循数据最小化原则,只收集必要信息
- 实现适当的数据过期和销毁机制
## 性能优化指南
- 实现资源懒加载和按需加载
- 优化关键渲染路径提高首屏加载速度
- 使用适当的缓存策略减少网络请求
- 批量处理DOM操作避免重排和重绘
- 定期进行性能审计和优化
## 可维护性原则
- 优先考虑代码可理解性而非简洁性
- 避免过早优化和不必要的抽象
- 关键决策需在代码中记录理由
- 避免使用黑魔法和难以理解的技巧
- 团队培训和知识共享机制
## 技术债务管理
- 使用TODO和FIXME标记需要改进的地方
- 定期安排时间清理技术债务
- 新功能实现前评估现有代码质量
- 记录所有已知问题和临时解决方案
- 使用代码质量工具定期评估项目健康度
## 用户体验设计规范
- 所有操作必须提供用户反馈
- 错误信息应该清晰并提供解决方案
- 降低操作复杂度,减少用户认知负担
- 界面风格保持一致性
- 考虑边缘情况和用户出错恢复路径

@ -0,0 +1,474 @@
# Weather Service 技术实现文档
## 1. Weather Service 项目结构
```
src/main/java/com/htfp/weather/
├── WeatherServiceApplication.java # 应用程序入口
├── config/ # 配置类
├── download/ # 数据下载模块
├── griddata/ # 网格数据处理
├── info/ # 信息处理模块
├── schedule/ # 定时任务模块
├── utils/ # 工具类
└── web/ # Web接口模块
```
## 2. 核心模块分析
### 2.1 应用程序入口
文件:`WeatherServiceApplication.java`
```java
@EnableRetry // 开启重试机制
@EnableScheduling // 开启定时任务
@SpringBootApplication
public class WeatherServiceApplication {
public static void main(String[] args) {
TimeZone.setDefault(TimeZone.getTimeZone("Asia/Shanghai"));
SpringApplication.run(WeatherServiceApplication.class, args);
}
}
```
主要特性:
1. 启用Spring Boot自动配置
2. 启用重试机制,支持失败操作重试
3. 启用定时任务调度功能
4. 设置默认时区为亚洲/上海
### 2.2 数据下载模块
目录:`download/`
#### 2.2.1 基础下载器BaseDataDownloader
文件:`BaseDataDownloader.java`
```java
@Data
@ToString
public abstract class BaseDataDownloader {
private String startTimeStr; // 起始时间北京时UTC+8
private LocalDateTime startTime; // 起始时间对象
private String refTimeStr; // 参考时间字符串
private OffsetDateTime refTime; // 数据实际起报UTC时间
// 获取所有下载文件信息
public abstract List<DownLoadFileInfo> getFilesInfo();
// 单个文件下载
public abstract DownLoadFileInfo download(DownLoadFileInfo downLoadFileInfo) throws IOException;
// 批量下载所有文件
public abstract List<DownLoadFileInfo> downloadAll(List<DownLoadFileInfo> downLoadFileInfoList);
}
```
#### 2.2.2 下载文件信息DownLoadFileInfo
文件:`DownLoadFileInfo.java`
```java
@Data
@ToString
public class DownLoadFileInfo {
private String fileName; // 保存文件名
private String refTimeStr; // 数据起报时间
private Integer forecastHour; // 预报时效
private String forecastBJTimeStr; // 预报时效对应的北京时间
private String forecastUTCTimeStr; // 预报时效对应的UTC时间
private String url; // 下载链接
private String savePath; // 保存路径
private boolean isDownloadSuccess; // 下载是否成功
private String errorMsg; // 错误信息
}
```
#### 2.2.3 GFS数据下载
目录:`download/gfs/`
该模块实现了GFS全球预报系统气象数据的下载功能
1. 支持配置下载起始时间
2. 提供单文件和批量下载能力
3. 包含下载状态跟踪和错误处理
4. 支持UTC和北京时间的转换
### 2.3 网格数据处理模块
目录:`griddata/`
#### 2.3.1 数据表配置TableConfig
文件:`griddata/common/TableConfig.java`
```java
@Data @Component @Slf4j
public class TableConfig {
private String dataTableName; // 数据表名
private String metaTableName; // 元数据表名
private String dataIndexName; // 数据索引名
private String metaIndexName; // 元数据索引名
private String dataDir; // 数据目录
private List<String> variableList; // 变量列表
private int lonSize; // 经度网格数
private int latSize; // 纬度网格数
private int levSize; // 层次数
private int timeToLive; // 数据生存时间
// 网格参数
private double[] lonList; // 经度列表
private double[] latList; // 纬度列表
private int[] pressureList; // 气压层次
private int[] heightList; // 高度层次
}
```
#### 2.3.2 数据导入GfsDataImport
文件:`griddata/operation/GfsDataImport.java`
主要功能:
1. 网格数据导入到阿里云表格存储
2. 支持多维度数据处理:
- 时间维度
- 空间维度(经纬度)
- 垂直维度(气压层/高度层)
3. 数据处理流程:
- 初始化元数据
- 读取NC文件
- 数据单位转换
- 网格数据写入
- 更新元数据状态
关键特性:
1. 支持并发导入
2. 错误处理和状态跟踪
3. 支持断点续传
4. 数据完整性验证
#### 2.3.3 数据操作类
目录:`griddata/operation/`
1. BaseTableOperation表操作基类
2. CreateTable创建数据表
3. GfsDataFetcher数据获取
4. QueryMeta元数据查询
5. UpdateTable表更新
6. DataDeleter数据删除
### 2.4 定时任务模块
目录:`schedule/`
#### 2.4.1 网格数据处理器GridDataProcessor
文件:`schedule/GridDataProcessor.java`
```java
@Component
@Slf4j
@DependsOn({"gfsDataImport", "gfsDownloader"})
public class GridDataProcessor {
// 定时任务每日北京时0,8,12,18点30分执行
@Scheduled(cron = "0 30 0,8,12,18 * * ?")
public void dailyDataProcess() {
if (download()) { // 下载数据
importToTableStore(); // 导入数据库
}
clearExpiredData(); // 清理过期数据
}
}
```
主要功能:
1. 数据下载管理
- 获取文件信息
- 执行下载
- 处理下载失败
2. 数据导入管理
- 导入表格存储
- 处理导入失败
3. 数据清理
- 清理3天前的过期数据
- 日志记录
#### 2.4.2 错误报告服务ErrorReportService
文件:`schedule/ErrorReportService.java`
```java
@Service
@Slf4j
public class ErrorReportService {
// 邮件发送,支持重试
@Retryable(maxAttempts = 4, backoff = @Backoff(delay = 2000, multiplier = 2))
public void sendSimpleEmail(final Mail mail) {
// 发送邮件逻辑
}
// 远程报告,支持重试
@Retryable(maxAttempts = 4, backoff = @Backoff(delay = 2000, multiplier = 2))
public void reportToRemote(final ErrorReport report) {
// 发送远程报告逻辑
}
}
```
特性:
1. 支持邮件通知
2. 支持远程报告
3. 失败重试机制
4. 多种通知方式
### 2.5 Web接口模块
目录:`web/`
#### 2.5.1 地面天气控制器SurfaceWeatherController
文件:`web/controller/SurfaceWeatherController.java`
```java
@RestController
@RequestMapping("/htfp/weather/v1/surface/")
public class SurfaceWeatherController {
// 地面实时气象信息查询
@PostMapping("/querySurfaceNowWeather")
public Result queryNowWeather(@Validated @RequestBody Position2D position2D)
// 地面24小时预报结果查询
@PostMapping("/querySurfaceForecast")
public Result querySurfaceForecast(@Validated @RequestBody Position2D position2D)
// 地面气象预警信息查询
@PostMapping("/queryWeatherWarning")
public Result queryWeatherWarning(@Validated @RequestBody Position2D position2D)
// 全国地面气象预警信息查询
@PostMapping("/queryAllWeatherWarning")
public Result queryWeatherWarning()
}
```
主要功能:
1. 地面实时天气查询
2. 24小时天气预报
3. 气象预警信息
4. 支持多数据源切换(和风天气、彩云天气等)
#### 2.5.2 高空天气控制器UpperWeatherController
文件:`web/controller/UpperWeatherController.java`
```java
@RestController
@RequestMapping("/htfp/weather/v1/upper/")
public class UpperWeatherController {
// 高空实时气象信息查询
@PostMapping("/queryUpperNowWeather")
public Result queryUpperNowWeather(@Validated @RequestBody Position3D position3D)
// 高空航点实时气象信息查询
@PostMapping("/queryUpperNowWeatherInMultiPoints")
public Result queryUpperNowWeatherInMultiPoints(@Validated @RequestBody MultiPointsRequest request)
// 高空24小时预报结果查询
@PostMapping("/queryUpperForecast")
public Result queryUpperForecast(@Validated @RequestBody Position3D position3D)
// 高度廓线查询
@RequestMapping("/queryProfileByPressure")
public Result queryProfileByAndPressure(@Validated @RequestBody ProfileRequest request)
// 平面网格数据查询
@PostMapping("/queryPlaneGrid")
public Result queryPlaneGrid(@Validated @RequestBody PlaneRequest request)
}
```
主要功能:
1. 高空实时天气查询
- 单点查询
- 多点查询(航线)
2. 高空天气预报
- 24小时预报
- 多点预报
3. 高度廓线分析
- 气压坐标
- 近地面高度坐标
4. 平面网格数据查询
#### 2.5.3 接口特性
1. RESTful API设计
2. 跨域支持(@CrossOrigin
3. 参数验证(@Validated
4. 统一响应格式Result
5. 接口日志记录(@ControllerLog
6. 异常处理ControllerExceptionAdvice
#### 2.5.4 数据模型
1. 请求参数:
- Position2D二维位置经纬度
- Position3D三维位置经纬度+高度)
- MultiPointsRequest多点请求
- ProfileRequest廓线请求
- PlaneRequest平面网格请求
2. 响应数据:
- NowWeatherStatus实时天气状态
- TimeSeriesDataset时间序列数据
- SurfaceWeatherWarning天气预警信息
- ProfileResponse廓线响应
- PlaneResponse平面网格响应
### 2.6 工具类模块
目录:`utils/`
#### 2.6.1 HTTP客户端工具HttpClientUtils
文件:`utils/HttpClientUtils.java`
提供HTTP请求相关功能
1. GET请求
2. POST请求
3. SSL证书处理
4. 请求头管理
5. 响应处理
#### 2.6.2 JSON工具JSONUtils
文件:`utils/JSONUtils.java`
JSON数据处理工具
1. 对象序列化
2. JSON解析
3. 文件读写
4. 类型转换
#### 2.6.3 气象工具MeteoUtils
文件:`utils/MeteoUtils.java`
气象数据处理工具:
1. 气压高度转换
2. 温度单位转换
3. 风向风速计算
4. 气象要素计算
#### 2.6.4 时间工具DateTimeUtils
文件:`utils/DateTimeUtils.java`
时间处理工具:
1. 时区转换
2. 格式化处理
3. 时间计算
4. UTC时间处理
#### 2.6.5 数组工具NdArrayUtils
文件:`utils/NdArrayUtils.java`
多维数组处理工具:
1. 数组维度转换
2. 数据提取
3. 数组操作
#### 2.6.6 Spring工具SpringUtil
文件:`utils/SpringUtil.java`
Spring相关工具
1. Bean管理
2. 上下文处理
3. 配置加载
## 3. 系统功能概述
### 3.1 核心功能
1. GFS气象数据处理
- 数据下载
- 数据解析
- 数据存储
- 数据查询
2. 天气信息服务
- 地面天气查询
- 高空天气查询
- 天气预警
- 天气预报
3. 数据分析能力
- 高度廓线分析
- 多点数据分析
- 网格数据分析
- 时序数据分析
### 3.2 技术特点
1. 数据处理
- 支持多维度数据
- 高效数据存储
- 实时数据更新
- 数据质量控制
2. 系统架构
- 模块化设计
- 可扩展性好
- 高可用性
- 易维护性
3. 接口设计
- RESTful规范
- 参数验证
- 异常处理
- 日志记录
4. 运维特性
- 监控告警
- 错误追踪
- 性能优化
- 安全保障
## 4. 后续开发建议
### 4.1 功能增强
1. 数据源扩展
- 增加更多气象数据源
- 支持数据源动态切换
- 实现数据源备份
2. 分析能力提升
- 增加更多气象要素分析
- 提供数据可视化能力
- 支持自定义分析模型
3. 接口优化
- 增加批量处理接口
- 优化接口响应时间
- 提供WebSocket接口
### 4.2 性能优化
1. 数据处理
- 优化数据存储结构
- 实现数据缓存
- 提高并发处理能力
2. 系统架构
- 引入微服务架构
- 实现服务集群
- 优化负载均衡
3. 运维能力
- 完善监控系统
- 优化告警机制
- 提升系统可用性
### 4.3 安全加强
1. 访问控制
- 实现用户认证
- 细化权限管理
- 加强接口安全
2. 数据安全
- 加密敏感数据
- 实现数据备份
- 审计日志完善
### 4.4 文档完善
1. 接口文档
- 详细API说明
- 示例代码
- 错误码说明
2. 运维文档
- 部署说明
- 配置说明
- 故障处理
3. 开发文档
- 架构说明
- 开发规范
- 测试用例

@ -0,0 +1,249 @@
# 项目整体规划和逻辑结构报告
## 1. 项目概述
### 1.1 项目定位
- 名称Weather Service
- 类型:气象数据服务系统
- 目标提供GFS全球预报系统数据的获取、存储和查询服务
### 1.2 核心功能
1. GFS数据获取
2. 数据存储管理
3. 数据查询服务
4. 数据分析处理
5. 系统监控告警
## 2. 系统架构
### 2.1 技术架构
```mermaid
graph TD
A[客户端] --> B[API网关]
B --> C[Weather Service]
C --> D[数据处理服务]
C --> E[存储服务]
D --> F[GFS数据源]
E --> G[阿里云表格存储]
```
### 2.2 模块划分
1. 数据采集模块
- GFS数据下载
- 数据格式转换
- 数据质量控制
2. 数据存储模块
- 元数据管理
- 数据索引
- 数据压缩
- 数据备份
3. 查询服务模块
- REST API接口
- 数据过滤
- 结果格式化
4. 监控模块
- 系统监控
- 数据监控
- 告警管理
## 3. 数据流设计
### 3.1 数据流程
```mermaid
sequenceDiagram
participant GFS
participant Service
participant Storage
participant Client
GFS->>Service: 气象数据
Service->>Service: 数据处理
Service->>Storage: 存储数据
Client->>Service: 查询请求
Service->>Storage: 检索数据
Storage->>Service: 返回数据
Service->>Client: 响应结果
```
### 3.2 数据模型
1. 气象数据模型
- 时间维度
- 空间维度
- 气象要素
- 预报时效
2. 元数据模型
- 数据源信息
- 数据质量
- 数据统计
- 更新记录
## 4. 接口设计
### 4.1 外部接口
1. 数据获取接口
```
GET /api/v1/weather/data
参数:
- time: 时间
- location: 位置
- variables: 气象要素
```
2. 数据查询接口
```
GET /api/v1/weather/query
参数:
- startTime: 开始时间
- endTime: 结束时间
- region: 区域范围
- elements: 要素列表
```
### 4.2 内部接口
1. 数据处理接口
2. 存储接口
3. 监控接口
## 5. 存储设计
### 5.1 表结构
1. 数据表(gfs_data_table)
- 主键设计
- 分区策略
- 索引设计
2. 元数据表(gfs_meta_table)
- 配置信息
- 统计信息
- 状态信息
### 5.2 存储优化
- 数据压缩
- 冷热数据分离
- 索引优化
- 缓存策略
## 6. 部署架构
### 6.1 环境规划
1. 开发环境
2. 测试环境
3. 预发布环境
4. 生产环境
### 6.2 部署模式
```mermaid
graph TD
A[负载均衡] --> B[服务集群]
B --> C[主存储节点]
B --> D[从存储节点]
E[监控系统] --> B
E --> C
E --> D
```
## 7. 安全设计
### 7.1 访问控制
- 身份认证
- 权限管理
- 接口鉴权
- 数据加密
### 7.2 数据安全
- 传输加密
- 存储加密
- 备份策略
- 审计日志
## 8. 监控告警
### 8.1 监控指标
1. 系统指标
- CPU使用率
- 内存使用率
- 磁盘使用率
- 网络流量
2. 业务指标
- 数据更新延迟
- 查询响应时间
- 数据完整性
- 服务可用性
### 8.2 告警策略
- 告警级别
- 告警渠道
- 告警规则
- 告警升级
## 9. 项目规划
### 9.1 开发计划
1. 第一阶段:基础架构
- 框架搭建
- 核心功能实现
- 基本测试
2. 第二阶段:功能完善
- 接口开发
- 数据处理
- 存储优化
3. 第三阶段:性能优化
- 性能测试
- 负载均衡
- 监控告警
### 9.2 运维计划
1. 部署策略
2. 备份策略
3. 监控策略
4. 应急预案
## 10. 风险控制
### 10.1 技术风险
- 数据源稳定性
- 存储容量规划
- 性能瓶颈
- 系统扩展性
### 10.2 业务风险
- 数据准确性
- 服务可用性
- 用户体验
- 运维成本
## 11. 评估指标
### 11.1 性能指标
- 响应时间:<500ms
- 并发能力:>1000QPS
- 数据延迟:<5min
- 可用性:>99.9%
### 11.2 质量指标
- 代码覆盖率:>80%
- 接口可用性:>99.9%
- 数据准确率:>99.99%
- 故障恢复时间:<30min
## 12. 总结与建议
### 12.1 项目优势
1. 架构清晰,模块化设计
2. 性能优化,可扩展性好
3. 安全可靠,运维友好
4. 监控完善,风险可控
### 12.2 改进建议
1. 持续优化数据处理流程
2. 加强系统安全性
3. 完善监控告警机制
4. 优化用户体验
5. 提高系统可用性

@ -0,0 +1,164 @@
# 项目文件分析报告
## 1. 项目结构概览
```
weather-service/
├── src/ # 源代码目录
│ ├── main/ # 主要源代码
│ └── test/ # 测试代码
├── target/ # 编译输出目录
├── docs/ # 项目文档
├── GFSData/ # GFS数据存储目录
├── logError/ # 错误日志目录
├── logInfo/ # 信息日志目录
├── .idea/ # IntelliJ IDEA配置目录
├── .vscode/ # VSCode配置目录
└── 配置文件
├── pom.xml # Maven项目配置文件
├── .cursorrules # 项目规范配置
├── gfsDataConfig.json # GFS数据配置
├── tableConf.json # 表格配置
└── tablestoreConf.json # 表格存储配置
```
## 2. 核心配置文件分析
### 2.1 pom.xml
- 类型Maven配置文件
- 作用:管理项目依赖和构建配置
- 主要配置:
- SpringBoot 2.1.4.RELEASE
- JDK 1.8
- Lombok
- 其他项目依赖
### 2.2 gfsDataConfig.json
- 类型GFS数据配置文件
- 作用配置GFS气象数据的获取参数
- 主要配置:
```json
{
"duration": 30,
"minLon": 70.0,
"maxLon": 140.0,
"minLat": 0.0,
"maxLat": 55.0,
"resolution": 0.25,
"variables": ["DZDT", "RH", "TMP", "UGRD", "VGRD", "TCDC", "PRATE"],
"pressureLevels": [400-1000],
"heightLevels": [2-100],
"saveRoot": "./GFSData"
}
```
- 配置说明:
- 经纬度范围:覆盖中国及周边区域
- 分辨率0.25度
- 气象变量:包括温度、湿度、风速等
- 气压层次15个层次
- 高度层次8个层次
### 2.3 tableConf.json
- 类型:数据表配置文件
- 作用:配置数据存储表结构
- 主要配置:
```json
{
"dataTableName": "gfs_data_table",
"metaTableName": "gfs_meta_table",
"dataIndexName": "gfs_data_index",
"metaIndexName": "gfs_meta_table_index",
"dataDir": "./GFSData/",
"variableList": ["temp", "cloud", "windSpeed", "wind360", "humidity", "precip"],
"lonSize": 281,
"latSize": 221,
"levSize": 15,
"timeToLive": 1
}
```
- 配置说明:
- 数据表结构定义
- 索引配置
- 变量列表
- 网格尺寸
- 数据生命周期
### 2.4 tablestoreConf.json
- 类型:阿里云表格存储配置文件
- 作用:配置表格存储服务连接参数
- 主要配置:
- endpoint服务端点
- accessId访问ID
- accessKey访问密钥
- instanceName实例名称
- 安全说明:建议使用环境变量管理敏感信息
## 3. 目录结构分析
### 3.1 src/
- main/:主要源代码目录
- 遵循Maven标准目录结构
- 包含Java源代码和资源文件
- test/:测试代码目录
- 单元测试
- 集成测试
### 3.2 数据目录
- GFSData/
- 存储GFS气象数据
- 按配置文件中的结构组织
### 3.3 日志目录
- logError/:错误日志存储
- logInfo/:信息日志存储
- 建议:
- 实现日志轮转
- 设置日志级别
- 定期清理
### 3.4 IDE配置目录
- .idea/IntelliJ IDEA配置
- .vscode/VSCode配置
- 建议:
- 统一团队IDE配置
- 添加必要的.gitignore规则
## 4. 文件依赖关系
```mermaid
graph TD
A[pom.xml] --> B[项目依赖管理]
C[gfsDataConfig.json] --> D[GFS数据获取]
D --> E[GFSData目录]
F[tableConf.json] --> G[数据表结构]
H[tablestoreConf.json] --> I[阿里云存储服务]
G --> I
D --> G
```
## 5. 建议与改进
1. 配置文件管理
- 将敏感信息移至环境变量
- 区分开发和生产环境配置
- 添加配置文件注释说明
2. 日志管理
- 实现日志分级存储
- 添加日志轮转策略
- 规范日志格式
3. 数据存储
- 优化数据存储结构
- 实现数据备份策略
- 添加数据验证机制
4. 代码组织
- 完善单元测试
- 添加API文档
- 规范代码注释
5. 安全性
- 加密敏感配置
- 实现访问控制
- 添加数据校验

@ -0,0 +1,132 @@
# 项目规范分析报告
## 1. 技术栈规范
### 1.1 基础框架
- 开发语言JDK 1.8
- 框架选型SpringBoot 2.1.4.RELEASE
- 项目管理Maven
- 代码简化Lombok
### 1.2 前端技术规范
- 开发语言TypeScript
- 代码规范ESLint + Prettier
- UI框架支持Material Design、Ant Design等
- 状态管理支持Redux、MobX、Context API
## 2. 开发规范
### 2.1 命名规范
- 组件命名PascalCase
- 函数命名camelCase
- 常量命名UPPER_CASE
- 接口命名I前缀如IUserService
- 类型定义T前缀如TUserData
### 2.2 文件组织
- 遵循Maven标准项目结构
- 文档统一存放在docs目录
- 使用Markdown格式编写文档
### 2.3 代码质量控制
- 函数长度≤50行
- 文件长度≤300行
- 嵌套深度≤3层
- 圈复杂度≤10
- 单一职责原则:每个函数只做一件事
## 3. 工程化规范
### 3.1 版本控制
- 语义化版本管理
- 语义化提交信息feat:, fix:, docs:等)
- 主分支代码必须经过测试
- 使用锁定注释标记稳定功能
### 3.2 依赖管理
- 优先使用国内镜像源
- Maven依赖阿里云镜像
- npm包淘宝镜像
- Docker镜像阿里云
- 记录依赖版本和来源
### 3.3 测试规范
- 单元测试覆盖关键功能
- 端到端测试验证用户流程
- 采用TDD开发方法
- 测试数据与生产环境隔离
## 4. 安全规范
### 4.1 数据安全
- 敏感数据传输加密
- 用户输入验证和清洗
- 实现数据备份策略
- 遵循数据最小化原则
- 数据生命周期管理
### 4.2 代码安全
- API向下兼容
- 环境配置分离
- 避免硬编码敏感信息
- 定期安全审计
## 5. 性能规范
### 5.1 前端性能
- 资源懒加载
- 优化首屏加载
- 合理使用缓存
- 优化DOM操作
- 响应式设计
### 5.2 后端性能
- 合理使用连接池
- 优化数据库查询
- 实现请求限流
- 异步处理长任务
- 定期性能监控
## 6. 文档规范
### 6.1 必要文档
- API接口文档
- 数据模型文档
- 架构设计文档
- 部署文档
- 使用手册
### 6.2 文档要求
- 及时更新
- 版本对应
- 结构清晰
- 示例完整
- 便于维护
## 7. 协作规范
### 7.1 开发流程
1. 问题分析
2. 方案设计
3. 功能实现
4. 测试验证
5. 代码审查
6. 部署上线
### 7.2 团队协作
- 统一开发环境
- 代码审查制度
- 知识共享机制
- 技术债务管理
- 定期代码重构
## 8. 总结
该项目规范体系完整,覆盖了从开发到部署的各个环节,有助于:
1. 提高代码质量
2. 规范开发流程
3. 保障系统安全
4. 优化性能表现
5. 促进团队协作
建议在实际开发中严格执行这些规范,并根据项目实际情况进行适当调整和优化。

@ -2,11 +2,20 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.htfp.weather</groupId>
<groupId>com.htfp</groupId>
<artifactId>weather-service</artifactId>
<version>0.0.1-SNAPSHOT</version>
<version>0.0.3</version>
<name>weather-service</name>
<description>weather-service</description>
<packaging>jar</packaging>
<!--声明父模块-->
<parent>
<groupId>com.htfp</groupId>
<artifactId>weather</artifactId>
<version>0.0.1</version>
<relativePath>../pom.xml</relativePath>
</parent>
<properties>
<java.version>1.8</java.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
@ -88,11 +97,10 @@
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-thymeleaf</artifactId>
</dependency>
<!--? <dependency>-->
<!--? <groupId>org.springframework.boot</groupId>-->
<!--? <artifactId>spring-boot-starter-thymeleaf</artifactId>-->
<!--? </dependency>-->
<dependency>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>okhttp</artifactId>
@ -128,48 +136,42 @@
</dependency>
<!-- springmvc的参数valid校验依赖 结束 -->
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>${spring-boot.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>${spring-boot.version}</version>
<configuration>
<executable>true</executable>
<layout>JAR</layout>
<!-- 指定该Main Class为全局的唯一入口 -->
<mainClass>com.htfp.weather.WeatherServiceApplication</mainClass>
<layout>ZIP</layout>
<!--打包排除所有jar包-->
<includes>
<include>
<groupId>nothing</groupId>
<artifactId>nothing</artifactId>
</include>
</includes>
<!--打包排除所有jar包-->
</configuration>
<executions>
<execution>
<id>repackage</id>
<goals>
<!--可以把依赖的包都打包到生成的Jar包中-->
<goal>repackage</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
</build>
</project>

@ -25,22 +25,22 @@ public abstract class BaseDataDownloader {
/**
*
*/
public abstract List<FileInfo> getFilesInfo();
public abstract List<DownLoadFileInfo> getFilesInfo();
/**
*
* @param fileInfo:
* @param downLoadFileInfo:
* @return /
* @throws IOException
*/
public abstract FileInfo download(FileInfo fileInfo) throws IOException;
public abstract DownLoadFileInfo download(DownLoadFileInfo downLoadFileInfo) throws IOException;
/**
*
* @param fileInfoList
* @param downLoadFileInfoList
* @return
*/
public abstract List<FileInfo> downloadAll(List<FileInfo> fileInfoList);
public abstract List<DownLoadFileInfo> downloadAll(List<DownLoadFileInfo> downLoadFileInfoList);
}

@ -9,7 +9,7 @@ import lombok.ToString;
@Data
@ToString
public class FileInfo {
public class DownLoadFileInfo {
/**保存文件名*/
private String fileName;
/**数据起报时间*/
@ -26,7 +26,8 @@ public class FileInfo {
private String savePath;
private boolean isDownloadSuccess;
public FileInfo() {
private String errorMsg;
public DownLoadFileInfo() {
}
}

@ -108,7 +108,7 @@ public class GfsDataConfig {
}
public void readConfig() {
try (InputStream f = new FileInputStream(configPath)){
GfsDataConfig gfsDataConfig = JSONUtils.json2pojo(IOUtils.toString(f), GfsDataConfig.class);
GfsDataConfig gfsDataConfig = JSONUtils.json2obj(IOUtils.toString(f), GfsDataConfig.class);
this.duration = gfsDataConfig.getDuration();
this.minLon = gfsDataConfig.getMinLon();
this.maxLon = gfsDataConfig.getMaxLon();

@ -1,7 +1,7 @@
package com.htfp.weather.download.gfs;
import com.htfp.weather.download.BaseDataDownloader;
import com.htfp.weather.download.FileInfo;
import com.htfp.weather.download.DownLoadFileInfo;
import com.htfp.weather.info.Constant;
import com.htfp.weather.utils.DateTimeUtils;
import com.htfp.weather.utils.HttpClientUtils;
@ -26,7 +26,7 @@ import java.util.concurrent.Future;
/**
* @author shiyi
* @
* @Description GFS,
*/
@Slf4j
@Component("gfsDownloader")
@ -62,33 +62,39 @@ public class GfsDownloader extends BaseDataDownloader {
}
@Override
public FileInfo download(FileInfo fileInfo) {
return download0(fileInfo, 2);
public DownLoadFileInfo download(DownLoadFileInfo downLoadFileInfo) {
return download0(downLoadFileInfo, 2);
}
private FileInfo download0(FileInfo fileInfo, int retryNum) {
String url = fileInfo.getUrl();
File destDir = new File(fileInfo.getSavePath());
File fileOut = new File(destDir, fileInfo.getFileName());
private DownLoadFileInfo download0(DownLoadFileInfo downLoadFileInfo, int retryNum) {
String url = downLoadFileInfo.getUrl();
File destDir = new File(downLoadFileInfo.getSavePath());
File fileOut = new File(destDir, downLoadFileInfo.getFileName());
log.info("[GFS Download] 文件下载中,保存至 {}", fileOut);
try {
// DONE: 改用okhttp FIXME 2024/6/8: 如果连接是httpsjar包启动下载会报错 javax.net.ssl.SSLException: Received fatal alert: internal_error
HttpClientUtils.downloadFileByUrl(url, fileOut.getPath());
log.info("[GFS Download] 文件下载成功: {}", fileOut);
fileInfo.setDownloadSuccess(fileValid(fileOut.getAbsolutePath()));
boolean fileIsValid = fileValid(fileOut.getAbsolutePath());
downLoadFileInfo.setDownloadSuccess(fileIsValid);
if (fileIsValid) {
log.info("[GFS Download] 文件下载成功: {}", fileOut);
} else {
log.error("[GFS Download] 文件下载失败,文件打开错误: {}", fileOut);
}
} catch (Exception e) {
// DONE 2024/5/24: 文件服务器在外网不稳定,增加重试机制
fileInfo.setDownloadSuccess(false);
downLoadFileInfo.setDownloadSuccess(false);
if (retryNum > 0) {
log.error("[GFS Download] 文件下载失败,重试中: {}", fileOut);
return download0(fileInfo, retryNum - 1);
return download0(downLoadFileInfo, retryNum - 1);
} else {
e.printStackTrace();
log.error("[GFS Download] 文件下载失败: {}", fileOut);
downLoadFileInfo.setErrorMsg(e.getMessage());
log.error("[GFS Download] 文件{}下载失败: {}", fileOut, e.getMessage());
}
}
return fileInfo;
return downLoadFileInfo;
}
/** 简单校验验证文件完整性,并生成索引文件*/
@ -97,18 +103,19 @@ public class GfsDownloader extends BaseDataDownloader {
ncFile.getLocation();
return true;
} catch (IOException e){
log.error("GFS文件{}校验失败: {}", file, e.getMessage());
return false;
}
}
@Override
public List<FileInfo> downloadAll(List<FileInfo> fileInfoList) {
log.info("[GFS Download] 下载任务启动,共 {} 个文件", fileInfoList.size());
List<Future<FileInfo>> futures = new ArrayList<>();
List<FileInfo> finishList = new ArrayList<>();
for (FileInfo fileInfo : fileInfoList) {
futures.add(executorService.submit(() -> download(fileInfo)));
public List<DownLoadFileInfo> downloadAll(List<DownLoadFileInfo> downLoadFileInfoList) {
log.info("[GFS Download] 下载任务启动,共 {} 个文件", downLoadFileInfoList.size());
List<Future<DownLoadFileInfo>> futures = new ArrayList<>();
List<DownLoadFileInfo> finishList = new ArrayList<>();
for (DownLoadFileInfo downLoadFileInfo : downLoadFileInfoList) {
futures.add(executorService.submit(() -> download(downLoadFileInfo)));
}
for (Future<FileInfo> future : futures) {
for (Future<DownLoadFileInfo> future : futures) {
try {
finishList.add(future.get());
} catch (InterruptedException | ExecutionException e) {
@ -120,7 +127,7 @@ public class GfsDownloader extends BaseDataDownloader {
}
@Override
public List<FileInfo> getFilesInfo() {
public List<DownLoadFileInfo> getFilesInfo() {
// 时间选项填充
String lonlatBoxStr = null;
String levelsStr = null;
@ -145,28 +152,28 @@ public class GfsDownloader extends BaseDataDownloader {
// 分辨率为3h整数截断包括目标时刻本身
final int nFiles = 1 + gfsDataConfig.getDuration() / forecastStep;
// 存储文件信息
List<FileInfo> fileInfoList = new ArrayList<>(nFiles);
List<DownLoadFileInfo> downLoadFileInfoList = new ArrayList<>(nFiles);
for (int i = 0; i < nFiles; i++) {
FileInfo fileInfo = new FileInfo();
DownLoadFileInfo downLoadFileInfo = new DownLoadFileInfo();
int forecastHour = i * forecastStep + hourDiff;
fileInfo.setForecastHour(forecastHour);
downLoadFileInfo.setForecastHour(forecastHour);
String baseURL = getBaseURL(forecastHour, levelsStr, variablesStr, lonlatBoxStr);
fileInfo.setUrl(baseURL);
fileInfo.setRefTimeStr(refTimeStr);
fileInfo.setForecastUTCTimeStr(
this.getRefTime().plusHours(fileInfo.getForecastHour())
downLoadFileInfo.setUrl(baseURL);
downLoadFileInfo.setRefTimeStr(refTimeStr);
downLoadFileInfo.setForecastUTCTimeStr(
this.getRefTime().plusHours(downLoadFileInfo.getForecastHour())
.format(DateTimeFormatter.ofPattern(Constant.UTC_TIME_STRING))
);
fileInfo.setForecastBJTimeStr(
DateTimeUtils.getLocalZoneDateTime(this.getRefTime().plusHours(fileInfo.getForecastHour()))
downLoadFileInfo.setForecastBJTimeStr(
DateTimeUtils.getLocalZoneDateTime(this.getRefTime().plusHours(downLoadFileInfo.getForecastHour()))
.format(DateTimeFormatter.ofPattern(Constant.BJT_TIME_STRING))
);
fileInfo.setFileName(String.format("%s-from-%s+%d.grib2", fileInfo.getForecastBJTimeStr(), fileInfo.getRefTimeStr(), forecastHour));
fileInfo.setSavePath(savePath);
fileInfoList.add(fileInfo);
downLoadFileInfo.setFileName(String.format("%s-from-%s+%d.grib2", downLoadFileInfo.getForecastBJTimeStr(), downLoadFileInfo.getRefTimeStr(), forecastHour));
downLoadFileInfo.setSavePath(savePath);
downLoadFileInfoList.add(downLoadFileInfo);
}
return fileInfoList;
return downLoadFileInfoList;
}

@ -43,8 +43,8 @@ public enum GfsVariableIsobaricEnum {
public static String getGfsVariableName(GfsVariableIsobaricEnum gfsVariableIsobaricEnum) {
return variableName.get(gfsVariableIsobaricEnum.nameInApi);
}
public static String getGfsVariableName(String tableVariableName) {
return variableName.get(tableVariableName);
public static String getGfsVariableName(String nameInApi) {
return variableName.get(nameInApi);
}
public static String getVariableNameInApi(String nameInFile) {

@ -4,6 +4,7 @@ import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.htfp.weather.download.gfs.GfsDataConfig;
import com.htfp.weather.utils.JSONUtils;
import com.htfp.weather.utils.MeteoUtils;
import lombok.Data;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.io.IOUtils;
@ -18,6 +19,7 @@ import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.nio.file.NoSuchFileException;
import java.util.Arrays;
import java.util.List;
@Data @Component @Slf4j
@ -51,6 +53,7 @@ public class TableConfig {
public double[] latList;
@JsonIgnore
public int[] pressureList;
public int[] pressureHeightList;
@JsonIgnore
public int[] heightList;
// @JsonIgnore
@ -63,7 +66,11 @@ public class TableConfig {
// private final String configPath = Objects.requireNonNull(this.getClass().getClassLoader().getResource("config/tableConf.json")).getPath();
private final String configPath = System.getProperty("user.dir") +"/tableConf.json";
@PostConstruct
private void initTableConfig() {
public void init() {
if (dataConfig == null){
dataConfig = new GfsDataConfig();
dataConfig.init();
}
readConfig();
initLonList();
initLatList();
@ -74,7 +81,7 @@ public class TableConfig {
// String pathSeparator = System.getProperty("file.separator");
try (InputStream f = new FileInputStream(configPath)){
String jsonStr = IOUtils.toString(f, StandardCharsets.UTF_8);
TableConfig tableConfig = JSONUtils.json2pojo(jsonStr, TableConfig.class);
TableConfig tableConfig = JSONUtils.json2obj(jsonStr, TableConfig.class);
this.dataTableName = tableConfig.getDataTableName();
this.metaTableName = tableConfig.getMetaTableName();
this.dataIndexName = tableConfig.getDataIndexName();
@ -136,6 +143,8 @@ public class TableConfig {
private void initLevList() {
pressureList = dataConfig.getPressureLevels();
// NOTE 2024/7/4: 气压对应的海拔高度,注意不是离地高度
pressureHeightList = Arrays.stream(pressureList).map(x-> (int) MeteoUtils.pressure2Elevation(x)).toArray();
heightList = dataConfig.getHeightLevels();
// this.levSize = Math.max(pressureList.length, heightList.length);
}

@ -17,11 +17,10 @@ import java.util.stream.Collectors;
/**
* @Author : shiyi
* @Date : 2024/1/15 19:10
* @Description : meta
* @Description : meta
*/
@Slf4j @Component
@Slf4j
public class TableConfigStatic {
@Autowired
GfsDataConfig dataConfig;
/**
* index
@ -48,7 +47,6 @@ public class TableConfigStatic {
// 数据过期时间,单位为秒
public static int TIME_TO_LIVE;
@PostConstruct
private void initTableConfig() {
initDatasetConfig();
initLonList();

@ -1,9 +1,15 @@
package com.htfp.weather.griddata.common;
import com.htfp.weather.utils.JSONUtils;
import lombok.Data;
import org.apache.commons.io.IOUtils;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;
import java.io.FileInputStream;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
@Data
@Configuration("tableStoreConf")
@ConfigurationProperties("tablestore")
@ -12,4 +18,16 @@ public class TableStoreConf {
private String accessId;
private String accessKey;
private String instanceName;
public static TableStoreConf init() {
try {
String path = System.getProperty("user.dir") + "./tablestoreConf.json";
InputStream f = new FileInputStream(path);
String jsonStr = IOUtils.toString(f, StandardCharsets.UTF_8);
return JSONUtils.json2obj(jsonStr, TableStoreConf.class);
} catch (Exception ex) {
throw new RuntimeException(ex);
}
}
}

@ -19,22 +19,43 @@ public abstract class BaseTableOperation{
private TableConfig tableConfig;
@Autowired
private TableStoreConf tableStoreConf;
TableStoreGridConfig config;
@PostConstruct
public void init() {
TableStoreGridConfig config = new TableStoreGridConfig();
config = new TableStoreGridConfig();
if (tableStoreConf == null){
tableStoreConf = TableStoreConf.init();
}
if (tableConfig == null) {
tableConfig = new TableConfig();
tableConfig.init();
}
config.setTableStoreEndpoint(tableStoreConf.getEndpoint());
config.setAccessId(tableStoreConf.getAccessId());
config.setAccessKey(tableStoreConf.getAccessKey());
config.setTableStoreInstance(tableStoreConf.getInstanceName());
config.setDataTableName(tableConfig.getDataTableName());
config.setMetaTableName(tableConfig.getMetaTableName());
tableStoreGrid = new TableStoreGrid(config);
connect();
}
/**
*
*/
public void connect() {
if (tableStoreGrid == null) {
tableStoreGrid = new TableStoreGrid(config);
}
}
/**
*
*/
public void close() {
if (tableStoreGrid != null) {
tableStoreGrid.close();
tableStoreGrid = null;
}
}
}

@ -4,10 +4,13 @@ import com.alicloud.openservices.tablestore.model.TableOptions;
import com.alicloud.openservices.tablestore.model.search.FieldSchema;
import com.alicloud.openservices.tablestore.model.search.FieldType;
import com.alicloud.openservices.tablestore.model.search.IndexSchema;
import com.aliyun.tablestore.grid.consts.AttributionEnum;
import com.htfp.weather.griddata.common.TableConfigStatic;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
/**
* @Author : shiyi
@ -33,19 +36,24 @@ public class CreateTable extends BaseTableOperation {
*/
private void createIndex() throws Exception {
IndexSchema indexSchema = new IndexSchema();
indexSchema.setFieldSchemas(Arrays.asList(
new FieldSchema("status", FieldType.KEYWORD).setIndex(true).setEnableSortAndAgg(true),
new FieldSchema("create_time", FieldType.LONG).setIndex(true).setEnableSortAndAgg(true),
new FieldSchema("ref_time", FieldType.LONG).setIndex(true).setEnableSortAndAgg(true)
));
AttributionEnum[] values = AttributionEnum.values();
ArrayList<FieldSchema> fieldSchemas = new ArrayList<>();
for (AttributionEnum attributionEnum : values) {
fieldSchemas.add(new FieldSchema(attributionEnum.getName(), attributionEnum.getType()).setIndex(true).setEnableSortAndAgg(true));
}
indexSchema.setFieldSchemas(fieldSchemas);
this.tableStoreGrid.createMetaIndex(TableConfigStatic.META_INDEX_NAME, indexSchema);
}
public static void main(String[] args) throws Exception {
TableConfigStatic.initDatasetConfig();
CreateTable example = new CreateTable();
example.init();
try {
example.createStore();
example.createIndex();
} catch (Exception e) {
e.printStackTrace();
} finally {
example.close();
}

@ -20,6 +20,7 @@ import org.springframework.util.CollectionUtils;
import ucar.ma2.Array;
import javax.annotation.PostConstruct;
import javax.annotation.PreDestroy;
import javax.annotation.Resource;
import java.time.Duration;
import java.time.Instant;
@ -33,7 +34,7 @@ import java.util.stream.Collectors;
/**
* @Author : shiyi
* @Date : 2024/1/22 13:53
* @Description :
* @Description : ,
*/
@Slf4j @Component
@DependsOn({"tableStoreConf", "tableConfig"})
@ -43,11 +44,14 @@ public class GfsDataFetcher extends BaseTableOperation {
@Resource
QueryMeta queryMeta;
GridDataSetMeta lastGridDataSetMeta;
@PreDestroy
public void destroy() {
close();
}
public GridDataSetMeta getLastGridDataSetMeta() {
if (lastGridDataSetMeta == null) {
updateLastGridDataSetMeta();
throw new AppException(ErrorCode.DATA_SET_EMPTY, ": e.getMessage()");
throw new AppException(ErrorCode.DATA_SET_EMPTY);
}
return lastGridDataSetMeta;
}
@ -132,14 +136,15 @@ public class GfsDataFetcher extends BaseTableOperation {
}
/**
*
* @param variable
* @param iLat
* @param iLon
* @return {@link Array}
*
* @param variableNameList
* @param targetTime
* @param latitude
* @param longitude
* @return {@link GridDataSet}
* @throws Exception
*/
public Array getProfileByVariableAndPressure(String variableName, OffsetDateTime targetTime, double latitude, double longitude) throws Exception {
public GridDataSet getProfileByPressure(List<String> variableNameList, OffsetDateTime targetTime, double latitude, double longitude) throws Exception {
lastGridDataSetMeta = getLastGridDataSetMeta();
GridDataFetcher fetcher = tableStoreGrid.getDataFetcher(lastGridDataSetMeta);
int iTime = getTargetTimeIndex(targetTime);
@ -148,25 +153,27 @@ public class GfsDataFetcher extends BaseTableOperation {
int[] origin = new int[]{iTime, 0, iLat, iLon};
int[] shape = new int[]{1,tableConfig.levSize, 1, 1};
// TODO 2024/6/17:
fetcher.setVariablesToGet(Collections.singletonList(variableName));
fetcher.setVariablesToGet(variableNameList);
fetcher.setOriginShape(origin, shape);
GridDataSet gridDataSet = fetcher.fetch();
Array array = gridDataSet.getVariable(variableName).toArray();
return array;
return fetcher.fetch();
}
public GridDataSet getProfileByPressure(List<String> variableNameList, OffsetDateTime targetTime, double latitude, double longitude) throws Exception {
/**获取变量随近地面高度的分布(目前只针对风速风向)*/
public GridDataSet getProfileByNearSurfaceHeight(List<String> variableNameList, OffsetDateTime targetTime, double latitude, double longitude) throws Exception {
lastGridDataSetMeta = getLastGridDataSetMeta();
GridDataFetcher fetcher = tableStoreGrid.getDataFetcher(lastGridDataSetMeta);
int iTime = getTargetTimeIndex(targetTime);
int iLat= getLatitudeIndex(latitude);
int iLon = getLongitudeIndex(longitude);
int[] origin = new int[]{iTime, 0, iLat, iLon};
int[] shape = new int[]{1,tableConfig.levSize, 1, 1};
//TODO NOTE 2024/7/4: 目前仅有风速支持近地面多个高度, 且只有102030405080100这7个高度2m高度只针对temp和humiditync文件本身高度坐标就有很多种难以统一到数据库中
int[] shape = new int[]{1,tableConfig.heightList.length - 1, 1, 1}; // heightList.length - 1 去掉2m高度
// TODO 2024/6/17:
fetcher.setVariablesToGet(variableNameList);
fetcher.setOriginShape(origin, shape);
return fetcher.fetch();
}
/**
*
* @param dataSetId ID

@ -3,9 +3,9 @@ package com.htfp.weather.griddata.operation;
import com.aliyun.tablestore.grid.GridDataWriter;
import com.aliyun.tablestore.grid.model.GridDataSetMeta;
import com.aliyun.tablestore.grid.model.StoreOptions;
import com.aliyun.tablestore.grid.model.grid.Grid;
import com.aliyun.tablestore.grid.model.grid.Grid2D;
import com.htfp.weather.download.FileInfo;
import com.htfp.weather.download.gfs.GfsVariableHeightEnum;
import com.htfp.weather.download.gfs.GfsVariableIsobaricEnum;
import com.htfp.weather.griddata.common.TableConfig;
@ -28,12 +28,12 @@ import ucar.nc2.Variable;
import javax.annotation.PreDestroy;
import javax.annotation.Resource;
import java.io.File;
import java.nio.ByteBuffer;
import java.time.OffsetDateTime;
import java.time.format.DateTimeFormatter;
import java.util.*;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.stream.Collectors;
/**
@ -50,9 +50,9 @@ public class GfsDataImport extends BaseTableOperation {
@Getter
boolean importing = false;
private final ExecutorService executorService = Executors.newFixedThreadPool(5);
GridDataWriter writer;
/**
* init meta data to table store.
*
*
* @param dataSetID
* @param dataType
@ -81,42 +81,35 @@ public class GfsDataImport extends BaseTableOperation {
}
/**
* update meta and set status to DONE when data import finished.
*
* @param meta
* @return
* @throws Exception
* meta
*/
public GridDataSetMeta updateMeta(GridDataSetMeta meta) throws Exception {
tableStoreGrid.updateDataSetMeta(meta);
return meta;
}
public GridDataSetMeta putMeta(GridDataSetMeta meta) throws Exception {
tableStoreGrid.putDataSetMeta(meta);
return meta;
}
/** 导入指定起报时间的所有文件*/
public List<ImportResult> importData(OffsetDateTime refTime) throws Exception {
public synchronized List<ImportFileInfo> importData(OffsetDateTime refTime) throws Exception {
importing = true;
long start = System.currentTimeMillis();
List<String> fileList = getFiles(refTime);
if (CollectionUtils.isEmpty(fileList)) {
throw new AppException(ErrorCode.NO_NC_OR_GRIB_FILES);
}
log.info("[tablestore] 数据导入开始, refTime = {}...", refTime);
// datasetId和起报时间绑定
String dataSetId = refTime.format(DateTimeFormatter.ofPattern(Constant.DATA_SET_ID_FORMAT_STRING));
List<String> fileVariables = getFileVariables(tableConfig.variableList);
int[] shape = new int[]{tableConfig.timeSizeMax, tableConfig.levSize, tableConfig.latSize, tableConfig.lonSize};
GridDataSetMeta meta = initMeta(dataSetId, tableConfig.dataType, fileVariables, shape);
List<String> forecastHours = new ArrayList<>(); // 所有有效预报时效坐标记录到数据库属性中
connect();
try {
long start = System.currentTimeMillis();
List<String> fileList = getFiles(refTime);
if (CollectionUtils.isEmpty(fileList)) {
throw new AppException(ErrorCode.NO_NC_OR_GRIB_FILES);
}
log.info("[tablestore] 数据导入开始, refTime = {}...", refTime);
// datasetId和起报时间绑定
String dataSetId = refTime.format(DateTimeFormatter.ofPattern(Constant.DATA_SET_ID_FORMAT_STRING));
List<String> fileVariables = getFileVariables(tableConfig.variableList);
int[] shape = new int[]{tableConfig.timeSizeMax, tableConfig.levSize, tableConfig.latSize, tableConfig.lonSize};
GridDataSetMeta meta = initMeta(dataSetId, tableConfig.dataType, fileVariables, shape);
writer = tableStoreGrid.getDataWriter(meta);
// todo 2024/5/13: 待优化,用于数据库的索引,必须连续,因此必须保证前一个时刻成功导入后才能导入下一个时刻,数据量大的时候导入时间较长
List<ImportResult> finishedList;
// List<Future<ImportResult>> futures = new ArrayList<>();
List<String> forecastHours = new ArrayList<>(); // 所有有效预报时效坐标记录到数据库属性中
// todo 2024/5/13: 待优化,用于数据库的索引,必须连续,因此必须保证前一个时刻成功导入后才能导入下一个时刻,数据量大的时候导入时间较长
List<ImportFileInfo> finishedList;
// List<Future<ImportResult>> futures = new ArrayList<>();
finishedList = new ArrayList<>();
for (int i = 0; i < fileList.size(); i++) {
String file = fileList.get(i);
@ -124,37 +117,40 @@ public class GfsDataImport extends BaseTableOperation {
int forecastHour = getForecastHourFromFilename(file);
// 2024/5/13:使用多线程并使用Future来获取结果但是实际瓶颈可能是网络带宽而非计算
// futures.add(executorService.submit(() -> importFromNcFile(meta, file, iTime,forecastHour)));
ImportResult importResult = importFromNcFile(meta, file, iTime, forecastHour);
finishedList.add(importResult);
if (importResult.isSuccess()) {
forecastHours.add(String.valueOf(importResult.getForecastHour()));
ImportFileInfo importFileInfo = importFromNcFile(meta, file, iTime, forecastHour);
finishedList.add(importFileInfo);
if (importFileInfo.isSuccess()) {
forecastHours.add(String.valueOf(importFileInfo.getForecastHour()));
}
}
// for (Future<ImportResult> future : futures) {
// ImportResult importResult = future.get();
// finishedList.add(importResult);
// forecastHours.add(String.valueOf(importResult.getForecastHour()));
// if (importResult.isSuccess()) {
// forecastHours.add(String.valueOf(importResult.getForecastHour()));
// }
// }
// for (Future<ImportResult> future : futures) {
// ImportResult importResult = future.get();
// finishedList.add(importResult);
// forecastHours.add(String.valueOf(importResult.getForecastHour()));
// if (importResult.isSuccess()) {
// forecastHours.add(String.valueOf(importResult.getForecastHour()));
// }
// }
List<ImportResult> failedList = finishedList.stream().filter(result -> !result.isSuccess()).collect(Collectors.toList());
if (!failedList.isEmpty()) {
log.warn("存在气象数据导入数据库失败,失败文件列表: {} ", failedList);
meta.addAttribute("status", "WRONG");
} else {
meta.addAttribute("status", "DONE");
List<ImportFileInfo> failedList = finishedList.stream().filter(result -> !result.isSuccess()).collect(Collectors.toList());
if (!failedList.isEmpty()) {
log.warn("存在气象数据导入数据库失败,失败文件列表: {} ", failedList);
meta.addAttribute("status", "WRONG");
} else {
meta.addAttribute("status", "DONE");
}
long end = System.currentTimeMillis();
log.info("[tablestore] 数据导入完成, 耗时: {} s, forecastHours: {}", (end - start) / 1000., forecastHours);
meta.setForecastHours(forecastHours);
meta.addAttribute("reference_time", refTime.toInstant().toEpochMilli());
putMeta(meta);
gfsDataFetcher.lastGridDataSetMeta = meta;
log.info("[tablestore]: 更新最新的数据元信息: {}", meta);
return finishedList;
} finally {
importing = false;
close();
}
long end = System.currentTimeMillis();
log.info("[tablestore] 数据导入完成, 耗时: {} s, forecastHours: {}", (end - start)/1000., forecastHours);
meta.setForecastHours(forecastHours);
meta.addAttribute("reference_time", refTime.toInstant().toEpochMilli());
putMeta(meta);
gfsDataFetcher.lastGridDataSetMeta = meta;
log.info("[tablestore]: 更新最新的数据元信息: {}", meta);
importing = false;
return finishedList;
}
/**
@ -164,10 +160,8 @@ public class GfsDataImport extends BaseTableOperation {
* @param iTime
* @param forecastHour
*/
public ImportResult importFromNcFile(GridDataSetMeta meta, String file, int iTime, int forecastHour) {
public ImportFileInfo importFromNcFile(GridDataSetMeta meta, String file, int iTime, int forecastHour) {
try (NetcdfFile ncFile = NetcdfFiles.open(file)) {
GridDataWriter writer = tableStoreGrid.getDataWriter(meta);
// 相对reftime的小时数文件可能缺失因此time可能是不连续的但是iTime是连续的
if (forecastHour == -1) {
forecastHour = (int) ncFile.findVariable("time").read().getDouble(0);
@ -176,6 +170,7 @@ public class GfsDataImport extends BaseTableOperation {
int ysize = meta.getySize();
for (String variableName : meta.getVariables()) {
// 风速在最后单独处理
if ("Wind_speed_gust_surface".equals(variableName) || "Wind_speed_isobaric".equals(variableName) || "Wind_direction_isobaric".equals(variableName)
|| "Wind_speed_height_above_ground".equals(variableName) || "Wind_direction_height_above_ground".equals(variableName)) {
continue;
@ -183,40 +178,52 @@ public class GfsDataImport extends BaseTableOperation {
Variable variable = ncFile.findVariable(variableName);
if (variable != null) {
int shapeLength = variable.getShape().length;
if (shapeLength== 4) {
if (shapeLength== 4) { // 存在垂直坐标,即变量随高度变化
// 气压坐标和高度坐标保持各自的索引
int zSize = variable.getShape(1);
for (int z = 0; z < zSize; z++) {
// 注意高度索引是递增排序
Array array = variable.read(new int[]{0, z, 0, 0}, new int[]{1, 1, xsize, ysize});
transferUnit(variableName, array);
Grid2D grid2D = new Grid2D(array.getDataAsByteBuffer(), variable.getDataType(),
ByteBuffer dataBuffer = transferUnit(
variableName, variable.read(new int[]{0, z, 0, 0}, new int[]{1, 1, xsize, ysize})
).getDataAsByteBuffer();
Grid2D grid2D = new Grid2D(dataBuffer, variable.getDataType(),
new int[]{0, 0}, new int[]{xsize, ysize});
writer.writeGrid2D(variableName, iTime, z, grid2D);
dataBuffer=null; grid2D = null;
}
} else if (shapeLength == 3) {
// DONE 2024/5/24: 导入不含高度坐标的数据
Array array = variable.read(new int[]{0, 0, 0}, new int[]{1,xsize, ysize});
transferUnit(variableName, array);
Grid2D grid2D = new Grid2D(array.getDataAsByteBuffer(), variable.getDataType(),
// DONE 2024/5/24: 导入不含高度坐标的数据,如降水量,阵风风速
ByteBuffer dataBuffer = transferUnit(
variableName, variable.read(new int[]{0, 0, 0}, new int[]{1,xsize, ysize}))
.getDataAsByteBuffer();
Grid2D grid2D = new Grid2D(dataBuffer, variable.getDataType(),
new int[]{0, 0}, new int[]{xsize, ysize});
writer.writeGrid2D(variableName, iTime, 0, grid2D);
dataBuffer= null; grid2D = null;
}
} else {
log.warn("[tablestore] 数据文件 {} 中没有变量 {}", ncFile.getLocation(), variableName);
}
}
// 导入风速风向
importWind(writer, meta, ncFile, iTime);
importWind(meta, ncFile, iTime);
} catch (Exception e) {
log.error("[tablestore] 导入文件数据失败,_t={}: {}", forecastHour, file, e);
return new ImportResult(false, file, forecastHour, iTime);
return new ImportFileInfo(false, file, forecastHour, iTime);
} finally {
System.gc();
}
log.info("[tablestore] 导入文件数据成功,_t={}: {}", forecastHour, file);
return new ImportResult(true, file, forecastHour, iTime);
return new ImportFileInfo(true, file, forecastHour, iTime);
}
private void transferUnit(String variableName, Array array) {
/**
*
*
* @return
*/
private Array transferUnit(String variableName, Array array) {
if (GfsVariableIsobaricEnum.TEMP.getNameInFile().equals(variableName)) {
MeteoUtils.kelvin2Celsius(array);
}
@ -226,10 +233,11 @@ public class GfsDataImport extends BaseTableOperation {
if (GfsVariableHeightEnum.PRECIP.getNameInFile().equals(variableName)) {
MeteoUtils.precipRate2mmPerHour(array);
}
return array;
}
/** 计算风速风向,包括地面坐标和气压坐标,并导入数据库*/
private void importWind(GridDataWriter writer, GridDataSetMeta meta, NetcdfFile ncFile, int iTime) throws Exception {
private void importWind( GridDataSetMeta meta, NetcdfFile ncFile, int iTime) throws Exception {
// TODO 2024/5/8: 风速风向最好单独保存为一个文件,如果后续需要二维可视化,可以直接本地读取出图,减少网络请求
for (String suffix : new String[]{"_isobaric", "_height_above_ground"}) {
Variable uwnd = ncFile.findVariable("u-component_of_wind" + suffix);
@ -253,6 +261,9 @@ public class GfsDataImport extends BaseTableOperation {
Grid2D dirGrid2D = new Grid2D(dirArray.getDataAsByteBuffer(), dirArray.getDataType(),
new int[]{0, 0}, new int[]{xsize, ysize});
writer.writeGrid2D("Wind_direction" + suffix, iTime, z, dirGrid2D);
uwndArray = null; vwndArray = null; speedArray = null; dirArray = null;
speedGrid2D = null; dirGrid2D = null;
}
}
}
@ -322,13 +333,13 @@ public class GfsDataImport extends BaseTableOperation {
}
@Data
public static class ImportResult {
public static class ImportFileInfo {
private boolean success;
private String file;
private int forecastHour;
private int iTime;
public ImportResult(boolean success, String file, int forecastHour, int iTime) {
public ImportFileInfo(boolean success, String file, int forecastHour, int iTime) {
this.success = success;
this.file = file;
this.forecastHour = forecastHour;

@ -3,6 +3,7 @@ package com.htfp.weather.griddata.operation;
import com.alicloud.openservices.tablestore.model.search.sort.FieldSort;
import com.alicloud.openservices.tablestore.model.search.sort.Sort;
import com.alicloud.openservices.tablestore.model.search.sort.SortOrder;
import com.aliyun.tablestore.grid.consts.AttributionEnum;
import com.aliyun.tablestore.grid.core.QueryBuilder;
import com.aliyun.tablestore.grid.model.GridDataSetMeta;
import com.aliyun.tablestore.grid.model.QueryGridDataSetResult;
@ -11,7 +12,10 @@ import com.htfp.weather.griddata.common.TableConfig;
import org.springframework.stereotype.Component;
import org.springframework.util.CollectionUtils;
import javax.annotation.PostConstruct;
import javax.annotation.PreDestroy;
import javax.annotation.Resource;
import javax.crypto.interfaces.PBEKey;
import java.util.Collections;
import java.util.List;
@ -24,6 +28,11 @@ import java.util.List;
public class QueryMeta extends BaseTableOperation {
@Resource
TableConfig tableConfig;
@PreDestroy
public void destroy() {
close();
}
/**
*
*
@ -37,7 +46,8 @@ public class QueryMeta extends BaseTableOperation {
QueryBuilder.and()
.equal("status", "DONE")
.build(),
new QueryParams(0, 10, new Sort(Collections.singletonList(new FieldSort("reference_time", SortOrder.DESC)))));
new QueryParams(0, 10, new Sort(Collections.singletonList(new FieldSort(AttributionEnum.REFERENCE_TIME.getName(), SortOrder.DESC))))
);
List<GridDataSetMeta> gridDataSetMetas = result.getGridDataSetMetas();
if (CollectionUtils.isEmpty(gridDataSetMetas)) {
throw new RuntimeException("meta table为空");

@ -1,11 +1,10 @@
package com.htfp.weather.schedule;
import com.htfp.weather.download.FileInfo;
import com.htfp.weather.download.DownLoadFileInfo;
import com.htfp.weather.download.gfs.GfsDataConfig;
import com.htfp.weather.download.gfs.GfsDownloader;
import com.htfp.weather.griddata.operation.GfsDataFetcher;
import com.htfp.weather.griddata.operation.GfsDataImport;
import com.htfp.weather.griddata.operation.GfsDataImport.ImportResult;
import com.htfp.weather.griddata.operation.GfsDataImport.ImportFileInfo;
import com.htfp.weather.info.Constant;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.io.FileUtils;
@ -51,8 +50,8 @@ public class GridDataProcessor {
private final ExecutorService executorService = Executors.newFixedThreadPool(3);
boolean test = false;
/**每日北京时0,8,12,18点执行*/
@Scheduled(cron = "0 0 0,8,12,18 * * ?")
/**每日北京时0,8,12,18点 30分执行*/
@Scheduled(cron = "0 30 0,8,12,18 * * ?")
public void dailyDataProcess() {
if (download()) {
importToTableStore();
@ -60,16 +59,13 @@ public class GridDataProcessor {
clearExpiredData();
}
public boolean download() {
private boolean download() {
gfsDownloader.iniTimeSetting();
try {
log.info("[schedule-start] 开始下载气象数据");
List<FileInfo> fileInfoList = gfsDownloader.getFilesInfo();
// if (test){
// throw new RuntimeException("测试异常");
// }
List<FileInfo> finishedList = gfsDownloader.downloadAll(fileInfoList);
List<FileInfo> failedList = finishedList.stream().filter(fileInfo -> !fileInfo.isDownloadSuccess()).collect(Collectors.toList());
List<DownLoadFileInfo> downLoadFileInfoList = gfsDownloader.getFilesInfo();
List<DownLoadFileInfo> finishedList = gfsDownloader.downloadAll(downLoadFileInfoList);
List<DownLoadFileInfo> failedList = finishedList.stream().filter(fileInfo -> !fileInfo.isDownloadSuccess()).collect(Collectors.toList());
if (!failedList.isEmpty()) {
log.error("[schedule] 下载失败文件列表: {}", failedList);
downloadFailedReport(gfsDownloader.getRefTimeStr(), finishedList, failedList);
@ -84,16 +80,13 @@ public class GridDataProcessor {
}
}
public boolean importToTableStore(){
private boolean importToTableStore(){
try {
OffsetDateTime refTime = gfsDownloader.getRefTime();
log.info("[schedule-start] 开始将数据导入数据库, 起报时间: {}", refTime);
// if (test){
// throw new RuntimeException("测试异常");
// }
List<ImportResult> importResults = gfsDataImport.importData(refTime);
List<ImportResult> finishedList = importResults.stream().filter(result -> result.isSuccess()).collect(Collectors.toList());
List<ImportResult> failedList = importResults.stream().filter(result -> !result.isSuccess()).collect(Collectors.toList());
List<ImportFileInfo> importFileInfos = gfsDataImport.importData(refTime);
List<ImportFileInfo> finishedList = importFileInfos.stream().filter(result -> result.isSuccess()).collect(Collectors.toList());
List<ImportFileInfo> failedList = importFileInfos.stream().filter(result -> !result.isSuccess()).collect(Collectors.toList());
if (!failedList.isEmpty()) {
log.error("[schedule-end] 气象数据导入数据库失败, 文件列表: {} ", failedList);
importFailedReport(gfsDownloader.getRefTimeStr(), finishedList, failedList);
@ -132,7 +125,7 @@ public class GridDataProcessor {
log.info("[schedule-end] 过期数据清理结束");
}
private void downloadFailedReport(String refTime, List<FileInfo> finishedList, List<FileInfo> failedList) {
private void downloadFailedReport(String refTime, List<DownLoadFileInfo> finishedList, List<DownLoadFileInfo> failedList) {
// 邮件
Mail mail = new Mail();
mail.setSubject("【航天飞鹏-气象服务】数据下载失败通知");
@ -185,7 +178,7 @@ public class GridDataProcessor {
}
}
private void importFailedReport(String refTime, List<ImportResult> finishedList, List<ImportResult> failedList) {
private void importFailedReport(String refTime, List<ImportFileInfo> finishedList, List<ImportFileInfo> failedList) {
Mail mail = new Mail();
mail.setSubject("【航天飞鹏-气象服务】数据入库失败通知");
mail.setTos(mailReceiver);

@ -2,6 +2,7 @@ package com.htfp.weather.utils;
import java.time.*;
import java.time.format.DateTimeFormatter;
import java.util.TimeZone;
/**
* @Author : shiyi
@ -11,6 +12,8 @@ import java.time.format.DateTimeFormatter;
public class DateTimeUtils {
public static final String DEFAULT_PATTERN = "yyyy-MM-dd'T'HH:mm:ssxxx";
public static final DateTimeFormatter DEFAULT_FORMATTER = DateTimeFormatter.ofPattern(DEFAULT_PATTERN);
public static final ZoneOffset DEFAULT_ZONE_OFFSET = ZoneOffset.ofHours(TimeZone.getDefault().getRawOffset()/3600000);
public static final ZoneId DEFAULT_ZONE_ID = ZoneId.systemDefault();
/**
* ,
*
@ -132,4 +135,13 @@ public class DateTimeUtils {
ZonedDateTime zonedDateTime = instant.atZone(ZoneId.systemDefault());
return zonedDateTime.format(DEFAULT_FORMATTER);
}
/** 将OffsetDateTime对象转换为指定时区*/
public static OffsetDateTime offsetDateTimeToTargetZone(OffsetDateTime offsetDateTime, int ZoneTo) {
return offsetDateTime.withOffsetSameInstant(ZoneOffset.ofHours(ZoneTo));
}
/** 将OffsetDateTime对象转换为系统时区*/
public static OffsetDateTime offsetDateTimeToSystemZone(OffsetDateTime offsetDateTime) {
return offsetDateTime.withOffsetSameInstant(DEFAULT_ZONE_OFFSET);
}
}

@ -1,9 +0,0 @@
package com.htfp.weather.utils;
/**
* @Author : shiyi
* @Date : 2024/4/25 18:31
* @Description : https://www.jianshu.com/p/3b269082cbbb
*/
public class DownloadUtils {
}

@ -1,5 +1,8 @@
package com.htfp.weather.utils;
import com.htfp.weather.utils.ssl.HostInterceptor;
import com.htfp.weather.utils.ssl.SelectiveHostnameVerifier;
import com.htfp.weather.utils.ssl.SelectiveTrustManager;
import lombok.extern.slf4j.Slf4j;
import okhttp3.OkHttpClient;
import okhttp3.Request;
@ -16,9 +19,17 @@ import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;
import java.nio.file.Files;
import java.nio.file.StandardCopyOption;
import java.security.KeyManagementException;
import java.security.NoSuchAlgorithmException;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
import java.util.concurrent.TimeUnit;
import javax.net.ssl.*;
import javax.net.ssl.SSLContext;
import java.security.SecureRandom;
/**
* @Author : shiyi
* @Date : 2024/1/23 16:19
@ -26,13 +37,36 @@ import java.util.concurrent.TimeUnit;
*/
@Slf4j
public class HttpClientUtils {
private static final List<String> TRUSTED_DOMAINS = Arrays.asList("nomads.ncep.noaa.gov");
private static final OkHttpClient OKHTTP_CLIENT = new OkHttpClient.Builder()
.connectTimeout(30, TimeUnit.SECONDS)
.readTimeout(60, TimeUnit.SECONDS)
.writeTimeout(60, TimeUnit.SECONDS)
.build();
private static final OkHttpClient DONLOAD_OKHTTP_CLIENT = new OkHttpClient.Builder()
.build();
private static final OkHttpClient DOWNLOAD_OKHTTP_CLIENT = buildOKHttpClient(TRUSTED_DOMAINS).build();
public static OkHttpClient.Builder buildOKHttpClient(List<String> trustedDomains) {
try {
// 1. 创建自定义 TrustManager
SelectiveTrustManager trustManager = new SelectiveTrustManager(trustedDomains);
// 2. 初始化 SSLContext
SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(null, new TrustManager[]{trustManager}, new SecureRandom());
final SSLSocketFactory sslSocketFactory = sslContext.getSocketFactory();
OkHttpClient.Builder builder = new OkHttpClient.Builder();
builder.sslSocketFactory(sslSocketFactory, trustManager);
builder.addInterceptor(new HostInterceptor());
builder.hostnameVerifier(new SelectiveHostnameVerifier(trustedDomains));
return builder;
} catch (NoSuchAlgorithmException | KeyManagementException e) {
e.printStackTrace();
return new OkHttpClient.Builder();
} catch (Exception e) {
e.printStackTrace();
return new OkHttpClient.Builder();
}
}
// public static <T> T sendGet(String url, Map<String, String> params, Class<T> responseClass) {
// MultiValueMap<String, String> queryParams = new LinkedMultiValueMap<>();
// if (params != null) {
@ -53,12 +87,16 @@ public class HttpClientUtils {
// }
public static <T> T sendGet(String url, Map<String, String> params, Class<T> responseClass) throws Exception {
String jsonStr = sendGet(url, params);
return JSONUtils.json2pojo(jsonStr, responseClass);
return JSONUtils.json2obj(jsonStr, responseClass);
}
public static String sendGet(String url, Map<String, String> params) throws IOException {
StringBuilder stringBuilder = new StringBuilder();
if (!CollectionUtils.isEmpty(params)) {
params.keySet().forEach(res -> {
for (String res : params.keySet()) {
if (res == null || params.get(res) == null) {
continue;
}
if (StringUtils.isNotBlank(stringBuilder)) {
stringBuilder.append("&");
} else {
@ -69,7 +107,7 @@ public class HttpClientUtils {
} catch (UnsupportedEncodingException e) {
throw new RuntimeException(e);
}
});
}
}
Request request = new Builder()
.url(url + stringBuilder)
@ -89,23 +127,23 @@ public class HttpClientUtils {
}
public static boolean downloadFileByUrl(String sourceUrl, String localPath) {
public static void downloadFileByUrl(String sourceUrl, String localPath) throws IOException {
File file = new File(localPath);
Request request = new Request.Builder()
.url(sourceUrl)
.build();
try (Response response = DONLOAD_OKHTTP_CLIENT.newCall(request).execute()) {
try (Response response = DOWNLOAD_OKHTTP_CLIENT.newCall(request).execute()) {
if (response.code() == 200) {
assert response.body() != null;
InputStream stream = response.body().byteStream();
Files.copy(stream, file.toPath(), StandardCopyOption.REPLACE_EXISTING);
long dataBytes = Files.copy(stream, file.toPath(), StandardCopyOption.REPLACE_EXISTING);
log.debug("文件{}下载完成,文件大小: {} bytes", localPath, dataBytes);
} else {
return false;
throw new RuntimeException("文件下载失败Response" + response);
}
} catch (IOException e) {
return false;
}
return true;
}
}

@ -1,13 +1,13 @@
package com.htfp.weather.utils;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.JavaType;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
@ -20,67 +20,58 @@ public class JSONUtils {
}
public static ObjectMapper getInstance() {
return objectMapper;
}
/**
* javaBean,list,array convert to json string
*/
public static String obj2json(Object obj) throws Exception {
public static String obj2json(Object obj) throws JsonProcessingException {
return objectMapper.writeValueAsString(obj);
}
/**
* json string convert to javaBean
*/
public static <T> T json2pojo(String jsonStr, Class<T> clazz)
throws Exception {
public static <T> T json2obj(String jsonStr, Class<T> clazz) throws IOException {
return objectMapper.readValue(jsonStr, clazz);
}
/**
* JSON, T
*/
public static <T> T json2obj(String json, TypeReference<T> typeReference) throws IOException {
return objectMapper.readValue(json, typeReference);
}
/**
* json string convert to map
*/
public static <T> Map<String, Object> json2map(String jsonStr)
throws Exception {
return objectMapper.readValue(jsonStr, Map.class);
public static Map<String, Object> json2map(String jsonStr) throws IOException {
return objectMapper.readValue(jsonStr, new TypeReference<Map<String, Object>>() {});
}
/**
* json string convert to map with javaBean
* JSON string convert to map with specific javaBean
*/
public static <T> Map<String, T> json2map(String jsonStr, Class<T> clazz)
throws Exception {
Map<String, Map<String, Object>> map = (Map<String, Map<String, Object>>) objectMapper.readValue(jsonStr,
new TypeReference<Map<String, T>>() {
});
Map<String, T> result = new HashMap<String, T>();
for (Map.Entry<String, Map<String, Object>> entry : map.entrySet()) {
result.put(entry.getKey(), map2pojo(entry.getValue(), clazz));
}
return result;
public static <T> Map<String, T> json2map(String jsonStr, Class<T> clazz) throws IOException {
JavaType type = objectMapper.getTypeFactory().constructMapType(Map.class, String.class, clazz);
return objectMapper.readValue(jsonStr, type);
}
/**
* json array string convert to list with javaBean
* json array string convert to list with specific javaBean
*/
public static <T> List<T> json2list(String jsonArrayStr, Class<T> clazz)
throws Exception {
List<Map<String, Object>> list = (List<Map<String, Object>>) objectMapper.readValue(jsonArrayStr,
new TypeReference<List<T>>() {
});
List<T> result = new ArrayList<T>();
for (Map<String, Object> map : list) {
result.add(map2pojo(map, clazz));
}
return result;
throws IOException {
return objectMapper.readValue(
jsonArrayStr,
objectMapper.getTypeFactory().constructCollectionType(List.class, clazz)
);
}
/**
* map convert to javaBean
*/
public static <T> T map2pojo(Map map, Class<T> clazz) {
public static <T> T map2obj(Map<String, T> map, Class<T> clazz) {
return objectMapper.convertValue(map, clazz);
}
@ -89,4 +80,7 @@ public class JSONUtils {
objectMapper.enable(SerializationFeature.INDENT_OUTPUT);
objectMapper.writeValue(new File(filePath), obj);
}
public static void main(String[] args) throws Exception {
}
}

@ -8,7 +8,7 @@ import java.util.Arrays;
/**
* @Author : shiyi
* @Date : 2024/1/24 11:32
* @Description :
* @Description :
*/
public class MeteoUtils {
@ -30,13 +30,64 @@ public class MeteoUtils {
return kph / 3.6;
}
// public static double pressure2Height(double pressure) {
//
// }
//
// public static double height2Pressure(double pressure) {
//
// }
/**
*
* @param kph (km/h)
* @return
*/
public static int kmPerHour2WindScale(double kph) {
return mPerSecond2WindScale(kmPerHour2mPerSecond(kph));
}
/**
*
* @param mps (m/s)
* @return
*/
public static int mPerSecond2WindScale(double mps) {
if (mps < 0) {
throw new IllegalArgumentException("风力等级计算错误风速不能为负值");
}
if (mps <= 0.2) {
return 0;
} else if (mps <= 1.5) {
return 1;
} else if (mps <= 3.3) {
return 2;
} else if (mps <= 5.4) {
return 3;
} else if (mps <= 7.9) {
return 4;
} else if (mps <= 10.7) {
return 5;
} else if (mps <= 13.8) {
return 6;
} else if (mps <= 17.1) {
return 7;
} else if (mps <= 20.7) {
return 8;
} else if (mps <= 24.4) {
return 9;
} else if (mps <= 28.4) {
return 10;
} else if (mps <= 32.6) {
return 11;
} else if (mps <= 36.9){
return 12;
} else {
return 13;
}
}
/**
*
* @param currentPressure hPa
* @return
*/
public static double pressure2Elevation(double currentPressure) {
double referencePressure = 101.325; // 参考大气压单位kPa
return 44330 * (1 - Math.pow(currentPressure/10. / referencePressure, 1 / 5.255));
}
/**
* @param array (kg/m^2/s)
* @return mm/h
@ -62,7 +113,7 @@ public class MeteoUtils {
//
// @Deprecated
// public static Array calculateWindSpeed(Array u, Array v) {
// // // FIXME 2024/5/13: io耗时太多对于一般大小的矩阵不使
// // note 2024/5/13: io耗时太多对于一般大小的矩阵不实
// uvValid(u, v);
// try (NDManager manager = NDManager.newBaseManager()) {
// int[] shapeArray = u.getShape();

@ -0,0 +1,22 @@
package com.htfp.weather.utils.ssl;
import okhttp3.Interceptor;
import okhttp3.Response;
import org.jetbrains.annotations.NotNull;
import java.io.IOException;
public class HostInterceptor implements Interceptor {
private static final ThreadLocal<String> hostnameHolder = new ThreadLocal<>();
@NotNull
@Override
public Response intercept(Chain chain) throws IOException {
hostnameHolder.set(chain.request().url().host());
return chain.proceed(chain.request());
}
public static String getCurrentHostname() {
return hostnameHolder.get();
}
}

@ -0,0 +1,31 @@
package com.htfp.weather.utils.ssl;
import org.springframework.util.CollectionUtils;
import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.HttpsURLConnection;
import javax.net.ssl.SSLSession;
import java.util.Arrays;
import java.util.List;
public class SelectiveHostnameVerifier implements HostnameVerifier {
private List<String> trustedDomains;
private final HostnameVerifier defaultVerifier = HttpsURLConnection.getDefaultHostnameVerifier();
public SelectiveHostnameVerifier(){}
public SelectiveHostnameVerifier(List<String> trustedDomains) {
this.trustedDomains = trustedDomains;
}
@Override
public boolean verify(String hostname, SSLSession session) {
if (!CollectionUtils.isEmpty(trustedDomains) && trustedDomains.contains(hostname)) {
// 白名单域名:跳过主机名验证
return true;
} else {
// 其他域名:使用默认验证
return defaultVerifier.verify(hostname, session);
}
}
}

@ -0,0 +1,79 @@
package com.htfp.weather.utils.ssl;
import lombok.extern.slf4j.Slf4j;
import org.springframework.util.CollectionUtils;
import javax.net.ssl.TrustManagerFactory;
import javax.net.ssl.X509TrustManager;
import java.security.KeyStore;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.Arrays;
import java.util.List;
@Slf4j
public class SelectiveTrustManager implements X509TrustManager {
// 目标域名白名单example.com
private final List<String> trustedDomains;
private final X509TrustManager defaultTrustManager;
private final X509TrustManager fakeTrustManager;
public SelectiveTrustManager(List<String> trustedDomains) throws Exception {
this.trustedDomains = trustedDomains;
// 获取默认 TrustManager
TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
tmf.init((KeyStore) null);
defaultTrustManager = (X509TrustManager) tmf.getTrustManagers()[0];
// 创建一个“信任所有证书”的 TrustManager仅用于白名单域名
fakeTrustManager = new X509TrustManager() {
@Override
public void checkClientTrusted(X509Certificate[] chain, String authType) {
}
@Override
public void checkServerTrusted(X509Certificate[] chain, String authType) {
}
@Override
public X509Certificate[] getAcceptedIssuers() {
return new X509Certificate[0];
}
};
}
@Override
public void checkClientTrusted(X509Certificate[] chain, String authType) {
// 客户端验证交给默认逻辑
try {
defaultTrustManager.checkClientTrusted(chain, authType);
} catch (CertificateException e) {
throw new RuntimeException(e);
}
}
@Override
public void checkServerTrusted(X509Certificate[] chain, String authType) throws CertificateException {
// 获取当前请求的域名
String hostname = getCurrentHostname(); // 需要额外逻辑获取当前域名
if (!CollectionUtils.isEmpty(trustedDomains) && trustedDomains.contains(hostname)) {
// 白名单域名:跳过证书验证
log.debug("{} is in trusted domains, skipping certificate verification", hostname);
fakeTrustManager.checkServerTrusted(chain, authType);
} else {
// 非白名单域名:使用默认验证
defaultTrustManager.checkServerTrusted(chain, authType);
}
}
@Override
public X509Certificate[] getAcceptedIssuers() {
return defaultTrustManager.getAcceptedIssuers();
}
// 需要结合 OkHttp 拦截器或反射获取当前请求域名(见后续优化)
private String getCurrentHostname() {
return HostInterceptor.getCurrentHostname();
}
}

@ -0,0 +1,15 @@
package com.htfp.weather.web.config;
import com.htfp.weather.web.exception.AppException;
import com.htfp.weather.web.exception.ErrorCode;
import org.apache.commons.lang3.StringUtils;
public class Config {
public static final String SECRET = "htfpweather";
public static void validSecret(String secret) {
if (StringUtils.isEmpty(secret) || !SECRET.equals(secret)) {
throw new AppException(ErrorCode.SECRET_ERROR);
}
}
}

@ -1,9 +0,0 @@
package com.htfp.weather.web.config;
/**
* @Author : shiyi
* @Date : 2024/6/12 14:12
* @Description :
*/
public class LogAspect {
}

@ -0,0 +1,16 @@
package com.htfp.weather.web.config.aspect;
import java.lang.annotation.*;
/**
* @Author : shiyi
* @Date : 2024/7/15 16:01
* @Description :
*/
@Documented
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface ControllerLog {
String info(); // 调用接口的简要信息
boolean isSaveToDatabase() default false; //该条操作日志是否需要持久化存储, 当前未配置数据库, 默认为false
}

@ -0,0 +1,131 @@
package com.htfp.weather.web.config.aspect;
import com.htfp.weather.web.param.response.Result;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.aspectj.lang.JoinPoint;
import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.Signature;
import org.aspectj.lang.annotation.*;
import org.aspectj.lang.reflect.MethodSignature;
import org.springframework.core.annotation.Order;
import org.springframework.stereotype.Component;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.context.request.RequestContextHolder;
import org.springframework.web.context.request.ServletRequestAttributes;
import javax.servlet.http.HttpServletRequest;
import java.lang.reflect.Method;
import java.lang.reflect.Parameter;
import java.util.*;
/**
* @Author : shiyi
* @Date : 2024/6/12 14:12
* @Description :
*/
@Aspect
@Component
@Order(1)
@Slf4j
public class ControllerLogAspect {
/**定义切点指定controller包下面的公共方法且被ControllerLog注解修饰*/
@Pointcut("execution(public * com.htfp.weather.web.controller.*.*(..))" +
"&& @annotation(com.htfp.weather.web.config.aspect.ControllerLog)"
)
public void logPointCut() {
}
// @Before(value = "logPointCut()&& @annotation(controllerLog)")
// public void doBefore(JoinPoint joinPoint, ControllerLog controllerLog) {
// log.info("[controllerLog] {} start: {}", controllerLog.info(), joinPoint.getSignature().toShortString());
// }
//
// @AfterReturning(value = "logPointCut()&& @annotation(controllerLog)", returning = "ret")
// public void doAfterReturning(JoinPoint joinPoint, Object ret, ControllerLog controllerLog) {
// log.info("[controllerLog] {} finished: {}, return: {}", controllerLog.info(), joinPoint.getSignature().toShortString(), ret);
// }
@AfterThrowing(value = "logPointCut()&& @annotation(controllerLog)", throwing = "cause")
public void doAfterThrowing(JoinPoint joinPoint, Throwable cause, ControllerLog controllerLog){
log.info("[controllerLog] {} failed: {}, errorMsg: {}", controllerLog.info(), joinPoint.getSignature().toShortString(), cause.getMessage());
}
@Around(value = "logPointCut()&& @annotation(controllerLog)")
public Object doAround(ProceedingJoinPoint joinPoint, ControllerLog controllerLog) throws Throwable {
//获取当前请求对象
ServletRequestAttributes attributes = (ServletRequestAttributes) RequestContextHolder.getRequestAttributes();
if (attributes == null) {
return Result.error(new RuntimeException("切点处理异常: " + this.getClass().getName()));
}
HttpServletRequest request = attributes.getRequest();
Signature signature = joinPoint.getSignature();
MethodSignature methodSignature = (MethodSignature) signature;
Method method = methodSignature.getMethod();
//记录请求信息
Object parameter = getParameter(method, joinPoint.getArgs());
log.info("[controllerLog] {} start: {}, request params: {}", controllerLog.info(), signature.toShortString(), parameter);
long startTime = System.currentTimeMillis();
// 调用目标切点方法
Object result = joinPoint.proceed();
long endTime = System.currentTimeMillis();
int timeCost = (int) (endTime - startTime);
if (controllerLog.isSaveToDatabase()) {
WebLog webLog = new WebLog();
webLog.setDescription(controllerLog.info());
webLog.setIp(request.getRemoteUser());
webLog.setRequestMethod(request.getMethod());
webLog.setParameter(parameter);
webLog.setResult(result);
webLog.setTimeCost(timeCost);
webLog.setStartTime(startTime);
webLog.setUri(request.getRequestURI());
webLog.setUrl(request.getRequestURL().toString());
}
log.debug("[controllerLog] {} finished ({}ms): {}, return: {}", controllerLog.info(), timeCost, signature.toShortString(), result);
log.info("[controllerLog] {} finished ({}ms): {}", controllerLog.info(), timeCost, signature.toShortString());
// 返回值将由controller返回
return result;
}
/**
*
*/
private Object getParameter(Method method, Object[] args) {
List<Object> argList = new ArrayList<>();
Parameter[] parameters = method.getParameters();
for (int i = 0; i < parameters.length; i++) {
//将RequestBody注解修饰的参数作为请求参数
RequestBody requestBody = parameters[i].getAnnotation(RequestBody.class);
if (requestBody != null) {
argList.add(args[i]);
}
//将RequestParam注解修饰的参数作为请求参数
RequestParam requestParam = parameters[i].getAnnotation(RequestParam.class);
if (requestParam != null) {
Map<String, Object> map = new HashMap<>();
String key = parameters[i].getName();
if (!StringUtils.isEmpty(requestParam.value())) {
key = requestParam.value();
}
map.put(key, args[i]);
argList.add(map);
}
// else {
// argList.add(args[i]);
// }
}
if (argList.isEmpty()) {
return null;
} else if (argList.size() == 1) {
return argList.get(0);
} else {
return argList;
}
}
}

@ -0,0 +1,35 @@
package com.htfp.weather.web.config.aspect;
import lombok.Data;
/**
* @Author : shiyi
* @Date : 2024/7/15 15:44
* @Description :
*/
@Data
public class WebLog {
private String description; // 操作简介
private String username;
private Long startTime;
private Integer timeCost;
private String basePath;
private String uri;
private String url;
private String requestMethod;
private String ip;
private Object parameter;
private Object result;
}

@ -0,0 +1,108 @@
package com.htfp.weather.web.config.interceptor;
import com.htfp.weather.utils.JSONUtils;
import com.htfp.weather.web.exception.ErrorCode;
import com.htfp.weather.web.param.response.Result;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Component;
import org.springframework.web.servlet.HandlerInterceptor;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.util.Arrays;
import java.util.Formatter;
import java.util.List;
/**
* @Author : shiyi
* @Date : 2024/7/5 10:26
* @Description :
*/
@Slf4j @Component
public class InterfaceInterceptor implements HandlerInterceptor {
private static final Integer SECONDS = 60 * 1000;
@Override
public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
String servletPath = request.getServletPath();
// log.info("InterfaceInterceptor.preHandle >> 进入拦截, {}, {}", servletPath, handler.getClass().getName());
// ip 校验
if (checkIp(request)) {
return true;
}
// 签名校验
final String signature = request.getHeader("signature");
final String timestamp = String.valueOf(request.getHeader("timestamp"));
final String requestURI = request.getRequestURI();
String checkSignature = createSignature(timestamp, requestURI);
if (!checkSignature.equalsIgnoreCase(signature)) {
failedResponse(response);
return false;
}
//判断时间是否是60s内
long nowTime = System.currentTimeMillis();
long reqTime = Long.parseLong(timestamp);
if (Math.abs(nowTime - reqTime) > SECONDS) {
failedResponse(response);
return false;
}
return true;
}
private void failedResponse(HttpServletResponse response) throws IOException {
response.getWriter().write(JSONUtils.obj2json(Result.error(ErrorCode.API_VALIDATE_ERROR)));
}
private boolean checkIp(HttpServletRequest request) {
String remoteAddr = request.getRemoteAddr();
List<String> whiteNameList = Arrays.asList("127.0.0.1", "182.92.130.23", "123.57.5.1", "0:0:0:0:0:0:0:1");
if (whiteNameList.contains(remoteAddr)) {
return true;
}
if (remoteAddr.startsWith("172.10.0.")) {
// 局域网测试
return true;
}
return false;
}
/**
* @param key -
* @param data- /htfp/weather/upper/queryPlaneGrid
* @return
* @throws NoSuchAlgorithmException
* @throws InvalidKeyException
*/
private String createSignature(String key, String data) throws NoSuchAlgorithmException, InvalidKeyException {
// TODO 签名算法待定
SecretKeySpec signingKey = new SecretKeySpec(key.getBytes(), "HmacSHA1");
Mac mac = Mac.getInstance("HmacSHA1");
mac.init(signingKey);
byte[] bytes = mac.doFinal(data.getBytes());
return toHexString(bytes);
}
/**
* 16便
* @param bytes
* @return
*/
private String toHexString(byte[] bytes) {
Formatter formatter = new Formatter();
for (byte b : bytes) {
formatter.format("%02x", b);
}
return formatter.toString();
}
}

@ -0,0 +1,35 @@
package com.htfp.weather.web.config.interceptor;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.servlet.config.annotation.InterceptorRegistry;
import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;
import javax.annotation.Resource;
/**
* @Author : shiyi
* @Date : 2024/7/5 11:41
* @Description :
*/
@Configuration
public class WebAppConfigurer implements WebMvcConfigurer {
@Resource
private InterfaceInterceptor interfaceInterceptor;
@Override
public void addInterceptors(InterceptorRegistry registry) {
// 可以添加多个拦截器,一般只添加一个
// addPathPatterns("/**") 表示对所有请求都拦截
// .excludePathPatterns("/base/index") 表示排除对/base/index请求的拦截
// 多个拦截器可以设置order顺序值越小preHandle越先执行postHandle和afterCompletion越后执行
// order默认的值是0如果只添加一个拦截器可以不显示设置order的值
registry.addInterceptor(interfaceInterceptor)
.addPathPatterns("/**")
// .excludePathPatterns("/base/index")
.order(0);
}
}

@ -1,20 +1,25 @@
package com.htfp.weather.web.controller;
import com.htfp.weather.download.FileInfo;
import com.htfp.weather.download.DownLoadFileInfo;
import com.htfp.weather.download.gfs.GfsDataConfig;
import com.htfp.weather.download.gfs.GfsDownloader;
import com.htfp.weather.griddata.common.TableConfig;
import com.htfp.weather.griddata.operation.GfsDataImport;
import com.htfp.weather.griddata.operation.GfsDataImport.ImportResult;
import com.htfp.weather.griddata.operation.GfsDataImport.ImportFileInfo;
import com.htfp.weather.info.Constant;
import com.htfp.weather.utils.DateTimeUtils;
import com.htfp.weather.web.config.Config;
import com.htfp.weather.web.config.aspect.ControllerLog;
import com.htfp.weather.web.exception.AppException;
import com.htfp.weather.web.exception.ErrorCode;
import com.htfp.weather.web.pojo.request.GfsConfigUpdate;
import com.htfp.weather.web.pojo.request.TableConfigUpdate;
import com.htfp.weather.web.pojo.response.Result;
import com.htfp.weather.web.param.request.GfsConfigUpdate;
import com.htfp.weather.web.param.request.TableConfigUpdate;
import com.htfp.weather.web.param.response.DownloadResult;
import com.htfp.weather.web.param.response.ImportResult;
import com.htfp.weather.web.param.response.Result;
import com.htfp.weather.web.service.FileService;
import com.htfp.weather.web.service.GfsDataServiceImpl;
import org.apache.commons.lang3.StringUtils;
import org.jetbrains.annotations.NotNull;
import org.springframework.util.CollectionUtils;
import org.springframework.validation.annotation.Validated;
import org.springframework.web.bind.annotation.*;
@ -23,11 +28,11 @@ import javax.annotation.Resource;
import java.io.File;
import java.io.IOException;
import java.time.OffsetDateTime;
import java.time.LocalDateTime;
import java.time.ZoneOffset;
import java.time.format.DateTimeFormatter;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
/**
* @Author : shiyi
@ -43,23 +48,28 @@ public class ConfigController {
@Resource
GfsDataConfig gfsDataConfig;
@Resource
GfsDownloader gfsDownloader;
@Resource
GfsDataImport gfsDataImport;
@Resource
FileService fileService;
private final String SECRET = "htfpweather";
@Resource
GfsDataServiceImpl gfsDataService;
private void validSecret(String secret) {
if (StringUtils.isEmpty(secret) || !SECRET.equals(secret)) {
throw new AppException(ErrorCode.SECRET_ERROR);
@Resource
SurfaceWeatherController surfaceWeatherController;
@PostMapping("/switchSurfaceDataSource")
public Result switchService(@RequestBody Map<String, String> params) {
Config.validSecret(params.get("secret"));
String dataSource = params.get("dataSource");
if (StringUtils.isEmpty(dataSource)) {
return Result.error(ErrorCode.VALIDATE_ERROR, "dataSource不能为空");
}
return surfaceWeatherController.switchSurfaceDataSource(dataSource);
}
/**
* tablestore
*/
@RequestMapping("/queryDatabaseConfig")
@ControllerLog(info = "查询tablestore数据库配置")
public Result queryDatabaseConfig() {
return Result.success(tableConfig);
}
@ -68,10 +78,11 @@ public class ConfigController {
* tablestore
*/
@PostMapping("/updateDatabaseConfig")
@ControllerLog(info = "更新tablestore数据库配置")
public Result updateDatabaseConfig(@RequestBody TableConfigUpdate request) {
String secret = request.getSecret();
TableConfig updateConfig = request.getConfig();
validSecret(secret);
Config.validSecret(secret);
if (updateConfig == null) {
throw new AppException(ErrorCode.CONFIG_ERROR, "配置文件不能为空");
}
@ -88,6 +99,7 @@ public class ConfigController {
* GFS
*/
@RequestMapping("/queryDataSourceConfig")
@ControllerLog(info = "查询GFS下载配置")
public Result queryDataSourceConfig() {
return Result.success(gfsDataConfig);
}
@ -96,10 +108,11 @@ public class ConfigController {
* GFS
*/
@PostMapping("/updateDataSourceConfig")
@ControllerLog(info = "更新GFS下载配置")
public Result updateDataSourceConfig(@Validated @RequestBody GfsConfigUpdate request) {
String secret = request.getSecret();
GfsDataConfig updateConfig = request.getConfig();
validSecret(secret);
Config.validSecret(secret);
if (updateConfig == null) {
throw new AppException(ErrorCode.CONFIG_ERROR, "配置文件不能为空");
}
@ -116,19 +129,14 @@ public class ConfigController {
* 24
*/
@PostMapping("/downloadAllFiles")
@ControllerLog(info = "下载当前时刻未来24小时的数据文件")
public Result downloadAllFiles(@RequestBody Map<String, String> params) {
String secret = params.get("secret");
validSecret(secret);
Config.validSecret(secret);
// List<FileInfo> allCompleted = false;
try {
gfsDownloader.iniTimeSetting();
List<FileInfo> fileInfoList = gfsDownloader.getFilesInfo();
List<FileInfo> finishedList = gfsDownloader.downloadAll(fileInfoList);
List<FileInfo> successList = finishedList.stream().filter(FileInfo::isDownloadSuccess).collect(Collectors.toList());
List<FileInfo> failedList = finishedList.stream().filter(fileInfo -> !fileInfo.isDownloadSuccess()).collect(Collectors.toList());
Map<String, List<FileInfo>> data = new HashMap<>();
data.put("successList", successList);
data.put("failedList", failedList);
DownloadResult data = gfsDataService.downloadNowAllFile();
List<DownLoadFileInfo> failedList = data.getFailedList();
if (CollectionUtils.isEmpty(failedList)) {
return Result.success(data);
} else {
@ -143,49 +151,53 @@ public class ConfigController {
*
*/
@PostMapping("/downloadSingleFile")
@ControllerLog(info = "下载指定时刻文件")
public Result downloadSingleFile(@RequestBody Map<String, String> params) {
String secret = params.get("secret");
validSecret(secret);
Config.validSecret(secret);
OffsetDateTime offsetDateTime = OffsetDateTime.parse(params.get("time"), Constant.API_TIME_FORMATTER);
String targetUtcStr = DateTimeUtils.getUTCDateTime(offsetDateTime).format(DateTimeFormatter.ofPattern(Constant.UTC_TIME_STRING));
try {
gfsDownloader.iniTimeSetting();
List<FileInfo> fileInfoList = gfsDownloader.getFilesInfo(); // TODO 2024/6/12: 可以优化不用生成24小时所有的文件信息
for (FileInfo fileInfo : fileInfoList) {
if (fileInfo.getForecastUTCTimeStr().equals(targetUtcStr)) {
FileInfo downloadResult = gfsDownloader.download(fileInfo);
if (downloadResult.isDownloadSuccess()) {
return Result.success(downloadResult);
} else {
return Result.error(ErrorCode.DOWNLOAD_ERROR);
}
}
}
return Result.error(ErrorCode.QUERY_TIME_ERROR, ": 请选择当前时间之后24小时范围内的时刻");
} catch (AppException e) {
return Result.error(ErrorCode.DOWNLOAD_START_ERROR, e.getMessage());
}
DownLoadFileInfo downLoadFileInfo = gfsDataService.downloadSingleFile(offsetDateTime);
return Result.success(downLoadFileInfo);
}
/**
*
*/
@PostMapping("/importToTablestoreByReferenceTime")
@ControllerLog(info = "根据起报时间将数据导入数据库")
public Result importData(@RequestBody Map<String, String> params) {
String secret = params.get("secret");
validSecret(secret);
Config.validSecret(secret);
OffsetDateTime time = OffsetDateTime.parse(params.get("time"), Constant.API_TIME_FORMATTER);
return getImportResponse(time);
}
/**
*
*/
@PostMapping("/downloadAndImport")
@ControllerLog(info = "下载并导入")
public Result downloadAndImport(@RequestBody Map<String, String> params) {
Result downloadResponse = downloadAllFiles(params);
if (downloadResponse.isSuccess()) {
DownloadResult data = (DownloadResult) downloadResponse.getData();
String refTimeStr = data.getSuccessList().get(0).getRefTimeStr();
OffsetDateTime time = OffsetDateTime.of(LocalDateTime.parse(refTimeStr, DateTimeFormatter.ofPattern(Constant.UTC_TIME_STRING)), ZoneOffset.UTC);
return getImportResponse(time);
} else {
return downloadResponse;
}
}
@NotNull
private Result getImportResponse(OffsetDateTime time) {
try {
List<ImportResult> finishedList = gfsDataImport.importData(time);
List<ImportResult> successList = finishedList.stream().filter(result -> result.isSuccess()).collect(Collectors.toList());
List<ImportResult> failedList = finishedList.stream().filter(result -> !result.isSuccess()).collect(Collectors.toList());
Map<String, List<ImportResult>> data = new HashMap<>();
data.put("successList", successList);
data.put("failedList", failedList);
ImportResult importResult = gfsDataService.importData(time);
List<ImportFileInfo> failedList = importResult.getFailedList();
if (CollectionUtils.isEmpty(failedList)) {
return Result.success(data);
return Result.success(importResult);
} else {
return new Result(false, ErrorCode.TABLESTORE_IMPORT_ERROR.getCode(), ErrorCode.TABLESTORE_IMPORT_ERROR.getMsg() + ": 数据导入未全部完成", data);
return new Result(false, ErrorCode.TABLESTORE_IMPORT_ERROR.getCode(), ErrorCode.TABLESTORE_IMPORT_ERROR.getMsg() + ": 数据导入未全部完成", importResult);
}
} catch (AppException e) {
return Result.error(e.getErrorCode());
@ -195,26 +207,29 @@ public class ConfigController {
}
@PostMapping("queryAllDataDir")
@ControllerLog(info = "查询所有数据文件夹")
public Result queryAllDataDir(@RequestBody Map<String, String> params) {
String secret = params.get("secret");
validSecret(secret);
Config.validSecret(secret);
File dataRoot = new File(gfsDataConfig.getSaveRoot());
List<String> subDirList = fileService.getSubDirList(dataRoot);
return Result.success(subDirList);
}
/**
*
*/
@PostMapping("/queryAllFileByReferenceTime")
@ControllerLog(info = "查询指定起报时间文件夹中的所有文件名")
public Result queryAllFileByReferenceTime(@RequestBody Map<String, String> params) {
String secret = params.get("secret");
validSecret(secret);
Config.validSecret(secret);
OffsetDateTime time = OffsetDateTime.parse(params.get("time"), Constant.API_TIME_FORMATTER);
String timeStr = DateTimeUtils.getUTCDateTime(time).format(DateTimeFormatter.ofPattern(Constant.DATA_FOLDER_STRING));
File dataDir = new File(gfsDataConfig.getSaveRoot(), timeStr);
// 查找目录下的文件
List<com.htfp.weather.web.pojo.response.FileInfo> fileList = fileService.getFileList(dataDir);
List<com.htfp.weather.web.param.response.FileInfo> fileList = fileService.getFileList(dataDir);
return Result.success(fileList);
}
@ -222,9 +237,10 @@ public class ConfigController {
*
*/
@PostMapping("/deleteDirByReferenceTime")
@ControllerLog(info = "删除指定起报时间对应的文件夹")
public Result deleteDirByReferenceTime(@RequestBody Map<String, String> params) {
String secret = params.get("secret");
validSecret(secret);
Config.validSecret(secret);
OffsetDateTime time = OffsetDateTime.parse(params.get("time"), Constant.API_TIME_FORMATTER);
String timeStr = DateTimeUtils.getUTCDateTime(time).format(DateTimeFormatter.ofPattern(Constant.DATA_FOLDER_STRING));
File dataDir = new File(gfsDataConfig.getSaveRoot(), timeStr);
@ -243,9 +259,10 @@ public class ConfigController {
*
*/
@PostMapping("/deleteTargetFile")
@ControllerLog(info = "删除指定文件")
public Result deleteTargetFile(@RequestBody Map<String, String> params) {
String secret = params.get("secret");
validSecret(secret);
Config.validSecret(secret);
String filePath = params.get("filePath");
File file = new File(filePath);
if (gfsDataImport.isImporting()) {

@ -2,7 +2,7 @@ package com.htfp.weather.web.controller;
import com.htfp.weather.web.exception.AppException;
import com.htfp.weather.web.exception.ErrorCode;
import com.htfp.weather.web.pojo.response.Result;
import com.htfp.weather.web.param.response.Result;
import org.springframework.validation.BindException;
import org.springframework.validation.ObjectError;
import org.springframework.web.bind.annotation.ExceptionHandler;

@ -0,0 +1,54 @@
package com.htfp.weather.web.controller;
import com.google.common.primitives.Ints;
import com.htfp.weather.griddata.common.TableConfig;
import com.htfp.weather.web.config.aspect.ControllerLog;
import com.htfp.weather.web.exception.ErrorCode;
import com.htfp.weather.web.param.response.LevelsAvailableResponse;
import com.htfp.weather.web.param.response.LevelsAvailableResponse.LevelsAvailable;
import com.htfp.weather.web.param.response.Result;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.web.bind.annotation.*;
import javax.annotation.Resource;
import java.util.*;
/**
* @Author : shiyi
* @Date : 2024/4/21 15:56
* @Description :
*/
@CrossOrigin
@RestController
@Slf4j
@RequestMapping("/htfp/weather/v1/info")
public class InfoController {
@Resource
TableConfig tableConfig;
@PostMapping("/queryLevelsAvailable")
@ControllerLog(info = "可选高度查询")
public Result queryLevelsAvailable(Map<String, String> params) {
String dataSource = params.get("dataSource");
List<LevelsAvailable> levelsAvailableList = new ArrayList<>();
if (StringUtils.isEmpty(dataSource)) {
levelsAvailableList.add(
new LevelsAvailable(
"GFS",
Ints.asList(tableConfig.getPressureList()),
Ints.asList(tableConfig.getPressureHeightList()))
);
} else {
if (dataSource.equals("GFS")) {
levelsAvailableList = Collections.singletonList(
new LevelsAvailable(
"GFS",
Ints.asList(tableConfig.getPressureList()),
Ints.asList(tableConfig.getPressureHeightList()))
);
}
}
return Result.success(new LevelsAvailableResponse(levelsAvailableList));
}
}

@ -1,19 +1,23 @@
package com.htfp.weather.web.controller;
import com.htfp.weather.web.pojo.hefeng.HeFengWarningResponse;
import com.htfp.weather.web.pojo.request.Position2D;
import com.htfp.weather.web.pojo.response.NowWeatherStatus;
import com.htfp.weather.web.pojo.response.Result;
import com.htfp.weather.web.pojo.response.SurfaceWeatherWarning;
import com.htfp.weather.web.pojo.response.TimeSeriesDataset;
import com.htfp.weather.web.config.aspect.ControllerLog;
import com.htfp.weather.web.exception.ErrorCode;
import com.htfp.weather.web.param.cma.CmaWarning;
import com.htfp.weather.web.param.request.Position2D;
import com.htfp.weather.web.param.response.NowWeatherStatus;
import com.htfp.weather.web.param.response.Result;
import com.htfp.weather.web.param.response.SurfaceWeatherWarning;
import com.htfp.weather.web.param.response.TimeSeriesDataset;
import com.htfp.weather.web.service.GfsDataServiceImpl;
import com.htfp.weather.web.service.surfaceapi.CaiYunServiceImpl;
import com.htfp.weather.web.service.surfaceapi.CmaServiceImpl;
import com.htfp.weather.web.service.surfaceapi.HeFengServiceImpl;
import com.htfp.weather.web.service.IDataService;
import com.htfp.weather.web.service.surfaceapi.ISurfaceDataService;
import lombok.extern.slf4j.Slf4j;
import org.springframework.validation.annotation.Validated;
import org.springframework.web.bind.annotation.*;
import javax.annotation.PostConstruct;
import javax.annotation.Resource;
import java.util.List;
@ -23,44 +27,80 @@ import java.util.List;
* @Description :
*/
@CrossOrigin
@RestController @Slf4j
@RequestMapping("/htfp/weather/surface/")
@RestController
@Slf4j
@RequestMapping("/htfp/weather/v1/surface/")
public class SurfaceWeatherController {
@Resource(name = "hefeng")
ISurfaceDataService surfaceDataService;
@Resource
CmaServiceImpl cmaService;
@Resource
HeFengServiceImpl heFengService;
@Resource
CaiYunServiceImpl caiYunService;
@Resource
GfsDataServiceImpl gfsDataService;
ISurfaceDataService service;
@PostConstruct
public void init() {
service = gfsDataService;
}
public Result switchSurfaceDataSource(String dataSource) {
switch (dataSource.toLowerCase()) {
case "hefeng":
service = heFengService;
return Result.success();
case "caiyun":
service = caiYunService;
return Result.success();
default:
return Result.error(ErrorCode.VALIDATE_ERROR, "未知数据源");
}
}
@PostMapping("/querySurfaceNowWeather")
@ControllerLog(info = "地面实时气象信息查询")
public Result queryNowWeather(@Validated @RequestBody Position2D position2D) throws Exception {
double lat = position2D.getLatitude();
double lon = position2D.getLongitude();
log.info("[data-server] 地面实时气象信息查询 start: param={}", position2D);
NowWeatherStatus nowWeatherStatus = surfaceDataService.getNowSurfaceWeatherStatus(lat, lon);
NowWeatherStatus nowWeatherStatus = service.getNowSurfaceWeatherStatus(lat, lon);
nowWeatherStatus.setLatitude(lat);
nowWeatherStatus.setLongitude(lon);
log.info("[data-server] 地面实时气象信息查询 end");
nowWeatherStatus.setDataSource(service.getDataSource());
return Result.success(nowWeatherStatus);
}
@PostMapping("/querySurfaceForecast")
@ControllerLog(info = "地面24小时预报结果查询")
public Result querySurfaceForecast(@Validated @RequestBody Position2D position2D) throws Exception {
double lat = position2D.getLatitude();
double lon = position2D.getLongitude();
log.info("[data-server] 地面24小时预报结果查询 start: param={}", position2D);
TimeSeriesDataset forecastSeries = surfaceDataService.getForecastSeries(lat, lon);
TimeSeriesDataset forecastSeries = service.getForecastSeries(lat, lon);
forecastSeries.setLatitude(lat);
forecastSeries.setLongitude(lon);
log.info("[data-server] 地面24小时预报结果查询");
forecastSeries.setDataSource(service.getDataSource());
return Result.success(forecastSeries);
}
@PostMapping ("/queryWeatherWarning")
@PostMapping("/queryWeatherWarning")
@ControllerLog(info = "地面气象预警信息查询")
public Result queryWeatherWarning(@Validated @RequestBody Position2D position2D) throws Exception {
double lat = position2D.getLatitude();
double lon = position2D.getLongitude();
log.info("[data-server] 地面气象预警信息查询 start: param={}", position2D);
List<SurfaceWeatherWarning> warning = surfaceDataService.getSurfaceWarning(lat, lon);
log.info("[data-server] 地面气象预警信息查询 end");
// List<SurfaceWeatherWarning> warning = surfaceDataService.getSurfaceWarning(lat, lon);
List<SurfaceWeatherWarning> warning = cmaService.getSurfaceWarning(lat, lon);
return Result.success(warning);
}
@PostMapping("/queryAllWeatherWarning")
@ControllerLog(info = "全国地面气象预警信息查询")
public Result queryWeatherWarning() throws Exception {
// List<SurfaceWeatherWarning> warning = surfaceDataService.getSurfaceWarning(lat, lon);
List<CmaWarning> cmaWarningList = cmaService.getSurfaceWeatherWarnings(null);
cmaService.updateCmaWarningCache(cmaWarningList);
return Result.success(cmaWarningList);
}
}

@ -1,29 +0,0 @@
package com.htfp.weather.web.controller;
import com.htfp.weather.griddata.common.TableConfig;
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.web.bind.annotation.GetMapping;
import javax.annotation.Resource;
/**
* @Author : shiyi
* @Date : 2024/4/20 17:30
* @Description :
*/
@Controller
public class ThymeleafTest {
@Resource
TableConfig tableConfig;
@GetMapping("index")// 页面的url地址
public String getindex(Model model, String secret) {
if (!"shiyi".equals(secret)){
return "wrong";
}
model.addAttribute("tableConfig", tableConfig);
return "index";// 与templates中index.html对应
}
}

@ -1,12 +1,14 @@
package com.htfp.weather.web.controller;
import com.htfp.weather.utils.DateTimeUtils;
import com.htfp.weather.web.config.aspect.ControllerLog;
import com.htfp.weather.web.exception.AppException;
import com.htfp.weather.web.exception.ErrorCode;
import com.htfp.weather.web.pojo.request.*;
import com.htfp.weather.web.pojo.response.*;
import com.htfp.weather.web.param.request.*;
import com.htfp.weather.web.param.response.*;
import com.htfp.weather.web.service.GfsDataServiceImpl;
import lombok.extern.slf4j.Slf4j;
import org.springframework.util.CollectionUtils;
import org.springframework.validation.annotation.Validated;
import org.springframework.web.bind.annotation.*;
@ -21,106 +23,126 @@ import java.util.List;
* @Description :
*/
@CrossOrigin
@RestController @Slf4j
@RequestMapping("/htfp/weather/upper/")
@RestController
@Slf4j
@RequestMapping("/htfp/weather/v1/upper/")
public class UpperWeatherController {
@Resource(name = "tablestore-gfs")
GfsDataServiceImpl gfsDataService;
@PostMapping("/queryUpperNowWeather")
@ControllerLog(info = "高空实时气象信息查询")
public Result queryUpperNowWeather(@Validated @RequestBody Position3D position3D) {
double latitude = position3D.getLatitude();
double longitude = position3D.getLongitude();
int level = position3D.getLevel();
log.info("[data-server] 高空实时气象信息查询 start: param={}", position3D);
NowWeatherStatus nowWeather = gfsDataService.getNowWeather(level, latitude, longitude);
log.info("[data-server] 高空实时气象信息查询 end");
return Result.success(nowWeather);
}
@PostMapping("/queryUpperNowWeatherInMultiPoints")
@ControllerLog(info = "高空航点实时气象信息查询")
public Result queryUpperNowWeatherInMultiPoints(@Validated @RequestBody MultiPointsRequest multiPointsRequest) {
List<Double> latitudeList = new ArrayList<>();
List<Double> longitudeList = new ArrayList<>();
for (Position2D position : multiPointsRequest.getPointList()) {
Double latitude = position.getLatitude();
Double longitude = position.getLongitude();
if (latitude != null && longitude != null) {
latitudeList.add(latitude);
longitudeList.add(longitude);
} else {
throw new AppException(ErrorCode.MULTI_INDEX_ERROR);
}
}
int level = multiPointsRequest.getLevel().intValue();
log.info("[data-server] 高空航点实时气象信息查询 start: param={}", multiPointsRequest);
List<Position2D> pointList = multiPointsRequest.getPointList();
int size = pointList.size();
List<Double> latitudeList = new ArrayList<>(size);
List<Double> longitudeList = new ArrayList<>(size);
checkLatitudeLongitude(latitudeList, longitudeList, pointList);
int level = multiPointsRequest.getLevel();
List<NowWeatherStatus> nowWeatherByMultiPoint = gfsDataService.getNowWeatherByMultiPoint(level, latitudeList, longitudeList);
log.info("[data-server] 高空航点实时气象信息查询 end");
return Result.success(nowWeatherByMultiPoint);
}
@PostMapping("/queryUpperForecast")
@ControllerLog(info = "高空24小时预报结果查询")
public Result queryUpperForecast(@Validated @RequestBody Position3D position3D) {
double latitude = position3D.getLatitude();
double longitude = position3D.getLongitude();
int level = position3D.getLevel();
log.info("[data-server] 高空24小时预报结果查询 start: param={}", position3D);
TimeSeriesDataset forecastSeries = gfsDataService.getForecastSeries(level, latitude, longitude);
log.info("[data-server] 高空24小时预报结果查询 end");
return Result.success(forecastSeries);
}
@PostMapping("/queryUpperForecastInMultiPoints")
@ControllerLog(info = "高空航点24小时预报结果查询")
public Result queryUpperForecastInMultiPoints(@Validated @RequestBody MultiPointsRequest multiPointsRequest) {
List<Double> latitudeList = new ArrayList<>();
List<Double> longitudeList = new ArrayList<>();
for (Position2D position : multiPointsRequest.getPointList()) {
Double latitude = position.getLatitude();
Double longitude = position.getLongitude();
if (latitude != null && longitude != null) {
latitudeList.add(latitude);
longitudeList.add(longitude);
} else {
throw new AppException(ErrorCode.MULTI_INDEX_ERROR);
}
}
int level = multiPointsRequest.getLevel().intValue();
log.info("[data-server] 高空航点24小时预报结果查询 start: param={}", multiPointsRequest);
List<TimeSeriesDataset> nowWeatherByMultiPoint = gfsDataService.getForecastSeriesByMultiPoint(latitudeList, longitudeList, level);
log.info("[data-server] 高空航点24小时预报结果查询 end");
List<Position2D> pointList = multiPointsRequest.getPointList();
int size = pointList.size();
List<Double> latitudeList = new ArrayList<>(size);
List<Double> longitudeList = new ArrayList<>(size);
checkLatitudeLongitude(latitudeList, longitudeList, pointList);
int level = multiPointsRequest.getLevel();
List<TimeSeriesDataset> nowWeatherByMultiPoint = gfsDataService.getForecastSeriesByMultiPoint(level, latitudeList, longitudeList);
return Result.success(nowWeatherByMultiPoint);
}
/**查询指定变量随气压的分布*/
/**
*
*/
@RequestMapping("/queryProfileByVariableAndPressure")
@ControllerLog(info = "指定变量高度廓线(气压坐标)查询")
public Result queryProfileByVariableAndPressure(@Validated @RequestBody ProfileRequest profileRequest) {
OffsetDateTime time = OffsetDateTime.parse(profileRequest.getTime());
OffsetDateTime utcDateTime = DateTimeUtils.getUTCDateTime(time);
String variableName = profileRequest.getVariableName();
double latitude = profileRequest.getLatitude();
double longitude = profileRequest.getLongitude();
log.info("[data-server] 高度廓线查询 start: param={}", profileRequest);
ProfileResponse profileResponse = gfsDataService.getProfileByVariableAndPressure(variableName, utcDateTime, latitude, longitude);
log.info("[data-server] 高度廓线查询 end");
return Result.success(profileResponse);
}
/**查询全部变量随气压的分布*/
/**
*
*/
@RequestMapping("/queryProfileByPressure")
@ControllerLog(info = "全部变量高度廓线(气压坐标)查询")
public Result queryProfileByAndPressure(@Validated @RequestBody ProfileRequest profileRequest) {
OffsetDateTime time = OffsetDateTime.parse(profileRequest.getTime());
OffsetDateTime utcDateTime = DateTimeUtils.getUTCDateTime(time);
double latitude = profileRequest.getLatitude();
double longitude = profileRequest.getLongitude();
log.info("[data-server] 高度廓线查询 start: param={}", profileRequest);
ProfileDataset profile = gfsDataService.getProfileByPressure(utcDateTime, latitude, longitude);
log.info("[data-server] 高度廓线查询 end");
return Result.success(profile);
}
/**
*
*/
@RequestMapping("/queryProfileByVariableAndNearSurfaceHeight")
@ControllerLog(info = "指定变量高度廓线(近地面高度坐标)查询")
public Result queryProfileByVariableAndNearSurfaceHeight(@Validated @RequestBody ProfileRequest profileRequest) {
OffsetDateTime time = OffsetDateTime.parse(profileRequest.getTime());
OffsetDateTime utcDateTime = DateTimeUtils.getUTCDateTime(time);
String variableName = profileRequest.getVariableName();
double latitude = profileRequest.getLatitude();
double longitude = profileRequest.getLongitude();
ProfileResponse profile = gfsDataService.getProfileByVariableAndNearSurfaceHeight(variableName, utcDateTime, latitude, longitude);
return Result.success(profile);
}
/**
*
*/
@RequestMapping("/queryProfileByNearSurfaceHeight")
@ControllerLog(info = "全部变量高度廓线(近地面高度坐标)查询")
public Result queryProfileByNearSurfaceHeight(@Validated @RequestBody ProfileRequest profileRequest) {
OffsetDateTime time = OffsetDateTime.parse(profileRequest.getTime());
OffsetDateTime utcDateTime = DateTimeUtils.getUTCDateTime(time);
double latitude = profileRequest.getLatitude();
double longitude = profileRequest.getLongitude();
log.info("[data-server] 高度廓线(近地面高度坐标)查询 start: param={}", profileRequest);
ProfileDataset profile = gfsDataService.getProfileByNearSurfaceHeight(utcDateTime, latitude, longitude);
return Result.success(profile);
}
@PostMapping("/queryPlaneGrid")
@ControllerLog(info = "平面网格数据查询")
public Result queryPlaneGrid(@Validated @RequestBody PlaneRequest planeRequest) {
planeRequest.valid();
String variableName = planeRequest.getVariableName();
@ -131,10 +153,27 @@ public class UpperWeatherController {
double maxLat = planeRequest.getMaxLatitude();
double minLon = planeRequest.getMinLongitude();
double maxLon = planeRequest.getMaxLongitude();
log.info("[data-server] 平面网格数据查询 start: param={}", planeRequest);
PlaneResponse forecastSeries = gfsDataService.getPlane(utcDateTime, variableName, level, minLat, maxLat, minLon, maxLon);
log.info("[data-server] 平面网格数据查询 end");
return Result.success(forecastSeries);
}
private void checkLatitudeLongitude(List<Double> latitudeList, List<Double> longitudeList, List<Position2D> pointList) {
if (CollectionUtils.isEmpty(pointList)) {
throw new AppException(ErrorCode.EMPTY_INDEX_ERROR, "航点列表为空");
}
int size = pointList.size();
for (Position2D position : pointList) {
Double latitude = position.getLatitude();
Double longitude = position.getLongitude();
if (latitude != null && longitude != null) {
latitudeList.add(latitude);
longitudeList.add(longitude);
} else {
throw new AppException(ErrorCode.MULTI_INDEX_ERROR, "经纬度不允许为空");
}
}
}
}

@ -12,10 +12,11 @@ public class AppException extends RuntimeException{
return errorCode;
}
public AppException(ErrorCode errorCode) {
super(errorCode.getMsg());
this.errorCode = errorCode;
}
public AppException(ErrorCode errorCode, String message) {
super(message);
super(errorCode.getMsg()+ ", " +message);
this.errorCode = errorCode;
}
public AppException(ErrorCode errorCode, Throwable e) {

@ -1,5 +1,7 @@
package com.htfp.weather.web.exception;
import org.checkerframework.checker.units.qual.A;
/**
* @Author : shiyi
* @Date : 2024/3/14 16:30
@ -8,7 +10,7 @@ package com.htfp.weather.web.exception;
public enum ErrorCode {
//
OTHER_ERROR(-1, "其他错误"),
API_VALIDATE_ERROR(-1, "接口校验错误"),
VALIDATE_ERROR(1001, "参数校验错误 "),
CONFIG_ERROR(1002, "配置相关错误 "),
SECRET_ERROR(1003, "无权限访问"),
@ -21,9 +23,12 @@ public enum ErrorCode {
// 数据查询
HE_FENG_THIS_AREA_HAVE_NO_DATA(3001, "查询的数据或地区不存在"),
HE_FENG_REQUEST_ERROR(3002, "查询请求错误"),
CAI_YUN_REQUEST_ERROR(4002, "查询请求错误"),
CAI_YUN_DATA_PARSE_ERROR(4003, "数据解析错误"),
CAI_YUN_REQUEST_ERROR(3003, "查询请求错误"),
CAI_YUN_DATA_PARSE_ERROR(3004, "数据解析错误"),
CMA_REQUEST_ERROR(3005, "查询请求错误"),
CMA_DATA_PARSE_ERROR(3006, "数据解析错误"),
TIANDITU_REQUEST_ERROR(3007, "查询请求错误"),
TIANDITU_DATA_PARSE_ERROR(3008, "数据解析错误"),
IMPORT_DATA_INITIAL_FAILED(5001, "导入数据初始化失败"),
NO_NC_OR_GRIB_FILES(5002, "文件夹不存在或中没有对应的气象数据文件"),
@ -36,6 +41,7 @@ public enum ErrorCode {
NO_SUCH_VARIABLE(6006, "找不到变量名映射"),
TABLESTORE_QUERY_ERROR(6007, "数据库查询错误"),
RESPONSE_DATA_BUILD_ERROR(6008, "查询数据结果构建错误"),
EMPTY_INDEX_ERROR(6009, "坐标索引为空"),
;
private final int code;
private final String msg;

@ -1,9 +1,7 @@
package com.htfp.weather.web.pojo.caiyun;
package com.htfp.weather.web.param.caiyun;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonProperty;
import lombok.Data;
import lombok.EqualsAndHashCode;
import java.util.List;

@ -1,9 +1,8 @@
package com.htfp.weather.web.pojo.caiyun;
package com.htfp.weather.web.param.caiyun;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonProperty;
import lombok.Data;
import lombok.EqualsAndHashCode;
import java.util.List;

@ -1,9 +1,8 @@
package com.htfp.weather.web.pojo.caiyun;
package com.htfp.weather.web.param.caiyun;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonProperty;
import lombok.Data;
import lombok.EqualsAndHashCode;
/**
* @Author : shiyi

@ -1,4 +1,4 @@
package com.htfp.weather.web.pojo.caiyun;
package com.htfp.weather.web.param.caiyun;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonProperty;

@ -0,0 +1,50 @@
package com.htfp.weather.web.param.cma;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonProperty;
import lombok.Data;
/**
* @Author : shiyi
* @Date : 2024/7/2 16:13
* @Description :
*/
@Data
public class AntiGeoCodeResponse {
Result result;
String msg;
String status;
@Data @JsonIgnoreProperties(value = {"formattedAddress", "location"}, ignoreUnknown = true)
public static class Result {
AddressComponent addressComponent;
@Data @JsonIgnoreProperties(ignoreUnknown = true)
public static class AddressComponent {
private String address;
private String city;
@JsonProperty("county_code")
private String countyCode;
private String nation;
@JsonProperty("poi_position")
private String poiPosition;
private String county;
@JsonProperty("city_code")
private String cityCode;
@JsonProperty("address_position")
private String addressPosition;
private String poi;
@JsonProperty("province_code")
private String provinceCode;
private String province;
private String road;
@JsonProperty("road_distance")
private int roadDistance;
@JsonProperty("poi_distance")
private int poiDistance;
@JsonProperty("address_distance")
private int addressDistance;
private String info;
}
}
}

@ -0,0 +1,32 @@
package com.htfp.weather.web.param.cma;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import lombok.Data;
/**
* @Author : shiyi
* @Date : 2024/7/1 17:58
* @Description :
*/
@Data @JsonIgnoreProperties(value = {"warnDefine"}, ignoreUnknown = true)
public class CmaWarning {
private String expires;
private String procince;
private Integer leveltype;
private String signalypecode;
private Double lon;
private Double lat;
private Integer zoom;
private String issuecontent; // 发布内容
private String signallevelcode; // 颜色等级: YELLOW, BLUE, ORANGE, RED
private String reference;
private String areaId; // 县级行政编码
private String signallevel; // 预警等级:黄色,蓝色,橙色,红色
private String dataid; // 预警数据ID
private String sender; // 数据发布单位
private String codename; // 预警信号名称
private String name; // 行政单位名称
private String time; // 发布时间
private String msgtype; // 消息类型ALERT、UPDATE
private String procincecode; // 省级行政编码(前两位)
}

@ -1,9 +1,8 @@
package com.htfp.weather.web.pojo.hefeng;
package com.htfp.weather.web.param.hefeng;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import lombok.Data;
import lombok.EqualsAndHashCode;
import org.jetbrains.annotations.NotNull;
import java.util.List;

@ -1,4 +1,4 @@
package com.htfp.weather.web.pojo.hefeng;
package com.htfp.weather.web.param.hefeng;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import lombok.Data;

@ -1,4 +1,4 @@
package com.htfp.weather.web.pojo.hefeng;
package com.htfp.weather.web.param.hefeng;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import lombok.Data;

@ -1,4 +1,4 @@
package com.htfp.weather.web.pojo.request;
package com.htfp.weather.web.param.request;
import com.htfp.weather.download.gfs.GfsDataConfig;
import lombok.Data;

@ -1,5 +1,7 @@
package com.htfp.weather.web.pojo.request;
package com.htfp.weather.web.param.request;
import com.htfp.weather.download.gfs.GfsLevelsEnum;
import com.htfp.weather.web.valid.EnumValid;
import lombok.Data;
import javax.validation.constraints.NotNull;
@ -12,7 +14,10 @@ import java.util.List;
*/
@Data
public class MultiPointsRequest {
@NotNull
List<Position2D> pointList;
@NotNull(message = "高度层次 (level) 不能为空")
@EnumValid(message = "高度层次 (level) 数值错误", enumClass = GfsLevelsEnum.class)
Integer level;
}

@ -1,4 +1,4 @@
package com.htfp.weather.web.pojo.request;
package com.htfp.weather.web.param.request;
import com.htfp.weather.download.gfs.GfsVariableIsobaricEnum;
import com.htfp.weather.download.gfs.GfsLevelsEnum;

@ -1,4 +1,4 @@
package com.htfp.weather.web.pojo.request;
package com.htfp.weather.web.param.request;
import lombok.Data;
@ -14,13 +14,13 @@ import javax.validation.constraints.NotNull;
@Data
public class Position2D {
@NotNull(message = "纬度 (latitude) 不能为空")
@Min(value = -90, message = "纬度 (latitude) 最小值为-90南纬9")
@Max(value = 90, message = "纬度 (latitude) 最大值为90北纬90°)")
@Min(value = 0, message = "纬度 (latitude) 最小值为0")
@Max(value = 55, message = "纬度 (latitude) 最大值为55北纬55°)")
Double latitude;
@NotNull(message = "经度 (longitude) 不能为空")
@Min(value = -180, message = "经度 (longitude) 最小值为-180西经18")
@Max(value = 180, message = "经度 (longitude) 最大值为180东经18")
@Min(value = 70, message = "经度 (longitude) 最小值为 70东经 7")
@Max(value = 140, message = "经度 (longitude) 最大值为 140东经 14")
Double longitude;
}

@ -1,9 +1,9 @@
package com.htfp.weather.web.pojo.request;
package com.htfp.weather.web.param.request;
import javax.validation.constraints.Max;
import javax.validation.constraints.Min;
import javax.validation.constraints.NotNull;
import com.htfp.weather.download.gfs.GfsLevelsEnum;
import com.htfp.weather.web.valid.EnumValid;
import lombok.Data;
import lombok.EqualsAndHashCode;
@ -16,5 +16,6 @@ import lombok.EqualsAndHashCode;
@Data
public class Position3D extends Position2D {
@NotNull(message = "高度层次 (level) 不能为空")
@EnumValid(message = "高度层次 (level) 数值错误", enumClass = GfsLevelsEnum.class, usedField = "getCode")
Integer level;
}

@ -1,7 +1,8 @@
package com.htfp.weather.web.pojo.request;
package com.htfp.weather.web.param.request;
import com.htfp.weather.info.Constant;
import com.htfp.weather.download.gfs.GfsVariableIsobaricEnum;
import com.htfp.weather.web.valid.DateTimeStr;
import com.htfp.weather.web.valid.EnumValid;
import lombok.Data;
import javax.validation.constraints.Max;
@ -18,7 +19,7 @@ public class ProfileRequest {
@NotNull(message = "时间不能为空")
@DateTimeStr(message = "时间格式错误")
String time;
@EnumValid(enumClass = GfsVariableIsobaricEnum.class, usedField = "getNameInApi", message = "变量不存在")
String variableName;
@NotNull(message = "纬度 (latitude) 不能为空")

@ -1,4 +1,4 @@
package com.htfp.weather.web.pojo.request;
package com.htfp.weather.web.param.request;
import com.htfp.weather.griddata.common.TableConfig;
import lombok.Data;

@ -0,0 +1,17 @@
package com.htfp.weather.web.param.response;
import com.htfp.weather.download.DownLoadFileInfo;
import lombok.Data;
import java.util.List;
/**
* @Author : shiyi
* @Date : 2024/7/3 11:37
* @Description :
*/
@Data
public class DownloadResult {
List<DownLoadFileInfo> successList;
List<DownLoadFileInfo> failedList;
}

@ -1,4 +1,4 @@
package com.htfp.weather.web.pojo.response;
package com.htfp.weather.web.param.response;
import lombok.Data;

@ -0,0 +1,17 @@
package com.htfp.weather.web.param.response;
import com.htfp.weather.griddata.operation.GfsDataImport.ImportFileInfo;
import lombok.Data;
import java.util.List;
/**
* @Author : shiyi
* @Date : 2024/7/3 11:38
* @Description :
*/
@Data
public class ImportResult {
List<ImportFileInfo> successList;
List<ImportFileInfo> failedList;
}

@ -0,0 +1,18 @@
package com.htfp.weather.web.param.response;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import java.util.List;
@Data @NoArgsConstructor @AllArgsConstructor
public class LevelsAvailableResponse {
List<LevelsAvailable> levelsAvailableList;
@Data @NoArgsConstructor @AllArgsConstructor
public static class LevelsAvailable {
String dataSource;
List<Integer> pressureLevels;
List<Integer> heightLevels;
}
}

@ -1,5 +1,6 @@
package com.htfp.weather.web.pojo.response;
package com.htfp.weather.web.param.response;
import com.fasterxml.jackson.annotation.JsonInclude;
import lombok.Data;
/**
@ -7,7 +8,7 @@ import lombok.Data;
* @Date : 2024/1/24 11:28
* @Description :
*/
@Data
@Data @JsonInclude(JsonInclude.Include.NON_NULL)
public class NowWeatherStatus {
String time; // 数据时间,北京时
String weatherText; // 天气状况
@ -17,9 +18,10 @@ public class NowWeatherStatus {
Float windSpeed; // 风速m/s
Float wind360; // 风向360角度
// String windDir; // 风向, 文字描述,如“西北风”
// String windScale; // 风力等级
Integer windScale; // 风力等级
Float humidity; // 相对湿度 %
// Float pressure; // 大气压强,默认单位:百帕
Float cloud; // 云量
Float precip; //降水量
String dataSource;
}

@ -1,25 +1,22 @@
package com.htfp.weather.web.pojo.response;
package com.htfp.weather.web.param.response;
import com.fasterxml.jackson.annotation.JsonInclude;
import lombok.Data;
import java.util.ArrayList;
/**
* @Author : shiyi
* @Date : 2024/5/8 15:19
* @Description : 线
*/
@Data
@Data @JsonInclude(JsonInclude.Include.NON_NULL)
public class ProfileDataset {
int[] pressureLevels;
// int[ pressureHeight;
int[] heightLevel;
double latitude;
double longitude;
float[] temp;
float[] windSpeed;
float[] wind360;
// List<String> windScale;
float[] humidity;
// float[] pressure;
float[] cloud;
@ -33,7 +30,6 @@ public class ProfileDataset {
temp = new float[size];
windSpeed = new float[size];
wind360 = new float[size];
// windScale = new float[size;
humidity = new float[size];
// pressure = new float[size;
cloud = new float[size];

@ -0,0 +1,18 @@
package com.htfp.weather.web.param.response;
import com.fasterxml.jackson.annotation.JsonInclude;
import lombok.Data;
/**
* @Author : shiyi
* @Date : 2024/5/8 15:19
* @Description : 线
*/
@Data @JsonInclude(JsonInclude.Include.NON_NULL)
public class ProfileResponse {
int[] pressureLevels;
int[] heightLevels;
double latitude;
double longitude;
float[] values;
}

@ -1,4 +1,4 @@
package com.htfp.weather.web.pojo.response;
package com.htfp.weather.web.param.response;
import com.htfp.weather.web.exception.ErrorCode;
import lombok.Data;
@ -37,7 +37,7 @@ public class Result {
return new Result(false, error.getCode(), error.getMsg(), null);
}
public static Result error(ErrorCode error, String msg) {
return new Result(false, error.getCode(), error.getMsg() + msg, null);
return new Result(false, error.getCode(), msg, null);
}
public static Result error(Throwable cause) {

@ -1,5 +1,7 @@
package com.htfp.weather.web.pojo.response;
package com.htfp.weather.web.param.response;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.annotation.JsonInclude;
import lombok.Data;
/**
@ -7,13 +9,15 @@ import lombok.Data;
* @Date : 2024/4/17 15:44
* @Description :
*/
@Data
@Data @JsonInclude(JsonInclude.Include.NON_NULL)
public class SurfaceWeatherWarning {
String pubTime;
// String startTime;
String endTime;
String title;
String text;
String typeName;
String severityColor;
String source;
@JsonIgnore
String dataid;
}

@ -1,8 +1,8 @@
package com.htfp.weather.web.pojo.response;
package com.htfp.weather.web.param.response;
import com.fasterxml.jackson.annotation.JsonInclude;
import lombok.Data;
import java.sql.Time;
import java.util.ArrayList;
import java.util.List;
@ -11,7 +11,7 @@ import java.util.List;
* @Date : 2024/1/24 13:46
* @Description :
*/
@Data
@Data @JsonInclude(JsonInclude.Include.NON_NULL)
public class TimeSeriesDataset {
List<String> time;
Double latitude;
@ -19,19 +19,19 @@ public class TimeSeriesDataset {
float[] temp;
float[] windSpeed;
float[] wind360;
// List<String> windScale;
int[] windScale;
float[] humidity;
// float[] pressure;
float[] cloud;
float[] precip;
String dataSource;
public TimeSeriesDataset() {}
public TimeSeriesDataset(int size) {
time = new ArrayList<>(size);
temp = new float[size];
windSpeed = new float[size];
wind360 = new float[size];
// windScale = new float[size;
windScale = new int[size];
humidity = new float[size];
// pressure = new float[size;
cloud = new float[size];

@ -1,9 +0,0 @@
package com.htfp.weather.web.pojo.response;
/**
* @Author : shiyi
* @Date : 2024/6/6 15:24
* @Description :
*/
public class MultiPointResponse {
}

@ -1,24 +0,0 @@
package com.htfp.weather.web.pojo.response;
import lombok.Data;
import lombok.Getter;
/**
* @Author : shiyi
* @Date : 2024/5/8 15:19
* @Description : 线
*/
@Data
public class ProfileResponse {
int[] pressureLevels;
// int[ pressureHeight;
double latitude;
double longitude;
float[] values;
public ProfileResponse(int[] pressureLevels, float[] values) {
this.pressureLevels = pressureLevels;
// this.pressureHeight = pressureHeight;
this.values = values;
}
}

@ -1,17 +0,0 @@
package com.htfp.weather.web.pojo.response;
import lombok.Data;
import java.util.List;
/**
* @Author : shiyi
* @Date : 2024/1/24 13:38
* @Description :
*/
@Data
public class TimeSeries <T> {
String name;
List<String> time;
List<T> values;
}

@ -2,7 +2,7 @@ package com.htfp.weather.web.service;
import com.htfp.weather.web.exception.AppException;
import com.htfp.weather.web.exception.ErrorCode;
import com.htfp.weather.web.pojo.response.FileInfo;
import com.htfp.weather.web.param.response.FileInfo;
import org.apache.commons.io.FileUtils;
import org.springframework.stereotype.Component;
@ -14,7 +14,7 @@ import java.util.List;
/**
* @Author : shiyi
* @Date : 2024/6/12 11:25
* @Description :
* @Description :
*/
@Component
public class FileService {
@ -57,13 +57,7 @@ public class FileService {
}
return subDirList;
}
public void uploadFile(String filePath){
// 文件上传逻辑
}
public void downloadFile(String filePath){
// 文件下载逻辑
}
public void deleteFile(File file) throws IOException {
if (file == null || !file.exists()) {

@ -4,31 +4,36 @@ import com.aliyun.tablestore.grid.consts.AttributionEnum;
import com.aliyun.tablestore.grid.model.GridDataSet;
import com.aliyun.tablestore.grid.model.GridDataSetMeta;
import com.aliyun.tablestore.grid.model.grid.Grid4D;
import com.htfp.weather.download.DownLoadFileInfo;
import com.htfp.weather.download.gfs.GfsDownloader;
import com.htfp.weather.download.gfs.GfsLevelsEnum;
import com.htfp.weather.download.gfs.GfsVariableHeightEnum;
import com.htfp.weather.download.gfs.GfsVariableIsobaricEnum;
import com.htfp.weather.griddata.common.TableConfig;
import com.htfp.weather.griddata.operation.GfsDataFetcher;
import com.htfp.weather.griddata.common.ValueRange;
import com.htfp.weather.griddata.operation.GfsDataFetcher;
import com.htfp.weather.griddata.operation.GfsDataImport;
import com.htfp.weather.info.Constant;
import com.htfp.weather.download.gfs.GfsLevelsEnum;
import com.htfp.weather.utils.DateTimeUtils;
import com.htfp.weather.utils.MeteoUtils;
import com.htfp.weather.web.exception.AppException;
import com.htfp.weather.web.exception.ErrorCode;
import com.htfp.weather.web.pojo.response.*;
import com.htfp.weather.web.param.response.*;
import com.htfp.weather.web.service.surfaceapi.ISurfaceDataService;
import lombok.extern.slf4j.Slf4j;
import org.jetbrains.annotations.NotNull;
import org.springframework.stereotype.Service;
import org.springframework.util.CollectionUtils;
import ucar.ma2.Array;
import ucar.ma2.Index4D;
import javax.annotation.Resource;
import javax.annotation.security.DenyAll;
import java.lang.reflect.Method;
import java.time.Duration;
import java.time.Instant;
import java.time.OffsetDateTime;
import java.time.ZoneOffset;
import java.time.format.DateTimeFormatter;
import java.util.*;
import java.util.stream.Collectors;
/**
* @Author : shiyi
@ -37,14 +42,86 @@ import java.util.*;
*/
@Service("tablestore-gfs")
@Slf4j
public class GfsDataServiceImpl implements IDataService {
public class GfsDataServiceImpl implements IUpperDataService, ISurfaceDataService {
@Resource
GfsDataFetcher gfsDataFetcher;
@Resource
TableConfig tableConfig;
@Resource
GfsDownloader gfsDownloader;
@Resource
GfsDataImport gfsDataImport;
@Override
public String getDataSource() {
return dataSource;
}
String dataSource = "GFS";
// 地面站数据访问接口
@Override
public NowWeatherStatus getNowSurfaceWeatherStatus(double lat, double lon) throws Exception {
return getNowWeather(GfsLevelsEnum.SURFACE.getCode(), lat, lon);
}
@Override
public TimeSeriesDataset getForecastSeries(double lat, double lon) throws Exception {
return getForecastSeries(GfsLevelsEnum.SURFACE.getCode(), lat, lon);
}
@Override @DenyAll
public List<SurfaceWeatherWarning> getSurfaceWarning(double lat, double lon) throws Exception {
return null;
}
// 下载和导入数据
public DownloadResult downloadNowAllFile() {
gfsDownloader.iniTimeSetting();
List<DownLoadFileInfo> downLoadFileInfoList = gfsDownloader.getFilesInfo();
List<DownLoadFileInfo> finishedList = gfsDownloader.downloadAll(downLoadFileInfoList);
List<DownLoadFileInfo> successList = finishedList.stream().filter(DownLoadFileInfo::isDownloadSuccess).collect(Collectors.toList());
List<DownLoadFileInfo> failedList = finishedList.stream().filter(fileInfo -> !fileInfo.isDownloadSuccess()).collect(Collectors.toList());
DownloadResult downloadResult = new DownloadResult();
downloadResult.setSuccessList(successList);
downloadResult.setFailedList(failedList);
return downloadResult;
}
public DownLoadFileInfo downloadSingleFile(OffsetDateTime offsetDateTime) {
String targetUtcStr = DateTimeUtils.getUTCDateTime(offsetDateTime).format(DateTimeFormatter.ofPattern(Constant.UTC_TIME_STRING));
try {
gfsDownloader.iniTimeSetting();
List<DownLoadFileInfo> downLoadFileInfoList = gfsDownloader.getFilesInfo(); // TODO 2024/6/12: 可以优化不用生成24小时所有的文件信息
for (DownLoadFileInfo downLoadFileInfo : downLoadFileInfoList) {
if (downLoadFileInfo.getForecastUTCTimeStr().equals(targetUtcStr)) {
DownLoadFileInfo downloadResult = gfsDownloader.download(downLoadFileInfo);
if (downloadResult.isDownloadSuccess()) {
return downloadResult;
} else {
throw new AppException(ErrorCode.DOWNLOAD_ERROR, downloadResult.getErrorMsg());
}
}
}
throw new AppException(ErrorCode.QUERY_TIME_ERROR, ": 请选择当前时间之后24小时范围内的时刻");
} catch (Exception e) {
throw new AppException(ErrorCode.DOWNLOAD_START_ERROR, e.getMessage());
}
}
public ImportResult importData(OffsetDateTime time) throws Exception {
List<GfsDataImport.ImportFileInfo> finishedList = gfsDataImport.importData(time);
List<GfsDataImport.ImportFileInfo> successList = finishedList.stream().filter(result -> result.isSuccess()).collect(Collectors.toList());
List<GfsDataImport.ImportFileInfo> failedList = finishedList.stream().filter(result -> !result.isSuccess()).collect(Collectors.toList());
ImportResult importResult = new ImportResult();
importResult.setFailedList(failedList);
importResult.setSuccessList(successList);
return importResult;
}
// 从tablestore获取数据
/**
*
*
@ -53,6 +130,7 @@ public class GfsDataServiceImpl implements IDataService {
* @param level 9999
* @return @{@link NowWeatherStatus}
*/
@Override
public NowWeatherStatus getNowWeather(int level, double latitude, double longitude) {
OffsetDateTime nowTime = OffsetDateTime.now();
ValueRange<OffsetDateTime> timeRange = new ValueRange<>(Collections.singletonList(nowTime));
@ -70,8 +148,10 @@ public class GfsDataServiceImpl implements IDataService {
nowWeatherStatus.setLatitude(latitude);
nowWeatherStatus.setLongitude(longitude);
return nowWeatherStatus;
} catch (Exception e) {
throw new RuntimeException(e);
} catch (AppException e) {
throw e;
}catch (Exception e) {
throw new AppException(ErrorCode.TABLESTORE_QUERY_ERROR, e.getMessage());
}
}
@ -82,6 +162,7 @@ public class GfsDataServiceImpl implements IDataService {
* @param level
* @return
*/
@Override
public List<NowWeatherStatus> getNowWeatherByMultiPoint(int level, List<Double> latitude, List<Double> longitude) {
if (latitude.size() != longitude.size()) {
throw new AppException(ErrorCode.INDEX_SIZE_ERROR, ": 经纬度数组大小不一致");
@ -107,8 +188,10 @@ public class GfsDataServiceImpl implements IDataService {
nowWeatherStatusList.add(nowWeatherStatus);
}
return nowWeatherStatusList;
} catch (Exception e) {
throw new RuntimeException(e);
} catch (AppException e) {
throw e;
}catch (Exception e) {
throw new AppException(ErrorCode.TABLESTORE_QUERY_ERROR, e.getMessage());
}
}
@ -120,6 +203,7 @@ public class GfsDataServiceImpl implements IDataService {
* @param level 9999
* @return @{@link TimeSeriesDataset}
*/
@Override
public TimeSeriesDataset getForecastSeries(int level, double latitude, double longitude) {
List<String> variableNames = getVariableNamesInDatabase(level);
OffsetDateTime nowTime = OffsetDateTime.now();
@ -134,8 +218,10 @@ public class GfsDataServiceImpl implements IDataService {
);
final GfsLevelsEnum levelFlag = GfsLevelsEnum.getByCode(level);
return buildTimeSeriesDatasetInTargetPoint(gridDataSet, level, latitude, longitude, levelFlag);
} catch (Exception e) {
throw new RuntimeException(e);
} catch (AppException e) {
throw e;
}catch (Exception e) {
throw new AppException(ErrorCode.TABLESTORE_QUERY_ERROR, e.getMessage());
}
}
@ -146,7 +232,8 @@ public class GfsDataServiceImpl implements IDataService {
* @param level
* @return
*/
public List<TimeSeriesDataset> getForecastSeriesByMultiPoint(List<Double> latitudeList, List<Double> longitudeList, int level) {
@Override
public List<TimeSeriesDataset> getForecastSeriesByMultiPoint(int level, List<Double> latitudeList, List<Double> longitudeList) {
if (latitudeList.size() != longitudeList.size()) {
throw new AppException(ErrorCode.INDEX_SIZE_ERROR, ": 经纬度数组大小不一致");
}
@ -168,8 +255,10 @@ public class GfsDataServiceImpl implements IDataService {
timeSeriesDatasetList.add(timeSeriesDataset);
}
return timeSeriesDatasetList;
} catch (Exception e) {
throw new RuntimeException(e);
} catch (AppException e) {
throw e;
}catch (Exception e) {
throw new AppException(ErrorCode.TABLESTORE_QUERY_ERROR, e.getMessage());
}
}
@ -188,22 +277,30 @@ public class GfsDataServiceImpl implements IDataService {
throw new AppException(ErrorCode.VALIDATE_ERROR, variableName + "变量在数据库中不存在");
}
try {
int[] pressureLevels = tableConfig.getPressureList();
Array array = gfsDataFetcher.getProfileByVariableAndPressure(targetVariable, targetTime, latitude, longitude);
float[] values = (float[]) array.copyTo1DJavaArray();
ProfileResponse profileResponse = new ProfileResponse(pressureLevels, values);
GridDataSet gridDataSet = gfsDataFetcher.getProfileByPressure(Collections.singletonList(targetVariable), targetTime, latitude, longitude);
float[] values = (float[]) gridDataSet.getVariables().get(targetVariable).toArray().copyTo1DJavaArray();
ProfileResponse profileResponse = new ProfileResponse();
profileResponse.setPressureLevels(tableConfig.getPressureList());
profileResponse.setHeightLevels(tableConfig.getPressureHeightList());
profileResponse.setValues(values);
profileResponse.setLatitude(latitude);
profileResponse.setLongitude(longitude);
return profileResponse;
} catch (Exception e) {
} catch (AppException e) {
throw e;
}catch (Exception e) {
throw new AppException(ErrorCode.TABLESTORE_QUERY_ERROR, e.getMessage());
}
}
/** 获取全部变量随气压变化的廓线 (推荐) */
@Override
public ProfileDataset getProfileByPressure(OffsetDateTime targetTime, double latitude, double longitude) {
try {
List<String> variableNames = getVariableNamesInDatabase(GfsLevelsEnum.UPPER.getCode());
GfsLevelsEnum levelFlag = GfsLevelsEnum.UPPER;
List<String> variableNames = getVariableNamesInDatabase(levelFlag.getCode());
GridDataSet gridDataSet = gfsDataFetcher.getProfileByPressure(variableNames, targetTime, latitude, longitude);
ProfileDataset profile = buildProfileDataset(gridDataSet, GfsLevelsEnum.UPPER);
ProfileDataset profile = buildProfileDataset(gridDataSet, levelFlag);
profile.setLatitude(latitude);
profile.setLongitude(longitude);
return profile;
@ -214,6 +311,48 @@ public class GfsDataServiceImpl implements IDataService {
}
}
/** 获取指定变量随高度变化的廓线, 目前只支持风速风向*/
public ProfileResponse getProfileByVariableAndNearSurfaceHeight(String variableName, OffsetDateTime targetTime, double latitude, double longitude) {
String targetVariable = GfsVariableHeightEnum.getGfsVariableName(variableName);
if (targetVariable == null) {
throw new AppException(ErrorCode.VALIDATE_ERROR, variableName + "变量在数据库中不存在");
}
try {
int[] heightLevels = tableConfig.getHeightList();
GridDataSet gridDataSet = gfsDataFetcher.getProfileByNearSurfaceHeight(Collections.singletonList(targetVariable), targetTime, latitude, longitude);
float[] values = (float[]) gridDataSet.getVariables().get(targetVariable).toArray().copyTo1DJavaArray();
ProfileResponse profileResponse = new ProfileResponse();
profileResponse.setValues(values);
profileResponse.setHeightLevels(Arrays.copyOfRange(heightLevels, 1, heightLevels.length));
profileResponse.setLatitude(latitude);
profileResponse.setLongitude(longitude);
return profileResponse;
} catch (AppException e) {
throw e;
}catch (Exception e) {
throw new AppException(ErrorCode.TABLESTORE_QUERY_ERROR, e.getMessage());
}
}
/** 获取全部变量随高度变化的廓线, 目前只支持风速风向*/
public ProfileDataset getProfileByNearSurfaceHeight(OffsetDateTime targetTime, double latitude, double longitude) {
try {
GfsLevelsEnum levelFlag = GfsLevelsEnum.SURFACE;
List<String> variableNames = new ArrayList<>();
variableNames.add(GfsVariableHeightEnum.getGfsVariableName("windSpeed"));
variableNames.add(GfsVariableHeightEnum.getGfsVariableName("wind360"));
GridDataSet gridDataSet = gfsDataFetcher.getProfileByNearSurfaceHeight(variableNames, targetTime, latitude, longitude);
ProfileDataset profile = buildProfileDataset(gridDataSet, levelFlag);
profile.setHeightLevel(Arrays.copyOfRange(tableConfig.getHeightList(), 1, tableConfig.getHeightList().length));
profile.setLatitude(latitude);
profile.setLongitude(longitude);
return profile;
} catch (AppException e) {
throw e;
} catch (Exception e) {
throw new AppException(ErrorCode.TABLESTORE_QUERY_ERROR, e.getMessage());
}
}
/**
*
@ -227,6 +366,7 @@ public class GfsDataServiceImpl implements IDataService {
* @param maxLon
* @return @{@link PlaneResponse}
*/
@Override
public PlaneResponse getPlane(OffsetDateTime targetTime, String variableName, int level, double minLat, double maxLat, double minLon, double maxLon) {
String targetVariable;
if (GfsLevelsEnum.SURFACE.equals(GfsLevelsEnum.getByCode(level))) {
@ -250,8 +390,10 @@ public class GfsDataServiceImpl implements IDataService {
variableNames, timeRange, levRange, latRange, lonRange
).getVariable(targetVariable).toArray();
return buildPlane(array, minLat, maxLat, minLon, maxLon);
} catch (Exception e) {
throw new RuntimeException(e);
} catch (AppException e) {
throw e;
}catch (Exception e) {
throw new AppException(ErrorCode.TABLESTORE_QUERY_ERROR, e.getMessage());
}
}
@ -325,6 +467,7 @@ public class GfsDataServiceImpl implements IDataService {
Method method = aClass.getDeclaredMethod(setMethodName, aClass.getDeclaredField(variableNameInApi).getType());
// TODO 2024/6/20: 这里的toArray操作在数据量较大的时候会比较吃内存和cpu
float[] storage = (float[]) grid4D.toArray().section(
// levIndexGlobal-origin[1] 获取目标点在子区域中的索引
new int[]{0, levIndexGlobal-origin[1], latIndexGlobal-origin[2], lonIndexGlobal-origin[3]}, new int[]{tSize, 1, 1, 1}
).copyTo1DJavaArray();
method.invoke(result, storage);
@ -340,6 +483,17 @@ public class GfsDataServiceImpl implements IDataService {
List<String> timeList = getResultTimeStringList(gridDataSet, tSize, origin[0]);
result.setTime(timeList);
}
// 计算地面风力等级
if (levelFlag == GfsLevelsEnum.SURFACE) {
float[] windSpeed = result.getWindSpeed();
if (windSpeed != null) {
int[] windScale = new int[windSpeed.length];
for (int i = 0; i < windScale.length; i++) {
windScale[i] = MeteoUtils.mPerSecond2WindScale(windSpeed[i]);
}
result.setWindScale(windScale);
}
}
}
return result;
}
@ -400,6 +554,11 @@ public class GfsDataServiceImpl implements IDataService {
throw new AppException(ErrorCode.RESPONSE_DATA_BUILD_ERROR);
}
}
if (levelFlag == GfsLevelsEnum.SURFACE) {
if (result.getWindSpeed() != null) {
result.setWindScale(MeteoUtils.mPerSecond2WindScale(result.getWindSpeed()));
}
}
return result;
}
@ -427,12 +586,16 @@ public class GfsDataServiceImpl implements IDataService {
throw new AppException(ErrorCode.RESPONSE_DATA_BUILD_ERROR);
}
}
result.setPressureLevels(tableConfig.getPressureList());
if (levelFlag == GfsLevelsEnum.UPPER) {
result.setPressureLevels(tableConfig.getPressureList());
result.setHeightLevel(tableConfig.getPressureHeightList());
} else {
result.setHeightLevel(tableConfig.getHeightList());
}
return result;
}
@NotNull
private static String getVariableNameInApi(GfsLevelsEnum levelFlag, Map.Entry<String, Grid4D> variable) {
private String getVariableNameInApi(GfsLevelsEnum levelFlag, Map.Entry<String, Grid4D> variable) {
String variableNameInFile = variable.getKey();
String variableNameInApi;
if (GfsLevelsEnum.SURFACE.equals(levelFlag)) {
@ -446,41 +609,4 @@ public class GfsDataServiceImpl implements IDataService {
return variableNameInApi;
}
private Map<String, List> getFutureTimeIndexList(GridDataSetMeta lastGridDataSetMeta) {
long milli = (long) lastGridDataSetMeta.getAttributes().get(AttributionEnum.REFERENCE_TIME.getName());
OffsetDateTime refTime = OffsetDateTime.ofInstant(Instant.ofEpochMilli(milli), ZoneOffset.ofHours(0));
OffsetDateTime currentTime = DateTimeUtils.getUTCDateTime(OffsetDateTime.now());
if (currentTime.isBefore(refTime)) {
throw new AppException(ErrorCode.QUERY_TIME_ERROR);
}
int startHour = getForecastHourFromTargetTime(currentTime, refTime);
List<String> forecastHours = lastGridDataSetMeta.getForecastHours();
List<Integer> iTimeList = new ArrayList<>();
List<String> timeList = new ArrayList<>();
for (int i = 0; i < forecastHours.size(); i++) {
int hour = Integer.parseInt(forecastHours.get(i));
if (hour >= startHour) {
iTimeList.add(i);
String timeStr = DateTimeUtils.getLocalZoneDateTime(refTime.plusHours(hour))
.format(Constant.API_TIME_FORMATTER);
timeList.add(timeStr);
}
if (iTimeList.size() >= 24) {
break;
}
}
if (CollectionUtils.isEmpty(iTimeList) || CollectionUtils.isEmpty(timeList)) {
throw new AppException(ErrorCode.QUERY_TIME_ERROR);
}
Map<String, List> map = new HashMap<>(2);
map.put("iTimeList", iTimeList);
map.put("timeList", timeList);
return map;
}
private int getForecastHourFromTargetTime(OffsetDateTime targetTime, OffsetDateTime refTime) {
return (int) Duration.between(refTime, targetTime).toHours();
}
}

@ -1,9 +0,0 @@
package com.htfp.weather.web.service;
/**
* @Author : shiyi
* @Date : 2024/3/15 10:51
* @Description :
*/
public interface IDataService {
}

@ -0,0 +1,28 @@
package com.htfp.weather.web.service;
import com.htfp.weather.web.param.response.NowWeatherStatus;
import com.htfp.weather.web.param.response.PlaneResponse;
import com.htfp.weather.web.param.response.ProfileDataset;
import com.htfp.weather.web.param.response.TimeSeriesDataset;
import java.time.OffsetDateTime;
import java.util.List;
/**
* @Author : shiyi
* @Date : 2024/3/15 10:51
* @Description :
*/
public interface IUpperDataService {
NowWeatherStatus getNowWeather(int level, double latitude, double longitude);
List<NowWeatherStatus> getNowWeatherByMultiPoint(int level, List<Double> latitude, List<Double> longitude);
TimeSeriesDataset getForecastSeries(int level, double latitude, double longitude);
List<TimeSeriesDataset> getForecastSeriesByMultiPoint(int level, List<Double> latitudeList, List<Double> longitudeList);
ProfileDataset getProfileByPressure(OffsetDateTime targetTime, double latitude, double longitude);
PlaneResponse getPlane(OffsetDateTime targetTime, String variableName, int level, double minLat, double maxLat, double minLon, double maxLon);
}

@ -5,14 +5,15 @@ import com.htfp.weather.utils.DateTimeUtils;
import com.htfp.weather.utils.MeteoUtils;
import com.htfp.weather.web.exception.AppException;
import com.htfp.weather.web.exception.ErrorCode;
import com.htfp.weather.web.pojo.caiyun.*;
import com.htfp.weather.web.pojo.caiyun.CaiYunNowResponse.NowResult.RealTime;
import com.htfp.weather.web.pojo.caiyun.CaiYunForecastResponse.HourlyResult.Hourly;
import com.htfp.weather.web.pojo.caiyun.CaiYunWarningResponse.AlertResult.Alert.AlertContent;
import com.htfp.weather.web.pojo.response.NowWeatherStatus;
import com.htfp.weather.web.pojo.response.SurfaceWeatherWarning;
import com.htfp.weather.web.pojo.response.TimeSeriesDataset;
import com.htfp.weather.web.param.caiyun.*;
import com.htfp.weather.web.param.caiyun.CaiYunNowResponse.NowResult.RealTime;
import com.htfp.weather.web.param.caiyun.CaiYunForecastResponse.HourlyResult.Hourly;
import com.htfp.weather.web.param.caiyun.CaiYunWarningResponse.AlertResult.Alert.AlertContent;
import com.htfp.weather.web.param.response.NowWeatherStatus;
import com.htfp.weather.web.param.response.SurfaceWeatherWarning;
import com.htfp.weather.web.param.response.TimeSeriesDataset;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import java.time.OffsetDateTime;
@ -30,12 +31,14 @@ import static com.htfp.weather.utils.HttpClientUtils.sendGet;
*/
@Service("caiyun") @Slf4j
public class CaiYunServiceImpl implements ISurfaceDataService {
private final String key = "Tc9tgOYr5jlPPlEw";
private final HashMap<String, String> variableNameMap =
new HashMap<String, String>() {{
}};
String dataSource = "CaiYun";
@Value("${caiyun.key}")
private String key;
static final String STATUS_OK = "ok";
@Override
public String getDataSource() {
return dataSource;
}
public <T extends CaiYunBaseResponse> T caiYunRequest(String url, HashMap<String, String> params, Class<T> responseClass) {
params.put("units", "metric:v2");
try {
@ -126,6 +129,7 @@ public class CaiYunServiceImpl implements ISurfaceDataService {
nowWeatherStatus.setTemp(realtime.getTemperature());
nowWeatherStatus.setWindSpeed((float) MeteoUtils.kmPerHour2mPerSecond(realtime.getWind().getSpeed()));
nowWeatherStatus.setWind360(realtime.getWind().getDirection());
nowWeatherStatus.setWindScale(MeteoUtils.kmPerHour2WindScale(realtime.getWind().getSpeed()));
nowWeatherStatus.setHumidity(realtime.getHumidity());
nowWeatherStatus.setCloud(realtime.getCloudrate());

@ -0,0 +1,187 @@
package com.htfp.weather.web.service.surfaceapi;
import com.htfp.weather.info.Constant;
import com.htfp.weather.utils.DateTimeUtils;
import com.htfp.weather.utils.JSONUtils;
import com.htfp.weather.web.exception.AppException;
import com.htfp.weather.web.exception.ErrorCode;
import com.htfp.weather.web.param.cma.AntiGeoCodeResponse;
import com.htfp.weather.web.param.cma.AntiGeoCodeResponse.Result.AddressComponent;
import com.htfp.weather.web.param.cma.CmaWarning;
import com.htfp.weather.web.param.response.NowWeatherStatus;
import com.htfp.weather.web.param.response.SurfaceWeatherWarning;
import com.htfp.weather.web.param.response.TimeSeriesDataset;
import lombok.Setter;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
import javax.annotation.PostConstruct;
import java.time.OffsetDateTime;
import java.time.format.DateTimeFormatter;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import static com.htfp.weather.utils.HttpClientUtils.sendGet;
/**
* @Author : shiyi
* @Date : 2024/7/2 14:02
* @Description :
*/
@Slf4j
@Service("cma")
public class CmaServiceImpl implements ISurfaceDataService{
String dataSource = "CMA";
@Value("${tianditu.key}") @Setter
private String keyOfTianDiTu;
private Map<String, List<SurfaceWeatherWarning>> warningCache = new ConcurrentHashMap<>();
private final ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
@Override
public String getDataSource() {
return dataSource;
}
@Override
public NowWeatherStatus getNowSurfaceWeatherStatus(double lat, double lon) throws Exception {
return null;
}
@Override
public TimeSeriesDataset getForecastSeries(double lat, double lon) throws Exception {
return null;
}
@Override
public List<SurfaceWeatherWarning> getSurfaceWarning(double lat, double lon) throws Exception {
String countyId = getCountyIdByLatLon(lat, lon);
return getSurfaceWarningByCountyCode(countyId);
}
public List<SurfaceWeatherWarning> getSurfaceWarningByCountyCode(String countyCode) {
if (warningCache.containsKey(countyCode)) {
return warningCache.get(countyCode);
} else {
return new ArrayList<>();
}
// else {
// // 如果缓存中没有,则先查一遍
// updateCmaWarning(countyCode);
// return warningCache.get(countyCode);
// }
}
/**根据经纬度获取县级行政区代码*/
public String getCountyIdByLatLon(double lat, double lon) throws Exception {
String url = "http://api.tianditu.gov.cn/geocoder";
HashMap<String, String> params = new HashMap<>();
params.put("postStr", String.format("{'lon':%.6f,'lat':%.6f,'ver':1}", lon, lat));
params.put("type", "geocode");
params.put("tk", keyOfTianDiTu);
AntiGeoCodeResponse antiGeoCodeResponse = sendGet(url, params, AntiGeoCodeResponse.class);
if (antiGeoCodeResponse.getResult() == null){
throw new AppException(ErrorCode.TIANDITU_REQUEST_ERROR, "获取行政区划信息失败:" + antiGeoCodeResponse.getMsg());
}
AddressComponent addressComponent = antiGeoCodeResponse.getResult().getAddressComponent();
String countyCode = addressComponent.getCountyCode();
if (StringUtils.isEmpty(countyCode)) {
String info;
info = addressComponent.getInfo();
throw new AppException(ErrorCode.TIANDITU_DATA_PARSE_ERROR, "获取行政区代码失败:" + info);
} else {
return countyCode.substring(countyCode.length()-6);
}
}
/**
* 线
* */
@Scheduled(cron = "0 20,50 * * * ?")
@PostConstruct
public void updateCmaWarning() {
List<CmaWarning> cmaWarningList = null;// 更新全国预警数据
try {
cmaWarningList = getSurfaceWeatherWarnings(null);
} catch (Exception e) {
// 3分钟后重新执行此函数
scheduler.schedule(this::updateCmaWarning, 5, TimeUnit.MINUTES);
log.warn("获取全国预警数据失败3分钟后重新更新");
return;
}
updateCmaWarningCache(cmaWarningList);
}
/**
* @param countyCode */
public List<CmaWarning> getSurfaceWeatherWarnings(String countyCode) {
String url = "https://data.cma.cn/dataGis/disasterWarning/getWarningDataByCnty";
// 6小时前12小时后
OffsetDateTime startTime = DateTimeUtils.offsetDateTimeToSystemZone(OffsetDateTime.now()).minusHours(6);
OffsetDateTime endTime = startTime.plusHours(12);
DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd+HH:mm:ss");
HashMap<String, String> params = new HashMap<>();
params.put("startTime", startTime.format(formatter));
params.put("endTime", endTime.format(formatter));
params.put("provinceCode", countyCode);
log.info("获取预警数据: {}", params);
try {
String jsonStr = sendGet(url, params);
List<CmaWarning> cmaWarningList = JSONUtils.json2list(jsonStr, CmaWarning.class);
return cmaWarningList;
} catch (Exception e) {
log.error("[中央气象台] 预警信息请求结果处理错误, 预警数据更新失败, {}", e.getMessage());
throw new AppException(ErrorCode.CMA_REQUEST_ERROR, e.getMessage());
}
}
/**
*
* @param cmaWarningList
*/
public void updateCmaWarningCache(List<CmaWarning> cmaWarningList) {
log.info("更新预警数据, 区域数量: {}", cmaWarningList.size());
warningCache.clear();
for (CmaWarning cmaWarning : cmaWarningList) {
String countyId = cmaWarning.getAreaId();
String dataid = cmaWarning.getDataid();
SurfaceWeatherWarning surfaceWeatherWarning = buildSurfaceWeatherWarning(cmaWarning);
if (! warningCache.containsKey(countyId)) {
warningCache.put(countyId, new ArrayList<>());
}
List<SurfaceWeatherWarning> countyCacheList = warningCache.get(countyId);
if (countyCacheList.stream().noneMatch(w -> dataid.equals(w.getDataid()))) {
// 该数据未缓存过
warningCache.get(countyId).add(surfaceWeatherWarning);
}
}
}
private SurfaceWeatherWarning buildSurfaceWeatherWarning(CmaWarning cmaWarning) {
SurfaceWeatherWarning surfaceWeatherWarning = new SurfaceWeatherWarning();
surfaceWeatherWarning.setSource(cmaWarning.getSender());
surfaceWeatherWarning.setSeverityColor(cmaWarning.getSignallevelcode().toUpperCase());
surfaceWeatherWarning.setText(cmaWarning.getIssuecontent());
String startTime = cmaWarning.getTime(); // NOTE 2024/7/2: 中央气象台默认使用 GMT+8 北京时
if (StringUtils.isNotEmpty(startTime)) {
surfaceWeatherWarning.setPubTime(OffsetDateTime.parse(startTime + "+08:00", DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ssxxx")).format(Constant.API_TIME_FORMATTER)
);
}
String endTime = cmaWarning.getExpires();
if (StringUtils.isNotEmpty(endTime)) {
surfaceWeatherWarning.setEndTime(OffsetDateTime.parse(endTime + "+08:00", DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ssxxx")).format(Constant.API_TIME_FORMATTER)
);
}
surfaceWeatherWarning.setTypeName(cmaWarning.getCodename());
surfaceWeatherWarning.setTitle(String.format("%s发布:%s%s预警", cmaWarning.getSender(), cmaWarning.getCodename(), cmaWarning.getSignallevel()));
surfaceWeatherWarning.setDataid(cmaWarning.getDataid());
return surfaceWeatherWarning;
}
}

@ -5,13 +5,13 @@ import com.htfp.weather.utils.DateTimeUtils;
import com.htfp.weather.utils.MeteoUtils;
import com.htfp.weather.web.exception.AppException;
import com.htfp.weather.web.exception.ErrorCode;
import com.htfp.weather.web.pojo.response.SurfaceWeatherWarning;
import com.htfp.weather.web.pojo.response.TimeSeriesDataset;
import com.htfp.weather.web.pojo.response.NowWeatherStatus;
import com.htfp.weather.web.pojo.hefeng.*;
import com.htfp.weather.web.pojo.hefeng.HeFengNowResponse.HeFengNow;
import com.htfp.weather.web.pojo.hefeng.HeFengForecastResponse.HeFengForecastHour;
import com.htfp.weather.web.pojo.hefeng.HeFengWarningResponse.HeFengWarning;
import com.htfp.weather.web.param.response.SurfaceWeatherWarning;
import com.htfp.weather.web.param.response.TimeSeriesDataset;
import com.htfp.weather.web.param.response.NowWeatherStatus;
import com.htfp.weather.web.param.hefeng.*;
import com.htfp.weather.web.param.hefeng.HeFengNowResponse.HeFengNow;
import com.htfp.weather.web.param.hefeng.HeFengForecastResponse.HeFengForecastHour;
import com.htfp.weather.web.param.hefeng.HeFengWarningResponse.HeFengWarning;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
@ -34,6 +34,7 @@ import static com.htfp.weather.utils.HttpClientUtils.*;
@Service("hefeng")
@Slf4j
public class HeFengServiceImpl implements ISurfaceDataService {
String dataSource = "HeFeng";
@Value("${hefeng.key}")
private String key;
private final HashMap<String, String> variableNameMap =
@ -49,7 +50,10 @@ public class HeFengServiceImpl implements ISurfaceDataService {
}};
static final String STATUS_OK = "200";
static final String STATUS_NO_DATA = "204";
@Override
public String getDataSource() {
return dataSource;
}
public <T extends HeFengBaseResponse> T heFengRequest(String url, HashMap<String, String> params, Class<T> responseClass) throws Exception {
// String signature = getSignature(params, key);
// HashMap<String, String> params = new HashMap<>();
@ -138,7 +142,9 @@ public class HeFengServiceImpl implements ISurfaceDataService {
for (int i = 0; i < hourly.size(); i++) {
HeFengForecastHour heFengForecastHour = hourly.get(i);
processHeFengData(heFengForecastHour);
timeSeriesDataset.getTime().add(heFengForecastHour.getFxTime());
timeSeriesDataset.getTime().add(
DateTimeUtils.offsetDateTimeToSystemZone(OffsetDateTime.parse(heFengForecastHour.getFxTime())).format(Constant.API_TIME_FORMATTER)
);
timeSeriesDataset.getTemp()[i] = heFengForecastHour.getTemp();
timeSeriesDataset.getWindSpeed()[i] = heFengForecastHour.getWindSpeed();
timeSeriesDataset.getWind360()[i] = heFengForecastHour.getWind360();
@ -178,13 +184,13 @@ public class HeFengServiceImpl implements ISurfaceDataService {
private NowWeatherStatus buildNowSurfaceWeatherStatus(HeFengNow now) {
NowWeatherStatus nowWeatherStatus = new NowWeatherStatus();
processHeFengData(now);
nowWeatherStatus.setTime(now.getObsTime());
nowWeatherStatus.setTime(DateTimeUtils.offsetDateTimeToSystemZone(OffsetDateTime.parse(now.getObsTime())).format(Constant.API_TIME_FORMATTER));
nowWeatherStatus.setWeatherText(now.getText());
nowWeatherStatus.setTemp(now.getTemp());
nowWeatherStatus.setWindSpeed(now.getWindSpeed());
nowWeatherStatus.setWind360(now.getWind360());
// nowSurfaceWeatherStatus.setWindDir(now.getWindDir());
// nowSurfaceWeatherStatus.setWindScale(now.getWindScale());
// nowWeatherStatus.setWindDir(now.getWindDir());
nowWeatherStatus.setWindScale(MeteoUtils.mPerSecond2WindScale(now.getWindSpeed()));
nowWeatherStatus.setHumidity(now.getHumidity());
// nowWeatherStatus.setPressure(now.getPressure());
nowWeatherStatus.setCloud(now.getCloud());

Some files were not shown because too many files have changed in this diff Show More

Loading…
Cancel
Save