javaScript实现客户端直连华为云OBS实现文件上传、断点续传、断网重传-Toy模板网

这篇具有很好参考价值的文章主要介绍了javaScript实现客户端直连华为云OBS实现文件上传、断点续传、断网重传。希望对大家有所帮助。如果存在错误或未考虑完全的地方，请大家不吝赐教，您也可以点击"举报违法"按钮提交疑问。

写在前面：在做这个调研时我遇到的需求是前端直接对接华为云平台实现文件上传功能。上传视频文件通常十几个G、客户工作环境网络较差KB/s，且保证上传是稳定的，支持网络异常断点重试、文件断开支持二次拖入自动重传等。综合考虑使用的华为云的分段上传功能，基于分段的特性在应用层面上实现断点、断网重传功能。

主要参考华为云上传官方文档

文件上传_对象存储服务 OBS_BrowserJS_上传对象

同时我的另一篇博客介绍了使用海外AWS云平台上传文件的示例

javaScript实现客户端直连AWS S3（亚马逊云）文件上传、断点续传、断网重传-CSDN博客

最开始我使用的是华为云提供的文件直传功能uploadFile，通过在后端保留断点信息，用户二次上传同一个文件时通过从后端获取断点信息UploadCheckpoint进行续传。但是后期总会遇到uploadId失效，跟华为云方进行沟通无果，继而作罢，转而用分段上传，分段上传会创建唯一的uploadId，二次上传时用同一个uploadId并将未上传的分片上传即可。

公司最开始使用的是七牛云上传，七牛云可以根据同个key进行自动续传，而华为云或亚马逊云只能开发从应用层面自己实现自动续传，实现的原理大同小异，云平台会对分段上传文件进行一段时间保存，如果文件在有效期内上传完成即可完成续传。二次上传时将已成功上传的分段过滤即可

上传整体流程

首次上传

initiateMultipartUpload初始化分段上传获取uploadId

uploadPart 上传段

completeMultipartUpload 合并段

二次上传

getObjectMetadata获取上传完成的文件

listMultipartUploads获取上传信息，返回最近一次上传的uploadId

listParts 根据uploadId获取已上传的分片信息

uploadPart 上传未完成的段

completeMultipartUpload 合并段

安装华为云OBS

npm install esdk-obs-browserjs

使用OBS

import ObsClient from "esdk-obs-browserjs/src/obs";

获取秘钥

几乎所有的云平台管理都需要获取秘钥，秘钥通常调用后端接口获取

  const secret = await getHwSecret();
  //创建OBS对象
  const hwClient = new ObsClient({
    access_key_id: secret.ak,
    secret_access_key: secret.sk,
    server: secret.endPoint,
    timeout: 3000,
  });

Bucket、Key、Prefix作用

在上传时的配置参数，Bucket可以理解为文件夹名称，用于区分文件存放的路径；Key可以理解为文件名称，区分唯一的文件，Prefix可以理解为查找文件的匹配项，通常可以与key相同。

  const params = {
    Bucket: secret.bucketName,
    Key: key,
    Prefix: key,
  };

初始化分段上传

function initMultiPartUpload(hwClient: any, params: any) {
  return new Promise((resolve, reject) => {
    hwClient.initiateMultipartUpload(params, (err: any, result: any) => {
      if (err) {
        reject(err);
      } else if (result.CommonMsg.Status < 300 && result.InterfaceResult) {
        const uploadId = result.InterfaceResult.UploadId;
        resolve(uploadId);
      }
    });
  });
}

分段上传示例

单个分片上传，并获取上传完成的ETag信息，用于手动合并分片时的校验。

//单段上传，成功上传时返回ETag值
function uploadNextPart(
  n: number,
  hwClient: any,
  uploadId: any,
  file: any,
  params: any
) {
  const count = Math.ceil(file.size / PartSize);
  const lastPartSize = file.size % PartSize;
  return new Promise((resolve, reject) => {
    hwClient.uploadPart(
      {
        ...params,
        PartNumber: n,
        UploadId: uploadId,
        SourceFile: file,
        PartSize: count === n ? lastPartSize : PartSize,
        Offset: (n - 1) * PartSize,
      },
      (err: any, result: any) => {
        if (err) {
          reject(err);
        } else {
          if (result.CommonMsg.Status < 300 && result.InterfaceResult) {
            resolve(result.InterfaceResult.ETag);
          }
        }
      }
    );
  });
}

分片上传，parts不为空时实现续传功能

这里提供了一种分段上传续传逻辑，parts在后面会介绍怎么获取的，它是同一个文件已经上传的分片信息。这里的逻辑是过滤已上传分片，通过PartNubmer来过滤。通过Promise.all方法确保所有分片都执行完，进行手动合并

//分片上传，parts不为空时实现续传功能
async function uploadPart(
  fileState: FileState,
  file: File,
  uploadId: any,
  parts: any,
  hwClient: any,
  params: any
) {
  const completeParts = [...parts];
  const partNumbers = parts?.map((_: any) => _.PartNumber) || [];
  const count = Math.ceil(file.size / PartSize);
  if (partNumbers.length) {
    fileState.status = FileStatus.processing;
    fileState.percent = parseInt((completeParts.length * 100) / count);
  }
  let startTime = null as any;
  const uploadPromises = []; // 存储所有分片上传的 Promise 对象
  for (let n = 1; n <= count; n++) {
    if (!partNumbers.includes(n)) {
      if (!startTime) {
        startTime = new Date();
      }
      const promise = uploadNextPart(n, hwClient, uploadId, file, params)
        .then((data: any) => {
          completeParts.push({
            PartNumber: n,
            ETag: data,
          });
          //按分片数量计算上传速度
          const currentTime = new Date();
          const elapsedTime =
            (currentTime.getTime() - startTime.getTime()) / 1000;
          startTime = currentTime;
          //上传速度MB/s
          fileState.uploadSpeed = (10 / elapsedTime).toFixed(2);
          fileState.percent = parseInt((completeParts.length * 100) / count);
        })
        .catch((err: any) => {
          fileState.status = FileStatus.fail;
          throw err;
        });
      uploadPromises.push(promise);
    }
  }
  //所有promise上传结束，调用分片合并校验接口
  Promise.all(uploadPromises)
    .then(async () => {
      checkMultiUpload(uploadId, completeParts, fileState, hwClient, params);
    })
    .catch(() => {
      fileState.status = FileStatus.fail;
    });
}

合并分片校验

华为云对合并分片的校验规则是PartNumber必须是按序排列的，因此需要对分片进行排序

//校验分片是否全部上传成功
function completeMultiUpload(
  uploadId: any,
  parts: any,
  fileState: any,
  hwClient: any,
  params: any
) {
  const newParts = parts.sort((a: any, b: any) => a.PartNumber - b.PartNumber);
  hwClient.completeMultipartUpload(
    {
      ...params,
      UploadId: uploadId,
      Parts: newParts,
    },
    function (err: any, result: any) {
      if (err) {
        fileState.status = FileStatus.fail;
      } else {
        if (result.CommonMsg.Status < 300 && result.InterfaceResult) {
          fileState.status = FileStatus.success;
          fileState.percent = 100;
        }
      }
    }
  );
}

检测文件是否上传完成

如果需要断点续传，可以先使用listObjects方法，校验文件是否上传完成。传入首次上传时key信息，查找的关键字Prefix信息

  const params = {
    Bucket: secret.bucketName,
    Key: key,
    Prefix: key,
  };

function checkIsCompleted(hwClient: any, params: any): Promise<any> {
  return new Promise(async (resolve, reject) => {
    try {
      await hwClient.listObjects(params, (err: any, result: any) => {
        if (err) {
          reject(err);
        } else {
          if (
            result.CommonMsg.Status < 300 &&
            result.InterfaceResult.Contents.length
          ) {
            resolve(true);
          } else {
            resolve(false);
          }
        }
      });
    } catch (err: any) {
      reject(err);
    }
  });
}

获取上传上传的uploadId

如果文件没有上传完成，通过listMultipartUploads方法校验文件是否部分上传，并获取最近上传的uploadId

async function getHwCheckpoint(hwClient: any, params: any) {
  return new Promise((resolve, reject) => {
    hwClient.listMultipartUploads(params, (err: any, result: any) => {
      if (err) {
        reject(err);
      } else if (
        result.CommonMsg.Status < 300 &&
        result.InterfaceResult &&
        result.InterfaceResult.Uploads.length
      ) {
        const uploads = result.InterfaceResult.Uploads;
        resolve(uploads[uploads.length - 1].UploadId);
      } else {
        resolve("");
      }
    });
  });
}

获取已上传分片信息

根据返回的最近一次上传的uploadId信息，使用listParts方法获取详细的分片信息。当分片数量过大时，使用递归方式获取详细的分片信息。

//根据uploadId获取已上传分片信息
async function listAllUploadParts(
  uploadId: any,
  hwClient: any,
  params: any
): Promise<any> {
  let completeParts = [] as any;
  const listAll = async function (partNumberMarker = null) {
    return new Promise((resolve, reject) => {
      hwClient.listParts(
        {
          ...params,
          UploadId: uploadId,
          PartNumberMarker: partNumberMarker,
        },
        (err: any, result: any) => {
          if (err) {
            reject(err);
          } else {
            if (result.CommonMsg.Status < 300 && result.InterfaceResult) {
              const parts = result.InterfaceResult.Parts.map((_: any) => {
                return {
                  PartNumber: parseInt(_.PartNumber),
                  ETag: _.ETag,
                };
              });
              completeParts = [...completeParts, ...parts];
              if (result.InterfaceResult.IsTruncated === "true") {
                resolve(listAll(result.InterfaceResult.NextPartNumberMarker));
              } else {
                resolve(completeParts);
              }
            } else {
              reject(new Error("Failed to list parts"));
            }
          }
        }
      );
    });
  };
  await listAll();
  return completeParts;
}

主体完整代码示例

const PartSize = 5 * 1024 * 1024; 

async function hwRequest(
  fileState: FileState,
  file: File,
  key: string,
  mode: UploadMode
) {
  const secret = await getHwSecret();
  fileState.url = `${secret.domain}${key}`;
  //创建OBS对象
  const hwClient = new ObsClient({
    access_key_id: secret.ak,
    secret_access_key: secret.sk,
    server: secret.endPoint,
    timeout: 3000,
  });
  const params = {
    Bucket: secret.bucketName,
    Key: key,
    Prefix: key,
  };
  const isCompleted = await checkIsCompleted(hwClient, params);
  if (isCompleted) {
    //已全部上传
    fileState.percent = 100;
    fileState.status = FileStatus.success;
  } else {
      //检查是否部分上传
      const uploadId = await getHwCheckpoint(hwClient, params);
      if (!uploadId) {
        //首次上传
        const uploadId = await initMultiPartUpload(hwClient, params);
        const completeParts = [] as any;
        uploadPart(fileState, file, uploadId, completeParts, hwClient, params);
      } else {
        //二次上传续传
        const completeParts = await listAllUploadParts(
          uploadId,
          hwClient,
          params
        );
        uploadPart(fileState, file, uploadId, completeParts, hwClient, params);
    }
  }
}