fstream

README

FileStream: Event-Driven File Upload Engine

FileStream is an open-source project, which is aim to gain more experience in golang, the purpose of this project is to build robust file upload systems that are efficient, resumable, and ready for real-time integrations.

Purpose of this project

This project was born from a desire to learn Go deeply while building something practical and useful. The aim is to:

• Enhance skills in Go by working on a real-world project.
• Build a reusable, efficient engine for file uploads.
• Attract contributions from developers passionate about open-source collaboration.

Roadmap

Here are some exciting features in the pipeline:

• Chunked uploads - Upload large files in smaller, manageable chunks.
• WebSocket Notifications - Real time updates for uploading process.
• Event driven - Publish events via NATS or other event system.
• Storage - In starting level it's good to support Minio for example.
• Customizable options - Customizable options is like remove metadata or using virus scan.
• Resumable upload - Resume interrupted uploads without starting over.

How to contribute

Here’s how you can get involved:

1. Report issues: If you find any bug or issue, please open a issue
2. Fork and code: Check out the open issues and submit a pull request.

Project setup

1. Clone the repository:

   git clone https://github.com/Nicolas-ggd/filestream
   

Usage

Basic File Upload Example:

import(
    "github.com/gin-gonic/gin"
    "github.com/Nicolas-ggd/filestream"
)

func Upload(c *gin.Context) {
   // Retrieve the uploaded file
   uploadFile, err := c.FormFile("file")
   if err != nil {
     c.JSON(http.StatusBadRequest, gin.H{"error": "failed to get file"})
     return
   }

   // Open the uploaded file
   file, err := uploadFile.Open()
   if err != nil {
     fmt.Println("Error opening uploaded file:", err)
     c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to open uploaded file"})
     return
   }
   
   defer file.Close()


   // Construct the FileRequest
   fileReq := fstream.RFileRequest{
       File:            file,
       UploadFile:      uploadFile,
       MaxRange:        rangeMax,
       FileSize:        fileSize,
       UploadDirectory: "uploads/",
       FileUniqueName:  true, // remember that if you set true here, you will receive unique name file
   }


   // Call StoreChunk to handle the uploaded chunk
   prFile, err := fstream.StoreChunk(&fileReq)
   if err != nil {
      c.JSON(http.StatusBadRequest, gin.H{"error": "failed to store chunk"})
      return
   }
   
   // You can write your own if statement to check whatever you want
   if isLast {
    // You can perform your own logic here, before return 200 status, 
    // it's better to remove chunks which is uploaded and doesn't use anymore
     err = fstream.RemoveUploadedFile("/dir", "filename.jpeg")
     if err != nil {
        c.JSON(http.StatusBadRequest, gin.H{"error": err})	 
     }
   }

   c.JSON(http.StatusOK, gin.H{"message": "file chunk processed"})
}

Validate File Extensions:

   import(
     "github.com/Nicolas-ggd/filestream"
   )

    // Declare extension slice, slice contains all file extension which is ok for you to allow in your system
    var allowExtensions = []string{".jpg", ".jpeg", ".png", ".gif", ".bmp", ".webp"}

    // This function is very simple, but save your time to compare extension and current file to known if it's valid for you
    allowed := fstream.IsAllowExtension(allowExtensions, "filename.png")
	
    if allowed {
     // your logic goes here...	
    }

Remove EXIF Metadata:

   import(
     "github.com/Nicolas-ggd/filestream"
   )

   // filePath is the upload direction + filename
   err := fstream.RemoveExifMetadata(filePath)
   if err != nil {
     log.Fatalln(err)	
   }

License

FileStream is open-source software licensed under the MIT License.

Documentation

Index

• func IsAllowExtension(fileExtensions []string, fileName string) bool
• func RemoveExifMetadata(inputPath string) error
• func RemoveUploadedFile(uploadDir, fileName string) error
• type File
  • func StoreChunk(r *RFileRequest) (*File, error)
• type RFileRequest

Functions

func IsAllowExtension

func IsAllowExtension(fileExtensions []string, fileName string) bool

IsAllowExtension checks if a given file's extension is allowed based on a provided list of acceptable extensions.

func RemoveExifMetadata

func RemoveExifMetadata(inputPath string) error

RemoveExifMetadata returns error if something went wrong during the exif metadata removal process, functions takes inputPath which is location of the image. purpose of this function is that to open and re-encode image without metadata

func RemoveUploadedFile

func RemoveUploadedFile(uploadDir, fileName string) error

RemoveUploadedFile removes uploaded file from uploaded directory and returns error if something went wrong, it takes upload directory and fileName. Use this function in your handler after file is uploaded

Types

type File

type File struct {
	// Original uploaded file name
	FileName string
	// FileUniqueName is unique name
	FileUniqueName string
	// Uploaded file path
	FilePath string
	// Uploaded file extension
	FileExtension string
	// Uploaded file size
	FileSize string
}

File struct is final face of uploaded file, it includes necessary field to use them after file is uploaded

func StoreChunk

func StoreChunk(r *RFileRequest) (*File, error)

StoreChunk cares slice of chunks and returns final results and error. Functions creates new directory for chunks if it doesn't exist, if directory already exists it appends received chunks in current chunks and if entire file is uploaded then File struct is returned

type RFileRequest

type RFileRequest struct {
	// File is an interface to access the file part of a multipart message.
	File multipart.File
	// A FileHeader describes a file part of a multipart request.
	UploadFile *multipart.FileHeader
	// Maximum range of chunk uploads
	MaxRange int
	// Uploaded file size
	FileSize int
	// Upload directory
	UploadDirectory string
	// FileUniqueName is identifier to generate unique name for files
	FileUniqueName bool
}

RFileRequest struct is used for http request, use this struct to bind uploaded file