How to create an external blob provider

•  
•  
•  
•  

• sensenet.com
• Forum
• Blog
• [1]
• [2]

• 100%
• 6.5.5
• Enterprise
• Community
• Planned

Overview

This article provides an example implementation of the IBlobProvider interface. This particular implementation (LocalDiskChunkBlobProvider) saves binary data to files of predefined size inside a folder, each file containing one chunk. Please note that an optimal chunk size has not been tested for and the optimal size may very well depend on the particular system and its resources. The underlying IBlobProvider interface and in-depth description as to the purpose of the methods are explained in detail in the article Blob provider. As a context is saved for every binary stream in the Files table, it is very well possible to set an individual chunk size for every stream. So, one could come up with varying chunk sizes for different size ranges if deemed necessary. Consequently, it is also possible to adjust the chunk size any time after deployment.

This particular implementation is not an enterprise level solution and comes as-is with no warranty whatsoever.

The 10-byte chunk size is merely for testing purposes, it is highly recommended to have it changed to at least several magnitudes higher (e.g., 64000 for ~64kB or 1000000 for ~1MB).

Example

using SenseNet.ContentRepository.Storage.Data.SqlClient.Blob;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.IO;
using SenseNet.Diagnostics;
 
namespace SenseNet.ContentRepository.Tests.Data
{
    class LocalDiskChunkBlobProvider : IBlobProvider
    {
        private static StringBuilder _trace = new StringBuilder();
        public static string Trace { get { return _trace.ToString(); } }
 
 
        private static int _chunkByteSize = 10; //ten bytes by default
        public static int ChunkByteSize { get;set; }
 
        internal class LocalDiskChunkBlobProviderData
        {
            public Guid Id { get; set; }
            public string Trace { get; set; }
            public int ChunkSize { get; set; }
 
        }
 
        private static string _rootDirectory;
        public LocalDiskChunkBlobProvider()
        {
            _rootDirectory = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "App_Data\\" + GetType().Name);
            if (Directory.Exists(_rootDirectory))
            {
                foreach (var path in Directory.GetFiles(_rootDirectory))
                    System.IO.File.Delete(path);
            }
            else
            {
                Directory.CreateDirectory(_rootDirectory);
            }
        }
 
        public static void _ClearTrace()
        {
            _trace.Clear();
        }
 
        /*========================================================================== provider implementation */
 
        public object ParseData(string providerData)
        {
            return BlobStorageContext.DeserializeBlobProviderData<LocalDiskChunkBlobProviderData>(providerData);
        }
 
        public void Allocate(BlobStorageContext context)
        {
            //Create a guaranteed unique identifier that will
            //be used as the name of the new folder to store the chunks
            var id = Guid.NewGuid();
 
            //Pass the created id to the method that creates the actual
            //folder for the chunks
            CreateFolder(id);
 
            //Initialize the BlobProviderData with the unique id and the
            //set chunk size
            context.BlobProviderData = new LocalDiskChunkBlobProviderData { Id = id, ChunkSize = _chunkByteSize };
 
            //Optionally, make a note of this operation in the log
            SnTrace.Test.Write("LocalDiskChunkBlobProvider.Allocate: " + id);
        }
 
        public void Delete(BlobStorageContext context)
        {
            //By casting the BlobProviderData with the right type, 
            //we can acquire the id that is needed for the delete operation
            var id = ((LocalDiskChunkBlobProviderData)context.BlobProviderData).Id;
 
            //Call the delete method and pass the relevant id to remove all
            //unnecessary consituents of the previously stored content
            DeleteFolder(id);
 
            //Optionally, make a note of this operation in the log
            SnTrace.Test.Write("LocalDiskChunkBlobProvider.Delete: " + id);
        }
 
        public Stream GetStreamForRead(BlobStorageContext context)
        {
            //By casting the BlobProviderData, arriving with the context,
            //with the right type, we can create the required input
            //parameter for the following method call
            var providerData = (LocalDiskChunkBlobProviderData)context.BlobProviderData;
 
            //Retrieve and return the read-only stream assembled from
            //all chunks. Necessary parameters can be extracted from the context.
            return new FileSystemChunkReaderStream(providerData, context.Length, GetDirectoryPath(providerData.Id));
        }
 
        public Stream GetStreamForWrite(BlobStorageContext context)
        {
            //By casting the BlobProviderData, arriving with the context,
            //with the right type, we can create the required input
            //parameter for the following method call
            var providerData = (LocalDiskChunkBlobProviderData)context.BlobProviderData;
 
            //Retrieve and return the writable stream assembled from all
            //chunks. Necessary parameters can be extracted from the context.
            return new FileSystemChunkWriterStream(providerData, context.Length, GetDirectoryPath(providerData.Id));
        }
 
        public Stream CloneStream(BlobStorageContext context, Stream stream)
        {
            //Optionally, make a note of this operation in the log
            SnTrace.Test.Write("LocalDiskChunkBlobProvider.CloneStream: {0}", context.FileId);
 
            //Examine the incoming stream against the required type.
            //If it is not a match then throw an exception
            if (!(stream is FileSystemChunkReaderStream))
                throw new InvalidOperationException("Stream must be a FileSystemChunkReaderStream in the local disk provider.");
 
            //By taking advantage of already existing logic,
            //call GetStreamForRead with the incoming context
            return GetStreamForRead(context);
        }
 
        public void Write(BlobStorageContext context, long offset, byte[] buffer)
        {
            //By casting the BlobProviderData, arriving with the
            //context, with the right type, we can create the
            //providerdata and use its fields and properties
            //to retrieve necessary data
            var providerData = (LocalDiskChunkBlobProviderData)context.BlobProviderData;
 
            //Set the appropriate id so chunks will be written
            //in the right folder
            var originalFileId = providerData.Id;
 
            //Set appropriate chunk size as, inside a particular
            //folder, chunks must have the same length
            var originalChunkSize = providerData.ChunkSize;
 
            //Determine the length of the incoming binary
            var currentBlobSize = context.Length;
 
            //Check if the incoming offset (the point of start of write)
            //and the length of the binary data to be written are
            //consistent with settings (e.g., one may not start writing
            //content mid-chunk)
            AssertValidChunks(currentBlobSize, originalChunkSize, offset, buffer.Length);
 
            //The following piece of code takes apart the incoming 
            //byte array according to the set chunk size and writes
            //every chunk into a separate file
            var length = buffer.Length;
            var sourceOffset = 0;
            while (length > 0)
            {
                var chunkIndex = Convert.ToInt32( (offset / originalChunkSize) );
                var currentChunkLength = Math.Min(originalChunkSize, length);
                var bytes = new byte[currentChunkLength];
                Array.ConstrainedCopy(buffer, sourceOffset, bytes, 0, currentChunkLength);
 
                WriteChunk(((LocalDiskChunkBlobProviderData)context.BlobProviderData).Id, chunkIndex, bytes);
 
                length -= bytes.Length;
                offset += originalChunkSize;
                sourceOffset += originalChunkSize;
            }
 
            //Optionally, make a note of this operation in the log
            SnTrace.Test.Write("LocalDiskChunkBlobProvider.Write");
        }
 
        /*================================================================================== utility methods */
 
        private static string GetDirectoryPath(Guid id)
        {
            //Concatenate the root folder name with the incoming
            //id in order to return a complete path
            return Path.Combine(_rootDirectory, id.ToString());
        }
 
        private static string GetFilePath(Guid id, int chunkIndex)
        {
            //Concatenate the complete path with the incoming
            //chunkIndex in order to return the full path of the chunk file
            return Path.Combine(GetDirectoryPath(id), chunkIndex.ToString());
        }
 
        public static void WriteChunk(Guid id, int chunkIndex, byte[] bytes)
        {
            //Use existing logic to write chunk
            using (var stream = new FileStream(GetFilePath(id, chunkIndex), FileMode.OpenOrCreate))
                stream.Write(bytes, 0, bytes.Length);
        }
 
        private void CreateFolder(Guid id)
        {
            //Create target folder for the new content using the
            //incoming id (that is based on a guaranteed unique
            //identifier, or GUID)
            Directory.CreateDirectory(GetDirectoryPath(id));
        }
 
        private void AssertValidChunks(long currentBlobSize, int chunkSize, long offset, int size)
        {
            //Make sure there is no remainder when dividing
            //the offset (start of write) with the set chunk
            //size. If there is, the operation is illegal,
            //as write may not occur mid-chunk
            if (offset % chunkSize > 0)
                throw new Exception("Invalid offset");
        }
 
        private void DeleteFolder(Guid id)
        {
            //Retrieve the full directory path using the incoming id
            var myPath = GetDirectoryPath(id);
 
            //Only do the operation if the folder actually exists
            if (System.IO.Directory.Exists(myPath))
            {
                System.IO.DirectoryInfo dirinfo = new DirectoryInfo(myPath);
 
                //List and then remove all files under the folder to be
                //deleted before attempting to remove the folder itself
                foreach (FileInfo file in dirinfo.GetFiles())
                {
                    file.Delete();
                }
 
                //Finally, remove the folder
                foreach (DirectoryInfo dir in dirinfo.GetDirectories())
                {
                    dir.Delete(true);
                }
            }
        }
 
 
    }
}

Reader stream example

This class is a simple example related to the file system blob provider above.

using System;
using System.IO;
 
namespace SenseNet.ContentRepository.Tests.Data
{
    internal class FileSystemChunkReaderStream : Stream
    {
        private readonly string _directoryPath;
 
        private readonly int _chunkSize;
        private int _currentChunkIndex;
 
        private int _loadedChunkIndex = -1;
        private byte[] _loadedBytes;
 
        public FileSystemChunkReaderStream(LocalDiskChunkBlobProvider.LocalDiskChunkBlobProviderData providerData, long fullSize, string directoryPath)
        {
            _directoryPath = directoryPath;
            Length = fullSize;
            _chunkSize = providerData.ChunkSize;
        }
 
        public override bool CanRead => true;
        public override bool CanSeek => true;
        public override bool CanWrite => false;
        public override long Length { get; }
 
        private long __position;
        public override long Position
        {
            get { return __position; }
            set
            {
                _currentChunkIndex = (value / _chunkSize).ToInt();
                __position = value;
            }
        }
 
        public override void Flush()
        {
            throw new NotSupportedException();
        }
 
        public override int Read(byte[] buffer, int offset, int count)
        {
            count = Math.Min((Length - Position), count).ToInt();
 
            var totalCount = 0;
            while (count > 0)
            {
                var chunkOffset = (long)_chunkSize * _currentChunkIndex;
 
                if (_currentChunkIndex != _loadedChunkIndex)
                {
                    _loadedBytes = LoadChunk(_currentChunkIndex);
                    if (_loadedBytes == null)
                        throw new ApplicationException($"Chunk not found. ChunkIndex:{_currentChunkIndex}");
 
                    _loadedChunkIndex = _currentChunkIndex;
                }
                var copiedCount = CopyBytes(_loadedBytes, (Position - chunkOffset).ToInt(), buffer, offset, count);
 
                Position += copiedCount;
                offset += copiedCount;
                count -= copiedCount;
                totalCount += copiedCount;
                if (Position >= Length)
                    break;
            }
            return totalCount;
        }
 
        private byte[] LoadChunk(int chunkIndex)
        {
            var path = Path.Combine(_directoryPath, chunkIndex.ToString());
 
            byte[] bytes;
 
            using (var stream = new FileStream(path, FileMode.Open))
            {
                var streamLength = stream.Length.ToInt();
                bytes = new byte[streamLength];
                stream.Read(bytes, 0, streamLength);
            }
            return bytes;
        }
 
        private static int CopyBytes(byte[] source, int sourceOffset, byte[] target, int targetOffset, int expectedCount)
        {
            var availableSourceCount = source.Length - sourceOffset;
            var availableTargetCount = target.Length - targetOffset;
            var availableCount = Math.Min(Math.Min(availableSourceCount, availableTargetCount), expectedCount);
 
            Array.ConstrainedCopy(source, sourceOffset, target, targetOffset, availableCount);
 
            return availableCount;
        }
 
        public override long Seek(long offset, SeekOrigin origin)
        {
            var position = Position;
            switch (origin)
            {
                case SeekOrigin.Begin:
                    position = offset;
                    break;
                case SeekOrigin.Current:
                    position += offset;
                    break;
                case SeekOrigin.End:
                    position = Length - offset;
                    break;
            }
            if (position < 0 || position > Length - 1)
                throw new ApplicationException(String.Format("Invalid offset. Expected max:{0}, requested:{1}", Length, offset));
            Position = position;
            return position;
        }
 
        public override void SetLength(long value)
        {
            throw new NotSupportedException();
        }
 
        public override void Write(byte[] buffer, int offset, int count)
        {
            throw new NotSupportedException();
        }
    }
}

Writer stream example

This class is a simple example related to the file system blob provider above.

using System;
using System.IO;
 
namespace SenseNet.ContentRepository.Tests.Data
{
    internal class FileSystemChunkWriterStream : Stream
    {
        private readonly Guid _id;
        private readonly int _chunkSize;
 
        private int _currentChunkIndex;
        private int _currentChunkPosition;
        private byte[] _buffer;
        private bool _flushIsNecessary;
 
        public FileSystemChunkWriterStream(LocalDiskChunkBlobProvider.LocalDiskChunkBlobProviderData providerData, long fullSize)
        {
            Length = fullSize;
            _chunkSize = providerData.ChunkSize;
            _id = providerData.Id;
        }
 
        public override bool CanRead => false;
        public override bool CanSeek => false;
        public override bool CanWrite => true;
 
        public override long Length { get; }
 
        private long __position;
        public override long Position
        {
            get { return __position; }
            set
            {
                // set Position value only through the private SetPosition method
                throw new NotSupportedException();
            }
        }
        private void SetPosition(long position)
        {
            _currentChunkIndex = (position / _chunkSize).ToInt();
            _currentChunkPosition = (position % _chunkSize).ToInt();
            __position = position;
        }
 
        public override void Flush()
        {
            // nothing to write to the db
            if (!_flushIsNecessary)
                return;
 
            var bytesToWrite = _currentChunkPosition;
            var chunkIndex = _currentChunkIndex;
 
            // If the current chunk position is 0, that means we are at the beginning of the
            // next chunk, so we have to write all bytes (chunk size) from the buffer using
            // the previous chunk index.
            if (_currentChunkPosition == 0)
            {
                bytesToWrite = _chunkSize;
                chunkIndex = _currentChunkIndex - 1;
            }
 
            byte[] bytes;
            if (bytesToWrite == _buffer.Length)
            {
                bytes = _buffer;
            }
            else
            {
                bytes = new byte[bytesToWrite];
                Array.ConstrainedCopy(_buffer, 0, bytes, 0, bytesToWrite);
            }
 
            LocalDiskChunkBlobProvider.WriteChunk(_id, chunkIndex, bytes);
 
            _flushIsNecessary = false;
        }
 
        public override int Read(byte[] buffer, int offset, int count)
        {
            throw new NotSupportedException();
        }
 
        public override long Seek(long offset, SeekOrigin origin)
        {
            throw new NotSupportedException();
        }
 
        public override void SetLength(long value)
        {
            throw new NotSupportedException();
        }
 
        public override void Write(byte[] buffer, int offset, int count)
        {
            if (buffer == null)
                throw new ArgumentNullException(nameof(buffer));
            if (offset < 0 || count < 0 || buffer.Length < offset + count)
                throw new ArgumentException(string.Format("Invalid write parameters: buffer length {0}, offset {1}, count {2}.",
                    buffer.Length, offset, count));
 
            // nothing to write
            if (count == 0)
                return;
 
            if (Position >= Length)
                throw new InvalidOperationException("Stream length exceeded.");
 
            // Initialize buffer here and not in the constructor 
            // to allocate memory only when it is needed.
            if (_buffer == null)
                _buffer = new byte[_chunkSize];
 
            var bytesToWrite = count;
            while (bytesToWrite > 0)
            {
                // if the inner buffer is already full, write it to the db
                if (_currentChunkPosition >= _chunkSize || _currentChunkPosition == 0 && _flushIsNecessary)
                {
                    Flush();
 
                    if (_currentChunkPosition >= _chunkSize)
                    {
                        // reset inner buffer position and move to the next chunk index
                        _currentChunkPosition = 0;
                        _currentChunkIndex++;
                    }
                }
 
                // we can only write so much bytes in one round as many slots are left in the inner buffer
                var maxBytesToWrite = Math.Min(bytesToWrite, _chunkSize - _currentChunkPosition);
 
                Array.ConstrainedCopy(buffer, offset, _buffer, _currentChunkPosition, maxBytesToWrite);
 
                bytesToWrite -= maxBytesToWrite;
                offset += maxBytesToWrite;
                _currentChunkPosition += maxBytesToWrite;
                _flushIsNecessary = true;
            }
 
            SetPosition(Position + count);
        }
 
        protected override void Dispose(bool disposing)
        {
            try
            {
                this.Flush();
            }
            finally
            {
                base.Dispose(disposing);
            }
        }
    }
}

Related links

• Blob provider
• Content Repository
• Database structure

References

There are no external references for this article.