AMOS–STG  add-on

ANSARI's Micro Operating System Storage Add-On Module

» Posted on 28. Jun 2013

Related links:  AMOS | Overview | Storage


 

AMOS – Storage Add-On

 

AMOSstg_overview

 

AMOS-STG is an optional add-on module for extending the existing data storage capabilities of AMOS. It supports storing of any system or application data into non-volatile memories like FLASH, EEPROM or SD-card devices. Any needed operations like creating, reading, writing, modifying or deleting of data is supported on byte level access regardless of bank, sector and segment sizes usual in nonvolatile memory devices.

 

Basic Definitions

 
  • AMOS-STG add-on can handle multiple memory devices in a system
  • A memory device like a FLASH or SD-card is organized into one or more Partitions
  • Partitions are organized into Segments
  • Segments are the smallest erasable blocks of a specific memory device
  • Any data for storing is encapsulated into a Storage Unit (SU)
  • Storage units may overlap over segment boundaries
  • Files for storing are created using one or more SUs
  • Data of the first SU of each file is kept in Storage Unit Look-up Table (SULUT)
  • Each partition has its own, and only one SULUT
  • Files can not span over partitions

 

Supported Storage types

 
  • Internal Flash of the µController (direct access within the CPU address space)
  • External Flash devices (accessible via SPI-bus or direct access within the CPU address space)
  • EEPROM (accessible over I²C-bus or SPI-bus)
  • SD-cards (accessible over SPI-bus)

 

Application point of view for storing Data

System or user applications have different focuses for storing data. In general the AMOS-STG add-on distinguishes between data storage types described below and supports each data storing type differently to guarantee best system performance. Each storing type is defined by its needs and characteristics as following: 

LOG Files

When an application logs data, the file won’t be read normally anymore by the local application.

At a point, the stored log data needs to be transmitted to another remote system.

When the logging storage area is full, if enabled, it should be possible to overwrite old records with new records, even those records, which are not transmitted yet.

Log file should be able to act as a FIFO buffer between local application and a remote system to overcome bandwidth and/or connectivity issues.

It is helpful to create a log file with a smaller size, which expands while logged data amount grows.

When log data is transmitted, it should release allocated storage area automatically to maximize the capacity and efficiency of small storage devices.

 

CONSTANT Files

There is no need to update or modify these type of files at all.

Constant files may need huge amount of space with overlapping segment boundaries.

Reading of the data stored must be stream-able. 

Multiple applications may need to read at the same time different locations of same file.

 

DATA Files

The major number of files in a storage will be a big number of different application data files.

Data files may need to be shared between applications at the same time with simultaneous write access.

Data files may get modified frequently and normally block wise, but also byte wise.

Data files may need to grow with new data appended to the existing files.

 

Hardware point of view for storing Data

While the logic of handling stored data is fixed in AMOS, the storage devices can differ from hardware module to hardware module. Drivers standardize the interface between the STG add-on and the various possible storage devices. They handle completely the communication between each individual storage device and the STG add-on. Each hardware storage driver is then presenting  the same standardized interface to the STG add-on for reading, writing, erasing and getting status and device ID information.

 

AMOS Flash File System (AFFS)

AFFS is a special developed file system for Flash devices. It guarantees optimum usage of Flash segments and considering write cycles to whole flash, prohibiting frequent write cycles to specific locations of a Flash device.

Following figure illustrates an example how files are organized in a partition followed by details of data structures used by AMOS-STG add-on to create the AFFS:

 

AMOSstg_file_system

  

Storage Unit Structure

Saved data in a storage is always encapsulated in a SU-frame. A SU-frame is built by a header (orange) and a payload area (light blue). AMOS-STG add-on manages the storage with the data stored into the SU headers. The SU-header is transparent to the application.

SU_structure

A file in a storage is built up by one or several SUs in a chain. There is no limit for the number of SUs building a chain or a file. The first byte of a SU-heater defines the type of SU followed by a CRC value for specific areas of the header. The interpretation of the data within a header is based on the type of the SU and the validity checking using the CRC value stored. The size of one SU can vary  from 16 Bytes up to 64 kBytes. New data can easily appended to a file by creating a new SU with needed size and append the new SU to an existing chain of a file. 

 

Modified Storage Unit Structure

When a file needs to be modified, a SU-header can be redefined using the MSU structure below, which is a type 0x02 SU. The header differs from the type 0x01 and allows to reorganize the existing data within original SU in combination with the new data stored into the MSU.

MSU_structure

If a part of a file needs to be modified and the file itself is defined by one or several SUs, the header of the SU, which contains the obsolete data will be set as obsolete by writing a valid pointer into the vSUH field. The vSUH points now to a valid MSU containing only the new replacement data after its header. In the header of the MSU, pointers will define how the valid data is now compiled together by pointing to start and end points of valid data within the original SU and and this new MSU data area (mSUd[ ]). This action can be repeated recursive as long as there is free space in the partition.

 

Storage Unit Look-up Table

Each partition contains a Storage Unit Look-up Table (SULUT). In this table all starting SUs of any existing files in the partition is stored, with related file and storage information. SULUT helps AMOS-STG to find files faster and run storage de-fragmentation algorithms independently. The information in SULUT allows the storage add-on to fill a FLASH devices fully before any segment is erased to grantee a homogeneous usage of Flash areas resulting in better life-time for a Flash device. A SULUT is a type 0x03 SU with different header interpretation.

A physical to virtual segment translation table follows the header to reorganize fragmented free segments in the storage into a single continuous block of free space. From storage add-on write logic point of view, data is written into a partition sequentially until the partition area gets full. When files written before in the storage are removed, some storage areas gets free and fragmented free spaces are generated within the partition. Once a partition is written completely, the released segments are reorganized into a single virtual free block of storage area, accepting new sequential data to be written into the storage. This logic guarantees a homogeneous usage of the whole storage area, which is essential to Flash devices.

SULUT_structure

 

 

Implementation of STG add-on

When AMOS starts and the STG add-on is enabled, the storage service needs to:

 
  • Enumerate all storage devices in the hardware 
  • Enumerate all partitions and their SULUT locations in each partition
  • Create a related data structure in RAM for each SULUT
  • Check data fragmentation and initiate reorganization algorithms as needed
  • Publishing readiness of the service to the system and waiting for IPC packets

The initial data and hardware configuration needed by the add-on to start initialization is stored into the data storage and is structured as shown below:

STG_Data_Storage_structure

 

Submit a Comment