org.mapdb
Class StoreDirect

java.lang.Object
  org.mapdb.Store
      org.mapdb.StoreDirect

All Implemented Interfaces:

Engine

Direct Known Subclasses:

StoreWAL

public class StoreDirect
extends Store

Storage Engine which saves record directly into file. It has zero protection from data corruption and must be closed properly after modifications. It is used when Write-Ahead-Log transactions are disabled. Storage format ---------------- `StoreDirect` is composed of two files: Index file is sequence of 8-byte longs, it translates `recid` (offset in index file) to record size and offset in physical file. Records position may change, but it requires stable ID, so the index file is used for translation. This store uses data structure called `Long Stack` to manage (and reuse) free space, it is is linked LIFO queue of 8-byte longs. Index file -------------- Index file is translation table between permanent record ID (recid) and mutable location in physical file. Index file is sequence of 8-byte longs, one for each record. It also has some extra longs to manage free space and other metainfo. Index table and physical data could be stored in single file, but keeping index table separate simplifies compaction. Basic **structure of index file** is bellow. Each slot is 8-bytes long so `offset=slot*8` slot | in code | description --- | --- | --- 0 | HEADER | File header, format version and flags 1 | IO_INDEX_SIZE | Allocated file size of index file in bytes. 2 | IO_PHYS_SIZE | Allocated file size of physical file in bytes. 3 | IO_FREE_SIZE | Space occupied by free records in physical file in bytes. 4 | IO_INDEX_SUM | Checksum of all Index file headers. Checks if store was closed correctly 5..9 | | Reserved for future use 10..14 | | For usage by user 15 | IO_FREE_RECID |Long Stack of deleted recids, those will be reused and returned by Engine.put(Object, Serializer) 16..4111 | |Long Stack of free physical records. This contains free space released by record update or delete. Each slots corresponds to free record size. TODO check 4111 is right 4112 | IO_USER_START |Record size and offset in physical file for recid=1 4113 | |Record size and offset in physical file for recid=2 ... | ... |... snip ... N+4111 | |Record size and offset in physical file for recid=N Long Stack ------------ Long Stack is data structure used to store free records. It is LIFO queue which uses linked records to store 8-byte longs. Long Stack is identified by slot in Index File, which stores pointer to Long Stack head. The structure of of index pointer is following: byte | description --- |--- 0..1 | relative offset in head Long Stack Record to take value from. This value decreases by 8 each take 2..7 | physical file offset of head Long Stack Record, zero if Long Stack is empty Each Long Stack Record is sequence of 8-byte longs, first slot is header. Long Stack Record structure is following: byte | description --- |--- 0..1 | length of current Long Stack Record in bytes 2..7 | physical file offset of next Long Stack Record, zero of this record is last 8-15 | Long Stack value 16-23 | Long Stack value ... | and so on until end of Long Stack Record Physical pointer ---------------- Index slot value typically contains physical pointer (information about record location and size in physical file). First 2 bytes are record size (max 65536). Then there is 6 byte offset in physical file (max store size is 281 TB). Physical file offset must always be multiple of 16, so last 4 bites are used to flag extra record information. Structure of **physical pointer**: bite | in code | description --- | --- | --- 0-15 |`val>>>48` | record size 16-59 |`val&MASK_OFFSET` | physical offset 60 |`val&MASK_LINKED!=0` | linked record flag 61 |`val&MASK_DISCARD!=0` | to be discarded while storage is offline flag 62 |`val&MASK_ARCHIVE!=0` | record modified since last backup flag 63 | | not used yet Records in Physical File --------------------------- Records are stored in physical file. Maximal record size size is 64KB, so larger records must be stored in form of the linked list. Each record starts by Physical Pointer from Index File. There is flag in Physical Pointer indicating if record is linked. If record is not linked you may just read ByteBuffer from given size and offset. If record is linked, each record starts with Physical Pointer to next record. So actual data payload is record size-8. The last linked record does not have the Physical Pointer header to next record, there is MASK_LINKED flag which indicates if next record is the last one.

Author:

Jan Kotek

Field Summary

static String	DATA_FILE_EXT
protected boolean	deleteFilesAfterClose
protected long	freeSize
protected static int	HEADER 4 byte file header
protected Volume	index
protected long	indexSize
protected static int	IO_FREE_RECID index file offset where reference to longstack of free recid is stored
protected static int	IO_FREE_SIZE index file offset where space occupied by free phys records is stored
protected static int	IO_INDEX_SIZE index file offset where current size of index file is stored
protected static int	IO_INDEX_SUM checksum of all index file headers.
protected static int	IO_PHYS_SIZE index file offset where current size of phys file is stored
protected static int	IO_USER_START index file offset where first recid available to user is stored
protected static int	LONG_STACK_PREF_COUNT
protected static int	LONG_STACK_PREF_COUNT_ALTER
protected static long	LONG_STACK_PREF_SIZE
protected static long	LONG_STACK_PREF_SIZE_ALTER
protected static long	MASK_ARCHIVE
protected static long	MASK_DISCARD
protected static long	MASK_LINKED
protected static long	MASK_OFFSET
protected static int	MAX_REC_SIZE maximal non linked record size
protected long	maxUsedIoList maximal non zero slot in free phys record, access requires `structuralLock`
protected Volume	phys
protected static int	PHYS_FREE_SLOTS_COUNT number of free physical slots
protected long	physSize
protected boolean	readOnly
protected long	sizeLimit
protected boolean	spaceReclaimReuse
protected boolean	spaceReclaimSplit
protected boolean	spaceReclaimTrack
protected static short	STORE_VERSION 2 byte store version
protected boolean	syncOnCommitDisabled

Fields inherited from class org.mapdb.Store

checksum, CHECKSUM_FLAG_MASK, CHUNK_SIZE, CHUNK_SIZE_MOD_MASK, compress, COMPRESS_FLAG_MASK, encrypt, ENCRYPT_FLAG_MASK, encryptionXTEA, locks, LOG, LZF, newRecidLock, password, recycledDataOuts, serializerPojo, serializerPojoInitLock, structuralLock

Fields inherited from interface org.mapdb.Engine

CATALOG_RECID, CHECK_RECORD, CLASS_INFO_RECID, LAST_RESERVED_RECID

Constructor Summary

StoreDirect(Volume.Factory volFac)

StoreDirect(Volume.Factory volFac, boolean readOnly, boolean deleteFilesAfterClose, int spaceReclaimMode, boolean syncOnCommitDisabled, long sizeLimit, boolean checksum, boolean compress, byte[] password, boolean disableLocks, int sizeIncrement)

Method Summary

String	calculateStatistics() get some statistics about store.
boolean	canRollback()
protected void	checkHeaders()
void	clearCache() clears any underlying cache
void	close() Close store/cache.
void	commit() Makes all changes made since the previous commit/rollback permanent.
void	compact()
protected void	compactPostUnderLock() subclasses put additional cleanup after compaction finishes here
protected void	compactPreUnderLock() subclasses put additional checks before compaction starts here
<A> boolean	compareAndSwap(long recid, A expectedOldValue, A newValue, Serializer<A> serializer) Updates existing record in atomic (Compare And Swap) manner.
protected long	countLongStackItems(long ioList)
protected void	createStructure()
<A> void	delete(long recid, Serializer<A> serializer) Remove existing record from store/cache Recid must be a number returned by 'put' method.
protected void	freeIoRecidPut(long ioRecid)
protected long	freeIoRecidTake(boolean ensureAvail)
protected void	freePhysPut(long indexVal, boolean recursive)
protected long	freePhysTake(int size, boolean ensureAvail, boolean recursive)
<A> A	get(long recid, Serializer<A> serializer) Get existing record.
protected <A> A	get2(long ioRecid, Serializer<A> serializer)
long	getCurrSize() returns current size occupied by physical store (does not include index).
Iterator<Long>	getFreeRecids()
long	getFreeSize() returns free size in physical store (does not include index).
protected long[]	getLinkedRecordsIndexVals(long indexVal)
long	getMaxRecid()
ByteBuffer	getRaw(long recid)
long	getSizeLimit() returns maximal store size or `0` if there is no limit
protected long	indexHeaderChecksum()
boolean	isClosed() Checks whether Engine was closed.
boolean	isReadOnly() Check if you can write into this Engine.
protected void	longStackPut(long ioList, long offset, boolean recursive)
protected long	longStackTake(long ioList, boolean recursive)
protected long[]	physAllocate(int size, boolean ensureAvail, boolean recursive)
long	preallocate() Preallocates recid for not yet created record.
void	preallocate(long[] recids) Preallocates recids for not yet created record.
<A> long	put(A value, Serializer<A> serializer) Insert new record.
protected void	put2(DataOutput2 out, long ioRecid, long[] indexVals)
void	rollback() Undoes all changes made in the current transaction.
protected static long	roundTo16(long offset)
protected static long	size2ListIoRecid(long size)
<A> void	update(long recid, A value, Serializer<A> serializer) Update existing record with new value.
protected void	update2(DataOutput2 out, long ioRecid)
void	updateRaw(long recid, ByteBuffer data)

Methods inherited from class org.mapdb.Store

canSnapshot, closeListenerRegister, closeListenerUnregister, deserialize, expectedMasks, forDB, forEngine, getSerializerPojo, lockAllWrite, lockPos, newDataOut2, printStatistics, serialize, snapshot, unlockAllWrite

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MASK_OFFSET

protected static final long MASK_OFFSET

See Also:

Constant Field Values

MASK_LINKED

protected static final long MASK_LINKED

See Also:

Constant Field Values

MASK_DISCARD

protected static final long MASK_DISCARD

See Also:

Constant Field Values

MASK_ARCHIVE

protected static final long MASK_ARCHIVE

See Also:

Constant Field Values

HEADER

protected static final int HEADER

4 byte file header

See Also:

Constant Field Values

STORE_VERSION

protected static final short STORE_VERSION

2 byte store version

See Also:

Constant Field Values

MAX_REC_SIZE

protected static final int MAX_REC_SIZE

maximal non linked record size

See Also:

Constant Field Values

PHYS_FREE_SLOTS_COUNT

protected static final int PHYS_FREE_SLOTS_COUNT

number of free physical slots

See Also:

Constant Field Values

IO_INDEX_SIZE

protected static final int IO_INDEX_SIZE

index file offset where current size of index file is stored

See Also:

Constant Field Values

IO_PHYS_SIZE

protected static final int IO_PHYS_SIZE

index file offset where current size of phys file is stored

See Also:

Constant Field Values

IO_FREE_SIZE

protected static final int IO_FREE_SIZE

index file offset where space occupied by free phys records is stored

See Also:

Constant Field Values

IO_INDEX_SUM

protected static final int IO_INDEX_SUM

checksum of all index file headers. Used to verify store was closed correctly

See Also:

Constant Field Values

IO_FREE_RECID

protected static final int IO_FREE_RECID

index file offset where reference to longstack of free recid is stored

See Also:

Constant Field Values

IO_USER_START

protected static final int IO_USER_START

index file offset where first recid available to user is stored

See Also:

Constant Field Values

DATA_FILE_EXT

public static final String DATA_FILE_EXT

See Also:

Constant Field Values

LONG_STACK_PREF_COUNT

protected static final int LONG_STACK_PREF_COUNT

See Also:

Constant Field Values

LONG_STACK_PREF_SIZE

protected static final long LONG_STACK_PREF_SIZE

See Also:

Constant Field Values

LONG_STACK_PREF_COUNT_ALTER

protected static final int LONG_STACK_PREF_COUNT_ALTER

See Also:

Constant Field Values

LONG_STACK_PREF_SIZE_ALTER

protected static final long LONG_STACK_PREF_SIZE_ALTER

See Also:

Constant Field Values

index

protected Volume index

phys

protected Volume phys

physSize

protected long physSize

indexSize

protected long indexSize

freeSize

protected long freeSize

deleteFilesAfterClose

protected final boolean deleteFilesAfterClose

readOnly

protected final boolean readOnly

syncOnCommitDisabled

protected final boolean syncOnCommitDisabled

spaceReclaimReuse

protected final boolean spaceReclaimReuse

spaceReclaimSplit

protected final boolean spaceReclaimSplit

spaceReclaimTrack

protected final boolean spaceReclaimTrack

sizeLimit

protected final long sizeLimit

maxUsedIoList

protected long maxUsedIoList

maximal non zero slot in free phys record, access requires `structuralLock`

Constructor Detail

StoreDirect

public StoreDirect(Volume.Factory volFac,
                   boolean readOnly,
                   boolean deleteFilesAfterClose,
                   int spaceReclaimMode,
                   boolean syncOnCommitDisabled,
                   long sizeLimit,
                   boolean checksum,
                   boolean compress,
                   byte[] password,
                   boolean disableLocks,
                   int sizeIncrement)

StoreDirect

public StoreDirect(Volume.Factory volFac)

Method Detail

checkHeaders

protected void checkHeaders()

createStructure

protected void createStructure()

indexHeaderChecksum

protected long indexHeaderChecksum()

preallocate

public long preallocate()

Description copied from interface: Engine

Preallocates recid for not yet created record. It does not insert any data into it.

Returns:

new recid

preallocate

public void preallocate(long[] recids)

Description copied from interface: Engine

Preallocates recids for not yet created record. It does not insert any data into it. This is done in batch of given size (determied by size of array in argument)

Parameters:

recids - array to put result into

put

public <A> long put(A value,
                    Serializer<A> serializer)

Description copied from interface: Engine

Insert new record.

Parameters:

value - records to be added

serializer - used to convert record into/from binary form

Returns:

recid (record identifier) under which record is stored.

put2

protected void put2(DataOutput2 out,
                    long ioRecid,
                    long[] indexVals)

get

public <A> A get(long recid,
                 Serializer<A> serializer)

Description copied from interface: Engine

Get existing record.

Recid must be a number returned by 'put' method. Behaviour for invalid recid (random number or already deleted record) is not defined, typically it returns null or throws 'EndOfFileException'

Parameters:

recid - (record identifier) under which record was persisted

serializer - used to deserialize record from binary form

Returns:

record matching given recid, or null if record is not found under given recid.

get2

protected <A> A get2(long ioRecid,
                     Serializer<A> serializer)
          throws IOException

Throws:

IOException

update

public <A> void update(long recid,
                       A value,
                       Serializer<A> serializer)

Description copied from interface: Engine

Update existing record with new value.

Recid must be a number returned by 'put' method. Behaviour for invalid recid (random number or already deleted record) is not defined, typically it throws 'EndOfFileException', but it may also corrupt store.

Parameters:

recid - (record identifier) under which record was persisted.

value - new record value to be stored

serializer - used to serialize record into binary form

update2

protected void update2(DataOutput2 out,
                       long ioRecid)

compareAndSwap

public <A> boolean compareAndSwap(long recid,
                                  A expectedOldValue,
                                  A newValue,
                                  Serializer<A> serializer)

Description copied from interface: Engine

Updates existing record in atomic (Compare And Swap) manner. Value is modified only if old value matches expected value. There are three ways to match values, MapDB may use any of them:

1. Equality check oldValue==expectedOldValue when old value is found in instance cache
2. Deserializing oldValue using serializer and checking oldValue.equals(expectedOldValue)
3. Serializing expectedOldValue using serializer and comparing binary array with already serialized oldValue

Recid must be a number returned by 'put' method. Behaviour for invalid recid (random number or already deleted record) is not defined, typically it throws 'EndOfFileException', but it may also corrupt store.

Parameters:

recid - (record identifier) under which record was persisted.

expectedOldValue - old value to be compared with existing record

newValue - to be written if values are matching

serializer - used to serialize record into binary form

Returns:

true if values matched and newValue was written

delete

public <A> void delete(long recid,
                       Serializer<A> serializer)

Description copied from interface: Engine

Remove existing record from store/cache

Recid must be a number returned by 'put' method. Behaviour for invalid recid (random number or already deleted record) is not defined, typically it throws 'EndOfFileException', but it may also corrupt store.

Parameters:

recid - (record identifier) under which was record persisted

serializer - which may be used in some circumstances to deserialize and store old object

getLinkedRecordsIndexVals

protected long[] getLinkedRecordsIndexVals(long indexVal)

physAllocate

protected long[] physAllocate(int size,
                              boolean ensureAvail,
                              boolean recursive)

roundTo16

protected static long roundTo16(long offset)

close

public void close()

Description copied from interface: Engine

Close store/cache. This method must be called before JVM exits to flush all caches and prevent store corruption. Also it releases resources used by MapDB (disk, memory..).

Engine can no longer be used after this method was called. If Engine is used after closing, it may throw any exception including NullPointerException

There is an configuration option DBMaker.closeOnJvmShutdown() which uses shutdown hook to automatically close Engine when JVM shutdowns.

isClosed

public boolean isClosed()

Description copied from interface: Engine

Checks whether Engine was closed.

Returns:

true if engine was closed

commit

public void commit()

Description copied from interface: Engine

Makes all changes made since the previous commit/rollback permanent. In transactional mode (on by default) it means creating journal file and replaying it to storage. In other modes it may flush disk caches or do nothing at all (check your config options)

rollback

public void rollback()
              throws UnsupportedOperationException

Description copied from interface: Engine

Undoes all changes made in the current transaction. If transactions are disabled it throws UnsupportedOperationException.

Throws:

UnsupportedOperationException - if transactions are disabled

isReadOnly

public boolean isReadOnly()

Description copied from interface: Engine

Check if you can write into this Engine. It may be readonly in some cases (snapshot, read-only files).

Returns:

true if engine is read-only

canRollback

public boolean canRollback()

Returns:

true if engine supports rollback

clearCache

public void clearCache()

Description copied from interface: Engine

clears any underlying cache

compact

public void compact()

compactPreUnderLock

protected void compactPreUnderLock()

subclasses put additional checks before compaction starts here

compactPostUnderLock

protected void compactPostUnderLock()

subclasses put additional cleanup after compaction finishes here

longStackTake

protected long longStackTake(long ioList,
                             boolean recursive)

longStackPut

protected void longStackPut(long ioList,
                            long offset,
                            boolean recursive)

freeIoRecidPut

protected void freeIoRecidPut(long ioRecid)

freeIoRecidTake

protected long freeIoRecidTake(boolean ensureAvail)

size2ListIoRecid

protected static long size2ListIoRecid(long size)

freePhysPut

protected void freePhysPut(long indexVal,
                           boolean recursive)

freePhysTake

protected long freePhysTake(int size,
                            boolean ensureAvail,
                            boolean recursive)

getMaxRecid

public long getMaxRecid()

Specified by:

getMaxRecid in class Store

getRaw

public ByteBuffer getRaw(long recid)

Specified by:

getRaw in class Store

getFreeRecids

public Iterator<Long> getFreeRecids()

Specified by:

getFreeRecids in class Store

updateRaw

public void updateRaw(long recid,
                      ByteBuffer data)

Specified by:

updateRaw in class Store

getSizeLimit

public long getSizeLimit()

Description copied from class: Store

returns maximal store size or `0` if there is no limit

Specified by:

getSizeLimit in class Store

getCurrSize

public long getCurrSize()

Description copied from class: Store

returns current size occupied by physical store (does not include index). It means file allocated by physical file

Specified by:

getCurrSize in class Store

getFreeSize

public long getFreeSize()

Description copied from class: Store

returns free size in physical store (does not include index).

Specified by:

getFreeSize in class Store

calculateStatistics

public String calculateStatistics()

Description copied from class: Store

get some statistics about store. This may require traversing entire store, so it can take some time.

Specified by:

calculateStatistics in class Store

countLongStackItems

protected long countLongStackItems(long ioList)