org.mapdb

Class Pump

java.lang.Object

org.mapdb.Pump

public final class Pump
extends Object

Data Pump moves data from one source to other. It can be used to import data from text file, or copy store from memory to disk.

Constructor Summary

Constructors

Constructor and Description
Pump()

Method Summary

Methods

Modifier and Type	Method and Description
static void	archiveTreeMap(Iterator<Fun.Pair> source, String file, Volume.VolumeFactory factory, DB.BTreeMapMaker config)
static void	archiveTreeMap(NavigableMap source, File target, DB.BTreeMapMaker config)
protected static void	arraySort(Object[] array, int arrayLen, Comparator comparator, Executor executor)
static void	backupFull(DB db, OutputStream out)
static DB	backupFullRestore(DBMaker.Maker maker, InputStream in)
static void	backupIncremental(DB db, File backupDir)
static DB	backupIncrementalRestore(DBMaker.Maker maker, File backupDir)
static <E,K,V> long	buildTreeMap(Iterator<E> source, Engine engine, Fun.Function1<K,E> keyExtractor, Fun.Function1<V,E> valueExtractor, boolean ignoreDuplicates, int nodeSize, boolean valuesStoredOutsideNodes, long counterRecid, BTreeKeySerializer keySerializer, Serializer<V> valueSerializer, Executor executor) Build BTreeMap (or TreeSet) from presorted data.
static void	copy(DB src, DB target) Deprecated. not yet implemented
static <K,V,A> void	fillHTreeMap(HTreeMap<K,V> m, Iterator<A> pumpSource, Fun.Function1<K,A> pumpKeyExtractor, Fun.Function1<V,A> pumpValueExtractor, int pumpPresortBatchSize, boolean pumpIgnoreDuplicates, Serializer<A> sortSerializer, Executor executor)
static <E> Iterator<E>	merge(Executor executor, Iterator... iters) Merges multiple iterators into single iterator.
protected static String	nameWithoutExt(File f)
static <E> Iterator<E>	poisonPillIterator(BlockingQueue<E> q, Object poisonPill)
static <E> Iterator<E>	sort(Comparator<E> comparator, boolean mergeDuplicates, Iterator<E>... iterators) Merge presorted iterators into single sorted iterator.
static <E> Iterator<E>	sort(Iterator<E> source, boolean mergeDuplicates, int batchSize, Comparator comparator, Serializer<E> serializer, Executor executor) Sorts large data set by given Comparator.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Pump

public Pump()

Method Detail

sort

public static <E> Iterator<E> sort(Iterator<E> source,
                   boolean mergeDuplicates,
                   int batchSize,
                   Comparator comparator,
                   Serializer<E> serializer,
                   Executor executor)

Sorts large data set by given Comparator. Data are sorted with in-memory cache and temporary files.

Parameters:

source - iterator over unsorted data

mergeDuplicates - should be duplicate keys merged into single one?

batchSize - how much items can fit into heap memory

comparator - used to sort data

serializer - used to store data in temporary files

Returns:

iterator over sorted data set

arraySort

protected static void arraySort(Object[] array,
             int arrayLen,
             Comparator comparator,
             Executor executor)

sort

public static <E> Iterator<E> sort(Comparator<E> comparator,
                   boolean mergeDuplicates,
                   Iterator<E>... iterators)

Merge presorted iterators into single sorted iterator.

Parameters:

comparator - used to compare data

mergeDuplicates - if duplicate keys should be merged into single one

iterators - array of already sorted iterators

Returns:

sorted iterator

merge

public static <E> Iterator<E> merge(Executor executor,
                    Iterator... iters)

Merges multiple iterators into single iterator. Result iterator will return entries from all iterators. It does not do sorting or any other special functionality. Does not allow null elements.

Parameters:

iters - - iterators to be merged

Returns:

union of all iterators.

poisonPillIterator

public static <E> Iterator<E> poisonPillIterator(BlockingQueue<E> q,
                                 Object poisonPill)

buildTreeMap

public static <E,K,V> long buildTreeMap(Iterator<E> source,
                        Engine engine,
                        Fun.Function1<K,E> keyExtractor,
                        Fun.Function1<V,E> valueExtractor,
                        boolean ignoreDuplicates,
                        int nodeSize,
                        boolean valuesStoredOutsideNodes,
                        long counterRecid,
                        BTreeKeySerializer keySerializer,
                        Serializer<V> valueSerializer,
                        Executor executor)

Build BTreeMap (or TreeSet) from presorted data. This method is much faster than usual import using Map.put(key,value) method. It is because tree integrity does not have to be maintained and tree can be created in linear way with. This method expect data to be presorted in **reverse order** (highest to lowest). There are technical reason for this requirement. To sort unordered data use sort(java.util.Iterator, boolean, int, java.util.Comparator, Serializer, Executor) This method does not call commit. You should disable Write Ahead Log when this method is used DBMaker.Maker.transactionDisable()

Parameters:

source - iterator over source data, must be reverse sorted

keyExtractor - transforms items from source iterator into keys. If null source items will be used directly as keys.

valueExtractor - transforms items from source iterator into values. If null BTreeMap will be constructed without values (as Set)

ignoreDuplicates - should be duplicate keys merged into single one?

nodeSize - maximal BTree node size before it is splited.

valuesStoredOutsideNodes - if true values will not be stored as part of BTree nodes

counterRecid - TODO make size counter friendly to use

keySerializer - serializer for keys, use null for default value

valueSerializer - serializer for value, use null for default value

Throws:

DBException.PumpSourceNotSorted - if source iterator is not reverse sorted

DBException.PumpSourceDuplicate - if source iterator has duplicates

fillHTreeMap

public static <K,V,A> void fillHTreeMap(HTreeMap<K,V> m,
                        Iterator<A> pumpSource,
                        Fun.Function1<K,A> pumpKeyExtractor,
                        Fun.Function1<V,A> pumpValueExtractor,
                        int pumpPresortBatchSize,
                        boolean pumpIgnoreDuplicates,
                        Serializer<A> sortSerializer,
                        Executor executor)

copy

public static void copy(DB src,
        DB target)

Deprecated. not yet implemented

backupFull

public static void backupFull(DB db,
              OutputStream out)

backupFullRestore

public static DB backupFullRestore(DBMaker.Maker maker,
                   InputStream in)

backupIncremental

public static void backupIncremental(DB db,
                     File backupDir)

backupIncrementalRestore

public static DB backupIncrementalRestore(DBMaker.Maker maker,
                          File backupDir)

nameWithoutExt

protected static String nameWithoutExt(File f)

archiveTreeMap

public static void archiveTreeMap(NavigableMap source,
                  File target,
                  DB.BTreeMapMaker config)

archiveTreeMap

public static void archiveTreeMap(Iterator<Fun.Pair> source,
                  String file,
                  Volume.VolumeFactory factory,
                  DB.BTreeMapMaker config)