Class SnappyCompressorOutputStream
- java.lang.Object
-
- java.io.OutputStream
-
- org.apache.commons.compress.compressors.CompressorOutputStream
-
- org.apache.commons.compress.compressors.snappy.SnappyCompressorOutputStream
-
- All Implemented Interfaces:
java.io.Closeable
,java.io.Flushable
,java.lang.AutoCloseable
public class SnappyCompressorOutputStream extends CompressorOutputStream
CompressorOutputStream for the raw Snappy format.This implementation uses an internal buffer in order to handle the back-references that are at the heart of the LZ77 algorithm. The size of the buffer must be at least as big as the biggest offset used in the compressed stream. The current version of the Snappy algorithm as defined by Google works on 32k blocks and doesn't contain offsets bigger than 32k which is the default block size used by this class.
The raw Snappy format requires the uncompressed size to be written at the beginning of the stream using a varint representation, i.e. the number of bytes needed to write the information is not known before the uncompressed size is known. We've chosen to make the uncompressedSize a parameter of the constructor in favor of buffering the whole output until the size is known. When using the
FramedSnappyCompressorOutputStream
this limitation is taken care of by the warpping framing format.- Since:
- 1.14
- See Also:
- Snappy compressed format description
-
-
Field Summary
Fields Modifier and Type Field Description private LZ77Compressor
compressor
private ByteUtils.ByteConsumer
consumer
private boolean
finished
private static int
FOUR_BYTE_COPY_TAG
private static int
FOUR_SIZE_BYTE_MARKER
private static int
MAX_LITERAL_SIZE_WITH_ONE_SIZE_BYTE
private static int
MAX_LITERAL_SIZE_WITH_THREE_SIZE_BYTES
private static int
MAX_LITERAL_SIZE_WITH_TWO_SIZE_BYTES
private static int
MAX_LITERAL_SIZE_WITHOUT_SIZE_BYTES
private static int
MAX_MATCH_LENGTH
private static int
MAX_MATCH_LENGTH_WITH_ONE_OFFSET_BYTE
private static int
MAX_OFFSET_WITH_ONE_OFFSET_BYTE
private static int
MAX_OFFSET_WITH_TWO_OFFSET_BYTES
private static int
MIN_MATCH_LENGTH
private static int
MIN_MATCH_LENGTH_WITH_ONE_OFFSET_BYTE
private static int
ONE_BYTE_COPY_TAG
private static int
ONE_SIZE_BYTE_MARKER
private byte[]
oneByte
private java.io.OutputStream
os
private static int
THREE_SIZE_BYTE_MARKER
private static int
TWO_BYTE_COPY_TAG
private static int
TWO_SIZE_BYTE_MARKER
-
Constructor Summary
Constructors Constructor Description SnappyCompressorOutputStream(java.io.OutputStream os, long uncompressedSize)
Constructor using the default block size of 32k.SnappyCompressorOutputStream(java.io.OutputStream os, long uncompressedSize, int blockSize)
Constructor using a configurable block size.SnappyCompressorOutputStream(java.io.OutputStream os, long uncompressedSize, Parameters params)
Constructor providing full control over the underlying LZ77 compressor.
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description void
close()
static Parameters.Builder
createParameterBuilder(int blockSize)
Returns a builder correctly configured for the Snappy algorithm using the gven block size.void
finish()
Compresses all remaining data and writes it to the stream, doesn't close the underlying stream.void
write(byte[] data, int off, int len)
void
write(int b)
private void
writeBackReference(LZ77Compressor.BackReference block)
private void
writeBackReferenceWithFourOffsetBytes(int len, int offset)
private void
writeBackReferenceWithLittleEndianOffset(int tag, int offsetBytes, int len, int offset)
private void
writeBackReferenceWithOneOffsetByte(int len, int offset)
private void
writeBackReferenceWithTwoOffsetBytes(int len, int offset)
private void
writeLiteralBlock(LZ77Compressor.LiteralBlock block)
private void
writeLiteralBlockFourSizeBytes(LZ77Compressor.LiteralBlock block, int len)
private void
writeLiteralBlockNoSizeBytes(LZ77Compressor.LiteralBlock block, int len)
private void
writeLiteralBlockOneSizeByte(LZ77Compressor.LiteralBlock block, int len)
private void
writeLiteralBlockThreeSizeBytes(LZ77Compressor.LiteralBlock block, int len)
private void
writeLiteralBlockTwoSizeBytes(LZ77Compressor.LiteralBlock block, int len)
private void
writeLiteralBlockWithSize(int tagByte, int sizeBytes, int len, LZ77Compressor.LiteralBlock block)
private void
writeLittleEndian(int numBytes, int num)
private void
writeUncompressedSize(long uncompressedSize)
-
-
-
Field Detail
-
compressor
private final LZ77Compressor compressor
-
os
private final java.io.OutputStream os
-
consumer
private final ByteUtils.ByteConsumer consumer
-
oneByte
private final byte[] oneByte
-
finished
private boolean finished
-
MAX_LITERAL_SIZE_WITHOUT_SIZE_BYTES
private static final int MAX_LITERAL_SIZE_WITHOUT_SIZE_BYTES
- See Also:
- Constant Field Values
-
MAX_LITERAL_SIZE_WITH_ONE_SIZE_BYTE
private static final int MAX_LITERAL_SIZE_WITH_ONE_SIZE_BYTE
- See Also:
- Constant Field Values
-
MAX_LITERAL_SIZE_WITH_TWO_SIZE_BYTES
private static final int MAX_LITERAL_SIZE_WITH_TWO_SIZE_BYTES
- See Also:
- Constant Field Values
-
MAX_LITERAL_SIZE_WITH_THREE_SIZE_BYTES
private static final int MAX_LITERAL_SIZE_WITH_THREE_SIZE_BYTES
- See Also:
- Constant Field Values
-
ONE_SIZE_BYTE_MARKER
private static final int ONE_SIZE_BYTE_MARKER
- See Also:
- Constant Field Values
-
TWO_SIZE_BYTE_MARKER
private static final int TWO_SIZE_BYTE_MARKER
- See Also:
- Constant Field Values
-
THREE_SIZE_BYTE_MARKER
private static final int THREE_SIZE_BYTE_MARKER
- See Also:
- Constant Field Values
-
FOUR_SIZE_BYTE_MARKER
private static final int FOUR_SIZE_BYTE_MARKER
- See Also:
- Constant Field Values
-
MIN_MATCH_LENGTH_WITH_ONE_OFFSET_BYTE
private static final int MIN_MATCH_LENGTH_WITH_ONE_OFFSET_BYTE
- See Also:
- Constant Field Values
-
MAX_MATCH_LENGTH_WITH_ONE_OFFSET_BYTE
private static final int MAX_MATCH_LENGTH_WITH_ONE_OFFSET_BYTE
- See Also:
- Constant Field Values
-
MAX_OFFSET_WITH_ONE_OFFSET_BYTE
private static final int MAX_OFFSET_WITH_ONE_OFFSET_BYTE
- See Also:
- Constant Field Values
-
MAX_OFFSET_WITH_TWO_OFFSET_BYTES
private static final int MAX_OFFSET_WITH_TWO_OFFSET_BYTES
- See Also:
- Constant Field Values
-
ONE_BYTE_COPY_TAG
private static final int ONE_BYTE_COPY_TAG
- See Also:
- Constant Field Values
-
TWO_BYTE_COPY_TAG
private static final int TWO_BYTE_COPY_TAG
- See Also:
- Constant Field Values
-
FOUR_BYTE_COPY_TAG
private static final int FOUR_BYTE_COPY_TAG
- See Also:
- Constant Field Values
-
MIN_MATCH_LENGTH
private static final int MIN_MATCH_LENGTH
- See Also:
- Constant Field Values
-
MAX_MATCH_LENGTH
private static final int MAX_MATCH_LENGTH
- See Also:
- Constant Field Values
-
-
Constructor Detail
-
SnappyCompressorOutputStream
public SnappyCompressorOutputStream(java.io.OutputStream os, long uncompressedSize) throws java.io.IOException
Constructor using the default block size of 32k.- Parameters:
os
- the outputstream to write compressed data touncompressedSize
- the uncompressed size of data- Throws:
java.io.IOException
- if writing of the size fails
-
SnappyCompressorOutputStream
public SnappyCompressorOutputStream(java.io.OutputStream os, long uncompressedSize, int blockSize) throws java.io.IOException
Constructor using a configurable block size.- Parameters:
os
- the outputstream to write compressed data touncompressedSize
- the uncompressed size of datablockSize
- the block size used - must be a power of two- Throws:
java.io.IOException
- if writing of the size fails
-
SnappyCompressorOutputStream
public SnappyCompressorOutputStream(java.io.OutputStream os, long uncompressedSize, Parameters params) throws java.io.IOException
Constructor providing full control over the underlying LZ77 compressor.- Parameters:
os
- the outputstream to write compressed data touncompressedSize
- the uncompressed size of dataparams
- the parameters to use by the compressor - note that the format itself imposes some limits like a maximum match length of 64 bytes- Throws:
java.io.IOException
- if writing of the size fails
-
-
Method Detail
-
write
public void write(int b) throws java.io.IOException
- Specified by:
write
in classjava.io.OutputStream
- Throws:
java.io.IOException
-
write
public void write(byte[] data, int off, int len) throws java.io.IOException
- Overrides:
write
in classjava.io.OutputStream
- Throws:
java.io.IOException
-
close
public void close() throws java.io.IOException
- Specified by:
close
in interfacejava.lang.AutoCloseable
- Specified by:
close
in interfacejava.io.Closeable
- Overrides:
close
in classjava.io.OutputStream
- Throws:
java.io.IOException
-
finish
public void finish() throws java.io.IOException
Compresses all remaining data and writes it to the stream, doesn't close the underlying stream.- Throws:
java.io.IOException
- if an error occurs
-
writeUncompressedSize
private void writeUncompressedSize(long uncompressedSize) throws java.io.IOException
- Throws:
java.io.IOException
-
writeLiteralBlock
private void writeLiteralBlock(LZ77Compressor.LiteralBlock block) throws java.io.IOException
- Throws:
java.io.IOException
-
writeLiteralBlockNoSizeBytes
private void writeLiteralBlockNoSizeBytes(LZ77Compressor.LiteralBlock block, int len) throws java.io.IOException
- Throws:
java.io.IOException
-
writeLiteralBlockOneSizeByte
private void writeLiteralBlockOneSizeByte(LZ77Compressor.LiteralBlock block, int len) throws java.io.IOException
- Throws:
java.io.IOException
-
writeLiteralBlockTwoSizeBytes
private void writeLiteralBlockTwoSizeBytes(LZ77Compressor.LiteralBlock block, int len) throws java.io.IOException
- Throws:
java.io.IOException
-
writeLiteralBlockThreeSizeBytes
private void writeLiteralBlockThreeSizeBytes(LZ77Compressor.LiteralBlock block, int len) throws java.io.IOException
- Throws:
java.io.IOException
-
writeLiteralBlockFourSizeBytes
private void writeLiteralBlockFourSizeBytes(LZ77Compressor.LiteralBlock block, int len) throws java.io.IOException
- Throws:
java.io.IOException
-
writeLiteralBlockWithSize
private void writeLiteralBlockWithSize(int tagByte, int sizeBytes, int len, LZ77Compressor.LiteralBlock block) throws java.io.IOException
- Throws:
java.io.IOException
-
writeLittleEndian
private void writeLittleEndian(int numBytes, int num) throws java.io.IOException
- Throws:
java.io.IOException
-
writeBackReference
private void writeBackReference(LZ77Compressor.BackReference block) throws java.io.IOException
- Throws:
java.io.IOException
-
writeBackReferenceWithOneOffsetByte
private void writeBackReferenceWithOneOffsetByte(int len, int offset) throws java.io.IOException
- Throws:
java.io.IOException
-
writeBackReferenceWithTwoOffsetBytes
private void writeBackReferenceWithTwoOffsetBytes(int len, int offset) throws java.io.IOException
- Throws:
java.io.IOException
-
writeBackReferenceWithFourOffsetBytes
private void writeBackReferenceWithFourOffsetBytes(int len, int offset) throws java.io.IOException
- Throws:
java.io.IOException
-
writeBackReferenceWithLittleEndianOffset
private void writeBackReferenceWithLittleEndianOffset(int tag, int offsetBytes, int len, int offset) throws java.io.IOException
- Throws:
java.io.IOException
-
createParameterBuilder
public static Parameters.Builder createParameterBuilder(int blockSize)
Returns a builder correctly configured for the Snappy algorithm using the gven block size.- Parameters:
blockSize
- the block size.- Returns:
- a builder correctly configured for the Snappy algorithm using the gven block size
-
-