Home Dashboard Directory Help
Search

Deflatestream cannot read as compressed or write as uncompressed by David R Tribble


Status: 

Closed
 as By Design Help for as By Design


0
0
Sign in
to vote
Type: Bug
ID: 393748
Opened: 1/9/2009 3:19:50 PM
Access Restriction: Public
1
Workaround(s)
view
0
User(s) can reproduce this bug

Description

System.IO.Compression.Deflatestream allows one to read from a stream of compressed data, decompressing (inflating) it as it is read. However, it is not possible to read a stream of normal data, compressing it as it is read. (This throws an InvalidOperationException.)

Likewise, one can write normal data to a stream, compressing (deflating) the data as it is written. However, it is not possible to write compressed data to a stream, uncompressing it into normal data as it is written. (There is no exception documented for this, but presumably it's the same InvalidOperationException.)

In other words, data compression and decompression are supported in one direction only, writing to compress and reading to decompress. The reverse directions are not: writing to decompress and reading to compress.

Example

Consider an application that reads plaintext data from a file, compresses it, then encrypts it, writing it to an output file (or network socket). The reverse operation reads the encrypted ciphertext data, decrypts it, then decompresses it, writing the recovered plaintext data to an output file (or socket).

Work-around

The only work-arounds I can think of are:
1) Use an in-memory MemoryStream to hold the entire data stream. This is obviously impractical for large amounts of data or for streaming (network) data.

2) Write Stream classes that utilize a Deflatestream object, intercepting the Read and Write calls to handle the compression/decompression. The code operates in an inside-out manner.

3) Use temporary files to buffer large compressed/uncompressed data. Data to be read as compressed is first read and written compressed to a temp file, then the file is opened in read mode to read the compressed data, then the temp file is deleted. The opposite is done for data to be written as uncompressed.

None of these work-arounds is a good solution.
Details
Sign in to post a comment.
Posted by David R Tribble on 1/16/2009 at 1:21 PM
The code I posted sort of works. It still needs work to handle the 'count' argument correctly.

Away, it gives you an idea of how this kind of thing could be done in an "inside-out" fashion.
Posted by David R Tribble on 1/16/2009 at 1:16 PM
See the attachments for more complete source code as well as a test driver program.

The complementary OutputInflaterStream class is much more difficult to write, and so far I haven't been able to do so.
Posted by David R Tribble on 1/16/2009 at 1:12 PM
Correction: That last comment describes an InputDeflaterStream, which allows normal input data to be read as compressed data.
Posted by David R Tribble on 1/16/2009 at 1:10 PM
I managed to write a workable InflaterOutputStream, which allows compressed input data to be written to a stream as decompressed data. It works fairly well, but I don't recommend this as an actual solution.

public class InputDeflaterStream: Stream
{
    private Stream         m_in;
    private byte[]         m_inBuf =     new byte[4*1024];
    private byte[]         m_memBuf =     new byte[4*1024];
    private byte[]         m_tmpBuf =     new byte[1];
    private MemoryStream    m_memStream;
    private DeflateStream m_defl;

    public InputDeflaterStream(Stream inp)
    {
        m_in = inp;
        m_memStream = new MemoryStream(m_memBuf, true);
        m_defl = new DeflateStream(m_memStream,
            CompressionMode.Compress);
    }

    public override int Read(byte[] buf, int offset, int count)
    {
        int     nBytes;

        // Sanity checks
        if (m_in == null)
            throw new ObjectDisposedException("Stream is closed");
        if (buf == null)
            throw new ArgumentNullException("Null buffer");
        if (offset < 0)
            throw new ArgumentOutOfRangeException("Negative offset");
        if (offset + count > buf.Length)
            throw new ArgumentException("Bad offset or count");

        // Read data from the input stream, and write them to the
        // in-memory compression stream buffer.
        nBytes = 0;
        while (count > 0)
        {
            int     len;

            // Read the next chunk of input data
            len = m_in.Read(m_inBuf, 0, m_inBuf.Length);
            if (len <= 0)
                break;

            // Write the input data to the in-memory compression stream
            m_defl.Write(m_inBuf, 0, len);

            // Fill the read buffer with compressed data read from
            // the in-memory buffer
            m_defl.Flush();
            len = (int) m_memStream.Position;
            if (len > 0)
            {
                Array.Copy(m_memBuf, 0, buf, offset, len);
                offset += len;
                nBytes += len;
                m_memStream.Position = 0;
            }
        }
        return nBytes;
    }

    public override int ReadByte()
    {
        if (Read(m_tmpBuf, 0, 1) <= 0)
            return -1;
        return m_tmpBuf[0];
    }

//etc.
}
Posted by Microsoft on 1/15/2009 at 8:26 PM
Hi David,
Thanks for providing this feedback. You're right that, through DeflateStream itself, operations are unidirectional. We'll take note of this request for possible inclusion in a future version of .NET. (We're interested in adding related compression capabilities, such as seeking in a compressed stream.)

For now, the best I can offer is workaround suggestions. The MemoryStream approach would be ideal, but if your compressed streams will be very large, you're right that you'd want to avoid that. Adding your own Stream class can be tricky to get right, so you may prefer something more straightforward. I'd recommend using the temp file approach, but use the DeflateStream leaveOpen parameter which may simplify the code a bit.

Refer to the code below. This creates a FileStream (called deflateFileStream) that DeflateStream will write to, but the 3rd argument leaveOpen is true. After compressing, you set the FileStream position to 0 to read through the stream.


using System;
using System.IO;
using System.IO.Compression;

namespace DeflateStreamTests
{
    class Program
    {
        const int BufferSize = 4096;

        static void Main(string[] args)
        {
            try
            {
                FileStream src = new FileStream(@"C:\temp\fileToCompress.bin", FileMode.Open, FileAccess.Read);
                FileStream deflateFileStream = new FileStream("out.deflate", FileMode.OpenOrCreate, FileAccess.ReadWrite);

                DeflateStream deflateStream = new DeflateStream(deflateFileStream, CompressionMode.Compress, true);
                int index = 0;
                int recv = 0;
                int size = (int)src.Length;
                using (deflateStream)
                {
                    while (index < size)
                    {
                        byte[] buffer = new byte[BufferSize];
                        recv = src.Read(buffer, 0, buffer.Length);
                        if (recv <= 0) break;
                        deflateStream.Write(buffer, 0, recv);
                        index += recv;
                    }
                }

                src.Close();

                deflateFileStream.Position = 0;
                byte[] compressedBytes = new byte[BufferSize];
                while (deflateFileStream.Read(compressedBytes, 0, compressedBytes.Length) > 0)
                {
                    Console.WriteLine(compressedBytes[0]); // placeholder: just print out 1st element of array
                }
            }
            catch (Exception e)
            {
                Console.WriteLine(e);
            }
        }
    }
}

Again, this isn't ideal, but probably the simplest solution for the moment. Thanks again for the feedback. We'll make note of this request for a future version of .NET.

Thanks!
Kim Hamilton
Base Class Libraries
Posted by Microsoft on 1/13/2009 at 10:03 PM
Thank you for your feedback, We are currently reviewing the issue you have submitted. If this issue is urgent, please contact support directly(http://support.microsoft.com)
Posted by David R Tribble on 1/13/2009 at 12:44 PM
>> "it is not possible to read a stream of normal data, compressing
>> it as it is read. "

This means that I cannot open the DeflateStream for reading and compressing (deflating). I.e., I cannot read plain data from the underlying input stream so that the deflater compresses it, so that my read() calls return the data as compressed data bytes (instead of as plain data).

Likewise, I cannot open the DeflateStream for writing and decompressing (inflating). I.e., I cannot write compressed data to the stream so that the inflater decompresses the data as it gets written to the underlying stream, so that it is written as uncompressed data.

In other words, I would like
a) to read plain data from an input stream, compressing it as it is read, so that I receive a compressed (ZIP) stream of data, and
b) given a source of compressed (ZIP) data, write that data to an output stream decompressing it as it is written, so that the original uncompressed data bytes are written to the underlying output stream.
Posted by Microsoft on 1/12/2009 at 11:38 PM
Thanks for reporting the issue.

I'm not sure what the following sentence means:
"it is not possible to read a stream of normal data, compressing it as it is read. "

Could you please provide more details or upload a zipped project file about the problem?

Thank you,
Visual Studio Product Team

Posted by Microsoft on 1/12/2009 at 2:54 AM
Thank you for your feedback, We are currently reviewing the issue you have submitted. If this issue is urgent, please contact support directly(http://support.microsoft.com)
Sign in to post a workaround.
Posted by cheeso on 11/13/2009 at 8:17 AM
DotNetZip (http://dotnetzip.codeplex.com) has a DeflateStream that works both ways. You can compress or decompress during read or write.
It's free.