DESCRIPTION

SPY-58: Don't compress json objects

We don't actually parse the whole object. Instead we look
at the first character and if it looks like json then we
don't compress it.

Change-Id: If60423b56ebf22c4529bffe95496ffe8b833a65f


COMMENTS

author: Kenny Maynard
date: 2011-10-21 22:33:25.143000000

Uploaded patch set 3.

-------------------------------------
author: Noah Dean
date: 2011-10-21 23:21:59.541000000

Patch Set 3:

Won't we benefit greatly from compressing JSON objects?

-------------------------------------
author: Noah Dean
date: 2011-10-21 23:38:09.321000000

Patch Set 3:

Mike said in an email somewhere nobody could see it:  :)

> If we compress them then they will end up as attachments instead of json  > documents and we won't be able to run map-reduce queries on them.

They don't stop being JSON docs just because they're compressed.  We can solve that problem by expanding it along the way.  You could set a flag that identifies the object as JSON and if the backend sees that and the compressed flag set, it can try decompressing the object before storing it.  It can also have an attribute that says that it found it compressed and recompress it for memory.  Best of all worlds.

-------------------------------------
author: Kenny Maynard
date: 2011-10-22 00:27:54.931000000

Patch Set 3: Verified



-------------------------------------
author: Kenny Maynard
date: 2011-10-22 00:28:48.186000000

Patch Set 3:

Does this functionality currently exist in Couchbase.

-------------------------------------
author: Noah Dean
date: 2011-10-22 00:31:36.982000000

Patch Set 3:

Nobody's needed it yet.  Similarly, the try to avoid compression of JSON objects doesn't exist in all of the clients.  I don't think it makes much sense to add features to all of the clients to make a default workaround of a limitation we should just get rid of in the server.  The user can always instruct the client to not compress by setting a higher threshold where it's a problem, I'd think.

-------------------------------------
author: Kenny Maynard
date: 2011-10-22 00:38:33.241000000

Patch Set 3:

Makes sense to me. I'll let Matt make the final call.

-------------------------------------
author: Andreas Short
date: 2011-10-22 04:02:07.169000000

Patch Set 3:

I agree with the approach of compressing on the client and even storing it compressed on the server.  I've not raised it because there are so many other issues at the moment.  

There is no standard flag for compression, and the default needs to be interoperability with JSON.  Why else do it?  

For now we need this.  After we have a standard flag and compression scheme which is doable across 6 clients, then we can remove this check.  We need it now so we don't break things mysteriously.

-------------------------------------
author: Andreas Short
date: 2011-10-22 04:07:32.295000000

Patch Set 3:

Oh, and by the way, I think the compression will bring bigger benefit to the server than to the clients.

-------------------------------------
author: Noah Dean
date: 2011-10-22 04:22:48.257000000

Patch Set 3:

Well, today you could just not compress objects on the client by default unless they're really huge.  That seems easier than conditionally compressing them based on byte patterns just in case they want to do MR.

I understand compression is good.  I'm suggesting keeping it compressed in RAM, expand on disk (well, change compression algorithms on disk, is probably more accurate).

-------------------------------------
