![]() This would ideally be fixed by getting RTTI re-enabled in snappy, so I've gone ahead and opened The problem only manifests when our snappy plugin is dlopen()ed at runtime, and then the linker kicks in and can't find that missing symbol. Ceph still builds just fine, because the compressors are built as shared libraries. This is because RTTI was disabled in snappy 1.1.9, so the typeinfo for the snappy::Source class - which Ceph's Snapp圜ompressor creates a subclass of - isn't included in libsnappy.so. Oct 27 08:55:33 node1 ceph-osd: create cannot load compressor of type snappy Oct 27 08:55:33 node1 ceph-osd: load failed dlopen(): "/usr/lib64/ceph/compressor/libceph_snappy.so: undefined symbol: _ZTIN6snappy6SourceE" or "/usr/lib64/ceph/libceph_snappy.so: cannot open shared object file: No such file or directory" The OSD logs will show something like this: ceph health detail will tell you that each of your OSDs is "unable to load:snappy". ![]() Do not store plain text files in Snappy compressed form, instead use a container like SequenceFile.If you try to run Ceph with snappy 1.1.9 installed, ceph status will show HEALTH_WARN, and tell you that your OSDs "have broken BlueStore compression". Plain text files: Like Gzip, Snappy is not splittable. Permanent Storage: Snappy compression is not efficient space-wise and it is expensive to store data on HDFS (3-way replication) Please do make sure these intermediate files are cleaned up soon enough so we don’t have disk space issues on the cluster. Temporary Intermediate files (not available currently as of Pig 0.9.2, applicable only to native Map Reduce) : If you have a series of MR jobs chained together, Snappy compression is a good way to store the intermediate files. Map output: Snappy works great if you have large amounts of data flowing from Mappers to the Reducers (you might not see a significant difference if data volume between Map and Reduce is low) Snappy is not CPU intensive – which means MR tasks have more CPU for user operations.Reduce tasks run faster with better decompression speeds.Map tasks begin transferring data sooner compared to Gzip or Bzip (though more data needs to be transferred to Reduce tasks).tCompressOutput(conf, true) Ĭonf.set("",".compress.Snapp圜odec") tOutputCompressionType(conf, CompressionType.BLOCK) //Block level is better than Record level, in most cases Set Configuration parameters for Snappy compressed intermediate Sequence Files tOutputFormat(SequenceFileOutputFormat.class) Set Configuration parameters for Map output compression Configuration conf = new Configuration() Ĭonf.setBoolean("", true) Ĭonf.set(".codec",".compress.Snapp圜odec") ![]() There is work being done to be able to use Snappy for creating intermediate/temporary files between multiple MR jobs. You can read and write Snappy compressed files as well, though I would not recommend doing that as its not very efficient space-wise compared to other compression algorithms. This should get you going with using Snappy for Map output compression with Pig. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |