python - MemoryError when read a file larger than 2G -
a single file size greater 2g. call open(f, "rb").read() memoryerror. call open(f, "rb").read(1<<30) ok
how can eliminate 2g limit? have enough memory -- 16g
what using memory mapped files (mmap)? there's example in documentation on python.org. adapted below.
with open(f, "rb") fi:     # memory-map file, size 0 means whole file     mm = mmap.mmap(fi.fileno(), 0)      # stuff      mm.close()      
Comments
Post a Comment