Gzip split large files

The following should give you files about 1 GiB in size each. So depending on what you really want to do with it, you might want to go for smaller splits.

Or just leave it as gz , works well enough for zgrep and other tools, even if it has to be uncompressed every single time. If this is a log file you might want to fine tune your log rotation, to produce smaller splits naturally. Sign up to join this community. The best answers are voted up and rise to the top. Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams?

Learn more. Split gz file and decompress individually [duplicate] Ask Question. Asked 4 years, 9 months ago. Active 4 years, 9 months ago. Viewed 11k times. Improve this question. Noor Noor 3 3 gold badges 11 11 silver badges 18 18 bronze badges. StephenKitt, in the possible duplicate, it doesn't mention any command, i'm not well verse in the command so i'll need to commands as part of the answer too, thanks — Noor. Could you say something more about what the size limitation is? Also, what is your goal?

Is it to process the file somehow could be done without storing the decompressed data , or do you want to do something else with the split file? Noor Well, that's clear, but can't you decompress it because there's not enough space left on the filesystem, or is there some other space-related issue? Also, "get into the file" can be done in a number of ways without decompressing it, depending on what you want to do. Noor The command I gave above will not store the decompressed file.

Depending on what you want to do you may not need to store any part of the decompressed file, but you're not letting us know what you want to do, so we are struggling to help you. You may likewise grep the decompressed data with either zgrep directly, or with gunzip -c file. Show 8 more comments. Active Oldest Votes. Improve this answer. Add a comment. Well, split will happily split things for you in various ways.

Here's an attempt at a solution, lightly tested:. I would further improve this by using a temporary directory mktemp -d for the intermediate files and ensuring the script cleans up after itself at exit with a trap. Ideally it would also sanity check the arguments, possibly accepting a second argument indicating the number of lines per part, and inspect the contents of the current directory to ensure it doesn't clobber any preexisting files.

I don't think awk is for splitting gzip file into smaller pieces files, it's for text-processing. Below is my way to solve your issue, hope it helps:. Stack Overflow for Teams — Collaborate and share knowledge with a private group.

Create a free Team What is Teams? Collectives on Stack Overflow. Learn more. Split large gzip file while adding header line to each split Ask Question. Asked 4 years, 10 months ago. Active 4 years, 10 months ago.

Viewed times. Improve this question. Add a comment. Active Oldest Votes. I would approach it like this: gunzip the file use head to get the first line and save it off to another file use tail to get the rest of the file and pipe it to split to produce files of 10,, lines each use sed to insert the header into each file, or just cat the header with each file gzip each file You'll want to wrap this in a script or a function to make it easier to rerun at a later time.

Here's an attempt at a solution, lightly tested:! Improve this answer. Is it possible for example on that too. Thanks — add-semi-colons. What arguments would you want? There are plenty of examples on StackOverflow of shell scripts that accept arguments if that's all you need.

Haifeng Zhang Haifeng Zhang