SQUASHFS/squashfs-tools-4.4/README-4.4

   1         SQUASHFS 4.4 - A squashed read-only filesystem for Linux
   2
   3         Copyright 2002-2019 Phillip Lougher <phillip@squashfs.org.uk>
   4
   5         Released under the GPL licence (version 2 or later).
   6
   7 Welcome to Squashfs 4.4.  This is the first release in over 5 years, and
   8 there are substantial improvements: reproducible builds, new compressors,
   9 CVE fixes, security hardening and new options for Mksquashfs/Unsquashfs.
  10
  11 Please see the INSTALL file for instructions on installing the tools,
  12 and the USAGE file for documentation on how to use the tools.
  13
  14 Summary of changes (and sections below)
  15 ---------------------------------------
  16
  17 1. Mksquashfs now generates reproducible images by default.  Mkfs time and
  18    file timestamps can also be specified.
  19
  20 2. Support for the Zstandard (ZSTD) compression algorithm has been added.
  21
  22 3. Pseudo files now support symbolic links.
  23
  24 4. CVE-2015-4645 and CVE-2015-4646 have been fixed.
  25
  26 5. Unsquashfs has been further hardened against corrupted filestems.
  27
  28 6. Unsquashfs is now more strict about error handling.
  29
  30 7. Miscellaneous new options and major bug fixes for Mksquashfs.
  31
  32 8. Miscellaneous new options and major bug fixes for Unsquashfs.
  33
  34 9. Squashfs-tools 4.4 is compatible with all earlier 4.x filesystems
  35    and releases.
  36
  37
  38 1. Introducing reproducible builds
  39 ----------------------------------
  40
  41 Ever since Mksquashfs was parallelised back in 2006, there
  42 has been a certain randomness in how fragments and multi-block
  43 files are ordered in the output filesystem even if the input
  44 remains the same.
  45
  46 This is because the multiple parallel threads can be scheduled
  47 differently between Mksquashfs runs.  For example, the thread
  48 given fragment 10 to compress may finish before the thread
  49 given fragment 9 to compress on one run (writing fragment 10
  50 to the output filesystem before fragment 9), but, on the next
  51 run it could be vice-versa.  There are many different scheduling
  52 scenarios here, all of which can have a knock on effect causing
  53 different scheduling and ordering later in the filesystem too.
  54
  55 Mkquashfs doesn't care about the ordering of fragments and
  56 multi-block files within the filesystem, as this does not
  57 affect the correctness of the filesystem.
  58
  59 In fact not caring about the ordering, as it doesn't matter, allows
  60 Mksquashfs to run as fast as possible, maximising CPU and I/O
  61 performance.
  62
  63 But, in the last couple of years, Squashfs has become used in
  64 scenarios (cloud etc) where this randomness is causing problems.
  65 Specifically this appears to be where downloaders, installers etc.
  66 try to work out the differences between Squashfs filesystem
  67 updates to minimise the amount of data that needs to transferred
  68 to update an image.
  69
  70 Additionally, in the last couple of years has arisen the notion
  71 of reproducible builds, that is the same source and build
  72 environment etc should be able to (re-)generate identical
  73 output.  This is usually for verification and security, allowing
  74 binaries/distributions to be checked for malicious activity.
  75 See https://reproducible-builds.org/ for more information.
  76
  77 Mksquashfs now generates reproducible images by default.
  78 Images generated by Mksquashfs will be ordered identically to
  79 previous runs if the same input has been supplied, and the
  80 same options used.
  81
  82 1.1.1 Dealing with timestamps
  83
  84 Timestamps embedded in the filesystem will stiil cause differences.
  85 Each new run of Mksquashfs will produce a different mkfs (make filesystem)
  86 timestamp in the super-block.  Moreover if any file timestamps have changed
  87 (even if the content hasn't), this will produce a difference.
  88
  89 To prevent timestamps from producing differences, the following
  90 new Mksquashfs options have been added.
  91
  92 1.1.2 -mkfs-time <time>
  93
  94 This option takes a positive time value (which is the number
  95 of seconds since the epoch of 1970-01-01 00:00:00 UTC), and sets
  96 the file system timestamp to that.
  97
  98 Squashfs uses an unsigned 32-bit integer to store time, and the
  99 time given should be in that range.
 100
 101 Obviously you can use the date command to convert dates into
 102 this value, i.e.
 103
 104 % mksquashfs source source.sqsh -mkfs-time $(date +%s -d "Jan 1 2019 19:00")
 105
 106 1.1.3 -all-time <time>
 107
 108 This option takes a positive time value (which is the number
 109 of seconds since the epoch of 1970-01-01 00:00:00 UTC), and sets
 110 the timestamp on all files to that (but not the mkfs time).
 111
 112 1.1.4 environment variable SOURCE_DATE_EPOCH
 113
 114 As an alternative to the above command line options, you can
 115 set the environment variable SOURCE_DATE_EPOCH to a time value.
 116
 117 This value will be used to set the mkfs time.  Also any
 118 file timestamps which are after SOURCE_DATE_EPOCH will be
 119 clamped to SOURCE_DATE_EPOCH.
 120
 121 See https://reproducible-builds.org/docs/source-date-epoch/
 122 for more information.
 123
 124 Note: both SOURCE_DATE_EPOCH and the command line options cannot
 125 be used at the same time.  They are different ways to do the same thing,
 126 and both have FORCE sematics which mean they can't be over-ridden
 127 elsewhere (otherwise it would defeat the purpose).
 128
 129 1.1.5 -not-reproducible
 130
 131 This option tells Mksquashfs that the files do not have to be
 132 strictly ordered.  This will make Mksquashfs behave like version 4.3.
 133
 134
 135 2. Zstandard (ZSTD) compression added
 136 -------------------------------------
 137
 138 This is named zstd.  It supports the following
 139 compression options.
 140
 141         zstd
 142           -Xcompression-level <compression-level>
 143                 <compression-level> should be 1 .. 22 (default 15)
 144
 145
 146 3. Pseudo file symbolic link support added
 147 ------------------------------------------
 148
 149 Pseudo definition
 150
 151 Filename s mode uid gid symlink
 152
 153 uid and gid can be either specified as a decimal number, or by name.
 154
 155 Note mode is ignored, as symlinks always have "rwxrwxrwx" permissions.
 156
 157 For example:
 158
 159 symlink s 0 root root example
 160
 161 creates a symlink "symlink" to file "example" with root uid/gid.
 162
 163
 164 4. CVE-2015-2015-4645 and CVE-2015-4646
 165 ---------------------------------------
 166
 167 These CVEs were raised due to Unsquashfs having variable overflow and
 168 stack overflow in a number of vulnerable functions.
 169
 170 All instances of variable overflow and stack overflow have been
 171 removed.
 172
 173
 174 5. Unsquashfs hardened against corrupted filestems
 175 --------------------------------------------------
 176
 177 The filesystem super-block values and filesystem metadata tables
 178 are further sanity checked.  More importantly, these values are now
 179 checked for validity against other values in the metadata tables, and
 180 these values must match.
 181
 182
 183 6. Unsquashfs is now more strict about error handling
 184 -----------------------------------------------------
 185
 186 Unsquashfs splits errors into two categories: fatal errors and non-fatal
 187 errors.  In this release a significant number of errors that were previously
 188 non-fatal have been hardened to fatal.
 189
 190 Fatal errors are those which cause Unsquashfs to abort instantly.
 191 These are generally due to failure to read the filesystem (corruption),
 192 and/or failure to write files to the output filesystem, due to I/O error
 193 or out of space.  Generally anything which is unexpected is a fatal error.
 194
 195 Non-fatal errors are generally where support is lacking in the
 196 output filesystem, and it can be considered to be an expected failure.
 197 This includes the inability to write extended attributes (xattrs) to
 198 a filesystem that doesn't support them, the inability to create files on
 199 filesystem that doesn't support them (i.e. symbolic links on VFAT), and the
 200 inability to execute privileged operations as a user-process.
 201
 202 The user may well know the filesystem cannot support certain operations
 203 and would prefer Unsquashfs to ignore then without aborting.
 204
 205 Two new options have been added:
 206
 207 6.1. -ignore-errors
 208
 209 This makes Unsquashfs behave like previous versions, and treats more
 210 errors as non-fatal.
 211
 212 6.2. -strict-errors
 213
 214 This makes Unsquashfs treat every error as fatal, and it will abort
 215 instantly.
 216
 217
 218 7. Miscellaneous new options and major bug fixes for Mksquashfs
 219 ---------------------------------------------------------------
 220
 221 7.1. -root-mode <mode>
 222
 223 This sets the permissions of the root directory to the octal <mode>.
 224 This is mostly intended for when multiple sources are specified on
 225 the command line.  In this instance Mksquashfs produces a dummy top level
 226 directory with permissions 0777 (rwxrwxrwx).  This option allows the
 227 permissions to be changed.  But the option can also be used when a single
 228 source is specified.
 229
 230 7.2. -quiet
 231
 232 This suppresses all output from Mksquashfs, except the progress bar.
 233
 234 The progress bar can disabled with -no-progress to produce completely
 235 silent output.
 236
 237 This new option is useful for scripts.
 238
 239 7.3. -noId
 240
 241 This is similar to the pre-existing -noI option, except, it specifies that
 242 only the Id table (uids and gids) should be uncompressed.  This option was
 243 added to enable a use-case where uids and gids need to be updated after
 244 filesystem generation.
 245
 246 7.4. -offset <offset>
 247
 248 This option skips <offset> bytes at the beginning of the output filesystem.
 249
 250 Optionally a suffix of K, M or G can be given to specify Kbytes, Mbytes or
 251 Gbytes respectively.
 252
 253 7.5. Update lz4 wrapper to use new functions introduced in 1.7.0
 254
 255 7.5. Bug fix, don't allow "/" pseudo filenames
 256
 257 7.6. Bug fix, allow quoting of pseudo files, to better handle filenames with
 258      spaces
 259
 260 7.7. Fix compilation with glibc 2.25+
 261
 262
 263 8. Miscellaneous new options and major bug fixes for Unsquashfs
 264 ---------------------------------------------------------------
 265
 266 8.1. -lln[umeric]
 267
 268 This is similar to the "-lls" option but displays uids and gids numerically.
 269
 270 8.2. -lc option
 271
 272 This is similar to the "-ls" option except it only displays files and empty
 273 directories.
 274
 275 8.3. -llc option
 276
 277 As "-llc" option but displays file attributes.
 278
 279 8.4. -offset <offset>
 280
 281 This option skips <offset> bytes at the beginning of the input filesystem.
 282
 283 Optionally a suffix of K, M or G can be given to specify Kbytes, Mbytes or
 284 Gbytes respectively.
 285
 286 8.5. -quiet
 287
 288 This suppresses all output from Unsquashfs, except the progress bar.
 289
 290 The progress bar can disabled with -no-progress to produce completely
 291 silent output.
 292
 293 This new option is useful for scripts.
 294
 295 8.6. -UTC
 296
 297 This option makes Unsquashfs display all times in the UTC time zone rather
 298 than using the default local time zone.
 299
 300 8.7. Update lz4 wrapper to use new functions introduced in 1.7.0
 301
 302 8.8. Bug fix, fatal and non-fatal errors now set the exit code to 1
 303
 304 8.9. Bug fix, fix time setting for symlinks
 305
 306 8.10. Bug fix, try to set sticky-bit when running as a user process
 307
 308 8.11. Fix compilation with glibc 2.25+
 309
 310
 311 9. Compatiblity
 312 ---------------
 313
 314 Mksquashfs 4.4 generates 4.0 filesystems.  These filesystems are fully
 315 compatible/interchangable with filesystems generated by Mksquashfs 4.x and are
 316 mountable on 2.6.29 and later kernels.