cost-of-modules reports too small sizes

[nknapp]

I don't know, how important that is to you, but I think that the sizes that are reported are sometimes smaller than the actual sizes. For example

> mkdir test
> cd test/
> npm init -y 
Wrote to /home/nknappmeier/tmp/test/package.json:
[...]

> npm install --save thought
npm WARN prefer global [email protected] should be installed with -g
[email protected] /home/nknappmeier/tmp/test
└─┬ [email protected] 
  ├── [email protected] 
[...]

> cost-of-modules

Making sure dependencies are installed
npm install --production

npm WARN [email protected] No description
npm WARN [email protected] No repository field.

Calculating...


┌───────────┬──────────────┬───────┐
│ name      │ children     │ size  │
├───────────┼──────────────┼───────┤
│ thought   │ 118          │ 8.63M │
├───────────┼──────────────┼───────┤
│ 1 modules │ 113 children │ 8.43M │
└───────────┴──────────────┴───────┘

> du -hs .
13M	.

While cost-of-modules only reports 8.43M of this project (that has only a package.json and the thought-dependency), while du -hs reports 13M. Maybe it does not take into account, that even a very small file uses 4kb of space, depending on the filesystems block-size.

[nknapp]

I was motivated an wrote my own analzyer analyze-module-size which reports the sizes in a slightly different way (output the whole tree with aggregated block-based sizes for each dependency).

[siddharthkp]

Correct size is definitely the most important thing.

Can you share your package.json and I'll look into what's happening

[siddharthkp]

Great work on analyze module size!

[nknapp]

I have created my test-package with

npm init -y
npm install --save though

in an empty directory. The package.json is

{
  "name": "test",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "thought": "^1.1.2"
  }
}

[siddharthkp]

Looping @alshakero in here. Do you think we introduced a bug here: pull/26/files#diff

The results of fs.readdir do not match with a du -k

[nknapp]

I have noticed that "du -h" shows 4k size on my system, even if the file contents is only 100 bytes. It also shows 4k for directories. The stats show a block size of 4096 for my system, so I just round up file sizes to the next block in analyze-module-size. This quickly adds up if you have a lot of small files and a lot of small modules. I think even a small file uses a whole block, so counting the blocks should be the correct way of measuring the size. However, the blocks-property of the stats object seems to be based on 512-byte blocks, even if blockSize states 4096. On TravisCI, the block size seems to be different. Am 8. Mai 2017 15:15:28 MESZ schrieb Siddharth Kshetrapal <[email protected]>:

…

Looping @alshakero in here. Do you think we introduced a bug here: [pull/26/files#diff](https://github.com/siddharthkp/cost-of-modules/pull/26/files#diff-2113531435815bbbba24d4989c5e9451R2) The results of `fs.readdir` do not match with a `du -k` -- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: #40 (comment)

-- Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

[siddharthkp]

That makes sense.

We used to use du but that led to inconsistent results on different environments (let's not even get into windows). That's a bad developer experience especially if you want to use this in your CI ("works on my machine" bugs)

Honestly, between correctness and consistency, I don't know what to prefer, because, you do not control the environment in which your npm module will land (talking about libraries here)

@nknapp @alshakero What are your thoughts?

[nknapp]

Thinking about it, what is the actual bad experience when installing a bloated package? It's the time it takes for npm install to complete. This may be more related to the number and size of tars that are downloaded. And the time to unpack the tars probably depends more on the number of files than on the unpacked total size. I want to say that different metrics may be more interesting. Am 9. Mai 2017 05:20:55 MESZ schrieb Siddharth Kshetrapal <[email protected]>:

…

That makes sense. We used to use `du` but that led to inconsistent results on different environments (let's not even get into windows). That's a bad developer experience especially if you want to use this in your CI ("works on my machine" bugs) Honestly, between correctness and consistency, I don't know what to prefer, because, you do not control the environment in which your npm module will land (talking about libraries here) @nknapp @alshakero What are your thoughts? -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #40 (comment)

-- Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

[alshakero]

@nknapp said it perfectly. It has to do with block sizes. However, before I went for FS I compared to windows native stats. And FS delivered exactly the same results, as in ===.

So on Windows, I don't think we need to be any more accurate than Windows itself. And I took the numbers I put in fractures from Windows stats.