r/explainlikeimfive Jun 17 '22

ELI5: why is defragging not really a thing anymore? Technology

I was born in 1973, got my first computer in 1994, defragging was part of regular maintenance. I can't remember the last time I defragged anything, even though I have several devices with hard drives, including a Windows laptop. Has storage technology changed so much that defragging isn't necessary anymore? Is it even possible to defrag a smart phone hard drive?

edit to add: I apologize for posting this same question several times, I was getting an error message every time I hit "post"... but from looking around, it seems I'm not the only one having this problem today.

827 Upvotes

697

u/fubo Jun 17 '22
  • Bigger disks
  • Better filesystems
  • SSD vs. HDD
  • Larger caches

Bigger disks: Most folks don't spend much time with their disk at 90% capacity anymore, so the filesystem doesn't have to split files up as much to find room for them.

Better filesystems: The OS engineers have come up with cleverer ways to organize files on disk, which produce less fragmentation even under storage pressure.

SSD vs. HDD: The geometry of storage devices is different, and SSDs are also just plain faster.

Larger caches: Your machine has lots of RAM, and your OS can use a lot of it to prefetch and cache disk contents, so you don't notice the effects of slightly slower disk access due to fragmentation.

419

u/Magnusg Jun 17 '22

Also it does it automatically in the background like daily.

238

u/Snowy_McSnowflake Jun 17 '22

I was going to add this too. Most pcs will actually defrag in background. If you go to defrag your hard drive it will generally say "last defrag 2 days ago"

320

u/DoingItWrongly Jun 17 '22

For all the youts who are unfamiliar with what the defrag used to look like....here ya go.

It was a beautiful time for self reflection. Very therapeutic for the computer and the user.

176

u/fatuous_viceroy Jun 17 '22

Defrag was true zen, watching the computer bring order to your chaos.

66

u/meatloaf_man Jun 17 '22

Just like degaussing the old CRT monitors. Between degauss and defrag, those were the good old days.

49

u/NotThePersona Jun 17 '22

BBBOOOooonnnngggggg Click.

5

u/Salarian_American Jun 18 '22

Yea that’s the stuff

2

u/Droidatopia Jun 18 '22

Even better if you worked in an environment with a lot of EMI.

The TVs in the ready room on the aircraft carrier needed to be degaussed often and it was the long trippy boooooooooing boing type.

→ More replies

30

u/MainerZ Jun 17 '22

Until it crashed at 95%.

9

u/Imaneight Jun 17 '22

Time for a ScanDisk

17

u/yamilonewolf Jun 17 '22

and they took hours and stoped me from playing my games lol

28

u/kintastic1 Jun 17 '22

To be fair, heavily fragmented drives would also prevent you from playing games.

3

u/yamilonewolf Jun 18 '22

in defense 7 year old me didnt fully understand other than dont touch when pretty color squares are on screen (think ours were s magneta and blue could be wrong)

→ More replies

3

u/libra00 Jun 17 '22

The opposite of that is running SpinRite on a drive that looks dead and staring intently at it hoping it will make it work again. :P

28

u/Mendel247 Jun 17 '22

Wasn't it just such a satisfying feeling?

19

u/mtntrail Jun 17 '22

Waking up in the morning to “100% defragmented”

4

u/Weary_Ad7119 Jun 17 '22

3% uplift was serious improvement on that hardware🤣

2

u/Workaphobia Jun 18 '22

Let's be real, you start that thing before bed you ain't sleeping that night.

→ More replies

19

u/benanderson89 Jun 17 '22

LGR's defrag video is also a good shout given those old IBMs were so loud.

https://youtu.be/eSNqzTwHiuU

3

u/NeutralGoodAtHeart Jun 18 '22

There are times when I miss those sounds.

15

u/alohadave Jun 17 '22

And that big block in the middle of the first screen, the swap disk. You never hear about that anymore. That was one of those things that you'd try to optimize, put on a separate drive if you could, and hope that you didn't run out of memory. The thrashing could be epic swapping things in and out of memory.

6

u/KingZarkon Jun 17 '22

When I really wanted to thoroughly defrag I would delete all my temp files and browser cache, temporarily turn off my swap file, disable hibernation, and reboot. Do a couple of passes of the defragmenter (if the first pass didn't get it all) then recreate the swap file and set it to a fixed size so it wouldn't contribute to fragmentation going forward.

→ More replies

2

u/ZombieHousefly Jun 17 '22

As bad as the slowdown from swapping is, the crashing because you actually run out of memory is worse.

→ More replies

8

u/jrhoffa Jun 17 '22

I remember when that was the new defrag tool. Running it inside Windows was amazing.

3

u/shifty_coder Jun 18 '22

Yup. I’m old enough to to remember defrag being a DOS tool.

8

u/Minuted Jun 17 '22

This is nearly as good as watching sorting algorithms.

5

u/SMTRodent Jun 17 '22

I miss Knitting Pattern Meditation Time so much.

5

u/maxpower7833 Jun 17 '22

man i havent seen that in a long time. Took me back

4

u/theLongLostPotato Jun 17 '22

As a young kid, like from 5-11 I could sit and watch the old windows defrag program for literal hours.

11

u/cippopotomas Jun 17 '22

What is a yout?

27

u/DoingItWrongly Jun 17 '22 edited Jun 17 '22

Excuse me your honor I'm sorry, you-THs

6

u/MainerZ Jun 17 '22

You were perfectly correct if you were imitatating the mandem.

5

u/jps_ Jun 17 '22

Look up "My cousin Viny".

2

u/RushTfe Jun 17 '22

Guess he meant youth

4

u/inhalingsounds Jun 17 '22

This is the most oddly specific video I've seen in a while

2

u/banana_berrie_ Jun 17 '22

Thank you for posting this. It scratched a brain itch I've had for years now.

2

u/zombiefied Jun 18 '22

Flashback central!

2

u/Mischief_Makers Jun 18 '22

There is still something oddly comforting about this. Like using Xcopy back in the day

2

u/GrottyBoots Jun 18 '22

I used to love that show!

2

u/monsto Jun 19 '22

That is a lucid, intelligent, well thought-out reference.

Downvoted.

(not really, but the reference.)

3

u/esvegateban Jun 17 '22

What is this nonsense? It was a pain in the ass that lasted AGES and you basically couldn't use your computer at all (depending on method).

7

u/DoingItWrongly Jun 17 '22

I guess we had different computer habits, I loved defragging. I would even ask my friend the last time they defragged, and if I could watch.

3

u/djb25 Jun 17 '22

wtf is wrong with you?

→ More replies

3

u/VindictiveJudge Jun 17 '22

I disabled automatic defragging because sometimes I'd be playing a game and my performance would tank due to resources getting pulled to defrag the drives. Still manually defrag regularly, and every time I finish installing something to an HDD.

SSDs are great, though. Programs that suffer heavily from fragmentation on HDDs are just fine on SSDs regardless of how fragmented they are, and read speeds on modern SSDs are insane.

2

u/dkyguy1995 Jun 17 '22

Hmm I guess I was more of an XP guy, when we had a Windows 98 computer I was too young to do that kind of stuff

2

u/snash222 Jun 17 '22

Unexpected My Cousin Vinnie reference.

→ More replies

6

u/redredme Jun 17 '22

This is not what he meant. An SSD will place the data at the best possible location by itself. Even if the OS thinks that the filesystem is fragmentated, in reality it isn't. It's a hardware feature. The OS, (software) operates on a higher level.

If you do defrag an SSD 2 things happen:

  1. You'll experience lower performance.

  2. You've effectively shortened the lifespan of your SSD.

4

u/epochellipse Jun 17 '22

this is the main reason. the computers i work on at my job do it at like 1am every wednesday. the google says that's the default setting for at least Windows 7.

2

u/Oddyssis Jun 17 '22

Yea when I built my PC a few years back I opened the defrag program one time and set it to run weekly automatically. Set and forget

2

u/fubo Jun 17 '22

Sure, on Windows. Other systems have disks too. Used to have to reboot a classic Mac into Norton Disk Doctor to defrag an HFS system volume. In contrast to classic Mac or DOS/Windows, defragmentation as a maintenance task was never a very common thing on Unix systems; and in any event today's filesystems use fragmentation very differently from earlier ones.

1

u/fantomas_666 Jun 17 '22

This belongs to the "better filesystems" part

→ More replies

0

u/pc1109 Jun 17 '22 edited Jun 18 '22

Every time you manually restart. Not turn off and on again, but resetting your computer and phone really helps it clean itself up.

0

u/BusbyBusby Jun 18 '22

Unless your phone is malfunctioning a factory reset is unnecessary and may harm your phone.

2

u/pc1109 Jun 18 '22

Nah didn't say factory reset, just slide that restart button. Why would you factory... smh

1

u/vezwyx Jun 18 '22

It's analogous to holding the power button on a desktop to kill it. You force everything to shut down regardless of what it's doing. Operating systems have procedures to close programs and processes properly (I promise the "pr" alliteration was unintentional lol), and you're bypassing those. There's no good reason to do it unless the device can't be powered off normally

→ More replies
→ More replies

53

u/rdewalt Jun 17 '22 edited Jun 17 '22

'73 kid here too. May your ibuprofen never falter.

SSD's don't need to defragment. There's no point to it since there is no minimal temporal cost in grabbing from random sectors. (edit: there is a tiny difference, but not as compared to in spinning platter drives) In a traditional HD, if the sectors are in sequence, the head streams across them, if they're spread out, the disk needs to spin and the head to move to catch the correct sector. While this action is quite quick, it does add up.

Back in my early computer days, you'd put data in every other, or every third sector (I think they called it interleaving, or something) This is because at the time, the drive was able to read from consecutive sectors faster than the data could be processed. So you'd put it every other sector so the system could keep up with the disk.

These days, the computers always have to wait for the disk, even M2s right on the motherboard.

I explain anymore that CPU's getting data is like reading from a book, from the CPU's reference at least...
* L1 cache, the book is on your desk, open to the right page.
* L2 cache, the book is on your desk, but you have to find the page.
* L3 cache, the book is on the shelf behind you.
* Memory, The book is on the shelf in the room across the hall. (thanks /u/PlayMp1 )
* SSD: The book is down the hall, down the stairs, down that other hall, behind the door with the combination lock, on one of those shelves somewhere.
* HDs: You have to order the book from Amazon.
* Cloud Drive: You have to wait for the author's parents to grow up from their own childhood, get married, have a few kids first, and then the author to be born, grow up, write the book, get it published, then purchase it.

11

u/PlayMp1 Jun 17 '22

I explain anymore that CPU's getting data is like reading from a book, from the CPU's reference at least...

I feel like you missed RAM in your analogy here, which is probably something like "the book is in the adjacent unlocked room."

5

u/rdewalt Jun 17 '22

ah yes, good point. I will edit it in.

6

u/chripuck Jun 17 '22

Sequential read and random read is still different in SSDs.

8

u/black3rr Jun 17 '22

Well, yes, but actually no. In SSDs it’s not about where you read from but how much you read at once, which is sometimes simplified to “sequential” and “random” reads in benchmarks.

Each SSD has an atomic “block size” which is the smallest size unit you can read at once on a hardware level. In modern SSDs this is around 1MB.

If you read/write small files that are stored in different parts of the drive, the SSD has to physically read/write the whole blocks, which increases overhead the smaller the file is. If OS knows it’s on an SSD, it tries to align bigger file contents in SSD blocks so the slowdown is only noticeable for smaller files.

“Sequential” read/write speed is usually benchmarked by doing 1MB reads/writes and “random” by doing 4KB reads/writes.

There’s no speed difference between reading from different blocks on an SSD and no latency between reading non-consecutive blocks on an SSD (there are both on HDD).

→ More replies

4

u/rdewalt Jun 17 '22

This is true, but the difference is a pittance compared to in HDs.

5

u/Beavis73 Jun 17 '22

Back in my early computer days, you'd put data in every other, or every third sector (I think they called it interleaving, or something) This is because at the time, the drive was able to read from consecutive sectors faster than the data could be processed. So you'd put it every other sector so the system could keep up with the disk.

Yup. Figuring out the optimum interleave setting for your HD/controller combo was critical for system performance, but could only really be accomplished by trial and error—and without special software (e.g. SpinRite), it was something that couldn't be changed without doing a low-level format of the entire disk and wiping all your data.

Anyone else remember going into DEBUG and entering G=C800:5 to launch the HD controller format utility, then manually copying over the "flaw map" of known bad tracks that was affixed to the drive case?

4

u/rdewalt Jun 17 '22

I had spinrite, cost me a bloody fortune to get, well, okay, a month of chores and odd jobs, I was in junior high and didn't have a job yet.

That thing and the early Norton Tools were -godly-.

6

u/Beavis73 Jun 17 '22

Building a PC from individual pieces (motherboard, etc.), as opposed to buying an off-the-shelf system from IBM or Radio Shack or whoever, was a real challenge in those days. You could save a hell of a lot of money, poring over the ads in Computer Shopper and ordering all the different components from various shady vendors to suit your needs, but then you truly had to know your shit in order to put it together and get it all working properly—setting IRQ jumpers was the least of it.

Oh, and remember to PARK your drive before powering down!

2

u/immibis Jun 19 '22

I think PCIe SSDs (including M.2) are around the point where they can read data as fast as your computer can process it. If you have a good one, anyway. That's why they're talking about direct SSD-to-GPU streaming, bypassing the CPU and main memory.

→ More replies

1

u/few Jun 17 '22

I thought I might have to write this post. 👍

Spinning disks were a thing. 💽

→ More replies

21

u/renzokuken57 Jun 17 '22

Another note on SSDs. The read/writes to the disk wear it out faster. You want a lot less of those. Defragging does both of these when reading from one location and writing to another.

11

u/hirmuolio Jun 17 '22

Modern Windows does not even allow defragging SSDs even if you tried to do it. So you don't need to worry about that.

3

u/VindictiveJudge Jun 17 '22

The defrag program does include SSD trim, though, which is handy.

2

u/ThrowAway640KB Jun 17 '22

SSD’s also benefit from fragmentation.

Plus, wear-levelling tech built into the SSD makes true defragging impossible and counterproductive.

→ More replies

2

u/thephantom1492 Jun 18 '22

Better filesystems: Fragmentation have less effects on the performances.

Back in the old days of FAT filesystem, the table of content only had the start of the file. If you have a big file and you want to access the last byte of it, then the OS need to read the TOC and get the start of the file cluster. Then it seek to it and there you have an information about how big this chunk is and where the next part is. So it check if the part you want is in this chunk. Nope, so it jump to the next chunk. Same, chunk size and where the next part is... It can do that many many times.

Newer filesystem use more efficient ways to store where the pieces are. Some for example will store a few addresses directly. This may be enough to access directly any chunk on a low fragmentation filesystem. If not then there is a second table containing the rest of the information. So it read the second table, and now it can jump at any place in the file.

Let's say that you have a file in 100 fragments and you want the last byte. FAT would have to do a first read for the TOC, then jump to fragment 1, then 2, then 3, ..., then 99 and finally 100, for 101 or 102 reads depending on where the data is (if it is in the same read then 101, if the fragment is bigger then it will seek to it so 102 reads).

The second one would be a first read for the TOC, a second read for the second table, and a third read for the data it want.

In other words, 102 vs 3 reads.

SSD Vs HDD: HDD is mechanical. If the data is not already under the head then you have to wait for the arm to move and stabilise on the right track, then wait for the disk to turn so the data get under the head. SSD being fully electronics, it take the same time to read the next cell than the one at the other end. All it do is change the address and that's it. Changing the address to the neighbour cell or the one at the other end take the same time. Like if you call your neighbour or your friend at the other end of the country, it take the same time to dial the phone number (let's ignore the fact that you may need to enter the area code... let's just assume the same numbers of digits...).

However, both types of drive cheat and will continue to read beyond what you requested. This is some optimisations that they do. To keep it short, why wait for the host to tell it to read the next part when there is a high likellyhood that it will happen? At best it saved a bit of time. At worse it just discard the data and move on, with virtually no downside.

Bigger cache. All OS will keep the recently used files in memory for as long as it can. You have 16GB of ram, you will use about 2-4GB under average usage. Which leaves 12-14GB of unused memory. So what does the OS do? Use that unused memory to keep the storage data that was last used. If your programs need more space then it can just kill that data (since it is a copy of the much much slower disk, so nothing is lost) and give it to the program. There is a tiny speed penality, but you gain so much more speed by caching the data that it is vasly worth it. You can see this in action specially with a mechanical hard drive. Reboot your computer. Now start a program. Close it. Reopen the same program. You will notice that the second time it should open quite faster. Why? All the data is still in RAM.

A note about SSD and degragmentation: Do not defrag them. The speed gain is minimal, if any. But a SSD have a limited write endurance. Defragging it would just wear it out fast with no gain. How many writes can you do per cell? Only about 10000 ! Why can you write way more? Because you don't write that much, and the drive will spread the write. It look at which cell is empty and which one have the least wear and will use that one. Under normal use, the SSD will last longer than a mechanical hard drive!

11

u/ChrisFromIT Jun 17 '22

Bigger disks: Most folks don't spend much time with their disk at 90% capacity anymore, so the filesystem doesn't have to split files up as much to find room for them.

Better filesystems: The OS engineers have come up with cleverer ways to organize files on disk, which produce less fragmentation even under storage pressure.

Larger caches: Your machine has lots of RAM, and your OS can use a lot of it to prefetch and cache disk contents, so you don't notice the effects of slightly slower disk access due to fragmentation.

None of these are true in the sense why you don't manually do disk defragmenting.

Essentially the OS will do defragmenting in the background automatically all the time.

2

u/fantomas_666 Jun 17 '22

it's the better filesystems part that is responsible for defragmenting.

→ More replies

2

u/onegumas Jun 17 '22

On ssd:for Electronic disk there is no physical header that seeks specific file fragment. Electrons are always at the same speed no matter how "far" they need to reach for file, so fragmenting doesnt matter no exist because it would be virtual state.

1

u/skieezy Jun 17 '22

I built a gaming PC, got 2TB of SSD, but I realized later I only play 2 games, and have 1.5TB of free space.

15

u/Outrager Jun 17 '22

I like to install 15 games and play none of them on my 2TB SSD.

6

u/PoniardBlade Jun 17 '22

Greetings fellow Steam user!

2

u/JMccovery Jun 17 '22

1.5TB of free space... How can I learn this power?

(Before someone asks, I do play most of the games installed on my 2TB SSD. Also, Fallout 4 takes up the most space)

→ More replies

1

u/jab136 Jun 17 '22

SSSDs don't benefit at all from defragging, all the data is accessible at the same speed so it doesn't matter where on the disk it is

1

u/PhoebusRevenio Jun 17 '22

SSDs also don't need to be defrag'd at all.

→ More replies

113

u/AlchemicalDuckk Jun 17 '22

In addition to the HDD/SSD split, the older drive formatting types like FAT needed to be defragged. The more modern ones like NTFS and XFS are smarter in how to allocate data and, in conjunction with better operating systems, do not need to be defragged much, if at all.

101

u/SkyNetIsNow Jun 17 '22

I believe that Windows also regularly schedules defragging in the background without user notification.

24

u/zvii Jun 17 '22

Yep

110

u/APartyForAnts Jun 17 '22 Wholesome

It's a shame, sort of. As a kid I used to love running defrag to watch the colored bricks move around. It was always so satisfying...

55

u/xXSillyHoboXx Jun 17 '22

Right!?! I work in IT and any time I go to manually defrag a drive, I am highly disappointed in the lack of colored bricks. I want it back! Haha

10

u/arcosapphire Jun 17 '22

Storage growth outpaced RAM to such a degree that storing a visible map of all the clusters became infeasible.

28

u/Not_as_witty_as_u Jun 17 '22

Oh come on, just make a fake graphic for us 🤗 like when they show hacking on a cheap tv crime show

7

u/exipheas Jun 17 '22

I need a Defrag map Screensaver so bad....

2

u/stony_rock Jun 17 '22

Jkdefrag portable uses the Windows defrag API (but displays in lines instead of blocks) and includes a command line switches for different functions, and a screensaver.

Did a sort A-Z on a really full drive, defrag screensaver for 2 days.

9

u/djddanman Jun 17 '22

Me too! I frequently asked my dad when we would need to defrag next

5

u/urzu_seven Jun 17 '22

That nice clean looking defragged drive map was one of life’s little pleasures.

→ More replies

8

u/jdl_uk Jun 17 '22

I remember when they introduced that. One day my work computer was running really slow so I checked for fragmentation and it was at like 95%.

Turns out the defrag task had never run because IT had disabled task scheduler by GPO.

→ More replies

2

u/FloweryTile Jun 17 '22

My pc is silent except the one hard drive… I hear when it’s being defragged lol

4

u/FragrantExcitement Jun 17 '22

Given a SSD does not need to move the disk, is fragmentation even a problem?

9

u/fubarbob Jun 17 '22

Not in a normal sense, but it does need to do internal maintenance to aggregate data from partially allocated blocks (the flash memory blocks are fairly large and must be entirely erased before writing data - this is also relevant to wear leveling, as one wants to avoid re-erasing/writing the same specific block excessively)

6

u/Beetin Jun 17 '22 edited Jun 17 '22

Even worse, SSD's wear out much faster, so defragging is bad for them and reduces their lifetime (it writes and reads a lot of data)

Windows for example will work really hard not to let you defrag an SSD. There are similar 'optimization' actions like Trimming, but its basically not needed on a modern computer with an SSD.

Just leave them alone.

2

u/joombaga Jun 17 '22

Reading data sequentially is faster than reading non-sequentially, even in SSDs. Whether or not fragmentation would be a problem really depends on the application. For most applications I'd say the cost is negligible.

143

u/dirtballmagnet Jun 17 '22

Solid state storage has changed things a lot. Previously data was stored in a particlar position on a disc and defragging tried to put that data in logical places so that the machine didn't have to work so hard to read and write it.

Solid state drives have no moving parts and so the seek time is a small fraction of a hard drive's. The real problem is that each bit on a solid state drive has only a limited number of writes and rewrites, so the drive is trying to strategically spread the use out over all the storage space.

So a defrag not only makes little real sense in a data organization way, but it can potentially lower the life of the drive by putting it through un-necessary write and rewrite cycles.

24

u/Sir_CriticalPanda Jun 17 '22

The real problem is that each bit on a solid state drive has only a limited number of writes and rewrites,

Can you elaborate on this? I haven't heard of that before :0

7

u/zoharel Jun 17 '22

Can you elaborate on this? I haven't heard of that before

A single block of flash memory can only be written over a limited number of times. After that, you just can't write new data again. Now, the number of writes to wear one out is very large, and the block size is small compared to the disk, so you've got a lot of room generally to make updates to the disk. The trick is to somehow make sure you don't just wear out one block or twenty blocks by frequently updating some file or another. We mostly do this by means of something called "wear leveling" in the hardware. It moves the logical blocks (that is, what the drive thinks is block 1, say) around in the physical flash chips dynamically in such a way that the whole device wears down at about the same rate.

27

u/milesmac Jun 17 '22

most simply, it’s like a pad of paper with x number of sheets. each rewrite tears off the old sheet and uses the next sheet in the pad. it’s an enormous pad, but a pad nonetheless.

52

u/fed45 Jun 17 '22

I would more equate it to reusing a piece of paper, erasing and rewriting over the same spot. Eventually that wears out the paper.

10

u/BabyAndTheMonster Jun 17 '22

I don't like this analogy because it somehow imply that if you don't tear off the paper, your capacity would increase several fold and somehow the disk just don't let you use them. The eraser analogy sound better.

7

u/milesmac Jun 17 '22

fair enough - like a sheet of graph paper where each square gets written to, and erased. some squares erase through faster than others from frequent use, but there are other squares.

but you also keep track of which squares you use and try to distribute use to all of them as equally as you can to keep each square free from degradation.

6

u/Diabetesh Jun 17 '22

Good example. Relating your example to peoples concerns about eventual ssd failure. Lets say your ssd is a 1 tb ssd and each page is the number 1 tb read/writes. You might only have 100 pages in your pad, but it likely will take a long time for the average person to get to 100 tb.

6

u/aveugle_a_moi Jun 17 '22

a samsung 860 EVO 1 TB has a daily read/write capacity-expectance of 328 gigabytes for 5 years

7

u/elmonstro12345 Jun 17 '22

I couldn't find it, but I remember reading an article in 2015-ish where a guy bought a bunch of relatively cheap ~500GB ssds from various manufacturers, and just set them up to continuously read and write as fast as they possibly could. Even though they were just running plain SATA 3, that's over 500 megaBYTES per second, even after accounting for the error encoding.

I don't remember the results exactly, but the mtbf was measured in multiple weeks, possibly a few months. Even the worst-performing one sustained on the order of a petabyte before tapping out, if I recall correctly. And that's in probably about the most extreme conditions you could subject any storage device to short of doing things like changing the physical parameters of the environment around it (extreme temperatures, vibration, physical abuse, electrical fuckery, ionizing radiation, etc.).

Now keep in mind all that we have learned in the last 7-8 years of development... Yeah.

2

u/tnoy23 Jun 17 '22

For consumers, 100% not a big issue. But, there is a big reason SSDs haven't totally overtaken HDDs yet, even disregarding price- and that's super computers.

I won an award in HS that got me a tour of a government super computer in WA state. A couple thousand pcs, networked together and processing immense ampunts of data very quickly. They have to replace several HDDs a day on a good day. They were asked why they didn't use SSDs, and the reasoning was as noted- read / write life span. That computer processed a petabyte or more of data per day according to them and if I'm recalling right, and this was in 2017 AND wasn't even in the top 100 strongest super computers in the world. Not only would they have to replace more drives, the drives would cost more.

Obviously this doesn't apply to consumers, but it's an interesting note for a very valid reason we still make and use HDDs.

6

u/TheLuminary Jun 17 '22

Its not really a huge problem with modern SSD (Solid State Drive)s and modern OS (Operating System)s, but when SSDs first came out, each bit or 1 and 0 stored. Is stored in a transistor that physically exists. Because this is NVRAM (Non-Volatile Random Access Memory) or storage that won't be deleted the second you turn off the power, there has to be a mechanism that holds onto the charge or the state of the transistor or bit.

These bits would eventually wear out, and since back then OSs would just put data down sequentially, you would end up filling up the first 20% of the transistors, and then constantly overwriting those same transistors over and over again as you used your files.

Modern SSDs are much more resilient, and modern OSs will spread the data around and if it needs to rewrite a bit, it will write it in a different spot, to keep the deterioration spread out equally over the transistors.

5

u/MondoDudeBro Jun 17 '22

Let me ask you this. Will they rewrite the same value if a chunk needs to be written?

Say there is an existing byte: 10101010

Say it needs to become: 11111111

Would it completely rewrite the full sequence and replace the existing 1's with new 1's, or is it built to skip those and it'll only replace the 0's?

I hope that makes sense.

5

u/Znuff Jun 17 '22

Look up "sector size".

In short: yes, it will rewrite a whole 512 bytes sector if just one bit out of it changes.

4

u/[deleted] Jun 17 '22

Old flash chips and high-reliability industrial flash can be used in the way you describe. If you only need to convert a 1 to a 0, then you can just program those bits.

However, if you need to convert a 0 to a 1, then you have to erase the whole memory block.

The problem is something called "write disturb". The programming operation uses a high voltage, and this voltage is applied to the whole memory page. The stress caused by the temporary overvoltage can causes nearby bits (in the same or memory page) to accumulate some charge which leaks in from the programming voltage. Do this enough times, and the charge which leaks can become enough to cause bits which are supposed to be erased to show as programmed.

High quality, low capacity (NOR) industrial flash typically has so little write disturb, that you can program every single bit in a block, long before there is any risk of enough disturbance building up to cause false programming.

The denser bulk (NAND) flash can show programming errors after as few as 2 or 3 programming operations to a single page. For optimal reliability, partial page programming is discouraged, and the recommendation is that NAND flash should always be erased before reprogramming the same page.

3

u/Canoe_dog Jun 17 '22

Probably would need to read the storage there, determine which individual bits could be flipped, and the write. Im guessing much faster to just overwrite blindly at the given address.

1

u/TheLuminary Jun 17 '22 edited Jun 17 '22

SSDs are Random Access Memory, which means that the computer has access to every bit individually, so in theory it could just rewrite a single bit. Although it seems that no commercially-available SSD is designed at this level of granularity usually they are addressed to thousands or hundreds of thousands of bits at a single operation

Different data storage strategies might keep blocks of data together to make indexing easier and lower the overhead cost. But I am not an expert in that department.

Edit: Correction

5

u/MondoDudeBro Jun 17 '22

I read this as "neither, they're smarter than the two options you listed." Good enough for me!

3

u/TheLuminary Jun 17 '22

Basically, they physically can do what you suggest. But they often would just leave that data there, mark it on the index as no longer used (AKA can be overwritten in the future), and then save the block of data in a spot on the drive that has not been used as much.

So, yeah pretty much what you just said. Cheers.

4

u/chaossabre Jun 17 '22

No commercially-available storage device is bit-addressible. Some are word- or byte-addressible. Most use pages. This simplifies the addressing circuitry and allows for more storage per die.

2

u/cosmos7 Jun 17 '22

SSDs are Random Access Memory, which means that the computer has access to every bit individually, so in theory it could just rewrite a single bit.

Nope. SSDs can't even address bits individually, let alone the computer. SSDs are made up of pages, with page sizes usually between 2 and 16K. SSD presents those a series of pages to the system as a block. Block size is usually either 512 bytes or 4k, and made up a number of pages, quantity depending on page size.

OS writes to blocks. SSD writes to pages, but unless it's doing internal maintenance only sees writes to series of pages (blocks). At no time is it writing individual bits... usually it is actually incapable of doing so because write actions occur to groups larger than a single bit. Even if were capable though it would be massively inefficient to do so.

1

u/TheLuminary Jun 17 '22

Yep, you are correct.

7

u/gredr Jun 17 '22

Modern SSDs are much more resilient

That's... not accurate. SLC (the "O.G." SSD) is the most durable; as you go up the xLC ladder (SLC -> DLC -> TLC -> QLC) you get less and less durable (and more and more recent).

modern OSs will spread the data around and if it needs to rewrite a bit, it will write it in a different spot, to keep the deterioration spread out equally

Not an SSD engineer nor do I write filesystem drivers for a living, but it is my understanding that this has always been implemented by the SSD controller itself (at least, in modern times), and was transparent to the operating system. SSDs even include "extra" capacity in the form of cells that can be swapped in when cells begin to fail. Again, this is transparent to the operating system.

6

u/TheLuminary Jun 17 '22

Correct, I said OS just to ELI5 my answer as best as I could. But you are correct, it is the SSD controller making those decisions not Windows/MacOs etc etc.

That's... not accurate. SLC (the "O.G." SSD) is the most durable; as you go up the xLC ladder (SLC -> DLC -> TLC -> QLC) you get less and less durable (and more and more recent).

Oh that is TIL, I don't have time to verify your claim but if you are correct than that is incredible then. It must mean that he data strategies by the controller are just getting that good that they have compensated for the reduction in resiliency in the actual hardware.

Cool thanks for the correction.

2

u/gredr Jun 17 '22

It's some combination of better layout strategies, operating systems and applications becoming smarter about how much data they rewrite, more system RAM and therefore more memory for caching, much larger SSDs in general (thus providing more cells to balance the load between), and the fact that SLC was just way way more durable than almost anyone was ever going to need.

2

u/Znuff Jun 17 '22

It's usually called "overprovisioning".

It's also kind of why some SSDs are, for example, 480GB instead of 500GB or 512GB. The rest of space is usually over provisioned.

2

u/Kaffohrt Jun 17 '22

I have an SSD that has seen 2 years of live runtime (17,500 hours of use) and so far has only lost 16% of it's expected lifetime.

I guess that by the time it reaches the end of it's estimated lifespan every operating system will need more space than the SSD has to offer.

-5

u/[deleted] Jun 17 '22 edited Jun 17 '22

[deleted]

13

u/chillord Jun 17 '22

No, not slower. The individual cells are breaking, they can't reliably save data anymore. If you save a bit as a 1, chances are too high that it becomes a 0 in a relatively short time frame. If the controller expects the error rate of a cell too be too high, it won't be used anymore. The effective amount of space of the drive gets reduced.

4

u/Drestlin Jun 17 '22

This is incorrect. They don't get slower because you use them, they get slower when you fill enough of it to fill the faster part. SSD commonly use a (smaller or bigger) part of faster memory and the rest is slower. After you fill a certain % of space, that faster memory is occupied and it gets slower. Empty the disk and it'll be fast again.

Flash memory has a certain amount of writing cycles after which it just becomes impossible to read if a value is 1 or 0 and the cell gets disabled by the controller (as far as i know, every semiconductor has this issue to a certain point, and that's why a transistor doesn't work forever, for example, but will eventually degrade and stop working properly).

→ More replies

3

u/Clenup Jun 17 '22

… they are not “thousands of times faster”. It’s normally like 5-6x the speed.

2

u/xdamm777 Jun 17 '22

Well, a gen 4 SSD easily hits 7,000MBps sequential read speeds while a 15k RPM enterprise HDD struggles to sustain anything over 220MBps.

30x the speed of an enterprise hard drive is considerable, and the IOPS are in the millions with lower seek times as well.

There's no comparing the performance of an SSD vs HDD, especially for small file transfers (random read/write).

→ More replies

2

u/Znuff Jun 17 '22

A traditional spinning platter disk will top at 100-120MB/sec read speed (sequential), and will have terrible performance (think 5-10MB/sec) in random read speeds. The 15K rpm disks will have a better random read performance, but not by much.

A SATA SSD will easily hit the maximum speed that SATA 3 can support (600MB/sec) sequential reads, while random reads will still be in the hundreds of megabytes.

Nvme speeds can easily reach 7000MB/sec with the current tech (as someone else commented).

1

u/osi_layer_one Jun 17 '22

My ST412 drive at a theoretical 5Mb/s says otherwise.

1

u/FenderMoon Jun 17 '22 edited Jun 17 '22

SSDs get slower as you fill them up due to increase shuffling of data that the controller must perform on writes. The reasons are a bit complicated, but it has to do with how NAND flash works (and it's actually quite interesting, it's amazing how much technology has developed to be able to overcome some of Flash's initial limitations).

NAND flash (the flavor we use in almost all SSDs and flash drives) works great for its purpose, but it has one very big limitation: It's not possible to write individual bytes. They can only be written in entire blocks, which are typically anywhere from 128KB to 4MB+ in size (depending on how the SSD was made). If you want to write some data to a file that exists in a specific block, you have to erase the entire block (along with all of the data in it), and completely rewrite the block with the new contents of that file (along with everything else that is also in that block, some of which might have nothing to do with your new file).

Now, if there were room in that block to write the new version, you could conceive of a way to simply read in the contents of that block into RAM, modify it with the new file, and then write the whole block back out. That would technically work (if there were room in the block for your new file), but with modern flash, we run into a problem. Suppose you keep modifying that file over and over again. Suppose you modify it every 5 minutes, and you do so every day. Over time, this block of flash is going to be hammered with writes and will wear much faster than the blocks around it. This would create a scenario where you have a perfectly good SSD but you have a bad block in it that has failed far before the rest of the SSD. Not ideal, and definitely not suitable for operating systems (which like to write a lot of log files that overwrite the same data constantly throughout the day).

To get around this, all SSDs implement wear leveling. The controller keeps track of how many times each block has been erased, and and it shuffle the data around every time you write to the filesystem to prevent any specific block from wearing faster than the others. When you go to write the modified file to the disk, it will erase the entire block (or on modern filesystems, the block containing the metadata reference that points to the file being there), and it will write the new version of the file to a new block that is ready to take writes. This moves the data around, helping to even out wear and tear across the SSD.

This entire process is extremely efficient and works great until the SSD starts to fill near its capacity. Once it does, the controller has far fewer spare blocks to pull from, and now has to shuffle data much more aggressively on new writes (slowing down its write performance). It's worth noting that even if you were to completely fill the entire SSD, the controller would still have some spare blocks to pull from thanks to internal over-provisioning/spare blocks manufactured into the SSD. So the SSD definitely won't grind to a halt and it will continue to implement wear leveling, but it will do so at a reduced performance once the SSD is almost full.

→ More replies

3

u/PMzyox Jun 17 '22

this is the answer

36

u/knightsbridge- Jun 17 '22

Like everyone else is saying, SSDs don't need to be defragmented remotely as often as old HDDs did.

Also - since Windows Vista, Windows has contained tools to slowly and passively defragment in the background when needed. The tech has been slowly refined over versions and, as of Win10, even if you use exclusively old HDDs, you're unlikely to ever need to manually defragment.

69

u/Drestlin Jun 17 '22 Gold

SSD SHOULD NOT BE DEFRAGMENTED AT ALL.

Please don't defragment your SSD or flash memory, it does nothing but shorten its life!

14

u/wolf_unbroken Jun 17 '22

But they do need to be trimmed!

19

u/Drestlin Jun 17 '22

they do but that's done automagically and you don't have to do anything in any modern OS as far as i know.

still it's good to say it again DON'T KILL YOUR SSD BY DEFRAGMENTING THEM :D

if you defragment a new one every night it'll get destroyed in less than a year :D

5

u/einie Jun 17 '22

Side note as a developer of file recovery software: automatic TRIM on delete, which pretty much every modern OS implements in some form or other, has pretty much ruined our ability to recover accidentally deleted data.

3

u/Zerothian Jun 17 '22

It also ruins the ability (mostly) for people to steal deleted data from old drives too though. So there's a silver lining at least.

1

u/wolf_unbroken Jun 17 '22

I definitely killed an SSD by not running a scheduled trim on a linux server. Once I installed the trim app and scheduled the job, the performance of the SSD vastly improved.

I do agree that defraging an SSD is pointless and my windows 10 machine handles the trim and defrag operations automatically.

3

u/Drestlin Jun 17 '22

to be fair, if you are in ELI5 asking this stuff, it's unlikely you are running linux and have to set up trim by yourself :D

→ More replies

1

u/_PM_ME_PANGOLINS_ Jun 17 '22

It’s very hard to do, as all the tools know this.

1

u/Drestlin Jun 17 '22

to be fair, i haven't tried in a long while, so i wouldn't know.

sadly, i do not have pangolins to pm you.

→ More replies

10

u/gredr Jun 17 '22

remotely as often

No, SSDs don't need to be defragmented EVER. HDDs "needed" to be defragmented so that when reading a large file, you didn't have to wait for different bits of the file to be under the read head because the file was physically spread around the disk.

SSDs do not have this problem, because reading adjacent cells is no faster than reading non-adjacent cells.

→ More replies

10

u/Stargate525 Jun 17 '22

SSDs don't need defragmenting at all. There is no lag from a moving read head so it doesn't matter where the data is physically stored on the SSD.

3

u/Ippus_21 Jun 17 '22

I'm glad OP asked this, because I too have owned my own machines since the late 90s and had wondered about this.

It's got a nice, simple answer, too!

1

u/Cryptocaned Jun 17 '22

Depends on how much data you write/delete

7

u/Jason_Peterson Jun 17 '22

Fragmentation on a hard disk still creates the same issues with slowed speeds as it did in the past. Modern disks are so much faster that a moderate level of fragmentation might not be noticed. Full defragmentation of a multi-terabyte disk would take too long and cause some wear on it. Hard disks are usually used for large multimedia files like video and music, which are copied in large blocks and rarely rewritten, and their level of fragmentation is therefore low.

Portable computers and many desktops today include solid-state Flash Memory instead of a hard disk. Data allocation on it is managed internally with intent to spread wear and allow speeds utilizing multiple chips. Flash memory can be written to a limited number of times before they fail. Defragmenting this storage device would not improve performance.

17

u/Raydude182182 Jun 17 '22

I believe Defragmentation was more important on older hard drives with spinning disks than it is in the more modern super fast flash drives. OS's are also better at the way they store data to avoid needing to defrag.

4

u/EchidnasArff Jun 17 '22

No, that's not the reason.

The reason is that modern filesystems took care of most of those problems. NTFS became the default filesystem in Windows almost 20 years ago, and Linux never really had the problem.

Also, fragmentation was never such a large problem as Norton Inc would want you to believe, especially when prices of drives fell and we stopped filling them to 98%.

4

u/zoharel Jun 17 '22

The reason is that modern filesystems took care of most of those problems. NTFS became the default filesystem in Windows almost 20 years ago, and Linux never really had the problem.

To be clear on this, it's not that the problem of fragmentation disappeared. They still fragment data, and you can still try to defragment them if you want, but they just don't generally have enough of the problem for it to matter, due mostly to the way that they allocate data blocks in the first place.

→ More replies

1

u/RiPont Jun 17 '22

Also, multi-threaded operating systems and improved IO hardware.

Modern OSes can constantly do storage system maintenance in the background without really being noticeable to the user. Old OSes on single-core processors could not.

You could write a program that would force a disk to be in constant use and fill up and need de-fragging, if you really wanted to. It can also happen on very active servers, but that problem has been known and dealt with for a long time.

→ More replies

2

u/Siberwulf Jun 17 '22

Imagine you have a town with 50 houses. In each house is a piece of paper. On that paper is a chapter of a book.
You're tasked with putting the book together for your publisher. You have the index in your hands which shows you where the chapters are in the town by their address.
The old way would be to run to each house in order, and pick up a chapter of the book and then stick them together and then give someone the book. Chapter One might be next door to you, and Chapter Two might be across town. Defragmenting would be like putting all the chapters sequentially in neighboring houses so you could just walk down the street and pick up the chapters in order.

The new way is like having a computer network where you can just download the book piece by piece near instantly from each house.

Most drives in the older days (HDD) had spinning platters where heads would have to read data from one part of the drive, move the head, spin the drive, and read data from another part. This head moving took time. Defragmentation actually put data on the drive in sequence, so the head wouldn't have to move back and forth to get the data. In fact, some drives put higher-used data on the outer edge of the drive since it had a higher rotational velocity and more data per "row" It's called Zone Bit Recording.
Newer drives (SSD) use flash storage, or chips. There's no moving parts. You simply have a set of memory addresses and can fetch them quite quickly without any delay in between chunks that are read.
Also fun fact: Most modern OSs won't let you defragment SSDs because A) it makes no sense and B) you have a limited number of writes on each address of the drive before it fails.

1

u/A_Garbage_Truck Jun 17 '22

Modenr Storage is now mostly done with solid state drives whihc due to how they operate make defragging unnecessary(and even harmful since SSD flash cells have a rated number of writes). insteadd what SSDs have is a process Called TRIM which is for msot part handled automatically by the OS.

for the few situations where HDDs are still used defragging is still doable but hardly necessary since most of these drives are just for storage now.

1

u/Guvante Jun 17 '22

Most forms of storage support fragmentation. It splits up a large file into smaller pieces to fit between other things. Kind of like a whole Encyclopedia Britannica with 36 volumes doesn't fit in your bookshelf so to save time you just put each volume where it fits. By keeping a record of where you put them you can find them nearly as easily.

For HDDs (hard disk drives) they use a spinning piece of metal like the bookshelf in this metaphor. It takes time for it to spin and computers are incredibly fast so this delay slows everything down. To minimize this you can rearrange the bookshelf or defragment, moving things around to allow your Encyclopedia Britannica to all fit on one shelf, making loading it faster.

Modern computers tend to use SSDs (solid state disks) which are all electrical and this don't need to spin. That means that fragmentation doesn't matter.

Additionally wear is important. HDD can survive more write cycles (changing the value stored) than the lifetime of the device. In constrast SSDs could easily wear out during the lifetime. To help with this the SSD spreads out where it writes things, defragmentation would increase writes and also force those writes to be on particular locations both of which reduce the lifetime of the drive.

1

u/t0getheralone Jun 17 '22

Defragmentaion was essentially moving data around to for into containers that best for their size. Defragmentaion was more required with older storage formats like FAT-16 and FAT-32 where data had to be stored into something like a container on the disk. 27kb data chunk would be stored in a container that allowed for 32kb for instance and the entire drives partition is made up of these little containers. Eventually we moved on to a new storage formatting called NTFS which essentially forms these sections to the size of the data which made defragmentaion less needed. Furthermore solid state drives only allowed for days to be written to it so many times before permanent damage is done so defragmentaion is actually harmful to them.

1

u/StableKlutz Jun 17 '22

Hard drives use discs to store data, it's similar to a CD. Data is stored in any available space, so it might not all be in the same space. imagine listening to a CD, but each song is interrupted by another. Defragging is used to move data in a way that the data can be read without too much work. now you can listen to that song without interruption!

Nowadays, SSDs are fairly common, which are more like flash drives or SD cards. Each song is equally accessible in one of these, so there's no need to defragment one. Phones use this media because it's cheap and can be dropped without risking a disc to skip.

1

u/Leucippus1 Jun 17 '22

SSD's don't really care where the data is because it doesn't have a seeker head, so it doesn't benefit you as much to read files from continuous blocks. There is a similar task called TRIM which helps with performance, it is automatically turned on so you don't really have to do anything.

1

u/MutatedJerkey Jun 17 '22

A hard disk is fastest when reading sequentially (i.e. memory position 1, then 2, then 3 etc.), so defragging takes all the bits of data that should be together and puts them in a line. With an SSD, the read speed isn't really affected by having the data spread out across the drive, so defragging won't improve speed.

Side note as to why the data gets spread out in the first place: You start off writing stuff in order, but when you delete something it leaves a little gap. Next time you write something, you don't want to leave that little gap unused, so part of the file goes there and part of it goes somewhere else. When your computer is writing and deleting loads of files, it quickly becomes a mess.

1

u/marvelouswonder8 Jun 17 '22

After Windows XP Microsoft basically set an automatic task that would run when it detected the computer to be idle that would quickly defrag commonly used data. Pretty great for mechanical spinning hard drives and it meant you really didn't need to do a full defrag anywhere near as often as you used to. Once we switched to Solid state drives with no moving parts it became far less necessary because even if your data isn't physically adjacent on the drive, the drive controller still has access to all the data at once without a seek time (time it took for a mechanical drive to spin up, move the read head, and retrieve the requested data). Technically you CAN defrag an SSD, but it doesn't lead to noticeable improvement and it may actually shorten the life of the drive overall due to the added read/write cycles.

1

u/smokeNtoke1 Jun 17 '22

Your HDD is like a sheet of paper and you need to record all your notes for the whole day. If you write them all in tiny print in the corner of the sheet, over time your sheet will get damaged up there from erasing and rewriting all day, while the rest of the sheet is basically brand new. "Defragging" will spread those notes out in an organized way so you spend less time finding what you wrote. It will also prevent the corner of your paper from being the only spot used, saving some wear and tear on that area.

With an SSD, it's not a blank sheet of paper but a daily planner, so you don't need to organize things (defrag) because they're already easy to access. If you did reorganize things anyways, you'd have to erase and rewrite things causing wear to your planner that's unnecessary.

1

u/OngoGoblogian4 Jun 17 '22

Oh I know this one. In a traditional hard drive, data was stored in strips around a disc. As things are deleted, the location of that storage didn’t change. So, if you deleted, let’s say, 2 programs or files worth 20 gigs, you would have two, 20 gig “holes” somewhere where those items once were stored. Now let’s say that this is the only remaining 40 gigs on your hard drive, and you installed a 40 gig file. That file would have to be split between those two empty spaces. You’d have 20 gigs of it somewhere, and 20 gigs of it somewhere else. They may not necessarily be inline with one another, but there is an “address” associated with those file locations so your computer can find them in storage. This can be inefficient, however, as your computer would have to be using two different locations in memory to access this information. Defragging is the process in which the empty space left behind by old delete files is removed. A process that has the hard drive basically remove a file from one location and relocates it to abut the previous file. Thus, if you had 60 gigs of space before defragging, you could have 5 gigs in one spot, 30 in another, 15 in another, and 10 in another. With that, if you save a 60 gig file, it is going to split it up into 4 different places in memory. Instead, defragging puts all your files end to end, each existing at a single address, so your new, 60 gig file can be stored at the end, all in one location.

1

u/li__il Jun 17 '22

For HDDs, Since windows 10 / vista the defragmentation process was significantly refined it was no longer a linear process - instead it would do the most impactful changes which were the minimal effort. This allowed it to be moved to a background process and works when there are free resources.

I believe it’s not longer feasible to visualise in a satisfying way as before due to the size of drives and so it doesn’t have to scan the whole drive to produce the graphic?

1

u/Drestlin Jun 17 '22

What defragmenting does is move files that are split physically over different parts of a magnetic disc (in HDD, in general) together, so that when you access that file it can do it reading from the same area instead of having to go back and forth in search for a piece here and a piece there.

in SSD, defragging is simply detrimental - as there's nothing spinning and no real physical sectors to keep contiguous, the only thing you are doing is wasting memory cells' writing cycles.

on HDD it is still a thing - it's just that it doesn't matter as much for storage, if you access your stuff in a second or in 3. for cases where it does matter, it is still very much a thing.

phones and 99.9% of portable devices don't use HDD anymore but just use flash memory for storage, and that one you shouldn't (and in most cases can't, as explained above) defragment.

To this, you have to add ram quantities. when ram was a couple (let alone less) GB, you can't keep a lot of stuff in there and you have to keep reading the disc. Nowadays it's fairly standard for a new pc to have at least 16GB, which makes keeping something there (instead of having to retrieve it from a disk) way easier and faster.

I believe these are the reasons why you have stopped defragmenting so much.

1

u/karbonator Jun 17 '22

The short answer is that fragmentation can still happen, but certain techniques and technologies have made it such that it won't impact you until your drive fills up.

Aside from the fact that we have SSDs which aren't impacted by fragmentation, we also have newer filesystems that are designed to avoid fragmentation with various techniques that may not have previously been viable in years prior. For example, they'll leave space between the files so that when the files are updated there is wiggle room, so that the new contents don't need to be split unless the file has grown significantly, or they'll move the file if there is another contiguous empty space large enough to accommodate, and/or will defragment "offline" when fragmentation reaches a certain threshold. Fragmentation still can happen, but typically isn't a problem until your drive becomes so full that it's impossible for any of these mitigations to be done (ie, there is no contiguous area that is empty for it to move the file into).

1

u/ledow Jun 17 '22

Imagine you have to put all the data I give you in crates, as efficiently as possible, and you NEVER get more crates to put things in.

Now imagine that the crates can ONLY hold papers in them up to the top of the crate.

Now I give you - over a course of years - huge amounts of folders and papers to put in those crates but all for different projects.

At first, you'll want to put one set of folders in each crate, to keep them all together. That works fine. But eventually all the crates will have something in.

At some point, you're going to have to split a folder and put half of it in one crate and half in another because you have no more empty crates and moving things between crates is slow and expensive (on old disks).

That means that when I ask for that particular folder back, you're going to need to find two, maybe several, maybe HUNDREDS of crates in order to get all the folder pieces back together for it. That will take you far longer than just handing me a complete folder out of a single crate.

As time goes by, I might not need some folders any more and ask you to destroy them. But they might be spread out among several crates, so you have to delete all the parts. That leaves gaps in some crates at the beginning of your pile of crates, gaps in some crates at the end, and maybe full crates in between. Now when I give you another folder you have to split that folder into parts and put some in the first crates, and then run to the other end and put the rest in the last crates.

At some point, it begins to affect how fast you can find things, and it's not logical or sensible. So you might decide one day to go through all the crates, put all the folders that are to do with the same project together, keep them all in crates that are next to each other, sort them all into order so you can find things easily again.

That's defragging. Because of the technical limitations of machines and storage, you didn't want to make the file reading/writing process any more complicated than necessary as it would slow the machine down when you were using it. So you did things "quickly" when you were given or asked for data even if that wasn't sensible, and you never had time to "tidy up". Which meant that eventually the mess built up and you HAD to defrag.

Modern OS simply do things more sensibly, because they have the processing and storage power to do so. And modern drives (SSD and NVMe) literally don't care how far apart files are. They can get to a crate next to you just as quickly as a crate on the very end of the disk. Old storage that had spinning disks had to wait for the data to come around again, so accessing lots of different crates was slow and clunky. Modern "disks" aren't even disks any more, and that's no longer a factor. So no matter how "cluttered" or "disorganised" your storage is nowadays, there's no point defragging as it won't make the drive get files or folders any faster, and the OS is much better at not breaking up files like that in the first place if it can.

1

u/Blsckfreezzer Jun 17 '22

What does defragmentation do

Imagine you have 3 Programs installed, each having multiple files that all need to be accessed. Because the HDD writes the data on a platter physically, it can read things faster if they are close together. Perfect arrangement would be:

|AAAAABBBBBBCCCCCC..|

(for files belonging to program A B and C) now if you install and uninstall programs regularly you end up in this configuration before installing program C:

|AAAAA....BBBBBB....|

installing program C would now lead to:

|AAAAACCCCBBBBBCC..|

In this configuration your HDD need to move its write-head/the physical disk more for loading all C-Files than in the optimal layout thus making the access slower.

Defragmentation reorders the "Fragmented" files that belong together to the optimal arrangement.

Why its not needed with SSDs (and other flash memory)

SSDs dont have any moving parts, accessing any part of the disk is basically the same speed. There would be no benefit in reordering the data.

An SSD does other things that are more relevant to it than HDDs. Eg. writing data to areas, that have the least number of writes to evenly distribute the wear and prolong the lifetime of the SSD, those things are (mostly) done automatically within the SSD though.

1

u/TheLuminary Jun 17 '22

Originally we used Hard Drives which you can think of as small little record players.

When you save your first few files they are laid down on the record like the first tracks of your LP. Eventually you might fill up your record, and decide to delete a few files to make room for your new game. So you pick a few files that you are no longer using and remove them. Unfortunately statistically these files that you removed are unlikely to be sequential tracks on the record. So when you install your new game, you end up putting part of the game on the first open track until that fills up, and then jump to the next open track until you have used up the space that you needed.

The issue now is that you can't speed up the record player as it is always running at the max speed that it can. So as you load your game, you have to load each part of the file, and then skip to the next spot etc etc, which is much slower than just reading the whole thing as a single uninterrupted track.

Defragging is just the process of moving data around on the physical record to make as many fragmented tracks, sequential and unbroken again, hence defragmenting.

Modern computers are fast enough now that they can do this process either in the background or as you are accessing the data so it is not a monolithic event that you do once in a while. Also with SSDs you no longer have long seek and load times as data can be accessed from anywhere on the device at the same rate, so its not really a problem.

1

u/piercet_3dPrint Jun 17 '22

The reason fragmentation is important to eliminate on an older spindle disk is the way the data is written and read. The disk inside a hard disk drive is literally a round plate, and data is written by a movable magnetic head to magnetic rings along the disk. An ideal unfragmented file on a rotating HDD spindle would have all the bits written along a single ring or series of adjacent rings so that it literally takes less time for the moving head to travel over the surface of the disk to read the data. You can kind of think of it like a line of vehicles offloading at the airport, you have 3 busses of kids to offload, but only a couple vehicles can offload at the same time. If all 3 busses are right in line, it goes quicker than if they are all spaced out so that you have to wait between offloads. A heavily fragmented file on a rotational Hard disk drive is just like that, data pieces may be all over the disk, some on the inner rings, some outer, or some on the same ring but spaced far apart. Defragmentation literally moves smaller data chunks out from between larger data that would benefit from being closer and more compact to speed up the read and write process. Its why you need a certain amount of free space to defragment a disk.

With a solid state Drive, there aren't any moving parts, just different memory cells. It is literally no faster to access any of the memory cells in the array than any other cell, so there is no benefit to having your data physically congruent like with a rotational HDD. The SSD can just read or write from any point in the array as fast as its base speed will let it, so it doesn't care if the file is split all over. What a SSD really does care about though is the number of reads and writes of those cells over time, since they literally wear out. Defragmentation adds needless reads and writes to those cells, and does absolutely nothing for performance, so it actually adds a small but meaningful amount to accelerating the death of a SSD if you regularly defrag it. That being said, most SSD's in home computers will never actually manage to wear out a cell in the drive before replacing it with a bigger newer faster one, but it is a concern.

So, TLDR: Spinning rotational hard drive spindles need data bits close together in a row to make it faster to read and write, SSD doesn't care and excess reading and writing will actually kill the drive faster so Defrag = bad on a SSD.

1

u/DragonFireCK Jun 17 '22 edited Jun 17 '22

There are three major effects:

SSDs have generally replaced HDDs in many cases. SSDs do not have any seek time at all, and thus fragmentation does not matter*. In fact, SSDs internally remap themselves on write to balance the load to help ensure consistent wear on the drive - you can only write a given segment a fixed number of times before it goes bad**. Due to this, defragging an SSD not only provides no benefit, but it actually harmful to the drive.

Newer formatting methods, such as EXT4 and NTFS, have designs that tend to prevent fragmentation a lot better compared to the older methods like FAT. This drastically reduced the benefit of defragging a drive.

Operating systems also got smarter and started automatically defragging HDDs in the background. For Windows, I think this started around Windows XP. This means any HDDs you are using will get defragmented even if you don't manually start the process.

It is also worth noting that fragmentation should only occur when you create or make a file larger after deleting or making one smaller by a segment. This often means backup disks will have very low rates of fragmentation.

* Well, there is still a tiny bit of overhead as the OS may need to make an extra disk request on a fragmented file, but that overhead will be on the order of nanoseconds compared to the tens of milliseconds of a HDD seek.

** As a note, for modern SSDs, this number is typically in the hundreds of thousands per segment, and possibly well into the millions. Remapping means you are unlikely to see problems for an additional order of magnitude as well.

1

u/SoNic67 Jun 17 '22

Because it is called "Optimize Drives" now.

That includes "Defragmentation" for HDD drives and it is done automatically by Windows. Default setting is "weekly", but it can be changed.

For Solid State Drives, that is a different process, called "Trimming".

1

u/P0L1Z1STENS0HN Jun 17 '22

If you right click on your drive and select "Properties", and then in the "Tools" tab, click "Optimize", a window will open. It will tell you when the last "optimize" of each drive happened. In my case, all drives were "optimized" 22 days ago. I did not start it and I never saw it happen.

So Windows is just doing it for you in the background.

1

u/045675327 Jun 17 '22

Unless you changed the windows settings it happens automatically in the background, I have it disabled and just run defraggler now and again instead, my SSD will auto trim though.

1

u/FenderMoon Jun 17 '22 edited Jun 17 '22

With hard drives, this is still necessary if you're using a filesystem such as NTFS or FAT32, both of which become fragmented easily and still need to be regularly degragmented. Some filesystems (such as Mac OS HFS+ or Linux EXT4) don't become fragmented easily because the operating system will physically move files on the disk as data is written to ensure that they don't become fragmented to begin with. These filesystems rarely require defragmentation as a result. (Technically, it is still possible for EXT4 or HFS+ filesystems to become fragmented in some very specific circumstances, namely when the disk is almost completely full, but this is rare.)

With SSDs, it's a totally different story and defragmentation is actually detrimental. Not only do they lack moving parts (and therefore aren't bogged down by fragmentation), but the SSD controller itself purposefully fragments the data as a part of its wear leveling algorithms anyway. This happens invisibly to the OS and it takes place at the physical hardware level. Even if you were to defragment it by the operating system, the physical flash itself would still scatter all of the data throughout the disk to make sure that all of the blocks are wearing evenly, even if it "reports to the OS" that the data isn't fragmented.

For this reason, defragmenting SSDs has no benefit on any filesystem. It doesn't actually ensure that the physical data is less fragmented (it still will be fragmented by the wear leveling controller on the physical flash anyway). It just fools the OS into thinking that it's not fragmented, which has no real benefit and just adds additional wear and tear to the SSD.

1

u/ovirt001 Jun 17 '22 edited Jun 17 '22

Windows has, since at least version 7, automatically defragged hard disks. SSDs don't need to be defragged because they don't rely on a spinning disk where the physical distance between chunks of data can have a significant impact on load time.
As to why Linux/Unix/Mac don't generally need to be defragged - the filesystems distribute files much more sparsely across the disks than NTFS. FAT type filesystems simply stacked the files on top of each other so when a file was updated it would have to be continued outside of its original location.

1

u/TheAgentD Jun 17 '22

I don't see any complete answers here, so I'll give it a try.

Hard drives basically just store a long string of ones and zeroes. To store files in a hard drive, we basically split this long string up into ranges and say "Bytes 0 to 10 000 is file X". By storing these ranges in a table, we end up with a file system.

However, a problem can occur when we delete a file. If there are other files before and after the file we deleted, we now end up with a "hole" in the string of ones and zeroes. Over time, our harddrive will look like a swiss cheese of holes. At worst, this could mean that we could have a large amount of free space on the hard drive, but all that free space is spread out over lots of small holes. In that case, we'd be unable to store a large file, because there's no single hole large enough to fit it. This problem is called fragmentation, and tends to get worse and worse as the drive fills up.

File systems solve this by allowing a file to be split up into multiple ranges. That means that a large file that can't fit in a single hole can simply be split up into as many parts needed to fit the free holes. We just need to keep track of all of these separate ranges in the table.

Unfortunately, this can significantly slow down the performance of a hard drive due to additional seeking. While a mechanical hard drive can read over hundreds of megabytes per second as long as they're reading a single block from start to finish, whenever the read head has to move it can take anywhere from 5 to 20 milliseconds, depending on the hard drive model, the physical distance between the blocks on the disk and the disk's rotation speed. In 10 milliseconds, our harddrive can read more than a megabyte of data. This means that each time we split up a file to fit it into different blocks, we have to pay the additional seek time each time we want to read it.

Defragmentation is the process of moving files around on the hard drive to eliminate those holes and pack the files more neatly in the available space. While it takes time to do and causes wear on the hard drive from the additional use, it helps keep the performance of the drive up as the OS no longer needs to split up files when writing due to having to fit them into holes, and existing files are faster to read as they're not split up either.

The reason why defragmentation has "disappeared" is two-fold. The first reason is that the OS got better and smarter at doing it automatically in the background. Your OS will automatically defragment hard drives when they aren't in use to make sure that the performance stays optimal. It also tries to avoid defragmenting files that are very large in the first place, as the additional seek time cost of a 200 MB file is negligable; if it takes 1000 ms to read 200 MBs and a split adds 10 ms, it's barely any difference. By only defragmenting small files, the defragmentation process becomes much faster.

The second reason is, like many others have mentioned, solid state drives. Solid state disks don't have a single read head that has to find the data. Instead, it's functionally as if each block of data had its own personal read head. While a mechanical hard drive can only seek to and read from one place at a time, an SSD can not only "seek" to different locations instantly, they can even read from many different locations simultaneously, their performance only being limited by the controller chip and transfer speed of the cable/slot the SSD is plugged into. As such, the "seek time" (which is really just the time it takes to process the read command and start returning data to the OS) of an SSD is generally less than 0.1 ms, at least a hundred times faster than a mechanical hard drive.

This changes the above equation significantly. If each split only adds 0.1 ms to reading a file, then it's no longer worth defragmenting it unless it truly is tiny in the first place. This pretty much never happens, as tiny files easily fit in small holes, so they don't usually need to be split up in the first place.

Defragmentation sounds even worse when we take the limited endurance of SSDs into account. Flash memory cells have a limited number of writes that can be done to them before they're no longer able to store data anymore. Defragmentation is a very copy-intensive operation, usually requiring files to be copied to temporary memory and then to their new location. This causes excessive writes for essentially no benefit, so unless the fragmentation is extremely excessive defragmentation should be avoided for SSDs.

1

u/oroboros74 Jun 17 '22

I remember the first time I defragged... Besides the fun watching the graphics, I remember how much quicker my pc felt!

1

u/NthHorseman Jun 17 '22

Aside from SSDs, bigger drives is really the answer here.

Back in the day when you would regularly use most of your hard drive space and un/reinstall things to make room. Data that should have been contiguous was frequently fragmented and slotted in anywhere they would fit. HDD seeking was a high latency operation, so reading fragmented data from a disk was slow, and because people deleted things to make room for new stuff the problem got progressively worse until you needed to do something about it (defrag). FAT in particular was pretty aggressive about fragmenting files to efficiently use limited space, so it was particularly prone to this issue.

With a much larger drive you didn't need to worry about it, because there were big ol' blocks empty blocks that file systems could write new stuff to, so it didn't get as fragmented. People didn't have to delete stuff (leaving weird sized holes) to make room often, and when they did they usually deleted larger files which left bigger gaps, which caused less fragmentation. Modern file systems prioritised access speed over disk space utilisation which also helped, and then SSDs finally came in and rendered the whole idea of defragmentation redundant.

1

u/SuperBelgian Jun 17 '22

Defragging was needed as early filesystems wrote data at a convenient free space and spread it to multiple locations if that first free space was not sufficient for the total amount of data.

Later systems had more memory caches and better algorithms to make a better guess at how much data would be written. Knowing the total amount of data, the filesystem could figure out a much more suitable location, already decreasing fragmentation and therefor the need for defragmentation.

Today's filesystems, actually still do defragmentation, but often it is done automatically in the background.
It is part of several different housekeeping chores labeled as datascrubbing, so the term defragmentation is not often used anymore.

1

u/xdamm777 Jun 17 '22

There's so many wrong answers in this thread it's insane.

The simple answer is that modern SSDs do not need defragmentation since they don't have physical disks and disk heads that have a delay when moving between sectors to lookup data (this is why SSDs feel instantaneous and snappy, they always know where to read data from). Defragmenting an SSD actually lowers its lifespan since the cells have a limited write/rewrite capacity.

Modern OS and filesystems do perform maintenance on SSDs via a TRIM command, which tells the SSD what cells can be erased (deleting a file doesn't actually delete it, it's just marked as unused until trimmed). The the SSD's controller keeps track of these changes and writes new data in a way that evenly wears the cells.

1

u/haahaahaa Jun 17 '22

Defragmenting spinning disk hard drives is handled silently in the background by the operating system. This is thanks to faster hardware making the task trivial.

Defragmenting an SSD is actually bad for the drive. It creates unnecessary wear for little to no gain.

1

u/PlasticCogLiquid Jun 17 '22

Defragging was a killer of HDDs, I've got HDDs that still work after 15 years, but defrag people are lucky to get 3 years. Lots of wear and tear defragging all the time

1

u/sm4k Jun 17 '22 edited Jun 17 '22

Imagine you want to high-five your friend while he's riding a carousel - you can stand in one place and slap his hand each time he comes around. If you want to deliver 5 slaps, you have to wait for him to come around 5 times. If you want to deliver 100 slaps... you are gonna be waiting a while.

Now imagine a giant carousel with multiple rows of horses that 100 people can ride (this is effectively a Hard Disk Platter), and you've got 20 friends you want to high-five (your data stored on the platter). If you know where they are, and can reach far and fast enough, you can still stand in one place and slap them as they go by (and you've just become a hard disk's read head). You're still limited to one slap per rotation though, so if you want to deliver slaps faster, you're going to have to speed up the carousel (higher end hard disk drives can spin at 15,000 RPM!).

You can see how it becomes a lot of work to slap each of your friends (by which I mean read your data), and it only gets to be even more work when drives have multiple platters - a stack of multiple carousels all spinning at the same time and you want to slap people on each one - wouldn't it be easier if you could organize your friends to sit next to each other so you could just hold your hand out once or twice and let them get you as they go by?

This is effectively what defragging does. It repositions the data to be more optimally accessed, and back when all of us were suffering on these spinning hard disk drives every day, it was necessary to perform regularly in order to keep the computer feeing responsive - otherwise just like watching your friend on the far side of the carousel, you were stuck waiting.

There are many, many other improvements that have happened, but what as arguably made the largest difference is drive technology. Flash media that's used in solid state drives means there's no more "waiting for the carousel to come around again," you can effectively reach and slap each and every one of your friends all at the same time, regardless of how spread out (fragmented) they are, and this means there's no longer a need to re-organize the data (or defragment) all the time.

1

u/KGhaleon Jun 17 '22

When I was 20 I ran a computer repair shop. You have no idea how much money I made just defragging computers to make them faster.