Discussion

What makes Intel Optane stand out

FpUser: I feel sorry about the situation. From my perspective Optane was a godsend for databases. I was contemplating building a system. Could've been a pinnacle of vertical scalability for cheap.

walterbell: [delayed]

amelius: For a good technical explanation at the physical level of a memory cell:https://pcper.com/2017/06/how-3d-xpoint-phase-change-memory-...

walterbell: Related: "High-bandwidth flash progress and future" (15 comments), https://news.ycombinator.com/item?id=46700384

hbogert: It stands out, because it didn't sell. Which is weird because there were some pretty big pros about using them. The latency for updating 1 byte was crazy good. Some databases or journals for something like zfs really benefited from this.

bombcar: It feels like everyone figured out what to do with them and how just about when they stopped making them.

timschmidt: Same for the Larabee / Knights architecture. Would sure be fun to play around with a 500 core Knights CPU with a couple TB of optane for LLM inference.Intel's got an amazing record of axing projects as soon as they've done the hard work of building an ecosystem.

trollbridge: Yeah, I've wondered if we might see a revival of this kind of technology.

ece: Fabs are expensive and all, but maybe running a right-sized fab could have still been profitable at making optane for low-latency work that it was so good at. Even moreso with ram prices as they are.

ksec: >Which is weird....It isn't weird at all. I would be surprised if it ever succeed in the first place.Cost was way too high. Intel not sharing the tech with others other than Micron. Micron wasn't committed to it either, and since unused capacity at the Fab was paid by Intel regardless they dont care. No long term solution or strategy to bring cost down. Neither Intel or Micron have a vision on this. No one wanted another Intel only tech lock in. And despite the high price, it barely made any profits per unit compared to NAND and DRAM which was at the time making historic high profits. Once the NAND and DRAM cycle went down again cost / performance on Optane wasn't as attractive. Samsung even made some form of SLC NAND that performs similar to Optane but cheaper, and even they end up stopped developing for it due to lack of interest.

zozbot234: > 500 coreThe newest fully E-core based Xeon CPUs have reached that figure by now, at least in dual-socket configs.

dangoodmanUT: Optane was crazy good tech, it way just too expensive at the time for mass adoption, but the benefits were so good.Looking at those charts, besides the DWPD it feels like normal NVMe has mostly caught up. I occassionally wonder where a gen 7/8(?) optane would be today if it caught on, it'd probably be nuts.

zozbot234: > besides the DWPD it feels like normal NVMe has mostly caught up.So what you mean is that on the most important metric of them all for many workloads, Flash-based NVMe has not caught up at all. When you run a write heavy workload on storage with a limited DWPD (including heavy swapping from RAM) higher performance actually hurts your durability.

newsclues: in an era of shortages, if there was an optane factory today ready to print money...

walterbell: [delayed]

gigatexal: I’m still sad they discontinued them. What’s the alternative now does anything come close?

readitalready: These are absolute beasts for database servers, and definitely needs to make a comeback.They suck for large sequential file access, but incredible for small random access: databases.

gozzoo: Maybe we can also mention the HP Memristor here.

cogman10: IMO, the reason they didn't sell is the ideal usage for them is pairing them with some slow spinning disks. The issue Optane had is that SSD capacity grew dramatically while the price plummeted. The difference between Optane and SSDs was too small. Especially since the M.2 standard proliferated and SSDs took advantage of PCI-E performance.I believe Optane retained a performance advantage (and I think even today it's still faster than the best SSDs) but SSDs remain good enough and fast enough while being a lot cheaper.The ideal usage of optane was as a ZIL in ZFS.

zozbot234: > (~$1/GB)Isn't that actually crazy good, even insane value for the performance and DWPD you get with Optane, especially with DRAM being ~$15/GB or so? I don't think ~$1/GB NAND is anywhere that good on durability, even if the raw performance is quite possibly higher.

rkagerer: My understanding is Optane is still unbeaten when it comes to latency. Has anyone examined its use as an OS volume, compared to today's leading SSD's? I know the throughput won't be as high, but in my experience that's not as important to how responsive your machine feels as latency.

speedgoose: I configured a hetzner ax101 bare metal server with a 480GB 3d xpoint ssd some years ago. It’s used as the boot volume and it seems fast despite the server being heavily over provisioned, but I can’t really compare because I don’t have a baseline without.

rkagerer: Before people claim it doesn't matter due to OS write buffering, I should point out a) today's bloated software and the many-layered, abstracted I/O stack it's built on tends to issue lots of unnecessary flushes, b) read latency is just as important as write (if not moreso) to how responsive your OS feels, particularly if the whole thing doesn't fit in (or preload to) memory.

pgwalsh: Sure, they were expensive but they have great endurance and sustained read and write speeds. I use one in my car for camera recordings. I had gone through several other drives but this one has been going on 3 or 4 years now without issue. I have a couple more in use too. It's a shame this tech is going away because it's excellent.

exmadscientist: Around the time of Optane's discontinuation, the rumor mill was saying that the real reason it got the axe was that it couldn't be shrunk any, so its costs would never go down. Does anyone know if that's true? I never heard anything solid, but it made a lot of sense given what we know about Optane's fab process.And if no shrink was possible, is that because it was (a) possible but too hard; (b) known blocks to a die shrink; or (c) execs didn't want to pay to find out?

hamdingers: > Has anyone examined its use as an OS volume, compared to today's leading SSD's?Late last year I switched from a 1.5tb Optane 905P to a 4tb WD Blue SN5000 NVMe drive in a gaming machine and saw improved load times, which makes sense given the read and write speeds are ~double. No observable difference otherwise.I'm sure that's not the use case you were looking for. I could probably tease out the difference in latency with benchmarks but that's not how I use the computer.The 905P is now in service as an SSD cache for a large media server and that came with a big performance boost but the baseline I'm comparing to is just spinning drives.

rkagerer: Thanks, that's helpful real-world feedback (not that I wouldn't also be interested in some synthetic benchmark comparisons from someone else).

exmadscientist: > The ideal usage of optane was as a ZIL in ZFS.It was also the best boot drive money could buy. Still is, I think, though other comments in the thread ask how it compares against today's best, which I'd also love to see.

gozzoo: This concept was very popular back in the days when computers used to boot from HDD, but now it doesn't make much sense. I wouldn't notice If my laptop boots for 5 sec instead of 10.

amluto: Intel did a spectacularly poor job with the ecosystem around the memory cells. They made two plays, and both were flops.1. “Optane” in DIMM form factor. This targeted (I think) two markets. First, use as slower but cheaper and higher density volatile RAM. There was actual demand — various caching workloads, for example, wanted hundreds of GB or even multiple TB in one server, and Optane was a route to get there. But the machines and DIMMs never really became available. Then there was the idea of using Optane DIMMs as persistent storage. This was always tricky because the DDR interface wasn’t meant for this, and Intel also seems to have a lot of legacy tech in the way (their caching system and memory controller) and, for whatever reason, they seem to be barely capable of improving their own technology. They had multiple serious false starts in the space (a power-supply-early-warning scheme using NMI or MCE to idle the system, a horrible platform-specific register to poke to ask the memory controller to kindly flush itself, and the stillborn PCOMMIT instruction).2. Very nice NVMe devices. I think this was more of a failure of marketing. If they had marketed a line of SSDs that, coupled with an appropriate filesystem, could give 99% fsync latency of 5 microseconds and they had marketed this, I bet people would have paid. But they did nothing of the sort — instead they just threw around the term “Optane” inconsistently.These days one could build a PCM-backed CXL-connected memory mapped drive, and the performance might be awesome. Heck, I bet it wouldn’t be too hard to get a GPU to stream weights directly off such a device at NVLink-like speeds. Maybe Intel should try it.

orion138: One of the many problems was trying to limit the use of Optane to Intel devices. They should have manufactured and sold Optane memory and let other players build on top of it at a low level.

amluto: > Optane memoryWhich “Optane memory”? The NVMe product always worked on non-Intel. The NVDIMM products that I played with only ever worked on a very small set of rather specialized Intel platforms. I bet AMD could have supported them about as easily as Intel, and Intel barely ever managed to support them.

wtallis: The consumer "Optane memory" products were a combination of NVMe and Intel's proprietary caching software, the latter of which was locked to Intel's platforms. They also did two generations of hybrid Optane+QLC drives that only worked on certain Intel platforms, because they ran a PCIe x2+x2 pair of links over a slot normally used for a single X2 or x4 link.Yes, the pure-Optane consumer "Optane memory" products were at a hardware level just small, fast NVMe drives that could be use anywhere, but they were never marketed that way.

twotwotwo: One potential application I briefly had hope for was really good power loss protection in front of a conventional Flash SSD. You only need a little compared to the overall SSD capacity to be able to correctly report the write was persisted, and it's always running, so there's less of a 'will PLP work when we really need it?' question. (Maybe there's some use as a read cache too? Host RAM's probably better for that, though.) It's going to be rewritten lots of times, but it's supposed to be ready for that.It seems like there's a very small window, commercially, for new persistent memories. Flash throughput scales really cost-efficiently, and a lot is already built around dealing with the tens-of-microseconds latencies (or worse--networked block storage!). Read latencies you can cache your way out of, and writers can either accept commit latency or play it a little fast and loose (count a replicated write as safe enough or...just not be fully safe). You have to improve on Flash by enough to make it worth the leap while remaining cheaper than other approaches to the same problem, and you have to be confident enough in pulling it off to invest a ton up front. Not easy!

zozbot234: > It seems like there's a very small window, commercially, for new persistent memories. Flash throughput scales really cost-efficientlyFlash is no bueno for write-heavy workloads, and the random-access R/W performance is meh compared to Optane. MLC and SLC have better durability and performance, but still very mid.

hedora: I think it was killed primarily because the DIMM version had a terrible programming API. There was no way to pin a cache line, update it and flush, so no existing database buffer pool algorithms were compatible with it. Some academic work tried to address this, but I don’t know of any products.The SSD form factor wasn’t any faster at writes than NAND + capacitor-backed power loss protection. The read path was faster, but only in time to first byte. NAND had comparable / better throughput. I forget where the cutoff was, but I think it was less than 4-16KB, which are typical database read sizes.So, the DIMMs were unprogrammable, and the SSDs had a “sometimes faster, but it depends” performance story.

myself248: It sounds like they didn't do a good job of putting the DIMM version in the hands of folks who'd write the drivers just for fun.The read path is sort of a wash, but writes are still unequalled. NAND writes feel like you're mailing a letter to the floating gate...

deepsquirrelnet: I worked at Micron in the SSD division when Optane (originally called crosspoint “Xpoint”) was being made. In my mind, there was never a real serious push to productize it. But it’s not clear to me whether that was due to unattractive terms of the joint venture or lack of clear product fit.There was certainly a time when it seemed they were shopping for engineers opinions of what to do with it, but I think they quickly determined it would be a much smaller market anyway from ssds and didn’t end up pushing on it too hard. I could be wrong though, it’s a big company and my corner was manufacturing and not product development.

chrneu: I worked at Intel for a while and might be able to explain this.There were/are often projects that come down from management that nobody thinks are worth pursuing. When i say nobody, it might not just be engineers but even say 1 or 2 people in management who just do a shit roll out. There are a lot of layers of Intel and if even one layer in the Intel Sandwich drag their feet it can kill an entire project. I saw it happen a few times in my time there. That one specific node that intel dropped the ball on kind of came back to 2-3 people in one specific department, as an example.Optane was a minute before I got there, but having been excited about it at the time and somewhat following it, that's the vibe I get from Optane. It had a lot of potential but someone screwed it up and it killed the momentum.

osnium123: Are you referring to the Intel 10nm struggles in your reference to 2-3 people?

georgeburdell: Flash has the same shrink problem. And the solution for Optane was the same: go 3D

jamiek88: Oh I was so excited for that. I devoured any news or blogs or rumours about that immediately!

exmadscientist: At the time of their introduction Optane drives were noticeably faster to boot your machine than even the fastest available Flash SSD. So in a workstation with multiple hard drives installed anyway, buying one to boot off of made decent sense.If they had been cheaper, I think they'd have been really, really popular.

exmadscientist: Unfortunately a gaming machine workload is so read-heavy that I wouldn't expect Optane to square up well. Gaming is all about read speed and overall capacity. You need that heavy I/O mix, especially with low latency deadlines, to see gains from Optane. That limited target use case, coupled with ignorant benchmarking, always limited them.

epistasis: When most people are running databases on AWS RDS, or on ridiculous EBS drives with insanely low throughput and latency, it makes sense to me.There are very few applications that benefit from such low latency, and if one has to go off the standard path of easy, but slow and expensive and automatically backup up, people will pick the ease.Having the best technology performance is not enough to have product market fit. The execution required from the side of executives at Intel is far far beyond their capability. They developed a platform and wanted others to do the work of building all the applications. Without that starting killer app, there's not enough adoption to build an ecosystem.

amluto: > There are very few applications that benefit from such low latencyBasically any RDBMS? MySQL and Postgres both benefit from high performance storage, but too many customers have moved into the cloud where you can’t get NVMe-like performance for durable storage for anything remotely close to a worthwhile price.

epistasis: I'm saying that there are very few downstream applications that use databases that benefit from reducing latency beyond the slow performance of the cloud. Running your database on VMs or baremetal gives better performance, but almost no applications built on databases bother to do it.

empiricus: This is actually insane. Do you mean 2-4 people in one department basically killed Intel? Roll to disbelief.

exmadscientist: I don't think the shrink problem is at all the same for the two technologies. There are some really weird materials and production steps in Optane that are simply not present when making Flash cells.