Exploring the Quality of SSD – A Testing and Evaluation Report for SSDs in the Market Today

Solid State Drive (SSD), an emerging technology to the manufacturers and end-users in the marketplace, is considered as one of the innovative storage media that features high speed, low power consuming, high endurance and light-weight. It becomes a focus since the product has launched. At the current stage, SSD has been widely adopted in the product lines targeting on enterprise use. However, SSD still remains as a prospective product for many consumers as they often require a further evaluation before making a purchasing decision. The price and quality of the product are mainly the key factors that consumers are concerned about.

SSD uses NAND-based flash memory, and thus, the selling prices tend to be high as the unit price for NAND Flash fluctuates periodically. Since Single-Level Cell (SLC) is applied, the manufacturing cost per MB is relatively high. Also, the uncertainty for MTBF naturally makes end-users suspicious of the product life, quality and reliability. End-users are mostly unwilling to spend on products with doubtful product life and high failure rate.

To investigate and verify for the concerns in SSD, Allion Test Labs, Inc. (Allion) has selected a few branded SSDs that are available in the marketplace to carry out the testing in order to examine if the products indeed have potential risks toward their quality or performance in the real life using scenarios. Allion analyzes, diagnoses based on the test results and anticipates investigating the findings from an objective perspective. This can be used as a reference for vendors while developing new products and for consumers while making purchasing decisions.

Brand Name	Model Name	Firmware	Size
Corsair	P128	VBM19C1Q	128 GB
Crucial	RealSSD C300	0002	128 GB
KINGSTON	SSDNOW V	C091126a	128 GB
SanDisk	G3 60GB	1.00	60 GB
Corsair	R120	1.0	128 GB
Intel	X25-M	2CV102HD	160 GB
OCZ	Vertex	1.5	128 GB
WDC	SiliconEdge Blue	5.12	128 GB

In this test experiment, we select eight pieces of SSDs with the brands of Corsair, Crucial, Kingston, SanDisk, Intel, OCZ, and WDC. Corsair provides with two different models. For testing methodologies, we base them on indices we have used in the past to measure overall performance of SSD. The indices include four aspects: Reliability, Performance Evaluation, Performance Stability and Performance Degradation. By evaluating the test results according to each of the aspects of the indices, we rate the products “A” as the Best, “B” as the Second, and “C” as the Last that needs further improvement.

The chart below illustrates the final evaluation results. Intel X25-M and OCZ Vertex are the two SSD models with the best performing quality. How do we derive these test results? Please refer to the following sections for testing evaluation analyses.

Model	Reliability	Performance Evaluation	Performance Stability	Performance Degradation	Result
Corsair P128	B	C	C	A
Crucial RealSSD C300	C	B	B	A
KINGSTON SSDNOW V	A	C	C	A
SanDisk G3 60GB	B	C	B	A
Corsair R120	C	B	C	C
Intel X25-M	A	A	A	A
OCZ Vertex	A	B	A	A
WDC SiliconEdge Blue	B	B	B	A

Index 1: Reliability

Category	Reliability
Item	Surface of SSD	SSD Status	Data Integrity

The main purpose to examine the reliability of SSD is to observe and confirm if there is any Data Corruption/Data Error or Data Loss. SSD, similar to most of the storage devices, plays an important role in storage, and its main task is to accurately store and retain data. The capability to preserve the data under different using scenarios and under the conditions of dissimilar humidity and unstable power supply seem to be a big challenge for designing and developing SSD. If any of the problems such as Data Corruption/Data Error or Data Loss arises, end-users may encounter an unpredictable yet sever lost. Therefore, the reliability of SSD is the first evaluation index to be verified in this test experiment.

In the reliability aspect amid the indices, it can be divided into topics of “surface of SSD,” “SSD Status,” and “Data Integrity.” In the test experiment, we do not see any unusual problems based on the subjects of “surface of SSD, “and “SSD Status.” The specifications of the product appearance meet the standard, and while SSDs are connected with other computers, their devices can be detected from the hosts. For data retention, only KINGSTON, Intel and OCZ can accurately save all data; yet, over half of the brands encounter a Data Error problem. This means while the data is written into SSD after the read/write process, the data stored has been corrupted and is different from the original files.

Index 2: Performance Evaluation

Category	Performance Evaluation
Item	Sequential 64K Read	Sequential 64K Write	Random 16K Read	Random 16K Write	Read/ Write Performance

One of the well-known strengths that SSD has is its high speed transmission. Therefore, we utilize performance evaluation tools to simulate various using scenarios and execute the evaluation test. We select two kinds of sizes: 16K and 64K to carry out the Random and Sequential Read/Write tests. The reason we pick on these two kinds of sizes is that among the data file sizes, 16K is the size that end-users mostly use for general data saving while 64K is the size mostly utilized for media file saving. The purpose to put this test into practice is to observe how the SSDs perform while they are under the read/write processes. The resulting numbers are derived from the average of 100 test rounds. The chart above illustrates how unlikely the eight branded SSDs perform. Among these brands, Crucial, Intel and OCZ better performed whereas KINGSTON and SanDisk weakly performed.

To observe while the sequential 64K reading process is undertaking, we notice that 7 out of 8 branded SSDs perform approximately at the same level. SanDisk, nevertheless, is the only one that performs with less satisfaction. The speed of 99.24MB/s is only able to reach approximately half of the average performance compared to the remaining seven SSDs. To observe while the sequential 64K writing process is undertaking, we notice that OCZ’s performance is the best whereas SanDisk’s is less ideal. On the other hand, to observe while the random 16K reading process is undertaking, we notice that Crucial reaches 181.99MB/s whereas Intel reaches 204.38MB/s, and the rests reach below 70MB/s. Moreover, to observe while the random 16K writing process is undertaking, we notice that the products of Corsai-R120, Intel and WDC have the capability to exceed the speed of 100MB/s, and SanDisk only performs with a speed of 11.62MB/s. Lastly, to compare the average read/write performance among these eight SSDs, we discover that Crucial, Intel and OCZ are the outstanding ones that demonstrate the performance to exceed 60MB/s.

According to the data derived, we find out that every brand of SSD performs dissimilarly based on different using scenarios. For instance, while we implement the random 16K reading process test, we discern that Crucial performs as the second best. However, it becomes the last five in the ranking among all SSDs in the random 16K writing process test. To compare it with other brands, we notice that there is a great difference between the reading and writing speed. The fact reveals that end-users sense not only such discrepancy, but also the product itself has to enhance its controller IC towards the read/write computing techniques.

Index 3: Performance Stability

Category	Performance Stability
Item	Performance Normal Distribution	High/ Low Gap over mean	Standard Deviation over mean

This index helps to validate if the stability of SSD remains steady and consistent while undertaking the read/write process. It is especially important to pay more attention to this part because while SSD is used dynamically for longer hours and bigger storing coverage, the problem of performance instability easily occurs. If the performance stability is consistent, this benefits not only for end-users, but also guarantees the corresponding devices to operate steadily.

We employ the 3^rd index to evaluate SSD’s “Performance Stability.” In order to obtain the most accurate result, we evaluate according to the three aspects: Performance Distribution, High/Low Gap, and Standard Deviation over Mean.

Performance Normal Distribution

＊Diagram of Normal Distribution Curve

“Normal Distribution” means that among all the statistic data, all data points fall into the range between -3σ to 3σ. The performance Normal Distribution here means how the test results gained reveal as the data points generated for the bell curve shown on the graph. For the test result data per round, the closer to “μ”, the more stable the performance is. On the other hand, the further the test result data is, the greater the differences are, and the unstable the performance is. For example, the charts shown in the following illustrate the performances of Crucial C300 and Corsair P128.

The blue curves represent the test results collected for every round, and the pink curves represent the accumulated test results. These plots help conclude that the performance of Crucial is better than that of Corsair.

＊Crucial C300	＊Corsair P128

In addition, according to the test results, Crucial has 93.8% of the test results that fall into the average read/write performance (MB/s) within the range of +/–10%. In contrast, Corsair only has 48% of the test results that fall into the average read/write performance (MB/s) within the range of +/–10%.

Among all tested SSDs, OCZ is the one with the best performance. OCZ has not only 98.1% of the test results that fall into the average read/write performance (MB/s) within the range of +/–10%, but also no extraordinary results found outside the range of +/–20%. The chart in the following displays the degree of stability of OCZ’s performance. The navy blue line represents the test result for every round. All test results reveal that the speed of OCZ performs steadily, and the results fall into the red circled area (+/–10%). They are obviously close to the light blue dotty line (average speed). It is concluded that the speed for its performance likely remains at a certain level with consistency. On the contrary, the chart on the right hand side discloses a great difference for Corsair. The navy blue line fluctuates a lot, and there are extraordinary parts that exceed the range of +/–20%. It also signifies the degree differences for the performance results every time.

＊OCZ Vertex	＊Corsair R120	＊Legend

In short, except for OCZ, the rest of the SSDs all divulge a problem of instability.

High/ Low Gap over Mean

High/Low Gap over Mean is an indicator for the gap between the best and worst performance for the designated SSD. The bigger the gap is, the more unstable it is. In order to distinguish the degree differences among SSDs’ instability, we have applied this indicator to compare and contrast each SSD, then evaluate to perceive the average number for the differences between two target groups: the top 10% best performed SSDs and the last 10% least performed ones found in the test results. The methods used to compare include the calculation for the “absolute difference” and that of “relative difference”. “Absolute Difference” practically indicates the difference number between the leading group and the lagging group. “Relative Difference” on the other hand, implies on the calculation for the ratio of the difference number to the overall average performance, which also means the percentage of overall average performance divided by the difference number.

According to the test results, Intel and OCZ are especially the two brands that perform well. The “Absolute Difference” between the leading group and the lagging group conducted by Intel is 12.1MB/s, and the ratio of it to the overall average performance, as known as the “Relative Difference” is 16.2%. In contrast, the “Absolute Difference” for OCZ is only 4.6MB/s, and the “Relative Difference” is 7%. Besides Intel and OCZ, the rest of the SSDs all have a problem on exceeding difference number. For example, the “Absolute Difference” of Corsair P128 is 26.4MB/s, and the “Relative Difference” is 55%. In short, this SSD has the problem of exceeding difference number for its performance over half of the test rounds.

 

Standard Deviation over Mean

Here we make use of statistics to understand the “Standard Deviation over Mean” of SSD’s performance. This is to observe how much dispersion there is from the average. For instance, based on the test result of the selected SSD of SanDisk, “μ” is 47.2MB/s; “σ” is 3.2; “σ/μ” is 7%. This explains that the data all falls into the range of its average number of +/–3.2MB/s among most of the test rounds. In comparison, the “μ” of KINGSTON is 32.6MB/s whereas “σ” is 4.6; “σ/μ” is 14%. Therefore, it is obvious for users to conclude that the performance of SanDisk G3 60GB is better than that of KINGSTON SSDNOW V.

Among all of the tested SSDs, except for Corsair P 128, Corsair R 120 and KINGSTON SSDNOW V, the rest of the SSDs are able to meet the expectation with the Standard Deviation over Mean between 2% ~ 10% under this section.

Index 4: Performance Degradation

Category	Performance Degradation
Item	Performance Degradation	Overall Performance

Performance degradation is defined as performance decline, and this happens gradually for certain reasons such as while the operating time gets longer, the using frequencies increase, and the remaining capacity diminishes. The primary reason is the sub-optimal TRIM and Garbage Collection algorithm design in SSD controller chip which leads to non-ideal space recovery and subsequently insufficient storage for the product to spread out read/write operations among different storage blocks (Wear Leveling). Furthermore, during the usage, while SSD does not provide the situation of full operation or data saturation with sufficient spaces (as known as the term -Space Provision-), it easily directs to the problem of Performance Degradation.

＊Example of SSD Performance Degradation

There are two kinds of circumstances for Performance Degradation. One is that SSD’s performance gradually declines after being used. The diagram shown on the right hand side represents the situation mentioned above. In this instance, the pink line stands for the performance for each of the round. The stable blue line symbolizes for the remaining capacity. Except for the situation that the speed of 200MB/s can be reached at the initiating stage, after the 20^th round, approximately only 75MB/s can be reached. Another situation is that while SSD’s remaining storage space is decreasing, the speed relatively declines. Furthermore, while the storage space is released again, the speed will recover eventually and reach a better performance standard.

＊SanDisk G3

Based on the test experiment result, both KINGSTON and SanDisk obviously reflect on Performance Degradation, and the reason belongs to the second situation mentioned in the previous paragraph. The diagram beside illustrates how the performance of SanDisk G3 60GB varies based on various test rounds. The navy blue line fluctuates with a regular pattern and represents the SSD’s performance level based on how much the storage space is left (indicated as the light purple line).

On the contrary, for the rest of the selected SSDs, there is no Performance Degradation identified, and there are no extraordinary conditions found for the overall performance. Lastly, according to the test experiment, Corsair R120 encounters malfunction, and it has not been able to complete all test rounds. As a result, this product has a bad review for its Reliability, Performance Stability and Performance Degradation.

Comprehensive Verification to Make SSD a Golden Quality

According to the test experiments, we can conclude that even though they are all SSDs with a declaration of high performance efficiency, their performances tremendously differentiate in reality. Based on the designed test experiment with 4 evaluation indices and 13 evaluation standards, we find numerous problems with variant degrees among all chosen branded SSDs. The issues and concerns arisen behind are especially important for the product designers to be cautious. The further improvement and enhancement are mandatory as they will be the prerequisite for the branding manufacturers to initiate and work on during the product design and verification stage.

Among all the emerging technology products, SSD is still yet to become mature as opposed to replacing the traditional hard drives. Due to the complexity math of its design, every single designing detail may be the factor to impact on SSD’s reliability, performance stability, and overall performance and so on. Furthermore, SSD can be applied either for built-in in the notebook or external use as a portable consumer electronic product. Its storage function can also be utilized for enterprise and military use. Multifarious usage requires diverse performance level and thus demands appropriate tests in order to assure if SSD’s performance can indeed conform to its intended purposes. Based from Allion’s professional test and consulting experience for more than 20 years, we induct four key factors – interface, controller, NAND Flash and application – that may affect the quality and performance of SSD. To ensure its quality, we suggest implement a variety of verification methodologies then find the root causes for the problems.

＊The Complexity Math of SSD Design

Although there are many choices from multitudinous brands, consumers tend to have difficulty choosing the right products to fit their needs. By publishing this, we, Allion, anticipate pointing out the SSD quality issues that end-users might not be aware of. Meanwhile, the test results can be a reference for manufacturers to develop, modify and improve while designing products, and this leads to end-users to further enjoy high quality products.

Remark: The SSDs selected for this test experiment were purchased in 2010. Below price list reflects on the retail price at the time of purchase for readers’ reference