lichess.org
Donate

Effectively estimating Stockfish Levels' ratings.

smach1,656 viewsEnglish (US)
Chess engineChess botLichessChess
This article outlines a practical and rather simple method for estimating the ratings of Fairy-Stockfish Levels using Chess.com bots.

Setting the Stage

In both recent [1] [2] and not-so-recent [3] [4] posts, there have been attempts to estimate Stockfish Levels ratings. However, these articles either lack thorough explanation of the methodology behind the estimation, or do not specify the source of the information at all, which makes it really difficult to ascertain their reliability. In this article, we aim to provide a practical and more realistic approach to estimating these ratings.

To achieve this, we will use Chess.com bots, have the engines engage in duels, and assign a ratings to each of the levels. Afterward, we can convert the obtained Chess.com ratings to Lichess ratings, using a recent comprehensive study [5] that establishes correlations between FIDE ratings, Chess.com ratings and Lichess ratings.

TL;DR

Except for the final row, which shows the maximum rating possible, you may interpret the ratings as "at most".

LevelChess.comFIDELichess BlitzBulletRapidClassical
1250-----
2400-----
31100-1485131516651675
4150015451800167519401870
5170016151940184520401985
6210020152205218522702155
7290025902850290027352580
83200+-----

The Method

First and foremost, it's important to emphasize that our primary objective here is not to conduct an in-depth analysis or provide a highly precise mathematical estimation of any kind. Instead, our goal is to conduct a rapid analysis that offers a meaningful albeit inherently imprecise estimation of the strengths of each Stockfish Levels.

That being said, let's move on to describe the format of the challenges between the engines, and after that, we will present all the information used to arrive at the final results.

The matches will follow a simple two-match knockout format, with a preference for Lichess engines in the event of a draw. Lichess will always initiate the games with the White pieces. If Lichess secures a victory in the first game of the match, it proceeds to face the next Chess.com opponent, as the worst-case scenario would be a draw by the end of the second match.

However, if Lichess loses the first game, it proceeds to a second game of the same match. If the second game results in a win, Lichess advances to the next match the same way. However, if the second game ends either in a draw or with Lichess losing, considering the prior game's loss, Lichess is eliminated, making way for the next Lichess Stockfish engine level to enter the arena against the same opponent.

This process continues until there are no more Stockfish Levels to play or until we exhaust all Chess.com opponents. Naturally, we will commence by pitting Stockfish Level 1 against the first Chess.com bot with the lowest available rating.

Standings

LichessChess.comResult
FSF-L1Martin (250)1-0, Checkmate.
FSF-L1Wayne (250)0-1, Resignation.
Wayne (250)FSF-L11-0, Checkmate.
FSF-L2Wayne (250)1-0, Checkmate.
FSF-L2Fabian (250)1-0, Checkmate.
FSF-L2Juan (400)0-1, Checkmate.
Juan (400)FSF-L21-0, Checkmate.
FSF-L3Juan (400)1-0 Checkmate.
FSF-L3Filip (400)1-0, Checkmate.
FSF-L3Elani (400)1-0, Checkmate.
FSF-L3Noel (550)1-0, Checkmate.
FSF-L3Oliver (550)1-0, Checkmate.
FSF-L3Milica (550)1-0, Checkmate.
FSF-L3Janjay (700)1-0, Checkmate.
FSF-L3Aron (700)1-0, Checkmate.
FSF-L3Mina (700)1-0, Checkmate.
FSF-L3Zara (850)1-0, Checkmate.
FSF-L3Santiago (850)1-0, Checkmate.
FSF-L3Karim (850)1-0, Checkmate.
FSF-L3Emir (1000)1-0, Checkmate.
FSF-L3Maxim (1000)1-0, Checkmate.
FSF-L3Hans (1000)1-0, Checkmate.
FSF-L3Azeez (1100)0-1, Checkmate.
Azeez (1100)FSF-L31-0, Checkmate.
FSF-L4Azeez (1100)1-0, Checkmate.
FSF-L4Laura (1100)1-0, Checkmate.
FSF-L4Sven (1100)1-0, Checkmate.
FSF-L4Maria (1200)1-0, Checkmate.
FSF-L4Elena (1200)0-1, Checkmate.
Elena (1200)FSF-L40-1, Checkmate.
FSF-L4Wilson (1200)1-0, Checkmate.
FSF-L4Jade (1300)1-0, Checkmate.
FSF-L4Nelson (1300)1-0, Checkmate.
FSF-L4Vinh (1300)1-0, Checkmate.
FSF-L4David (1400)1-0, Checkmate.
FSF-L4Ali (1400)0-1, Checkmate.
Ali (1400)FSF-L40-1, Checkmate.
FSF-L4Mateo (1400)0-1, Checkmate.
Mateo (1400)FSF-L40-1, Checkmate.
FSF-L4Antonio (1500)½-½, Stalemate.
Antonio (1500)FSF-L41-0, Checkmate.
FSF-L5Antonio (1500)1-0, Checkmate.
FSF-L5Wendy (1500)1-0, Checkmate.
FSF-L5Pierre (1500)1-0, Checkmate.
FSF-L5Pablo (1600)1-0, Checkmate.
FSF-L5Joel (1600)1-0, Checkmate.
FSF-L5Isabel (1600)1-0, Checkmate.
FSF-L5Arthur (1700)0-1, Checkmate.
Arthur (1700)FSF-L51-0, Checkmate.
FSF-L6Arthur (1700)1-0, Checkmate.
FSF-L6Jonas (1700)1-0, Checkmate.
FSF-L6Isla (1700)1-0, Checkmate.
FSF-L6Lorenzo (1800)1-0, Checkmate.
FSF-L6Wally (1800)1-0, Checkmate.
FSF-L6Julia (1800)1-0, Checkmate.
FSF-L6Miguel (1900)1-0, Checkmate.
FSF-L6Xavier (1900)1-0, Checkmate.
FSF-L6Olga (1900)1-0, Checkmate.
FSF-L6Li (2000)1-0, Checkmate.
FSF-L6Charles (2000)0-1, Checkmate.
Charles (2000)FSF-L60-1, Checkmate.
FSF-L6Fatima (2000)1-0, Checkmate.
FSF-L6Manuel (2100)0-1, Checkmate.
Manuel (2100)FSF-L6½-½, Insufficient Material.
FSF-L7Manuel (2100)1-0, Checkmate.
FSF-L7Oscar (2100)1-0, Checkmate.
FSF-L7Noam (2200)1-0, Checkmate.
FSF-L7Nora (2200)1-0, Checkmate.
FSF-L7Ahmed (2200)1-0, Checkmate.
FSF-L7Sakura (2200)1-0, Checkmate.
FSF-L7Arjun (2300)1-0, Checkmate.
FSF-L7Francis (2300)1-0, Checkmate.
FSF-L7Sofia (2300)1-0, Checkmate.
FSF-L7Alexander (2450)1-0, Checkmate.
FSF-L7Luke (2450)1-0, Checkmate.
FSF-L7Wei (2450)1-0, Checkmate.
FSF-L7Morphy (2500)1-0, Checkmate.
FSF-L7Krikor (2550)½-½, Stalemate.
Krikor (2550)FSF-L70-1, Checkmate.
FSF-L7Danya (2650)1-0, Checkmate.
FSF-L7Judit Polgar (2735)1-0, Checkmate.
FSF-L7Ian (2795)1-0, Checkmate.
FSF-L7Fabiano (2840)1-0, Checkmate.
FSF-L7Engine Level 24, Super Grandmaster (2900)0-1, Checkmate.
FSF-L8Engine Level 24, Super Grandmaster (2900)1-0, Checkmate.
FSF-L8Engine Level 25, Maximum (3200)½-½, Repetition.

(*) FSF-L1 resigned with King+Queen+Rook against a lonely King, after being unable to checkmate in several moves.

Considerations

One of the most evident challenges with this approach is the limited variety of sampling. A simple victory implies immediately skipping to the next opponent. But it's worth noticing that on Chess.com, players typically encounter three bots for each rating level, this offers a more comprehensive metric by effectively attempting three matches at the same level before advancing to the next one.

Another issue with this approach is the lack of opportunities for comebacks from either side, which would probably change the outcome reasonably. Nevertheless, this simplification may still (hopefully) serve our purposes well and help us reach a useful result more swiftly.

We should also consider that, as the engines run locally in the client's browser, the available resources can directly impact performance in unexpected ways. The tests were conducted with both engines running on the same computer, using Chrome, on a state-of-the-art Linux desktop. Results for the same engines on a mobile device, for example, could be entirely different and were not tested.

Finally, for the ratings conversions, we had no choice but to rely on Chess.com Blitz Ratings, as this was the only available data that allowed a proper comparison. We initially aimed to use Rapid ratings, but unfortunately, that option was not viable. It's safer to base your assessment solely on the directly obtained ratings. The converted ratings are primarily provided for the sake of completeness, aiming to fill in gaps and enhance our overall understanding of the strengths.

Conclusion

Despite the ratio between the number of games (75) and the number of Stockfish Levels (8) being less than ideal, the engines maintained a consistent pattern throughout the matches and yielded results in line with expectations, considering the increasing difficulty on both sides.

The lower levels exhibited a somewhat random and notably non-human behavior. In contrast, the intermediate levels, while also displaying forced behaviors at times, generally replicated more natural moves.

Particularly in endgames, some intermediate levels appeared to struggle when seeking simple checkmates, whereas higher levels even seek more "elegant" or "less obvious" ways to win.

Furthermore, I believe that the direct estimates are sufficiently reliable, whereas the conversions, though potentially useful, may have accumulated too many errors to be taken literally.

Games

FSF-L1 - Martin (250), 1-0, Checkmate.

https://lichess.org/wvdhDHV5/black

FSF-L1 - Wayne (250), 0-1, Resignation.
https://lichess.org/xPZvkd8l/black

Wayne (250) - FSF-L1, 1-0, Checkmate.
https://lichess.org/U5XRWkQY/white

FSF-L2 - Wayne (250), 1-0, Checkmate.
https://lichess.org/0Ki2mmTl/black

FSF-L2 - Fabian (250), 1-0, Checkmate.
https://lichess.org/6AUR2yZV/black

FSF-L2 - Juan (400), 0-1, Checkmate.
https://lichess.org/UKL32SBv/black

Juan (400) - FSF-L2, 1-0, Checkmate.
https://lichess.org/I7ySdYQp/white

FSF-L3 - Juan (400), 1-0, Checkmate.
https://lichess.org/qRBaiwUQ/black

FSF-L3 - Filip (400), 1-0, Checkmate.
https://lichess.org/Dl0Mkteu/black

FSF-L3 - Elani (400), 1-0, Checkmate.
https://lichess.org/cRsSOLvc/black

FSF-L3 - Noel (550), 1-0, Checkmate.
https://lichess.org/79RLcZ8J/black

FSF-L3 - Oliver (550), 1-0, Checkmate.
https://lichess.org/sqL9wCS3/black

FSF-L3 - Milica (550), 1-0, Checkmate.
https://lichess.org/V4vGVCw0/black

FSF-L3 - Janjay (700), 1-0, Checkmate.
https://lichess.org/0ylavcPW/black

FSF-L3 - Aron (700), 1-0, Checkmate.
https://lichess.org/YDUTCESp/black

FSF-L3 - Mina (700), 1-0, Checkmate.
https://lichess.org/bVns8dGT/black

FSF-L3 - Zara (850), 1-0, Checkmate.
https://lichess.org/mYVNOYo6/black

FSF-L3 - Santiago (850), 1-0, Checkmate.
https://lichess.org/adMneOrM/black

FSF-L3 - Karim (850), 1-0, Checkmate.
https://lichess.org/hVXD6tWK/black

FSF-L3 - Emir (1000), 1-0, Checkmate.
https://lichess.org/2J4mPJak/black

FSF-L3 - Maxim (1000), 1-0, Checkmate.
https://lichess.org/JJjTAeQV/black

FSF-L3 - Hans (1000), 1-0, Checkmate.
https://lichess.org/XoV0CdZ0/black

FSF-L3 - Azeez (1100), 0-1, Checkmate.
https://lichess.org/v2RSokZr/black

Azeez (1100) - FSF-L3, 1-0, Checkmate.
https://lichess.org/xdXyxshn/white

FSF-L4 - Azeez (1100), 1-0, Checkmate.
https://lichess.org/n2AhORwt/black

FSF-L4 - Laura (1100), 1-0, Checkmate.
https://lichess.org/8u51y9oQ/black

FSF-L4 - Sven (1100), 1-0, Checkmate.
https://lichess.org/9nrWiXpU/black

FSF-L4 - Maria (1200), 1-0, Checkmate.
https://lichess.org/DLvzLNu0/black

FSF-L4 - Elena (1200), 0-1, Checkmate.
https://lichess.org/AN0og3NR/black

Elena (1200) - FSF-L4, 0-1, Checkmate.
https://lichess.org/BBhAs9aE/white

FSF-L4 - Wilson (1200), 1-0, Checkmate.
https://lichess.org/ZMv5jzHD/black

FSF-L4 - Jade (1300), 1-0, Checkmate.
https://lichess.org/DhxrJDRk/black

FSF-L4 - Nelson (1300), 1-0, Checkmate.
https://lichess.org/PRvM9Flo/black

FSF-L4 - Vinh (1300), 1-0, Checkmate.
https://lichess.org/NUIgkjPu/black

FSF-L4 - David (1400), 1-0, Checkmate.
https://lichess.org/BCaecgNe/black

FSF-L4 - Ali (1400), 0-1, Checkmate.
https://lichess.org/Dt5UquJz/black

Ali (1400) - FSF-L4, 0-1, Checkmate.
https://lichess.org/GV6HmBu1/white

FSF-L4 - Mateo (1400), 0-1, Checkmate.
https://lichess.org/11mN9yD2/black

Mateo (1400) - FSF-L4, 0-1, Checkmate.
https://lichess.org/czcz1ofo/white

FSF-L4 - Antonio (1500), 1/2-1/2, Stalemate.
https://lichess.org/0sTuWymM/black

Antonio (1500) - FSF-L4, 1-0, Checkmate.
https://lichess.org/MNBFOPPM/white

FSF-L5 - Antonio (1500), 1-0, Checkmate.
https://lichess.org/ChVnVxgm/black

FSF-L5 - Wendy (1500), 1-0, Checkmate.
https://lichess.org/OEiz3CZH/black

FSF-L5 - Pierre (1500), 1-0, Checkmate.
https://lichess.org/tywjdieI/black

FSF-L5 - Pablo (1600), 1-0, Checkmate.
https://lichess.org/xXr7Gdwi/black

FSF-L5 - Joel (1600), 1-0, Checkmate.
https://lichess.org/MgmoF82l/black

FSF-L5 - Isabel (1600), 1-0, Checkmate.
https://lichess.org/tK7S8FdF/black

FSF-L5 - Arthur (1700), 0-1, Checkmate.
https://lichess.org/UnhaC5oR/black

Arthur (1700) - FSF-L5, 1-0, Checkmate.
https://lichess.org/eHJ7EtKQ/white

FSF-L6 - Arthur (1700), 1-0, Checkmate.
https://lichess.org/BDr9peWM/black

FSF-L6 - Jonas (1700), 1-0, Checkmate.
https://lichess.org/WNQLA4kx/black

FSF-L6 - Isla (1700), 1-0, Checkmate.
https://lichess.org/zgLmuFzu/black

FSF-L6 - Lorenzo (1800), 1-0, Checkmate.
https://lichess.org/VSsRTw3G/black

FSF-L6 - Wally (1800), 1-0, Checkmate.
https://lichess.org/M04kHMy3/black

FSF-L6 - Julia (1800), 1-0, Checkmate.
https://lichess.org/aQSw4Kab/black

FSF-L6 - Miguel (1900), 1-0, Checkmate.
https://lichess.org/zezfn5ND/black

FSF-L6 - Xavier (1900), 1-0, Checkmate.
https://lichess.org/ddb2uEml/black

FSF-L6 - Olga (1900), 1-0, Checkmate.
https://lichess.org/aeaxDF6X/black

FSF-L6 - Li (2000), 1-0, Checkmate.
https://lichess.org/ee0VsAZZ/black

FSF-L6 - Charles (2000), 0-1, Checkmate.
https://lichess.org/zhhOzRpn/black

Charles (2000) - FSF-L6, 0-1, Checkmate.
https://lichess.org/Dt6IiGtp/white

FSF-L6 - Fatima (2000), 1-0, Checkmate.
https://lichess.org/W3cppMkO/black

FSF-L6 - Manuel (2100), 0-1, Checkmate.
https://lichess.org/LcvDMPPS/black

Manuel (2100) - FSF-L6, 1/2-1/2, Insufficient Material.
https://lichess.org/TRTrPM36/white

FSF-L7 - Manuel (2100), 1-0, Checkmate.
https://lichess.org/Rn7jviPZ/black

FSF-L7 - Oscar (2100), 1-0, Checkmate.
https://lichess.org/UC4CijfA/black

FSF-L7 - Noam (2200), 1-0, Checkmate.
https://lichess.org/KyQqlwfz/black

FSF-L7 - Nora (2200), 1-0, Checkmate.
https://lichess.org/5BjidQtL/black

FSF-L7 - Ahmed (2200), 1-0, Checkmate.
https://lichess.org/7IoullET/black

FSF-L7 - Sakura (2200), 1-0, Checkmate.
https://lichess.org/rvlo83ZO/black

FSF-L7 - Arjun (2300), 1-0, Checkmate.
https://lichess.org/ht1YBACX/black

FSF-L7 - Francis (2300), 1-0, Checkmate.
https://lichess.org/aSspACjM/black

FSF-L7 - Sofia (2300), 1-0, Checkmate.
https://lichess.org/mjIz3f34/black

FSF-L7 - Alexander (2450), 1-0, Checkmate.
https://lichess.org/S7LxG7Vz/black

FSF-L7 - Luke (2450), 1-0, Checkmate.
https://lichess.org/a3NT1WHu/black

FSF-L7 - Wei (2450), 1-0, Checkmate.
https://lichess.org/j4jCb4fO/black

FSF-L7 - Morphy (2500), 1-0, Checkmate.
https://lichess.org/HtQ5Whri/black

FSF-L7 - Krikor (2550), 1/2-1/2, Stalemate.
https://lichess.org/GN2eWUzV/black

Krikor (2550) - FSF-L7, 0-1, Checkmate.
https://lichess.org/47hAZ0fN/white

FSF-L7 - Danya (2650), 1-0, Checkmate.
https://lichess.org/P2ISgkvh/black

FSF-L7 - Judit Polgar (2735), 1-0, Checkmate.
https://lichess.org/abrTWYt2/black

FSF-L7 - Ian (2795), 1-0, Checkmate.
https://lichess.org/U9rbGqK7/black

FSF-L7 - Fabiano (2840), 1-0, Checkmate.
https://lichess.org/ijVYBFiU/black

FSF-L7 - Engine Level 24, Super Grandmaster (2900), 0-1, Checkmate.
https://lichess.org/xLvympAH/black

FSF-L8 - Engine Level 24, Super Grandmaster (2900), 1-0, Checkmate.
https://lichess.org/vnLaCr59/black

FSF-L8 - Engine Level 25, Maximum (3200), 1/2-1/2, Repetition.
https://lichess.org/rAOd4Kjc/black

References

[1] Stockfish level and its rating
[2] Stockfish level and its ratings, Forum post
[3] Stockfish AI levels with LiChess and ratings
[4] The meaning of stockfish levels on lichess 1-8
[5] Rating Comparisons

The image at the top of this article is accessible via the following link:
https://images.chesscomfiles.com/uploads/v1/article/24337.8d1dfeba.668x375o.59ae4c67a29e@2x.jpeg

Document version 0.2, last updated: October 3, 2023.

Discuss this blog post in the forum

More blog posts by smach