Thursday, December 8, 2022

Analyzing 2022 USTA NTRP year-end ratings - How often do appeals stick? How many are inappropriate?

I just wrote about 2022 year-end ratings and how many players have already appealed their rating, but a natural follow up question is how many of these appeals are really appropriate?

Allowing appeals is seemingly done for a few reasons:

  • Players get bumped up but want to continue to play their their old team and friends
  • Players get bumped up and there is no flight at the higher level and appealing down allows them to play
  • Players are improving but just missed a bump up and want the validation of the higher rating by their name
  • A team at the higher level needs players but roster limits require a minimum number of at-level players so if someone appeals up, they help meet that minimum number

One can debate which of the above, or other reasons there might be, are legitimate reasons to appeal, but one could also make the case that if someone appeals and then goes back to their old rating after a year, perhaps that appeal wasn't appropriate or justified.

Whether you agree with the characterization of this scenario indicating an inappropriate or unjustified appeal, it is a statistic we can look at so I went above doing so.  What I'll be doing is looking at players that appealed in year X but in year X+1 went back to their old level.

I will note that my data is not necessarily 100% perfect, but I think it is good enough for us to get a good idea of what happens with players that appeal.

First, looking at players that appealed their year-end level up:

  • 2021 - 1,948 appeals up with 645, or 33%, bumped back down at 2022 year-end
  • 2019 - 2,443 / 729 / 30%
  • 2018 - 2,018 / 630 / 31%
  • 2017 - 1,765 / 552 / 31%
  • 2016 - 1,866 / 574 / 31%

This shows a pretty clear trend of around a third of appeal ups being "inappropriate".

What about appeal downs?  Here that is:

  • 2021 - 3,307 appeals down with 1,060, or 32%, bumped back up at 2022 year-end
  • 2019 - 4,706 / 1,544 / 33%
  • 2018 - 3,123 / 1,400 / 45%
  • 2017 - 2,151 / 834 / 39%
  • 2016 - 2,122 / 834 / 39%
Here we see a higher percentage, but not quite as consistent year to year, of appeal downs being "inappropriate".

What do you think?  Do these stats tell us anything about whether the appeal system is working as intended?  Is it a concern that 30-40% of appeal players go back to their pre-appeal level?

5 comments:

  1. Kevin, I have a different question about appeals, if you happen to know? Taking the age thresholds like 60, 65, etc., can a player keep appealing throughout the year and have it actually work after multiple attempts?

    For example, Player A gets a 3.5 rating at year-end. He immediately appeals that rating back to 3.0, and is denied. He then plays a 3.5 match the following week, appeals again and is denied again. 2 months later after 4 additional 3.5 matches, he appeals again and is granted his appeal back to 3.0. Can this actually happen? My understanding is no chance, but I've heard some odd stories where maybe it could.

    ReplyDelete
    Replies
    1. No, appealing your rating again will not change the result until a new year-end rating is published, or the rules/criteria applied to the appeal are different.

      Delete
  2. Agreed, I'd like to know as well. I'm not sure whether or not this was already answered.

    ReplyDelete
  3. As a USTA captain, I have had many players appeal for various reason. The system was initially designed to allow players to play at the level at which they should be rated. In that case, I have had many 3.0 players that get bumped to 3.5 too soon. In this case, if you can get an appeal down to 3.0 and have one more year to develop your game, then go for it. As a 3.0, you can always play up and gain more experience playing in the 3.5 leagues also. On the flip side, as a low 3.5, it's absolute purgatory. Low end 3.5s get demolished by good 3.5s, and most certainly won't play 4.0 league.

    The dark side of the appeal is the fact this is used so often to manipulate rosters in order to gain good players with a higher rating. In bigger cities, where competition is extremely fierce, I see 4.0s appealing down to 3.5 all the time. The dynamic consequences of this are polarizing. For example, a good 4.0 has a bad year and wins his appeal to 3.5. As a 3.5 guy in the upcoming season, he becomes a key player for the 3.5 men's team (while still being able to play 4.0). As a 3.5, he now can play 6.0, 7.0, and 8.0 mixed leagues. As a 3.5 guy, in 7.0 mix he can pair now with the best 3.5 girl (instead of a 3.0 girl) , and in 8.0 mix he can now pair with a 4.5 girl (instead of a 4.0 girl).

    ReplyDelete
    Replies
    1. Honestly, I wouldn't be too concerted aobut the 30% to 40% of appeal players returning back to their original ratings. I would be more concerned if that number was above 50% which would mean the appeal was being used way too much to manipulate rosters.

      Delete