[Zar]

*** hrothgar wrote: "I think that you and I have a different definition of the world "normalize". Fixing one point on an axis does not normalize the distribution.

More specifically, saying that Zar points are distributed along (.73, 1.0) while
BUM_RAP points are distributed along (.67, 1.0) is does not normalize the distribution in any normal sense of the word...
<

Bum and Binki are not on the horizon anyway - we are talking about methods that have chances of being played at the table - the 5 methods that I compaere do qualify I believe (let me know if you have any other suggestions).

We can certainly run computer-based methods and Zar Points will be happy to participate (not with the "human-version" that you know, though :-).

So the normalization is also not an issue (I think).

ZAR

[tysen2k]

It's sort of backwards to look at the hands that make a certain number of tricks and then look at a standard deviation of points (normalized or not). It's better to look at the collection of hands that your evaluator predicts belong at a certain level and then look at the standard deviation of the obtained tricks.

So instead of looking at hands that take 10 tricks and then seeing how many hands have >51 Zar, we should be looking at those hands that have 52-56 Zar and looking at the standard deviation of the # of tricks taken. Then repeat that for 26-28 Goren, etc. They both have std deviations in terms of tricks so it's an apples-to-apples comparison.

My previous experiments did this (although for all hands, not by level). I could break it out by level and see if one method is more accurate at certain levels. But when you look at overall performance by combining them again, you'll get what I started with.

Tysen

[mikestar]

hrothgar, on May 19 2004, 07:57 PM, said:

... Start by normalizing the different evaluation criteria to use the same scale.
I'm fond of 0-11.  I'm perfectly happy with 0-40, or 0-1, or whatever...

It is simpler and just as correct to calculate the mean and standard devaition for each method in terms of its own points and then compare relative error (Standard deviation divided by the mean).

Finding the theoretical best method doen't entirely answer the question of which to use, complexity is a factor as well. Binky may be the best of all but it is unusable at the table. Zar is no more intrinically complex than the others, but it is unfamiliar. Teaching a beginner to count 1-3-5 instead of 1-2-3 isn't such a big step and will improve his accuracy.

By the way, while the double dummy solver approach is the best available, it does have some biases.

When the points are equally distributed, the times that the contract makes double dummy when but goes down at the table are compensated by the times the contract goes down double dummy but makes at the table--perhaps even overcompensated, as there are two defenders to make a mistake, wrong guess, etc. vs. one declarer.

But as the defenders hands get weaker and weaker, double dummy defense gets less and less useful. Let's say we're in a grand missing the Queen of trumps and have to guess a two way finesse. Double dummy declarer play guaratees we make every time, while double dummy defense doesn't affect the result at all.

Maybe in part Zar seems to perform best on grands and worst on partials because of this bias. I believe this bias will affect all methods, so that the ordering of the relative errors will be relaible, but the magnitude of the differences in relative errors will not be.

And part of the question is "how much better?"

[tysen2k]

mikestar, on May 19 2004, 04:23 PM, said:

By the way, while the double dummy solver approach is the best available, it does have some biases.

Yes it does.

Someone (I think Peter Cheung) did a large study of okbridge hands and compared how single-dummy declarers did compared to double-dummy data. He found that overall, the DD result was within 0.1 tricks of the SD result. At high contracts, DD declarer has more of an advantage (can drop those queens). At partscores the DD declarer was at a disadvantage, and at games both DD and SD were very close.

So DD data does have its disadvantages, but it's the best we have right now. Just understand its limitations and always use common sense.

[Cascade]

Ben I think you overlooked the implications of:

Quote

Start by normalizing the different evaluation criteria to use the same scale.

[tysen2k]

Okay I've split the data that I posted on my yahoo site by tricks (the tricks the evaluator predicts, not the actual tricks taken).

Here are the results:

            8      9      10     11     12     13    Overall
HCP       1.21   0.43   0.69   0.73   0.59   N/A    0.66
HCP+321   1.07   0.31   0.65   0.68   0.54   0.40   0.59
HCP+531   1.05   0.26   0.62   0.69   0.61   0.93   0.56
Zar       1.05   0.27   0.62   0.70   0.62   0.60   0.56
BUM+321   1.07   0.30   0.63   0.66   0.55   0.50   0.57
BUM+531   1.04   0.24   0.60   0.66   0.60   0.59   0.54

These numbers represent the average number of tricks in error that each evaluator predicted away from those that were actually taken (smaller is better).

Note that this is from the sample of hands that always take at least 9 tricks (thus the 1+ error that is always seen when only 8 tricks are predicted). I'll use this dataset since it's the one that has consistantly put Zar in the best light that I've seen.

HCP by itself never predicts that it will make a grand (the highest combined on these 13,000 hands is 34 HCP). This is another artifact of limiting the hands by eliminating those hands where NT is the best contract (again to make Zar look better).

Also notice that the evaluators that only use 321 for distribution look like they are the most accurate for slams. Don't let this fool you; it only means that if the hands have enough points to predict a slam with only 321 then it's a pretty solid one. It misses a lot of potential slams when it predicts less tricks.

Comments are welcome

Tysen

[Zar]

*** mikestar wrote: “Finding the theoretical best method doen't entirely answer the question of which to use, complexity is a factor as well. Binky may be the best of all but it is unusable at the table. Zar is no more intrinically complex than the others, but it is unfamiliar. Teaching a beginner to count 1-3-5 instead of 1-2-3 isn't such a big step and will improve his accuracy.
<

I believe I mentioned somewhere in the threads that 5-3-1 has the SAME problems Goren does – it makes 7-card suits look like 5-card suit, 6-card suits look like 4-cards suits etc. Just an example – the 7-3-3-0 is the same as 5-4-4-0, 6-3-3-1 is the same as 4-4-4-1 etc.

Even if you make it 10-5-1 instead of 3-2-1 or 5-3-1, it’s is not gonna change a thing.

If you want to teach your students how to think in patterns and evaluate a hand (NOT in terms of Zar Points if you don’t like them, but JUST Goren and Bergen instead) simply let them use the Zar Bid Machine – it will teach them not only the patterns, but how to look at a hand as an entity of Controls, HCP, patterns etc rather than a bunch of 13 cards. Have a look at the home page of the website – you may like it.

>
Maybe in part Zar seems to perform best on grands and worst on partials because of this bias.
<

It’s not that Zar Points perform worst in part-scores, it’s that the LOWER the level the CLOSER the methods become in terms of performance. As I mentioned, for a contract of “1 Club” all methods are equal :-) As you can see from the posting of the comparison (well, not in terms of std deviation but simply counting the contracts, true) you’ll see that while HCP-5-3-1 gets 10 times less Grands from the standard GIB boards, the difference drops to 2 times of the GIB Games compared to the Zar Points performance, and it will go FURTHER down in the Level 3 partscores etc. But the point is that the HCP-5-3-1 does NOT become better than Zar Points in the partscore region – they just get closer and I am sure it’s natural from anyone’s perspective.

ZAR

[whereagles]

Hum... I didn't have time to read the whole thing, but it seems to me ZAR requires effort from the players which doesn't compensate for the extra precision.

[inquiry]

whereagles, on May 20 2004, 04:33 AM, said:

Hum... I didn't have time to read the whole thing, but it seems to me ZAR requires effort from the players which doesn't compensate for the extra precision.

The thread, currently, has centered on comparing ZAR points to something called BUM RAP + 5-3-1.

Of the two, I think ZAR point is much easier to calculate. Count number of cards in your long suit, multiple by 2, count number of cards in second suit, add those together, subtract number of cards in your shortest suit, add your hcp, add you number of controls. Really, this is much simplier that even this simple description sounds. There are subtle sadjustments, like adding 1 point of each honor in trump suit (but never more than 1 pt), and adding points for superfits, and subtracting points for misfits.

Bum Rap + 5-3-1 is similiar, but to me, more compicated. Add 5 points for a void, 3 points for singleton and 1 point for doubleton to your hcp. But high card points here are 4.5 for Ace, 3 for king, 1.5 for Queen, 0.75 for a jack and 0.25 for a ten.

For me, there is no comparison. I don't want to deal with fractions of a point. Second, the ZAR method has clearly stated values suitable for game (52), slam (62) and grand slam (67). It is hard to tell what the comparable numbers are for BUM RAP + 531. I think one might assume 25, 31, and 34. But I guess it could be something like 25.25 for game, 30.75 for slam, and 34.75 for grand. Tysen hasn't told us yet.

Finally, as thread starter, let me give you my take on the issue. I was highly skeptical at first when I read Zar's theory's. But he did something nice, he shared his data set with us (you can check for yourself). I find that, given reality checks to make sure you are not off top winners in a suit, when you have a fit, ZAR point work very well indeed. When you have wild distribution but a total misfit, ZAR points fail miserably. Of course, that fail miserably is relative, because with horrible misfits, you need to be subtracting points like crazy.

I suggest you at least give a read to ZAR's theories, for I suspect people will be talking about them for a long, long time to come. So even if you don't believe them/like them, at least you will understand what is being discussed (remember the first time someone at a table said something about the rule of 20 and you had no clue what was being discussed...I hate it when something like that happens to me at the table). Read his article at this link never miss game - Zar points .

Then after reading it, check how well it would have worked for you when you have a disaster (or when you have a success) without using it. Maybe, you will change from skeptic to believer like I have.

Ben

[hrothgar]

Tysen:

Thanks for the explanation regarding why you advocate "swapping" the metric I proposed.

[I originally suggested collecting a bucket of hands that make precisely 10 tricks and then calculating the standard deviation of hand strength while you suggested taking a sample of hands that "should" take 10 tricks and then calculating the standard deviation of the expected number of tricks]

The reason that I suggested starting with a bucket of hands that make precisely 10 tricks is linked to a comment that Ben has just raised: Off the top of my head, I don't know what range of [BUM-RAP + 531] "points" corresponds to

(1) making 10 tricks in a suit contract
(2) making 11 tricks in a suit contract
(3) making 12 tricks in a suit contract
(4) making 13 tricks in a suit contact

Thanks very much for posting the "standard error" for each level of bidding.
Any chance that we can get the mean as well?

[mikestar]

Zar,

When I stated that Zar does best for grands and worst for part scores (I should have said "least good"), I was not intending to compare it to other methods which undoubtledly show the same characteristic. I apologize for not being clearer.

My real intention was to call attention to the bias of double dummy solvers. In point of fact Zar's at the table accuracy for part scores and slams may be more nearly the same than your figures indicate, and this will be true in varying degrees of other methods.

I accept that your figures show Zar superior to 1-3-5 at all levels, with increasing difference in accuracy as the level goes up. My understanding is that we can be fairly confident that Zar is more accurate, but not as certain how nuch more accurate--the double dummy bias is highest at the grand slam level and this will tend to magnify small differences in accuracy.

I hope that this is a clearer statement of my points.

[Zar]

*** mikestar wrote: "My understanding is that we can be fairly confident that Zar is more accurate, but not as certain how nuch more accurate--the double dummy bias is highest at the grand slam level and this will tend to magnify small differences in accuracy.
<

Well ... what is the alternative? At least DD is VERY close to the actual play ata the table, meaning that the 1.0 more tricks that it makes (due to the fact that it "sees" all cards) are compensated by the 1.1 trick the average defender gives away statistically (I believe I have mentioned this somewhere in this forum). Bias in terms of uneven "performance" across levels - yes, but the question again "what's the alternative"?

Anaysis of real-life events (see Ben's thread about the Cavendish) is clearly one way to compare and see ... but you'll never get an exact answer to the question "how much EXACTLY the method A is better than method B" I guess, in terms of making everybody happy :-)


My take is that you have to personally be comfortable with it, whatever your personal criteria is.

ZAR

[tysen2k]

inquiry, on May 20 2004, 07:43 AM, said:

For me, there is no comparison. I don't want to deal with fractions of a point.

I really don't want to use fractions of a point either. I can't handle those fractions at the table (although I do know some people who love them). I myself have been using straight HCP+531 and then just making a mental note if I have lots of aces or queens in my hand. It's not as accurate as either BUM+531 or Zar, but it's pretty close to Zar in terms of accuracy.

The thing that pushes me to use just HCP instead of the more accurate Zar is the fact that I'm familiar with the ranges of numbers. My system is written in terms of these familiar points and I don't want to have to convert my weak NT into 25-28 or whatever it turns out to be.

However, I've been playing around with alternatives in case you are comfortable with Zar's point ranges. Here's what I've found.

Zar is very accurate in terms of high card values. The ratio of 3-2-1-0.5 has been known for decades. It's the distribution scheme of Zar that loses the accuracy. So my alternative is to use Zar's 6/4/2/1 for high cards and one of these distribution schemes:

• 8/4/2 for void/singleton/doubleton or
  
• 1*longest + 1*next - 2*shortest

Both of these are practically the same in terms of accuracy so take your pick. They are both right between Zar and BUM+531 in terms of accuracy.

Note that for the first case you'll have to add a base of 8 points to get back to Zar's scale. In the second case you'll have to add 6 points. Alternatively you could subtract 16 or 12 points from your requirements for game, slam, etc. So if you pick the second option, you'll be using 40 for game, 50 for a slam and 55 for a grand. Hey, nice round numbers...

Tysen

[Zar]

*** tysen2k wroe: "It's the distribution scheme of Zar that loses the accuracy. So my alternative is to use Zar's 6/4/2/1 for high cards and one of these distribution schemes: 8/4/2 for void/singleton/doubleton
<

Some people don't read, only write :-)

I mentioned in a previous post that assigning points ONLY for short suit doesn't work, regardless of whether it is 3-2-1, 5-3-1, and even if you make it 10-5-1 it wouldn't work (because it makes 7-card suits look like 5-card, 6-card suits like 4- card etc.). Tysen offers now a new "perl" 8-4-2. Sorry I didn't mentioned this one :-)

So, for Tysen (and Tysen only :-):

8-4-2 DOESN'T work. And 16-8-4 wouldn't in case you are thinking of pushing it further - just forhet it :-)

ZAR

[tysen2k]

Zar, on May 20 2004, 04:11 PM, said:

I mentioned in a previous post that assigning points ONLY for short suit doesn't work, regardless of whether it is 3-2-1, 5-3-1, and even if you make it 10-5-1 it wouldn't work (because it makes 7-card suits look like 5-card, 6-card suits like 4- card etc.).

Ah, but it does seem to work. If you look at how the different distributions actually behave, you'll see this. Look at this segment of my previous data for hands with voids only:

           Real    Zar     5-3-1   8-4-2
5-4-4-0   1.519   1.200   1.667   1.600
6-4-3-0   1.624   1.600   1.667   1.600
5-5-3-0   1.643   1.400   1.667   1.600
7-3-3-0   1.697   1.800   1.667   1.600

(To refresh your memory this is the number of tricks you are better than a 4333 hand). These are vastly different shapes but they all practically take the same number of tricks. The fact that some have only 5&4 card suits and others 7 card suits doesn't seem to make much of a difference on their trick taking ability. Zar's predictions are all over the map but the ones that count shortness only are right on.

Every method that counts points in some way will have certain hand types with the same value. For example Zar assigns the same value to 6331 and 5440 even though they have vastly different trick taking abilities (and 5332 with 4441).

Tysen

[Zar]

*** tysen2k wrote: "The fact that some have only 5&4 card suits and others 7 card suits doesn't seem to make much of a difference on their trick taking ability.
<

OK man - we just run on different stadiums then ...

ZAR

[Zar]

Hi, guys:

I ran the worst-case scenarios for all the Standard GIB boards (part-score and 3 NT contracts that is).
As expected, as we get down to the part-score area, the methods get really close to each other.
Here are the results (and the way they have been calculated in parans):

Part score results ( All Standard GIB boards for 3H and 3S contracts)

================Overall Results ============================

The WTC ( 10 > number of tricks > 8) got 22057 contracts
GOREN 5-3-1 points( 26 > HCP+5-3-1> 22 ) got 29432 contracts
GOREN 3-2-1 points( 26 > HCP+3-2-1> 22 ) got 32262 contracts
Fit+3 Zar Points ( 52 > +3 extra trump > 45) got 30957 contracts
Basic Zar Points ( 52 > no fit points > 45 ) got 35090 contracts


The only area where Zar Points demonstrated slightly WORSE performance were actually the 3NT Standard GIB boards.
Here are the results (and the way they have been calculated in parans):

3NT results ( All Standard GIB boards for 3 NT contracts)

================Overall Results ============================

The WTC ( number of tricks > 8) got 292 contracts
GOREN 3-2-1 ( HCP+3-2-1> 25 ) got 1042 contracts
GOREN 5-3-1 ( HCP+5-3-1> 25 ) got 1127 contracts
Basic Zar Points ( no fit ) >51 got 952 contracts
Fit Zar Points (+3 extra trump)>51 got 968 contracts

Again close results, but this time Zar Points are slightly worse than the Goren counterparts - on average about 10% less.
And the performance of Fit and Basic is virtually the same, as should be expected.

The outperformance in the areas of Games, Slams, and Grands though runs in the order of times rather than percentages, as you can remember from the previous postings.

As usual, I'll post the results ZIPped on the website so you can look at every board separately if you have the time.


Make it a great day:

ZAR

[Zar]

Hi, guys:

Here are the offensive bidding numbers, AFTER partner HAS OPENED,

There were several parallel runs by different people and the numbers below are the direct results of the runs made by John Gallucci (thanx, John!).

The corresponding numbers for the DEFENSIVE bidding are published in the Zar Points defensive bidding thread.



======================== ZAR POINTS DISTRIBUTIONS

Total hands with 26 Zar Points or more = 468043 or 46.8043%
Total hands with 25 Zar Points or less = 531957 or 53.1957%


1,000,000 Total Hands



HCP 26 31 36 41 46 51 56 Total Percent
--- -- -- -- -- -- -- -- ----- -------
2 0 0 0 0 0 0 0 0 0
3 1 0 0 0 0 0 0 1 0
4 17 0 0 0 0 0 0 17 0
5 126 0 0 0 0 0 0 126 0
6 567 0 0 0 0 0 0 567 0.1
7 3224 2 0 0 0 0 0 3226 0.7
8 10042 43 0 0 0 0 0 10085 2.2
9 22368 265 0 0 0 0 0 22633 4.8
10 39225 1131 1 0 0 0 0 40357 8.6
11 53707 5721 43 0 0 0 0 59471 12.7
12 57037 11149 213 0 0 0 0 68399 14.6
13 45935 18435 594 1 0 0 0 64965 13.9
14 29820 24890 1682 6 0 0 0 56398 12.0
15 11336 27565 5204 58 0 0 0 44163 9.4
16 3649 21338 7683 173 0 0 0 32843 7.0
17 845 13418 8980 494 1 0 0 23738 5.1
18 76 6591 8559 946 7 0 0 16179 3.5
19 1 2251 6941 1282 9 0 0 10484 2.2
20 0 537 4297 1557 39 0 0 6430 1.4
21 0 71 2101 1481 81 1 0 3735 0.8
22 0 0 816 1252 117 0 0 2185 0.5
23 0 0 172 748 112 2 0 1034 0.2
24 0 0 34 370 141 2 0 547 0.1
25 0 0 5 176 102 2 0 285 0.1
26 0 0 0 36 71 4 0 111 0
27 0 0 0 7 26 3 0 36 0
28 0 0 0 1 12 8 0 21 0
29 0 0 0 0 3 3 0 6 0
30 0 0 0 0 0 0 0 0 0
-- -- -- -- -- -- -- -----

277976 133407 47325 8588 721 26 0 468043


59.4 28.5 10.1 1.8 0.2 0.0 0.0 % using base of 468043
27.8 13.3 4.7 0.9 0.1 0.0 0.0 % using base of 1000000


RAW COUNT ------------- Responder's Range -------------
Opener's Range 10- 11-15 16-20 21-25 26-30 31+
-------------- ----- ----- ----- ----- ----- -----
26 - 30 1221 18075 60273 81550 38047 20371
31 - 35 894 12476 36132 40612 15422 6174
36 - 40 458 5999 15069 14574 4355 1305
41 - 45 116 1573 3145 2375 583 120
46 - 50 21 174 297 160 31 3
51 - 55 0 13 10 2 1 0

PERCENTAGE'S ------------- Responder's Range -------------
Opener's Range 10- 11-15 16-20 21-25 26-30 31+
-------------- ----- ----- ----- ----- ----- -----
26 - 30 0.12 1.81 6.03 8.16 3.80 2.04
31 - 35 0.09 1.25 3.61 4.06 1.54 0.62
36 - 40 0.05 0.60 1.51 1.46 0.44 0.13
41 - 45 0.01 0.16 0.31 0.24 0.06 0.01
46 - 50 0.00 0.02 0.03 0.02 0.00 0.00
51 - 55 0.00 0.00 0.00 0.00 0.00 0.00
56 - 60 0.00 0.00 0.00 0.00 0.00 0.00



Table below shows the spread of probability of 8 to 57 Zar Points

0 1 2 3 4 5 6 7 8 9
--- --- --- --- --- --- --- --- --- ---
0 > 0 0 0 0 0 0 0 0 493 973
10 > 2376 4491 7642 10674 15948 19875 26738 32751 39695 45235
20 > 51738 55273 59379 60652 62067 35957 82255 56361 51201 46710
30 > 41449 35847 30814 26166 22115 18465 14934 11738 9171 6683
40 > 4799 3253 2237 1502 969 627 334 201 114 41
50 > 31 13 6 6 1 0 0 0 0 0
60 > 0 0 0 0 0 0 0 0 0 0




Table below shows the spread of probability PERCENTAGES of 8 to 57 Zar Points

0 1 2 3 4 5 6 7 8 9
--- --- --- --- --- --- --- --- --- ---
0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1
10 > 0.2 0.4 0.8 1.1 1.6 2.0 2.7 3.3 4.0 4.5
20 > 5.2 5.5 5.9 6.1 6.2 3.6 8.2 5.6 5.1 4.7
30 > 4.1 3.6 3.1 2.6 2.2 1.8 1.5 1.2 0.9 0.7
40 > 0.5 0.3 0.2 0.2 0.1 0.1 0.0 0.0 0.0 0.0
50 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
60 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0



Opener's Range Game 52+ Small Slam 62+ Grand Slam 67+
-------------- ---------- -------------- --------------
26 - 30 74825 7.48 7200 0.720 1369 0.137
31 - 35 66540 6.65 7624 0.762 2202 0.220
36 - 40 28791 2.88 6914 0.691 2094 0.209
41 - 45 3920 0.39 2807 0.281 1137 0.114
46 - 50 150 0.02 284 0.028 252 0.025
51 - 55 0 0.00 9 0.001 17 0.002
---------- ------------ ------------
174226 17.42 24838 2.48 7071 0.7 = 20.6 %




Compared to the case when OPPONENTS open, the numbers for Game + Slam + Grand were only 16.5% (see the research on the cases when we are in DEFENSIVE bidding).

Defensive bidding numbers are posted in the defensive bidding thread.

Cheers:

ZAR

[Zar]

Hm ...

The tables columns get screwdup ... Tell me if you can copy-paste them in a document and get the numbers lined-up.

I'll also try to post all these on the website when I get the time (hopefully this weekend).

Cheers:

ZAR

[inquiry]

Zar, on Jun 29 2004, 12:27 PM, said:

Hm ...

The tables columns get screwdup ... Tell me if you can copy-paste them in a document and get the numbers lined-up.

I'll also try to post all these on the website when I get the time (hopefully this weekend).

Cheers:

ZAR

Paste your tables inside "quotes". To do this type a press the "quote" button from above the enter your post field... just hit enter to get something like this

Quote

undefined

Undefined is the default text. Delete that word and paste you text with tables. In the normal window, all the spaces disappear, but inside quotes, spaces are retained. Alternatively, you can enter spaces that "Stay" by typing the following for each space...

Quote

   so to get three spaces, type      This means that 1♠  2♠  will look like 1♠ 2♠. but 1♠   2♠ will show the three spaces... see example below

Same text as in quote printed outside the quotes....

  so to get three spaces, type     This means that 1♠ 2♠ will look like 1♠ 2♠. but 1♠   2♠ will show the three spaces... see example below


Note the space at the beginning of the paragraph (two acutally) due to the code followed by a space in the example. Also the three spaces between the second 1S-2S, but only one between the first.

hope that helps.

Ben