Incorrect Encouter data in: gold, silver, crystal, heartgold, soulsilver, (Rarity totals don't add up to 100 for certain sets of certain condition values)

[RyanVereque]

I've seen some other issues relating to bad encounter data, but they were limited in scope or closed. I've queried the DB for combinations of location_area_id, version_id, encounter_method_id, combination of encounter_condition_values and summed the rarities. Only 25 / 141,822 combos (excluding gift-like encouter_methods) did not add up to 100.

The 25 combos are presented below (id based and human-readable)

ID Based Version
(h_ecv_N means what is the encounter_condition value corresponding to encounter_condition = N in our hypothetical scenario. A value of null implies no such encounter_condition = N exists in encounter_condition_value_map for the location_area_id, version_id, encounter_method_id combo):

location_area_id	version_id	encounter_method_id	h_ecv_1	h_ecv_2	h_ecv_3	h_ecv_4	h_ecv_5	h_ecv_6	total_rarity
186	5	4	NULL	NULL	NULL	NULL	NULL	NULL	300
192	4	3	1	NULL	NULL	NULL	NULL	NULL	65
192	5	3	1	NULL	NULL	NULL	NULL	NULL	65
192	5	4	NULL	NULL	NULL	NULL	NULL	NULL	200
192	6	3	1	NULL	NULL	NULL	NULL	NULL	65
200	5	4	NULL	NULL	NULL	NULL	NULL	NULL	300
205	5	4	NULL	NULL	NULL	NULL	NULL	NULL	300
224	5	4	NULL	NULL	NULL	NULL	NULL	NULL	300
225	5	4	NULL	NULL	NULL	NULL	NULL	NULL	300
235	5	4	NULL	NULL	NULL	NULL	NULL	NULL	300
236	6	1	NULL	3	NULL	NULL	NULL	NULL	101
236	6	1	NULL	4	NULL	NULL	NULL	NULL	101
243	4	2	1	NULL	NULL	NULL	NULL	NULL	15
243	4	3	1	NULL	NULL	NULL	NULL	NULL	65
243	5	2	1	NULL	NULL	NULL	NULL	NULL	15
243	5	3	1	NULL	NULL	NULL	NULL	NULL	65
243	5	4	NULL	NULL	NULL	NULL	NULL	NULL	200
277	5	4	NULL	NULL	NULL	NULL	NULL	NULL	300
306	6	1	NULL	NULL	NULL	NULL	NULL	NULL	300
349	15	3	NULL	NULL	NULL	NULL	NULL	NULL	125
349	15	4	NULL	NULL	NULL	NULL	NULL	NULL	200
349	15	5	NULL	NULL	NULL	NULL	NULL	NULL	160
349	16	3	NULL	NULL	NULL	NULL	NULL	NULL	125
349	16	4	NULL	NULL	NULL	NULL	NULL	NULL	200
349	16	5	NULL	NULL	NULL	NULL	NULL	NULL	160

Human-readable version (I believe ordered the same):

location_name	location_area_name	identifier	encounter_method_identifier	encounter_condition_values_concat
Cherrygrove City	NULL	silver	super-rod	NULL
Route 32	NULL	gold	good-rod	During a swarm
Route 32	NULL	silver	good-rod	During a swarm
Route 32	NULL	silver	super-rod	NULL
Route 32	NULL	crystal	good-rod	During a swarm
Union Cave	B2F	silver	super-rod	NULL
Route 34	NULL	silver	super-rod	NULL
Olivine City	NULL	silver	super-rod	NULL
Sea Route 40	NULL	silver	super-rod	NULL
Cianwood City	NULL	silver	super-rod	NULL
Route 42	NULL	crystal	walk	In the morning
Route 42	NULL	crystal	walk	During the day
Route 44	NULL	gold	old-rod	During a swarm
Route 44	NULL	gold	good-rod	During a swarm
Route 44	NULL	silver	old-rod	During a swarm
Route 44	NULL	silver	good-rod	During a swarm
Route 44	NULL	silver	super-rod	NULL
Sea Route 19	NULL	silver	super-rod	NULL
Route 13	NULL	crystal	walk	NULL
Vermilion City	S.S. Anne dock	heartgold	good-rod	NULL
Vermilion City	S.S. Anne dock	heartgold	super-rod	NULL
Vermilion City	S.S. Anne dock	heartgold	surf	NULL
Vermilion City	S.S. Anne dock	soulsilver	good-rod	NULL
Vermilion City	S.S. Anne dock	soulsilver	super-rod	NULL
Vermilion City	S.S. Anne dock	soulsilver	surf	NULL

Lets check whats going on for the first record:

• Get all associated encounters and their rarity values in the first location_area_id, version_id, encounter_method_id combo

  • Query

    SELECT encounters.*, encounter_slots.rarity
    FROM (SELECT * FROM encounters WHERE location_area_id = 186 AND version_id = 5) as encounters
             INNER JOIN
             (SELECT * FROM encounter_slots WHERE encounter_method_id = 4) as encounter_slots
             ON encounters.encounter_slot_id = encounter_slots.id
    ;

  • Result

    id	version_id	location_area_id	encounter_slot_id	pokemon_id	min_level	max_level	rarity
    55117	5	186	529	98	40	40	40
    55118	5	186	530	222	40	40	30
    55119	5	186	531	98	40	40	20
    55120	5	186	532	99	40	40	10
    55121	5	186	529	98	40	40	40
    55122	5	186	530	222	40	40	30
    55123	5	186	531	98	40	40	20
    55124	5	186	532	99	40	40	10
    55125	5	186	529	98	40	40	40
    55126	5	186	530	120	40	40	30
    55127	5	186	531	98	40	40	20
    55128	5	186	532	99	40	40	10

  • Notice that the rarity sum of these encounters is > 100 ? But this is irrelevant unless ALL of these encounters can be fulfilled at once by a given set of values for encounter_conditions.

• Now lets simply join with encounter_condition_value_map to see if there are any conditions on these encounters.

  • Query

    SELECT encounters.*, encounter_slots.rarity
    FROM (SELECT * FROM encounters WHERE location_area_id = 186 AND version_id = 5) as encounters
             INNER JOIN
             (SELECT * FROM encounter_slots WHERE encounter_method_id = 4) as encounter_slots
             ON encounters.encounter_slot_id = encounter_slots.id
             INNER JOIN encounter_condition_value_map
                        ON encounter_condition_value_map.encounter_id = encounters.id
    ;

  • Result

    Nothing!

  • This means that previous set of encounters are satisfied by any combo of encounter_condition_values, and yet
    their rarity sum is > 100. Problem detected.

• Furthermore if we check Bulbapedia - Cherrygrove_City we can that there are encounter conditions for Cherrygrove City in Gen II (silver) using the Super Rod. I suppose some updates to encounter_condition_value_map are needed.

If interested in the wall of queries and temp tables I used to automatically detect these inconsistencies, let me know.

[magical]

Nice catch! Given that there are other problems (#313) with the GSC encounters, i wonder if they should be reverted until a proper rip can be done.

I'm not sure what's going on with the S.S. Anne Dock encounters in HG/SS.

[RyanVereque]

For fun I've checked the last 2 of the 25 records.

The encounters of last one would sum to 100 if not for the inclusion of encounter 26620 which is a no-condition 60% wingull (perhaps this was meant to have the 'during a swarm' condition attached as Bulbapedia mentions that that is the one encounter for swarm).

The second to last seems confusing. Among its encounters there are several Magikarp encounters that sum to 100 all on their own. The other encounters are of the right mons, but their rarities don't match with Bulbapedia.

In fact location_area_id = 349 (S.S. Anne dock) seems to have no encounters with any attached conditions (no swarm at all).

[magical]

Interesting. Maybe the conditions got accidentally dropped somehow? I was also wondering if maybe encounter data from two different areas accidentally got combined.

[RyanVereque]

Could be. I'm also thinking about being skeptical towards Bulbapedia rarities for this location. I don't see how rarities of 3% or 7% can exist when encounter_slots for the HG/SS version group encounter method Super Rod can only be [5,10,15,30,40] which can never sum to 3 or 7. Maybe in the second to last example, your rarities are good (sum to 100) with the exception of the magikarps (additional 100) somehow ? I also just checked the third to last and there aren't any clues besides re-use of the same encounter_slot_id (which is true for the other examples aswell).

[RyanVereque]

I made a different kind of lookup where instead of looking for bad rarity sums, it looks for duplicated sets of :

(location_area_id,version_id,encounter_method_id,h_ecv_1,h_ecv_2,h_ecv_3,h_ecv_4,h_ecv_5,h_ecv_6,encounter_slot_id)

From this query and a quick visual comparison, they all seem to correspond to some within the previous 25 bad combos, or rather a subset actually, but nothing new. Some have counts of 2 or even 3. I cannot guarantee that removing them or adding conditions to them or rearranging them to different (location_area_id,version_id,encounter_method_id) within those 25 would fix the sums, but one would hope since two kinds of issues seem to have the same set of offending tuples involved.

[magical]

Looks like the HG/SS S.S. Anne Port encounters were added in a separate commit from the rest of the HG/SS encounters: 373ec78. That commit didn't add any encounter conditions, so i guess they've been missing since the beginning.

[magical]

Looking at the ROM, i don't actually see any separate encounter data for the S.S. Anne Dock. I could be missing something, but is it possible that the Dock encounters were simply copied over from the main Vermilion City encounters?

[magical]

On top of that, it's not called the S.S. Anne in HG/SS, it's called the S.S. Aqua, and from watching a walkthrough video it doesn't look like there's any way to surf or fish on the dock (the sides are roped off). I think these encounters are bogus.