I was getting error “shmget: [EEXIST][17]: key 52b14804: shared memory already exists” in online log and I notice OS physical memory and kernel parameters (shmmax) have much more available memory. But Informix instance unable to allocate new shared memory virtual segment during run-time from OS kernel. I notice there is no AF file (Assert Failed) generated.
I have raised a case to IBM Informix and confirm it was encountering IDS bug with product defect number idsdb00206627.
Applies to:
since IDS 11.50.FC6W4 until IDS 11.50.FC7
Bug Description:
idsdb00206627: SERVER HANGS/CRASHES WHEN PERFORMING CONTINUOUS ATTACH/DETACH FROM SHARED MEMORY SEGMENTS
We create a new segment of requested size in mt_shm_create_segment() and issue mt_notifyvp(MT_NOTIFYVP_NEWSHMSEG, attach). We’re still in mt_shm_create_segment() when a call to mt_notifyvp(newseg, attach) fails because one of the VPs didn’t succeed in attaching, so we immediately issue mt_notifyvp(newseg, detach). However, in mt_notifyvp() we can break out of the loop in case of an error, failing to collect remaining acknowledgments for mt_notifyvp(attach), so while some of the VPs are still working on attaching to the new segment, some other VPs have started detaching from the same segment. With a previous fix for idsdb00198802 we make an OS call shmctl(shmid, IPC_STAT, addr), which basically reads a shmid_ds struct from the shared memory, identified by shmid and writes this struct into provided address. This is a way to verify the address given is valid within the address space of a VP. Since attach/detach suppose to be executed consecutively without stepping on each other’s toes, this normally wouldn’t be a problem. However, due to a mentioned problem in mt_notifyvp() those segments still working on attachseg() start failing all over the place once we write junk to the address we’re trying to verify for detach (ideally, we should have given the address of the first segment in segment_t->se_shmid[] to shmctl() so we deal with the same struct, but that still doesn’t solve the mt_notifyvp() problem). So we end up having a total mess – attaches/detaches are running in parallel overwriting same addresses they both use, we ignore some acknowledgments due to errors in scope of some notifies and expect other notifies to process them causing hangs/crashes, etc.
Solution:
Apply patch IDS 11.50.FC7W1 (release on Jul 2010)