Niacin is also known as Vitamin B3. Interestingly, it’s also called “nicotinic acid,” and the similarity of the name to “nicotine” isn’t coincidental:

Take a look at nicotine and you’ll see the structural similarity.
Interestingly, high doses of niacin have been noted to modulate cholesterol (hundreds of times the RDA). A few problems can occur, though – the most common is prostaglandin-mediated flushing of the face. The most serious are heart and liver problems.
Merck was going after a cholesterol drug that incorporated niacin, with a prostaglandin blocker to mitigate the side effects. Modulating prostaglandin metabolism isn’t without tox liability – this is what Vioxx and other COX-2 inhibitors were monkeying with. However, this strategy was essentially working later in the game – not inhibiting prostaglandin production, but blocking those that are produced.
Today they got their not approvable letter for this drug, so there were obviously some (unspecified) concerns.
The COX-2 agents caused everyone to take a second look at the class, to the point where we’re questioning even the OTC NSAIDs (except aspirin). A prostaglandin antagonist isn’t a COX inhibitor, but some people are gun-shy about the whole pathway now. Additionally, theres the liver tox liability – cholesterol biosynthesis occurs in the liver, so you’re altering liver function with these drugs. This is why you come back to the doctor for blood tests when you’re on statins (and you can’t just use an OTC cholesterol test). Since niacin is known to sometimes cause liver problems (although usually with “extended-release” niacinamide), that could be the issue.
[The author is not a physician and holds no Merck position].