In a binary min heap of n elements, the (n-1)th element (i.e. the next to largest) will either be on the last level of the tree, or next-to-last level of the heap. In a full heap, there are (n+1)/2 items on the last level, and (n+1)/4 items on the next-to-last level.
It's an O(n) operation because you potentially have to search (3*(n+1))/4 elements.
It's not O(1) because a heap is not necessarily sorted. You can't guarantee that the 99th smallest item in a heap of 100 items is at the 99th position in the array. Consider these two min heaps:
1 1
2 3 5 2
4 5 6 7 6 7 3 4
Both of those are valid min heaps, but in the second one, the second smallest item is at position 3 in the array.
Update
Actually, in a full binary heap, the next-to-last item has to be on the leaf level. In a not-full heap, it can be anywhere on the leaf level, or on the level, or on the level above. Consider, for example:
1
2 6
3 5 7
About the (3*(n+1)/2) question:
The next-to-smallest can be anywhere on the last level (4 nodes) or the next-to-last level (2 nodes). Total of 6 possible nodes. That works out to (7+1)/2 nodes for the last level, and (7+1)/4 for the next-to-last level. Simplifying, that's (3*(7+1))/4.