By default, the RadGridView control will automatically put in scrollbars for the horizontal and vertical scrolling of it's rows if there is not enough room to display the content. You are right that the ScrollViewer simply lets the RadGridView determine it's own size unconstrained, and therefore it will fill up the width to accomodate all of it's columns, and the height to accomodate all of the rows.
You want the RadGridView to be constrained by it's container, so you'll want to put it into a grid cell (unless it is supposed to take up the entire view). You shouldn't have any Width or Height set on the RadGridView, because you want them set as Auto (the default) to grow or shrink appropriately. It sounds like something else might be getting in the way of that, so I suggest you post some code.
Here is my RadGridView, and it fills up the container and scrolls like you want. Note that I have overridden the row definition and am using a custom usercontrol, which shouldn't affect the scrolling at all.
<telerik:RadGridView ItemsSource="{Binding Shipments}" RowStyle="{StaticResource rowStyle}"
RowDetailsVisibilityMode="Collapsed"
RowIndicatorVisibility="Collapsed"
CanUserDeleteRows="False"
CanUserInsertRows="False"
CanUserSelect="False" telerik:StyleManager.Theme="Windows7" />
Here is my customized rowtemplate (in my local resourcedictionary):
<ControlTemplate x:Key="MyCustomRowTemplate" TargetType="telerik:GridViewRow">
<Border x:Name="rowsContainer" BorderBrush="#FFA0AFC3" BorderThickness="0,0,0,1" >
<Grid Width="Auto" HorizontalAlignment="Stretch">
<Grid.RowDefinitions>
<RowDefinition Height="Auto"/>
</Grid.RowDefinitions>
<Grid.ColumnDefinitions>
<ColumnDefinition Width="*" />
</Grid.ColumnDefinitions>
<view:ActiveReleaseItemView DataContext="{Binding}" />
</Grid>
</Border>
</ControlTemplate>
<Style x:Key="rowStyle" TargetType="telerik:GridViewRow">
<Setter Property="Template" Value="{StaticResource MyCustomRowTemplate}" />
</Style>